feat(core): install a store's declared indexes as definition-hashed SQLite indexes

[nickruigrok]

Second of five stacked PRs for GRA-243. Stacked on #386; this PR's own changes are the last three commits.

What

• One field-expression emitter (apps/core/src/store-expressions.ts), shared by index DDL, ranges, filters, ORDER BY and cursor seeks.

  • Managed fields map to their fixed columns.
  • A declared field maps to CASE json_type(...) WHEN 'null' THEN -9e999 ELSE json_extract(...) END. A field left out (SQL NULL), an explicit null (negative infinity, which no JSON number can be) and a value stay distinct, ordered missing < null < value (decision D3).
  • Field names must be identifiers and appear only inside JSON-path literals. Table IDs must be host-minted UUIDs.

• The index planner (apps/core/src/store-indexes.ts).

  • Each declared index becomes CREATE INDEX sdk_ix_<sha256(definition)[:32]> ON sdk_records ([owner_id,] <expressions>, record_id) WHERE table_id = '<table ID>'.
  • The definition is the emitter version, the table, the owner prefix (when the table is read by owner only) and the fields.
  • A changed definition is a new index; nothing is dropped, so pinned consumers keep theirs.
  • Installing is idempotent, compares against sqlite_master, and refuses an index whose name SQLite holds with another definition.

• Per-store cap: 256 physical indexes, retained ones included (storeMaxIndexes). The refusal (data.too_many_indexes) comes before any index is created.

  • Why: every write evaluates every partial index's WHERE and opens each index, so cost follows the store's total.
  • Measured on SQLite 3.51 with 32 indexes on the written table plus N others: 25 µs per insert at N = 32, 90 µs at 224, 356 µs at 352, 784 µs at 480.

• defineTables installs the indexes and records the manifest by hash in the new sdk_schemas table, in one transaction (decision D2). Activation stays GRA-246's.

• Fixed default-order indexes (table_id, created_at, record_id) and (table_id, owner_id, created_at, record_id) in the host layout (migration 0003), so 64 tables don't cost 64 partial indexes.

• Purge fix: the deleted-store purge now also clears sdk_schemas. Its "anything left?" check moved from a UNION to EXISTS terms, because the DO's SQLite caps a compound SELECT at five terms; adding the sixth term failed in tests.

• Two-valued conditions and value-only ranges, as completed for D3 on GRA-243. filterSql and rangeSql in the emitter write every filter condition and range bound as SQL that is 0 or 1, never NULL, so NOT keeps records whose field is null or left out:

  • comparisons and neq match values only;
  • eq/in with null match an explicit null only;
  • exists tests for a field left out;
  • an upper bound alone is bounded below by the field's base type, so lt excludes null and a field left out;
  • null is refused on a field that can't hold it;
  • in binds its list as one JSON value read with json_each, so its SQL stays flat (an OR chain of 99 values overran SQLite's expression depth), with null as a separate sentinel term;
  • a number or boolean bounded only below is also bounded above by text (< ''), which sorts above every number.

Tests

apps/core/test/store-indexes.test.ts, run in workerd against the store's real SQLite:

• the exact installed DDL, and the owner prefix;
• idempotence, including a reordered declaration;
• a renamed index reuses its physical index, and a changed one is added beside it;
• a tampered definition is refused;
• the cap (288 requested): refused, with nothing created and no schema recorded;
• EXPLAIN QUERY PLAN for equality, ranges and order on a declared index uses sdk_ix_… with no TEMP B-TREE and no table scan;
• the default-order indexes are used;
• missing < null < value through the index, with null equality and value-only ranges.

apps/core/test/store-expressions.test.ts evaluates the emitter in the store's SQLite over notes whose field holds a value, another value, null or nothing:

• every condition;
• NOT over each condition (null and left out included);
• nested logic;
• range bounds (lt/lte exclude null and a field left out);
• type refusals;
• in over numbers (including 2 vs 2.5 vs text "2"), strings and booleans, and a 100-value in binding one parameter;
• lower bounds never reaching text.

store-indexes.test.ts also runs the largest descriptor the boundary accepts (depth 8, the longest in, a full 16-field range and the owner: 65 parameters) in the store's SQLite. Its plan tests now read the emitter's own rangeSql/filterSql output, null equality and one-sided bounds included, and a cap test counts indexes retained for an older schema.

Results: vp check is clean. vp test on store-indexes, data-stores, data-store-lifecycle and data-store-receipts passes 107 tests.

[nickruigrok]

@greptileai review

[greptile-apps]

Confidence Score: 5/5

[Critical impact] The PR appears safe to merge; the latest change fixes the rejected-schema table leak.

Summary

This PR installs definition-hashed SQLite indexes and records each schema alongside them. It also adds shared field expressions, an index cap, and fixed indexes for default ordering.

• A store installs declared indexes when an app defines its schema.
• Declared-field filters and ranges keep missing fields and nulls distinct.
• Default record lists use two shared order indexes.
• Deleted-store cleanup checks each table without a long UNION.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Read current tables] --> B[Choose table IDs and hash index definitions]
  B --> C[Start transaction and check store]
  C --> D{Tables still match?}
  D -- No --> E[Rebuild plan, up to three attempts]
  E --> A
  D -- Yes --> F[Insert tables and install indexes]
  F --> G[Record schema and commit]
  F -- Refused --> H[Roll back transaction]

Loading

_{Reviews (2) · Last reviewed commit: "fix(core): define a store's tables, inde..." · Reviewed by Greptile}

[nickruigrok]

@greptileai review