Repository navigation
feat(sdk): build bounded, typed store reads for ctx.db - #386
Merged
Merged
Conversation
Adds the reading half of `ctx.db` (spec 9.2): `get(table, id, options?)` and `query(table)` with `withIndex`, `filter`, `order` and the bounded terminals `take`, `first`, `unique`, `count`, `collect`, `paginate` and `aggregate`, typed by the App's schema. A query is an immutable value that only produces a descriptor (`@grasp-os/shared/store-queries`), handed to an executor the trusted host supplies; nothing is read until a terminal is called, and no descriptor carries SQL, physical IDs, owners or roles. Also adds `paginationOptionsValidator` and `paginationResultValidator`, whose two strict page shapes keep the cursor and `hasMore` consistent.
…filters two-valued From the review of the first draft: - The wire schema checks a filter's depth, size and `in` lists iteratively before any recursive reading, so a deep or wide filter is refused without overflowing the stack or the clock; ranges are checked for an index range's shape (equalities first, then a lower and an upper bound on one more field, at most 17 bounds). - Conditions are two-valued, as completed on GRA-243: comparisons and `neq` match values only, `eq: null` and `in` with null match an explicit null only, `exists: false` matches a field left out only, and `NOT` is plain negation. Aggregates take declared numeric fields only. - Chained filters sit side by side under one `and`; `in` lists are copied; a range belongs to the index it was built for; a query is ordered once, in types and at run time; empty conditions, empty `AND` and `OR`, nulls on fields that can't hold one and malformed options are refused as the wire would refuse them. - The descriptor carries its version, and its shapes and structural checks live in a module without Zod (`@grasp-os/shared/store-reads`), so App bundles don't carry Zod.
…ts bounds A query's range and filter together bind at most 64 values as SQL parameters (an `in` counts once, as the host binds its list as one JSON value; null and `exists` bind nothing), which with the owner and a cursor's seek keeps a store's statement under its SQLite's limit of 100 parameters. The wire checks a range's length before reading its bounds. `neq: null` is documented as any value, and `get` uses the shared identifier limit, now in a module without Zod.
nickruigrok
marked this pull request as ready for review
October 11, 2026 02:16
Contributor
Author
|
@greptileai review |
|
…et filters while built From the review of #386: - A range belongs to the index definition of the reader's own schema, not to table and index names, so a range from another reader over a schema with the same names is refused. - A query's state is frozen plain data, and each read hands its executor a copy of its own, so an executor that changes what it was given changes no later read. - A filter's nodes count against the 64-node budget as they are built, with the query's earlier filters and a list's length counted before any member is built, so a huge list is refused at once.
Contributor
Author
|
@greptileai review |
nickruigrok
added a commit
that referenced
this pull request
Oct 11, 2026
…QLite indexes (#388) Second of five stacked PRs for GRA-243. **Stacked on #386**; this PR's own changes are the last three commits. ## What - **One field-expression emitter** (`apps/core/src/store-expressions.ts`), shared by index DDL, ranges, filters, ORDER BY and cursor seeks. - Managed fields map to their fixed columns. - A declared field maps to `CASE json_type(...) WHEN 'null' THEN -9e999 ELSE json_extract(...) END`. A field left out (SQL NULL), an explicit null (negative infinity, which no JSON number can be) and a value stay distinct, ordered missing < null < value (decision D3). - Field names must be identifiers and appear only inside JSON-path literals. Table IDs must be host-minted UUIDs. - **The index planner** (`apps/core/src/store-indexes.ts`). - Each declared index becomes `CREATE INDEX sdk_ix_<sha256(definition)[:32]> ON sdk_records ([owner_id,] <expressions>, record_id) WHERE table_id = '<table ID>'`. - The definition is the emitter version, the table, the owner prefix (when the table is read by owner only) and the fields. - A changed definition is a new index; nothing is dropped, so pinned consumers keep theirs. - Installing is idempotent, compares against `sqlite_master`, and refuses an index whose name SQLite holds with another definition. - **Per-store cap: 256 physical indexes, retained ones included** (`storeMaxIndexes`). The refusal (`data.too_many_indexes`) comes before any index is created. - Why: every write evaluates every partial index's WHERE and opens each index, so cost follows the store's total. - Measured on SQLite 3.51 with 32 indexes on the written table plus N others: 25 µs per insert at N = 32, 90 µs at 224, 356 µs at 352, 784 µs at 480. - **`defineTables`** installs the indexes and records the manifest by hash in the new `sdk_schemas` table, in one transaction (decision D2). Activation stays GRA-246's. - **Fixed default-order indexes** `(table_id, created_at, record_id)` and `(table_id, owner_id, created_at, record_id)` in the host layout (migration `0003`), so 64 tables don't cost 64 partial indexes. - **Purge fix:** the deleted-store purge now also clears `sdk_schemas`. Its "anything left?" check moved from a UNION to EXISTS terms, because the DO's SQLite caps a compound SELECT at five terms; adding the sixth term failed in tests. - **Two-valued conditions and value-only ranges**, as completed for D3 on GRA-243. `filterSql` and `rangeSql` in the emitter write every filter condition and range bound as SQL that is 0 or 1, never NULL, so `NOT` keeps records whose field is null or left out: - comparisons and `neq` match values only; - `eq`/`in` with null match an explicit null only; - `exists` tests for a field left out; - an upper bound alone is bounded below by the field's base type, so `lt` excludes null and a field left out; - null is refused on a field that can't hold it; - `in` binds its list as one JSON value read with `json_each`, so its SQL stays flat (an OR chain of 99 values overran SQLite's expression depth), with null as a separate sentinel term; - a number or boolean bounded only below is also bounded above by text (`< ''`), which sorts above every number. ## Tests `apps/core/test/store-indexes.test.ts`, run in workerd against the store's real SQLite: - the exact installed DDL, and the owner prefix; - idempotence, including a reordered declaration; - a renamed index reuses its physical index, and a changed one is added beside it; - a tampered definition is refused; - the cap (288 requested): refused, with nothing created and no schema recorded; - `EXPLAIN QUERY PLAN` for equality, ranges and order on a declared index uses `sdk_ix_…` with no TEMP B-TREE and no table scan; - the default-order indexes are used; - missing < null < value through the index, with null equality and value-only ranges. `apps/core/test/store-expressions.test.ts` evaluates the emitter in the store's SQLite over notes whose field holds a value, another value, null or nothing: - every condition; - `NOT` over each condition (null and left out included); - nested logic; - range bounds (`lt`/`lte` exclude null and a field left out); - type refusals; - `in` over numbers (including 2 vs 2.5 vs text "2"), strings and booleans, and a 100-value `in` binding one parameter; - lower bounds never reaching text. `store-indexes.test.ts` also runs the largest descriptor the boundary accepts (depth 8, the longest `in`, a full 16-field range and the owner: 65 parameters) in the store's SQLite. Its plan tests now read the emitter's own `rangeSql`/`filterSql` output, null equality and one-sided bounds included, and a cap test counts indexes retained for an older schema. Results: `vp check` is clean. `vp test` on `store-indexes`, `data-stores`, `data-store-lifecycle` and `data-store-receipts` passes 107 tests.
nickruigrok
added a commit
that referenced
this pull request
Oct 11, 2026
…schema's access (#391) Third of five stacked PRs for GRA-243. **Security**: needs human review before merge. **Stacked on #386 and #388**; this PR's own changes are the last three commits. ## Threat model (written first; also in `apps/core/src/store-queries.ts`) Who reads: App code, through core, which resolves the caller and hands the store a `ReadScope`. The descriptor is whatever App code built and is trusted for nothing. | Threat | How it's closed | | --- | --- | | A caller reads records it may not see, through any terminal | The read mode comes from the store's own record of the pinned schema (`sdk_schemas`), never from the request. `none` refuses. `appMembers`/`appBuilders` refuse callers core didn't find to have the role (`data.access_denied`). `owner` puts `owner_id = principal` in the one statement every terminal runs, so counts, aggregates, `unique` and results cover the caller's records only. The owner leads the index, so other owners' records aren't even scanned. Nothing is filtered after the fact. | | A record ID discloses another table's or owner's record | `get` looks a record up by table and ID under the same predicate and answers null in every case. `expectedRevision` is checked only on a visible record. | | A descriptor smuggles SQL or names what it shouldn't | Names resolve through the pinned manifest. Table IDs and index names come from the store's catalog and planner. Field names reach SQL only through the shared emitter. Values are bound. | | A query scans without bound | `INDEXED BY` the named index, so SQLite fails rather than scans. The range is checked against the index's fields in order. At most 10,000 entries and 16 MiB of documents per terminal, then `data.scan_limit`, never a partial answer. `collect` fails past 100 (`data.result_limit`); `unique` past one (`data.not_unique`). | | A descriptor overruns SQLite | Filter depth, nodes and lists, plus 64 bound values, are checked at the boundary (#386). The emitter's SQL is flat (#388). | | A condition drops a record through SQL NULL | Conditions are two-valued (#388), so `NOT` keeps null and missing. | | A pinned consumer sees fields it doesn't know, or misses defaults | Documents are projected to the pinned schema at every depth: objects keep only declared keys, with defaults; records, arrays and unions project what they hold (spec 32.7). | | An aggregate overflows | A non-finite result fails with `data.aggregate_overflow`. | | A read creates state, or a missing index makes it scan | A read refuses a store with no row (`data.unknown_schema`) without creating one. An index SQLite doesn't hold is `data.index_required`. | | A caller claims a schema or a role | Both come from core (`ReadScope`). An unknown schema hash is `data.unknown_schema`. | ## What this PR trusts, and who answers for it - **`ReadScope` is taken as given.** That covers the principal, roles, schema hash and consumer. Only core holds the store binding, and core must build the scope per invocation from its own records. This is a hard requirement on GRA-244. - **An older schema hash keeps its own access mode.** If a table goes from `appMembers` to `owner` in a newer schema, a read under the older hash still lets members read it. The security review probed this: v1 `appMembers`, v2 `owner`, and a read under v1's hash returns another owner's records. It is closed by two hard requirements: GRA-244 takes the hash from the consumer's own pinned manifest, and GRA-246 retires superseded hashes once nothing is pinned to them. No code change here. ## What - `StoreHost.read(storeId, scope, request)`, exposed as `DataStore.read` over RPC and `OpenStore.read` from core, with `readScopeSchema` in `@grasp-os/shared/store-queries`. - `apps/core/src/store-queries.ts`: - access resolution; - range-to-index checking; - `planQuery`, one statement with the filter as a computed column, ordered by the index columns past the equality prefix and limited to one entry past the scan bound; - a bounded match iterator; - every terminal except `paginate` (PR 4); - projection. - The store caches each pinned schema's manifest and index plan per object; neither changes for a hash. - New errors: `data.unknown_schema`, `data.unknown_index`, `data.access_denied`, `data.scan_limit`, `data.result_limit`, `data.not_unique`. - The store doesn't apply defaults on write; that's GRA-244's operation boundary. - Aggregates take declared numeric fields only, never managed ones (D3). ## Boundary GRA-244 builds `ctx.db` with `databaseReader(schema, (request) => store.read(scope, request))` and resolves the scope's role flags and `operationAccess` narrowing. PR 5 adds the host-side read set and `readConsistently`. ## Tests `apps/core/test/store-queries.test.ts`, run in workerd through `OpenStore` and the DataStore DO, with the SDK reader building the descriptors: - **Owner-only tables:** every terminal (collect, count, aggregates, `unique` with another owner's match, filters, `get`) answers from the caller's records only; `expectedRevision` conflicts only on visible records. - **Index plans:** real `EXPLAIN QUERY PLAN` of the store's own statement (via `planQuery`) uses `sdk_ix_…` with the owner and range terms, and the owner default-order index, with no TEMP B-TREE. - **Roles:** member and builder modes admit and refuse by role; `none` refuses everyone, by query and by ID. - **Names:** unknown schema, table, another schema's table, unknown index or field, out-of-order ranges, a range without an index, and managed aggregate fields are all refused. - **Injection:** quotes in values match literally; injection in a field name and raw SQL keys are refused. - **D3 through the store:** `lt` ranges exclude null and missing, the index orders missing < null < value, `NOT` keeps missing, `neq` is value-only. - **Bounds:** `collect` > 100 and `unique` > 1 are refused, `take(100)` and `take(0)` work, and 10,001 entries trip `data.scan_limit` for count and for a filtered `first` while an early stop and an index range still answer; 140 × 120 KB documents trip the 16 MiB bound while an unfiltered count reads none. - **Projection:** records are projected to an older pinned schema (a newer field omitted) and a newer one (defaults filled in). Undeclared nested keys in objects, arrays of objects and records of objects are left out. - **Since the security review:** - another owner's 10,005 records cause neither `scan_limit` nor `result_limit` for Ada; - a `get` with another table's ID answers null; - exactly 10,000 entries are answered; - min/max over nothing are null and sum over nothing is 0; - 1e308 + 1e308 gives `data.aggregate_overflow`; - a sum on a declared string field is refused; - a read creates no store row; - a dropped index gives `data.index_required`; - plan tests now explain the host's own statement (`StoreHost.explain`); - a union is projected to the member the stored value validates against, not a smaller earlier one; the first member whose projection validates is used only when none does, and validators are cached per descriptor (tested: second member, first member, and a later key); - aggregates read JSON numbers only (`numberExpression`), never booleans, and a missing index is logged as `store.index_missing`. Results: `vp check` is clean. `vp test` on store-queries, store-expressions, store-indexes, the data-store suites, the SDK database and schema suites and shared store-queries passes 200 tests. After the review fixes, the store and SDK suites pass 154 tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First of five stacked PRs for GRA-243 (bounded typed queries over business stores).
What
The reading half of
ctx.db(spec 9.2), as SDK types and a pure builder, exported on the new subpath@grasp-os/sdk/database(databaseReader,DatabaseReader,Doc,QueryDefinitionError):get(table, id, { expectedRevision? })andquery(table), typed by the App's schema (Doc,Id, index names and their field tuples).withIndex(name, range?)(initializer only),filter(repeatable; chained filters sit side by side under oneand) andorder(once, enforced in types and at runtime).take(0..100),first,unique,count,collect,paginate({ pageSize, cursor })(cursor at most 4,096 characters) andaggregate(sum/min/maxover declared numeric fields only).eq,neq,gt,gte,lt,lte,in,exists,AND,OR,NOT), turned into one tree.inlists are copied. Empty conditions and emptyAND/ORare refused, and so is null on fields that can't hold it.paginationOptionsValidatorandpaginationResultValidatoron the SDK root. The two strict page shapes keep the cursor andhasMoreconsistent.A query is an immutable value. Nothing is read until a terminal is called, and the builder only produces a versioned descriptor (
descriptorVersion: 1) for an executor the trusted host supplies. A descriptor never carries SQL, physical IDs, owners or roles.The descriptor's shapes and structural checks are in
@grasp-os/shared/store-reads, which has no Zod, so App bundles don't carry it. The host's Zod schema is@grasp-os/shared/store-queries(queryRequestSchema,getRequestSchema,readRequestSchema, with the TS types tied bysatisfies).inlists (100) iteratively, before any recursive parse, so deep or wide input is refused without overflowing the stack or the clock.queryMaxBoundValues; anincounts once, and null andexistsbind nothing). That keeps a store's statement, with the owner and a cursor's seek, under the Durable Object SQLite limit of 100 parameters.indexMaxFields).Semantics (D3 as completed on GRA-243)
Conditions are two-valued:
neqmatch values only, andneq: nullmatches any value;eq: nullandincontaining null match an explicit null only;exists: falsematches a field left out only;NOTis plain negation, so it keeps records whose field is left out;These are documented on
FilterNode,RangeBound,FieldConditionandIndexRangeBuilder, with examples. The host emitter is in #388.Boundaries
ctx.dbfromdatabaseReader(schema, execute)and the host-side reader in PR 5.Tests
packages/sdk/test/database.test.ts(workerd): descriptors are built at the terminal, are immutable, and pass the host schema; 12 chained filters stay flat;inis copied; every refusal happens before anything is read (unknown table or index, range order or shape, a range from another index, null on non-nullable fields, empty conditions, ordering twice including order → filter → order, out-of-bounds terminals and cursors, count with a field, managed aggregate fields, malformedgetoptions); the page contract.packages/shared/test/store-queries.test.ts: unknown keys and versions, non-identifier names, malformed ranges, oversized and empty filter parts, a 100,000-deep filter refused without throwing, a 262,144-wide filter refused in under a second, out-of-bounds terminals, the read request union, a 1,000,000-bound range refused by length, and the 64-value budget.packages/sdk/test/database.test-d.ts: type fixtures reject wrong tables, IDs from another table, wrong indexes and range order, bounds after an upper bound, null bounds, wrong field types, non-scalar filter fields, predicates,take(101),pageSize: 500,alland iteration, ordering twice, and managed or non-numeric aggregate fields.vp checkis clean;vp teston these suites plus the schema and manifest suites passes 145 tests (the ids and audit-log suites included).