Skip to content

feat(sdk): build bounded, typed store reads for ctx.db - #386

Merged
nickruigrok merged 4 commits into
mainfrom
feat/store-query-builder
Oct 11, 2026
Merged

nickruigrok merged 4 commits into
mainfrom
feat/store-query-builder

Conversation

@nickruigrok

@nickruigrok nickruigrok commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

First of five stacked PRs for GRA-243 (bounded typed queries over business stores).

What

The reading half of ctx.db (spec 9.2), as SDK types and a pure builder, exported on the new subpath @grasp-os/sdk/database (databaseReader, DatabaseReader, Doc, QueryDefinitionError):

  • Reads: get(table, id, { expectedRevision? }) and query(table), typed by the App's schema (Doc, Id, index names and their field tuples).
  • Modifiers: withIndex(name, range?) (initializer only), filter (repeatable; chained filters sit side by side under one and) and order (once, enforced in types and at runtime).
  • Terminals: take(0..100), first, unique, count, collect, paginate({ pageSize, cursor }) (cursor at most 4,096 characters) and aggregate (sum/min/max over declared numeric fields only).
  • Index ranges: equality on an index prefix, then a lower and/or upper bound on the next field. Checked by the types and at runtime; a range belongs to the index it was built for.
  • Filters: Baseflare's structured grammar (eq, neq, gt, gte, lt, lte, in, exists, AND, OR, NOT), turned into one tree. in lists are copied. Empty conditions and empty AND/OR are refused, and so is null on fields that can't hold it.
  • Page contract: paginationOptionsValidator and paginationResultValidator on the SDK root. The two strict page shapes keep the cursor and hasMore consistent.

A query is an immutable value. Nothing is read until a terminal is called, and the builder only produces a versioned descriptor (descriptorVersion: 1) for an executor the trusted host supplies. A descriptor never carries SQL, physical IDs, owners or roles.

The descriptor's shapes and structural checks are in @grasp-os/shared/store-reads, which has no Zod, so App bundles don't carry it. The host's Zod schema is @grasp-os/shared/store-queries (queryRequestSchema, getRequestSchema, readRequestSchema, with the TS types tied by satisfies).

  • It checks a filter's depth (8), nodes (64) and in lists (100) iteratively, before any recursive parse, so deep or wide input is refused without overflowing the stack or the clock.
  • It caps the values a query binds at 64 across range and filter (queryMaxBoundValues; an in counts once, and null and exists bind nothing). That keeps a store's statement, with the owner and a cursor's seek, under the Durable Object SQLite limit of 100 parameters.
  • It checks a range's length before reading its bounds, then its shape: equalities first, each on a field of its own, then at most a lower and an upper bound on one more field, nothing after an upper bound, and at most 17 bounds (from the shared indexMaxFields).

Semantics (D3 as completed on GRA-243)

Conditions are two-valued:

  • comparisons and neq match values only, and neq: null matches any value;
  • eq: null and in containing null match an explicit null only;
  • exists: false matches a field left out only;
  • NOT is plain negation, so it keeps records whose field is left out;
  • range bounds match values only.

These are documented on FilterNode, RangeBound, FieldCondition and IndexRangeBuilder, with examples. The host emitter is in #388.

Boundaries

  • GRA-244 builds ctx.db from databaseReader(schema, execute) and the host-side reader in PR 5.
  • The store executes descriptors in PR 3.

Tests

  • packages/sdk/test/database.test.ts (workerd): descriptors are built at the terminal, are immutable, and pass the host schema; 12 chained filters stay flat; in is copied; every refusal happens before anything is read (unknown table or index, range order or shape, a range from another index, null on non-nullable fields, empty conditions, ordering twice including order → filter → order, out-of-bounds terminals and cursors, count with a field, managed aggregate fields, malformed get options); the page contract.
  • packages/shared/test/store-queries.test.ts: unknown keys and versions, non-identifier names, malformed ranges, oversized and empty filter parts, a 100,000-deep filter refused without throwing, a 262,144-wide filter refused in under a second, out-of-bounds terminals, the read request union, a 1,000,000-bound range refused by length, and the 64-value budget.
  • packages/sdk/test/database.test-d.ts: type fixtures reject wrong tables, IDs from another table, wrong indexes and range order, bounds after an upper bound, null bounds, wrong field types, non-scalar filter fields, predicates, take(101), pageSize: 500, all and iteration, ordering twice, and managed or non-numeric aggregate fields.
  • Results: vp check is clean; vp test on these suites plus the schema and manifest suites passes 145 tests (the ids and audit-log suites included).

Adds the reading half of `ctx.db` (spec 9.2): `get(table, id, options?)`
and `query(table)` with `withIndex`, `filter`, `order` and the bounded
terminals `take`, `first`, `unique`, `count`, `collect`, `paginate` and
`aggregate`, typed by the App's schema. A query is an immutable value
that only produces a descriptor (`@grasp-os/shared/store-queries`),
handed to an executor the trusted host supplies; nothing is read until a
terminal is called, and no descriptor carries SQL, physical IDs, owners
or roles.

Also adds `paginationOptionsValidator` and `paginationResultValidator`,
whose two strict page shapes keep the cursor and `hasMore` consistent.
…filters two-valued

From the review of the first draft:
- The wire schema checks a filter's depth, size and `in` lists
  iteratively before any recursive reading, so a deep or wide filter is
  refused without overflowing the stack or the clock; ranges are checked
  for an index range's shape (equalities first, then a lower and an upper
  bound on one more field, at most 17 bounds).
- Conditions are two-valued, as completed on GRA-243: comparisons and
  `neq` match values only, `eq: null` and `in` with null match an
  explicit null only, `exists: false` matches a field left out only, and
  `NOT` is plain negation. Aggregates take declared numeric fields only.
- Chained filters sit side by side under one `and`; `in` lists are
  copied; a range belongs to the index it was built for; a query is
  ordered once, in types and at run time; empty conditions, empty `AND`
  and `OR`, nulls on fields that can't hold one and malformed options are
  refused as the wire would refuse them.
- The descriptor carries its version, and its shapes and structural
  checks live in a module without Zod (`@grasp-os/shared/store-reads`),
  so App bundles don't carry Zod.
…ts bounds

A query's range and filter together bind at most 64 values as SQL
parameters (an `in` counts once, as the host binds its list as one JSON
value; null and `exists` bind nothing), which with the owner and a
cursor's seek keeps a store's statement under its SQLite's limit of 100
parameters. The wire checks a range's length before reading its bounds.
`neq: null` is documented as any value, and `get` uses the shared
identifier limit, now in a module without Zod.
@nickruigrok
nickruigrok marked this pull request as ready for review October 11, 2026 02:16
@nickruigrok

Copy link
Copy Markdown
Contributor Author

@greptileai review

@greptile-apps

greptile-apps Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[High impact] The PR appears safe to merge; no new blocking issue was found.

Summary

Adds bounded, schema-typed reads through @grasp-os/sdk/database, shared request checks, and page validators.

  • Typed database reads send bounded requests to the host.
  • The host checks bounded, versioned read descriptors.
  • Page requests and results share one strict contract.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[App code] --> B[Build immutable query]
  B --> C[Call bounded terminal]
  C --> D[Copy descriptor]
  D --> E[Host-supplied executor]
  E --> F[Host checks request and access]
Loading

Reviews (2) · Last reviewed commit: "fix(sdk): bind ranges to their reader's ..." · Reviewed by Greptile

Comment thread packages/sdk/src/database.ts
Comment thread packages/sdk/src/database.ts
Comment thread packages/sdk/src/database.ts
…et filters while built

From the review of #386:
- A range belongs to the index definition of the reader's own schema,
  not to table and index names, so a range from another reader over a
  schema with the same names is refused.
- A query's state is frozen plain data, and each read hands its executor
  a copy of its own, so an executor that changes what it was given
  changes no later read.
- A filter's nodes count against the 64-node budget as they are built,
  with the query's earlier filters and a list's length counted before
  any member is built, so a huge list is refused at once.
@nickruigrok

Copy link
Copy Markdown
Contributor Author

@greptileai review

@nickruigrok
nickruigrok merged commit 2a1dfbf into main Oct 11, 2026
9 checks passed
@nickruigrok
nickruigrok deleted the feat/store-query-builder branch October 11, 2026 02:34
nickruigrok added a commit that referenced this pull request Oct 11, 2026
…QLite indexes (#388)

Second of five stacked PRs for GRA-243. **Stacked on #386**; this PR's
own changes are the last three commits.

## What
- **One field-expression emitter**
(`apps/core/src/store-expressions.ts`), shared by index DDL, ranges,
filters, ORDER BY and cursor seeks.
  - Managed fields map to their fixed columns.
- A declared field maps to `CASE json_type(...) WHEN 'null' THEN -9e999
ELSE json_extract(...) END`. A field left out (SQL NULL), an explicit
null (negative infinity, which no JSON number can be) and a value stay
distinct, ordered missing < null < value (decision D3).
- Field names must be identifiers and appear only inside JSON-path
literals. Table IDs must be host-minted UUIDs.
- **The index planner** (`apps/core/src/store-indexes.ts`).
- Each declared index becomes `CREATE INDEX
sdk_ix_<sha256(definition)[:32]> ON sdk_records ([owner_id,]
<expressions>, record_id) WHERE table_id = '<table ID>'`.
- The definition is the emitter version, the table, the owner prefix
(when the table is read by owner only) and the fields.
- A changed definition is a new index; nothing is dropped, so pinned
consumers keep theirs.
- Installing is idempotent, compares against `sqlite_master`, and
refuses an index whose name SQLite holds with another definition.
- **Per-store cap: 256 physical indexes, retained ones included**
(`storeMaxIndexes`). The refusal (`data.too_many_indexes`) comes before
any index is created.
- Why: every write evaluates every partial index's WHERE and opens each
index, so cost follows the store's total.
- Measured on SQLite 3.51 with 32 indexes on the written table plus N
others: 25 µs per insert at N = 32, 90 µs at 224, 356 µs at 352, 784 µs
at 480.
- **`defineTables`** installs the indexes and records the manifest by
hash in the new `sdk_schemas` table, in one transaction (decision D2).
Activation stays GRA-246's.
- **Fixed default-order indexes** `(table_id, created_at, record_id)`
and `(table_id, owner_id, created_at, record_id)` in the host layout
(migration `0003`), so 64 tables don't cost 64 partial indexes.
- **Purge fix:** the deleted-store purge now also clears `sdk_schemas`.
Its "anything left?" check moved from a UNION to EXISTS terms, because
the DO's SQLite caps a compound SELECT at five terms; adding the sixth
term failed in tests.

- **Two-valued conditions and value-only ranges**, as completed for D3
on GRA-243. `filterSql` and `rangeSql` in the emitter write every filter
condition and range bound as SQL that is 0 or 1, never NULL, so `NOT`
keeps records whose field is null or left out:
  - comparisons and `neq` match values only;
  - `eq`/`in` with null match an explicit null only;
  - `exists` tests for a field left out;
- an upper bound alone is bounded below by the field's base type, so
`lt` excludes null and a field left out;
  - null is refused on a field that can't hold it;
- `in` binds its list as one JSON value read with `json_each`, so its
SQL stays flat (an OR chain of 99 values overran SQLite's expression
depth), with null as a separate sentinel term;
- a number or boolean bounded only below is also bounded above by text
(`< ''`), which sorts above every number.

## Tests
`apps/core/test/store-indexes.test.ts`, run in workerd against the
store's real SQLite:
- the exact installed DDL, and the owner prefix;
- idempotence, including a reordered declaration;
- a renamed index reuses its physical index, and a changed one is added
beside it;
- a tampered definition is refused;
- the cap (288 requested): refused, with nothing created and no schema
recorded;
- `EXPLAIN QUERY PLAN` for equality, ranges and order on a declared
index uses `sdk_ix_…` with no TEMP B-TREE and no table scan;
- the default-order indexes are used;
- missing < null < value through the index, with null equality and
value-only ranges.

`apps/core/test/store-expressions.test.ts` evaluates the emitter in the
store's SQLite over notes whose field holds a value, another value, null
or nothing:
- every condition;
- `NOT` over each condition (null and left out included);
- nested logic;
- range bounds (`lt`/`lte` exclude null and a field left out);
- type refusals;
- `in` over numbers (including 2 vs 2.5 vs text "2"), strings and
booleans, and a 100-value `in` binding one parameter;
- lower bounds never reaching text.

`store-indexes.test.ts` also runs the largest descriptor the boundary
accepts (depth 8, the longest `in`, a full 16-field range and the owner:
65 parameters) in the store's SQLite. Its plan tests now read the
emitter's own `rangeSql`/`filterSql` output, null equality and one-sided
bounds included, and a cap test counts indexes retained for an older
schema.

Results: `vp check` is clean. `vp test` on `store-indexes`,
`data-stores`, `data-store-lifecycle` and `data-store-receipts` passes
107 tests.
nickruigrok added a commit that referenced this pull request Oct 11, 2026
…schema's access (#391)

Third of five stacked PRs for GRA-243. **Security**: needs human review
before merge. **Stacked on #386 and #388**; this PR's own changes are
the last three commits.

## Threat model (written first; also in
`apps/core/src/store-queries.ts`)
Who reads: App code, through core, which resolves the caller and hands
the store a `ReadScope`. The descriptor is whatever App code built and
is trusted for nothing.

| Threat | How it's closed |
| --- | --- |
| A caller reads records it may not see, through any terminal | The read
mode comes from the store's own record of the pinned schema
(`sdk_schemas`), never from the request. `none` refuses.
`appMembers`/`appBuilders` refuse callers core didn't find to have the
role (`data.access_denied`). `owner` puts `owner_id = principal` in the
one statement every terminal runs, so counts, aggregates, `unique` and
results cover the caller's records only. The owner leads the index, so
other owners' records aren't even scanned. Nothing is filtered after the
fact. |
| A record ID discloses another table's or owner's record | `get` looks
a record up by table and ID under the same predicate and answers null in
every case. `expectedRevision` is checked only on a visible record. |
| A descriptor smuggles SQL or names what it shouldn't | Names resolve
through the pinned manifest. Table IDs and index names come from the
store's catalog and planner. Field names reach SQL only through the
shared emitter. Values are bound. |
| A query scans without bound | `INDEXED BY` the named index, so SQLite
fails rather than scans. The range is checked against the index's fields
in order. At most 10,000 entries and 16 MiB of documents per terminal,
then `data.scan_limit`, never a partial answer. `collect` fails past 100
(`data.result_limit`); `unique` past one (`data.not_unique`). |
| A descriptor overruns SQLite | Filter depth, nodes and lists, plus 64
bound values, are checked at the boundary (#386). The emitter's SQL is
flat (#388). |
| A condition drops a record through SQL NULL | Conditions are
two-valued (#388), so `NOT` keeps null and missing. |
| A pinned consumer sees fields it doesn't know, or misses defaults |
Documents are projected to the pinned schema at every depth: objects
keep only declared keys, with defaults; records, arrays and unions
project what they hold (spec 32.7). |
| An aggregate overflows | A non-finite result fails with
`data.aggregate_overflow`. |
| A read creates state, or a missing index makes it scan | A read
refuses a store with no row (`data.unknown_schema`) without creating
one. An index SQLite doesn't hold is `data.index_required`. |
| A caller claims a schema or a role | Both come from core
(`ReadScope`). An unknown schema hash is `data.unknown_schema`. |

## What this PR trusts, and who answers for it
- **`ReadScope` is taken as given.** That covers the principal, roles,
schema hash and consumer. Only core holds the store binding, and core
must build the scope per invocation from its own records. This is a hard
requirement on GRA-244.
- **An older schema hash keeps its own access mode.** If a table goes
from `appMembers` to `owner` in a newer schema, a read under the older
hash still lets members read it. The security review probed this: v1
`appMembers`, v2 `owner`, and a read under v1's hash returns another
owner's records. It is closed by two hard requirements: GRA-244 takes
the hash from the consumer's own pinned manifest, and GRA-246 retires
superseded hashes once nothing is pinned to them. No code change here.

## What
- `StoreHost.read(storeId, scope, request)`, exposed as `DataStore.read`
over RPC and `OpenStore.read` from core, with `readScopeSchema` in
`@grasp-os/shared/store-queries`.
- `apps/core/src/store-queries.ts`:
  - access resolution;
  - range-to-index checking;
- `planQuery`, one statement with the filter as a computed column,
ordered by the index columns past the equality prefix and limited to one
entry past the scan bound;
  - a bounded match iterator;
  - every terminal except `paginate` (PR 4);
  - projection.
- The store caches each pinned schema's manifest and index plan per
object; neither changes for a hash.
- New errors: `data.unknown_schema`, `data.unknown_index`,
`data.access_denied`, `data.scan_limit`, `data.result_limit`,
`data.not_unique`.
- The store doesn't apply defaults on write; that's GRA-244's operation
boundary.
- Aggregates take declared numeric fields only, never managed ones (D3).

## Boundary
GRA-244 builds `ctx.db` with `databaseReader(schema, (request) =>
store.read(scope, request))` and resolves the scope's role flags and
`operationAccess` narrowing. PR 5 adds the host-side read set and
`readConsistently`.

## Tests
`apps/core/test/store-queries.test.ts`, run in workerd through
`OpenStore` and the DataStore DO, with the SDK reader building the
descriptors:
- **Owner-only tables:** every terminal (collect, count, aggregates,
`unique` with another owner's match, filters, `get`) answers from the
caller's records only; `expectedRevision` conflicts only on visible
records.
- **Index plans:** real `EXPLAIN QUERY PLAN` of the store's own
statement (via `planQuery`) uses `sdk_ix_…` with the owner and range
terms, and the owner default-order index, with no TEMP B-TREE.
- **Roles:** member and builder modes admit and refuse by role; `none`
refuses everyone, by query and by ID.
- **Names:** unknown schema, table, another schema's table, unknown
index or field, out-of-order ranges, a range without an index, and
managed aggregate fields are all refused.
- **Injection:** quotes in values match literally; injection in a field
name and raw SQL keys are refused.
- **D3 through the store:** `lt` ranges exclude null and missing, the
index orders missing < null < value, `NOT` keeps missing, `neq` is
value-only.
- **Bounds:** `collect` > 100 and `unique` > 1 are refused, `take(100)`
and `take(0)` work, and 10,001 entries trip `data.scan_limit` for count
and for a filtered `first` while an early stop and an index range still
answer; 140 × 120 KB documents trip the 16 MiB bound while an unfiltered
count reads none.
- **Projection:** records are projected to an older pinned schema (a
newer field omitted) and a newer one (defaults filled in). Undeclared
nested keys in objects, arrays of objects and records of objects are
left out.
- **Since the security review:**
- another owner's 10,005 records cause neither `scan_limit` nor
`result_limit` for Ada;
  - a `get` with another table's ID answers null;
  - exactly 10,000 entries are answered;
  - min/max over nothing are null and sum over nothing is 0;
  - 1e308 + 1e308 gives `data.aggregate_overflow`;
  - a sum on a declared string field is refused;
  - a read creates no store row;
  - a dropped index gives `data.index_required`;
- plan tests now explain the host's own statement (`StoreHost.explain`);
- a union is projected to the member the stored value validates against,
not a smaller earlier one; the first member whose projection validates
is used only when none does, and validators are cached per descriptor
(tested: second member, first member, and a later key);
- aggregates read JSON numbers only (`numberExpression`), never
booleans, and a missing index is logged as `store.index_missing`.

Results: `vp check` is clean. `vp test` on store-queries,
store-expressions, store-indexes, the data-store suites, the SDK
database and schema suites and shared store-queries passes 200 tests.
After the review fixes, the store and SDK suites pass 154 tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant