Repository navigation
Commit 8a44ce7
Fixes #20041
Clause-②: yes (narrowing)
## What changed
On the SQLite faces `$like` / `$ilike` compile to `GLOB`, and SQLite
reads a pattern only up to its first U+0000. A pattern holding U+0000
was cut there and answered a different question, with nothing raised.
There is no NUL-safe SQLite pattern primitive to compile to instead
(`LIKE` cuts the same way, `replace()` cannot target U+0000, `instr()`
has no wildcards; measured on #20024). So the accept set becomes one
contract: such a pattern is refused, the way a pattern ending in a lone
unpaired backslash already is.
- **`packages/spec`:** one exported predicate,
`hasNulInLikePattern(pattern)`, beside `hasDanglingLikeEscape` in
`src/data/filter.zod.ts`, with its own docblock. `hasDanglingLikeEscape`
is byte-identical. The converters (`likePatternToRegexSource`,
`likePatternToGlobPattern`, `matchesLikePattern`) are unchanged (see
H3).
- **Every door that refused a dangling escape now asks the new predicate
right after it**, in that door's existing envelope, `INVALID_FILTER` /
400:
| door | file | envelope |
|:--|:--|:--|
| `SqlDriver`'s filter walk (`classifyFilterKey`), before a dialect is
chosen | `driver-sql/src/sql-driver.ts` | new builder
`nulLikePatternError`, born in the withheld seam (`withheldFilterError`,
the operator map as the node, as for the dangling escape) |
| `RemoteTransport.buildWhereSQL`'s `$like` / `$ilike` arm, before
anything is sent | `driver-turso/src/remote-transport.ts` | new method
`nulLikePattern`, through `withheldRefusal`; its withheld sentence is
`driver-sql`'s behind the `[RemoteTransport]` prefix |
| `driver-memory`'s shape gate (`assertFieldConstraintShape`: the query
path and the reference matcher `match()`) |
`driver-memory/src/filter-refusal.ts` | new exported
`nulLikePatternError`, `driver-sql`'s author text word for word |
| `driver-memory`'s QueryAST `comparison` `like` / `ilike` arm, and the
`$like` translator floor | `driver-memory/src/memory-driver.ts` | the
same builder |
- The new check runs AFTER the dangling-escape check at every door, so a
pattern with both keeps the refusal it already had.
- **Messages.** The withheld class statement names neither the operator
variant, the field, the path nor the pattern. The author text (and the
server log) reads, for example: `Operator "$like" on field "v" at
filter.v.$like has a pattern holding the NUL character U+0000
("%\u0000"). ...` — the pattern goes through `JSON.stringify`, so U+0000
reaches no message as a raw byte.
- **Bounded in-place fix, declared:** the `$like` operator's
`.describe()` (`LIKE_DESCRIPTION`, same file) said "A pattern ending in
a lone unpaired backslash is refused (INVALID_FILTER)". It now reads "A
pattern ending in a lone unpaired backslash, or holding the NUL
character U+0000, is refused (INVALID_FILTER)", so the declared contract
names the refusal this PR enforces.
`content/docs/references/data/filter.mdx` is its regeneration
(`gen:schema && gen:docs`): 5 rows, that phrase only.
- **A pending release note corrected, which needs your confirmation:**
the last bullet of `.changeset/20024-sqlite-glob-stored-nul.md` (not yet
released) said "a pattern holding U+0000 is still cut there". At this
head that sentence is false, so it now says the pattern is refused by
every driver that answers `$like`. `node
scripts/check-empty-changeset.mjs` is RED on this by design: its
DELIBERATE CORRECTION class says "do NOT restore it -- say so on the PR
and get it confirmed". This is that statement. The alternative is to
drop the edit and leave that false sentence in the release's changelog
beside this PR's own entry.
## H1: every face, base vs head (measured, the PM's hypothesis holds on
the SQLite faces)
A probe (scratch, not committed) drove the public `find` of each face
over one row set: 13 stored values, 12 non-NULL, U+0000 at the start,
middle and end, alone, and none (`'a'` + U+0000 + `'b'`, `'ab'` +
U+0000, U+0000 + `'z'`, U+0000 alone, `'A'` + U+0000 + `'B'`, `plain`,
`''`, `ab`, `a`, `b`, `axb`, `AB`, NULL). Cases: 10 patterns holding
U+0000 (`'%'`+NUL, NUL+`'%'`, `'%'`+NUL+`'%'`, `'a'`+NUL+`'b'`,
`'a'`+NUL+`'%'`, `'_'`+NUL+`'_'`, backslash+NUL, NUL alone, `$ilike`
`'%'`+NUL+`'B'` and `'AB'`+NUL) and 9 NUL-free controls, each bare and
under `$not`. Base is `8d76c2d38c`, head is this branch (dists rebuilt
at each).
| face | U+0000 pattern, base | U+0000 pattern, head | NUL-free control,
base = head |
|:--|:--|:--|:--|
| `SqlDriver` on better-sqlite3 | 20 of 20 differ from `formula`, 0
refused | 20 of 20 refused, `INVALID_FILTER` / 400 | 0 of 18 differ on
the NUL-free rows; 12 of 18 differ over rows holding a stored U+0000
(#20024 item 2 (ii), not this card) |
| `SqliteWasmDriver` | 20 of 20 differ, 0 refused | 20 of 20 refused |
same as above |
| `TursoDriver` local | 20 of 20 differ, 0 refused | 20 of 20 refused |
same as above |
| `TursoDriver` remote, `makeLibsqlSqliteStub` | 20 of 20 differ, 0
refused | 20 of 20 refused | same as above |
| `TursoDriver` remote, real `@libsql/client` `file::memory:` | 20 of 20
differ, 0 refused | 20 of 20 refused | same as above |
| `InMemoryDriver.find` (`$`-spelling) | 0 of 20 differ (answers
correctly), 0 refused | 20 of 20 refused | 0 of 18 differ |
| `InMemoryDriver.find`, QueryAST `comparison` `like` / `ilike` |
answers (`like '%'`+NUL gives the one value ending in U+0000) | refused
| `like 'ab%'` answers the same rows |
| `driver-memory` `match()` | answers (`true` for `'ab'`+NUL) | refused
| pinned equal to the query path |
| `@objectstack/formula` | the oracle | unchanged (see H2) | the oracle
|
| `driver-mongodb` `translateFilter` | 38 of 38 cases refused: `$like`
is not translated at all | unchanged, not touched (held by draft PR
#19947) | refused |
- The five SQLite faces gave byte-identical answer lists on every case,
at base and at head. The control answers are byte-identical base vs head
on all six driver faces.
- Examples at base, SQLite faces: `$like: '%'` + U+0000 returned all 12
non-NULL rows, where `formula` returns the two ending in U+0000; `$like:
'a'` + U+0000 + `'b'` also returned `'a'`; `$like: '_'` + U+0000 + `'_'`
also returned `'a'` and `'b'`; `$ilike: 'AB'` + U+0000 also returned
`'AB'` and `'ab'`; `$not $like '%'` + U+0000 returned only the NULL row.
- Also measured at base, not touched: objectql `applyHaving` refuses
every `$like` (`INVALID_FILTER` / 400, "Unsupported operator '$like' in
`having`"); service-analytics `compileScopedFilterToSql` refuses it
(`READ_SCOPE_COMPILE_FAILED` / 500, fail-closed) and
`normalizeAnalyticsFilterTree` refuses it (`INVALID_FILTER` / 400).
`lowerAnalyticsWhere` alone passes the node through; the tree build
after it refuses.
- The Postgres and MySQL arms of `driver-sql` were not measured live (no
server here). At head the refusal fires on the walk before the dialect
is chosen, pinned by compiling with the `pg` and `mysql2` clients and no
server.
## H2: the doors (census)
`git grep -n -E '\bNAME\(' HEAD -- 'packages/**/*.ts' ':!**/*.test.ts'`,
comments and the definition excluded:
- `hasDanglingLikeEscape`: 7 call sites at base and at head. Five are
face doors: `filter-refusal.ts` (1), `memory-driver.ts` (2),
`sql-driver.ts` (1), `remote-transport.ts` (1). Two are the converters'
own backstops in `filter.zod.ts`. Positive control: the PM's list
(`sql-driver.ts`, `remote-transport.ts`, `memory-driver.ts` x2,
`filter-refusal.ts`) is exactly the five doors.
- `hasNulInLikePattern`: 0 at base, 5 at head, one beside each of the
five doors.
- Every one of the five doors refused a dangling escape at base, and
every one must refuse U+0000. `formula` does NOT refuse a dangling
escape: `matchesLikePattern` throws, and the arm answers `false`
(measured: `matchesFilterCondition({ v: 'abc\\' }, { v: { $like: 'abc\\'
} })` is `false`). So by the claim's condition its file is not touched,
and it still evaluates a U+0000 pattern.
## H3: where the predicate is called, option (a)
Converter consumer census, same grep over non-test sources:
- `likePatternToGlobPattern`: 3 calls: `sql-driver.ts`
(`likePatternPredicate`), `remote-transport.ts` (`pushLikePattern`), and
`driver-memory/src/memory-analytics.ts` (`globSubstringPattern`, the
analytics echo of a `$contains` comparand).
- `likePatternToRegexSource`: 3 calls: `memory-driver.ts` (2) and the
spec's own `matchesLikePattern`.
- `matchesLikePattern`: 2 calls: `driver-memory/src/memory-matcher.ts`
and `formula/src/matches-filter.ts`.
- 8 calls in all: `packages/drivers/**` 6, `packages/formula` 1,
`packages/spec` 1. `packages/services/**`, `packages/objectql` and
`packages/plugins/**`: 0 (the same grep, whose drivers count is the
positive control).
Option (b), a throw inside the converters, would have changed three
consumers beyond this surface: `memory-analytics`' `$contains` echo
would throw a plain `Error` on a comparand holding U+0000; `formula`'s
`$like` would answer `false` (its caught throw) instead of the right
rows; the reference matcher the same (though its shape gate runs first).
So the predicate is called at each door (a), and
`filter-like-nul-pattern.test.ts` pins that the JS translation keeps its
meaning.
## H4: the withheld seam
- `nulLikePatternError` has a row in
`sql-driver-compile-refusal-seam.test.ts` and `nulLikePattern` in
`remote-transport-compile-refusal-seam.test.ts`; without the row, "every
builder that goes through the seam is driven by a row below" is red. The
remote pin's local-vs-remote table gained the class too (the withheld
sentence is one sentence on both compilers).
- policy-marked: `INVALID_FILTER` / 400, the error's own keys exactly
`code,status`, no field or pattern on the wire, both in the log / sink.
`'author'`: the full text. Unmarked: byte-identical to policy. Merged
`$and`: the refusing arm's mark decides, in both arm orders.
- Through `TursoDriver` in remote mode no mark survives
`toRemoteFilter`, so an author gets the class statement there (the
#20093 fail-closed cost, pinned in
`turso-20041-like-nul-pattern.test.ts`).
## H5: declaration
- `Clause-②: yes (narrowing)`: the PR adds one public export,
`hasNulInLikePattern` on `@objectstack/spec/data`
(`api-surface/data.json` +1), so the value is `yes` (AGENTS.md's
Clause-② rule, the PR #20104 precedent), and the arm is `(narrowing)`
for the refused patterns. The claim carried `no (narrowing)`; corrected
in the patch round after contract review 5828387721. The changeset
grades `@objectstack/spec`, `driver-sql`, `driver-sqlite-wasm`,
`driver-turso` and `driver-memory` `minor`, with **BREAKING** and a `!`
title, following PR #19971's changeset.
- `node scripts/check-changeset-no-major.mjs --base 8d76c2d`: "✓ This
diff introduces no `major` bump." (exit 0)
- `node scripts/check-adr-0087-registration.mjs --base 8d76c2d`: "✓
check-adr-0087-registration: 1 declared-breaking changeset(s), each
carrying an ADR-0087 disposition." — `[BREAKING+bang+clause-②-narrowing]
not-required (no-migration-prescription)` (exit 0). The gate chose
`not-required`: no key, schema, object definition or stored
representation moves, and no rewrite of a stored pattern keeps its
meaning.
## Compile surfaces (`references/compile-surfaces.md`, re-verified with
its grep: 92 hits in non-test sources)
| # | face | this PR |
|:--|:--|:--|
| 1 | `driver-sql` `applyFilterCondition` (`sql-driver.ts:15953`);
`driver-sqlite-wasm` and Turso local by inheritance | CHANGED: the walk
refuses a U+0000 pattern on every dialect |
| 2 | Turso `RemoteTransport.buildWhereSQL` (`remote-transport.ts:2668`)
| CHANGED: the `$like` / `$ilike` arm refuses it |
| 3 | service-analytics `compileScopedFilterToSql`
(`read-scope-sql.ts:602`) | not touched (claim excludes it): refuses
every `$like` already, measured |
| 4 | service-analytics `lowerAnalyticsWhere`
(`filter-normalizer.ts:2015`) | not touched: passes the node through;
`normalizeAnalyticsFilterTree` refuses every `$like`, measured |
| 5 | `formula` `matchesFilterCondition` (`matches-filter.ts:212`) | not
touched: it refuses no dangling escape, so it is not one of the doors;
it evaluates a U+0000 pattern correctly (the oracle above) |
| half | objectql `applyHaving` / `matchesHaving`
(`having-filter.ts:279` / `:292`) | not touched: refuses every `$like`,
measured |
| thawed | `driver-memory` `checkCondition` (`memory-matcher.ts:361`)
and the query path | CHANGED: the shape gate both run first refuses it;
the QueryAST arm and the translator floor too |
| thawed | `driver-mongodb` `translateFieldOperators`
(`mongodb-filter.ts:832`) | not touched (draft PR #19947 holds it):
refuses every `$like` through its `default:` arm, measured through
`translateFilter` |
## Tests
New pins, each asserting `code`, `status` and the path (never a bare
`toThrow()`), plus NUL-free controls and the dangling escape beside
U+0000:
- `packages/spec/src/data/filter-like-nul-pattern.test.ts` (5): the
predicate, the escaped U+0000, its independence from the dangling
escape, and the converters unchanged.
- `driver-sql/src/sql-driver-20041-like-nul-pattern.test.ts` (84): 9
patterns x bare / `$not` / `$or` / `$and`, unmarked (class only, path in
the log, through `find` and `count`) and author-marked (operator, field,
path); the dangling escape keeps its refusal also beside U+0000; 8
controls equal `formula`; `sqlite`, `pg` and `mysql2` compiles refuse on
the walk.
- `driver-sqlite-wasm/src/sqlite-wasm-20041-like-nul-pattern.test.ts`
(16), the same through sql.js, controls checked against `formula`.
- `driver-turso/src/turso-20041-like-nul-pattern.test.ts` (17): local,
remote over the stub and remote over a real libSQL engine; no `FROM`
statement reaches the engine for a refused filter; 8 controls, equal on
all three transports.
- `driver-memory/src/memory-20041-like-nul-pattern.test.ts` (21): the
query path, the QueryAST spelling and `match()`, with controls.
- Consumer pins changed: the two seam enumerations above (one row each;
the remote one also one local-vs-remote row). No other consumer pin
reads a U+0000 `$like`: `git grep` over every `*.test.ts` that names
`$like` / `$ilike` and U+0000 finds none outside this PR.
Suites (`vitest run --maxWorkers=2`), at `5f2e6f246d` (code equal to
this head but for a comment in `memory-driver.ts`), dists rebuilt:
- `driver-sql`: 186 files passed, 11 skipped; 3024 tests passed, 170
skipped.
- `driver-turso`: 69 files, 1629 tests passed.
- `driver-sqlite-wasm`: 33 files, 622 tests passed.
- `driver-memory`: 53 files, 1269 tests passed (re-run at `049b3a6c95`,
same).
- `formula`: 35 files, 978 tests passed.
- `spec`: 570 files, 16369 tests passed, 2 todo.
- `plugin-auth` (consumer): 114 files, 2440 tests passed.
- `typecheck` exits 0 in `spec`, `driver-sql`, `driver-sqlite-wasm`,
`driver-turso`, `driver-memory`. `tsc --listFiles` counts each new
driver test once in its package program; the spec test once in
`tsconfig.test.json`, which `check:test-typecheck` compiles.
## Before-red and ablations (one-off, no file left behind)
All from committed state, through `scripts/ablation-replace.mjs` (anchor
1 -> 0 on disk, blob changed), restored with the blob equal to `HEAD`,
`git diff HEAD` empty and `git status --porcelain` empty.
1. **Every driver's predicate replaced by a never-true local** (the four
source files at once, which is base behaviour at every door).
`driver-sql` rebuilt and `ablation-dist-preflight` found the marker in
`dist/` before the run. Predicted and observed exactly:
- `sql-driver-20041`: 75 failed, 9 passed (the controls, the dangling
escape, and nothing else green);
- `sql-driver-compile-refusal-seam`: 4 failed, 91 passed (the new row);
- `sqlite-wasm-20041`: 10 failed, 6 passed; `turso-20041`: 8 failed, 9
passed; `remote-transport-compile-refusal-seam`: 6 failed, 96 passed;
`memory-20041`: 15 failed, 6 passed;
- the dangling-escape suites `sql-driver-like-pattern` (22) and
`memory-like-pattern` (15) stayed green.
Restored, rebuilt, and `ablation-dist-preflight --absent` passed with
the tree clean.
2. **The new builder bypassing the seam** (`nulLikePatternError` calling
`unsupportedFilterError`, `nulLikePattern` calling
`invalidFilterError`). Predicted and observed: `driver-sql` 78 failed of
179 (both enumeration assertions, the row's 4, and the 72 path / log
cases of the new suite); remote seam pin 6 failed of 102.
## Gates (at `049b3a6c95`)
- `node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` from the real diff (17 paths, merge base
`8d76c2d38c`): 110 commands, the PM's 79 plus 31 the docs,
`driver-memory` and changeset paths add. All 110 run after a full `turbo
run build --filter='./packages/*' --filter='./packages/*/*'` (72 tasks);
`--ran`: "110 derived, 110 run, 0 NOT-MEASURED, 0 UNRUN".
- 109 exit 0. **One exits 1, on purpose:** `node
scripts/check-empty-changeset.mjs --base origin/main`, the DELIBERATE
CORRECTION of `.changeset/20024-sqlite-glob-stored-nul.md` above.
- Roster families beside these paths, run by hand:
`check-changeset-fixed`, `check:meta-url-spelling`,
`check:spec-changes`, `check:authz-resolver`, `check:error-code-casing`,
`check:filter-alias-parity`, `check:engine-double-contract`,
`check:error-status-conformance`: all exit 0.
- `pnpm --filter @objectstack/spec check:generated`: all 15 artifacts up
to date after `gen:api-surface`, `gen:export-origins`, `gen:schema` and
`gen:docs` (`api-surface/data.json` and `export-origins/data.json` gain
`hasNulInLikePattern`; `authorable-surface.base.json` untouched).
- `pnpm check:driver-conformance`: "50 covered cell(s), 0 in the DEBT
ledger, 0 exempt" at base and at head.
- `node scripts/check-issue-citations.mjs --base 8d76c2d`: 17
citations across 5 files, all resolve.
- `pnpm check:nul-bytes` passes, and a control-byte scan of the changed
files finds none. U+0000 is spelled `String.fromCharCode(0x00)` in every
file.
- **Re-run at the merged head `b88ca46d90`** (merge base `6780e34af5`,
17 paths, after a full `turbo run build`): the derivation gives the same
110 commands; 109 exit 0 and `check-empty-changeset` exits 1 on the
#20024 correction; `--ran`: "110 derived, 110 run, 0 NOT-MEASURED, 0
UNRUN". `check-changeset-no-major` and `check-adr-0087-registration`
exit 0; `check:generated` all 15 up to date.
- ESLint, narrowed: `--no-inline-config --format json` over the 12
changed `.ts` files reports 12 files, 0 errors, 0 warnings.
`eslint.config.mjs` enables no type-aware linting (the printed
`parserOptions` for `sql-driver.ts` are
`{"ecmaVersion":"latest","sourceType":"module"}`), so this diff cannot
move the verdict of a file it does not touch. The full `pnpm lint` is
CI's.
## Acceptance notes
- **Not this card:** a NUL-free pattern matched against a STORED value
that holds U+0000 still differs on the SQLite faces (12 of 18 control
cases in the probe, unchanged). That is #20024 item 2 (ii), and #20024
remains open for it.
- `content/docs/protocol/objectql/query-syntax.mdx` still names only the
dangling escape among refused patterns. It is not false, only
incomplete, and it is outside this claim's file surface, so it is left
for the next PR that touches the page.
- `formula` keeps evaluating a U+0000 pattern (correctly). A write-side
`check` with such a pattern therefore answers where the read side
refuses; the read side refusing loudly means the two can no longer
silently disagree.
- `origin/main` at `6780e34af5` is merged into this branch (merge commit
`2b8dd90200`, no conflicts): the 10 commits past the old merge base
`8d76c2d38c` are `aa04ea2964`, `fa00ebf447`, `7b27bd00c7`, `7a13e0562a`,
`7c1039b388`, `55daf89d74`, `226e00c038`, `7b068877ce`, `0d73ff6245`,
`6780e34af5`. One of them touches this PR's paths: `55daf89d74` (PR
#20114) changes `packages/drivers/driver-turso/src/remote-transport.ts`
and `remote-transport-compile-refusal-seam.test.ts`; both auto-merged,
and the branch's delta against the new merge base is still exactly its
17 files, +1017/−16. After the merge: `turso-20041-like-nul-pattern`
17/17 and `remote-transport-compile-refusal-seam` 136/136 (102 plus
#20114's rows), `driver-turso` 69 files / 1663 tests;
`sql-driver-compile-refusal-seam` and
`sql-driver-20041-like-nul-pattern` 179/179; `sqlite-wasm-20041` 16/16,
memory 36/36, spec 31/31.
---------
Co-authored-by: Claude <noreply@anthropic.com>
1 parent fc646cf commit 8a44ce7
17 files changed
Lines changed: 1017 additions & 16 deletions
File tree
- .changeset
- content/docs/references/data
- packages
- drivers
- driver-memory/src
- driver-sqlite-wasm/src
- driver-sql/src
- driver-turso/src
- spec
- api-surface
- export-origins
- src/data
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
29 | | - | |
| 29 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
0 commit comments