Repository navigation
Commit 682873d
fix(core): the JSON-column refusal reads true on every face and reaches a REST caller whole (#21213)
Fixes #21067
Clause-②: no
## What this changes
`jsonColumnOperatorRefusalText`
(`packages/core/src/utils/json-column-operator-refusal.ts`) is the one
builder of the `INVALID_FILTER` / 400 refusal a filter gets for a scalar
comparison or text operator on a multi-value or JSON field. It is
rewritten once, for both of the card's reasons together (seat 2's
carried note folded in: one rewrite, not two passes):
1. **Under the REST bound.** The withheld message was 748 characters.
The `/data` door cuts a 4xx message of 500 or more to 499 plus an
ellipsis, so the wire ended `…Refused rather than compiled because the
answ…`, and no caller read the sentence saying the field and the
operator were withheld. It is now **486** characters, one constant text.
2. **True on every face.** The old reason named `driver-sql`'s storage
("a field this driver stores as a JSON TEXT column") and the two wrong
answers SQL used to give. Both are untrue on the engine's
per-aggregation `filter` and on `driver-memory`, which print the same
text. The reason is now the field's declaration: `it aims a scalar
comparison or text operator at a multi-value or JSON field, which it
cannot test for one member.`
3. **The null comparand's repair (added at the PM's note on this
card).** The refused set also catches `{ f: null }`, `$eq: null` and
`$ne: null`, which ask whether the field has a value; `$contains` cannot
express that. One constant clause, with no branch on the comparand,
names the presence spellings: `For no value, use "$null" or "$empty".`
Both answer on a multi-value or JSON field on every face (they are
outside the refused set), and the REST pin now proves `$null: true`,
`$null: false` and `$empty: true` answer on both the `where` and the
per-aggregation faces.
The new message, in full:
```text
A constraint in this filter WAS NOT APPLIED: it aims a scalar comparison or text operator at a multi-value or JSON field, which it cannot test for one member. Use "$contains" for membership ({ "FIELD": { "$contains": "a" } }), or an $or of "$contains" for any-of ({ "$or": [{ "FIELD": { "$contains": "a" } }, { "FIELD": { "$contains": "b" } }] }). For no value, use "$null" or "$empty". The field and the operator are withheld from the message; the full diagnostic is in the server log.
```
The diagnostic (server log, and the author-disclosed wire text, see
below) shares that reason, names the operator in it (`it aims "$in", a
scalar comparison or text operator, at …`; the bare spelling's operator
as `=`, as before), names the field, and spells the remedy with the
field's name, presence clause included. It drops the same storage and
SQL history. One `refusalReason()` and one `containsRemedy()` (which
ends in `PRESENCE_REMEDY`) serve both texts, so the two cannot drift.
The SQL mechanism and the measured wrong answers stay in the builder's
docblock.
**Unchanged:** `code`, `status`, the refused operator set
(`JSON_COLUMN_INCOMPATIBLE_OPERATORS`, 27 spellings), the `$contains`
remedy, no new export, no new code.
**Faces.** Re-derived at `cb45469e` by name and by text: every face that
prints this sentence calls the builder, and none builds its own copy.
They are `driver-sql` `jsonColumnOperatorError` (`sql-driver.ts`),
`objectql` `having-filter.ts` (the judgment and the per-row backstop,
three call sites), and `driver-memory` `jsonStoredFieldOperatorError`
(`filter-refusal.ts`). No face file changes except one stale docblock in
`sql-driver.ts`. `driver-turso`'s `remote-transport.ts` and
`driver-mongodb` share phrases with other refusals, not this sentence.
No docs page or skill quotes it (`content/docs/**`, `skills/**`,
`docs/**`, `apps/docs/**`, by builder name and by seven distinctive
phrases: zero hits).
## Measured, before and after
| reading | `BASE` `cb45469e` | this PR |
|:--|:--|:--|
| withheld message, constant | 748 chars | 486 chars |
| its wire `error`, SQLite and live PostgreSQL 16.14, `mapDataError` |
499 plus an ellipsis, ending `Refused rather than compiled because the
answ…` | the whole message |
| author-disclosed diagnostic, field `owners`, `$in` | 643 chars | 402
chars |
| that, at `POST /api/v1/data/:object/query` through the real
`SecurityPlugin` | 499 plus an ellipsis, ending `Refused rather than
compiled beca…` | the whole diagnostic (measured at 377 before the
presence clause; the 402 text is pinned whole through `mapDataError`) |
| field-name length from which the diagnostic is cut on the wire | every
length | 27 characters (the presence clause goes first) |
| field-name length from which the cut takes the any-of example | 10
characters | 37 characters |
### A dispatch hypothesis this falsified (H5)
The dispatch read the diagnostic as server-log text the envelope never
bounds. Measured, it is also a wire text. `driver-sql`'s `'author'`
provenance arm (`resolveWithheldFilterRefusal`) swaps the refusal for
the diagnostic when the predicate is the caller's own, and
`plugin-security` marks the caller's verbatim `where` `'author'`
(`security-plugin.ts`, the `markFilterSubtreeProvenance(callerWhere,
'author')` call). Through `POST /api/v1/data/:object/query` with a real
`SecurityPlugin`, a member caller sending `{ owners: { $in: ['u1'] } }`
received the diagnostic: cut to 500 at `BASE`, whole (377) at
`caf0e3c7`, before the presence clause raised it to 402. That was a
one-off measurement file, run twice and not committed; the `BASE` leg
rebuilt `core`'s `dist/` with the old text and restored it, with dist
preflight proofs both ways.
The diagnostic was in the rewrite already, because its reason clause was
the same untrue `driver-sql` mechanism seat 2's note names. All four
conditions of the bounded in-place fix hold: same defect class (a
refusal cut at the envelope), a mechanical fix in the shape triage
pinned, the claimed file, and the same gate families. So it is fixed
here, not filed. Its content is kept: field and operator named, remedy
with the field's name.
## Pins
New, compared with the builder's output and the bound's own function
(`truncateClientMessage` / `mapDataError` from `@objectstack/types`),
never with a copied sentence or a retyped 500. `CLIENT_MESSAGE_MAX`
itself is module-private in
`packages/types/src/data-error-classification.ts`, so it is not
imported; `truncateClientMessage` is already exported there for `rest`,
and nothing new is exported. A sibling that rewords nothing and only
calls the builder cannot flip these.
- `packages/core/src/utils/json-column-operator-refusal.test.ts`:
re-captured hashes and lengths. For every refused spelling,
`truncateClientMessage(message)` returns it unchanged. The message
carries the one-member, any-of and no-value remedy and ends with the
withheld sentence. Neither text names a storage form or a backend's
wrong answer. The diagnostic gives the same reason with the operator
named.
-
`packages/drivers/driver-sql/src/sql-driver-json-column-refusal-wire-bound.test.ts`
(new, `DIALECT_CELLS`): for 14 operators plus bare equality, on a
multi-value lookup, a `tags` field and a `json` field,
`mapDataError(err).body.error` equals the shared message (unmarked,
carrying the presence clause) and the shared diagnostic (author-marked),
with the remedy spelling. SQLite always; **PostgreSQL and MySQL in CI's
`Temporal Conformance (live PG + MySQL)` job**, which runs this
package's whole suite with both URLs set.
- `packages/rest/src/aggregation-filter-json-column-refusal.test.ts`:
through `POST /api/v1/data/:object/query`, the `where` twin's body
equals the builder's message whole, not just the per-aggregation face's
(`toBe`, plus the any-of remedy, the presence clause and the withheld
sentence), on the null-comparand rows too; three new controls show
`$null: true`, `$null: false` and `$empty: true` answer with equal
counts on both faces.
Flipped, each to the new substance:
- `sql-driver-json-column-operator-refusal.test.ts`: `'JSON TEXT
column'` becomes the operator-named reason, and the remedy is asserted
with the field's own name.
- `sql-driver-target-field-provenance.test.ts`: the class fragment that
survives redaction, `'JSON TEXT column'`, becomes `'at a multi-value or
JSON field'`.
- `sql-driver-json-column-refusal-shared-text.test.ts`: the docblock's
"did not change by one byte" claim.
ADR-0112 `code` plus `status` assertions are untouched everywhere.
### Reverse verification
At `fefb6e1f`, `BASE`'s builder was written to disk (tree only, never
staged) and its landing checked by grep: old text 1, new 0. Then:
- `sql-driver-json-column-refusal-wire-bound.test.ts` (SQLite plus live
PostgreSQL 16.14): **90 failed** (45 per cell), 1 skipped (MySQL). The
red direction, as expected.
- The core pin: **8 failed**, 4 passed.
Restored with `git checkout HEAD -- PATH` under an `EXIT INT TERM` trap.
The blob is `2f17d7f3`, equal to HEAD's, and `git diff HEAD` is empty.
## Verification (on `d352319a`, the final head: the branch, one merge of
`origin/main` `0d421041`, which touches none of these packages, and the
presence-clause commit)
The targeted files below ran on `d352319a`, with `OS_TEST_POSTGRES_URL`
set to a private PostgreSQL 16.14:
- core: 12 passed
- driver-sql, the five JSON-column files: 360 passed, 2 skipped (the
MySQL cells)
- rest `aggregation-filter-json-column-refusal`: 136 passed, 68 skipped
(MySQL)
- driver-memory `memory-20444-*` and `memory-21066-*`: 125 passed
- objectql `engine-aggregate-filter-json-column-refusal`,
`engine-aggregate-filter`, `engine-aggregate-filter-array-membership`
and `engine-cascade-delete-multivalue-probe`: 263 passed
Full suites, on the pre-merge commits (before the presence clause; that
commit touches only the builder, its unit pin and the two wire pins
above, which were re-run):
- `@objectstack/core` local: 74 files, 2107 passed
- `@objectstack/driver-sql`, SQLite: 208 files passed, 11 skipped; 3469
passed, 192 skipped
- `@objectstack/rest` local: 255 files, 4834 passed, 301 skipped
Typecheck: `core`, `driver-sql` and `rest` all pass, and `tsc
--listFiles` confirms the four touched driver-sql test files are in its
program.
**Gates** (`node scripts/pm/dispatch-gates.mjs --repo
objectstack-ai/objectstack --commands`, re-derived and re-run on
`d352319a`): 65 derived, 65 exit 0, 0 NOT MEASURED. `--ran` with exit
codes reconciles to 65 accounted, 0 unrun.
- `check:dual-build-cjs-loads` measured this time: 105 require entries
across 66 packages load. The first run on `8450f66b` was exit 3
`PREREQUISITE NOT MET`, since a whole-repo build was missing then.
- `check:driver-conformance`, before the first edit and after the last
commit: 50 covered, 0 DEBT, 0 exempt; dialect axis 8 suites, 0 in the
DIALECT ledger. The ledger did not move.
**Lint**, narrowed:
1. The population comes from eslint's own config: `--format json`
reports every listed file with 0 warnings, so none was ignored.
2. On `d352319a`, the 8 `.ts` files of this diff were linted, with 0
errors and 0 warnings.
3. `eslint.config.mjs` enables no type-aware linting (no
`parserOptions.project`), so this diff cannot move the verdict on any
untouched file.
The repo-wide `pnpm lint` is CI's to run.
**Patch round 1 (`89209290`).** `Lint & Repo Gates` was red at
`d352319a` on `pnpm check:live-db-isolation`:
`packages/rest/src/aggregation-filter-json-column-refusal.test.ts:258
use names the literal "$null"`. The fix is test-only: the presence
clause is asserted in two pieces. On `89209290`:
- `pnpm check:live-db-isolation`: exit 0, 34 live-server files scanned.
- That test file: 136 passed, 68 skipped, with SQLite and live
PostgreSQL 16.14 running and MySQL a named skip.
- The 65 derived families re-run: 65 exit 0.
`check:dual-build-cjs-loads` gave exit 3 PREREQUISITE NOT MET until
another family built the missing `dist/`, then exit 0 on re-run. `--ran`
reconciles to 65 run, 0 NOT MEASURED.
- `check:live-db-isolation` is a declared WIDE-population family, so
per-card derivation never names it.
## Siblings that print this sentence (both landed before this PR; seat
edit)
Both siblings this PR named as later landers merged first, so they are
earlier landers now:
- #21178 → `862f12c0b` (PR #21208): the `driver-turso` remote face. It
prints the builder's output and pins it by equality with the builder, so
this rewrite does not flip it.
- #20822 group 3b → `e18fea6dc` (PR #21196): new `having-filter.ts`
callers of the builder.
This branch is behind both. The merge queue rebuilds it onto `main`, so
their pins run against this text in the merge group.
## Acceptance notes
- **The diagnostic's boundary.** It names the field four times, so its
length grows with the name. It is whole on the wire up to 26-character
field names (measured over `$in`, `$startsWith` and bare equality). From
27 characters the cut takes the presence clause first, and from 37 the
any-of example; the one-member remedy and both names come before both.
Field names declare no maximum length, so no text that repeats the name
can be bounded. The withheld message, the card's subject, is constant
and bounded.
- **A gate false positive, on the test's spelling.**
`check:live-db-isolation`'s statement-head needle matches the verb USE
followed by a quoted operand (`STATEMENT` in
`scripts/check-live-db-isolation.mjs`). It read the assertion string
`'For no value, use "$null" or "$empty".'` at
`packages/rest/src/aggregation-filter-json-column-refusal.test.ts:258`
as a MySQL USE naming a database `"$null"`. Fixed in `89209290` by
asserting the clause in two pieces; the gate and the product sentence
are unchanged.
- `packages/objectql/src/engine-cascade-delete-multivalue-probe.test.ts`
keeps a test double whose refusal paraphrases the old wording ("is
stored as a JSON TEXT column"). It asserts only `code` and `status`, so
it is not a pin of this text. Left as is.
- `rest`'s own PostgreSQL and MySQL cells of
`aggregation-filter-json-column-refusal.test.ts` are still provisioned
by no CI job (its header says so). The PostgreSQL wire pin that CI does
run is the new `driver-sql` file.
- The local PostgreSQL leg ran against a private PostgreSQL 16.14
started for this run on a random port (UTC server, so `driver-sql`'s
temporal files were not run against it). It was stopped afterwards.
---
_Generated by [Claude
Code](https://claude.ai/code/session_017xfMoEjKUuSh2xYB8sCozp)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 1ecb871 commit 682873d
9 files changed
Lines changed: 389 additions & 55 deletions
File tree
- .changeset
- packages
- core/src/utils
- drivers/driver-sql/src
- rest/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
13 | | - | |
14 | | - | |
15 | | - | |
16 | | - | |
17 | | - | |
18 | | - | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
19 | 28 | | |
20 | 29 | | |
21 | | - | |
22 | | - | |
| 30 | + | |
23 | 31 | | |
24 | 32 | | |
25 | 33 | | |
26 | 34 | | |
| 35 | + | |
27 | 36 | | |
28 | 37 | | |
29 | 38 | | |
| |||
51 | 60 | | |
52 | 61 | | |
53 | 62 | | |
54 | | - | |
55 | | - | |
| 63 | + | |
| 64 | + | |
56 | 65 | | |
57 | 66 | | |
58 | | - | |
59 | | - | |
60 | | - | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
61 | 70 | | |
62 | 71 | | |
63 | 72 | | |
| |||
83 | 92 | | |
84 | 93 | | |
85 | 94 | | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
29 | 53 | | |
30 | 54 | | |
31 | 55 | | |
| |||
108 | 132 | | |
109 | 133 | | |
110 | 134 | | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
111 | 138 | | |
112 | 139 | | |
113 | | - | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
114 | 145 | | |
115 | 146 | | |
116 | 147 | | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
117 | 182 | | |
118 | 183 | | |
119 | 184 | | |
| |||
147 | 212 | | |
148 | 213 | | |
149 | 214 | | |
150 | | - | |
151 | | - | |
152 | | - | |
153 | | - | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
154 | 228 | | |
155 | 229 | | |
156 | 230 | | |
157 | 231 | | |
158 | 232 | | |
159 | 233 | | |
160 | | - | |
| 234 | + | |
161 | 235 | | |
162 | | - | |
163 | | - | |
| 236 | + | |
164 | 237 | | |
165 | 238 | | |
166 | | - | |
167 | | - | |
168 | | - | |
169 | | - | |
170 | | - | |
171 | | - | |
172 | | - | |
173 | | - | |
174 | | - | |
175 | | - | |
176 | | - | |
177 | | - | |
178 | | - | |
179 | | - | |
180 | | - | |
181 | | - | |
182 | | - | |
183 | | - | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
184 | 243 | | |
185 | 244 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
120 | 120 | | |
121 | 121 | | |
122 | 122 | | |
123 | | - | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
124 | 126 | | |
125 | | - | |
126 | | - | |
| 127 | + | |
| 128 | + | |
127 | 129 | | |
128 | 130 | | |
129 | 131 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | | - | |
11 | | - | |
12 | | - | |
13 | | - | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
14 | 15 | | |
15 | 16 | | |
16 | 17 | | |
| |||
0 commit comments