Skip to content

Commit 9bfbacb

Browse files
fix(driver-sql): the local json backfill lets the codec decide, so JSON nested past SQLite's depth limit is no longer quoted into a string (#19972)
Fixes #19912 Clause-②: no ## What was wrong `SqlDriver.backfillCanonicalJsonEncoding` (the local SQLite `Field.json` storage backfill from #12380) ran one statement per json column on every `syncSchema` / `initObjects` over an existing table: ```sql update ?? set ?? = json_quote(??) where typeof(??) = 'text' and json_valid(??) = 0 ``` SQLite's `json_valid()` answers 0 for JSON nested more than 1000 levels deep, while the driver reads such a cell with `JSON.parse` without trouble. So a deep array the current write door stored correctly was quoted into a JSON string on the next schema sync and read back as a string from then on. No error, no log line: a correct cell silently became a wrong one. ## The fix SQL now only pre-filters. The driver's codec decides. - **One rule, shared by both faces.** New `packages/drivers/driver-sql/src/unencoded-json-text.ts` exports `recoverUnencodedJsonText(stored)`: when `JSON.parse` accepts the text it returns `null` and the cell is left as stored; when `JSON.parse` rejects it, the cell is a legacy plain string and the function returns `JSON.stringify(stored)`, which is what the write path stores. It is exported from `@objectstack/driver-sql`, and the Turso remote residue backfill (`remote-codec-residue-backfill.ts`, from PR #19904) now imports it. That remote change is an import plus one line in place of its inline copy, and the remote behaviour does not change. - **The local pass** (`convergeJsonColumn`) reads the `typeof(col) = 'text' and json_valid(col) = 0` candidates 500 at a time in `rowid` order, applies the rule to each, and rewrites only the cells the rule rejects. Each rewrite is a compare-and-set, `where rowid = ? and typeof(col) = 'text' and col = ?` bound to the text the decision read, so a value written concurrently is never overwritten. One transaction covers each page's writes. The pass walks every page, so a column still converges in a single sync, and memory stays bounded to one page. - **Idempotence holds.** A converted cell is outside the pre-filter. A withheld cell is re-read on every sync but never written. A re-run over a converged table changes zero rows, and that is pinned with `total_changes()`. - **Logging.** The failure posture is unchanged: `warn`, and the rows stay legacy. The warn now also reports how many rows the committed pages converted. Withheld cells get one `info` line per column. ## Measured before acting (the dispatch's H1 to H5) - **H1, confirmed** at `2c1011b`: the method and the statement are exactly as quoted above, guarded by `if (!this.isSqlite || !fields || fields.length === 0) return;` and `if (!tableExisted) return;`. - **H2, confirmed. It is a build-time constant, and every bundled build has 1000.** `json_valid` of an N-level array, measured on each bundled engine: | engine | SQLite | N=1000 | N=1001 | N=2000 | N=2001 | |---|---|---|---|---|---| | better-sqlite3 13.0.3 | 3.53.4 | 1 | 0 | 0 | 0 | | sql.js 1.14.1 | 3.49.1 | 1 | 0 | 0 | 0 | | `@libsql/client` 0.17.4 (local) | 3.45.1 | 1 | 0 | 0 | 0 | `JSON.parse` (Node v22.22.2) reads 1001, 1999, 2000 and 2001 levels. The bundled `sqlite3.c` defines `JSON_MAX_DEPTH 1000` unless `SQLITE_JSON_MAX_DEPTH` is set, and notes the limit "was 2000 prior to version 3.42.0". A Turso server build, or any custom or older build, can therefore differ. The fix does not depend on the value. - **H3, confirmed.** No override of the method exists outside `sql-driver.ts`. `SqliteWasmDriver` forces `isSqlite` and inherits `syncSchema` / `initObjects` unchanged. In local mode, `TursoDriver.syncSchema` goes to `super.syncSchema`, then `initObjects`, then `super.initObjects`; `initObjects` goes to `super`; `syncSchemasBatch` calls `super.syncSchema` once per schema. All three doors reach the backfill, and the Turso local face runs on better-sqlite3 (`toKnexConfig`). The Turso doors are pinned in a test. The wasm face was measured with a scratch probe but is not pinned (see Acceptance notes): through both `syncSchema` and `initObjects`, the 1001-level array read as an array after two backfills, and `bare` was converted. - **H4: the decision rule is shared, the paging machinery is not.** The remote pass runs over libsql `execute`/`batch` (`RemoteBackfillClient`), with a probe and a per-invocation budget. The local pass runs over knex. Sharing the walker would take more than an import swap on the remote side. The pre-filter SQL reads the same on both faces, but it only narrows the scan and never decides. - **H5, confirmed. The two are identical by construction.** The SQLite json arm of `formatOutput` is `JSON.parse(text)` inside a try/catch. The rule asks the same question, and a test pins the link through the driver: the backfill rewrites a cell exactly when the driver's read of that cell answers the stored text itself. **Preservation, byte for byte.** Stored and then read on better-sqlite3, `json_quote(col)` equals `JSON.stringify` of the text read back, for every non-surrogate BMP code point plus astral characters and an embedded NUL: 63,499 samples, 0 mismatches. Lone surrogates never reach disk; better-sqlite3 stores them as U+FFFD. The compare-and-set only writes when the stored bytes equal what was read, so every cell written is written with the bytes the retired statement produced. ## Tests New `packages/drivers/driver-sql/src/sql-driver-json-backfill-depth-limit.test.ts` (7 tests): 1. The card's reproduction: `bare`, a 1001-level array and `[1,2]`. Only `bare` is quoted, and the array still reads as an array after a **second** backfill. 2. Idempotence: `total_changes()` does not move over two re-runs, including with a withheld cell present. 3. Preservation: 11 legacy bare texts (empty, control characters, an embedded NUL, U+2028/2029, a leading U+FEFF, emoji and CJK) are each converted to the exact bytes `select json_quote(?)` gives, and each still reads as itself. 4. The rule is the read codec. The cases are 1000-, 1001- and 2001-level arrays, a 1001-level object, parseable and unparseable texts, `1e400` and an escaped lone surrogate. A cell is rewritten if and only if its pre-backfill read answers its own text, and no read moves. 5. Compare-and-set: a second connection writes the deep array into a candidate row between the page read and the write, and the write leaves it alone. 6. Paging: 1,100 rows, with deep cells at rows 0, 499, 500, 999 and 1099. All 1,095 legacy rows converge in one pass, and the log payloads report `rowsConverted: 1095` and `rowsWithheld: 5`. 7. A unit test of `recoverUnencodedJsonText`. New `packages/drivers/driver-turso/src/turso-local-json-backfill-depth-limit.test.ts` (3 tests) runs the dev's reproduction on `new TursoDriver({ url: ':memory:' })`, once through each local door: `syncSchema`, `initObjects` and `syncSchemasBatch`. Suites run at `74ff944`: - driver-sql: 181 files passed, 11 skipped; 2677 tests passed, 170 skipped. The skips are the live PG and MySQL cells; CI runs them in `Temporal Conformance`. - driver-turso: 60 files, 1334 tests passed. This includes the unchanged `turso-remote-codec-residue-backfill.test.ts`, run against the rebuilt driver-sql `dist`. - driver-sqlite-wasm: 29 files, 518 tests passed. - `tsc --noEmit` exit 0 for driver-sql and driver-turso. `--listFiles` shows each new test file inside its package's program. ## Reverse verification Both ablations went through `scripts/ablation-replace.mjs` (anchor hit 1 time, blob changed, and restore proven as blob == HEAD with an empty `git diff HEAD`) from the committed fix at `74ff944`. - **Ablation 1: the retired statement put back** in place of `convergeJsonColumn`. Afterwards driver-sql was rebuilt. `ablation-dist-preflight` found the marker `json_quote(??)` in `dist/index.js` and `dist/index.mjs`; before the mutation it was absent from both. - driver-sql file: **4 red, 3 green**. Red: the reproduction, the read-codec rule, paging, and compare-and-set. Compare-and-set is red because the old code never reads candidates, so the injection never fired (`expected +0 to be 1`); ablation 2 isolates it. Green: idempotence, preservation, and the unit test, which is the expected direction because the old statement is also idempotent and byte-identical on bare text. - Turso file: **3 of 3 red**, all with `expected false to be true` on `Array.isArray`. - Restore leg: rebuilt, the preflight with `--absent` found the marker in none of the 6 built files, and the tests went back to 7/7 and 3/3. - **Ablation 2: the compare-and-set predicate removed.** The update became `where rowid = ?`. **Exactly one test went red**, compare-and-set: `raced` was overwritten with `"bare"`. After the restore, the file is 7/7 green again. ## Gates `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` at `74ff944` derived 62 families: the dispatch's 56, plus `check:engine-double-contract`, `check:objectql-double-limit`, `check:query-options-erasure`, `check:type-check-coverage`, `check:type-check-debt` and `check:where-matcher`. Reconciled with `--ran` (exit codes recorded): **60 exit 0 and 2 NOT MEASURED** (both exit 3, a missing prerequisite). - `check:dual-build-cjs-loads` needs every workspace package's `dist`, and 78 published entry points have no `dist` here. As a narrowed probe: the CJS and ESM entries of driver-sql and driver-turso both load, and `recoverUnencodedJsonText` resolves from both driver-sql builds. - `check:type-check-debt` is missing the built type entries of 23 dependencies. The two packages this diff touches type-check clean (above). - `check:plugin-teardown-shape --self-test` first exited 3 on the shallow clone (its pinned fixture commit was unreachable). After fetching that one commit it passed, and `check:lean-entry-closure` passed after the `@objectstack/objectql` closure was built. - `check:driver-conformance` gave the same line before (`2c1011b`) and after (`74ff944`): `OK — 50 covered cell(s), 0 in the DEBT ledger, 0 exempt`. - `check:durability-log-level` exit 0: 36 durability-critical seams, all loud. - Narrowed lint, not the repo-wide `pnpm lint`: `eslint --no-inline-config --format json` over the 6 changed TS files gave 6 results, 0 errors and 0 warnings, at `74ff944`. Every one of the 6 falls under the `packages/**/*.{ts,tsx,mts,cts}` blocks of `eslint.config.mjs`, and the `.changeset` file is outside every lint glob. The config enables no type-aware linting (no `parserOptions.project`, no typed rules, as its own comment records), so this diff cannot move the verdict for an untouched file. ## Acceptance notes - **Behaviour on the wasm face, measured only.** sql.js reads text through a decoder that drops a leading U+FEFF, and its bind path truncates at an embedded NUL. Where the read differs from the stored bytes, the compare-and-set does not match, so the cell is not written and keeps reading as it did. The retired statement rewrote such cells, because `json_quote()` works on the stored bytes. For a BOM-prefixed JSON text, that would change this face's read from an array to a string. That last point is reasoned from the two measured reads; it was not run against the old code. The wasm face also used to run a zero-row `UPDATE` on every sync for every json column. Its dirty flag is set by statement kind, so the file-backed image was marked dirty each time. It now runs an `UPDATE` only when a candidate is rejected. A committed wasm pin would be a new file in `driver-sqlite-wasm`, which is outside this card's declared file surface, so none was added. - **Withheld cells cost a read on every sync.** A deep cell stays a pre-filter candidate forever, so it is read and parsed on each sync, with one `info` line per column. These cells are rare by shape. - **The json pre-filter has the same embedded-NUL exception as the date one.** `json_valid()` stops at a NUL, so a text that is valid JSON up to a NUL is never selected. That is the safe direction, and the local doc now says so. The remote module's header names only the date exception ("with one exception in the safe direction (the date bullet)"). This is a gap in the docs, not a defect. - **A cursor stall guard.** If a rowid comes back rounded (past 2^53) and the cursor stops advancing, the pass stops, leaving the rest of the column legacy, which reads the same. --- _Generated by [Claude Code](https://claude.ai/code/session_01Bvd69VPa6puiNzzPUroDBx)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent c839986 commit 9bfbacb

7 files changed

Lines changed: 597 additions & 38 deletions

File tree

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
---
2+
"@objectstack/driver-sql": minor
3+
"@objectstack/driver-turso": patch
4+
---
5+
6+
fix(driver-sql): the local SQLite `Field.json` storage backfill no longer turns a deeply nested array or object into a string on the next schema sync (#19912)
7+
8+
The backfill that converges legacy json cells on their JSON-encoded form (it came with the change that made the SQLite write path JSON-encode every json value; the issue number that change cites no longer resolves, and its live record is the `SqlDriver.backfillCanonicalJsonEncoding` doc block and `sql-driver-12380-json-roundtrip.test.ts`) ran one `UPDATE … set col = json_quote(col)` over every TEXT cell SQLite's `json_valid()` rejects. `json_valid()` answers 0 for JSON nested more than 1000 levels deep (SQLite's JSON depth limit in every build this repository bundles), while the driver reads such a cell with `JSON.parse` without trouble. So a deep array written correctly through the driver was quoted into a JSON string by the next `syncSchema` / `initObjects`, and read back as a string from then on — silently, with no error.
9+
10+
SQL now only pre-selects the candidate cells, a page at a time. The driver's own codec decides each one: a cell `JSON.parse` reads is left exactly as stored; a cell it cannot read is a legacy plain string and is rewritten to `JSON.stringify` of that string, byte-for-byte what the old statement wrote for it wherever the engine reads the stored bytes back verbatim. A cell the engine does not read back verbatim is left as stored and keeps reading as it did, where the old statement rewrote it: on `SqliteWasmDriver`, a legacy text with a leading U+FEFF or an embedded NUL (sql.js drops both when it reads the text); on any engine, text holding invalid UTF-8. Each rewrite is a compare-and-set on the text it was decided from, so a value written concurrently is never overwritten, and a re-run over a converged table still writes nothing. This covers every local SQLite face that inherits the backfill: `SqlDriver` on every client it treats as SQLite (`better-sqlite3`, `sqlite3` and its alias `sqlite`), `SqliteWasmDriver`, and `TursoDriver` in local mode.
11+
12+
The decision rule is exported from `@objectstack/driver-sql` as `recoverUnencodedJsonText(stored)`, and `@objectstack/driver-turso`'s remote codec-residue backfill now imports it instead of carrying its own copy, so the local and remote backfills apply one rule. That is a new public export on `@objectstack/driver-sql`'s root entry, and the reason this package takes `minor`: the function returns `null` for text `JSON.parse` accepts and `JSON.stringify(stored)` for text it rejects, and it is exported so that `SqlDriver.backfillCanonicalJsonEncoding` and the remote backfill's `recoverResidueCell` decide each cell by one shared rule rather than by two copies that could drift apart. The remote backfill's behaviour is unchanged.

‎packages/drivers/driver-sql/src/index.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,10 @@ export { resolveSqliteAbsentFileTarget } from './sql-driver.js';
1414
// stops an embedder from re-deriving the seam (or, worse, putting the text back
1515
// on the wire by spreading the error, which the symbol key exists to prevent).
1616
export { withheldFilterDiagnosticOf } from './sql-driver.js';
17+
// [#19912] The rule that decides which json TEXT cell a storage backfill
18+
// rewrites. Exported because the Turso remote backfill applies the same rule to
19+
// its own residue, and a second copy there is how the two faces would drift.
20+
export { recoverUnencodedJsonText } from './unencoded-json-text.js';
1721
// [#11991] The #11756 emission-scope refusal. Exported because a host that
1822
// renders its own diagnostics (Studio, `os migrate plan`, an installer) needs
1923
// the structured `client` / `supportedClients` and the stable `code` — the
Lines changed: 303 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,303 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
3+
/**
4+
* [#19912] The local `Field.json` storage backfill
5+
* (`SqlDriver.backfillCanonicalJsonEncoding`, #12380) lets the driver's own
6+
* codec decide which cell to rewrite. SQL only pre-filters.
7+
*
8+
* The defect: the backfill was one `UPDATE … set col = json_quote(col) where
9+
* typeof(col) = 'text' and json_valid(col) = 0`. SQLite's `json_valid()`
10+
* answers 0 for JSON nested past its depth limit (1000 levels in the bundled
11+
* better-sqlite3), while `JSON.parse` — what `formatOutput` reads every json
12+
* TEXT cell with — reads it. So a deep array the current write door stored
13+
* correctly was quoted into a JSON string on the next `syncSchema`, and read
14+
* back as a string from then on. Nothing failed; the cell was simply wrong.
15+
*
16+
* What is pinned:
17+
*
18+
* 1. The card's reproduction: `bare`, a 1001-level array and `[1,2]` → only
19+
* `bare` is quoted, and the array still reads as an array after a SECOND
20+
* backfill.
21+
* 2. Idempotence: a re-run over a converged table (withheld cell included)
22+
* changes zero rows.
23+
* 3. Preservation: legacy bare text that is not JSON is still converted to
24+
* exactly the bytes the old `json_quote()` statement wrote.
25+
* 4. The rule is the read codec: a cell is rewritten exactly when the driver
26+
* reads it back as its own text.
27+
* 5. Compare-and-set: a value written between the page read and the write is
28+
* not overwritten.
29+
* 6. Paging: a column with more candidates than one page converges in one pass.
30+
*
31+
* The same round-trip through `TursoDriver`'s local face lives in
32+
* `driver-turso/src/turso-local-json-backfill-depth-limit.test.ts`.
33+
*/
34+
35+
import { describe, it, expect, afterEach, vi } from 'vitest';
36+
import { mkdtempSync, rmSync } from 'node:fs';
37+
import { createRequire } from 'node:module';
38+
import { tmpdir } from 'node:os';
39+
import { join } from 'node:path';
40+
import { SqlDriver } from './sql-driver.js';
41+
import { recoverUnencodedJsonText } from './unencoded-json-text.js';
42+
43+
/**
44+
* A second, raw connection — another process on the same file. Loaded through
45+
* `createRequire` because the package carries no `@types/better-sqlite3`; the
46+
* two members used are typed here.
47+
*/
48+
const Database = createRequire(import.meta.url)('better-sqlite3') as new (file: string) => {
49+
prepare(sql: string): { run(...args: unknown[]): unknown };
50+
close(): void;
51+
};
52+
53+
const T = 'json_depth_19912';
54+
const FIELDS = { label: { type: 'text' }, val: { type: 'json' } };
55+
const SCHEMA = { name: T, fields: FIELDS };
56+
57+
/** An array nested `levels` deep: `[]` is 1, `[[]]` is 2. */
58+
function deepArray(levels: number): unknown {
59+
let v: unknown = [];
60+
for (let i = 1; i < levels; i++) v = [v];
61+
return v;
62+
}
63+
64+
function deepObject(levels: number): unknown {
65+
let v: unknown = {};
66+
for (let i = 1; i < levels; i++) v = { a: v };
67+
return v;
68+
}
69+
70+
function arrayDepth(v: unknown): number {
71+
let d = 0;
72+
while (Array.isArray(v)) {
73+
d++;
74+
v = v[0];
75+
}
76+
return d;
77+
}
78+
79+
const open: SqlDriver[] = [];
80+
const dirs: string[] = [];
81+
82+
async function makeDriver(filename = ':memory:'): Promise<SqlDriver> {
83+
const d = new SqlDriver({ client: 'better-sqlite3', connection: { filename }, useNullAsDefault: true });
84+
(d as any).logger = { warn: vi.fn(), info: vi.fn(), error: vi.fn(), debug: vi.fn() };
85+
open.push(d);
86+
await d.connect();
87+
return d;
88+
}
89+
90+
afterEach(async () => {
91+
for (const d of open.splice(0)) await d.disconnect();
92+
for (const dir of dirs.splice(0)) rmSync(dir, { recursive: true, force: true });
93+
});
94+
95+
/** A cell the way a pre-#12380 door (or any raw writer) left it: bound as-is. */
96+
async function plantRaw(d: SqlDriver, id: string, val: unknown): Promise<void> {
97+
await d.execute(`insert into "${T}" ("id", "label", "val") values (?, ?, ?)`, [id, id, val as any]);
98+
}
99+
100+
async function disk(d: SqlDriver, id: string): Promise<{ t: string; v: unknown }> {
101+
const rows = (await d.execute(`select typeof("val") as t, "val" as v from "${T}" where "id" = ?`, [id])) as any[];
102+
return rows[0];
103+
}
104+
105+
async function readAll(d: SqlDriver): Promise<Map<string, unknown>> {
106+
const rows = (await d.find(T, {})) as Array<Record<string, unknown>>;
107+
return new Map(rows.map((r) => [r.id as string, r.val]));
108+
}
109+
110+
async function totalChanges(d: SqlDriver): Promise<number> {
111+
const rows = (await d.execute('select total_changes() as n')) as any[];
112+
return Number(rows[0].n);
113+
}
114+
115+
describe('[#19912] the local json backfill leaves JSON nested past SQLite\'s depth limit as stored', () => {
116+
it('the card\'s reproduction: only `bare` is quoted; the 1001-level array reads as an array after a SECOND backfill', async () => {
117+
const d = await makeDriver();
118+
await d.syncSchema(T, SCHEMA); // creates the table: no backfill
119+
await plantRaw(d, 'bare', 'bare'); // the pre-#12380 form of the string 'bare'
120+
await d.create(T, { id: 'deep', label: 'deep', val: deepArray(1001) }, { bypassTenantAudit: true });
121+
await d.create(T, { id: 'pair', label: 'pair', val: [1, 2] }, { bypassTenantAudit: true });
122+
123+
const deepText = JSON.stringify(deepArray(1001));
124+
// The current door stored the array encoded — and the pre-filter selects it,
125+
// so it is the codec, not SQL, that must keep it.
126+
expect(await disk(d, 'deep')).toEqual({ t: 'text', v: deepText });
127+
const valid = (await d.execute(`select json_valid("val") as ok from "${T}" where "id" = 'deep'`)) as any[];
128+
expect(valid[0].ok).toBe(0);
129+
expect(arrayDepth((await readAll(d)).get('deep'))).toBe(1001);
130+
const pairBefore = await disk(d, 'pair');
131+
132+
await d.syncSchema(T, SCHEMA); // first backfill
133+
expect(await disk(d, 'bare')).toEqual({ t: 'text', v: '"bare"' });
134+
expect(await disk(d, 'deep')).toEqual({ t: 'text', v: deepText });
135+
expect(await disk(d, 'pair')).toEqual(pairBefore);
136+
137+
await d.syncSchema(T, SCHEMA); // second backfill
138+
const read = await readAll(d);
139+
expect(Array.isArray(read.get('deep'))).toBe(true);
140+
expect(arrayDepth(read.get('deep'))).toBe(1001);
141+
expect(read.get('bare')).toBe('bare');
142+
expect(read.get('pair')).toStrictEqual([1, 2]);
143+
expect(await disk(d, 'deep')).toEqual({ t: 'text', v: deepText });
144+
});
145+
146+
it('is idempotent: a re-run over a converged table changes zero rows, withheld cell included', async () => {
147+
const d = await makeDriver();
148+
await d.syncSchema(T, SCHEMA);
149+
await plantRaw(d, 'bare', 'bare');
150+
await d.create(T, { id: 'deep', label: 'deep', val: deepArray(1001) }, { bypassTenantAudit: true });
151+
await d.syncSchema(T, SCHEMA); // converges `bare`, withholds `deep`
152+
153+
const before = await totalChanges(d);
154+
await d.syncSchema(T, SCHEMA);
155+
await d.syncSchema(T, SCHEMA);
156+
expect(await totalChanges(d)).toBe(before);
157+
});
158+
159+
it('preservation: legacy bare text that is not JSON is converted to exactly the bytes json_quote() wrote', async () => {
160+
const d = await makeDriver();
161+
await d.syncSchema(T, SCHEMA);
162+
const LEGACY = [
163+
'America/New_York',
164+
'',
165+
'{bad json',
166+
'[1,2',
167+
'quote " and backslash \\ and slash /',
168+
'line\nbreak\ttab\rreturn',
169+
'a\u0000b', // an embedded NUL: better-sqlite3 stores and reads it verbatim
170+
'\u0001\u001f\u007f',
171+
'

',
172+
'leading BOM',
173+
'emoji 😀 and 中文',
174+
];
175+
for (const [i, s] of LEGACY.entries()) await plantRaw(d, `l${i}`, s);
176+
for (const [i, s] of LEGACY.entries()) expect(await disk(d, `l${i}`), `l${i} planted`).toEqual({ t: 'text', v: s });
177+
178+
await d.syncSchema(T, SCHEMA);
179+
const read = await readAll(d);
180+
for (const [i, s] of LEGACY.entries()) {
181+
// What the retired statement wrote, computed by SQLite itself.
182+
const quoted = ((await d.execute('select json_quote(?) as q', [s])) as any[])[0].q;
183+
expect(await disk(d, `l${i}`), `l${i}`).toEqual({ t: 'text', v: quoted });
184+
expect(quoted, `l${i}: json_quote and JSON.stringify agree`).toBe(JSON.stringify(s));
185+
expect(read.get(`l${i}`), `l${i} reads as the string it is`).toBe(s);
186+
}
187+
});
188+
189+
it('the rule is the read codec: a cell is rewritten exactly when the driver reads it back as its own text', async () => {
190+
const d = await makeDriver();
191+
await d.syncSchema(T, SCHEMA);
192+
const TEXTS = [
193+
JSON.stringify(deepArray(1000)),
194+
JSON.stringify(deepArray(1001)),
195+
JSON.stringify(deepArray(2001)),
196+
JSON.stringify(deepObject(1001)),
197+
'{"a":1}',
198+
'"already quoted"',
199+
' [1] ',
200+
'1e400',
201+
'"\\ud800"',
202+
'bare',
203+
'',
204+
'{bad json',
205+
'[1,]',
206+
];
207+
for (const [i, s] of TEXTS.entries()) await plantRaw(d, `t${i}`, s);
208+
const readBefore = await readAll(d);
209+
210+
await d.syncSchema(T, SCHEMA);
211+
for (const [i, s] of TEXTS.entries()) {
212+
const readsAsItself = readBefore.get(`t${i}`) === s;
213+
const rule = recoverUnencodedJsonText(s);
214+
expect(rule !== null, `t${i}: the rule rewrites iff the read answers the text itself`).toBe(readsAsItself);
215+
expect((await disk(d, `t${i}`)).v, `t${i} on disk`).toBe(rule ?? s);
216+
}
217+
// …and no read moved.
218+
const readAfter = await readAll(d);
219+
for (const [i] of TEXTS.entries()) {
220+
expect(readAfter.get(`t${i}`), `t${i} reads the same`).toStrictEqual(readBefore.get(`t${i}`));
221+
}
222+
});
223+
224+
it('compare-and-set: a value written between the page read and the write is not overwritten', async () => {
225+
const dir = mkdtempSync(join(tmpdir(), 'os-19912-'));
226+
dirs.push(dir);
227+
const file = join(dir, 'cas.db');
228+
const d = await makeDriver(file);
229+
await d.syncSchema(T, SCHEMA);
230+
await plantRaw(d, 'raced', 'bare');
231+
await plantRaw(d, 'calm', 'also bare');
232+
233+
// A second connection — another process on the same file — writes the deep
234+
// array into `raced` right after the backfill has read its candidates.
235+
const other = new Database(file);
236+
const deepText = JSON.stringify(deepArray(1001));
237+
const knex = (d as any).knex;
238+
let injected = 0;
239+
const onResponse = (_res: unknown, q: { sql?: string }) => {
240+
if (injected === 0 && /json_valid/.test(q.sql ?? '') && /^select/i.test(q.sql ?? '')) {
241+
other.prepare(`update "${T}" set "val" = ? where "id" = 'raced'`).run(deepText);
242+
injected++;
243+
}
244+
};
245+
knex.on('query-response', onResponse);
246+
try {
247+
await d.syncSchema(T, SCHEMA);
248+
} finally {
249+
knex.removeListener('query-response', onResponse);
250+
other.close();
251+
}
252+
253+
expect(injected).toBe(1);
254+
// The concurrent write survives: the backfill decided on 'bare', and the
255+
// cell no longer holds 'bare'.
256+
expect(await disk(d, 'raced')).toEqual({ t: 'text', v: deepText });
257+
expect(arrayDepth((await readAll(d)).get('raced'))).toBe(1001);
258+
// The row nobody raced converged as usual.
259+
expect(await disk(d, 'calm')).toEqual({ t: 'text', v: '"also bare"' });
260+
});
261+
262+
it('pages: a column with more candidates than one page converges in one pass, withheld cells on the page edges', async () => {
263+
const d = await makeDriver();
264+
await d.syncSchema(T, SCHEMA);
265+
const knex = (d as any).knex;
266+
const deepText = JSON.stringify(deepArray(1001));
267+
const DEEP_AT = new Set([0, 499, 500, 999, 1099]);
268+
const rows = Array.from({ length: 1100 }, (_, i) => ({
269+
id: `r${String(i).padStart(4, '0')}`,
270+
label: 'x',
271+
val: DEEP_AT.has(i) ? deepText : `legacy ${i}`,
272+
}));
273+
await knex.batchInsert(T, rows, 200);
274+
275+
await d.syncSchema(T, SCHEMA);
276+
277+
const left = (await d.execute(
278+
`select "id", "val" from "${T}" where json_valid("val") = 0 order by "id"`,
279+
)) as Array<{ id: string; val: string }>;
280+
expect(left.map((r) => r.id)).toEqual([...DEEP_AT].map((i) => `r${String(i).padStart(4, '0')}`));
281+
for (const r of left) expect(r.val).toBe(deepText);
282+
const converted = (await d.execute(
283+
`select count(*) as n from "${T}" where "val" = json_quote('legacy ' || (substr("id", 2) + 0))`,
284+
)) as any[];
285+
expect(Number(converted[0].n)).toBe(1100 - DEEP_AT.size);
286+
287+
const info = ((d as any).logger.info as ReturnType<typeof vi.fn>).mock.calls.map((c) => c[1]);
288+
expect(info).toContainEqual({ rowsConverted: 1100 - DEEP_AT.size });
289+
expect(info).toContainEqual({ rowsWithheld: DEEP_AT.size });
290+
});
291+
});
292+
293+
describe('[#19912] recoverUnencodedJsonText — the shared rule', () => {
294+
it('leaves text that parses; rewrites text that does not as its JSON string', () => {
295+
expect(recoverUnencodedJsonText(JSON.stringify(deepArray(1001)))).toBeNull();
296+
expect(recoverUnencodedJsonText('{"a":1}')).toBeNull();
297+
expect(recoverUnencodedJsonText('123')).toBeNull();
298+
expect(recoverUnencodedJsonText('"x"')).toBeNull();
299+
expect(recoverUnencodedJsonText('bare')).toBe('"bare"');
300+
expect(recoverUnencodedJsonText('')).toBe('""');
301+
expect(recoverUnencodedJsonText('a\u0000b')).toBe('"a\\u0000b"');
302+
});
303+
});

0 commit comments

Comments
 (0)