Skip to content

Commit 794b744

Browse files
committed
test(driver-turso): load the writers' dependencies before tsx is registered
A writer's start-up was about 40 ms of compiling the package's source and about 2 s of loading its dependencies' built code through tsx's require hook, which plain Node loads in about 1.2 s. The writer now requires those dependencies first, then registers tsx through its public CJS API and requires the source, whose requires hit the cache. The phase times join the measurement line. Claude-Session: https://claude.ai/code/session_01P88ehrZ6hM19hBzdpsQTTo Co-authored-by: Claude <noreply@anthropic.com>
1 parent 7e0c6cc commit 794b744

1 file changed

Lines changed: 44 additions & 19 deletions

File tree

‎packages/drivers/driver-turso/src/turso-remote-autonumber-concurrency.test.ts‎

Lines changed: 44 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,8 @@
1515
* read-then-write one. Neither can two `@libsql/client` instances in ONE
1616
* Node process, whose native calls run on the one JS thread.
1717
*
18-
* So this file spawns two child processes (through `tsx`, over the package's
19-
* SOURCE, the way `packages/spec`'s process-boundary pins do). For each round
18+
* So this file spawns two child processes, each running the package's SOURCE
19+
* compiled by `tsx`, as `packages/spec`'s process-boundary pins do. For each round
2020
* the parent creates a fresh database, and each writer opens its own
2121
* `@libsql/client` connection to it — the native `file:` backend, which is
2222
* libSQL itself, the engine behind a Turso endpoint — drives its own
@@ -34,19 +34,25 @@
3434
* # What the file waits on, and what it no longer waits on
3535
*
3636
* Measured at `d4aaeaf0e` on a 4-vCPU box, one round of the earlier shape
37-
* (writers spawned per round, 200 writes each, the default
38-
* `synchronous = FULL`) took about 4.4 s: about 2.2 s to start each writer
39-
* (`tsx` plus the driver's source graph, mostly `@objectstack/driver-sql`),
40-
* and about 2.2 s of writes. The writes were fsync-bound: one `create` is three
37+
* (writers spawned per round under `node --import tsx`, 200 writes each, the
38+
* default `synchronous = FULL`) took about 4.4 s: about 2.2 s to start each
39+
* writer and about 2.2 s of writes. Of a writer's start-up, the package's own
40+
* source took about 40 ms to compile; the rest was its dependencies' built code
41+
* (`@objectstack/driver-sql` and what it imports, `@objectstack/spec`,
42+
* `@libsql/client`), which loads in about 1.2 s under plain Node and about 2 s
43+
* under tsx's hook. The writes were fsync-bound: one `create` is three
4144
* statements (reserve, insert, read back), and a single writer took 5.9 ms per
4245
* `create` with fsync and 1.2 ms with `synchronous = OFF`. CI's hourly full
4346
* battery measured the file at up to 29.3 s, over 10 s in 6 of the 7 runs that
4447
* executed it, and every retry round paid both start-ups again.
4548
*
46-
* Two changes, neither touching what is asserted:
49+
* Three changes, none touching what is asserted:
4750
*
4851
* - The writers start ONCE per test and serve every round over stdin/stdout.
4952
* A retry round therefore costs its writes, not two more process start-ups.
53+
* - A writer loads the driver's dependencies with plain Node first, then
54+
* registers tsx and requires the source. The source is still compiled by
55+
* tsx, and the dependencies are the same modules, loaded without the hook.
5056
* - Each writer sets `PRAGMA synchronous = OFF` on its own connection.
5157
* `synchronous` decides only whether SQLite flushes to disk at a commit,
5258
* which matters for a power cut and not for a throwaway temp file. The lock
@@ -115,7 +121,15 @@ import { TursoDriver } from './index.js';
115121

116122
const HERE = fileURLToPath(new URL('.', import.meta.url));
117123
const PKG_ROOT = join(HERE, '..');
118-
const DRIVER_ENTRY = new URL('./index.ts', import.meta.url).href;
124+
const DRIVER_ENTRY = fileURLToPath(new URL('./index.ts', import.meta.url));
125+
/**
126+
* What the driver's source imports at runtime from `node_modules` (its type-only
127+
* imports are erased). The writer loads these BEFORE tsx is registered: they
128+
* are built code, not this package's source, and tsx's require hook roughly
129+
* doubles their load. A dependency missing from this list still loads, under
130+
* tsx, so the list is a cost, never a correctness, question.
131+
*/
132+
const PRELOADED_DEPENDENCIES = ['@libsql/client', '@objectstack/driver-sql', '@objectstack/spec/api', '@objectstack/spec/data'];
119133

120134
const WRITES_PER_WORKER = 200;
121135
/** Rounds a loaded box gets to let the two writers overlap before the overlap evidence is declared NOT MEASURED. */
@@ -146,9 +160,12 @@ interface WorkerReport {
146160
}
147161

148162
/**
149-
* The writer. Runs under `tsx` so it imports the driver's SOURCE, like the
150-
* test itself does, and pays that start-up once for every round. It reads one
151-
* JSON command per line on stdin and answers one JSON line on stdout:
163+
* The writer. Loads the driver's dependencies with plain Node, then registers
164+
* tsx through its public CJS API and requires the driver's SOURCE, like the
165+
* test itself does — so the source is compiled by tsx, to CJS (this package
166+
* has no `"type": "module"`), and its requires of those dependencies are cache
167+
* hits. It pays that start-up once for every round. It reads one JSON command
168+
* per line on stdin and answers one JSON line on stdout:
152169
*
153170
* - `{ open: <db path> }` — connect to that round's database, register the
154171
* schema (the tables already exist — the parent created them — so
@@ -159,13 +176,13 @@ interface WorkerReport {
159176
* stdin closing ends the loop, and the process exits.
160177
*/
161178
const WORKER_SOURCE = `
162-
import { createClient } from '@libsql/client';
179+
import { createRequire } from 'node:module';
163180
import { createInterface } from 'node:readline';
164-
// tsx compiles this package's .ts to CJS (no "type": "module"), so the
165-
// class may arrive under \`default\` — the spelling packages/spec's
166-
// process-boundary pins use.
167-
const driverModule = await import(${JSON.stringify(DRIVER_ENTRY)});
168-
const TursoDriver = driverModule.TursoDriver ?? driverModule.default?.TursoDriver;
181+
const require = createRequire(${JSON.stringify(join(PKG_ROOT, 'package.json'))});
182+
for (const dependency of ${JSON.stringify(PRELOADED_DEPENDENCIES)}) require(dependency);
183+
require('tsx/cjs/api').register();
184+
const { TursoDriver } = require(${JSON.stringify(DRIVER_ENTRY)});
185+
const { createClient } = require('@libsql/client');
169186
const { OS_TEST_TAG } = process.env;
170187
const reply = (message) => process.stdout.write(JSON.stringify(message) + '\\n');
171188
let driver = null;
@@ -209,7 +226,7 @@ interface Writer {
209226
function startWriter(tag: string): Writer {
210227
const child = spawn(
211228
process.execPath,
212-
['--import', 'tsx', '--input-type=module', '-e', WORKER_SOURCE],
229+
['--input-type=module', '-e', WORKER_SOURCE],
213230
{ cwd: PKG_ROOT, env: { ...process.env, OS_TEST_TAG: tag }, stdio: ['pipe', 'pipe', 'pipe'] },
214231
);
215232
const inbox: unknown[] = [];
@@ -286,10 +303,13 @@ async function round(writers: Writer[]): Promise<number> {
286303

287304
// Release both only once both are connected and registered, so the
288305
// writes contend rather than queue behind connection set-up.
306+
const opened = Date.now();
289307
for (const w of writers) w.send({ open: dbPath });
290308
await Promise.all(writers.map((w) => w.next('ready')));
309+
const released = Date.now();
291310
for (const w of writers) w.send({ go: WRITES_PER_WORKER });
292311
const reports = await Promise.all(writers.map((w) => w.next<WorkerReport>('report')));
312+
const written = Date.now();
293313

294314
// Different processes, and not this one.
295315
const pids = reports.map((r) => r.pid);
@@ -334,7 +354,12 @@ async function round(writers: Writer[]): Promise<number> {
334354
const inOrder = [...owner.keys()].sort((a, b) => a - b).map((n) => owner.get(n)!);
335355
let switches = 0;
336356
for (let i = 1; i < inOrder.length; i++) if (inOrder[i] !== inOrder[i - 1]) switches++;
337-
process.stdout.write(`[turso-remote-autonumber-concurrency] pids ${pids.join('/')} · ${total} writes · ${new Set(all).size} distinct · ${switches} writer change(s)\n`);
357+
// The phase times say where a slow run went: "ready" includes both process
358+
// start-ups on the first round only, "writes" is the contention itself.
359+
process.stdout.write(
360+
`[turso-remote-autonumber-concurrency] pids ${pids.join('/')} · ${total} writes · ${new Set(all).size} distinct · ` +
361+
`${switches} writer change(s) · ready ${released - opened}ms · writes ${written - released}ms\n`,
362+
);
338363
return switches;
339364
}
340365

0 commit comments

Comments
 (0)