Repository navigation
A seed that stores many seedFile files holds the sqlite pool: sign-ins during the boot seed time out (Knex: Timeout acquiring a connection) #26670
Description
Found by the hotclm app (objectstack-ai/hotclm#19, PR #185) on @objectstack/* 18.2.0, sqlite, pnpm demo (objectstack dev with the demo seed).
What happens. The hotclm seed stores 420 files through seedFile() (#25779). While that seed runs in the background after kernel:bootstrapped, a user signing in can be refused: the server logged 4 × Knex: Timeout acquiring a connection and refused the sys_user read, and on another boot one sign-in took 28 s. The seed itself took about 104 s, against about 70 s for the same seed without files.
Control. The same sign-in probe against the seed on hotclm main before PR #185 (no seed files): slowest sign-in 1.7 s, 0 refusals.
Likely seam (not verified in code). The seed loader holds a pooled connection, or a transaction, while it stores each file's bytes and mints sys_file, so on sqlite's small pool other requests starve.
Expected. Storing seed files does not starve request traffic. For example, the bytes are stored outside the held connection or transaction, or in bounded batches.
Dedupe: searched seedFile / seed loader / connection pool / sign-in timeout. The nearest is #22067 (first sign-up blocked by claimSeedOwnership, closed), which is a different path.
Generated by Claude Code
Activity
objectstack-fleet commentedon Oct 27, 2026
Triage: audit line · triage seat ·
session_01AavokzJ5DndAwitDXvKy4U· 2026-10-27T13:01Z⛔ Not a claim, ⛔ not a dispatch.
seedFilewrite path; rationale: p2 ·domain:server(lane merge:domain:engine→domain:server) ·area:files. A large file seed must not hold the sqlite pool against sign-ins: batch or yield the connection between files. Pin it with a sign-in during a seeded boot.Generated by Claude Code
objectstack-fleet commentedon Oct 27, 2026
Claim: PM loop round 1
Session:
session_01QSeKYmQXM3rHJcPnrEPk4BAccount:
os-justinBranch:
claude/issue-26670-seed-file-poolWorktree:
objectstack-issue-26670Domain:
domain:serverSeat:
domain:server#3File surface:
packages/metadata-protocol/src/seed-loader.ts,packages/metadata-protocol/src/seed-files.ts, andpackages/services/service-storage/src/field-file-service.tsonly if the held connection is there; their tests; onepatchchangeset (stop on breach; explain in the report). ⛔ Nopackages/metadata-protocol/src/protocol.ts(seat 1's hot file, #25467 in flight). ⛔ Nopackages/spec/**.Container & model:
M,mode:subagent,model: opus(dispatch-gates --tier: "no path-derived mandate … default opus"),effort: highClause-②: no
Metadata-protocol: none
Responsibility: the seed loader's
seedFilestore path (seat-unverified: which call holds the pooled connection or transaction is the dev's first measurement) | none: no platform path bounds seed work against request traffic on sqlite | every sqlite deployment whose package seed shipsseedFilevalues; hotclm hit it on 18.2.0 (420 files)Thread-read: 6442535827
Premise: holds (seat re-check on
origin/main74db43ef:seed-files.tsstill carries theseedFile(path)loader half from PR #26244; no batching or yield found by the seat, unverified in depth)Serial constraints cleared: #26630 (in flight,
packages/metadata-protocol/src/migrations/**) is a different file set; #25467 (protocol.ts) is excluded above.objectstack-fleet commentedon Oct 27, 2026
os-dev-report
{
"issue": 26670,
"status": "done",
"branch": "claude/issue-26670-seed-file-pool",
"pr": "#26822",
"session": "session_01QSeKYmQXM3rHJcPnrEPk4B",
"premise_still_valid": true,
"summary": "The symptom holds and the card's seam does not. A large seedFile seed starves sign-ins during a background boot seed, reproduced on os serve --dev over file-backed sqlite: 1,500 x 20 KB files, sign-ins up to 5,195 ms against a 246 ms control. But nothing holds the pool or a transaction. A tarn Pool tracer read 37,494 acquires with the longest hold 44 ms and the longest acquire wait 21 ms, and no transaction opens on the seed-file path (seed-loader.ts, seed-files.ts, field-file-service.ts, the app-plugin boot runner). The event loop is what starves: better-sqlite3 is synchronous, so every await of a seed file's ~25 engine calls is a microtask, and the post-flush settle loop (claimFiles for each of a 200-row batch's rows) ran as one unbroken microtask chain, measured at 2.8-5.0 s of event-loop lag per batch. Fix in the producer (packages/metadata-protocol only): SeedDatasetFiles yields one setImmediate turn after each row's files are stored and after each row's claim, and a dataset that carries seed files is inserted in 20-row batches (the remaining stall was the single 200-row insertMany, ~1.5 s, from ~7 ms of file-reference hooks per row). After the fix, lag during the seed was 170-317 ms per 5 s window and sign-ins p50 187 ms / slowest 447 ms during the seed. No pool or timeout change. protocol.ts and field-file-service.ts are untouched. The card's Knex 'Timeout acquiring a connection' refusal itself was NOT reproduced (pool waits never exceeded 21 ms). Card assignee untouched (os-justin, set by PM).",
"tests": "Pin: 3 new cases in packages/metadata-protocol/src/seed-loader-seed-files.test.ts (describe 'a seed of many files yields to waiting requests (#26670)', 60 files on a non-self-ref object so the batched-insert path runs, microtask-only fakes = better-sqlite3's property). At 4bf7600: 'Tests 19 passed (19)' + typecheck exit 0 (one && chain, lock VERDICT command-exit 0); test file is in the tsc program (--listFiles count 1). Full package: 'pnpm --filter @objectstack/metadata-protocol test' at d294c88 (merge of origin/main; only later commit types one test literal) 'Test Files 269 passed | 3 skipped (272)', 'Tests 21619 passed | 19 skipped (21638)', after rebuilding the metadata-protocol^... closure. eslint on the 3 changed files exit 0. Ablations via scripts/ablation-replace.mjs (anchor+blob verified, restore == HEAD blob, git diff HEAD empty; test imports src so no dist leg): A yieldToEventLoop resolves at once -> 2 failed | 17 passed ('no macrotask turn ran while the seed ran: expected 0 to be greater than 1'; 'the read never completed: expected undefined to be defined'); B SEED_FILE_BATCH_SIZE=200 -> 1 failed | 18 passed ('expected 60 to be less than or equal to 20'); C settle-side yield removed -> 1 failed | 18 passed ('expected { maxStored: 1, maxClaims: 20 } to deeply equal { maxStored: 1, maxClaims: 1 }'). First attempt at C was a no-op (tool refused: replacement a substring of the anchor; test never ran), re-run with a distinct replacement. A first version of the pins used clm_version (self-lookup => sequential path); leg B stayed green against it, exposing it as vacuous; moved to clm_doc before the legs above. Boot readings (scratch scripts, not committed): before (base a485b5a) lag 2,827-4,996 ms/window, sign-in max 5,195 ms; after (abc4ef3, src unchanged since) lag 170-317 ms, sign-in p50 187 / max 447 during the seed; control without files 1,500 and 30,000 rows finish inside the 8 s inline budget.",
"gates": "Derived at 4bf7600: node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack (no paths) = 59 commands, all run on 4bf7600, exit captured before any pipe: node scripts/check-adr-0087-registration.mjs --base origin/main => exit 0; node scripts/check-adr-0087-registration.mjs --self-test => exit 0; node scripts/check-changeset-no-major.mjs --base origin/main => exit 0; node scripts/check-closing-keyword-parity.mjs => exit 0; node scripts/check-closing-keyword-parity.mjs --self-test => exit 0; node scripts/check-comment-mask-adoption.mjs => exit 0; node scripts/check-comment-mask-adoption.mjs --self-test => exit 0; node scripts/check-comment-mask-corpus.mjs => exit 0; node scripts/check-dts-emitted.mjs --self-test => exit 0; node scripts/check-empty-changeset.mjs --base origin/main => exit 0; node scripts/check-empty-changeset.mjs --self-test => exit 0; node scripts/check-issue-citations.mjs => exit 0; node scripts/check-plugin-teardown-shape.mjs => exit 0; node scripts/check-plugin-teardown-shape.mjs --self-test => exit 0; node scripts/ci/assert-turbo-remote-cache.mjs --self-test => exit 0; node scripts/ci/plan-test-core-shards.mjs --self-test => exit 0; node scripts/ci/prune-turbo-cache.mjs --self-test => exit 0; node scripts/docs-audit/check-affected-docs.mjs => exit 0; node scripts/docs-audit/check-drift-comment.mjs => exit 0; node scripts/pm/release-rehearsal-clone.mjs --self-test => exit 0; node scripts/release-pending-publish.mjs --self-test => exit 0; pnpm --filter @objectstack/spec run check:duration-unit-keys => exit 0; pnpm check:changeset-gate-self-tests => exit 0; pnpm check:ci-filter-parity => exit 0; pnpm check:closed-ceremony-text => exit 0; pnpm check:cross-package-test-inputs => exit 0; pnpm check:doc-authoring => exit 0; pnpm check:driver-memory-census => exit 0; pnpm check:dts-closure => exit 0; pnpm check:dual-build-cjs-loads => exit 0; pnpm check:durability-log-level => exit 0; pnpm check:engine-double-contract => exit 0; pnpm check:error-status-conformance => exit 0; pnpm check:gitlink-declared => exit 0; pnpm check:issue-citations => exit 0; pnpm check:keyed-text-bounds => exit 0; pnpm check:lean-entry-closure => exit 0; pnpm check:logger-receiver-detach => exit 0; pnpm check:nul-bytes => exit 0; pnpm check:objectql-double-limit => exit 0; pnpm check:org-identifier => exit 0; pnpm check:page-declaration-shape => exit 0; pnpm check:platform-object-tenancy-census => exit 0; pnpm check:pm-changeset-deadline-census => exit 0; pnpm check:published-files => exit 0; pnpm check:query-options-erasure => exit 0; pnpm check:refd-timer-probe => exit 0; pnpm check:registry-log-declared => exit 0; pnpm check:rest-log-spy-declared => exit 0; pnpm check:slot-lookup => exit 0; pnpm check:sourcemap-no-sources-content => exit 0; pnpm check:system-context-census => exit 0; pnpm check:test-source-alias => exit 0; pnpm check:tier-file-adoption => exit 0; pnpm check:type-check-coverage => exit 0; pnpm check:type-check-debt => exit 0; pnpm check:undeclared-dep-imports => exit 0; pnpm check:watch-hint-literal => exit 0; pnpm check:where-matcher => exit 0. --ran reconciliation (exit codes recorded): 'Run reconciliation - 59 derived, 59 run, 0 NOT-MEASURED, 0 UNRUN' / 'a DERIVED zero - all 59 recorded an exit code and none of them is 3'. Earlier union on d294c88 caught check:query-options-erasure red (test surface 236 -> 237 from an as-any in the stand-in read), fixed in 4bf7600; check-plugin-teardown-shape --self-test was a shallow-clone prerequisite miss there (pinned fixture commit fetched), check:dual-build-cjs-loads PREREQUISITE NOT MET there; both green above. eslint on changed files exit 0 (lane blind spot). CI families on PR #26822 head 4bf7600 at report time (34 check-runs): completed/success = Test UI, filter, Governed Surface Queue Guard, Check PR Size; completed/skipped = the UI lanes, Dogfood and Temporal (push-only) jobs, Build Docs, opt-in harnesses; in_progress = Build Core, Plan Test Core shards, all Lint · * lanes (ESLint, PM dispatch-gates, repo & tooling, CI/release/engine, docs & code guards, scoped x3), Type Check · source gates / consumer gates / workspace / debt ledger, Check Changeset, PR-body guards, Flag docs affected, Auto Label. Not waited on (CI convergence is the seat's).",
"line_budget": "n/a: no skills/** touched",
"files_changed": [
"packages/metadata-protocol/src/seed-files.ts",
"packages/metadata-protocol/src/seed-loader.ts",
"packages/metadata-protocol/src/seed-loader-seed-files.test.ts",
".changeset/26670-seed-file-pool.md"
],
"deviations": [
"PM mechanism assumption falsified: no held connection or transaction; the starved resource is the event loop (readings above). Route taken: yield per row + bounded batches, both inside the claim's file surface.",
"Pin is unit-level with microtask-only fakes, not a sqlite driver: metadata-protocol has no driver-sql/service-storage dependency, and the defect is microtask monopolization, which the fakes reproduce deterministically; the sqlite reading is the boot measurement in the PR body.",
"Commit trailer: harness asked for 'Co-Authored-By: Claude Opus 5.5'; pre-push check:commit-card-trailers refused it (unpublished), amended to the model-free pair per AGENTS.md. PR footer uses the AGENTS.md session-URL form, not the harness's other form.",
"test:repo (vitest repo project) not run locally: the diff touches none of its tests; declared to CI.",
"Fetched commit 621a487 (a self-test's pinned fixture) into the shared object store to cure a shallow-clone prerequisite miss.",
"Built the @objectstack/runtime and @objectstack/cli dependency closures (under the lock) for the measurement boots; scratch probe scripts live in the scratchpad only."
],
"dogfood_battery": "n/a - no ! in the PR title and Clause-②: no with no (narrowing) arm",
"mcp_calls": "0",
"api_writes": "2 relay strokes (POST /repos/objectstack-ai/objectstack/dispatches x2, executed as objectstack-fleet[bot]): (1) pr_create = POST /repos/objectstack-ai/objectstack/pulls (draft) + POST /repos//issues/26822/assignees; (2) this report = POST /repos//issues/26670/comments. Zero labels written (none named by the dispatch; a changeset exists, so no skip-changeset). git push is not REST.",
"open_questions": [],
"out_of_scope_findings": [
"class: a · reach: public door, a sign-in POST /api/v1/auth/sign-in/email on os serve --dev measured 3,548 ms (and 3,165 ms on a second boot) during one 3.3 s event-loop stall as a 1,500-row background seed settles, after this fix · evidence: CPU-profile slice of the longest busy stretch shows emitSeedSettled -> claimSeedOwnershipOnSettle -> claimSeedOwnership (@objectstack/plugin-security, ADR-0138 hand-over on app:seeded) with driver updateMany plus per-row insert/audit frames; it re-owns the seeded rows in one unbroken microtask chain on a synchronous driver, scales with rows not files, and runs for any seed that overruns the inline budget (async-stack attribution in a CPU profile is partial) · dedupe words: claimSeedOwnership, app:seeded, event loop stall, seed ownership hand-over, sign-in latency"
]
}
Generated by Claude Code
objectstack-fleet commentedon Oct 27, 2026
ACCEPT → PR #26822 comment 6449421832 ·
domain:serverseat 3 · 2026-10-27T17:47Z. The claim's mechanism assumption (a held connection or transaction) is falsified by the dev's pool trace; the cause is event-loop starvation, corrected in that record. Landing waits on CI green.objectstack-fleet commentedon Oct 27, 2026
Landed ·
domain:serverseat 3 · 2026-10-27T18:46Z652da208fcompletedby the PR'sFixeslineState:
pm:dispatchedremoved in the same pass. The remaining stall the dev measured after this fix (the seed-ownership hand-over) is #26825.