Skip to content

4-hourly full run: red on main (CI) #27076

Closed

Description

@github-actions

os-4-hourly-full-run-ci — machine-findable marker for this generated card. ⛔ Do not delete this line: it is how the 4-hourly full run finds this card instead of filing a new one every 4 hours.

4-hourly full run: red on main (CI)

Swept 2026-10-28T12:25:20.367Z · run log · commit d745bb5e5ff7ea034b77bae0f16fcdd4c8dd21d3 · conclusion failure.
CI runs the FULL battery on main every 4 hours (#16467, #25771). A push run on
main tests only the packages the merge touched, and a merge-queue build tests only what the group
contained — so this scheduled run is the only thing that says whether the WHOLE tree is green, and
this card is the only channel that reports it. Nothing is blocked by it.
⛔ The remedy is never to narrow, skip or delete the failing check to make the 4-hourly run green.
Fix what it names and let the next 4-hourly run refresh this card; a card nobody reopens is closed by
the next green run being uneventful, not by editing this one.

What landed since the last green 4-hourly run

Range: 3217fc5dcf (run 40606905083, the last green CI schedule run) .. d745bb5e5f — compare

Activity

  1. added
    area:recordsBusiness objects, records, the views that show data, usable forms, search
    domain:serverDomain lane — seat card indexed by label:pm:seat
    priority:p0Critical: blocker, must ship before MVP
    and removed
    priority:p1High: required for production / M2
    on Oct 28, 2026
  2. objectstack-fleet commented on Oct 28, 2026

    @objectstack-fleet
    Contributor

    Triage: p1 → p0 · domain:server · pm:blocked on #27004 · triage seat · session_01AavokzJ5DndAwitDXvKy4U · 2026-10-28T04:26Z

    ⛔ Not a claim.


    Generated by Claude Code

  3. objectstack-fleet commented on Oct 28, 2026

    @objectstack-fleet
    Contributor

    triage#1 · os-zhuang · session_01AavokzJ5DndAwitDXvKy4U

    Triage: the 08:05Z run is red on a different failure · p0 kept · 2026-10-28T08:57Z

    ⛔ Not a claim.


    Generated by Claude Code

  4. objectstack-fleet commented on Oct 28, 2026

    @objectstack-fleet
    Contributor

    Claim: PM loop round 5 · card thread · p0 main-red · lane refill under the shared cap of 4
    Session: session_0186vfjHHMsRXC9EUT9hTFMo (the seat); the card thread is session_01FEkQ5oKUikM4XcQpd1HJ6U
    Account: os-project-manager (the seat's linked user; the card's assignee)
    Branch: claude/project-thread-9j22mo (the card thread's harness-pinned branch; it names no issue, so this claim is the identity record)
    Worktree: the card thread's own container checkout
    Domain: domain:server
    Seat: domain:server#2
    Ruling-ref: triage 6460647889 (p1 → p0 under main-red convention ①: "⛔ Never raise the 5 s budget, skip the case or shrink its rows"). Blocked-by #27004 is cleared: PR #27073 merged at 05:03Z (693a7298). The 08:05Z schedule run 40654333991 on 24351cfe (a descendant) is red again, in Test Core (6/8), packages/drivers/driver-sqlite-wasm pnpm run test exited 1. The failing test is not yet read (the log blob answers 403; the MCP tail stops at the last 5,000 lines). The dev identifies it first and hands back needs_decision if it cannot
    File surface: whatever file the identified red's root cause lives in, its tests, and changesets per package the diff publishes from. ⛔ No raised timeout, skip, smaller fixture or retry wrapper on any test
    Container & model: M, mode:thread, default model and effort (set at thread open)
    Clause-②: no (expected; the dev measures and declares)
    Metadata-protocol: none
    Responsibility: the 4-hourly full run's red on main after #27004
    Thread-read: 6460647889
    Premise: holds (read 2026-10-28T09:11Z at origin/main 0f1045bc: card open, unassigned, pm:queue; the 08:05Z schedule run is red after #27073 landed); the dev re-measures it first, and a refutation is a premise_still_valid: false hand-back
    Resume-from: none — fresh from origin/main
    Serial constraints cleared: yes — the dev re-reads open PRs on the file it fixes before the first edit and before pr_create

    State: pm:queue → pm:dispatched, assigned to os-project-manager, in this act.


    Generated by Claude Code

  5. objectstack-fleet commented on Oct 28, 2026

    @objectstack-fleet
    Contributor

    os-dev-report
    {
    "issue": 27076,
    "status": "needs_decision",
    "branch": "claude/project-thread-9j22mo",
    "pr": null,
    "session": "session_01FEkQ5oKUikM4XcQpd1HJ6U",
    "premise_still_valid": true,
    "summary": "Not identified, not reproduced, so no fix (order step 3). The red of run 40654333991 (24351cf) is real: Test Core (6/8) turbo reports driver-sqlite-wasm pnpm run test exited (1) (annotation at log line 25,901 of 55,143), but every readable channel stops short of the failing test, and the suite passes every local reproduction at 24351cf: alone x2, beside 3 CPU burners x8, and inside the shard's real whole-package leg (the 10 packages of Test Core 6/8, turbo --concurrency=4, OS_TEST_TIERS=queue) x3, 746/746 tests every time. Two facts narrow it without naming it: (1) the shard's check-test-completeness step printed OK (11 of 11 packages reported, 16621 tests declared and all accounted for), so the package printed its summary and no test went missing; exit 1 with a complete summary means either a failed case or a run-level vitest error (an unhandled error / worker RPC fault) after the cases, and the readable log cannot tell which; (2) the same signature happened once before: schedule run 39853791549 (49185cd, 2026-10-22), Test Core (6/6), where driver-sqlite-wasm#test AND runtime#test:repo both exited 1 in one turbo run. Of the 240 most recent failed CI runs (60 each of schedule, push, merge_group, pull_request) these two are the only driver-sqlite-wasm exits; no push, merge_group or PR run shows it. It is not the #27004 family: no wasm case is anywhere near the 5 s budget (slowest case 0.28 s alone, 0.57 s under load; slowest file 1.6 s inside the shard leg), and nothing in packages/drivers/driver-sqlite-wasm changed between the last green schedule run (3217fc5) and 24351cf except the version bump; its dependencies changed in driver-sql (#26960 cut 19, a move) and spec.",
    "tests": "All at 24351cf in a detached worktree, closure built under os-verify-lock (driver-sqlite-wasm^... then the shard's 10-package ^... closure, VERDICT command-exit 0 both). (a) vitest run alone x2: exit 0, Test Files 39 passed (39), Tests 746 passed (746). (b) 8 runs beside 3 node -e for(;;){} burners on the 4-core box, one lock hold of 448 s: 8/8 exit 0, 746/746. (c) the shard's whole-package leg as ci.yml runs it: OS_TEST_TIERS=queue pnpm turbo run test test:repo filtered to plugin-auth, plugin-approvals, lint, service-datasource, driver-sqlite-wasm, service-queue, example-todo, service-job, service-cluster, connector-imap, --concurrency=4 --log-order=stream (reps 2-3 with --force --only): 3/3 exit 0, driver-sqlite-wasm 746/746 each (suite wall 94.9 s, 106.5 s, 87.8 s; Tasks 73/73, 10/10, 10/10). Slowest wasm cases by max over runs, alone: like-stored-nul grid 283 ms, reclaim-space 282 ms, autonumber-resync 279 ms, returning-persist 260 ms, driver-autonumber 239 ms; under burners: reclaim-space 574 ms, autonumber-batch-resync 520 ms, driver-autonumber 519 ms, date-bucket 480 ms, autonumber-resync 439 ms; slowest files inside the shard leg: tenant-scope 1,612 ms, 20024-like-stored-nul 1,408 ms, driver-advanced 1,358 ms. The CI-side difference I cannot reproduce here is the cli 4/4 slice that runs after the whole leg on the same runner (it does not overlap the wasm suite: the legs are sequential in ci.yml).",
    "gates": "n/a: no diff, no PR. No dispatch-gates derivation run (there is no change set).",
    "dogfood_battery": "n/a: no PR",
    "mcp_calls": "11, all reads: issue_read x2 (card, comments), get_job_logs x4 (job 122702224962 tail: 4,999 lines, capped, all cli; aggregator job 122707051142: attestation and timing tables only; plan job 122701700200: shard 6 package list; 10-22 job 120086810803 tail: turbo Failed line only), actions_list x1 (run artifacts), actions_get x1 (artifact download url), get_check_run x1 (empty output), search_issues x2",
    "api_writes": "1: this os-dev-report comment on #27076 through scripts/pm/post-stamped.mjs. REST reads only otherwise (gh api: check-run annotations, run jobs, schedule run list, failed-run lists, open pulls and their files). Read attempts that failed: the artifact zip test-core-run-summary-6-of-8 on productionresultssa11.blob.core.windows.net, curl exit 56, proxy CONNECT 403 (connect_rejected); the job-log redirect goes to the same blob host class.",
    "open_questions": [
    {
    "question": "How do we get the failing test of Test Core (6/8) in run 40654333991 named, since this container cannot read past the last 5,000 log lines or download artifacts?",
    "options": [
    "A: someone with browser access to GitHub (the maintainer, or a seat whose host reaches the artifact blob) opens job 122702224962 at log line ~25,901 (search driver-sqlite-wasm:test for FAIL / Unhandled / Error) or downloads artifact test-core-run-summary-6-of-8, and pastes the failing test name and its error onto #27076; the seat then re-dispatches with that name. Cost: one manual read, about 5 minutes. Artifact retention ends 2026-11-01, log retention longer.",
    "B: wait for the next 4-hourly schedule run. Green closes the card under its own rule, and the case reads as a rare load flake (2 hits among the 60 most recent failed schedule runs, both on the shard carrying driver-sqlite-wasm); red on wasm again gives a fresh run whose log has the same reach problem, so it needs A anyway.",
    "C: a CI-side observability change so a failing package's own failure lines reach the readable tail of the shard log (for example, re-print the failing turbo task's FAIL and Errors blocks at the end of the test step). This is a new card in the CI lane, not this one; it fixes the reach problem for every future shard red, not this red."
    ],
    "recommendation": "A, with B in parallel. Real business need: the card is p0 and closes only on a green full run; A is the only option that names the cause of a red that has now happened twice, while B only tells whether it recurs. Long-term soundness: guessing a fix without the failing test would be the workaround the order forbids; A keeps the fix contract-first. Preventing AI mistakes: A hands the next dev a named failing case, so the fix can be red-first instead of speculative. Startup focus: A and B add nothing to the product or the gate set; C is a new CI capability, so I do not recommend it here: it is a separate card only if the maintainer names it."
    }
    ],
    "out_of_scope_findings": [
    "carrier: none · noted, not filed: the fleet's readers cannot see a Test Core shard failure that sits more than 5,000 lines before the end of the job log (get_job_logs tail cap; log and artifact blobs 403 through the card-thread proxy). Same reach problem triage hit on this card. Recorded for option C only."
    ]
    }

    Generated by Claude Code

  6. objectstack-fleet commented on Oct 28, 2026

    @objectstack-fleet
    Contributor

    Seat note · #27076 parked, waiting on a log read or the next schedule run

    domain:server seat 2 · session_0186vfjHHMsRXC9EUT9hTFMo · 2026-10-28T10:23Z

    The card thread handed back needs_decision with no code (os-dev-report 6466478513). The driver-sqlite-wasm red in schedule run 40654333991 (job 122702224962, Test Core (6/8), near log line 25,901) cannot be identified from this environment:

    • the MCP log tail stops at the last 4,999 lines;
    • the log blob and the run-summary artifact answer proxy 403;
    • the check-run annotations name only the package.

    Local reproduction at 24351cfe stayed green in every configuration. The package passed 746 of 746 each time:

    • alone, twice;
    • beside three CPU burners, 8 times;
    • inside the shard's real 10-package leg, 3 times.

    This is not the #27004 budget family. The slowest wasm case takes 0.28 s alone and 0.57 s under load.

    The shard counted every test. Its completeness step accounted for all 16,621, so the exit 1 is either a failed case or a run-level vitest error. The same signature appeared once before, in schedule run 39853791549 (49185cdb).

    Next, both in parallel:

    • A: a reader with browser access opens job 122702224962 near line 25,901, or downloads artifact test-core-run-summary-6-of-8 (retained to 11-01), and pastes the failing test and error here. The seat then re-dispatches with a named target.
    • B: the next 4-hourly run (≈12:05Z) either closes this card green under triage's rule or confirms a recurrence with a fresh log.

    State: pm:dispatched → pm:blocked. The assignee stays on the seat. ⛔ No budget raise, skip or retry wrapper in the meantime.


    Generated by Claude Code

  7. hotlong commented on Oct 28, 2026

    @hotlong
    Contributor

    Failing test named: read from the full job log, 2026-10-28

    维护者速读

    失败的用例是 SqliteWasmDriver (file persistence) > should persist data on-write,5000 ms 超时(实际跑了 5026 ms)。同一文件里它的兄弟用例 on-disconnect 也跑了 4236 ms,离预算只差一点。10-22 那次同样的红是 5 条 wasm 文件持久化用例一起超时。所以这和 #27004 是同一类:本地 0.3 s 的落盘用例在 CI 分片的负载下能慢到 5 s 以上。两次失败都发生在落盘路径上,不是某一条用例的逻辑错误。

    Readback

    The full log of job 122702224962 was downloaded from a maintainer machine: 55,143 lines, the same count the seat saw. Excerpt, ANSI stripped:

    ❯ src/sqlite-wasm-driver.test.ts (10 tests | 1 failed) 9528ms
        ✓ should persist data across disconnect/reconnect (on-disconnect)  4236ms
        × should persist data on-write 5026ms
    
    FAIL  src/sqlite-wasm-driver.test.ts > SqliteWasmDriver (file persistence) > should persist data on-write
    Error: Test timed out in 5000ms.
     ❯ src/sqlite-wasm-driver.test.ts:136:3
    
    Test Files  1 failed | 38 passed (39)
         Tests  1 failed | 745 passed (746)
      Duration  66.08s (transform 24.46s, setup 0ms, import 143.94s, tests 32.04s, environment 20ms)
    

    The earlier occurrence, schedule run 39853791549, job 120086810803 (49185cdb, 10-22), was read the same way. Five wasm cases timed out at 5000 ms, all of them on-write or flush paths:

    • sqlite-wasm-background-flush-failure.test.ts: three cases, at 9879, 7792 and 6869 ms.
    • sqlite-wasm-driver-returning-persist.test.ts: flushes an UPDATE … RETURNING under on-write, 5921 ms.
    • sqlite-wasm-driver-durability.test.ts: writes the database file atomically, 5759 ms.

    In that run, on-write itself passed in 2751 ms. The runtime#test:repo red in the same run was a separate 60 s tree-walk timeout.

    What this changes in the 10:22Z report

    The report's line "not the #27004 family … slowest case 0.28 s alone, 0.57 s under load" holds on a 4-core dev box. On the shard it does not hold: the persistence cases cost 2.7–9.9 s there. Note import 143.94s for a 66 s run, which means the runner was heavily oversubscribed. The target is now named, so the seat can re-dispatch. ⛔ No raised budget, skip or retry wrapper (the card's rule); the fix belongs in why a file flush costs seconds under load.

  8. objectstack-fleet commented on Oct 28, 2026

    @objectstack-fleet
    Contributor

    os-dev-report
    {
    "issue": 27076,
    "status": "done",
    "branch": "claude/project-thread-9j22mo",
    "pr": "#27245",
    "session": "session_01FEkQ5oKUikM4XcQpd1HJ6U",
    "premise_still_valid": true,
    "summary": "Round 2, after the maintainer named the failing case on the card (comment 6468136240): sqlite-wasm-driver.test.ts 'should persist data on-write' timed out at 5,026 ms on Test Core 6/8; run 39853791549 timed out five on-write/flush cases. Root cause measured: each flush ends in atomicWriteFile's handle.sync(), and that fsync queues behind the other suites' disk I/O on the shard runner. The case is about 30 ms on an idle disk; with 4 dd conv=fsync writers on the same disk the same persistence cases (and only those) rise to 1.4-3.6 s, with 8 writers they time out (5 timeouts in 3 rounds, including on-write at 5,358 ms, the CI failure). CPU load alone does not do it. Fix (PR #27245, one file): packages/drivers/driver-sqlite-wasm/vitest.config.ts sets test.env TMPDIR to /dev/shm when writable (else unchanged) and restates the preset's disableConsoleIntercept; every test file already reaches its temp dir via os.tmpdir(). Product code, cases, assertions and the 5 s budget are unchanged; sync() and rename() still run. This replaces round 1's needs_decision (comment 6466478513): the case is identified, it IS a load-only timing red, but disk-load, not the #27004 per-row family.",
    "tests": "Head 2c4bfce. Red-first A/B, one os-verify-lock hold (VERDICT command-exit 0, 236 s), 8 dd if=/dev/zero bs=1M count=512 conv=fsync writers on the same disk, the 4 persistence files (29 cases), base and head alternated x3. Base leg = scripts/ablation-replace.mjs replacing the env line with env: {} (anchor 1 -> 0, blob 08fbf987a6e2 -> ea1e118ddf13; a probe case printed TMPDIR_SEEN=/tmp under it, /dev/shm on head), restored after each round (ok restored: blob == HEAD, git diff HEAD empty). base-1: 2 failed (returning-persist UPDATE RETURNING 6061 ms, transaction 7227 ms timeouts); head-1: 29/29, slowest 178 ms; base-2: 2 failed (INSERT RETURNING 6641 ms; driver.test.ts should persist data on-write 5358 ms, Test timed out in 5000ms); head-2: 29/29, slowest 223 ms; base-3: 1 failed (UPDATE RETURNING 6223 ms); head-3: 29/29, slowest 217 ms. Mechanism readings without the config: 4 writers, TMPDIR on disk x3: on-write 1388-3630 ms, 7-11 cases over 1 s per run; same 4 writers, TMPDIR=/dev/shm x3: on-write 12-28 ms, slowest case 0.54 s. Step probe idle: create 3, insert 1, flush 4, reopen+find 3 ms, raw 8 KiB sync 1 ms; under 2 writers raw sync up to 575 ms. Whole package at head: pnpm --filter @objectstack/driver-sqlite-wasm test, VERDICT command-exit 0, Test Files 39 passed (39), Tests 746 passed (746); /dev/shm has no leftover after the run. No test file added; no ablation needed beyond the A/B (the change is a config).",
    "gates": "Local at 2c4bfce, change set 1 path (vitest.config.ts). dispatch-gates --commands --repo objectstack-ai/objectstack: 42 commands, each exit captured before any pipe; --ran: 42 derived famil(ies) accounted for, 40 run, 2 NOT-MEASURED. NOT MEASURED: pnpm check:dual-build-cjs-loads and pnpm check:lean-entry-closure, exit 3 PREREQUISITE NOT MET (need a full workspace build; CI runs both). node scripts/check-plugin-teardown-shape.mjs --self-test exit 3 on the shallow clone, exit 0 after git fetch --depth=1 of 621a487 (48 cases). All other 37 derived commands exit 0. Hand gates: node scripts/check-console-intercept-disarm.mjs exit 0 (OK: 77 vitest-running packages); pnpm exec eslint --no-inline-config packages/drivers/driver-sqlite-wasm exit 0; pnpm --filter @objectstack/driver-sqlite-wasm typecheck exit 0; node scripts/check-commit-card-trailers.mjs --range origin/main..HEAD exit 0. (A first try of the disarm gate as a pnpm script exited 254: no such script, re-run by its node path.) CI check-runs on PR #27245: in_progress at report time, not waited on.",
    "dogfood_battery": "n/a: no ! in the PR title and Clause-2 is no with no (narrowing) arm",
    "mcp_calls": "0 this round (all GitHub reads this round through gh api REST). Round 1: 11, all reads (issue_read x2, get_job_logs x4, actions_list x1, actions_get x1, get_check_run x1, search_issues x2). No MCP write tool used.",
    "api_writes": "3 in the card thread so far: round-1 os-dev-report comment 6466478513 (post-stamped); relay pr_create of PR #27245 (fleet-write run 40687896969, body read back identical 9405 bytes, assignee os-project-manager, label skip-changeset confirmed by REST read of the pull); this os-dev-report comment. Plus one git push (not a REST write).",
    "open_questions": [],
    "out_of_scope_findings": [
    "carrier: none · noted, not filed: other packages' tests that fsync real files under os.tmpdir() could hit the same disk-queue stall on a shard; not surveyed (no red names one)."
    ]
    }


    Generated by Claude Code

  9. objectstack-fleet commented on Oct 28, 2026

    @objectstack-fleet
    Contributor

    Landing record · #27076 · PR #27245

    domain:server seat 2 · session_0186vfjHHMsRXC9EUT9hTFMo · 2026-10-28T13:09Z

    Reading Probe Result
    Queue membership git ls-remote origin 'refs/heads/gh-readonly-queue/main/pr-27245-*' 0 refs (released)
    Ancestry git merge-base --is-ancestor 4d3a077f origin/main yes (main 4d3a077f)
    Content git grep -c MEMORY_BACKED_TMP origin/main -- packages/drivers/driver-sqlite-wasm/vitest.config.ts 2 hits

    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area:recordsBusiness objects, records, the views that show data, usable forms, searchbugSomething isn't workingdomain:serverDomain lane — seat card indexed by label:pm:seatpriority:p0Critical: blocker, must ship before MVP

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions