Skip to content

Backend suite showed 4 failures in 1 of 4 local runs on an unchanged commit — unidentified #406

Description

@Taleef7

Observation

On cabcfc8 (main, 2026-08-06/07), four consecutive local runs of pnpm test:

run result
1 1902 pass / 4 fail
2 1906 pass / 0 fail
3 1906 pass / 0 fail
4 1906 pass / 0 fail

Same commit, no changes between runs. 1902 + 4 = 1906, so exactly four tests that failed then passed.

I did not capture run 1's output, so the four are unidentified. That is the main cost of this — the observation exists but cannot be acted on.

What is known

  • CI is unaffected so far. Every CI run on the branch and on main has been green, including the Postgres-backed Backend-TS job, which exercises strictly more than the local SQLite floor.
  • Run 1 was the first suite run after a pnpm install + pnpm typecheck in the same shell invocation, so lingering background work from install (esbuild, better-sqlite3 rebuild) is a plausible source of contention.
  • The suite contains many timing-sensitive tests: several execution-diff cases run 8–14 s, WITHOUT preparation the official artifact reads the whole roster out-of-population takes ~6–9 s, and the MEASURE/EMPLOYEE run-pipeline tests take 1–3 s each. A loaded machine is the obvious suspect.

Why it is worth tracking rather than dismissing

A suite that fails ~1-in-4 locally will eventually fail in CI, and when it does it will look like a real regression on whatever PR is unlucky. Finding it from a red CI run on an unrelated change is the expensive way.

It is also the shape this repo keeps cataloguing: a control that appears to pass reliably and does not.

Next step

Capture full output on every suite run so the next occurrence names itself:

pnpm test 2>&1 | tee /tmp/suite-$(date +%s).txt

Then the four are identifiable and this becomes a real bug report instead of an anecdote. Nothing to fix until then — guessing at candidates would be inventing a cause.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions