Skip to content

A fast local preflight by default; test totals counted at a release - #841

Merged
irparent merged 2 commits into
mainfrom
test/local-test-timeout
Oct 5, 2026
Merged

irparent merged 2 commits into
mainfrom
test/local-test-timeout

Conversation

@irparent

@irparent irparent commented Oct 5, 2026

Copy link
Copy Markdown
Member

What changes

Contributing to this repository had become slower than it needed to be. Three changes:

Before Now
npm run preflight (and the pre-push hook) every check CI runs, one after another: 23 min on a desktop the checks a branch fails CI on for no reason but a missed step (claims and renders, lint, types, the dashboard's and the website's checks when the branch touches them, the branch's own test files): 74 s on this branch. --full keeps the old run
Test totals in .claims.json compared on every PR, so a PR that added a test had to run the whole suite again to count it, locally and in CI, and the file conflicted on every rebase compared only with generate.mjs --check --with-test-counts; counted at a release, as the release roll already does. claims-alignment no longer runs the suite a second time
Per-test timeout vitest's 5 s everywhere 30 s on a developer's machine, 5 s in CI

Why

Measured on 2026-10-05:

  • CI takes about 11 minutes, its jobs in parallel. The local preflight took 23, serially. Of five preflight runs that day, two failed on the test totals (the branch had added tests) and two on a 5 s timeout that a busy desktop hit. Two passed.
  • The whole suite takes 12 minutes on a Windows desktop, with or without coverage (725 s and 738 s). Most of that is starting git and node processes; website-build-scope.test.ts alone runs a few hundred git commands. So the fast preflight does not run the whole suite, and not every test a change reaches either: a change to a module every test imports reaches all of it. CI runs the suite.
  • The 5 s timeout measured the machine. A test that takes 45 ms alone timed out at 5 s in two of those runs, and a process-starting test in one. CI's runners do nothing else and keep the 5 s.

Tests

  • tests/preflight-mirrors-ci.test.ts: every pull-request job is still a --full step or CI-only. The fast mode runs the branch's own test files, never the whole suite or the builds; the dashboard's and the website's checks only when the branch touches them; and --full runs none of the fast-only steps.
  • npm run preflight (fast) passed on this commit in 74 s.

🤖 Generated with Claude Code

irparent and others added 2 commits October 5, 2026 10:35
npm run preflight runs the suite under coverage on a developer's machine
beside whatever else it is doing, and there the 5 s default measured the
machine, not the test: in four preflight runs on 2026-10-05, a test that
takes 45 ms alone timed out at 5 s twice and a process-starting test once,
each failing a 25-minute run that CI would have passed. CI's runners keep
the default, so a genuinely slow test still fails there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…elease

The preflight ran every check CI runs, one after another: 23 minutes on a
desktop against CI's 11 in parallel, and twice whenever a branch added a
test, because the committed truthbase carried the test totals and the only
way to match them was to run the whole suite again and count. CI did the
same: claims-alignment ran the suite a second time just to count it.

- npm run preflight now runs, in a few minutes, the checks a branch fails
  CI on for no reason but a missed step: claims and renders, lint, the type
  checks, the dashboard's and the website's checks when the branch touches
  them, and the test files the branch adds or changes. CI runs the whole
  suite, in parallel. `--full` keeps the old run for a change wide enough
  to want it.
- generate.mjs --check keeps the committed test totals unless
  --with-test-counts asks for them; claims-alignment no longer recounts.
  The release roll captures them, as it already did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 5, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
website Ignored Ignored Oct 5, 2026 6:32pm UTC

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

Iris gate — 1 of 2 tripped --fail-on detector_veto

iris-eval ingest: 3 stored, 1 tripped --fail-on detector_veto (2 of 3 evaluated in dataset "release-gate")

Trace Verdict Basis Rules, classes or missing inputs Evidence
b1b42bb540f9518c6209f8b835af5d14 failed detector_veto + risk_over_loss no_pii, pii_leak, credential_leak no_pii: AWS Access Key (output 45–65)
Verdict basis Traces
detector_veto 2
clean 1

Unjudged questions: task_completed (3), tool_use_correct (3) — a trace that did not carry what a rule needs.

tests/fixtures/ci-gate/traces.ndjson · 3 evaluated · dataset release-gate: 2 in the gate · exit 1 · what the bases mean

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown

Iris gate — 1 stored, nothing tripped --fail-on any

iris-eval ingest: 1 stored, 0 tripped --fail-on any

Verdict basis Traces
clean 1

Unjudged questions: task_completed (1), tool_use_correct (1) — a trace that did not carry what a rule needs.

tests/fixtures/ci-gate/clean.ndjson · 1 evaluated · exit 0 · what the bases mean

@irparent
irparent merged commit 7a6936a into main Oct 5, 2026
74 checks passed
@irparent
irparent deleted the test/local-test-timeout branch October 5, 2026 18:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant