Repository navigation
Guard shared test hosts, protect Blaxel worker lifecycles, and rewrite the test-runner specs - #53
Merged
Merged
Conversation
Cover routine recovery from dead mutex owners and route production-location evidence through the canonical CLI. Separate recorded Blaxel validation from committed tests, and tighten fixture isolation, native context, and refusal assertions.
Group selection and retries, mutex ownership, and native entrypoints with guidance. State red/green evidence, safe fixture placement, and each slice baseline. Keep live-run records with contribution validation while the proof states the standing requirement.
Bound native config checks to non-executing invocations during red steps and regressions. Clarify proof assignments, early isolated-workflow preparation, setup prerequisites, and the remaining production CI evidence at draft handoff.
Resolve local files from the working tree and remote files from the requested commit. Admit focused marked-host runs, redirect broader work, and keep zero retries explicit across native and worker handoffs.
Blaxel red/green: 294f256 reproduces the unreadable same-user process; 2156f26 passes all 20 worker-lease and lease-contract cases with default retries, no failures, leaks or infrastructure failures. Retained run directories are 2026-10-27T22-22-48-239Z-p2976-re22b246ab160f1ce and 2026-10-27T22-23-18-728Z-p3090-r6956a6b83ed3f866. Remove the completed checkpoint under docs prep and a trailing blank line.
Integrates origin/thopter/test-harness-resource-cleanup (#12) on top of the already-merged local test-runner guardrails (#49). Both branch from main a8e73fb. Policies from #49 applied to #12's additions - Registered native configs refuse direct runs. #12 registers two new experiment surfaces; their configs (test/runner-fixtures/blaxel-harness/{driver,fixtures}/vitest.config.ts) now import scripts/test/native-entrypoint.mjs, like the other 36. The registered config sweep in test-runner-native-entrypoints.test.ts covers all 38. - One explicit local file on marked hosts. #12 documented the harness driver as `npm test -- local --surface experiment:blaxel-harness`, which a marked host now refuses. The harness README, the runner spec's Blaxel harness section and the Blaxel proof's live declaration use `npm test -- local --file test/runner-fixtures/blaxel-harness/driver/harness.test.ts`, accepted on marked and unmarked hosts (checked by dry run on a marked host). - Targeted native execution moved to scripts/test/target-command.mjs and target-native.mjs. #12 had edited the old in-place targetCommand() so that targeted Vitest used a runner config written by supervised-command.mjs (TV_TARGET_CONFIG / TV_TARGET_VITEST_CONFIG). #49's target-native.mjs already writes the attempt-recording config at the same attempts path, so #12's copy is dropped: no TV_TARGET_CONFIG export, no second config. supervised-command.mjs keeps #12's resource diagnostics and derives the attempts path from TV_TARGET_NATIVE_RESULT for them. test-runner-resource-parent.test.ts's targeted case now runs the real targetCommand() instead of a hand-written native command that depended on the removed config. - `## Testing` headings renamed to `## Testing guidance`. #12's Blaxel proof links now point at #testing-guidance. Changes from #12 reconciled with #49 - supervised-command.mjs: the test command gets runnerEnv() context (#49) and the parent records resource diagnostics and the attempt sidecar (#12). - The targeted task record keeps one attemptResultPath (both sides added one for the same file), #49's generatedInputPaths, and #12's resourceDiagnosticsPaths. - Remote selection reads the pinned commit's registry (#49), so the harness fixture surfaces resolve for Blaxel once committed; the CLI accepts #12's --lock-ttl-seconds alongside #49's stricter option parsing. Conflicted files - test.config.mjs: both experiment groups kept (both use order 88; ties sort by name). - scripts/test/supervised-command.mjs, scripts/run-blaxel-testshards.mjs: as above. - specs/arch/test-runner/test-runner.md: suites are subject to local guardrails (#49) and narrower Blaxel work includes planned package/runner/tag selection (#12). - specs/arch/test-runner/test-registry.md: #49's explicit-selection wording, without the "two placement classes" sentence #12 removed, plus #12's harness surface paragraph. - proofs/arch/testing-policy.md: #12's coverage-model structure opening with #49's coverage paragraph. - proofs/arch/test-runner/test-runner.md: #49's coverage paragraphs, #12's Blaxel exception paragraph, and #49's tv serve recipe (with --print-links, matching main's spec). - proofs/arch/test-runner/reporting.md: #49's private lock-location paragraph, then #12's diagnostic coverage paragraphs. - specs/index.md, proofs/index.md: regenerated with npm run specs:index. Checked: lint and type-check pass; test-runner-resource-parent.test.ts passes locally (all four cases); full validation on Blaxel follows the push.
…ked hosts The marked-host refusal named local routes only for agent, PostHog and daemon surfaces, so a dedicated coordinator selecting experiment:blaxel-harness by surface or tag was sent to npm test -- blaxel, which the driver's placement check refuses. The driver is now a local-only selection whose refusal names npm test -- local --file test/runner-fixtures/blaxel-harness/driver/harness.test.ts. The driver surface's root is that file, so the existing route code can name it. The marked-home guardrail tests cover both selections and the one-file form.
Rewrites specs/arch/test-runner/ as a set under the spec policy, so that each fact has one owner and every statement is one derivation could not be trusted to get right, or that something outside this version depends on. The registry, preflight and flaky-test content folds into test-runner.md; preflight.md, flaky-tests.md and test-registry.md are removed. The ledger in docs/working/test-runner-spec-rewrite/ records why each statement is there and lists the inbound citations that now need repair. Proofs, tests and code are unchanged. The proofs of the removed specs are orphaned and many proof citations no longer resolve, so the spec-link test fails until the proofs are re-derived after review.
Pins the summary fields readers use and the event-stream fields baseline update reads from earlier runs, and states the rule for changing the stream. Defines what the duration baseline measures. Exempts an authorized attested skip from the join's shard-agreement condition. States the Firefox omission outside GitHub Actions, including that Blaxel runs and their attestations cover Chromium only. Scopes the run-directory, best-effort, process-backend, failure cap, mutex and override statements to what the code does. Adds the review as second-opinion-r1.md and the writer's response to each of its findings to the ledger.
baseline update rejects stored event streams whose events lack schemaVersion 1, a shared runId, a contiguous zero-based sequence, or a recognized status on every event but process leaks, and reads only Blaxel runs. reporting.md now states and pins that envelope, the status union and the Blaxel identity beside the fields the update uses. It also describes optional diagnostics as best-effort whenever they are collected, rather than only after finalization. Adds the review as second-opinion-r2.md and the writer's responses to the ledger.
Adds the spec-rewrite method's removal account. It compares the nine old specs at d283755 with the six new ones at 9257497, passage by passage, and says whether each is kept, reworded, moved, now derivable, or dropped, and why. It lists the citations outside the rewritten files that point at removed files, headings or anchors, and the judgment calls a reviewer may want to see first. Also adds the passing round-three second opinion.
Six reviewers each read one rewritten spec at ee599ee and raised 20 findings in all. The reviews are committed under docs/working/test-runner-spec-rewrite/file-reviews/, and the ledger responds to each finding. Specs now match the code where they overstated it: - test-runner.md: --file matches must belong to one surface; only the local and blaxel shortcuts default to all; the registry error names the three discovered config files; verify --plan still selects a provider; teardown scans as soon as services and commands stop. - blaxel-testshards.md: the lease record includes the deletion reservation every acquisition refuses; workers use origin, with --repo-url left to the coordinator script; cleanup rejects mount boundaries; what a run's coordinator and workers exchange crosses revisions when they differ, as with --commit. - sharded-execution.md: a pass accounts for every file rather than each file running exactly once; the shard-count recommendation is bounded by the pool's size. - reporting.md: passing tests are not listed or counted, and error excerpts appear when available; selection.suite may be null; a Blaxel run creates its directory after preflight and selection checks. - github-ci.md: recipe coverage is for surfaces in all; shard jobs read their run directory from the shard worker's summary. - attestation.md: eligibility compares trees, not commits; the first policy read precedes the run's tests. Two findings were met differently from the reviewer's suggestion: the run-directory promise is kept and two late selection checks in runBlaxel are left as a code gap, and the attestation policy-read rule is narrowed instead of reordering CI. The ledger gives the reasons. The removal account corrects the entries the reviews showed to be wrong. One passage moves from derivable to dropped.
Round 2 passed test-runner.md, blaxel-testshards.md, github-ci.md and attestation.md. The reports are committed under docs/working/test-runner-spec-rewrite/file-reviews/*-r2.md, and the ledger responds to the two remaining findings. sharded-execution.md: the plan, the duration baseline and the shard summary cross runner revisions in a --commit run, and spec policy asks for pinned types for such data unless a spec states an exception. The worker checks the plan against a digest of its whole content, so there is no small subset of fields to pin. blaxel-testshards.md#runs now states the exception: these shapes are left to the code under the compatibility and infrastructure-failure rule. sharded-execution.md points to it. reporting.md: runBlaxel computed its selection arguments, including the one-surface check for targeted runs and the supported-suite check, after remote preflight had created the run directory, so those refusals left a directory without a summary. They now run first, with the coordinator's arguments unchanged. An invalid --run-dir-output value is also rejected before any directory exists, on local and Blaxel runs. The guardrails test shows each refusal precedes remote preflight and writes no run-directory identity.
The set-level second opinion (second-opinion-r4.md) and the round 3 per-file reviews of blaxel-testshards.md, reporting.md and sharded-execution.md all pass at 6ffed2c with no findings.
The test-runner rewrite folded the registry, preflight and flaky-test specs into test-runner.md and renamed most headings in its companions. This repoints every link, glossary entry and code comment outside proofs/arch/test-runner/ to where that content now lives: - spec links in AGENTS.md, the workflow, the testing policy, the desktop, licensing, Node-version and telemetry specs, the migration map and the tvdev-setup skill; - glossary entries for terms the folded specs owned, dropping "execution group" and "normalized surface result", which no spec defines now; - heading citations in workflow, script and test comments. Two tests compared the registry and stale-owner rules against the old spec text. They now read the sentences that state those rules in github-ci.md and blaxel-testshards.md.
The registry, preflight and flaky-test proofs fold into the test-runner proof as sections, keeping every assertion and anchor, the same way their specs folded into test-runner.md. Proof links to removed headings and anchors are repointed, and statements the specs now contradict are corrected: attestation eligibility is by tree, CI recipe coverage is for surfaces in the all suite, and the shard-count recommendation allows exactly two minutes and falls back to the pool size. Assertions that cover detail the specs leave to the code are kept and marked superfluous. Promises the rewrite pins for the first time are cited to the tests that already cover them: the owner-token format, plan validation by digest, refusal of a worker reserved for deletion, --run-dir-output, and one rule of the timing-event envelope. Two promises had no test: a local guardrail refusal leaves no run directory, and the summary keeps the fields RunSummary pins. A guardrails test and expectations in two reporting tests now cover them.
The tests for "a refusal leaves no run directory" checked only that no --run-dir-output identity was written. Creating the directory and writing its identity are separate steps, so that could not show the directory was never made. The refusal cases now run in a disposable checkout, with a second fixture surface committed for Blaxel selection, and check that the checkout has no .test-runs/ afterwards. The proof describes that observation. Two glossary entries said more than their owning specs: broad run is now a definition with a pointer to the one-file limit, and test failure becomes failed shard, a pointer to the shard classification.
Pre-merge docs prep. The ledger and the removal account move to docs/archive/2026-10/test-runner-spec-rewrite/ with preambles: they hold why each statement in the rewritten specs is there, how each review finding was answered, and what became of each passage of the old specs, none of which the diff records. The review reports and the merge validation note were checkpoints and are deleted, leaving docs/working/ empty. The four documents archived on the #49 and #12 branches now name this pull request, which is the one that merges them.
ThopterBot
marked this pull request as ready for review
October 28, 2026 22:21
This was referenced Oct 28, 2026
The pull request description becomes a short orientation. This document keeps the detail it held: how the guardrails, worker leases, cleanup and diagnostics work, the judgment calls in the spec rewrite, the proof and test repairs, and the validation evidence.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #49 and #12.
Several agents often share one development host, and their concurrent test runs exhaust its memory and get processes killed. On Blaxel, persistent workers collect generated test runtimes in memory-backed
/tmp, and a worker's lock can expire, or be released before the run's reports are downloaded, without proof that the run's processes stopped. This change brings together #49's local guardrails and #12's Blaxel worker leases, cleanup and diagnostics. It also rewrites the specs underspecs/arch/test-runner/from an empty page, so that they state only what the code cannot be trusted to settle on its own.What changes
scripts/test/). Local runs and local verify take a mutex shared by all of an operating-system user's checkouts. On hosts marked with~/.tvdev-use-blaxel, a local run selects at most one test file, and broader work goes to Blaxel. The runner hands Vitest and Playwright exact files, broad zero-retry runs need an override, and directvitestorplaywrightinvocations refuse. Each refusal exits2and names the command to run instead, and each guardrail has an override whose name containsagainst-test-guidance.test/runner-fixtures/blaxel-harness/exercises leases, recovery and deletion against disposable workers. It runs as two experiment surfaces outsidealland CI.preflight.md,flaky-tests.mdandtest-registry.mdare deleted. The proofs, references across the repository and contributor guidance follow the new layout.Where to look
Spec and guidance edits outside
specs/arch/test-runner/. These come from two sources.AGENTS.md,specs/arch/testing-policy.md,specs/spec-workflow.mdand thetvdev-contributeandtvdev-reviewskill sources;specs/arch/desktop/e2e-harness.mdandspecs/arch/telemetry/sink.md;specs/terms.md.specs/product/desktop-app.mdcited preflight's testing guidance to explain its required Mac verify run. It now says what that run exercises: the runner's macOS process backend.specs/terms.mdpoints each test-runner term at its new owner. It redefines broad run as a run that selects more than one test file, and restates preflight, shard, attempt and infrastructure failure to match the rewritten specs. Failed shard and recovered flake replace test failure and flaky test. Execution group and normalized surface result are dropped, because no spec defines them.Code and test scan. These are the parts most likely to raise questions about scope or cost:
New production modules in
scripts/test/:local-test-mutex.mjs, which uses Node's built-in SQLite;blaxel-worker-lease.mjs,blaxel-lease-lifecycle.mjsandblaxel-pool-lifecycle.mjs;resource-diagnostics.mjsandtest-guidance.mjs.No dependency is added.
New test machinery:
test/runner-fixtures/guidance/;test/helpers/worker-builds.tsandtest/helpers/application-requests.ts.Edits outside the runner's own files. Every registered Vitest and Playwright config gains one import, through which a direct invocation refuses. Five browser acceptances now load the built application through
ProductServer.What the rewrite changed in code, tests and proofs:
scripts/test/cli.mjschecks a Blaxel selection and--run-dir-outputbefore creating the run directory, so a refusal leaves no run directory behind.summary.jsonfields the spec pins; and two existing tests that now read the rewritten spec text.proofs/arch/test-runner/test-runner.md. No assertion is removed. Assertions about detail the specs leave to the code are marked superfluous.Validation
npm run verify -- blaxelpassed at the head,a2c4f922: 22 surfaces on 36 shards, 3,812 tests passed, 8 skipped and none failed, with no recovered flakes, process leaks or infrastructure failures. It published the head tree's attestation. GitHub CI run 40838234780 passed at the head. Lint, type-check, the package manifest checks and the vibe-mode check ran, and the heavy jobs skipped because of that attestation. Blaxel has no Firefox, so the Firefox projects ran in neither (TV-1011).Detail
docs/archive/2026-10/test-runner-spec-rewrite/change-detail.mddescribes how each mechanism works, the judgment calls in the spec rewrite and the proof and test repairs. It also holds the full validation record, including live Blaxel probes of targeted runs and the evidence carried over from #49 and #12. In the same folder,ledger.mdgives the reason for each statement in the six specs, andremoval-account.mdsays what became of each passage of the nine specs they replace.