Skip to content

Clean up test runtimes and protect Blaxel worker lifecycles - #12

Closed
ThopterBot wants to merge 65 commits into
mainfrom
thopter/test-harness-resource-cleanup
Closed

ThopterBot wants to merge 65 commits into
mainfrom
thopter/test-harness-resource-cleanup

Conversation

@ThopterBot

@ThopterBot ThopterBot commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Persistent Blaxel workers accumulate generated source-CLI and web-bundle runtimes. The worker lock also expires without proving that descendants stopped and releases before report downloads finish, allowing the next checkout to overwrite results. This PR gives generated builds a worker lifetime and protects each Blaxel attempt through cleanup and verified retrieval.

Reusable CLI and versioned web builds publish only after preparation succeeds. Playwright and Vitest worker teardown stop servers, wait for pending builds, and remove their outputs while preserving reuse across cases. Failed preparation and per-launch copies clean themselves up. Five browser acceptances—two theme lifecycle walks and three demo-mode cases—load built application assets while retaining their authentication, restart, navigation and rendering assertions. Their request inventories establish the shipped-asset boundary; incremental memory savings after Vite's full-response correction are unmeasured.

The Blaxel lifecycle uses one worker-local lease, serialized by a short flock guard. Dispatch registers ownership before checkout, and only the coordinator renews. Normal release follows attributed-process teardown and validated report download. After 20 minutes without renewal by default, the next acquisition stops the abandoned attempt's processes by owner token and PID/start identity, audits, clears its state and takes the lease. Recovery can discard undownloaded reports and retains a receipt describing that loss. Uncertain or unreadable ownership holds the worker. Pool provisioning publishes workers only after verification; deletion reserves them and removes dispatch eligibility before calling the provider.

Each attempt has private scratch and an inherited TMPDIR. Under the lease and a clean owner audit, pre-run cleanup removes only direct /tmp/television-source-cli-e2e-*, /tmp/television-web-bundle-* directories and stale runner scratch. Post-run cleanup removes the attempt's scratch after processes stop. Persistent caches and unrelated temporary files are preserved. This also reclaims known residue from test revisions without the helper fixes.

Surface parents retain available RAM, shared memory, temporary free space and OOM-counter observations at start/end, plus bounded kernel OOM lines for failed or recovered native attempts. Diagnostic links survive normalization, retry aggregation and download. Limits are 64 KiB of kernel lines, 128 KiB per sidecar and 8 MiB per shard attempt; unavailable probes stay explicit and do not change native outcomes.

The governing changes are in Blaxel shards, reporting, testing policy, and the runner/registry placement rules, with mirrored proofs. The scope is test infrastructure and acceptance setup. The archived proposal and plan preserve rationale and validation structure.

Validation:

  • Red/green contracts and real Playwright/Vitest worker teardown passed on Blaxel. The live qualification passed on disposable private workers: competing acquisition and deletion, delayed retrieval, active and quiet coordinator crashes, recovery receipts, diagnostic transport, and actual Node/Chromium temporary-file placement. Its two experiment surfaces require explicit selection and are excluded from ordinary all/CI.
  • npm run verify -- blaxel --pool poc passed at 7c66d160, containing main a8e73fbf: all 36 shards and 22 surfaces; 3,705 tests passed, eight skipped, zero failed/recovered tests, process leaks or infrastructure failures. Lint, type checking and package manifests passed. The gate retained 231 linked resource sidecars with complete start/end observations and zero OOM deltas. Normal cleanup reclaimed 1,510 runtime directories totaling 5,003,157,504 allocated bytes. Peak memory and long-term renewal-loss frequency remain unmeasured.
  • The worker-lease and lease-contract files passed all 20 cases on private Blaxel workers at 2156f266. Process-permission coverage runs the real command unprivileged and proves that unreadable production audits preserve the lease and leftovers. A root-only case checks a same-user nondumpable child; fixture process enumeration is explicitly supplied, while production audits the whole worker. Both runs retained their results and reported zero failures, leaks or infrastructure failures. A renewal already in flight at release can produce a closing warning after its lease is gone. All private workers and the coordinator were deleted after artifact retrieval.
  • GitHub CI passed all 26 jobs at 9787188d, including the full unit suite, desktop tests, all 16 e2e shards, static checks and attestation publication. The unprivileged unit job passed all nine applicable lease-fixture cases; the root-only permission case passed on Blaxel.

jwhiting added a commit that referenced this pull request Oct 1, 2026
Connect the Mac app with one link containing the server address and,
when required, its access token. The setup screen supplies a prompt for
the person's agent and a link field. A successful connection is saved
for later launches; **Television › Disconnect from Server** (⌘,) forgets
it and returns to setup, including from the upgrade gate.

- `tv links` checks the running server's authentication requirements and
prints its connect links. CLI startup and `tv links` use terminal
hyperlinks for interactive output and plain URLs for pipes.
- Connection dialogs distinguish an unreachable server from a rejected
token or required desktop upgrade. A dropped connection keeps the
interface and live artifact documents mounted, escalates after three
failed reconnect attempts, and recovers automatically. Authorization
failures stop retries; a displayed upgrade gate stays until reload.
- The desktop main process owns connection checks, persistence,
cancellation, retries and navigation. The packaged page paints Connected
before navigation; only that page can call the connection IPC. A failed
remote page load returns to saved-connection recovery.
- Setup and local dialogs share the production controls and modal
presentation, with bundled Clouds, fonts, icons and their notices. The
dialog drag strip sits inside the native dialog so it remains usable
during modality. The served app composes the upgrade gate, keeping its
markdown dependencies out of the local renderer. Large controls also
reach canonical v2; frozen v1 stays unchanged.
- The administrator guide teaches connect links and `tv links` and gives
Mac or browser instructions appropriate to the person's computer.
PR-authoring guidance defines the reader, useful validation evidence and
where working records belong.

Server APIs, WebSocket messages, served-page parameters, the remote
native bridge and `connection.json` retain their contracts. The required
desktop version remains **1.3.1**, and the recommendation remains
**1.4.0**. Older desktop apps can still reconnect, though the served
page's recovery instruction names **Disconnect from Server** while those
apps retain **Connect to server…**.

**Validation.** `npm run verify -- blaxel` passed at `6cd71e7c`,
covering static checks and the CLI, browser and real-Electron suites.
The daemon-acceptance suite also passed real installation, refresh,
boot, stop and uninstall. Subsequent commits change only documentation
and developer guidance. The theme-reset test exercises the built
application with its existing assertions, timeouts and retry settings;
[PR #12](#12)
addresses Chromium response buffers exhausting RAM-backed `/tmp` during
repeated reloads of unbundled pages on Blaxel.

**Signed desktop product checks.** The [candidate
build](https://github.com/telepath-computer/television/actions/runs/36935934106)
is **261001hquaj28w0**, version **1.4.15**, built from
**6cd71e7c8dc3869651bfafab6f22ced522f1a73a** without a version override.
It remains unreleased: [Apple Silicon
DMG](https://download.todesktop.com/260923p52umxx/Television%201.4.15%20-%20Build%20261001hquaj28w0-arm64.dmg),
[Mac
zip](https://download.todesktop.com/260923p52umxx/Television%201.4.15%20-%20Build%20261001hquaj28w0-arm64-mac.zip).
The current application code matches that build.

Josh ran the following checks on his Mac against servers running the PR
code:

| Check | Steps and observed result |
| --- | --- |
| Mac install — PASS | Installed and opened the candidate without a
Gatekeeper warning. Dock and application menu showed Television; About
showed **1.4.15**. The earlier saved connection opened successfully. |
| Connect screen — PASS | Copy reached the real system clipboard; window
dragging worked. A rejected token and an unreachable address produced
validation errors. A valid connect link connected successfully. |
| Bridge and IPC — PASS | **Disconnect from Server** worked through the
menu and **⌘,**, and was disabled on setup. Launching with the server
down showed the local error page and recovered when the server returned.
Dropping an established connection showed **Disconnected**, then **Can't
connect with server**, and recovered automatically. Token rejection
showed **Access token required** on both the served and local pages; the
local Disconnect button worked, and a new link restored the connection.
|
| Updates — PASS for notice and restart | ToDesktop's simulated
**update-available** scenario showed the desktop update notice.
**Restart to update** quit and reopened the app. |
| Upgrade gate — PASS | Launched the same **1.4.15** candidate with
`--runtime-simulate-updates=update-available` and connected to a server
run from source at `8e7cd67b` with `TV_TEST_VERSION=1.4.15` and
`TV_TEST_REQUIRED_DESKTOP_VERSION=2.0.0`. The gate appeared initially
without **Restart to update**, then showed it once the simulated update
was ready. The top strip dragged the window while gated. **Television ›
Disconnect from Server** returned to the connect screen. |

An agent checked the same build's Mac zip: **Electron license files and
bundled notices PASS**. The agent extracted and read
`Contents/Resources/LICENSE.electron.txt` and
`Contents/Resources/LICENSES.chromium.html`, confirming Electron's MIT
license and Chromium's credits are present and readable. Inside
`Contents/Resources/app.asar`, `dist/THIRD-PARTY-NOTICES.txt` and
`dist/clouds/THIRD-PARTY-NOTICES.txt` matched the corresponding build
outputs byte for byte. The desktop notices include Phosphor Icons,
Hind's font license, and the Clouds photo and palette terms; the Clouds
folder carries its own photo and palette terms. The app's plist and
packaged manifest report **1.4.15**.

**Not checked on the signed build:** appearance switching and Reduce
Motion.

A server whose connection check succeeds but whose page repeatedly fails
to load can cycle through saved-connection recovery without backoff;
Disconnect from Server remains available. Local appearance follows the
Mac at launch and retains the server's native appearance input during
that run; persistence across launches is tracked in
[TV-912](https://linear.app/telepath-computer/issue/TV-912).
Specify coordinator renewal, owned-process cleanup before reuse, and the disposition of abandoned reports. Narrow built-app conversion to two files after the Vite response fix, and address proposal review refinements on scratch cleanup, old branches, browser evidence, and remote credentials.
Policy reconciliation:
- This branch has investigation, proposal, and review documents only. Its
  proposed runner behavior remains unimplemented; retain that distinction.
- Preserve main's built theme-reset acceptance and full Vite responses. The
  resource proposal must assess remaining conversions against both changes.
- Preserve main's demo-mode fixture connection cleanup: that flake has a
  diagnosed teardown cause and is not evidence for resource exhaustion.
- Preserve origin-repository checkout, SDK runtime assets, printed connect
  links, single-origin app loading, and the stable-front proxy API. The five
  proposed browser conversions still need their declared restart boundaries.
- Main's rewritten spec policy governs derivation: keep decisions and external
  contracts in specs, with implementation and proof detail in derived work.
- Retain main's artifact isolation, resources, placement, desktop, UI,
  onboarding, release, and staging changes. This branch has no executable
  changes to reconcile against them; removed staging commands are not part
  of the contribution's validation plan.

Git reported no textual conflicts. Reviewed both histories and the affected
policies and harness changes. No tests, builds, or verification ran: this host
forbids execution. Live validation remains due on a pushed Blaxel revision.
Derive the spec checkpoint from the converged cleanup proposal. Reserve
Blaxel workers through owned-process teardown and report retrieval, recover
abandoned runs after lost renewal, and restrict temporary-file cleanup to
known test leftovers and per-attempt scratch. Define bounded resource
evidence, reusable worker build lifetimes, and the scoped built-app default.

Apply the independent proposal review's bounded refinements: one 20-minute
recovery interval using the existing option, an explicit deletion path for
abandoned workers that fail cleanup, legacy-work skipping, and provisioning
exclusion. Preserve the review and record the derivation and its limits.

No proofs or implementation changes. Source/policy review and Git diff
checks only; no tests or builds ran. Independent spec review is next, with
Josh's spec acceptance at PR time and execution evidence later on Blaxel.
@ThopterBot ThopterBot changed the title Clean up test runtimes and retain Blaxel resource diagnostics Clean up test runtimes and protect Blaxel worker lifecycles Oct 27, 2026
@ThopterBot
ThopterBot marked this pull request as ready for review October 27, 2026 21:52
@ThopterBot
ThopterBot marked this pull request as draft October 27, 2026 22:07
Remove the completed validation checkpoint under pre-merge docs prep. The private Blaxel worker-lease and lease-contract files passed all 20 tests at 704f484 after reproducing the permission failure at 6b5721c. The proof retains the scope and limits of the added process-permission cases.
@ThopterBot
ThopterBot marked this pull request as ready for review October 27, 2026 22:14
@ThopterBot
ThopterBot marked this pull request as draft October 27, 2026 22:18
Blaxel red/green: 294f256 reproduces the unreadable same-user process; 2156f26 passes all 20 worker-lease and lease-contract cases with default retries, no failures, leaks or infrastructure failures. Retained run directories are 2026-10-27T22-22-48-239Z-p2976-re22b246ab160f1ce and 2026-10-27T22-23-18-728Z-p3090-r6956a6b83ed3f866. Remove the completed checkpoint under docs prep and a trailing blank line.
@ThopterBot
ThopterBot marked this pull request as ready for review October 27, 2026 22:24
jwhiting added a commit that referenced this pull request Oct 28, 2026
Lists the full Blaxel verifications of bcc65d9 and 8e6cfcb, the targeted and
planned Blaxel probes of the reconciled targeted path on 8e6cfcb, and the
source PRs' live evidence that carries forward unchanged.
jwhiting added a commit that referenced this pull request Oct 28, 2026
Pre-merge docs prep. The ledger and the removal account move to
docs/archive/2026-10/test-runner-spec-rewrite/ with preambles: they
hold why each statement in the rewritten specs is there, how each
review finding was answered, and what became of each passage of the
old specs, none of which the diff records. The review reports and the
merge validation note were checkpoints and are deleted, leaving
docs/working/ empty.

The four documents archived on the #49 and #12 branches now name this
pull request, which is the one that merges them.
@ThopterBot

Copy link
Copy Markdown
Collaborator Author

Superseded by #53, which integrates this change with the test-runner spec rewrite.

@ThopterBot ThopterBot closed this Oct 28, 2026
jwhiting added a commit that referenced this pull request Oct 29, 2026
…e the test-runner specs (#53)

Supersedes #49 and #12.

Several agents often share one development host, and their concurrent
test runs exhaust its memory and get processes killed. On Blaxel,
persistent workers collect generated test runtimes in memory-backed
`/tmp`, and a worker's lock can expire, or be released before the run's
reports are downloaded, without proof that the run's processes stopped.
This change brings together #49's local guardrails and #12's Blaxel
worker leases, cleanup and diagnostics. It also rewrites the specs under
`specs/arch/test-runner/` from an empty page, so that they state only
what the code cannot be trusted to settle on its own.

## What changes

- **Local guardrails** (`scripts/test/`). Local runs and local verify
take a mutex shared by all of an operating-system user's checkouts. On
hosts marked with `~/.tvdev-use-blaxel`, a local run selects at most one
test file, and broader work goes to Blaxel. The runner hands Vitest and
Playwright exact files, broad zero-retry runs need an override, and
direct `vitest` or `playwright` invocations refuse. Each refusal exits
`2` and names the command to run instead, and each guardrail has an
override whose name contains `against-test-guidance`.
- **Blaxel worker leases.** A shard attempt leases its worker from
before checkout until its processes are confirmed gone and its reports
are downloaded. An abandoned lease is taken over only after its
processes are stopped, and pool deletion refuses leased workers.
- **Temporary files and diagnostics.** Each attempt gets private scratch
space on its worker, generated runtimes are cleaned up, and reusable
test builds last as long as their test worker. Each Blaxel surface
execution records memory, shared memory, free temporary space and
out-of-memory kills. These records never change a test outcome.
- **Live qualification.** `test/runner-fixtures/blaxel-harness/`
exercises leases, recovery and deletion against disposable workers. It
runs as two experiment surfaces outside `all` and CI.
- **Specs.** The nine test-runner specs (28,168 words) become six
(10,627 words), and `preflight.md`, `flaky-tests.md` and
`test-registry.md` are deleted. The proofs, references across the
repository and contributor guidance follow the new layout.

## Where to look

**Spec and guidance edits outside `specs/arch/test-runner/`.** These
come from two sources.

- *From #49 and #12:*
- the contributor guidance in `AGENTS.md`,
`specs/arch/testing-policy.md`, `specs/spec-workflow.md` and the
`tvdev-contribute` and `tvdev-review` skill sources;
- commands the guardrails accept in `specs/arch/desktop/e2e-harness.md`
and `specs/arch/telemetry/sink.md`;
  - the *marked host* term in `specs/terms.md`.
- *From the rewrite:*
- Links in twelve files now point to where their content lives. Where a
link's target changed, the words around it change to match.
- `specs/product/desktop-app.md` cited preflight's testing guidance to
explain its required Mac verify run. It now says what that run
exercises: the runner's macOS process backend.
- `specs/terms.md` points each test-runner term at its new owner. It
redefines *broad run* as a run that selects more than one test file, and
restates *preflight*, *shard*, *attempt* and *infrastructure failure* to
match the rewritten specs. *Failed shard* and *recovered flake* replace
*test failure* and *flaky test*. *Execution group* and *normalized
surface result* are dropped, because no spec defines them.

**Code and test scan.** These are the parts most likely to raise
questions about scope or cost:

- **New production modules in `scripts/test/`:**
  - `local-test-mutex.mjs`, which uses Node's built-in SQLite;
- `blaxel-worker-lease.mjs`, `blaxel-lease-lifecycle.mjs` and
`blaxel-pool-lifecycle.mjs`;
  - `resource-diagnostics.mjs` and `test-guidance.mjs`.

  No dependency is added.
- **New test machinery:**
  - the qualification fixture;
  - the guidance fixtures in `test/runner-fixtures/guidance/`;
- `test/helpers/worker-builds.ts` and
`test/helpers/application-requests.ts`.
- **Edits outside the runner's own files.** Every registered Vitest and
Playwright config gains one import, through which a direct invocation
refuses. Five browser acceptances now load the built application through
`ProductServer`.
- **What the rewrite changed in code, tests and proofs:**
- *Code:* one behavior change. `scripts/test/cli.mjs` checks a Blaxel
selection and `--run-dir-output` before creating the run directory, so a
refusal leaves no run directory behind.
- *Tests:* four new refusal cases, run in a throwaway checkout of the
repository; checks on the `summary.json` fields the spec pins; and two
existing tests that now read the rewritten spec text.
- *Proofs:* the proofs for the registry, preflight and flaky tests
become sections of `proofs/arch/test-runner/test-runner.md`. No
assertion is removed. Assertions about detail the specs leave to the
code are marked *superfluous*.

## Validation

`npm run verify -- blaxel` passed at the head, `a2c4f922`: 22 surfaces
on 36 shards, 3,812 tests passed, 8 skipped and none failed, with no
recovered flakes, process leaks or infrastructure failures. It published
the head tree's attestation. [GitHub CI run
40838234780](https://github.com/telepath-computer/television/actions/runs/40838234780)
passed at the head. Lint, type-check, the package manifest checks and
the vibe-mode check ran, and the heavy jobs skipped because of that
attestation. Blaxel has no Firefox, so the Firefox projects ran in
neither (TV-1011).

## Detail


[`docs/archive/2026-10/test-runner-spec-rewrite/change-detail.md`](https://github.com/telepath-computer/television/blob/a2c4f9220f061e85086fa435a4241a97a36055b8/docs/archive/2026-10/test-runner-spec-rewrite/change-detail.md)
describes how each mechanism works, the judgment calls in the spec
rewrite and the proof and test repairs. It also holds the full
validation record, including live Blaxel probes of targeted runs and the
evidence carried over from #49 and #12. In the same folder, `ledger.md`
gives the reason for each statement in the six specs, and
`removal-account.md` says what became of each passage of the nine specs
they replace.

---------

Co-authored-by: Josh Whiting <167151+jwhiting@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants