Repository navigation
Clean up test runtimes and protect Blaxel worker lifecycles - #12
Closed
ThopterBot wants to merge 65 commits into
Closed
ThopterBot wants to merge 65 commits into
ThopterBot wants to merge 65 commits into
Conversation
jwhiting
added a commit
that referenced
this pull request
Oct 1, 2026
Connect the Mac app with one link containing the server address and, when required, its access token. The setup screen supplies a prompt for the person's agent and a link field. A successful connection is saved for later launches; **Television › Disconnect from Server** (⌘,) forgets it and returns to setup, including from the upgrade gate. - `tv links` checks the running server's authentication requirements and prints its connect links. CLI startup and `tv links` use terminal hyperlinks for interactive output and plain URLs for pipes. - Connection dialogs distinguish an unreachable server from a rejected token or required desktop upgrade. A dropped connection keeps the interface and live artifact documents mounted, escalates after three failed reconnect attempts, and recovers automatically. Authorization failures stop retries; a displayed upgrade gate stays until reload. - The desktop main process owns connection checks, persistence, cancellation, retries and navigation. The packaged page paints Connected before navigation; only that page can call the connection IPC. A failed remote page load returns to saved-connection recovery. - Setup and local dialogs share the production controls and modal presentation, with bundled Clouds, fonts, icons and their notices. The dialog drag strip sits inside the native dialog so it remains usable during modality. The served app composes the upgrade gate, keeping its markdown dependencies out of the local renderer. Large controls also reach canonical v2; frozen v1 stays unchanged. - The administrator guide teaches connect links and `tv links` and gives Mac or browser instructions appropriate to the person's computer. PR-authoring guidance defines the reader, useful validation evidence and where working records belong. Server APIs, WebSocket messages, served-page parameters, the remote native bridge and `connection.json` retain their contracts. The required desktop version remains **1.3.1**, and the recommendation remains **1.4.0**. Older desktop apps can still reconnect, though the served page's recovery instruction names **Disconnect from Server** while those apps retain **Connect to server…**. **Validation.** `npm run verify -- blaxel` passed at `6cd71e7c`, covering static checks and the CLI, browser and real-Electron suites. The daemon-acceptance suite also passed real installation, refresh, boot, stop and uninstall. Subsequent commits change only documentation and developer guidance. The theme-reset test exercises the built application with its existing assertions, timeouts and retry settings; [PR #12](#12) addresses Chromium response buffers exhausting RAM-backed `/tmp` during repeated reloads of unbundled pages on Blaxel. **Signed desktop product checks.** The [candidate build](https://github.com/telepath-computer/television/actions/runs/36935934106) is **261001hquaj28w0**, version **1.4.15**, built from **6cd71e7c8dc3869651bfafab6f22ced522f1a73a** without a version override. It remains unreleased: [Apple Silicon DMG](https://download.todesktop.com/260923p52umxx/Television%201.4.15%20-%20Build%20261001hquaj28w0-arm64.dmg), [Mac zip](https://download.todesktop.com/260923p52umxx/Television%201.4.15%20-%20Build%20261001hquaj28w0-arm64-mac.zip). The current application code matches that build. Josh ran the following checks on his Mac against servers running the PR code: | Check | Steps and observed result | | --- | --- | | Mac install — PASS | Installed and opened the candidate without a Gatekeeper warning. Dock and application menu showed Television; About showed **1.4.15**. The earlier saved connection opened successfully. | | Connect screen — PASS | Copy reached the real system clipboard; window dragging worked. A rejected token and an unreachable address produced validation errors. A valid connect link connected successfully. | | Bridge and IPC — PASS | **Disconnect from Server** worked through the menu and **⌘,**, and was disabled on setup. Launching with the server down showed the local error page and recovered when the server returned. Dropping an established connection showed **Disconnected**, then **Can't connect with server**, and recovered automatically. Token rejection showed **Access token required** on both the served and local pages; the local Disconnect button worked, and a new link restored the connection. | | Updates — PASS for notice and restart | ToDesktop's simulated **update-available** scenario showed the desktop update notice. **Restart to update** quit and reopened the app. | | Upgrade gate — PASS | Launched the same **1.4.15** candidate with `--runtime-simulate-updates=update-available` and connected to a server run from source at `8e7cd67b` with `TV_TEST_VERSION=1.4.15` and `TV_TEST_REQUIRED_DESKTOP_VERSION=2.0.0`. The gate appeared initially without **Restart to update**, then showed it once the simulated update was ready. The top strip dragged the window while gated. **Television › Disconnect from Server** returned to the connect screen. | An agent checked the same build's Mac zip: **Electron license files and bundled notices PASS**. The agent extracted and read `Contents/Resources/LICENSE.electron.txt` and `Contents/Resources/LICENSES.chromium.html`, confirming Electron's MIT license and Chromium's credits are present and readable. Inside `Contents/Resources/app.asar`, `dist/THIRD-PARTY-NOTICES.txt` and `dist/clouds/THIRD-PARTY-NOTICES.txt` matched the corresponding build outputs byte for byte. The desktop notices include Phosphor Icons, Hind's font license, and the Clouds photo and palette terms; the Clouds folder carries its own photo and palette terms. The app's plist and packaged manifest report **1.4.15**. **Not checked on the signed build:** appearance switching and Reduce Motion. A server whose connection check succeeds but whose page repeatedly fails to load can cycle through saved-connection recovery without backoff; Disconnect from Server remains available. Local appearance follows the Mac at launch and retains the server's native appearance input during that run; persistence across launches is tracked in [TV-912](https://linear.app/telepath-computer/issue/TV-912).
Specify coordinator renewal, owned-process cleanup before reuse, and the disposition of abandoned reports. Narrow built-app conversion to two files after the Vite response fix, and address proposal review refinements on scratch cleanup, old branches, browser evidence, and remote credentials.
Policy reconciliation: - This branch has investigation, proposal, and review documents only. Its proposed runner behavior remains unimplemented; retain that distinction. - Preserve main's built theme-reset acceptance and full Vite responses. The resource proposal must assess remaining conversions against both changes. - Preserve main's demo-mode fixture connection cleanup: that flake has a diagnosed teardown cause and is not evidence for resource exhaustion. - Preserve origin-repository checkout, SDK runtime assets, printed connect links, single-origin app loading, and the stable-front proxy API. The five proposed browser conversions still need their declared restart boundaries. - Main's rewritten spec policy governs derivation: keep decisions and external contracts in specs, with implementation and proof detail in derived work. - Retain main's artifact isolation, resources, placement, desktop, UI, onboarding, release, and staging changes. This branch has no executable changes to reconcile against them; removed staging commands are not part of the contribution's validation plan. Git reported no textual conflicts. Reviewed both histories and the affected policies and harness changes. No tests, builds, or verification ran: this host forbids execution. Live validation remains due on a pushed Blaxel revision.
Derive the spec checkpoint from the converged cleanup proposal. Reserve Blaxel workers through owned-process teardown and report retrieval, recover abandoned runs after lost renewal, and restrict temporary-file cleanup to known test leftovers and per-attempt scratch. Define bounded resource evidence, reusable worker build lifetimes, and the scoped built-app default. Apply the independent proposal review's bounded refinements: one 20-minute recovery interval using the existing option, an explicit deletion path for abandoned workers that fail cleanup, legacy-work skipping, and provisioning exclusion. Preserve the review and record the derivation and its limits. No proofs or implementation changes. Source/policy review and Git diff checks only; no tests or builds ran. Independent spec review is next, with Josh's spec acceptance at PR time and execution evidence later on Blaxel.
Archive the proposal and plan because they preserve the design rationale and the coherent validation structure. Delete completed investigation notebooks, review findings and checkpoint records under the docs policy; retained run artifacts and the PR description carry validation evidence.
ThopterBot
marked this pull request as ready for review
October 27, 2026 21:52
ThopterBot
marked this pull request as draft
October 27, 2026 22:07
ThopterBot
marked this pull request as ready for review
October 27, 2026 22:14
ThopterBot
marked this pull request as draft
October 27, 2026 22:18
Blaxel red/green: 294f256 reproduces the unreadable same-user process; 2156f26 passes all 20 worker-lease and lease-contract cases with default retries, no failures, leaks or infrastructure failures. Retained run directories are 2026-10-27T22-22-48-239Z-p2976-re22b246ab160f1ce and 2026-10-27T22-23-18-728Z-p3090-r6956a6b83ed3f866. Remove the completed checkpoint under docs prep and a trailing blank line.
ThopterBot
marked this pull request as ready for review
October 27, 2026 22:24
jwhiting
added a commit
that referenced
this pull request
Oct 28, 2026
Pre-merge docs prep. The ledger and the removal account move to docs/archive/2026-10/test-runner-spec-rewrite/ with preambles: they hold why each statement in the rewritten specs is there, how each review finding was answered, and what became of each passage of the old specs, none of which the diff records. The review reports and the merge validation note were checkpoints and are deleted, leaving docs/working/ empty. The four documents archived on the #49 and #12 branches now name this pull request, which is the one that merges them.
Collaborator
Author
|
Superseded by #53, which integrates this change with the test-runner spec rewrite. |
jwhiting
added a commit
that referenced
this pull request
Oct 29, 2026
…e the test-runner specs (#53) Supersedes #49 and #12. Several agents often share one development host, and their concurrent test runs exhaust its memory and get processes killed. On Blaxel, persistent workers collect generated test runtimes in memory-backed `/tmp`, and a worker's lock can expire, or be released before the run's reports are downloaded, without proof that the run's processes stopped. This change brings together #49's local guardrails and #12's Blaxel worker leases, cleanup and diagnostics. It also rewrites the specs under `specs/arch/test-runner/` from an empty page, so that they state only what the code cannot be trusted to settle on its own. ## What changes - **Local guardrails** (`scripts/test/`). Local runs and local verify take a mutex shared by all of an operating-system user's checkouts. On hosts marked with `~/.tvdev-use-blaxel`, a local run selects at most one test file, and broader work goes to Blaxel. The runner hands Vitest and Playwright exact files, broad zero-retry runs need an override, and direct `vitest` or `playwright` invocations refuse. Each refusal exits `2` and names the command to run instead, and each guardrail has an override whose name contains `against-test-guidance`. - **Blaxel worker leases.** A shard attempt leases its worker from before checkout until its processes are confirmed gone and its reports are downloaded. An abandoned lease is taken over only after its processes are stopped, and pool deletion refuses leased workers. - **Temporary files and diagnostics.** Each attempt gets private scratch space on its worker, generated runtimes are cleaned up, and reusable test builds last as long as their test worker. Each Blaxel surface execution records memory, shared memory, free temporary space and out-of-memory kills. These records never change a test outcome. - **Live qualification.** `test/runner-fixtures/blaxel-harness/` exercises leases, recovery and deletion against disposable workers. It runs as two experiment surfaces outside `all` and CI. - **Specs.** The nine test-runner specs (28,168 words) become six (10,627 words), and `preflight.md`, `flaky-tests.md` and `test-registry.md` are deleted. The proofs, references across the repository and contributor guidance follow the new layout. ## Where to look **Spec and guidance edits outside `specs/arch/test-runner/`.** These come from two sources. - *From #49 and #12:* - the contributor guidance in `AGENTS.md`, `specs/arch/testing-policy.md`, `specs/spec-workflow.md` and the `tvdev-contribute` and `tvdev-review` skill sources; - commands the guardrails accept in `specs/arch/desktop/e2e-harness.md` and `specs/arch/telemetry/sink.md`; - the *marked host* term in `specs/terms.md`. - *From the rewrite:* - Links in twelve files now point to where their content lives. Where a link's target changed, the words around it change to match. - `specs/product/desktop-app.md` cited preflight's testing guidance to explain its required Mac verify run. It now says what that run exercises: the runner's macOS process backend. - `specs/terms.md` points each test-runner term at its new owner. It redefines *broad run* as a run that selects more than one test file, and restates *preflight*, *shard*, *attempt* and *infrastructure failure* to match the rewritten specs. *Failed shard* and *recovered flake* replace *test failure* and *flaky test*. *Execution group* and *normalized surface result* are dropped, because no spec defines them. **Code and test scan.** These are the parts most likely to raise questions about scope or cost: - **New production modules in `scripts/test/`:** - `local-test-mutex.mjs`, which uses Node's built-in SQLite; - `blaxel-worker-lease.mjs`, `blaxel-lease-lifecycle.mjs` and `blaxel-pool-lifecycle.mjs`; - `resource-diagnostics.mjs` and `test-guidance.mjs`. No dependency is added. - **New test machinery:** - the qualification fixture; - the guidance fixtures in `test/runner-fixtures/guidance/`; - `test/helpers/worker-builds.ts` and `test/helpers/application-requests.ts`. - **Edits outside the runner's own files.** Every registered Vitest and Playwright config gains one import, through which a direct invocation refuses. Five browser acceptances now load the built application through `ProductServer`. - **What the rewrite changed in code, tests and proofs:** - *Code:* one behavior change. `scripts/test/cli.mjs` checks a Blaxel selection and `--run-dir-output` before creating the run directory, so a refusal leaves no run directory behind. - *Tests:* four new refusal cases, run in a throwaway checkout of the repository; checks on the `summary.json` fields the spec pins; and two existing tests that now read the rewritten spec text. - *Proofs:* the proofs for the registry, preflight and flaky tests become sections of `proofs/arch/test-runner/test-runner.md`. No assertion is removed. Assertions about detail the specs leave to the code are marked *superfluous*. ## Validation `npm run verify -- blaxel` passed at the head, `a2c4f922`: 22 surfaces on 36 shards, 3,812 tests passed, 8 skipped and none failed, with no recovered flakes, process leaks or infrastructure failures. It published the head tree's attestation. [GitHub CI run 40838234780](https://github.com/telepath-computer/television/actions/runs/40838234780) passed at the head. Lint, type-check, the package manifest checks and the vibe-mode check ran, and the heavy jobs skipped because of that attestation. Blaxel has no Firefox, so the Firefox projects ran in neither (TV-1011). ## Detail [`docs/archive/2026-10/test-runner-spec-rewrite/change-detail.md`](https://github.com/telepath-computer/television/blob/a2c4f9220f061e85086fa435a4241a97a36055b8/docs/archive/2026-10/test-runner-spec-rewrite/change-detail.md) describes how each mechanism works, the judgment calls in the spec rewrite and the proof and test repairs. It also holds the full validation record, including live Blaxel probes of targeted runs and the evidence carried over from #49 and #12. In the same folder, `ledger.md` gives the reason for each statement in the six specs, and `removal-account.md` says what became of each passage of the nine specs they replace. --------- Co-authored-by: Josh Whiting <167151+jwhiting@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Persistent Blaxel workers accumulate generated source-CLI and web-bundle runtimes. The worker lock also expires without proving that descendants stopped and releases before report downloads finish, allowing the next checkout to overwrite results. This PR gives generated builds a worker lifetime and protects each Blaxel attempt through cleanup and verified retrieval.
Reusable CLI and versioned web builds publish only after preparation succeeds. Playwright and Vitest worker teardown stop servers, wait for pending builds, and remove their outputs while preserving reuse across cases. Failed preparation and per-launch copies clean themselves up. Five browser acceptances—two theme lifecycle walks and three demo-mode cases—load built application assets while retaining their authentication, restart, navigation and rendering assertions. Their request inventories establish the shipped-asset boundary; incremental memory savings after Vite's full-response correction are unmeasured.
The Blaxel lifecycle uses one worker-local lease, serialized by a short
flockguard. Dispatch registers ownership before checkout, and only the coordinator renews. Normal release follows attributed-process teardown and validated report download. After 20 minutes without renewal by default, the next acquisition stops the abandoned attempt's processes by owner token and PID/start identity, audits, clears its state and takes the lease. Recovery can discard undownloaded reports and retains a receipt describing that loss. Uncertain or unreadable ownership holds the worker. Pool provisioning publishes workers only after verification; deletion reserves them and removes dispatch eligibility before calling the provider.Each attempt has private scratch and an inherited
TMPDIR. Under the lease and a clean owner audit, pre-run cleanup removes only direct/tmp/television-source-cli-e2e-*,/tmp/television-web-bundle-*directories and stale runner scratch. Post-run cleanup removes the attempt's scratch after processes stop. Persistent caches and unrelated temporary files are preserved. This also reclaims known residue from test revisions without the helper fixes.Surface parents retain available RAM, shared memory, temporary free space and OOM-counter observations at start/end, plus bounded kernel OOM lines for failed or recovered native attempts. Diagnostic links survive normalization, retry aggregation and download. Limits are 64 KiB of kernel lines, 128 KiB per sidecar and 8 MiB per shard attempt; unavailable probes stay explicit and do not change native outcomes.
The governing changes are in Blaxel shards, reporting, testing policy, and the runner/registry placement rules, with mirrored proofs. The scope is test infrastructure and acceptance setup. The archived proposal and plan preserve rationale and validation structure.
Validation:
all/CI.npm run verify -- blaxel --pool pocpassed at7c66d160, containing maina8e73fbf: all 36 shards and 22 surfaces; 3,705 tests passed, eight skipped, zero failed/recovered tests, process leaks or infrastructure failures. Lint, type checking and package manifests passed. The gate retained 231 linked resource sidecars with complete start/end observations and zero OOM deltas. Normal cleanup reclaimed 1,510 runtime directories totaling 5,003,157,504 allocated bytes. Peak memory and long-term renewal-loss frequency remain unmeasured.2156f266. Process-permission coverage runs the real command unprivileged and proves that unreadable production audits preserve the lease and leftovers. A root-only case checks a same-user nondumpable child; fixture process enumeration is explicitly supplied, while production audits the whole worker. Both runs retained their results and reported zero failures, leaks or infrastructure failures. A renewal already in flight at release can produce a closing warning after its lease is gone. All private workers and the coordinator were deleted after artifact retrieval.9787188d, including the full unit suite, desktop tests, all 16 e2e shards, static checks and attestation publication. The unprivileged unit job passed all nine applicable lease-fixture cases; the root-only permission case passed on Blaxel.