diff --git a/docs/archive/2026-10/test-harness-resource-cleanup-plan.md b/docs/archive/2026-10/test-harness-resource-cleanup-plan.md new file mode 100644 index 00000000..e73c6596 --- /dev/null +++ b/docs/archive/2026-10/test-harness-resource-cleanup-plan.md @@ -0,0 +1,98 @@ +> **Archived 2026-10 from PR #12.** This plan's three slices and private qualification followed by standard shared-pool validation were completed. It preserves one coherent view of the derivation boundaries and why the two-worker full gate was rejected in favor of the standard partition. The body below is unchanged from its working state and is a clue to the change, not a record of it. + +# Test-harness resource cleanup: implementation plan + +Owner: rose. Branch: `thopter/test-harness-resource-cleanup`. Draft PR #12. Derived from the converged [proposal](test-harness-resource-cleanup-proposal.md), [spec checkpoint](test-harness-resource-cleanup-review-response.md#Spec round 1: convergence and proof derivation), and [proof round 2](test-harness-resource-cleanup-proof-review-r2.md) at `3fa15d47`. Status: [plan round 2](test-harness-resource-cleanup-plan-review-r2.md) passed at `04a6b275`; [slice 1 has converged](test-harness-resource-cleanup-slice-1-review-r1.md), and [slice 2 has converged](test-harness-resource-cleanup-slice-2-review-r1.md). [Slice 3 has converged](test-harness-resource-cleanup-slice-3-review-r2.md) at `dc1d9f2e`. [Integrated validation](test-harness-resource-cleanup-integrated.md) passed; independent integrated review is next. The supervisor arranges reviews. Josh reviews the spec deltas at PR time. + +## Slice judgment + +Use three slices, each with tests and implementation reviewed together. Helper ownership and shipped-asset loading can be completed independently of the remote coordinator. Diagnostic capture has a separate native-runner/reporting boundary. Lease acquisition, dispatch registration, recovery, scratch cleanup, pool mutation and result retrieval must remain together: splitting that lifecycle would leave intermediate combinations whose safety is hard to assess. + +The order is helper lifetime, diagnostics, then worker lifecycle and live qualification. The last slice uses the earlier diagnostics and helper fixes on real workers, so the remote acceptance runs prove their composition once. Each slice receives independent implementation review, followed by review and full verification of the integrated result. This is one PR; these are review boundaries, not separate releases. + +The specs and proofs remain the source of requirements. Paths below identify the current implementation areas, not a fixed inventory of files that may change. No production application behavior, retry default, standard shard count or browser version changes belong here. + +## Execution and evidence + +Nothing in this plan runs tests, builds, lint, type checks, browsers or servers on the shared development host. Edit, commit and push each red or green revision there. Every execution uses that pushed revision on Blaxel through the canonical runner. A failed import, missing dependency, unavailable credential or setup failure is not the intended red result: establish that the narrow test reaches the missing behavior before counting it as red. + +Use the named `tv-cleanup-coordinator-rose` sandbox outside all worker-pool labels and private pool `rose-cleanup`, with workers `tv-testshard-x64-rose-cleanup-01` and `-02`. Inventory, credentials, architecture, runtime and available capacity must be established before execution. The documented 72-worker shared pool plus these three sandboxes would reach the documented 75-concurrent-shard limit if every sandbox counted; whether idle sandboxes count is unknown. Inspect actual workspace occupancy and limits before provisioning, and report insufficient capacity without changing shared workers. The coordinator needs the repository's selected Node, dependencies and the local preflight capabilities required by the selected driver. The proposal owns ephemeral credential injection and credential-free logs. Focused development runs and every worker experiment use the private pool. Ordinary full verification uses shared `poc` workers only after private qualification and deletion, as specified below. + +Run ordinary focused tests from the coordinator as `npm test -- blaxel --pool rose-cleanup --file `, adding `--grep ` when narrowing a known failure. Expand to affected complete files and directly affected helper callers there. A small whole-surface run also names `--pool rose-cleanup`; `--surface` always targets one worker even when `--shards` is supplied. Leave whole-surface coverage of large selections, including `e2e:browser-app`, to the standard-shard full gate below. Its recorded file medians alone total about 376 seconds, leaving little of the 480-second targeted-worker timeout for setup and retries. Keep the default native and infrastructure retry budgets. Use a shortened existing recovery interval only for the declared abandoned-run experiments; it changes no production default. Slice 3 adds the canonical forwarding needed for that explicit override. + +Run the live driver on the coordinator as `TV_BLAXEL_HARNESS_COORDINATOR=tv-cleanup-coordinator-rose TV_BLAXEL_HARNESS_POOL=rose-cleanup npm test -- local --surface experiment:blaxel-harness`. These are this contribution's resources, supplied through the reusable driver's configuration rather than fixed in its code. It calls the canonical Blaxel runner and pool manager for that private pool; it is never itself a worker selection. Do not run the entire experiment suite or any isolated-GitHub-only lifecycle fixture. Before its fresh-provisioning acceptance, retrieve earlier private-run results and delete the now-idle private pool through the pool manager. The driver owns its coordinator subprocesses and pool cleanup, and reports any ownership or cleanup it cannot confirm. + +For each slice, retain the red and green commit IDs, exact commands, canonical run-directory paths, relevant assertion failures and recovered retries in the working response. After a targeted pass, expand only to affected files/surfaces. A broad failure returns to the failing case or file; a full gate is not an iteration probe. Deliberately failing native fixtures keep their failed reports even when the driver correctly passes by observing those failures. + +## Slice 1 — worker-owned builds and built application walks + +**Result.** The source CLI and versioned web outputs belong to a test worker, remain reusable across its cases, and are removed at teardown. Failed preparation and failed per-launch copying leave no private output. Five selected browser cases load shipped assets while keeping their product assertions. + +**Implementation area.** `test/helpers/product-server.ts`, `versioned-web-bundle.ts`, the shared Playwright registration and the necessary Vitest lifecycle adapters. Reuse a small common build-lifetime owner for these two actual callers. Preserve version-keyed reuse and the web builder's existing serialization. Register ownership before preparation can fail and publish a result only after preparation succeeds. Per-case product-server disposal must not remove the worker cache. Real framework teardown, rather than a test-only disposal call, must close the lifetime. + +Convert only the persistence and executable-lifecycle cases in `packages/web/test/e2e/theme-product-acceptance.test.ts`, and the three cases sharing `packages/web/test/e2e/browser-demo-mode.test.ts`'s page setup. Use the existing stable-front proxy for restart continuity and authentication; retain backend port `0`. Serve the built unsupported-URL view in demo mode. Preserve PR #9's reset walk, the separate Settings development-module walk, other deferred callers and main's demo fixture teardown correction. + +**Red/green and review criteria.** Implement the [build-lifetime contract and real worker seam](../../proofs/arch/testing-policy.md#Worker-owned builds). Begin with narrow red cases for early publication/failed preparation and outputs surviving worker teardown. Contract builders are the declared substitutes; the Playwright and Vitest seam must build real CLI/web output, use it, reuse it within a worker, retain it after case disposal and remove it after worker teardown. Exercise the absent-bundle copy failure within that same fixture, then a valid launch using the reusable build. Put cheap contracts in the ordinary repository suite and the real build/server crossing in the existing Node e2e surface, with fixture configs owned through the existing nested-runner arrangement. + +For the five converted walks, make their built-loading boundary observable in the existing runs before changing page setup, then preserve all existing product assertions through the adaptation. Use Playwright request events in those walks to retain a request inventory showing the built page rather than the development-module graph; this adds no separate browser launch or source-text compliance scan. Confirm same-origin authenticated operation and restart/reload continuity. Grade the affected [theme](../../proofs/product/themes-and-appearance.md), [delivery](../../proofs/arch/themes/delivery.md), [artifact](../../proofs/product/artifacts.md) and [bridge](../../proofs/arch/artifact-frame/artifact-bridge.md) citations only after the real adapted walks pass. + +Expand from the new helper tests and the five cases to both affected browser files and directly affected helper callers where the change predicts impact. Whole large-surface regression coverage waits for the integrated gate under the execution rule above. Preserve existing per-case process cleanup. This slice is green for its assigned obligations; killed-worker scratch recovery remains explicitly assigned to slice 3. + +## Slice 2 — bounded diagnostic capture and report transport + +**Result.** Surface parents retain start/end resource observations and bounded failure evidence independently of the native test child's survival. Sidecars keep native outcomes, recovered retries and attempt identities intact through the reporting pipeline. + +**Implementation area.** A shared collector/writer used by the real surface parent, planned shard execution and the targeted Blaxel path; `scripts/test/reporting.mjs`, provider normalization/output aggregation, their declarations, and the coordinator's artifact download. Keep the existing native reporters and result tree. Add `resourceDiagnosticsPaths` only for retained files. A common writer handles the stated sidecar and attempt budgets, including receipts later produced by slice 3; do not add a sampler, archive or retention daemon. + +**Red/green and review criteria.** Discharge [resource bounds](../../proofs/arch/test-runner/reporting.md#^reporting-t-resource-bounds), [parent capture](../../proofs/arch/test-runner/reporting.md#^reporting-t-resource-parent), and [retained links](../../proofs/arch/test-runner/reporting.md#^reporting-t-resource-links). Start with narrow writer/normalizer red assertions, then cross the actual parent/native-runner boundary on Linux. Both planned and targeted entry points must retain a baseline before preparation, finalize after teardown, and keep a missing final sample explicit after parent interruption. Inspect real probe sources/units without asserting transient host totals or manufacturing an OOM. + +Use the reviewed injected clock, probe readings and write failures for unavailable/reset/oversized cases. Preserve 64 KiB of kernel lines at most, 128 KiB per sidecar and 8 MiB per shard attempt including receipts; preserve compact observations before optional excerpts. Exclude secrets, arbitrary environments, arguments, bodies and query strings. Diagnostic failure warns without changing the native result. Earlier attempt links survive a later successful infrastructure attempt, and missing native output cannot create a passing selected-surface result. + +The ordinary parent/child seams use the existing guarded local registry hook inside their Blaxel-hosted execution. It is never forwarded to workers. These tests use real native reports, not fabricated replacements. Expand to the affected reporting, shard-worker, provider and native-recorder tests after the new focused cases pass. + +**Explicit remaining baseline.** The real Blaxel download/reuse and abandoned-run receipt evidence is not discharged here. `^reporting-t-resource-live` stays marked pending and belongs to slice 3's shared live runs; no second paid-service driver or diagnostic-only crash scenario is added. The current lease/download race is also still present until slice 3. Do not describe this checkpoint as safe for concurrent worker reuse or as a completed contribution. + +## Slice 3 — exclusive worker lifecycle and live qualification + +**Result.** One worker remains exclusively owned from acquisition through confirmed process teardown and retained results. Lost renewal permits automatic cleanup before reuse. Scratch and known leftovers are removed only under the reviewed ownership conditions. Pool operations follow that same exclusion. Live qualification proves the whole path, including slice 2's diagnostics. + +**Implementation area.** `scripts/run-blaxel-testshards.mjs`, `manage-blaxel-testshards.mjs`, their lease/discovery/dispatch helpers, the existing process owner/audit mechanisms, and the shared/targeted runner paths. Keep acquisition and entry checks usable before checkout; they cannot depend on code in a checkout another run may still own. Preserve the outer shard-attempt token across nested surface owners. A short OS-released guard serializes worker-local ownership changes; it is never held across report downloads. + +In `scripts/test/cli.mjs`, forward an explicitly supplied `--lock-ttl-seconds` to the coordinator for the live driver's shortened recovery interval. Add a small contract beside the existing CLI/provider assertions: the explicit option reaches the delegated arguments and is absent by default. The live recovery run proves it arrives at the worker. Ordinary invocations without that option retain the spec's default interval; this does not add worker-timeout forwarding or change retry defaults. + +Keep acquisition, renewal, dispatch registration, normal release, interruption and recovery in one lifecycle. Renew through setup, execution, teardown and retrieval; remove shell-exit unlock. Normal release follows confirmed descendants and verified downloads. The one expired-renewal procedure stops attributed owners, audits, records abandoned report loss and only then reuses the worker. Preserve the reviewed legacy behavior and fail-closed cases. Pool deletion refuses valid leases and removes dispatch eligibility before deletion; fresh provisioning publishes eligibility only after successful verification. No new service, caller-attestation system or mixed-version framework is needed. + +Give each shard attempt its specified private scratch and `TMPDIR`. Route uploads, plan, askpass and setup sidecars there, finalizing evidence outside it before removal. Pre-run cleanup removes only the two known direct `/tmp` prefixes and non-current attempt directories after the lease and clean owner audit. Preserve unrelated files, links' outside targets, current scratch and caches; reject unsafe roots/mounts. Stop processes before post-run removal and retain evidence/ownership when cleanup is uncertain. + +**Red/green and review criteria.** First establish red at the [lease decision and ordering contracts](../../proofs/arch/test-runner/blaxel-testshards.md#Lease, recovery, and scratch), then implement the lifecycle and cross the real Linux guard/registration and filesystem seams. Cover competing acquisition, old-owner mutation, delayed entry, nested tokens, token/PID-start-validated termination, incomplete records, cleanup errors and permitted legacy behavior at their reviewed shapes. The ordinary guard seam restricts its audit to fixture tokens as declared; it supplies no live-provider credit. Replace the existing TTL-only assertions rather than leaving tests that endorse the superseded release behavior. Test pool publication/deletion ordering and discovery of unavailable partial workers with the declared SDK substitutions before the live pool crossing. + +Register `experiment:blaxel-harness` and `experiment:blaxel-harness-fixtures`, giving the fixture surface at least two native files. Discharge the [registry selection contract](../../proofs/arch/test-runner/test-registry.md#^registry-t-blaxel-harness-selection): neither surface enters `all`/ordinary CI, the fixture tag selects only fixtures, and the driver never dispatches itself. The fixtures include failed and recovered native outcomes, bounded owned child processes and the small real Chromium page. The driver uses the reviewed pre-download rendezvous; ordinary runs never wait there. + +After the focused contract/seam tests are green, run the real driver from the dedicated coordinator on the pushed revision. These observations share one small fresh pool: + +- Planned two-shard execution and a targeted run cross acquisition, checkout, setup, native execution and actual transport. Retain uploaded-script paths, `temporary-cleanup` phase/counts, native outcomes and diagnostics. A competing run and pool deletion are refused while download is gated and renewal continues. After retrieval, scratch is gone, allowed legacy residue is gone, unrelated sentinels/caches remain, and normal reuse works. +- Kill the actual lease-renewing coordinator in the two reviewed scenarios: an active supervisor/owned descendant, and quiet work before retrieval. After the deliberately shortened recovery interval, the next acquisition terminates the old owned processes, confirms disuse, retains the receipt and reuses the worker. No discarded report can become a pass. Do not substitute killing only the shard or merely stopping a mocked renewal callback. +- Inspect real Node-created paths and Chromium shared-memory descriptors under the attempt's `TMPDIR`, including unlinked files. Record the browser version and executable/cache path. This verifies placement without claiming peak memory or recreating exhaustion. +- The failed fixture stays failed; the recovered fixture retains all native attempts. Downloaded diagnostic links survive scratch removal, carry the correct identities and expose unavailable OOM evidence honestly. Delete the private pool through the real manager after reports are retained, confirming the provider metadata/deletion boundary. + +These discharge `^blaxel-t-retrieval`, `^blaxel-t-recovery-live`, `^blaxel-t-tmpdir-live`, `^blaxel-t-pool-live` and `^reporting-t-resource-live`. The source-level registry contract and SDK mocks do not substitute for any of them. All temporary baselines from slices 1 and 2 are resolved before this slice is called complete. + +## Integrated result and final validation + +After slice convergence, independently review the integrated specs, proofs, tests and code. Verify every new pending citation is either discharged with its real evidence or reported as unfinished; intentional fixture failures remain distinguishable from test-runner failures. Preserve the separate Settings development path, PR #9 reset, normal stale-owner protection and isolated-GitHub fixture restrictions. + +Fetch the PR target, integrate it under the logical-merging workflow if needed, then commit and push the tree before the qualifying gate. Any target movement requires validation of the updated tree. After private qualification, retain its reports and confirm the live driver's deletion of both private workers. Keep `tv-cleanup-coordinator-rose` to invoke ordinary full verification against the shared pool at the standard 36 shards: + +```sh +npm run verify -- blaxel --pool poc +``` + +This runs every `all` shard index plus lint, type checking and package-manifest validation on Blaxel and has the standard [publication-qualifying](../../specs/arch/test-runner/test-runner.md#^test-publication-qualifying) shape. The two opt-in experiment surfaces remain separately qualified by the private driver. No deliberate interruption, contamination, pool deletion or provisioning targets `poc`; ordinary acquisition, cleanup and retrieval exercise the implemented lifecycle under its intended shared-pool conditions. Unavailable leases remain an infrastructure failure, not permission to steal a lock or alter the pool. + +The committed [timing baseline](../../test/timing-baseline.json), through 2026-09-29, records 882.506 seconds across 311 Blaxel file medians: about 441 seconds per worker if divided over two, before setup and retries. The canonical runner's cold profile permits 480 seconds per worker and does not forward a worker-timeout override. These historical totals are not a measured duration for the eventual tree, but they provide too little allowance to plan a two-worker full gate. Use the existing standard partition and pool instead of adding timeout controls or private capacity for this contribution. + +Retrieve all evidence before deleting the dedicated coordinator, and report cleanup that cannot be confirmed. Retain working documents while the PR is draft; perform pre-merge docs preparation only after implementation and integrated validation are complete. Josh's spec acceptance and the repository's required GitHub checks remain merge obligations; neither this plan nor an independent PASS claims they have happened. + +## Current limits + +Slice 1's helper contracts, real framework teardown and five built-application walks have converged. Slice 2's diagnostics have converged. Slice 3's focused contracts, Linux seams and live private qualification have passed; its evidence record carries the failed attempts, fixes, final revisions and retained run paths. Both private workers were deleted through the manager and confirmed absent. Slice 3 has converged. Target reconciliation found no incoming main commits. The standard 36-shard full gate passed at `7c66d160` with no failed/recovered tests, leaks, infrastructure failures or renewal losses; integrated review remains outstanding. The integrated record names eight native skips, four skipped acquisition candidates, the non-fatal attestation publication error, retained evidence and final private-coordinator cleanup. Provider inventory differs from the historical pool count; account-specific quota, idle-sandbox accounting and the cause of one deliberately shortened lease's timing lapse remain unknown. Incremental memory savings after the Vite full-response fix remain outside the acceptance claim. diff --git a/docs/archive/2026-10/test-harness-resource-cleanup-proposal.md b/docs/archive/2026-10/test-harness-resource-cleanup-proposal.md new file mode 100644 index 00000000..03a852a5 --- /dev/null +++ b/docs/archive/2026-10/test-harness-resource-cleanup-proposal.md @@ -0,0 +1,254 @@ +> **Archived 2026-10 from PR #12.** This proposal shaped the implemented test-harness cleanup and recovery requirements. It preserves the resource investigation's quantitative context, the reasons for limiting built-application conversion to five cases, and the tradeoffs behind one renewable recovery deadline. The body below is unchanged from its working state and is a clue to the change, not a record of it. + +# Test-harness resource cleanup + +Converged proposal, revised 2026-10-27 with bounded proposal/spec review refinements. Owner: rose. Branch: `thopter/test-harness-resource-cleanup`, draft PR #12, originally based on `b5477211`. Main at `a8e73fbf` was integrated in `74d22941` before spec derivation. The supervisor arranges independent review of the proposal, specs, proofs, and implementation; Josh reviews every spec delta at PR time. No production behavior changes are proposed. + +## Problem and intended result + +In the investigated runs, repeated unbundled application loads consumed several GiB of Chromium response-buffer memory. Persistent Blaxel workers also retain temporary test runtimes across runs, reducing the memory and filesystem capacity available to browsers. The [investigation](chromium-insufficient-resources.md) establishes renderer OOM kills and reproduces `ERR_INSUFFICIENT_RESOURCES`; the exact failing allocation syscall remains unobserved. + +The fixes have different effects. The historical theme-reset walk added roughly 3.2 GB of shared memory in a passing run that reached about 0.55 GB available RAM. Built loading removes hundreds of development-module requests per load, but those historical numbers cannot predict its incremental saving after the Vite response correction described below. By comparison, the two leaked runtime prefixes occupied 1.16 GB on the worst worker inspected and 0.079 GB on the other. All of `/tmp` occupied 1.20 GB and 0.16 GB respectively. Cleanup recovers accumulated capacity; it does not guarantee browser headroom. The original worker also retained roughly 1.4 GiB of checkout/dependencies, 0.9 GiB of browser installations, and other intentional caches. Those are outside this cleanup. + +This contribution fixes the cached runtime lifetimes, moves five acceptance cases in two files onto the built application, and retains compact resource evidence when an acceptance fails. It also makes the worker reservation cover preparation, execution, process teardown, and result download, with automatic recovery when a coordinator stops renewing its reservation. After taking that lock and confirming the worker is idle, each run removes the two known leaked-runtime prefixes and stale run scratch. This repeatedly reclaims residue from older test revisions as well as newer ones. Cleanup preserves provider services, deliberate caches, and the current run; abandoned runs and reports have the bounded recovery rule below. + +PR #9 merged as `f81a2465` on 2026-10-01 and owns the built-application conversion of **“executable theme reset reloads connected Chromium documents under confirmed state.”** Preserve that integrated change; do not duplicate it. Main also includes `982e564d` (PR #32), making registry Vite services answer conditional requests in full. Its runner spec records Chromium 145 retaining about 2 MB per 304 response, roughly 465 MB per application reload. That newer mitigation changes the baseline for future measurements. The investigation numbers above describe the historical runs, not current main's residual memory cost; neither the helper leak nor the lock/download gaps is repaired by the Vite response change. + +## Evidence available before implementation + +The investigation report and its evidence were copied from `33ad4184`. Additional read-only inspections on 2026-10-01 observed: + +| Observation | `poc-131`, the original failing worker | `poc-03`, the GC experiment's worker | +| --- | ---: | ---: | +| Source-CLI runtime directories | 267 | 17 | +| Source-CLI runtimes, allocated bytes | 1,059,090,432 | 55,930,880 | +| Versioned web-bundle directories | 146 | 27 | +| Versioned web bundles, allocated bytes | 104,255,488 | 22,622,208 | +| Total `/tmp`, allocated bytes | 1,197,846,528 | 156,442,624 | +| Root filesystem available bytes at inventory | 2,137,575,424 | 3,700,719,616 | +| Active shard lock / test owner tokens | None observed | None observed | + +The [poc-131 inventory](test-harness-resource-cleanup-evidence/tv-testshard-x64-poc-131.json), [poc-03 inventory](test-harness-resource-cleanup-evidence/tv-testshard-x64-poc-03.json), and their [poc-131 sizes](test-harness-resource-cleanup-evidence/tv-testshard-x64-poc-131-sizes.json) / [poc-03 sizes](test-harness-resource-cleanup-evidence/tv-testshard-x64-poc-03-sizes.json) retain the measurements and collection methods. These are later snapshots, not reconstructions of the failed run. Inventory read filenames, process identities, relevant ownership tokens, and `/tmp` descriptor targets; it did not read fixture credentials or arbitrary process environments into artifacts. Size measurements used `du -B1 --max-depth=1 /tmp`. An earlier `du -x` attempt omitted file storage on this sandbox mount and was discarded. + +There are two proven helper leaks: + +- [`getSourceCLIEntry()`](../../test/helpers/product-server.ts) caches a path in module state after creating `television-source-cli-e2e-*`. Its cache is assigned before the build succeeds. `disposeAllProductServers()` removes per-launch homes and bundled copies, but has no cleanup for this shared runtime. +- [`buildVersionedWebBundle()`](../../test/helpers/versioned-web-bundle.ts) caches one promise per version and creates `television-web-bundle-*`, with no removal on worker completion or build failure. This is additional evidence supporting run-owned scratch space. Its worker cleanup belongs in the same contribution: it is another concrete cached test build, not a hypothetical extension. + +Other `/tmp` entries include Playwright artifacts and transform caches, Node's compile cache, `tsx` caches, Chromium temporary files, X11 state, an `xvfb-run` authority directory, the Git askpass helper, hundreds of uploaded shard scripts, test-runner fixtures, and `multipart-uploads`. The idle process tables include Blaxel's `sandbox-api`, `ukp-fs`, and entrypoint processes. The precise owner of every miscellaneous entry was not established; in particular, the upload directory must be treated as outside Television's disposal ownership. Neither idle snapshot had an open `/tmp` descriptor or cwd reference, which establishes only those instants. + +### Browser versions in the evidence + +The supplied failure trace records **Chromium `148.0.7778.96`**. During the investigation, the coordinator made cached build 1223 available at the build-1208 path expected by Playwright 1.58.2, whose manifest names Chromium `145.0.7632.6`. PR #32's separate 304-response evidence concerns Chromium 145. These are distinct observations; the 145 finding does not establish the installed version or exact retention behavior on every Blaxel worker. + +**The version workers run today is unknown.** At inspected main `a8e73fbf`, the coordinator still creates those aliases when build 1223 exists and the expected build-1208 path does not. That conditional code cannot establish the current contents of each persistent worker's browser cache. During the planned Blaxel validation, record the actual launched browser version and resolved executable/cache path with the run evidence before interpreting resource measurements or `TMPDIR` observations. Do not infer the running browser from the Playwright package version. Browser-version alignment remains outside this contribution. + +## Worker exclusion, result retrieval, and safe `/tmp` cleanup + +Josh has selected routine cleanup under an exclusive worker lock. There is no pool drain-and-recreate plan or outstanding rollout choice. Before each run, remove only the two known legacy prefixes and stale scratch in the dedicated runner namespace; after execution, remove that attempt's scratch only after its processes stop and its evidence has been finalized outside scratch. Never delete all of `/tmp` or use age alone as proof of disuse. + +### What the existing code establishes + +The canonical coordinator intends one shard job per sandbox. [`acquireAny()` and `lockCommand()`](../../scripts/run-blaxel-testshards.mjs) acquire `/tmp/tv-testshard.lock`; another coordinator skips an unexpired lock. Within a shard, surface tasks execute sequentially, but a surface can have several test workers and many child processes. This does not prove that a worker never has overlapping jobs: + +1. Lock expiry allows reclamation based on time alone. The record identifies the coordinator host/PID, not the remote shard supervisor. Setup is not fully covered by surface owner tokens. The configured TTL is 900/1200 seconds against a 240/480-second worker timeout, but provider timeout is not evidence that every descendant has stopped. Main's spec at `a8e73fbf` explicitly uses TTL as the final availability bound for uncertain cleanup; this contribution replaces that requirement. +2. The shard shell's `EXIT` trap releases the lock before `runShard()` downloads results from `/workspace/television/.testshards/results`. A following checkout clears that fixed tree. This leaves an unprotected interval in which the previous job's evidence can disappear or be replaced. Report identity checks can detect some mismatches but cannot restore deleted files. This source-level race is present at both the branch base and inspected main; no collision was observed during the Chromium investigation. +3. Direct Blaxel process API calls bypass this repository lock. The inventory used that API. `manage-blaxel-testshards.mjs` also provisions and checks workers through `process.exec`, and deletes them through the provider API. The idle snapshots establish neither provider serialization nor a history-wide absence of overlap. + +These are demonstrated protocol gaps, not demonstrated causes of the Chromium failures. The download-race follow-up is incorporated here. Replace blind TTL reclamation with a renewable reservation whose expiry requires process cleanup before reuse. + +### One lock through process teardown and download + +Josh's calibration is that only our runner and pool scripts touch these workers. Correct those code paths using the existing lock; do not add a worker controller, persistent compatibility fence, access-control system, or defenses against hypothetical API callers. + +For a participating coordinator, retain the worker lock until both conditions hold: **all execution for the attempt is stopped, and its results are downloaded and validated at the coordinator.** Keep atomic directory acquisition and the existing `lockId` ownership check. Write `expiresAt: 0` for compatibility: the inspected older runner reclaims only a positive timestamp in the past, so it leaves these locks alone. The updated runner uses a separate renewable recovery deadline, described below. That deadline authorizes cleanup of an abandoned attempt, never immediate reuse beneath surviving processes. Normal release removes the directory, as it does today. + +The runner's order becomes: + +1. Acquire the lock, recovering an abandoned attempt if necessary, then confirm the worker has no prior test/shard work before removing known leftovers. Extend owner-token attribution to the outer shard, including checkout and setup children. Preserve that attempt token alongside surface tokens, which nested supervisors replace today, so detached descendants remain attributable to the attempt. Record the remote process name and PID/start identity so recovery can distinguish the run from PID reuse; the coordinator's host/PID alone is insufficient. +2. Renew the reservation while preparing, executing, cleaning up, and downloading. Keep renewal active while waiting at the existing all-shards-acquired barrier. Keep sequential surfaces and attempt/retry behavior. Record that dispatch was requested before making the API call. If its outcome is unknown, retain ownership while resolving it; a coordinator that exits leaves the bounded recovery path to the next acquirer. +3. On success, failure, timeout, or interruption, stop and audit the run's attributed processes, including descendants of an exited shell. Reuse the existing owner audit and TERM/KILL cleanup. A shell exit or successful kill API response alone is insufficient. The shell finalizes reports but no longer releases the lock in its `EXIT` trap. +4. Download the existing summary, native reports, logs, setup/cleanup evidence, and new diagnostic sidecars to the coordinator's run directory. Validate the existing run/attempt identities and required files. Missing or corrupt evidence remains an infrastructure failure; an early setup failure records which reports were never produced and retrieves what is available. +5. Release the matching lock only after teardown and retrieval complete. Keep the fixed checked results tree: protecting it through download closes the checkout-reset race without a separate artifact archive. Release unused locks after acquisition/upload failures only when execution was never requested and there are no pending writes or produced results to retain. + +All runner exit paths must follow that order, including acquisition rollback, architecture rejection, both signal paths, retry, and exception cleanup. An interrupted download retains the lock; retry retrieval without repeating the tests. If the coordinator exits before completing that work, it stops renewing and reports the reservation's recovery deadline. A failed teardown or uncertain dispatch excludes that worker from immediate retries. Keep the default retry budget. + +### Automatic recovery with one renewable deadline + +**A coordinator keeps its reservation by renewing it; after renewal stops for the recovery interval, the next acquirer stops and audits that run's owned processes, clears its abandoned state, and takes the lock.** The same procedure handles an already quiet worker and a surviving shard supervisor or test descendant. An expired deadline alone never makes the worker available. + +Use the coordinator to renew throughout ownership, including result download. Each successful renewal records worker-local time and sets `recoverAfter` to that time plus **20 minutes**, independent of the worker execution timeout. Reuse `--lock-ttl-seconds` to override this recovery interval; do not add another option or carry the worker timeout in the lock record. Renew comfortably within the configured interval, with cadence derived from it so short private-worker checks work too. Renewal requires the same owner and a still-valid deadline; it cannot revive an expired reservation. Only the coordinator renews: a surviving supervisor cannot preserve a crashed coordinator's reservation forever. No daemon, cross-host PID probe, or separate heartbeat service is needed. + +The automatic entry point is the canonical runner's `acquireAny()` / lock helper. On finding an expired `recoverAfter`, it performs this bounded worker-local sequence before publishing a new owner: + +1. Re-read the matching lock identity and deadline under a short filesystem critical section shared by acquisition, renewal, dispatch registration, and release. Use an OS-released advisory lock (for example `flock` on a fixed guard file outside the disposable lock directory), so a killed recovery process cannot leave a second permanent mutex. Hold it only for worker-local state changes and bounded cleanup, not network downloads. A competing acquirer or late renewal cannot act on a stale read. +2. Stop only the old run's attributed supervisor and descendants, using its owner token and the existing PID/start-identity checks, TERM/KILL escalation, and owner audit. The current stale-owner reaper deliberately leaves live supervisors' children alone; recovery must stop the abandoned run's live supervisors as well, without weakening the ordinary reaper's protection for other owners. Confirm no live or stale owners or remaining attributed processes before deleting their files. A dead supervisor is not enough; detached descendants still count. A late remote dispatch must check the matching, unexpired reservation and register ownership under the same critical section **before** checkout/setup can start. This prevents an API request delayed across recovery from launching the old run into the new checkout. Uploads use attempt-specific scratch paths. +3. Remove the abandoned attempt's scratch and the allowed legacy leftovers, then reset its results as part of the next run's normal preparation. **Discard undownloaded results of this abandoned attempt after the recovery interval.** Do not create a second archive or require an orphan-report download that can itself block reuse. Record the prior worker/run/lock, last renewal and deadline, stopped process identities, cleanup outcome, and whether reports were discarded in a bounded recovery receipt with the incoming attempt's setup sidecars, outside the old results tree being reset. Loss of reports never becomes a passing result for the abandoned run. +4. Publish the incoming reservation only after cleanup succeeds. Recovery is repeatable: if its process dies midway, the old expired identity remains until another acquirer finishes the same work. Normal release and renewal check `lockId`; a returning coordinator that has lost it must stop work and report infrastructure failure, never release or mutate the new owner's attempt. + +This defines coordinator loss operationally: it has stopped renewing for the recovery interval. It does not prove that a remote OS process died. A network partition or suspended coordinator beyond that interval has the same consequence as a crash. A healthy coordinator keeps renewing through a long download, so elapsed shard time alone does not end retrieval protection. Once renewal is lost, orphaned reports are the explicit exception to normal download-before-release. Their file timestamps do not extend the deadline; a hung child must not defeat recovery by continuing to write. + +The capacity cost is bounded by the remaining recovery interval, the next acquisition attempt, and bounded process cleanup. Quiet crash leftovers wait the same interval as active ones; accept that availability cost to avoid a second liveness test, report-age clock, or special quiet-run recovery procedure. Twenty minutes preserves the existing cold-profile operating allowance while removing its coupling to execution time; it is not a measured provider guarantee. Verify renewal and cleanup timing on private workers before relying on it. + +Two correctness limits remain. Failure to identify or stop a process must hold and report the worker rather than run tests over it; an unkillable process or broken provider needs operator repair, not another expiry that ignores the failed audit. Missing or corrupt **lock ownership** is also different from missing scratch metadata: dispatch cannot precede complete ownership registration, and an empty acquisition left before dispatch can be reclaimed after a clean worker audit, but ambiguous live work cannot be killed by a name guess. These limits prevent unsafe reuse; an ordinary crashed coordinator with attributable surviving processes follows automatic recovery and does not need an operator. + +### Older branches and pool operations + +Older branches matter in two concrete ways. Their helpers still leave the two known directory prefixes, which every updated run removes. Their coordinators also unlock at shell exit and use timestamp expiry. New locks' `expiresAt: 0` stops the inspected old acquisition code from taking an updated run's worker, but cannot correct an old coordinator's own early release. Branches take the correction when they integrate main; updating every active branch or waiting for all old runs is not a prerequisite for this contribution or its final verification. + +A legacy lock has no renewable deadline or complete outer-run attribution. Respect its unexpired TTL. After expiry, reclaim it only following a clean worker audit; skip and report one with live or ambiguous work. The inspected old shard's exit trap removes only its own matching `lockId`, so ordinary old work frees itself when it ends. Do not infer an owner token from a process name or build a separate legacy recovery system. An old run that never exits remains an operator case; the pool deletion path below remains available. No permanent old-client exclusion mechanism or pool recreation is proposed. + +`manage-blaxel-testshards.mjs` is the other actual API caller to correct. Deletion checks ownership under the same worker-local guard and refuses an unexpired reservation. After expiry, explicit pool deletion may reserve and delete even a worker whose process cleanup failed: destroying the sandbox is the cleanup. Remove that worker from test discovery before calling the delete API, so a failed or delayed deletion cannot admit a test run. Retain and report it as unavailable if deletion fails; do not silently return it to the pool. Unreadable or incomplete records follow one rule for acquisition and deletion: reclaim only after a clean worker audit, otherwise hold and report. A fresh worker never published for test dispatch may be deleted without a lease. Any other worker whose ownership check cannot run is held for operator repair; current discovery ineligibility alone does not prove an earlier run stopped. The manager continues to list such workers for operator repair. `ensure` remains fresh-only, so partial deletion does not refill capacity; the target pool must be deleted and recreated to replace missing workers. + +For fresh provisioning, publish the labels used by test-worker discovery only after runtime setup and architecture verification succeed. This closes the interval between sandbox creation and acquiring a lock without relying on a pool-wide dispatch stop. The manager must still identify and report partially provisioned workers, and must not silently reuse them through `createIfNotExists`. Listing/status is read-only. This covers our pool script without a generic API gateway or a new pool-draining protocol. + +Preserve the runner's read-only `--audit-stale-owners` mode: it must not recover another run or remove residue before observing it. It may reserve an available worker without recovery, holding that reservation through its existing dependency setup and report collection. A held lock is reported/skipped. This exception preserves the diagnostic's purpose; ordinary test acquisition uses automatic recovery, and pool deletion respects the same reservation deadline with the destruction exception above. + +### A directory for each shard attempt + +Give every locked shard **attempt** a fresh, short, private directory beneath a runner-owned root, for example `/tmp/tv-testshards//`. Record its run, shard, attempt, lock identity, and remote supervisor identity outside its disposable `tmp/` child for diagnostics. The current attempt's directory is explicitly excluded from pre-run cleanup, including before dispatch. Missing or incomplete metadata in an earlier attempt's scratch does not add a cleanup gate once the lock and worker audit establish disuse. Upload/dispatch failure removes only this attempt's directory when execution is known not to have started and pending requests have settled. An unknown outcome leaves the lock and scratch for recovery. Use a short path because browser and Electron processes can create Unix sockets with limited pathname length. + +Export an absolute `TMPDIR` pointing to that child before running preparation and test commands. Keep scripts, plans, askpass, and setup sidecars within the attempt directory as well. Preserve `TMPDIR` through child environments. Continue to keep deliberately persistent npm, Electron, browser-download, and dependency-hash caches at their existing locations outside scratch space. Playwright's transform cache may become attempt-local initially; measure any resulting setup cost before deciding it deserves a separately bounded persistent cache. + +The mechanism is supported by both source and installed dependencies: + +- On non-Windows systems, [Node's `os.tmpdir()`](https://nodejs.org/api/os.html#ostmpdir) checks `TMPDIR` before `TMP`, `TEMP`, and `/tmp`. Both cached build helpers use `os.tmpdir()`. +- [Chromium 148's `GetShmemTempDir()` and `GetTempDir()`](https://github.com/chromium/chromium/blob/148.0.7778.96/base/files/file_util_posix.cc#L1283-L1302) select the temporary directory when `--disable-dev-shm-usage` is present; `GetTempDir()` reads `TMPDIR`. The investigation observed this flag in the actual browser launch. +- Installed Playwright 1.58.2 uses `os.tmpdir()` for browser profiles and artifact directories in `playwright-core/lib/server/browserType.js`, inherits the parent environment by default, and includes that Chromium flag. Its transform cache also defaults to `os.tmpdir()`. + +This establishes the intended wiring; a live Blaxel test must still observe Node-created paths and actual Chromium shared-memory descriptor paths under the attempt directory. We have not run that check at the proposal stage. Setting `TMPDIR` cannot relocate hard-coded `/tmp` paths or X11's global socket directory; the runner's explicit temporary paths must be moved deliberately, and external infrastructure remains untouched. Chromium also unlinks shared-memory files while holding them open, so directory deletion cannot substitute for stopping the owning processes. + +### Exact deletion scope and order + +Before preparation for **every run**, while holding the worker reservation and after confirming no prior execution or retrieval remains, remove: + +- Real directories directly under `/tmp` whose basenames match `television-source-cli-e2e-*` or `television-web-bundle-*`. The helper source and inventories establish these as leaked test builds; Josh has authorized their recurring removal. Missing per-directory metadata on these legacy copies is expected. The whole-worker exclusion and idle check supply the disuse evidence that the prefix alone cannot supply. +- Every attempt directory directly beneath the dedicated `/tmp/tv-testshards/` root except the current one, after the worker owner audit confirms no prior live or stale owners remain. These directories belong to our runner. As with the legacy prefixes, the held lock and clean worker audit establish disuse; missing, half-written, or inconsistent per-directory metadata does not prevent removal. This does not reclaim another run's held lock or decide how dead-coordinator locks are recovered. + +Revalidate directory identity immediately before deletion; reject symlink roots, unexpected mount boundaries, or paths escaping the exact allowed parent. Recursive removal must not follow nested links to another tree. Preserve the lock and guard, provider uploads, X11 state, unrelated temporary files, checkouts, and deliberate caches. Reports stay outside scratch; only abandoned-attempt recovery can discard them before download. Do not broaden the legacy patterns to other Television-looking names. Report per-prefix counts, allocated bytes removed, filesystem free space before/after, and skips or errors; those two space measurements need not be equal when files remain open or the filesystem accounts storage differently. + +After execution, finalize evidence outside scratch, stop all attributed processes, then remove this attempt's scratch while the reservation remains held through download. Setup failure, acceptance failure, timeout, and interruption use the same ownership rule. If teardown is uncertain, preserve scratch and keep the worker reserved. If deletion fails after confirmed teardown, preserve the original test outcome and add cleanup failure evidence; the worker cannot start another test until pre-run cleanup succeeds. Evidence must still be downloaded. A partial deletion must never be reported as reclaimed capacity for the whole directory. + +An inconclusive worker audit or a live consumer means skip deletion **and exclude that worker from test dispatch**, with a recoverable infrastructure failure. Missing scratch metadata alone does not make that audit inconclusive. There is no fallback to deleting unrelated files to obtain headroom. Known residue left by old test revisions is removed on subsequent updated runs without recreating workers or losing their caches. This proposal revision performs no worker cleanup or configuration changes. + +## Worker lifetime for cached runtimes + +Keep one lazily built source-CLI runtime per test worker and reuse it across launches and test cases. Publish the cached entry only after the build and asset installation succeed. A partial build removes its own directory and leaves the cache retryable. Apply the equivalent ownership rule to the cached versioned web bundles, retaining their per-version reuse. + +Test cleanup continues to stop each case's product servers promptly. Worker teardown then stops any remaining servers and removes the shared cached runtime and versioned bundles. Cleanup is idempotent and waits for pending builds before removing their outputs. The worker teardown must cover both the shared Playwright registration helper and the Vitest callers; a per-test deletion would discard reuse and add unnecessary builds. A failed server teardown cannot justify removing files beneath a surviving server. The outer shard's process ownership and scratch cleanup provide the backstop when a worker is killed and cannot execute its own teardown. + +The relevant entry points are `test/helpers/playwright.ts`, the two cached build helpers, and the Node/Vitest callers of those helpers. The design should use the runners' existing teardown facilities rather than a general-purpose cleanup registry. Failure during a per-launch runtime copy must also dispose that copy and its newly created home: the current bundle preparation occurs before the launch disposer is registered. + +The helper fix matters even with shard scratch: it releases files between test workers within a shard and protects callers on hosts without the Blaxel shard script, including ordinary CI and developer environments. The reviewer independently counted 1,388 source-CLI and 727 versioned-bundle directories on this development host; that is reviewer evidence, not a new inspection by this contributor. Cleaning those existing host files is the host owner's decision and outside this contribution. The helper fix prevents new leakage on updated branches; it does not retrospectively remove unowned files. + +## Built application for product acceptances + +### Assessment after PR #32 + +**Convert five cases in two files, rather than the earlier 17-file set.** Josh delegated the scope judgment to this contribution. PR #32 (`982e564d`, merged 2026-10-09) removes conditional request headers at the registry Vite service boundary, so those services return full responses. Its recorded Chromium 145 evidence identifies the 2 MB buffer retained per 304 until context closure. That change directly addresses the repeated-reload mechanism for tests using those services, including tests that remain on Vite. Routine cleanup also recovers the leaked runtime capacity on each updated run. Neither mechanism now justifies a broad application-loader migration. + +The two remaining source-loaded theme lifecycle cases are worth converting because they already claim built-application behavior in parts of their proofs and exercise repeated loads and server restarts. Browser demo mode is a small adjacent conversion: its three cases share one product-page helper, with repeated application reloads in two cases. Both files already have a real `ProductServer` and built assets available. This improves their shipped-asset boundary and removes hundreds of module requests per load without a new browser harness. The incremental memory and time benefit after #32 is unmeasured. Main's `f9180241` (PR #30) fixes the demo-mode flake by closing an unused HTTP connection that held fixture teardown open; that flake does not support the resource rationale for this conversion. + +Diminishing returns set in after these two files. The other product walks could gain packaged-asset coverage, but reviewing their loader, authentication, restart, and proof changes adds regression work without evidence of a remaining resource failure that those conversions would cure. Keep #32 for every remaining registry Vite service. Preserve the assertions and #9's merged theme-reset conversion. Validate the selected paths functionally and confirm their request inventory; any before/after resource comparison must use a baseline containing #32. This scope is based on source and proof inspection, not a post-#32 memory experiment. + +### Conversion boundaries + +Use the already-built web application served by `ProductServer`. Existing registry pre-commands build the web and artifact documents before both browser-app surfaces, and `installProductRuntimeAssets()` already copies the application index and assets. Preserve the tests' assertions, real browser behavior, CSS-motion declarations, authentication, and server lifecycle. + +The complete conversion set under `packages/web/test/e2e/` is: + +| Tests | Reason and boundary | +| --- | --- | +| `theme-product-acceptance.test.ts`: “theme settings persist rendered app and artifact outcomes through a built-server restart” and “executable themes run only in the Chromium application lifecycle and isolate failures” | Keep the two real documents, executable isolation, appearance, package watching, and server-restart assertions. Current main's [product proof](../../proofs/product/themes-and-appearance.md) calls the executable and appearance walks built-application acceptances, while its selection description still names Vite loading. The [delivery proof](../../proofs/arch/themes/delivery.md) likewise calls the executable frame walk built. Make the setup and proof descriptions agree on shipped assets. PR #9's theme-reset case and the already-built cases are untouched. | +| All three cases in `browser-demo-mode.test.ts` | Change the shared `openApp` helper. The marker-on/off walk reloads twice; return-to-original reloads once. The bridge-inert case shares the same helper and real marker/server boundary, so keeping its loader identical needs no separate harness or additional conversion machinery. Preserve external navigation, unsupported-view behavior, and the inert bridge assertions. | + +A server URL is sufficient where a backend stays up. Where restart is part of the walk, reuse the existing [`startStableFrontProxy()`](../../test/helpers/stable-front-proxy.ts) to serve the built document and retarget HTTP/WebSocket traffic after restart. Do not weaken a reconnect test by navigating to a replacement origin or clearing browser state. Keep development loading available explicitly for module and fixture seams; avoid changing every `appURL()` caller implicitly, including PR #9's test and the Electron dispatcher seams. + +The product runtime also needs the built `views/url-unsupported` assets for the existing external-navigation walk, which currently obtains that view from Vite. Add the missing test-runtime asset copy, not a production routing change. This is a prerequisite for preserving that assertion honestly. + +For new or substantially changed **browser acceptances using `ProductServer`**, prefer its built application unless the assertion needs development modules. This testing directive does not require retroactive conversion of other files or apply to in-process `Server` walks using main's `appURLForServer()`, component fixtures, or Electron module/dispatcher seams. Those boundaries remain as authored. Existing built-app acceptances remain built-app acceptances. + +### Remaining development-page exposure + +At `b5477211`, a read-only inventory found 38 e2e test files with a literal `/packages/web/src/index.html` reference and two more using `guerilla-artifact-sharing.helpers.ts`: 40 test files, or 41 source-loading files when that helper is counted. That historical count is not today's loader count or a measurement of concurrent pages or memory. Repository tests that inspect HTML as text and component fixtures importing individual modules are excluded. The narrowed scope deliberately leaves the following exposures: + +| Remaining tests | Observed reload/page behavior and why deferred | +| --- | --- | +| `channel-product.test.ts`, `channel-crud.test.ts`, `channel-sidebar-remote.test.ts`, `channel-switcher-context-menu.test.ts`, `tab-pages.test.ts`, `keyboard-navigation-product.test.ts`, `artifact-view-wheel.test.ts`, product walks in `top-bar.test.ts` and `auth.01.test.ts` | Deferred from the earlier conversion set. Channel, tab, and keyboard walks include observer pages; the tab walk also reloads. They remain legitimate conversion candidates, but their present product assertions do not require this contribution to change their delivery path. The channel drag flake has no established resource cause. | +| `markdown-content-sync.spec.ts`, `path-artifact-saga.test.ts`, `path-artifact-real-stack-coverage.test.ts`, `navigable-history-real-stack.01/.02.test.ts`, `onboarding-browser.test.ts` | Deferred from the earlier set. These cover real editing, persistence, navigation, or boot/restart; onboarding in particular requires stable-origin continuity. After #32 and recurring cleanup, a larger migration is not justified by the resource evidence. Keep their proof boundaries and assertions intact. | +| Main's web `artifact-isolation.test.ts`, `artifact-move.test.ts`, and desktop `artifact-isolation.test.ts` | Additional `appURL()` users identified by round 2 review on main. They are explicitly outside the two-file conversion. Source inspection finds a reload in web isolation and a second page in web move; no headroom claim is made. | +| Web `telemetry-session.test.ts` | One case reloads the application; another opens two tabs in one persistent context, closes it, then opens a new context. This is a concrete remaining multi-page exposure. It uses an in-process server with an injected telemetry sink; converting that boundary needs its own proof review. | +| Web `artifact-navigation.01.test.ts`, `artifact-navigation-history.test.ts` | Their `beforeEach` loads the development app, clears storage, then reloads it: two full graph loads per case. They use directly controlled `Store`/`Server` instances. These are sensible next conversion candidates, not proven-safe exceptions. | +| Web `auth.02.test.ts` | The silent reconnect case explicitly reloads the app. Other cases inject authentication failures into an in-process server. Reconcile this base-revision inventory against PR #9's merged changes before implementation. | +| Web `reload-after-reconnect.01/.02.test.ts`, `guerilla-artifact-sharing.test.ts`, `guerilla-artifact-sharing-resilience.test.ts` | These open one application page and reload/recover artifact documents or reconnect the event socket. The reconnect continuity case explicitly expects no top-level navigation. They control server internals/producer-consumer seams; an artifact reload is not evidence of a full application graph reload. | +| Web `settings.test.ts`, `platform-marker.test.ts`, `artifact-navigation.02.test.ts`, `appearance-delivery.test.ts`, `theme.test.ts`, `artifact-full-screen.test.ts` | The inspected source-app cases use one application page. Other cases in these files use fixtures or built assets. Retain their in-process server, platform, appearance, and navigation boundaries for this contribution; no claim that development modules are inherently necessary. | +| Desktop `all-webview-dispatcher.01/.02.test.ts`, `webview-event-parity.test.ts` | The executable-theme parity walk explicitly waits for a top-level document reset and later closes/relaunches Electron. The navigation-history walk quits and relaunches. These remain exposed; application resets, fresh Electron processes, and guest-webview reloads must be distinguished in any follow-up conversion. | +| Desktop `artifact-view-webview-wheel.test.ts`, `guerilla-artifact-sharing.test.ts`, `path-webview-content-reload.test.ts`, `artifact-navigation.test.ts`, `native-navigation-key.test.ts`, `artifact-navigation-external.test.ts` | Launch the development application in Electron; several reload or navigate guest documents. Native input, preload, and webview seams are outside this browser-product conversion. No measurement establishes their peak memory. | + +This contribution does not establish a post-#32 memory saving, eliminate unbundled loading, or certify every remaining test's headroom. Preserve the remaining walks in full validation and use the new diagnostics if they fail. The testing directive is limited to new or substantially changed `ProductServer` browser acceptances. + +### Reconciliation with current main + +Main at `a8e73fbf` is integrated in `74d22941`. Preserve PR #9's theme-reset conversion, the stable-front proxy's current API, SDK runtime assets, and single-origin application loading. The inventory above is explicitly tied to `b5477211`, not a claim that all listed lines remain on today's main. Do not convert an already converted case again. The five selected cases remain source-loaded at this checkpoint; the derived proofs now describe their built-application boundary and explicitly mark the adaptation pending. + +Main also now owns full responses from registry Vite services and selection of a contributor's origin repository for remote checkout. Preserve both. Measure built-app memory savings against this new baseline rather than attributing PR #32's benefit to PR #12. + +## Failure diagnostics and retention + +The minimum that would have identified this incident is **available RAM, shared memory, free bytes on the effective temporary filesystem, and the Linux OOM-kill counter at surface start and end, plus kernel OOM lines when failure occurs**. Commit to that small set. Read it from the parent surface process so a browser or test-worker crash does not destroy the initial observation. Include the surface preparation/teardown interval in both planned and targeted paths; retain the start record immediately, and mark a missing final observation as unknown after interruption. + +Keep one resource sidecar per surface execution, linked to the native attempt evidence already recording file/title/project/retry and failures. Retain it when a test attempt fails even if a later retry passes. When available, the shared browser failure hook can add a snapshot at the failed attempt boundary; this is an enhancement, not the minimum contract. Start/end samples provide context and an OOM delta, not a peak or exact allocation-time measurement. A system OOM-counter increase alone cannot attribute a kill to a browser; readable kernel lines provide process identities. If native results are missing, retain the sidecar without inventing an acceptance result. + +Read kernel output through a bounded capture and retain at most **64 KiB of relevant OOM lines** for the execution interval, with explicit truncation and unavailable-probe reasons. The two snapshots and bounded excerpt produce at most **128 KiB per surface sidecar and 8 MiB of resource diagnostics per shard attempt**. Enforce these limits when writing; a cap is necessary for bounded retention, but a continuous sampler/ring-buffer framework is not. Retain each surface's small start/end record before using the remaining budget for excerpts. Omit credentials, request query strings, bodies, and arbitrary process environments. Probe failure adds an observability warning and preserves the original test outcome. + +Periodic sampling, per-process descriptor inventories, cgroup probes, detailed browser/request event records, and browser-revision comparison remain optional investigation tools. They are not new merge obligations or spec requirements in this contribution. Do not add default videos, heap snapshots, or NetLogs. Existing native retry evidence and trace policy remain unchanged. + +Write sidecars under the existing checked `.testshards/results` tree, outside disposable scratch, and download them with the other reports. Keep at most one attempt's bounded resource diagnostics per worker. A failed download leaves that attempt locked while the coordinator retries retrieval and renews; it does not start a growing queue of retained runs. Normally the next results reset follows download and release. If the coordinator stops renewing past the recovery deadline, automatic recovery may discard those abandoned reports and records that loss. Its recovery receipt shares the incoming attempt's 8 MiB diagnostic budget, with explicit truncation; it needs no separate size limit. Coordinator-side `.test-runs/` retention is unchanged. No cross-run archive or retention daemon is needed. + +## Authority and derivation + +After proposal convergence, derive the smallest changes in the existing owners: + +- [`testing-policy.md`](../../specs/arch/testing-policy.md): built application as the default for new or substantially changed `ProductServer` browser acceptances unless development-module behavior is under test, excluding in-process `Server`/`appURLForServer()` walks; worker-owned reusable test runtimes and cleanup expectations. +- [`blaxel-testshards.md`](../../specs/arch/test-runner/blaxel-testshards.md): exclusive reservation through setup, process teardown and result retrieval; renewal, automatic recovery after lost renewal, owned-process cleanup before reuse, and the explicit abandoned-report exception; actual older-runner behavior; pool-script locking; attempt temporary directories, recurring exact-prefix cleanup and bounded worker retention. Replace its blind TTL escape and shell-exit unlock expectations rather than leaving conflicting requirements. +- [`reporting.md`](../../specs/arch/test-runner/reporting.md): minimum resource observations and artifact references, bounded content, unavailable evidence, retention alongside failed/recovered native attempts, and recovery receipts that disclose discarded orphan reports. + +Use existing lifecycle and reporting contracts in `test-runner.md` and `sharded-execution.md`, adding the cross-references or ownership corrections required by the outer Blaxel lock. The proof review also exposed missing authority for the automated live driver and its deliberate-failure fixtures: the `test-registry.md` and `test-runner.md` delta places them in separately selected experiment surfaces, with the driver on a dedicated credentialed Blaxel coordinator and the fixtures on its private pool. This keeps repeatable live coverage without adding a kind or suite. Proof review round 2 approved that placement delta; its dependent proof binding is complete. Existing isolated-GitHub-only lifecycle fixtures keep their placement. No `preflight.md` delta is planned. Optional investigation instrumentation remains optional. + +Update the mirrored proofs after independent spec convergence. Test setup changes may also require proof-only corrections to descriptions of the affected browser walks; they do not change their product promises. Do not prescribe a new framework or adopt unrelated harness internals into spec authority. + +## Validation and contribution boundaries + +All execution is on Blaxel at committed, pushed revisions. The shared development host performs file inspection, edits, Git operations, and lightweight remote coordination only. No tests, builds, browser launches, lint, or type checks have run for this proposal. Apply the test-iteration discipline: red/green at the narrowest relevant file, then affected surfaces, then full verification with the default retry budget. + +Derive proofs for the principal paths without a combinatorial fault matrix: + +- One run excludes a competing coordinator through preparation, teardown, and a deliberately delayed result download while renewing. Passing the former TTL never admits a second run beneath that valid reservation. Exercise both targeted and planned dispatch through the real provider. +- Stopping coordinator renewal permits automatic acquisition after the recovery interval, both with a quiet worker and with an owned supervisor/descendant still alive. Reuse follows confirmed termination; the receipt records any discarded reports. Contract coverage uses a controlled clock for renewal/deadline calculation, the recovery-interval override, and error paths. A private-worker seam crosses real dispatch, recovery, and delayed retrieval, including a late dispatch rejected after ownership changes. It must not reduce to mocked API responses or simulate a coordinator crash by killing only the shard. +- Concurrent recovery attempts cannot both acquire; delayed renewal/release cannot alter a successor's reservation. Keep this at the lock helper's real filesystem boundary. The inspected older runner does not reclaim a new held lock with `expiresAt: 0`. Legacy helper residue is removed on the next updated run; live legacy work is skipped and failed process cleanup prevents reuse. Pool deletion respects the same deadline but can destroy an abandoned worker that failed cleanup. Fresh workers are unavailable to tests until provisioning succeeds. +- Exact-prefix and stale-scratch cleanup preserves unrelated files, live consumers, and the current attempt directory, while removing earlier scratch even when its metadata is incomplete. Real runtime builds demonstrate once-per-worker reuse, final disposal, and partial-build cleanup. A Blaxel observation verifies Node temporary paths and Chromium shared-memory descriptors under the attempt's `TMPDIR`, recording the actual browser version used. +- A failed or recovered acceptance retains bounded start/end/OOM evidence through retrieval, including when its worker exits. Contract coverage handles unavailable probes, unchanged original test outcomes, truncation, and limits. + +The live recovery checks use small, disposable, owned processes on private workers. No deliberate OOM, process storm, or invocation of the repository's isolated-GitHub-only destructive fixtures is authorized. If a required assertion cannot be proven safely within Blaxel-only execution, report that specific blocker at proof review; do not silently substitute local or GitHub execution. + +Use a dedicated two-worker pool named **`rose-cleanup`**, with workers `tv-testshard-x64-rose-cleanup-01` and `-02`, for cleanup, orphan, and interruption experiments. Explicitly pass `--pool rose-cleanup` through the canonical runner; reduced planned runs also need the documented shard-count override. Exercise interruption with disposable owned subprocesses there, without intentionally exhausting memory or running repository fixtures restricted to isolated GitHub hosts. Never run those experiments on shared `poc` workers. Two workers permit a distinct replacement after a contaminated attempt; they do not guarantee capacity for every retry after repeated contamination, which remains an infrastructure failure with the default retry budget intact. + +Re-run the converted acceptances without weakening their user-visible assertions. A recorded built-page request inventory should show that they no longer load the unbundled application graph. Validate restart continuity and authenticated navigation directly. Repeated use of one isolated worker should leave no generated runtime directories after cleanup and no growing sequence of resource reports. + +For the final gate, retain the separate sandbox named **`tv-cleanup-coordinator-rose`**, excluded from testshard pool labels. After private qualification and deletion of the two experimental workers, it will fetch the committed, pushed branch and run `npm run verify -- blaxel --pool poc` at the standard 36 shards, so the invoking machine's lint/type-check/package-manifest phases also execute on Blaxel. The two experimental workers and coordinator are a plan, not currently provisioned capacity. Provision and verify their identities before starting validation; if unavailable, report blocked rather than run anything on this host or silently substitute GitHub execution. The user has already authorized outside-pool Blaxel hosts. + +Supply the coordinator's credentials through the Blaxel SDK's `process.exec` environment: `BL_WORKSPACE` and an authorized `BL_API_KEY` for its outbound Blaxel calls, plus `BLAXEL_TV_GH_TOKEN` for repository access. The launching process reads the existing authorized secrets into memory and passes them directly in that environment object; do not interpolate their values into shell commands, uploaded scripts, output, or reports. The inspected SDK accepts the Blaxel pair without a CLI login/config file, and `readBlaxelGithubToken()` accepts `BLAXEL_TV_GH_TOKEN` when no `.blaxel-gh-token` file is present. Use that environment path in the fresh coordinator, with `GIT_ASKPASS` reading the environment for its initial fetch; the existing runner passes only the GitHub token onward to workers as `GH_TOKEN`. Do not copy local credential files or persist tokens in the coordinator's filesystem, sandbox configuration, or Git configuration. Confirm that the required credential is available and valid for the run before dispatch; its availability is not established by this proposal. End the credential-bearing process after downloading its reports and dispose the dedicated coordinator. This describes ephemeral process injection, not a claim about provider-internal retention. + +The experiment plan adds at most three sandboxes. The spec's pool-size and tier examples are not a current capacity inventory: inspect actual workspace usage and provider limits before provisioning/dispatch. Do not claim slots are reserved or alter shared workers to obtain them. The [implementation plan](test-harness-resource-cleanup-plan.md#Integrated result and final validation) keeps focused development and all worker experiments on the named private hosts, then uses the standard shared pool for ordinary full verification. The committed file medians leave inadequate setup/retry allowance for a two-worker full run within the canonical runner's 480-second worker timeout. Retrieve private reports before the pool-deletion acceptance and all remaining reports before deleting the coordinator. Record the exact tested revision, failures, recovered flakes, and any measurement limits. Ready-PR CI remains the repository's merge requirement; the contribution's execution evidence stays on Blaxel as instructed. + +Keep browser-version alignment, the default shard count, browser request limits, retry defaults, and production memory behavior outside this contribution. Forced GC is an investigation tool, not a proposed mitigation. Preserve main's demo-mode teardown fix; the drag flake has no established resource-exhaustion cause. + +## Review status + +Proposal round 1 at `1652787e`: FAIL on lease scope and mixed-version consequences. The response at `99b7ed93` narrowed the proposal while those human scope decisions were pending. Josh's 2026-10-27 instruction brings lock correction and recurring legacy cleanup into this PR and removes the pool-recreation choice. The [review response](test-harness-resource-cleanup-review-response.md) records the current dispositions without changing the independent review's historical verdict. + +Proposal round 2 at `d3570bf4`: FAIL because dead-coordinator lock recovery is undefined. The [round 2 review](test-harness-resource-cleanup-review-r2.md) is preserved unchanged. Josh selected automatic recovery, required that an orphaned active run cannot hold a worker forever, and delegated the recovery mechanism and built-app conversion scope. His supervisor clarified that the suggested predicate and grace-period wording were illustrations, not requirements. The revision at `57a701d0` supplied one renewable recovery deadline and narrowed conversion to two files. + +Proposal round 3 at `57a701d0`: **PASS**, relayed by the supervisor. The [review](test-harness-resource-cleanup-review-r3.md) is preserved unchanged. Its bounded refinements are applied here: fixed recovery interval, deletion of abandoned workers despite failed process cleanup, ordinary skipping of live legacy work, and trimmed receipt/provisioning language. The [response](test-harness-resource-cleanup-review-response.md#Round 3: convergence and spec derivation) records the judgments. The proposal and spec deltas have converged: spec round 1 at `cd8aa453` also passed. Its bounded refinements and the derived proof checkpoint are recorded in the response; independent proof review is next. + +Proof round 1 at `9fd51f49`: **FAIL** on missing spec-owned placement for live qualification and deliberate-failure fixtures. The [review](test-harness-resource-cleanup-proof-review-r1.md) and [response](test-harness-resource-cleanup-review-response.md#Proof round 1: placement finding and refinements) retain that history. Proof round 2 at `3fa15d47`: **PASS**, including the placement delta. Its [report](test-harness-resource-cleanup-proof-review-r2.md) permits the bounded proof binding now applied. Plan round 1 at `70f27ccb`: **FAIL** on the two-worker full-gate budget. The [review](test-harness-resource-cleanup-plan-review-r1.md) and [response](test-harness-resource-cleanup-review-response.md#Plan round 1: full-gate placement and refinements) record the correction to ordinary shared-pool verification and the bounded refinements. The [implementation plan](test-harness-resource-cleanup-plan.md) is ready for independent re-review; tests and implementation have not begun. No tests, builds, remote experiments, or worker operations were performed for this checkpoint. Unknowns are live provider behavior during failed dispatch/recovery, today's worker browser versions, actual `TMPDIR` placement and post-mitigation memory savings, current capacity/credential availability, the implemented tree's full-gate duration, and observed recovery timing. Recovery waits at most 20 minutes after the last renewal by default, then acquisition and bounded cleanup; failed cleanup prevents test reuse and explicit pool deletion remains available. Josh's spec acceptance remains due at PR time; there is no pending shared-pool cleanup rollout decision. diff --git a/packages/web/test/e2e/browser-demo-mode.test.ts b/packages/web/test/e2e/browser-demo-mode.test.ts index 47e0e344..520e6006 100644 --- a/packages/web/test/e2e/browser-demo-mode.test.ts +++ b/packages/web/test/e2e/browser-demo-mode.test.ts @@ -6,6 +6,7 @@ import os from "node:os"; import path from "node:path"; import { test, expect } from "../../../../test/helpers/playwright.ts"; import { launchProductServer, type ProductServer } from "../../../../test/helpers/product-server.ts"; +import { recordApplicationRequests } from "../../../../test/helpers/application-requests.ts"; import { APPLICATION_SHELL_STATES, waitForApplicationRender } from "../../../../test/helpers/application-readiness.ts"; import { configureTestMotion, pickChannelByName } from "./helpers.ts"; @@ -72,11 +73,10 @@ async function createURLArtifact( return ((await response.json()) as { artifact: { id: string } }).artifact.id; } -async function openApp(page: Page, server: ProductServer, baseURL: string | undefined): Promise { - if (!baseURL) throw new Error("Expected Playwright baseURL"); - const appBaseURL = await server.appURL(baseURL); - await page.goto(`${appBaseURL}/packages/web/src/index.html`); +async function openApp(page: Page, server: ProductServer, requests: ReturnType): Promise { + await page.goto(server.serverURL); await waitForApp(page); + requests.expectBuilt(); } async function waitForApp(page: Page): Promise { @@ -106,8 +106,10 @@ test.describe("browser demo mode", () => { let home: string; let site: Awaited>; let server: ProductServer | null = null; + let requests: ReturnType; - test.beforeEach(async () => { + test.beforeEach(async ({ page }) => { + requests = recordApplicationRequests([page]); home = mkdtempSync(path.join(os.tmpdir(), "television-demo-home-")); site = await startExternalSite(); server = await launchProductServer({ @@ -117,6 +119,7 @@ test.describe("browser demo mode", () => { }); test.afterEach(async () => { + await requests.attach(test.info()); await server?.dispose(); server = null; await site.close(); @@ -124,7 +127,7 @@ test.describe("browser demo mode", () => { }); // spec: proofs/product/artifacts.md#^af-ac-demo-mode - test("the home-directory marker turns direct external pages on and off at app load (^af-ac-demo-mode)", async ({ page, baseURL }) => { + test("the home-directory marker turns direct external pages on and off at app load (^af-ac-demo-mode)", async ({ page }) => { if (!server) throw new Error("Product server was not started"); const [channel] = await channels(server); if (!channel) throw new Error("Expected a default channel"); @@ -132,7 +135,7 @@ test.describe("browser demo mode", () => { await createURLArtifact(server, pageURL, channel.id); const returnButton = page.getByRole("button", { name: RETURN_LABEL }); - await openApp(page, server, baseURL); + await openApp(page, server, requests); await expect(artifactFrame(page)).toHaveAttribute("src", /\/views\/url-unsupported\//); await expect(returnButton).toHaveCount(0); @@ -152,7 +155,7 @@ test.describe("browser demo mode", () => { }); // spec: proofs/product/artifacts.md#^af-ac-demo-return - test("the artifact icon, an app reload, and a channel round trip each return to the original page (^af-ac-demo-return)", async ({ page, baseURL }) => { + test("the artifact icon, an app reload, and a channel round trip each return to the original page (^af-ac-demo-return)", async ({ page }) => { if (!server) throw new Error("Product server was not started"); writeFileSync(path.join(home, MARKER), ""); const [channel] = await channels(server); @@ -160,7 +163,7 @@ test.describe("browser demo mode", () => { await createChannel(server, "Elsewhere"); await createURLArtifact(server, `${site.origin}/one`, channel.id); - await openApp(page, server, baseURL); + await openApp(page, server, requests); await expect(frameHeading(page)).toHaveText("Fixture page one"); await followNext(page); @@ -181,7 +184,7 @@ test.describe("browser demo mode", () => { }); // spec: proofs/arch/artifact-frame/artifact-bridge.md#^ab-ac-demo-inert-seam - test("bridge-shaped messages from a demo-mode page change nothing (^ab-ac-demo-inert-seam)", async ({ page, baseURL }) => { + test("bridge-shaped messages from a demo-mode page change nothing (^ab-ac-demo-inert-seam)", async ({ page }) => { if (!server) throw new Error("Product server was not started"); writeFileSync(path.join(home, MARKER), ""); const [channel] = await channels(server); @@ -191,7 +194,7 @@ test.describe("browser demo mode", () => { await createURLArtifact(server, pageURL, channel.id); await createURLArtifact(server, `${site.origin}/two`, channel.id, "Second page"); - await openApp(page, server, baseURL); + await openApp(page, server, requests); const selected = page.getByRole("option", { name: channel.name, exact: true }); await expect(selected).toHaveAttribute("aria-selected", "true"); const tabs = page.locator(".top-bar .tab"); diff --git a/packages/web/test/e2e/gate-boot-barrier.spec.ts b/packages/web/test/e2e/gate-boot-barrier.spec.ts index 6759b9b8..82dacf48 100644 --- a/packages/web/test/e2e/gate-boot-barrier.spec.ts +++ b/packages/web/test/e2e/gate-boot-barrier.spec.ts @@ -1,4 +1,5 @@ -import { expect, test, type Page } from "@playwright/test"; +import { type Page } from "@playwright/test"; +import { expect, test } from "../../../../test/helpers/playwright.ts"; import { mkdtempSync, rmSync } from "node:fs"; import os from "node:os"; import path from "node:path"; diff --git a/packages/web/test/e2e/theme-product-acceptance.test.ts b/packages/web/test/e2e/theme-product-acceptance.test.ts index f3180668..071fffe8 100644 --- a/packages/web/test/e2e/theme-product-acceptance.test.ts +++ b/packages/web/test/e2e/theme-product-acceptance.test.ts @@ -13,6 +13,8 @@ import { TelevisionClient } from "@telepath-computer/television-shared"; import { test, expect } from "../../../../test/helpers/playwright.ts"; import { launchProductServer, type ProductServer } from "../../../../test/helpers/product-server.ts"; import { seedThemePackage } from "../../../../test/helpers/theme-package.ts"; +import { recordApplicationRequests } from "../../../../test/helpers/application-requests.ts"; +import { startStableFrontProxy } from "../../../../test/helpers/stable-front-proxy.ts"; import { waitForApplicationShell } from "./helpers.ts"; import { APPLICATION_SHELL_STATES, @@ -611,12 +613,12 @@ test("Firefox refreshes a ready Markdown editor from a fixed-dark theme to light // proofs/arch/themes/delivery.md#^theme-delivery-t-active-package-refresh test("theme settings persist rendered app and artifact outcomes through a built-server restart", async ({ browser, - baseURL, }) => { - if (!baseURL) throw new Error("Expected Playwright baseURL"); const context = await browser.newContext({ colorScheme: null }); const page = await context.newPage(); + const requests = recordApplicationRequests([page]); const product = await launchProductServer(); + const front = await startStableFrontProxy(product.serverURL); try { const themeAPath = seedThemePackage( product.home, @@ -691,9 +693,10 @@ test("theme settings persist rendered app and artifact outcomes through a built- "https://example.com/appearance", ); - const appURL = await product.appURL(baseURL); - await page.goto(`${appURL}/packages/web/src/index.html`); + const appURL = front.url; + await page.goto(appURL); await waitForApplicationShell(page); + requests.expectBuilt(); await openSettings(page); await page.getByRole("button", { name: "Refresh themes" }).click(); @@ -843,6 +846,7 @@ test("theme settings persist rendered app and artifact outcomes through a built- )).toHaveCount(0); await product.restart(); + front.setTarget(product.serverURL); await waitForApplicationShell(page); await expectAppearance("dark", FIXED_DARK); const persistedDisplay = await fetch(new URL("/display", product.serverURL)); @@ -868,6 +872,8 @@ test("theme settings persist rendered app and artifact outcomes through a built- expect((await expectSandboxFrame(page, "overlay")).instance) .not.toBe(beforeAdaptiveOverlay.instance); } finally { + await requests.attach(test.info()); + await front.dispose(); await product.dispose(); await context.close(); } @@ -880,7 +886,6 @@ test("theme settings persist rendered app and artifact outcomes through a built- // proofs/arch/themes/delivery.md#^theme-delivery-t-frame-appearance-browser test("executable themes run only in the Chromium application lifecycle and isolate failures", async ({ page, - baseURL, browserName, }) => { test.skip(browserName !== "chromium", "Chromium owns executable-theme browser acceptance"); @@ -890,11 +895,12 @@ test("executable themes run only in the Chromium application lifecycle and isola const reconnectBackoffBudget = 30_000; const themeResetNavigationBudget = 30_000; test.setTimeout(test.info().timeout + reconnectBackoffBudget + 2 * themeResetNavigationBudget); - if (!baseURL) throw new Error("Expected Playwright baseURL"); const product = await launchProductServer(); + const front = await startStableFrontProxy(product.serverURL); const secondPage = await page.context().newPage(); const applicationPages = [page, secondPage] as const; + const requests = recordApplicationRequests(applicationPages); const selectReloadingTheme = async (value: string): Promise => { // A consented script is cleared by replacing both application documents. // Observe that navigation before applying ordinary post-render assertions. @@ -1062,13 +1068,14 @@ document.querySelector("#artifact-probe").addEventListener("click", () => { "Executable Theme Artifact", artifactPath, ); - const appURL = await product.appURL(baseURL); + const appURL = front.url; await Promise.all(applicationPages.map(async (applicationPage) => { await applicationPage.goto( - `${appURL}/packages/web/src/index.html`, + appURL, ); await waitForApplicationShell(applicationPage); })); + requests.expectBuilt(); await openSettings(page); await page.getByRole("button", { name: "Refresh themes" }).click(); const themeSelect = page.locator('tv-select[name="theme"]'); @@ -1783,6 +1790,7 @@ document.querySelector("#artifact-probe").addEventListener("click", () => { await expectExecutableRuns(applicationPages, 3); await product.restart(); + front.setTarget(product.serverURL); // Allow one capped retry interval plus the usual 5s shell-ready budget. await Promise.all(applicationPages.map((applicationPage) => waitForApplicationRender( @@ -1845,6 +1853,8 @@ document.querySelector("#artifact-probe").addEventListener("click", () => { expect(scriptRequests.length).toBeGreaterThan(0); expect(scriptRequests.every((owner) => owner === "application")).toBe(true); } finally { + await requests.attach(test.info()); + await front.dispose(); await product.dispose(); await secondPage.close(); } diff --git a/packages/web/test/e2e/update-journey.spec.ts b/packages/web/test/e2e/update-journey.spec.ts index 42db665c..5ad0664c 100644 --- a/packages/web/test/e2e/update-journey.spec.ts +++ b/packages/web/test/e2e/update-journey.spec.ts @@ -1,4 +1,5 @@ -import { expect, test, type Page } from "@playwright/test"; +import { type Page } from "@playwright/test"; +import { expect, test } from "../../../../test/helpers/playwright.ts"; import { mkdtempSync, rmSync } from "node:fs"; import os from "node:os"; import path from "node:path"; diff --git a/packages/web/test/e2e/update-notification.01.spec.ts b/packages/web/test/e2e/update-notification.01.spec.ts index 2efacd40..f7841cc8 100644 --- a/packages/web/test/e2e/update-notification.01.spec.ts +++ b/packages/web/test/e2e/update-notification.01.spec.ts @@ -1,4 +1,5 @@ -import { expect, test, type Locator, type Page } from "@playwright/test"; +import { type Locator, type Page } from "@playwright/test"; +import { expect, test } from "../../../../test/helpers/playwright.ts"; import { mkdtempSync, rmSync } from "node:fs"; import os from "node:os"; import path from "node:path"; diff --git a/packages/web/test/e2e/update-notification.02.spec.ts b/packages/web/test/e2e/update-notification.02.spec.ts index a5cbbc2c..32f84770 100644 --- a/packages/web/test/e2e/update-notification.02.spec.ts +++ b/packages/web/test/e2e/update-notification.02.spec.ts @@ -1,4 +1,5 @@ -import { expect, test, type Page } from "@playwright/test"; +import { type Page } from "@playwright/test"; +import { expect, test } from "../../../../test/helpers/playwright.ts"; import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; import os from "node:os"; import path from "node:path"; diff --git a/packages/web/test/e2e/update-reload.spec.ts b/packages/web/test/e2e/update-reload.spec.ts index 8df87161..63589ffc 100644 --- a/packages/web/test/e2e/update-reload.spec.ts +++ b/packages/web/test/e2e/update-reload.spec.ts @@ -1,4 +1,5 @@ -import { expect, test, type Page } from "@playwright/test"; +import { type Page } from "@playwright/test"; +import { expect, test } from "../../../../test/helpers/playwright.ts"; import { mkdtempSync, rmSync } from "node:fs"; import os from "node:os"; import path from "node:path"; diff --git a/packages/web/test/foundation-copy.test.ts b/packages/web/test/foundation-copy.test.ts index 0fb0a976..ca180f14 100644 --- a/packages/web/test/foundation-copy.test.ts +++ b/packages/web/test/foundation-copy.test.ts @@ -1,6 +1,7 @@ import { globSync, readFileSync, statSync } from "node:fs"; import path from "node:path"; -import { describe, expect, test } from "vitest"; +import { describe, expect } from "vitest"; +import { test } from "../../../test/helpers/vitest.ts"; import { enumerateTrackedPaths } from "../../../scripts/test/file-inventory.mjs"; import { buildVersionedWebBundle } from "../../../test/helpers/versioned-web-bundle.ts"; import { diff --git a/proofs/arch/artifact-frame/artifact-bridge.md b/proofs/arch/artifact-frame/artifact-bridge.md index 2f1af862..ec5e3e85 100644 --- a/proofs/arch/artifact-frame/artifact-bridge.md +++ b/proofs/arch/artifact-frame/artifact-bridge.md @@ -79,7 +79,7 @@ Readiness assertions observe lifecycle state, never shielding, pointer suspensio - **Contract** (browser-bridge contract harness; the parent is mocked; authored platform identities cover macOS and Windows/Linux): the bridge captures option+arrow on macOS and ctrl+arrow on Windows/Linux for all four arrows, including in text-editing targets and with `repeat`; calls `preventDefault`; and posts `navigation-key`. Shift, every other extra modifier, composition, and non-arrow keys do not forward, while a descendant handler's `stopPropagation()` cannot starve the window capture listener — *(covered by tests in `packages/artifact/test/unit/artifact-bridge.test.ts`: "matches the platform navigation chord for every arrow", "rejects composition, non-arrows, missing chords, and every extra modifier", "captures matching chords in editing targets without stopping propagation", "leaves unmatched keydown behavior untouched", and "produces runnable source that behaves like installBridge when eval'd in a fresh window")*. ^ab-ac-navigation-key - **Contract** (browser-bridge and preload contract harnesses; parent `postMessage` and Electron IPC peers are mocked, forfeiting real handoff to the acceptance assertions below): each producer captures trusted pointer move, down, up, cancel, and click events and emits one `artifact-pointer` carrying the raw guest-viewport coordinates and button fields without preventing default, stopping propagation, or taking pointer capture; untrusted events are ignored — *(covered by `packages/artifact/test/unit/artifact-bridge.test.ts` “reports trusted artifact pointer input without interfering” and `packages/desktop/test/webview-bridge-preload.test.ts` “reports trusted webview pointer input without interfering”)*. ^ab-ac-pointer-producers - **Contract** (host contract harness plus production theme frames in a browser DOM; source windows, frame geometry, and pointer notifications are authored inputs, while browsing-context delivery is forfeited to the acceptance assertions): the current iframe or subscribed webview's source-valid `artifact-pointer` is accepted without readiness, mapped from guest coordinates through the frame's rendered rectangle and layout dimensions, normalized by the theme owner, and delivered once to each current theme frame. A zero layout dimension drops it, and the same payload from another window or replaced webview is ignored — *(covered by `packages/web/test/artifact-view.test.ts` “relays only current artifact pointer input to theme frames in application coordinates”)*. ^ab-ac-pointer-host -- **Acceptance** (outer boundary: physical pointer input in a real Chromium artifact iframe against a running production server and real theme package; the artifact fixture records its trusted event coordinates without altering input or delivery, and no mocks or production test hooks are used): move, down, up, and click cross the injected bridge and trusted channel to both theme frames in application-viewport coordinates, mapped from the coordinates actually received by the artifact (including Chromium's integer click coordinates), while the artifact's own click action remains effective — *(covered by `packages/web/test/e2e/theme-product-acceptance.test.ts` “executable themes run only in the Chromium application lifecycle and isolate failures”)*. ^ab-ac-pointer-browser +- **Acceptance** (outer boundary: physical pointer input in a real Chromium artifact iframe within the built application, served through the real stable-front proxy as harness transport, against a running production server and real theme package; the artifact fixture records its trusted event coordinates without altering input or delivery, and no mocks or production test hooks are used): move, down, up, and click cross the injected bridge and trusted channel to both theme frames in application-viewport coordinates, mapped from the coordinates actually received by the artifact (including Chromium's integer click coordinates), while the artifact's own click action remains effective — *(policy-grade test: `packages/web/test/e2e/theme-product-acceptance.test.ts` “executable themes run only in the Chromium application lifecycle and isolate failures”)*. ^ab-ac-pointer-browser - **Acceptance** (outer boundary: native mouse input in an ordinary HTML artifact webview in the real Electron app against a running production server and real theme package; no mocks or test hooks): the preload and subscribed IPC path deliver move, down, up, and click to both theme frames in application-viewport coordinates while the artifact's own click action remains effective — *(covered by `packages/desktop/test/e2e/webview-event-parity.test.ts` “executes an executable theme in Electron and resets on remote consent and selection changes”)*. ^ab-ac-pointer-electron - **Seam** (browser bridge → host navigation handler, crossed by real `postMessage` in the browser seam harness; an authored document and tab/channel state are fixtures, with no mock on the handoff): each of the four forwarded key values reaches the app's navigation handling while focus is inside the document — *(covered by test: `packages/web/test/e2e/artifact-view.test.ts` "forwards every browser navigation arrow from a focused artifact document")*; product outcomes compose with [product/keyboard-navigation.md](../../../specs/product/keyboard-navigation.md). ^ab-ac-navigation-key-seam - **Contract** (desktop shell side of the native-key delivery seam; authored native key events and platform identities cover macOS and Windows/Linux; the renderer navigation receiver is mocked, forfeiting the real handoff to [#^ab-ac-electron-key-seam](#^ab-ac-electron-key-seam)): the matcher implements the same modifier, arrow, repeat, and composition matrix as [#^ab-ac-navigation-key](#^ab-ac-navigation-key), consumes a match, and delivers its arrow value once to app navigation — *(covered by tests: all cases under `packages/desktop/test/native-navigation-key.test.ts` "Electron native navigation chord", plus `packages/desktop/test/main.test.ts` "consumes a matching webview chord and sends its arrow once to the embedding renderer" and `packages/web/test/renderer.test.ts` "delivers Electron native arrows to the existing application navigation edge")*. ^ab-ac-electron-key-contract @@ -116,6 +116,6 @@ The recorder's two branches hand their reports to the host through one function, Demo-mode frames are not bridge participants ([^ab-demo-frame-inert](../../../specs/arch/artifact-frame/artifact-bridge.md#^ab-demo-frame-inert)): -- **Seam** (Chromium against a really-running server under the server-home hook and cross-origin fixture server declared in [product/artifacts.md](../../product/artifacts.md#test-hooks); two external page artifacts on one channel and a second, newer channel are fixtures created through the production API; code evaluated inside the fixture page's frame posts bridge-shaped `bridge-ready`, a horizontal and a vertical `navigation-key` each chosen to have somewhere to move, `navigation-request`, and `artifact-pointer` messages to its parent through real `postMessage`; the default motion override is the only mock): after the page posts them, the focused channel and selected tab are unchanged, the fixture's first page is still framed at its own URL, and Back and Forward remain absent. Theme pointer delivery is not observed here; [#^ab-ac-demo-inert-host](#^ab-ac-demo-inert-host) carries it — *(covered by test: `packages/web/test/e2e/browser-demo-mode.test.ts` "bridge-shaped messages from a demo-mode page change nothing (^ab-ac-demo-inert-seam)")*. ^ab-ac-demo-inert-seam +- **Seam** (the built application in Chromium against a really-running server under the server-home hook and cross-origin fixture server declared in [product/artifacts.md](../../product/artifacts.md#test-hooks); two external page artifacts on one channel and a second, newer channel are fixtures created through the production API; code evaluated inside the fixture page's frame posts bridge-shaped `bridge-ready`, a horizontal and a vertical `navigation-key` each chosen to have somewhere to move, `navigation-request`, and `artifact-pointer` messages to its parent through real `postMessage`; the default motion override is the only mock): after the page posts them, the focused channel and selected tab are unchanged, the fixture's first page is still framed at its own URL, and Back and Forward remain absent. Theme pointer delivery is not observed here; [#^ab-ac-demo-inert-host](#^ab-ac-demo-inert-host) carries it — *(policy-grade test: `packages/web/test/e2e/browser-demo-mode.test.ts` "bridge-shaped messages from a demo-mode page change nothing (^ab-ac-demo-inert-seam)")*. ^ab-ac-demo-inert-seam - **Contract** (host contract harness; an authored external page artifact posed under browser demo mode, with synthetic `MessageEvent`s whose source is the frame's own window; the transport mock forfeits the real crossing to [#^ab-ac-demo-inert-seam](#^ab-ac-demo-inert-seam)): every message type in the [message protocol](../../../specs/arch/artifact-frame/artifact-bridge.md#message-protocol) produces no navigation-handler, pointer-forwarder, navigation-state, reload, or status-reply call and leaves the frame's trusted identity null — *(covered by test: `packages/web/test/artifact-view.test.ts` "browser demo mode artifact frame > acts on no bridge message from a demo-mode frame (^ab-ac-demo-inert-host)")*. ^ab-ac-demo-inert-host diff --git a/proofs/arch/test-runner/blaxel-testshards.md b/proofs/arch/test-runner/blaxel-testshards.md index 58b93ef0..0d0d8353 100644 --- a/proofs/arch/test-runner/blaxel-testshards.md +++ b/proofs/arch/test-runner/blaxel-testshards.md @@ -8,9 +8,13 @@ Proves [specs/arch/test-runner/blaxel-testshards.md](../../../specs/arch/test-ru The host marker’s provider-selection behavior is covered by [the runner proof](test-runner.md#^t-blaxel-host-marker). The corporate credential location is setup information checked by human review, not a new token-source mechanism. The existing token-source assertions remain unchanged; no test reads the vault or asserts its item name. -Coverage declarations are carried inside the migrated assertion blocks below. +Lease decisions and failure ordering are checked cheaply at the worker/coordinator contracts. Real Linux processes and files cross the guard, owner-registration, and deletion seams. These tests do not claim Blaxel transport coverage. -`test/repo/blaxel-stale-owner-audit.test.ts` checks diagnostic construction and coordinator ordering on every host. Its live diagnostic cases require Linux procfs and run only on Linux; they establish no macOS backend coverage. `test/repo/test-runner-provider.test.ts` exercises shell activation ordering with an authored nvm replacement in a shell isolated from host startup files. That contract check does not install nvm or establish real worker activation. +The live acceptances below enter through the canonical CLI on a dedicated pool: normal retrieval, refusal while another coordinator holds the worker, recovery after coordinator death, and pool creation/deletion. They share a small committed fixture selection and fresh workers rather than repeating a full suite for every failure permutation. Both planned and targeted execution paths are crossed. Resource retention composes with [reporting's diagnostics assertions](reporting.md#Resource diagnostics); helper teardown belongs to [the testing-policy proof](../testing-policy.md#Worker-owned builds). + +The spec's [Testing](../../../specs/arch/test-runner/blaxel-testshards.md#Testing) directives require real API/transport, fresh provisioning, and observed Node/Chromium temporary paths. Private workers and bounded disposable processes supply those observations; no deliberately exhausted host or shared-pool contamination is needed. The live driver and its native fixtures live in `experiment:blaxel-harness` and `experiment:blaxel-harness-fixtures`, excluded from `all` and ordinary CI by their experiment kind. Mocks below forfeit those crossings to the named live assertions. The live driver shares these observations within one fresh private pool; focused contracts carry uncommon failure permutations without repeating paid-service runs. + +The lease contracts and filesystem seams run only on Linux because their production helpers inspect procfs and mount records. This does not establish macOS worker support. `test/repo/blaxel-stale-owner-audit.test.ts` checks diagnostic construction and coordinator ordering on every host. Its live diagnostic cases require Linux procfs and run only on Linux; they establish no macOS backend coverage. `test/repo/test-runner-provider.test.ts` exercises shell activation ordering with an authored nvm replacement in a shell isolated from host startup files. That contract check does not install nvm or establish real worker activation. Repository selection composes several pieces across clean seams. A contract test proves URL conversion. A seam test runs real Git against a temporary checkout's `origin` to prove selection, and a process-boundary test runs the real coordinator to prove that an unusable selection stops it before dispatch. A further seam test executes the worker's generated `checkout` script under real `bash` and Git against two local bare repositories with shared history, proving that one persistent checkout moves between repositories in both directions without re-cloning or deleting ignored dependency files. That test reaches the repositories through local paths, so it forfeits GitHub HTTPS transport and token authentication. The coordinator's use of the selector and the generated script is a source assertion. The live runs that the spec's [Testing](../../../specs/arch/test-runner/blaxel-testshards.md#Testing) directive orders for a repository-selection change carry the GitHub transport, token, and real worker handoff: they run from checkouts of two repositories on the same pool, including workers whose checkout last served the other repository. When the two runs' commits have identical dependency declarations, those workers' logs must show no clone and a `deps` cache hit, which is the live evidence that switching repositories keeps installed dependencies. A focused test runs the worker's generated `deps` script after real checkouts of both repositories, with a stand-in `npm`. It proves that the cache decision follows the dependency declarations and ignores which repository supplied them. @@ -27,11 +31,19 @@ Pool discovery also has a contract test that runs the shared discovery helper us The SDK reads `BL_API_URL`, `BL_API_KEY`, and `BL_WORKSPACE`; the discovery contract test supplies a loopback API and fixture credentials in an isolated working directory and home, with an explicit child environment that preserves the test owner token and disables SDK telemetry and HTTP/2. Production uses the configured Blaxel API and credentials. +The worker lock and temporary-cleanup helpers accept filesystem roots for disposable seam fixtures; production uses the specified worker paths. The lease fixture supplies `auditPids`, its explicit process set, before procfs inspection. Production supplies no process set and enumerates the whole worker. This substitutes enumeration only and forfeits whole-worker coverage to the live acceptance; identity/token reads and signal revalidation remain real. Their decision layer accepts a clock, owner-audit/signal operations, and filesystem probes for contract failure injection; production uses worker time, procfs, signals, and the filesystem. + +A coordinator milestone hook, enabled only by `TV_TEST_RUNNER_SELFTEST`, can wait on a fixture-controlled rendezvous immediately before report download while renewal continues; ordinary runs do not wait. It changes scheduling, not lease state, process results, or report bytes. Live recovery uses the existing `--lock-ttl-seconds` override to shorten the abandonment wait, not a fake clock or altered retry budget. + +`retainAttemptEvidence` accepts the lease’s SDK filesystem and a coordinator download directory. The prepared-but-undispatched contract substitutes reads backed by real fixture files; worker inventory creation, teardown and local writes remain real. Production reads the worker results tree through the installed SDK. + +Pool-management contract tests may supply recording/failing SDK operations; ordinary pool commands use the installed SDK. These substitutions forfeit provider behavior to the live pool acceptance below. + ## Assertions ### Test assertions -- Pool discovery includes matching workers on later API pages and excludes workers with other project, purpose, pool, or architecture labels, as required by the spec's [Pool](../../../specs/arch/test-runner/blaxel-testshards.md#Pool) contract. **Contract:** the consumer is the shared discovery helper and installed SDK, called in a child process; the service is the HTTP mock described above. The helper collects the unfiltered listing before filtering locally. Both the manager mode (all matching workers) and coordinator mode (only matching `DEPLOYED` workers) must traverse all pages, including an earlier page containing no matching workers. A page-fetch failure rejects discovery rather than returning a partial pool. Fixture sandbox records omit a region so discovery needs no data-plane connection. (policy-grade test: `test/repo/blaxel-pool-discovery.test.ts`) ^blaxel-pool-pagination +- Pool discovery includes matching workers on later API pages and excludes workers with other project, purpose, pool, or architecture labels, as required by the spec's [Pool](../../../specs/arch/test-runner/blaxel-testshards.md#Pool) contract. **Contract:** the consumer is the shared discovery helper and installed SDK, called in a child process; the service is the HTTP mock described above. The helper collects the unfiltered listing before filtering locally. Both manager and coordinator modes must traverse all pages, including an earlier page containing no matching workers. The coordinator sees only published, matching `DEPLOYED` workers; the manager also retains the pool's unpublished and partially provisioned workers. A page-fetch failure rejects discovery rather than returning a partial pool. Fixture sandbox records omit a region so discovery needs no data-plane connection. (policy-grade test: `test/repo/blaxel-pool-discovery.test.ts`, including unpublished and partially provisioned worker records) ^blaxel-pool-pagination The repository-selection assertions: @@ -44,6 +56,7 @@ The repository-selection assertions: The remaining coverage lives in `test/repo/test-runner-provider.test.ts`. +- The explicit recovery-interval forwarding contract also pins the coordinator's 1200-second default in every timeout profile. This source check proves the configured values; the lease seam and live driver carry actual renewal behavior. - The canonical recommended shard count is 36, and the pool default remains 72 workers using `blaxel/playwright-chromium:latest`, 8192 MB, and the existing labels and naming scheme. - Pool-manager source assertions require pinned nvm `v0.40.3`, `.nvmrc` selection, fresh-worker provisioning before architecture verification, and no alternate image or size defaults. - Coordinator source assertions require nvm activation after checkout and before dependency/test work, and the generated `deps` script includes `.nvmrc` and the root manifest in its dependency cache identity. @@ -54,7 +67,35 @@ The remaining coverage lives in `test/repo/test-runner-provider.test.ts`. - Provider summaries and normalized run events preserve each shard's restore/wake, checkout, dependency cache outcome, readiness, surface compute, and report-download phases without parsing human logs. - The normal-lease acquisition scheduler never exceeds its concurrency bound, attempts each requested shard exactly once, and preserves input order in its outcomes. A partial batch writes a provider report containing acquired-but-not-run, unavailable, and thrown-error shard outcomes before releasing every partial success. - Signal shutdown writes completed, running, and pending shard outcomes before lease cleanup; a report-write failure does not suppress cleanup. -- Unconfirmed ordinary execution cleanup orders named stop, owner-token reaping, confirmed zero-owner state, then lock release; failed owner confirmation retains the lease until TTL. Completed and undispatched paths do not run descendant cleanup. +- Ordinary cleanup/release coverage is superseded by [the lease ordering contract](#^blaxel-t-lease-order) and [the live retrieval acceptance](#^blaxel-t-retrieval); shell completion alone no longer permits release. - A stale-owner setup failure records the contaminated sandbox and its lifecycle evidence, excludes that sandbox from subsequent attempts, and permits a configured infrastructure retry only on a distinct sandbox that independently passes lease acquisition and owner audit. -**Boundary gaps — stated honestly per [testing-policy.md](../../../specs/arch/testing-policy.md):** the classifier, normalization, bounded acquisition scheduler, and provisioning/activation/cache command shapes are covered without calling Blaxel. The scheduler test uses an injected acquisition callback and therefore does not cross the Blaxel API boundary. The code-only checks cannot prove nvm installation, selected Node/npm versions, or cache behavior inside the base image. After review, deleting and recreating `poc` and running an operator-approved targeted shard provide that seam evidence along with the real coordinator path: leasing sandboxes, lock protocol, architecture validation, checkout/runtime/dependency setup, streaming logs, and result download. There is no recorded-fixture stand-in for that external boundary today. Repository switching on real workers has no committed test; the live runs from two repositories' checkouts described in the coverage model provide that evidence for a repository-selection change. +**Boundary gaps — stated honestly per [testing-policy.md](../../../specs/arch/testing-policy.md):** the classifier, normalization, bounded acquisition scheduler, and provisioning/activation/cache command shapes are covered without calling Blaxel. The scheduler test uses an injected acquisition callback and therefore does not cross the Blaxel API boundary. Code-only checks cannot prove installation or operation inside the base image. The live assertions below supply that provider evidence. Repository switching on real workers has no committed test; the live runs from two repositories' checkouts described in the coverage model remain required for a repository-selection change. + +### Lease, recovery, and scratch + +- **Contract — lease decisions.** The shared lock decision layer consumes authored current, expired, legacy, incomplete, and unreadable records with a supplied worker clock and audit/signal results. Under [Locking](../../../specs/arch/test-runner/blaxel-testshards.md#Locking) and [recovery](../../../specs/arch/test-runner/blaxel-testshards.md#^blaxel-recovery), renewal preserves ownership only for the matching unexpired lease; the default recovery interval is 1200 seconds regardless of execution timeout. Expiry permits cleanup, not dispatch: both quiet and active abandoned runs reach a new lease only after confirmed cleanup and a clean audit. Unknown identity or surviving owners hold the worker. Incomplete/unreadable records have the same clean-audit requirement for acquisition and deletion. Legacy positive TTLs are respected, live legacy owners are skipped, and new records use `expiresAt: 0`. Audit mode never recovers or deletes residue. Mock clock and audit/signal operations forfeit real serialization and termination to the guard seam and live recovery below. (policy-grade tests: `test/repo/blaxel-lease-contracts.test.ts` “expired active and quiet leases use the same cleanup; uncertain owners hold”, “legacy, incomplete, unreadable and audit leases fail closed without clean ownership”, and “cleanup failures keep the old identity, while explicit deletion reserves an expired worker”; `test/repo/blaxel-worker-lease.test.ts` renewable-default and old-identity cases) ^blaxel-t-lease-decisions + +- **Contract — coordinator ordering and failures.** The coordinator's lease lifecycle uses recording/failing transport, download, and cleanup callbacks. Under [release](../../../specs/arch/test-runner/blaxel-testshards.md#^blaxel-release), renewal spans acquisition, execution, cleanup, and retrieval. A successful shell exit still requires descendant confirmation; unknown dispatch or failed cleanup cannot release. Successful retrieval and identity validation precede release; a download retry neither reruns tests nor resets worker results. An unused lease waits for pending writes and retains produced evidence. On interruption, the authoritative interrupted report precedes cleanup/retrieval and cannot become an ordinary completion report; cleanup still runs after a report-write error. A coordinator that loses ownership stops issuing mutations and reports infrastructure failure. These callbacks forfeit API, signals, and downloads to the live retrieval/recovery assertions; existing signal-report fixtures supply breadth without another native suite. (policy-grade tests: `test/repo/blaxel-lease-contracts.test.ts` “retrieval retries after teardown without rerunning work; uncertainty never releases”, “renewal continues while retrieval waits and stops on ownership loss”, and “preparation evidence is downloaded before an undispatched attempt releases”; `test/repo/test-runner-provider.test.ts` interrupted-report and explicit TTL-forwarding cases; `test/repo/blaxel-worker-lease.test.ts` delayed-entry case) ^blaxel-t-lease-order + +- **Seam — guarded registration and ownership.** Production worker-entry and lock helpers cross real Linux process/filesystem boundaries in a private fixture root as an ordinary repository test on Linux. It requires no Blaxel API or credentials; it runs only on Linux because it uses procfs, signals and the OS guard. Two competing acquisitions yield only one owner; killing a process holding the short guard releases the OS guard without deleting another lease. A request paused before entry registration, then resumed after ownership changes, cannot touch a checkout sentinel. Old-ID renewal/release cannot change the new record. Registered setup descendants preserve the outer token when a nested surface replaces its own token; recovery stops only the attributed PID/start identities, including a descendant that ignores TERM and needs KILL. A separate live fixture owner remains untouched and prevents reuse. The fixtures are small processes, records produced by the real helper, and rendezvous files. The supplied process set includes the registered supervisor, its reported descendant and the unrelated-owner sentinel. This mocks whole-worker enumeration and forfeits that protection to live recovery; procfs inspection, identity revalidation, signals and filesystem operations remain real. This proves [registration](../../../specs/arch/test-runner/blaxel-testshards.md#^blaxel-reservation) and [recovery](../../../specs/arch/test-runner/blaxel-testshards.md#^blaxel-recovery), not provider dispatch. (policy-grade tests: `test/repo/blaxel-worker-lease.test.ts` competing acquisition, delayed entry, old-identity and registered-descendant recovery cases) ^blaxel-t-worker-guard + +- **Seam — bounded filesystem cleanup.** The production cleaner crosses the real filesystem in a disposable parent containing both allowed legacy prefixes, stale scratch with complete and missing metadata, the current scratch, unrelated files/directories, and symlinks to outside sentinels. With a lease and clean-audit precondition supplied as a mock, it removes exactly the permitted real directories, leaves the current directory and outside sentinels intact, rejects symlink roots, and records counts, allocated bytes and free-space observations under [known leftovers](../../../specs/arch/test-runner/blaxel-testshards.md#^blaxel-known-leftovers). The audit mock forfeits real disuse confirmation to the live retrieval/recovery cases. No assertion equates allocated bytes with the filesystem's free-space increase. (policy-grade tests: `test/repo/blaxel-lease-contracts.test.ts` “only known real directories are removed, with allocated-byte evidence and outside links preserved”; `test/repo/blaxel-worker-lease.test.ts` “attempt identity and recovery evidence survive checkout resetting the result tree”) ^blaxel-t-scratch-cleanup + +- **Contract — cleanup uncertainty.** The cleaner's decision layer consumes authored filesystem-probe results and removal errors. An unexpected mount boundary is skipped, and an incomplete removal is reported without claiming the whole directory was reclaimed; neither permits new test dispatch under [Temporary files](../../../specs/arch/test-runner/blaxel-testshards.md#Temporary files). These mocked conditions forfeit actual filesystem operations to the cleanup seam and live retrieval assertion; a real mount or full disk adds no necessary coverage. (policy-grade tests: `test/repo/blaxel-lease-contracts.test.ts` “mount boundaries and incomplete removal remain reported failures without reclaimed counts” and “cleanup failures keep the old identity, while explicit deletion reserves an expired worker”) ^blaxel-t-scratch-errors + +- **Seam — unreadable process ownership.** The generated lock command runs without root privileges against real Linux procfs. On a root test worker the fixture drops the command to UID/GID 65534; otherwise it uses the current user. Its explicit process set permits acquisition without reading unrelated processes' environments, including a same-user nondumpable process on a root test host. The unscoped production audit refuses preparation when a process environment cannot be read, preserves the lease and known leftovers, and reports the PID and permission failure without environment contents. A root-only case changes a child's credentials after exec to make that same-user process nondumpable: the audit must still refuse it when included in the supplied process set. This checks [release](../../../specs/arch/test-runner/blaxel-testshards.md#^blaxel-release)'s unreadable-audit rule. Credential changes and procfs reads are real; the root-only case is skipped on unprivileged CI and runs on the private Blaxel worker. (policy-grade tests: `test/repo/blaxel-worker-lease.test.ts` “fixture audit runs without root access to unrelated process environments”, “an unreadable whole-worker audit preserves the lease and its evidence”, and “fixture scope still refuses an unreadable process in its process set”) ^blaxel-t-unreadable-audit + +Shared live declaration: the following acceptances run in `experiment:blaxel-harness` via `npm test -- local --surface experiment:blaxel-harness` on a dedicated credentialed Blaxel coordinator sandbox outside the worker pool, from a committed and pushed checkout, never a shared developer host or a leased worker, under [harness placement](../../../specs/arch/test-runner/test-runner.md#^blaxel-harness-placement). The caller supplies `TV_BLAXEL_HARNESS_COORDINATOR` and `TV_BLAXEL_HARNESS_POOL` as described in the [harness instructions](../../../test/runner-fixtures/blaxel-harness/README.md); no personal resource name is prescribed. They spawn the canonical CLI with real argv and inspect its exit and retained run directory. A fresh dedicated pool, at least two committed native fixture files in `experiment:blaxel-harness-fixtures`, known leftover directories and unrelated sentinels are fixtures. The two files permit the planned run's file-atomic two-shard assignment. Under [surface selection](../../../specs/arch/test-runner/test-registry.md#^blaxel-harness-surfaces), `npm test -- blaxel --tag blaxel-harness-fixture --pool --shards 2 --allow-shard-count-override 1` delegates `--suite selection --surfaces experiment:blaxel-harness-fixtures`; `npm test -- blaxel --surface experiment:blaxel-harness-fixtures --pool ` takes the targeted path. Workers read the committed registry. The coordinator, SDK, worker scripts, native runners, transport, ownership audits and filesystem operations are real. The report-download rendezvous above is the only scheduling hook; no mechanism is mocked. A bounded test driver owns and finally cleans up its coordinators and private workers. These tests do not invoke fixtures restricted to isolated GitHub hosts. + +- **Acceptance — reports before reuse.** A small planned run with at least two shards traverses real acquisition, architecture verification, checkout, runtime/dependency setup, execution and log/report transport under [the shard lifecycle](../../../specs/arch/test-runner/blaxel-testshards.md#Shard lifecycle). Each uploaded script path lies inside its current shard attempt directory. The retained setup phases include `temporary-cleanup`, with cleanup counts and free-space evidence. While it waits at the download rendezvous, observed renewals keep the leases valid; a competing canonical run cannot acquire the occupied capacity or alter its result files. After retrieval, the original native reports, setup/cleanup evidence and diagnostics have the original identities and readable local paths, scratch is gone, and the workers become available to a targeted run. Legacy-prefix and stale-scratch fixtures disappear, unrelated sentinels and persistent caches survive, and the next run does not accumulate an older results tree. The targeted run crosses its distinct dispatch path too. This proves [release](../../../specs/arch/test-runner/blaxel-testshards.md#^blaxel-release) and [scratch lifetime](../../../specs/arch/test-runner/blaxel-testshards.md#^blaxel-scratch); the shared diagnostics observations are owned by [reporting](reporting.md#^reporting-t-resource-live). (policy-grade test: `test/runner-fixtures/blaxel-harness/driver/harness.test.ts` “qualifies exclusive retrieval, active and quiet recovery, temporary placement, and pool lifecycle”, with `fixtures/outcomes.test.ts` and `fixtures/temporary.test.ts` under the same harness directory) ^blaxel-t-retrieval + +- **Acceptance — abandoned run recovery.** In two small scenarios, kill the real coordinator: once while a shard supervisor and a token-bearing child remain, and once after the shard stops but before download. The next canonical run, after the shortened configured recovery interval, stops the old owned processes where present, confirms a clean worker, clears allowed residue, and runs on that worker. Its downloaded recovery receipt retains the old identity, renewal/deadline, cleanup result and discarded-report flag across results reset. No old report becomes a pass. This proves the same [automatic recovery path](../../../specs/arch/test-runner/blaxel-testshards.md#^blaxel-recovery) works for active and quiet orphans without waiting for their voluntary exit. Failure permutations are carried by the contracts, not more live crash runs. (policy-grade test: `test/runner-fixtures/blaxel-harness/driver/harness.test.ts` “qualifies exclusive retrieval, active and quiet recovery, temporary placement, and pool lifecycle”, with `fixtures/outcomes.test.ts` and `fixtures/temporary.test.ts` under the same harness directory) ^blaxel-t-recovery-live + +- **Seam — temporary-directory inheritance.** In the live planned and targeted fixture runs above, the shard environment reaches real Node and Chromium: Node's `os.tmpdir()` and created files are inside the current `tmp/`, and actual Chromium shared-memory descriptor paths resolve there, including unlinked files. Inspect the launched browser's descriptors on the worker; an environment-variable echo alone is insufficient. Nested surface tokens must not erase the outer ownership token or `TMPDIR`. After process teardown, current scratch is absent while retained sidecars are readable. This honors [Temporary files](../../../specs/arch/test-runner/blaxel-testshards.md#^blaxel-scratch) and [Testing](../../../specs/arch/test-runner/blaxel-testshards.md#Testing). A small HTML page is the browser fixture; no browser or path mechanism is mocked, and no memory-pressure or exact byte-saving claim is made. (policy-grade test: `test/runner-fixtures/blaxel-harness/driver/harness.test.ts` “qualifies exclusive retrieval, active and quiet recovery, temporary placement, and pool lifecycle”, with `fixtures/outcomes.test.ts` and `fixtures/temporary.test.ts` under the same harness directory) ^blaxel-t-tmpdir-live + +### Pool mutation + +- **Contract — publication and deletion ordering.** The pool manager uses authored sandbox states and recording/failing SDK operations. Under [Pool](../../../specs/arch/test-runner/blaxel-testshards.md#Pool) and [pool operations](../../../specs/arch/test-runner/blaxel-testshards.md#Older runners and pool operations), it publishes eligibility only after successful provisioning/verification, refuses to refill partial pools, refuses deletion of valid leases, and unpublishes before deleting an idle or expired worker. An expired lease can be deleted despite failed process cleanup; a failed deletion stays unavailable and visible. Unreadable records follow the common contract above; command failure on a previously eligible worker holds/reports it, while a fresh worker never published may be deleted without a lease. SDK substitutions forfeit actual metadata changes and deletion to the live pool acceptance. (policy-grade tests: `test/repo/blaxel-lease-contracts.test.ts` “pool eligibility follows verification, and deletion reserves then unpublishes” and “failed deletion of never-published workers remains retryable without their runtime”) ^blaxel-t-pool-order + +- **Acceptance — fresh pool and safe deletion.** Through canonical pool commands and the real API, create the dedicated pool used by the live assertions, observe workers become eligible only after runtime/architecture verification, and run the fixtures on them. Attempt deletion while the retrieval gate holds valid leases: the command reports refusals and exits nonzero, and the original run still completes. After release, delete the pool and confirm its workers are absent; a new coordinator cannot dispatch there. This proves the [pool mutation](../../../specs/arch/test-runner/blaxel-testshards.md#Older runners and pool operations) and [operations](../../../specs/arch/test-runner/blaxel-testshards.md#Pool provisioning) spine through actual Blaxel label changes and deletion. These observations share the retrieval scenario rather than provisioning another pool. (policy-grade test: `test/runner-fixtures/blaxel-harness/driver/harness.test.ts` “qualifies exclusive retrieval, active and quiet recovery, temporary placement, and pool lifecycle”, with `fixtures/outcomes.test.ts` and `fixtures/temporary.test.ts` under the same harness directory) ^blaxel-t-pool-live diff --git a/proofs/arch/test-runner/reporting.md b/proofs/arch/test-runner/reporting.md index a214d39b..2ec31c9f 100644 --- a/proofs/arch/test-runner/reporting.md +++ b/proofs/arch/test-runner/reporting.md @@ -6,7 +6,11 @@ Proves [specs/arch/test-runner/reporting.md](../../../specs/arch/test-runner/rep ## Coverage model -Coverage declarations are carried inside the migrated assertion blocks below. +Native-report contracts carry normalization breadth; real native-runner and CLI crossings carry attempt capture and final output. The existing declarations below name those boundaries. + +Resource diagnostics compose a bounded writer contract, a real Linux probe/parent-lifecycle seam, and the dedicated Blaxel acceptances owned by [the provider proof](blaxel-testshards.md#Lease, recovery, and scratch). A small native failure and a recovered retry establish retained evidence; they do not reproduce resource exhaustion. Authored unavailable/reset/oversized probe inputs cover uncommon diagnostic failures without filling a disk or causing an OOM kill. Those mocks forfeit kernel reading and transport to the real seams. + +Planned and targeted execution share the collector; the parent seams cover capture and the dedicated Blaxel acceptances cover download. The same sidecars pass through provider normalization and attempt history; a recovered infrastructure attempt must not replace earlier evidence. Recovery receipts are observed in the provider's crash acceptance, not a duplicate crash here. No test claims start/end samples measure a peak or that an unavailable kernel log excludes OOM. The dedicated live acceptance checks download, reuse and recovery receipts alongside the focused bounds, parent capture and retained-link assertions. ## Test hooks @@ -15,6 +19,8 @@ Coverage declarations are carried inside the migrated assertion blocks below. `TV_VITEST_ATTEMPT_FILE` supplies the NDJSON output path to the production Vitest attempt recorder. The canonical runner and the planned shard worker set it to the current native run's attempt-result path. A Vitest invocation that configures the recorder outside those entrypoints must set it; the recorder fails without it. +The resource collector accepts a probe reader and clock; contract tests provide authored readings/errors and times, while production reads Linux counters, memory information and filesystem statistics at the current time. The writer accepts an output root and write operation; tests use disposable directories and may inject a write failure, while production writes into the current results tree. Local parent/child seam fixtures use the runner’s existing guarded registry hook, not a replacement reporter or fabricated native result. That hook is not forwarded to Blaxel workers. + ## Assertions ### Test assertions @@ -40,3 +46,12 @@ The event-normalization assertions are **contract tests** over recorded native-r - A planned shard worker running under GitHub Actions finalizes its own run directory and retains its shard summary, plan, native results, and logs inside it. - A repository workflow assertion pins full run-directory artifact upload on success and failure and a 30-day artifact retention period. +### Resource diagnostics + +- **Contract — bounded, honest observations.** The collector/writer consumes authored start/end memory, filesystem, OOM-counter and kernel-line readings, including unavailable probes, a counter reset and over-budget excerpts, and writes real files in a temporary results root. Under [resource diagnostics](../../../specs/arch/test-runner/reporting.md#^resource-diagnostics) and [bounds](../../../specs/arch/test-runner/reporting.md#^resource-diagnostic-bounds), values keep units/time, unknowns keep reasons rather than zero, and resets produce no negative OOM delta. A failed native attempt, even if later recovered, or incomplete execution requests the interval's kernel evidence. Authored timestamped lines prove that a whole-second read is filtered to the exact observation interval and untimestamped OOM lines are counted as omitted. The bounded reader/writer retains at most 64 KiB of kernel lines and 128 KiB per sidecar, and 8 MiB total per shard attempt including receipts; compact start/end evidence survives before optional excerpts, with explicit truncation. Only allowed fields are serialized: supplied credential, query/body, environment and argument sentinels do not appear. Injected probe and write failures warn without changing the original native outcome. Mock readings/write failures forfeit OS reading to the parent seam and actual download to the live assertion below. ([`test/repo/resource-diagnostics.test.ts`](../../../test/repo/resource-diagnostics.test.ts)) ^reporting-t-resource-bounds + +- **Seam — parent capture survives the child.** The production surface parent crosses real Linux procfs/filesystem probes and an installed native runner's child-process boundary in a disposable results root. Authored tiny native fixture tests produce a failure/retry and an exit before native output; both planned-worker and targeted execution wiring are exercised. The retained baseline exists before preparation begins and the final observation follows teardown; a terminated child cannot erase the baseline. A separate interrupted-parent fixture leaves the baseline with an explicitly missing final observation rather than invented zeroes. Real probe values are checked for their source/units and availability reasons, not exact memory totals. No kernel OOM is manufactured; inaccessible kernel evidence must remain unknown. No mechanisms are mocked. This proves the capture boundary in [resource diagnostics](../../../specs/arch/test-runner/reporting.md#^resource-diagnostics); the live assertion carries remote retrieval. ([`test/repo/test-runner-resource-parent.test.ts`](../../../test/repo/test-runner-resource-parent.test.ts); combined-project member attribution in [`test/repo/test-runner-shard-worker.test.ts`](../../../test/repo/test-runner-shard-worker.test.ts), “runs exact files from two Vitest projects in one process”) ^reporting-t-resource-parent + +- **Contract — retained links and attempt history.** Provider normalization and infrastructure-attempt aggregation consume authored shard/task envelopes referring to real temporary sidecar files, with recorded native failed/recovered fixtures. Under [normalized results](../../../specs/arch/test-runner/reporting.md#Normalized surface result) and [resource diagnostics](../../../specs/arch/test-runner/reporting.md#Resource diagnostics), each advertised `resourceDiagnosticsPaths` entry exists relative to the final run directory; earlier attempt evidence survives a later successful infrastructure attempt. Missing native output remains incomplete and cannot manufacture a selected-surface result. Samples and kernel excerpts appear in neither compact console output nor timing events. Mocked provider envelopes forfeit remote creation/download to the live assertion; recorded native inputs forfeit attempt capture to the real parent seam and existing native-recorder assertions above. ([`test/repo/resource-diagnostics-links.test.ts`](../../../test/repo/resource-diagnostics-links.test.ts)) ^reporting-t-resource-links + +- **Acceptance — evidence reaches the caller.** Share the dedicated-pool canonical CLI runs in [the provider retrieval acceptance](blaxel-testshards.md#^blaxel-t-retrieval), under its shared `experiment:blaxel-harness` declaration and dedicated coordinator placement. Its `experiment:blaxel-harness-fixtures` selection includes a committed fail-once native fixture that recovers under the default retry budget and a fixture that fails all attempts. Planned and targeted runs retain the native outcomes and their start/end sidecars after worker scratch is removed; every normalized diagnostic link opens a downloaded file with the correct run/shard/surface identity and native-attempt references. The failed run still fails, the recovered case retains both attempts, and any inaccessible kernel evidence says so. The provider's [crash acceptance](blaxel-testshards.md#^blaxel-t-recovery-live) supplies the downloaded recovery receipt and its disclosed report loss. Boundary: real CLI, Blaxel API, native runners and report files, with the provider proof's scheduling hook and no mocks. This proves [resource retention](../../../specs/arch/test-runner/reporting.md#Resource diagnostics) end to end without a resource-exhaustion experiment. (policy-grade test: `test/runner-fixtures/blaxel-harness/driver/harness.test.ts` “qualifies exclusive retrieval, active and quiet recovery, temporary placement, and pool lifecycle”, using the harness’s real failed/recovered native fixtures) ^reporting-t-resource-live diff --git a/proofs/arch/test-runner/sharded-execution.md b/proofs/arch/test-runner/sharded-execution.md index 8f289374..2e118ab8 100644 --- a/proofs/arch/test-runner/sharded-execution.md +++ b/proofs/arch/test-runner/sharded-execution.md @@ -8,6 +8,8 @@ Proves [specs/arch/test-runner/sharded-execution.md](../../../specs/arch/test-ru Coverage declarations are carried inside the migrated assertion blocks below. +The [Blaxel guard seam](blaxel-testshards.md#^blaxel-t-worker-guard) and [live temporary-directory seam](blaxel-testshards.md#^blaxel-t-tmpdir-live) prove that the outer shard attempt's ownership and `TMPDIR` survive the shared worker and nested surface environments required by [Shared shard worker](../../../specs/arch/test-runner/sharded-execution.md#Shared shard worker). The [planned live acceptance](blaxel-testshards.md#^blaxel-t-retrieval) also supplies real plan transfer. Diagnostic artifact fields and attempt-history retention in [the provider model](../../../specs/arch/test-runner/sharded-execution.md#Blaxel provider report shape) compose with [reporting's link contract](reporting.md#^reporting-t-resource-links) and [live retrieval](reporting.md#^reporting-t-resource-live). Those assertions have passed; this proof relies on their retained evidence without duplicate provider runs. + ## Test hooks @@ -49,4 +51,3 @@ Planner and baseline assertions are **contract tests** over temporary registries - Shard-count recommendations select the smallest simulated count under 120 seconds, obey the supplied capacity, and flag an atomic file over target. The baseline update reports only the Blaxel recommendation, simulated up to the Blaxel pool size. **Boundary statement:** real local runner processes prove explicit file selection for Vitest and the default Playwright command, report shape, and join logic. They do not prove that the Electron wrapper consumes a generated Playwright list, that a Blaxel worker accepts the transferred plan, or that independently started GitHub matrix jobs derive the same plan on hosted machines. Live Electron, Blaxel, and GitHub runs must validate those execution boundaries before activation. - diff --git a/proofs/arch/test-runner/test-registry.md b/proofs/arch/test-runner/test-registry.md index ff3af3aa..5a7edb9f 100644 --- a/proofs/arch/test-runner/test-registry.md +++ b/proofs/arch/test-runner/test-registry.md @@ -32,7 +32,7 @@ These run in-process as pure functions over the loaded registry, plus real files - `validateRegistry` ignores a runner config placed outside the canonical include roots (e.g. under `prototypes/` or `experiments/`). - The `all` suite resolves to the unit and non-agent e2e execution groups, in `order`, includes `unit:build-config`, and the `unit:workspaces` group contains the package unit surfaces in declared order. - Every lifecycle fault surface retains `kind: "experiment"` and `tag:isolated-github-only`; none is selected by `all`. +- **Contract — live harness selection.** Load the committed registry with the real selector and inspect the committed driver's dispatch selections. Both [Blaxel harness surfaces](../../../specs/arch/test-runner/test-registry.md#^blaxel-harness-surfaces) have `kind: "experiment"`; `all` and `e2e-ci` select neither, and tag `blaxel-harness-fixture` selects only `experiment:blaxel-harness-fixtures`. The driver's planned and targeted selections name only that fixture surface, never the driver, as required by [harness placement](../../../specs/arch/test-runner/test-runner.md#^blaxel-harness-placement). This is a source-level contract with no mocks or provider calls; real dispatch is carried by [the provider retrieval acceptance](blaxel-testshards.md#^blaxel-t-retrieval). (policy-grade test: `test/repo/test-runner-config.test.ts` “keeps Blaxel harness fixtures separate from the driver and ordinary suites”) ^registry-t-blaxel-harness-selection - The `unit:build-config` surface owns `test/repo/build-config-integrity.test.ts`; `unit:root` excludes that file. - The `telemetry-posthog-roundtrip` suite resolves to the `telemetry-posthog-roundtrip:integration` surface, and that surface owns `packages/server/test/telemetry-posthog.integration.test.ts`; `unit:server` excludes that file. - The `daemon-acceptance` suite resolves to `daemon-acceptance:cli`, and that surface exclusively owns `test/node/daemon-acceptance.test.ts`; `e2e:node` excludes that file. The surface declares the `daemon-test-host` preflight and is absent from `all`. - diff --git a/proofs/arch/test-runner/test-runner.md b/proofs/arch/test-runner/test-runner.md index 44eef1fc..d2d5ae0e 100644 --- a/proofs/arch/test-runner/test-runner.md +++ b/proofs/arch/test-runner/test-runner.md @@ -10,6 +10,8 @@ Coverage declarations are carried inside the migrated assertion blocks below. Pr The [combined-project seam](sharded-execution.md#^combined-vitest-projects) crosses the shared `vitestProjectsArgs` handoff that writes the combined config and selects it for the Vitest command used by local and sharded runs. It exercises project roots, exact file selection and attempt recording. The local wrapper's pre-commands, execution without an explicit file list and result splitting remain within the real local end-to-end boundary gap below; the shared-helper seam does not claim that whole path. +The Blaxel exception to ordinary live-owner protection in [Surface process lifecycle](../../../specs/arch/test-runner/test-runner.md#Surface process lifecycle) is proven by [the worker guard seam](blaxel-testshards.md#^blaxel-t-worker-guard) and [automatic recovery acceptance](blaxel-testshards.md#^blaxel-t-recovery-live). The same provider proof carries preservation of the outer shard attempt token through nested surface scopes. These assertions do not weaken or relocate the existing ordinary-owner cleanup tests, including fixtures restricted to isolated GitHub execution. + Two test-infrastructure rules concern the product's `tv serve` and are proven where the product is. The recipe for a test that spawns `tv serve` gives the server a temporary [Television home](../../../specs/product/cli.md#^cli-home) whose config file sets port `0`, passes that home with `--home` to the server and every client command, and passes the acquired port as client `--port`. It is a way of writing tests, honored by the design of the tests that spawn the server: the [product CLI proof](../../product/cli.md#Acceptance criteria: CLI spine) declares it for its acceptances, and review checks it elsewhere. No separate assertion is ordered for it; the repository scan below guards only numeric test-service URLs. The daemon-acceptance surface's form of installation (a temporary home, a released stable port written with `tv config set port`, then `tv --home serve --persist`) is carried by the same proof's [home-based persisted-install assertion](../../product/cli.md#^cli-ac-persist-home-install). ## Test hooks @@ -37,6 +39,7 @@ The runner's own tests run the **real `scripts/test/cli.mjs` process** with `spa - Targeted selectors (`--file`, `--surface`, `--package`, `--runner`, `--tag`) and the `--suite unit` / `--suite e2e` / `--suite telemetry-posthog-roundtrip` / `--suite daemon-acceptance` selections pass the guardrail unless a selected surface carries an execution-placement tag. - `daemon-acceptance` fails preflight before its build when `TV_DAEMON_TEST_HOST` is not exactly `1`, and passes that check when the designated-host acknowledgement is present. - `isolated-github-only` surfaces are rejected before preflight for developer local and every Blaxel path; the exact dedicated GitHub workflow identity plus explicit opt-in permits only its local execution path. +- **Acceptance — harness placement configuration.** The real canonical CLI accepts explicitly named coordinator and private-pool environment values for local harness selection without prescribing personal resource names. Missing configuration, an inherited shard owner, and remote driver selection are refused before preflight. Authored environment values and the existing guarded dry-run hook replace only downstream execution; this proves the CLI decision, not actual sandbox placement. The [live harness](blaxel-testshards.md#^blaxel-t-retrieval) checks coordinator labels and crosses the configured private pool. Proves [harness placement](../../../specs/arch/test-runner/test-runner.md#^blaxel-harness-placement). (policy-grade tests: `test/repo/test-runner-guardrails.test.ts` “accepts a configured Blaxel harness coordinator and private pool without fixed names” and “refuses missing Blaxel harness configuration and leased or remote placement before preflight”) ^t-blaxel-harness-placement - `--suite all` includes `unit:build-config`, which validates the ordinary CLI output and an isolated release-configured `--outfile` bundle without network or secrets, as proven by [the build seam](../telemetry/sink.md#^t-build-config-integrity). - **Acceptance** (real canonical CLI process, real temporary home with no marker): default `verify` and explicit `--provider auto` select local phases without bypass or Blaxel preflight/fallback diagnostics. The [preflight hook](preflight.md#Test hooks) `TV_TEST_RUNNER_FAKE_REMOTE_PREFLIGHT` is supplied with `TV_TEST_RUNNER_SELFTEST` explicitly removed from the child environment and must remain untouched: reaching remote preflight would fail the command, so a successful local plan proves it was not consulted. The same unauthorized input on a marked home must fail with the remote hook’s authorization error, confirming the detector is active. No remote mechanism is mocked on the unmarked path; `--plan` proves selection and phase construction, not suite execution. Proves [provider resolution](../../../specs/arch/test-runner/test-runner.md#Verify orchestration). Evidence: `test/repo/test-runner-guardrails.test.ts`. ^t-local-verify-default - **Acceptance** (real CLI process and temporary marked home; remote checks replaced by the declared preflight hook): a passing remote result makes default `verify --plan` select Blaxel phases. This proves provider selection, forfeiting real service and Git checks to the [preflight proof](preflight.md#Assertions), which records their committed boundary-test gap. The [live-evidence directive](../../../specs/arch/test-runner/preflight.md#Testing) governs live validation. Passing self-test results are refused outside `verify --plan`, before any remote dispatch. Proves [provider resolution](../../../specs/arch/test-runner/test-runner.md#Verify orchestration). Evidence: `test/repo/test-runner-guardrails.test.ts`. ^t-marked-verify-default diff --git a/proofs/arch/testing-policy.md b/proofs/arch/testing-policy.md index deca7180..78774cef 100644 --- a/proofs/arch/testing-policy.md +++ b/proofs/arch/testing-policy.md @@ -1,7 +1,25 @@ -*How the testing policy’s development discipline is checked.* +*How worker-owned test builds are proven, and how testing discipline is reviewed.* # Testing policy — proof Proves [specs/arch/testing-policy.md](../../specs/arch/testing-policy.md). -The policy governs authoring and execution discipline, which is checked by review of the contribution and its recorded validation. It orders no tests of its wording or of an agent’s compliance. Runtime enforcement belongs to the test-runner specs and their proofs; its pointer to the shared-branch workflow moves no runtime promise into this spec and orders no additional tests. +## Coverage model + +Authoring and execution discipline is checked by review of the contribution and its recorded validation. It orders no tests of its wording or of an agent's compliance. Runner enforcement belongs to the test-runner proofs. + +Reusable helper builds are runtime behavior owned here. A small contract covers cache publication and failed preparation; real Playwright and Vitest worker lifecycles prove build reuse and final disposal. These are helper tests, not additional product browser walks. Real builder output must be exercised at least once; injected builders earn only lifecycle coverage. + +The built-application default is honored by the setup of the product tests that need it. [Themes](../product/themes-and-appearance.md) and [demo mode](../product/artifacts.md#^af-ac-demo-mode) retain their product assertions while changing their asset-loading boundary. No source-text test asserts that every acceptance uses a particular helper. + +Killed workers cannot promise teardown. The [Blaxel scratch/recovery assertions](test-runner/blaxel-testshards.md#Lease, recovery, and scratch) carry the backstop rather than duplicating a crash experiment here. + +## Test hooks + +The shared build-lifetime helper accepts a temporary parent and preparation callback; contract tests may supply a builder that writes fixture files, waits on a rendezvous, or rejects, while real callers run the source-CLI or versioned-web preparation. Its teardown entry point is the same one the runner-worker adapters call; exposing it for a contract does not stand in for testing those adapters. + +## Worker-owned builds + +- **Contract — publish successful builds and dispose owned work.** With a real temporary filesystem and recording preparation callbacks, concurrent requests in one worker reuse a pending/successful build for the same key, while distinct version keys remain distinct. A failed preparation removes its partial output and is not published; a later request may prepare it again. Teardown waits for in-flight preparation to settle and removes all owned outputs, including one that failed after creating files. Per-case server disposal is covered by the real worker seam below. These callbacks mock build/asset preparation and forfeit actual executable output to the worker seam below. This proves [worker build lifetime](../../specs/arch/testing-policy.md#^worker-build-lifetime). (policy-grade tests: `test/repo/worker-builds.test.ts`, all four cases) ^testing-t-build-lifetime + +- **Seam — real runner workers own their builds.** The installed Playwright and Vitest runners each execute a small authored fixture through the actual shared helper/adaptor path, using a private temporary root. Two cases assigned to the same worker reuse the source-CLI preparation and a same-version web bundle; a later worker owns a separate lifetime. Source CLI output starts a real product server, and the real web output is served/read, establishing that published results are usable. Within this same fixture, a bundle path naming an absent child makes the real copy fail after allocation of a per-launch runtime/home; those private outputs are removed, and a subsequent valid launch still uses the worker build. The missing path is fixture data. After per-case server cleanup the shared outputs still exist; after normal worker teardown they do not. No runner, builder, filesystem or server mechanism is mocked. This crosses each framework's teardown boundary under [worker build lifetime](../../specs/arch/testing-policy.md#^worker-build-lifetime), without a browser or an extra product assertion. [The real version-stamp seam](updates/version-advertisement.md#^t-web-stamp) supplies version breadth; this seam need not rebuild a matrix of versions. (policy-grade tests: `test/node/worker-build-lifetime.e2e.test.ts`, the Playwright and Vitest cases with their authored native fixtures in `test/fixtures/worker-builds/`) ^testing-t-build-worker-seam diff --git a/proofs/arch/themes/delivery.md b/proofs/arch/themes/delivery.md index 96b5843d..9a468515 100644 --- a/proofs/arch/themes/delivery.md +++ b/proofs/arch/themes/delivery.md @@ -6,6 +6,8 @@ Proves [specs/arch/themes/delivery.md](../../../specs/arch/themes/delivery.md). ## Coverage model +The persistence and executable-lifecycle browser walks share the built-application setup declared in [the product coverage model](../../product/themes-and-appearance.md#Coverage model). Their retained request inventories establish that loading boundary; these are the same product walks, with no duplicate browser launch here. + Coverage declarations are carried inside the migrated assertion blocks below. The active-route acceptance crosses a really-running server and real package folders for ordinary files, escapes, and validators. JavaScript-route breadth covers all three reserved entries at that same public HTTP boundary: main-entry consent matches and mismatches exact IDs, each manifest declaration is independently enabled, disabled, or absent, and invalid syntax remains fixture bytes rather than being executed. A sibling JavaScript file remains an ordinary static asset. One missing-root recovery walk removes a package with every entry enabled and main consent granted, then proves empty CSS and JavaScript entries, unavailable ordinary assets, and exact restored bytes and validators. The package-root `README.md` exclusion retains its own assertion. Main-script coverage composes a mocked-DOM contract for the resource owner's element mechanics with application-composition contracts for confirmed consent, connection and event wiring, forced reload, and one-shot Settings restoration. DOM, service, history, and navigation mocks forfeit loading, execution, transport, and actual browser reload to [the product executable-theme browser acceptance](../../product/themes-and-appearance.md#^theme-ac-executable-browser) and [reset acceptance](../../product/themes-and-appearance.md#^theme-ac-executable-reset); they buy deterministic lifecycle breadth only. The reset contract deliberately follows installed include presence rather than inferred execution, so it also covers the harmless conservative reload when consent installed an include whose server-side manifest gate returned empty JavaScript. A same-ID manifest refresh remains an in-place include replacement even when it closes that gate. The same product proof carries the required real-Electron execution path. Static ownership and exclusion from standalone entry graphs live in [UI architecture](../ui/index.md#^ui-t-theme-resources). @@ -58,7 +60,7 @@ The [visible application-theme seam](#^theme-delivery-t-visible-app) is covered - **Acceptance** (outer boundary: the built application in Chromium and Firefox against a real server and scratch data directory; fixture: an installed theme importing a nested stylesheet with relative day/night SVG URLs; no mocks or test hooks): selecting the theme resolves its registered `--app-wallpaper-image` relative to the declaring stylesheet, loads the correct asset through the application's composed `--app-wallpaper`, changes the image with appearance, and clears it when selecting None — *(covered by test: `packages/web/test/e2e/theme-product-acceptance.test.ts` “theme-relative wallpaper URLs render through the built app token in Chromium and Firefox”)*. ^theme-delivery-t-wallpaper-url -- **Acceptance** (outer boundary: a real Chromium browser loads production shell modules and a live-canonical HTML artifact against a really-running server; a selected package with a byte-stable entry stylesheet, nested imported stylesheet, computed-presentation probes, and temporary data directory are fixtures; no mocks or test hooks): changing only the nested stylesheet crosses the real `theme-changed` transport and application-link replacement, then changes imported computed presentation in both document classes without registry refresh — *(covered by `packages/web/test/e2e/theme-product-acceptance.test.ts` “theme settings persist rendered app and artifact outcomes through a built-server restart”)*. ^theme-delivery-t-active-package-refresh +- **Acceptance** (outer boundary: a real Chromium browser loads the built application and a live-canonical HTML artifact against a really-running server; a selected package with a byte-stable entry stylesheet, nested imported stylesheet, computed-presentation probes, and temporary data directory are fixtures; the real stable-front proxy supplies harness transport; no mocks or test hooks): changing only the nested stylesheet crosses the real `theme-changed` transport and application-link replacement, then changes imported computed presentation in both document classes without registry refresh — *(policy-grade test: `packages/web/test/e2e/theme-product-acceptance.test.ts` “theme settings persist rendered app and artifact outcomes through a built-server restart”)*. ^theme-delivery-t-active-package-refresh - **Contract** (application theme-resource owner against a mocked DOM, forfeiting browser loading and execution to [the product browser acceptance](../../product/themes-and-appearance.md#^theme-ac-executable-browser)): a refresh creates one cache-busted classic external script in the document head at the connected server's absolute `/theme/main.js`; replacement detaches the prior element before appending the new one, clear removes it, and main-script operations neither replace nor reorder the stylesheet link or sandboxed frames — *(covered by `packages/web/test/theme.test.ts` “installs one cache-busted classic script from the connected server”, “removes the prior script before appending its replacement and clears it”, and “leaves stylesheet and sandboxed frame identity and ordering unchanged”)*. ^theme-delivery-t-app-script-module - **Contract** (renderer composition in a mocked DOM, with connection and application services replaced by controllable boundary mocks, forfeiting real connection event generation and browser resource loading to the browser and Electron assertions below): connection installs the application theme link before display state is ready; a selection change or same-ID resource-refreshing `theme-changed` event replaces it; a consent-membership-only event preserves it; and disconnect removes it — *(covered by test: `packages/web/test/renderer.test.ts` “ties the theme link to connection and theme events before display readiness” and “ties the theme script to confirmed consent, refresh, and connection lifecycle”)*. ^theme-delivery-t-app-link-renderer - **Contract** (application composition against controllable connection and application-service boundaries in a mocked DOM, forfeiting real event transport and script execution to [the product browser acceptance](../../product/themes-and-appearance.md#^theme-ac-executable-browser)): connection leaves the main script absent before confirmed display state; confirmed matching active-ID consent appends the first include only after first-party runtime setup; reconnect reevaluates confirmed state; opt-in injects without replacing the active stylesheet link or frame elements; active-package and registry events with unchanged selection and consent replace the stylesheet and main include; unrelated consent membership preserves all active resource identities; effective appearance changes preserve the stylesheet link and main include while frame replacement follows [the frame-appearance lifecycle](#^theme-delivery-t-frame-appearance-lifecycle); and disconnect removes them. Each path calls the one resource owner rather than using Settings registry state as an execution gate — *(covered by `packages/web/test/renderer.test.ts` “ties the theme script to confirmed consent, refresh, and connection lifecycle” and “reconciles theme frames across confirmed setup, refresh, selection, and connection state”)*. ^theme-delivery-t-app-script-renderer @@ -71,8 +73,8 @@ The [visible application-theme seam](#^theme-delivery-t-visible-app) is covered - **Contract** (application composition against controllable connection, display-state, and registry boundaries in a mocked DOM; deferred registry results and failures are authored outcomes; the mocks forfeit real transport and iframe replacement to [#^theme-delivery-t-frame-browser](#^theme-delivery-t-frame-browser)): selection change and same-ID active-package or registry refresh destroy both prior frames before resolving the current registry, then recreate only enabled destination frames. If initial or replacement lookup fails, both frames remain absent rather than retaining stale executable content; that failure does not interrupt the stylesheet, main-script, or application paths, and reconnect or a later selection, active-package, or registry trigger retries. A resolution or rejection from a superseded connection, theme ID, or lookup can neither install a frame nor remove frames installed by the current lookup. Consent-only events preserve both identities. Null selection and disconnect remove both frames; a registry refresh that disables either declaration leaves that frame absent while independently recreating the other enabled frame; and frame creation, replacement, or removal never requests a full application reload — *(covered by `packages/web/test/renderer.test.ts` “installs ready theme frames after deferred application mount”, “reconciles theme frames across confirmed setup, refresh, selection, and connection state”, “fails theme frame lookup closed while stylesheet, main script, and application continue”, “discards superseded theme frame resolutions and rejections”, and `packages/web/test/theme.test.ts` “remembers ordinary application focus and restores accidental current-frame focus”)*. ^theme-delivery-t-frame-lifecycle - **Contract** (the production application appearance-to-frame composition path in a mocked DOM; an effective root `data-theme` mutation, enabled registry snapshots, deferred fresh lookups, lookup failure, and rapid superseding mutations are authored outcomes; posing the root mutation and replacing the registry and DOM boundaries are mocks that forfeit the real appearance producer and event transport, script execution, and browsing-context replacement to [#^theme-delivery-t-frame-appearance-browser](#^theme-delivery-t-frame-appearance-browser)): under the [sandboxed-frame lifecycle](../../../specs/arch/themes/delivery.md#sandboxed-frames), observing an effective appearance change synchronously removes both current frames in the observer's callback turn before beginning the fresh registry lookup, preserves the active stylesheet and main-script elements without requesting application reload, then recreates every enabled frame for the new value. A failed initial or replacement appearance lookup leaves both absent; a later effective appearance change performs another fresh lookup and can restore enabled frames. A result superseded by a later appearance cannot install stale frames — *(covered by `packages/web/test/renderer.test.ts` “reconciles theme frames across confirmed setup, refresh, selection, and connection state”, “fails theme frame lookup closed while stylesheet, main script, and application continue”, and “discards superseded theme frame resolutions and rejections”)*. ^theme-delivery-t-frame-appearance-lifecycle - **Contract** (production host pointer forwarder against current frame-window and application-event boundary mocks in a browser DOM; the frame windows record `postMessage` and the event methods record interference, forfeiting real dispatch and hit testing to [#^theme-delivery-t-frame-browser](#^theme-delivery-t-frame-browser)): capture-phase listeners forward each trusted application-document `pointermove`, `pointerdown`, `pointerup`, `pointercancel`, and `click` once to each currently installed frame and never to a removed frame. The same internal forwarder accepts a source-authorized artifact input after the artifact owner maps it to application coordinates. Both paths map respectively to `television-theme-pointer-move`, `television-theme-pointer-down`, `television-theme-pointer-up`, `television-theme-pointer-cancel`, and `television-theme-pointer-click`; every message has only `type`, `clientX`, `clientY`, `button`, and `buttons`. Move sends `button: -1` and the input's current `buttons`; down sends the changed input `button` and post-transition `buttons` including the new press; up sends the changed input `button` and post-transition `buttons` excluding the release; click sends the clicked input `button` and `buttons: 0`; and cancel sends `button: -1` and `buttons: 0`, including for an authored cancel input reporting nonzero `buttons`. Each message uses the target frame's current `contentWindow` with destination `"*"`. Untrusted application events are ignored, messages are neither queued nor replayed, and forwarding never prevents default, stops propagation, or takes pointer capture. Any `message` event sourced from either current frame produces no application operation or reply regardless of payload, including an artifact-pointer lookalike — *(covered by `packages/web/test/theme.test.ts` “forwards the closed trusted pointer protocol without interfering or replaying” and “accepts no frame-originated command or reply path”, plus `packages/web/test/artifact-view.test.ts` “relays only current artifact pointer input to theme frames in application coordinates”)*. ^theme-delivery-t-frame-pointer -- **Acceptance** (outer boundary: the built application in Chromium against a really-running server and real package folders; current registry state, production host elements, public entry routes, the injected artifact bridge, and both theme iframe windows remain on the path; no mocks or test hooks): independently and jointly enabled entries execute automatically without main consent in separate `allow-scripts` opaque-origin documents, expose their cache-busted route through `document.currentScript.src`, cannot inspect application DOM, and contain load, syntax, and runtime failures. Real pointer move, down, up, cancel, and click input in the application document reaches both frames with the exact viewport coordinates and button fields while the same application sequence and click remain effective; a drag that produces no click forwards only its down, motion, and up transitions. Real move, down, up, and click input in a bridge-capable artifact also reaches both frames in application-viewport coordinates while the artifact's click remains effective. Ordinary application focus changes remain untouched. When either frame unexpectedly acquires focus, focus returns without scrolling to the last connected application element or, when unavailable, to `#app`; the production Settings popover remains operable in its document layer and an application modal dialog remains operable in the browser top layer — *(covered by `packages/web/test/e2e/theme-product-acceptance.test.ts` “executable themes run only in the Chromium application lifecycle and isolate failures”)*. This is the same browser walk as [product acceptance](../../product/themes-and-appearance.md#^theme-ac-sandboxed-frames), not a second launch. ^theme-delivery-t-frame-browser -- **Acceptance** (outer boundary: the production Settings appearance control in the built Chromium application against a really-running server; a real package enabling both sandbox entries, entry-script startup probes, application-color probes visible through transparent regions of both frames, and a temporary data directory are fixtures; no mocks or test hooks): under the [sandboxed-frame appearance contract](../../../specs/arch/themes/delivery.md#sandboxed-frames), an effective light-to-dark appearance change records both old frame removals before either replacement, leaves no old-appearance frame, and reruns both entries. Each replacement document root carries `data-theme="dark"`; its used root scheme and its host element's used scheme are both dark. Rendered-pixel probes behind otherwise empty portions of the background and overlay frames remain visible, proving transparent iframe canvas composition rather than only transparent computed backgrounds — *(covered by `packages/web/test/e2e/theme-product-acceptance.test.ts` “executable themes run only in the Chromium application lifecycle and isolate failures”)*. This is the same browser walk as [product frame-appearance acceptance](../../product/themes-and-appearance.md#^theme-ac-frame-appearance), not a second launch. ^theme-delivery-t-frame-appearance-browser +- **Acceptance** (outer boundary: the built application in Chromium against a really-running server and real package folders; current registry state, production host elements, public entry routes, the injected artifact bridge, and both theme iframe windows remain on the path; the real stable-front proxy supplies harness transport; no mocks or test hooks): independently and jointly enabled entries execute automatically without main consent in separate `allow-scripts` opaque-origin documents, expose their cache-busted route through `document.currentScript.src`, cannot inspect application DOM, and contain load, syntax, and runtime failures. Real pointer move, down, up, cancel, and click input in the application document reaches both frames with the exact viewport coordinates and button fields while the same application sequence and click remain effective; a drag that produces no click forwards only its down, motion, and up transitions. Real move, down, up, and click input in a bridge-capable artifact also reaches both frames in application-viewport coordinates while the artifact's click remains effective. Ordinary application focus changes remain untouched. When either frame unexpectedly acquires focus, focus returns without scrolling to the last connected application element or, when unavailable, to `#app`; the production Settings popover remains operable in its document layer and an application modal dialog remains operable in the browser top layer — *(policy-grade test: `packages/web/test/e2e/theme-product-acceptance.test.ts` “executable themes run only in the Chromium application lifecycle and isolate failures”)*. This is the same browser walk as [product acceptance](../../product/themes-and-appearance.md#^theme-ac-sandboxed-frames), not a second launch. ^theme-delivery-t-frame-browser +- **Acceptance** (outer boundary: the production Settings appearance control in the built Chromium application against a really-running server; a real package enabling both sandbox entries, entry-script startup probes, application-color probes visible through transparent regions of both frames, and a temporary data directory are fixtures; the real stable-front proxy supplies harness transport; no mocks or test hooks): under the [sandboxed-frame appearance contract](../../../specs/arch/themes/delivery.md#sandboxed-frames), an effective light-to-dark appearance change records both old frame removals before either replacement, leaves no old-appearance frame, and reruns both entries. Each replacement document root carries `data-theme="dark"`; its used root scheme and its host element's used scheme are both dark. Rendered-pixel probes behind otherwise empty portions of the background and overlay frames remain visible, proving transparent iframe canvas composition rather than only transparent computed backgrounds — *(policy-grade test: `packages/web/test/e2e/theme-product-acceptance.test.ts` “executable themes run only in the Chromium application lifecycle and isolate failures”)*. This is the same browser walk as [product frame-appearance acceptance](../../product/themes-and-appearance.md#^theme-ac-frame-appearance), not a second launch. ^theme-delivery-t-frame-appearance-browser - **Acceptance** (outer boundary: the real Electron app connected to a really-running server; an installed theme, its exact computed application-shell color, and an isolated Electron profile are fixtures; the declared desktop shell-version test hook forfeits only installed-version provenance to upgrade-gate coverage): selecting the installed theme renders its application style, and quitting and relaunching with the saved connection renders the same persisted theme without another selection — *(covered by test: `packages/desktop/test/e2e/webview-event-parity.test.ts` “renders the persisted app theme after quitting and relaunching”)*. ^theme-delivery-t-app-link-electron-startup - **Seam** (the production application theme-link module in Chromium against a really-running server on the app's origin and authored theme package resources; no mocks or test hooks): setup, selection, and watched-file refresh keep one cache-busted link last; each replaced link is detached; nested imports, font, and image resources load from the stable path; and appearance changes alter neither link identity nor request count — *(covered by test: `packages/web/test/e2e/appearance-delivery.test.ts` “theme link stays final and mode changes reuse its nested resources”)*. ^theme-delivery-t-app-link - **Acceptance** (outer boundary: the built application and a canonical-v2-styled HTML artifact in Chromium against a really-running server and its installed Clouds package; no mocks or test hooks): the marked app document receives the Clouds wallpaper through the stable route, the artifact keeps its canonical background without requesting or painting that wallpaper, and selecting `None` restores both bare baselines through the null theme while `/theme/theme.css` is empty — *(covered by `packages/web/test/e2e/appearance-delivery.test.ts` “Clouds targets the marked app without painting canonical artifacts”)*. ^theme-delivery-t-app-document diff --git a/proofs/index.md b/proofs/index.md index 1c3fbb7d..85f11e1d 100644 --- a/proofs/index.md +++ b/proofs/index.md @@ -14,7 +14,7 @@ Generated from the first-line description of every proof by `scripts/specs-index - `licensing.md` (33 lines) — *How Television's licensing promises are proven where users receive them: the license gate over the real tree, the packed CLI tarball, the desktop upload directory, the served browser notices, and Electron's license files in the installed desktop app.* - `tab-pages.md` (30 lines) — *How the promises in Tab pages are proven.* - `telemetry.md` (89 lines) — *How the promises in Telemetry are proven.* - - `themes-and-appearance.md` (55 lines) — *How the promises in Themes and appearance are proven.* + - `themes-and-appearance.md` (57 lines) — *How the promises in Themes and appearance are proven.* - `update-notifications.md` (88 lines) — *How Television's update promises are proven: real browser and Electron journeys for the server-update notice, auto-reload, the desktop self-update notice, the desktop recommendation and the gate, and the desktop product's update check on a candidate build for the desktop app's own updates.* - `versioning.md` (27 lines) — *How Television's release versions are proven where users receive them: the packed CLI package and the desktop upload directory built from the same commit, the built `tv` process and running server, update details, and the native About panel in the desktop product's Mac install check.* - **onboarding/** @@ -28,7 +28,7 @@ Generated from the first-line description of every proof by `scripts/specs-index - `licensing.md` (24 lines) — *How the licensing machinery is proven: generator, gate and loader contracts over authored inputs, and seams through every real shipping build, the suite's inventory handoff and the committed asset manifest.* - `making-skills.md` (18 lines) — *How the promises in Making skills are proven.* - `node-versions.md` (18 lines) — *How the Node version contracts are proven: repository declarations, the advisory floor in the packed CLI package, and the toolchain checker at its two developer entry points.* - - `testing-policy.md` (7 lines) — *How the testing policy’s development discipline is checked.* + - `testing-policy.md` (25 lines) — *How worker-owned test builds are proven, and how testing discipline is reviewed.* - **artifact-frame/** - `artifact-bridge.md` (121 lines) — *How the promises in Artifact bridge are proven.* - `index.md` (24 lines) — *How the promises in Artifact frame (architecture) are proven.* @@ -78,18 +78,18 @@ Generated from the first-line description of every proof by `scripts/specs-index - `sink.md` (58 lines) — *How the promises in Telemetry sink are proven.* - **test-runner/** - `attestation.md` (66 lines) — *How the promises in Test Attestation are proven.* - - `blaxel-testshards.md` (60 lines) — *How the promises in Blaxel Test Shards are proven.* + - `blaxel-testshards.md` (96 lines) — *How the promises in Blaxel Test Shards are proven.* - `flaky-tests.md` (16 lines) — *How the promises in Flaky Tests are proven.* - `github-ci.md` (53 lines) — *How the promises in GitHub Actions CI are proven.* - `preflight.md` (42 lines) — *How the promises in Preflight are proven.* - - `reporting.md` (42 lines) — *How the promises in Reporting are proven.* - - `sharded-execution.md` (52 lines) — *How the promises in Sharded Execution are proven.* + - `reporting.md` (57 lines) — *How the promises in Reporting are proven.* + - `sharded-execution.md` (53 lines) — *How the promises in Sharded Execution are proven.* - `test-registry.md` (38 lines) — *How the promises in Test Registry are proven.* - - `test-runner.md` (75 lines) — *How the promises in Test Runner are proven.* + - `test-runner.md` (77 lines) — *How the promises in Test Runner are proven.* - **themes/** - `authoring.md` (35 lines) — *How theme guidance, generated vocabulary and the authored application reference are proven at the built `television` skill boundary.* - `bundled-installation.md` (42 lines) — *How the promises in Bundled theme installation are proven.* - - `delivery.md` (87 lines) — *How the promises in Theme delivery are proven.* + - `delivery.md` (89 lines) — *How the promises in Theme delivery are proven.* - `index.md` (63 lines) — *How the promises in Theme architecture are proven.* - **ui/** - `conformance.md` (15 lines) — *How production UI conformance is checked while automated markup comparison is scheduled for TV-649.* diff --git a/proofs/product/artifacts.md b/proofs/product/artifacts.md index e402e310..17124fc6 100644 --- a/proofs/product/artifacts.md +++ b/proofs/product/artifacts.md @@ -6,7 +6,7 @@ Proves [specs/product/artifacts.md](../../specs/product/artifacts.md). ## Coverage model -Coverage declarations are carried inside the migrated assertion blocks below. +Coverage declarations are carried inside the migrated assertion blocks below. The two demo-mode acceptances and the bridge non-participation seam share the built application and built unsupported-URL view from the real product-server helper. Each walk retains a request-event inventory of its built document and assets; these observations use the same browser session as the product assertions. ## Test hooks @@ -49,12 +49,12 @@ Coverage relationship: serving an artifact's content at `/artifact//` to a b Coverage relationship: fresh loading after a channel switch is crossed by [arch/artifact-frame/reload-navigation.md#^rn-ac-persist-browser](../arch/artifact-frame/reload-navigation.md#^rn-ac-persist-browser), whose real-browser case removes every frame while another channel is current and observes a newly loaded frame on return. Its history restoration is reload-navigation's promise; this spec adds no second channel-switch walk. Rearrangement continuity is deliberately not promised, so it creates no assertion. -Browser demo mode ([#^af-demo-mode](../../specs/product/artifacts.md#^af-demo-mode)) is proven by two Chromium acceptance walks, with a routing contract for the breadth of which artifacts it reaches. Shared declaration for the two walks: the outer boundary is Chromium driving the production app against a really-running server launched with the server-home hook above; the external page is a two-page HTML fixture served by a local HTTP server on an origin other than the Television server's, never a public site; the external page artifact is created through the production API; the default motion override is the only mock. Playwright reads the cross-origin frame's document to observe which fixture page is shown. +Browser demo mode ([#^af-demo-mode](../../specs/product/artifacts.md#^af-demo-mode)) is proven by two Chromium acceptance walks, with a routing contract for the breadth of which artifacts it reaches. Shared declaration for the two walks: the outer boundary is Chromium driving the built application and built unsupported-URL view against a really-running server launched with the server-home hook above; the external page is a two-page HTML fixture served by a local HTTP server on an origin other than the Television server's, never a public site; the external page artifact is created through the production API; the default motion override is the only mock. Playwright reads the cross-origin frame's document to observe which fixture page is shown. -- **Acceptance** (shared demo-mode walk): with no marker file, the external page artifact's frame shows the unsupported-URL error page. After the marker file is created and the app is reloaded, the frame's document is the fixture's first page at its own URL, the frame shows no Back or Forward control, and the artifact icon is a button named by [content.yml#return_aria_label](../../specs/ui/app/artifact-frame/content.yml#return_aria_label). After the marker file is deleted and the app is reloaded again, the frame shows the unsupported-URL error page and the icon is not a button — *(covered by test: `packages/web/test/e2e/browser-demo-mode.test.ts` "the home-directory marker turns direct external pages on and off at app load (^af-ac-demo-mode)")*. ^af-ac-demo-mode -- **Acceptance** (shared demo-mode walk, marker present): following the fixture's link shows its second page; pressing the artifact icon shows the first page again. Following the link again and reloading the app shows the first page. Following the link once more, switching to another channel, and returning shows the first page. Back and Forward remain absent throughout — *(covered by test: `packages/web/test/e2e/browser-demo-mode.test.ts` "the artifact icon, an app reload, and a channel round trip each return to the original page (^af-ac-demo-return)")*. ^af-ac-demo-return +- **Acceptance** (shared demo-mode walk): with no marker file, the external page artifact's frame shows the unsupported-URL error page. After the marker file is created and the app is reloaded, the frame's document is the fixture's first page at its own URL, the frame shows no Back or Forward control, and the artifact icon is a button named by [content.yml#return_aria_label](../../specs/ui/app/artifact-frame/content.yml#return_aria_label). After the marker file is deleted and the app is reloaded again, the frame shows the unsupported-URL error page and the icon is not a button — *(policy-grade test: `packages/web/test/e2e/browser-demo-mode.test.ts` "the home-directory marker turns direct external pages on and off at app load (^af-ac-demo-mode)")*. ^af-ac-demo-mode +- **Acceptance** (shared demo-mode walk, marker present): following the fixture's link shows its second page; pressing the artifact icon shows the first page again. Following the link again and reloading the app shows the first page. Following the link once more, switching to another channel, and returning shows the first page. Back and Forward remain absent throughout — *(policy-grade test: `packages/web/test/e2e/browser-demo-mode.test.ts` "the artifact icon, an app reload, and a channel round trip each return to the original page (^af-ac-demo-return)")*. ^af-ac-demo-return - **Contract** (the production artifact-frame view under jsdom with an authored external page artifact and a fake application service posing demo mode on — mocks that forfeit real document loading to [#^af-ac-demo-return](#^af-ac-demo-return); theme-change and artifact-content-change events are injected on that service): neither event reassigns the demo page's frame `src`, which stays the artifact's own URL — *(covered by test: `packages/web/test/artifact-view.test.ts` "browser demo mode artifact frame > leaves a demo-mode page's URL untouched through theme and content change events (^af-ac-demo-untouched)")*. ^af-ac-demo-untouched - **Contract** (the production routing functions `artifactRenderRoute` and `artifactDocumentRoute` over authored artifact records and recorded current URLs; no mocks): with demo mode on in a browser, `http:` and `https:` artifact URLs that are not Television artifact addresses route to a direct iframe at the artifact's own URL, including a URL on the Television host itself; a Television artifact address keeps the shared-artifact route both on another host and on the connected Television host; a non-web URL keeps the unsupported-URL page. Path and Markdown artifacts, a Television-served artifact whose recorded current URL is an external web page, and every artifact in the desktop app route identically with demo mode on and off — *(covered by tests: `packages/web/src/artifact-dispatcher.test.ts` "browser demo mode routing", all four cases)*. ^af-ac-demo-scope -Coverage relationship: the two walks cross the real server's marker-file check, its delivery to the browser, routing, and frame rendering, and compose with [#^af-ac-demo-scope](#^af-ac-demo-scope), which carries the classification and routing breadth off the walked path, and [#^af-ac-demo-untouched](#^af-ac-demo-untouched), which keeps Television's own reload triggers away from the page. The walks reach the server through the development proxy, so app and server share an origin; that the flag route sends no CORS headers is [artifact isolation's route contract](../arch/artifact-frame/isolation.md#^iso-t-routes). That demo pages cannot forward the navigation chord or feed theme pointer effects is carried by the bridge's non-participation assertions [arch/artifact-frame/artifact-bridge.md#^ab-ac-demo-inert-seam](../arch/artifact-frame/artifact-bridge.md#^ab-ac-demo-inert-seam) and [arch/artifact-frame/artifact-bridge.md#^ab-ac-demo-inert-host](../arch/artifact-frame/artifact-bridge.md#^ab-ac-demo-inert-host). A site that refuses framing is outside Television's behavior, since Television only leaves the browser's own outcome in place, and orders no test. Which views show the return button, and what pressing it requests, are the frame's contracts at [ui/app/artifact-frame/index.md#^af-ui-ac-return-scope](../ui/app/artifact-frame/index.md#^af-ui-ac-return-scope) and [ui/app/artifact-frame/index.md#^af-ui-ac-frame-actions](../ui/app/artifact-frame/index.md#^af-ui-ac-frame-actions). +Coverage relationship: the two walks cross the real server's marker-file check, its delivery to the browser, routing, and frame rendering, and compose with [#^af-ac-demo-scope](#^af-ac-demo-scope), which carries the classification and routing breadth off the walked path, and [#^af-ac-demo-untouched](#^af-ac-demo-untouched), which keeps Television's own reload triggers away from the page. The built application and server share an origin; that the flag route sends no CORS headers is [artifact isolation's route contract](../arch/artifact-frame/isolation.md#^iso-t-routes). That demo pages cannot forward the navigation chord or feed theme pointer effects is carried by the bridge's non-participation assertions [arch/artifact-frame/artifact-bridge.md#^ab-ac-demo-inert-seam](../arch/artifact-frame/artifact-bridge.md#^ab-ac-demo-inert-seam) and [arch/artifact-frame/artifact-bridge.md#^ab-ac-demo-inert-host](../arch/artifact-frame/artifact-bridge.md#^ab-ac-demo-inert-host). A site that refuses framing is outside Television's behavior, since Television only leaves the browser's own outcome in place, and orders no test. Which views show the return button, and what pressing it requests, are the frame's contracts at [ui/app/artifact-frame/index.md#^af-ui-ac-return-scope](../ui/app/artifact-frame/index.md#^af-ui-ac-return-scope) and [ui/app/artifact-frame/index.md#^af-ui-ac-frame-actions](../ui/app/artifact-frame/index.md#^af-ui-ac-frame-actions). diff --git a/proofs/product/themes-and-appearance.md b/proofs/product/themes-and-appearance.md index 210a5ae0..605d4336 100644 --- a/proofs/product/themes-and-appearance.md +++ b/proofs/product/themes-and-appearance.md @@ -6,6 +6,8 @@ Proves [specs/product/themes-and-appearance.md](../../specs/product/themes-and-a ## Coverage model +The persistence and executable-lifecycle walks in `theme-product-acceptance.test.ts` load the built application through the real stable-front proxy, keeping the browser origin and authentication across backends that bind port `0` on restart. Request-event inventories retained by both walks establish the built document and asset paths across their navigations. Their product outcomes, fixtures and motion declarations stay the same. The separate `settings.test.ts` registry/control walk keeps its existing development-module boundary. Reset already uses the built application and is unchanged. + Coverage declarations are carried inside the migrated assertion blocks below. The executable-theme browser spine uses two connected application documents in one real Chromium session against a running production server. Real package fixtures drive main-enabled, unflagged, iframe-enabled, syntax-error, and uncaught-exception paths through Settings, persisted display state, the three public script routes, the production application resource owner, two real iframe boundaries, and an artifact document. Theme-owned markers, execution counters, package assets, and frame messages make absence before main consent, main execution in both application documents, automatic sandbox execution, repeat execution, document-lifetime effects, reload reset, classic-script-relative URL resolution, and frame replacement observable without a test hook or mock. Browser error events and ordinary application controls establish failure isolation. @@ -39,16 +41,16 @@ Clouds' association with Television 1.3 composes with [product versioning's rele ### Acceptance criteria -- **Acceptance** (outer boundaries: real Settings controls over HTTP and websocket connections; the persistence matrix loads production shell modules through the test Vite server against a built server, while the cross-browser `None`/Clouds case loads that server's built shell; valid, invalid, and dot-prefixed theme folders and real temporary homes are fixtures; no mocks or test hooks): a fresh installation offers None plus the themes in [the bundled inventory](../../specs/ui/themes/bundled.yml); refresh shows valid packages and readable invalid-folder errors while every dot-prefixed folder is absent from options and errors; selection propagates to a second client; `None` selects the null theme and clears the active theme ID; the selected theme renders in Chromium and Firefox; and the server-confirmed selection survives a server restart and reconnection in the same browser page — *(covered by `packages/web/test/e2e/settings.test.ts` “settings controls cross the registry and confirmed display state” and `packages/web/test/e2e/theme-product-acceptance.test.ts` “the real dropdown renders its selected built-shell theme in Chromium and Firefox” and “theme settings persist rendered app and artifact outcomes through a built-server restart”)*. ^theme-ac-selection +- **Acceptance** (outer boundaries: real Settings controls over HTTP and websocket connections; the persistence and cross-browser `None`/Clouds walks load the built application, while the separate registry/control walk in `settings.test.ts` retains its development-module boundary; valid, invalid, and dot-prefixed theme folders and real temporary homes are fixtures; the persistence walk uses the real stable-front proxy as harness transport; no mocks or test hooks): a fresh installation offers None plus the themes in [the bundled inventory](../../specs/ui/themes/bundled.yml); refresh shows valid packages and readable invalid-folder errors while every dot-prefixed folder is absent from options and errors; selection propagates to a second client; `None` selects the null theme and clears the active theme ID; the selected theme renders in Chromium and Firefox; and the server-confirmed selection survives a server restart and reconnection in the same browser page — *(policy-grade tests: `packages/web/test/e2e/settings.test.ts` “settings controls cross the registry and confirmed display state” and `packages/web/test/e2e/theme-product-acceptance.test.ts` “the real dropdown renders its selected built-shell theme in Chromium and Firefox” and “theme settings persist rendered app and artifact outcomes through a built-server restart”)*. ^theme-ac-selection - **Acceptance** (outer boundary: Firefox drives the real Settings controls in the built application against a really-running production server and its real event transport; fixed-dark and light theme packages, a ready Markdown editor with authored contents, and a temporary home are fixtures; no mocks or test hooks): selecting the light theme changes both the shell and the same loaded Markdown editor from the fixed-dark presentation to the light presentation without replacing the editor document or losing its contents — *(covered by test: `packages/web/test/e2e/theme-product-acceptance.test.ts` “Firefox refreshes a ready Markdown editor from a fixed-dark theme to light”)*. The same browser walk supplies the delivery evidence at [theme-delivery-t-markdown-refresh-firefox](../arch/themes/delivery.md#^theme-delivery-t-markdown-refresh-firefox). ^theme-ac-markdown-refresh-firefox -- **Acceptance** (outer boundary: two real Chromium application documents drive the built application's Settings controls against a running production server and its real event transport; main-enabled, unflagged, syntax-error, and uncaught-exception theme packages, a package-relative asset, an artifact document, and a temporary home are fixtures; no mocks or test hooks): before consent, the selected `enableMainJS: true` theme has no `main.js` include or request in either application document. Granting exact-ID consent runs its classic main script in both documents after the application runtime is available, exposes `document.currentScript.src`, resolves the package asset from that URL, survives a built-server restart, and leaves the artifact script-free. Appearance changes do not rerun it. An unrelated active-package edit, restoration after complete package-root loss, and server reconnection each rerun it; the missing-root interval executes nothing. An unflagged `main.js` executes nowhere. Separately consented syntax-error and uncaught-exception main scripts report browser errors and leave an ordinary Settings control operational — *(covered by `packages/web/test/e2e/theme-product-acceptance.test.ts` “executable themes run only in the Chromium application lifecycle and isolate failures”)*. ^theme-ac-executable-browser +- **Acceptance** (outer boundary: two real Chromium application documents drive the built application's Settings controls against a running production server and its real event transport; main-enabled, unflagged, syntax-error, and uncaught-exception theme packages, a package-relative asset, an artifact document, and a temporary home are fixtures; the real stable-front proxy supplies harness transport; no mocks or test hooks): before consent, the selected `enableMainJS: true` theme has no `main.js` include or request in either application document. Granting exact-ID consent runs its classic main script in both documents after the application runtime is available, exposes `document.currentScript.src`, resolves the package asset from that URL, survives a built-server restart, and leaves the artifact script-free. Appearance changes do not rerun it. An unrelated active-package edit, restoration after complete package-root loss, and server reconnection each rerun it; the missing-root interval executes nothing. An unflagged `main.js` executes nowhere. Separately consented syntax-error and uncaught-exception main scripts report browser errors and leave an ordinary Settings control operational — *(policy-grade test: `packages/web/test/e2e/theme-product-acceptance.test.ts` “executable themes run only in the Chromium application lifecycle and isolate failures”)*. ^theme-ac-executable-browser - **Acceptance** (outer boundary: the same two real Chromium application documents and running production server; a main-enabled theme whose script installs document-lifetime probes, a second theme, and persisted Settings-open state are fixtures; no mocks or test hooks): while the main include is active, withdrawing consent, selecting the second theme, and selecting `None` each reload both connected application documents, clear the prior script's document-lifetime effects, and start under the confirmed destination state. Only a document whose Settings popover was open reopens it; its URL marker is consumed before reopening and causes no later reload. After selecting the second theme, selecting the still-consented main-enabled theme again reinstalls its include and executes it without another consent grant. Changing themes while no main include is installed remains in place — *(covered by `packages/web/test/e2e/theme-product-acceptance.test.ts` “executable theme reset reloads connected Chromium documents under confirmed state”)*. ^theme-ac-executable-reset -- **Acceptance** (outer boundary: the same real Chromium application and running production server; a package independently enabling both sandbox entries, a destination theme, syntax-error and uncaught-exception iframe entries, Settings and application modal-dialog controls, and pointer-observation probes are fixtures; no mocks or test hooks): without main-page consent, the background and foreground entries execute in separate opaque-origin `allow-scripts` frames at their specified layers and cannot access the application DOM. Theme selection and an unrelated active-package edit replace both frame documents; package-root loss replaces them with empty entries, restoration executes fresh frames, and registry refresh that disables either declaration removes only that frame. Syntax and uncaught runtime errors stay in their affected frames while the application operates. Trusted pointer move, down, up, cancel, and click events in the application document reach both frames with viewport coordinates and button state while the same underlying application sequence, including click, remains effective; a drag that produces no click forwards its down, movement, and up but no click. Pointer move, down, up, and click inside the bridge-capable artifact reach both frames in application-viewport coordinates while the artifact's own click remains effective. Unexpected focus returns first to the last connected application element and then to `#app` when that element is unavailable. The foreground layers do not prevent using the production Settings popover in its document layer or an application modal dialog in the browser top layer — *(covered by `packages/web/test/e2e/theme-product-acceptance.test.ts` “executable themes run only in the Chromium application lifecycle and isolate failures”)*. ^theme-ac-sandboxed-frames -- **Acceptance** (outer boundary: the production Settings appearance control in a real Chromium application against a running production server; a real active package enabling both sandbox entries, entry-script startup probes, visible application probes behind transparent regions of both frames, and a temporary home are fixtures; no mocks or test hooks): under the [frame lifecycle](../../specs/product/themes-and-appearance.md#^theme-iframe-lifecycle), an effective light-to-dark appearance change removes both old frame documents before installing replacements, leaves no old-appearance frame, and reruns both entries in documents whose root `data-theme` is `dark`. Each replacement host and document root has a used dark scheme, and the application probes remain visibly unchanged through the frames' transparent regions — *(covered by `packages/web/test/e2e/theme-product-acceptance.test.ts` “executable themes run only in the Chromium application lifecycle and isolate failures”)*. ^theme-ac-frame-appearance -- **Acceptance** (outer boundary: a real Chromium browser loads production shell modules and a live-canonical HTML artifact against a running built server and its real `/events` transport; a real active package with a byte-stable entry stylesheet, nested imported stylesheet, computed-presentation probes, and a temporary home are fixtures; no mocks or test hooks): changing only the nested stylesheet updates the application and artifact presentation; moving the complete active package away produces their foundation and canonical baselines while the exact selected theme ID remains confirmed; restoring it at the same path restores both presentations and keeps that ID, without registry refresh or another selection — *(covered by `packages/web/test/e2e/theme-product-acceptance.test.ts` “theme settings persist rendered app and artifact outcomes through a built-server restart”)*. ^theme-ac-active-package +- **Acceptance** (outer boundary: the same real Chromium application and running production server; a package independently enabling both sandbox entries, a destination theme, syntax-error and uncaught-exception iframe entries, Settings and application modal-dialog controls, and pointer-observation probes are fixtures; the real stable-front proxy supplies harness transport; no mocks or test hooks): without main-page consent, the background and foreground entries execute in separate opaque-origin `allow-scripts` frames at their specified layers and cannot access the application DOM. Theme selection and an unrelated active-package edit replace both frame documents; package-root loss replaces them with empty entries, restoration executes fresh frames, and registry refresh that disables either declaration removes only that frame. Syntax and uncaught runtime errors stay in their affected frames while the application operates. Trusted pointer move, down, up, cancel, and click events in the application document reach both frames with viewport coordinates and button state while the same underlying application sequence, including click, remains effective; a drag that produces no click forwards its down, movement, and up but no click. Pointer move, down, up, and click inside the bridge-capable artifact reach both frames in application-viewport coordinates while the artifact's own click remains effective. Unexpected focus returns first to the last connected application element and then to `#app` when that element is unavailable. The foreground layers do not prevent using the production Settings popover in its document layer or an application modal dialog in the browser top layer — *(policy-grade test: `packages/web/test/e2e/theme-product-acceptance.test.ts` “executable themes run only in the Chromium application lifecycle and isolate failures”)*. ^theme-ac-sandboxed-frames +- **Acceptance** (outer boundary: the production Settings appearance control in a real Chromium application against a running production server; a real active package enabling both sandbox entries, entry-script startup probes, visible application probes behind transparent regions of both frames, and a temporary home are fixtures; the real stable-front proxy supplies harness transport; no mocks or test hooks): under the [frame lifecycle](../../specs/product/themes-and-appearance.md#^theme-iframe-lifecycle), an effective light-to-dark appearance change removes both old frame documents before installing replacements, leaves no old-appearance frame, and reruns both entries in documents whose root `data-theme` is `dark`. Each replacement host and document root has a used dark scheme, and the application probes remain visibly unchanged through the frames' transparent regions — *(policy-grade test: `packages/web/test/e2e/theme-product-acceptance.test.ts` “executable themes run only in the Chromium application lifecycle and isolate failures”)*. ^theme-ac-frame-appearance +- **Acceptance** (outer boundary: a real Chromium browser loads the built application and a live-canonical HTML artifact against a running built server and its real `/events` transport; a real active package with a byte-stable entry stylesheet, nested imported stylesheet, computed-presentation probes, and a temporary home are fixtures; the real stable-front proxy supplies harness transport; no mocks or test hooks): changing only the nested stylesheet updates the application and artifact presentation; moving the complete active package away produces their foundation and canonical baselines while the exact selected theme ID remains confirmed; restoring it at the same path restores both presentations and keeps that ID, without registry refresh or another selection — *(policy-grade test: `packages/web/test/e2e/theme-product-acceptance.test.ts` “theme settings persist rendered app and artifact outcomes through a built-server restart”)*. ^theme-ac-active-package - **Acceptance** (outer boundary: the real Electron app and another client connected to a running server; a theme enabling main, background, and foreground entries, a second theme, an artifact webview, and an isolated Electron profile are fixtures; the declared desktop shell-version test hook forfeits only installed-version provenance to upgrade-gate coverage): before consent the main script does not execute, while both sandboxed frames are created and execute in the application. Pointer move, down, up, and click inside the ordinary HTML artifact webview reach both theme frames in application-viewport coordinates while the artifact's click remains effective. Granting consent through Settings executes `main.js` in the application renderer but no theme JavaScript in the artifact webview, and the renderer exposes the same Node-facing globals and preload bridge before and after activation. Consent survives application restart; remote main opt-out and remote selection of the second theme each reload the application document, clear its document-lifetime probe, and apply confirmed state. A remote same-theme active-package event replaces both frame documents without itself causing a full application reload — *(covered by `packages/desktop/test/e2e/webview-event-parity.test.ts` “executes an executable theme in Electron and resets on remote consent and selection changes”)*. ^theme-ac-executable-electron - **Acceptance** (outer boundary: the real Electron app connected to a running server; an installed theme and isolated Electron profile are fixtures; the declared desktop shell-version test hook forfeits only installed-version provenance to upgrade-gate coverage): the selected application theme renders again after quitting and relaunching through the saved connection — *(covered by `packages/desktop/test/e2e/webview-event-parity.test.ts` “renders the persisted app theme after quitting and relaunching”)*. ^theme-ac-selection-electron -- **Acceptance** (outer boundary: the built application in a real Chromium browser against a really-running server; one adaptive package and one fixed-dark package, local Television-managed HTML and markdown artifacts, a third-party URL artifact, both sandboxed entries with startup probes, dedicated computed-style probes, controlled system appearance, and a real temporary home are fixtures; no mocks or test hooks): under the adaptive theme, `Adapt to system`, `Light`, and `Dark` produce their expected effective appearance in the application and applicable artifacts. Under the fixed-dark theme, selecting and persisting an opposing Light preference leaves the application root, foundation mode values and native scheme, both artifact classes, and both frame host/document pairs dark; it does not reload app or artifacts, and it keeps the existing frame documents because effective appearance did not change. The same fixed results survive a built-server restart. Returning to the adaptive theme applies the latest Light preference; that effective change replaces and reruns both frames while leaving their transparent composition intact — *(covered by `packages/web/test/e2e/theme-product-acceptance.test.ts` “theme settings persist rendered app and artifact outcomes through a built-server restart” and “executable themes run only in the Chromium application lifecycle and isolate failures”)*. ^theme-ac-appearance +- **Acceptance** (outer boundary: the built application in a real Chromium browser against a really-running server; one adaptive package and one fixed-dark package, local Television-managed HTML and markdown artifacts, a third-party URL artifact, both sandboxed entries with startup probes, dedicated computed-style probes, controlled system appearance, and a real temporary home are fixtures; the real stable-front proxy supplies harness transport; no mocks or test hooks): under the adaptive theme, `Adapt to system`, `Light`, and `Dark` produce their expected effective appearance in the application and applicable artifacts. Under the fixed-dark theme, selecting and persisting an opposing Light preference leaves the application root, foundation mode values and native scheme, both artifact classes, and both frame host/document pairs dark; it does not reload app or artifacts, and it keeps the existing frame documents because effective appearance did not change. The same fixed results survive a built-server restart. Returning to the adaptive theme applies the latest Light preference; that effective change replaces and reruns both frames while leaving their transparent composition intact — *(policy-grade tests: `packages/web/test/e2e/theme-product-acceptance.test.ts` “theme settings persist rendered app and artifact outcomes through a built-server restart” and “executable themes run only in the Chromium application lifecycle and isolate failures”)*. ^theme-ac-appearance - **Acceptance** (outer boundary: the real Electron app connected to a really-running server; one adaptive and one fixed-dark installed package, an existing Television-managed artifact webview, a third-party URL artifact webview, native-appearance observation, computed-style probes, and an isolated profile are fixtures; the declared desktop shell-version test hook forfeits only installed-version provenance to upgrade-gate coverage): the adaptive theme carries `system`, `light`, and `dark` through `nativeTheme` to both retained webviews and their media-query results. Under the fixed-dark theme, an opposing Light preference persists while the application, `nativeTheme`, native-surface appearance source, and both current webviews remain dark and loaded. Returning to the adaptive theme applies that latest Light preference; the third-party URL webview stays loaded, while the Television-managed artifact follows its ordinary theme-selection reload lifecycle — *(covered by `packages/desktop/test/e2e/webview-event-parity.test.ts` “applies manifest-resolved appearance to Electron and retained webviews”)*. This is the same Electron walk as [desktop appearance acceptance](../arch/desktop/appearance.md#^desktop-appearance-t-electron), not a second launch. ^theme-ac-appearance-electron - **Acceptance** (outer boundary: a real browser loads artifacts from a really running production `Server`; the committed old-style v1 document, an authored v2 control document, fresh production canonical tree, temporary storage, and installed loud unscoped theme are fixtures; no mocks or test hooks): the v1 artifact loads its wrapper, base, components, and font but makes no theme request, renders the fixed light canonical probe, and does not render the loud theme value; the v2 control requests the theme and renders its loud value and dark scheme — *(covered by `packages/server/test/e2e/canonical.spec.ts` “frozen v1 stays fixed light while live v2 receives the active theme”)*. ^theme-ac-frozen-v1 - **Acceptance** (outer boundary: a spawned built CLI server against a real fresh temporary home; the packaged bundled-theme tree is the fixture; no mocks or test hooks): first boot installs every bundled theme byte-for-byte, selects Clouds from an unset theme, exposes its exact stylesheet and wallpaper assets, and records the one-time decision — *(covered by `test/node/theme-packaging.test.ts` “Built CLI installs every packaged theme and initially selects Clouds”)*. ^theme-ac-bundled diff --git a/scripts/manage-blaxel-testshards.mjs b/scripts/manage-blaxel-testshards.mjs index 58996712..4c41f705 100644 --- a/scripts/manage-blaxel-testshards.mjs +++ b/scripts/manage-blaxel-testshards.mjs @@ -2,6 +2,9 @@ import { SandboxInstance } from "@blaxel/core"; import { readFileSync } from "node:fs"; import process from "node:process"; +import { randomUUID } from "node:crypto"; +import { workerLeaseCommand } from "./test/blaxel-worker-lease.mjs"; +import { publishWorker, deleteWorker } from "./test/blaxel-pool-lifecycle.mjs"; import { listBlaxelPoolSandboxes } from "./test/blaxel-pool.mjs"; import { DEFAULT_BLAXEL_POOL_SIZE } from "./testshard-constants.mjs"; @@ -33,7 +36,7 @@ if (command === "ensure" && !/^\d+$/.test(nodeSelector)) fail(".nvmrc must conta if (command === "list") { const sandboxes = await listPoolSandboxes(); for (const sandbox of sandboxes) { - console.log(`${sandbox.metadata?.name}\t${sandbox.status}\t${sandbox.spec?.runtime?.memory ?? "?"}MB\t${sandbox.spec?.image ?? "?"}`); + console.log(`${sandbox.metadata?.name}\t${sandbox.status}\t${sandbox.metadata?.labels?.purpose}\t${sandbox.spec?.runtime?.memory ?? "?"}MB\t${sandbox.spec?.image ?? "?"}`); } process.exit(0); } @@ -56,14 +59,12 @@ if (command === "ensure") { region, labels: { project: "television", - purpose: "testshards", + purpose: "testshards-provisioning", pool, arch, }, }); - await sandbox.wait(); - await provisionSandboxRuntime(sandbox); - await verifySandboxArchitecture(sandbox); + await publishWorker(sandbox, { provision: provisionSandboxRuntime, verify: verifySandboxArchitecture, update: SandboxInstance.updateMetadata }); console.log(` ${sandbox.metadata?.name}: ${sandbox.status}`); } process.exit(0); @@ -71,11 +72,20 @@ if (command === "ensure") { if (command === "delete") { const sandboxes = await listPoolSandboxes(); + let refused = false; for (const sandbox of sandboxes) { - console.log(`deleting ${sandbox.metadata?.name}`); - await SandboxInstance.delete(sandbox.metadata.name); + try { + await deleteWorker(sandbox, { reserve: reserveDeletion, update: SandboxInstance.updateMetadata, remove: SandboxInstance.delete }); + console.log(`deleted ${sandbox.metadata.name}`); + } catch (error) { refused = true; console.error(`held ${sandbox.metadata.name}: ${error.message}`); } } - process.exit(0); + process.exit(refused ? 1 : 0); +} + +async function reserveDeletion(sandbox) { + const script = ["set -eo pipefail", 'export NVM_DIR="${NVM_DIR:-$HOME/.nvm}"', '. "$NVM_DIR/nvm.sh"', "nvm use --silent default", "set -u", workerLeaseCommand({ operation: "delete", lockId: `delete-${randomUUID()}` })].join("; "); + const result = await sandbox.process.exec({ name: `tv-delete-reservation-${randomUUID()}`, command: `bash -lc ${shellQuote(script)}`, waitForCompletion: true, timeout: 30 }); + if (result.exitCode !== 0) throw new Error(`ownership check refused deletion: ${result.logs ?? result.exitCode}`); } async function provisionSandboxRuntime(sandbox) { diff --git a/scripts/run-blaxel-testshards.mjs b/scripts/run-blaxel-testshards.mjs index c0d6dfbe..4f1efca2 100644 --- a/scripts/run-blaxel-testshards.mjs +++ b/scripts/run-blaxel-testshards.mjs @@ -10,9 +10,13 @@ import { blaxelCheckoutScript, blaxelDependencyScript, resolveBlaxelRepositoryUr import { transferShardInputs } from "./test/blaxel-shard-dispatch.mjs"; import { buildLinuxStaleOwnerAuditInvocation, parseLinuxStaleOwnerAuditOutput } from "./test/blaxel-stale-owner-audit.mjs"; import { normalizeSetupTimingStatus, normalizeTimingPhase } from "./test/phase-metrics.mjs"; +import { RESOURCE_SIDECAR_BYTES, RESOURCE_ATTEMPT_BYTES, writeResourceDiagnostic } from "./test/resource-diagnostics.mjs"; import { spawnSync } from "node:child_process"; import fs from "node:fs"; import os from "node:os"; +import { randomUUID } from "node:crypto"; +import { renewLease, finishLease, retainAttemptEvidence } from "./test/blaxel-lease-lifecycle.mjs"; +import { workerLeaseCommand, workerEntryCommand, shardScratchPath } from "./test/blaxel-worker-lease.mjs"; import path from "node:path"; import process from "node:process"; @@ -39,6 +43,7 @@ const timeouts = resolveTimeouts(timeoutProfile, options); const playwrightBrowsersPath = options["playwright-browsers-path"] ?? "/home/playwright/.cache/ms-playwright"; const runId = `${new Date().toISOString().replace(/[:.]/g, "-")}-${process.pid}`; const activeLeases = new Map(); +const pendingAcquisitions = new Set(); const attempts = []; let finalReport = null; let activeAttempt = null; @@ -136,6 +141,7 @@ async function runStaleOwnerAudit() { const sandboxes = rotatePool(await listPoolSandboxes(), `${runId}:stale-owner-audit`, 1); if (sandboxes.length === 0) fail(`No deployed x64 Blaxel worker is available in pool ${pool}.`); const lease = await acquireAny(sandboxes, { shardIndex: 1, slot: 0, requestedWorkers: 1, verifyArchitecture: false }); + if (shuttingDown) await shutdownPromise; if (!lease) fail(`No available Blaxel worker in pool ${pool} for the read-only stale-owner audit.`); const { command, args } = buildLinuxStaleOwnerAuditInvocation(); @@ -143,15 +149,17 @@ async function runStaleOwnerAudit() { lease.processName = processName; try { console.log(`acquired stale-owner audit lease: ${lease.sandbox.metadata.name}`); - const setup = await runSystemDependencySetup(lease); - lease.processName = processName; - console.log(`read-only diagnostic command: ${[command, ...args].map(shellQuote).join(" ")}`); - const proc = await lease.sandbox.process.exec({ - name: processName, - command: [command, ...args].map(shellQuote).join(" "), - waitForCompletion: true, - timeout: 30, - }); + const remoteScript = `${lease.input.scratch}/audit.sh`; + lease.pendingWrites = lease.sandbox.fs.write(remoteScript, `#!/usr/bin/env bash\nset -e\n${systemDependenciesCommand()}\n${[command, ...args].map(shellQuote).join(" ")}\n`); + await lease.pendingWrites; + await leaseOperation(lease, "prepare"); + await leaseOperation(lease, "dispatch", { processName }); + lease.executionRequested = true; + const setup = { name: "system-deps" }; + const proc = await (lease.dispatchRequest = lease.sandbox.process.exec({ + name: processName, command: workerEntryCommand({ ...lease.input, processName }, remoteScript), + waitForCompletion: true, timeout: timeouts.workerTimeoutSeconds, + })); const rawLogs = await lease.sandbox.process.logs(processName, "all").catch(() => proc.logs ?? ""); const logs = chunkToText(rawLogs); if (proc.exitCode !== 0) throw new Error(`read-only stale-owner diagnostic exited ${proc.exitCode}: ${logs.trim()}`); @@ -170,6 +178,7 @@ async function runStaleOwnerAudit() { ...audit, }; console.log(`${JSON.stringify(result, null, 2)}`); + lease.retrieved = true; if (!audit.capabilities.procfs.readable || audit.processScan.status !== "ok") return 2; if (!audit.capabilities.ss.available || !audit.capabilities.ss.usable) return 2; if (staleOwnerCount > 0) return 3; @@ -181,24 +190,6 @@ async function runStaleOwnerAudit() { } } -async function runSystemDependencySetup(lease) { - const args = ["-lc", systemDependenciesCommand()]; - const processName = `tv-system-deps-${runId}`.replace(/[^a-zA-Z0-9-]/g, "-"); - lease.processName = processName; - console.log(`system dependency setup: ${["bash", ...args].map(shellQuote).join(" ")}`); - const proc = await lease.sandbox.process.exec({ - name: processName, - command: ["bash", ...args].map(shellQuote).join(" "), - waitForCompletion: true, - timeout: timeouts.workerTimeoutSeconds, - }); - const rawLogs = await lease.sandbox.process.logs(processName, "all").catch(() => proc.logs ?? ""); - const logs = chunkToText(rawLogs); - if (logs.trim()) console.log(logs.trimEnd()); - if (proc.exitCode !== 0) throw new Error(`Blaxel system dependency setup exited ${proc.exitCode}: ${logs.trim()}`); - return { name: "system-deps", command: ["bash", ...args], exitCode: proc.exitCode }; -} - function validateStaleOwnerAuditOptions(parsed) { const allowed = new Set(["audit-stale-owners", "pool", "arch"]); const unexpected = Object.keys(parsed).filter((key) => !allowed.has(key)); @@ -238,7 +229,7 @@ async function runAttempt({ attempt, shardIndices, excludedSandboxes = [], outpu maxConcurrency: MAX_PARALLEL_LEASE_ACQUISITIONS, acquire: async ({ shardIndex, slot }) => { attemptState.shardStartedAt.set(shardIndex, Date.now()); - const lease = await acquireAny(sandboxes, { shardIndex, slot, requestedWorkers: shardIndices.length }); + const lease = await acquireAny(sandboxes, { shardIndex, slot, requestedWorkers: shardIndices.length, attempt: attempt + 1 }); if (lease) attemptState.leases.set(shardIndex, lease); return lease; }, @@ -254,6 +245,7 @@ async function runAttempt({ attempt, shardIndices, excludedSandboxes = [], outpu attemptState.leases.set(outcome.shardIndex, outcome.lease); acquired.push(outcome.lease); } + if (shuttingDown) await shutdownPromise; const acquisitionIncomplete = acquisitionOutcomes.some((outcome) => !outcome.lease); let results; if (acquisitionIncomplete) { @@ -265,11 +257,12 @@ async function runAttempt({ attempt, shardIndices, excludedSandboxes = [], outpu console.log(`acquired: ${formatShardLeases(acquired)}`); console.log(`starting: ${formatShardLeases(acquired)}`); results = await Promise.all(acquired.map(async (lease) => { - const result = await runShard(lease, outputDir, { + const result = await (lease.workPromise = runShard(lease, outputDir, { + attempt, onExecutionRequested: () => attemptState.executionRequested.add(lease.shardIndex), onExecutionStarted: () => attemptState.executionStarted.add(lease.shardIndex), onResultProgress: (progress) => attemptState.partialResults.set(lease.shardIndex, progress), - }); + })); attemptState.partialResults.delete(lease.shardIndex); attemptState.completedResults.set(lease.shardIndex, result); return result; @@ -296,6 +289,7 @@ async function runAttempt({ attempt, shardIndices, excludedSandboxes = [], outpu durationMs: Date.now() - startedAt, shards: results.sort((a, b) => a.shardIndex - b.shardIndex), }; + if (shuttingDown) await shutdownPromise; writeProviderReports(outputDir, report); return report; } finally { @@ -331,33 +325,69 @@ function hashString(value) { return hash >>> 0; } -async function acquireAny(sandboxes, { shardIndex, slot, requestedWorkers, verifyArchitecture = true }) { - const lockId = `${runId}-shard-${shardIndex}`; - const expiresAt = Math.floor(Date.now() / 1000) + timeouts.lockTtlSeconds; +async function leaseOperation(lease, operation, extra = {}) { + if (lease.lost) throw lease.lost; + if (operation === "prepare") lease.preparationRequested = true; + const body = ["set -eo pipefail", 'export NVM_DIR="${NVM_DIR:-$HOME/.nvm}"', '. "$NVM_DIR/nvm.sh"', "nvm use --silent default", "set -u", workerLeaseCommand({ ...lease.input, operation, ...extra })].join("; "); + const result = await lease.sandbox.process.exec({ name: `tv-${operation}-${randomUUID()}`, command: `bash -lc ${shellQuote(body)}`, waitForCompletion: true, timeout: 30 }); + const lines = chunkToText(result.logs ?? "").trim().split("\n"); + let response; + try { response = JSON.parse(lines.at(-1)); } catch { + const details = chunkToText(await lease.sandbox.process.logs(result.name, "all")).slice(-4000); + throw new Error(`worker ${operation} returned no ownership evidence (exit ${result.exitCode}): ${details}`); + } + if (result.exitCode !== 0 || !response.ok) throw new Error(`worker ${operation}: ${response.reason ?? "failed"}`); + return response; +} + +function acquireAny(sandboxes, options) { + const pending = acquireWorker(sandboxes, options); + pendingAcquisitions.add(pending); + return pending.finally(() => pendingAcquisitions.delete(pending)); +} + +async function acquireWorker(sandboxes, { shardIndex, slot, requestedWorkers, verifyArchitecture = true, attempt = 1 }) { + const lockId = `${runId}-s${shardIndex}-a${attempt}-${randomUUID().slice(0, 8)}`; + const input = { lockId, runId, shardIndex, attempt, attemptId: lockId, owner: `${os.hostname()}:${process.pid}`, ttlSeconds: timeouts.lockTtlSeconds, audit: auditStaleOwners, summaryIdentity: shardSummaryIdentity(shardIndex) }; + input.scratch = shardScratchPath(input); for (const sandbox of acquisitionOrder(sandboxes, { slot, requestedWorkers })) { - if (sandbox.__leased) continue; - const command = `bash -lc ${shellQuote(lockCommand({ lockId, shardIndex, expiresAt }))}`; - const proc = await sandbox.process.exec({ name: `tv-lock-${lockId}`, command, waitForCompletion: true, timeout: 30 }); - if (proc.exitCode === 0) { - let retainLock = false; - try { - if (verifyArchitecture && !(await verifySandboxArchitecture(sandbox))) { - sandbox.__leased = true; - continue; - } - sandbox.__leased = true; - const lease = { sandbox, lockId, shardIndex, processName: null }; - activeLeases.set(lockId, lease); - retainLock = true; - return lease; - } finally { - if (!retainLock) await releaseLock(sandbox, lockId); - } + if (sandbox.__leased || shuttingDown) continue; + const lease = { sandbox, lockId, shardIndex, input: { ...input, worker: sandbox.metadata.name }, processName: null }; + try { + await leaseOperation(lease, "acquire"); + } catch (error) { + console.error(`Skipping ${sandbox.metadata.name}: ${error.message}`); + continue; } + sandbox.__leased = true; + activeLeases.set(lockId, lease); + lease.stopRenewal = renewLease({ + intervalMs: Math.max(250, Math.min(30_000, timeouts.lockTtlSeconds * 1000 / 4)), + renew: () => leaseOperation(lease, "renew"), + onLost: (error) => { lease.lost = error; console.error(`Lease renewal lost on ${sandbox.metadata.name}: ${error.message}`); }, + }); + try { + if (!verifyArchitecture || await verifySandboxArchitecture(sandbox)) return lease; + } catch (error) { + console.error(`Architecture check failed on ${sandbox.metadata.name}: ${error.message}`); + } + await stopAndReleaseLease(lease, { killProcess: false }); + activeLeases.delete(lockId); } return null; } +function shardSummaryIdentity(shardIndex) { + return { + shardIndex, shardTotal, + ...(shardPlan ? { + planId: shardPlan.planId, timingProvider: shardPlan.timingProvider, + testedCommit: shardPlan.testedCommit, testedTree: shardPlan.testedTree, + assignedFiles: shardPlan.shards[shardIndex - 1].surfaces.flatMap(surface => surface.files.map(file => file.path)).sort(), + } : {}), + }; +} + async function verifySandboxArchitecture(sandbox) { const check = [ "set -euo pipefail", @@ -393,24 +423,10 @@ function acquisitionOrder(sandboxes, { slot, requestedWorkers }) { return ordered; } -function lockCommand({ lockId, shardIndex, expiresAt }) { - const owner = `${os.hostname()}:${process.pid}`; - const freshJson = JSON.stringify({ lockId, runId, shard: `${shardIndex}/${shardTotal}`, owner, expiresAt }); - const reclaimedJson = JSON.stringify({ lockId, runId, shard: `${shardIndex}/${shardTotal}`, owner, expiresAt, reclaimed: true }); - return [ - "set -euo pipefail", - "lock=/tmp/tv-testshard.lock", - "now=$(date +%s)", - `if mkdir "$lock" 2>/dev/null; then node -e ${shellQuote(`const fs=require("fs"); const data=${JSON.stringify(freshJson)}; const obj=JSON.parse(data); obj.acquiredAt=Number(process.env.now||${Date.now()}); fs.writeFileSync("/tmp/tv-testshard.lock/owner.json", JSON.stringify(obj));`)}; exit 0; fi`, - "expires=0", - "if [ -f \"$lock/owner.json\" ]; then expires=$(node -e 'try{console.log(JSON.parse(require(\"fs\").readFileSync(process.argv[1],\"utf8\")).expiresAt||0)}catch{console.log(0)}' \"$lock/owner.json\"); fi", - `if [ "$expires" -gt 0 ] && [ "$expires" -lt "$now" ]; then rm -rf "$lock"; mkdir "$lock"; node -e ${shellQuote(`const fs=require("fs"); const data=${JSON.stringify(reclaimedJson)}; const obj=JSON.parse(data); obj.acquiredAt=Number(process.env.now||${Date.now()}); fs.writeFileSync("/tmp/tv-testshard.lock/owner.json", JSON.stringify(obj));`)}; exit 0; fi`, - "exit 42", - ].join("; "); -} async function runShard(lease, outputDir, { onExecutionRequested = () => {}, onExecutionStarted = () => {}, onResultProgress = () => {} } = {}) { const { sandbox, shardIndex, lockId } = lease; + const diagnosticAttemptId = lease.input.attemptId; const startedAt = Date.now(); if (lease.restorePhase) { lease.restorePhase.completedAt = new Date(startedAt).toISOString(); @@ -419,50 +435,86 @@ async function runShard(lease, outputDir, { onExecutionRequested = () => {}, onE const overallStarted = lease.restorePhase ? Date.parse(lease.restorePhase.startedAt) : startedAt; const processName = `tv-testshard-${runId}-s${shardIndex}`.replace(/[^a-zA-Z0-9-]/g, "-"); lease.processName = processName; - const remoteScript = `/tmp/${processName}.sh`; + const remoteScript = `${lease.input.scratch}/worker.sh`; + lease.remoteScript = remoteScript; const logPath = path.join(outputDir, `shard-${shardIndex}.log`); lease.logPath = logPath; const summaryPath = path.join(outputDir, `shard-${shardIndex}.json`); - const planRemotePath = shardPlanRaw ? `/tmp/${processName}.plan.json` : null; - const dispatch = await transferShardInputs({ - write: (remotePath, contents) => sandbox.fs.write(remotePath, contents), + const planRemotePath = shardPlanRaw ? `${lease.input.scratch}/plan.json` : null; + const dispatch = await (lease.pendingWrites = transferShardInputs({ + write: (remotePath, contents) => { + if (lease.lost || shuttingDown) throw lease.lost ?? new Error("coordinator interrupted before upload"); + return sandbox.fs.write(remotePath, contents); + }, planRemotePath, planRaw: shardPlanRaw, remoteScript, - remoteScriptContents: shardScript({ shardIndex, lockId, planRemotePath }), - }); + remoteScriptContents: shardScript({ shardIndex, lockId, planRemotePath, diagnosticAttemptId, scratch: lease.input.scratch }), + })); if (!dispatch.ok) { appendLog(logPath, `\n[coordinator] ${dispatch.failureMessage}\n`); return dispatchFailureResult({ lease, outputDir, startedAt, logPath, dispatch }); } - const command = `bash ${remoteScript}`; + const command = workerEntryCommand({ ...lease.input, processName }, remoteScript); let stream; let retainedResultProgress = null; + let resourceDiagnosticsPaths = []; + const copiedArtifacts = new Map(); let failureStep = "worker-execution"; + let exitCode = null; + let executionError = null; try { + await leaseOperation(lease, "prepare"); + await leaseOperation(lease, "dispatch", { processName }); + if (lease.lost || shuttingDown) throw lease.lost ?? new Error("coordinator interrupted before dispatch"); lease.executionRequested = true; onExecutionRequested(); - const proc = await sandbox.process.exec({ - name: processName, - command, - env: { GH_TOKEN: blaxelGithubToken }, - timeout: timeouts.workerTimeoutSeconds, - keepAlive: true, - }); + const proc = await (lease.dispatchRequest = sandbox.process.exec({ name: processName, command, env: { GH_TOKEN: blaxelGithubToken }, timeout: timeouts.workerTimeoutSeconds, keepAlive: true })); onExecutionStarted(); stream = sandbox.process.streamLogs(processName, { - onStdout: (chunk) => appendLog(logPath, chunk), - onStderr: (chunk) => appendLog(logPath, chunk), + onStdout: (chunk) => appendLog(logPath, chunk), onStderr: (chunk) => appendLog(logPath, chunk), onError: (error) => console.error(`Log stream failed for shard ${shardIndex}: ${error.message}`), }); const completed = await sandbox.process.wait(processName, { maxWait: timeouts.workerTimeoutSeconds * 1000, interval: 5000 }); lease.executionCompletionConfirmed = true; - stream?.close?.(); - const logs = await sandbox.process.logs(processName, "all").catch(() => ""); - if (logs && !fs.existsSync(logPath)) fs.writeFileSync(logPath, logs); - failureStep = "report-download"; + exitCode = completed.exitCode ?? proc.exitCode ?? null; + } catch (error) { executionError = error; } + stream?.close?.(); + let result; + try { + await finishLease({ + settleWrites: async () => { await lease.pendingWrites; }, + assertOwned: () => { if (lease.lost) throw lease.lost; }, + teardown: () => confirmTeardown(lease), + download: async () => { + await beforeDownload(lease); + let logError = null; + if (lease.executionRequested) { + try { + const logs = await sandbox.process.logs(processName, "all"); + appendLog(logPath, `\n[coordinator] final worker log:\n${chunkToText(logs)}`); + } catch (error) { logError = error; } + } + failureStep = "report-download"; + result = await downloadResult(); + // Retain available artifacts even when a required log read failed. + if (logError) throw logError; + lease.retrieved = true; + }, + release: async () => { await leaseOperation(lease, "release", { retrieved: true }); lease.released = true; }, + }); + if (executionError) return { ...result, status: "infra-failed", ...classifyBlaxelCoordinatorFailure({ failureStep: "worker-execution", message: executionError.message }), failureMessage: executionError.message }; + return result; + } catch (error) { + appendLog(logPath, `\n[coordinator] ${error.stack || error.message}\n`); + return { ...retainedResultProgress, shardIndex, shardTotal, sandbox: sandbox.metadata.name, outputDir, status: "infra-failed", exitCode, startedAt: new Date(overallStarted).toISOString(), completedAt: new Date().toISOString(), durationMs: Date.now() - overallStarted, logPath, resourceDiagnosticsPaths, tasks: retainedResultProgress?.tasks ?? [], ...classifyBlaxelCoordinatorFailure({ failureStep, message: error.message }), failureMessage: error.message }; + } finally { await lease.stopRenewal?.(); } + + async function downloadResult() { const reportStarted = Date.now(); - const exitCode = completed.exitCode ?? proc.exitCode ?? null; + await leaseOperation(lease, "check"); + resourceDiagnosticsPaths = await copyResourceDiagnostics(sandbox, { outputDir, shardIndex, diagnosticAttemptId, copiedArtifacts }); + await retainAttemptEvidence(lease, outputDir); const plannedShard = shardPlan?.shards?.[shardIndex - 1]; let shardSummary = null; let shardSummaryError = null; @@ -484,6 +536,7 @@ async function runShard(lease, outputDir, { onExecutionRequested = () => {}, onE summaryPath: fs.existsSync(summaryPath) ? summaryPath : null, summaryError: shardSummaryError, nativeResultPath, + resourceDiagnosticsPaths: [...resourceDiagnosticsPaths], logPath: fs.existsSync(logPath) ? logPath : null, setupTimings, planId: shardSummary?.planId ?? shardPlan?.planId, @@ -510,8 +563,7 @@ async function runShard(lease, outputDir, { onExecutionRequested = () => {}, onE appendLog(logPath, `\n[coordinator] ${shardSummaryError}\n`); } } catch (error) { - shardSummaryError = `Failed to read shard summary: ${error.message}`; - appendLog(logPath, `\n[coordinator] ${shardSummaryError}\n`); + throw new Error(`Failed to read shard summary: ${error.message}`); } retainResultProgress(); const remoteShardNativePath = `/workspace/television/.testshards/results/native-shard-${shardIndex}.json`; @@ -525,9 +577,12 @@ async function runShard(lease, outputDir, { onExecutionRequested = () => {}, onE } } catch {} retainResultProgress(); - const copiedArtifacts = new Map(); for (const [taskIndex, task] of (shardSummary?.tasks ?? []).entries()) { const copiedTask = { ...tasks[taskIndex] }; + copiedTask.resourceDiagnosticsPaths = [ + ...(task.resourceDiagnosticsPaths ?? []).map((remotePath) => copiedArtifacts.get(remotePath)).filter(Boolean), + ...resourceDiagnosticsPaths.filter((file) => path.basename(file) === "recovery.json"), + ]; for (const field of ["nativeResultPath", "attemptResultPath", "workspaceNativeResultPath", "lifecycleResultPath", "processLeakResultPath"]) { if (!task[field]) continue; copiedTask[field] = await copyRemoteArtifact(sandbox, task[field], { outputDir, shardIndex, copiedArtifacts }); @@ -549,9 +604,7 @@ async function runShard(lease, outputDir, { onExecutionRequested = () => {}, onE const setupRaw = await sandbox.fs.read(`/workspace/television/.testshards/results/setup-shard-${shardIndex}.json`); fs.writeFileSync(path.join(outputDir, `setup-shard-${shardIndex}.json`), setupRaw.endsWith("\n") ? setupRaw : `${setupRaw}\n`); setupTimings = [lease.restorePhase, ...JSON.parse(setupRaw)].filter(Boolean); - } catch { - setupTimings = [lease.restorePhase].filter(Boolean); - } + } catch (error) { throw new Error(`Failed to retain setup timings: ${error.message}`); } retainResultProgress(); const identityError = shardPlan && shardSummary ? validateDownloadedShardSummary(shardSummary, shardPlan, shardIndex) : null; const resultError = downloadedResultError(shardSummary, exitCode, shardIndex); @@ -559,6 +612,7 @@ async function runShard(lease, outputDir, { onExecutionRequested = () => {}, onE shardSummaryError = identityError ?? resultError; appendLog(logPath, `\n[coordinator] ${shardSummaryError}\n`); } + if (!shardSummary || identityError) throw new Error(shardSummaryError ?? "missing shard summary"); const hasSummaryFile = fs.existsSync(summaryPath); const status = classifyDownloadedShardStatus({ identityError, shardSummary, exitCode, hasSummaryFile, setupTimings }); const completedAt = new Date().toISOString(); @@ -569,37 +623,7 @@ async function runShard(lease, outputDir, { onExecutionRequested = () => {}, onE failureStep: hasSummaryFile ? "summary-parse" : "summary-read", failureMessage: shardSummaryError ?? `Shard ${shardIndex}/${shardTotal} produced no readable summary.`, } : null; - return { shardIndex, shardTotal, shard: `${shardIndex}/${shardTotal}`, sandbox: sandbox.metadata.name, outputDir, status, exitCode, startedAt: new Date(overallStarted).toISOString(), completedAt, durationMs: Date.now() - overallStarted, summaryPath: fs.existsSync(summaryPath) ? summaryPath : null, summaryError: shardSummaryError, nativeResultPath, logPath, setupTimings, planId: shardSummary?.planId ?? shardPlan?.planId, timingProvider: shardSummary?.timingProvider ?? shardPlan?.timingProvider, testedTree: shardSummary?.testedTree ?? shardPlan?.testedTree, testedCommit: shardSummary?.testedCommit ?? shardPlan?.testedCommit, predictedDurationMs: shardSummary?.predictedDurationMs, phases: [...(shardSummary?.phases ?? []), reportPhase], assignedFiles: shardSummary?.assignedFiles ?? [], collectedFiles: shardSummary?.collectedFiles ?? [], failureKind: identityError ? "plan-validation" : resultError ? "result-report" : missingSummaryFailure?.failureKind ?? shardSummary?.failureKind ?? failedInfrastructureTask?.failureKind, failureStep: identityError ? "plan-validation" : resultError ? "summary-validation" : missingSummaryFailure?.failureStep ?? shardSummary?.failureStep ?? failedInfrastructureTask?.failureStep, failureMessage: identityError ?? resultError ?? missingSummaryFailure?.failureMessage ?? shardSummary?.failureMessage ?? failedInfrastructureTask?.failureMessage, processLeaks: shardSummary?.processLeaks ?? [], tasks }; - } catch (error) { - stream?.close?.(); - appendLog(logPath, `\n[coordinator] ${error.stack || error.message}\n`); - const completedAt = new Date().toISOString(); - const plannedShard = shardPlan?.shards?.[shardIndex - 1]; - const failure = classifyBlaxelCoordinatorFailure({ failureStep, message: error.message }); - return { - ...retainedResultProgress, - shardIndex, - shardTotal, - shard: `${shardIndex}/${shardTotal}`, - sandbox: sandbox.metadata.name, - outputDir, - status: "infra-failed", - error: error.message, - ...failure, - failureMessage: error.message, - planId: retainedResultProgress?.planId ?? shardPlan?.planId, - timingProvider: retainedResultProgress?.timingProvider ?? shardPlan?.timingProvider, - testedCommit: retainedResultProgress?.testedCommit ?? shardPlan?.testedCommit, - testedTree: retainedResultProgress?.testedTree ?? shardPlan?.testedTree, - predictedDurationMs: retainedResultProgress?.predictedDurationMs ?? plannedShard?.predictedDurationMs, - startedAt: retainedResultProgress?.startedAt ?? new Date(overallStarted).toISOString(), - completedAt, - durationMs: Date.now() - overallStarted, - logPath, - assignedFiles: retainedResultProgress?.assignedFiles ?? plannedShard?.surfaces?.flatMap((surface) => surface.files.map((file) => file.path)).sort() ?? [], - collectedFiles: retainedResultProgress?.collectedFiles ?? [], - tasks: retainedResultProgress?.tasks ?? [], - }; + return { shardIndex, shardTotal, shard: `${shardIndex}/${shardTotal}`, sandbox: sandbox.metadata.name, outputDir, status, exitCode, startedAt: new Date(overallStarted).toISOString(), completedAt, durationMs: Date.now() - overallStarted, summaryPath: fs.existsSync(summaryPath) ? summaryPath : null, summaryError: shardSummaryError, nativeResultPath, resourceDiagnosticsPaths, remoteScript, scratch: lease.input.scratch, logPath, setupTimings, planId: shardSummary?.planId ?? shardPlan?.planId, timingProvider: shardSummary?.timingProvider ?? shardPlan?.timingProvider, testedTree: shardSummary?.testedTree ?? shardPlan?.testedTree, testedCommit: shardSummary?.testedCommit ?? shardPlan?.testedCommit, predictedDurationMs: shardSummary?.predictedDurationMs, phases: [...(shardSummary?.phases ?? []), reportPhase], assignedFiles: shardSummary?.assignedFiles ?? [], collectedFiles: shardSummary?.collectedFiles ?? [], failureKind: identityError ? "plan-validation" : resultError ? "result-report" : missingSummaryFailure?.failureKind ?? shardSummary?.failureKind ?? failedInfrastructureTask?.failureKind, failureStep: identityError ? "plan-validation" : resultError ? "summary-validation" : missingSummaryFailure?.failureStep ?? shardSummary?.failureStep ?? failedInfrastructureTask?.failureStep, failureMessage: identityError ?? resultError ?? missingSummaryFailure?.failureMessage ?? shardSummary?.failureMessage ?? failedInfrastructureTask?.failureMessage, processLeaks: shardSummary?.processLeaks ?? [], tasks }; } } @@ -647,6 +671,41 @@ function retainTaskWithoutRemoteArtifacts(task) { if (retained[field]) retained[field] = null; } if (Array.isArray(retained.generatedInputPaths)) retained.generatedInputPaths = []; + retained.resourceDiagnosticsPaths = []; + return retained; +} + +async function copyResourceDiagnostics(sandbox, { outputDir, shardIndex, diagnosticAttemptId, copiedArtifacts }) { + const root = "/workspace/television/.testshards/results/resources"; + const retained = []; + let bytes = 0; + try { + const listing = await sandbox.fs.ls(root); + for (const entry of (listing.files ?? []).slice(0, 1024)) { + if (!/^[A-Za-z0-9_.-]+\.json$/.test(entry.name)) continue; + try { + if (entry.size > RESOURCE_SIDECAR_BYTES || bytes + entry.size > RESOURCE_ATTEMPT_BYTES) throw new Error("budget"); + const remotePath = `${root}/${entry.name}`; + const raw = await sandbox.fs.read(remotePath); + const size = Buffer.byteLength(raw); + if (size > RESOURCE_SIDECAR_BYTES || bytes + size > RESOURCE_ATTEMPT_BYTES) throw new Error("budget"); + const record = JSON.parse(raw); + if (record.schemaVersion !== 1 || record.identity?.runId !== runId || record.identity?.shardIndex !== shardIndex || record.identity?.attemptId !== diagnosticAttemptId) throw new Error("identity"); + const localPath = path.join(outputDir, "artifacts", `shard-${shardIndex}`, "resources", entry.name); + fs.mkdirSync(path.dirname(localPath), { recursive: true }); + if (record.kind === "recovery") { + // The bootstrap receipt must exist before checkout can supply this + // shared writer. Retention adopts its whitelist and attempt budget. + if (!writeResourceDiagnostic(path.dirname(path.dirname(localPath)), path.basename(entry.name, ".json"), record)) continue; + } else fs.writeFileSync(localPath, raw); + copiedArtifacts.set(remotePath, localPath); + retained.push(localPath); bytes += size; + } catch { console.error(`[resource diagnostics] shard ${shardIndex}: sidecar unavailable, invalid or over budget`); } + } + if ((listing.files?.length ?? 0) > 1024) console.error(`[resource diagnostics] shard ${shardIndex}: file listing truncated`); + } catch (error) { + if (error?.status !== 404 && error?.response?.status !== 404) console.error(`[resource diagnostics] shard ${shardIndex}: directory unavailable`); + } return retained; } @@ -707,7 +766,7 @@ function targetCommand() { if (target.runner) { const args = ["cd", shellQuote(target.cwd), "&&"]; if (target.runner === "vitest") { - args.push("npx", "vitest", "run", "--config", shellQuote(target.config), "--reporter=json", '--outputFile "$TV_TARGET_NATIVE_RESULT"'); + args.push("npx", "vitest", "run", "--config", '"$TV_TARGET_VITEST_CONFIG"', "--reporter=json", '--outputFile "$TV_TARGET_NATIVE_RESULT"'); if (target.file) args.push(shellQuote(target.file)); if (target.grep) args.push("-t", shellQuote(target.grep)); if (target.retries) args.push(`--retry=${target.retries}`); @@ -766,14 +825,11 @@ function systemDependenciesCommand() { return 'if ! command -v python3 >/dev/null 2>&1 || ! command -v xauth >/dev/null 2>&1 || ! command -v ss >/dev/null 2>&1 || ! ldconfig -p 2>/dev/null | grep -q "libgtk-3.so.0" || ! ldconfig -p 2>/dev/null | grep -q "libXtst.so.6"; then apt-get update && apt-get install -y --no-install-recommends python3 xauth libgtk-3-0 libxtst6 iproute2; fi'; } -function shardScript({ shardIndex, lockId, planRemotePath }) { +function shardScript({ shardIndex, lockId, planRemotePath, diagnosticAttemptId, scratch }) { return `#!/usr/bin/env bash set -euo pipefail -lock=/tmp/tv-testshard.lock shard_plan=${shellQuote(planRemotePath ?? "")} -release_lock() { - if [ -f "$lock/owner.json" ] && grep -q '"lockId":"${lockId}"' "$lock/owner.json"; then rm -rf "$lock"; fi -} +export TMPDIR=${shellQuote(`${scratch}/tmp`)} export CI=1 export SKIP_ACP_TESTS=1 export NVM_DIR="\${NVM_DIR:-$HOME/.nvm}" @@ -783,11 +839,14 @@ export npm_config_fund=false export PLAYWRIGHT_BROWSERS_PATH=${playwrightBrowsersPath} export TEST_RETRIES=${testRetries} export TV_TEST_TIMING_PROVIDER=blaxel-playwright-x64-4vcpu +export TV_TEST_RUN_ID=${shellQuote(runId)} +export TV_TEST_SHARD_INDEX=${shardIndex} +export TV_TEST_SHARD_ATTEMPT=${shellQuote(diagnosticAttemptId)} mkdir -p /workspace /cache/npm if [ -z "\${GH_TOKEN:-}" ]; then echo "GH_TOKEN missing" >&2; exit 2; fi -setup_metrics=/tmp/tv-testshard-setup-${shardIndex}.jsonl -deps_cache_status=/tmp/tv-testshard-deps-cache-${shardIndex}.txt -owner_report=/tmp/tv-stale-owner-${shardIndex}.json +setup_metrics=${shellQuote(`${scratch}/setup.jsonl`)} +deps_cache_status=${shellQuote(`${scratch}/deps.txt`)} +owner_report=${shellQuote(`${scratch}/owners.json`)} : > "$setup_metrics" : > "$deps_cache_status" export TV_TEST_DEPS_CACHE_STATUS_FILE="$deps_cache_status" @@ -825,23 +884,32 @@ runtime_versions() { node --version npm --version } +restore_attempt_evidence() { + mkdir -p /workspace/television/.testshards/results + cp -R ${shellQuote(`${scratch}/evidence/.`)} /workspace/television/.testshards/results/ +} write_setup_metrics() { - mkdir -p /workspace/television/.testshards/results 2>/dev/null || true + restore_attempt_evidence node - "$setup_metrics" "/workspace/television/.testshards/results/setup-shard-${shardIndex}.json" "/workspace/television/.testshards/results/shard-${shardIndex}.json" "$deps_cache_status" <<'METRICS_NODE' const fs = require('fs'); const [input, setupOutput, shardOutput, depsCacheStatusFile] = process.argv.slice(2); const depsCacheStatus = fs.existsSync(depsCacheStatusFile) ? fs.readFileSync(depsCacheStatusFile, 'utf8').trim() : ''; const rows = (fs.existsSync(input) ? fs.readFileSync(input, 'utf8').split(/\\n/).filter(Boolean).map((line) => JSON.parse(line)) : []) .map((row) => row.name === 'deps' && ['hit', 'miss'].includes(depsCacheStatus) ? { ...row, cacheStatus: depsCacheStatus } : row); +const cleanupPath = '/workspace/television/.testshards/results/temporary-cleanup.json'; +if (fs.existsSync(cleanupPath)) rows.unshift({ name: 'temporary-cleanup', status: 0, durationMs: 0, cleanup: JSON.parse(fs.readFileSync(cleanupPath, 'utf8')) }); fs.writeFileSync(setupOutput, JSON.stringify(rows, null, 2) + "\\n"); if (!fs.existsSync(shardOutput)) { const failed = rows.find((row) => row.status && row.status !== 0); if (failed) { const summary = { + ...${JSON.stringify(shardSummaryIdentity(shardIndex))}, status: 'infra-failed', - failureKind: failed.name === 'checkout' ? 'checkout' : 'setup', + collectedFiles: [], + neverProduced: [failed.name === 'test-run' ? 'final shard summary' : 'test execution'], + failureKind: failed.name === 'checkout' ? 'checkout' : failed.name === 'test-run' ? 'worker-execution' : 'setup', failureStep: failed.name, - failureMessage: failed.name === 'checkout' ? 'Repository checkout failed on the Blaxel worker. Check GitHub token validity and repository access from the sandbox.' : 'Setup step ' + failed.name + ' failed before tests ran.', + failureMessage: failed.name === 'checkout' ? 'Repository checkout failed on the Blaxel worker. Check GitHub token validity and repository access from the sandbox.' : failed.name === 'test-run' ? 'Test execution ended without a finalized shard summary.' : 'Setup step ' + failed.name + ' failed before tests ran.', suite: ${JSON.stringify(suite)}, shardIndex: ${shardIndex}, shardTotal: ${shardTotal}, @@ -857,20 +925,20 @@ if (!fs.existsSync(shardOutput)) { } METRICS_NODE } -trap 'write_setup_metrics; rm -f "$owner_report" "$deps_cache_status"; if [ -n "$shard_plan" ]; then rm -f "$shard_plan"; fi; release_lock' EXIT -rm -rf /workspace/television/.testshards 2>/dev/null || true -cat > /tmp/tv-git-askpass.sh <<'ASKPASS' +trap 'write_setup_metrics' EXIT +cat > ${scratch}/git-askpass.sh <<'ASKPASS' #!/usr/bin/env bash case "$1" in *Username*) echo x-access-token ;; *) echo "$GH_TOKEN" ;; esac ASKPASS -chmod 700 /tmp/tv-git-askpass.sh -export GIT_ASKPASS=/tmp/tv-git-askpass.sh +chmod 700 ${scratch}/git-askpass.sh +export GIT_ASKPASS=${scratch}/git-askpass.sh export GIT_TERMINAL_PROMPT=0 run_step checkout bash -lc ${shellQuote(blaxelCheckoutScript({ repoUrl, commit }))} cd /workspace/television +restore_attempt_evidence run_step runtime-activation activate_runtime run_step runtime-versions runtime_versions run_step deps bash -c ${shellQuote(blaxelDependencyScript())} @@ -934,6 +1002,7 @@ elif [ ${shellQuote(suite)} = target ]; then export TV_TARGET_RUN_ID=${shellQuote(`${runId}:${shardIndex}`)} export TV_TARGET_NAME=${shellQuote(`${targetOptions()?.workspace ?? "target"} ${targetOptions()?.file ?? ""} ${targetOptions()?.grep ?? ""}`)} export TV_TARGET_RUNNER=${shellQuote(targetOptions()?.runner ?? "target")} + export TV_TARGET_CONFIG=${shellQuote(targetOptions()?.config ? path.posix.join("/workspace/television", targetOptions().cwd, targetOptions().config) : "")} export TV_TARGET_COMMAND=${shellQuote(targetCommand())} export TV_TARGET_LIFECYCLE_RESULT="/workspace/television/.testshards/results/target-lifecycle-${shardIndex}.json" export TV_TARGET_NATIVE_RESULT="/workspace/television/.testshards/results/native-shard-${shardIndex}.json" @@ -970,6 +1039,8 @@ const task = { durationMs: Date.now() - Date.parse(startedAt), nativeResultPath: fs.existsSync(process.env.TV_TARGET_NATIVE_RESULT) ? process.env.TV_TARGET_NATIVE_RESULT : null, lifecycleResultPath: process.env.TV_TARGET_LIFECYCLE_RESULT, + attemptResultPath: lifecycle.attemptResultPath || null, + resourceDiagnosticsPaths: lifecycle.resourceDiagnosticsPaths || [], processLeaks, failureKind: lifecycle.failureKind || null, failureStep: lifecycle.failureStep || null, @@ -997,63 +1068,47 @@ exit "$test_status" `; } -async function releaseLock(sandbox, lockId) { - const command = `bash -lc ${shellQuote(`lock=/tmp/tv-testshard.lock; if [ -f "$lock/owner.json" ] && grep -q '\"lockId\":\"${lockId}\"' "$lock/owner.json"; then rm -rf "$lock"; fi`)}`; - await sandbox.process.exec({ name: `tv-release-${lockId}`, command, waitForCompletion: true, timeout: 30 }).catch(() => {}); +function confirmTeardown(lease) { + lease.teardownPromise ??= (async () => { + const result = await leaseOperation(lease, "teardown"); + lease.teardownConfirmed = true; + if (lease.logPath) appendLog(lease.logPath, `\n[coordinator] owner teardown: ${JSON.stringify(result.record.teardown)}\n`); + })(); + return lease.teardownPromise; } -async function stopAndReleaseLease(lease, { killProcess }) { - if (killProcess && lease.processName) { - await lease.sandbox.process.kill(lease.processName).catch(() => {}); - try { - await reapUnconfirmedShardOwners(lease); - } catch (error) { - if (lease.logPath) appendLog(lease.logPath, `\n[coordinator] Owner cleanup was not confirmed; retaining lease ${lease.lockId} until TTL: ${error.stack || error.message}\n`); - throw error; - } + +async function beforeDownload(lease) { + if (process.env.TV_TEST_RUNNER_SELFTEST !== "1" || !process.env.TV_BLAXEL_DOWNLOAD_GATE) return; + const gate = process.env.TV_BLAXEL_DOWNLOAD_GATE; + fs.mkdirSync(gate, { recursive: true }); + fs.writeFileSync(path.join(gate, `shard-${lease.shardIndex}.json`), JSON.stringify({ pid: process.pid, lockId: lease.lockId, sandbox: lease.sandbox.metadata.name, scratch: lease.input.scratch })); + while (!shuttingDown && !fs.existsSync(path.join(gate, "continue"))) { + if (lease.lost) throw lease.lost; + await new Promise((resolve) => setTimeout(resolve, 100)); } - await releaseLock(lease.sandbox, lease.lockId); } -async function reapUnconfirmedShardOwners(lease) { - const processName = `tv-owner-cleanup-${lease.lockId}`.replace(/[^a-zA-Z0-9-]/g, "-"); - const reportPath = `/tmp/${processName}.json`; - const script = [ - "set -eo pipefail", - 'export NVM_DIR="${NVM_DIR:-$HOME/.nvm}"', - '. "$NVM_DIR/nvm.sh"', - "nvm use --silent default", - "set -u", - "cd /workspace/television", - "status=0", - `node scripts/test/stale-owner-reaper-cli.mjs --output ${shellQuote(reportPath)} || status=$?`, - `cat ${shellQuote(reportPath)}`, - 'exit "$status"', - ].join("; "); - const completed = await lease.sandbox.process.exec({ - name: processName, - command: `bash -lc ${shellQuote(script)}`, - waitForCompletion: true, - timeout: 30, - }); - const raw = await lease.sandbox.fs.read(reportPath); - let report; - try { - report = JSON.parse(raw); - } catch (error) { - throw new Error(`Unconfirmed shard owner cleanup returned invalid JSON: ${error.message}`); - } - const validExit = completed.exitCode === 0 || completed.exitCode === 1; - const consistentExit = completed.exitCode === (report.staleDetected ? 1 : 0); - const cleanupConfirmed = report.cleanupConfirmed === true && - Array.isArray(report.liveOwners) && report.liveOwners.length === 0 && - Array.isArray(report.remainingStaleOwners) && report.remainingStaleOwners.length === 0 && - Array.isArray(report.processLeaks); - if (!validExit || !consistentExit || !cleanupConfirmed) { - throw new Error(`Unconfirmed shard owner cleanup failed closed (exit=${completed.exitCode}, staleDetected=${report.staleDetected}, cleanupConfirmed=${report.cleanupConfirmed}, liveOwners=${report.liveOwners?.length ?? "invalid"}, remainingStaleOwners=${report.remainingStaleOwners?.length ?? "invalid"})`); - } - if (lease.logPath) appendLog(lease.logPath, `\n[coordinator] unconfirmed execution owner cleanup: ${JSON.stringify(report)}\n`); - return report; +async function stopAndReleaseLease(lease, { killProcess }) { + if (lease.released) return; + if (lease.finishing) return lease.finishing; + lease.finishing = (async () => { + try { + await lease.pendingWrites; + await lease.dispatchRequest?.catch(() => {}); + // Even a successful named process wait is not a descendant audit. + await confirmTeardown(lease); + // Teardown always stops owned processes. On interruption, also wait for + // the in-flight work promise to finish retaining its reports. + if (killProcess && lease.workPromise) await lease.workPromise; + if (lease.released) return; + if (!lease.preparationRequested && !lease.executionRequested) lease.retrieved = true; + if (!lease.retrieved) throw new Error("results not fully retained; keeping lease for recovery"); + await leaseOperation(lease, "release", { retrieved: true }); + lease.released = true; + } finally { await lease.stopRenewal?.(); } + })(); + return lease.finishing; } function buildInterruptedReportSnapshot(signal) { @@ -1114,9 +1169,10 @@ function installSignalHandlers({ killProcesses }) { process.exit(130); } shuttingDown = true; - const leases = [...activeLeases.values()]; - console.error(`Received ${signal}; preserving the interrupted attempt, then ${killProcesses ? "stopping remote processes and " : ""}releasing ${leases.length} Blaxel lease(s)...`); + console.error(`Received ${signal}; waiting for in-flight acquisitions, then retaining results and cleaning up owned workers...`); shutdownPromise = (async () => { + await Promise.allSettled([...pendingAcquisitions]); + const leases = [...activeLeases.values()]; let interruptedSnapshot = null; const { reportError, releaseError, finalizeError } = await finishInterruptedRun({ preserveReport: () => { @@ -1124,10 +1180,12 @@ function installSignalHandlers({ killProcesses }) { writeInterruptedReportSnapshot(interruptedSnapshot); }, releaseLeases: async () => { - await Promise.allSettled(leases.map((lease) => stopAndReleaseLease(lease, { killProcess: killProcesses }))); - for (const lease of leases) activeLeases.delete(lease.lockId); + const results = await Promise.allSettled(leases.map((lease) => stopAndReleaseLease(lease, { killProcess: killProcesses }))); + for (const lease of leases) if (lease.released) activeLeases.delete(lease.lockId); + const failures = results.flatMap((result, index) => result.status === "rejected" ? [`${leases[index].sandbox.metadata.name}: ${result.reason.message}`] : []); + if (failures.length) throw new Error(failures.join("; ")); }, - finalizeReport: () => writeInterruptedReportSnapshot(interruptedSnapshot), + finalizeReport: () => writeInterruptedReportSnapshot(buildInterruptedReportSnapshot(signal) ?? interruptedSnapshot), }); if (reportError) console.error(`Failed to preserve interrupted Blaxel report: ${reportError.stack || reportError.message || reportError}`); if (releaseError) console.error(`Blaxel shard cleanup failed: ${releaseError.stack || releaseError.message || releaseError}`); @@ -1146,13 +1204,14 @@ function listPoolSandboxes() { function resolveTimeouts(profile, overrides) { const profiles = { - normal: { workerTimeoutSeconds: 240, lockTtlSeconds: 900 }, + normal: { workerTimeoutSeconds: 240, lockTtlSeconds: 1200 }, cold: { workerTimeoutSeconds: 480, lockTtlSeconds: 1200 }, repair: { workerTimeoutSeconds: 480, lockTtlSeconds: 1200 }, }; const selected = { ...profiles[profile] }; if (overrides["worker-timeout-seconds"]) selected.workerTimeoutSeconds = Number.parseInt(overrides["worker-timeout-seconds"], 10); if (overrides["lock-ttl-seconds"]) selected.lockTtlSeconds = Number.parseInt(overrides["lock-ttl-seconds"], 10); + if (!Number.isSafeInteger(selected.lockTtlSeconds) || selected.lockTtlSeconds <= 0) fail("--lock-ttl-seconds must be a positive integer"); return selected; } diff --git a/scripts/test/blaxel-classification.mjs b/scripts/test/blaxel-classification.mjs index 87ebbab3..9547c243 100644 --- a/scripts/test/blaxel-classification.mjs +++ b/scripts/test/blaxel-classification.mjs @@ -36,7 +36,12 @@ export function validateDownloadedShardSummary(summary, plan, shardIndex) { const assignedFiles = [...(summary.assignedFiles ?? [])].sort(); const collectedFiles = [...(summary.collectedFiles ?? [])].sort(); if (JSON.stringify(assignedFiles) !== JSON.stringify(expectedFiles)) return `downloaded shard ${shardIndex} assigned files disagree with its plan`; - if (summary.failureKind === "stale-owner" && collectedFiles.length === 0 && (summary.tasks ?? []).length === 0) return null; + if (collectedFiles.length === 0 && (summary.tasks ?? []).length === 0) { + if (summary.failureKind === "stale-owner") return null; + // A setup failure or killed shell can retain its assignment without ever + // producing native results. This remains infrastructure failure, not a pass. + if (summary.status === "infra-failed" && (summary.neverProduced ?? []).some(value => ["test execution", "final shard summary"].includes(value))) return null; + } const taskAssignedFiles = (summary.tasks ?? []).flatMap((task) => task.assignedFiles ?? []).sort(); const taskCollectedFiles = (summary.tasks ?? []).flatMap((task) => task.collectedFiles ?? []).sort(); if (JSON.stringify(taskAssignedFiles) !== JSON.stringify(expectedFiles)) return `downloaded shard ${shardIndex} task assignments disagree with its plan`; diff --git a/scripts/test/blaxel-lease-acquisition.d.mts b/scripts/test/blaxel-lease-acquisition.d.mts index 70b2c46a..103f5b23 100644 --- a/scripts/test/blaxel-lease-acquisition.d.mts +++ b/scripts/test/blaxel-lease-acquisition.d.mts @@ -22,3 +22,4 @@ export function acquireShardLeases(options: { maxConcurrency: number; acquire: (assignment: { shardIndex: number; slot: number }) => Promise; }): Promise>>; +export function leaseRecoveryArgs(options: Record): unknown[]; diff --git a/scripts/test/blaxel-lease-acquisition.mjs b/scripts/test/blaxel-lease-acquisition.mjs index 4f2d767e..6501d361 100644 --- a/scripts/test/blaxel-lease-acquisition.mjs +++ b/scripts/test/blaxel-lease-acquisition.mjs @@ -83,3 +83,7 @@ export async function acquireShardLeases({ shardIndices, maxConcurrency, acquire function errorMessage(error) { return error instanceof Error ? error.message : String(error); } + +export function leaseRecoveryArgs(options) { + return options["lock-ttl-seconds"] === undefined ? [] : ["--lock-ttl-seconds", options["lock-ttl-seconds"]]; +} diff --git a/scripts/test/blaxel-lease-lifecycle.d.mts b/scripts/test/blaxel-lease-lifecycle.d.mts new file mode 100644 index 00000000..e0c288a5 --- /dev/null +++ b/scripts/test/blaxel-lease-lifecycle.d.mts @@ -0,0 +1,3 @@ +export function renewLease(options: {renew: () => Promise; intervalMs: number; onLost: (error: Error) => void}): () => Promise; +export function finishLease(options: {settleWrites: () => Promise; teardown: () => Promise; download: () => Promise; release: () => Promise; assertOwned: () => void; attempts?: number}): Promise; +export function retainAttemptEvidence(lease: {preparationRequested?: boolean; executionRequested?: boolean; retrieved?: boolean; teardownConfirmed?: boolean; shardIndex: number; input: Record; sandbox: {fs: {read(file: string): Promise; ls(directory: string): Promise<{files?: Array<{name: string}>; subdirectories?: Array<{name: string}>}>}}}, directory: string): Promise; diff --git a/scripts/test/blaxel-lease-lifecycle.mjs b/scripts/test/blaxel-lease-lifecycle.mjs new file mode 100644 index 00000000..4c4980de --- /dev/null +++ b/scripts/test/blaxel-lease-lifecycle.mjs @@ -0,0 +1,63 @@ +import fs from "node:fs"; +import path from "node:path"; + +/** Coordinator-local lifecycle; transport operations are supplied at one seam. */ +export function renewLease({ renew, intervalMs, onLost }) { + let stopped = false; + let pending = Promise.resolve(); + const tick = () => { + if (stopped) return; + pending = renew().catch((error) => { stopped = true; clearInterval(timer); onLost(error); }); + pending.finally(() => { if (!stopped) timer = setTimeout(tick, intervalMs); }); + }; + let timer = setTimeout(tick, intervalMs); + return async () => { stopped = true; clearTimeout(timer); await pending; }; +} + +export async function finishLease({ settleWrites, teardown, download, release, assertOwned, attempts = 3 }) { + await settleWrites(); + assertOwned(); + // Failed teardown still allows evidence retrieval, but can never release. + let teardownError = null; + try { await teardown(); } catch (error) { teardownError = error; } + let retrievalError = null; + for (let attempt = 0; attempt < attempts; attempt++) { + assertOwned(); + try { await download(); retrievalError = null; break; } catch (error) { retrievalError = error; } + } + if (teardownError) throw teardownError; + if (retrievalError) throw retrievalError; + assertOwned(); + await release(); +} + +export async function retainAttemptEvidence(lease, directory) { + if (!lease.preparationRequested && !lease.executionRequested) { lease.retrieved = true; return; } + const root = "/workspace/television/.testshards/results"; + const identity = JSON.parse(await lease.sandbox.fs.read(`${root}/attempt.json`)); + for (const key of ["lockId", "runId", "shardIndex", "attemptId"]) if (identity[key] !== lease.input[key]) throw new Error(`attempt evidence identity mismatch: ${key}`); + let evidence; + try { evidence = JSON.parse(await lease.sandbox.fs.read(`${root}/evidence.json`)); } + catch (error) { + if (lease.teardownConfirmed || error?.status !== 404) throw error; + // Teardown uncertainty must not prevent retaining already-produced evidence. + const files = []; + const collect = async (directory) => { + const listing = await lease.sandbox.fs.ls(directory); + for (const file of listing.files ?? []) files.push(path.posix.relative(root, `${directory}/${file.name}`)); + for (const child of listing.subdirectories ?? []) await collect(`${directory}/${child.name}`); + }; + await collect(root); + evidence = { runId: lease.input.runId, attemptId: lease.input.attemptId, files }; + } + if (evidence.runId !== lease.input.runId || evidence.attemptId !== lease.input.attemptId || !Array.isArray(evidence.files)) throw new Error("invalid retained-evidence inventory"); + for (const name of evidence.files) { + // Optional diagnostic reads use their bounded, warning-only transport below. + // They must not make a native result fail or bypass its retention budget. + if (typeof name === "string" && name.startsWith("resources/")) continue; + if (typeof name !== "string" || name.startsWith("/") || name.split("/").includes("..")) throw new Error("invalid evidence path"); + const raw = await lease.sandbox.fs.read(`${root}/${name}`); + const target = path.join(directory, "artifacts", `shard-${lease.shardIndex}`, name); + fs.mkdirSync(path.dirname(target), { recursive: true }); fs.writeFileSync(target, raw); + } +} diff --git a/scripts/test/blaxel-pool-lifecycle.d.mts b/scripts/test/blaxel-pool-lifecycle.d.mts new file mode 100644 index 00000000..7a4390db --- /dev/null +++ b/scripts/test/blaxel-pool-lifecycle.d.mts @@ -0,0 +1,3 @@ +export interface Worker {metadata: {name: string; labels: Record}; wait: () => Promise} +export function publishWorker(worker: Worker, operations: {provision: (worker: Worker) => Promise; verify: (worker: Worker) => Promise; update: (name: string, metadata: {labels: Record}) => Promise}): Promise; +export function deleteWorker(worker: Pick, operations: {reserve: (worker: Pick) => Promise; update: (name: string, metadata: {labels: Record}) => Promise; remove: (name: string) => Promise}): Promise; diff --git a/scripts/test/blaxel-pool-lifecycle.mjs b/scripts/test/blaxel-pool-lifecycle.mjs new file mode 100644 index 00000000..0f742989 --- /dev/null +++ b/scripts/test/blaxel-pool-lifecycle.mjs @@ -0,0 +1,18 @@ +/** The manager's SDK seam; fresh/unavailable states retain pool identity. */ +export async function publishWorker(sandbox, { provision, verify, update }) { + await sandbox.wait(); + await provision(sandbox); + await verify(sandbox); + return update(sandbox.metadata.name, { labels: { ...sandbox.metadata.labels, purpose: "testshards" } }); +} + +export async function deleteWorker(sandbox, { reserve, update, remove }) { + // A fresh sandbox was never eligible, so no coordinator can own its checkout. + if (sandbox.metadata.labels.purpose !== "testshards-provisioning") { + await reserve(sandbox); + await update(sandbox.metadata.name, { labels: { ...sandbox.metadata.labels, purpose: "testshards-unavailable" } }); + } + // Keep the never-published state if deletion fails: provisioning may have + // failed before installing the runtime needed by the ownership check. + await remove(sandbox.metadata.name); +} diff --git a/scripts/test/blaxel-pool.d.mts b/scripts/test/blaxel-pool.d.mts new file mode 100644 index 00000000..275c21ab --- /dev/null +++ b/scripts/test/blaxel-pool.d.mts @@ -0,0 +1,2 @@ +import type { SandboxInstance } from "@blaxel/core"; +export function listBlaxelPoolSandboxes(options: {pool: string; arch: string; deployedOnly?: boolean}): Promise; diff --git a/scripts/test/blaxel-pool.mjs b/scripts/test/blaxel-pool.mjs index ab385c43..cfedce7c 100644 --- a/scripts/test/blaxel-pool.mjs +++ b/scripts/test/blaxel-pool.mjs @@ -5,7 +5,7 @@ export async function listBlaxelPoolSandboxes({ pool, arch, deployedOnly = false return sandboxes.filter((sandbox) => { const labels = sandbox.metadata?.labels ?? {}; return labels.project === "television" - && labels.purpose === "testshards" + && (deployedOnly ? labels.purpose === "testshards" : ["testshards", "testshards-provisioning", "testshards-unavailable"].includes(labels.purpose)) && labels.pool === pool && labels.arch === arch && (!deployedOnly || sandbox.status === "DEPLOYED"); diff --git a/scripts/test/blaxel-worker-lease.d.mts b/scripts/test/blaxel-worker-lease.d.mts new file mode 100644 index 00000000..794478b5 --- /dev/null +++ b/scripts/test/blaxel-worker-lease.d.mts @@ -0,0 +1,9 @@ +import type { Stats } from "node:fs"; +export interface LeaseInput { lockId: string; runId?: string; shardIndex?: number; shardTotal?: number; attempt?: number; attemptId?: string; owner?: string; ttlSeconds?: number; expiresAt?: number; tmpRoot?: string; resultsRoot?: string; scratch?: string; auditPids?: number[]; audit?: boolean; operation?: string; processName?: string; pid?: number; pidFromEnv?: boolean; retrieved?: boolean; worker?: string; summaryIdentity?: Record } +export interface CleanupReport { ok: boolean; before: {availableBytes: number; freeBytes: number}; after: {availableBytes: number; freeBytes: number}; removed: Record; skips: unknown[]; errors: unknown[] } +export interface LeaseRecord { lockId: string; runId: string; attemptId: string; recoverAfter: number; lastRenewal: number; expiresAt: number; recovery: {discardedReports: boolean} | null } +export function shardScratchPath(input: LeaseInput): string; +export function workerLeaseCommand(input: LeaseInput): string; +export function workerEntryCommand(input: LeaseInput, script: string): string; +export function workerLeaseOperation(input: LeaseInput, hooks?: {now?: () => number; scanOwners?: () => Promise>>; stopOwners?: (token: string) => Promise; cleanTemporary?: (input: {tmpRoot: string; current: string | null; onlyCurrent: boolean}) => Promise}): Promise<{ok: boolean; reason?: string; deleting?: boolean; record: LeaseRecord}>; +export function cleanWorkerTemporaryDirectories(input: {tmpRoot: string; current: string | null; onlyCurrent?: boolean}, probes?: {lstat?: (file: string) => Stats; remove?: (file: string) => void}): Promise; diff --git a/scripts/test/blaxel-worker-lease.mjs b/scripts/test/blaxel-worker-lease.mjs new file mode 100644 index 00000000..209ff0d5 --- /dev/null +++ b/scripts/test/blaxel-worker-lease.mjs @@ -0,0 +1,274 @@ +import { createHash } from "node:crypto"; +import { createRequire } from "node:module"; +const require = createRequire(import.meta.url); + +export function shardScratchPath({ tmpRoot = "/tmp", lockId, shardIndex, attempt = 1 }) { + const digest = createHash("sha256").update(lockId).digest("hex").slice(0, 20); + return `${tmpRoot}/tv-testshards/r-${digest}-s${shardIndex}-a${attempt}`; +} + +// This code runs before checkout, including after another revision ran here. +// Serialize the actual implementation; never read a helper from the worker clone. +export function workerLeaseCommand(input) { + const tmpRoot = input.tmpRoot ?? "/tmp"; + const source = `${cleanWorkerTemporaryDirectories.toString()}\n${workerLeaseOperation.toString()}\nworkerLeaseOperation(JSON.parse(Buffer.from(process.argv[1], 'base64url'))).then(result => { console.log(JSON.stringify(result)); if (!result.ok) process.exitCode = 42; }).catch(error => { console.error(error.message); process.exitCode = 2; });`; + return `flock -w 20 -F ${quote(`${tmpRoot}/tv-testshard.guard`)} node -e ${quote(source)} ${quote(Buffer.from(JSON.stringify({ ...input, scratch: input.scratch ?? shardScratchPath(input) })).toString("base64url"))}`; +} + +export function workerEntryCommand(input, script) { + // exec keeps the registered shell PID/start identity. The token exists at + // process creation, so procfs can observe it even before entry registration. + const register = workerLeaseCommand({ ...input, operation: "enter", pidFromEnv: true }); + return `env TV_TEST_SHARD_OWNER=${quote(input.lockId)} bash -c ${quote(`export TV_SHARD_ENTRY_PID=$$; ${register} && exec bash ${quote(script)}`)}`; +} + +function quote(value) { return `'${String(value).replaceAll("'", `'"'"'`)}'`; } + +/** Must run under the short flock guard. Hooks substitute only declared seams. */ +export async function workerLeaseOperation(input, hooks = {}) { + const fs = require("node:fs"); + const path = require("node:path"); + const now = hooks.now ?? Date.now; + const tmpRoot = input.tmpRoot ?? "/tmp"; + const resultsRoot = input.resultsRoot ?? "/workspace/television/.testshards/results"; + const lock = path.join(tmpRoot, "tv-testshard.lock"); + const ownerFile = path.join(lock, "owner.json"); + const operation = input.operation ?? "acquire"; + const ttlMs = (input.ttlSeconds ?? 1200) * 1000; + if (!Number.isSafeInteger(ttlMs) || ttlMs <= 0) throw new Error("invalid recovery interval"); + if (!/^[A-Za-z0-9_.:-]+$/.test(input.lockId ?? "")) throw new Error("invalid lock identity"); + if (fs.lstatSync(tmpRoot).isSymbolicLink()) throw new Error("temporary root is a symlink"); + if (fs.existsSync(lock) && (!fs.lstatSync(lock).isDirectory() || fs.lstatSync(lock).isSymbolicLink())) throw new Error("unsafe lock root"); + let record = null; + const exists = fs.existsSync(lock); + try { record = JSON.parse(fs.readFileSync(ownerFile, "utf8")); } catch (error) { if (!exists && error.code !== "ENOENT") throw error; } + const valid = record && typeof record.lockId === "string" && typeof record.runId === "string" && typeof record.attemptId === "string" && Number.isFinite(record.lastRenewal) && Number.isFinite(record.recoverAfter); + const expired = valid && record.recoverAfter <= now(); + const held = valid ? !expired : Number.isFinite(record?.expiresAt) && record.expiresAt > now() / 1000; + const write = (value) => { + fs.mkdirSync(lock, { recursive: true }); + fs.writeFileSync(`${ownerFile}.next`, JSON.stringify(value)); + fs.renameSync(`${ownerFile}.next`, ownerFile); + }; + const denied = (reason) => ({ ok: false, reason }); + + // Inspect all tokens, including malformed tokens, conservatively. No arguments + // or environments enter the evidence. Kernel threads and zombies own no work. + function readProcess(pid) { + const base = `/proc/${pid}`; + const stat = fs.readFileSync(`${base}/stat`, "utf8"); + const fields = stat.slice(stat.lastIndexOf(")") + 1).trim().split(/\s+/); + if (fields.length < 20) throw new Error("unreadable process identity"); + if (["Z", "X"].includes(fields[0])) return null; + const environment = fs.readFileSync(`${base}/environ`, "utf8").split("\0"); + const value = (key) => environment.find((entry) => entry.startsWith(`${key}=`))?.slice(key.length + 1); + const shardToken = value("TV_TEST_SHARD_OWNER"); + const surfaceToken = value("TV_TEST_SURFACE_OWNER"); + if (!shardToken && !surfaceToken) return null; + let command = "unavailable"; + try { command = path.basename(fs.readlinkSync(`${base}/exe`)); } catch (error) { if (!["ENOENT", "ESRCH"].includes(error.code)) throw error; } + return { pid, startTime: fields[19], shardToken, surfaceToken, command }; + } + async function scan() { + const owners = []; + // Seam fixtures supply their process set before inspection. Production + // enumerates the whole worker and refuses any unreadable ownership. + const pids = input.auditPids ?? fs.readdirSync("/proc").filter((name) => /^\d+$/.test(name)).map(Number); + for (const pid of pids) { + try { const entry = readProcess(pid); if (entry) owners.push(entry); } + catch (error) { + if (["EACCES", "EPERM"].includes(error.code)) throw new Error(`worker owner audit unreadable at PID ${pid} (${error.code}); worker reuse refused`); + if (!["ENOENT", "ESRCH"].includes(error.code)) throw error; + } + } + return owners; + } + const scanOwners = hooks.scanOwners ?? scan; + const pause = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); + async function stop(token) { + const actions = new Map(); + const signal = async (name) => { + for (const candidate of (await scanOwners()).filter((entry) => entry.shardToken === token)) { + let current; + try { current = readProcess(candidate.pid); } catch (error) { if (["ENOENT", "ESRCH"].includes(error.code)) continue; throw error; } + if (!current || current.startTime !== candidate.startTime || current.shardToken !== token) continue; + try { process.kill(current.pid, name); } + catch (error) { if (error.code === "ESRCH") continue; throw error; } + actions.set(`${current.pid}:${current.startTime}`, { pid: current.pid, startTime: current.startTime, command: current.command, outcome: name }); + } + }; + await signal("SIGTERM"); + const deadline = Date.now() + 2000; + while ((await scanOwners()).some((entry) => entry.shardToken === token) && Date.now() < deadline) await pause(50); + await signal("SIGKILL"); + const killDeadline = Date.now() + 2000; + while ((await scanOwners()).some((entry) => entry.shardToken === token) && Date.now() < killDeadline) await pause(50); + if ((await scanOwners()).some((entry) => entry.shardToken === token)) throw new Error("owned processes survived cleanup"); + return [...actions.values()]; + } + const stopOwners = hooks.stopOwners ?? stop; + const cleanTemporary = hooks.cleanTemporary ?? cleanWorkerTemporaryDirectories; + const clean = async (current, onlyCurrent = false) => { + const result = await cleanTemporary({ tmpRoot, current, onlyCurrent }); + if (!result.ok) throw new Error(`temporary cleanup incomplete: ${JSON.stringify(result)}`); + return result; + }; + const auditClean = async () => (await scanOwners()).length === 0; + + if (["acquire", "delete"].includes(operation)) { + if (record?.deleting) return operation === "delete" ? { ok: true, deleting: true } : denied("worker reserved for deletion"); + if (exists && (held || input.audit)) return denied("worker lease held"); + let recovery = null; + let recoveryCleanup = null; + if (operation === "delete" && valid && expired) { + // Deleting the sandbox is itself cleanup; do not require successful signals. + } else if (valid && expired) { + const processes = await stopOwners(record.lockId); + if (!await auditClean()) return denied("other live or stale owners remain"); + recoveryCleanup = await clean(null); + recovery = { kind: "recovery", previous: { runId: record.runId, lockId: record.lockId, worker: input.worker ?? "unknown", lastRenewal: record.lastRenewal, recoverAfter: record.recoverAfter }, cleanup: "confirmed", discardedReports: true, processes: processes.slice(0, 128), omittedProcesses: Math.max(0, processes.length - 128) }; + } else if (!input.audit && !await auditClean()) return denied("live or stale owners remain"); + if (operation === "delete") { + write({ lockId: input.lockId, deleting: true }); + return { ok: true, deleting: true }; + } + if (recovery) fs.rmSync(resultsRoot, { recursive: true, force: true }); + const acquiredAt = now(); + const scratch = input.scratch; + if (!scratch || path.dirname(scratch) !== path.join(tmpRoot, "tv-testshards")) throw new Error("invalid scratch path"); + const parent = path.dirname(scratch); + if (fs.existsSync(parent) && fs.lstatSync(parent).isSymbolicLink()) throw new Error("scratch root is a symlink"); + fs.mkdirSync(path.join(scratch, "tmp"), { recursive: true, mode: 0o700 }); + record = { lockId: input.lockId, runId: input.runId, shardIndex: input.shardIndex, attemptId: input.attemptId, owner: input.owner, acquiredAt, lastRenewal: acquiredAt, recoverAfter: acquiredAt + ttlMs, expiresAt: 0, scratch, audit: Boolean(input.audit), dispatch: null, recovery, recoveryCleanup, summaryIdentity: input.summaryIdentity }; + write(record); + return { ok: true, record }; + } + + if (!valid || record.lockId !== input.lockId || expired || record.deleting) return denied("lease lost or expired"); + if (operation === "check") { + return { ok: true, record }; + } else if (operation === "renew") { + record.lastRenewal = now(); record.recoverAfter = record.lastRenewal + ttlMs; write(record); + } else if (operation === "prepare") { + if (record.dispatch || record.prepared) return denied("attempt already prepared or dispatched"); + if (!record.audit && !await auditClean()) return denied("worker audit not clean"); + const cleanup = record.audit ? null : await clean(record.scratch); + fs.rmSync(resultsRoot, { recursive: true, force: true }); + fs.mkdirSync(resultsRoot, { recursive: true }); + fs.writeFileSync(path.join(resultsRoot, "attempt.json"), JSON.stringify({ runId: record.runId, shardIndex: record.shardIndex, attemptId: record.attemptId, lockId: record.lockId })); + fs.writeFileSync(path.join(resultsRoot, "temporary-cleanup.json"), JSON.stringify(record.recoveryCleanup ? { ...cleanup, recovery: record.recoveryCleanup } : cleanup)); + if (record.recovery) { + // First evidence in a freshly reset tree. The bounded receipt is also + // accounted for by the surface writer's shared attempt budget. + const receipt = { schemaVersion: 1, ...record.recovery, identity: { runId: record.runId, shardIndex: record.shardIndex, attemptId: record.attemptId, surfaceId: "worker-recovery" } }; + const raw = JSON.stringify(receipt); + if (Buffer.byteLength(raw) > 128 * 1024) throw new Error("recovery receipt exceeds sidecar budget"); + fs.mkdirSync(path.join(resultsRoot, "resources")); + fs.writeFileSync(path.join(resultsRoot, "resources/recovery.json"), raw); + } + // Checkout can clone/reset the entire repository. Keep this evidence in + // attempt scratch until the shell or teardown restores it after checkout. + fs.cpSync(resultsRoot, path.join(record.scratch, "evidence"), { recursive: true }); + record.prepared = true; write(record); + } else if (operation === "dispatch") { + if (!record.prepared || record.dispatch) return denied("invalid dispatch state"); + record.dispatch = { processName: input.processName, requestedAt: now() }; write(record); + } else if (operation === "enter") { + if (!record.dispatch || record.dispatch.processName !== input.processName || record.dispatch.pid) return denied("dispatch not current"); + const pid = input.pidFromEnv ? Number(process.env.TV_SHARD_ENTRY_PID) : input.pid; + const entry = readProcess(pid); + if (!entry || entry.shardToken !== record.lockId) return denied("entry attribution missing"); + record.dispatch.pid = entry.pid; record.dispatch.startTime = entry.startTime; write(record); + } else if (operation === "teardown") { + // A request that never registered may still arrive. Keep it excluded until + // recovery, whose entry check will reject that old identity. + if (record.dispatch && !record.dispatch.pid) return denied("dispatch outcome unknown"); + const processes = await stopOwners(record.lockId); + if (!record.audit && !await auditClean()) return denied("worker audit not clean"); + if (record.prepared) { + fs.mkdirSync(resultsRoot, { recursive: true }); + const preserved = path.join(record.scratch, "evidence"); + if (fs.existsSync(preserved)) fs.cpSync(preserved, resultsRoot, { recursive: true }); + const setupFile = path.join(resultsRoot, `setup-shard-${record.shardIndex}.json`); + if (!fs.existsSync(setupFile)) { + const inputFile = path.join(record.scratch, "setup.jsonl"); + const rows = fs.existsSync(inputFile) ? fs.readFileSync(inputFile, "utf8").split("\n").filter(Boolean).map((line) => JSON.parse(line)) : []; + fs.writeFileSync(setupFile, JSON.stringify(rows)); + } + const summaryFile = path.join(resultsRoot, `shard-${record.shardIndex}.json`); + if (!fs.existsSync(summaryFile)) fs.writeFileSync(summaryFile, JSON.stringify({ ...record.summaryIdentity, collectedFiles: [], status: "infra-failed", failureKind: "worker-execution", failureStep: "interrupted-before-summary", failureMessage: "The owned process ended without a finalized shard summary.", shardIndex: record.shardIndex, tasks: [], neverProduced: ["final shard summary"] })); + } + const cleanup = await clean(record.scratch, true); + record.teardown = { confirmed: true, processes, cleanup }; write(record); + if (record.prepared) { + fs.writeFileSync(path.join(resultsRoot, "teardown.json"), JSON.stringify(record.teardown)); + const files = []; + const inventory = (dir) => { for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + const file = path.join(dir, entry.name); + if (entry.isSymbolicLink()) throw new Error("symlink in retained evidence"); + if (entry.isDirectory()) inventory(file); + else if (entry.name !== "evidence.json") files.push(path.relative(resultsRoot, file)); + } }; + inventory(resultsRoot); + fs.writeFileSync(path.join(resultsRoot, "evidence.json"), JSON.stringify({ runId: record.runId, attemptId: record.attemptId, files })); + } + } else if (operation === "release") { + if (!record.teardown?.confirmed || !input.retrieved) return denied("teardown and retained evidence required"); + fs.rmSync(lock, { recursive: true }); + } else throw new Error(`unknown lease operation ${operation}`); + return { ok: true, record }; +} + +/** Direct allowed directories only; lstat traversal never follows symlinks. */ +export async function cleanWorkerTemporaryDirectories({ tmpRoot = "/tmp", current, onlyCurrent = false }, probes = {}) { + const fs = require("node:fs"); + const path = require("node:path"); + const lstat = probes.lstat ?? fs.lstatSync; + const remove = probes.remove ?? ((file) => fs.rmSync(file, { recursive: true, force: false })); + const scratchRoot = path.join(tmpRoot, "tv-testshards"); + const mountPoints = new Set(fs.readFileSync("/proc/self/mountinfo", "utf8").trim().split("\n").map((line) => line.split(" ")[4].replace(/\\([0-7]{3})/g, (_match, code) => String.fromCharCode(parseInt(code, 8))))); + const free = () => { const value = fs.statfsSync(tmpRoot); return { availableBytes: value.bavail * value.bsize, freeBytes: value.bfree * value.bsize }; }; + const report = { ok: true, before: free(), after: null, removed: {}, skips: [], errors: [] }; + for (const root of [tmpRoot, scratchRoot]) { + if (!fs.existsSync(root)) continue; + if (lstat(root).isSymbolicLink() || !lstat(root).isDirectory()) throw new Error(`unsafe cleanup root ${root}`); + if (root !== tmpRoot && (mountPoints.has(root) || lstat(root).dev !== lstat(tmpRoot).dev)) throw new Error("scratch root crosses a mount boundary"); + } + function allocated(file, device) { + const stat = lstat(file); + // Overlay files may expose their backing device while directories expose + // overlay's device. A regular file's st_dev alone is not a mount boundary. + if (mountPoints.has(file) || (stat.isDirectory() && stat.dev !== device)) throw new Error("unexpected mount boundary"); + let bytes = stat.blocks * 512; + if (stat.isDirectory()) for (const name of fs.readdirSync(file)) bytes += allocated(path.join(file, name), device); + return bytes; + } + const candidates = []; + if (onlyCurrent) { + if (!current || path.dirname(current) !== scratchRoot) throw new Error("current scratch escapes allowed root"); + candidates.push([current, "current"]); + } else { + for (const name of fs.readdirSync(tmpRoot)) { + const prefix = ["television-source-cli-e2e-", "television-web-bundle-"].find((value) => name.startsWith(value)); + if (prefix) candidates.push([path.join(tmpRoot, name), prefix]); + } + if (fs.existsSync(scratchRoot)) for (const name of fs.readdirSync(scratchRoot)) { + const file = path.join(scratchRoot, name); if (file !== current) candidates.push([file, "stale-scratch"]); + } + } + for (const [file, prefix] of candidates) { + try { + if (!fs.existsSync(file)) continue; + const stat = lstat(file); + if (!stat.isDirectory() || stat.isSymbolicLink()) { report.skips.push({ prefix, reason: "not a real directory" }); continue; } + const bytes = allocated(file, lstat(tmpRoot).dev); + const currentIdentity = lstat(file); + if (!currentIdentity.isDirectory() || currentIdentity.isSymbolicLink() || currentIdentity.dev !== stat.dev || currentIdentity.ino !== stat.ino) throw new Error("directory identity changed before removal"); + remove(file); + const counts = report.removed[prefix] ??= { directories: 0, allocatedBytes: 0 }; + counts.directories++; counts.allocatedBytes += bytes; + } catch (error) { report.ok = false; report.errors.push({ prefix, reason: error.code ?? error.message }); } + } + report.after = free(); + return report; +} diff --git a/scripts/test/cli.mjs b/scripts/test/cli.mjs index ac1aba81..c7d03d7e 100644 --- a/scripts/test/cli.mjs +++ b/scripts/test/cli.mjs @@ -1,4 +1,5 @@ #!/usr/bin/env node +import { leaseRecoveryArgs } from "./blaxel-lease-acquisition.mjs"; import { spawn, spawnSync } from "node:child_process"; import fs from "node:fs"; import os from "node:os"; @@ -480,6 +481,7 @@ async function runBlaxel(selectionOptions, surfaces, testRetries = 0) { const args = ["scripts/run-blaxel-testshards.mjs", "--pool", selectionOptions.pool ?? "poc", "--commit", selectionOptions.commit ?? "HEAD", "--output-dir", outputDir]; if (selectionOptions["timeout-profile"]) args.push("--timeout-profile", selectionOptions["timeout-profile"]); else args.push("--timeout-profile", "cold"); + args.push(...leaseRecoveryArgs(selectionOptions)); if (selectionOptions.shards) args.push("--shards", selectionOptions.shards); if (selectionOptions["shard-indices"]) args.push("--shard-indices", selectionOptions["shard-indices"]); if (selectionOptions["allow-shard-count-override"]) args.push("--allow-shard-count-override", selectionOptions["allow-shard-count-override"]); @@ -866,6 +868,11 @@ function assertRegistryValid() { } function enforceExecutionPlacement(provider, surfaces) { + if (surfaces.some((surface) => surface.id === "experiment:blaxel-harness")) { + if (provider !== "local" || process.env.TV_TEST_SHARD_OWNER || !process.env.TV_BLAXEL_HARNESS_COORDINATOR?.trim() || !process.env.TV_BLAXEL_HARNESS_POOL?.trim()) { + fail("The Blaxel harness driver requires its dedicated coordinator outside the worker pool. Set TV_BLAXEL_HARNESS_COORDINATOR and TV_BLAXEL_HARNESS_POOL explicitly."); + } + } const isolated = surfaces.filter((surface) => surface.tags.includes("isolated-github-only")); if (isolated.length === 0) return; const ids = isolated.map((surface) => surface.id).join(", "); diff --git a/scripts/test/planned-shard-worker.mjs b/scripts/test/planned-shard-worker.mjs index cf536201..9fba1c64 100644 --- a/scripts/test/planned-shard-worker.mjs +++ b/scripts/test/planned-shard-worker.mjs @@ -14,6 +14,7 @@ import { runnerEnv } from "./runner-env.mjs"; import { createSurfaceSupervisor } from "./surface-supervisor.mjs"; import { startSurfaceServices } from "./surface-services.mjs"; import { reapStaleOwnerProcesses } from "./stale-owner-reaper.mjs"; +import { surfaceResourceDiagnostics } from "./resource-diagnostics.mjs"; export async function runPlannedShard(rawOptions = {}, env = process.env) { const planPath = path.resolve(rawOptions.plan ?? env.TEST_SHARD_PLAN ?? ""); @@ -179,6 +180,7 @@ function relocateTaskArtifacts(summary, resultsDir, providerOutputDir, nativeOut workspaceNativeResultPath: relocateNative(task.workspaceNativeResultPath), lifecycleResultPath: relocateProvider(task.lifecycleResultPath), processLeakResultPath: relocateProvider(task.processLeakResultPath), + resourceDiagnosticsPaths: (task.resourceDiagnosticsPaths ?? []).map(relocateProvider), generatedInputPaths: (task.generatedInputPaths ?? []).map(relocateProvider), })), }; @@ -208,10 +210,12 @@ async function runVitestWorkspace(tasks, { resultsDir, shardIndex, skipPreComman const nativeResultPath = path.join(resultsDir, `native-unit-workspaces-shard-${shardIndex}.json`); const attemptResultPath = path.join(resultsDir, `native-unit-workspaces-shard-${shardIndex}.attempts.ndjson`); const supervisor = createSurfaceSupervisor({ runId: lifecycleRunId, surfaceIds: tasks.map((task) => task.surfaceId) }); + const diagnostics = tasks.map((task) => surfaceResourceDiagnostics({ resultsDir, surfaceId: task.surfaceId, env, runId: lifecycleRunId })); let pre; let exitCode = 0; let processLeaks = []; let commandStarted = null; + let splitStatuses = new Map(); fs.rmSync(attemptResultPath, { force: true }); const commandArgs = vitestProjectsArgs({ configPath: workspacePath, configPaths: tasks.map((task) => path.resolve(task.config)) }); try { @@ -225,10 +229,16 @@ async function runVitestWorkspace(tasks, { resultsDir, shardIndex, skipPreComman exitCode = pre.exitCode; } } finally { - processLeaks = await supervisor.finish(); + let cleanupConfirmed = false; + try { processLeaks = await supervisor.finish(); cleanupConfirmed = true; } + finally { + try { splitStatuses = splitVitestWorkspaceResult({ nativeResultPath, tasks }); } + finally { + for (const [index, task] of tasks.entries()) task.resourceDiagnosticsPaths = diagnostics[index].finish({ runner: "vitest", nativeResultPath: task.nativeResultPath, attemptResultPath, exitCode, incomplete: !cleanupConfirmed || !pre || pre.failed || processLeaks.length > 0 }); + } + } } if (pre.failed) return tasks.map((task) => failedTask(task, pre.exitCode, started, { preCommandsSkipped: skipPreCommands, failureKind: pre.failureKind, failureStep: "pre-command", failureMessage: pre.failureMessage, phases: pre.phases, generatedInputPaths: [workspacePath], processLeaks })); - const splitStatuses = splitVitestWorkspaceResult({ nativeResultPath, tasks }); return tasks.map((task) => { const status = exitCode === 0 ? 0 : (splitStatuses.get(task.surfaceId) ?? exitCode); const collected = collectedVitestFiles(task.nativeResultPath, task); @@ -239,6 +249,7 @@ async function runVitestWorkspace(tasks, { resultsDir, shardIndex, skipPreComman async function runStandalone(task, { resultsDir, shardIndex, skipPreCommands, env, lifecycleRunId }) { const started = Date.now(); const supervisor = createSurfaceSupervisor({ runId: lifecycleRunId, surfaceIds: [task.surfaceId] }); + const diagnostics = surfaceResourceDiagnostics({ resultsDir, surfaceId: task.surfaceId, env, runId: lifecycleRunId }); let pre; let command; let args; @@ -300,7 +311,9 @@ async function runStandalone(task, { resultsDir, shardIndex, skipPreCommands, en pre?.phases.push(phase("surface-services-stop", serviceStopped, 1)); } } - processLeaks = await supervisor.finish(); + let cleanupConfirmed = false; + try { processLeaks = await supervisor.finish(); cleanupConfirmed = true; } + finally { task.resourceDiagnosticsPaths = diagnostics.finish({ runner: task.runner, nativeResultPath: task.nativeResultPath, attemptResultPath, exitCode, incomplete: !cleanupConfirmed || !pre || pre.failed || Boolean(serviceFailure) || processLeaks.length > 0 }); } } if (pre.failed) return failedTask(task, pre.exitCode, started, { preCommandsSkipped: skipPreCommands, failureKind: pre.failureKind, failureStep: "pre-command", failureMessage: pre.failureMessage, phases: pre.phases, processLeaks }); const collected = task.runner === "vitest" ? collectedVitestFiles(task.nativeResultPath, task) : collectedPlaywrightFiles(task.nativeResultPath, task); @@ -366,6 +379,7 @@ function resultForTask(task, exitCode, started, extras) { attemptResultPath: extras.attemptResultPath && fs.existsSync(extras.attemptResultPath) ? extras.attemptResultPath : null, workspaceNativeResultPath: extras.workspaceNativeResultPath ?? null, generatedInputPaths: extras.generatedInputPaths ?? [], + resourceDiagnosticsPaths: task.resourceDiagnosticsPaths ?? [], preCommandsSkipped: extras.preCommandsSkipped, phases: extras.phases ?? [], processLeaks, ...(mismatch diff --git a/scripts/test/provider-normalization.mjs b/scripts/test/provider-normalization.mjs index b2277b56..8b0afe50 100644 --- a/scripts/test/provider-normalization.mjs +++ b/scripts/test/provider-normalization.mjs @@ -1,7 +1,7 @@ import fs from "node:fs"; import path from "node:path"; import { normalizeSetupTimingStatus } from "./phase-metrics.mjs"; -import { normalizeSurfaceResult } from "./reporting.mjs"; +import { normalizeSurfaceResult, retainedResourcePaths } from "./reporting.mjs"; export function normalizeProviderShardSurfaces({ provider, providerSummary, outputDir, surfaces, delegatedCommand, exitCode, coordinatorLogPath }) { const bySurface = new Map(surfaces.map((surface) => [surface.id, surface])); @@ -49,6 +49,7 @@ export function normalizeProviderShardSurfaces({ provider, providerSummary, outp durationMs: shard.durationMs ?? providerSummary.durationMs ?? 0, logPath: path.relative(process.cwd(), shard.logPath ?? coordinatorLogPath), nativeResultPath: path.relative(process.cwd(), shard.nativeResultPath), + resourceDiagnosticsPaths: shard.resourceDiagnosticsPaths, })); } } @@ -201,6 +202,7 @@ export function mergeSurfaceParts(parts) { durationSource: "task-wall", nativeResultPath: group.map((part) => part.nativeResultPath).filter(Boolean)[0] ?? null, nativeResultPaths: group.map((part) => part.nativeResultPath).filter(Boolean), + resourceDiagnosticsPaths: retainedResourcePaths(group.flatMap((part) => part.resourceDiagnosticsPaths ?? [])), failureKind: firstInfrastructureFailure?.failureKind ?? first.failureKind, failureStep: firstInfrastructureFailure?.failureStep ?? first.failureStep, failureMessage: firstInfrastructureFailure?.failureMessage ?? first.failureMessage, @@ -248,6 +250,7 @@ function normalizeTaskPart({ task, surface, shard, outputDir, coordinatorLogPath logPath: path.relative(process.cwd(), shard.logPath ?? coordinatorLogPath), nativeResultPath, attemptResultPath, + resourceDiagnosticsPaths: task.resourceDiagnosticsPaths, processLeaks: task.processLeaks ?? [], }); const infraStatus = task.infraStatus === "incomplete" || normalized.infraStatus === "incomplete" ? "incomplete" : "completed"; @@ -357,5 +360,6 @@ function providerFailureSurfaces({ provider, failures, delegatedCommand, coordin infraFailedShards: failure.status === "infra-failed" && failure.shardIndex ? [failure.shardIndex] : [], logPath: retainedLogPath(failure.logPath, coordinatorLogPath), nativeResultPath: null, + resourceDiagnosticsPaths: retainedResourcePaths(failure.resourceDiagnosticsPaths), })); } diff --git a/scripts/test/provider-outputs.d.mts b/scripts/test/provider-outputs.d.mts index c8e2809a..75625c42 100644 --- a/scripts/test/provider-outputs.d.mts +++ b/scripts/test/provider-outputs.d.mts @@ -10,6 +10,7 @@ export interface ProviderTask { exitCode?: number | null; durationMs?: number; nativeResultPath?: string | null; + resourceDiagnosticsPaths?: string[]; [key: string]: unknown; } @@ -28,6 +29,7 @@ export interface ProviderShard { leaseAcquired?: boolean; executionStarted?: boolean; nativeResultPath?: string | null; + resourceDiagnosticsPaths?: string[]; [key: string]: unknown; } diff --git a/scripts/test/provider-outputs.mjs b/scripts/test/provider-outputs.mjs index 2f8d3ff2..c2e0247b 100644 --- a/scripts/test/provider-outputs.mjs +++ b/scripts/test/provider-outputs.mjs @@ -1,6 +1,7 @@ import fs from "node:fs"; import path from "node:path"; import { normalizeSetupTimingStatus, normalizeTimingPhase } from "./phase-metrics.mjs"; +import { retainedResourcePaths } from "./reporting.mjs"; export function providerCommandExitCode({ delegatedExitCode, reportStatus }) { if (delegatedExitCode !== 0) return delegatedExitCode || 1; @@ -63,6 +64,7 @@ export function compactProviderSummary(report, { outputDir = null } = {}) { logPath: shard.logPath, summaryPath: shard.summaryPath, outputDir: shard.outputDir, + resourceDiagnosticsPaths: retainedResourcePaths(shard.resourceDiagnosticsPaths), })), failedTasks, infraFailedShards, @@ -150,6 +152,7 @@ export function summarizeInfrastructureAttempts(reports) { logPath: shard.logPath, summaryPath: shard.summaryPath, outputDir: shard.outputDir, + resourceDiagnosticsPaths: retainedResourcePaths(shard.resourceDiagnosticsPaths), })), })); } @@ -189,7 +192,13 @@ export function combineAttemptReports(attempts, expectedShardIndices, { shardTot message: `No infrastructure attempt returned a result for shard ${shardIndex}/${shardTotal}.`, })); } - const shards = expectedShardIndices.map((shardIndex) => latestByShard.get(shardIndex)).sort((a, b) => a.shardIndex - b.shardIndex); + const shards = expectedShardIndices.map((shardIndex) => ({ ...latestByShard.get(shardIndex) })).sort((a, b) => a.shardIndex - b.shardIndex); + // Retain only links from superseded attempts, never their test outcomes. + for (const shard of shards) { + const history = attempts.flatMap((attempt) => (attempt.shards ?? []).filter((part) => part.shardIndex === shard.shardIndex)); + shard.resourceDiagnosticsPaths = retainedResourcePaths(history.flatMap((part) => [...(part.resourceDiagnosticsPaths ?? []), ...(part.tasks ?? []).flatMap((task) => task.resourceDiagnosticsPaths ?? [])])); + shard.tasks = (shard.tasks ?? []).map((task) => ({ ...task, resourceDiagnosticsPaths: retainedResourcePaths(history.flatMap((part) => (part.tasks ?? []).filter((candidate) => candidate.surfaceId === task.surfaceId).flatMap((candidate) => candidate.resourceDiagnosticsPaths ?? []))) })); + } const last = attempts.at(-1); const failedTasks = failedTasksFromShards(shards, { shardTotal, outputDir }); const startedAt = attempts[0]?.startedAt ?? attempts[0]?.run?.createdAt ?? new Date().toISOString(); diff --git a/scripts/test/reporting.d.mts b/scripts/test/reporting.d.mts index eef8a7d0..c68c7bb9 100644 --- a/scripts/test/reporting.d.mts +++ b/scripts/test/reporting.d.mts @@ -20,6 +20,7 @@ export interface NormalizedSurfaceResult { logPath: string | null; nativeResultPath?: string | null; nativeResultPaths?: string[]; + resourceDiagnosticsPaths?: string[]; counts: Counts; failedTests: Array>; flakyRecoveredTests: Array>; @@ -62,4 +63,5 @@ export function buildReports(args: { startedAt: string; completedAt: string; }): { summary: ReportSummary; results: ReportResults }; -export function normalizeSurfaceResult(args: { surface: Pick & Partial & { durationSource?: NormalizedSurfaceResult["durationSource"] }; command: string[]; exitCode: number | null; durationMs: number; logPath: string; nativeResultPath: string | null; attemptResultPath?: string | null; processLeaks?: Array> }): NormalizedSurfaceResult; +export function normalizeSurfaceResult(args: { surface: Pick & Partial & { durationSource?: NormalizedSurfaceResult["durationSource"] }; command: string[]; exitCode: number | null; durationMs: number; logPath: string; nativeResultPath: string | null; attemptResultPath?: string | null; resourceDiagnosticsPaths?: string[]; processLeaks?: Array> }): NormalizedSurfaceResult; +export function retainedResourcePaths(paths?: string[]): string[]; diff --git a/scripts/test/reporting.mjs b/scripts/test/reporting.mjs index 809b9689..13242288 100644 --- a/scripts/test/reporting.mjs +++ b/scripts/test/reporting.mjs @@ -79,6 +79,7 @@ export function buildReports({ runId, provider = null, commandProvider = provide logPath: surface.logPath, nativeResultPath: Array.isArray(surface.nativeResultPath) ? (surface.nativeResultPath[0] ?? null) : surface.nativeResultPath, nativeResultPaths: surface.nativeResultPaths ?? (Array.isArray(surface.nativeResultPath) ? surface.nativeResultPath : [surface.nativeResultPath].filter(Boolean)), + resourceDiagnosticsPaths: retainedResourcePaths(surface.resourceDiagnosticsPaths).map((file) => path.relative(runDir, file).replaceAll("\\", "/")), failureKind: surface.failureKind ?? null, failureStep: surface.failureStep ?? null, failureMessage: surface.failureMessage ?? null, @@ -135,7 +136,7 @@ function compactObject(value) { return Object.fromEntries(Object.entries(value).filter(([, item]) => item !== undefined && item !== null)); } -export function normalizeSurfaceResult({ surface, command, exitCode, durationMs, logPath, nativeResultPath, attemptResultPath = null, processLeaks = [] }) { +export function normalizeSurfaceResult({ surface, command, exitCode, durationMs, logPath, nativeResultPath, attemptResultPath = null, resourceDiagnosticsPaths = [], processLeaks = [] }) { const nativeReport = readJsonResult(nativeResultPath); const native = nativeReport.value; const base = { @@ -148,6 +149,7 @@ export function normalizeSurfaceResult({ surface, command, exitCode, durationMs, logPath, nativeResultPath, attemptResultPath, + resourceDiagnosticsPaths: retainedResourcePaths(resourceDiagnosticsPaths), infraStatus: "completed", processLeaks: deduplicateProcessLeaks(processLeaks), }; @@ -170,6 +172,12 @@ export function normalizeSurfaceResult({ surface, command, exitCode, durationMs, return { ...base, status: exitCode === 0 ? "passed" : "failed", counts: emptyCounts(), failedTests: [], flakyRecoveredTests: [], files: [] }; } +// Exact paths only: a basename lookup could attach another infrastructure +// attempt's observation when this attempt's download is absent. +export function retainedResourcePaths(paths = []) { + return [...new Set(paths.filter((file) => typeof file === "string" && fs.existsSync(file)).map((file) => path.resolve(file)))].sort(); +} + function normalizePlaywright(native, exitCode, surface) { const tests = []; collectPlaywrightSpecs(native.suites ?? [], tests, [], surface); diff --git a/scripts/test/resource-diagnostics.d.mts b/scripts/test/resource-diagnostics.d.mts new file mode 100644 index 00000000..b69fb4d1 --- /dev/null +++ b/scripts/test/resource-diagnostics.d.mts @@ -0,0 +1,7 @@ +export function startResourceDiagnostics(options: { root: string; identity: Record; readProbe?: (name: string, interval?: Record) => unknown; now?: () => Date; write?: (file: string, contents: string) => void; warn?: (message: string) => void }): { file: string | null; finish: (outcome: { runner: string; nativeResultPath: string | null; attemptResultPath?: string | null; exitCode: number; incomplete?: boolean }) => string[] }; +export function writeResourceDiagnostic(root: string, name: string, record: Record, options?: { write?: (file: string, contents: string) => void; warn?: (message: string) => void }): string | null; +export const RESOURCE_SIDECAR_BYTES: number; +export const RESOURCE_ATTEMPT_BYTES: number; +export function surfaceResourceDiagnostics(options: { resultsDir: string; surfaceId: string; env?: NodeJS.ProcessEnv; runId: string }): ReturnType; +export function readLinuxResourceProbe(name: string, options?: Record): unknown; +export function kernelLinesInInterval(lines: string[], interval: { since: string; until: string }): { lines: string[]; omittedUnrecognizedLines: number }; diff --git a/scripts/test/resource-diagnostics.mjs b/scripts/test/resource-diagnostics.mjs new file mode 100644 index 00000000..5d1d35c7 --- /dev/null +++ b/scripts/test/resource-diagnostics.mjs @@ -0,0 +1,239 @@ +import { BASELINE_TIMING_PROVIDER } from "./timing-baseline.mjs"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { spawnSync } from "node:child_process"; +import { randomUUID } from "node:crypto"; +import { normalizeSurfaceResult } from "./reporting.mjs"; + +export const RESOURCE_SIDECAR_BYTES = 128 * 1024; +export const RESOURCE_ATTEMPT_BYTES = 8 * 1024 * 1024; +const KERNEL_BYTES = 64 * 1024; +const COMPACT_RESERVE_BYTES = 8 * 1024; +const probes = { + memoryAvailable: { unit: "bytes", source: "/proc/meminfo:MemAvailable" }, + sharedMemoryUsed: { unit: "bytes", source: "/proc/meminfo:Shmem" }, + temporaryFree: { unit: "bytes", source: "statfs" }, + oomKills: { unit: "count", source: "/proc/vmstat:oom_kill" }, +}; +const warnDefault = (message) => console.error(`[resource diagnostics] ${message}`); + +/** Shared by planned tasks and the targeted surface parent, never the native child. */ +export function surfaceResourceDiagnostics({ resultsDir, surfaceId, env = process.env, runId }) { + if (env.TV_TEST_TIMING_PROVIDER !== BASELINE_TIMING_PROVIDER) return { file: null, finish: () => [] }; + return startResourceDiagnostics({ root: resultsDir, identity: { + runId: env.TV_TEST_RUN_ID ?? runId, + shardIndex: Number(env.TV_TEST_SHARD_INDEX ?? "1"), + attemptId: env.TV_TEST_SHARD_ATTEMPT ?? "unavailable", + surfaceId, + } }); +} + +export function startResourceDiagnostics({ root, identity, readProbe = readLinuxResourceProbe, now = () => new Date(), write, warn = warnDefault }) { + const name = `surface-${randomUUID()}`; + const tmpdir = os.tmpdir(); + const start = observe(readProbe, now, tmpdir, warn); + const record = { + kind: "surface", identity: cleanIdentity(identity), start, + end: { status: "missing", reason: "parent-did-not-finalize" }, + nativeEvidence: { status: "pending", failedAttempt: null, recovered: null }, + oomDelta: { value: null, unit: "count", reason: "missing-final-observation" }, + kernel: { status: "not-requested", reason: "parent-did-not-finalize" }, + }; + let file = writeResourceDiagnostic(root, name, record, { write, warn }); + return { + get file() { return file; }, + finish(outcome) { + try { + record.end = observe(readProbe, now, tmpdir, warn); + record.oomDelta = counterDelta(start.oomKills, record.end.oomKills); + record.nativeEvidence = nativeEvidence(outcome, path.join(root, "resources"), warn); + if (outcome.exitCode !== 0 || outcome.incomplete || record.nativeEvidence.status !== "complete" || record.nativeEvidence.failedAttempt) { + const interval = { since: start.observedAt, until: record.end.observedAt }; + try { record.kernel = cleanKernel(readProbe("kernelOOM", interval)); } + catch (error) { record.kernel = { status: "unavailable", reason: errorCode(error) }; warn(`kernel evidence unavailable: ${errorCode(error)}`); } + } else record.kernel = { status: "not-requested", reason: "no-failed-native-attempt" }; + file = writeResourceDiagnostic(root, name, record, { write, warn }) ?? file; + } catch (error) { warn(`final observation unavailable: ${errorCode(error)}`); } + return file && fs.existsSync(file) ? [file] : []; + }, + }; +} + +function observe(readProbe, now, tmpdir, warn) { + const observedAt = now().toISOString(); + const sample = { status: "observed", observedAt }; + for (const [name, metadata] of Object.entries(probes)) { + const value = { ...metadata, observedAt, ...(name === "temporaryFree" ? { path: tmpdir } : {}) }; + try { + const reading = readProbe(name, { tmpdir }); + if (!Number.isSafeInteger(reading) || reading < 0) throw new Error("Invalid numeric probe"); + sample[name] = { ...value, value: reading }; + } catch (error) { + sample[name] = { ...value, value: null, reason: errorCode(error) }; + warn(`${name} unavailable: ${errorCode(error)}`); + } + } + return sample; +} + +export function readLinuxResourceProbe(name, options = {}) { + if (process.platform !== "linux") throw Object.assign(new Error("Linux required"), { code: "ENOTSUP" }); + if (name === "temporaryFree") { + const stats = fs.statfsSync(options.tmpdir ?? os.tmpdir()); + return stats.bavail * stats.bsize; + } + if (name === "kernelOOM") { + // The worker's util-linux 2.38 accepts whole-second calendar bounds. Read + // an enclosing UTC window, then filter its output to our exact interval. + const calendar = (value) => new Date(value).toISOString().slice(0, 19).replace("T", " "); + const result = spawnSync("dmesg", ["--kernel", "--time-format", "iso", "--since", calendar(Date.parse(options.since) - 1), "--until", calendar(Date.parse(options.until) + 1000)], { encoding: "utf8", env: { ...process.env, TZ: "UTC", LC_ALL: "C" }, timeout: 2000, maxBuffer: 128 * 1024 }); + const truncated = result.error?.code === "ENOBUFS"; + if (!truncated && (result.error || result.status !== 0)) { + const permissionCode = /operation not permitted/i.test(result.stderr ?? "") ? "EPERM" : /permission denied/i.test(result.stderr ?? "") ? "EACCES" : "unavailable"; + throw Object.assign(new Error("Kernel log unavailable"), { code: result.error?.code ?? permissionCode }); + } + return { ...kernelLinesInInterval((result.stdout ?? "").split("\n"), options), truncated }; + } + const file = name === "oomKills" ? "/proc/vmstat" : "/proc/meminfo"; + const key = name === "oomKills" ? "oom_kill" : name === "memoryAvailable" ? "MemAvailable:" : "Shmem:"; + const line = fs.readFileSync(file, "utf8").split("\n").find((line) => line.startsWith(`${key} `)); + if (!line) throw Object.assign(new Error("Counter unavailable"), { code: "ENOENT" }); + return Number(line.trim().split(/\s+/)[1]) * (name === "oomKills" ? 1 : 1024); +} + +function counterDelta(start, end) { + if (start.value === null || end.value === null) return { value: null, unit: "count", reason: "unavailable" }; + if (end.value < start.value) return { value: null, unit: "count", reason: "counter-reset" }; + return { value: end.value - start.value, unit: "count" }; +} + +export function kernelLinesInInterval(lines, interval) { + const since = Date.parse(interval.since); + const until = Date.parse(interval.until); + let omittedUnrecognizedLines = 0; + const retained = lines.filter((line) => { + if (!/out of memory|oom[-_ ]kill|killed process/i.test(line)) return false; + const timestamp = line.split(/\s/, 1)[0].replace(",", ".").replace(/\.(\d{3})\d+/, ".$1"); + const time = Date.parse(timestamp); + if (!Number.isFinite(time)) { omittedUnrecognizedLines++; return false; } + return time >= since && time <= until; + }); + return { lines: retained, omittedUnrecognizedLines }; +} + +function nativeEvidence(outcome, diagnosticDir, warn) { + const nativeExists = outcome.nativeResultPath && fs.existsSync(outcome.nativeResultPath); + const attemptExists = outcome.attemptResultPath && fs.existsSync(outcome.attemptResultPath); + const references = { + resultPath: nativeExists ? path.relative(diagnosticDir, outcome.nativeResultPath) : null, + attemptResultPath: attemptExists ? path.relative(diagnosticDir, outcome.attemptResultPath) : null, + }; + if (!nativeExists) return { ...references, status: "missing", failedAttempt: null, recovered: null }; + try { + const native = normalizeSurfaceResult({ surface: { id: "diagnostic", runner: outcome.runner }, command: [], logPath: null, durationMs: 0, ...outcome }); + const attempts = native.files.flatMap((file) => file.attempts ?? []); + const complete = native.infraStatus !== "incomplete" && native.files.every((file) => file.complete); + return { ...references, status: complete ? "complete" : "incomplete", failedAttempt: attempts.some((attempt) => ["failed", "timed-out", "interrupted"].includes(attempt.status)) || native.counts.testsFailed > 0, recovered: native.counts.testsFlakyRecovered > 0 }; + } catch (error) { + warn(`native attempt evidence unavailable: ${errorCode(error)}`); + return { ...references, status: "incomplete", failedAttempt: null, recovered: null }; + } +} + +// Kernel output is not a trusted text channel. Retain only timestamps, executable +// names and numeric OOM fields, never raw messages, paths, arguments or values. +function cleanKernel(input) { + const lines = []; + let bytes = 0; + let omitted = integer(input?.omittedUnrecognizedLines) ?? 0; + let truncated = Boolean(input?.truncated); + for (const raw of input?.lines ?? []) { + const line = String(raw); + if (!/out of memory|oom[-_ ]kill|killed process/i.test(line)) continue; + const killed = /Killed process (\d+) \(([A-Za-z0-9_.+-]{1,80})\)/.exec(line); + if (!killed) { omitted++; continue; } + const timestamp = /^\d{4}-\d\d-\d\dT[\d:.,+-]+/.exec(line)?.[0]; + const fields = [...line.matchAll(/\b(total-vm|anon-rss|file-rss|shmem-rss|pgtables|UID|oom_score_adj):\s*(-?\d+)(kB)?/g)].map((match) => `${match[1]}:${match[2]}${match[3] ?? ""}`); + const retained = [timestamp, `Killed process ${killed[1]} (${killed[2]})`, ...fields].filter(Boolean).join(" "); + const cost = Buffer.byteLength(retained) + (lines.length ? 1 : 0); + if (bytes + cost > KERNEL_BYTES) { truncated = true; break; } + lines.push(retained); bytes += cost; + } + return { status: "available", lines, truncated, omittedUnrecognizedLines: omitted }; +} + +/** One budget for surface records and recovery receipts in the current attempt. */ +export function writeResourceDiagnostic(root, name, input, { write = atomicWrite, warn = warnDefault } = {}) { + try { + if (!/^[A-Za-z0-9_.-]+$/.test(name)) throw new Error("Invalid diagnostic filename"); + const directory = path.resolve(root, "resources"); + fs.mkdirSync(directory, { recursive: true }); + const file = path.join(directory, `${name}.json`); + // Reserve each baseline's final compact observation before optional excerpts + // can consume the remaining budget. Parents write sequentially per attempt. + const otherUsage = fs.readdirSync(directory).filter((entry) => entry.endsWith(".json") && path.join(directory, entry) !== file) + .reduce((sum, entry) => sum + Math.max(COMPACT_RESERVE_BYTES, fs.statSync(path.join(directory, entry)).size), 0); + const budget = Math.min(RESOURCE_SIDECAR_BYTES, RESOURCE_ATTEMPT_BYTES - otherUsage); + if (budget < COMPACT_RESERVE_BYTES) { warn("attempt budget exhausted; diagnostic omitted"); return null; } + const record = cleanRecord(input); + let contents = serialize(record); + if (Buffer.byteLength(contents) > budget) { + record.truncated = true; + const field = record.kind === "recovery" ? "processes" : null; + const values = field ? record[field] : record.kernel?.lines ?? []; + if (!field && record.kernel) record.kernel.truncated = true; + let low = 0; let high = values.length; + while (low < high) { + const count = Math.ceil((low + high) / 2); + if (field) record[field] = values.slice(0, count); else if (record.kernel) record.kernel.lines = values.slice(0, count); + if (Buffer.byteLength(serialize(record)) <= budget) low = count; else high = count - 1; + } + if (field) { record[field] = values.slice(0, low); record.omittedProcesses = (integer(input.omittedProcesses) ?? 0) + values.length - low; } + else if (record.kernel) record.kernel.lines = values.slice(0, low); + contents = serialize(record); + // The omission count itself can add a few bytes at the boundary. + while (Buffer.byteLength(contents) > budget && field && record[field].length) { record[field].pop(); record.omittedProcesses++; contents = serialize(record); } + warn("diagnostic truncated to its byte budget"); + } + if (Buffer.byteLength(contents) > budget) { warn("compact diagnostic exceeds its budget; omitted"); return null; } + write(file, contents); + return fs.existsSync(file) ? file : null; + } catch (error) { warn(`diagnostic write failed: ${errorCode(error)}`); return null; } +} + +function cleanRecord(input) { + const base = { schemaVersion: 1, kind: input.kind === "recovery" ? "recovery" : "surface", identity: cleanIdentity(input.identity ?? {}) }; + if (base.kind === "recovery") return { + ...base, + previous: select(input.previous, ["runId", "lockId", "worker", "lastRenewal", "recoverAfter"]), + cleanup: token(input.cleanup), discardedReports: Boolean(input.discardedReports), + omittedProcesses: integer(input.omittedProcesses) ?? 0, + processes: (input.processes ?? []).map((entry) => ({ pid: integer(entry.pid), startTime: token(entry.startTime), command: executable(entry.command), outcome: token(entry.outcome) })), + }; + return { + ...base, start: cleanObservation(input.start), end: cleanObservation(input.end), + oomDelta: select(input.oomDelta, ["value", "unit", "reason"]), + nativeEvidence: select(input.nativeEvidence, ["status", "failedAttempt", "recovered", "resultPath", "attemptResultPath"]), + kernel: input.kernel?.status === "available" ? cleanKernel(input.kernel) : select(input.kernel, ["status", "reason"]), + }; +} + +function cleanObservation(input) { + const record = select(input, ["status", "reason", "observedAt"]); + for (const key of Object.keys(probes)) if (input?.[key]) record[key] = select(input[key], ["value", "unit", "source", "observedAt", "path", "reason"]); + return record; +} + +function cleanIdentity(input) { return { runId: token(input.runId), shardIndex: integer(input.shardIndex), attemptId: token(input.attemptId), surfaceId: token(input.surfaceId) }; } +function token(value) { return typeof value === "string" ? value.replace(/[^a-zA-Z0-9_.:/@+-]/g, "_").slice(0, 256) : null; } +function executable(value) { return typeof value === "string" ? token(path.basename(value).split(/\s/)[0]) : null; } +function integer(value) { return Number.isSafeInteger(value) && value >= 0 ? value : null; } +function select(input, fields) { return Object.fromEntries(fields.filter((key) => input?.[key] !== undefined).map((key) => [key, input[key]])); } +function serialize(value) { return `${JSON.stringify(value)}\n`; } +function errorCode(error) { return ["EACCES", "EPERM", "ENOENT", "ENOSPC", "EROFS", "EIO", "ENOTSUP", "ETIMEDOUT", "ENOBUFS"].includes(error?.code) ? error.code : "unavailable"; } +function atomicWrite(file, contents) { + const temporary = `${file}.${process.pid}.tmp`; + try { fs.writeFileSync(temporary, contents); fs.renameSync(temporary, file); } + finally { fs.rmSync(temporary, { force: true }); } +} diff --git a/scripts/test/supervised-command.mjs b/scripts/test/supervised-command.mjs index f2b9fa7e..761a25a7 100644 --- a/scripts/test/supervised-command.mjs +++ b/scripts/test/supervised-command.mjs @@ -4,17 +4,29 @@ import path from "node:path"; import process from "node:process"; import { createSurfaceSupervisor } from "./surface-supervisor.mjs"; import { startSurfaceServices } from "./surface-services.mjs"; +import { writeVitestRunnerConfig } from "./execution-groups.mjs"; +import { surfaceResourceDiagnostics } from "./resource-diagnostics.mjs"; const { runId, surfaceIds, resultPath, preCommand, services, command, args } = parseArgs(process.argv.slice(2)); const supervisor = createSurfaceSupervisor({ runId, surfaceIds }); +const diagnostics = surfaceResourceDiagnostics({ resultsDir: path.dirname(resultPath), surfaceId: surfaceIds.join(","), runId }); +let executionEnv = process.env; +let attemptResultPath = null; let commandResult = { exitCode: 127, signal: null, error: null }; let cleanupError = null; let executionFailure = null; let processLeaks = []; let serviceScope = null; try { + if (process.env.TV_TARGET_RUNNER === "vitest" && process.env.TV_TARGET_CONFIG) { + const runnerConfig = path.join(path.dirname(resultPath), "target.runner.config.mjs"); + attemptResultPath = `${process.env.TV_TARGET_NATIVE_RESULT}.attempts.ndjson`; + fs.rmSync(attemptResultPath, { force: true }); + writeVitestRunnerConfig({ configPath: runnerConfig, baseConfigPath: process.env.TV_TARGET_CONFIG, runnerPath: "scripts/test/vitest-attempt-reporter.mjs" }); + executionEnv = { ...process.env, TV_TARGET_VITEST_CONFIG: runnerConfig, TV_VITEST_ATTEMPT_FILE: attemptResultPath }; + } if (preCommand) { - const preResult = await supervisor.run(preCommand[0], preCommand.slice(1), { cwd: process.cwd(), env: process.env, stdio: "inherit" }); + const preResult = await supervisor.run(preCommand[0], preCommand.slice(1), { cwd: process.cwd(), env: executionEnv, stdio: "inherit" }); const preLeaks = await supervisor.checkForLeaks(); if (preResult.exitCode !== 0 || preLeaks.length > 0) { commandResult = preResult; @@ -23,14 +35,14 @@ try { } if (!executionFailure && services.length > 0) { try { - serviceScope = await startSurfaceServices(services, { env: supervisor.childEnv(process.env) }); + serviceScope = await startSurfaceServices(services, { env: supervisor.childEnv(executionEnv) }); } catch (error) { commandResult = { exitCode: 1, signal: null, error }; executionFailure = { kind: "surface-service", message: error.message ?? String(error), step: "surface-service-start" }; } } if (!executionFailure) { - commandResult = await supervisor.run(command, args, { cwd: process.cwd(), env: serviceScope?.env ?? process.env, stdio: "inherit" }); + commandResult = await supervisor.run(command, args, { cwd: process.cwd(), env: serviceScope?.env ?? executionEnv, stdio: "inherit" }); } } finally { if (serviceScope) { @@ -50,6 +62,7 @@ try { } const failedForLifecycle = processLeaks.length > 0 || cleanupError !== null; +const resourceDiagnosticsPaths = diagnostics.finish({ runner: process.env.TV_TARGET_RUNNER, nativeResultPath: process.env.TV_TARGET_NATIVE_RESULT, attemptResultPath, exitCode: commandResult.exitCode, incomplete: failedForLifecycle || Boolean(executionFailure) }); const publishedServices = serviceScope?.services ?? []; const result = { schemaVersion: 1, @@ -62,6 +75,8 @@ const result = { exitCode: commandResult.exitCode, signal: commandResult.signal, processLeaks, + resourceDiagnosticsPaths, + attemptResultPath: attemptResultPath && fs.existsSync(attemptResultPath) ? attemptResultPath : null, failureKind: cleanupError?.name === "UnconfirmedSurfaceCleanupError" ? "unconfirmed-process-cleanup" : executionFailure?.kind ?? (failedForLifecycle ? "process-leak" : null), failureStep: executionFailure?.step ?? null, failureMessage: cleanupError?.message ?? executionFailure?.message ?? (processLeaks.length > 0 ? `${processLeaks.length} owned process leak(s) were reaped` : null), diff --git a/scripts/test/surface-supervisor.mjs b/scripts/test/surface-supervisor.mjs index 508be8aa..368962bd 100644 --- a/scripts/test/surface-supervisor.mjs +++ b/scripts/test/surface-supervisor.mjs @@ -59,7 +59,7 @@ export class SurfaceSupervisor { } childEnv(env = process.env) { - return { ...env, [OWNER_TOKEN_ENV]: this.ownerToken }; + return { ...env, TMPDIR: env.TMPDIR ?? process.env.TMPDIR, ...(process.env.TV_TEST_SHARD_OWNER ? { TV_TEST_SHARD_OWNER: process.env.TV_TEST_SHARD_OWNER } : {}), [OWNER_TOKEN_ENV]: this.ownerToken }; } run(command, args = [], { diff --git a/specs/arch/test-runner/blaxel-testshards.md b/specs/arch/test-runner/blaxel-testshards.md index 94f05a97..02e46df0 100644 --- a/specs/arch/test-runner/blaxel-testshards.md +++ b/specs/arch/test-runner/blaxel-testshards.md @@ -1,10 +1,10 @@ -*The Blaxel shard coordinator: leasing a persistent sandbox pool, selecting the repository workers fetch, sourcing a GitHub token, preparing and locking workers, running shards with dependency-hash caching, and classifying Blaxel shard outcomes.* +*The Blaxel shard coordinator: persistent workers, exclusive leases and abandoned-run recovery, owned temporary files, remote checkout and execution, and retained shard results.* # Blaxel Test Shards Blaxel is the recommended way for agents and humans to run broad validation during development. It cuts the time a full run takes substantially — usually under three minutes, sometimes under two — which is the whole point. It works by keeping a pool of machines on hot standby, preconfigured so a run pays none of the usual setup cost: copying the container image, installing Playwright browsers, and the like. Routing runs to infrastructure like this, for speed and reliability, is also the principal reason Television has a single centralized test runner. -This spec is authoritative for the Blaxel coordinator: how it leases a persistent sandbox pool, selects the repository workers fetch and sources the GitHub token they fetch it with, prepares and locks each worker, runs the shard, caches dependencies by hash, classifies a Blaxel shard's outcome, and its options and output layout. The selected Node/npm values and `.nvmrc` authority are owned by [arch/node-versions.md](../node-versions.md). It extends the shared model in [sharded-execution.md](./sharded-execution.md); the git-safety and auth checks before dispatch are [preflight.md](./preflight.md); the canonical command surface that invokes it is [test-runner.md](./test-runner.md). +This spec is authoritative for the Blaxel coordinator: how it leases a persistent sandbox pool, recovers abandoned runs, owns temporary files, selects the repository workers fetch and sources the GitHub token they fetch it with, prepares and locks each worker, runs the shard, caches dependencies by hash, classifies a Blaxel shard's outcome, and its options and output layout. Resource evidence is owned by [reporting.md](./reporting.md#Resource diagnostics). The selected Node/npm values and `.nvmrc` authority are owned by [arch/node-versions.md](../node-versions.md). It extends the shared model in [sharded-execution.md](./sharded-execution.md); the git-safety and auth checks before dispatch are [preflight.md](./preflight.md); the canonical command surface that invokes it is [test-runner.md](./test-runner.md). Blaxel is the only remote provider. `npm test -- blaxel ` runs the coordinator `scripts/run-blaxel-testshards.mjs`; `npm test -- pool blaxel ` manages the pool through `scripts/manage-blaxel-testshards.mjs`. @@ -30,19 +30,60 @@ The pool manager `ensure` creates sandboxes named `tv-testshard---NN Provisioning requires an absent pool: delete the target pool before `ensure`, then let `ensure` create and provision every worker. Blaxel's `createIfNotExists` applies a creation specification only when a sandbox is absent; it does not reconcile provisioning commands or runtime state inside an existing sandbox. `ensure` therefore refuses a populated target pool, keeping provisioning fresh-only instead of carrying convergence logic for unknown nvm/Node state. +Fresh workers receive the labels that make them discoverable for test dispatch only after provisioning and architecture verification succeed. Pool management still identifies and reports partially provisioned workers; `ensure` must not treat them as absent or silently reuse them. Pool mutations follow [Locking](#Locking). A partly deleted pool is not refilled by `ensure`: replacing missing capacity requires deleting the remaining target pool and provisioning it afresh. + ## Locking -The coordinator leases one worker per shard through an in-sandbox lock directory `/tmp/tv-testshard.lock` with an `owner.json` recording `lockId`, `runId`, shard, owner, `expiresAt`, and `acquiredAt`. Normal runs use a bounded worker pool with at most 32 concurrent lease acquisitions (`MAX_PARALLEL_LEASE_ACQUISITIONS`); the pool starts another acquisition as each active attempt settles. Each attempt retains its unique shard lock ID, validates `uname -m` and `process.arch`, and records a lease as soon as it succeeds. The coordinator waits for every requested lease before starting any shard process. If pool discovery fails, the deployed pool is undersized, or acquisition is incomplete, no shard process starts. The attempt records one infrastructure-failed result per requested shard, including whether a lease was acquired, `executionStarted: false`, the assignment, and the exact discovery, acquisition, or barrier cause. Every successful partial lease is released, and the incomplete shards remain eligible for infrastructure retry. A lock whose `expiresAt` has passed may be reclaimed. +The coordinator leases one worker per shard through `/tmp/tv-testshard.lock`. Its `owner.json` records the unique `lockId`, run/shard/attempt identity, owner and acquisition time, worker-local last renewal and `recoverAfter` deadline, and dispatch ownership. Acquisition, renewal, dispatch registration, recovery, and release serialize their worker-local changes under one short OS-released guard outside the disposable lock directory. Rechecking identity under that guard prevents an earlier renewal or release from overwriting a new owner. The guard is not held during network downloads. ^blaxel-reservation + +Normal runs use at most 32 concurrent lease acquisitions (`MAX_PARALLEL_LEASE_ACQUISITIONS`), starting another as each settles. Each shard attempt retains its unique lock ID, validates `uname -m` and `process.arch`, and records a lease as soon as it succeeds. The coordinator waits for every requested lease before starting any shard process. Discovery failure, an undersized deployed pool, or incomplete acquisition starts no shard. The attempt records one infrastructure-failed result per requested shard, including whether a lease was acquired, `executionStarted: false`, the assignment, and the exact cause. Partial leases follow the release conditions below; incomplete shards remain eligible for infrastructure retry. + +Only the coordinator renews a lease, throughout acquisition barriers, preparation, execution, teardown, and result download. Each successful renewal sets `recoverAfter` to worker-local time plus the recovery interval: 20 minutes by default, overridden by `--lock-ttl-seconds`. This interval is independent of the shard execution timeout. Renewal requires the same owner and an unexpired deadline; it cannot revive an expired lease. A surviving shard cannot renew on its coordinator's behalf. + +The outer shard has a unique shard attempt owner token inherited by setup commands, surface supervisors, and their descendants. It remains available alongside the surface token that nested supervisors replace. Dispatch is recorded before the API request; the remote entry point checks the matching, unexpired lease and registers its process name and PID/start identity under the guard before touching checkout or starting setup. A delayed request cannot launch obsolete work after recovery. Uploads use paths specific to the shard attempt. Unknown dispatch outcomes retain the lease until resolved or recovered. + +Normal release requires confirmed process teardown and completed retrieval of that shard attempt's reports. The shard shell finalizes evidence but never unlocks on exit. Named Blaxel process stop or successful shell completion does not establish that descendants are gone: audit the attributed processes and use token- and PID/start-validated TERM/KILL cleanup for survivors. Preserve the audit in the shard log. Remote cleanup activates nvm before enabling nounset, with strict handling in the cleanup body. An unreadable audit, uncertain dispatch, or surviving owner retains the lease and excludes the worker from immediate test retries. ^blaxel-release + +While still renewing and holding the lock, download summaries, native reports, logs, and setup/cleanup/diagnostic artifacts; check their identities and required evidence. Missing or corrupt evidence remains an infrastructure failure. An early setup failure records which outputs were never produced and retains those that were. Incomplete retrieval keeps the lease while retrieval is retried, without rerunning tests. An unused lease can be released only when execution was never requested, pending writes have settled, and any produced evidence has been retained. The fixed worker results tree must not be reset by a later checkout before this boundary. + +On `SIGINT` or `SIGTERM` while an attempt exists, the attempt becomes interrupted even if its shard promises have all just completed. Completed shard results remain intact. An unfinished shard retains downloaded summaries, native results, task artifacts, parsed task evidence, assignments, collections, and logs, and records `interrupted/coordinator-interruption` plus its known lease and execution state. A pending shard has no synthetic task result. A dispatched process whose start is unconfirmed omits `executionStarted`; absence means unknown, while `false` means execution was not requested. The coordinator makes this interrupted snapshot authoritative before cleanup and retrieval, and cleanup cannot replace it with an ordinary completion report. A report-write failure does not suppress process cleanup. The first signal exits 130 after cleanup; a second exits immediately. Any lease that cannot be released safely stops renewing when the coordinator exits and follows abandoned-run recovery. A signal before the first attempt has no shard result to preserve; the canonical runner treats absent provider evidence as incomplete. Console summaries name only retained log files. + +### Abandoned-run recovery + +After renewal stops for the recovery interval, the next ordinary acquisition stops and audits the abandoned shard attempt's owned processes, clears its leftovers, and takes the lock. The same sequence applies to a quiet worker and to a surviving supervisor or test descendant. Expiry authorizes cleanup, never reuse beneath live processes. This operational definition of abandonment also covers a coordinator suspended or disconnected longer than the interval; it does not claim to prove the coordinator's OS process died. Quiet abandoned workers wait the same interval to keep recovery to one procedure. ^blaxel-recovery + +Under the guard, recheck the old identity and deadline, terminate only its attributed processes with revalidated identity, and confirm the worker has no remaining live or stale test owners. This recovery may stop the abandoned shard attempt's still-live supervisors; ordinary stale-owner cleanup continues to protect other live owners. Only then remove abandoned scratch and the known leftovers below, discard that shard attempt's undownloaded results, and publish the new lease. A bounded [recovery receipt](./reporting.md#Resource diagnostics) identifies the old run, deadline, process cleanup, and discarded reports; it survives the results reset as part of the incoming shard attempt's evidence. This is the explicit exception to normal download-before-release. Lost evidence can never establish a pass for the abandoned run. + +Interrupted recovery leaves the old expired identity until another acquisition completes cleanup. A returning coordinator cannot renew, release, dispatch, or mutate a shard attempt whose ownership it lost; it reports infrastructure failure. Failure to identify or stop owners holds and reports the worker instead of dispatching tests. An unreadable or incomplete lock record may be reclaimed by ordinary acquisition or pool deletion only after a clean worker audit; otherwise hold and report it. Execution cannot precede complete registration, and missing ownership evidence cannot authorize signalling by name. Ordinary attributable coordinator crashes recover automatically; persistent cleanup/provider failures require operator action. -Ordinary attempt cleanup stops a dispatched remote process when completion was not confirmed, then runs the existing owner-lifecycle reaper while retaining the lease. Its remote shell activates nvm before enabling nounset because nvm activation is not nounset-safe; strict error, pipe, and unset-variable handling remains active for the reaper body. Named Blaxel process stop is not a process-tree guarantee: the reaper uses owner tokens and revalidated process identity/group signalling to remove detached descendants. Cleanup releases the lease only when the reaper confirms no live or stale owner-token processes remain; an unreadable, inconsistent, or contaminated report fails closed and leaves the exact lock to its TTL rather than exposing a dirty worker. The reaper report is appended to the shard log as lifecycle evidence. A process whose wait completed successfully is not stopped or reaped again, preserving the normal completed-process path; a lease whose execution was never requested is only released. This ordering prevents a timed-out or transport-failed process or descendant from continuing in an unlocked persistent sandbox while keeping completed and pre-dispatch cleanup unchanged. +### Older runners and pool operations -On `SIGINT` or `SIGTERM` while an attempt exists, the attempt becomes interrupted even if its shard promises have all just completed. Completed shard results remain intact. An unfinished shard retains downloaded summaries, native results, task artifacts, parsed task evidence, assignments, collections, and logs, and records `interrupted/coordinator-interruption` plus its known lease and execution state. A pending shard has no synthetic task result. A dispatched process whose start is unconfirmed omits `executionStarted`; absence means unknown, while `false` means execution was not requested. The coordinator makes this interrupted snapshot authoritative before cleanup, stops running shard processes, applies the same owner-confirmation boundary before releasing each known lease, and ensures cleanup cannot replace it with an ordinary completion report. A report-write failure does not suppress lease and process cleanup. A cleanup that cannot confirm zero owners retains its lock until TTL. The first signal exits 130 after cleanup; a second signal exits immediately, with lock expiry as the final bound. A signal handled before the first attempt has no shard result to preserve; the canonical runner treats the absent provider evidence as incomplete when it receives the coordinator exit. Console summaries name only retained log files. +**Transitional, while branches still run the timestamp-only lock protocol:** write `expiresAt: 0` in new records: older runners reclaim only positive past timestamps, so they leave these leases alone. For a legacy record with no renewable deadline, respect its unexpired TTL; after expiry, require a clean worker audit before acquisition. Skip and report live or ambiguous legacy work. Its normal exit trap removes only its matching `lockId`. Updated runners cannot protect an older coordinator's downloads after that coordinator releases early; branches acquire the corrected behavior when they integrate it. + +Pool deletion checks the same identity and deadline under the guard and refuses unexpired leases. An unlocked worker may be reserved for deletion after a clean owner audit. Explicit deletion may also reserve and destroy a worker with an expired lease even when process cleanup failed, because sandbox deletion is the cleanup. Before issuing deletion, remove the worker from test dispatch eligibility; failed or delayed deletion must not admit a new test run. The manager continues to identify and report it as unavailable for operator repair. Unreadable records follow the recovery rule above. A newly created worker that has never been published for test dispatch may be deleted without a lease. Any other worker that cannot answer the ownership check is held for operator repair; discovery ineligibility alone does not establish that an earlier run has stopped. Fresh provisioning follows [Pool](#Pool). + +`--audit-stale-owners` reserves an available worker without recovering a held lease or deleting residue before observing it. It holds its lease through its prerequisite setup and report collection, then releases normally. Held workers are reported/skipped. This preserves the diagnostic's read-only view of stale owners. + +## Temporary files + +Every leased shard attempt has a fresh private directory under `/tmp/tv-testshards/`, identified with its run, shard, provider attempt, and lock. An absolute `TMPDIR` points to its `tmp/` child before preparation or tests begin and is preserved by child environments. Uploaded scripts, plans, askpass helpers, and setup sidecars are written in the shard attempt directory. Setup sidecars are finalized into the results tree before that directory is removed. Paths stay short enough for browser Unix sockets. Intentional npm, Electron, browser-download, and dependency-hash caches retain their existing persistent locations. ^blaxel-scratch + +After taking the lease and confirming no prior execution remains, each run removes only: + +- Real directories directly under `/tmp` matching `television-source-cli-e2e-*` or `television-web-bundle-*`. +- Every non-current shard attempt directory directly under `/tmp/tv-testshards/`, including ones with missing or incomplete metadata. + +The lock and clean worker audit establish disuse; metadata or age alone does not. Prior result retrieval is protected by the lease, with only the abandoned-run exception above. Reject symlink roots, unexpected mount boundaries, and paths escaping the allowed parent; recursive removal must not follow links into another tree. Preserve the current shard attempt, lock and guard, provider uploads, X11 state, unrelated temporary files, checkouts, and persistent caches. Report counts and allocated bytes removed per prefix, filesystem free space before/after, and skips/errors. ^blaxel-known-leftovers + +After success, failure, timeout, or interruption, stop attributed processes, finalize evidence outside scratch, and remove the current shard attempt directory while retaining the lock through download. An uncertain teardown preserves scratch. Failed cleanup retains its cause and native outcomes, but prevents new test dispatch until cleanup succeeds; an inconclusive worker audit likewise skips deletion and dispatch. Download available evidence even when cleanup fails. Directory deletion cannot release Chromium's open, unlinked shared-memory files: their owning processes must stop first. + +Retain at most the current shard attempt's bounded resource evidence on the worker under the existing `.testshards/results` tree. The next reset follows protected download, or the explicitly recorded abandonment decision above. There is no accumulating cross-run diagnostic archive. Coordinator-side run retention is unchanged. Evidence contents and byte limits belong to [reporting.md](./reporting.md#Resource diagnostics). ## Shard lifecycle After acquiring the complete batch, the coordinator uploads each shard's plan and worker script before starting its remote process. A rejected upload returns an infrastructure-failed shard result with the sandbox, assignment, and `transport/worker-dispatch` cause; it cannot reject the whole attempt without a report. -Each shard runs a script on its worker whose steps are recorded as setup timings: `checkout` (clone if the worker has no checkout, point `origin` at the run's repository, then fetch and hard-reset to the commit), `runtime-activation`, `runtime-versions`, `deps`, `system-deps` (`xauth`, `libgtk-3-0`, and `iproute2` for `ss`), `owner-lifecycle`, `playwright-cache`, `electron-repair`, then `test-run`. The `electron-repair` phase invokes the installed Electron package's own `install.js` with `/cache/electron` as its archive cache. Electron's installer reuses a complete runtime or prepares an absent one; the phase carries no separate download/extraction implementation and does not delete runtime files. ^blaxel-electron-repair After checkout, `runtime-activation` loads the provisioned nvm and runs `nvm use` from the checked-out `.nvmrc`; it does not run a second version checker or install/repair a runtime. All dependency, repair, and test commands inherit the activated Node/npm path. The `system-deps` step asserts these tools on every lease and installs the package set when any prerequisite is absent. `owner-lifecycle` leaves live-supervisor tokens untouched, reaps stale token-bearing groups with revalidated TERM/KILL, and fails before `test-run` whenever it found contamination. That sandbox is excluded from every later attempt by the coordinator. A configured infrastructure retry may run the shard on a different sandbox, whose own lease and owner audit must pass before `test-run`; without an exact contaminated-sandbox identity or enough distinct clean capacity, the shard remains infrastructure-failed. The test run is `scripts/run-test-shard.mjs` ([sharded-execution.md](./sharded-execution.md)) for suite and selection runs, or a single targeted command for a `target` run. A target executes its registry-declared `preCommand` and then its registry `command` when one is declared, otherwise its native runner, inside one surface process scope carrying the selected surface ID and owner token. Selectors, retry arguments, and the native JSON reporter are forwarded through a declared command, so wrappers such as the headless-Linux Electron launcher remain active remotely. It always writes a target shard summary after `test-run`, including native results and any normalized process leaks, so a native test failure is classified as `failed` rather than a setup failure. The worker exports `CI=1` and `TEST_RETRIES`, so child runners see the `FLAKY_TEST_RETRIES` default ([flaky-tests.md](./flaky-tests.md)). +Each shard runs a script on its worker whose steps are recorded as setup timings: `temporary-cleanup` (the pre-run cleanup above), `checkout` (clone if the worker has no checkout, point `origin` at the run's repository, then fetch and hard-reset to the commit), `runtime-activation`, `runtime-versions`, `deps`, `system-deps` (`xauth`, `libgtk-3-0`, and `iproute2` for `ss`), `owner-lifecycle`, `playwright-cache`, `electron-repair`, then `test-run`. The `electron-repair` phase invokes the installed Electron package's own `install.js` with `/cache/electron` as its archive cache. Electron's installer reuses a complete runtime or prepares an absent one; the phase carries no separate download/extraction implementation and does not delete runtime files. ^blaxel-electron-repair After checkout, `runtime-activation` loads the provisioned nvm and runs `nvm use` from the checked-out `.nvmrc`; it does not run a second version checker or install/repair a runtime. All dependency, repair, and test commands inherit the activated Node/npm path. The `system-deps` step asserts these tools on every lease and installs the package set when any prerequisite is absent. `owner-lifecycle` leaves live-supervisor tokens untouched, reaps stale token-bearing groups with revalidated TERM/KILL, and fails before `test-run` whenever it found contamination. That sandbox is excluded from every later attempt by the coordinator. A configured infrastructure retry may run the shard on a different sandbox, whose own lease and owner audit must pass before `test-run`; without an exact contaminated-sandbox identity or enough distinct clean capacity, the shard remains infrastructure-failed. The test run is `scripts/run-test-shard.mjs` ([sharded-execution.md](./sharded-execution.md)) for suite and selection runs, or a single targeted command for a `target` run. A target executes its registry-declared `preCommand` and then its registry `command` when one is declared, otherwise its native runner, inside one surface process scope carrying the selected surface ID and owner token. Selectors, retry arguments, and the native JSON reporter are forwarded through a declared command, so wrappers such as the headless-Linux Electron launcher remain active remotely. It always writes a target shard summary after `test-run`, including native results and any normalized process leaks, so a native test failure is classified as `failed` rather than a setup failure. The worker exports `CI=1` and `TEST_RETRIES`, so child runners see the `FLAKY_TEST_RETRIES` default ([flaky-tests.md](./flaky-tests.md)). If a setup step fails before the test run and no shard summary was produced, the worker synthesizes an `infra-failed` shard summary naming the failed step (`checkout` failures are reported distinctly). Native result paths rooted at the worker checkout (`/workspace/television`) are normalized to repository-relative paths after download. @@ -69,7 +110,7 @@ The coordinator attributes failures through remote wait and final log capture to ## Options -The coordinator's option set, with code-true defaults. The canonical runner ([test-runner.md](./test-runner.md)) overrides some of these, noted inline. +The coordinator's options and defaults follow. The canonical runner ([test-runner.md](./test-runner.md)) overrides some of these, noted inline. | Option | Default | Notes | |---|---|---| @@ -81,22 +122,23 @@ The coordinator's option set, with code-true defaults. The canonical runner ([te | `--arch` | `x64` | other values rejected. | | `--commit` | `HEAD` | the committed revision to test. | | `--timeout-profile` | `normal` | the canonical runner passes `cold`. See drift note below. | -| `--worker-timeout-seconds`, `--lock-ttl-seconds` | per profile | override the profile values. | +| `--worker-timeout-seconds` | per profile | override the shard execution timeout. | +| `--lock-ttl-seconds` | `1200` | positive recovery interval since the last successful renewal, independent of timeout profile; expiry requires cleanup before test reuse. | | `--retry-infra` | `0` | the canonical runner passes `2`. | | `--allow-dirty` | off | dispatch from a dirty tree, ignoring local edits. The canonical runner's `--ignore-uncommitted` is translated to this. | | `--test-retries` | `0` | non-unit retry count passed to the worker. | | `--repo-url` | the checkout's `origin` as a GitHub HTTPS URL | repository fetched in the sandbox ([#Repository](#Repository)). | | `--playwright-browsers-path` | `/home/playwright/.cache/ms-playwright` | browser cache path. | | `--output-dir` | `.blaxel-testshards/latest` | run output location. | -| `--audit-stale-owners` | off | lease one worker, install/check system prerequisites, run the read-only stale-owner diagnostic, print its JSON result, and release the lease without dispatching tests. It may be combined only with `--pool` and `--arch`. | +| `--audit-stale-owners` | off | reserve one available worker without recovery or residue removal, install/check system prerequisites, run the read-only stale-owner diagnostic, print its JSON result, and release without dispatching tests. It may be combined only with `--pool` and `--arch`. | -Timeout profiles (code-true): `normal` = worker 240 s, lock TTL 900 s; `cold` = worker 480 s, lock TTL 1200 s; `repair` = worker 480 s, lock TTL 1200 s. +Worker execution timeout profiles: `normal` = 240 s; `cold` = 480 s; `repair` = 480 s. All profiles use the same default recovery interval above. **Default profile.** The coordinator's own default timeout profile is `normal`, but the canonical runner invokes Blaxel with `--timeout-profile cold`, so a `npm test -- blaxel` run uses `cold` unless overridden. ## Output -The run output directory (default `.blaxel-testshards/latest`) contains `request.json`, combined `report.json`, compact `summary.json`, and any retained per-shard `shard-.log` and `shard-.json` files ([sharded-execution.md](./sharded-execution.md)). A path is exposed only after its local file is retained; shards that never produced a log or result do not advertise an inferred filename. Planned suite runs also contain `shard-plan.json`. Completed shard downloads add `setup-shard-.json` and, when the worker produced it, top-level `native-shard-.json`. Per-surface native reports, Vitest attempt sidecars, lifecycle results, process-leak results, and generated runner inputs are downloaded under `artifacts/shard-/`, preserving their worker-relative paths. Infrastructure retries write `attempt-/` subdirectories with the same applicable layout. Each shard summary exposes normalized restore/wake, checkout, dependency cache hit/miss, system setup, readiness, test-compute, and report-download phases; the coordinator prints those rows and passes them into the canonical run events and summary. Ordinary completion exits `0` only for a passed combined report and `1` for a failed report. A handled `SIGINT` or `SIGTERM` exits `130`; when an attempt exists it writes the interrupted provider report first. When the canonical runner receives that coordinator exit, it finalizes the normalized incomplete run and propagates `130`. +The run output directory (default `.blaxel-testshards/latest`) contains `request.json`, combined `report.json`, compact `summary.json`, and any retained per-shard `shard-.log` and `shard-.json` files ([sharded-execution.md](./sharded-execution.md)). A path is exposed only after its local file is retained; shards that never produced a log or result do not advertise an inferred filename. Planned suite runs also contain `shard-plan.json`. Completed shard downloads add `setup-shard-.json` and, when the worker produced it, top-level `native-shard-.json`. Per-surface native reports, Vitest attempt sidecars, lifecycle results, process-leak results, resource sidecars, recovery receipts, and generated runner inputs are downloaded under `artifacts/shard-/`, preserving their worker-relative paths. Infrastructure retries write `attempt-/` subdirectories with the same applicable layout. Each shard summary exposes normalized restore/wake, checkout, dependency cache hit/miss, system setup, readiness, test-compute, and report-download phases; the coordinator prints those rows and passes them into the canonical run events and summary. Ordinary completion exits `0` only for a passed combined report and `1` for a failed report. A handled `SIGINT` or `SIGTERM` exits `130`; when an attempt exists it writes the interrupted provider report first. When the canonical runner receives that coordinator exit, it finalizes the normalized incomplete run and propagates `130`. ## Operations @@ -113,7 +155,7 @@ The GitHub token the workers use is obtained from the corporate 1Password item * ### Pool provisioning -The canonical runner uses pool `poc`. Provision it by deleting the complete pool with `npm test -- pool blaxel delete --pool poc`, then recreating all 72 fresh workers with `npm test -- pool blaxel ensure --pool poc` ([#Pool](#Pool)). Use `list --pool poc` only for inspection. `ensure` refuses a populated pool and intentionally does not reconcile existing runtime state. A bare command targets the manager's `default` pool, which the canonical runner does not use. +The canonical runner uses pool `poc`. Provision it by deleting the complete pool with `npm test -- pool blaxel delete --pool poc`, then recreating all 72 fresh workers with `npm test -- pool blaxel ensure --pool poc` ([#Pool](#Pool)). Recreation needs an idle pool: deletion reports workers it refused and exits nonzero if any requested worker was not deleted. Use `list --pool poc` for inspection. `ensure` refuses a populated pool and intentionally does not reconcile existing runtime state. A bare command targets the manager's `default` pool, which the canonical runner does not use. ### Opt-in local suite exclusions @@ -125,7 +167,8 @@ Under the testing policy's [rule for integration against paid services or comple `--suite target` is not sufficient for a change that affects plan transfer; [sharded-execution.md's Testing section](./sharded-execution.md#Testing) defines the required evidence. -A change to pool provisioning or runtime selection requires the live run to use freshly recreated `poc` workers, since `ensure` refuses a populated pool ([#Pool](#Pool)). +A change to pool provisioning or runtime selection requires the live run to use freshly provisioned workers, since `ensure` refuses a populated pool ([#Pool](#Pool)). A dedicated pool may supply this evidence without deleting shared `poc` workers. -A change to repository selection, or to how a worker's checkout moves between repositories, requires live runs from checkouts of two repositories on the same pool, including a run on workers whose checkout last served the other repository. +Recovery and cleanup experiments use disposable owned processes on a dedicated pool, never intentionally contaminating shared workers. Automated qualification uses the runner's [Blaxel harness placement](./test-runner.md#^blaxel-harness-placement) and the registry's explicitly selected experiment surfaces, excluded from `all` and ordinary CI. Temporary-file placement needs live observation of Node-created paths and Chromium shared-memory descriptor paths under the attempt's `TMPDIR`. +A change to repository selection, or to how a worker's checkout moves between repositories, requires live runs from checkouts of two repositories on the same pool, including a run on workers whose checkout last served the other repository. diff --git a/specs/arch/test-runner/reporting.md b/specs/arch/test-runner/reporting.md index 0cc4fab1..a0eea8ad 100644 --- a/specs/arch/test-runner/reporting.md +++ b/specs/arch/test-runner/reporting.md @@ -1,10 +1,10 @@ -*The run directory, normalized reports, timing events, and CI run artifacts every test run produces.* +*The run directory, normalized reports, resource diagnostics, timing events, and CI run artifacts test runs produce.* # Reporting Every test run leaves a compact result for people and agents, detailed files for debugging, and machine-readable timing records. The timing records that Blaxel runs leave in their run directories are the input for balancing future test runs across parallel machines. -This spec is authoritative for the *run directory*, the *normalized surface result* model, the per-run timing event stream, compact-summary discipline, status accounting, and the run artifacts GitHub Actions jobs upload. Local runs, regular GitHub CI jobs, and Blaxel runs all produce this model; Blaxel provider results are translated into it by [sharded-execution.md](./sharded-execution.md). The runner that drives reporting is [test-runner.md](./test-runner.md); the testing discipline behind the honesty of these reports is [testing-policy.md](../testing-policy.md). Duration rollups and shard planning are owned by [sharded-execution.md](./sharded-execution.md). +This spec is authoritative for the *run directory*, the *normalized surface result* model, Blaxel resource diagnostics, the per-run timing event stream, compact-summary discipline, status accounting, and the run artifacts GitHub Actions jobs upload. Local runs, regular GitHub CI jobs, and Blaxel runs all produce the shared reporting model; Blaxel provider results are translated into it by [sharded-execution.md](./sharded-execution.md). The runner that drives reporting is [test-runner.md](./test-runner.md); the testing discipline behind the honesty of these reports is [testing-policy.md](../testing-policy.md). Duration rollups and shard planning are owned by [sharded-execution.md](./sharded-execution.md). ## Run directory @@ -82,6 +82,7 @@ interface NormalizedSurfaceResult { logPath: string | null; nativeResultPath?: string | null; nativeResultPaths?: string[]; + resourceDiagnosticsPaths?: string[]; counts: Counts; failedTests: Array>; flakyRecoveredTests: Array>; @@ -210,6 +211,20 @@ A process leak makes every surface named by `owningSurfaceIds` failed with `infr `events.ndjson` is an explicit exception to compact-summary discipline. It enumerates every test attempt because its purpose is machine analysis; it is not printed into agent context as a normal command result. +## Resource diagnostics + +Each Blaxel surface execution, planned or targeted, records available RAM, shared-memory usage, free bytes on the effective temporary filesystem, and the Linux OOM-kill counter at start and end. The parent captures and immediately retains the start observation before surface preparation, then captures the end after teardown, so a browser or test-worker crash does not erase the baseline. Each value records its units and observation time; unavailable probes and missing final observations are explicit, never substituted with zero. Counter resets cannot be reported as negative OOM events. ^resource-diagnostics + +One sidecar identifies the run, shard attempt, and surface execution and links to its native attempt evidence. Retain it for failed tests, failures that recover on retry, and missing native output. The surface result's `resourceDiagnosticsPaths` links only retained files relative to the owning run directory. Attempt history keeps its own sidecars when an infrastructure retry supersedes that attempt; aggregation does not overwrite them with the successful retry's evidence. A sidecar with missing native evidence does not invent a test result or a selected-surface outcome. + +If any native attempt failed, or execution evidence is incomplete, capture relevant kernel OOM lines for the execution interval through a bounded read. Keep at most 64 KiB of these lines per surface execution, 128 KiB for the entire sidecar, and 8 MiB of resource diagnostics per shard attempt, including recovery receipts. Reserve space for compact start/end records before optional excerpts. State truncation or omission and unavailable-probe reasons. Probe or diagnostic-write failure adds an observability warning without changing the test outcome or masking its original failure. ^resource-diagnostic-bounds + +Start/end observations give context and an OOM-counter delta, not a peak-memory measurement or the exact failing allocation. A system counter increase alone does not identify which process was killed. Kernel evidence may identify the process; absent or inaccessible kernel output is unknown, not evidence of no OOM. No periodic sampler, full process inventory, heap snapshot, video, or NetLog is required by this contract. + +A recovery receipt records the prior worker/run/lock identity, last renewal and deadline, process identities stopped and cleanup outcome, and whether undownloaded reports were discarded. Preserve it with the incoming attempt's setup evidence across results reset and download, within the same attempt diagnostic budget. A receipt disclosing discarded reports supplies no passing evidence for that abandoned run. [Blaxel's lifecycle](./blaxel-testshards.md#Abandoned-run recovery) owns when such discard is permitted and how worker retention stays bounded. + +These artifacts contain no credentials, request query strings or bodies, or arbitrary process environments; process identities exclude arguments. They remain outside disposable scratch and are downloaded with the provider artifacts. Resource samples and kernel excerpts stay out of the timing event stream and compact console output; links make the detailed evidence available. Existing native retry records and trace policy remain unchanged. + ## Timing event stream `events.ndjson` is UTF-8 newline-delimited JSON. Each event is serialized with RFC 8785 JSON canonicalization followed by one LF byte; the file also ends in LF. Each line is one `TimingEvent`. The stream is complete only after the run has finished; it is produced from native reports and provider output, not treated as a live progress protocol. diff --git a/specs/arch/test-runner/sharded-execution.md b/specs/arch/test-runner/sharded-execution.md index 07f702b8..825df010 100644 --- a/specs/arch/test-runner/sharded-execution.md +++ b/specs/arch/test-runner/sharded-execution.md @@ -225,6 +225,8 @@ Blaxel workers and regular GitHub CI matrix jobs run `scripts/run-test-shard.mjs - `--fail-fast` stops the shard after the first failed group or task. Every later task that had files assigned by the plan remains in the shard summary with `status: "skipped"`, `skipReason: "fail-fast"`, its assigned files, and no collected files or runner artifacts. - The worker writes `shard-.json` into its results directory, default `.testshards/results`, and exits non-zero if any task failed or plan validation failed. +On Blaxel, the shared worker preserves the outer shard attempt's ownership and temporary-directory environment; [the Blaxel lifecycle](./blaxel-testshards.md#Locking) owns these, including teardown and download before release. Resource sidecars travel with the task artifacts under [reporting's resource-diagnostics contract](./reporting.md#Resource diagnostics), including evidence from superseded infrastructure attempts. GitHub workers retain their existing host lifecycle. + ## Blaxel provider report shape The Blaxel coordinator collects the plan and per-shard summaries into a provider report. Regular GitHub CI retains each direct worker's shard summary and timing stream as job artifacts; the required agreement join is owned by [github-ci.md](./github-ci.md) under the invariants above. @@ -250,6 +252,7 @@ interface ProviderTask { failureStep?: string | null; processLeaks?: NormalizedProcessLeak[]; nativeResultPath?: string | null; + resourceDiagnosticsPaths?: string[]; [key: string]: unknown; } @@ -338,4 +341,3 @@ A seam must also pass the generated `--test-list` file through the real Electron Under the testing policy's [rule for integration against paid services or complex host dependencies](../testing-policy.md#integration-against-paid-services-or-complex-host-dependencies-is-important), a live planned Blaxel suite run must cross the real Blaxel API and the real transport to the remote process. The run must show that the worker validates and executes the `ShardPlan` it receives. A `--suite target` run carries no provider plan, so it cannot prove this handoff. Live GitHub evidence follows the [isolated CI requirements in github-ci.md](./github-ci.md#Testing). The evidence must show that independently started matrix jobs derive the same valid plan from their shared inputs. The strict join must validate the jobs' summaries and exact file assignments. - diff --git a/specs/arch/test-runner/test-registry.md b/specs/arch/test-runner/test-registry.md index 3a6737d6..7308b283 100644 --- a/specs/arch/test-runner/test-registry.md +++ b/specs/arch/test-runner/test-registry.md @@ -92,7 +92,9 @@ The registered suites: - `daemon-acceptance` → `kind:daemon-acceptance` - `all` → `kind:unit`, `kind:e2e` -`all` is the broad suite and is deliberately unit + non-agent e2e: agent, experiment, live PostHog roundtrip, and production-identity daemon acceptance surfaces are not part of `all`. Agent, telemetry, and daemon acceptance surfaces must be selected explicitly with their named suites. The experiment suite contains two placement classes. Lifecycle-fault experiments carry `isolated-github-only`, so selection alone cannot bypass their execution-placement guard. `experiment:dynamic-services` is a normal runnable service fixture without that tag; its protection is exclusion from `all`, so it runs only when selected explicitly. `telemetry-posthog-roundtrip` owns the live telemetry ↔ PostHog integration surface that requires secrets. `daemon-acceptance` owns the host-mutating packed-CLI and production-service lifecycle surface. Both are excluded from CI and broad verification. Build-config-integrity coverage is a normal unit surface, `unit:build-config`, so it runs in `all` and CI without secrets. +`all` is the broad suite and is deliberately unit + non-agent e2e: agent, experiment, live PostHog roundtrip, and production-identity daemon acceptance surfaces are not part of `all`. Agent, telemetry, and daemon acceptance surfaces must be selected explicitly with their named suites. Lifecycle-fault experiments carry `isolated-github-only`, so selection alone cannot bypass their execution-placement guard. `experiment:dynamic-services` is a normal runnable service fixture without that tag; its protection is exclusion from `all`, so it runs only when selected explicitly. `telemetry-posthog-roundtrip` owns the live telemetry ↔ PostHog integration surface that requires secrets. `daemon-acceptance` owns the host-mutating packed-CLI and production-service lifecycle surface. Both are excluded from CI and broad verification. Build-config-integrity coverage is a normal unit surface, `unit:build-config`, so it runs in `all` and CI without secrets. + +Live Blaxel harness qualification uses two separately owned `kind: "experiment"` surfaces: `experiment:blaxel-harness` is the automated driver, and `experiment:blaxel-harness-fixtures` contains the native fixtures it dispatches, including deliberate failures and recovered retries. The fixture surface carries the selection tag `blaxel-harness-fixture` and never dispatches another run. Both are excluded from `all` and ordinary CI; explicit selection cannot recursively include the driver in its fixture runs. Their execution placement is owned by [Blaxel harness qualification](./test-runner.md#Blaxel harness qualification). ^blaxel-harness-surfaces ## Execution groups @@ -178,4 +180,3 @@ Registry discovery and validation scan only the *canonical include roots* `packa - a surface whose `roots` or `excludeRoots` entry does not exist, - a service with a duplicate `id`, a missing Vite config, an unsupported `kind`, an invalid `publishUrlEnv` name, or a `publishUrlEnv` duplicated within its surface, - an unregistered runner config file found under the include roots — any file matching `vitest.config.ts`, `vitest.e2e.config.ts`, or `playwright.config.ts` that no surface declares. - diff --git a/specs/arch/test-runner/test-runner.md b/specs/arch/test-runner/test-runner.md index 3a376055..b1c3b9bc 100644 --- a/specs/arch/test-runner/test-runner.md +++ b/specs/arch/test-runner/test-runner.md @@ -67,7 +67,7 @@ A *selector* chooses which tests run; the runner resolves selectors against [tes | `--runner ` | Surfaces using a runner. | | `--tag ` | Surfaces carrying a registry tag, such as `browser` or `electron`. | -The local provider accepts every registered suite. A Blaxel suite selection accepts only `all`, `unit`, or `e2e`; narrower remote work uses surface, file, or grep selection through the target path. +The local provider accepts every registered suite. A Blaxel suite selection accepts only `all`, `unit`, or `e2e`; narrower remote work uses surface, file, or grep selection through the target path, or package, runner, or tag selection through the planned selection path. `--file` must resolve to exactly one owning surface: if no surface owns the file the runner fails; if several do, it fails and asks for `--surface` to disambiguate. On the local execution path, a `--file` or `--grep` selection against a surface that does not declare support for it is rejected while building that surface's command. The targeted Blaxel path does not currently apply this `supports` check. The runner parses these selectors itself and rejects unknown positional arguments; the sole positional accepted is the `verify` provider. @@ -89,7 +89,7 @@ Production attestations have exactly two source classes: a publication-qualifyin A *broad run* is a provider shortcut that runs the full `all` set with no narrowing selector. A broad run skips lint and type-check, so the provider shortcuts refuse it with a message directing the caller to `npm run verify`. -The `telemetry-posthog-roundtrip` and `daemon-acceptance` suites are not broad verification. They are specialized, opt-in local suites and are excluded from `all`. The first crosses the live telemetry ↔ PostHog boundary; the second replaces the active global `tv` package and the literal `com.television.server` user service on a designated developer host. Build-config-integrity tests are not part of either suite: `unit:build-config` is a normal unit surface in `all` and uses no secrets, network, or host service mutation. +The `telemetry-posthog-roundtrip` and `daemon-acceptance` suites and [Blaxel harness surfaces](#Blaxel harness qualification) are not broad verification and are excluded from `all`. The two suites are specialized, opt-in local suites: the first crosses the live telemetry ↔ PostHog boundary; the second replaces the active global `tv` package and the literal `com.television.server` user service on a designated developer host. The harness surfaces have the separate coordinator/worker placement below. Build-config-integrity tests are not part of either suite: `unit:build-config` is a normal unit surface in `all` and uses no secrets, network, or host service mutation. The guardrail trips only on the effective `all` suite with nothing narrowing it: a bare provider (which defaults to `--suite all`), an explicit `--suite all` or `--all`, or a bare `--grep` with no other selector. Everything else passes through: @@ -144,6 +144,12 @@ The surface builds the publishable CLI, packs it, and force-installs that tarbal Persisted serve rejects port `0`, so the surface cannot keep a kernel-owned port reservation while handing a stable number to the service's config file. For each installation it creates a temporary [Television home](../../product/cli.md#^cli-home), binds loopback port `0`, reads the assigned nonzero port, closes the reservation, writes that port with `tv --home config set port `, and immediately installs with `tv --home serve --persist` and probes the daemon on that number. This is a narrow declared exception to [the normal continuous-reservation rule](#^test-dynamic-ports). `TV_DAEMON_TEST_DIRTY_START=1` asks the surface to create a production-identity installation before its foundation bulldozer; this is a test-harness mode for proving convergence from prior service residue, and it remains protected by `TV_DAEMON_TEST_HOST=1`. +## Blaxel harness qualification + +Live tests of the Blaxel harness use the opt-in [driver and fixture surfaces](./test-registry.md#^blaxel-harness-surfaces). Run the automated driver with `npm test -- local --surface experiment:blaxel-harness` on a dedicated credentialed Blaxel coordinator sandbox outside the worker pool, using a committed and pushed checkout. It must not run on a shared developer host or a leased shard worker. It owns a disposable private pool and uses canonical Blaxel commands against that pool; it never selects itself for remote execution. Ordinary repository contracts and Linux process/filesystem seams remain ordinary repository tests and need no live-provider credentials. ^blaxel-harness-placement + +The driver selects only the committed fixture surface for worker runs: `--tag blaxel-harness-fixture` delegates a planned `--suite selection --surfaces experiment:blaxel-harness-fixtures` run; `--surface experiment:blaxel-harness-fixtures` delegates a targeted run. Workers load these fixtures from the committed registry, without a forwarded replacement-registry hook. Deliberate fixture failures remain failed native results; the driver asserts their expected outcomes and retained evidence. The driver cleans up its coordinators and private pool when it finishes, retaining reports and identifying any cleanup it could not complete. + ## Retries Non-unit Vitest and Playwright surfaces retry twice at the runner level by default, including targeted file/grep/surface runs; unit surfaces receive no runner-level retries. `--retries ` overrides the non-unit count and is ignored (with a notice) for a unit-only selection. This is the runner's blanket retry budget; the per-test flaky annotation that opts a single case into retries is owned by [flaky-tests.md](./flaky-tests.md). @@ -219,6 +225,8 @@ A combined Vitest run uses a generated config with one `test.projects` entry per Persistent workers inspect Television owner tokens before accepting a new scope. A token is stale only when its recorded supervisor PID and process start time no longer identify a live supervisor; a concurrent run with a live supervisor is left untouched. A stale process is captured and reaped, and the worker reports an infrastructure preflight failure naming the token and process; it does not run a new surface in contaminated state. An ephemeral provider still performs the check so its correctness does not depend on host disposal. +Blaxel additionally reserves the whole worker through result retrieval. Its [abandoned-run recovery](./blaxel-testshards.md#Abandoned-run recovery) may stop that abandoned attempt's live supervisors after coordinator renewal expires; it does not weaken the ordinary stale-owner rule for other runs. Surface supervisors and helpers preserve the outer shard attempt attribution alongside their own surface tokens. [Reporting](./reporting.md#Resource diagnostics) owns the resource evidence captured across Blaxel surface preparation, execution, and teardown. + ## End-of-surface leak detection After registered teardown and its graceful interval, the scope owner enumerates every live process carrying the owner token. Each candidate is revalidated immediately before cleanup. A candidate that exits naturally before any cleanup signal is sent is not a *process leak* and is omitted; this closes the normal child-exit race where a runner exits just before one of its helper processes. A process that requires `SIGTERM` or `SIGKILL` remains a leak even if a later revalidation finds that it has exited. For every actual leak, the owner captures its identity and listening sockets before sending `SIGTERM`, escalates to `SIGKILL` after the bounded interval, and scans once more to confirm that no survivor remains. Before each signal it confirms that the PID still has the same process start time and owner token, preventing PID reuse from targeting another process. The diagnostic identifies at least the surface owner, PID, parent PID, process group, process start time, executable command, and every attributable listening TCP socket. [reporting.md](./reporting.md) owns the normalized record that carries this attribution. ^test-process-leak @@ -258,4 +266,3 @@ Under the testing policy's [mocking policy](../testing-policy.md#Mocking policy) Under the testing policy's [compositional coverage rule](../testing-policy.md#Compositional coverage across clean seams), this spec owns the decision that selects a provider automatically from the home marker and, on marked hosts, a remote preflight result. [Preflight's Testing section](./preflight.md#Testing) requires a real Git process to run the production git-safety sequence in a repository with an `origin` remote. It also requires live evidence for the Blaxel capability checks. The live planned run required by [sharded execution's Testing section](./sharded-execution.md#Testing) supplies the real provider report for the remote run. Reading `.github/workflows/ci.yml` proves only which command the file contains. The isolated live run required by [GitHub CI's Testing section](./github-ci.md#Testing) establishes that GitHub starts the job and that the job invokes the test runner. - diff --git a/specs/arch/testing-policy.md b/specs/arch/testing-policy.md index 9d4f5086..e291467d 100644 --- a/specs/arch/testing-policy.md +++ b/specs/arch/testing-policy.md @@ -207,6 +207,10 @@ In order to liberally cover boundary seams across the codebase, complex harness Prefer real, committed fixtures over invented data. Better yet, generate fixtures dynamically from current code paths, driven by simpler inputs with fewer assumptions, whenever possible. +A reusable build created by a test helper belongs to its worker: it is shared within that worker, published only after preparation succeeds, and removed at worker teardown rather than per test case. The [Blaxel shard attempt lifecycle](test-runner/blaxel-testshards.md#Temporary files) supplies the backstop when a killed worker cannot run its teardown. ^worker-build-lifetime + +For new or substantially changed browser acceptances that use the `ProductServer` helper, the default is its built application unless the assertion needs development-module behavior. This exercises shipped assets and avoids loading hundreds of development modules for a product assertion. This default does not require retroactive conversion of other tests or apply to in-process `Server` walks using `appURLForServer()`, component fixtures, or Electron module/dispatcher seams. ^product-acceptance-built-app + ## Timing, observable status, cleanup, and flakiness Prefer real readiness and state observation over sleeps. Take the time to make complex tasks reliably observable for testing purposes. ^flake-avoidance diff --git a/specs/index.md b/specs/index.md index 2e5833ec..4b647979 100644 --- a/specs/index.md +++ b/specs/index.md @@ -40,7 +40,7 @@ Generated from the first-line description of every spec by `scripts/specs-index. - `licensing.md` (294 lines) — *The licensing implementation: bundle-derived third-party notices generation for the esbuild, Vite, and skill outputs, the config file for elections, notice texts, and ignored packages, the vendored-asset provenance manifest, the committed source-surface notices, the suite-run allowlist gate, and the assertions on the CLI tarball and the desktop upload directory — wired into the existing build, verify, and publish pipeline.* - `making-skills.md` (30 lines) — *Arch spec: how a bundled skill gets made — authority, bundle derivation, and final consumer delivery.* - `node-versions.md` (68 lines) — *The Node and npm versions used for repository work, automation, publishing, and published-package support.* - - `testing-policy.md` (234 lines) — *How Television is tested: coverage requirements, the fixture/mock distinction, the three test shapes and their declaration schema, and mock vs integration discipline.* + - `testing-policy.md` (238 lines) — *How Television is tested: coverage requirements, the fixture/mock distinction, the three test shapes and their declaration schema, and mock vs integration discipline.* - **artifact-frame/** - `artifact-bridge.md` (354 lines) — *The artifact bridge: how an embedded artifact document and the app cooperate across the iframe/webview boundary — lifecycle and readiness, navigation reporting, input observation, keyboard forwarding, and live updates for shared artifacts.* - `index.md` (51 lines) — *The artifact-frame architecture root: the domain's module map, the frame lifecycle rules, and the deliberate boundary between what is specified here and what remains governed by code.* @@ -90,14 +90,14 @@ Generated from the first-line description of every spec by `scripts/specs-index. - `sink.md` (63 lines) — *The telemetry sink: the PostHog transport, its bounded in-memory buffer, fire-and-forget delivery that never blocks or crashes the server, and the platform operations (region, key, agent access).* - **test-runner/** - `attestation.md` (132 lines) — *Tree-hash test attestation: recording that an exact tree passed full validation as a bare `refs/testpass//` ref, and letting validation-bearing CI runs skip their heavy test jobs — never their cheap ones — when the identical tree is already attested under the current version.* - - `blaxel-testshards.md` (131 lines) — *The Blaxel shard coordinator: leasing a persistent sandbox pool, selecting the repository workers fetch, sourcing a GitHub token, preparing and locking workers, running shards with dependency-hash caching, and classifying Blaxel shard outcomes.* + - `blaxel-testshards.md` (174 lines) — *The Blaxel shard coordinator: persistent workers, exclusive leases and abandoned-run recovery, owned temporary files, remote checkout and execution, and retained shard results.* - `flaky-tests.md` (74 lines) — *The flaky-test policy: the runner's retry budget, the per-test `FLAKY_TEST_RETRIES` annotation, and how recovered flakes are surfaced.* - `github-ci.md` (137 lines) — *The GitHub Actions CI execution contract: PR concurrency, per-job dependency and browser environment, build-once fan-out with an artifact manifest, cleanup tool prerequisites, worker counts, and the retirement record for the Blacksmith-dispatched shard provider.* - `preflight.md` (92 lines) — *The checks the test runner runs before tests: local capability checks for the selected surfaces, and the git-safety and provider-auth checks that gate a committed-revision remote run.* - - `reporting.md` (411 lines) — *The run directory, normalized reports, timing events, and CI run artifacts every test run produces.* - - `sharded-execution.md` (341 lines) — *The shared remote execution model: deterministic duration-aware plans, per-worker execution, provider reports, retries, and normalization.* - - `test-registry.md` (181 lines) — *The test registry: how `test.config.mjs` declares the surfaces, suites, and execution groups the test runner selects, owns files against, and validates.* - - `test-runner.md` (261 lines) — *The canonical `npm test` and `npm run verify` command surface: how a caller selects tests, picks where they run, and runs them, with the guardrails that keep broad runs honest.* + - `reporting.md` (426 lines) — *The run directory, normalized reports, resource diagnostics, timing events, and CI run artifacts test runs produce.* + - `sharded-execution.md` (343 lines) — *The shared remote execution model: deterministic duration-aware plans, per-worker execution, provider reports, retries, and normalization.* + - `test-registry.md` (182 lines) — *The test registry: how `test.config.mjs` declares the surfaces, suites, and execution groups the test runner selects, owns files against, and validates.* + - `test-runner.md` (268 lines) — *The canonical `npm test` and `npm run verify` command surface: how a caller selects tests, picks where they run, and runs them, with the guardrails that keep broad runs honest.* - **themes/** - `authoring.md` (204 lines) — *Theme-authoring guidance: the self-contained theming document bundled inside the `television` skill, the authoritative UI material it carries, and the workflow it gives an agent.* - `bundled-installation.md` (77 lines) — *Bundled theme installation: validated package assets, CLI path resolution, minimum-version installation and replacement with backup copies, and the one-time default-theme selection.* diff --git a/test.config.mjs b/test.config.mjs index 1b279a3d..4cafb960 100644 --- a/test.config.mjs +++ b/test.config.mjs @@ -92,6 +92,10 @@ export default { executionGroup("agent:root", "agent: root", 14, [ surface("agent:root", "vitest", "test/agent/vitest.config.ts", { kind: "agent", roots: ["test/agent"], agent: true, tags: ["agent"] }), ]), + executionGroup("experiment:blaxel-harness", "experiment: Blaxel harness", 88, [ + surface("experiment:blaxel-harness", "vitest", "test/runner-fixtures/blaxel-harness/driver/vitest.config.ts", { kind: "experiment", roots: ["test/runner-fixtures/blaxel-harness/driver"], tags: ["blaxel-harness-driver"] }), + surface("experiment:blaxel-harness-fixtures", "vitest", "test/runner-fixtures/blaxel-harness/fixtures/vitest.config.ts", { kind: "experiment", roots: ["test/runner-fixtures/blaxel-harness/fixtures"], tags: ["blaxel-harness-fixture"] }), + ]), executionGroup("experiment:dynamic-services", "experiment: dynamic services", 89, [ surface("experiment:dynamic-services", "playwright", "test/runner-fixtures/dynamic-services/playwright.config.ts", { kind: "experiment", diff --git a/test/fixtures/resource-diagnostics/native.fixture.ts b/test/fixtures/resource-diagnostics/native.fixture.ts new file mode 100644 index 00000000..9254bdc3 --- /dev/null +++ b/test/fixtures/resource-diagnostics/native.fixture.ts @@ -0,0 +1,8 @@ +import { afterAll, expect, test } from "vitest"; +import { writeFileSync } from "node:fs"; +import path from "node:path"; + +afterAll(() => writeFileSync(path.join(process.env.RESOURCE_FIXTURE_ROOT!, "teardown.json"), JSON.stringify({ observedAt: new Date().toISOString() }))); +test("retains a failed attempt before recovery", (context) => { + expect(context.task.result?.retryCount ?? 0).toBeGreaterThan(0); +}); diff --git a/test/fixtures/resource-diagnostics/prepare.mjs b/test/fixtures/resource-diagnostics/prepare.mjs new file mode 100644 index 00000000..b1f8b222 --- /dev/null +++ b/test/fixtures/resource-diagnostics/prepare.mjs @@ -0,0 +1,12 @@ +import fs from "node:fs"; +import path from "node:path"; + +const root = process.env.RESOURCE_FIXTURE_ROOT; +const resources = path.join(root, "results", "resources"); +const records = fs.existsSync(resources) ? fs.readdirSync(resources).filter((name) => name.endsWith(".json")).map((name) => JSON.parse(fs.readFileSync(path.join(resources, name), "utf8"))) : []; +fs.writeFileSync(path.join(root, "preparation.json"), JSON.stringify({ observedAt: new Date().toISOString(), records })); +if (process.env.RESOURCE_FIXTURE_MODE === "interrupt") { + // Abruptly end our own fixture parent, then exit this only descendant too. + // TERM permits the real supervisor to finalize, which is a different path. + process.kill(process.ppid, "SIGKILL"); +} diff --git a/test/fixtures/resource-diagnostics/setup.mjs b/test/fixtures/resource-diagnostics/setup.mjs new file mode 100644 index 00000000..e1930cac --- /dev/null +++ b/test/fixtures/resource-diagnostics/setup.mjs @@ -0,0 +1,3 @@ +export default function setup() { + if (process.env.RESOURCE_FIXTURE_MODE === "exit") process.exit(17); +} diff --git a/test/fixtures/worker-builds/exercise.ts b/test/fixtures/worker-builds/exercise.ts new file mode 100644 index 00000000..9804d025 --- /dev/null +++ b/test/fixtures/worker-builds/exercise.ts @@ -0,0 +1,52 @@ +import assert from "node:assert/strict"; +import { appendFileSync, existsSync, readFileSync, readdirSync } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { disposeAllProductServers, launchProductServer } from "../../helpers/product-server.ts"; +import { buildVersionedWebBundle } from "../../helpers/versioned-web-bundle.ts"; + +export async function exerciseWorkerBuilds(caseNumber: number): Promise { + const temporaryRoot = os.tmpdir(); + const recordPath = process.env.TV_WORKER_BUILD_RECORD; + assert.ok(recordPath, "the parent supplies a private observation file"); + + if (caseNumber === 1) { + await assert.rejects(launchProductServer({ + onboardingBundle: path.join(temporaryRoot, "absent-bundle"), + env: { VITEST: "" }, + }), /ENOENT/); + assert.deepEqual(readdirSync(temporaryRoot).filter((entry) => entry.startsWith("television-product-e2e-")), [], + "the failed launch removes its home and private runtime before another launch"); + } + + const server = await launchProductServer({ env: { VITEST: "" } }); + let webBundle: string; + try { + const response = await fetch(server.serverURL); + assert.equal(response.status, 200); + const document = await response.text(); + const script = document.match(/src="([^"]+\.js)"/); + assert.ok(script, "the source CLI serves the built application"); + const asset = await fetch(new URL(script[1]!, server.serverURL)); + assert.equal(asset.status, 200); + assert.ok((await asset.text()).length > 0); + + webBundle = await buildVersionedWebBundle("worker-lifetime-fixture"); + const builtDocument = readFileSync(path.join(webBundle, "index.html"), "utf8"); + const builtScript = builtDocument.match(/src="([^"]+\.js)"/); + assert.ok(builtScript, "the real versioned builder emits an application entry"); + assert.ok(readFileSync(path.join(webBundle, builtScript[1]!)).length > 0); + } finally { + await server.dispose(); + await disposeAllProductServers(); + } + + const sourceDirectories = readdirSync(temporaryRoot) + .filter((entry) => entry.startsWith("television-source-cli-e2e-")) + .map((entry) => path.join(temporaryRoot, entry)); + assert.equal(sourceDirectories.length, 1, "one source build is shared across cases"); + assert.ok(existsSync(path.join(sourceDirectories[0]!, "tv.cjs"))); + assert.ok(existsSync(path.join(webBundle!, "index.html")), "case cleanup preserves the shared bundle"); + assert.equal(existsSync(server.home), false, "case cleanup removes its private home"); + appendFileSync(recordPath, `${JSON.stringify({ caseNumber, pid: process.pid, source: sourceDirectories[0], webBundle })}\n`); +} diff --git a/test/fixtures/worker-builds/playwright.fixture.ts b/test/fixtures/worker-builds/playwright.fixture.ts new file mode 100644 index 00000000..44be09a9 --- /dev/null +++ b/test/fixtures/worker-builds/playwright.fixture.ts @@ -0,0 +1,5 @@ +import { test } from "../../helpers/playwright.ts"; +import { exerciseWorkerBuilds } from "./exercise.ts"; + +test("first case prepares usable builds", async () => exerciseWorkerBuilds(1)); +test("second case reuses the worker builds", async () => exerciseWorkerBuilds(2)); diff --git a/test/fixtures/worker-builds/vitest.fixture.ts b/test/fixtures/worker-builds/vitest.fixture.ts new file mode 100644 index 00000000..8bc4c7a4 --- /dev/null +++ b/test/fixtures/worker-builds/vitest.fixture.ts @@ -0,0 +1,5 @@ +import { test } from "../../helpers/vitest.ts"; +import { exerciseWorkerBuilds } from "./exercise.ts"; + +test("first case prepares usable builds", async () => exerciseWorkerBuilds(1)); +test("second case reuses the worker builds", async () => exerciseWorkerBuilds(2)); diff --git a/test/helpers/application-requests.ts b/test/helpers/application-requests.ts new file mode 100644 index 00000000..08fad8d5 --- /dev/null +++ b/test/helpers/application-requests.ts @@ -0,0 +1,35 @@ +import { expect, type Page, type TestInfo } from "@playwright/test"; + +/** Retain paths only: authentication fragments and query values are never recorded. */ +export function recordApplicationRequests(pages: readonly Page[]) { + const inventories = pages.map((page) => { + const entries = new Set(); + page.on("request", (request) => { + entries.add(JSON.stringify({ + path: new URL(request.url()).pathname, + type: request.resourceType(), + application: request.frame() === page.mainFrame(), + })); + }); + return entries; + }); + const snapshot = () => inventories.map((entries) => [...entries].sort().map((entry) => + JSON.parse(entry) as { path: string; type: string; application: boolean } + )); + return { + expectBuilt(): void { + for (const requests of snapshot()) { + const application = requests.filter((request) => request.application); + expect(application.some((request) => /^\/(packages|@vite|@id|@fs|node_modules)\//.test(request.path)), + "the application loads shipped assets without development modules").toBe(false); + expect(application.some((request) => request.type === "document" && request.path === "/")).toBe(true); + expect(application.some((request) => request.type === "script" && /^\/assets\/.+\.js$/.test(request.path))).toBe(true); + } + }, + async attach(info: TestInfo): Promise { + await info.attach("application-request-inventory", { + body: Buffer.from(JSON.stringify(snapshot(), null, 2)), contentType: "application/json", + }); + }, + }; +} diff --git a/test/helpers/owned-process.ts b/test/helpers/owned-process.ts index 4410672f..ac0ed595 100644 --- a/test/helpers/owned-process.ts +++ b/test/helpers/owned-process.ts @@ -35,8 +35,10 @@ export function spawnOwnedProcess(command: string, args: readonly string[] = [], const { termGraceMs = TEST_PROCESS_TERM_GRACE_MS, env: requestedEnv, ...spawnOptions } = options; if (!Number.isFinite(termGraceMs) || termGraceMs < 0) throw new Error("termGraceMs must be a non-negative finite number"); const env: NodeJS.ProcessEnv = { ...(requestedEnv ?? process.env) }; - const ownerToken = process.env[OWNER_TOKEN_ENV]; - if (ownerToken !== undefined) env[OWNER_TOKEN_ENV] = ownerToken; + for (const key of [OWNER_TOKEN_ENV, "TV_TEST_SHARD_OWNER"]) { + if (process.env[key] !== undefined) env[key] = process.env[key]; + } + env.TMPDIR ??= process.env.TMPDIR; const detached = process.platform !== "win32"; const child = spawn(command, [...args], { ...spawnOptions, env, detached }); if (!child.pid) { diff --git a/test/helpers/playwright.ts b/test/helpers/playwright.ts index d2e51024..39bef2cb 100644 --- a/test/helpers/playwright.ts +++ b/test/helpers/playwright.ts @@ -1,10 +1,20 @@ import { expect, test as base } from "@playwright/test"; -import { disposeAllProductServers } from "./product-server.ts"; +import { disposeAllProductServers, disposeSourceCLIBuilds } from "./product-server.ts"; +import { disposeVersionedWebBuilds } from "./versioned-web-bundle.ts"; export { expect }; /** Playwright test registration with per-test cleanup for product servers, including interrupted launches, and their app proxies. */ -export const test = base.extend<{ productServerCleanup: void }>({ +export const test = base.extend<{ productServerCleanup: void }, { workerBuildCleanup: void }>({ + workerBuildCleanup: [async ({}, use) => { + try { + await use(); + } finally { + // Keep runtime files if a live server could not be stopped. + await disposeAllProductServers(); + await Promise.all([disposeSourceCLIBuilds(), disposeVersionedWebBuilds()]); + } + }, { scope: "worker", auto: true }], productServerCleanup: [async ({}, use) => { try { await use(); diff --git a/test/helpers/product-server.ts b/test/helpers/product-server.ts index 15a3f3cb..a8fc6eb3 100644 --- a/test/helpers/product-server.ts +++ b/test/helpers/product-server.ts @@ -9,11 +9,13 @@ import { fileURLToPath } from "node:url"; import { parseConnectURL } from "@telepath-computer/television-shared"; import { spawnOwnedProcess, type OwnedProcess } from "./owned-process.ts"; import { writeHomeConfig } from "./television-home.ts"; +import { WorkerBuilds } from "./worker-builds.ts"; const HERE = path.dirname(fileURLToPath(import.meta.url)); const REPO_ROOT = path.resolve(HERE, "../.."); const CLI_BUILD_SCRIPT = path.join(REPO_ROOT, "packages/cli/build.mjs"); -let sourceCLIEntry: string | null = null; +const sourceCLIBuilds = new WorkerBuilds("television-source-cli-e2e-"); +export const disposeSourceCLIBuilds = (): Promise => sourceCLIBuilds.dispose(); const serverStartupTimeoutMs = 30_000; export interface ProductServer { @@ -64,16 +66,15 @@ export interface LaunchProductServerOptions { env?: Record; } -function getSourceCLIEntry(): string { - if (sourceCLIEntry) return sourceCLIEntry; - const buildDir = mkdtempSync(path.join(os.tmpdir(), "television-source-cli-e2e-")); - sourceCLIEntry = path.join(buildDir, "tv.cjs"); - execFileSync(process.execPath, [CLI_BUILD_SCRIPT, "--outfile", sourceCLIEntry], { - cwd: REPO_ROOT, - stdio: "pipe", +async function getSourceCLIEntry(): Promise { + const buildDir = await sourceCLIBuilds.get("source", (directory) => { + execFileSync(process.execPath, [CLI_BUILD_SCRIPT, "--outfile", path.join(directory, "tv.cjs")], { + cwd: REPO_ROOT, + stdio: "pipe", + }); + installProductRuntimeAssets(directory); }); - installProductRuntimeAssets(buildDir); - return sourceCLIEntry; + return path.join(buildDir, "tv.cjs"); } /** @@ -81,28 +82,33 @@ function getSourceCLIEntry(): string { * The shared cached layout stays bundle-free so ordinary launches keep the * missing-bundle behavior. */ -function getCLIEntryWithBundles(options: { +function getCLIEntryWithBundles(sharedEntry: string, options: { onboardingBundle?: string; bundledThemesBundle?: string; }): { cliEntry: string; dispose: () => void } { - const sharedEntry = getSourceCLIEntry(); const runDir = mkdtempSync(path.join(os.tmpdir(), "television-product-e2e-bundle-")); - cpSync(path.dirname(sharedEntry), runDir, { recursive: true }); - if (options.onboardingBundle !== undefined) { - cpSync(options.onboardingBundle, path.join(runDir, "onboarding"), { recursive: true }); - } - if (options.bundledThemesBundle !== undefined) { - cpSync(options.bundledThemesBundle, path.join(runDir, "themes"), { recursive: true }); + try { + cpSync(path.dirname(sharedEntry), runDir, { recursive: true }); + if (options.onboardingBundle !== undefined) { + cpSync(options.onboardingBundle, path.join(runDir, "onboarding"), { recursive: true }); + } + if (options.bundledThemesBundle !== undefined) { + cpSync(options.bundledThemesBundle, path.join(runDir, "themes"), { recursive: true }); + } + return { + cliEntry: path.join(runDir, path.basename(sharedEntry)), + dispose: () => rmSync(runDir, { recursive: true, force: true }), + }; + } catch (error) { + rmSync(runDir, { recursive: true, force: true }); + throw error; } - return { - cliEntry: path.join(runDir, path.basename(sharedEntry)), - dispose: () => rmSync(runDir, { recursive: true, force: true }), - }; } function installProductRuntimeAssets(buildDir: string): void { const webDist = path.join(REPO_ROOT, "packages/web/dist"); const artifactMissingView = path.join(webDist, "views/artifact-missing"); + const urlUnsupportedView = path.join(webDist, "views/url-unsupported"); const markdownView = path.join(REPO_ROOT, "packages/view-markdown/dist"); const canonical = path.join(REPO_ROOT, "packages/canonical/dist/canonical"); const sdk = path.join(REPO_ROOT, "packages/server/dist/sdk"); @@ -111,6 +117,9 @@ function installProductRuntimeAssets(buildDir: string): void { if (!existsSync(path.join(artifactMissingView, "index.html"))) { throw new Error(`Web build produced no artifact-missing view at ${artifactMissingView}`); } + if (!existsSync(path.join(urlUnsupportedView, "index.html"))) { + throw new Error(`Web build produced no url-unsupported view at ${urlUnsupportedView}`); + } if (!existsSync(path.join(markdownView, "index.html"))) { throw new Error(`View build produced no markdown view at ${markdownView}`); } @@ -133,6 +142,7 @@ function installProductRuntimeAssets(buildDir: string): void { cpSync(sdk, path.join(buildDir, "sdk"), { recursive: true }); const staticDir = path.join(buildDir, "web"); mkdirSync(staticDir, { recursive: true }); + cpSync(urlUnsupportedView, path.join(staticDir, "views/url-unsupported"), { recursive: true }); cpSync(webIndex, path.join(staticDir, "index.html")); cpSync(webAssets, path.join(staticDir, "assets"), { recursive: true }); } @@ -157,22 +167,12 @@ export async function disposeAllProductServers(): Promise { export async function launchProductServer(options: LaunchProductServerOptions = {}): Promise { const createdHome = options.home === undefined; const home = options.home ?? mkdtempSync(path.join(os.tmpdir(), "television-product-e2e-")); - // The test-runner recipe for spawned `tv serve`: config port 0, with the - // acquired port read from startup output and passed to client commands. - writeHomeConfig(home, options.auth ? { port: 0 } : { port: 0, auth: false }); const { TELEVISION_ACP_AGENT: _ignoredAgent, ...inheritedEnv } = process.env; const env = { ...inheritedEnv, ...options.env }; - let bundledRuntime = options.onboardingBundle !== undefined || options.bundledThemesBundle !== undefined - ? getCLIEntryWithBundles({ - ...(options.onboardingBundle !== undefined - ? { onboardingBundle: options.onboardingBundle } - : {}), - ...(options.bundledThemesBundle !== undefined - ? { bundledThemesBundle: options.bundledThemesBundle } - : {}), - }) - : undefined; - const cliEntry = (): string => bundledRuntime?.cliEntry ?? getSourceCLIEntry(); + let sharedEntry: string; + let preparation: Promise | undefined; + let bundledRuntime: ReturnType | undefined; + const cliEntry = (): string => bundledRuntime?.cliEntry ?? sharedEntry; const proxies = new Map(); let current: { process: ProductServerProcess; serverURL: string } | null = null; let pendingProcess: ProductServerProcess | null = null; @@ -201,23 +201,32 @@ export async function launchProductServer(options: LaunchProductServerOptions = const dispose: ProductServerDisposer = () => { disposalRequested = true; disposalPromise ??= (async () => { - try { - await Promise.all([...proxies.values()].map((proxy) => proxy.dispose())); - const processes = new Set([current?.process, pendingProcess].filter((process): process is ProductServerProcess => process != null)); - await Promise.all([...processes].map(stopProcess)); - bundledRuntime?.dispose(); - if (options.cleanupHome ?? createdHome) { - rmSync(home, { recursive: true, force: true }); - } - } finally { - activeProductServerDisposers.delete(dispose); + // A disposal requested during preparation must not race removal of its home. + await preparation?.catch(() => undefined); + await Promise.all([...proxies.values()].map((proxy) => proxy.dispose())); + const processes = new Set([current?.process, pendingProcess].filter((process): process is ProductServerProcess => process != null)); + await Promise.all([...processes].map(stopProcess)); + bundledRuntime?.dispose(); + if (options.cleanupHome ?? createdHome) { + rmSync(home, { recursive: true, force: true }); } + // Failed cleanup remains visible to worker teardown, which preserves builds. + activeProductServerDisposers.delete(dispose); })(); return disposalPromise; }; activeProductServerDisposers.add(dispose); try { + // Register cleanup before configuration, shared preparation or copying can fail. + // Port 0 remains the recipe for every real backend, including restarts. + writeHomeConfig(home, options.auth ? { port: 0 } : { port: 0, auth: false }); + preparation = getSourceCLIEntry(); + sharedEntry = await preparation; + if (disposalRequested) throw new Error("Product server was disposed during preparation"); + if (options.onboardingBundle !== undefined || options.bundledThemesBundle !== undefined) { + bundledRuntime = getCLIEntryWithBundles(sharedEntry, options); + } current = await startServe(); pendingProcess = null; if (disposalRequested) { @@ -246,7 +255,7 @@ export async function launchProductServer(options: LaunchProductServerOptions = rmSync(sibling, { recursive: true, force: true }); cpSync(restartOptions.onboardingBundle, sibling, { recursive: true }); } else { - bundledRuntime = getCLIEntryWithBundles({ + bundledRuntime = getCLIEntryWithBundles(sharedEntry, { onboardingBundle: restartOptions.onboardingBundle, ...(options.bundledThemesBundle !== undefined ? { bundledThemesBundle: options.bundledThemesBundle } diff --git a/test/helpers/versioned-web-bundle.ts b/test/helpers/versioned-web-bundle.ts index 0a926926..083038c1 100644 --- a/test/helpers/versioned-web-bundle.ts +++ b/test/helpers/versioned-web-bundle.ts @@ -1,8 +1,7 @@ -import { mkdtempSync } from "node:fs"; import { createRequire } from "node:module"; -import os from "node:os"; import path from "node:path"; import { pathToFileURL, fileURLToPath } from "node:url"; +import { WorkerBuilds } from "./worker-builds.ts"; // Versioned web-bundle builder for the update-notifications harnesses: a real // `vite build` of packages/web stamped at a chosen version via the sanctioned @@ -15,7 +14,8 @@ import { pathToFileURL, fileURLToPath } from "node:url"; const REPO_ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", ".."); const WEB_PACKAGE_DIR = path.join(REPO_ROOT, "packages", "web"); -const distByVersion = new Map>(); +const builds = new WorkerBuilds("television-web-bundle-"); +export const disposeVersionedWebBuilds = (): Promise => builds.dispose(); // Builds are serialized: the version reaches the vite config through // process.env, which concurrent in-process builds would race on. let buildChain: Promise = Promise.resolve(); @@ -36,9 +36,8 @@ async function loadVite(): Promise { return typeof loaded.build === "function" ? loaded : loaded.default!; } -async function runBuild(version: string | undefined): Promise { +async function runBuild(version: string | undefined, outDir: string): Promise { const vite = await loadVite(); - const outDir = mkdtempSync(path.join(os.tmpdir(), "television-web-bundle-")); const previous = process.env.TV_TEST_WEB_VERSION; if (version === undefined) delete process.env.TV_TEST_WEB_VERSION; else process.env.TV_TEST_WEB_VERSION = version; @@ -52,7 +51,6 @@ async function runBuild(version: string | undefined): Promise { if (previous === undefined) delete process.env.TV_TEST_WEB_VERSION; else process.env.TV_TEST_WEB_VERSION = previous; } - return outDir; } /** @@ -61,11 +59,9 @@ async function runBuild(version: string | undefined): Promise { */ export function buildVersionedWebBundle(version?: string): Promise { const key = version ?? ""; - let dist = distByVersion.get(key); - if (dist === undefined) { - dist = buildChain.then(() => runBuild(version), () => runBuild(version)); - buildChain = dist.catch(() => undefined); - distByVersion.set(key, dist); - } - return dist; + return builds.get(key, (directory) => { + const build = buildChain.then(() => runBuild(version, directory)); + buildChain = build.catch(() => undefined); + return build; + }); } diff --git a/test/helpers/vitest.ts b/test/helpers/vitest.ts new file mode 100644 index 00000000..8adfb808 --- /dev/null +++ b/test/helpers/vitest.ts @@ -0,0 +1,16 @@ +import { test as base } from "vitest"; +import { disposeAllProductServers, disposeSourceCLIBuilds } from "./product-server.ts"; +import { disposeVersionedWebBuilds } from "./versioned-web-bundle.ts"; + +/** Test registration for Vitest callers of the reusable product builders. */ +export const test = base.extend<{ workerBuildCleanup: void }>({ + workerBuildCleanup: [async ({}, use) => { + try { + await use(); + } finally { + await disposeAllProductServers(); + await Promise.all([disposeSourceCLIBuilds(), disposeVersionedWebBuilds()]); + } + }, { scope: "worker", auto: true }], +}); +export const it = test; diff --git a/test/helpers/worker-builds.ts b/test/helpers/worker-builds.ts new file mode 100644 index 00000000..d6c7a6e7 --- /dev/null +++ b/test/helpers/worker-builds.ts @@ -0,0 +1,50 @@ +import { mkdtempSync, rmSync } from "node:fs"; +import os from "node:os"; +import path from "node:path"; + +/** Owns reusable preparation for one runner worker. */ +export class WorkerBuilds { + private readonly builds = new Map>(); + private readonly directories = new Set(); + private disposal: Promise | undefined; + private readonly prefix: string; + private readonly parent: string; + + constructor(prefix: string, parent = os.tmpdir()) { + this.prefix = prefix; + this.parent = parent; + } + + get(key: string, prepare: (directory: string) => void | Promise): Promise { + if (this.disposal) return Promise.reject(new Error("Worker build lifetime has ended")); + let build = this.builds.get(key); + if (!build) { + const directory = mkdtempSync(path.join(this.parent, this.prefix)); + this.directories.add(directory); + build = Promise.resolve().then(() => prepare(directory)).then(() => directory, (error: unknown) => { + this.builds.delete(key); + this.remove(directory); + throw error; + }); + this.builds.set(key, build); + } + return build; + } + + dispose(): Promise { + this.disposal ??= (async () => { + await Promise.allSettled(this.builds.values()); + const failures: unknown[] = []; + for (const directory of this.directories) { + try { this.remove(directory); } catch (error) { failures.push(error); } + } + if (failures.length) throw new AggregateError(failures, "Failed to remove worker builds"); + })(); + return this.disposal; + } + + private remove(directory: string): void { + rmSync(directory, { recursive: true, force: true }); + this.directories.delete(directory); + } +} diff --git a/test/node/acp-home-context.test.ts b/test/node/acp-home-context.test.ts index a6d18c89..5f7319fb 100644 --- a/test/node/acp-home-context.test.ts +++ b/test/node/acp-home-context.test.ts @@ -2,7 +2,8 @@ import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "nod import os from "node:os"; import path from "node:path"; import { fileURLToPath } from "node:url"; -import { describe, expect, it } from "vitest"; +import { describe, expect } from "vitest"; +import { it } from "../helpers/vitest.ts"; import { WebSocket } from "ws"; import { ACPClient } from "../../packages/web/src/services/acp-client.ts"; import { launchProductServer } from "../helpers/product-server.ts"; diff --git a/test/node/product-server-proxy.e2e.test.ts b/test/node/product-server-proxy.e2e.test.ts index 7370f107..50583715 100644 --- a/test/node/product-server-proxy.e2e.test.ts +++ b/test/node/product-server-proxy.e2e.test.ts @@ -1,6 +1,7 @@ import { createServer, type IncomingMessage, type ServerResponse } from "node:http"; import net, { type Socket } from "node:net"; -import { afterEach, describe, expect, test } from "vitest"; +import { afterEach, describe, expect } from "vitest"; +import { test } from "../helpers/vitest.ts"; import { disposeAllProductServers, launchProductServer, startDevelopmentProxy, type DevelopmentProxy } from "../helpers/product-server.ts"; import { ownedProcessCount } from "../helpers/owned-process.ts"; @@ -14,11 +15,13 @@ afterEach(async () => { describe("product server lifecycle", () => { test("disposes a child process while product-server startup is still in flight", async () => { - const startup = launchProductServer(); + const startup = launchProductServer({ env: { VITEST: "" } }); + const rejected = expect(startup).rejects.toThrow("tv serve exited before startup"); + await expect.poll(() => ownedProcessCount(), { interval: 5 }).toBe(1); await disposeAllProductServers(); - await expect(startup).rejects.toThrow("tv serve exited before startup"); + await rejected; expect(ownedProcessCount()).toBe(0); }); }); diff --git a/test/node/update-reload-guard.e2e.test.ts b/test/node/update-reload-guard.e2e.test.ts index 1ff46e11..567a3183 100644 --- a/test/node/update-reload-guard.e2e.test.ts +++ b/test/node/update-reload-guard.e2e.test.ts @@ -1,4 +1,5 @@ -import { describe, it } from "vitest"; +import { describe } from "vitest"; +import { it } from "../helpers/vitest.ts"; import { expect as pwExpect } from "@playwright/test"; import { buildVersionedWebBundle, diff --git a/test/node/update-reload.e2e.test.ts b/test/node/update-reload.e2e.test.ts index 1f359bbe..b8c68916 100644 --- a/test/node/update-reload.e2e.test.ts +++ b/test/node/update-reload.e2e.test.ts @@ -1,4 +1,5 @@ -import { describe, it } from "vitest"; +import { describe } from "vitest"; +import { it } from "../helpers/vitest.ts"; import { expect as pwExpect } from "@playwright/test"; import { buildVersionedWebBundle, diff --git a/test/node/worker-build-lifetime.e2e.test.ts b/test/node/worker-build-lifetime.e2e.test.ts new file mode 100644 index 00000000..21020fd7 --- /dev/null +++ b/test/node/worker-build-lifetime.e2e.test.ts @@ -0,0 +1,72 @@ +import { mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs"; +import { createRequire } from "node:module"; +import os from "node:os"; +import path from "node:path"; +import { expect, test } from "vitest"; +import { spawnOwnedProcess } from "../helpers/owned-process.ts"; + +const require = createRequire(import.meta.url); +const root = path.resolve(import.meta.dirname, "../.."); +type Observation = { caseNumber: number; pid: number; source: string; webBundle: string }; + +// proofs/arch/testing-policy.md#^testing-t-build-worker-seam +test.each(["playwright", "vitest"] as const)("%s workers reuse builds and remove them at teardown", async (runner) => { + let previous: Observation | undefined; + for (let worker = 0; worker < 2; worker += 1) { + const temporaryRoot = mkdtempSync(path.join(os.tmpdir(), "television-worker-build-seam-")); + const record = path.join(temporaryRoot, "observations.jsonl"); + // Nested fixtures belong to this acceptance, not independently runnable surfaces. + // Their native configs are temporary, as in the runner-reporting acceptance. + const config = path.join(temporaryRoot, `${runner}.config.mjs`); + writeFileSync(config, `export default ${JSON.stringify(runner === "playwright" ? { + testDir: path.join(root, "test/fixtures/worker-builds"), + testMatch: "playwright.fixture.ts", workers: 1, fullyParallel: false, + retries: 0, timeout: 90_000, reporter: "line", outputDir: path.join(temporaryRoot, "results"), + } : { test: { + include: ["test/fixtures/worker-builds/vitest.fixture.ts"], maxWorkers: 1, + fileParallelism: false, retry: 0, testTimeout: 90_000, + } })};\n`); + const cli = runner === "playwright" + ? require.resolve("@playwright/test/cli") + : path.join(path.dirname(require.resolve("vitest/package.json")), "vitest.mjs"); + const owned = spawnOwnedProcess(process.execPath, [ + cli, runner === "playwright" ? "test" : "run", + "--config", config, + ], { + cwd: root, + env: { ...process.env, TMPDIR: temporaryRoot, TV_WORKER_BUILD_RECORD: record }, + stdio: ["ignore", "pipe", "pipe"], + }); + let output = ""; + owned.child.stdout!.on("data", (chunk: Buffer) => { output += chunk.toString(); }); + owned.child.stderr!.on("data", (chunk: Buffer) => { output += chunk.toString(); }); + try { + const exitCode = await new Promise((resolve, reject) => { + owned.child.once("error", reject); + owned.child.once("close", resolve); + }); + expect(exitCode, output).toBe(0); + const observations = readFileSync(record, "utf8").trim().split("\n").map((line) => JSON.parse(line) as Observation); + expect(observations).toHaveLength(2); + const [first, second] = observations as [Observation, Observation]; + expect(first.caseNumber).toBe(1); + expect(second).toEqual({ ...first, caseNumber: 2 }); + if (previous) { + expect(first.source).not.toBe(previous.source); + expect(first.webBundle).not.toBe(previous.webBundle); + } + previous = first; + + const remaining = readdirSync(temporaryRoot).filter((name) => + name.startsWith("television-source-cli-e2e-") || + name.startsWith("television-web-bundle-") || + name.startsWith("television-product-e2e-") + ); + expect(remaining, `${runner} left worker builds or failed-launch copies after teardown`).toEqual([]); + } finally { + const cleanup = await owned.dispose(); + expect(cleanup.outcome, output).not.toBe("survived"); + rmSync(temporaryRoot, { recursive: true, force: true }); + } + } +}); diff --git a/test/repo/blaxel-lease-contracts.test.ts b/test/repo/blaxel-lease-contracts.test.ts new file mode 100644 index 00000000..e305eb82 --- /dev/null +++ b/test/repo/blaxel-lease-contracts.test.ts @@ -0,0 +1,192 @@ +import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, rmSync, existsSync, symlinkSync, lstatSync } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, expect, test as platformTest, vi } from "vitest"; +import { workerLeaseOperation, cleanWorkerTemporaryDirectories } from "../../scripts/test/blaxel-worker-lease.mjs"; +import { finishLease, renewLease, retainAttemptEvidence } from "../../scripts/test/blaxel-lease-lifecycle.mjs"; +import { publishWorker, deleteWorker } from "../../scripts/test/blaxel-pool-lifecycle.mjs"; + +// Contract clocks, owner audits/signals and SDK operations are substitutions. +// Filesystem assertions cross the real cleaner; the live driver proves transport. +const test = platformTest.skipIf(process.platform !== "linux"); +const roots: string[] = []; +afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); }); +function fixture(record?: unknown) { + const tmpRoot = mkdtempSync(path.join(os.tmpdir(), "lease-contract-")); roots.push(tmpRoot); + const ownerFile = path.join(tmpRoot, "tv-testshard.lock/owner.json"); + if (record !== undefined) { mkdirSync(path.dirname(ownerFile)); writeFileSync(ownerFile, typeof record === "string" ? record : JSON.stringify(record)); } + const input = { lockId: "incoming", runId: "incoming-run", shardIndex: 1, attemptId: "incoming-attempt", tmpRoot, scratch: path.join(tmpRoot, "tv-testshards/current"), resultsRoot: path.join(tmpRoot, "results") }; + return { tmpRoot, input, ownerFile }; +} +const expired = { lockId: "old", runId: "old-run", attemptId: "old-attempt", lastRenewal: 1, recoverAfter: 2, expiresAt: 0 }; + +test("expired active and quiet leases use the same cleanup; uncertain owners hold", async () => { + for (const active of [false, true]) { + const f = fixture(expired); const calls: string[] = []; + let owners = active ? [{ shardToken: "old" }] : []; + const result = await workerLeaseOperation(f.input, { now: () => 3000, scanOwners: async () => owners, stopOwners: async (token: string) => { calls.push(token); owners = []; return []; } }); + expect(result.ok).toBe(true); expect(calls).toEqual(["old"]); + expect(result.record.recovery?.discardedReports).toBe(true); + } + const f = fixture(expired); + const result = await workerLeaseOperation(f.input, { now: () => 3000, stopOwners: async () => [], scanOwners: async () => [{ shardToken: "unrelated" }] }); + expect(result).toMatchObject({ ok: false }); + expect(JSON.parse(readFileSync(f.ownerFile, "utf8")).lockId).toBe("old"); +}); + +test("legacy, incomplete, unreadable and audit leases fail closed without clean ownership", async () => { + for (const record of [{ lockId: "legacy", expiresAt: 1 }, {}, "unreadable"]) { + for (const operation of ["acquire", "delete"]) { + const f = fixture(record); + expect(await workerLeaseOperation({ ...f.input, operation }, { now: () => 3000, scanOwners: async () => [{}] })).toMatchObject({ ok: false }); + expect((await workerLeaseOperation({ ...f.input, operation }, { now: () => 3000, scanOwners: async () => [] })).ok).toBe(true); + } + } + for (const record of [expired, { lockId: "legacy", expiresAt: 100 }]) { + const f = fixture(record); + expect(await workerLeaseOperation({ ...f.input, audit: true }, { now: () => 3000, scanOwners: async () => { throw Error("must not inspect held worker"); } })).toMatchObject({ ok: false }); + } + const f = fixture({ lockId: "legacy", expiresAt: 100 }); + expect(await workerLeaseOperation(f.input, { now: () => 3000, scanOwners: async () => [] })).toMatchObject({ ok: false }); + const audit = fixture(); + const residue = path.join(audit.tmpRoot, "television-web-bundle-preserve"); mkdirSync(residue); + const staleScratch = path.join(audit.tmpRoot, "tv-testshards/stale"); mkdirSync(staleScratch, { recursive: true }); + const auditInput = { ...audit.input, audit: true }; + const auditHooks = { scanOwners: async () => { throw Error("audit must observe existing owners before cleanup"); }, stopOwners: async () => [] }; + expect((await workerLeaseOperation(auditInput, auditHooks)).ok).toBe(true); + expect((await workerLeaseOperation({ ...auditInput, operation: "prepare" }, auditHooks)).ok).toBe(true); + expect((await workerLeaseOperation({ ...auditInput, operation: "teardown" }, auditHooks)).ok).toBe(true); + expect(existsSync(residue)).toBe(true); expect(existsSync(staleScratch)).toBe(true); +}); + +test("cleanup failures keep the old identity, while explicit deletion reserves an expired worker", async () => { + const f = fixture(expired); + await expect(workerLeaseOperation(f.input, { now: () => 3000, scanOwners: async () => [], stopOwners: async () => { throw Error("cannot signal"); } })).rejects.toThrow("cannot signal"); + expect(JSON.parse(readFileSync(f.ownerFile, "utf8")).lockId).toBe("old"); + expect(await workerLeaseOperation({ ...f.input, operation: "delete" }, { now: () => 3000, scanOwners: async () => { throw Error("unavailable"); } })).toMatchObject({ ok: true, deleting: true }); + expect(await workerLeaseOperation(f.input)).toMatchObject({ ok: false }); +}); + +test("only known real directories are removed, with allocated-byte evidence and outside links preserved", async () => { + const f = fixture(); const outside = path.join(f.tmpRoot, "unrelated"); mkdirSync(outside); writeFileSync(path.join(outside, "keep"), "keep"); + for (const name of ["television-source-cli-e2e-old", "television-web-bundle-old", "tv-testshards/stale", "tv-testshards/no-metadata", "tv-testshards/current"]) { mkdirSync(path.join(f.tmpRoot, name), { recursive: true }); writeFileSync(path.join(f.tmpRoot, name, "data"), "payload"); } + symlinkSync(outside, path.join(f.tmpRoot, "television-web-bundle-link")); + symlinkSync(outside, path.join(f.tmpRoot, "television-source-cli-e2e-old/link")); + const result = await cleanWorkerTemporaryDirectories({ tmpRoot: f.tmpRoot, current: f.input.scratch }); + expect(result.ok).toBe(true); + expect(result.removed["stale-scratch"].directories).toBe(2); + expect(result.removed["television-source-cli-e2e-"].allocatedBytes).toBeGreaterThanOrEqual(0); + expect(result.before.availableBytes).toBeGreaterThan(0); expect(result.after.availableBytes).toBeGreaterThan(0); + expect(existsSync(path.join(outside, "keep"))).toBe(true); expect(existsSync(f.input.scratch)).toBe(true); + expect(lstatSync(path.join(f.tmpRoot, "television-web-bundle-link")).isSymbolicLink()).toBe(true); + await cleanWorkerTemporaryDirectories({ tmpRoot: f.tmpRoot, current: f.input.scratch, onlyCurrent: true }); + expect(existsSync(f.input.scratch)).toBe(false); + const linked = path.join(f.tmpRoot, "link"); symlinkSync(outside, linked); + await expect(cleanWorkerTemporaryDirectories({ tmpRoot: linked, current: null })).rejects.toThrow("unsafe cleanup root"); +}); + +test("mount boundaries and incomplete removal remain reported failures without reclaimed counts", async () => { + for (const failure of ["mount", "identity", "remove"]) { + const f = fixture(); const dir = path.join(f.tmpRoot, "television-web-bundle-old"); mkdirSync(dir); + let directoryReads = 0; + const probes = failure !== "remove" ? { lstat: (file: string) => { + const stat = lstatSync(file); + if (file === dir) { + directoryReads++; + if (failure === "mount") Object.defineProperty(stat, "dev", { value: stat.dev + 1 }); + if (failure === "identity" && directoryReads === 3) Object.defineProperty(stat, "ino", { value: stat.ino + 1 }); + } + return stat; + } } : { remove: () => { throw Object.assign(Error("incomplete"), { code: "EIO" }); } }; + const result = await cleanWorkerTemporaryDirectories({ tmpRoot: f.tmpRoot, current: null }, probes); + expect(result.ok).toBe(false); expect(result.removed).toEqual({}); expect(result.errors).toHaveLength(1); expect(existsSync(dir)).toBe(true); + } +}); + +test("retrieval retries after teardown without rerunning work; uncertainty never releases", async () => { + const calls: string[] = []; let attempts = 0; + const operations = { settleWrites: async () => { calls.push("writes"); }, assertOwned: () => {}, teardown: async () => { calls.push("teardown"); }, download: async () => { calls.push("download"); if (++attempts === 1) throw Error("transport"); }, release: async () => { calls.push("release"); } }; + await finishLease(operations); + expect(calls).toEqual(["writes", "teardown", "download", "download", "release"]); + for (const broken of ["teardown", "download", "ownership"] as const) { + calls.length = 0; + await expect(finishLease({ ...operations, ...(broken === "teardown" ? { teardown: async () => { throw Error("unknown dispatch"); } } : broken === "download" ? { download: async () => { throw Error("transport"); } } : { assertOwned: () => { throw Error("lost lease"); } }) })).rejects.toThrow(); + expect(calls).not.toContain("release"); + } +}); + +test("pool eligibility follows verification, and deletion reserves then unpublishes", async () => { + const calls: string[] = []; + const sandbox = { metadata: { name: "fixture", labels: { purpose: "testshards-provisioning", pool: "private" } }, wait: async () => { calls.push("wait"); } }; + const update = async (_name: string, metadata: { labels: Record }) => { calls.push(metadata.labels.purpose); }; + await publishWorker(sandbox, { provision: async () => { calls.push("provision"); }, verify: async () => { calls.push("verify"); }, update }); + expect(calls).toEqual(["wait", "provision", "verify", "testshards"]); + calls.length = 0; + await expect(publishWorker(sandbox, { provision: async () => {}, verify: async () => { throw Error("wrong architecture"); }, update })).rejects.toThrow(); + expect(calls).not.toContain("testshards"); + sandbox.metadata.labels.purpose = "testshards"; + calls.length = 0; + const ops = { reserve: async () => { calls.push("reserve"); }, update, remove: async () => { calls.push("delete"); throw Error("provider unavailable"); } }; + await expect(deleteWorker(sandbox, ops)).rejects.toThrow(); + expect(calls).toEqual(["reserve", "testshards-unavailable", "delete"]); + calls.length = 0; + await expect(deleteWorker(sandbox, { ...ops, reserve: async () => { throw Error("valid lease"); } })).rejects.toThrow(); + expect(calls).toEqual([]); +}); + + +test("renewal continues while retrieval waits and stops on ownership loss", async () => { + vi.useFakeTimers(); + try { + let lost: unknown = null; + const renew = vi.fn().mockResolvedValue(undefined); + const stop = renewLease({ renew, intervalMs: 10, onLost: (error: unknown) => { lost = error; } }); + let continueDownload!: () => void; + const waiting = new Promise(resolve => { continueDownload = resolve; }); + const release = vi.fn(); + const finishing = finishLease({ settleWrites: async () => {}, teardown: async () => {}, download: () => waiting, release, assertOwned: () => { if (lost) throw lost; } }); + await vi.advanceTimersByTimeAsync(35); + expect(renew).toHaveBeenCalledTimes(3); expect(release).not.toHaveBeenCalled(); + continueDownload(); await finishing; await stop(); + await vi.advanceTimersByTimeAsync(30); + expect(renew).toHaveBeenCalledTimes(3); expect(release).toHaveBeenCalledOnce(); + renew.mockRejectedValueOnce(Error("ownership lost")); + const stopLost = renewLease({ renew, intervalMs: 10, onLost: (error: unknown) => { lost = error; } }); + await vi.advanceTimersByTimeAsync(100); + expect(renew).toHaveBeenCalledTimes(4); expect(lost).toBeInstanceOf(Error); + await stopLost(); + } finally { vi.useRealTimers(); } +}); + +test("failed deletion of never-published workers remains retryable without their runtime", async () => { + const sandbox = { metadata: { name: "partial", labels: { purpose: "testshards-provisioning" } } }; + const ops = { reserve: vi.fn().mockRejectedValue(Error("no runtime")), update: vi.fn(), remove: vi.fn().mockRejectedValueOnce(Error("provider unavailable")).mockResolvedValueOnce(undefined) }; + await expect(deleteWorker(sandbox, ops)).rejects.toThrow("provider unavailable"); + await deleteWorker(sandbox, ops); + expect(ops.reserve).not.toHaveBeenCalled(); expect(ops.update).not.toHaveBeenCalled(); expect(ops.remove).toHaveBeenCalledTimes(2); +}); + +test("preparation evidence is downloaded before an undispatched attempt releases", async () => { + const f = fixture(); + const hooks = { scanOwners: async () => [], stopOwners: async () => [] }; + await workerLeaseOperation(f.input, hooks); + await workerLeaseOperation({ ...f.input, operation: "prepare" }, hooks); + writeFileSync(path.join(f.input.resultsRoot, "preparation.txt"), "produced before dispatch"); + const directory = path.join(f.tmpRoot, "download"); + const lease = { preparationRequested: true, executionRequested: false, teardownConfirmed: true, + shardIndex: 1, input: f.input, sandbox: { fs: { + read: async (file: string) => readFileSync(file.replace("/workspace/television/.testshards/results", f.input.resultsRoot), "utf8"), + ls: async () => { throw Error("confirmed teardown must supply its inventory"); }, + } }, + }; + await finishLease({ + settleWrites: async () => {}, assertOwned: () => {}, + teardown: async () => { expect((await workerLeaseOperation({ ...f.input, operation: "teardown" }, hooks)).ok).toBe(true); }, + download: () => retainAttemptEvidence(lease, directory), + release: async () => { + expect(existsSync(path.join(directory, "artifacts/shard-1/preparation.txt"))).toBe(true); + expect(readFileSync(path.join(directory, "artifacts/shard-1/preparation.txt"), "utf8")).toBe("produced before dispatch"); + expect((await workerLeaseOperation({ ...f.input, operation: "release", retrieved: true }, hooks)).ok).toBe(true); + }, + }); +}); diff --git a/test/repo/blaxel-pool-discovery.test.ts b/test/repo/blaxel-pool-discovery.test.ts index 7d546e46..591cc3ee 100644 --- a/test/repo/blaxel-pool-discovery.test.ts +++ b/test/repo/blaxel-pool-discovery.test.ts @@ -40,7 +40,7 @@ async function withApi(run: (discover: (deployedOnly: boolean) => Promise Promise Promise { await withApi(async (discover, requests) => { - expect(await discover(deployedOnly)).toEqual(deployedOnly ? ["a-worker", "b-worker"] : ["a-worker", "b-worker", "c-failed"]); + expect(await discover(deployedOnly)).toEqual(deployedOnly ? ["a-worker", "b-worker"] : ["a-worker", "b-worker", "c-failed", "d-provisioning", "e-unavailable"]); expect(requests).toEqual(["/sandboxes", "/sandboxes?cursor=second", "/sandboxes?cursor=third"]); }); }); diff --git a/test/repo/blaxel-stale-owner-audit.test.ts b/test/repo/blaxel-stale-owner-audit.test.ts index 5bddc464..8abb11f9 100644 --- a/test/repo/blaxel-stale-owner-audit.test.ts +++ b/test/repo/blaxel-stale-owner-audit.test.ts @@ -82,7 +82,9 @@ describe("Blaxel stale-owner audit", () => { expect(source.indexOf(later), `${later} must follow the audit branch`).toBeGreaterThan(auditBranch); } expect(source).toContain("acquireAny(sandboxes, { shardIndex: 1, slot: 0, requestedWorkers: 1, verifyArchitecture: false })"); - expect(source).toContain("const setup = await runSystemDependencySetup(lease)"); + const auditScript = source.slice(source.indexOf("const remoteScript = `${lease.input.scratch}/audit.sh`"), source.indexOf("const rawLogs =", source.indexOf("async function runStaleOwnerAudit"))); + expect(auditScript.indexOf("systemDependenciesCommand()")).toBeLessThan(auditScript.indexOf("[command, ...args]")); + expect(auditScript).toContain('workerEntryCommand({ ...lease.input, processName }, remoteScript)'); expect(source).toContain("run_step system-deps bash -lc ${shellQuote(systemDependenciesCommand())}"); expect(source).toContain("apt-get install -y --no-install-recommends python3 xauth libgtk-3-0 libxtst6 iproute2"); expect(source).not.toContain("psmisc"); diff --git a/test/repo/blaxel-worker-lease.test.ts b/test/repo/blaxel-worker-lease.test.ts new file mode 100644 index 00000000..b2c93a05 --- /dev/null +++ b/test/repo/blaxel-worker-lease.test.ts @@ -0,0 +1,194 @@ +import { spawn, spawnSync, execFile } from "node:child_process"; +import { chownSync, mkdtempSync, mkdirSync, readFileSync, rmSync, writeFileSync, existsSync } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { promisify } from "node:util"; +import { afterEach, expect, test } from "vitest"; +import { workerLeaseCommand, workerEntryCommand, shardScratchPath } from "../../scripts/test/blaxel-worker-lease.mjs"; + +// Real generated lock command, Linux flock/procfs and disposable files. The +// supplied process IDs substitute only the whole-worker audit's enumeration. +const roots: string[] = []; +afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); }); +function fixture(unprivileged = false) { + // A root Blaxel worker must reproduce the unprivileged CI runner too. Use + // /tmp directly because the outer shard's private TMPDIR is not traversable. + const credentials = unprivileged && process.getuid?.() === 0 ? { uid: 65534, gid: 65534 } : {}; + const tmpRoot = mkdtempSync(path.join(unprivileged ? "/tmp" : os.tmpdir(), "lease-seam-")); roots.push(tmpRoot); + if (credentials.uid !== undefined) chownSync(tmpRoot, credentials.uid, credentials.gid!); + const input = { lockId: "fixture-lock", runId: "fixture-run", shardIndex: 1, shardTotal: 1, attemptId: "fixture-attempt", owner: "fixture", expiresAt: Math.floor(Date.now() / 1000) + 1200, tmpRoot, resultsRoot: path.join(tmpRoot, "results"), auditPids: [] as number[] }; + const recordPath = path.join(tmpRoot, "tv-testshard.lock", "owner.json"); + const run = (extra = {}) => spawnSync("bash", ["-c", workerLeaseCommand({ ...input, ...extra })], { encoding: "utf8", env: process.env, timeout: 15_000, ...credentials }); + return { input, recordPath, run, record: () => JSON.parse(readFileSync(recordPath, "utf8")) }; +} + +test.skipIf(process.platform !== "linux")("acquisition writes a renewable deadline and protects the lease from timestamp-only runners", () => { + const f = fixture(); + const before = Date.now(); + const acquired = f.run(); + expect(acquired.status, acquired.stderr).toBe(0); + expect(f.record()).toMatchObject({ lockId: f.input.lockId, expiresAt: 0, attemptId: f.input.attemptId }); + expect(f.record().lastRenewal).toBeGreaterThanOrEqual(before); + expect(f.record().recoverAfter - f.record().lastRenewal).toBe(1_200_000); +}); + +test.skipIf(process.platform !== "linux")("fixture audit runs without root access to unrelated process environments", async () => { + const f = fixture(true); + const ready = path.join(f.input.tmpRoot, "unrelated-ready"); + // A same-user service can be nondumpable too. It belongs outside this + // fixture's process set even though a user filter would include it. + const child = process.getuid?.() === 0 ? spawn(process.execPath, ["-e", `process.setgid(65534);process.setuid(65534);require('fs').writeFileSync(${JSON.stringify(ready)},'ready');setInterval(()=>{},1000)`], { stdio: "ignore", env: process.env }) : null; + try { + if (child) await until(() => existsSync(ready)); + const acquired = f.run(); + expect(acquired.status, acquired.stderr).toBe(0); + expect(f.record().lockId).toBe(f.input.lockId); + } finally { if (child) { child.kill("SIGKILL"); await until(() => child.signalCode !== null); } } +}); + +test.skipIf(process.platform !== "linux")("an unreadable whole-worker audit preserves the lease and its evidence", () => { + const f = fixture(true); + expect(f.run().status).toBe(0); + const before = readFileSync(f.recordPath, "utf8"); + const sentinel = path.join(f.input.tmpRoot, "television-web-bundle-preserve"); + mkdirSync(sentinel); + // No fixture scope: production must refuse reuse when ownership is unknown. + const prepared = f.run({ operation: "prepare", auditPids: undefined }); + expect(prepared.status).toBe(2); + expect(prepared.stderr).toMatch(/worker owner audit unreadable at PID \d+ \(EACCES\); worker reuse refused/); + expect(readFileSync(f.recordPath, "utf8")).toBe(before); + expect(existsSync(sentinel)).toBe(true); +}); + +test.skipIf(process.platform !== "linux" || process.getuid?.() !== 0)("fixture scope still refuses an unreadable process in its process set", async () => { + const f = fixture(true); + const ready = path.join(f.input.tmpRoot, "unreadable-ready"); + // Changing credentials after exec disables dumpability. procfs files become + // root-owned even though /proc/PID/status still identifies UID 65534. + const source = `process.setgid(65534);process.setuid(65534);require('fs').writeFileSync(${JSON.stringify(ready)},'ready');setInterval(()=>{},1000)`; + const child = spawn(process.execPath, ["-e", source], { stdio: "ignore", env: { ...process.env, TV_TEST_SHARD_OWNER: "fixture-lock" } }); + try { + await until(() => existsSync(ready)); + const acquired = f.run({ auditPids: [child.pid!] }); + expect(acquired.status).toBe(2); + expect(acquired.stderr).toContain(`worker owner audit unreadable at PID ${child.pid} (EACCES); worker reuse refused`); + expect(existsSync(f.recordPath)).toBe(false); + } finally { child.kill("SIGKILL"); await until(() => child.signalCode !== null); } +}); + +test.skipIf(process.platform !== "linux")("a quiet expired renewable lease recovers through cleanup before replacement", () => { + const f = fixture(); + mkdirSync(path.dirname(f.recordPath)); + writeFileSync(f.recordPath, JSON.stringify({ lockId: "abandoned", runId: "old-run", shardIndex: 1, attemptId: "old-attempt", expiresAt: 0, lastRenewal: 1, recoverAfter: 2, dispatch: null })); + const acquired = f.run(); + expect(acquired.status, acquired.stderr).toBe(0); + expect(f.record().lockId).toBe(f.input.lockId); + expect(f.record().recovery).toMatchObject({ previous: { lockId: "abandoned", runId: "old-run" }, cleanup: "confirmed", discardedReports: true }); +}); + +test.skipIf(process.platform !== "linux")("renewal and release reject old identities, expiry, and missing retrieval", () => { + const f = fixture(); + expect(f.run({ ttlSeconds: 60 }).status).toBe(0); + expect(f.run({ operation: "renew", lockId: "other" }).status).toBe(42); + expect(f.run({ operation: "release", retrieved: true }).status).toBe(42); + expect(f.run({ operation: "teardown" }).status).toBe(0); + expect(f.run({ operation: "release" }).status).toBe(42); + expect(f.run({ operation: "renew", ttlSeconds: 90 }).status).toBe(0); + expect(f.record().recoverAfter - f.record().lastRenewal).toBe(90_000); + writeFileSync(f.recordPath, JSON.stringify({ ...f.record(), recoverAfter: 1 })); + expect(f.run({ operation: "renew" }).status).toBe(42); + expect(f.run({ operation: "release", retrieved: true }).status).toBe(42); +}); + +const exec = promisify(execFile); +async function until(predicate: () => boolean, timeout = 5000) { + const deadline = Date.now() + timeout; + while (!predicate()) { if (Date.now() > deadline) throw Error("fixture rendezvous timeout"); await new Promise(resolve => setTimeout(resolve, 25)); } +} +function shell(value: string) { return `'${value.replaceAll("'", `'"'"'`)}'`; } + +test.skipIf(process.platform !== "linux")("competing acquisitions serialize, and killing a guard holder releases the OS guard", async () => { + const f = fixture(); + const command = (lockId: string) => exec("bash", ["-c", workerLeaseCommand({ ...f.input, lockId })], { env: process.env, timeout: 10_000 }); + const outcomes = await Promise.allSettled([command("fixture-lock"), command("new-lock")]); + expect(outcomes.filter(result => result.status === "fulfilled")).toHaveLength(1); + const owner = f.record().lockId; + rmSync(path.dirname(f.recordPath), { recursive: true }); + const ready = path.join(f.input.tmpRoot, "guard-ready"); + const child = spawn("flock", ["-F", path.join(f.input.tmpRoot, "tv-testshard.guard"), process.execPath, "-e", `require('fs').writeFileSync(${JSON.stringify(ready)}, String(process.pid));setInterval(()=>{},1000)`], { detached: true, stdio: "ignore", env: process.env }); + try { + await until(() => existsSync(ready)); + expect(Number(readFileSync(ready, "utf8"))).toBe(child.pid); + child.kill("SIGKILL"); + await command(owner); + expect(f.record().lockId).toBe(owner); + } finally { try { process.kill(-child.pid!, "SIGKILL"); } catch {} } +}); + +test.skipIf(process.platform !== "linux")("an obsolete delayed entry cannot touch checkout after recovery", async () => { + const f = fixture(); const processName = "delayed"; + expect(f.run().status).toBe(0); expect(f.run({ operation: "prepare" }).status).toBe(0); + const script = path.join(shardScratchPath(f.input), "worker.sh"); + const sentinel = path.join(f.input.tmpRoot, "checkout-touched"); + writeFileSync(script, `touch ${shell(sentinel)}\n`); + expect(f.run({ operation: "dispatch", processName }).status).toBe(0); + const entry = workerEntryCommand({ ...f.input, processName }, script); + expect(f.run({ operation: "teardown" }).status).toBe(42); + expect(f.run({ operation: "release", retrieved: true }).status).toBe(42); + // The real entry command is delayed until after the old identity is replaced. + writeFileSync(f.recordPath, JSON.stringify({ ...f.record(), recoverAfter: 1 })); + expect(f.run({ lockId: "new-lock", attemptId: "new-attempt" }).status).toBe(0); + const delayed = spawnSync("bash", ["-c", entry], { encoding: "utf8", timeout: 10_000 }); + expect(delayed.status).not.toBe(0); expect(existsSync(sentinel)).toBe(false); + expect(f.record().lockId).toBe("new-lock"); +}); + +test.skipIf(process.platform !== "linux")("recovery kills registered descendants by token, including TERM-resistant children, and holds for another owner", async () => { + const f = fixture(); const processName = "live-fixture"; + expect(f.run().status).toBe(0); expect(f.run({ operation: "prepare" }).status).toBe(0); + const script = path.join(shardScratchPath(f.input), "worker.sh"); + const ready = path.join(f.input.tmpRoot, "child-ready.json"); + const childSource = `process.on('SIGTERM',()=>{});require('fs').writeFileSync(${JSON.stringify(ready)},JSON.stringify({pid:process.pid,outer:process.env.TV_TEST_SHARD_OWNER,surface:process.env.TV_TEST_SURFACE_OWNER}));setInterval(()=>{},1000)`; + // A real nested SurfaceSupervisor replaces the surface token but preserves the + // outer attempt, even with a restricted requested child environment. + const supervisor = path.join(process.cwd(), "scripts/test/surface-supervisor.mjs"); + const parentSource = `import {SurfaceSupervisor} from ${JSON.stringify(supervisor)};const s=new SurfaceSupervisor({runId:'nested-fixture',surfaceIds:['nested']});await s.run(process.execPath,['-e',${JSON.stringify(childSource)}],{env:{PATH:process.env.PATH}});`; + writeFileSync(script, `exec ${shell(process.execPath)} --input-type=module -e ${shell(parentSource)}\n`); + expect(f.run({ operation: "dispatch", processName }).status).toBe(0); + const child = spawn("bash", ["-c", workerEntryCommand({ ...f.input, processName }, script)], { detached: true, stdio: "ignore", env: process.env }); + const unrelated = spawn(process.execPath, ["-e", "setInterval(()=>{},1000)"], { detached: true, stdio: "ignore", env: { ...process.env, TV_TEST_SHARD_OWNER: "unrelated-lock" } }); + try { + await until(() => existsSync(ready)); + const observed = JSON.parse(readFileSync(ready, "utf8")); + f.input.auditPids.push(child.pid!, observed.pid, unrelated.pid!); + expect(observed.outer).toBe(f.input.lockId); expect(observed.surface).toMatch(/^tv1\./); + expect(f.record().dispatch.pid).toBeGreaterThan(0); + writeFileSync(f.recordPath, JSON.stringify({ ...f.record(), recoverAfter: 1 })); + const held = f.run({ lockId: "new-lock", attemptId: "new-attempt" }); + expect(held.status, held.stderr).toBe(42); + expect(() => process.kill(unrelated.pid!, 0)).not.toThrow(); + const stat = existsSync(`/proc/${observed.pid}/stat`) ? readFileSync(`/proc/${observed.pid}/stat`, "utf8") : ""; + expect(stat === "" || stat.slice(stat.lastIndexOf(")") + 2).startsWith("Z")).toBe(true); + unrelated.kill("SIGKILL"); await until(() => unrelated.signalCode !== null); + const acquired = f.run({ lockId: "new-lock", attemptId: "new-attempt" }); + expect(acquired.status, acquired.stderr).toBe(0); + } finally { + try { process.kill(-child.pid!, "SIGKILL"); } catch {} + try { process.kill(-unrelated.pid!, "SIGKILL"); } catch {} + } +}, 20_000); + + +test.skipIf(process.platform !== "linux")("attempt identity and recovery evidence survive checkout resetting the result tree", () => { + const f = fixture(); + mkdirSync(path.dirname(f.recordPath)); + writeFileSync(f.recordPath, JSON.stringify({ lockId: "abandoned", runId: "old-run", attemptId: "old-attempt", expiresAt: 0, lastRenewal: 1, recoverAfter: 2 })); + expect(f.run().status).toBe(0); + expect(f.run({ operation: "prepare" }).status).toBe(0); + rmSync(f.input.resultsRoot, { recursive: true }); // checkout's real reset boundary + expect(f.run({ operation: "teardown" }).status).toBe(0); + expect(JSON.parse(readFileSync(path.join(f.input.resultsRoot, "attempt.json"), "utf8"))).toMatchObject({lockId: f.input.lockId, runId: f.input.runId}); + expect(JSON.parse(readFileSync(path.join(f.input.resultsRoot, "resources/recovery.json"), "utf8"))).toMatchObject({previous: {lockId: "abandoned"}, discardedReports: true}); + expect(JSON.parse(readFileSync(path.join(f.input.resultsRoot, "evidence.json"), "utf8")).files).toContain("temporary-cleanup.json"); + expect(existsSync(shardScratchPath(f.input))).toBe(false); +}); diff --git a/test/repo/build-config-integrity.test.ts b/test/repo/build-config-integrity.test.ts index 7b740667..aa457832 100644 --- a/test/repo/build-config-integrity.test.ts +++ b/test/repo/build-config-integrity.test.ts @@ -15,7 +15,8 @@ import os from "node:os"; import path from "node:path"; // @ts-expect-error — jsdom is intentionally installed without a declaration package import { JSDOM } from "jsdom"; -import { afterAll, beforeAll, describe, expect, test } from "vitest"; +import { afterAll, beforeAll, describe, expect } from "vitest"; +import { test } from "../helpers/vitest.ts"; // @ts-expect-error — repository build script has no declaration file import { buildCanonicalVersions } from "../../packages/canonical/scripts/build-canonical.mjs"; import { buildVersionedWebBundle } from "../helpers/versioned-web-bundle.ts"; diff --git a/test/repo/resource-diagnostics-links.test.ts b/test/repo/resource-diagnostics-links.test.ts new file mode 100644 index 00000000..7938f384 --- /dev/null +++ b/test/repo/resource-diagnostics-links.test.ts @@ -0,0 +1,56 @@ +import { copyFileSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { expect, test } from "vitest"; +import { combineAttemptReports, compactProviderSummary, summarizeInfrastructureAttempts } from "../../scripts/test/provider-outputs.mjs"; +import { normalizeProviderShardSurfaces } from "../../scripts/test/provider-normalization.mjs"; +import { buildReports } from "../../scripts/test/reporting.mjs"; +import { buildTimingEvents, serializeTimingEvents } from "../../scripts/test/timing-events.mjs"; + +const surfaces = [{ id: "e2e:fixture", runner: "playwright", roots: ["probe.test.ts"], cwd: "." }] as any; +const start = "2026-10-27T00:00:00.000Z"; +const end = "2026-10-27T00:00:01.000Z"; + +test("keeps only retained resource links across infrastructure replacement, normalization and final reporting", () => { + const root = mkdtempSync(path.join(os.tmpdir(), "tv-resource-links-")); + try { + const attempts = [0, 1].map((index) => { + const dir = path.join(root, `attempt-${index}`); mkdirSync(dir); + const diagnostic = path.join(dir, "resources.json"); + writeFileSync(diagnostic, JSON.stringify({ identity: { attemptId: index }, start: { memoryAvailable: { value: 12345 } }, kernel: { lines: ["KERNEL-DETAIL-SENTINEL"] } })); + const native = path.join(dir, "native.json"); copyFileSync("test/repo/fixtures/test-runner/playwright-retry.json", native); + return { status: index === 0 ? "failed" : "passed", startedAt: start, completedAt: end, shardIndices: [1], outputDir: dir, shards: [{ shardIndex: 1, status: index === 0 ? "infra-failed" : "passed", resourceDiagnosticsPaths: [diagnostic], outputDir: dir, tasks: [{ surfaceId: "e2e:fixture", runner: "playwright", status: "passed", exitCode: 0, durationMs: 25, nativeResultPath: native, resourceDiagnosticsPaths: [diagnostic, path.join(dir, "missing.json")] }] }] } as any; + }); + const combined = combineAttemptReports(attempts, [1], { shardTotal: 1, outputDir: root }); + expect(combined.status).toBe("passed"); + const history = summarizeInfrastructureAttempts(attempts) as any[]; + expect(history[0].shards[0].resourceDiagnosticsPaths).toEqual([path.join(root, "attempt-0/resources.json")]); + const normalized = normalizeProviderShardSurfaces({ provider: "blaxel", providerSummary: combined, outputDir: root, surfaces, delegatedCommand: ["coordinator"], exitCode: 0, coordinatorLogPath: path.join(root, "coordinator.log") }); + expect(normalized).toHaveLength(1); + expect(normalized[0].counts.testsFlakyRecovered).toBe(1); + expect(normalized[0].failedTests).toEqual([]); + const { results, summary } = buildReports({ runId: "resource-run", provider: "blaxel", runDir: root, selection: { surfaces: ["e2e:fixture"] }, surfaces: normalized, startedAt: start, completedAt: end }); + expect(results.surfaces[0].resourceDiagnosticsPaths).toEqual(["attempt-0/resources.json", "attempt-1/resources.json"]); + for (const link of results.surfaces[0].resourceDiagnosticsPaths as string[]) expect(existsSync(path.join(root, link))).toBe(true); + expect(summary.run.status).toBe("passed"); + const events = serializeTimingEvents(buildTimingEvents({ run: { runId: "resource-run", commandProvider: "blaxel", timingProvider: "blaxel-playwright-x64-4vcpu", status: "passed", startedAt: start, completedAt: end, durationMs: 1000, selection: { surfaces: ["e2e:fixture"] }, git: { commit: null, tree: null, dirty: false }, environment: { ci: true, os: "linux", arch: "x64", node: process.version }, phases: [] }, shards: [{ index: 1, total: 1, status: "passed", phases: [], surfaces: normalized as any }] })); + for (const compact of [JSON.stringify(summary), JSON.stringify(compactProviderSummary(combined)), events]) { + expect(compact).not.toContain("KERNEL-DETAIL-SENTINEL"); + expect(compact).not.toContain("memoryAvailable"); + } + expect(readFileSync(path.join(root, "attempt-0/resources.json"), "utf8")).toContain("KERNEL-DETAIL-SENTINEL"); + } finally { rmSync(root, { recursive: true, force: true }); } +}); + +test("baseline-only provider evidence cannot manufacture a selected-surface outcome", () => { + const root = mkdtempSync(path.join(os.tmpdir(), "tv-resource-missing-")); + try { + const diagnostic = path.join(root, "baseline.json"); writeFileSync(diagnostic, JSON.stringify({ end: { status: "missing" } })); + const normalized = normalizeProviderShardSurfaces({ provider: "blaxel", providerSummary: { status: "failed", shards: [{ shardIndex: 1, status: "infra-failed", resourceDiagnosticsPaths: [diagnostic], tasks: [] }] }, outputDir: root, surfaces, delegatedCommand: ["coordinator"], exitCode: 1, coordinatorLogPath: path.join(root, "coordinator.log") }); + const { summary, results } = buildReports({ runId: "missing-native", provider: "blaxel", runDir: root, selection: { surfaces: ["e2e:fixture"] }, surfaces: normalized, startedAt: start, completedAt: end }); + expect(summary.run).toMatchObject({ status: "incomplete", testStatus: "unknown" }); + expect(summary.counts).toMatchObject({ surfacesRun: 0, testsFailed: 0 }); + expect(results.surfaces.map((surface) => surface.id)).toEqual(["provider:blaxel"]); + expect(results.surfaces[0].resourceDiagnosticsPaths).toEqual(["baseline.json"]); + } finally { rmSync(root, { recursive: true, force: true }); } +}); diff --git a/test/repo/resource-diagnostics.test.ts b/test/repo/resource-diagnostics.test.ts new file mode 100644 index 00000000..43e78a75 --- /dev/null +++ b/test/repo/resource-diagnostics.test.ts @@ -0,0 +1,98 @@ +import { copyFileSync, existsSync, mkdtempSync, readFileSync, readdirSync, rmSync, statSync } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, expect, test } from "vitest"; +import { kernelLinesInInterval, startResourceDiagnostics, writeResourceDiagnostic } from "../../scripts/test/resource-diagnostics.mjs"; + +const roots: string[] = []; +const identity = { runId: "run-one", shardIndex: 1, attemptId: "lock-one", surfaceId: "e2e:fixture" }; +const startTime = "2026-10-27T00:00:00.000Z"; +const endTime = "2026-10-27T00:00:01.000Z"; +const secret = "credential-query-body-environment-argument-SENTINEL"; +const temporary = () => { const dir = mkdtempSync(path.join(os.tmpdir(), "tv-resource-contract-")); roots.push(dir); return dir; }; +const read = (file: string | null) => JSON.parse(readFileSync(file!, "utf8")); +afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); }); + +test("restricts a whole-second kernel read to the observed interval and discloses missing timestamps", () => { + const line = (time: string) => `${time} Out of memory: Killed process 23 (node)`; + const within = line("2026-10-27T00:00:00,500000+00:00"); + expect(kernelLinesInInterval([ + line("2026-10-27T00:00:00,100000+00:00"), within, + line("2026-10-27T00:00:00,900000+00:00"), + "Killed process 24 (node)", "unrelated kernel message", + ], { since: "2026-10-27T00:00:00.200Z", until: "2026-10-27T00:00:00.800Z" })).toEqual({ lines: [within], omittedUnrecognizedLines: 1 }); +}); + +test("retains a baseline, honest units and reset evidence, and kernel evidence for a recovered native attempt", () => { + const root = temporary(); + const nativeResultPath = path.join(root, "native.json"); + copyFileSync("test/repo/fixtures/test-runner/playwright-retry.json", nativeResultPath); + let ended = false; + const probes: Array<[string, unknown]> = []; + const collector = startResourceDiagnostics({ + root, identity: { ...identity, env: secret, args: [secret] }, now: () => new Date(ended ? endTime : startTime), + readProbe: (name, interval) => { + probes.push([name, interval]); + if (name === "temporaryFree") throw Object.assign(new Error(secret), { code: "EACCES" }); + if (name === "oomKills") return ended ? 2 : 10; + if (name === "kernelOOM") return { lines: [`2026-10-27T00:00:00,500000+00:00 Out of memory: Killed process 23 (node) total-vm:4096kB ${secret}`] }; + return 4096; + }, + warn: () => {}, + }); + expect(read(collector.file)).toMatchObject({ identity, start: { observedAt: startTime, memoryAvailable: { value: 4096, unit: "bytes", source: "/proc/meminfo:MemAvailable" }, temporaryFree: { value: null, reason: "EACCES" } }, end: { status: "missing" } }); + ended = true; + expect(collector.finish({ runner: "playwright", nativeResultPath, exitCode: 0 })).toEqual([collector.file]); + const record = read(collector.file); + expect(record).toMatchObject({ end: { observedAt: endTime }, oomDelta: { value: null, reason: "counter-reset" }, nativeEvidence: { status: "complete", failedAttempt: true, recovered: true }, kernel: { status: "available" } }); + expect(probes.find(([name]) => name === "kernelOOM")?.[1]).toMatchObject({ since: startTime, until: endTime }); + expect(record.kernel.lines.join("\n")).toContain("Killed process 23 (node)"); + expect(JSON.stringify(record)).not.toContain(secret); +}); + +test("requests kernel evidence for missing output and preserves unavailable probes without zeroes", () => { + const root = temporary(); + const warnings: string[] = []; + const collector = startResourceDiagnostics({ root, identity, warn: (message) => warnings.push(message), readProbe: () => { throw Object.assign(new Error(secret), { code: "EPERM" }); } }); + collector.finish({ runner: "vitest", nativeResultPath: path.join(root, "absent.json"), exitCode: 17 }); + const record = read(collector.file); + expect(record).toMatchObject({ nativeEvidence: { status: "missing", failedAttempt: null }, start: { oomKills: { value: null, unit: "count", reason: "EPERM" } }, end: { memoryAvailable: { value: null } }, kernel: { status: "unavailable", reason: "EPERM" }, oomDelta: { value: null, reason: "unavailable" } }); + expect(warnings.length).toBeGreaterThan(0); + expect(warnings.join("\n")).not.toContain(secret); +}); + +test("bounds sidecars and the attempt including receipts, reserving compact observations before excerpts", () => { + const root = temporary(); + const warnings: string[] = []; + const native = path.join(root, "native.json"); + copyFileSync("test/repo/fixtures/test-runner/playwright-retry.json", native); + const collector = startResourceDiagnostics({ root, identity, warn: (message) => warnings.push(message), readProbe: (name) => name === "kernelOOM" ? { lines: Array.from({ length: 8000 }, () => "Out of memory: Killed process 23 (node) total-vm:4096kB")} : 1 }); + collector.finish({ runner: "playwright", nativeResultPath: native, exitCode: 0 }); + const record = read(collector.file); + expect(record.start.memoryAvailable.value).toBe(1); + expect(record.end.memoryAvailable.value).toBe(1); + expect(record.kernel.truncated).toBe(true); + expect(Buffer.byteLength(record.kernel.lines.join("\n"))).toBeLessThanOrEqual(64 * 1024); + expect(statSync(collector.file!).size).toBeLessThanOrEqual(128 * 1024); + const pending = startResourceDiagnostics({ root, identity, readProbe: () => 2, warn: (message) => warnings.push(message) }); + const receipt = { kind: "recovery", identity, previous: { runId: "old", lockId: "old-lock", worker: "private-worker", lastRenewal: startTime, recoverAfter: endTime, args: secret }, discardedReports: true, cleanup: "confirmed", processes: Array.from({ length: 3000 }, (_, i) => ({ pid: i + 1, startTime: "123", command: "node", outcome: "terminated", args: secret })), env: secret }; + for (let i = 0; i < 80; i++) writeResourceDiagnostic(root, `receipt-${i}`, receipt, { warn: (message) => warnings.push(message) }); + pending.finish({ runner: "vitest", nativeResultPath: null, exitCode: 17 }); + expect(read(pending.file).end.memoryAvailable.value).toBe(2); + const files = readdirSync(path.join(root, "resources")).map((name) => path.join(root, "resources", name)); + expect(files.reduce((sum, file) => sum + statSync(file).size, 0)).toBeLessThanOrEqual(8 * 1024 * 1024); + expect(files.every((file) => statSync(file).size <= 128 * 1024)).toBe(true); + expect(files.every((file) => !readFileSync(file, "utf8").includes(secret))).toBe(true); + expect(read(collector.file).end.memoryAvailable.value).toBe(1); + expect(warnings.length).toBeGreaterThan(0); +}); + +test("diagnostic write failure warns and returns no unretained link without throwing over a native outcome", () => { + const root = temporary(); + const warnings: string[] = []; + const collector = startResourceDiagnostics({ root, identity, readProbe: () => 1, write: () => { throw Object.assign(new Error(secret), { code: "ENOSPC" }); }, warn: (message) => warnings.push(message) }); + expect(collector.finish({ runner: "vitest", nativeResultPath: null, exitCode: 17 })).toEqual([]); + expect(collector.file === null || !existsSync(collector.file)).toBe(true); + expect(warnings.join("\n")).toContain("ENOSPC"); + expect(warnings.join("\n")).not.toContain(secret); +}); diff --git a/test/repo/test-runner-config.test.ts b/test/repo/test-runner-config.test.ts index b2d76a1e..73dcc8a3 100644 --- a/test/repo/test-runner-config.test.ts +++ b/test/repo/test-runner-config.test.ts @@ -242,3 +242,15 @@ function visit(dir: string, root: string, out: Set): void { else if (/\.(test|spec)\.ts$/.test(entry.name)) out.add(path.relative(root, full)); } } + +// Source contract; the private driver carries real planned/targeted transport. +test("keeps Blaxel harness fixtures separate from the driver and ordinary suites", () => { + const registry = loadTestConfig(); + for (const suite of ["all", "e2e-ci"]) expect(selectSurfaces(registry, { suite }).some(surface => surface.id.startsWith("experiment:blaxel-harness"))).toBe(false); + expect(selectSurfaces(registry, { tag: "blaxel-harness-fixture" }).map(surface => surface.id)).toEqual(["experiment:blaxel-harness-fixtures"]); + for (const surface of registry.surfaces.filter(surface => surface.id.startsWith("experiment:blaxel-harness"))) expect(surface.kind).toBe("experiment"); + const source = readFileSync("test/runner-fixtures/blaxel-harness/driver/harness.test.ts", "utf8"); + expect(source).toContain('["--tag", "blaxel-harness-fixture", "--shards", "2"'); + expect(source).toContain('["--surface", "experiment:blaxel-harness-fixtures"]'); + expect(source).not.toContain('["--surface", "experiment:blaxel-harness"]'); +}); diff --git a/test/repo/test-runner-guardrails.test.ts b/test/repo/test-runner-guardrails.test.ts index 4a3facfb..b1af33eb 100644 --- a/test/repo/test-runner-guardrails.test.ts +++ b/test/repo/test-runner-guardrails.test.ts @@ -737,6 +737,39 @@ describe("test runner guardrails", () => { expect(result.stderr).toContain("--provider must be one of: local, blaxel"); }); + // These two placement cases clear the outer shard token only for an immediate + // CLI dry run; the child retains ordinary surface ownership and starts no work. + test("accepts a configured Blaxel harness coordinator and private pool without fixed names", () => { + const result = runCli(["local", "--surface", "experiment:blaxel-harness"], selftestEnv({ + TV_TEST_RUNNER_DRY_RUN: "1", + TV_TEST_SHARD_OWNER: undefined, + TV_BLAXEL_HARNESS_COORDINATOR: "dedicated-harness-coordinator", + TV_BLAXEL_HARNESS_POOL: "disposable-harness-pool", + })); + expect(result.status, result.stderr).toBe(0); + expect(result.stdout).toContain("surfaces=experiment:blaxel-harness"); + }); + + test("refuses missing Blaxel harness configuration and leased or remote placement before preflight", () => { + const configured = { + TV_TEST_RUNNER_DRY_RUN: "1", + TV_TEST_SHARD_OWNER: undefined, + TV_BLAXEL_HARNESS_COORDINATOR: "dedicated-harness-coordinator", + TV_BLAXEL_HARNESS_POOL: "disposable-harness-pool", + }; + for (const [provider, env] of [ + ["local", { ...configured, TV_BLAXEL_HARNESS_COORDINATOR: " " }], + ["local", { ...configured, TV_BLAXEL_HARNESS_POOL: undefined }], + ["local", { ...configured, TV_TEST_SHARD_OWNER: "fixture-owner" }], + ["blaxel", configured], + ] as const) { + const result = runCli([provider, "--surface", "experiment:blaxel-harness"], selftestEnv(env)); + expect(result.status, result.stderr).toBe(2); + expect(result.stderr).toContain("dedicated coordinator outside the worker pool"); + expect(result.stdout).not.toContain("[dry-run]"); + } + }); + test("rejects isolated lifecycle fixtures on developer-host local execution before preflight", () => { const result = runCli(["local", "--surface", "experiment:lifecycle-leak"], selftestEnv({ TV_TEST_RUNNER_DRY_RUN: "1", diff --git a/test/repo/test-runner-provider.test.ts b/test/repo/test-runner-provider.test.ts index 8bf3c27e..65f8c530 100644 --- a/test/repo/test-runner-provider.test.ts +++ b/test/repo/test-runner-provider.test.ts @@ -5,7 +5,7 @@ import path from "node:path"; import { describe, expect, test } from "vitest"; import { combineAttemptReports, compactProviderSummary, failedTasksFromShards, formatShardLogs, infrastructureRetryShardIndices, providerCommandExitCode, readShardSummaries, runInfrastructureAttempts, summarizeInfrastructureAttempts, writeProviderReports, type AttemptReport } from "../../scripts/test/provider-outputs.mjs"; import { classifyBlaxelCoordinatorFailure, classifyBlaxelShardStatus, classifyDownloadedShardStatus, validateDownloadedShardSummary } from "../../scripts/test/blaxel-classification.mjs"; -import { acquireShardLeases, excludeSandboxesByName, leaseAcquisitionFailureShards } from "../../scripts/test/blaxel-lease-acquisition.mjs"; +import { acquireShardLeases, excludeSandboxesByName, leaseAcquisitionFailureShards, leaseRecoveryArgs } from "../../scripts/test/blaxel-lease-acquisition.mjs"; import { normalizeBlaxelSurfaces, normalizeProviderShardSurfaces, normalizeProviderTimingShards, resolveProviderRunGit } from "../../scripts/test/provider-normalization.mjs"; import { loadTestConfig, selectSurfaces } from "../../scripts/test/config.mjs"; import { normalizeProcessLeak } from "../../scripts/test/timing-events.mjs"; @@ -27,6 +27,16 @@ function writeVitestNative(file: string, total: number) { } describe("provider result normalization", () => { + test("forwards only an explicitly supplied recovery interval to the coordinator", () => { + expect(leaseRecoveryArgs({})).toEqual([]); + expect(leaseRecoveryArgs({ "lock-ttl-seconds": "60" })).toEqual(["--lock-ttl-seconds", "60"]); + expect(readFileSync("scripts/test/cli.mjs", "utf8")).toContain("args.push(...leaseRecoveryArgs(selectionOptions))"); + const coordinator = readFileSync("scripts/run-blaxel-testshards.mjs", "utf8"); + for (const profile of ["normal", "cold", "repair"]) { + expect(coordinator).toMatch(new RegExp(`${profile}: \\{ workerTimeoutSeconds: \\d+, lockTtlSeconds: 1200 \\}`)); + } + }); + test("constructs Blaxel restore/wake timing with a typed passed status", () => { const source = readFileSync("scripts/run-blaxel-testshards.mjs", "utf8"); expect(source).toContain('restorePhase = { name: "sandbox-restore-wake", status: "passed"'); @@ -55,16 +65,13 @@ describe("provider result normalization", () => { expect(source).toContain("Delete the pool before ensure so every provisioned worker is fresh."); expect(source.indexOf("const existing = await listPoolSandboxes()")) .toBeLessThan(source.indexOf("SandboxInstance.createIfNotExists")); - expect(source.indexOf("await sandbox.wait()")) - .toBeLessThan(source.indexOf("await provisionSandboxRuntime(sandbox)")); - expect(source.indexOf("await provisionSandboxRuntime(sandbox)")) - .toBeLessThan(source.indexOf("await verifySandboxArchitecture(sandbox)")); + expect(source).toContain("await publishWorker(sandbox"); expect(source).toContain("options.size ?? String(DEFAULT_BLAXEL_POOL_SIZE)"); expect(source).toContain('const memory = Number.parseInt(options.memory ?? "8192", 10)'); expect(source).toContain('const image = options.image ?? "blaxel/playwright-chromium:latest"'); expect(source).toContain('return `tv-testshard-${arch}-${pool}-${String(index).padStart(2, "0")}`'); - for (const label of ['project: "television"', 'purpose: "testshards"', "pool,", "arch,"]) { + for (const label of ['project: "television"', 'purpose: "testshards-provisioning"', "pool,", "arch,"]) { expect(source).toContain(label); } }); @@ -118,9 +125,9 @@ describe("provider result normalization", () => { }); const source = readFileSync(path.join(process.cwd(), "scripts/run-blaxel-testshards.mjs"), "utf8"); - expect(source).toContain("const dispatch = await transferShardInputs({"); + expect(source).toContain("const dispatch = await (lease.pendingWrites = transferShardInputs({"); expect(source).toContain("return dispatchFailureResult({ lease, outputDir, startedAt, logPath, dispatch });"); - expect(source.indexOf("const dispatch = await transferShardInputs({")).toBeLessThan(source.indexOf("sandbox.process.exec({", source.indexOf("async function runShard"))); + expect(source.indexOf("const dispatch = await (lease.pendingWrites = transferShardInputs({")).toBeLessThan(source.indexOf("sandbox.process.exec({", source.indexOf("async function runShard"))); }); test("preserves completed, running, and pending shard evidence before signal cleanup", async () => { @@ -234,7 +241,7 @@ describe("provider result normalization", () => { const source = readFileSync(path.join(process.cwd(), "scripts/run-blaxel-testshards.mjs"), "utf8"); expect(source).toContain("interruptedSnapshot = buildInterruptedReportSnapshot(signal)"); expect(source).toContain("releaseLeases: async () => {"); - expect(source).toContain("finalizeReport: () => writeInterruptedReportSnapshot(interruptedSnapshot)"); + expect(source).toContain("finalizeReport: () => writeInterruptedReportSnapshot(buildInterruptedReportSnapshot(signal) ?? interruptedSnapshot)"); expect(source).toContain("completedResults: activeAttempt.completedResults"); expect(source).toContain("partialResults: activeAttempt.partialResults"); expect(source).toContain("executionRequested: activeAttempt.executionRequested"); @@ -333,30 +340,19 @@ describe("provider result normalization", () => { expect(source).toContain("MAX_PARALLEL_LEASE_ACQUISITIONS = 32"); expect(source).toContain("await acquireShardLeases({"); expect(source).toContain("acquire: async ({ shardIndex, slot }) => {"); - expect(source).toContain("const lease = await acquireAny(sandboxes, { shardIndex, slot, requestedWorkers: shardIndices.length });"); - expect(source).toContain("const lockId = `${runId}-shard-${shardIndex}`"); + expect(source).toContain("const lease = await acquireAny(sandboxes, { shardIndex, slot, requestedWorkers: shardIndices.length, attempt: attempt + 1 });"); + expect(source).toContain("const lockId = `${runId}-s${shardIndex}-a${attempt}-"); expect(source).toContain("activeLeases.set(lockId, lease)"); expect(source).toContain("lease.executionRequested = true"); expect(source).toContain("lease.executionCompletionConfirmed = true"); expect(source).toContain("killProcess: lease.executionRequested === true && lease.executionCompletionConfirmed !== true"); expect(source).not.toContain("acquired.map((lease) => stopAndReleaseLease(lease, { killProcess: false }))"); expect(source).toContain("lease.logPath = logPath"); - const stopIndex = source.indexOf("await lease.sandbox.process.kill(lease.processName)"); - const ownerCleanupIndex = source.indexOf("await reapUnconfirmedShardOwners(lease)", stopIndex); - const releaseIndex = source.indexOf("await releaseLock(lease.sandbox, lease.lockId)", ownerCleanupIndex); - expect(stopIndex).toBeGreaterThan(0); - expect(ownerCleanupIndex).toBeGreaterThan(stopIndex); - expect(releaseIndex).toBeGreaterThan(ownerCleanupIndex); - expect(source).toContain("node scripts/test/stale-owner-reaper-cli.mjs --output"); - const ownerCleanupStart = source.indexOf("async function reapUnconfirmedShardOwners"); - const ownerCleanupEnd = source.indexOf("function buildInterruptedReportSnapshot", ownerCleanupStart); - const ownerCleanupSource = source.slice(ownerCleanupStart, ownerCleanupEnd); - expect(ownerCleanupSource).toContain('"set -eo pipefail"'); - expect(ownerCleanupSource).not.toContain('"set -euo pipefail"'); - expect(ownerCleanupSource).toContain('"nvm use --silent default"'); - expect(ownerCleanupSource).not.toContain(' "nvm use --silent",'); + expect(source).toContain("await finishLease({"); + expect(source).toContain('teardown: () => confirmTeardown(lease)'); + expect(source).not.toContain("release_lock()"); + const ownerCleanupSource = source.slice(source.indexOf("async function leaseOperation"), source.indexOf("function acquireAny")); expect(ownerCleanupSource.indexOf('. "$NVM_DIR/nvm.sh"')).toBeLessThan(ownerCleanupSource.indexOf('"set -u"')); - expect(ownerCleanupSource.indexOf('"set -u"')).toBeLessThan(ownerCleanupSource.indexOf("node scripts/test/stale-owner-reaper-cli.mjs --output")); const nvmFixture = mkdtempSync(path.join(os.tmpdir(), "tv-provider-nvm-nounset-")); const outsideRepository = path.join(nvmFixture, "outside-repository"); @@ -401,10 +397,7 @@ describe("provider result normalization", () => { rmSync(nvmFixture, { recursive: true, force: true }); } - expect(source).toContain("report.cleanupConfirmed === true"); - expect(source).toContain("report.liveOwners.length === 0"); - expect(source).toContain("report.remainingStaleOwners.length === 0"); - expect(source).toContain("Owner cleanup was not confirmed; retaining lease"); + }); test("plans before Blaxel leases and invokes the production explicit-file worker", () => { @@ -1019,6 +1012,13 @@ describe("provider result normalization", () => { }; expect(validateDownloadedShardSummary(summary, plan, 1)).toBeNull(); + for (const missing of ["test execution", "final shard summary"]) { + const incomplete = { ...summary, status: "infra-failed", failureKind: "setup", tasks: [], collectedFiles: [], neverProduced: [missing] }; + expect(validateDownloadedShardSummary(incomplete, plan, 1)).toBeNull(); + expect(classifyDownloadedShardStatus({ shardSummary: incomplete, exitCode: 1 })).toBe("infra-failed"); + expect(validateDownloadedShardSummary({ ...incomplete, planId: "other" }, plan, 1)).toContain("planId mismatch"); + expect(validateDownloadedShardSummary({ ...incomplete, status: "passed" }, plan, 1)).toContain("task assignments disagree"); + } expect(validateDownloadedShardSummary({ ...summary, status: "passed" }, plan, 1)).toContain("reports passed with a skipped assigned task"); expect(validateDownloadedShardSummary({ ...summary, diff --git a/test/repo/test-runner-resource-parent.test.ts b/test/repo/test-runner-resource-parent.test.ts new file mode 100644 index 00000000..07e6f719 --- /dev/null +++ b/test/repo/test-runner-resource-parent.test.ts @@ -0,0 +1,101 @@ +import { execFileSync, spawnSync } from "node:child_process"; +import { existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs"; +import path from "node:path"; +import { expect, test } from "vitest"; +import { createShardPlan } from "../../scripts/test/shard-plan.mjs"; +import { serializeTimingBaseline } from "../../scripts/test/timing-baseline.mjs"; +import { spawnOwnedProcess } from "../helpers/owned-process.ts"; + +const fixtureRoot = "test/fixtures/resource-diagnostics"; +const sourceFile = `${fixtureRoot}/native.fixture.ts`; +const provider = "blaxel-playwright-x64-4vcpu"; +const read = (file: string) => JSON.parse(readFileSync(file, "utf8")); + +for (const entry of ["planned", "targeted"] as const) { + test(`${entry} parent retains real Linux observations around preparation, recovered native attempts and teardown`, { timeout: 30_000 }, () => { + const fixture = createFixture(entry, "retry"); + try { + const result = spawnSync(process.execPath, fixture.args, { encoding: "utf8", env: fixture.env, timeout: 25_000 }); + expect(result.status, result.stderr).toBe(0); + const prepared = read(path.join(fixture.dir, "preparation.json")); + expect(prepared.records).toHaveLength(1); + expect(prepared.records[0].end.status).toBe("missing"); + const record = readDiagnostics(fixture.dir)[0]; + expect(record.identity).toMatchObject({ runId: "resource-parent-run", attemptId: "resource-parent-attempt", shardIndex: 1, surfaceId: "fixture:resources" }); + expect(Date.parse(record.start.observedAt)).toBeLessThanOrEqual(Date.parse(prepared.observedAt)); + expect(Date.parse(record.end.observedAt)).toBeGreaterThanOrEqual(Date.parse(read(path.join(fixture.dir, "teardown.json")).observedAt)); + expect(record.start.memoryAvailable).toMatchObject({ unit: "bytes", source: "/proc/meminfo:MemAvailable" }); + expect(record.start.sharedMemoryUsed).toMatchObject({ unit: "bytes", source: "/proc/meminfo:Shmem" }); + expect(record.start.temporaryFree).toMatchObject({ unit: "bytes", source: "statfs", path: fixture.dir }); + expect(record.start.oomKills).toMatchObject({ unit: "count", source: "/proc/vmstat:oom_kill" }); + for (const key of ["memoryAvailable", "sharedMemoryUsed", "temporaryFree", "oomKills"]) { + const probe = record.start[key]; + if (probe.value === null) expect(probe.reason).toEqual(expect.any(String)); + else expect(probe.value).toBeGreaterThanOrEqual(0); + } + expect(record.nativeEvidence).toMatchObject({ status: "complete", failedAttempt: true, recovered: true }); + expect(["available", "unavailable"]).toContain(record.kernel.status); + if (record.kernel.status === "unavailable") { + // Expected host limitations stay unknown; malformed reader invocations + // must not masquerade as a successful probe of this installed tool. + expect(["EPERM", "EACCES", "ENOENT", "ENOTSUP", "ETIMEDOUT", "ENOBUFS"]).toContain(record.kernel.reason); + } + const resultRecord = entry === "planned" ? read(path.join(fixture.dir, "results/shard-1.json")).tasks[0] : read(path.join(fixture.dir, "results/lifecycle.json")); + expect(resultRecord.resourceDiagnosticsPaths).toHaveLength(1); + expect(existsSync(resultRecord.resourceDiagnosticsPaths[0])).toBe(true); + expect(resultRecord.processLeaks).toEqual([]); + const attemptPath = path.resolve(path.dirname(resultRecord.resourceDiagnosticsPaths[0]), record.nativeEvidence.attemptResultPath); + const attempts = readFileSync(attemptPath, "utf8").trim().split("\n").map((line) => JSON.parse(line)); + expect(attempts.map((row: any) => row.status)).toEqual(["failed", "passed"]); + } finally { fixture.cleanup(); } + }); +} + +test("targeted parent preserves a baseline after the installed native runner exits before its report", { timeout: 20_000 }, () => { + const fixture = createFixture("targeted", "exit"); + try { + const result = spawnSync(process.execPath, fixture.args, { encoding: "utf8", env: fixture.env, timeout: 15_000 }); + expect(result.status, result.stderr).toBe(17); + expect(read(path.join(fixture.dir, "preparation.json")).records).toHaveLength(1); + const record = readDiagnostics(fixture.dir)[0]; + expect(record).toMatchObject({ end: { status: "observed" }, nativeEvidence: { status: "missing" } }); + expect(existsSync(path.join(fixture.dir, "results/native.json"))).toBe(false); + } finally { fixture.cleanup(); } +}); + +test("interrupted surface parent leaves an explicit missing final observation", { timeout: 20_000 }, async () => { + const fixture = createFixture("targeted", "interrupt"); + const child = spawnOwnedProcess(process.execPath, fixture.args, { env: fixture.env, stdio: "ignore" }); + try { + await expect.poll(() => existsSync(path.join(fixture.dir, "preparation.json")), { timeout: 10_000, interval: 20 }).toBe(true); + await expect.poll(() => child.child.signalCode, { timeout: 5_000, interval: 20 }).toBe("SIGKILL"); + expect((await child.dispose()).outcome).not.toBe("survived"); + const records = readDiagnostics(fixture.dir); + expect(records).toHaveLength(1); + expect(records[0]).toMatchObject({ start: { status: "observed" }, end: { status: "missing", reason: "parent-did-not-finalize" } }); + } finally { await child.dispose(); fixture.cleanup(); } +}); + +function readDiagnostics(dir: string) { + const root = path.join(dir, "results/resources"); + return existsSync(root) ? readdirSync(root).filter((name) => name.endsWith(".json")).map((name) => read(path.join(root, name))) : []; +} + +function createFixture(entry: "planned" | "targeted", mode: string) { + mkdirSync(".test-runs", { recursive: true }); + const dir = mkdtempSync(path.join(process.cwd(), ".test-runs/resource-parent-")); + const results = path.join(dir, "results"); + mkdirSync(results); + const nativeConfig = path.join(dir, "native.config.mjs"); + writeFileSync(nativeConfig, `export default ${JSON.stringify({ test: { include: [path.resolve(sourceFile)], globalSetup: [path.resolve(`${fixtureRoot}/setup.mjs`)], pool: "forks", maxWorkers: 1, retry: 2 } })};\n`); + const preCommand = [process.execPath, path.resolve(`${fixtureRoot}/prepare.mjs`)]; + const surface = { id: "fixture:resources", runner: "vitest", kind: "e2e", package: null, config: nativeConfig, absoluteConfig: nativeConfig, roots: [sourceFile], excludeRoots: [], cwd: ".", supports: ["file"], preflight: [], tags: [], agent: false, preCommand, command: null, executionGroup: { id: "fixture:resources", name: "resource fixture", order: 1 } }; + const registryPath = path.join(dir, "registry.mjs"); + writeFileSync(registryPath, `export default ${JSON.stringify({ suites: { all: { include: ["kind:e2e"] } }, surfaces: [surface] })};\n`); + const plan = createShardPlan({ inventory: [{ path: sourceFile, surfaceId: surface.id, runner: "vitest" }], surfaces: [surface] as any, baseline: { schemaVersion: 1, sourceThrough: null, files: {} }, baselineBytes: serializeTimingBaseline({ schemaVersion: 1, sourceThrough: null, files: {} }), timingProvider: provider, testedCommit: execFileSync("git", ["rev-parse", "HEAD"], { encoding: "utf8" }).trim(), testedTree: execFileSync("git", ["rev-parse", "HEAD^{tree}"], { encoding: "utf8" }).trim(), shardTotal: 1 }); + const planPath = path.join(dir, "plan.json"); + writeFileSync(planPath, JSON.stringify(plan)); + const env = { ...process.env, GITHUB_ACTIONS: "false", TV_TEST_RUNNER_SELFTEST: "1", TEST_SHARD_CONFIG_MODULE: registryPath, TV_TEST_TIMING_PROVIDER: provider, TV_TEST_RUN_ID: "resource-parent-run", TV_TEST_SHARD_INDEX: "1", TV_TEST_SHARD_ATTEMPT: "resource-parent-attempt", RESOURCE_FIXTURE_ROOT: dir, RESOURCE_FIXTURE_MODE: mode, TMPDIR: dir, TV_TARGET_RUNNER: "vitest", TV_TARGET_CONFIG: nativeConfig, TV_TARGET_NATIVE_RESULT: path.join(results, "native.json"), TV_VITEST_ATTEMPT_FILE: path.join(results, "native.attempts.ndjson") }; + const args = entry === "planned" ? ["scripts/run-test-shard.mjs", "--plan", planPath, "--shard", "1", "--results-dir", results, "--test-retries", "2"] : ["scripts/test/supervised-command.mjs", "--run-id", "resource-parent-run", "--surface", surface.id, "--result", path.join(results, "lifecycle.json"), "--pre-command-json", JSON.stringify(preCommand), "--", "bash", "-c", 'exec node node_modules/vitest/vitest.mjs run --config "${TV_TARGET_VITEST_CONFIG:-$TV_TARGET_CONFIG}" --reporter=json --outputFile "$TV_TARGET_NATIVE_RESULT"' ]; + return { dir, args, env, cleanup: () => rmSync(dir, { recursive: true, force: true }) }; +} diff --git a/test/repo/test-runner-shard-worker.test.ts b/test/repo/test-runner-shard-worker.test.ts index 93e948e5..c4cfe5c7 100644 --- a/test/repo/test-runner-shard-worker.test.ts +++ b/test/repo/test-runner-shard-worker.test.ts @@ -37,9 +37,9 @@ describe("planned shard worker", () => { const fixture = workerFixture(["fixture:vitest-one", "fixture:vitest-two"], [ inventory("vitest-one.fixture.ts", "fixture:vitest-one", "vitest"), inventory("vitest-two.fixture.ts", "fixture:vitest-two", "vitest"), - ]); + ], "blaxel-playwright-x64-4vcpu"); try { - const result = runWorker(fixture, { TV_SHARD_RETRY_FIXTURE: "1" }); + const result = runWorker(fixture, { TV_SHARD_RETRY_FIXTURE: "1", TV_TEST_RUN_ID: "combined-resource-fixture", TV_TEST_SHARD_ATTEMPT: "combined-attempt", TV_TEST_SHARD_INDEX: "1" }); expect(result.status, result.stderr).toBe(0); const summary = shardSummary(fixture); expect(summary.preflights).toEqual([ @@ -73,6 +73,17 @@ describe("planned shard worker", () => { }); expect(normalized.find((surface) => surface.id === "fixture:vitest-one")?.counts.testsFlakyRecovered).toBe(0); expect(normalized.find((surface) => surface.id === "fixture:vitest-two")?.counts.testsFlakyRecovered).toBe(1); + for (const [index, task] of summary.tasks.entries()) { + expect(task.resourceDiagnosticsPaths).toHaveLength(1); + const diagnosticPath = task.resourceDiagnosticsPaths[0]; + const diagnostic = JSON.parse(readFileSync(diagnosticPath, "utf8")); + expect(diagnostic).toMatchObject({ + identity: { runId: "combined-resource-fixture", attemptId: "combined-attempt", shardIndex: 1, surfaceId: task.surfaceId }, + start: { status: "observed" }, end: { status: "observed" }, + nativeEvidence: { status: "complete", failedAttempt: index === 1, recovered: index === 1 }, + }); + expect(path.resolve(path.dirname(diagnosticPath), diagnostic.nativeEvidence.resultPath)).toBe(task.nativeResultPath); + } } finally { fixture.cleanup(); } }); diff --git a/test/repo/worker-builds.test.ts b/test/repo/worker-builds.test.ts new file mode 100644 index 00000000..9630b5cb --- /dev/null +++ b/test/repo/worker-builds.test.ts @@ -0,0 +1,79 @@ +import { existsSync, mkdtempSync, readdirSync, rmSync, writeFileSync } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, expect, test } from "vitest"; +import { WorkerBuilds } from "../helpers/worker-builds.ts"; + +const roots: string[] = []; +function rendezvous() { + let resolve!: (value: T) => void; + const promise = new Promise((done) => { resolve = done; }); + return { promise, resolve }; +} +function fixture(): { root: string; builds: WorkerBuilds } { + const root = mkdtempSync(path.join(os.tmpdir(), "worker-build-contract-")); + roots.push(root); + return { root, builds: new WorkerBuilds("output-", root) }; +} +afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); }); + +// proofs/arch/testing-policy.md#^testing-t-build-lifetime +test("concurrent and completed requests reuse a build while version keys remain distinct", async () => { + const { root, builds } = fixture(); + const ready = rendezvous(); + let calls = 0; + const first = builds.get("v1", async (directory) => { + calls += 1; + writeFileSync(path.join(directory, "entry.js"), "version one"); + await ready.promise; + }); + const concurrent = builds.get("v1", () => { throw new Error("prepared twice"); }); + expect(concurrent).toBe(first); + ready.resolve(); + const directory = await first; + expect(await builds.get("v1", () => { throw new Error("prepared again"); })).toBe(directory); + const second = await builds.get("v2", (output) => { writeFileSync(path.join(output, "entry.js"), "version two"); }); + expect(second).not.toBe(directory); + expect(calls).toBe(1); + expect(existsSync(path.join(directory, "entry.js"))).toBe(true); + await builds.dispose(); + expect(readdirSync(root)).toEqual([]); +}); + +test("failed preparation removes partial output and permits a fresh build", async () => { + const { root, builds } = fixture(); + let failedDirectory = ""; + await expect(builds.get("source", (directory) => { + failedDirectory = directory; + writeFileSync(path.join(directory, "partial.js"), "incomplete"); + throw new Error("asset preparation failed"); + })).rejects.toThrow("asset preparation failed"); + expect(readdirSync(root)).toEqual([]); + const directory = await builds.get("source", (output) => { writeFileSync(path.join(output, "entry.js"), "complete"); }); + expect(directory).not.toBe(failedDirectory); + expect(existsSync(path.join(directory, "entry.js"))).toBe(true); + await builds.dispose(); +}); + +test.each([false, true])("teardown waits for pending preparation, including rejection (%s)", async (reject) => { + const { root, builds } = fixture(); + const started = rendezvous(); + const ready = rendezvous(); + const build = builds.get("pending", async (directory) => { + writeFileSync(path.join(directory, "partial.js"), "pending"); + started.resolve(directory); + await ready.promise; + if (reject) throw new Error("late preparation failure"); + }); + const settled = Promise.allSettled([build]); + const directory = await started.promise; + let disposed = false; + const cleanup = builds.dispose().then(() => { disposed = true; }); + await Promise.resolve(); + expect(disposed).toBe(false); + expect(existsSync(directory)).toBe(true); + ready.resolve(); + expect((await settled)[0]!.status).toBe(reject ? "rejected" : "fulfilled"); + await cleanup; + expect(readdirSync(root)).toEqual([]); +}); diff --git a/test/runner-fixtures/blaxel-harness/README.md b/test/runner-fixtures/blaxel-harness/README.md new file mode 100644 index 00000000..77e877b2 --- /dev/null +++ b/test/runner-fixtures/blaxel-harness/README.md @@ -0,0 +1,34 @@ +# Live Blaxel harness qualification + +Run on a dedicated credentialed Blaxel coordinator sandbox outside every worker +pool, from a committed and pushed checkout with the repository's selected Node, +dependencies and local process-lifecycle prerequisites installed. This driver must +not run on a shared developer host or a leased worker. + +Supply the coordinator's sandbox name and an exclusively owned disposable pool: + +```sh +TV_BLAXEL_HARNESS_COORDINATOR= \ +TV_BLAXEL_HARNESS_POOL= \ +npm test -- local --surface experiment:blaxel-harness +``` + +The driver deletes that pool before provisioning two fresh workers and deletes it +again after qualification. Retain any earlier reports first. Never name a shared +pool. The coordinator remains allocated for the caller to retrieve evidence and +dispose of it. Native retry defaults apply. + +The CLI requires both names, local execution and no shard owner token. The driver +also checks that the named coordinator lacks the worker `purpose=testshards` label. +These checks prevent accidental placement; the operator supplies the actual +dedicated coordinator's name. + +Cleanup fixtures and sentinels include the selected pool in their names. Each +fresh worker has one fixture-mode input, `/tmp/tv-blaxel-harness-mode.json`, read +by the committed native fixtures. It needs no additional production environment +forwarding and disappears with the private worker. + +Inspect the canonical run directory and `.test-runs/blaxel-live-*/` for the +individual commands, child run pointers, recovery evidence, completion and cleanup +records. Expected native fixture failures are checked by the driver; a passing +driver does not turn those child reports into passing suites. diff --git a/test/runner-fixtures/blaxel-harness/driver/harness.test.ts b/test/runner-fixtures/blaxel-harness/driver/harness.test.ts new file mode 100644 index 00000000..6acff495 --- /dev/null +++ b/test/runner-fixtures/blaxel-harness/driver/harness.test.ts @@ -0,0 +1,227 @@ +import { spawn, type ChildProcess } from "node:child_process"; +import { existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, writeFileSync } from "node:fs"; +import path from "node:path"; +import { SandboxInstance } from "@blaxel/core"; +import { expect, test } from "vitest"; +import { listBlaxelPoolSandboxes } from "../../../../scripts/test/blaxel-pool.mjs"; + +// Acceptance: real canonical subprocesses, SDK, fresh private workers and native +// fixtures. The sole scheduling hook pauses immediately before report download. +const pool = process.env.TV_BLAXEL_HARNESS_POOL?.trim() ?? ""; +const coordinatorName = process.env.TV_BLAXEL_HARNESS_COORDINATOR?.trim() ?? ""; +const planned = ["--tag", "blaxel-harness-fixture", "--shards", "2", "--allow-shard-count-override", "1"]; +const targeted = ["--surface", "experiment:blaxel-harness-fixtures"]; +const resultsRoot = "/workspace/television/.testshards/results"; +// Each fresh worker belongs to one private pool. This fixed fixture input needs +// no additional environment forwarding through the production coordinator. +const modePath = "/tmp/tv-blaxel-harness-mode.json"; +const fixtureSuffix = `harness-${encodeURIComponent(pool)}`; +const sourceResidue = `/tmp/television-source-cli-e2e-${fixtureSuffix}`; +const webResidue = `/tmp/television-web-bundle-${fixtureSuffix}`; +const staleScratch = `/tmp/tv-testshards/${fixtureSuffix}`; +const unrelatedSentinel = `/tmp/tv-${fixtureSuffix}-unrelated/keep`; +const cacheSentinel = `/cache/tv-${fixtureSuffix}/keep`; +const repositoryRoot = path.resolve(import.meta.dirname, "../../../.."); +const cli = path.join(repositoryRoot, "scripts/test/cli.mjs"); +const children = new Set(); +const quote = (value: string) => `'${value.replaceAll("'", `'"'"'`)}'`; +const pause = (ms: number) => new Promise(resolve => setTimeout(resolve, ms)); +async function until(probe: () => Promise, timeout = 120_000): Promise { + const deadline = Date.now() + timeout; + while (Date.now() < deadline) { const result = await probe(); if (result) return result; await pause(250); } + throw Error("live harness rendezvous timed out"); +} +function readJson(file: string) { return JSON.parse(readFileSync(file, "utf8")); } +async function remoteJson(worker: SandboxInstance, file: string) { try { return JSON.parse(await worker.fs.read(file)); } catch (error: any) { if (error?.status === 404) return null; throw error; } } + +function command(root: string, name: string, args: string[], env: NodeJS.ProcessEnv = {}) { + const log = path.join(root, `${name}.log`); + const child = spawn(process.execPath, [cli, ...args], { cwd: repositoryRoot, detached: true, env: {...process.env, ...env}, stdio: ["ignore", "pipe", "pipe"] }); + children.add(child); + const chunks: Buffer[] = []; + child.stdout!.on("data", data => chunks.push(data)); child.stderr!.on("data", data => chunks.push(data)); + const done = new Promise((resolve, reject) => { + child.on("error", reject); + child.on("close", code => { children.delete(child); writeFileSync(log, Buffer.concat(chunks)); resolve(code); }); + }); + return {child, done, log}; +} +function run(root: string, name: string, selection: string[], gate?: string, recoverySeconds?: number) { + const pointer = path.join(root, `${name}.run`); + const task = command(root, name, ["blaxel", "--pool", pool, ...selection, ...(recoverySeconds ? ["--lock-ttl-seconds", String(recoverySeconds)] : []), "--run-dir-output", pointer], gate ? {TV_TEST_RUNNER_SELFTEST: "1", TV_BLAXEL_DOWNLOAD_GATE: gate} : {}); + return {...task, directory: async () => until(async () => existsSync(pointer) ? readFileSync(pointer, "utf8").trim() : null)}; +} +async function poolCommand(root: string, name: string, operation: string) { + const code = await command(root, name, ["pool", "blaxel", operation, "--pool", pool, ...(operation === "ensure" ? ["--size", "2"] : [])]).done; + // The API accepts deletion before its listing stops returning the worker. + // Observe absence before fresh provisioning or claiming the pool is gone. + if (operation === "delete" && code === 0) await until(async () => { + if ((await listBlaxelPoolSandboxes({pool, arch: "x64"})).length === 0) return true; + await pause(1000); return null; + }); + return code; +} +async function gates(gate: string) { + // Cold worker execution is allowed 480 seconds, before teardown and retrieval. + return until(async () => [1,2].every(index => existsSync(path.join(gate, `shard-${index}.json`))) ? [1,2].map(index => readJson(path.join(gate, `shard-${index}.json`))) : null, 600_000); +} +function retained(runDir: string) { + const summary = readJson(path.join(runDir, "summary.json")); + const provider = readJson(path.join(runDir, "provider/blaxel/report.json")); + const normalized = readJson(path.join(runDir, "results.json")); + return { summary, provider, normalized }; +} +function assertDiagnostics(runDir: string, expectedRecovery = false) { + const {provider, normalized} = retained(runDir); + const sidecars = [...new Set(normalized.surfaces.flatMap((surface: any) => surface.resourceDiagnosticsPaths ?? []))]; + expect(sidecars.length).toBeGreaterThan(0); + const records = sidecars.map(file => readJson(path.resolve(runDir, file))); + for (const [index, record] of records.entries()) { + expect(provider.shards.some((shard: any) => shard.shardIndex === record.identity.shardIndex)).toBe(true); + expect(record.identity.runId).toBe(provider.runId); + expect(record.identity.attemptId).toContain(provider.runId); + if (record.kind === "surface") { + expect(record.identity.surfaceId).toBe("experiment:blaxel-harness-fixtures"); + expect(record.start.status).toBe("observed"); expect(record.end.status).toBe("observed"); + expect(record.nativeEvidence.status).toBe("complete"); + for (const reference of [record.nativeEvidence.resultPath, record.nativeEvidence.attemptResultPath].filter(Boolean)) { + expect(existsSync(path.resolve(runDir, path.dirname(sidecars[index]), reference))).toBe(true); + } + if (record.nativeEvidence.failedAttempt) expect(["available", "unavailable"]).toContain(record.kernel?.status); + if (record.kernel?.status === "unavailable") expect(record.kernel.reason).toBeTruthy(); + } + } + expect(records.some(record => record.nativeEvidence?.recovered === true)).toBe(true); + if (expectedRecovery) { + const receipts = records.filter(record => record.kind === "recovery"); + expect(receipts.length).toBe(2); + for (const receipt of receipts) { expect(receipt.discardedReports).toBe(true); expect(receipt.cleanup).toBe("confirmed"); expect(receipt.previous.runId).not.toBe(provider.runId); } + } + return records; +} + +test("qualifies exclusive retrieval, active and quiet recovery, temporary placement, and pool lifecycle", async () => { + if (!coordinatorName || !pool || process.env.TV_TEST_SHARD_OWNER) throw Error("Set TV_BLAXEL_HARNESS_COORDINATOR and TV_BLAXEL_HARNESS_POOL on the dedicated Blaxel coordinator, outside every leased worker"); + const coordinator = await SandboxInstance.get(coordinatorName); + expect(coordinator.metadata.labels?.purpose).not.toBe("testshards"); + const root = mkdtempSync(path.join(repositoryRoot, ".test-runs/blaxel-live-")); + writeFileSync(path.join(root, "identity.json"), JSON.stringify({coordinatorName, pool, revision: readFileSync(path.join(repositoryRoot, ".git/HEAD"), "utf8").trim()})); + let workers: SandboxInstance[] = []; + try { + // The caller retained earlier development reports before this fresh-pool run. + expect(await poolCommand(root, "initial-delete", "delete")).toBe(0); + expect(await poolCommand(root, "ensure", "ensure")).toBe(0); + workers = await listBlaxelPoolSandboxes({pool, arch: "x64", deployedOnly: true}); + expect(workers).toHaveLength(2); + for (const worker of workers) { + const directories = [sourceResidue, webResidue, staleScratch, path.dirname(unrelatedSentinel), path.dirname(cacheSentinel)]; + const setup = [`mkdir -p ${directories.map(quote).join(" ")}`, + ...[sourceResidue, webResidue].map(directory => `printf fixture > ${quote(`${directory}/data`)}`), + ...[unrelatedSentinel, cacheSentinel].map(file => `printf keep > ${quote(file)}`)].join("; "); + const seeded = await worker.process.exec({name: "tv-harness-seed-fixtures", command: `bash -c ${quote(setup)}`, waitForCompletion: true, timeout: 10}); + expect(seeded.exitCode).toBe(0); + await worker.fs.write(modePath, JSON.stringify({mode: "normal"})); + } + const gate = path.join(root, "normal-download"); mkdirSync(gate); + const first = run(root, "planned", planned, gate); + const waiting = await gates(gate); + const before = await Promise.all(workers.map(worker => remoteJson(worker, "/tmp/tv-testshard.lock/owner.json"))); + const reportsBefore = await Promise.all(workers.map((worker, index) => worker.fs.read(`${resultsRoot}/shard-${before[index].shardIndex}.json`))); + await until(async () => { + const after = await Promise.all(workers.map(worker => remoteJson(worker, "/tmp/tv-testshard.lock/owner.json"))); + return after.every((record, index) => record.lastRenewal > before[index].lastRenewal) ? after : null; + }); + const competed = run(root, "competing", planned); + expect(await competed.done).not.toBe(0); + expect(retained(await competed.directory()).summary.run.infraStatus).toBe("incomplete"); + expect(await poolCommand(root, "held-delete", "delete")).toBe(1); + for (const worker of workers) expect((await remoteJson(worker, "/tmp/tv-testshard.lock/owner.json")).runId).toBe(before[0].runId); + const reportsAfter = await Promise.all(workers.map((worker, index) => worker.fs.read(`${resultsRoot}/shard-${before[index].shardIndex}.json`))); + expect(reportsAfter).toEqual(reportsBefore); + writeFileSync(path.join(gate, "continue"), "continue"); + expect(await first.done).toBe(1); // intentional native fixture failure + const firstDir = await first.directory(); const firstResult = retained(firstDir); + expect(firstResult.summary.run.infraStatus).toBe("completed"); + expect(firstResult.summary.counts.testsFailed).toBe(1); expect(firstResult.summary.counts.testsFlakyRecovered).toBe(1); + assertDiagnostics(firstDir); + const placements: any[] = []; + for (const shard of firstResult.provider.shards) { + expect(shard.remoteScript).toBe(`${shard.scratch}/worker.sh`); + expect(shard.setupTimings.some((phase: any) => phase.name === "temporary-cleanup")).toBe(true); + const artifact = path.join(shard.outputDir, "artifacts", `shard-${shard.shardIndex}`); + const cleanup = readJson(path.join(artifact, "temporary-cleanup.json")); + expect(cleanup.removed["television-source-cli-e2e-"].directories).toBe(1); + expect(cleanup.removed["television-web-bundle-"].directories).toBe(1); + expect(cleanup.removed["stale-scratch"].directories).toBe(1); + const placement = path.join(artifact, "temporary-placement.json"); + if (existsSync(placement)) placements.push(readJson(placement)); + } + expect(placements).toHaveLength(1); + expect(placements[0].browserVersion).toMatch(/^\d+\./); + expect(placements[0].executable).toContain("/home/playwright/.cache/ms-playwright/"); + expect(placements[0].nodeDir.startsWith(placements[0].tmp + "/")).toBe(true); + expect(placements[0].descriptors.some((entry: any) => entry.path.endsWith(" (deleted)"))).toBe(true); + for (const worker of workers) { + const ownGate = waiting.find(entry => entry.sandbox === worker.metadata.name)!; + expect(await remoteJson(worker, "/tmp/tv-testshard.lock/owner.json")).toBeNull(); + await expect(worker.fs.ls(ownGate.scratch)).rejects.toMatchObject({status: 404}); + expect(await worker.fs.read(unrelatedSentinel)).toBe("keep"); + expect(await worker.fs.read(cacheSentinel)).toBe("keep"); + } + const target = run(root, "targeted", targeted); + expect(await target.done).toBe(1); + expect(retained(await target.directory()).summary.run.infraStatus).toBe("completed"); + assertDiagnostics(await target.directory()); + const targetShard = retained(await target.directory()).provider.shards[0]; + expect(existsSync(path.join(targetShard.outputDir, "artifacts/shard-1/temporary-placement.json"))).toBe(true); + + for (const scenario of ["active", "quiet"]) { + for (const worker of workers) await worker.fs.write(modePath, JSON.stringify({mode: scenario === "active" ? "active" : "normal"})); + const crashGate = path.join(root, `${scenario}-download`); mkdirSync(crashGate); + const abandoned = run(root, `${scenario}-abandoned`, planned, crashGate, 60); + let owners: any[]; + if (scenario === "quiet") { await gates(crashGate); owners = await Promise.all(workers.map(worker => remoteJson(worker, "/tmp/tv-testshard.lock/owner.json"))); } + else owners = await until(async () => { + const active = await Promise.all(workers.map(worker => remoteJson(worker, `${resultsRoot}/active-fixture.json`))); + return active.every(Boolean) ? Promise.all(workers.map(worker => remoteJson(worker, "/tmp/tv-testshard.lock/owner.json"))) : null; + }); + const coordinatorPid = Number(owners[0].owner.split(":").at(-1)); + expect(coordinatorPid).toBeGreaterThan(1); expect(coordinatorPid).not.toBe(process.pid); + process.kill(coordinatorPid, "SIGKILL"); // actual renewer, not just its shard + expect(await abandoned.done).not.toBe(0); + for (const worker of workers) await worker.fs.write(modePath, JSON.stringify({mode: "normal"})); + owners = await until(async () => { + const current = await Promise.all(workers.map(worker => remoteJson(worker, "/tmp/tv-testshard.lock/owner.json"))); + return current.every((record, index) => record?.lockId === owners[index].lockId && Date.now() > record.recoverAfter + 500) ? current : null; + }, 120_000); + const recovered = run(root, `${scenario}-recovered`, planned); + expect(await recovered.done).toBe(1); + const directory = await recovered.directory(); + expect(retained(directory).summary.run.infraStatus).toBe("completed"); + const records = assertDiagnostics(directory, true); + for (const receipt of records.filter(record => record.kind === "recovery")) { + const previous = owners.find(owner => owner.lockId === receipt.previous.lockId); + expect(previous).toBeTruthy(); + expect(receipt.previous.lastRenewal).toBe(previous.lastRenewal); + expect(receipt.previous.recoverAfter).toBe(previous.recoverAfter); + } + if (scenario === "active") expect(records.filter(record => record.kind === "recovery").every(record => record.processes.length > 0)).toBe(true); + writeFileSync(path.join(root, `${scenario}-recovery-evidence.json`), JSON.stringify({owners, recoveredRun: directory})); + } + expect(await poolCommand(root, "final-delete", "delete")).toBe(0); + expect(await listBlaxelPoolSandboxes({pool, arch: "x64"})).toHaveLength(0); + const absent = run(root, "absent-pool", targeted); expect(await absent.done).not.toBe(0); + writeFileSync(path.join(root, "completed.json"), JSON.stringify({status: "passed", runs: readdirSync(root).filter(file => file.endsWith(".run"))})); + } finally { + const pending = [...children]; + const stopped = Promise.all(pending.map(child => new Promise(resolve => child.once("close", () => resolve())))); + for (const child of pending) { try { process.kill(-child.pid!, "SIGTERM"); } catch {} } + await Promise.race([stopped, pause(10_000)]); + for (const child of children) { try { process.kill(-child.pid!, "SIGKILL"); } catch {} } + // A valid lease is never bypassed. With our coordinators stopped, the same + // manager may delete an expired private worker, which is itself cleanup. + let cleanup = await poolCommand(root, "finally-delete", "delete"); + if (cleanup !== 0) { await pause(65_000); cleanup = await poolCommand(root, "finally-delete-expired", "delete"); } + writeFileSync(path.join(root, "cleanup.json"), JSON.stringify({exitCode: cleanup})); + } +}); diff --git a/test/runner-fixtures/blaxel-harness/driver/vitest.config.ts b/test/runner-fixtures/blaxel-harness/driver/vitest.config.ts new file mode 100644 index 00000000..290cbc8b --- /dev/null +++ b/test/runner-fixtures/blaxel-harness/driver/vitest.config.ts @@ -0,0 +1,2 @@ +import { defineConfig } from "vitest/config"; +export default defineConfig({ root: import.meta.dirname, test: { include: ["*.test.ts"], maxWorkers: 1, fileParallelism: false, testTimeout: 1_200_000, hookTimeout: 120_000 } }); diff --git a/test/runner-fixtures/blaxel-harness/fixtures/outcomes.test.ts b/test/runner-fixtures/blaxel-harness/fixtures/outcomes.test.ts new file mode 100644 index 00000000..39b8894d --- /dev/null +++ b/test/runner-fixtures/blaxel-harness/fixtures/outcomes.test.ts @@ -0,0 +1,7 @@ +import { expect, test } from "vitest"; +import { activeFixture, mode } from "./support.ts"; + +test("bounded active owned process", activeFixture); +let attempt = 0; +test("recovers a deliberate first attempt", () => { if (mode() === "normal") expect(++attempt).toBeGreaterThan(1); }); +test("retains a deliberate native failure", () => { if (mode() === "normal") expect("fixture failure").toBe("fixture success"); }); diff --git a/test/runner-fixtures/blaxel-harness/fixtures/support.ts b/test/runner-fixtures/blaxel-harness/fixtures/support.ts new file mode 100644 index 00000000..b1786840 --- /dev/null +++ b/test/runner-fixtures/blaxel-harness/fixtures/support.ts @@ -0,0 +1,20 @@ +import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import path from "node:path"; +import { spawnOwnedProcess } from "../../../helpers/owned-process.ts"; + +export const results = "/workspace/television/.testshards/results"; +export function mode() { + const file = "/tmp/tv-blaxel-harness-mode.json"; + return existsSync(file) ? JSON.parse(readFileSync(file, "utf8")).mode : "normal"; +} +export function evidence(name: string, value: unknown) { + mkdirSync(results, { recursive: true }); + writeFileSync(path.join(results, name), JSON.stringify(value)); +} +export async function activeFixture() { + if (mode() !== "active") return; + const child = spawnOwnedProcess(process.execPath, ["-e", "process.on('SIGTERM',()=>{});setTimeout(()=>process.exit(0),180000);setInterval(()=>{},1000)"], { stdio: "ignore" }); + evidence("active-fixture.json", { pid: child.pid, supervisor: process.pid, token: process.env.TV_TEST_SHARD_OWNER }); + try { await new Promise(resolve => setTimeout(resolve, 180_000)); } + finally { await child.dispose(); } +} diff --git a/test/runner-fixtures/blaxel-harness/fixtures/temporary.test.ts b/test/runner-fixtures/blaxel-harness/fixtures/temporary.test.ts new file mode 100644 index 00000000..78edc8ea --- /dev/null +++ b/test/runner-fixtures/blaxel-harness/fixtures/temporary.test.ts @@ -0,0 +1,33 @@ +import { expect, test } from "vitest"; +import { mkdtempSync, readdirSync, readFileSync, readlinkSync, rmSync, writeFileSync } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { chromium } from "playwright"; +import { activeFixture, evidence } from "./support.ts"; + +test("bounded active owned process", activeFixture); +test("Node and Chromium place real temporary files under the shard attempt", async () => { + const tmp = os.tmpdir(); + expect(tmp).toMatch(/^\/tmp\/tv-testshards\/[^/]+\/tmp$/); + const nodeDir = mkdtempSync(path.join(tmp, "node-placement-")); + writeFileSync(path.join(nodeDir, "created"), "node temporary file"); + const browser = await chromium.launch({ headless: true }); + try { + const page = await browser.newPage(); + await page.setContent(''); + await page.screenshot(); + const descriptors: Array<{pid: number; fd: string; path: string}> = []; + for (const pid of readdirSync("/proc").filter(value => /^\d+$/.test(value))) { + try { + const env = readFileSync(`/proc/${pid}/environ`, "utf8"); + const exe = readlinkSync(`/proc/${pid}/exe`); + if (!env.split("\0").includes(`TV_TEST_SHARD_OWNER=${process.env.TV_TEST_SHARD_OWNER}`) || !/chrome|chromium/.test(exe)) continue; + for (const fd of readdirSync(`/proc/${pid}/fd`)) { + try { const target = readlinkSync(`/proc/${pid}/fd/${fd}`); if (target.startsWith(tmp + "/")) descriptors.push({pid: Number(pid), fd, path: target}); } catch {} + } + } catch {} + } + evidence("temporary-placement.json", { tmp, nodeDir, browserVersion: browser.version(), executable: chromium.executablePath(), token: process.env.TV_TEST_SHARD_OWNER, surface: process.env.TV_TEST_SURFACE_OWNER, descriptors }); + expect(descriptors.some(entry => entry.path.includes("Chromium") && entry.path.endsWith(" (deleted)"))).toBe(true); + } finally { await browser.close(); rmSync(nodeDir, {recursive: true, force: true}); } +}); diff --git a/test/runner-fixtures/blaxel-harness/fixtures/vitest.config.ts b/test/runner-fixtures/blaxel-harness/fixtures/vitest.config.ts new file mode 100644 index 00000000..9676b195 --- /dev/null +++ b/test/runner-fixtures/blaxel-harness/fixtures/vitest.config.ts @@ -0,0 +1,2 @@ +import { defineConfig } from "vitest/config"; +export default defineConfig({ root: import.meta.dirname, test: { include: ["*.test.ts"], maxWorkers: 1, fileParallelism: false, testTimeout: 240_000 } });