Repository navigation
ci(prd-480): Linux CI runs on the owner machine - #404
Merged
Merged
Conversation
Phase 1 of PRD-480: the image, its entrypoint, and the host-side
lifecycle script. No runners are started and nothing is registered
yet — the script fails closed without a token, and `status` reports an
empty pool.
No compose.yml (owner-approved deviation, noted in the PRD): a host-side
loop of `docker run --rm` is what makes each job a fresh container
filesystem and hostname, which `restart: always` on one long-lived
container cannot.
Evidence:
- `docker build -t tn-ci-runner tools/ci-runners` — clean.
- smoke: cmake 3.28.3, clang 18.1.3, ninja 1.11.1, /usr/bin/{Xvfb,ccache,jq},
`ls -A /tmp | wc -l` → 0.
- `TN_RUNNERS_ENV=/nonexistent scripts/ci-runners.sh up` → exit 2, nothing built
or started. `up 0` and an unknown subcommand also exit 2.
- `docker run --rm tn-ci-runner` without the env file → exit 1 naming
RUNNER_ADMIN_TOKEN.
- `scripts/ci-runners.sh status` → TN_RUNNER=unset, 0 containers, 0 online.
- `bash -n` clean on both scripts (no shellcheck installed here).
Phase 2 of PRD-480. 23 `runs-on` lines (18 in ci.yml, one each in the five
integration-*.yml) now read:
runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }}
Deleting the TN_RUNNER repository variable is the whole kill switch, and a fork
PR always runs hosted. `supply-chain` keeps ubuntu-latest because gitleaks needs
`docker run` and the socket would hand every job root on the host;
integration-decals.yml's ubuntu-24.04-arm job and every macOS/Windows job are untouched, as
is native-platforms.yml (Phase 3).
A new ci-structure.spec.ts test enforces it: every job in ci.yml and the
integration workflows either carries that exact expression, is on the hosted
allow-list, or is a non-Linux runner.
Evidence:
- red, no workflow edited: "ci.yml scope runs on `runs-on: ubuntu-latest`".
- red, after routing, ci.yml:1051 (budgets) put back on ubuntu-latest: same test
failed naming that job, then it was restored.
- green: `pnpm exec vitest run scripts/__tests__/ci-structure.spec.ts
scripts/__tests__/ci-needs.spec.ts` — 142 passed, 0 failed.
- `pnpm lint` exit 0, `pnpm check:docs` 2496 links, `pnpm sync:agents --check` 22 mirrors.
- all six workflows parse as YAML and every runs-on resolves to one plain scalar.
One pre-existing assertion was narrowed rather than deleted: template-nonvisual
forbade the substring `pull_request` anywhere in the job, which the routing
expression mentions in every job without gating on it. It now forbids an `if:`
gate naming pull_request.
…ig.sh lives First live up: 0 of 4 online, the pool tore itself down and cleared TN_RUNNER. With the fix, 4 tn-local runners come online.
jonit-dev
marked this pull request as ready for review
October 2, 2026 20:30
…4.04 does The scope job runs node straight after checkout; on tn-local it failed with "node: command not found" (run 37061064658).
…d per slot PR #404 run 2: test-native failed on "rustc: command not found"; three unit tests timed out because a --cpus quota leaves nproc at 24, so vitest ran 8 workers in a 4-core budget. Each slot now owns two whole cores (nproc = 4, like hosted), and the image adds rustup stable, 7zip, ffmpeg, imagemagick, zstd, file, lsof, procps and mesa-vulkan-drivers.
…t for the host 12 GB per slot with no extra swap (5 x 12 = 60 of 62 GB worst case, typical jobs far less) and --oom-score-adj 800, so memory pressure kills a CI job before anything of the owner's. up refuses a pool that would take the host's two reserved cores.
…as none Workflows that need Mesa install it themselves. With it baked in, the tone and decals captures went blank on tn-local (run 37062759406: TN_CAPTURE_BLANK) while the same job passed on the image without it.
…ght lane Records the pool shape measured on the first live runs and the owner's stability constraint, and adds Astra's light-lane review point as a Phase 2 box.
A ten-second verdict queued behind five twenty-minute builds is the wait this removes. `ci-runners.sh up` now starts one light slot alongside the heavy ones: same image and host loop, `--cpus 1 --memory 2g --memory-swap 2g --oom-score-adj 900`, no cpuset, registering `tn-local-light` (entrypoint takes the label from `TN_RUNNER_LABELS`), and it advertises `TN_RUNNER_LIGHT` only once that runner is online — so the kill switch stays one deletion and `down` clears both variables before it stops anything. `scope`, `golden-path`, `build`, `ci-required` and `run-summary` route to it: the five ci.yml jobs whose whole work is a script or a summary, none of which builds the workspace or runs a test suite. Every other movable Linux job keeps the TN_RUNNER expression. `ci-structure.spec.ts` now requires each Linux job to use exactly one of the two expressions, and asserts the light allow-list in both directions, so a heavy job on the light lane and a light job off it both fail.
test-native on tn-local (run 37062759791): the Android packaging tests drive a fake gradlew built on jar, javac and java; without a JDK every one exited 127.
test-unit 3/3 on tn-local: the playtest doctor spec failed on "adb was not found".
…tions Inside bash -c the first argument is $0, so the label landed where the image name goes and docker tried to pull tn-local:latest. up then failed closed.
test-native run 37068328553: javac read MystralActivity.java as US-ASCII and rejected an em dash, so the Android metadata probe failed. 1 of 1593 left red.
4 tasks
…pool Phase 3 of PRD-480. Eight Linux jobs in native-platforms.yml and the `starter-linux/linux-x64` matrix row now pick their runner from the same expression ci.yml uses, so the pool is still changed by setting or deleting one repository variable. `scope` and `networking-matrix` — the two whose whole work is a script with no `pnpm install` — take the light lane; everything else is heavy. `github.event` inside a `workflow_call` is the caller's event payload, so the fork guard reads the same flag ci.yml did when it called this file. Kept hosted, with reasons in the file and in the spec's allow-list: `publish-android-v8` is the release write itself (`contents: write` plus `gh release upload`), and macOS, Windows, `linux-arm64` and iOS have no tn-local counterpart. The `desktop` matrix has no Linux row at all. The image gains the SDK the routed Android steps pin: cmdline-tools 16.0 with licenses accepted, platform-tools, platforms 35 and 36 (compileSdk), build-tools 35.0.0 and NDK 28.2.13676358 (ndkVersion), owned by `runner` because sdkmanager writes into ANDROID_HOME at job time. `emulator` and the system image stay out: android-emulator-runner installs those itself. 9.28 GB against the pool image's 5.25 GB, chowned inside the install layer so the SDK is not written twice. Heavy slots pass `--device /dev/kvm` where the host has one, and the entrypoint sets the node's mode before the job reads it: docker creates it 0660 root:kvm, which android-emulator-runner cannot open. That also means the lane's `udevadm control`/`udevadm trigger` lines can never succeed in a container — no udev, read-only /sys — so they are now `|| true`, which is what the step's own "report the mode, never assert it" rule already says. The spec compares the two emulator lanes' provisioning with that suffix normalised. Evidence: `docker build -t tn-ci-runner:phase3 tools/ci-runners` succeeded, and `docker run --rm --device /dev/kvm --entrypoint bash tn-ci-runner:phase3 -c 'sdkmanager --version && ls $ANDROID_HOME && ls -l /dev/kvm'` printed sdkmanager 16.0 with build-tools, cmdline-tools, licenses, ndk, platform-tools and platforms. `emulator -accel-check` inside the container reports "KVM (version 12) is installed and usable" with no missing shared libraries. `bash -n scripts/ci-runners.sh` passes. `pnpm exec vitest run scripts/__tests__/ci-structure.spec.ts scripts/__tests__/ci-needs.spec.ts` — 142 passed; red first on `.github/workflows/native-platforms.yml scope runs on ubuntu-latest`, and again on a `starter-linux/linux-x64` row put back to bare `ubuntu-24.04`. The six sibling workflow specs (ci-efficiency, ci-local, compiler-cache-setup, hosted-software-callers, workspace-dist-cache, native-release-proof) pass at 149. `pnpm lint` exits 0, `pnpm check:docs` checks 2496 links. Not run here and not claimed: a live `native-platforms` run on the pool, which is what the two phase boxes still wait for. `pnpm typecheck` and `pnpm budgets` fail in this worktree on missing `packages/*/dist` build output, in files this change does not touch.
… TN_RUNNER is set Run 37070815769: 39 jobs green, 33 on tn-local, 5 on tn-local-light, supply-chain hosted. 38 distinct runner names, so each job got a fresh container. ci-required started 3 s after its last need. AC-2 measured and not met (317 min summed wait, 42 min wall vs 39 hosted); owner decision recorded as pending.
jonit-dev
added
prd:50%
Run 37074908699: the emulator lane failed writing its udev rule because the image had no /etc/udev/rules.d; and starter-desktop.test.mjs pinned a bare ubuntu-24.04 for the linux-x64 row that now routes through TN_RUNNER. Census regenerated for the runtime-native test edit.
Run 37079046851 and the develop promotion both failed Windows desktop core on "nsis not installed. The package was not found" for a version the feed still lists. Three attempts, 30 s apart; makensis -VERSION still fails closed.
The five integration-*.yml files folded into integration.yml on develop; its paths gate takes the light lane and every other Linux job the TN_RUNNER switch. lint keeps develop's reused-tree condition with the routing expression. The routing spec now reads integration.yml. PRD-481 shard box ticked from run 37078784853 (worst leg 329 s).
jonit-dev
enabled auto-merge
October 3, 2026 00:37
On tn-local the emulator got KVM but renders through SwiftShader on the CPU; run 37082733117 spent 41 of its 45 minutes still running APKs (hosted ~30 min). The hosted allow-list names it with that reason; every other Linux leg still routes.
jonit-dev
added a commit
that referenced
this pull request
Oct 3, 2026
…use, image deps, PRD close-out) (#409) * ci(prd-380): an ordinary pull request runs the Linux native rows only The native matrix is one of the three multipliers that starves the runner pool: ci.yml ran its full macOS/Windows/iOS/arm64 legs for any `selection == 'full'` pull request. Phase 2 makes the scope planner decide how much of that matrix a run owes, once, and the workflow read the answer. The plan gains `nativeTier` (`full` | `reduced` | `none`), computed in `selectionPlan` from whether the change reached native code and the branch it targets, recomputed and compared by `validatePlan`, and published as the `native_tier` output. `native-platforms.yml` consumes it: `desktop` (macOS, Windows) and `ios-simulator` skip unless the tier is `full`, and `starter-linux` runs `linux-x64` alone because a job's `if` cannot read `matrix`, so the `scope` job shapes the rows instead and an unreadable tier keeps every row. `web-reference`, `android-v8-source`, `android-emulator-parity` and `desktop-parity` are Linux rows and still run on an ordinary pull request. A `main` push, the nightly, a `workflow_dispatch` audit and any merge group or pull request into `main` classify `full`; ci.yml now passes the merge group's own base branch, which is what tells those two apart. PRD-481's reuse profile follows: a reused plan keeps the native reach of the full plan it stands in for, and `ci-required` records the tier and demands it of the source run instead of blanket-exempting `native-platforms` on develop, so a reduced pass can never satisfy a full requirement — nor a skipped lane satisfy a reduced one. No label exempts the matrix. Evidence (2026-10-02, red before green): - scripts/__tests__/ci-efficiency.spec.ts + ci-structure + ci-needs: 216 passed; 2 new ci-efficiency assertions and 4 ci-needs cases failed against the pre-change files. - packages/runtime-native tests/native-platform-workflow + starter-desktop: 79 passed. The arm64 matrix rows moved from the job body into the `scope` step that shapes them, and that test's row assertions follow. - pnpm lint, pnpm typecheck, pnpm check:docs, pnpm census: clean. * docs(prd-480): phase 3 proven by run 37089715252 Every native-platforms leg green; Linux legs on tn-local and tn-local-light. * ci: the janitor also cancels a dequeued PR's merge-group runs Dequeueing, admin-merging or closing a pull request drops it from the develop merge queue but does not cancel the merge-group run GitHub already started, so that run keeps holding self-hosted runners for a verdict nothing will read: run 37093098790 ran 9 jobs for #404 after #404 had merged. The janitor only swept the head ref, which never names that branch. It now also cancels queued and in-progress runs whose branch starts with `gh-readonly-queue/<base>/pr-<number>-`, built from the event's own base ref and PR number through `env:` — no `github.*` context reaches the shell — and filtered in jq off one `gh run list --json databaseId,status,headBranch --limit 100`. The fork, develop/main-head and self-run guards are unchanged. Evidence: - `pnpm exec vitest run scripts/__tests__/ci-structure.spec.ts scripts/__tests__/ci-efficiency.spec.ts` — the janitor case now pins the env-built prefix and the `startswith` filter: red on the old workflow (1 failed | 134 skipped), green on the new one (182 passed). - `pnpm lint` clean. - The step's shell, extracted and run against a stub `gh`, cancelled the head ref's in-progress run plus both of PR 404's queued/in-progress merge-group runs, and spared a completed run, a `main`-based group, another PR's group, `pr-4040` and the janitor's own run id. * fix(ci): tree reuse compares runner class per job, so mixed boards can reuse currentRun() demanded one runner class for the whole run and returned a miss otherwise, but every board is mixed on purpose: supply-chain is hosted, the small joins run on tn-local-light and the platform legs are hosted. Reuse could therefore never trigger. currentRun() now returns the class each job routes to, read from that job's own labels (known before the job starts), and coverageMiss() compares the source's class for each required job against it. A job whose class is not assigned yet — a reuse run skips every leg it replaces — falls back to the class most of the run's known jobs report, so an unprovable leg is a miss unless it agrees with that routing. Profile coverage, matrix legs and success-only are unchanged. Spec: the mixed-board case now reuses, and a source whose `test` ran hosted while this run routes it to tn-local runs full naming the job. * docs(prd): CI PRD status lines match what is verified * docs(prd-480): the fallback runs fully hosted (dispatch run 37093694594) * ci(runners): bake the native jobs' build packages into the image test-native spent 52 s per run on apt-get install (run 37089715252); the -dev packages and xcompmgr now ship in the image. Mesa stays per-job. * docs(prd-380): phase 3 proven by live janitor runs * ci(runners): three light slots so small joins stop queueing behind one runner * ci(janitor): force-cancel, since a plain cancel can hang on a job queued for a self-hosted label * ci(runners): a balancer makes the local pool overflow capacity on top of free hosted runners * ci(runners): six 3-thread slots instead of five 4-thread ones, same reserved desktop cores * ci(scope): a CI-configuration-only pull request runs the `ci` selection A workflow-only PR to develop spent the whole board — test-native, both browser and playtest lanes, the 28-leg template sweep, golden-path, benchmark, the native matrix — to prove that a YAML file still parses. None of those jobs has a consumer for CI configuration. `ci` keeps exactly the gates that read it: typecheck, lint, budgets, the secrets scan, and the test-unit shards, which are where every `ci-*.spec.ts` runs. build-artifacts comes with them because those three jobs consume its upload; dropping it would leave three required jobs waiting on a producer that never ran. Qualifying paths are `.github/workflows/*.yml`, `scripts/ci-*.{mjs,sh,ts}`, `scripts/__tests__/ci-*.spec.ts`, `tools/ci-runners/**`, and anything already classified prose or instructions. `.github/actions/**` is excluded because the native matrix consumes it, and a lockfile because it changes what every job installs. A diff touching any native path keeps `full` — that includes `native-platforms.yml`, which is the matrix's own definition, so narrowing it would waive the lane that has to prove the change. Never for the merge queue, a push, the nightly, an explicit audit or a promotion into main. Those qualify a tree rather than review a change, and the queue already runs the full board before anything lands. Reuse needed no new guard: `coverageMiss` already refuses a source on the matrix legs it skipped, and a new test in ci-needs.spec.ts proves a `ci`-lane run cannot stand in for a full requirement. `classify`'s complexity is unchanged from its pre-existing warning — the family loop moved to `diffSelection` rather than pushing it further over. `pnpm ci:local --affected` accepted only full/prose/instructions, so a CI-configuration branch failed closed on an unknown selection; it now accepts `ci` and runs the same bounded docs lane as the other narrowed selections. * ci(janitor): a CI run that failed only on known flakes reruns its failed jobs once workflow_run on CI completion: when every failed primary job shows a known environmental signature (SwiftShader device lost, bridge budget, adapter provenance, dev server not up, 5xx, runner shutdown) and it is attempt 1, rerun only the failed jobs. Any other failure stays red. Takes effect from main. * ci: template legs use the balanced routing, so idle local slots run them too * fix(ci): starter-linux matrix rows are {include:[...]}, not a bare array Run 37100382876: the native scope step wrote the rows as an array, GitHub created no starter-linux job, and the reusable workflow still concluded failure, so ci-required read a red native lane with every leg green. --------- Co-authored-by: Joao Paulo Furtado <jfurtado141@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR: Linux CI jobs run on ephemeral
tn-localrunners on the owner's machine when theTN_RUNNERrepo variable is set, and fall back to hosted runners when it is deleted or the PR comes from a fork. PRD-480.Why
Run 37043413533 spent 239 min waiting for runners against 146 min of work. A local pool removes the queue.
What changed
tools/ci-runners/(Dockerfile + entrypoint): Ubuntu 24.04 actions-runner image with the build/Xvfb/Playwright deps. The entrypoint mints a registration token, unsets the admin token beforerun.shso no job sees it, and registers--ephemeralwith labeltn-local.scripts/ci-runners.sh up [N] | down | status: one host-sidedocker run --rmloop per slot (a fresh container and hostname for every job).upsetsTN_RUNNERonly after N runners are online;downdeletes it first. It fails closed when the env file or token is missing.ci.ymlandintegration-*.yml: every movable Linuxruns-onuses the routing expression.supply-chainstays hosted; arm64 andnative-platforms.ymlare untouched.ci-structure.spec.tsenforces the routing.Deviation from the PRD: no
compose.yml.restart: alwaysrestarts the same container, so its filesystem and hostname would carry over between jobs.Evidence
scripts/ci-runners.sh up: 4 online;statuslists 4tn-localrunners online.ci-structure.spec.ts+ci-needs.spec.ts: 142 passed. The new spec failed withbudgetsset back toubuntu-latest.Checklist
tn-local(AC-1)TN_RUNNERunset