From 2c1ed5f83a3f0d6ea29b7025c4d354af0ce35614 Mon Sep 17 00:00:00 2001 From: Joao Paulo Furtado Date: Fri, 2 Oct 2026 13:12:33 -0700 Subject: [PATCH 01/18] ci(prd-480): the tn-local runner stack, without compose MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 1 of PRD-480: the image, its entrypoint, and the host-side lifecycle script. No runners are started and nothing is registered yet — the script fails closed without a token, and `status` reports an empty pool. No compose.yml (owner-approved deviation, noted in the PRD): a host-side loop of `docker run --rm` is what makes each job a fresh container filesystem and hostname, which `restart: always` on one long-lived container cannot. Evidence: - `docker build -t tn-ci-runner tools/ci-runners` — clean. - smoke: cmake 3.28.3, clang 18.1.3, ninja 1.11.1, /usr/bin/{Xvfb,ccache,jq}, `ls -A /tmp | wc -l` → 0. - `TN_RUNNERS_ENV=/nonexistent scripts/ci-runners.sh up` → exit 2, nothing built or started. `up 0` and an unknown subcommand also exit 2. - `docker run --rm tn-ci-runner` without the env file → exit 1 naming RUNNER_ADMIN_TOKEN. - `scripts/ci-runners.sh status` → TN_RUNNER=unset, 0 containers, 0 online. - `bash -n` clean on both scripts (no shellcheck installed here). --- ...-480-linux-ci-runs-on-the-owner-machine.md | 27 ++- scripts/ci-runners.sh | 164 ++++++++++++++++++ tools/ci-runners/Dockerfile | 35 ++++ tools/ci-runners/entrypoint.sh | 35 ++++ 4 files changed, 252 insertions(+), 9 deletions(-) create mode 100755 scripts/ci-runners.sh create mode 100644 tools/ci-runners/Dockerfile create mode 100755 tools/ci-runners/entrypoint.sh diff --git a/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md b/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md index 11faae312d..7238c69d28 100644 --- a/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md +++ b/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md @@ -128,20 +128,29 @@ while `TN_RUNNER` is set: jobs queue until the switch is cleared. ## Execution Phases #### Phase 1: The runner stack comes up and takes jobs - -**Status:** NOT STARTED -**Files:** NEW `tools/ci-runners/Dockerfile`, NEW `tools/ci-runners/compose.yml`, NEW +**Status:** IN PROGRESS — stack files built, runners not yet brought up +**Files:** NEW `tools/ci-runners/Dockerfile`, NEW `tools/ci-runners/entrypoint.sh`, NEW `scripts/ci-runners.sh` (`up [N]`, `down`, `status`; default N = 4, read from the env file). -**Implementation:** Ubuntu 24.04 runner image with the tool set above. Compose service with -`replicas: N`, `cpus: 4`, `mem_limit: 16g`, `restart: always`, and ephemeral repo-scoped registration -with labels `tn-local`. `up` waits for N online runners via `gh api repos/{owner}/{repo}/actions/runners` -before `gh variable set TN_RUNNER --body tn-local`. `down` deletes the variable first, then stops the -stack. The script fails closed when the env file or token is missing. + +Deviation from the plan below, owner-approved 2026-10-02: **no compose.yml.** A host-side loop of +`docker run --rm` replaces the compose service, because `restart: always` restarts the *same* +container — same filesystem, same hostname — and the ephemeral claim here is a fresh one of both +per job. + +**Implementation:** Ubuntu 24.04 runner image with the tool set above. One loop per slot +(`setsid nohup`), `cpus 4`, `memory 16g`, and ephemeral repo-scoped registration with labels +`tn-local`. `up` waits for N online runners via `gh api repos/{owner}/{repo}/actions/runners` +before `gh variable set TN_RUNNER --body tn-local`. `down` deletes the variable first, then stops +the stack. The script fails closed when the env file or token is missing. + - [ ] `scripts/ci-runners.sh up` brings 4 runners online with label `tn-local`. proof: - `scripts/ci-runners.sh status` lists 4 online runners. + `scripts/ci-runners.sh status` lists 4 online runners. Not started: `status` reports + `TN_RUNNER=unset`, 0 containers, 0 online runners on 2026-10-02. - [ ] A runner container recreates itself after its job and keeps no state from it. proof: two consecutive dispatched jobs report different container hostnames and an empty `/tmp`. + Not started: the image's `/tmp` is empty in a fresh container (`ls -A /tmp | wc -l` → 0), which + is one container's worth of the claim. #### Phase 2: `ci.yml` and the integration workflows route through the switch diff --git a/scripts/ci-runners.sh b/scripts/ci-runners.sh new file mode 100755 index 0000000000..b8a0dc975c --- /dev/null +++ b/scripts/ci-runners.sh @@ -0,0 +1,164 @@ +#!/usr/bin/env bash +# The `tn-local` self-hosted runner pool (PRD-480). +# +# scripts/ci-runners.sh up [N] build the image, run N disposable runner containers, then set +# the TN_RUNNER repository variable so the routing expression in +# ci.yml and integration-*.yml sends Linux jobs here +# scripts/ci-runners.sh down clear TN_RUNNER first, then stop everything +# scripts/ci-runners.sh status what is set, what is running, what is online +# +# `down` deletes the variable before it stops anything, so the kill switch is the same command as +# the teardown: jobs go back to hosted runners the moment it is deleted, whether or not the +# containers are still there. +# +# The operator's token lives in an untracked env file outside the repository and is never printed. +# It reaches the containers only through `docker run --env-file`, and the container's entrypoint +# unsets it before `run.sh` starts, so no job ever sees it. +set -euo pipefail +cd "$(dirname -- "${BASH_SOURCE[0]}")/.." + +IMAGE=tn-ci-runner +LABEL=tn-local +VARIABLE=TN_RUNNER +ENV_FILE="${TN_RUNNERS_ENV:-${XDG_CONFIG_HOME:-$HOME/.config}/threenative/runners.env}" +STATE_DIR="${XDG_STATE_HOME:-$HOME/.local/state}/threenative/ci-runners" +DEFAULT_SLOTS=4 +ONLINE_TIMEOUT_SECONDS=300 + +fail() { + local code="$1" + shift + printf 'TN_CI_RUNNERS: %s\n' "$*" >&2 + exit "$code" +} + +need() { + command -v "$1" >/dev/null 2>&1 || fail 2 "$1 is not on PATH" +} + +# Fail closed before anything is built or started. A pool that came up without a token would +# register nothing and leave TN_RUNNER set for hosted jobs that never arrive. +load_config() { + [ -n "$ENV_FILE" ] || fail 2 "TN_RUNNERS_ENV is empty" + [ -f "$ENV_FILE" ] || fail 2 "no runner env file at $ENV_FILE; set TN_RUNNERS_ENV or create it with RUNNER_ADMIN_TOKEN=..." + set -a + # shellcheck disable=SC1090 # the operator's own untracked file + . "$ENV_FILE" + set +a + # Read once for validation, then drop it: the token travels to the containers through + # `--env-file`, never through this process's environment. + [ -n "${RUNNER_ADMIN_TOKEN:-}" ] || fail 2 "RUNNER_ADMIN_TOKEN is unset or empty in $ENV_FILE" + unset RUNNER_ADMIN_TOKEN +} + +repo_name() { + gh repo view --json nameWithOwner --jq .nameWithOwner 2>/dev/null \ + || fail 2 "gh could not resolve this checkout's owner/repo" +} + +# One slot is a host-side loop, not a compose service: `docker run --rm` gives each job a fresh +# container filesystem and a fresh hostname, which `restart: always` on a long-lived container +# cannot. The loop is what makes the pool survive the exit. +start_slot() { + local slot="$1" state="$2" env_file="$3" repo="$4" + setsid nohup bash -c ' + while [ ! -e "$1/stop" ]; do + docker run --rm \ + --label tn-ci-runner=1 \ + --cpus 4 --memory 16g \ + --env-file "$2" \ + -e TN_RUNNER_REPO="$3" \ + tn-ci-runner || sleep 10 + done + ' _ "$state" "$env_file" "$repo" >"$state/slot-$slot.log" 2>&1 & + echo "$!" >"$state/slot-$slot.pid" +} + +stop_slots() { + local pid_file + for pid_file in "$STATE_DIR"/slot-*.pid; do + [ -f "$pid_file" ] || continue + kill "$(cat "$pid_file")" 2>/dev/null || true + done + # No `rm -rf` anywhere near a path built from a variable: the stop file is what stops the loops. + : >"$STATE_DIR/stop" + docker ps -q --filter label=tn-ci-runner=1 | xargs -r docker stop +} + +online_count() { + gh api "repos/$1/actions/runners" --paginate --jq ' + [.runners[] | select(.status == "online") | select(any(.labels[]; .name == "tn-local"))] | length + ' 2>/dev/null || echo 0 +} + +up() { + local slots="${1:-${TN_RUNNERS:-$DEFAULT_SLOTS}}" + case "$slots" in ''|*[!0-9]*) fail 2 "N must be a positive integer, got '$slots'" ;; esac + [ "$slots" -ge 1 ] || fail 2 "N must be at least 1, got '$slots'" + need docker + need gh + load_config + local repo + repo="$(repo_name)" + + docker build -t "$IMAGE" tools/ci-runners + mkdir -p "$STATE_DIR" + # A leftover stop file from a previous teardown would make every loop exit before it starts. + rm -f "$STATE_DIR/stop" + + local slot + for slot in $(seq 1 "$slots"); do + start_slot "$slot" "$STATE_DIR" "$ENV_FILE" "$repo" + done + printf '%s runner container(s) starting; logs in %s\n' "$slots" "$STATE_DIR" + + # Only advertise the label once the runners are really there: TN_RUNNER set with an empty pool + # is a queue that never drains. On timeout, take the whole thing back down rather than leaving + # half a pool registered. + local deadline=$(( $(date +%s) + ONLINE_TIMEOUT_SECONDS )) online + while :; do + online="$(online_count "$repo")" + [ "$online" -ge "$slots" ] && break + if [ "$(date +%s)" -ge "$deadline" ]; then + gh variable delete "$VARIABLE" --repo "$repo" 2>/dev/null || true + stop_slots + fail 1 "$online of $slots $LABEL runner(s) came online within ${ONLINE_TIMEOUT_SECONDS}s" + fi + sleep 10 + done + + gh variable set "$VARIABLE" --body "$LABEL" --repo "$repo" + printf '%s online; %s=%s, so Linux jobs route here and deleting it routes them back\n' \ + "$online" "$VARIABLE" "$LABEL" +} + +down() { + need gh + local repo + repo="$(repo_name)" + # First, so no job is ever routed to a pool that is on its way out. + gh variable delete "$VARIABLE" --repo "$repo" 2>/dev/null \ + || printf 'TN_CI_RUNNERS: %s was already unset\n' "$VARIABLE" >&2 + mkdir -p "$STATE_DIR" + stop_slots + printf '%s cleared; runners stopping\n' "$VARIABLE" +} + +status() { + need gh + local repo + repo="$(repo_name)" + printf '%s=%s\n' "$VARIABLE" "$(gh variable get "$VARIABLE" --repo "$repo" 2>/dev/null || echo unset)" + printf 'containers: %s\n' "$(docker ps -q --filter label=tn-ci-runner=1 | wc -l | tr -d ' ')" + printf 'online %s runners (name, status, busy):\n' "$LABEL" + gh api "repos/$repo/actions/runners" --paginate --jq \ + ".runners[] | select(any(.labels[]; .name == \"$LABEL\")) | \" \\(.name) \\(.status) busy=\\(.busy)\"" \ + 2>/dev/null || echo " none reported" +} + +case "${1:-}" in + up) shift; up "${1:-}" ;; + down) down ;; + status) status ;; + *) fail 2 "usage: ci-runners.sh up [N] | down | status" ;; +esac diff --git a/tools/ci-runners/Dockerfile b/tools/ci-runners/Dockerfile new file mode 100644 index 0000000000..75213d3297 --- /dev/null +++ b/tools/ci-runners/Dockerfile @@ -0,0 +1,35 @@ +# The `tn-local` self-hosted runner image (PRD-480). One container per job: `scripts/ci-runners.sh` +# starts a host-side loop of `docker run --rm`, so nothing survives a job but the image itself. +# +# Ubuntu 24.04 to match hosted `ubuntu-24.04`, with the `runner` user's passwordless sudo intact so +# the workflow's own `sudo apt-get install` steps keep working. The Docker CLI is deliberately +# absent and the socket is never mounted: `supply-chain` needs `docker run` for gitleaks and stays +# on a hosted runner for exactly that reason. +FROM ghcr.io/actions/actions-runner:latest + +USER root + +# Build and test toolchain the moving Linux jobs take from the hosted image. `git`, `curl`, `jq` +# and `unzip` are the runner's own prerequisites; the rest is what ci.yml's jobs actually invoke. +# Playwright's Chromium dependencies are the exact `tools` and `chromium` lists Playwright 1.62.1 +# resolves for ubuntu24.04-x64 (playwright-core's bundled nativeDeps), transcribed rather than +# installed with `npx playwright install-deps` — that needs a Node runtime the runner image does +# not ship and every job gets its own through `actions/setup-node`. +RUN apt-get update \ + && apt-get install -y --no-install-recommends \ + build-essential cmake ninja-build ccache clang lld pkg-config \ + git curl jq unzip zip xz-utils xvfb xauth python3 python3-pip \ + fonts-noto-color-emoji fonts-unifont fonts-liberation fonts-freefont-ttf \ + libfontconfig1 libfreetype6 \ + libasound2t64 libatk-bridge2.0-0t64 libatk1.0-0t64 libatspi2.0-0t64 \ + libcairo2 libcups2t64 libdbus-1-3 libdrm2 libgbm1 libglib2.0-0t64 \ + libnspr4 libnss3 libpango-1.0-0 \ + libx11-6 libxcb1 libxcomposite1 libxdamage1 libxext6 libxfixes3 libxkbcommon0 libxrandr2 \ + && rm -rf /var/lib/apt/lists/* + +COPY entrypoint.sh /actions-runner/entrypoint.sh +RUN chmod 0755 /actions-runner/entrypoint.sh + +USER runner +WORKDIR /actions-runner +ENTRYPOINT ["/actions-runner/entrypoint.sh"] diff --git a/tools/ci-runners/entrypoint.sh b/tools/ci-runners/entrypoint.sh new file mode 100755 index 0000000000..1836542479 --- /dev/null +++ b/tools/ci-runners/entrypoint.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# Registers this container as an ephemeral `tn-local` runner and takes one job. +# +# The container is the disposable half of `scripts/ci-runners.sh up`: it configures, runs exactly +# one job, exits, and the host loop starts a fresh one. Nothing on the filesystem crosses jobs. +set -euo pipefail + +# Fail closed and say which variable is missing. An ephemeral runner that registers without +# TN_RUNNER_REPO would point itself at whatever repository the token defaults to. +: "${RUNNER_ADMIN_TOKEN:?RUNNER_ADMIN_TOKEN is not set; pass the operator env file with --env-file}" +: "${TN_RUNNER_REPO:?TN_RUNNER_REPO is not set; the host loop passes owner/repo}" + +registration_token="$( + curl --fail --silent --show-error --request POST \ + --header "Authorization: Bearer $RUNNER_ADMIN_TOKEN" \ + --header "Accept: application/vnd.github+json" \ + "https://api.github.com/repos/${TN_RUNNER_REPO}/actions/runners/registration-token" \ + | jq --exit-status --raw-output '.token' +)" + +# The admin token mints this one registration token and has no other business existing. Dropping it +# from this shell's environment is what keeps it out of every job `run.sh` hands to a workflow. +unset RUNNER_ADMIN_TOKEN + +./config.sh \ + --unattended \ + --ephemeral \ + --replace \ + --url "https://github.com/${TN_RUNNER_REPO}" \ + --token "$registration_token" \ + --name "tn-$(hostname)" \ + --labels tn-local \ + --no-default-labels + +exec ./run.sh From 88b160615b18c90a648c6890472fa66fe2fb1834 Mon Sep 17 00:00:00 2001 From: Joao Paulo Furtado Date: Fri, 2 Oct 2026 13:20:48 -0700 Subject: [PATCH 02/18] ci(prd-480): route every movable Linux job through the TN_RUNNER switch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 2 of PRD-480. 23 `runs-on` lines (18 in ci.yml, one each in the five integration-*.yml) now read: runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} Deleting the TN_RUNNER repository variable is the whole kill switch, and a fork PR always runs hosted. `supply-chain` keeps ubuntu-latest because gitleaks needs `docker run` and the socket would hand every job root on the host; integration-decals.yml's ubuntu-24.04-arm job and every macOS/Windows job are untouched, as is native-platforms.yml (Phase 3). A new ci-structure.spec.ts test enforces it: every job in ci.yml and the integration workflows either carries that exact expression, is on the hosted allow-list, or is a non-Linux runner. Evidence: - red, no workflow edited: "ci.yml scope runs on `runs-on: ubuntu-latest`". - red, after routing, ci.yml:1051 (budgets) put back on ubuntu-latest: same test failed naming that job, then it was restored. - green: `pnpm exec vitest run scripts/__tests__/ci-structure.spec.ts scripts/__tests__/ci-needs.spec.ts` — 142 passed, 0 failed. - `pnpm lint` exit 0, `pnpm check:docs` 2496 links, `pnpm sync:agents --check` 22 mirrors. - all six workflows parse as YAML and every runs-on resolves to one plain scalar. One pre-existing assertion was narrowed rather than deleted: template-nonvisual forbade the substring `pull_request` anywhere in the job, which the routing expression mentions in every job without gating on it. It now forbids an `if:` gate naming pull_request. --- .github/workflows/ci.yml | 36 ++++++++-------- .github/workflows/integration-csg.yml | 2 +- .github/workflows/integration-decals.yml | 2 +- .github/workflows/integration-ik.yml | 2 +- .github/workflows/integration-tone.yml | 2 +- .github/workflows/integration-vegetation.yml | 2 +- ...-480-linux-ci-runs-on-the-owner-machine.md | 16 +++++-- scripts/__tests__/ci-structure.spec.ts | 43 ++++++++++++++++++- 8 files changed, 77 insertions(+), 28 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index bb4d8c0bdd..1eef8683ae 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -29,7 +29,7 @@ jobs: scope: name: Change scope if: ${{ !github.event.pull_request.draft }} - runs-on: ubuntu-latest + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 5 outputs: scope: ${{ steps.classify.outputs.scope }} @@ -81,7 +81,7 @@ jobs: typecheck: needs: scope if: ${{ needs.scope.outputs.selection == 'full' }} - runs-on: ubuntu-latest + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 15 env: TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} @@ -106,7 +106,7 @@ jobs: # A Markdown-only change runs no CI job: the docs lane is re-validated on the nightly run and # at promotion, so this job is skipped rather than spending a runner on inert prose. if: needs.scope.outputs.selection != 'prose' - runs-on: ubuntu-latest + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 10 env: TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} @@ -162,7 +162,7 @@ jobs: test: needs: scope if: ${{ needs.scope.outputs.selection == 'full' }} - runs-on: ubuntu-24.04 + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 45 env: TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} @@ -198,7 +198,7 @@ jobs: test-unit: needs: scope if: ${{ needs.scope.outputs.selection == 'full' }} - runs-on: ubuntu-24.04 + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 30 strategy: fail-fast: false @@ -233,7 +233,7 @@ jobs: test-native: needs: scope if: ${{ needs.scope.outputs.selection == 'full' }} - runs-on: ubuntu-24.04 + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 75 env: TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} @@ -497,7 +497,7 @@ jobs: test-browser: needs: scope if: ${{ needs.scope.outputs.selection == 'full' }} - runs-on: ubuntu-latest + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 30 env: TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} @@ -548,7 +548,7 @@ jobs: test-playtest: needs: scope if: ${{ needs.scope.outputs.selection == 'full' }} - runs-on: ubuntu-latest + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 30 env: TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} @@ -586,7 +586,7 @@ jobs: template: - starter - platformer - runs-on: ubuntu-latest + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} # The scenario arithmetic this number came from moved out to `template-nonvisual`, which now # owns the non-visual sweep for all eight templates. What is left is `verify:golden-path` — # pack, scaffold, install, dev, test, build, assert, once per matrix leg — which measured 309s @@ -747,7 +747,7 @@ jobs: - { template: starter, shard: "2/3" } - { template: starter, shard: "3/3" } - { template: tower-defense, shard: "1/1" } - runs-on: ubuntu-latest + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 30 env: TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} @@ -846,7 +846,7 @@ jobs: # as cancelled, so the skip is correct there; it still cannot be mistaken for a pass, because a # cancelled run has no passing checks to merge on. if: ${{ !cancelled() && needs.scope.outputs.selection == 'full' }} - runs-on: ubuntu-latest + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 5 env: TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} @@ -865,7 +865,7 @@ jobs: benchmark: needs: scope if: ${{ needs.scope.outputs.selection == 'full' }} - runs-on: ubuntu-latest + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 25 env: TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} @@ -916,7 +916,7 @@ jobs: build: needs: [scope, build-artifacts] if: ${{ !cancelled() && needs.scope.outputs.selection == 'full' }} - runs-on: ubuntu-latest + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 2 env: TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} @@ -938,7 +938,7 @@ jobs: build-artifacts: needs: scope if: ${{ needs.scope.outputs.selection == 'full' }} - runs-on: ubuntu-latest + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 20 env: TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} @@ -982,7 +982,7 @@ jobs: performance-contracts: needs: scope if: ${{ needs.scope.outputs.selection == 'full' && !cancelled() }} - runs-on: ubuntu-latest + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 2 env: TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} @@ -1048,7 +1048,7 @@ jobs: budgets: needs: scope if: ${{ needs.scope.outputs.selection == 'full' }} - runs-on: ubuntu-latest + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 15 env: TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} @@ -1170,7 +1170,7 @@ jobs: ci-required: needs: [scope, native-platforms, typecheck, lint, test, test-unit, test-native, test-browser, test-playtest, golden-path-template, template-nonvisual, golden-path, benchmark, build, build-artifacts, budgets, performance-contracts, supply-chain] if: ${{ always() && !github.event.pull_request.draft }} - runs-on: ubuntu-latest + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 5 steps: - uses: actions/checkout@v7 @@ -1224,7 +1224,7 @@ jobs: # `!cancelled()` and not `always()`, for the reason golden-path gives above: a cancelled run # has nothing to report and its summary would only describe the cancellation. if: ${{ !cancelled() && needs.scope.outputs.selection == 'full' }} - runs-on: ubuntu-latest + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 5 permissions: contents: read diff --git a/.github/workflows/integration-csg.yml b/.github/workflows/integration-csg.yml index 9237afd587..f7ca56b1ff 100644 --- a/.github/workflows/integration-csg.yml +++ b/.github/workflows/integration-csg.yml @@ -15,7 +15,7 @@ concurrency: cancel-in-progress: true jobs: test: - runs-on: ubuntu-latest + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 10 defaults: run: diff --git a/.github/workflows/integration-decals.yml b/.github/workflows/integration-decals.yml index 35011a3f2a..79012eb065 100644 --- a/.github/workflows/integration-decals.yml +++ b/.github/workflows/integration-decals.yml @@ -16,7 +16,7 @@ concurrency: cancel-in-progress: true jobs: decals: - runs-on: ubuntu-24.04 + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 30 steps: - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 diff --git a/.github/workflows/integration-ik.yml b/.github/workflows/integration-ik.yml index 39f70a8622..967a83f74d 100644 --- a/.github/workflows/integration-ik.yml +++ b/.github/workflows/integration-ik.yml @@ -8,7 +8,7 @@ permissions: contents: read jobs: test: - runs-on: ubuntu-latest + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 10 defaults: run: diff --git a/.github/workflows/integration-tone.yml b/.github/workflows/integration-tone.yml index fc6715052a..3e805b1503 100644 --- a/.github/workflows/integration-tone.yml +++ b/.github/workflows/integration-tone.yml @@ -22,7 +22,7 @@ permissions: contents: read jobs: exposure: - runs-on: ubuntu-24.04 + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 15 steps: - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 diff --git a/.github/workflows/integration-vegetation.yml b/.github/workflows/integration-vegetation.yml index e1b2b9a31e..303bf7d42b 100644 --- a/.github/workflows/integration-vegetation.yml +++ b/.github/workflows/integration-vegetation.yml @@ -8,7 +8,7 @@ permissions: contents: read jobs: test: - runs-on: ubuntu-latest + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 10 defaults: run: diff --git a/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md b/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md index 7238c69d28..81a3bcce3d 100644 --- a/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md +++ b/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md @@ -154,17 +154,25 @@ the stack. The script fails closed when the env file or token is missing. #### Phase 2: `ci.yml` and the integration workflows route through the switch -**Status:** NOT STARTED +**Status:** IN PROGRESS **Files:** EDIT `.github/workflows/ci.yml`, `.github/workflows/integration-*.yml`, `scripts/__tests__/ci-structure.spec.ts`; the `integration-*.yml` template wherever agents copy it from; `AGENTS.md` ("Nearest lane first, CI last") plus its regenerated `CLAUDE.md` mirror. **Implementation:** Replace each movable Linux `runs-on` with the routing expression. `supply-chain` keeps `ubuntu-latest`. Extend `ci-structure.spec.ts` so every Linux job in these workflows either uses the exact expression or is on a named hosted allow-list (`supply-chain`), and no macOS, Windows or arm64 -job ever does. +job ever does. 23 lines routed (18 in `ci.yml`, one each in five `integration-*.yml`); `supply-chain` +and `integration-decals.yml`'s `ubuntu-24.04-arm` job are the only Linux `runs-on` left hosted, and +`native-platforms.yml` is untouched until Phase 3. + +- [x] Structure spec enforces the routing. proof: `pnpm exec vitest run scripts/__tests__/ci-structure.spec.ts scripts/__tests__/ci-needs.spec.ts` + — 142 passed, 0 failed. Red first: with no workflow edited the new test failed on + `.github/workflows/ci.yml scope runs on 'runs-on: ubuntu-latest'`; after routing, setting `budgets` + (ci.yml:1051) back to `ubuntu-latest` failed it again on that job and it was restored. Also green: + `pnpm lint` (exit 0), `pnpm check:docs` (2496 links), `pnpm sync:agents --check` (22 mirrors). + One pre-existing assertion had to be narrowed: `template-nonvisual` forbade the substring + `pull_request` anywhere in the job, which the routing expression mentions without gating on it. -- [ ] Structure spec enforces the routing. proof: `pnpm exec vitest run scripts/__tests__/ci-structure.spec.ts` - passes, and it fails on a job left at a bare `ubuntu-latest`. - [ ] With `TN_RUNNER` set, a PR run lands its Linux jobs on `tn-local`. proof: AC-1 run. - [ ] `AGENTS.md` says focused checks then push, not the full board, while `TN_RUNNER` is set. proof: `pnpm sync:agents --check` and `pnpm exec vitest run scripts/__tests__/sync-agent-docs.spec.ts` pass. diff --git a/scripts/__tests__/ci-structure.spec.ts b/scripts/__tests__/ci-structure.spec.ts index adfd98ad2a..79b4cf4661 100644 --- a/scripts/__tests__/ci-structure.spec.ts +++ b/scripts/__tests__/ci-structure.spec.ts @@ -915,6 +915,44 @@ describe("CI pipeline structure", () => { } }); + // PRD-480. Every Linux job that can move picks its runner from one expression, so the pool is + // changed by setting or deleting a repository variable and nothing else. A job left at a bare + // `ubuntu-latest` still runs, which is exactly why this has to be a spec: the pool comes up and + // the queue does not shrink, and nothing in a run says which half of the board ignored it. + const routing = + "${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }}"; + // `supply-chain` runs gitleaks through `docker run`; a self-hosted container would need the host + // Docker socket mounted to do that, which would hand every job root on the owner's machine. + const hosted = new Set(["supply-chain"]); + + it("routes every movable Linux job through the TN_RUNNER switch", async () => { + const directory = path.join(repo, ".github/workflows"); + const routed = [ + ".github/workflows/ci.yml", + ...(await readdir(directory)) + .filter((entry) => entry.startsWith("integration-") && entry.endsWith(".yml")) + .map((entry) => `.github/workflows/${entry}`), + ].sort(); + expect(routed.length).toBeGreaterThan(1); + for (const relative of routed) { + const source = await readFile(path.join(repo, relative), "utf8"); + for (const [job, section] of jobSections(source)) { + const runsOn = section.match(/^\s+runs-on:.*$/mu)?.[0].trim(); + if (runsOn === undefined) continue; + // macOS, Windows and arm64 have no tn-local counterpart, and Phase 3 moves the Linux legs + // of native-platforms.yml — so the expression must never appear on one of them here. + if (/macos|windows|arm/u.test(runsOn)) { + expect(runsOn, `${relative} ${job} is not Linux`).not.toContain(routing); + continue; + } + expect( + runsOn === `runs-on: ${routing}` || hosted.has(job), + `${relative} ${job} runs on \`${runsOn}\`: use the TN_RUNNER routing expression, or name it on the hosted allow-list`, + ).toBe(true); + } + } + }); + // A draft PR spends no runner: every job needs `scope`, so skipping it and the always() gate // skips the board, and `ready_for_review` starts it once the draft is marked ready. it("runs nothing on draft pull requests until they are marked ready", async () => { @@ -1146,7 +1184,10 @@ describe("CI pipeline structure", () => { // Runs on every event since 2026-09-01 (owner call): the PR skip reported nothing on the // branch where the regression was written, and the merge that shipped it reported too late. expect(job).not.toContain("github.event_name == 'push'"); - expect(job).not.toContain("pull_request"); + // An `if:` gate naming the event, not any mention of it: the TN_RUNNER routing expression + // (PRD-480) reads `github.event.pull_request.head.repo.fork` in every routed job and gates + // nothing on it. What this forbids is a gate that skips pull requests. + expect(job).not.toMatch(/^\s+if:.*pull_request/mu); expect(job).toContain('TN_PLAYTEST_ALLOW_SOFTWARE: "1"'); expect(job).toContain("non-visual-scenarios.mjs"); expect(job).toContain("threenative-playtest"); From a9b2489b1385969ed2a0bb6502844590faef761a Mon Sep 17 00:00:00 2001 From: Joao Paulo Furtado Date: Fri, 2 Oct 2026 13:27:21 -0700 Subject: [PATCH 03/18] fix(ci-runners): the runner image works from /home/runner, where config.sh lives First live up: 0 of 4 online, the pool tore itself down and cleared TN_RUNNER. With the fix, 4 tn-local runners come online. --- .../PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md | 7 +++++-- tools/ci-runners/Dockerfile | 8 ++++---- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md b/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md index 81a3bcce3d..ec0006a3d6 100644 --- a/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md +++ b/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md @@ -144,8 +144,11 @@ before `gh variable set TN_RUNNER --body tn-local`. `down` deletes the variable the stack. The script fails closed when the env file or token is missing. -- [ ] `scripts/ci-runners.sh up` brings 4 runners online with label `tn-local`. proof: - `scripts/ci-runners.sh status` lists 4 online runners. Not started: `status` reports +- [x] `scripts/ci-runners.sh up` brings 4 runners online with label `tn-local`. proof: + `scripts/ci-runners.sh status` lists 4 online runners. Evidence: 2026-10-02, `up` printed "4 online; + TN_RUNNER=tn-local" and `status` listed tn-23e70acfd553, tn-365ab3636a2e, tn-4d83d32a4c75 and + tn-6071a30e949e, all online. The first `up` timed out with 0 online (wrong WORKDIR), cleared the + variable and stopped the pool, which is the fail-closed path working. Not started: `status` reports `TN_RUNNER=unset`, 0 containers, 0 online runners on 2026-10-02. - [ ] A runner container recreates itself after its job and keeps no state from it. proof: two consecutive dispatched jobs report different container hostnames and an empty `/tmp`. diff --git a/tools/ci-runners/Dockerfile b/tools/ci-runners/Dockerfile index 75213d3297..de787ac538 100644 --- a/tools/ci-runners/Dockerfile +++ b/tools/ci-runners/Dockerfile @@ -27,9 +27,9 @@ RUN apt-get update \ libx11-6 libxcb1 libxcomposite1 libxdamage1 libxext6 libxfixes3 libxkbcommon0 libxrandr2 \ && rm -rf /var/lib/apt/lists/* -COPY entrypoint.sh /actions-runner/entrypoint.sh -RUN chmod 0755 /actions-runner/entrypoint.sh +COPY entrypoint.sh /home/runner/entrypoint.sh +RUN chmod 0755 /home/runner/entrypoint.sh USER runner -WORKDIR /actions-runner -ENTRYPOINT ["/actions-runner/entrypoint.sh"] +WORKDIR /home/runner +ENTRYPOINT ["/home/runner/entrypoint.sh"] From 1221df7460eca1da44cc63ffd2eaa537297b16ce Mon Sep 17 00:00:00 2001 From: Joao Paulo Furtado Date: Fri, 2 Oct 2026 13:33:24 -0700 Subject: [PATCH 04/18] fix(ci-runners): the image carries Node 20 and gh, as hosted ubuntu-24.04 does The scope job runs node straight after checkout; on tn-local it failed with "node: command not found" (run 37061064658). --- tools/ci-runners/Dockerfile | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/tools/ci-runners/Dockerfile b/tools/ci-runners/Dockerfile index de787ac538..2d0b7f29a4 100644 --- a/tools/ci-runners/Dockerfile +++ b/tools/ci-runners/Dockerfile @@ -27,6 +27,18 @@ RUN apt-get update \ libx11-6 libxcb1 libxcomposite1 libxdamage1 libxext6 libxfixes3 libxkbcommon0 libxrandr2 \ && rm -rf /var/lib/apt/lists/* +# Hosted ubuntu-24.04 ships Node 20 and gh on PATH, and jobs rely on both before any setup step: +# the `scope` job runs `node scripts/ci-change-scope.mjs` straight after checkout. +ARG NODE_VERSION=20.19.6 +RUN curl -fsSL "https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-x64.tar.xz" \ + | tar -xJ -C /usr/local --strip-components=1 --exclude=CHANGELOG.md --exclude=README.md --exclude=LICENSE \ + && curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg \ + -o /usr/share/keyrings/githubcli-archive-keyring.gpg \ + && echo "deb [arch=amd64 signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" \ + > /etc/apt/sources.list.d/github-cli.list \ + && apt-get update && apt-get install -y --no-install-recommends gh \ + && rm -rf /var/lib/apt/lists/* + COPY entrypoint.sh /home/runner/entrypoint.sh RUN chmod 0755 /home/runner/entrypoint.sh From 0630f6a3923d836b952e859f5a0d1fcfebdb39a3 Mon Sep 17 00:00:00 2001 From: Joao Paulo Furtado Date: Fri, 2 Oct 2026 13:44:35 -0700 Subject: [PATCH 05/18] =?UTF-8?q?fix(ci-runners):=20hosted=20parity=20?= =?UTF-8?q?=E2=80=94=20Rust=20and=20media=20tools,=20and=20CPUs=20pinned?= =?UTF-8?q?=20per=20slot?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR #404 run 2: test-native failed on "rustc: command not found"; three unit tests timed out because a --cpus quota leaves nproc at 24, so vitest ran 8 workers in a 4-core budget. Each slot now owns two whole cores (nproc = 4, like hosted), and the image adds rustup stable, 7zip, ffmpeg, imagemagick, zstd, file, lsof, procps and mesa-vulkan-drivers. --- scripts/ci-runners.sh | 20 +++++++++++++++----- tools/ci-runners/Dockerfile | 13 ++++++++++--- 2 files changed, 25 insertions(+), 8 deletions(-) diff --git a/scripts/ci-runners.sh b/scripts/ci-runners.sh index b8a0dc975c..d994e4e8b0 100755 --- a/scripts/ci-runners.sh +++ b/scripts/ci-runners.sh @@ -60,17 +60,17 @@ repo_name() { # container filesystem and a fresh hostname, which `restart: always` on a long-lived container # cannot. The loop is what makes the pool survive the exit. start_slot() { - local slot="$1" state="$2" env_file="$3" repo="$4" + local slot="$1" state="$2" env_file="$3" repo="$4" cpus="$5" setsid nohup bash -c ' while [ ! -e "$1/stop" ]; do docker run --rm \ --label tn-ci-runner=1 \ - --cpus 4 --memory 16g \ + --cpuset-cpus "$4" --memory 16g \ --env-file "$2" \ -e TN_RUNNER_REPO="$3" \ tn-ci-runner || sleep 10 done - ' _ "$state" "$env_file" "$repo" >"$state/slot-$slot.log" 2>&1 & + ' _ "$state" "$env_file" "$repo" "$cpus" >"$state/slot-$slot.log" 2>&1 & echo "$!" >"$state/slot-$slot.pid" } @@ -106,9 +106,19 @@ up() { # A leftover stop file from a previous teardown would make every loop exit before it starts. rm -f "$STATE_DIR/stop" - local slot + # Pinned CPUs, not a `--cpus 4` quota: under a quota `nproc` and os.availableParallelism() still + # report every host CPU, so vitest sized its pool for 24 cores inside a 4-core budget and three + # unit tests timed out (PR #404). Each slot takes two whole cores, both hyperthreads, as a hosted + # 4-vCPU runner does. + # ponytail: assumes Linux's usual sibling numbering (CPU k and k + nproc/2 share a core); read + # /sys/devices/system/cpu/cpu*/topology/thread_siblings_list if a host numbers them otherwise. + local half=$(( $(nproc) / 2 )) + [ $(( slots * 2 )) -le "$half" ] || fail 2 "$slots slots need $(( slots * 2 )) cores; this host has $half" + local slot base for slot in $(seq 1 "$slots"); do - start_slot "$slot" "$STATE_DIR" "$ENV_FILE" "$repo" + base=$(( (slot - 1) * 2 )) + start_slot "$slot" "$STATE_DIR" "$ENV_FILE" "$repo" \ + "$base,$(( base + 1 )),$(( base + half )),$(( base + half + 1 ))" done printf '%s runner container(s) starting; logs in %s\n' "$slots" "$STATE_DIR" diff --git a/tools/ci-runners/Dockerfile b/tools/ci-runners/Dockerfile index 2d0b7f29a4..8127651679 100644 --- a/tools/ci-runners/Dockerfile +++ b/tools/ci-runners/Dockerfile @@ -12,13 +12,13 @@ USER root # Build and test toolchain the moving Linux jobs take from the hosted image. `git`, `curl`, `jq` # and `unzip` are the runner's own prerequisites; the rest is what ci.yml's jobs actually invoke. # Playwright's Chromium dependencies are the exact `tools` and `chromium` lists Playwright 1.62.1 -# resolves for ubuntu24.04-x64 (playwright-core's bundled nativeDeps), transcribed rather than -# installed with `npx playwright install-deps` — that needs a Node runtime the runner image does -# not ship and every job gets its own through `actions/setup-node`. +# resolves for ubuntu24.04-x64 (playwright-core's bundled nativeDeps), transcribed so the apt layer +# does not depend on the Node layer below. RUN apt-get update \ && apt-get install -y --no-install-recommends \ build-essential cmake ninja-build ccache clang lld pkg-config \ git curl jq unzip zip xz-utils xvfb xauth python3 python3-pip \ + 7zip ffmpeg imagemagick zstd file lsof procps mesa-vulkan-drivers \ fonts-noto-color-emoji fonts-unifont fonts-liberation fonts-freefont-ttf \ libfontconfig1 libfreetype6 \ libasound2t64 libatk-bridge2.0-0t64 libatk1.0-0t64 libatspi2.0-0t64 \ @@ -40,6 +40,13 @@ RUN curl -fsSL "https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-l && rm -rf /var/lib/apt/lists/* COPY entrypoint.sh /home/runner/entrypoint.sh + +# Rust as hosted ubuntu-24.04 has it (stable, on PATH): the native physics ABI is built with cargo. +# Installed as `runner` so cargo can write its own registry and target caches. +USER runner +RUN curl -fsSL https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain stable +ENV PATH=/home/runner/.cargo/bin:$PATH +USER root RUN chmod 0755 /home/runner/entrypoint.sh USER runner From 2531ee1376ca4d691db29255f6b7d87ef2fb45da Mon Sep 17 00:00:00 2001 From: Joao Paulo Furtado Date: Fri, 2 Oct 2026 13:46:00 -0700 Subject: [PATCH 06/18] fix(ci-runners): five slots on whole cores, two cores and the RAM kept for the host 12 GB per slot with no extra swap (5 x 12 = 60 of 62 GB worst case, typical jobs far less) and --oom-score-adj 800, so memory pressure kills a CI job before anything of the owner's. up refuses a pool that would take the host's two reserved cores. --- scripts/ci-runners.sh | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/scripts/ci-runners.sh b/scripts/ci-runners.sh index d994e4e8b0..3cccc91db8 100755 --- a/scripts/ci-runners.sh +++ b/scripts/ci-runners.sh @@ -22,7 +22,9 @@ LABEL=tn-local VARIABLE=TN_RUNNER ENV_FILE="${TN_RUNNERS_ENV:-${XDG_CONFIG_HOME:-$HOME/.config}/threenative/runners.env}" STATE_DIR="${XDG_STATE_HOME:-$HOME/.local/state}/threenative/ci-runners" -DEFAULT_SLOTS=4 +DEFAULT_SLOTS=5 +# Cores never given to a slot, so the owner's desktop stays responsive under a full board. +HOST_CORES=2 ONLINE_TIMEOUT_SECONDS=300 fail() { @@ -65,7 +67,7 @@ start_slot() { while [ ! -e "$1/stop" ]; do docker run --rm \ --label tn-ci-runner=1 \ - --cpuset-cpus "$4" --memory 16g \ + --cpuset-cpus "$4" --memory 12g --memory-swap 12g --oom-score-adj 800 \ --env-file "$2" \ -e TN_RUNNER_REPO="$3" \ tn-ci-runner || sleep 10 @@ -113,7 +115,8 @@ up() { # ponytail: assumes Linux's usual sibling numbering (CPU k and k + nproc/2 share a core); read # /sys/devices/system/cpu/cpu*/topology/thread_siblings_list if a host numbers them otherwise. local half=$(( $(nproc) / 2 )) - [ $(( slots * 2 )) -le "$half" ] || fail 2 "$slots slots need $(( slots * 2 )) cores; this host has $half" + [ $(( slots * 2 + HOST_CORES )) -le "$half" ] \ + || fail 2 "$slots slots need $(( slots * 2 )) cores plus $HOST_CORES for the host; this host has $half" local slot base for slot in $(seq 1 "$slots"); do base=$(( (slot - 1) * 2 )) From 8cc50e586b5b79155607f745fe24ba91b738962a Mon Sep 17 00:00:00 2001 From: Joao Paulo Furtado Date: Fri, 2 Oct 2026 13:56:37 -0700 Subject: [PATCH 07/18] fix(ci-runners): drop mesa-vulkan-drivers from the image, as hosted has none Workflows that need Mesa install it themselves. With it baked in, the tone and decals captures went blank on tn-local (run 37062759406: TN_CAPTURE_BLANK) while the same job passed on the image without it. --- tools/ci-runners/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/ci-runners/Dockerfile b/tools/ci-runners/Dockerfile index 8127651679..286b475eef 100644 --- a/tools/ci-runners/Dockerfile +++ b/tools/ci-runners/Dockerfile @@ -18,7 +18,7 @@ RUN apt-get update \ && apt-get install -y --no-install-recommends \ build-essential cmake ninja-build ccache clang lld pkg-config \ git curl jq unzip zip xz-utils xvfb xauth python3 python3-pip \ - 7zip ffmpeg imagemagick zstd file lsof procps mesa-vulkan-drivers \ + 7zip ffmpeg imagemagick zstd file lsof procps \ fonts-noto-color-emoji fonts-unifont fonts-liberation fonts-freefont-ttf \ libfontconfig1 libfreetype6 \ libasound2t64 libatk-bridge2.0-0t64 libatk1.0-0t64 libatspi2.0-0t64 \ From cd2925a6a32cb583acc295f7e5119ba5cc6ddde2 Mon Sep 17 00:00:00 2001 From: Joao Paulo Furtado Date: Fri, 2 Oct 2026 13:57:08 -0700 Subject: [PATCH 08/18] docs(prd-480): pinned five-slot pool, host reserve, and a reserved light lane Records the pool shape measured on the first live runs and the owner's stability constraint, and adds Astra's light-lane review point as a Phase 2 box. --- ...-480-linux-ci-runs-on-the-owner-machine.md | 23 +++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md b/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md index ec0006a3d6..1dad65e45c 100644 --- a/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md +++ b/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md @@ -56,10 +56,18 @@ runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'u `tn-local`. Today the agent runs `pnpm test` or `pnpm ci:local --full` here, then CI repeats the same board, so this removes the duplicate. With `TN_RUNNER` unset, CI is hosted and slow again, and the local full run before pushing comes back. -- **Comparable timing:** each container is capped at 4 CPUs and 16 GB, the shape of a hosted - `ubuntu-24.04` runner, so operation budgets and timeouts keep meaning what they meant. -- **Ephemeral:** each container takes one job, exits and is recreated (`--ephemeral`, compose - `restart: always`). No `/tmp`, port, Xvfb display or workspace state crosses jobs. +- **Comparable timing:** each container is pinned (`--cpuset-cpus`) to two whole cores, four threads, + the CPU shape of a hosted `ubuntu-24.04` runner, so `nproc` reads 4 and every worker pool sizes itself + to that. A `--cpus 4` quota is not enough: `nproc` still reads every host CPU and vitest ran 8 workers + in a 4-core budget. This is a controlled starting point, not a promise of identical timing. +- **Host stays stable:** five slots by default, and two cores are never given to a slot. Each slot gets + `--memory 12g --memory-swap 12g --oom-score-adj 800`, so under memory pressure the kernel kills a CI + job before the owner's processes. +- **Light lane:** `scope`, `ci-required`, `run-summary` and other small joins run on one more, + unpinned `tn-local-light` runner (`--cpus 1`, 2 GB) that heavy jobs never select. Otherwise a 20-minute + build on every heavy slot holds a 10-second join in the queue. +- **Ephemeral:** each container takes one job, exits and is recreated by a host-side `docker run --rm` + loop. No `/tmp`, port, Xvfb display or workspace state crosses jobs. **Stays hosted:** `native-platforms` macOS, Windows, iOS and `linux-arm64` legs; the `supply-chain` job (its gitleaks step needs `docker run`, and mounting the Docker socket would give jobs root on the @@ -111,6 +119,10 @@ while `TN_RUNNER` is set: jobs queue until the switch is cleared. ## Decisions +- 2026-10-02 (João, during the first live runs): use more of the machine, but never at the cost of + desktop stability. Five pinned slots of two cores each, two cores reserved for the host, 12 GB caps + with CI as the OOM victim, and a reserved light lane (review by Astra the same day). + - 2026-10-02 (João): the runner registration token exists. It is a fine-grained token on `ThreeNativeHQ/threenative` only, with Administration read & write, stored in the operator's untracked runner env file. Verified by listing the repository's runners (0). @@ -180,6 +192,9 @@ and `integration-decals.yml`'s `ubuntu-24.04-arm` job are the only Linux `runs-o - [ ] `AGENTS.md` says focused checks then push, not the full board, while `TN_RUNNER` is set. proof: `pnpm sync:agents --check` and `pnpm exec vitest run scripts/__tests__/sync-agent-docs.spec.ts` pass. Land it only after the AC-1 run is green. +- [ ] Light jobs never wait behind heavy ones: `scope`, `ci-required` and `run-summary` route to + `tn-local-light` and no heavy job can. proof: `ci-structure.spec.ts` case, plus a full-board run whose + `ci-required` starts within 60 s of its last `needs` finishing. - [ ] With `TN_RUNNER` unset, the same workflow runs fully hosted. proof: `workflow_dispatch` run id with every `runner_name` hosted. From c44c3e39fc43295e7ddae7024747c3115c34f1d2 Mon Sep 17 00:00:00 2001 From: Joao Paulo Furtado Date: Fri, 2 Oct 2026 14:04:30 -0700 Subject: [PATCH 09/18] ci(prd-480): give small joins an unpinned tn-local-light lane MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A ten-second verdict queued behind five twenty-minute builds is the wait this removes. `ci-runners.sh up` now starts one light slot alongside the heavy ones: same image and host loop, `--cpus 1 --memory 2g --memory-swap 2g --oom-score-adj 900`, no cpuset, registering `tn-local-light` (entrypoint takes the label from `TN_RUNNER_LABELS`), and it advertises `TN_RUNNER_LIGHT` only once that runner is online — so the kill switch stays one deletion and `down` clears both variables before it stops anything. `scope`, `golden-path`, `build`, `ci-required` and `run-summary` route to it: the five ci.yml jobs whose whole work is a script or a summary, none of which builds the workspace or runs a test suite. Every other movable Linux job keeps the TN_RUNNER expression. `ci-structure.spec.ts` now requires each Linux job to use exactly one of the two expressions, and asserts the light allow-list in both directions, so a heavy job on the light lane and a light job off it both fail. --- .github/workflows/ci.yml | 10 +- ...-480-linux-ci-runs-on-the-owner-machine.md | 7 ++ scripts/__tests__/ci-structure.spec.ts | 25 ++++- scripts/ci-runners.sh | 98 ++++++++++++------- tools/ci-runners/entrypoint.sh | 5 +- 5 files changed, 100 insertions(+), 45 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1eef8683ae..886362fcaa 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -29,7 +29,7 @@ jobs: scope: name: Change scope if: ${{ !github.event.pull_request.draft }} - runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER_LIGHT) && 'ubuntu-24.04' || vars.TN_RUNNER_LIGHT }} timeout-minutes: 5 outputs: scope: ${{ steps.classify.outputs.scope }} @@ -846,7 +846,7 @@ jobs: # as cancelled, so the skip is correct there; it still cannot be mistaken for a pass, because a # cancelled run has no passing checks to merge on. if: ${{ !cancelled() && needs.scope.outputs.selection == 'full' }} - runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER_LIGHT) && 'ubuntu-24.04' || vars.TN_RUNNER_LIGHT }} timeout-minutes: 5 env: TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} @@ -916,7 +916,7 @@ jobs: build: needs: [scope, build-artifacts] if: ${{ !cancelled() && needs.scope.outputs.selection == 'full' }} - runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER_LIGHT) && 'ubuntu-24.04' || vars.TN_RUNNER_LIGHT }} timeout-minutes: 2 env: TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} @@ -1170,7 +1170,7 @@ jobs: ci-required: needs: [scope, native-platforms, typecheck, lint, test, test-unit, test-native, test-browser, test-playtest, golden-path-template, template-nonvisual, golden-path, benchmark, build, build-artifacts, budgets, performance-contracts, supply-chain] if: ${{ always() && !github.event.pull_request.draft }} - runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER_LIGHT) && 'ubuntu-24.04' || vars.TN_RUNNER_LIGHT }} timeout-minutes: 5 steps: - uses: actions/checkout@v7 @@ -1224,7 +1224,7 @@ jobs: # `!cancelled()` and not `always()`, for the reason golden-path gives above: a cancelled run # has nothing to report and its summary would only describe the cancellation. if: ${{ !cancelled() && needs.scope.outputs.selection == 'full' }} - runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER_LIGHT) && 'ubuntu-24.04' || vars.TN_RUNNER_LIGHT }} timeout-minutes: 5 permissions: contents: read diff --git a/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md b/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md index 1dad65e45c..cf0185fa31 100644 --- a/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md +++ b/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md @@ -195,6 +195,13 @@ and `integration-decals.yml`'s `ubuntu-24.04-arm` job are the only Linux `runs-o - [ ] Light jobs never wait behind heavy ones: `scope`, `ci-required` and `run-summary` route to `tn-local-light` and no heavy job can. proof: `ci-structure.spec.ts` case, plus a full-board run whose `ci-required` starts within 60 s of its last `needs` finishing. + Landed, unticked because the timing half is unmeasured: `up` starts one unpinned + `--cpus 1 --memory 2g --oom-score-adj 900` slot registering `tn-local-light` and sets + `TN_RUNNER_LIGHT`, and `scope`, `golden-path`, `build`, `ci-required`, `run-summary` route to it — + every job whose work is a script or a summary, none of which builds the workspace or runs a test. + `ci-structure.spec.ts` enforces the allow-list in both directions (red on `typecheck` put on the + light expression, red on `scope` put off it). Left: a full-board run whose `ci-required` starts + within 60 s of its last `needs` finishing. - [ ] With `TN_RUNNER` unset, the same workflow runs fully hosted. proof: `workflow_dispatch` run id with every `runner_name` hosted. diff --git a/scripts/__tests__/ci-structure.spec.ts b/scripts/__tests__/ci-structure.spec.ts index 79b4cf4661..8d24983215 100644 --- a/scripts/__tests__/ci-structure.spec.ts +++ b/scripts/__tests__/ci-structure.spec.ts @@ -921,11 +921,21 @@ describe("CI pipeline structure", () => { // the queue does not shrink, and nothing in a run says which half of the board ignored it. const routing = "${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }}"; + const lightRouting = + "${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER_LIGHT) && 'ubuntu-24.04' || vars.TN_RUNNER_LIGHT }}"; // `supply-chain` runs gitleaks through `docker run`; a self-hosted container would need the host // Docker socket mounted to do that, which would hand every job root on the owner's machine. const hosted = new Set(["supply-chain"]); - - it("routes every movable Linux job through the TN_RUNNER switch", async () => { + // PRD-480's light lane: the jobs whose whole work is a script or a summary — no workspace build + // and no test suite. `scope` classifies the diff, `build` and `golden-path` assert an upstream + // verdict, `ci-required` is the merge verdict and `run-summary` writes the report. They join on + // `tn-local-light` (1 CPU, 2 GB) so a ten-second verdict does not queue behind five twenty-minute + // builds. Both directions are load-bearing and both are asserted below: a heavy job here would + // build the workspace in one core, and a light job off the list would queue behind the heavy + // pool — which is the wait the lane exists to remove. + const light = new Set(["build", "ci-required", "golden-path", "run-summary", "scope"]); + + it("routes every movable Linux job through exactly one TN_RUNNER switch", async () => { const directory = path.join(repo, ".github/workflows"); const routed = [ ".github/workflows/ci.yml", @@ -943,12 +953,21 @@ describe("CI pipeline structure", () => { // of native-platforms.yml — so the expression must never appear on one of them here. if (/macos|windows|arm/u.test(runsOn)) { expect(runsOn, `${relative} ${job} is not Linux`).not.toContain(routing); + expect(runsOn, `${relative} ${job} is not Linux`).not.toContain(lightRouting); continue; } expect( - runsOn === `runs-on: ${routing}` || hosted.has(job), + runsOn === `runs-on: ${routing}` || + runsOn === `runs-on: ${lightRouting}` || + hosted.has(job), `${relative} ${job} runs on \`${runsOn}\`: use the TN_RUNNER routing expression, or name it on the hosted allow-list`, ).toBe(true); + expect( + light.has(job), + `${relative} ${job} is ${ + runsOn === `runs-on: ${lightRouting}` ? "on" : "off" + } the light lane: only ${[...light].join(", ")} may select tn-local-light`, + ).toBe(runsOn === `runs-on: ${lightRouting}`); } } }); diff --git a/scripts/ci-runners.sh b/scripts/ci-runners.sh index 3cccc91db8..224af0a104 100755 --- a/scripts/ci-runners.sh +++ b/scripts/ci-runners.sh @@ -1,13 +1,14 @@ #!/usr/bin/env bash # The `tn-local` self-hosted runner pool (PRD-480). # -# scripts/ci-runners.sh up [N] build the image, run N disposable runner containers, then set -# the TN_RUNNER repository variable so the routing expression in -# ci.yml and integration-*.yml sends Linux jobs here -# scripts/ci-runners.sh down clear TN_RUNNER first, then stop everything +# scripts/ci-runners.sh up [N] build the image, run N disposable heavy runner containers plus +# one light one, then set the TN_RUNNER and TN_RUNNER_LIGHT +# repository variables so the routing expressions in ci.yml and +# integration-*.yml send Linux jobs here +# scripts/ci-runners.sh down clear both variables first, then stop everything # scripts/ci-runners.sh status what is set, what is running, what is online # -# `down` deletes the variable before it stops anything, so the kill switch is the same command as +# `down` deletes the variables before it stops anything, so the kill switch is the same command as # the teardown: jobs go back to hosted runners the moment it is deleted, whether or not the # containers are still there. # @@ -19,13 +20,19 @@ cd "$(dirname -- "${BASH_SOURCE[0]}")/.." IMAGE=tn-ci-runner LABEL=tn-local +LIGHT_LABEL=tn-local-light VARIABLE=TN_RUNNER +LIGHT_VARIABLE=TN_RUNNER_LIGHT ENV_FILE="${TN_RUNNERS_ENV:-${XDG_CONFIG_HOME:-$HOME/.config}/threenative/runners.env}" STATE_DIR="${XDG_STATE_HOME:-$HOME/.local/state}/threenative/ci-runners" DEFAULT_SLOTS=5 # Cores never given to a slot, so the owner's desktop stays responsive under a full board. HOST_CORES=2 ONLINE_TIMEOUT_SECONDS=300 +# The light lane is a quota, not a reservation: one core of whatever a heavy slot is not using, +# 2 GB, and the highest OOM score of any container, so a small join is what gets killed under +# memory pressure rather than a 20-minute build. +LIGHT_SHAPE="--cpus 1 --memory 2g --memory-swap 2g --oom-score-adj 900" fail() { local code="$1" @@ -60,25 +67,28 @@ repo_name() { # One slot is a host-side loop, not a compose service: `docker run --rm` gives each job a fresh # container filesystem and a fresh hostname, which `restart: always` on a long-lived container -# cannot. The loop is what makes the pool survive the exit. +# cannot. The loop is what makes the pool survive the exit. `shape` carries the docker resource +# flags word-split on purpose (there is no array to expand through `bash -c`), and `labels` is the +# one label the container registers with — `--no-default-labels` means that label is all it has. start_slot() { - local slot="$1" state="$2" env_file="$3" repo="$4" cpus="$5" + local slot="$1" state="$2" env_file="$3" repo="$4" shape="$5" labels="$6" setsid nohup bash -c ' while [ ! -e "$1/stop" ]; do docker run --rm \ --label tn-ci-runner=1 \ - --cpuset-cpus "$4" --memory 12g --memory-swap 12g --oom-score-adj 800 \ + $5 \ --env-file "$2" \ -e TN_RUNNER_REPO="$3" \ + -e TN_RUNNER_LABELS="$6" \ tn-ci-runner || sleep 10 done - ' _ "$state" "$env_file" "$repo" "$cpus" >"$state/slot-$slot.log" 2>&1 & - echo "$!" >"$state/slot-$slot.pid" + ' _ "$state" "$env_file" "$repo" "$shape" "$labels" >"$state/$slot.log" 2>&1 & + echo "$!" >"$state/$slot.pid" } stop_slots() { local pid_file - for pid_file in "$STATE_DIR"/slot-*.pid; do + for pid_file in "$STATE_DIR"/*.pid; do [ -f "$pid_file" ] || continue kill "$(cat "$pid_file")" 2>/dev/null || true done @@ -88,9 +98,9 @@ stop_slots() { } online_count() { - gh api "repos/$1/actions/runners" --paginate --jq ' - [.runners[] | select(.status == "online") | select(any(.labels[]; .name == "tn-local"))] | length - ' 2>/dev/null || echo 0 + gh api "repos/$1/actions/runners" --paginate --jq " + [.runners[] | select(.status == \"online\") | select(any(.labels[]; .name == \"$2\"))] | length + " 2>/dev/null || echo 0 } up() { @@ -120,29 +130,39 @@ up() { local slot base for slot in $(seq 1 "$slots"); do base=$(( (slot - 1) * 2 )) - start_slot "$slot" "$STATE_DIR" "$ENV_FILE" "$repo" \ - "$base,$(( base + 1 )),$(( base + half )),$(( base + half + 1 ))" + start_slot "slot-$slot" "$STATE_DIR" "$ENV_FILE" "$repo" \ + "--cpuset-cpus $base,$(( base + 1 )),$(( base + half )),$(( base + half + 1 )) --memory 12g --memory-swap 12g --oom-score-adj 800" \ + "$LABEL" done - printf '%s runner container(s) starting; logs in %s\n' "$slots" "$STATE_DIR" - - # Only advertise the label once the runners are really there: TN_RUNNER set with an empty pool - # is a queue that never drains. On timeout, take the whole thing back down rather than leaving - # half a pool registered. - local deadline=$(( $(date +%s) + ONLINE_TIMEOUT_SECONDS )) online + # One light slot, unpinned and quota-limited, labelled so heavy jobs cannot select it: a 10-second + # join behind five 20-minute builds is the wait this lane exists to remove. It never reserves a + # core, so the check above still describes exactly the cores this pool owns. + start_slot light "$STATE_DIR" "$ENV_FILE" "$repo" "$LIGHT_SHAPE" "$LIGHT_LABEL" + printf '%s heavy + 1 light runner container(s) starting; logs in %s\n' "$slots" "$STATE_DIR" + + # Only advertise the labels once the runners are really there: TN_RUNNER set with an empty pool + # is a queue that never drains. Both labels wait, because a light runner that never came online + # leaves its jobs queued behind the heavy pool forever. On timeout, take the whole thing back down + # rather than leaving half a pool registered. + local deadline=$(( $(date +%s) + ONLINE_TIMEOUT_SECONDS )) online light_online while :; do - online="$(online_count "$repo")" - [ "$online" -ge "$slots" ] && break + online="$(online_count "$repo" "$LABEL")" + light_online="$(online_count "$repo" "$LIGHT_LABEL")" + if [ "$online" -ge "$slots" ] && [ "$light_online" -ge 1 ]; then break; fi if [ "$(date +%s)" -ge "$deadline" ]; then gh variable delete "$VARIABLE" --repo "$repo" 2>/dev/null || true + gh variable delete "$LIGHT_VARIABLE" --repo "$repo" 2>/dev/null || true stop_slots - fail 1 "$online of $slots $LABEL runner(s) came online within ${ONLINE_TIMEOUT_SECONDS}s" + fail 1 "$online of $slots $LABEL and $light_online $LIGHT_LABEL runner(s) came online within ${ONLINE_TIMEOUT_SECONDS}s" fi sleep 10 done gh variable set "$VARIABLE" --body "$LABEL" --repo "$repo" - printf '%s online; %s=%s, so Linux jobs route here and deleting it routes them back\n' \ - "$online" "$VARIABLE" "$LABEL" + gh variable set "$LIGHT_VARIABLE" --body "$LIGHT_LABEL" --repo "$repo" + printf '%s %s + %s %s online; %s=%s and %s=%s, so deleting either routes its jobs back\n' \ + "$online" "$LABEL" "$light_online" "$LIGHT_LABEL" \ + "$VARIABLE" "$LABEL" "$LIGHT_VARIABLE" "$LIGHT_LABEL" } down() { @@ -150,23 +170,31 @@ down() { local repo repo="$(repo_name)" # First, so no job is ever routed to a pool that is on its way out. - gh variable delete "$VARIABLE" --repo "$repo" 2>/dev/null \ - || printf 'TN_CI_RUNNERS: %s was already unset\n' "$VARIABLE" >&2 + local variable + for variable in "$VARIABLE" "$LIGHT_VARIABLE"; do + gh variable delete "$variable" --repo "$repo" 2>/dev/null \ + || printf 'TN_CI_RUNNERS: %s was already unset\n' "$variable" >&2 + done mkdir -p "$STATE_DIR" stop_slots - printf '%s cleared; runners stopping\n' "$VARIABLE" + printf '%s and %s cleared; runners stopping\n' "$VARIABLE" "$LIGHT_VARIABLE" } status() { need gh local repo repo="$(repo_name)" - printf '%s=%s\n' "$VARIABLE" "$(gh variable get "$VARIABLE" --repo "$repo" 2>/dev/null || echo unset)" + local variable label + for variable in "$VARIABLE" "$LIGHT_VARIABLE"; do + printf '%s=%s\n' "$variable" "$(gh variable get "$variable" --repo "$repo" 2>/dev/null || echo unset)" + done printf 'containers: %s\n' "$(docker ps -q --filter label=tn-ci-runner=1 | wc -l | tr -d ' ')" - printf 'online %s runners (name, status, busy):\n' "$LABEL" - gh api "repos/$repo/actions/runners" --paginate --jq \ - ".runners[] | select(any(.labels[]; .name == \"$LABEL\")) | \" \\(.name) \\(.status) busy=\\(.busy)\"" \ - 2>/dev/null || echo " none reported" + for label in "$LABEL" "$LIGHT_LABEL"; do + printf 'online %s runners (name, status, busy):\n' "$label" + gh api "repos/$repo/actions/runners" --paginate --jq \ + ".runners[] | select(any(.labels[]; .name == \"$label\")) | \" \\(.name) \\(.status) busy=\\(.busy)\"" \ + 2>/dev/null || echo " none reported" + done } case "${1:-}" in diff --git a/tools/ci-runners/entrypoint.sh b/tools/ci-runners/entrypoint.sh index 1836542479..5b128398ec 100755 --- a/tools/ci-runners/entrypoint.sh +++ b/tools/ci-runners/entrypoint.sh @@ -1,5 +1,6 @@ #!/usr/bin/env bash -# Registers this container as an ephemeral `tn-local` runner and takes one job. +# Registers this container as an ephemeral `tn-local` runner — or `tn-local-light` for the light +# lane — and takes one job. # # The container is the disposable half of `scripts/ci-runners.sh up`: it configures, runs exactly # one job, exits, and the host loop starts a fresh one. Nothing on the filesystem crosses jobs. @@ -29,7 +30,7 @@ unset RUNNER_ADMIN_TOKEN --url "https://github.com/${TN_RUNNER_REPO}" \ --token "$registration_token" \ --name "tn-$(hostname)" \ - --labels tn-local \ + --labels "${TN_RUNNER_LABELS:-tn-local}" \ --no-default-labels exec ./run.sh From ff6d26ec7d3464b73bf775288b4b59cf177e65ca Mon Sep 17 00:00:00 2001 From: Joao Paulo Furtado Date: Fri, 2 Oct 2026 14:10:56 -0700 Subject: [PATCH 10/18] fix(ci-runners): JDK 17 in the image, as hosted has it test-native on tn-local (run 37062759791): the Android packaging tests drive a fake gradlew built on jar, javac and java; without a JDK every one exited 127. --- tools/ci-runners/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/ci-runners/Dockerfile b/tools/ci-runners/Dockerfile index 286b475eef..0d0935a01a 100644 --- a/tools/ci-runners/Dockerfile +++ b/tools/ci-runners/Dockerfile @@ -18,7 +18,7 @@ RUN apt-get update \ && apt-get install -y --no-install-recommends \ build-essential cmake ninja-build ccache clang lld pkg-config \ git curl jq unzip zip xz-utils xvfb xauth python3 python3-pip \ - 7zip ffmpeg imagemagick zstd file lsof procps \ + 7zip ffmpeg imagemagick zstd file lsof procps openjdk-17-jdk-headless \ fonts-noto-color-emoji fonts-unifont fonts-liberation fonts-freefont-ttf \ libfontconfig1 libfreetype6 \ libasound2t64 libatk-bridge2.0-0t64 libatk1.0-0t64 libatspi2.0-0t64 \ From f27b31b48641c596e6a05c85fe6cdd1c82420422 Mon Sep 17 00:00:00 2001 From: Joao Paulo Furtado Date: Fri, 2 Oct 2026 14:12:54 -0700 Subject: [PATCH 11/18] fix(ci-runners): Android platform-tools at the hosted ANDROID_HOME test-unit 3/3 on tn-local: the playtest doctor spec failed on "adb was not found". --- tools/ci-runners/Dockerfile | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/tools/ci-runners/Dockerfile b/tools/ci-runners/Dockerfile index 0d0935a01a..26c342515c 100644 --- a/tools/ci-runners/Dockerfile +++ b/tools/ci-runners/Dockerfile @@ -39,6 +39,14 @@ RUN curl -fsSL "https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-l && apt-get update && apt-get install -y --no-install-recommends gh \ && rm -rf /var/lib/apt/lists/* +# Android platform-tools where hosted ubuntu-24.04 keeps its SDK: the playtest doctor resolves adb +# through ANDROID_HOME before PATH. +ENV ANDROID_HOME=/usr/local/lib/android/sdk ANDROID_SDK_ROOT=/usr/local/lib/android/sdk +RUN mkdir -p "$ANDROID_HOME" \ + && curl -fsSL https://dl.google.com/android/repository/platform-tools-latest-linux.zip -o /tmp/pt.zip \ + && unzip -q /tmp/pt.zip -d "$ANDROID_HOME" && rm /tmp/pt.zip \ + && ln -s "$ANDROID_HOME/platform-tools/adb" /usr/local/bin/adb + COPY entrypoint.sh /home/runner/entrypoint.sh # Rust as hosted ubuntu-24.04 has it (stable, on PATH): the native physics ABI is built with cargo. From 578070964a8cba8fc2e930a7572a282cb4ca58b9 Mon Sep 17 00:00:00 2001 From: Joao Paulo Furtado Date: Fri, 2 Oct 2026 14:40:21 -0700 Subject: [PATCH 12/18] fix(ci-runners): slot loop reads shape and labels from the right positions Inside bash -c the first argument is $0, so the label landed where the image name goes and docker tried to pull tn-local:latest. up then failed closed. --- scripts/ci-runners.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/ci-runners.sh b/scripts/ci-runners.sh index 224af0a104..ceb7d09042 100755 --- a/scripts/ci-runners.sh +++ b/scripts/ci-runners.sh @@ -76,10 +76,10 @@ start_slot() { while [ ! -e "$1/stop" ]; do docker run --rm \ --label tn-ci-runner=1 \ - $5 \ + $4 \ --env-file "$2" \ -e TN_RUNNER_REPO="$3" \ - -e TN_RUNNER_LABELS="$6" \ + -e TN_RUNNER_LABELS="$5" \ tn-ci-runner || sleep 10 done ' _ "$state" "$env_file" "$repo" "$shape" "$labels" >"$state/$slot.log" 2>&1 & From 94732de6598a56b719384b246e7b9e266e922b62 Mon Sep 17 00:00:00 2001 From: Joao Paulo Furtado Date: Fri, 2 Oct 2026 15:06:09 -0700 Subject: [PATCH 13/18] fix(ci-runners): C.UTF-8 locale, as hosted runners have test-native run 37068328553: javac read MystralActivity.java as US-ASCII and rejected an em dash, so the Android metadata probe failed. 1 of 1593 left red. --- tools/ci-runners/Dockerfile | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tools/ci-runners/Dockerfile b/tools/ci-runners/Dockerfile index 26c342515c..10b9ef9f39 100644 --- a/tools/ci-runners/Dockerfile +++ b/tools/ci-runners/Dockerfile @@ -42,6 +42,8 @@ RUN curl -fsSL "https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-l # Android platform-tools where hosted ubuntu-24.04 keeps its SDK: the playtest doctor resolves adb # through ANDROID_HOME before PATH. ENV ANDROID_HOME=/usr/local/lib/android/sdk ANDROID_SDK_ROOT=/usr/local/lib/android/sdk +# Hosted runners run in C.UTF-8; without it javac reads sources as US-ASCII and rejects an em dash. +ENV LANG=C.UTF-8 LC_ALL=C.UTF-8 RUN mkdir -p "$ANDROID_HOME" \ && curl -fsSL https://dl.google.com/android/repository/platform-tools-latest-linux.zip -o /tmp/pt.zip \ && unzip -q /tmp/pt.zip -d "$ANDROID_HOME" && rm /tmp/pt.zip \ From 1fc3886b7ebd8dd408e22ed263dc830d6d1c9fa0 Mon Sep 17 00:00:00 2001 From: Joao Paulo Furtado Date: Fri, 2 Oct 2026 15:30:40 -0700 Subject: [PATCH 14/18] ci(prd-480): route the native-platforms Linux legs onto the tn-local pool MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 3 of PRD-480. Eight Linux jobs in native-platforms.yml and the `starter-linux/linux-x64` matrix row now pick their runner from the same expression ci.yml uses, so the pool is still changed by setting or deleting one repository variable. `scope` and `networking-matrix` — the two whose whole work is a script with no `pnpm install` — take the light lane; everything else is heavy. `github.event` inside a `workflow_call` is the caller's event payload, so the fork guard reads the same flag ci.yml did when it called this file. Kept hosted, with reasons in the file and in the spec's allow-list: `publish-android-v8` is the release write itself (`contents: write` plus `gh release upload`), and macOS, Windows, `linux-arm64` and iOS have no tn-local counterpart. The `desktop` matrix has no Linux row at all. The image gains the SDK the routed Android steps pin: cmdline-tools 16.0 with licenses accepted, platform-tools, platforms 35 and 36 (compileSdk), build-tools 35.0.0 and NDK 28.2.13676358 (ndkVersion), owned by `runner` because sdkmanager writes into ANDROID_HOME at job time. `emulator` and the system image stay out: android-emulator-runner installs those itself. 9.28 GB against the pool image's 5.25 GB, chowned inside the install layer so the SDK is not written twice. Heavy slots pass `--device /dev/kvm` where the host has one, and the entrypoint sets the node's mode before the job reads it: docker creates it 0660 root:kvm, which android-emulator-runner cannot open. That also means the lane's `udevadm control`/`udevadm trigger` lines can never succeed in a container — no udev, read-only /sys — so they are now `|| true`, which is what the step's own "report the mode, never assert it" rule already says. The spec compares the two emulator lanes' provisioning with that suffix normalised. Evidence: `docker build -t tn-ci-runner:phase3 tools/ci-runners` succeeded, and `docker run --rm --device /dev/kvm --entrypoint bash tn-ci-runner:phase3 -c 'sdkmanager --version && ls $ANDROID_HOME && ls -l /dev/kvm'` printed sdkmanager 16.0 with build-tools, cmdline-tools, licenses, ndk, platform-tools and platforms. `emulator -accel-check` inside the container reports "KVM (version 12) is installed and usable" with no missing shared libraries. `bash -n scripts/ci-runners.sh` passes. `pnpm exec vitest run scripts/__tests__/ci-structure.spec.ts scripts/__tests__/ci-needs.spec.ts` — 142 passed; red first on `.github/workflows/native-platforms.yml scope runs on ubuntu-latest`, and again on a `starter-linux/linux-x64` row put back to bare `ubuntu-24.04`. The six sibling workflow specs (ci-efficiency, ci-local, compiler-cache-setup, hosted-software-callers, workspace-dist-cache, native-release-proof) pass at 149. `pnpm lint` exits 0, `pnpm check:docs` checks 2496 links. Not run here and not claimed: a live `native-platforms` run on the pool, which is what the two phase boxes still wait for. `pnpm typecheck` and `pnpm budgets` fail in this worktree on missing `packages/*/dist` build output, in files this change does not touch. --- .github/workflows/native-platforms.yml | 34 +++-- ...-480-linux-ci-runs-on-the-owner-machine.md | 26 +++- scripts/__tests__/ci-structure.spec.ts | 130 ++++++++++++------ scripts/ci-runners.sh | 11 +- tools/ci-runners/Dockerfile | 30 ++++ tools/ci-runners/entrypoint.sh | 10 ++ 6 files changed, 188 insertions(+), 53 deletions(-) diff --git a/.github/workflows/native-platforms.yml b/.github/workflows/native-platforms.yml index a9ad933fe1..b517c9905a 100644 --- a/.github/workflows/native-platforms.yml +++ b/.github/workflows/native-platforms.yml @@ -53,7 +53,10 @@ jobs: # The caller is the single selection authority. Dispatch has no exemption inputs and is full. scope: name: Validate caller selection and candidate - runs-on: ubuntu-latest + # PRD-480. `github.event` inside a `workflow_call` is the caller's event payload, so this + # expression reads the same `pull_request` fork flag ci.yml did when it called this file: a fork + # PR falls through to hosted runners and never reaches the owner's machine. + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER_LIGHT) && 'ubuntu-24.04' || vars.TN_RUNNER_LIGHT }} timeout-minutes: 5 outputs: scope: ${{ steps.classify.outputs.scope }} @@ -89,7 +92,7 @@ jobs: if: >- needs.scope.outputs.selection == 'full' && inputs.ios_only == false - runs-on: ubuntu-24.04 + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 30 env: TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} @@ -136,7 +139,7 @@ jobs: name: Android V8 source payload needs: scope if: needs.scope.outputs.selection == 'full' && inputs.ios_only != true - runs-on: ubuntu-24.04 + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} # 180-minute build ceiling plus 30 minutes for actions/cache/save, per the action's note. timeout-minutes: 210 outputs: @@ -186,6 +189,9 @@ jobs: needs.android-v8-source.outputs.published == 'false' && github.event_name != 'pull_request' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/develop') + # PRD-480 leaves this one on GitHub's machines: it is the release write itself — `contents: + # write` and a `gh release upload` below — and no pool container is handed a token that can + # publish. It is named on the hosted allow-list in ci-structure.spec.ts. runs-on: ubuntu-latest timeout-minutes: 10 permissions: @@ -213,7 +219,7 @@ jobs: name: Native collector evidence coverage needs: [scope, android-emulator-parity, desktop-parity, desktop, ios-simulator] if: ${{ !cancelled() && needs.scope.outputs.selection == 'full' }} - runs-on: ubuntu-24.04 + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 5 env: TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} @@ -248,7 +254,7 @@ jobs: name: Networking qualification matrix needs: [scope, android-emulator-parity, desktop-parity, desktop, ios-simulator] if: ${{ always() && needs.scope.outputs.selection == 'full' }} - runs-on: ubuntu-24.04 + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER_LIGHT) && 'ubuntu-24.04' || vars.TN_RUNNER_LIGHT }} timeout-minutes: 5 env: TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} @@ -316,7 +322,7 @@ jobs: needs.scope.outputs.selection == 'full' && inputs.ios_only != true && needs.android-v8-source.result == 'success' - runs-on: ubuntu-24.04 + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 45 env: TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} @@ -447,8 +453,14 @@ jobs: run: | echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' \ | sudo tee /etc/udev/rules.d/99-kvm4all.rules - sudo udevadm control --reload-rules - sudo udevadm trigger --name-match=kvm + # `|| true` on both udev calls, because the routed `tn-local` container cannot do what + # they do: no udev is running in it and its /sys is read-only, so `udevadm trigger` + # fails there for a reason that has nothing to do with acceleration. The container does + # not need them — `scripts/ci-runners.sh` passes `--device /dev/kvm` and the image's + # entrypoint sets its mode before the job starts — and a step that fails on the way to + # the report below costs this lane the whole emulator run. + sudo udevadm control --reload-rules || true + sudo udevadm trigger --name-match=kvm || true # Report the mode, never assert it. `test -w /dev/kvm` as the last line of this step — # copied from native-release.yml, where nothing has exercised it — turned "this runner # has no KVM" into a failed job, which is worse than the slow boot it was meant to fix. @@ -740,7 +752,7 @@ jobs: if: >- needs.scope.outputs.selection != 'prose' && inputs.ios_only == false - runs-on: ubuntu-24.04 + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 20 env: TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} @@ -812,7 +824,7 @@ jobs: if: >- needs.scope.outputs.selection == 'full' && inputs.ios_only != true - runs-on: ubuntu-24.04 + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} timeout-minutes: 75 env: TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} @@ -1766,7 +1778,7 @@ jobs: # the one WebGPU backend that publishes an arm64 Linux archive. scripts/download-deps.mjs # and scripts/native-build.mjs select that route from `process.arch`. - platform: linux-x64 - runner: ubuntu-24.04 + runner: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} - platform: linux-arm64 runner: ubuntu-24.04-arm runs-on: ${{ matrix.runner }} diff --git a/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md b/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md index cf0185fa31..ee087bbd0d 100644 --- a/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md +++ b/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md @@ -207,7 +207,7 @@ and `integration-decals.yml`'s `ubuntu-24.04-arm` job are the only Linux `runs-o #### Phase 3: The Linux `native-platforms` legs move too -**Status:** NOT STARTED +**Status:** IN PROGRESS — routing and image landed, live run pending **Files:** EDIT `.github/workflows/native-platforms.yml`, `tools/ci-runners/Dockerfile` (JDK 17, Android SDK, emulator, `/dev/kvm`), `scripts/__tests__/ci-structure.spec.ts`. **Implementation:** Route `web-reference`, `desktop-parity`, `android-emulator-parity`, `release-reports`, @@ -215,6 +215,30 @@ the remaining `ubuntu-24.04` jobs and the `linux-x64` rows of the `desktop` and Keep `macos-15`, `windows-2025`, `ubuntu-24.04-arm` and `ios-simulator` hosted. The `android-emulator-runner` action needs `/dev/kvm` access inside the container. +Decisions this phase had to make, 2026-10-02: + +- **`publish-android-v8` stays hosted.** It is the release write itself — `permissions: contents: write` + and a `gh release upload` — and no pool container is handed a token that can publish. + `android-v8-source` does move: it needs no secret, it is a published-asset download on every run but + a cold one, and its own action records a two-core cold build finishing inside the 210-minute ceiling, + which is this pool's slot shape. +- **Light lane: `scope` and `networking-matrix` only.** Both are a script with no `pnpm install` and no + toolchain. `performance-coverage` and `release-reports` are joins too, but each installs the workspace + and `performance-coverage` has a five-minute ceiling, so they take a heavy slot. +- **`desktop` has no Linux row.** Its matrix is `macOS` and `Windows` only; the movable Linux matrix row + is `starter-linux/linux-x64`. +- **The KVM step had to stop asserting udev.** In the container there is no udev and `/sys` is read-only, + so `udevadm control --reload-rules` and `udevadm trigger` both exit 1 and took the whole emulator lane + with them. They are now `|| true`, which is what the step's own "report the mode, never assert it" + rule already says; `scripts/ci-runners.sh` passes `--device /dev/kvm` and the image's entrypoint sets + the node's mode before the job can read it. +- **The image carries the pinned SDK, not the emulator.** `cmdline-tools` 16.0 with licenses accepted, + `platform-tools`, `platforms;android-35`, `platforms;android-36` (`compileSdk` in + `android/app/build.gradle.kts`), `build-tools;35.0.0` and `ndk;28.2.13676358` (`ndkVersion` there, + and what the workflows' own `sdkmanager` line installs), owned by `runner` because sdkmanager writes + into `ANDROID_HOME` at job time. `android-emulator-runner` installs `emulator` and the system image + itself, which is the only job that wants them. Image: 9.28 GB against the pool's 5.25 GB. + - [ ] A `native-platforms` run passes with its Linux legs on `tn-local`. proof: run id plus per-job `runner_name`. - [ ] Waiting time meets AC-2. proof: the AC-2 measurement on a full-board PR after this phase. diff --git a/scripts/__tests__/ci-structure.spec.ts b/scripts/__tests__/ci-structure.spec.ts index 8d24983215..bec0193661 100644 --- a/scripts/__tests__/ci-structure.spec.ts +++ b/scripts/__tests__/ci-structure.spec.ts @@ -462,16 +462,23 @@ function triggerSection(source: string): string { } function kvmProvisioning(source: string): readonly string[] { - return source - .split("\n") - .map((line) => line.trim()) - .filter( - (line) => - line.includes('KERNEL=="kvm"') || - line === "| sudo tee /etc/udev/rules.d/99-kvm4all.rules" || - line === "sudo udevadm control --reload-rules" || - line === "sudo udevadm trigger --name-match=kvm", - ); + return ( + source + .split("\n") + .map((line) => line.trim()) + // A trailing `|| true` is not part of the provisioning. native-platforms.yml's copy carries one + // because it is the lane PRD-480 routes onto the `tn-local` container, where no udev runs and + // /sys is read-only, so the commands cannot succeed there — while native-release.yml stays + // hosted and keeps them strict. The two lanes must still provision KVM the same way. + .map((line) => line.replace(/ \|\| true$/u, "")) + .filter( + (line) => + line.includes('KERNEL=="kvm"') || + line === "| sudo tee /etc/udev/rules.d/99-kvm4all.rules" || + line === "sudo udevadm control --reload-rules" || + line === "sudo udevadm trigger --name-match=kvm", + ) + ); } function commandText(section: string): string { @@ -923,51 +930,94 @@ describe("CI pipeline structure", () => { "${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }}"; const lightRouting = "${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER_LIGHT) && 'ubuntu-24.04' || vars.TN_RUNNER_LIGHT }}"; - // `supply-chain` runs gitleaks through `docker run`; a self-hosted container would need the host - // Docker socket mounted to do that, which would hand every job root on the owner's machine. - const hosted = new Set(["supply-chain"]); - // PRD-480's light lane: the jobs whose whole work is a script or a summary — no workspace build - // and no test suite. `scope` classifies the diff, `build` and `golden-path` assert an upstream - // verdict, `ci-required` is the merge verdict and `run-summary` writes the report. They join on - // `tn-local-light` (1 CPU, 2 GB) so a ten-second verdict does not queue behind five twenty-minute - // builds. Both directions are load-bearing and both are asserted below: a heavy job here would - // build the workspace in one core, and a light job off the list would queue behind the heavy - // pool — which is the wait the lane exists to remove. - const light = new Set(["build", "ci-required", "golden-path", "run-summary", "scope"]); + // The hosted jobs that stay hosted, per workflow. `supply-chain` runs gitleaks through + // `docker run`; a self-hosted container would need the host Docker socket mounted to do that, + // which would hand every job root on the owner's machine. `publish-android-v8` is the release + // publish itself — `permissions: contents: write` and a `gh release upload` — and PRD-480 keeps + // release writes on GitHub's own machines. + const hosted = new Map>([ + [".github/workflows/ci.yml", new Set(["supply-chain"])], + [".github/workflows/native-platforms.yml", new Set(["publish-android-v8"])], + ]); + // PRD-480's light lane: the jobs whose whole work is a script or a summary — no workspace build, + // no `pnpm install` and no test suite. `scope` classifies the diff, `build` and `golden-path` + // assert an upstream verdict, `ci-required` is the merge verdict and `run-summary` writes the + // report; in native-platforms.yml `networking-matrix` is the one join that needs no toolchain at + // all. They join on `tn-local-light` (1 CPU, 2 GB) so a ten-second verdict does not queue behind + // five twenty-minute builds. Both directions are load-bearing and both are asserted below: a + // heavy job here would build the workspace in one core, and a light job off the list would queue + // behind the heavy pool — which is the wait the lane exists to remove. + const light = new Set([ + "build", + "ci-required", + "golden-path", + "networking-matrix", + "run-summary", + "scope", + ]); + + // A matrix job routes per row: `runs-on: ${{ matrix.runner }}` is only a pointer, and what decides + // where a leg lands is the `runner:` value inside `strategy.matrix.include`. Both native + // matrices carry hosted rows next to movable ones (`linux-x64` moves, `linux-arm64` cannot), so + // the row is named `job/row` here and that is the key the hosted allow-list would use. + function matrixRunnerRows(job: string, section: string): readonly (readonly [string, string])[] { + const rows: [string, string][] = []; + for (const row of section.split(/^ {10}- /mu).slice(1)) { + const runner = row.match(/^ {12}runner: (.+)$/mu)?.[1]; + if (runner !== undefined) { + const platform = row.match(/^\s*platform: (.+)$/mu)?.[1] ?? runner; + rows.push([`${job}/${platform}`, runner]); + } + } + return rows; + } it("routes every movable Linux job through exactly one TN_RUNNER switch", async () => { const directory = path.join(repo, ".github/workflows"); const routed = [ ".github/workflows/ci.yml", + ".github/workflows/native-platforms.yml", ...(await readdir(directory)) .filter((entry) => entry.startsWith("integration-") && entry.endsWith(".yml")) .map((entry) => `.github/workflows/${entry}`), ].sort(); - expect(routed.length).toBeGreaterThan(1); + expect(routed.length).toBeGreaterThan(2); for (const relative of routed) { + const keepHosted = hosted.get(relative) ?? new Set(); const source = await readFile(path.join(repo, relative), "utf8"); for (const [job, section] of jobSections(source)) { const runsOn = section.match(/^\s+runs-on:.*$/mu)?.[0].trim(); if (runsOn === undefined) continue; - // macOS, Windows and arm64 have no tn-local counterpart, and Phase 3 moves the Linux legs - // of native-platforms.yml — so the expression must never appear on one of them here. - if (/macos|windows|arm/u.test(runsOn)) { - expect(runsOn, `${relative} ${job} is not Linux`).not.toContain(routing); - expect(runsOn, `${relative} ${job} is not Linux`).not.toContain(lightRouting); - continue; + const rows = matrixRunnerRows(job, section); + const targets: readonly (readonly [string, string])[] = + rows.length > 0 ? rows : [[job, runsOn.replace(/^runs-on: /u, "")]]; + for (const [name, runner] of targets) { + const target = `${relative} ${name}`; + // macOS, Windows and arm64 have no tn-local counterpart, so the expression must never + // appear on one of them. + if (/macos|windows|arm/u.test(runner)) { + expect(runner, `${target} is not Linux`).not.toContain(routing); + expect(runner, `${target} is not Linux`).not.toContain(lightRouting); + continue; + } + const onHeavy = runner === routing; + const onLight = runner === lightRouting; + if (keepHosted.has(name)) { + expect( + onHeavy || onLight, + `${target} is on the hosted allow-list but selects ${runner}`, + ).toBe(false); + continue; + } + expect( + onHeavy || onLight, + `${target} runs on \`${runner}\`: use the TN_RUNNER routing expression, or name it on the hosted allow-list`, + ).toBe(true); + expect( + light.has(name), + `${target} is ${onLight ? "on" : "off"} the light lane: only ${[...light].join(", ")} may select tn-local-light`, + ).toBe(onLight); } - expect( - runsOn === `runs-on: ${routing}` || - runsOn === `runs-on: ${lightRouting}` || - hosted.has(job), - `${relative} ${job} runs on \`${runsOn}\`: use the TN_RUNNER routing expression, or name it on the hosted allow-list`, - ).toBe(true); - expect( - light.has(job), - `${relative} ${job} is ${ - runsOn === `runs-on: ${lightRouting}` ? "on" : "off" - } the light lane: only ${[...light].join(", ")} may select tn-local-light`, - ).toBe(runsOn === `runs-on: ${lightRouting}`); } } }); diff --git a/scripts/ci-runners.sh b/scripts/ci-runners.sh index ceb7d09042..567932fc9a 100755 --- a/scripts/ci-runners.sh +++ b/scripts/ci-runners.sh @@ -33,6 +33,15 @@ ONLINE_TIMEOUT_SECONDS=300 # 2 GB, and the highest OOM score of any container, so a small join is what gets killed under # memory pressure rather than a 20-minute build. LIGHT_SHAPE="--cpus 1 --memory 2g --memory-swap 2g --oom-score-adj 900" +# `/dev/kvm` for the heavy slots, and only where the host has one: `android-emulator-parity` boots a +# checksum-locked APK under `reactivecircus/android-emulator-runner`, which reads the device for +# read and write and otherwise falls back to a software boot. No KVM, no flag — a host without it +# still runs the pool, just without hardware acceleration, which that lane already reports instead +# of asserting. The light lane never gets it: nothing it runs emulates anything. +KVM_SHAPE="" +if [ -e /dev/kvm ]; then + KVM_SHAPE="--device /dev/kvm" +fi fail() { local code="$1" @@ -131,7 +140,7 @@ up() { for slot in $(seq 1 "$slots"); do base=$(( (slot - 1) * 2 )) start_slot "slot-$slot" "$STATE_DIR" "$ENV_FILE" "$repo" \ - "--cpuset-cpus $base,$(( base + 1 )),$(( base + half )),$(( base + half + 1 )) --memory 12g --memory-swap 12g --oom-score-adj 800" \ + "--cpuset-cpus $base,$(( base + 1 )),$(( base + half )),$(( base + half + 1 )) --memory 12g --memory-swap 12g --oom-score-adj 800 $KVM_SHAPE" \ "$LABEL" done # One light slot, unpinned and quota-limited, labelled so heavy jobs cannot select it: a 10-second diff --git a/tools/ci-runners/Dockerfile b/tools/ci-runners/Dockerfile index 10b9ef9f39..78de8c2452 100644 --- a/tools/ci-runners/Dockerfile +++ b/tools/ci-runners/Dockerfile @@ -49,6 +49,36 @@ RUN mkdir -p "$ANDROID_HOME" \ && unzip -q /tmp/pt.zip -d "$ANDROID_HOME" && rm /tmp/pt.zip \ && ln -s "$ANDROID_HOME/platform-tools/adb" /usr/local/bin/adb +# The SDK every routed native-platforms Android step calls into. `sdkmanager` is the whole of it: +# `android-actions/setup-android` uses an already-installed `cmdline-tools/latest` when the version +# matches, the workflows run their own `sdkmanager "platforms;android-35" "build-tools;35.0.0" +# "ndk;28.2.13676358"`, and `android-emulator-runner` installs what it is missing from it. The +# versions are the ones this repository pins — build-tools 35.0.0, which is also the emulator +# action's own constant; platforms 35 for the parity lane's api-level and 36, which +# packages/runtime-native/android/app/build.gradle.kts declares as compileSdk; and the NDK, which +# that same file names in `ndkVersion` and which the Android V8 build refuses to start without. +# Licenses are accepted at build time so none of those job-time installs stalls on a prompt. +# `emulator` and the system image are deliberately absent: android-emulator-runner installs both +# itself, which is the only Android job that wants them. +ARG CMDLINE_TOOLS=12266719 +RUN set -eu; \ + curl -fsSL "https://dl.google.com/android/repository/commandlinetools-linux-${CMDLINE_TOOLS}_latest.zip" -o /tmp/cli.zip; \ + mkdir -p "$ANDROID_HOME/cmdline-tools"; \ + unzip -q /tmp/cli.zip -d "$ANDROID_HOME/cmdline-tools"; \ + rm /tmp/cli.zip; \ + mv "$ANDROID_HOME/cmdline-tools/cmdline-tools" "$ANDROID_HOME/cmdline-tools/latest"; \ + export PATH="$ANDROID_HOME/cmdline-tools/latest/bin:$PATH"; \ + yes | sdkmanager --licenses > /dev/null; \ + sdkmanager --install "platform-tools" "platforms;android-35" "platforms;android-36" \ + "build-tools;35.0.0" "ndk;28.2.13676358" > /dev/null; \ + rm -rf "$ANDROID_HOME/.temp"; \ + # `runner` owns the SDK, because sdkmanager writes into ANDROID_HOME at job time: the workflows \ + # install their own packages and android-emulator-runner installs the emulator, the platform and \ + # the system image, and a root-owned SDK fails the write as the unprivileged job user. Chowned \ + # here rather than in a later layer, which would copy the whole SDK up and double its size. \ + chown -R runner:runner "$ANDROID_HOME" +ENV PATH=/usr/local/lib/android/sdk/cmdline-tools/latest/bin:$PATH + COPY entrypoint.sh /home/runner/entrypoint.sh # Rust as hosted ubuntu-24.04 has it (stable, on PATH): the native physics ABI is built with cargo. diff --git a/tools/ci-runners/entrypoint.sh b/tools/ci-runners/entrypoint.sh index 5b128398ec..4a9675d418 100755 --- a/tools/ci-runners/entrypoint.sh +++ b/tools/ci-runners/entrypoint.sh @@ -6,6 +6,16 @@ # one job, exits, and the host loop starts a fresh one. Nothing on the filesystem crosses jobs. set -euo pipefail +# `docker run --device /dev/kvm` creates the node 0660 root:kvm, which the `runner` user cannot +# open: `android-emulator-runner` reads `/dev/kvm` for read and write before it starts anything and +# falls back to a software boot, which is the 474-second one native-platforms.yml's parity lane +# exists to avoid. udev cannot fix it here — no udev runs in the container and its /sys is +# read-only, which is why that lane's `udevadm` lines are tolerant — so the mode is set here, +# before the job can look. Root through the runner image's passwordless sudo; no-op without KVM. +if [ -e /dev/kvm ]; then + sudo chmod 0666 /dev/kvm +fi + # Fail closed and say which variable is missing. An ephemeral runner that registers without # TN_RUNNER_REPO would point itself at whatever repository the token defaults to. : "${RUNNER_ADMIN_TOKEN:?RUNNER_ADMIN_TOKEN is not set; pass the operator env file with --env-file}" From b1ae34f2cc60622ed34ec045f46bbe6d768b454b Mon Sep 17 00:00:00 2001 From: Joao Paulo Furtado Date: Fri, 2 Oct 2026 15:53:02 -0700 Subject: [PATCH 15/18] docs(prd-480): AC-1 green on tn-local; focused checks then push while TN_RUNNER is set Run 37070815769: 39 jobs green, 33 on tn-local, 5 on tn-local-light, supply-chain hosted. 38 distinct runner names, so each job got a fresh container. ci-required started 3 s after its last need. AC-2 measured and not met (317 min summed wait, 42 min wall vs 39 hosted); owner decision recorded as pending. --- AGENTS.md | 2 +- CLAUDE.md | 2 +- ...-480-linux-ci-runs-on-the-owner-machine.md | 31 ++++++++++--------- 3 files changed, 18 insertions(+), 17 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index b78c59b2e0..0b7b0e36ad 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -87,7 +87,7 @@ TypeScript 5.9 `strict`, **ESM only**; relative imports end in `.js`. Versions c Results go in the existing PRD, PR or response; a separate evidence file only when the user asks or a workflow consumes it. Prose edits need only the doc checks and mirror regeneration — no implementation checkpoints or artificial negative controls. **Fail closed**: malformed input throws, a missing observation fails, an empty assertion set fails. `pnpm test` proves units; **a playtest scenario proves the game** — every runtime-behaviour change gets one, rerun on each later change to that behaviour. -**Nearest lane first, CI last.** A CI round trip costs over an hour; prove it locally (unit test, playtest, emulator, attached device) and push once green. **Prefer an emulator over hardware** when it can hold the claim; a stopped emulator means start it, not "blocked". Never call `xvfb-run` (its exit status is its own cleanup kill; `sh scripts/xvfb.sh ` is the wrapper). An unnamed WebGPU adapter may be SwiftShader: pass `--browser-recipe webgpu` and check `adapter.info`. Free a port with `lsof -ti tcp: | xargs -r kill`; `pkill -f vite` kills your own shell. +**Nearest lane first, CI last.** A CI round trip costs over an hour; prove it locally (unit test, playtest, emulator, attached device) and push once green. While `TN_RUNNER` is set, run only the focused checks for what you changed and push: the full board runs once, on `tn-local`. **Prefer an emulator over hardware** when it can hold the claim; a stopped emulator means start it, not "blocked". Never call `xvfb-run` (its exit status is its own cleanup kill; `sh scripts/xvfb.sh ` is the wrapper). An unnamed WebGPU adapter may be SwiftShader: pass `--browser-recipe webgpu` and check `adapter.info`. Free a port with `lsof -ti tcp: | xargs -r kill`; `pkill -f vite` kills your own shell. Long gates write `artifacts/gates/status.json`: read it with `pnpm gate:status`, probe a stale phase with `pnpm gate:doctor`; `pnpm gate:resume` continues only while worktree, branch, HEAD, lease and artifact still match. None of them repair or remove a worktree. ## Working outside the repo diff --git a/CLAUDE.md b/CLAUDE.md index a210fa8869..71f7a5bd41 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -89,7 +89,7 @@ TypeScript 5.9 `strict`, **ESM only**; relative imports end in `.js`. Versions c Results go in the existing PRD, PR or response; a separate evidence file only when the user asks or a workflow consumes it. Prose edits need only the doc checks and mirror regeneration — no implementation checkpoints or artificial negative controls. **Fail closed**: malformed input throws, a missing observation fails, an empty assertion set fails. `pnpm test` proves units; **a playtest scenario proves the game** — every runtime-behaviour change gets one, rerun on each later change to that behaviour. -**Nearest lane first, CI last.** A CI round trip costs over an hour; prove it locally (unit test, playtest, emulator, attached device) and push once green. **Prefer an emulator over hardware** when it can hold the claim; a stopped emulator means start it, not "blocked". Never call `xvfb-run` (its exit status is its own cleanup kill; `sh scripts/xvfb.sh ` is the wrapper). An unnamed WebGPU adapter may be SwiftShader: pass `--browser-recipe webgpu` and check `adapter.info`. Free a port with `lsof -ti tcp: | xargs -r kill`; `pkill -f vite` kills your own shell. +**Nearest lane first, CI last.** A CI round trip costs over an hour; prove it locally (unit test, playtest, emulator, attached device) and push once green. While `TN_RUNNER` is set, run only the focused checks for what you changed and push: the full board runs once, on `tn-local`. **Prefer an emulator over hardware** when it can hold the claim; a stopped emulator means start it, not "blocked". Never call `xvfb-run` (its exit status is its own cleanup kill; `sh scripts/xvfb.sh ` is the wrapper). An unnamed WebGPU adapter may be SwiftShader: pass `--browser-recipe webgpu` and check `adapter.info`. Free a port with `lsof -ti tcp: | xargs -r kill`; `pkill -f vite` kills your own shell. Long gates write `artifacts/gates/status.json`: read it with `pnpm gate:status`, probe a stale phase with `pnpm gate:doctor`; `pnpm gate:resume` continues only while worktree, branch, HEAD, lease and artifact still match. None of them repair or remove a worktree. ## Working outside the repo diff --git a/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md b/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md index ee087bbd0d..f956ce4b29 100644 --- a/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md +++ b/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md @@ -98,12 +98,16 @@ while `TN_RUNNER` is set: jobs queue until the switch is cleared. ## Acceptance Criteria -- [ ] AC-1 [shared]: proof: CI run id plus `runner_name` per job from `gh api …/runs//jobs`. A non-draft +- [x] AC-1 [shared]: proof: CI run id plus `runner_name` per job from `gh api …/runs//jobs`. A non-draft PR's `CI` run executes every Linux job except `supply-chain` on a `tn-local` runner, and `ci-required` passes. Evidence: pending. + Evidence: CI run 37070815769 (PR #404, head 94732de65, 2026-10-02): conclusion success, `ci-required` success. 39 jobs succeeded: 33 on `tn-local`, 5 on `tn-local-light`, 1 hosted (`supply-chain`); `native-platforms` skipped by scope. - [ ] AC-2 [shared]: proof: per-job `started_at − created_at` summed with the baseline's script. Total time jobs spent waiting for a runner on a full-board PR drops below 60 min (baseline 239 min, run - 37043413533). Evidence: pending. + 37043413533). Measured, not met: run 37070815769 waited 317 min in total across 40 jobs on 5 heavy + + 1 light slots, with 42 min wall against the baseline's 39 min wall and zero hosted minutes. A 40-job board on + six slots queues by construction, so this criterion needs either more slots or overflow to hosted + runners. Owner decision pending. ## Blocked on @@ -162,10 +166,11 @@ the stack. The script fails closed when the env file or token is missing. tn-6071a30e949e, all online. The first `up` timed out with 0 online (wrong WORKDIR), cleared the variable and stopped the pool, which is the fail-closed path working. Not started: `status` reports `TN_RUNNER=unset`, 0 containers, 0 online runners on 2026-10-02. -- [ ] A runner container recreates itself after its job and keeps no state from it. proof: two +- [x] A runner container recreates itself after its job and keeps no state from it. proof: two consecutive dispatched jobs report different container hostnames and an empty `/tmp`. Not started: the image's `/tmp` is empty in a fresh container (`ls -A /tmp | wc -l` → 0), which is one container's worth of the claim. + Evidence: CI run 37070815769 (PR #404, head 94732de65, 2026-10-02): 38 `tn-local*` jobs ran on 38 distinct runner names (`tn-`) from 6 slots, so every job got a fresh container; each starts from the image, whose `/tmp` is empty (`ls -A /tmp | wc -l` = 0 in the smoke). #### Phase 2: `ci.yml` and the integration workflows route through the switch @@ -188,20 +193,16 @@ and `integration-decals.yml`'s `ubuntu-24.04-arm` job are the only Linux `runs-o One pre-existing assertion had to be narrowed: `template-nonvisual` forbade the substring `pull_request` anywhere in the job, which the routing expression mentions without gating on it. -- [ ] With `TN_RUNNER` set, a PR run lands its Linux jobs on `tn-local`. proof: AC-1 run. -- [ ] `AGENTS.md` says focused checks then push, not the full board, while `TN_RUNNER` is set. proof: +- [x] With `TN_RUNNER` set, a PR run lands its Linux jobs on `tn-local`. proof: AC-1 run. + Evidence: CI run 37070815769 (PR #404, head 94732de65, 2026-10-02), as AC-1. +- [x] `AGENTS.md` says focused checks then push, not the full board, while `TN_RUNNER` is set. proof: `pnpm sync:agents --check` and `pnpm exec vitest run scripts/__tests__/sync-agent-docs.spec.ts` pass. Land it only after the AC-1 run is green. -- [ ] Light jobs never wait behind heavy ones: `scope`, `ci-required` and `run-summary` route to - `tn-local-light` and no heavy job can. proof: `ci-structure.spec.ts` case, plus a full-board run whose - `ci-required` starts within 60 s of its last `needs` finishing. - Landed, unticked because the timing half is unmeasured: `up` starts one unpinned - `--cpus 1 --memory 2g --oom-score-adj 900` slot registering `tn-local-light` and sets - `TN_RUNNER_LIGHT`, and `scope`, `golden-path`, `build`, `ci-required`, `run-summary` route to it — - every job whose work is a script or a summary, none of which builds the workspace or runs a test. - `ci-structure.spec.ts` enforces the allow-list in both directions (red on `typecheck` put on the - light expression, red on `scope` put off it). Left: a full-board run whose `ci-required` starts - within 60 s of its last `needs` finishing. + Evidence: landed after AC-1; `pnpm sync:agents` + sync-agent-docs, primary-docs and instruction-budget specs, 24 passed (2026-10-02). +- [x] Light jobs never wait behind heavy ones (proof: `ci-structure.spec.ts` case, plus a full-board run + whose `ci-required` starts within 60 s of its last `needs` finishing): `scope`, `golden-path`, `build`, + `ci-required` and `run-summary` route to `tn-local-light`, and no heavy job can. + Evidence: ci-structure.spec.ts routing case (red with `typecheck` on the light lane), and CI run 37070815769 (PR #404, head 94732de65, 2026-10-02): `ci-required` started 22:48:50Z, 3 s after the last job it needs finished at 22:48:47Z. - [ ] With `TN_RUNNER` unset, the same workflow runs fully hosted. proof: `workflow_dispatch` run id with every `runner_name` hosted. From c15d2fdcc7da024ddf367a15e30ef9dfd4471e0d Mon Sep 17 00:00:00 2001 From: Joao Paulo Furtado Date: Fri, 2 Oct 2026 16:44:17 -0700 Subject: [PATCH 16/18] fix(ci-runners): emulator lane and starter matrix pass on tn-local Run 37074908699: the emulator lane failed writing its udev rule because the image had no /etc/udev/rules.d; and starter-desktop.test.mjs pinned a bare ubuntu-24.04 for the linux-x64 row that now routes through TN_RUNNER. Census regenerated for the runtime-native test edit. --- docs/verification/native-runtime-census-2026-08-16.md | 4 ++-- packages/runtime-native/tests/starter-desktop.test.mjs | 6 +++++- tools/ci-runners/Dockerfile | 3 ++- 3 files changed, 9 insertions(+), 4 deletions(-) diff --git a/docs/verification/native-runtime-census-2026-08-16.md b/docs/verification/native-runtime-census-2026-08-16.md index ebb40e53e1..330c22b0c5 100644 --- a/docs/verification/native-runtime-census-2026-08-16.md +++ b/docs/verification/native-runtime-census-2026-08-16.md @@ -17,7 +17,7 @@ considered, and a KEEP/DELETE verdict. | --- | ---: | --- | --- | --- | --- | | `src/` | 57,757 | PRD-045, PRD-047, PRD-048, PRD-050, PRD-053, PRD-116, PRD-143, PRD-153, PRD-154, PRD-155, PRD-292 | `src/physics/native_bindings.cpp:586`; desktop V8 and native runtime commands | Move host shims into each game or delete the native host | **KEEP** — this is the owned native host and its physics boundary. | | `conformance/` | 9,477 | PRD-053, PRD-054, PRD-055, PRD-076 | `conformance/run-conformance.mjs`; root `pnpm parity` | Replace cross-target registry/proofs with untested per-game scripts | **KEEP** — shared executable conformance evidence. | -| `tests/` | 57,849 | PRD-045, PRD-046, PRD-048, PRD-049, PRD-050, PRD-053, PRD-054, PRD-055, PRD-076, PRD-116, PRD-127, PRD-143, PRD-153, PRD-154, PRD-155, PRD-292 | `tests/physics_actuation_bindings_test.cpp:141`; runtime-native Vitest and native tests | Delete fail-closed tests to reduce the trigger | **KEEP** — removal would conceal regressions. | +| `tests/` | 57,853 | PRD-045, PRD-046, PRD-048, PRD-049, PRD-050, PRD-053, PRD-054, PRD-055, PRD-076, PRD-116, PRD-127, PRD-143, PRD-153, PRD-154, PRD-155, PRD-292 | `tests/physics_actuation_bindings_test.cpp:141`; runtime-native Vitest and native tests | Delete fail-closed tests to reduce the trigger | **KEEP** — removal would conceal regressions. | | `__tests__/` | 936 | PRD-228 | `__tests__/host-gap-gpu-drain.spec.ts`; the runtime-native Vitest suite via `vitest.config.ts` | Fold it into `tests/`, where every other spec in this package lives | **KEEP** — it is one source-reading spec proving the GPU-drain profile stays diagnostic-only and never ships, and it is picked up by the same suite either way. Its one-file directory is an accident of where it landed, not a design; moving it into `tests/` is a tidy-up and not this row's argument. | | `scripts/` | 30,902 | PRD-045, PRD-048, PRD-049, PRD-050, PRD-053, PRD-054, PRD-116, PRD-127, PRD-153, PRD-155, PRD-292 | `scripts/verify-desktop-physics.mjs:206-228`; build, platform, and device-condition verifiers | Make every game own packaging, device, and verifier orchestration | **KEEP** — no smaller shared alternative preserves the proof. | | `scenarios/` | 109 | PRD-217 | `scenarios/starter-ui-overlay-desktop.playtest.json`; run by the hosted `native-platforms` desktop-core job through `scripts/verify-starter-ui-overlay.mjs` | Keep the starter-HUD input assertion inside the desktop-core job script instead of a standalone scenario file | **KEEP** — the scenario is the portable playtest that proves the starter HUD receives real input on the packed desktop runtime; the hosted desktop-core job is its caller, so the area is one owned proof rather than reusable game content. | @@ -35,7 +35,7 @@ considered, and a KEEP/DELETE verdict. | `js-engine-versions.json` | 38 | PRD-222 | `tests/js-engine-version-skew.test.mjs`; `scripts/download-deps.mjs` consumes the pinned engine versions | Leave engine/version compatibility as prose or duplicate pins across download scripts | **KEEP** — one fail-closed manifest prevents desktop, Android, and iOS engine pins from drifting silently. | | `build-matrix.json` | 231 | PRD-235 | `scripts/check-build-matrix.ts` from `scripts/check-budgets.ts:467` via `pnpm budgets`; proven by `scripts/__tests__/check-build-matrix.spec.ts` | Leave "which build directory does this test executable live in" as prose in `AGENTS.md` and tribal knowledge | **KEEP** — the prose answer was already wrong once: a contract test that only links in `build/tn-linux` is invisible to anyone reading `CMakeLists.txt`. This is the machine-readable contract, and the gate cross-checks every preset, cache variable, and test executable it names against CMake, so either side drifting is a named failure. | | `native-deps.lock.json` | 1,050 | PRD-059 | `scripts/download-deps.mjs` reads it before any acquisition; `tests/dependency-provenance.test.mjs` proves lock coverage | Leave dependency URLs/versions/hashes as inline mutable code in `download-deps.mjs` | **KEEP** — the reviewed immutable payload matrix is what makes reconstruction mean the same bytes; inline authority already served one 404. | -| **Total** | **180,355** | | `pnpm budgets` current measurement | | **No area rejected.** | +| **Total** | **180,359** | | `pnpm budgets` current measurement | | **No area rejected.** | **Regenerated 2026-09-02 for PRD-292 recovery:** the native physics ABI, its contract test, and the continuous-collision measurement harness changed the counted `src/`, `include/`, `tests/`, diff --git a/packages/runtime-native/tests/starter-desktop.test.mjs b/packages/runtime-native/tests/starter-desktop.test.mjs index f1be41f965..024f0b506f 100644 --- a/packages/runtime-native/tests/starter-desktop.test.mjs +++ b/packages/runtime-native/tests/starter-desktop.test.mjs @@ -387,7 +387,11 @@ test('native workflow proves the Linux release container on x64 and arm64', () = assert.ok(starter); // The arm64 leg is the point of the matrix: a free hosted ARM runner, the same release steps, // and the same scope selection as the x64 leg because it is the same job. - assert.match(starter, /- platform: linux-x64\n\s+runner: ubuntu-24\.04/u); + // linux-x64 routes through the TN_RUNNER switch (PRD-480) and falls back to the same hosted image. + assert.match( + starter, + /- platform: linux-x64\n\s+runner: \$\{\{ \(github\.event\.pull_request\.head\.repo\.fork \|\| !vars\.TN_RUNNER\) && 'ubuntu-24\.04' \|\| vars\.TN_RUNNER \}\}/u, + ); assert.match(starter, /- platform: linux-arm64\n\s+runner: ubuntu-24\.04-arm/u); assert.match(starter, /runs-on: \$\{\{ matrix\.runner \}\}/u); // The runner has no GPU, so the software Vulkan ICD and its X display are provisioned. diff --git a/tools/ci-runners/Dockerfile b/tools/ci-runners/Dockerfile index 78de8c2452..96995b0e61 100644 --- a/tools/ci-runners/Dockerfile +++ b/tools/ci-runners/Dockerfile @@ -44,7 +44,8 @@ RUN curl -fsSL "https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-l ENV ANDROID_HOME=/usr/local/lib/android/sdk ANDROID_SDK_ROOT=/usr/local/lib/android/sdk # Hosted runners run in C.UTF-8; without it javac reads sources as US-ASCII and rejects an em dash. ENV LANG=C.UTF-8 LC_ALL=C.UTF-8 -RUN mkdir -p "$ANDROID_HOME" \ +# The emulator lane writes a udev rule before it starts; hosted has the directory, the base image does not. +RUN mkdir -p /etc/udev/rules.d "$ANDROID_HOME" \ && curl -fsSL https://dl.google.com/android/repository/platform-tools-latest-linux.zip -o /tmp/pt.zip \ && unzip -q /tmp/pt.zip -d "$ANDROID_HOME" && rm /tmp/pt.zip \ && ln -s "$ANDROID_HOME/platform-tools/adb" /usr/local/bin/adb From 11d686c68f0cb7af88cfdc3f840087a8ae51f416 Mon Sep 17 00:00:00 2001 From: Joao Paulo Furtado Date: Fri, 2 Oct 2026 17:18:32 -0700 Subject: [PATCH 17/18] fix(ci): Windows packaging retries the Chocolatey installs Run 37079046851 and the develop promotion both failed Windows desktop core on "nsis not installed. The package was not found" for a version the feed still lists. Three attempts, 30 s apart; makensis -VERSION still fails closed. --- .github/workflows/native-platforms.yml | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/.github/workflows/native-platforms.yml b/.github/workflows/native-platforms.yml index b517c9905a..8283ac9855 100644 --- a/.github/workflows/native-platforms.yml +++ b/.github/workflows/native-platforms.yml @@ -1254,8 +1254,12 @@ jobs: # The release container is a ZIP on macOS and Windows and desktop-distribution.mjs shells # out to `zip -r -q`. A missing archiver must be a red job, never a silent pass. if [ "${{ matrix.platform }}" = "Windows" ]; then - choco install zip -y --no-progress - choco install nsis --version=3.11.0 -y --no-progress + # The community feed drops requests now and then ("package was not found" for a listed + # version, run 37079046851). Retry twice; makensis -VERSION below still fails closed. + for attempt in 1 2 3; do + choco install zip -y --no-progress && choco install nsis --version=3.11.0 -y --no-progress && break + [ "$attempt" = 3 ] || sleep 30 + done nsis='/c/Program Files (x86)/NSIS' "$nsis/makensis.exe" -VERSION cygpath -w "$nsis" >> "$GITHUB_PATH" From 49e2f5eabf5b4dcd7eb06a39d0eb4dd606952ce8 Mon Sep 17 00:00:00 2001 From: Joao Paulo Furtado Date: Fri, 2 Oct 2026 19:23:38 -0700 Subject: [PATCH 18/18] ci(prd-480): Android emulator parity stays hosted On tn-local the emulator got KVM but renders through SwiftShader on the CPU; run 37082733117 spent 41 of its 45 minutes still running APKs (hosted ~30 min). The hosted allow-list names it with that reason; every other Linux leg still routes. --- .github/workflows/native-platforms.yml | 4 +++- docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md | 2 ++ scripts/__tests__/ci-structure.spec.ts | 7 ++++++- 3 files changed, 11 insertions(+), 2 deletions(-) diff --git a/.github/workflows/native-platforms.yml b/.github/workflows/native-platforms.yml index 8283ac9855..982022f728 100644 --- a/.github/workflows/native-platforms.yml +++ b/.github/workflows/native-platforms.yml @@ -322,7 +322,9 @@ jobs: needs.scope.outputs.selection == 'full' && inputs.ios_only != true && needs.android-v8-source.result == 'success' - runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} + # Hosted, measured: SwiftShader renders the emulator's GPU on the CPU, and on a 4-thread tn-local + # slot run 37082733117 spent 41 min of this job's 45 still running APKs (hosted finishes in ~30). + runs-on: ubuntu-24.04 timeout-minutes: 45 env: TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} diff --git a/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md b/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md index f956ce4b29..35a58a727f 100644 --- a/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md +++ b/docs/PRDs/CI/PRD-480-linux-ci-runs-on-the-owner-machine.md @@ -242,4 +242,6 @@ Decisions this phase had to make, 2026-10-02: - [ ] A `native-platforms` run passes with its Linux legs on `tn-local`. proof: run id plus per-job `runner_name`. + Exception, measured: `android-emulator-parity` stays hosted. SwiftShader renders its GPU on the CPU, and on a + 4-thread slot run 37082733117 spent 41 of the job's 45 minutes still running APKs, where hosted takes ~30. - [ ] Waiting time meets AC-2. proof: the AC-2 measurement on a full-board PR after this phase. diff --git a/scripts/__tests__/ci-structure.spec.ts b/scripts/__tests__/ci-structure.spec.ts index 9833b6cfc8..7193bafec3 100644 --- a/scripts/__tests__/ci-structure.spec.ts +++ b/scripts/__tests__/ci-structure.spec.ts @@ -1031,7 +1031,12 @@ describe("CI pipeline structure", () => { // release writes on GitHub's own machines. const hosted = new Map>([ [".github/workflows/ci.yml", new Set(["supply-chain"])], - [".github/workflows/native-platforms.yml", new Set(["publish-android-v8"])], + [ + ".github/workflows/native-platforms.yml", + // publish-android-v8 writes the release; android-emulator-parity is CPU-bound SwiftShader that + // overran its 45-minute budget on a pinned 4-thread slot (run 37082733117). + new Set(["android-emulator-parity", "publish-android-v8"]), + ], ]); // PRD-480's light lane: the jobs whose whole work is a script or a summary — no workspace build, // no `pnpm install` and no test suite. `scope` classifies the diff, `build` and `golden-path`