From 841946ee905c2751d46d6bbac6b03f442e6d6d1f Mon Sep 17 00:00:00 2001 From: Joao Paulo Furtado Date: Sun, 4 Oct 2026 11:10:23 -0700 Subject: [PATCH 1/2] ci: keep representative template smoke and classify integration dependencies --- AGENTS.md | 2 +- CLAUDE.md | 2 +- .../__tests__/ci-integration-scope.spec.ts | 221 +++++++++++++++++- .../__tests__/ci-template-selection.spec.ts | 140 ++++++++++- scripts/ci-change-scope.mjs | 57 ++++- scripts/ci-integration-scope.mjs | 97 +++++++- 6 files changed, 506 insertions(+), 13 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 657e664c96..c0b64e040c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -77,7 +77,7 @@ pnpm native:build # opt-in; downloads deps, compiles th pnpm native:verify:desktop # 300 native frames + a non-blank screenshot ``` -**CI.** The pre-push hook runs `pnpm ci:fast` drift checks only; it proves no runtime behaviour. `pnpm ci:local --full` runs the full board; `pnpm ci:local --affected --base origin/develop --target develop` runs what CI would select (dirty or unknown inputs fall back to full). A change touching only Markdown that no fixture consumes (not `AGENTS.md`/`CLAUDE.md`) runs no product CI job. PRs and merge groups into develop use the complete candidate diff: CI-only changes run CI contracts, exact template changes run scaffolder contracts and affected templates, and shared/unknown executable inputs keep exhaustive coverage. Main qualification and scheduled audits remain exhaustive. Template scenarios run once per kit; selective changes run one representative golden journey and full qualification retains both starter and platformer journeys. `ci-required` rejects missing or failed selected checks; retained contexts include `supply-chain`, `template-nonvisual`, `golden-path`, and native `desktop-parity` when applicable. Native host/platform tests run for native or shared runtime consumers, main qualification and unresolved scopes. Unsafe matrix/reusable verdict reuse stays disabled until authoritative complete coverage and per-leg routing exist. A new candidate runs fresh tests; only exact-run immutable build artifacts are shared. Details: `docs/PRDs/CI/PRD-483-impact-driven-ci-without-repeated-template-setup.md` and `docs/PRDs/done/PRD-373-selective-ci-and-develop-promotion.md`. +**CI.** The pre-push hook runs `pnpm ci:fast` drift checks only; it proves no runtime behaviour. `pnpm ci:local --full` runs the full board; `pnpm ci:local --affected --base origin/develop --target develop` runs what CI would select (dirty or unknown inputs fall back to full). A change touching only Markdown that no fixture consumes (not `AGENTS.md`/`CLAUDE.md`) runs no product CI job. PRs and merge groups into develop use the complete candidate diff: CI-only changes run CI contracts, exact template changes run scaffolder contracts and affected templates, and shared executable inputs keep the full safety board with starter as the default template smoke plus explicitly changed runtime/scenario kits. Template instruction and Markdown changes retain their contract lanes without adding runtime kits. Unknown executable inputs, main qualification and scheduled audits remain exhaustive. Template scenarios run once per selected kit; develop changes run one representative golden journey and full qualification retains both starter and platformer journeys. `ci-required` rejects missing or failed selected checks; retained contexts include `supply-chain`, `template-nonvisual`, `golden-path`, and native `desktop-parity` when applicable. Native host/platform tests run for native or shared runtime consumers, main qualification and unresolved scopes. Unsafe matrix/reusable verdict reuse stays disabled until authoritative complete coverage and per-leg routing exist. A new candidate runs fresh tests; only exact-run immutable build artifacts are shared. Details: `docs/PRDs/CI/PRD-483-impact-driven-ci-without-repeated-template-setup.md` and `docs/PRDs/done/PRD-373-selective-ci-and-develop-promotion.md`. Registry commands read auth from `~/.npmrc`; pass a checkout-local untracked `.npmrc` with `npm --userconfig .npmrc `. Never print either file. TypeScript 5.9 `strict`, **ESM only**; relative imports end in `.js`. Versions come from the `catalog:` in `pnpm-workspace.yaml`, templates excepted. Biome formats — never hand-format. Interfaces are `I`-prefixed; classes and type aliases are not. Unit tests: `/__tests__/*.spec.ts`, vitest, node environment, DOM and GPU stubbed. diff --git a/CLAUDE.md b/CLAUDE.md index 8a6cabc2b8..325c74b1b4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -79,7 +79,7 @@ pnpm native:build # opt-in; downloads deps, compiles th pnpm native:verify:desktop # 300 native frames + a non-blank screenshot ``` -**CI.** The pre-push hook runs `pnpm ci:fast` drift checks only; it proves no runtime behaviour. `pnpm ci:local --full` runs the full board; `pnpm ci:local --affected --base origin/develop --target develop` runs what CI would select (dirty or unknown inputs fall back to full). A change touching only Markdown that no fixture consumes (not `AGENTS.md`/`CLAUDE.md`) runs no product CI job. PRs and merge groups into develop use the complete candidate diff: CI-only changes run CI contracts, exact template changes run scaffolder contracts and affected templates, and shared/unknown executable inputs keep exhaustive coverage. Main qualification and scheduled audits remain exhaustive. Template scenarios run once per kit; selective changes run one representative golden journey and full qualification retains both starter and platformer journeys. `ci-required` rejects missing or failed selected checks; retained contexts include `supply-chain`, `template-nonvisual`, `golden-path`, and native `desktop-parity` when applicable. Native host/platform tests run for native or shared runtime consumers, main qualification and unresolved scopes. Unsafe matrix/reusable verdict reuse stays disabled until authoritative complete coverage and per-leg routing exist. A new candidate runs fresh tests; only exact-run immutable build artifacts are shared. Details: `docs/PRDs/CI/PRD-483-impact-driven-ci-without-repeated-template-setup.md` and `docs/PRDs/done/PRD-373-selective-ci-and-develop-promotion.md`. +**CI.** The pre-push hook runs `pnpm ci:fast` drift checks only; it proves no runtime behaviour. `pnpm ci:local --full` runs the full board; `pnpm ci:local --affected --base origin/develop --target develop` runs what CI would select (dirty or unknown inputs fall back to full). A change touching only Markdown that no fixture consumes (not `AGENTS.md`/`CLAUDE.md`) runs no product CI job. PRs and merge groups into develop use the complete candidate diff: CI-only changes run CI contracts, exact template changes run scaffolder contracts and affected templates, and shared executable inputs keep the full safety board with starter as the default template smoke plus explicitly changed runtime/scenario kits. Template instruction and Markdown changes retain their contract lanes without adding runtime kits. Unknown executable inputs, main qualification and scheduled audits remain exhaustive. Template scenarios run once per selected kit; develop changes run one representative golden journey and full qualification retains both starter and platformer journeys. `ci-required` rejects missing or failed selected checks; retained contexts include `supply-chain`, `template-nonvisual`, `golden-path`, and native `desktop-parity` when applicable. Native host/platform tests run for native or shared runtime consumers, main qualification and unresolved scopes. Unsafe matrix/reusable verdict reuse stays disabled until authoritative complete coverage and per-leg routing exist. A new candidate runs fresh tests; only exact-run immutable build artifacts are shared. Details: `docs/PRDs/CI/PRD-483-impact-driven-ci-without-repeated-template-setup.md` and `docs/PRDs/done/PRD-373-selective-ci-and-develop-promotion.md`. Registry commands read auth from `~/.npmrc`; pass a checkout-local untracked `.npmrc` with `npm --userconfig .npmrc `. Never print either file. TypeScript 5.9 `strict`, **ESM only**; relative imports end in `.js`. Versions come from the `catalog:` in `pnpm-workspace.yaml`, templates excepted. Biome formats — never hand-format. Interfaces are `I`-prefixed; classes and type aliases are not. Unit tests: `/__tests__/*.spec.ts`, vitest, node environment, DOM and GPU stubbed. diff --git a/scripts/__tests__/ci-integration-scope.spec.ts b/scripts/__tests__/ci-integration-scope.spec.ts index 0204664693..54b6539a1a 100644 --- a/scripts/__tests__/ci-integration-scope.spec.ts +++ b/scripts/__tests__/ci-integration-scope.spec.ts @@ -1,10 +1,17 @@ import { spawnSync } from "node:child_process"; -import { mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { existsSync, mkdirSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs"; import path from "node:path"; +import ts from "typescript"; import { describe, expect, it } from "vitest"; import { makeTempDirSync } from "../../test-support/temp-dir.js"; const url = new URL("../ci-integration-scope.mjs", import.meta.url).href; -const { integrationSelection, integrationGitSelection } = await import(url); +const { + integrationSelection, + integrationGitSelection, + INDEPENDENT_CORE_MODULES, + INDEPENDENT_PACKAGES, + INDEPENDENT_PROOFS, +} = await import(url); const workflow = readFileSync( new URL("../../.github/workflows/integration.yml", import.meta.url), "utf8", @@ -342,11 +349,11 @@ describe("fluid runtime independence", () => { expect(lanes["fluid-particles"]).toBe(true); }); it.each(["packages/core/src/renderer.ts", "packages/core/src/render/chain.ts"])( - "does not run fluid proofs for unrelated core render source: %s", + "retains fluid consumers for shared renderer and observation source: %s", (file) => { const lanes = select(workflow, [file]); - expect(lanes["fluid-native"]).toBe(false); - expect(lanes["fluid-particles"]).toBe(false); + expect(lanes["fluid-native"]).toBe(true); + expect(lanes["fluid-particles"]).toBe(true); }, ); it("fails closed for an unknown executable harness", () => { @@ -446,3 +453,207 @@ describe("fluid CI observer ownership", () => { }, ); }); + +describe("reviewed producer and independent feature boundaries", () => { + it("keeps fluid for the shared game loop and core build config", () => { + for (const file of [ + "packages/core/src/game.ts", + "packages/core/src/clustered-mesh.ts", + "packages/core/tsup.config.ts", + "packages/core/patches/three.patch", + ]) { + const lanes = select(workflow, [file]); + expect(lanes["fluid-particles"], file).toBe(true); + expect(lanes["fluid-native"], file).toBe(true); + expect(lanes.csg, file).toBe(false); + expect(lanes.ik, file).toBe(false); + expect(lanes.vegetation, file).toBe(false); + } + }); + it("does not select fluid for independent terrain or animation modules", () => { + for (const file of [ + "packages/core/src/terrain-jobs-worker.ts", + "packages/core/src/terrain-jobs.ts", + "packages/core/src/world-tiles.ts", + "packages/core/src/animation.ts", + "scripts/verify-animation-reversal.ts", + "scripts/temporal-aa-quality.ts", + ]) { + const lanes = select(workflow, [file]); + expect(lanes["fluid-particles"], file).toBe(false); + expect(lanes["fluid-native"], file).toBe(false); + } + }); + it("still selects fluid when an independent change shares a producer configuration delta", () => { + const lanes = select(workflow, [ + "packages/core/src/terrain-jobs-worker.ts", + "packages/core/tsup.config.ts", + ]); + expect(lanes["fluid-particles"]).toBe(true); + expect(lanes["fluid-native"]).toBe(true); + expect(lanes.csg).toBe(false); + }); + it("retains unknown executable and scheduling fallback", () => { + expect(select(workflow, ["scripts/unclassified.ts"])).toEqual(all); + expect(select(workflow, ["packages/unknown/src/runtime.ts"])).toEqual(all); + }); +}); + +const dependencyRoot = path.resolve(import.meta.dirname, "../.."); +function runtimeSource(filename: string): string { + const source = filename.replace("/dist/", "/src/"); + const candidates = [ + source.replace(/\.js$/u, ".ts"), + `${source}.ts`, + source, + `${source}/index.ts`, + ]; + const resolved = candidates.find( + (candidate) => existsSync(candidate) && statSync(candidate).isFile(), + ); + if (!resolved) throw new Error(`Unresolved fluid runtime source: ${filename}`); + return resolved; +} +function moduleEdges(file: string): { module: string; names: string[] }[] { + const source = ts.createSourceFile( + file, + readFileSync(file, "utf8"), + ts.ScriptTarget.Latest, + true, + ); + const edges: { module: string; names: string[] }[] = []; + const visit = (node: ts.Node) => { + if ( + ts.isImportDeclaration(node) && + !node.importClause?.isTypeOnly && + ts.isStringLiteral(node.moduleSpecifier) + ) { + const bindings = node.importClause?.namedBindings; + const names = + bindings && ts.isNamedImports(bindings) + ? bindings.elements + .filter((entry) => !entry.isTypeOnly) + .map((entry) => (entry.propertyName ?? entry.name).text) + : ["*"]; + if (names.length) edges.push({ module: node.moduleSpecifier.text, names }); + } + if ( + ts.isExportDeclaration(node) && + !node.isTypeOnly && + node.moduleSpecifier && + ts.isStringLiteral(node.moduleSpecifier) + ) + edges.push({ module: node.moduleSpecifier.text, names: ["*"] }); + if (ts.isCallExpression(node) && node.expression.kind === ts.SyntaxKind.ImportKeyword) { + const argument = node.arguments[0]; + if (!argument || !ts.isStringLiteral(argument)) + throw new Error(`Unresolved runtime import in ${file}`); + edges.push({ module: argument.text, names: ["*"] }); + } + ts.forEachChild(node, visit); + }; + visit(source); + return edges; +} +function workspaceTargets(module: string, names: string[]): string[] { + const match = /^@threenative\/([^/]+)(?:\/(.*))?$/u.exec(module); + if (!match?.[1]) return []; + const entry = runtimeSource( + path.join(dependencyRoot, "packages", match[1], "src", match[2] ?? "index.ts"), + ); + if (names.includes("*")) return [entry]; + // Resolve the public named exports, instead of pretending an unused barrel export is exercised. + const source = ts.createSourceFile( + entry, + readFileSync(entry, "utf8"), + ts.ScriptTarget.Latest, + true, + ); + const exports = source.statements + .filter(ts.isExportDeclaration) + .filter( + (node) => + !node.isTypeOnly && node.moduleSpecifier && ts.isStringLiteral(node.moduleSpecifier), + ); + const targets = new Set(); + for (const name of names) { + const exported = exports.find( + (node) => + node.exportClause && + ts.isNamedExports(node.exportClause) && + node.exportClause.elements.some( + (element) => !element.isTypeOnly && element.name.text === name, + ), + ); + if (exported?.moduleSpecifier && ts.isStringLiteral(exported.moduleSpecifier)) + targets.add(runtimeSource(path.resolve(path.dirname(entry), exported.moduleSpecifier.text))); + else targets.add(entry); + } + return [...targets]; +} +function fluidRuntimeClosure(extra: { module: string; names: string[] }[] = []): Set { + const pending = [ + "examples/prd476-fluid-particles/src/main.ts", + "examples/prd476-fluid-particles/src/collision-proof.ts", + "scripts/verify-fluid-collision.ts", + "scripts/verify-fluid-consumers.ts", + "scripts/verify-fluid-collision-native.ts", + ].map((file) => path.join(dependencyRoot, file)); + pending.push(...extra.flatMap(({ module, names }) => workspaceTargets(module, names))); + const seen = new Set(); + while (pending.length) { + const file = pending.pop() as string; + if (seen.has(file)) continue; + seen.add(file); + for (const edge of moduleEdges(file)) { + if (edge.module.startsWith(".")) + pending.push(runtimeSource(path.resolve(path.dirname(file), edge.module))); + else pending.push(...workspaceTargets(edge.module, edge.names)); + } + } + return seen; +} +it("guards independent modules against the actual fluid fixture/harness runtime graph", () => { + const seen = fluidRuntimeClosure(); + expect( + [...seen].filter((file) => INDEPENDENT_PROOFS.test(path.relative(dependencyRoot, file))), + ).toEqual([]); + for (const name of INDEPENDENT_PACKAGES) + expect( + [...seen].some((file) => file.includes(`/packages/${name}/src/`)), + name, + ).toBe(false); + for (const name of INDEPENDENT_CORE_MODULES) + expect(seen.has(path.join(dependencyRoot, `packages/core/src/${name}.ts`)), name).toBe(false); + for (const name of ["clustered-mesh", "renderer", "render/chain", "gpu-readback"]) + expect(seen.has(path.join(dependencyRoot, `packages/core/src/${name}.ts`)), name).toBe(true); +}); +it("detects a new fixture dependency through a named public export", () => { + const seen = fluidRuntimeClosure([{ module: "@threenative/core", names: ["AnimationPlayer"] }]); + expect(seen.has(path.join(dependencyRoot, "packages/core/src/animation.ts"))).toBe(true); +}); +it("rejects an unresolved new workspace-package import instead of accepting an exclusion", () => { + expect(() => + fluidRuntimeClosure([{ module: "@threenative/unclassified", names: ["newDependency"] }]), + ).toThrow("Unresolved fluid runtime source"); +}); + +it("fails closed when a shared producer reaches a newly added unclassified consumer", () => { + const added = [ + workflow + .replace( + " outputs:\n", + " outputs:\n new-core: ${{ steps.filter.outputs.new-core }}\n", + ) + .replace( + " TN_LANE_FILTERS: |\n", + " TN_LANE_FILTERS: |\n new-core ^examples/new-core/.*$\n", + ), + "\n new-core:\n needs: paths\n if: ${{ needs.paths.outputs.new-core == 'true' && !github.event.pull_request.draft }}\n runs-on: ubuntu-24.04\n steps:\n - run: pnpm build\n", + ].join(""); + expect(select(added, [".github/workflows/integration.yml"])).toEqual({ + ...none, + "new-core": true, + }); + expect(select(added, ["packages/core/tsup.config.ts"])).toEqual({ ...all, "new-core": true }); +}); diff --git a/scripts/__tests__/ci-template-selection.spec.ts b/scripts/__tests__/ci-template-selection.spec.ts index 054d4d4503..26f5380383 100644 --- a/scripts/__tests__/ci-template-selection.spec.ts +++ b/scripts/__tests__/ci-template-selection.spec.ts @@ -110,7 +110,13 @@ describe("impact-driven template coverage", () => { eventName: "pull_request", }); expect(plan.selection).toBe("full"); - expect(plan.templateMatrix.template).toEqual(TEMPLATE_NAMES); + expect(plan.templateMatrix.template).toEqual( + file.startsWith("unknown/") || + file.includes("lookalike") || + file === "scripts/stamp-template-render.ts" + ? TEMPLATE_NAMES + : ["starter"], + ); if (file.startsWith("packages/core/")) expect(plan.jobs["test-native"].required).toBe(true); } finally { rmSync(f.root, { recursive: true, force: true }); @@ -298,3 +304,135 @@ it("keeps the exhaustive board at one complete job per manifest plus both retain TEMPLATE_NAMES.length + 2, ); }); + +describe("representative develop smoke with exhaustive qualification", () => { + it.each([ + "packages/core/src/renderer.ts", + "packages/core/tsup.config.ts", + "packages/runtime-native/src/main.cpp", + ])("keeps starter for the classified producer %s", (file) => { + const plan = selectionPlan( + "full", + "resolved dependency", + [file], + "a".repeat(40), + true, + 0, + "develop", + ); + expect(plan.templateMatrix.template).toEqual(["starter"]); + expect(plan.goldenMatrix.template).toEqual(["starter"]); + expect(validatePlan(plan)).toEqual(plan); + expect(() => validatePlan({ ...plan, templateMatrix: { template: ["minimal"] } })).toThrow(); + }); + it("unions explicit runtime kits but ignores kit instructions and inert Markdown", () => { + const files = [ + "packages/core/src/game.ts", + "packages/create-threenative/templates/shooter/src/game.ts", + "packages/create-threenative/templates/snow/AGENTS.md", + "packages/create-threenative/templates/rain/CLAUDE.md", + "packages/create-threenative/templates/racing/README.md", + ]; + const plan = selectionPlan( + "full", + "resolved dependency", + files, + "a".repeat(40), + true, + 0, + "develop", + ); + expect(changedTemplates(files)).toEqual(["shooter"]); + expect(plan.templateMatrix.template).toEqual(["shooter", "starter"]); + expect(plan.goldenMatrix.template).toEqual(["starter"]); + }); + it.each(["main", ""])("keeps exhaustive qualification for %s", (target) => { + const plan = selectionPlan( + "full", + "qualification", + ["packages/core/src/game.ts"], + "a".repeat(40), + true, + 0, + target, + ); + expect(plan.templateMatrix.template).toEqual(TEMPLATE_NAMES); + expect(plan.goldenMatrix.template).toEqual(["starter", "platformer"]); + }); + it.each([{ files: [] }, { files: ["unknown/new.ts"] }, { files: ["scripts/unclassified.ts"] }])( + "keeps unknown or explicit audit inputs exhaustive: %s", + ({ files }) => { + const plan = selectionPlan( + "full", + "unresolved or audit", + files, + "a".repeat(40), + true, + 0, + "develop", + ); + expect(plan.templateMatrix.template).toEqual(TEMPLATE_NAMES); + }, + ); + it("checks scaffold/instruction contracts without runtime template legs for template instruction changes", () => { + const f = fixture(); + try { + const head = f.change("packages/create-threenative/templates/snow/AGENTS.md"); + const plan = classify({ + root: f.root, + base: f.base, + head, + candidateSha: head, + target: "develop", + eventName: "pull_request", + }); + expect(plan.selection).toBe("ci"); + expect(plan.checks.instructions).toBe(true); + expect(plan.jobs["template-nonvisual"].required).toBe(false); + expect(plan.jobs["test-unit"].required).toBe(true); + } finally { + rmSync(f.root, { recursive: true, force: true }); + } + }); +}); + +it("retains both renamed kit endpoints and deleted kit source coverage", () => { + const f = fixture(); + try { + const base = f.change("packages/create-threenative/templates/shooter/src/old.ts"); + mkdirSync(path.join(f.root, "packages/create-threenative/templates/snow/src"), { + recursive: true, + }); + f.git( + "mv", + "packages/create-threenative/templates/shooter/src/old.ts", + "packages/create-threenative/templates/snow/src/new.ts", + ); + f.git("commit", "-qm", "rename"); + const head = f.git("rev-parse", "HEAD"); + const plan = classify({ + root: f.root, + base, + head, + candidateSha: head, + target: "develop", + eventName: "merge_group", + }); + expect(plan.templateMatrix.template).toEqual(["shooter", "snow"]); + f.git("rm", "packages/create-threenative/templates/snow/src/new.ts"); + f.git("commit", "-qm", "delete"); + const deleted = f.git("rev-parse", "HEAD"); + expect( + classify({ + root: f.root, + base: head, + head: deleted, + candidateSha: deleted, + target: "develop", + eventName: "merge_group", + }).templateMatrix.template, + ).toEqual(["snow"]); + } finally { + rmSync(f.root, { recursive: true, force: true }); + } +}); diff --git a/scripts/ci-change-scope.mjs b/scripts/ci-change-scope.mjs index b9fca79667..67e39b7af7 100644 --- a/scripts/ci-change-scope.mjs +++ b/scripts/ci-change-scope.mjs @@ -50,11 +50,44 @@ export function changedTemplates(files) { for (const file of files) { const match = /^packages\/create-threenative\/(?:templates|template-playtests)\/([^/]+)\/.+/u.exec(file); - if (match && TEMPLATE_NAMES.includes(match[1])) names.add(match[1]); + if (match && TEMPLATE_NAMES.includes(match[1]) && !file.endsWith(".md")) names.add(match[1]); } return [...names].sort(); } +// These producer families keep the existing full build/unit/browser/native board. Only the +// redundant kit fanout narrows: each changed kit still proves itself, and starter exercises the +// default tarball/scaffold/physics/React/asset/MCP/boot route. Unknown executables keep all kits. +const REPRESENTATIVE_INPUTS = [ + /^docs\/(?:verification|benchmark)\/.*\.(?:png|jpe?g|webp|json|gz|txt)$/u, + /^packages\/(?:core|physics|ui|assets|playtest|runtime-native|create-threenative|terrain|metahuman|raw-unreal|ueformat|engine-mcp|blender-mcp)\//u, + /^examples\//u, + /^scripts\/__tests__\/[^/]+\.spec\.ts$/u, + /^scripts\/(?:api-surface\.json|run-test-suite\.sh|verify-animation-reversal\.ts|temporal-aa-(?:evidence|quality)\.ts|velocity-(?:capture|cost)-proof\.ts|verify-temporal-(?:aa|motion)\.ts|verify-velocity-history\.ts)$/u, + /^\.github\/(?:workflows\/[^/]+\.yml|actions\/(?:workspace-dist|pnpm|playwright-chromium|scaffold-from-tarballs|android-v8-source)\/)/u, + /^(?:package\.json|pnpm-(?:lock|workspace)\.yaml|tsconfig(?:\.base)?\.json|biome\.json|\.gitignore)$/u, +]; +function representativeTemplates(files, target) { + if ( + files.some((file) => { + const kit = + /^packages\/create-threenative\/(?:templates|template-playtests)\/([^/]+)\//u.exec(file); + return kit && !TEMPLATE_NAMES.includes(kit[1]); + }) + ) + return false; + return ( + target === "develop" && + files.length > 0 && + files.every( + (file) => + file.endsWith(".md") || + REPRESENTATIVE_INPUTS.some((pattern) => pattern.test(file)) || + CI_CONFIG_PATHS.some((pattern) => pattern.test(file)), + ) + ); +} + const SELECTIONS = new Set(["full", "prose", "instructions", "reused", "ci", "warm", "template"]); // CI configuration: the workflows, the scripts that decide and report on the board, their own specs // and the runner images. A diff made only of these cannot change a line of engine, template or @@ -126,13 +159,14 @@ export function selectionPlan( const ciLane = selection === "ci"; const warm = selection === "warm"; const template = selection === "template"; + const representative = full && representativeTemplates(files, target); const checks = { docs: !reused, ci: files.some((file) => CI_CONFIG_PATHS.some((pattern) => pattern.test(file))), instructions: full || selection === "instructions" || - (template && files.some((file) => /(?:^|\/)(?:AGENTS|CLAUDE)\.md$/u.test(file))), + files.some((file) => /(?:^|\/)(?:AGENTS|CLAUDE)\.md$/u.test(file)), // A warm run builds the workspace, because publishing it is the run's whole product. workspace: full || warm || ciLane || template, native: full && native, @@ -225,10 +259,20 @@ export function selectionPlan( nativeTier, unitMatrix: { shard: ciLane || template ? ["1/1"] : ["1/4", "2/4", "3/4", "4/4"] }, templateMatrix: { - template: full ? TEMPLATE_NAMES : template ? changedTemplates(files) : ["starter"], + template: full + ? representative + ? [...new Set(["starter", ...changedTemplates(files)])].sort() + : TEMPLATE_NAMES + : template + ? changedTemplates(files) + : ["starter"], }, goldenMatrix: { - template: template ? changedTemplates(files).slice(0, 1) : ["starter", "platformer"], + template: template + ? changedTemplates(files).slice(0, 1) + : representative + ? ["starter"] + : ["starter", "platformer"], }, checks, jobs, @@ -684,6 +728,11 @@ function parseNameStatus(output) { } function pathFamily(file, selective) { + if ( + /^packages\/create-threenative\/templates\/[^/]+\/(?:AGENTS|CLAUDE)\.md$/u.test(file) && + selective + ) + return "ci"; if (/(?:^|\/)(?:AGENTS|CLAUDE)\.md$/u.test(file)) { // Root and playtest instruction consumers have explicit contracts in the instruction lane. // Shipped template/native instructions may affect scaffolding and platform contracts too. diff --git a/scripts/ci-integration-scope.mjs b/scripts/ci-integration-scope.mjs index 2b129b490b..628bebf7f1 100644 --- a/scripts/ci-integration-scope.mjs +++ b/scripts/ci-integration-scope.mjs @@ -6,6 +6,89 @@ import { pathToFileURL } from "node:url"; const WORKFLOW = ".github/workflows/integration.yml"; const SELECTOR = "scripts/ci-integration-scope.mjs"; +// The package lanes share workspace producers, unlike csg/ik/vegetation's standalone npm +// projects. Producer configuration reaches every package lane; library source reaches its +// runtime consumers. This inventory adds coverage before the unknown-executable fallback. +const PACKAGE_LANES = [ + "exposure", + "cold-boot", + "decals", + "fog", + "fluid-particles", + "fluid-native", + "native-assets", + "tone", + "world-capture", +]; +// The fluid fixtures use FluidParticles3D, Scene/defineGame and core/playtest. These independent +// modules are not imported by that runtime closure; their unit/build proofs stay on the CI board. +// The dependency-boundary spec guards this exception against a new import from the exercised roots. +export const INDEPENDENT_CORE_MODULES = [ + "animation", + "clip-audit", + "rig-preparation", + "skeletal-mesh", + "world-tiles", + "world-cells", + "terrain-jobs", + "terrain-jobs-worker", + "world-topology", + "world-validate", +]; +export const INDEPENDENT_PACKAGES = [ + "terrain", + "metahuman", + "raw-unreal", + "ueformat", + "engine-mcp", + "blender-mcp", +]; +const OWNED_PACKAGES = + "core|physics|playtest|assets|ui|runtime-native|create-threenative|terrain|metahuman|raw-unreal|ueformat|engine-mcp|blender-mcp"; +const PACKAGE_CONTRACT = new RegExp( + `^packages/(?:${OWNED_PACKAGES})/(?:__tests__/.*\\.spec\\.ts|tests/.*\\.test\\.mjs)$`, + "u", +); +const PACKAGE_PRODUCER = new RegExp( + `^packages/(?:${OWNED_PACKAGES})/(?:package\\.json|tsconfig[^/]*\\.json|tsup\\.config\\.ts|scripts/.*|patches/.*)$`, + "u", +); +export const INDEPENDENT_PROOFS = + /^scripts\/(?:run-test-suite\.sh|verify-animation-reversal\.ts|temporal-aa-(?:evidence|quality)\.ts|velocity-(?:capture|cost)-proof\.ts|verify-temporal-(?:aa|motion)\.ts|verify-velocity-history\.ts)$/u; +function dependencyLanes(file, current) { + if (PACKAGE_PRODUCER.test(file)) return PACKAGE_LANES; + if (/^packages\/core\/src\//u.test(file)) { + const module = /^packages\/core\/src\/([^/]+)\.ts$/u.exec(file)?.[1]; + return INDEPENDENT_CORE_MODULES.includes(module) + ? ["exposure", "cold-boot", ...(file.includes("/world-") ? ["world-capture"] : [])] + : ["fluid-particles", "fluid-native"]; + } + if (PACKAGE_CONTRACT.test(file)) { + return [ + ...new Set( + [...current.jobs] + .filter(([, body]) => body.includes(file)) + .map(([id]) => current.lanesByJob.get(id)) + .filter(Boolean), + ), + ]; + } + if ( + INDEPENDENT_PROOFS.test(file) || + /^packages\/(?:terrain\/__tests__\/fixtures\/(?:glb|png)\.mjs|core\/__tests__\/(?:three-attributes\.d\.ts|velocity-render-fixture\.ts))$/u.test( + file, + ) + ) + return []; + // New terrain/editor and import-package sources are built/unit-tested by the ordinary board, + // and none of the retained integration fixtures imports these packages. + if ( + /^packages\/(?:terrain|metahuman|raw-unreal|ueformat|engine-mcp|blender-mcp)\/src\//u.test(file) + ) + return []; + return undefined; +} + function parse(source) { if (source.includes("\t") || source.includes("\r")) throw new Error("ambiguous indentation"); const split = source.split("\njobs:\n"); @@ -146,7 +229,19 @@ export function integrationSelection({ files, before, after }) { if (files.includes(SELECTOR)) return all("shared selector changed"); const lanes = new Set(); for (const file of files) { - let matched = false; + const dependency = dependencyLanes(file, current); + if ( + dependency === PACKAGE_LANES && + [...current.filters.keys()].some( + (lane) => !PACKAGE_LANES.includes(lane) && !["csg", "ik", "vegetation"].includes(lane), + ) + ) + return all("unclassified additive producer consumer"); + let matched = dependency !== undefined; + for (const lane of dependency ?? []) { + if (!current.filters.has(lane)) return all("unresolved producer lane"); + lanes.add(lane); + } for (const [lane, pattern] of current.filters) { if (pattern.test(file)) { lanes.add(lane); From 6bc848dda35a77ee6b95f0283e33c639ec7b352f Mon Sep 17 00:00:00 2001 From: Joao Paulo Furtado Date: Sun, 4 Oct 2026 12:25:54 -0700 Subject: [PATCH 2/2] ci: keep platform prerequisites hosted and allocate whole runner cores --- .github/workflows/native-platforms.yml | 10 +- scripts/__tests__/ci-runners.spec.ts | 145 +++++++++++++++++++++++++ scripts/__tests__/ci-structure.spec.ts | 37 ++++++- scripts/ci-runners.sh | 93 ++++++++++------ tools/ci-runners/README.md | 17 ++- 5 files changed, 261 insertions(+), 41 deletions(-) create mode 100644 scripts/__tests__/ci-runners.spec.ts diff --git a/.github/workflows/native-platforms.yml b/.github/workflows/native-platforms.yml index 3fad8e2b40..b051b4ab06 100644 --- a/.github/workflows/native-platforms.yml +++ b/.github/workflows/native-platforms.yml @@ -124,7 +124,8 @@ jobs: if: >- needs.scope.outputs.selection == 'full' && inputs.ios_only == false - runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} + # Keep this platform prerequisite out of the local queue during CI/Integration bursts. + runs-on: ubuntu-24.04 timeout-minutes: 30 env: TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} @@ -171,7 +172,8 @@ jobs: name: Android V8 source payload needs: scope if: needs.scope.outputs.selection == 'full' && inputs.ios_only != true - runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER) && 'ubuntu-24.04' || vars.TN_RUNNER }} + # Keep this platform prerequisite out of the local queue during CI/Integration bursts. + runs-on: ubuntu-24.04 # 180-minute build ceiling plus 30 minutes for actions/cache/save, per the action's note. timeout-minutes: 210 outputs: @@ -193,8 +195,8 @@ jobs: - uses: android-actions/setup-android@v4 with: packages: platform-tools - # No Actions cache for the NDK: the runner image already carries it (this step takes two - # seconds on a miss), while saving it took 90 seconds and a GB-scale share of the cache budget. + # Install the pinned NDK on the selected runner. Caching it previously took 90 seconds + # and a GB-scale share of the cache budget. - name: Install and select the pinned Android NDK run: | set -euo pipefail diff --git a/scripts/__tests__/ci-runners.spec.ts b/scripts/__tests__/ci-runners.spec.ts new file mode 100644 index 0000000000..4a558a3527 --- /dev/null +++ b/scripts/__tests__/ci-runners.spec.ts @@ -0,0 +1,145 @@ +import { spawnSync } from "node:child_process"; +import { chmodSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import path from "node:path"; +import { describe, expect, it, vi } from "vitest"; +import { makeTempDirSync } from "../../test-support/temp-dir.js"; + +const script = path.resolve(import.meta.dirname, "../ci-runners.sh"); + +function plan( + rows: readonly string[], + slots = "3", + available = rows.length, +): ReturnType { + const fixture = makeTempDirSync("runner-cpu-plan-"); + const bin = path.join(fixture, "bin"); + mkdirSync(bin); + for (const [name, output] of [ + ["lscpu", rows.join("\n")], + ["nproc", String(available)], + ]) { + const command = path.join(bin, name ?? ""); + writeFileSync(command, `#!/bin/sh\ncat <<'DATA'\n${output}\nDATA\n`); + chmodSync(command, 0o755); + } + // Planning must not use Docker, credentials, GitHub, or the service's mutable state. + return spawnSync("bash", [script, "plan", slots], { + encoding: "utf8", + env: { ...process.env, PATH: `${bin}:${process.env.PATH}`, TN_RUNNERS_ENV: "/missing-env" }, + }); +} + +const topology = Array.from({ length: 24 }, (_, cpu) => `${cpu},${cpu % 12},0,Y`); + +describe.skipIf(process.platform !== "linux")("local runner physical core allocation", () => { + it("keeps whole SMT cores separate between heavy slots, light pool and host", () => { + const result = plan(topology); + expect(result.status).toBe(0); + expect(result.stdout).toBe( + "heavy:1:0,12,1,13,2,14\nheavy:2:3,15,4,16,5,17\nheavy:3:6,18,7,19,8,20\nlight:9,21\nhost:10,22,11,23\n", + ); + }); + + it("groups actual socket/core identities instead of assuming CPU plus half is its sibling", () => { + const rows = Array.from({ length: 12 }, (_, core) => [ + `${core * 2},${core % 6},${Math.floor(core / 6)},Y`, + `${core * 2 + 1},${core % 6},${Math.floor(core / 6)},Y`, + ]).flat(); + const result = plan(rows); + expect(result.status).toBe(0); + expect(result.stdout).toContain("heavy:1:0,1,2,3,4,5\n"); + expect(result.stdout).toContain("light:18,19\nhost:20,21,22,23\n"); + }); + + it("passes only whole-core shapes to new slots without changing devices or memory limits", async () => { + const fixture = makeTempDirSync("runner-shape-"); + const bin = path.join(fixture, "bin"); + mkdirSync(bin); + const shapes = path.join(fixture, "shapes"); + const environment = path.join(fixture, "env"); + writeFileSync(environment, "RUNNER_ADMIN_TOKEN=test-only-placeholder\n"); + const commands = { + lscpu: `cat <<'DATA'\n${topology.join("\n")}\nDATA`, + nproc: "echo 24", + docker: "exit 0", + gh: 'case "$*" in *"repo view"*) echo test/repo;; *offline*) ;; *runners*) echo 3;; esac', + setsid: `printf '%s\n' "\${9:-balancer}" >> '${shapes}'`, + }; + for (const [name, body] of Object.entries(commands)) { + const command = path.join(bin, name); + writeFileSync(command, `#!/bin/sh\n${body}\n`); + chmodSync(command, 0o755); + } + const result = spawnSync("bash", [script, "up", "3"], { + encoding: "utf8", + env: { + ...process.env, + PATH: `${bin}:${process.env.PATH}`, + TN_RUNNERS_ENV: environment, + XDG_STATE_HOME: fixture, + }, + }); + expect(result.status).toBe(0); + await vi.waitFor(() => { + expect(readFileSync(shapes, "utf8").trim().split("\n")).toHaveLength(7); + }); + const launched = readFileSync(shapes, "utf8").trim().split("\n"); + const heavy = launched.filter((shape) => shape.includes("--memory 9g")); + const light = launched.filter((shape) => shape.includes("--memory 2g")); + expect(heavy).toHaveLength(3); + for (const cpus of ["0,12,1,13,2,14", "3,15,4,16,5,17", "6,18,7,19,8,20"]) { + expect( + heavy.some((shape) => + shape.includes(`--cpuset-cpus ${cpus} --memory 9g --memory-swap 9g --oom-score-adj 800`), + ), + ).toBe(true); + } + expect(light).toEqual( + Array(3).fill("--cpuset-cpus 9,21 --cpus 1 --memory 2g --memory-swap 2g --oom-score-adj 900"), + ); + expect(launched.join("\n")).not.toMatch(/--privileged|--gpus|--mount|--volume/u); + }); + + it.each([ + Array.from({ length: 24 }, (_, index) => `0,${index % 12},0,Y`), + [...topology.slice(0, 23), "23,unknown,0,Y"], + ])("rejects invalid or duplicated topology before producing an allocation", (...rows) => { + const result = plan(rows); + expect(result.status).not.toBe(0); + expect(result.stderr).toContain("invalid or duplicate CPU topology"); + expect(result.stdout).toBe(""); + }); + + it("rejects six slots before any runner mutation because complete cores cannot fit", () => { + const result = plan(topology, "6"); + expect(result.status).not.toBe(0); + expect(result.stderr).toContain("whole physical cores"); + expect(result.stdout).toBe(""); + }); + + it("cannot advertise unavailable CPUs through an inherited OpenMP override", () => { + const fixture = makeTempDirSync("runner-affinity-"); + const bin = path.join(fixture, "bin"); + mkdirSync(bin); + const lscpu = path.join(bin, "lscpu"); + writeFileSync(lscpu, `#!/bin/sh\ncat <<'DATA'\n${topology.join("\n")}\nDATA\n`); + chmodSync(lscpu, 0o755); + const allowed = readFileSync("/proc/self/status", "utf8").match( + /^Cpus_allowed_list:\s*(\d+)/mu, + )?.[1]; + expect(allowed).toBeDefined(); + const result = spawnSync("taskset", ["-c", allowed ?? "", "bash", script, "plan", "3"], { + encoding: "utf8", + env: { ...process.env, PATH: `${bin}:${process.env.PATH}`, OMP_NUM_THREADS: "24" }, + }); + expect(result.status).not.toBe(0); + expect(result.stderr).toContain("affinity"); + expect(result.stdout).toBe(""); + }); + + it("fails closed when inherited affinity cannot cover the discovered online CPUs", () => { + const result = plan(topology, "3", 23); + expect(result.status).not.toBe(0); + expect(result.stderr).toContain("affinity"); + }); +}); diff --git a/scripts/__tests__/ci-structure.spec.ts b/scripts/__tests__/ci-structure.spec.ts index 95a032cabc..8fe0d0c579 100644 --- a/scripts/__tests__/ci-structure.spec.ts +++ b/scripts/__tests__/ci-structure.spec.ts @@ -1078,9 +1078,44 @@ describe("CI pipeline structure", () => { ".github/workflows/native-platforms.yml", // publish-android-v8 writes the release; android-emulator-parity is CPU-bound SwiftShader that // overran its 45-minute budget on a pinned 4-thread slot (run 37082733117). - new Set(["android-emulator-parity", "publish-android-v8"]), + new Set([ + "android-emulator-parity", + "publish-android-v8", + "web-reference", + "android-v8-source", + ]), ], ]); + it.each([ + ["web-reference", "native-web-reference-${{ needs.scope.outputs.candidate_sha }}"], + ["android-v8-source", "android-v8-${{ needs.scope.outputs.candidate_sha }}"], + ])( + "keeps native platform prerequisite %s hosted with exact candidate artifacts", + async (job, artifact) => { + const native = await readFile( + path.join(repo, ".github/workflows/native-platforms.yml"), + "utf8", + ); + const section = requiredJob(native, job); + // An idle-pool variable is not a reservation. These producers must remain hosted even + // when both CI and Integration select the same advertised local capacity in a burst. + expect(section.match(/^ {4}runs-on: (.+)$/mu)?.[1]).toBe("ubuntu-24.04"); + expect(declaredNeeds(section)).toEqual(["scope"]); + expect(section).toContain("TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }}"); + expect(section).toContain("ref: ${{ needs.scope.outputs.candidate_sha }}"); + expect(section).toContain(`name: ${artifact}`); + expect(section).toContain("if-no-files-found: error"); + const android = requiredJob(native, "android-emulator-parity"); + expect(declaredNeeds(android)).toEqual(["scope", "web-reference", "android-v8-source"]); + expect(android).toContain(`name: ${artifact}`); + if (job === "android-v8-source") { + expect(section).toContain("uses: ./.github/actions/android-v8-source"); + expect(section).toContain("recipe-hash: ${{ steps.v8.outputs.recipe-hash }}"); + expect(section).toContain("published: ${{ steps.v8.outputs.published }}"); + } + }, + ); + // PRD-480's light lane: the jobs whose whole work is a script or a summary — no workspace build, // no `pnpm install` and no test suite. `scope` classifies the diff, `build` and `golden-path` // assert an upstream verdict, `ci-required` is the merge verdict and `run-summary` writes the diff --git a/scripts/ci-runners.sh b/scripts/ci-runners.sh index b55d58f55b..d8dc9e86ab 100755 --- a/scripts/ci-runners.sh +++ b/scripts/ci-runners.sh @@ -30,16 +30,16 @@ VARIABLE=TN_RUNNER LIGHT_VARIABLE=TN_RUNNER_LIGHT ENV_FILE="${TN_RUNNERS_ENV:-${XDG_CONFIG_HOME:-$HOME/.config}/threenative/runners.env}" STATE_DIR="${XDG_STATE_HOME:-$HOME/.local/state}/threenative/ci-runners" -DEFAULT_SLOTS=6 -THREADS_PER_SLOT=3 -# 6 x 9 GB heavy + 3 x 2 GB light stays under 62 GB even if every slot hits its cap at once. +DEFAULT_SLOTS=3 +CORES_PER_SLOT=3 +LIGHT_CORES=1 +# 3 x 9 GB heavy + 3 x 2 GB light caps the pool at 33 GB. SLOT_MEMORY=9g # Cores never given to a slot, so the owner's desktop stays responsive under a full board. HOST_CORES=2 ONLINE_TIMEOUT_SECONDS=300 -# The light lane is a quota, not a reservation: one core of whatever a heavy slot is not using, -# 2 GB, and the highest OOM score of any container, so a small join is what gets killed under -# memory pressure rather than a 20-minute build. +# The three short join slots share a physical core, separate from heavy and host cores. +# Each retains a one-CPU quota, 2 GB, and the highest OOM score under memory pressure. LIGHT_SHAPE="--cpus 1 --memory 2g --memory-swap 2g --oom-score-adj 900" # Several light slots: an ephemeral runner needs 30-60 s to re-register after each job, and every # board brings ~7 joins, so one light slot serialised every pull request's scope and verdict. @@ -133,10 +133,51 @@ online_count() { " 2>/dev/null || echo 0 } +# Read-only plan: group the CPUs by actual socket/core identity, keeping all SMT siblings +# together. Refuse inherited restricted affinity rather than allocate CPUs we cannot use. +plan() { + local slots="${1:-${TN_RUNNERS:-$DEFAULT_SLOTS}}" topology cpu_count cores=() slot first cpus + case "$slots" in ''|*[!0-9]*) fail 2 "N must be a positive integer" ;; esac + [ "$slots" -ge 1 ] || fail 2 "N must be at least 1" + topology="$(lscpu -p=CPU,CORE,SOCKET,ONLINE)" || fail 2 "cannot read CPU topology" + cpu_count="$(awk -F, ' + $1 !~ /^#/ { + if (NF != 4 || $1 !~ /^[0-9]+$/ || $2 !~ /^[0-9]+$/ || + $3 !~ /^[0-9]+$/ || $4 !~ /^(Y|N)$/ || seen[$1]++) bad=1 + if ($4 == "Y") n++ + } + END {if (bad || !n) exit 2; print n} + ' <<< "$topology")" || fail 2 "invalid or duplicate CPU topology" + [ "$cpu_count" -eq "$(env -u OMP_NUM_THREADS -u OMP_THREAD_LIMIT nproc)" ] || fail 2 "restricted affinity: online topology does not match available CPUs" + mapfile -t cores < <(awk -F, ' + $1 !~ /^#/ && $4 == "Y" { + key=$3 ":" $2 + if (!(key in cpus)) order[++n]=key + cpus[key]=cpus[key] (cpus[key] == "" ? "" : ",") $1 + } + END {for (i=1;i<=n;i++) print cpus[order[i]]} + ' <<< "$topology") + [ $(( slots * CORES_PER_SLOT + LIGHT_CORES + HOST_CORES )) -le "${#cores[@]}" ] \ + || fail 2 "$slots slots need $CORES_PER_SLOT whole physical cores each plus light/host reserves; only ${#cores[@]} exist" + for slot in $(seq 1 "$slots"); do + first=$(( (slot - 1) * CORES_PER_SLOT )) + cpus="$(IFS=,; echo "${cores[*]:$first:$CORES_PER_SLOT}")" + printf 'heavy:%s:%s\n' "$slot" "$cpus" + done + first=$(( ${#cores[@]} - HOST_CORES - LIGHT_CORES )) + cpus="$(IFS=,; echo "${cores[*]:$first:$LIGHT_CORES}")" + printf 'light:%s\n' "$cpus" + first=$(( ${#cores[@]} - HOST_CORES )) + cpus="$(IFS=,; echo "${cores[*]:$first:$HOST_CORES}")" + printf 'host:%s\n' "$cpus" +} + up() { local slots="${1:-${TN_RUNNERS:-$DEFAULT_SLOTS}}" case "$slots" in ''|*[!0-9]*) fail 2 "N must be a positive integer, got '$slots'" ;; esac [ "$slots" -ge 1 ] || fail 2 "N must be at least 1, got '$slots'" + local allocation + allocation="$(plan "$slots")" || return $? need docker need gh load_config @@ -149,31 +190,20 @@ up() { # A leftover stop file from a previous teardown would make every loop exit before it starts. rm -f "$STATE_DIR/stop" - # Pinned CPUs, not a `--cpus N` quota: under a quota `nproc` and os.availableParallelism() still - # report every host CPU, so vitest sized its pool for 24 cores inside a 4-core budget and three - # unit tests timed out (PR #404). Threads are handed out core by core (both hyperthreads of a core - # before the next), THREADS_PER_SLOT to a slot, never touching the HOST_CORES left for the desktop. - # Three threads a slot (nproc 3) rather than four buys a sixth slot from the same 20 threads; most - # CI time is installs, downloads and browser waits, so parallel jobs beat fatter ones. - # ponytail: assumes Linux's usual sibling numbering (CPU k and k + nproc/2 share a core); read - # /sys/devices/system/cpu/cpu*/topology/thread_siblings_list if a host numbers them otherwise. - local half=$(( $(nproc) / 2 )) core threads=() - for core in $(seq 0 $(( half - HOST_CORES - 1 ))); do threads+=("$core" "$(( core + half ))"); done - [ $(( slots * THREADS_PER_SLOT )) -le "${#threads[@]}" ] \ - || fail 2 "$slots slots of $THREADS_PER_SLOT threads need $(( slots * THREADS_PER_SLOT )); this host has ${#threads[@]} after $HOST_CORES cores for the desktop" - local slot cpus first - for slot in $(seq 1 "$slots"); do - first=$(( (slot - 1) * THREADS_PER_SLOT )) - cpus="$(IFS=,; echo "${threads[*]:$first:$THREADS_PER_SLOT}")" - start_slot "slot-$slot" "$STATE_DIR" "$ENV_FILE" "$repo" \ - "--cpuset-cpus $cpus --memory ${SLOT_MEMORY} --memory-swap ${SLOT_MEMORY} --oom-score-adj 800 $KVM_SHAPE" \ - "$LABEL" - done - # One light slot, unpinned and quota-limited, labelled so heavy jobs cannot select it: a 10-second - # join behind five 20-minute builds is the wait this lane exists to remove. It never reserves a - # core, so the check above still describes exactly the cores this pool owns. + # Heavy slots have whole physical cores; light joins never share their SMT siblings. + local kind slot cpus light_cpus + while IFS=: read -r kind slot cpus; do + if [ "$kind" = heavy ]; then + start_slot "slot-$slot" "$STATE_DIR" "$ENV_FILE" "$repo" \ + "--cpuset-cpus $cpus --memory ${SLOT_MEMORY} --memory-swap ${SLOT_MEMORY} --oom-score-adj 800 $KVM_SHAPE" \ + "$LABEL" + elif [ "$kind" = light ]; then + light_cpus="$slot" + fi + done <<< "$allocation" for slot in $(seq 1 "$LIGHT_SLOTS"); do - start_slot "light-$slot" "$STATE_DIR" "$ENV_FILE" "$repo" "$LIGHT_SHAPE" "$LIGHT_LABEL" + start_slot "light-$slot" "$STATE_DIR" "$ENV_FILE" "$repo" \ + "--cpuset-cpus $light_cpus $LIGHT_SHAPE" "$LIGHT_LABEL" done printf '%s heavy + %s light runner container(s) starting; logs in %s\n' "$slots" "$LIGHT_SLOTS" "$STATE_DIR" @@ -357,11 +387,12 @@ uninstall() { } case "${1:-}" in + plan) shift; plan "${1:-}" ;; up) shift; up "${1:-}" ;; down) down ;; status) status ;; balance) balance ;; install) install ;; uninstall) uninstall ;; - *) fail 2 "usage: ci-runners.sh up [N] | down | status | balance | install | uninstall" ;; + *) fail 2 "usage: ci-runners.sh plan [N] | up [N] | down | status | balance | install | uninstall" ;; esac diff --git a/tools/ci-runners/README.md b/tools/ci-runners/README.md index 3ce425936a..c5e3aab4e9 100644 --- a/tools/ci-runners/README.md +++ b/tools/ci-runners/README.md @@ -29,7 +29,7 @@ pool that is down never strands a job. ## Set it up (once, on the runner machine) Needs: Linux with systemd, Docker usable without `sudo`, `gh` logged in with admin rights on the repo, -and enough cores for the slots (3 threads each; 2 whole cores always stay free for the desktop). +and enough cores for the slots (3 whole physical cores each; 2 whole cores stay free for the desktop). 1. Create a fine-grained token for this repository only, with **Administration: Read and write**. It mints runner registration tokens. Add **Variables: Read and write** too if the pool must also start @@ -57,6 +57,11 @@ and enough cores for the slots (3 threads each; 2 whole cores always stay free f ## Day to day +`bash scripts/ci-runners.sh plan [N]` prints the CPU allocation without credentials or mutations. +It groups actual socket/core identities and refuses restricted inherited affinity or insufficient +whole cores. Activation uses the existing service restart only after the pool is drained and the +reviewed script is on `develop`; a restart cancels active jobs. + | Want | Run | |---|---| | What is running, what is routed | `pnpm ci:runners status` (balancer log: `~/.local/state/threenative/ci-runners/balance.log`) | @@ -74,9 +79,11 @@ restart or reboot. - An ephemeral container per job (`docker run --rm`): fresh filesystem, fresh hostname, no state shared between jobs. The image is Ubuntu 24.04 plus what hosted `ubuntu-24.04` jobs rely on: Node 20, `gh`, Rust, JDK 17, Android SDK and platform-tools, build tools, Xvfb and Playwright's dependencies. -- 6 heavy slots of 3 pinned threads each (`nproc` reads 3) with 9 GB of memory and no extra swap. More, - thinner slots beat fewer fat ones: most CI time is installs, downloads and browser waits. Under memory pressure the kernel kills a CI job before anything else. -- 3 light slots: 1 CPU and 2 GB each, for joins only. One slot serialised every pull request's scope and +- 3 heavy slots of 3 whole physical cores each (6 threads on a two-way SMT host), with 9 GB of memory + and no extra swap. Sibling threads stay in one slot. This halves job concurrency while keeping the + heavy pool's total CPU allocation; compare board throughput after activation. +- 3 light slots: 1 CPU quota and 2 GB each, sharing a separate physical core, for joins only. + These three slots contend for two logical CPUs on a two-way SMT host. One slot serialised every pull request's scope and verdict, because an ephemeral runner takes 30-60 s to re-register after each job. - The admin token mints one registration token per container and is unset before the job starts, so no job can read it. @@ -92,5 +99,5 @@ restart or reboot. flaky on both pools (see `docs/PRDs/CI/EXECUTION-ORDER.md`). Rerun the failed jobs once. - **The runner list shows offline `tn-*` runners.** Runners stopped while idle stay registered. `up` and `down` delete them. -- **Timing.** The pool adds about 6 slots to hosted's ~20; with several boards in flight both pools +- **Timing.** The pool adds 3 heavy slots plus 3 light slots to hosted's ~20; with several boards in flight both pools saturate, and wall time follows total demand.