diff --git a/AGENTS.md b/AGENTS.md index ebfa3733..68baf293 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,7 +30,7 @@ On a machine that already has clones of Makepad, OctoScript or Octoscript-Makepa ## Local CI -`tools/ci-local.sh --only all` (or `desktop`, `phone`, `apps`, `rom`) runs the workflows' `run:` steps verbatim on this machine, prints PASS, FAIL or SKIPPED per step, and writes `target/ci-local/last.json`. At most two runs share a machine; the others wait. When the GitHub macOS queue is saturated, you may merge on a local pass. First rebase the branch on `origin/main`, then run it on the exact head, then run `tools/ci-local-merge.sh `. That script checks the evidence, comments the summary on the PR, and merges with `--admin`. A SKIPPED step did not pass. GitHub CI runs on every push to `main`, including these merges. Main runs share one group per workflow (`-main`) that cancels older ones, so only the newest main commit's run completes. A failure there is fixed before anything else is merged. See [docs/local-ci.md](docs/local-ci.md). +`tools/ci-local.sh --only all` (or `desktop`, `phone`, `apps`, `rom`) runs the workflows' `run:` steps verbatim on this machine, prints PASS, FAIL or SKIPPED per step, and writes `target/ci-local/last.json`. At most two runs share a machine; the others wait. When the GitHub macOS queue is saturated, you may merge on a local pass. First rebase the branch on `origin/main`, then run it on the exact head, then run `tools/ci-local-merge.sh `. That script checks the evidence, comments the summary on the PR, and merges with `--admin`. A SKIPPED step did not pass. With `--linux-host`, the jobs GitHub runs on ubuntu, plus the Linux-only process sandbox tests (Landlock, seccomp), run on the Linux build host configured in `~/.config/octosense/build.env`, as the ssh user in `~/octosense-ci/` and never with sudo; their steps join the same table and `last.json` marked `linux`, bound to the exact commit. Never write the host's address, its key path or any credential into the repository, a log or a PR. GitHub CI runs on every push to `main`, including these merges. Main runs share one group per workflow (`-main`) that cancels older ones, so only the newest main commit's run completes. A failure there is fixed before anything else is merged. See [docs/local-ci.md](docs/local-ci.md). ## Rules diff --git a/docs/local-ci.md b/docs/local-ci.md index 06a97a6d..2348a5dc 100644 --- a/docs/local-ci.md +++ b/docs/local-ci.md @@ -10,15 +10,31 @@ tools/ci-local.sh --only all # or desktop | phone | apps | rom, co tools/ci-local.sh --only phone --keep-going --jobs 8 tools/ci-local.sh --list # the plan; runs nothing tools/ci-local.sh --check-drift # the local mapping still fits the workflows +tools/ci-local.sh --only apps --linux-host # the ubuntu jobs on the Linux build host (below) ``` - **Same commands as GitHub.** The runner reads the workflow files and runs each job's `run:` steps verbatim: same working directory, same `env:`, same `bash --noprofile --norc -eo pipefail`. Nothing is copied, so nothing drifts. The few things it maps (the actions: checkout, setup-python, rust-toolchain, rust-cache, the kernel cache, setup-node; expressions like `${{ runner.temp }}`) are checked by `--check-drift`. Every run checks them too, and so does `tools/test_ci_local.py` (part of the desktop job). A workflow that starts using a new action, job-level `if:`, a matrix or an unknown expression fails that check until `tools/ci_local.py` is taught about it. - **Output.** Each step is printed as PASS, FAIL, SKIPPED or NOT RUN (after a failure, without `--keep-going`), followed by a summary table with times. The full output goes to `target/ci-local/.log`. `target/ci-local/last.json` records the commit, whether the tree was dirty, the workflows run, and every step's result. The exit status is non-zero on any FAIL. -- **Skips are never passes.** A step that cannot run here is SKIPPED and the summary says why. Today that is the rom product tests, unless a JDK works (`javac -version`, from `PATH` or `JAVA_HOME`; macOS's `/usr/bin/javac` is only a stub), the rom `Check generated Agent Binder client` step, unless the Android SDK has `build-tools;35.0.0` and `platforms;android-35` (found through `ANDROID_HOME`), and the web installer job, unless `node`/`npm`/`npx` are on `PATH`. Jobs GitHub runs on `ubuntu-latest` (apps `services` and `kernel-security`, rom) run on the Mac. Their `#[cfg(target_os = "linux")]` code is not exercised, and the summary notes this. There is no Linux-host mode yet. +- **Skips are never passes.** A step that cannot run here is SKIPPED and the summary says why. Today that is the rom product tests, unless a JDK works (`javac -version`, from `PATH` or `JAVA_HOME`; macOS's `/usr/bin/javac` is only a stub), the rom `Check generated Agent Binder client` step, unless the Android SDK has `build-tools;35.0.0` and `platforms;android-35` (found through `ANDROID_HOME`), and the web installer job, unless `node`/`npm`/`npx` are on `PATH`. Without `--linux-host`, jobs GitHub runs on `ubuntu-latest` (apps `services` and `kernel-security`, rom) run on the Mac. Their `#[cfg(target_os = "linux")]` code is not exercised, and the summary notes this. - **The octos kernel.** The real-kernel steps (phone's app-peers relay and two-lane scenario, apps' `kernel-security`) need `octos` at the revision `Cargo.lock` pins. The workflow's own `Build octos (unless cached)` step builds it once (`tools/kernel-artifact.py --host`). The binary is then kept in a per-user cache that every clone shares, keyed by OS, architecture and octos revision (`~/Library/Caches/octosense-ci-local/octos-kernel/`, `$XDG_CACHE_HOME` or `OCTOSENSE_CI_LOCAL_CACHE`). The copy is written atomically, under a per-revision lock, so two clones never build the same kernel at once. - **Sharing the machine.** At most two runs execute at once (`--slots N` or `OCTOSENSE_CI_LOCAL_SLOTS`), coordinated by mkdir locks in `${TMPDIR}/octosense-ci-local/` (or `OCTOSENSE_CI_LOCAL_LOCKS`). A run waits for a free slot and says so. With `--no-wait` it exits with status 75 instead. A slot whose owner process has died is taken over. `--jobs N` sets `CARGO_BUILD_JOBS` (default: half the CPUs). - **Tools.** `cargo` (with clippy and rustfmt) comes from `PATH`, or `~/.cargo/bin` if it is not on `PATH`. `python3` also comes from `PATH`: GitHub uses 3.12, and a different version is noted in the summary. `cmake` is not needed by any of these steps. If a build ever asks for it, put one on `PATH`, e.g. `python3 -m venv && /bin/pip install cmake`. +## Run the Linux jobs on the Linux build host + +`--linux-host` runs every job whose `runs-on` is ubuntu on the Linux build host configured in `~/.config/octosense/build.env` (`OCTOSENSE_BUILD_HOST`, the ssh target, and `OCTOSENSE_BUILD_KEY`, its key; the environment overrides the file, and `--linux-host ` names another target). The macOS jobs still run on the Mac, at the same time. It also adds the Linux-only checks no workflow runs: today the process sandbox's Landlock and seccomp tests (`cargo test --locked -p octosense-shell --lib sandbox::`, the `#[cfg(target_os = "linux")]` tests in `crates/shell/src/sandbox/`), as the job `linux-host / sandbox`. They come with any run that includes a workflow a change to the shell triggers (desktop, phone, apps). On a kernel without Landlock the tests skip themselves; that step then fails instead of passing. + +```sh +tools/ci-local.sh --only apps --linux-host # services, kernel-security and the sandbox tests on Linux; apps on the Mac +tools/ci-local.sh --only all --linux-host --list # the plan, with where each job runs +``` + +- **What runs there.** The exact commit under test (`HEAD`; uncommitted changes stay on the Mac, and the run is marked dirty as usual) goes to the host as a git bundle, incrementally after the first run, into `~/octosense-ci/repo.git`. It is checked out as `~/octosense-ci/runs/-`, and the host runs `tools/setup.py` itself from the same lock files (its sources are worktrees of hub clones in `~/octosense-ci/cache/hub/`). The newest five run directories are kept. The host then runs this checkout's own `tools/ci_local.py`, so the same `run:` steps run there, extracted from the same workflow files. +- **Results.** Each remote step appears in the same PASS/FAIL/SKIPPED table, marked `(linux)`, and in `last.json` with `"host": "linux"` and `"sha"`, the commit the host checked out. The summary's `linux_host` records that host's OS, architecture, CPUs, tool versions and timings, never its address. A host that cannot be reached, a missing result or a result for another commit is a FAIL, never a pass. The output lines prefixed `linux|` are the host's; each job's full log stays in its run directory under `target/ci-remote/`. +- **Sharing the host.** At most four runs use the host at once (`--linux-slots N` or `OCTOSENSE_CI_LINUX_SLOTS`), through mkdir locks in `~/octosense-ci/locks/` on the host, separate from the Mac's two slots. A run's Linux jobs run in parallel, each with its own cargo target in `~/octosense-ci/cache/target/slot-/` (a warm run reuses the slot's built dependencies; the workspace's own crates and `.sources/` rebuild, since cargo keys them by the run directory's path) and `CARGO_BUILD_JOBS` of the host's CPUs divided by the slots (`--linux-jobs N`). The octos kernel cache is `~/octosense-ci/cache/octos-kernel/`. +- **Security.** The host runs code from pull-request branches of a public repository, as any CI does. Everything runs as the ssh user in `~/octosense-ci/`, never with sudo. The remote steps get an allow-listed environment (`HOME`, `USER`, `LOGNAME`, `LANG`, a fixed `PATH`, and the runner's own variables): no tokens and nothing from the Mac's environment. Never copy credentials there. The host's address and the key path are kept out of the log and `last.json`. +- **One-time setup on the host** (as the ssh user; no sudo): rustup with the stable toolchain and clippy and rustfmt, `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --no-modify-path --profile minimal -c clippy -c rustfmt`, and a Python venv at `~/octosense-ci/venv` (`python3 -m venv --without-pip ~/octosense-ci/venv` when the distribution's `python3-venv` is not installed; the steps need no Python packages). The hub clones are made on the first run. Building the shell (the sandbox tests) needs Makepad's Linux build dependencies, which the host administrator installs (`pkg-config`, `clang`, `libx11-dev`, `libxcursor-dev`, `libxkbcommon-dev`, `libwayland-dev`, `libvulkan-dev`, `libasound2-dev`, `libpulse-dev`, `libgl-dev`). Steps that need more than that (a JDK for rom's product tests, the Android SDK, node for the web installer) are SKIPPED there with the reason, as on the Mac. + ## Merge on a local pass ```sh @@ -33,6 +49,7 @@ tools/ci-local-merge.sh # --dry-run to preview - passed on the PR's exact head commit, with a clean tree; - comes from a head that contains the current `origin/main`; - covers every workflow GitHub would run for the PR's files (their `pull_request` `paths`), with no FAIL, no NOT RUN and no unexpected SKIP in them. A workflow GitHub would not run for the PR does not block it, so `--only desktop,phone` is enough evidence for a PR that only triggers those two. +- for steps run with `--linux-host`: ran on the Linux host on that same head commit. A remote result bound to any other commit is stale and refused. The Linux-only checks (`linux-host / sandbox`) count like the workflows they cover: they block a PR that triggers desktop, phone or apps. It also refuses while the latest completed GitHub run of a workflow on `main` has failed. Pass `--fixes-main` only for the PR that fixes it. Once every check passes, it posts the summary table as a PR comment ("Local CI passed on `` …") and runs `gh pr merge --admin --merge --match-head-commit `, with the subject `Merge pull request # from /`. diff --git a/tools/ci-local.sh b/tools/ci-local.sh index 9b07902b..60ede9fc 100755 --- a/tools/ci-local.sh +++ b/tools/ci-local.sh @@ -2,6 +2,7 @@ # Run the GitHub workflows' checks locally (tools/ci_local.py; docs/local-ci.md): # # tools/ci-local.sh [--only desktop|phone|apps|rom|all] [--jobs N] [--keep-going] [--no-wait] +# tools/ci-local.sh --linux-host ... # ubuntu jobs on the Linux build host (~/.config/octosense/build.env) # # Prints PASS/FAIL/SKIPPED per step, writes target/ci-local/.log # and target/ci-local/last.json, and exits non-zero on any failure. diff --git a/tools/ci_local.py b/tools/ci_local.py index c38802cd..72c29567 100644 --- a/tools/ci_local.py +++ b/tools/ci_local.py @@ -17,9 +17,12 @@ A step that cannot run here is SKIPPED with the reason, never passed. Jobs GitHub runs on ubuntu-latest run on this Mac; `#[cfg(target_os = "linux")]` -code in them is not exercised here (the summary says so). +code in them is not exercised here (the summary says so), unless +--linux-host sends them to a Linux build host over ssh (see "The Linux host" +below and docs/local-ci.md). tools/ci-local.sh [--only desktop|phone|apps|rom|all[,...]] [--jobs N] [--keep-going] [--no-wait] + tools/ci-local.sh --linux-host [SSH] # ubuntu jobs on the Linux host tools/ci-local.sh --list # the plan, nothing run tools/ci-local.sh --check-drift # the local mapping still fits the workflows tools/ci-local-merge.sh # merge on a local pass (see docs/local-ci.md) @@ -28,6 +31,18 @@ run at once, through mkdir locks in ${TMPDIR}/octosense-ci-local/; a run waits for a slot (or, --no-wait, exits 75). Results: target/ci-local/.log and target/ci-local/last.json. + +The Linux host (--linux-host): every job whose runs-on is ubuntu, and the +Linux-only checks in LINUX_HOST_JOBS (the process sandbox's Landlock and +seccomp tests), run on the host as the ssh user, under ~/octosense-ci/ only. +The exact commit under test goes there as a git bundle and is checked out in +~/octosense-ci/runs/- (the newest five are kept); the host runs +tools/setup.py itself from the same locks. The same `run:` steps run there, +from this file, with an allow-listed environment (no tokens). At most +OCTOSENSE_CI_LINUX_SLOTS (default 4) runs share the host; each of a run's +jobs runs in parallel, with its own cargo target in ~/octosense-ci/cache/. +Their results join the table and last.json marked `"host": "linux"`, bound +to the commit the host checked out. """ import argparse import datetime @@ -37,11 +52,14 @@ from pathlib import Path import platform import re +import shlex import shutil +import signal import socket import subprocess import sys import tempfile +import threading import time ROOT = Path(__file__).resolve().parents[1] @@ -279,6 +297,65 @@ def load_workflow(name): }, } +# Checks that only mean something on Linux and that no workflow runs on +# Linux: --linux-host adds them, as jobs of the pseudo-workflow "linux-host", +# whenever the run includes a workflow that a change to `source` triggers. +# They have the shape of a workflow job and run like one. +LINUX_HOST = "linux-host" +LINUX_HOST_JOBS = { + # The process sandbox's #[cfg(target_os = "linux")] tests (Landlock and + # seccomp: crates/shell/src/sandbox/{linux,tests}.rs). The workflows run + # the shell's tests on macOS only, so without a Linux host these never run. + # The tests skip themselves on a kernel without Landlock; that is a + # failure here, not a pass. + "sandbox": { + "runs-on": "ubuntu-latest", + "source": "crates/shell/src/sandbox/linux.rs", + "steps": [ + {"uses": "actions/checkout@v6"}, + {"uses": "dtolnay/rust-toolchain@stable"}, + {"run": "python3 tools/setup.py\n"}, + {"name": "Process sandbox on Linux (Landlock, seccomp)", + "run": "cargo test --locked -p octosense-shell --lib sandbox:: -- --nocapture 2>&1" + " | tee \"$RUNNER_TEMP/sandbox-tests.txt\"\n" + "if grep -q 'no process sandbox on this machine' \"$RUNNER_TEMP/sandbox-tests.txt\"; then\n" + " echo '::error::the sandbox tests skipped themselves: this kernel has no Landlock'; exit 1\n" + "fi\n"}, + ], + }, +} + + +def linux_host_job_covers(job): + """The workflows whose runs include a LINUX_HOST_JOBS job: those a change + to its source triggers (ci-local-merge.sh requires it for the same PRs).""" + return triggered_workflows([job["source"]]) + + +def plan_jobs(workflows, linux_host=False): + """[(workflow, job id, job, where)], where is "local" or "linux": with a + Linux host, every job whose runs-on is ubuntu runs there, and the + LINUX_HOST_JOBS that the chosen workflows cover are added.""" + plan = [] + for workflow in workflows: + for job_id, job in jobs_of(workflow): + where = "linux" if linux_host and "ubuntu" in str(job.get("runs-on", "")) else "local" + plan.append((workflow, job_id, job, where)) + if linux_host: + for job_id, job in LINUX_HOST_JOBS.items(): + if set(linux_host_job_covers(job)) & set(workflows): + plan.append((LINUX_HOST, job_id, job, "linux")) + return plan + + +def job_definition(key): + """The job `:` names (a workflow's, or a LINUX_HOST_JOBS one).""" + workflow, job_id = key.split(":", 1) + if workflow == LINUX_HOST: + return LINUX_HOST_JOBS[job_id] + return dict(jobs_of(workflow))[job_id] + + # GitHub expressions the runner can evaluate. Anything else fails --check-drift. EXPRESSION = re.compile(r"\$\{\{\s*(.*?)\s*\}\}") KNOWN_EXPRESSIONS = [re.compile(p) for p in ( @@ -360,6 +437,15 @@ def check_drift(workflows=None): for key in STEP_REQUIREMENTS: if key.split(":")[0] in names and key not in seen_steps: problems.append(f"{key}: listed in STEP_REQUIREMENTS but no step has that name") + if workflows is None: + for job_id, job in LINUX_HOST_JOBS.items(): + if not (ROOT / job["source"]).is_file(): + problems.append(f"{LINUX_HOST}:{job_id}: its source {job['source']} is gone: update LINUX_HOST_JOBS") + elif not linux_host_job_covers(job): + problems.append(f"{LINUX_HOST}:{job_id}: no workflow is triggered by {job['source']}") + for step in job["steps"]: + if "uses" in step and action_name(step["uses"]) not in ACTIONS: + problems.append(f"{LINUX_HOST}:{job_id}: uses {step['uses']}, which has no local mapping") return problems @@ -539,24 +625,43 @@ def link_kernel(runner_temp, cached): class Run: - def __init__(self, args): + def __init__(self, args, out_dir=None): self.args = args - stamp = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") - self.out_dir = ROOT / "target/ci-local" + self.stamp = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") + self.out_dir = Path(out_dir) if out_dir else ROOT / "target/ci-local" self.out_dir.mkdir(parents=True, exist_ok=True) - self.log_path = self.out_dir / f"{stamp}.log" + self.log_path = self.out_dir / f"{self.stamp}.log" self.log_file = open(self.log_path, "w", buffering=1) self.runner_temp = self.out_dir / "runner-temp" + self.runner_temp.mkdir(exist_ok=True) self.steps = [] self.actions = [] self.notes = [] self.failed = False + self.lock = threading.Lock() + # The Linux host's address and key path never reach the log or the results. + self.linux = getattr(args, "linux_settings", None) + self.scrub = [] + if self.linux: + target, key = self.linux + self.scrub = [v for v in (target, target.split("@")[-1], key) if v] + self.linux_info = None self.env = self.base_env() + def close(self): + self.log_file.close() + + def clean(self, text): + for secret in self.scrub: + text = text.replace(secret, "") + return text + def log(self, message, echo=True): - self.log_file.write(message + "\n") - if echo: - print(message, flush=True) + message = self.clean(message) + with self.lock: + self.log_file.write(message + "\n") + if echo: + print(message, flush=True) def base_env(self): env = dict(os.environ) @@ -589,18 +694,21 @@ def replace(match): raise RuntimeError(f"cannot evaluate ${{{{ {expression} }}}}") return EXPRESSION.sub(replace, str(text)) - def record(self, workflow, job, name, status, seconds=0.0, reason="", expected=True, command=None): - reason = reason.replace(str(Path.home()), "~") # results get posted: no home paths - self.steps.append({"workflow": workflow, "job": job, "name": name, "status": status, - "seconds": round(seconds, 1), "reason": reason, - "expected_skip": expected if status == SKIPPED else None, - "command": command}) - if status == FAIL: - self.failed = True + def record(self, workflow, job, name, status, seconds=0.0, reason="", expected=True, command=None, echo=True): + reason = self.clean(reason.replace(str(Path.home()), "~")) # results get posted: no home paths + step = {"workflow": workflow, "job": job, "name": name, "status": status, + "seconds": round(seconds, 1), "reason": reason, + "expected_skip": expected if status == SKIPPED else None, + "command": command, "host": "local"} + with self.lock: + self.steps.append(step) + if status == FAIL: + self.failed = True line = f"[{status}] {workflow} {job}: {name}" + (f" ({seconds:.0f}s)" if seconds else "") if reason: line += f" -- {reason}" - self.log(line) + self.log(line, echo=echo) + return step def android_sdk(self): for candidate in (self.env.get("ANDROID_HOME"), self.env.get("ANDROID_SDK_ROOT"), @@ -676,10 +784,17 @@ def run_step(self, workflow, job_id, job, index, step, outputs): output_file.write_text("") env["GITHUB_OUTPUT"] = str(output_file) building_kernel = "kernel-artifact.py --host" in script - kernel_lock = None + step_lock = None if building_kernel: - kernel_lock = self.prepare_kernel_build() - self.log(f"--> {workflow} {job_id}: {label} (cwd {cwd.relative_to(ROOT) if cwd != ROOT else '.'})") + step_lock = self.prepare_kernel_build() + elif "tools/setup.py" in script and getattr(self.args, "worker_job", None): + # A run's jobs share one checkout on the Linux host; setup.py, even + # with nothing to change, takes git's index locks in .sources/. + step_lock = DirLock(ROOT / "target/ci-remote/setup.lock") + while not step_lock.try_acquire({"root": str(ROOT)}): + time.sleep(1) + self.log(f"--> {workflow} {job_id}: {label} (cwd {cwd.relative_to(ROOT) if cwd != ROOT else '.'})", + echo=not getattr(self.args, "worker_job", None) or self.args.verbose) for line in script.rstrip().splitlines(): self.log(f" $ {line}", echo=self.args.verbose) started = time.time() @@ -707,8 +822,8 @@ def run_step(self, workflow, job_id, job, index, step, outputs): if code == 0 and building_kernel: self.finish_kernel_build() finally: - if kernel_lock: - kernel_lock.release() + if step_lock: + step_lock.release() for line in output_file.read_text().splitlines(): if "=" in line and step.get("id"): k, v = line.split("=", 1) @@ -769,7 +884,8 @@ def run_job(self, workflow, job_id, job): runs_on = str(job.get("runs-on", "")) self.log(f"\n=== {workflow} / {job_id} (GitHub: {runs_on}) ===") if JOBS.get(key, {}).get("linux_only_note") and "ubuntu" in runs_on and sys.platform == "darwin": - note = f"{key} runs on {runs_on} in GitHub; here on macOS, so #[cfg(target_os = \"linux\")] code in it is not exercised" + note = (f"{key} runs on {runs_on} in GitHub; here on macOS, so #[cfg(target_os = \"linux\")] code in it " + f"is not exercised (--linux-host runs it on Linux)") if note not in self.notes: self.notes.append(note) problem = self.job_problem(workflow, job_id, job) @@ -793,6 +909,14 @@ def git(self, *args): result = subprocess.run(["git", "-C", str(ROOT), *args], capture_output=True, text=True) return result.stdout.strip() if result.returncode == 0 else "" + def versions(self): + versions = {} + for tool, argv in (("cargo", ["cargo", "--version"]), ("rustc", ["rustc", "--version"]), + ("python3", ["python3", "--version"]), ("node", ["node", "--version"])): + if self.which(tool): + versions[tool] = subprocess.run(argv, env=self.env, capture_output=True, text=True).stdout.strip() + return versions + def execute(self, workflows): started = time.time() sha = self.git("rev-parse", "HEAD") @@ -800,11 +924,7 @@ def execute(self, workflows): self.log(f"ci-local: {ROOT} at {sha} ({self.git('rev-parse', '--abbrev-ref', 'HEAD')})" + (" with uncommitted changes" if dirty else "")) self.log(f"ci-local: workflows {', '.join(workflows)}; CARGO_BUILD_JOBS={self.args.jobs}; log {self.log_path}") - versions = {} - for tool, argv in (("cargo", ["cargo", "--version"]), ("rustc", ["rustc", "--version"]), - ("python3", ["python3", "--version"]), ("node", ["node", "--version"])): - if self.which(tool): - versions[tool] = subprocess.run(argv, env=self.env, capture_output=True, text=True).stdout.strip() + versions = self.versions() self.log("ci-local: " + "; ".join(f"{v}" for v in versions.values())) if not versions.get("python3", "").startswith("Python 3.12"): self.notes.append(f"GitHub runs Python 3.12; this run used {versions.get('python3', 'no python3')}") @@ -812,10 +932,22 @@ def execute(self, workflows): drift = check_drift() self.record("ci-local", "drift", "The local mapping fits the workflows", FAIL if drift else PASS, reason="; ".join(drift)) - for workflow in workflows: - data = load_workflow(workflow) - for job_id, job in jobs_of(workflow, data): + plan = plan_jobs(workflows, linux_host=self.linux is not None) + remote = [(w, j, job) for w, j, job, where in plan if where == "linux"] + self.linux_info = None + thread = None + if remote: + self.log(f"ci-local: on the Linux host: {', '.join(f'{w} / {j}' for w, j, _ in remote)}") + thread = threading.Thread(target=self.run_remote, args=(sha, bool(dirty), remote), daemon=True) + thread.start() + for workflow, job_id, job, where in plan: + if where == "local": self.run_job(workflow, job_id, job) + if thread: + while thread.is_alive(): + thread.join(timeout=1) + order = {(w, j): i for i, (w, j, _, _) in enumerate(plan)} + self.steps.sort(key=lambda s: order.get((s["workflow"], s["job"]), -1)) passed = not any(s["status"] in (FAIL, NOT_RUN) for s in self.steps) summary = { @@ -831,6 +963,7 @@ def execute(self, workflows): "started": datetime.datetime.fromtimestamp(started).isoformat(timespec="seconds"), "seconds": round(time.time() - started, 1), "host": {"system": platform.system(), "machine": platform.machine(), "cpus": os.cpu_count()}, + "linux_host": self.linux_info, "versions": versions, "jobs": self.args.jobs, "notes": self.notes, @@ -840,12 +973,370 @@ def execute(self, workflows): } self.log("\n" + format_table(summary)) tmp = self.out_dir / "last.json.tmp" - tmp.write_text(json.dumps(summary, indent=2) + "\n") + tmp.write_text(self.clean(json.dumps(summary, indent=2)) + "\n") os.replace(tmp, self.out_dir / "last.json") self.log(f"\nci-local: {'PASSED' if passed else 'FAILED'} in {fmt_seconds(summary['seconds'])}; " f"log {self.log_path}; result target/ci-local/last.json") return summary + # The Linux host: ship the commit, run the coordinator there (--linux-worker), + # relay its output, then merge its results into this run's steps. + def remote_fail(self, keys, reason): + for key in keys: + workflow, job_id = key.split(":", 1) + self.add_remote_steps([{"workflow": workflow, "job": job_id, "name": "Run on the Linux host", + "status": FAIL, "seconds": 0, "reason": reason, + "expected_skip": None, "command": None}], sha=None) + + def add_remote_steps(self, steps, sha): + with self.lock: + for step in steps: + step = dict(step, host="linux", sha=sha, reason=self.clean(step.get("reason") or "")) + self.steps.append(step) + if step["status"] != PASS: + line = f"[{step['status']}] {step['workflow']} {step['job']} (linux): {step['name']}" + self.log_file.write(self.clean(line + (f" -- {step['reason']}" if step["reason"] else "")) + "\n") + + def run_remote(self, sha, dirty, jobs): + keys = [f"{w}:{j}" for w, j, _ in jobs] + host = LinuxHost(self.linux, self.log) + try: + if dirty: + self.log(f"ci-local: the Linux host runs the commit {sha[:12]}, without this tree's uncommitted changes") + started = time.time() + run_dir = host.ship(sha, self.stamp) + self.log(f"ci-local: linux: {sha[:12]} checked out as ~/{CI_HOME}/runs/{run_dir} " + f"({fmt_seconds(time.time() - started)})") + result = host.run(run_dir, sha, keys, self.args) + except HostError as error: + self.log(f"ci-local: linux: {error}") + self.remote_fail(keys, f"the Linux host: {error}") + return + merge_remote_result(self, result, sha, keys) + + +def merge_remote_result(run, result, sha, keys): + """Add the coordinator's `result` for `keys` to `run`, bound to `sha`.""" + if result.get("error"): + run.remote_fail(keys, f"the Linux host: {result['error']}") + return + if result.get("sha") != sha: + run.remote_fail(keys, f"stale: the Linux host ran {str(result.get('sha'))[:12]}, not {sha[:12]}") + return + if result.get("busy"): + for key in keys: + workflow, job_id = key.split(":", 1) + run.add_remote_steps([{"workflow": workflow, "job": job_id, "name": "Run on the Linux host", + "status": SKIPPED, "seconds": 0, "expected_skip": False, "command": None, + "reason": "every Linux host slot is busy (--no-wait)"}], sha=sha) + return + for key in keys: + job = (result.get("jobs") or {}).get(key) + if not job: + run.remote_fail([key], "the Linux host returned no result for this job") + continue + run.add_remote_steps(job.get("steps") or [], sha=result["sha"]) + run.actions += [dict(a, host="linux") for a in job.get("actions") or []] + run.notes += [f"linux: {n}" for n in job.get("notes") or [] if f"linux: {n}" not in run.notes] + run.linux_info = {k: result.get(k) for k in ("sha", "run", "slot", "slots", "system", "machine", "cpus", + "kernel", "versions", "seconds", "prepare_seconds")} + python = (result.get("versions") or {}).get("python3", "no python3") + if not python.startswith("Python 3.12"): + run.notes.append(f"GitHub runs Python 3.12; the Linux host used {python}") + + +# --------------------------------------------------------------------------- +# The Linux host (--linux-host). This side ships the commit and starts the +# coordinator over ssh; the coordinator (--linux-worker, on the host) takes a +# host slot, prepares the checkout and runs one worker (--worker-job) per job +# in parallel. Nothing here needs more than the ssh user: no sudo. +# --------------------------------------------------------------------------- + +CI_HOME = "octosense-ci" # under the ssh user's home on the host +RESULT_MARK = "::ci-local-result:: " +KEEP_RUNS = 5 +LINUX_SLOTS = 4 +BUILD_ENV = Path.home() / ".config/octosense/build.env" +# The only environment remote steps get: no tokens, no keys, nothing from +# this machine. $HOME and the user are the host's own. +REMOTE_ENV = ('HOME="$HOME" USER="$(id -un)" LOGNAME="$(id -un)" LANG=C.UTF-8 ' + f'PATH="$HOME/{CI_HOME}/venv/bin:$HOME/.cargo/bin:/usr/local/bin:/usr/bin:/bin"') + + +class HostError(RuntimeError): + pass + + +def linux_host_settings(value): + """(ssh target, key path or None) from --linux-host, else the environment + (OCTOSENSE_BUILD_HOST, OCTOSENSE_BUILD_KEY), else ~/.config/octosense/build.env.""" + env = dict(os.environ) + if not env.get("OCTOSENSE_BUILD_HOST") and BUILD_ENV.is_file(): + for line in BUILD_ENV.read_text().splitlines(): + match = re.match(r"\s*(?:export\s+)?(OCTOSENSE_BUILD_[A-Z_]+)=(.*)$", line) + if match: + env.setdefault(match.group(1), match.group(2).strip().strip("'\"")) + target = value or env.get("OCTOSENSE_BUILD_HOST") + if not target: + return None, None + key = env.get("OCTOSENSE_BUILD_KEY") or None + return target, (os.path.expanduser(os.path.expandvars(key)) if key else None) + + +class LinuxHost: + def __init__(self, settings, log): + self.target, self.key = settings + self.log = log + + def ssh_argv(self, command): + return ["ssh", "-o", "BatchMode=yes", "-o", "ServerAliveInterval=30", "-o", "ServerAliveCountMax=6", + *(["-i", self.key] if self.key else []), self.target, command] + + def ssh(self, command, stdin=None): + result = subprocess.run(self.ssh_argv(command), stdin=stdin if stdin is not None else subprocess.DEVNULL, + capture_output=True, text=True) + if result.returncode != 0: + raise HostError(f"ssh exited {result.returncode}: {(result.stderr or result.stdout).strip()[-800:]}") + return result.stdout + + def ship(self, sha, stamp): + """Check out `sha` in a new run directory on the host; its name.""" + known = self.ssh(f"git -C ~/{CI_HOME}/repo.git cat-file -e {sha}^{{commit}} 2>/dev/null && echo have || " + f"{{ echo need; git -C ~/{CI_HOME}/repo.git for-each-ref --sort=-committerdate --count=50 " + f"--format='%(objectname)' refs/ci 2>/dev/null || true; }}").split() + run_dir = f"{sha}-{stamp}" + bundle = None + try: + if known[:1] != ["have"]: + bases = [r for r in known[1:] if re.fullmatch(r"[0-9a-f]{40}", r) and subprocess.run( + ["git", "-C", str(ROOT), "cat-file", "-e", f"{r}^{{commit}}"], capture_output=True).returncode == 0] + handle = tempfile.NamedTemporaryFile(suffix=".bundle", delete=False) + handle.close() + bundle = handle.name + ref = "refs/ci-local/linux-host" + subprocess.run(["git", "-C", str(ROOT), "update-ref", ref, sha], check=True) + try: + made = subprocess.run(["git", "-C", str(ROOT), "bundle", "create", "--quiet", bundle, ref, + *(f"^{b}" for b in bases)], capture_output=True, text=True) + finally: + subprocess.run(["git", "-C", str(ROOT), "update-ref", "-d", ref]) + if made.returncode != 0: + raise HostError(f"git bundle: {made.stderr.strip()}") + self.log(f"ci-local: linux: sending {sha[:12]} ({os.path.getsize(bundle) // 1024} KiB bundle, " + f"{len(bases)} known commits on the host)") + script = PREPARE_RUN.replace("@SHA@", sha).replace("@DIR@", run_dir).replace( + "@BUNDLE@", "1" if bundle else "0").replace("@CI@", CI_HOME) + with open(bundle or os.devnull, "rb") as stdin: + checked_out = self.ssh("bash -c " + shlex.quote(script), stdin=stdin).split() + finally: + if bundle: + os.unlink(bundle) + if checked_out[-1:] != [sha]: + raise HostError(f"the host checked out {checked_out[-1:]}, not {sha}") + return run_dir + + def run(self, run_dir, sha, keys, args): + options = [f"--linux-slots={args.linux_slots}", f"--expect-sha={sha}", f"--worker-jobs={','.join(keys)}"] + if args.linux_jobs: + options.append(f"--jobs={args.linux_jobs}") + options += [o for o, on in (("--keep-going", args.keep_going), ("--verbose", args.verbose), + ("--no-wait", args.no_wait)) if on] + command = (f"cd ~/{CI_HOME}/runs/{run_dir} && exec env -i {REMOTE_ENV} " + f"python3 tools/ci_local.py --linux-worker {' '.join(shlex.quote(o) for o in options)}") + process = subprocess.Popen(self.ssh_argv(command), stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, text=True, errors="replace") + result = None + tail = [] + for line in process.stdout: + line = line.rstrip("\n") + if line.startswith(RESULT_MARK): + result = json.loads(line[len(RESULT_MARK):]) + continue + tail = (tail + [line])[-20:] + self.log(f"linux| {line}") + code = process.wait() + if result is None: + raise HostError(f"no result (exit {code}): " + " / ".join(tail[-5:])) + return result + + +# Runs on the host (bash -c) with the bundle, if any, on stdin: fetch the +# commit into ~/octosense-ci/repo.git and check it out as a worktree. +PREPARE_RUN = r"""set -euo pipefail +ci="$HOME/@CI@" +mkdir -p "$ci/runs" "$ci/cache" "$ci/locks" +[ -d "$ci/repo.git" ] || git init --quiet --bare "$ci/repo.git" +exec 9>"$ci/locks/repo.flock" +flock 9 +if [ @BUNDLE@ = 1 ]; then + incoming="$ci/runs/.incoming-$$.bundle" + trap 'rm -f "$incoming"' EXIT + cat > "$incoming" + git -C "$ci/repo.git" fetch --quiet "$incoming" "refs/ci-local/linux-host:refs/ci/@SHA@" +fi +git -C "$ci/repo.git" update-ref "refs/ci/@SHA@" "@SHA@" +git -C "$ci/repo.git" worktree prune +git -C "$ci/repo.git" worktree add --quiet --detach "$ci/runs/@DIR@" "@SHA@" +git -C "$ci/runs/@DIR@" rev-parse HEAD +""" + + +def slug(key): + return re.sub(r"[^A-Za-z0-9_.-]+", "-", key) + + +def prepare_linux_checkout(ci, log): + """On the host, under the prepare lock: the source hubs (cloned once), + tools/setup.py in this checkout, and old run directories removed.""" + spec = importlib_util().spec_from_file_location("octosense_setup", ROOT / "tools/setup.py") + setup = importlib_util().module_from_spec(spec) + spec.loader.exec_module(setup) + hub = ci / "cache/hub" + for name, url in {"octoscript-makepad": setup.URL, **setup.RUNTIME_URLS}.items(): + if not (hub / name / ".git").exists(): + log(f"ci-local: cloning the {name} hub (once)") + partial = hub / f".{name}.partial" + shutil.rmtree(partial, ignore_errors=True) + subprocess.run(["git", "clone", "--quiet", "--no-checkout", "--filter=blob:none", url, str(partial)], + check=True) + os.replace(partial, hub / name) + result = subprocess.run([sys.executable, "tools/setup.py"], cwd=ROOT, capture_output=True, text=True) + if result.returncode != 0: + raise HostError("tools/setup.py failed: " + (result.stderr or result.stdout).strip()[-1500:]) + runs = sorted((p for p in (ci / "runs").iterdir() if p.is_dir() and not p.name.startswith(".")), + key=lambda p: p.stat().st_mtime, reverse=True) + for old in runs[KEEP_RUNS:]: + if old.resolve() == ROOT or run_is_active(old): + continue + log(f"ci-local: removing the old run {old.name}") + shutil.rmtree(old, ignore_errors=True) + for repo in [ci / "repo.git", *(p for p in hub.iterdir() if (p / ".git").exists())]: + subprocess.run(["git", "-C", str(repo), "worktree", "prune"], capture_output=True) + + +def importlib_util(): + import importlib.util + return importlib.util + + +def run_is_active(path): + try: + return pid_alive(int((path / "target/ci-remote/active").read_text())) + except (OSError, ValueError): + return False + + +def linux_worker(args): + """The coordinator, on the host: one slot, one prepare, a worker per job.""" + ci = Path.home() / CI_HOME + os.environ["OCTOSENSE_CI_LOCAL_LOCKS"] = str(ci / "locks/linux-slots") + os.environ["OCTOSENSE_CI_LOCAL_CACHE"] = str(ci / "cache") + os.environ["OCTOSENSE_SOURCES_HUB"] = str(ci / "cache/hub") + started = time.time() + + def emit(result): + print(RESULT_MARK + json.dumps(result), flush=True) + + def stop(signum, frame): + raise SystemExit(128 + signum) + signal.signal(signal.SIGTERM, stop) + signal.signal(signal.SIGHUP, stop) + sha = subprocess.run(["git", "-C", str(ROOT), "rev-parse", "HEAD"], capture_output=True, text=True).stdout.strip() + if sha != args.expect_sha: + emit({"sha": sha, "error": f"the checkout is at {sha[:12]}, not {args.expect_sha[:12]}"}) + return 1 + state = ROOT / "target/ci-remote" + state.mkdir(parents=True, exist_ok=True) + (state / "active").write_text(str(os.getpid())) + slot = acquire_slot(args.linux_slots, not args.no_wait, lambda m: print(m, flush=True), {"root": str(ROOT)}) + if slot is None: + emit({"sha": sha, "busy": True}) + return EXIT_BUSY + workers = [] + try: + number = int(slot.path.name.split("-")[-1]) + print(f"ci-local: Linux host slot {number + 1} of {args.linux_slots}", flush=True) + prepare = DirLock(ci / "locks/prepare") + while not prepare.try_acquire({"root": str(ROOT)}): + time.sleep(2) + try: + prepare_linux_checkout(ci, lambda m: print(m, flush=True)) + except (HostError, subprocess.CalledProcessError, OSError) as error: + emit({"sha": sha, "error": f"preparing the checkout: {error}"}) + return 1 + finally: + prepare.release() + prepare_seconds = time.time() - started + keys = [k for k in args.worker_jobs.split(",") if k] + jobs = args.jobs if args.jobs_given else max(1, (os.cpu_count() or 4) // args.linux_slots) + printing = threading.Lock() + + def relay(key, process): + for line in process.stdout: + with printing: + try: + sys.stdout.write(line if line.startswith("[") or line.startswith("ci-local:") + else f" {key}| {line}") + sys.stdout.flush() + except BrokenPipeError: + pass + + threads = [] + for key in keys: + env = dict(os.environ, CARGO_TARGET_DIR=str(ci / f"cache/target/slot-{number}" / slug(key))) + argv = [sys.executable, str(Path(__file__).resolve()), f"--worker-job={key}", + f"--worker-out={state / slug(key)}", f"--jobs={jobs}"] + argv += [o for o, on in (("--keep-going", args.keep_going), ("--verbose", args.verbose)) if on] + process = subprocess.Popen(argv, cwd=ROOT, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, + text=True, errors="replace", start_new_session=True) + workers.append(process) + thread = threading.Thread(target=relay, args=(key, process), daemon=True) + thread.start() + threads.append(thread) + for process in workers: + process.wait() + for thread in threads: + thread.join(timeout=5) + results = {} + versions = {} + for key in keys: + try: + data = json.loads((state / slug(key) / "result.json").read_text()) + except (OSError, ValueError): + continue + versions = versions or data.get("versions", {}) + results[key] = data + emit({"sha": sha, "run": ROOT.name, "slot": number + 1, "slots": args.linux_slots, + "system": platform.system(), "machine": platform.machine(), "cpus": os.cpu_count(), + "kernel": platform.release(), "versions": versions, "jobs": results, + "prepare_seconds": round(prepare_seconds, 1), "seconds": round(time.time() - started, 1)}) + return 0 + finally: + for process in workers: + if process.poll() is None: + try: + os.killpg(process.pid, signal.SIGTERM) + except OSError: + pass + slot.release() + try: + (state / "active").unlink() + except OSError: + pass + + +def run_worker(args): + """One job (--worker-job :), its result in --worker-out.""" + run = Run(args, out_dir=args.worker_out) + workflow, job_id = args.worker_job.split(":", 1) + run.run_job(workflow, job_id, job_definition(args.worker_job)) + result = {"key": args.worker_job, "steps": run.steps, "actions": run.actions, "notes": run.notes, + "versions": run.versions(), "log": str(run.log_path.relative_to(ROOT))} + tmp = run.out_dir / "result.json.tmp" + tmp.write_text(json.dumps(result, indent=2) + "\n") + os.replace(tmp, run.out_dir / "result.json") + return 0 + def fmt_seconds(seconds): seconds = int(round(seconds)) @@ -853,7 +1344,7 @@ def fmt_seconds(seconds): def format_table(summary, markdown=False): - rows = [(f"{s['workflow']} / {s['job']}", s["name"], s["status"] + ("" if s["expected_skip"] in (None, True) else " (!)"), + rows = [(f"{s['workflow']} / {s['job']}" + (" (linux)" if s.get("host") == "linux" else ""), s["name"], s["status"] + ("" if s["expected_skip"] in (None, True) else " (!)"), fmt_seconds(s["seconds"]) if s["seconds"] else "-") for s in summary["steps"]] lines = [] if markdown: @@ -873,7 +1364,8 @@ def format_table(summary, markdown=False): if skipped: lines.append("") lines.append("Skipped (these did NOT pass; (!) marks a skip that blocks tools/ci-local-merge.sh):") - lines += [f"- {s['workflow']} / {s['job']}: {s['name']}: {s['reason']}" for s in skipped] + lines += [f"- {s['workflow']} / {s['job']}{' (linux)' if s.get('host') == 'linux' else ''}: {s['name']}: {s['reason']}" + for s in skipped] if summary.get("notes"): lines.append("") lines.append("Notes:") @@ -881,26 +1373,25 @@ def format_table(summary, markdown=False): return "\n".join(lines) -def print_plan(workflows): - for workflow in workflows: - for job_id, job in jobs_of(workflow): - print(f"{workflow} / {job_id} (GitHub: {job.get('runs-on')})") - for index, step in enumerate(job.get("steps") or []): - label = step_label(step, index) - if "uses" in step: - print(f" [action] {step['uses']} -> {ACTIONS.get(action_name(step['uses']), 'NO MAPPING')}") - else: - cwd = step.get("working-directory") or ((job.get("defaults") or {}).get("run") or {}).get("working-directory") or "." - extra = " (needs: " + STEP_REQUIREMENTS[f"{workflow}:{job_id}:{label}"]["hint"] + ")" \ - if f"{workflow}:{job_id}:{label}" in STEP_REQUIREMENTS else "" - print(f" [run] {label} (cwd {cwd}){extra}") +def print_plan(workflows, linux_host=False): + for workflow, job_id, job, where in plan_jobs(workflows, linux_host): + print(f"{workflow} / {job_id} (GitHub: {job.get('runs-on')})" + (" -> the Linux host" if where == "linux" else "")) + for index, step in enumerate(job.get("steps") or []): + label = step_label(step, index) + if "uses" in step: + print(f" [action] {step['uses']} -> {ACTIONS.get(action_name(step['uses']), 'NO MAPPING')}") + else: + cwd = step.get("working-directory") or ((job.get("defaults") or {}).get("run") or {}).get("working-directory") or "." + extra = " (needs: " + STEP_REQUIREMENTS[f"{workflow}:{job_id}:{label}"]["hint"] + ")" \ + if f"{workflow}:{job_id}:{label}" in STEP_REQUIREMENTS else "" + print(f" [run] {label} (cwd {cwd}){extra}") def main(argv=None): parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) parser.add_argument("--only", default="all", metavar="|".join(sorted(GROUPS)), help="Which workflows to run, comma-separated (default all)") - parser.add_argument("--jobs", type=int, default=max(1, (os.cpu_count() or 2) // 2), + parser.add_argument("--jobs", type=int, default=None, help="CARGO_BUILD_JOBS for the run (default: half the CPUs)") parser.add_argument("--keep-going", action="store_true", help="Run every step even after a failure") parser.add_argument("--no-wait", action="store_true", help=f"Exit {EXIT_BUSY} instead of waiting for a free slot") @@ -909,14 +1400,38 @@ def main(argv=None): parser.add_argument("--verbose", "-v", action="store_true", help="Echo every step's output (always in the log)") parser.add_argument("--list", action="store_true", help="Print the plan and exit") parser.add_argument("--check-drift", action="store_true", help="Check the local mapping against the workflows and exit") - parser.add_argument("--linux-host", metavar="SSH", help=argparse.SUPPRESS) + parser.add_argument("--linux-host", nargs="?", const="", default=None, metavar="SSH", + help="Run the ubuntu jobs and the Linux-only checks on a Linux build host over ssh " + "(default: OCTOSENSE_BUILD_HOST, with the key OCTOSENSE_BUILD_KEY, from the " + "environment or ~/.config/octosense/build.env)") + parser.add_argument("--linux-slots", type=int, default=int(os.environ.get("OCTOSENSE_CI_LINUX_SLOTS", LINUX_SLOTS)), + help=f"How many runs may share the Linux host (default {LINUX_SLOTS}, or OCTOSENSE_CI_LINUX_SLOTS)") + parser.add_argument("--linux-jobs", type=int, default=0, + help="CARGO_BUILD_JOBS for each job on the Linux host (default: its CPUs / --linux-slots)") + # Internal: what runs on the Linux host. + parser.add_argument("--linux-worker", action="store_true", help=argparse.SUPPRESS) + parser.add_argument("--expect-sha", help=argparse.SUPPRESS) + parser.add_argument("--worker-jobs", default="", help=argparse.SUPPRESS) + parser.add_argument("--worker-job", help=argparse.SUPPRESS) + parser.add_argument("--worker-out", type=Path, help=argparse.SUPPRESS) args = parser.parse_args(argv) + args.jobs_given = args.jobs is not None + if args.jobs is None: + args.jobs = max(1, (os.cpu_count() or 2) // 2) + if args.linux_worker: + return linux_worker(args) + if args.worker_job: + return run_worker(args) unknown = [g for g in args.only.split(",") if g not in GROUPS] if unknown: parser.error(f"--only: unknown {', '.join(unknown)} (choose from {', '.join(sorted(GROUPS))})") workflows = [w for w in GROUPS["all"] if any(w in GROUPS[g] for g in args.only.split(","))] - if args.linux_host: - parser.error("--linux-host is not implemented: the ubuntu jobs run on this Mac (see docs/local-ci.md)") + args.linux_settings = None + if args.linux_host is not None: + args.linux_settings = linux_host_settings(args.linux_host) + if not args.linux_settings[0]: + parser.error("--linux-host: name the ssh target, or set OCTOSENSE_BUILD_HOST (and OCTOSENSE_BUILD_KEY) " + "in the environment or ~/.config/octosense/build.env") if args.check_drift: problems = check_drift() for problem in problems: @@ -924,7 +1439,7 @@ def main(argv=None): print("ci-local: the local mapping fits the workflows" if not problems else f"ci-local: {len(problems)} drift problem(s)") return 1 if problems else 0 if args.list: - print_plan(workflows) + print_plan(workflows, args.linux_settings is not None) return 0 clone_lock = DirLock(ROOT / "target/ci-local/running.lock") if not clone_lock.try_acquire({"root": str(ROOT)}): diff --git a/tools/ci_local_merge.py b/tools/ci_local_merge.py index 741cd521..d97e2c86 100644 --- a/tools/ci_local_merge.py +++ b/tools/ci_local_merge.py @@ -10,7 +10,12 @@ rebase main into the branch, push, and run tools/ci-local.sh again); - every workflow GitHub would run for the PR's files (its pull_request paths), with no FAIL, no step left NOT RUN and no unexpected SKIP in them; - a workflow GitHub would not run for the PR does not block it. + a workflow GitHub would not run for the PR does not block it; +- steps that ran on the Linux host (--linux-host; `"host": "linux"`) count + like local ones, but only if the host ran that same head commit: a remote + result bound to another commit is stale. The Linux-only checks + (ci_local.LINUX_HOST_JOBS) count for the PRs that trigger the workflows + they cover. It also refuses while the last completed push run of a workflow on main failed: a red main is fixed first (--fixes-main for the PR that fixes it). @@ -62,8 +67,20 @@ def evidence_problems(last, head, changed_files): problems.append(f"the run did not cover {', '.join(missing)} (GitHub runs them for this PR): " f"run tools/ci-local.sh --only all") scope = set(required) | {"ci-local"} + for job_id, job in ci_local.LINUX_HOST_JOBS.items(): + if set(ci_local.linux_host_job_covers(job)) & set(required): + scope.add(ci_local.LINUX_HOST) + stale = [s for s in last.get("steps", []) if s.get("host") == "linux" and s.get("sha") != head] + if stale: + ran = sorted({str(s.get("sha"))[:12] for s in stale}) + problems.append(f"stale: {len(stale)} step(s) on the Linux host ran on {', '.join(ran)}, not the PR head " + f"{head[:12]} (first: {stale[0]['workflow']} / {stale[0]['job']}: {stale[0]['name']})") for step in last.get("steps", []): where = f"{step['workflow']} / {step['job']}: {step['name']}" + if step in stale: + continue + if step.get("host") == "linux": + where += " (Linux host)" if step["workflow"] not in scope: continue # GitHub would not run it for this PR either if step["status"] == ci_local.FAIL: @@ -94,8 +111,14 @@ def red_main(repo_workflows): def comment_body(last, required): head = last["sha"] - lines = [f"Local CI passed on {head} (`tools/ci-local.sh --only {last['only']}`, " - f"{ci_local.fmt_seconds(last['seconds'])}, {last['host']['system']} {last['host']['machine']}).", + linux = last.get("linux_host") or {} + where = f"{last['host']['system']} {last['host']['machine']}" + flag = "" + if linux: + flag = " --linux-host" + where += f"; the ubuntu jobs on the Linux build host, {linux.get('system')} {linux.get('machine')}" + lines = [f"Local CI passed on {head} (`tools/ci-local.sh --only {last['only']}{flag}`, " + f"{ci_local.fmt_seconds(last['seconds'])}, {where}).", "", f"GitHub would run for this PR: {', '.join(required) or 'none'}. Merged on this local " f"pass (macOS runner queue); GitHub CI runs on the merge commit on main.", diff --git a/tools/test_ci_local.py b/tools/test_ci_local.py index 3d754194..71d2e314 100644 --- a/tools/test_ci_local.py +++ b/tools/test_ci_local.py @@ -5,6 +5,7 @@ from pathlib import Path import shutil import tempfile +from types import SimpleNamespace import unittest from unittest.mock import patch @@ -199,5 +200,149 @@ def test_the_comment_names_the_commit(self): self.assertIn("| phone.yml / home | Test | PASS |", body) +class LinuxHostPlan(unittest.TestCase): + def where(self, workflows, linux_host): + return {(w, j): where for w, j, _, where in ci.plan_jobs(workflows, linux_host)} + + def test_without_a_host_everything_runs_here_and_nothing_is_added(self): + plan = self.where(ci.GROUPS["all"], False) + self.assertEqual(set(plan.values()), {"local"}) + self.assertNotIn(ci.LINUX_HOST, {w for w, _ in plan}) + + def test_ubuntu_jobs_go_to_the_host_and_macos_jobs_stay(self): + plan = self.where(["apps.yml"], True) + self.assertEqual(plan[("apps.yml", "services")], "linux") + self.assertEqual(plan[("apps.yml", "kernel-security")], "linux") + self.assertEqual(plan[("apps.yml", "apps")], "local") + self.assertEqual(plan[(ci.LINUX_HOST, "sandbox")], "linux", "the sandbox tests come with apps") + for workflow in ci.GROUPS["all"]: + for job_id, job in ci.jobs_of(workflow): + expected = "linux" if "ubuntu" in job["runs-on"] else "local" + self.assertEqual(self.where([workflow], True)[(workflow, job_id)], expected, (workflow, job_id)) + + def test_the_sandbox_tests_follow_the_workflows_that_cover_the_shell(self): + self.assertIn((ci.LINUX_HOST, "sandbox"), self.where(["phone.yml"], True)) + self.assertNotIn((ci.LINUX_HOST, "sandbox"), self.where(["rom.yml"], True), + "rom.yml does not run for a change to the shell") + job = ci.job_definition(f"{ci.LINUX_HOST}:sandbox") + self.assertIn("octosense-shell", job["steps"][-1]["run"]) + self.assertEqual(ci.job_definition("apps.yml:services")["runs-on"], "ubuntu-latest") + + def test_a_linux_host_job_whose_source_moved_is_drift(self): + moved = {"sandbox": dict(ci.LINUX_HOST_JOBS["sandbox"], source="crates/shell/src/gone.rs")} + with patch.object(ci, "LINUX_HOST_JOBS", moved): + self.assertTrue(any("gone.rs" in p for p in ci.check_drift())) + + def test_the_host_comes_from_the_flag_the_environment_or_build_env(self): + with tempfile.TemporaryDirectory() as temp: + env_file = Path(temp) / "build.env" + env_file.write_text("OCTOSENSE_BUILD_HOST=someone@build.invalid\nexport OCTOSENSE_BUILD_KEY=\"$HOME/k/id\"\n") + clean = {k: v for k, v in os.environ.items() if not k.startswith("OCTOSENSE_BUILD_")} + with patch.object(ci, "BUILD_ENV", env_file), patch.dict(os.environ, clean, clear=True): + target, key = ci.linux_host_settings("") + self.assertEqual(target, "someone@build.invalid") + self.assertEqual(key, os.path.expanduser("~/k/id")) + self.assertEqual(ci.linux_host_settings("other@x.invalid")[0], "other@x.invalid") + with patch.object(ci, "BUILD_ENV", Path(temp) / "none"), patch.dict(os.environ, clean, clear=True): + self.assertEqual(ci.linux_host_settings(""), (None, None)) + + +def remote_run(temp, target="someone@build.invalid"): + args = SimpleNamespace(jobs=1, verbose=False, keep_going=False, linux_settings=(target, "/keys/id_build")) + run = ci.Run(args, out_dir=Path(temp)) + return run + + +def host_result(sha, steps_by_key): + return {"sha": sha, "run": f"{sha}-x", "slot": 1, "slots": 4, "system": "Linux", "machine": "x86_64", + "cpus": 72, "versions": {}, "seconds": 10, "prepare_seconds": 1, + "jobs": {key: {"steps": steps, "actions": [], "notes": []} for key, steps in steps_by_key.items()}} + + +class LinuxHostResults(unittest.TestCase): + sha = "d" * 40 + + def step(self, job="services", status="PASS", workflow="apps.yml"): + return {"workflow": workflow, "job": job, "name": "Kernel service", "status": status, "seconds": 5, + "reason": "", "expected_skip": None, "command": "cargo test", "host": "local"} + + def test_remote_steps_join_the_run_marked_linux_and_bound_to_the_commit(self): + with tempfile.TemporaryDirectory() as temp: + run = remote_run(temp) + self.addCleanup(run.close) + keys = ["apps.yml:services", "linux-host:sandbox"] + ci.merge_remote_result(run, host_result(self.sha, { + keys[0]: [self.step()], keys[1]: [self.step("sandbox", "FAIL", ci.LINUX_HOST)]}), self.sha, keys) + self.assertEqual([(s["job"], s["host"], s["sha"], s["status"]) for s in run.steps], + [("services", "linux", self.sha, "PASS"), ("sandbox", "linux", self.sha, "FAIL")]) + self.assertEqual(run.linux_info["machine"], "x86_64") + table = ci.format_table({"steps": run.steps, "notes": []}) + self.assertIn("apps.yml / services (linux)", table) + + def test_a_result_for_another_commit_is_refused(self): + with tempfile.TemporaryDirectory() as temp: + run = remote_run(temp) + self.addCleanup(run.close) + ci.merge_remote_result(run, host_result("e" * 40, {"apps.yml:services": [self.step()]}), + self.sha, ["apps.yml:services"]) + self.assertEqual([s["status"] for s in run.steps], ["FAIL"]) + self.assertIn("stale", run.steps[0]["reason"]) + + def test_a_missing_job_or_a_busy_host_never_passes(self): + with tempfile.TemporaryDirectory() as temp: + run = remote_run(temp) + self.addCleanup(run.close) + ci.merge_remote_result(run, host_result(self.sha, {}), self.sha, ["apps.yml:services"]) + ci.merge_remote_result(run, {"sha": self.sha, "busy": True}, self.sha, ["apps.yml:kernel-security"]) + ci.merge_remote_result(run, {"sha": self.sha, "error": "setup failed"}, self.sha, ["linux-host:sandbox"]) + self.assertEqual([(s["status"], s["expected_skip"]) for s in run.steps], + [("FAIL", None), ("SKIPPED", False), ("FAIL", None)]) + + def test_the_host_address_and_key_never_reach_the_log(self): + with tempfile.TemporaryDirectory() as temp: + run = remote_run(temp) + self.addCleanup(run.close) + run.log("ssh: connect to host build.invalid port 22 as someone@build.invalid with /keys/id_build", echo=False) + run.remote_fail(["apps.yml:services"], "ssh: Could not resolve hostname build.invalid") + run.log_file.flush() + text = run.log_path.read_text() + json.dumps(run.steps) + self.assertNotIn("build.invalid", text) + self.assertNotIn("id_build", text) + + def test_the_remote_environment_is_an_allow_list(self): + self.assertTrue(ci.REMOTE_ENV.startswith('HOME="$HOME"')) + for name in ("TOKEN", "KEY", "SECRET", "GH_", "GITHUB", "SSH_AUTH"): + self.assertNotIn(name, ci.REMOTE_ENV) + + +class MergeLinuxHost(unittest.TestCase): + head = "b" * 40 + + def remote(self, sha, workflow="apps.yml", job="services", status="PASS"): + return {"workflow": workflow, "job": job, "name": "Kernel service", "status": status, "seconds": 3, + "reason": "", "expected_skip": None, "host": "linux", "sha": sha} + + def test_a_remote_pass_on_the_head_is_evidence(self): + last = result(steps=[self.remote(self.head), self.remote(self.head, ci.LINUX_HOST, "sandbox")], + linux_host={"system": "Linux", "machine": "x86_64"}) + problems, _ = merge.evidence_problems(last, self.head, ["crates/kernel/src/lib.rs"]) + self.assertEqual(problems, []) + self.assertIn("--linux-host", merge.comment_body(last, ["apps.yml"])) + self.assertIn("| apps.yml / services (linux) | Kernel service | PASS |", merge.comment_body(last, ["apps.yml"])) + + def test_a_stale_remote_result_is_refused_even_when_the_run_is_on_the_head(self): + problems, _ = merge.evidence_problems(result(steps=[self.remote("c" * 40)]), self.head, ["crates/kernel/src/lib.rs"]) + self.assertTrue(any(p.startswith("stale: 1 step(s) on the Linux host ran on cccccccccccc") for p in problems), problems) + problems, _ = merge.evidence_problems(result(steps=[self.remote(None)]), self.head, ["crates/kernel/src/lib.rs"]) + self.assertTrue(any("stale" in p for p in problems), "a remote step bound to no commit is stale") + + def test_the_linux_only_checks_block_the_prs_that_trigger_what_they_cover(self): + sandbox = self.remote(self.head, ci.LINUX_HOST, "sandbox", "FAIL") + problems, _ = merge.evidence_problems(result(steps=[sandbox], passed=False), self.head, ["crates/shell/src/lib.rs"]) + self.assertTrue(any("failed: linux-host / sandbox" in p for p in problems), problems) + problems, _ = merge.evidence_problems(result(steps=[sandbox], passed=False), self.head, ["rom/README.md"]) + self.assertEqual(problems, [], "a rom-only PR does not run the shell's checks") + + if __name__ == "__main__": unittest.main()