diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 6b3443488..396fc4696 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -675,3 +675,5 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1791](https://github.com/mudler/vllm.cpp/issues/1791) | `SPEC-DSPARK` | **`scripts/dspark-paired-e2e.sh`'s `settle()` can never break early, so a wait for the GPU to drain always spends its full 360 s however fast the box actually drains.** The same idiom as [#1734](https://github.com/mudler/vllm.cpp/issues/1734), found by sweeping `scripts/` for it: `grep -c .` with an `\|\| echo 0` fallback makes `$n` the two-line string `0\n0`, so `[ "$n" -eq 0 ] && break` answers `integer expression expected` and returns 2 instead of deciding. The BUSY half of the guard works -- a positive count exits 0 and the fallback does not fire -- so only the FREE half is dead, and the failure is in the safe direction, which is why it was paid in silence. This is the FIFTH diagnosis of the idiom in this tree: `scripts/cpu-x86-llamacpp-floor.sh` already carries the removal and the reason in a comment, and a comment in one file is not reachable from another. FIXED IN FLOW with #1734: `\|\| true` keeps grep's own `0` and swallows only its status. The recurrence gate is `TheIdiomIsGoneFromEveryShellScript` in `tests/scripts/test_ltx25_ab_memwatch.py`, which sweeps every `scripts/*.sh` for a counting `grep`/`pgrep` paired with an `\|\| echo` fallback outside a comment; run against `27d8bfa70` it names all three live instances, this one included. It is a TRIPWIRE and says so: it reads text, and a `wc -l` with the same fallback walks past it | bug | | [#1796](https://github.com/mudler/vllm.cpp/issues/1796) | `SPEC-DFLASH2` | **[#1456](https://github.com/mudler/vllm.cpp/issues/1456)'s conclusion is retracted AT THE ARTIFACT, off-GPU, and the DFlash2 speed ratio's denominator inherits the retraction: `0.8016987337853048` MAY BE FLATTERING US.** #1456 measured a real source build at `CUDA_ARCHS=12.0` emitting `sm_80` for `_vllm_fa2_C` and `sm_75` for `_vllm_fa3_C`, and concluded the GB10 oracle has no `FLASH_ATTN` denominator. **The arch measurement stands; the conclusion does not, and no lease was needed to show it.** Both staged oracle wheels were opened with `zipfile` and walked as fatbinaries on the CPU dev box: `0.1.dev1+g66e5414c6` (this row's oracle, `/mnt/nas_share/rc/oracle-dflash2/`) and `0.1.dev1+g555967922` (the parity pin, `/mnt/nas_share/rc/oracle-vllm/`) each carry 76 FA2 fatbinaries and 192 FA3 fatbinaries, and **every one holds a PTX image beside its SASS image** -- arch 80 for FA2, arch 75 for FA3. The first FA2 PTX payload is zstd and decompresses to `.version 9.0` / `.target sm_80` for `flash_fwd_hdim128_bf16_causal_sm80`. That is the `+PTX` half of `FA2_ARCHS "8.0+PTX"` and it is the mechanism by which the module CAN reach sm_121 -- the artifact establishes a necessary condition, forward-JITtable code being shipped, and that the JIT then ran is an inference from that plus a run selecting `FLASH_ATTN` and generating; `cudaErrorUnsupportedPtxVersion` is the OPPOSITE failure, PTX ISA newer than the driver, which `.version 9.0` under driver 580.173.02 is not. vLLM says it directly too: `FlashAttentionBackend.supports_compute_capability` returns `capability >= DeviceCapability(8, 0)` (`vllm/v1/attention/backends/flash_attn.py:251-252` in the staged wheel). **THE DOWNSTREAM IS THE THING TO CHASE.** #1456's body records that the DFlash2 speed gate's denominator "will be vLLM pinned to `TRITON_ATTN`, by developer decision on 2026-08-20", explicitly because of that conclusion. AGENTS.md requires vLLM's PRODUCTION configuration as the denominator, and on this box vLLM's own auto-selection picks `FLASH_ATTN`: the 0.8017 gate run's log carries BOTH paths in one process, the forced branch taking `TRITON_ATTN` for the 27B target (`cuda.py:426`) and the auto branch taking `FLASH_ATTN` out of four valid backends (`cuda.py:486`), in `/mnt/nas_share/rc/dflash2-1673/out-n1673b/m-gate.log`. **So if `TRITON_ATTN` is the slower backend -- which vLLM's own priority ordering IMPLIES rather than states, and which nothing here measures -- then 16.279 tok/s is too LOW and the ratio is too HIGH. An error in our own favour is the one nobody chases, so it is recorded with its sign.** The ratio is NOT withdrawn and no denominator is substituted, because a wave must not replace a denominator the developer declared. **The five FA layers of [#1685](https://github.com/mudler/vllm.cpp/issues/1685) fall out of the same read**: `vllm/v1/worker/gpu/spec_decode/dflash/utils.py:31-46` sets the draft's `backend=speculative_config.attention_backend` UNCONDITIONALLY, so a harness that sets only the engine backend leaves the draft at `None` and the draft re-runs auto-selection; the siblings `dspark/utils.py:24-28` and `gemma4/speculator.py:66-89` both carry the target's backend through, and dspark's comment names this hazard by name. NOT reconciled in flow, and the reason is precise: what remains is ONE lease running vLLM against itself on this workload with `attention_backend=FLASH_ATTN` against `TRITON_ATTN`, each read back off the built engine, and **no GPU lease is authorised for this session**, so it is PENDING. Recommended collapse, recommended and not taken: #1456, [#1679](https://github.com/mudler/vllm.cpp/issues/1679) and #1685 observe ONE mechanism on two wheels, and #1685 is the one attached to a live number and should survive. Owed under `## Owed` O33 of [the DFlash2 spec](specs/dflash2-spec-decode.md), with caveat 5 beside the ratio in the same file | verification | | [#1794](https://github.com/mudler/vllm.cpp/issues/1794) | `LTX25-DIT-ATTN-ARM-PARSE` | **#1551 renamed the values of `VLLM_LTX2_DIT_FLASH_ATTN` and the two #1549-era LTX-2.5 A/B harnesses kept exporting the old ones, so three committed arms name a rung they do not select -- and one of the three says nothing.** At [#1549](https://github.com/mudler/vllm.cpp/issues/1549) the knob was BINARY: `=0` selected `vt::Attention` and every other value, unset included, selected `vt::AttentionDenseFlash`. [#1551](https://github.com/mudler/vllm.cpp/issues/1551) made it THREE-WAY, moved the unset default up a rung to `vt::AttentionDenseFa2` and gave the flash rung the exact spelling `flash`; it touched neither harness. Read at `27d8bfa70`: `scripts/ltx25-dit-attn-flash-pixel-ab.sh` exports `=1` for both its `flash` and its `flash-ctl` arms, and `scripts/ltx25-dit-attn-flash-ab.sh` leaves the variable UNSET for its `flash` arm -- all three of which selected `AttentionDenseFlash` before #1551 and select `AttentionDenseFa2` after it. `scripts/ltx25-dit-attn-fa2-hd128-ab.sh` is CORRECT and is the contrast that makes the other two legible: written after the rename, it exports `flash` / `""` / `0` and carries `assert_arm_op`. THE ASYMMETRY IS THE ISSUE: the pixel harness's `arm_report` counts `op=18` and `op=21` in the arm's own log and exits 46 on `ROUTING_BAD`, so its two `=1` arms abort -- after the render, an hour of a four-hour lease spent to learn a literal went stale -- while `ltx25-dit-attn-flash-ab.sh` has NO such assertion at all: its phase `[F]` PRINTS the op-provider selections and asserts nothing, so its `flash` arm renders FA-2, is reduced under the label `flash`, and the ratio it publishes is FA-2 against naive while the file's own header says flash against naive. THE RECORDED NUMBERS ARE NOT INVALIDATED: `.agents/specs/ltx25-dit-attn-flash.md` section 10 records `knob=1` announcing `op=21` and `ROUTING_OK=flash`, so the binary that ran it is one where `=1` still selected flash; what is wrong is the file as it stands against the binary `main` builds today. WHY A GATE AND NOT A REVIEWER: both harnesses were TRUE when written and both became false in a commit that touched neither file, because nothing connected the literal in a shell script to the literal in the C++ dispatch. FIXED IN FLOW with the red-first case that would have caught it -- `tests/scripts/test_ltx2_dit_attn_knob_arms.py` reads the accepted set out of `ltx2_device.cpp`'s own `std::strcmp(arm, "...")` calls rather than restating it, reads each harness's arm invocations, and asserts BOTH that every value is one the dispatch parses AND that every arm selects the rung its LABEL claims; the second half is the one that matters, because `unset` is a perfectly valid value and was still the wrong arm for a whole row. Registered on the preflight and CI record lanes, and it asserts both registrations itself. Found while fixing [#1751](https://github.com/mudler/vllm.cpp/issues/1751), which is the same defect one level down. Spec [`ltx25-dit-attn-arm-parse.md`](specs/ltx25-dit-attn-arm-parse.md) | bug | +| [#1754](https://github.com/mudler/vllm.cpp/issues/1754) | `GATE-CI-SITE-HUGO-LANE` | **`agent-record` is RED on `main` and on every pull request cut from it: `test_rendered_benchmark_index_links_resolve_to_emitted_pages` shells out to `hugo` and no job in `ci.yml` installs it, so `subprocess.run` raises `FileNotFoundError` before any assertion runs and `unittest` reports an ERROR.** Introduced by [#1714](https://github.com/mudler/vllm.cpp/pull/1714) at `1db7e59cf`; the scheduled baseline at `deb53c6a3` was green on this job and `08c81a892` and every scheduled run after it is red. FIXED IN FLOW by installing the renderer on the lane -- the same `peaceiris/actions-hugo@v3` action, the same `extended: true`, and the same `0.146.3` pin the `gh-pages` job publishes on -- plus `hugo version` as the step's first line and `tests/scripts/test_ci_site_lane.py` holding the two pins equal and the setup step present and ordered before the render. A SKIP GUARD WAS REJECTED AS THE FIX and is deliberately not in this change: it retires the red by arranging for the assertion to run nowhere, which [PR #1726](https://github.com/mudler/vllm.cpp/pull/1726) demonstrates -- its `agent-record` is SUCCESS having rendered no site. The guard is still wanted for a developer box without Hugo and is separately owned by [#1722](https://github.com/mudler/vllm.cpp/issues/1722) / #1726, which edits a disjoint file, so the two compose. [#1764](https://github.com/mudler/vllm.cpp/issues/1764) names this red as one of its three and is not closed by this change | bug | +| [#1828](https://github.com/mudler/vllm.cpp/issues/1828) | `GATE-CI-SITE-HUGO-LANE` | **`tests/scripts/test_check_site.py` asserts a literal `10` rendered benchmark detail links, which is a stored count of `docs/benchmarks/*.md` inside another file, and #1754's lane fix is what ARMS it**: until Hugo was installed the case never reached that line in CI. Measured on `d60692c89` with Hugo 0.146.3+extended: 10 slugs, 10 table hrefs, no duplicated target and no unlinked slug, so the relationship is a bijection and the literal is derivable at read time -- with its own non-vacuity floor, since `sorted(x) == sorted(y)` is satisfied by two empty sets. NOT FIXED IN FLOW: deriving it changes what the case asserts rather than how it spells a number, which `AGENTS.md` routes through the normal row, spec and fresh-review path, and the file already has an open pull request against the same case. Listed under `## Owed` in [gate-ci-site-hugo-lane.md](specs/gate-ci-site-hugo-lane.md) | bug | diff --git a/.agents/specs/gate-ci-site-hugo-lane.md b/.agents/specs/gate-ci-site-hugo-lane.md new file mode 100644 index 000000000..fca8ee31e --- /dev/null +++ b/.agents/specs/gate-ci-site-hugo-lane.md @@ -0,0 +1,200 @@ +# The lane that renders the docs site installs the renderer + +Identity: `GATE-CI-SITE-HUGO-LANE` + +Issue: [#1754](https://github.com/mudler/vllm.cpp/issues/1754) + +Related, separately owned: +[#1722](https://github.com/mudler/vllm.cpp/issues/1722) with +[PR #1726](https://github.com/mudler/vllm.cpp/pull/1726) (the developer-box +guard), and [#1764](https://github.com/mudler/vllm.cpp/issues/1764) (a +three-job umbrella that names this red as one of its three). + +Stored-count drift lock this change arms: +[#1828](https://github.com/mudler/vllm.cpp/issues/1828). See `## Owed`. + +## Now + +`agent-record` is red on `main` and on every pull request cut from it. +`tests/scripts/test_check_site.py`'s +`test_rendered_benchmark_index_links_resolve_to_emitted_pages`, added by +[#1714](https://github.com/mudler/vllm.cpp/pull/1714) at `1db7e59cf`, shells out +to `hugo`. No job in `.github/workflows/ci.yml` installs it, so `subprocess.run` +raises `FileNotFoundError`, `unittest` records an ERROR, and the step exits 1. +The regression window is measured: the scheduled baseline at `deb53c6a3` was +green on this job and `08c81a892` and every scheduled run after it is red. + +## Scope + +Provision Hugo on the CI lane that runs the docs-site suite, and gate that +provisioning so it cannot be removed silently. + +Out of scope: the contents of `tests/scripts/test_check_site.py`. This change +edits none of its lines, deliberately — see `## Design` and `## Owed`. + +## The design decision, and why the obvious fix is rejected + +The one-line repair is `shutil.which("hugo") or self.skipTest(...)`. It makes +the job green and it is what #1722 asks for, so it deserves a reason rather +than a preference. + +An absent binary has two failure modes and neither of them is a verdict on this +repository: + +| binary | guard | what CI reports | what was rendered | +|---|---|---|---| +| absent | none | ERROR, job red | nothing | +| absent | skip | `OK (skipped=1)`, job green | nothing | +| present | either | `ok`, job green | the whole site | + +Only the third row is a measurement. The first two differ in how loudly they +say the same thing, and the second says it in the register this repository has +been bitten by before: a skip wearing a pass. A guard alone would retire the red +by arranging for the assertion #1714 was written for to run **nowhere** — +PR #1726's `agent-record` is already SUCCESS for exactly that reason, having +rendered no site. + +So the guard is not the fix; it is the choice of which non-verdict a missing +binary produces. Installing the binary is the fix, and it is what +[#1754](https://github.com/mudler/vllm.cpp/issues/1754) names as option 2 and +calls, together with the guard, "the only combination where the test both cannot +error and actually runs". This change lands that half. #1726 lands the other, +and the two touch disjoint files so they compose without a conflict. + +The escalation clause in the dispatch — a skip is acceptable only when paired +with a CI-side assertion that the lane did not skip — is therefore not reached. +Nothing here prevented the install. + +## Design + +Three edits to `.github/workflows/ci.yml`, all inside the `agent-record` job. + +1. A job-level `HUGO_VERSION: "0.146.3"`, the same value + `.github/workflows/gh-pages.yml` pins at workflow level, and a + `peaceiris/actions-hugo@v3` step with `extended: true` — the same action, the + same major, the same build as the job that publishes the site. Reusing the + published lane's mechanism rather than inventing one is the point: a lane + rendering on a different Hugo returns a verdict about a site nobody visits. + The pin is not arbitrary. `website/hugo.toml` uses `excludeFiles`, which Hugo + deprecates from 0.153 in favour of `files`; bumping past that without + migrating the key publishes `docs/bench-evidence` and `docs/superpowers`. +2. `hugo version` as the first line of the step that runs the suite. It is the + line that fails legibly, and early, if the setup step ever stops producing a + binary. +3. Registration of the new guard suite on the same job. + +One new file, `tests/scripts/test_ci_site_lane.py`, holds four invariants over +the workflow and two over its own non-vacuity: + +- The lane installs Hugo, exactly once, **before** the step that renders. +- It asks for the extended build. +- Its pin equals `gh-pages.yml`'s, resolved through `${{ env.NAME }}` in both + files rather than compared as template strings, and it is an exact + `MAJOR.MINOR.PATCH` rather than a floating tag. +- The rendering step probes the binary before invoking the suite. + +The invariants are derived from the workflow, not written as a list of job +names: whichever job runs `tests/scripts/test_check_site.py` is the job that +must provision Hugo, so a step that moves to another job stays covered. + +Two of the six cases exist only to defeat vacuity, because that is the shape +this file would otherwise take. `test_the_resolver_finds_the_lane_it_is_about` +fails if the resolver stops seeing the suite, which every other case needs to be +true to mean anything, and the two loops over setup steps count what they saw +and fail on zero rather than passing an empty iteration. + +## Risks + +- **Job minutes.** `ci.yml:99` records `agent-record` as a 3.8-minute Ubuntu + job. `peaceiris/actions-hugo` downloads one release archive; the site itself + renders in 0.5 s locally, inside the suite that already runs. Accepted: the + alternative is a 3.8-minute job that renders nothing. +- **A second copy of the version.** Two files carrying one pin is the drift + shape this repository calls a lock. It is answered by holding them equal in a + test rather than by a shared file that every pull request would have to write. +- **A network dependency on the lane.** The action fetches from GitHub releases. + A fetch failure fails the step, which is a legible infrastructure red and not + a silent pass. + +## Tests + +- `tests/scripts/test_ci_site_lane.py` — six cases, run by `agent-record`. +- `tests/scripts/test_check_site.py` — unchanged, and now actually executed on + the lane rather than erroring on it. + +## Gates + +```sh +python3 tests/scripts/test_ci_site_lane.py +python3 scripts/check-site.py +python3 tests/scripts/test_check_site.py +scripts/agent-preflight.sh +``` + +## Evidence + +Host: `mudler-ubuntu-box`, x86_64, Python 3.12.3, `hugo +v0.146.3+extended linux/amd64`. Base `d60692c89`. + +**Red before, the reported failure.** `tests/scripts/test_check_site.py` at +sha256 `4ee0f66d…`, run with a `PATH` from which `/home/mudler/.local/bin` is +removed so `command -v hugo` reports nothing: + +``` +test_rendered_benchmark_index_links_resolve_to_emitted_pages ... ERROR +FileNotFoundError: [Errno 2] No such file or directory: 'hugo' +Ran 7 tests in 0.272s +FAILED (errors=1) +``` + +exit 1 — the CI text, reproduced locally. + +**Red before, the new guard.** `tests/scripts/test_ci_site_lane.py` against a +scratch tree holding `origin/main`'s two workflow files (`ci.yml` at sha256 +`f9f351e1…`, verified equal to `git show origin/main:.github/workflows/ci.yml`): +`Ran 6 tests`, `FAILED (failures=4)`, exit 1, naming +`agent-record: hugo setup steps at indices []` and `agent-record: the step runs +the site suite without probing hugo`. + +**Green after.** On this tree: + +- `tests/scripts/test_ci_site_lane.py -v`: `Ran 6 tests`, `OK`, exit 0, six + `ok`, zero skips. +- `tests/scripts/test_check_site.py -v` with Hugo on `PATH`: `Ran 7 tests`, + `OK`, exit 0, the case reporting `ok` and **not** `skipped`. Zero skips is the + load-bearing half of that line. + +**Mutations.** Each applied to the tree, proved applied by a diff, parsed or +compiled, then restored and re-verified by sha256 (`ci.yml` +`8f311a70…`, `test_check_site.py` `4ee0f66d…`). + +| mutation | result | +|---|---| +| delete only the `peaceiris/actions-hugo@v3` step | `FAILED (failures=3)`: absent setup, and both vacuity floors fire | +| `HUGO_VERSION` `0.146.3` → `0.147.0` | `FAILED (failures=1)`: `'0.147.0' != '0.146.3'`, naming the `excludeFiles` consequence | +| `assertEqual(len(detail_hrefs), 10)` → `11`, Hugo present | `FAILED (failures=1)`: `AssertionError: 10 != 11` | + +The third is the one that answers the question this change exists for. It fails +from inside the rendered site, which is only reachable when the binary is there, +so it proves the protected assertion **executed** rather than skipped. + +## Owed + +- [#1828](https://github.com/mudler/vllm.cpp/issues/1828) — `test_check_site.py` + asserts a literal `10` rendered detail links, a stored count of + `docs/benchmarks/*.md`. Until this change it was inert in CI, because the case + never reached the line; installing Hugo arms it. Measured on `d60692c89`: 10 + slugs, 10 table hrefs, no duplicate target, no unlinked slug — a bijection, + and therefore derivable at read time. Left out of scope because deriving it + changes what the case asserts, which `AGENTS.md` routes through the normal + row, spec and fresh-review path rather than the in-flow rule, and because + `tests/scripts/test_check_site.py` has an open pull request against the same + case. + +## Stop conditions + +- Stop and escalate if `peaceiris/actions-hugo@v3` cannot install 0.146.3 on + `ubuntu-latest`. The fallback is not a skip; it is a decision about which lane + renders the site. +- Stop if the pin has to move. Moving it past 0.153 is a `website/hugo.toml` + migration and belongs to `ENG-DOCS-SITE`, not here. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 82ad86ca9..4fec18dcd 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -153,10 +153,30 @@ jobs: # resolve a string and falls back instead of failing. needs: [last-gated-commit] runs-on: ubuntu-latest + env: + # The SAME pin as `.github/workflows/gh-pages.yml`, which publishes the + # site this job's suite renders. Two files carrying one version is a drift + # risk, so `tests/scripts/test_ci_site_lane.py` holds them equal rather + # than trusting whoever bumps one to remember the other. Read that file's + # comment before changing this value: 0.153 deprecates `excludeFiles`, + # which `website/hugo.toml` uses. + HUGO_VERSION: "0.146.3" steps: - uses: actions/checkout@v4 with: fetch-depth: 0 + # The docs-site suite RENDERS the site, so this lane needs the renderer. + # Without it `test_rendered_benchmark_index_links_resolve_to_emitted_pages` + # cannot run at all -- and neither of its two ways of not running is a + # verdict on this repository: unguarded it raises `FileNotFoundError` and + # reds the job (#1722, #1754), guarded it skips and the job goes green + # having rendered nothing. Installing the binary is what makes the + # assertion execute; the guard only decides which of the two non-verdicts + # an absent binary produces. + - uses: peaceiris/actions-hugo@v3 + with: + hugo-version: ${{ env.HUGO_VERSION }} + extended: true - name: Canonical roadmap tables and links are consistent # --report, wired the way check-device-leakage's is below: the record # anchor ratchet (ENG-RECORD-ANCHOR-RATCHET, #632) gates on a baseline, @@ -235,7 +255,12 @@ jobs: run: | python3 tests/scripts/test_env_agnostic_tooling.py - name: The docs site's content invariants hold + # `hugo version` first, deliberately. It is the one line that fails + # LEGIBLY when the setup step above stopped providing a binary, and it + # fails BEFORE the suite gets the chance to report that absence as + # either a red it did not cause or a skip nobody reads. run: | + hugo version python3 scripts/check-site.py python3 tests/scripts/test_check_site.py - name: Model add+RMSNorm glue routes through the fusion catalog @@ -246,6 +271,9 @@ jobs: run: | python3 scripts/check-attention-rung-consistency.py python3 tests/scripts/test_check_attention_rung_consistency.py + - name: The lane that renders the docs site actually provisions Hugo + run: | + python3 tests/scripts/test_ci_site_lane.py - name: Structural checkers ignore text the compiler never sees run: | python3 tests/scripts/test_checker_text.py diff --git a/tests/scripts/test_ci_site_lane.py b/tests/scripts/test_ci_site_lane.py new file mode 100644 index 000000000..9aad188fb --- /dev/null +++ b/tests/scripts/test_ci_site_lane.py @@ -0,0 +1,201 @@ +#!/usr/bin/env python3 +"""GATE-CI-SITE-HUGO-LANE (#1754) -- the docs-site lane provisions its renderer. + +`tests/scripts/test_check_site.py` renders the whole Hugo site and holds every +link in the rendered benchmark index to an emitted page. That assertion is only +worth its job minutes on a runner that HAS Hugo, and the failure this file exists +to prevent has two faces that a job name cannot tell apart: + + * No binary and no guard -- `subprocess.run` raises `FileNotFoundError`, the + suite ERRORs, and `agent-record` is red on `main` and on every branch cut + from it (#1722, #1754, #1764). + * No binary and a guard -- the case skips, the suite reports `OK (skipped=1)`, + and the job goes GREEN having rendered nothing. A skip wearing a pass is the + worse of the two, because the red at least gets looked at. + +Only installing the binary distinguishes them, so the invariants below are about +the LANE, not about the suite. They are derived from the workflow rather than +written as a list of job names: whichever job runs the site suite is the job that +must provision Hugo, so a step that moves to another job stays covered. + +Non-vacuity is asserted first and separately. A resolver that finds no job passes +every assertion after it, which is the mute switch this file would otherwise be. +""" + +from __future__ import annotations + +import re +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] +CI = ROOT / ".github/workflows/ci.yml" +PAGES = ROOT / ".github/workflows/gh-pages.yml" + +SITE_SUITE = "tests/scripts/test_check_site.py" +HUGO_ACTION = "peaceiris/actions-hugo" + +# `${{ env.NAME }}`, with the whitespace GitHub tolerates inside the braces. +_ENV_REF = re.compile(r"^\$\{\{\s*env\.([A-Za-z_][A-Za-z0-9_]*)\s*\}\}$") + + +def _load(path: Path) -> dict: + import yaml + + return yaml.safe_load(path.read_text(encoding="utf-8")) + + +def _resolve(value: object, *scopes: dict) -> object: + """Resolve a `${{ env.NAME }}` reference against the given env scopes. + + The pin is spelled through a workflow- or job-level `env` in both files, so + comparing the two `with:` values literally would compare two identical + template strings and prove nothing about the versions behind them. + """ + + if not isinstance(value, str): + return value + match = _ENV_REF.match(value.strip()) + if match is None: + return value + name = match.group(1) + for scope in scopes: + if name in scope: + return scope[name] + raise AssertionError(f"{value} names an env key no scope defines") + + +def _steps(job: dict) -> list[dict]: + return [step for step in (job.get("steps") or []) if isinstance(step, dict)] + + +def _runs_site_suite(step: dict) -> bool: + return SITE_SUITE in (step.get("run") or "") + + +def _is_hugo_setup(step: dict) -> bool: + return (step.get("uses") or "").startswith(HUGO_ACTION + "@") + + +def _jobs_running_the_site_suite(workflow: dict) -> list[tuple[str, dict]]: + return [ + (name, job) + for name, job in (workflow.get("jobs") or {}).items() + if any(_runs_site_suite(step) for step in _steps(job)) + ] + + +class SiteLaneProvisioningTests(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.ci = _load(CI) + cls.pages = _load(PAGES) + + # ---- non-vacuity ---------------------------------------------------- + + def test_the_resolver_finds_the_lane_it_is_about(self) -> None: + """Every assertion below is vacuous if this one does not hold.""" + lanes = _jobs_running_the_site_suite(self.ci) + self.assertEqual( + [name for name, _ in lanes], + ["agent-record"], + "the docs-site suite moved, or the resolver stopped seeing it", + ) + + def test_the_published_site_still_installs_hugo_through_the_action(self) -> None: + """The pin this file compares against has to exist to be compared.""" + setups = [ + step for step in _steps(self.pages["jobs"]["build"]) if _is_hugo_setup(step) + ] + self.assertEqual(len(setups), 1, f"gh-pages.yml hugo setup steps: {setups}") + + # ---- the invariant -------------------------------------------------- + + def test_the_lane_installs_hugo_before_it_renders(self) -> None: + for name, job in _jobs_running_the_site_suite(self.ci): + steps = _steps(job) + setup = [i for i, step in enumerate(steps) if _is_hugo_setup(step)] + self.assertEqual( + len(setup), 1, f"{name}: hugo setup steps at indices {setup}" + ) + renders = [i for i, step in enumerate(steps) if _runs_site_suite(step)] + self.assertTrue(renders, f"{name}: resolver disagrees with itself") + self.assertLess( + setup[0], + min(renders), + f"{name}: hugo is installed after the suite that needs it", + ) + + def test_the_lane_asks_for_the_extended_build(self) -> None: + """`gh-pages` renders `extended`. A lane on plain Hugo renders a + different site from the published one, so its verdict is about a site + nobody visits.""" + seen = 0 + for name, job in _jobs_running_the_site_suite(self.ci): + for step in _steps(job): + if _is_hugo_setup(step): + seen += 1 + self.assertIs( + (step.get("with") or {}).get("extended"), + True, + f"{name}: hugo setup is not the extended build", + ) + self.assertTrue(seen, "no hugo setup step to judge; this case was vacuous") + + def test_the_lane_and_the_published_site_pin_the_same_hugo(self) -> None: + published = None + for step in _steps(self.pages["jobs"]["build"]): + if _is_hugo_setup(step): + published = _resolve( + (step.get("with") or {}).get("hugo-version"), + self.pages.get("env") or {}, + ) + self.assertIsInstance(published, str, "gh-pages.yml pins no hugo version") + self.assertRegex( + str(published), + r"^\d+\.\d+\.\d+$", + "a floating hugo pin renders a different site every run", + ) + + seen = 0 + for name, job in _jobs_running_the_site_suite(self.ci): + for step in _steps(job): + if not _is_hugo_setup(step): + continue + seen += 1 + lane = _resolve( + (step.get("with") or {}).get("hugo-version"), + job.get("env") or {}, + self.ci.get("env") or {}, + ) + self.assertEqual( + lane, + published, + f"{name} renders on Hugo {lane}; the site publishes on " + f"{published}. website/hugo.toml uses `excludeFiles`, " + "deprecated from 0.153, so the two pins moving apart is a " + "difference in what gets published, not only in a version.", + ) + self.assertTrue(seen, "no hugo setup step to compare; this case was vacuous") + + def test_the_lane_proves_the_binary_resolved_before_running_the_suite(self) -> None: + """A setup step that stopped producing a binary must fail LEGIBLY. + + Without this line the suite reports the absence itself, as either a red + it did not cause or a skip nobody reads. + """ + for name, job in _jobs_running_the_site_suite(self.ci): + for step in _steps(job): + if not _runs_site_suite(step): + continue + body = step["run"].splitlines() + probe = [i for i, line in enumerate(body) if line.strip() == "hugo version"] + suite = [i for i, line in enumerate(body) if SITE_SUITE in line] + self.assertTrue( + probe, f"{name}: the step runs the site suite without probing hugo" + ) + self.assertLess(min(probe), min(suite), f"{name}: probe runs too late") + + +if __name__ == "__main__": + unittest.main()