agent-record is RED on main and therefore on every pull request, because
tests/scripts/test_check_site.py invokes hugo with no availability guard and
the job never installs it.
Measured
tests/scripts/test_check_site.py:126-132,
test_rendered_benchmark_index_links_resolve_to_emitted_pages:
result = subprocess.run(
[
"hugo",
...
- No guard of any kind.
grep -nE "shutil.which|skipTest|FileNotFoundError"
over that file returns nothing. A missing binary raises FileNotFoundError
from subprocess.run, which surfaces as an error rather than a skip.
- The
agent-record job does not install hugo. .github/workflows/ci.yml:127
onward has no hugo setup step.
- Introduced by #1714
(1db7e59cf docs: retire shared status and split benchmark details), the last
commit to touch that file.
scripts/main-baseline.py shows agent-record among the failures on main's own
baseline at 08c81a892189, and not at the earlier deb53c6a33d9 — consistent
with #1714 as the introducing change.
Why this is worth fixing rather than tolerating
It is not merely one more red. It is a red on a records gate, on a repository
where a large share of pull requests are records-heavy, at a moment when several
in-flight rows must each argue that every failing job is inherited.
That is the exact configuration in which a real agent-record failure would
be waved through as "the known one". A records PR that genuinely broke the issue
index would fail the same job with the same job name, and the standing red gives
every reviewer a ready-made reason not to look further.
Note also that audit-live-rows.py and check-agent-record.py both pass
locally on main — so the record checkers themselves are fine, and anyone
diagnosing this from the job name alone will look in the wrong place.
Repair options, in preference order
- Guard the test.
shutil.which("hugo") or self.skipTest("hugo not installed"). Cheapest, and correct: a docs-site rendering test has no business
erroring when the site generator is absent. But a silent skip in CI is its own
trap — if the job is meant to cover this, a skip means it does not.
- Install hugo in the
agent-record job. Restores the coverage the test was
written for. Costs job minutes on a job whose comment at ci.yml:99 notes it
is deliberately a 3.8-minute ubuntu job.
- Move the test out of
agent-record into whichever lane already has hugo,
if one does.
Option 1 plus option 2 together — guard and install — is the only combination
where the test both cannot error and actually runs.
Provenance
Found by a fresh reviewer working PR #1704, which needed to establish whether its
own agent-record red was inherited. The FileNotFoundError at line 130 is that
reviewer's reading of the CI log; the absent guard, the absent install step and
the #1714 attribution I verified directly in the tree.
agent-recordis RED onmainand therefore on every pull request, becausetests/scripts/test_check_site.pyinvokeshugowith no availability guard andthe job never installs it.
Measured
tests/scripts/test_check_site.py:126-132,test_rendered_benchmark_index_links_resolve_to_emitted_pages:grep -nE "shutil.which|skipTest|FileNotFoundError"over that file returns nothing. A missing binary raises
FileNotFoundErrorfrom
subprocess.run, which surfaces as an error rather than a skip.agent-recordjob does not install hugo..github/workflows/ci.yml:127onward has no hugo setup step.
(
1db7e59cf docs: retire shared status and split benchmark details), the lastcommit to touch that file.
scripts/main-baseline.pyshowsagent-recordamong the failures on main's ownbaseline at
08c81a892189, and not at the earlierdeb53c6a33d9— consistentwith #1714 as the introducing change.
Why this is worth fixing rather than tolerating
It is not merely one more red. It is a red on a records gate, on a repository
where a large share of pull requests are records-heavy, at a moment when several
in-flight rows must each argue that every failing job is inherited.
That is the exact configuration in which a real
agent-recordfailure wouldbe waved through as "the known one". A records PR that genuinely broke the issue
index would fail the same job with the same job name, and the standing red gives
every reviewer a ready-made reason not to look further.
Note also that
audit-live-rows.pyandcheck-agent-record.pyboth passlocally on
main— so the record checkers themselves are fine, and anyonediagnosing this from the job name alone will look in the wrong place.
Repair options, in preference order
shutil.which("hugo") or self.skipTest("hugo not installed"). Cheapest, and correct: a docs-site rendering test has no businesserroring when the site generator is absent. But a silent skip in CI is its own
trap — if the job is meant to cover this, a skip means it does not.
agent-recordjob. Restores the coverage the test waswritten for. Costs job minutes on a job whose comment at
ci.yml:99notes itis deliberately a 3.8-minute ubuntu job.
agent-recordinto whichever lane already has hugo,if one does.
Option 1 plus option 2 together — guard and install — is the only combination
where the test both cannot error and actually runs.
Provenance
Found by a fresh reviewer working PR #1704, which needed to establish whether its
own
agent-recordred was inherited. TheFileNotFoundErrorat line 130 is thatreviewer's reading of the CI log; the absent guard, the absent install step and
the #1714 attribution I verified directly in the tree.