fix(check-docs): saying a host runs no Alloy counted it as running one - #480
Merged
Merged
Conversation
count_alloy_agents tested `"alloy" in cell`, which cannot tell "Alloy agent (Docker)" from "runs no Alloy". Seven rows of docs/architecture.md say the second thing. The bug was survivable only by accident: every one of them is also marked Not built yet, so NOT_BUILT masked the negation — and it would have surfaced on the commit that dropped that marker, which is the commit already busy changing this count for a real reason, and where a wrong count reads as a stale sentence in hardware.md rather than as a miscount. Found 2026-09-16 when smaug's row said "No Alloy agent — this host is scraped" and pushed the count to five. bahamut, leviathan, titan, ramuh, carbuncle and siren all carry "runs no Alloy" and were queued to do the same, one at a time, as the lab domain gets built. Negations are stripped before the substring test rather than the affirmative being matched, and that choice is the part worth recording. Matching "alloy agent" instead looks tidier and is wrong: prometheus names Alloy in a service list — "docker-socket-proxy, Alloy" — and never says "Alloy agent" at all, so that test drops a host which genuinely runs one and quietly reports three. An over-count fails loudly against hardware.md's sentence; an under-count is a checker agreeing with a document that has gone stale, which is the failure this whole file exists to prevent. Verified: deleting prometheus's Alloy mention takes the count 4 -> 3, so it was being counted by that bare word alone. Measured rather than asserted, by building each row in memory and counting both ways: scenario OLD NEW today, as committed 4 4 build bahamut (runs no Alloy) 5 4 old overcounts build smaug (runs no Alloy) 5 4 old overcounts build titan (runs no Alloy) 5 4 old overcounts build trinity (has an agent) 5 5 both rise, correct The last line is the one that matters for not weakening the assertion: a host that really does declare an agent still moves the count the moment its marker comes off, which is exactly when hardware.md's sentence should be forced to change. All seven negative rows hold at four; trinity and odin both rise to five. The regex knows two shapes, "runs no Alloy" and "No Alloy agent", and the comment says plainly that a negation it does not know would overcount — loudly, but it should be extended here rather than worked around by reaching for different words in the table. smaug's row was reworded to dodge this on 2026-09-16 (c938c37); that reword is now belt and braces rather than the fix. Refs #88 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
count_alloy_agentstested"alloy" in cell, which cannot tellAlloy agent (Docker)fromruns no Alloy. Seven rows say the secondthing.
The bug was survivable only by accident: every one of them is also marked
Not built yet, so
NOT_BUILTmasked the negation — and it would havesurfaced on the commit that dropped that marker. Which is the commit already
busy changing this count for a real reason, and where a wrong count reads as a
stale sentence in
hardware.mdrather than as a miscount.Found 2026-09-16 when
smaug's row said "No Alloy agent — this host isscraped" and pushed the count to five.
bahamut,leviathan,titan,ramuh,carbuncleandsirenall carry "runs no Alloy" and were queued todo the same, one at a time, as the lab domain gets built.
Why negations are stripped rather than the affirmative matched
This is the part worth recording, because the tidier-looking fix is wrong.
Matching
alloy agentinstead breaksprometheus: its cell names Alloy ina service list —
…docker-socket-proxy, Alloy— and never says "Alloy agent"at all. That test drops a host which genuinely runs one and quietly reports
three.
An over-count fails loudly against
hardware.md's sentence. An under-count isa checker agreeing with a document that has gone stale — the exact failure this
whole file exists to prevent.
Verified: deleting
prometheus's Alloy mention takes the count 4 → 3, so itwas being counted by that bare word alone.
Measured, not asserted
Each row built in memory and counted both ways:
bahamut(runs no Alloy)smaug(runs no Alloy)titan(runs no Alloy)trinity(has an agent)The last row is the one that matters for not weakening the assertion: a
host that really does declare an agent still moves the count the moment its
marker comes off — exactly when
hardware.md's sentence should be forced tochange. All seven negative rows hold at four;
trinityandodinboth rise tofive.
Scope
The regex knows two shapes,
runs no AlloyandNo Alloy agent. The commentsays plainly that a negation it does not know would overcount — loudly — and
that it should be extended there rather than worked around by reaching for
different words in the table.
smaug's row was reworded to dodge this on 2026-09-16 (c938c37, in #479);that reword is now belt and braces rather than the fix.
make validatepasses;check_docsgreen at 10 assertions, count still 4.Refs #88
🤖 Generated with Claude Code