Skip to content

fix(check-docs): saying a host runs no Alloy counted it as running one - #480

Merged
Gerrrt merged 1 commit into
mainfrom
gerrrt/alloy-count-negation
Sep 16, 2026
Merged

Gerrrt merged 1 commit into
mainfrom
gerrrt/alloy-count-negation

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Sep 16, 2026

Copy link
Copy Markdown
Owner

count_alloy_agents tested "alloy" in cell, which cannot tell
Alloy agent (Docker) from runs no Alloy. Seven rows say the second
thing.

The bug was survivable only by accident: every one of them is also marked
Not built yet, so NOT_BUILT masked the negation — and it would have
surfaced on the commit that dropped that marker. Which is the commit already
busy changing this count for a real reason, and where a wrong count reads as a
stale sentence in hardware.md rather than as a miscount.

Found 2026-09-16 when smaug's row said "No Alloy agent — this host is
scraped" and pushed the count to five. bahamut, leviathan, titan,
ramuh, carbuncle and siren all carry "runs no Alloy" and were queued to
do the same, one at a time, as the lab domain gets built.

Why negations are stripped rather than the affirmative matched

This is the part worth recording, because the tidier-looking fix is wrong.

Matching alloy agent instead breaks prometheus: its cell names Alloy in
a service list — …docker-socket-proxy, Alloy — and never says "Alloy agent"
at all. That test drops a host which genuinely runs one and quietly reports
three.

An over-count fails loudly against hardware.md's sentence. An under-count is
a checker agreeing with a document that has gone stale — the exact failure this
whole file exists to prevent.

Verified: deleting prometheus's Alloy mention takes the count 4 → 3, so it
was being counted by that bare word alone.

Measured, not asserted

Each row built in memory and counted both ways:

scenario OLD NEW
today, as committed 4 4 both correct
build bahamut (runs no Alloy) 5 4 old overcounts
build smaug (runs no Alloy) 5 4 old overcounts
build titan (runs no Alloy) 5 4 old overcounts
build trinity (has an agent) 5 5 both rise, correct

The last row is the one that matters for not weakening the assertion: a
host that really does declare an agent still moves the count the moment its
marker comes off — exactly when hardware.md's sentence should be forced to
change. All seven negative rows hold at four; trinity and odin both rise to
five.

Scope

The regex knows two shapes, runs no Alloy and No Alloy agent. The comment
says plainly that a negation it does not know would overcount — loudly — and
that it should be extended there rather than worked around by reaching for
different words in the table.

smaug's row was reworded to dodge this on 2026-09-16 (c938c37, in #479);
that reword is now belt and braces rather than the fix.

make validate passes; check_docs green at 10 assertions, count still 4.

Refs #88

🤖 Generated with Claude Code

count_alloy_agents tested `"alloy" in cell`, which cannot tell "Alloy agent
(Docker)" from "runs no Alloy". Seven rows of docs/architecture.md say the
second thing. The bug was survivable only by accident: every one of them is
also marked Not built yet, so NOT_BUILT masked the negation — and it would have
surfaced on the commit that dropped that marker, which is the commit already
busy changing this count for a real reason, and where a wrong count reads as a
stale sentence in hardware.md rather than as a miscount.

Found 2026-09-16 when smaug's row said "No Alloy agent — this host is scraped"
and pushed the count to five. bahamut, leviathan, titan, ramuh, carbuncle and
siren all carry "runs no Alloy" and were queued to do the same, one at a time,
as the lab domain gets built.

Negations are stripped before the substring test rather than the affirmative
being matched, and that choice is the part worth recording. Matching "alloy
agent" instead looks tidier and is wrong: prometheus names Alloy in a service
list — "docker-socket-proxy, Alloy" — and never says "Alloy agent" at all, so
that test drops a host which genuinely runs one and quietly reports three. An
over-count fails loudly against hardware.md's sentence; an under-count is a
checker agreeing with a document that has gone stale, which is the failure this
whole file exists to prevent. Verified: deleting prometheus's Alloy mention
takes the count 4 -> 3, so it was being counted by that bare word alone.

Measured rather than asserted, by building each row in memory and counting both
ways:

  scenario                       OLD   NEW
  today, as committed              4     4
  build bahamut (runs no Alloy)    5     4   old overcounts
  build smaug   (runs no Alloy)    5     4   old overcounts
  build titan   (runs no Alloy)    5     4   old overcounts
  build trinity (has an agent)     5     5   both rise, correct

The last line is the one that matters for not weakening the assertion: a host
that really does declare an agent still moves the count the moment its marker
comes off, which is exactly when hardware.md's sentence should be forced to
change. All seven negative rows hold at four; trinity and odin both rise to
five.

The regex knows two shapes, "runs no Alloy" and "No Alloy agent", and the
comment says plainly that a negation it does not know would overcount — loudly,
but it should be extended here rather than worked around by reaching for
different words in the table. smaug's row was reworded to dodge this on
2026-09-16 (c938c37); that reword is now belt and braces rather than the fix.

Refs #88

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Gerrrt
Gerrrt merged commit 8053854 into main Sep 16, 2026
3 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/alloy-count-negation branch September 16, 2026 18:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant