[NEW-SPEC-8] Require spec and README code/CLI examples to be executable in CI (doctest / round-trip harness) #146
Description
Problem. THEME-J documents dead/broken doc references across the ecosystem, and REL-SPEC-10
(filed separately in this batch) documents at least two ports whose documented CLI quickstart cannot
actually be followed as written. Nothing today prevents either the spec document or a port's README
from shipping a stale example, because no example is ever actually executed.
Proposal. Add a new §20.7, plus a requirement under §13's README guidance: every fenced code/CLI
example marked normative in the spec, and every implementation's README quickstart, MUST be
machine-extractable and executed in CI (doctest-style extraction, or a round-trip harness) — a failing
example MUST fail the build. This extends REL-SPEC-10 from "one documented demo must work" to "every
normative example must be continuously proven to work," and applies the same discipline to the spec
repo itself so the document can't ship a stale snippet (as its own §12.1/§12.2 examples currently do —
see NEW-SPEC-6).
Filed from the 2026-07-29 ecosystem audit register; independently re-verified against current HEAD before filing.
Activity
feat: continuous example verification (§17 Requirement 19) (#187)
SoundMatt commentedon Aug 21, 2026
Implemented and merged in #187 (v2.8 → v2.9, MINOR).
What shipped
<!-- doctest:KIND -->marker convention, applied to all 7 of this document's existing normative JSON examples (§12.1, §12.2 ×2, §12.3, §15.8, §17.2, §20.6).KINDselects the validator — the same onerelay conformapplies to a real binary's output, not a separate check invented for the doctest.## RELAY conformancesection must be machine-extracted and executed in CI. Network-fetching commands (go get/go install) and commands needing a published container image are explicitly exempt.extractDoctestExamples(cmd/relay/doctest.go),TestSpecDoctestExamplesValidate(cmd/relay/doctest_test.go). NewREQ-RELAY-102.Real bugs found and fixed while building this
"816b402d...") instead of a real 64-char SHA-256. Replaced with genuine values — two are the actual hashes of the referenced golden vectors.patternkeyword at all — silently ignored per its own documented keyword list. Every schema constraining a SHA-256 field with"^[0-9a-f]{64}$"was unenforced:relay conformwould have accepted a malformed hash in any capabilities/manifest/attestation/vectors-manifest document without a single finding. Found because the truncated placeholders above initially validated cleanly — a false pass this mechanism exists to prevent. Fixed with aregexp-basedpatterncheck; newTestSchemaPattern; mutation-tested.v2.0whilespec/version.jsonwas atv2.8) — the exact class of drift §19.5 already guards against, just not in README.md. Fixed, andTestReadmeVersionMatchesVersionJSONextends the §19.5 mechanism so it can't silently recur.Verification
Local, pre-merge:
go build/vet/testclean,gofmt -l .clean,golangci-lint run0 issues, coverage 90.4% (≥90% gate),gofusa check0 errors/PASS,gofusa trace -req-coverage 100100%/100%.Post-merge, independently re-verified from a fresh clone at
6aefd24: build/vet/gofmt/test clean (coverage 90.4% confirmed),gofusa check0 errors/PASS,gofusa trace100%/100% (102/102 requirements traced), and end-to-end —relay conform --manifestreports 19 total requirements with Requirement 19 correctlyNOT_OBSERVABLE/implementation's own CI, and README's version line confirmed matchingspec/version.json.CI green on the
mainmerge commit (DCO, Lint, Docker build, CodeQL, Security, Test, Self-conformance, go-FuSa full safety lifecycle — all pass).