[NEW-SPEC-3] Runtime capabilities-to-binary cross-check; tighten missing-protocol / adapt:false from WARN to FAIL #141

Activity

1. SoundMatt

   added

   sev:high

   spec-compliance

   on Jul 30, 2026
2. SoundMatt

   closed this as completedin #182on Aug 21, 2026
3. SoundMatt commented on Aug 21, 2026

   

   SoundMatt

   on Aug 21, 2026

   OwnerAuthor

   More actions

   Implemented and merged in #182 (squash commit 54416fa), as a scoped-down version of the original proposal — per explicit direction to design a real fix rather than implement the tightening naively.

   The conflict found before implementing

   The issue proposed turning relay conform's missing-protocol/adapt:false findings from WARN to FAIL unconditionally. Literally implementing that would have broken RELAY's own reference CLI's passing self-conformance CI job: cmd/relay is deliberately, legitimately multi-protocol (protocol: null) and non-adapting (adapt: false) — confirmed by CI's own workflow comment, which already explained this is a correct classification, not a defect. relay conform's black-box CLI had no way to distinguish that legitimate case from a genuine single-protocol implementation bug.

   The fix

   New optional capabilities field multi_protocol (spec §12.2), defaulting to false. A tool that self-declares multi_protocol: true legitimately reports a null protocol/protocol_int and adapt: false — §10.3 scopes the Adapt() contract to protocol packages, so a multi-protocol aggregator has no single protocol to declare or per-protocol adapter to export. With that carve-out, §17 Requirements 1 and 6 tighten cleanly from WARN to FAIL for everyone else, closing the real audit-flagged gap (THEME-B: capabilities silently drifting from the shipped binary, undetected by non-strict relay conform).

   Scope decision (partial close)

   The issue also proposed verifying every declared commands string is invocable, and "exercising" every declared features string with a CLI probe. Both explicitly declined in this PR: there's no existing, spec-grounded signal distinct from the generic "invalid arguments" exit code (§11.3) to detect an unrecognized command across four languages' implementations, and §12.2 already states features are compiled-in and explicitly not runtime-probed. Implementing either now would mean inventing an unproven convention rather than tightening an existing one. Deferred to a dedicated follow-up rather than force-fit into this PR.

   What was implemented

   • New optional multi_protocol capabilities field + §12.2 prose/example + schema.
   • §17 Requirements 1 and 6 text tightened; black-box narrative and verifier table rows upgraded "Partial" → "Full".
   • validateCapabilitiesDoc in cmd/relay/conform.go: gated FAIL logic for null protocol and adapt: false, exempted when multi_protocol: true.
   • cmd/relay's own capabilities output now declares "multi_protocol": true.
   • New REQ-RELAY-097.
   • Fixed two stale doc references found in passing (Quickstart §step 9 — "three CI gates" / "partially checks Requirements 1, 6, and 12" was stale after the prior two PRs' new gates and this PR's Req 1/6 promotion).
   • SpecVersion bumped 2.4 → 2.5 (MINOR).

   Verification (double-checked, before and after merge)

   • 5 new/rewritten tests covering both tightened FAILs and the multi_protocol exemption.
   • Real mutation test: reverted the FAIL tightening back to WARN, confirmed the new tests correctly failed, restored, confirmed green again.
   • Manually verified — both pre-merge and independently from a fresh post-merge clone of main at 54416fa — that relay conform ./relay (RELAY's own binary) exits 0 with only the pre-existing softer version-doc WARN, no new FAIL; and that relay conform --manifest reports Requirements 1 and 6 as PASS for RELAY's own binary.
   • go build/vet/test, gofmt, golangci-lint run (latest) — all clean, both pre- and post-merge.
   • gofusa check (pinned v0.48.0): 0 errors. gofusa trace -req-coverage 100: 100% (97/97).
   • All 8 CI checks green on PR feat: tighten §17 Requirements 1 and 6 from WARN to FAIL, add multi_protocol capabilities field #182, and the main CI run for merge commit 54416fa independently confirmed success (Docker build, Lint, Security, Test, Self-conformance, go-FuSa full safety lifecycle).

   Closing this PR's scope. Commands-invocability and feature-probing remain open design questions for a future issue if wanted.