[NEW-SPEC-1] Machine-checkable conformance manifest (conformance.json) to turn self-reported conformance into an enforced artifact #139

Activity

1. SoundMatt

   added

   sev:high

   spec-compliance

   on Jul 30, 2026
2. SoundMatt

   closed this as completedin #180on Aug 21, 2026
3. SoundMatt commented on Aug 21, 2026

   

   SoundMatt

   on Aug 21, 2026

   OwnerAuthor

   More actions

   Implemented and merged in #180 (squash commit 44357a4).

   What was found

   Investigated the issue's original proposal against current spec state before implementing:

   • §17 had already grown to 14 numbered requirements (not "1-12" as the issue text speculated), so the new manifest entry became Requirement 15.
   • relay conform's own CLI flags were already informally deferred design space per existing spec text — §11.1 did not need editing.

   What was implemented

   • New §17 Requirement 15 — Conformance manifest. Implementations MUST commit a relay-conform/1 manifest (§17.2), generated by relay conform --manifest; CI MUST regenerate it and fail on drift or any FAIL entry.
   • New §17.2 defines the manifest JSON shape and a 4-value status vocabulary (PASS/FAIL/SHAPE_ONLY/NOT_OBSERVABLE), deliberately distinct from relay conform's existing PASS/WARN/FAIL finding severities. Core rule: a manifest MUST NOT report PASS for a requirement relay conform cannot actually observe.
   • New optional commit field on the §12.1 version document (backward-compatible) so a manifest can populate git_sha.
   • New §20.1 CI gate for manifest regeneration/diffing.
   • New embedded schema spec/schemas/relay-conform-manifest.json.
   • Reference implementation: --manifest flag + buildManifest() in cmd/relay/conform.go. Only Requirements 1, 6, 7, 12 get real PASS/FAIL/SHAPE_ONLY statuses (mirroring what relay conform can actually observe black-box). Everything else — including Requirement 15's own manifest-generation entry — is honestly NOT_OBSERVABLE.
   • New REQ-RELAY-095 in .fusa-reqs.json — minted fresh rather than reusing REQ-RELAY-052, since --manifest is materially new behavior.
   • SpecVersion bumped 2.2 → 2.3 (MINOR), matching the Requirement 13/14 precedent.

   Scope notes

   • Fixed a stray "§17.13" → "§17.2" citation left over from drafting.
   • 7 new tests cover manifest schema validity, capabilities_sha256 correctness (independently cross-checked against sha256sum), NOT_OBSERVABLE defaults, FAIL propagation, and CLI exit codes.

   Verification (double-checked, both before and after merge)

   • go build ./... && go vet ./... && go test ./... — clean, both pre-merge and from a fresh post-merge clone of main at 44357a4.
   • gofmt -l . — clean (caught and fixed one alignment issue that failed CI's Lint check on the first push).
   • gofusa check (pinned v0.48.0, matching CI): 0 errors.
   • gofusa trace -req-coverage 100: 100% (95/95 requirements traced).
   • Manually ran relay conform --manifest against a freshly built binary from the post-merge clone and asserted: valid JSON, spec_version: "2.3", all 15 requirement entries present, overall: "PASS".
   • All 7 CI checks green on PR feat: add relay conform --manifest (relay-conform/1), spec §17.2, Requirement 15 #180, and the main CI run for merge commit 44357a4 independently confirmed green (Docker build, Test, Security, Lint, go-FuSa full safety lifecycle, Self-conformance all success).

   Closing.