Repository navigation
[NEW-SPEC-7] Require a persisted, CI-committed cross-implementation interop matrix (not just a one-shot pass/fail) #145
Description
Problem. §20.1.3's existing behavioural-conformance gate (referenced from §20.1 item 3 and §20.3)
checks that a port is EQUIVALENT to the reference implementation over golden vectors, but not against
its other language siblings, and the result is not persisted anywhere diffable. This is the mechanism
gap behind THEME-B/C/D and behind INTEROP-01 through INTEROP-05 (filed separately in this batch):
without a persisted matrix, a cross-language equivalence regression has no artifact trail.
Proposal. Extend §20.1's behavioural-conformance gate and the interop command: beyond checking
EQUIVALENT-vs-reference, each port's CI MUST regenerate and commit a relay interop --format json
matrix comparing it against every other published sibling implementation of the same protocol (not
only the reference) over the full pinned vector set. A pull request that regresses any previously-
EQUIVALENT cell MUST FAIL. This makes cross-language equivalence a persisted, diffable artifact instead
of a one-shot CI check whose result vanishes after the run.
Note: the audit's own reference to "§20.1.3" does not correspond to an actual numbered subsection at
current HEAD (§20.1 is a flat numbered-list, not further subdivided into 20.1.1/20.1.2/20.1.3) — noting
this citation mismatch rather than silently keeping it; the underlying behavioural-conformance gate it
refers to is real and is item 3 of §20.1's CI-gates list.
Filed from the 2026-07-29 ecosystem audit register; independently re-verified against current HEAD before filing.
Activity
SoundMatt commentedon Aug 21, 2026
Implemented and merged in #186 (v2.7.1 → v2.8, MINOR).
What shipped
relay interop's JSON output is now a versioned artifact:kind: "relay-interop-matrix",matrix_version: "relay-interop-matrix/1", and an explicitparticipants[]list — so the set of compared implementations is itself part of the diffable artifact, not just the per-cell verdicts.--baseline FILEflag: compares a fresh run against a previously-committed matrix and reportsregressions[]— one entry per (vector, participant) cell that wasEQUIVALENTin the baseline but isn't now. A cell that was neverEQUIVALENTto begin with is not reported as a regression, matching the issue's precise wording.--baseline-check it on every change, failing the build on any regression.spec/schemas/relay-interop-matrix.json, newREQ-RELAY-101.Deliberately not pairwise (N²)
The issue's literal wording ("compare against every other sibling implementation, not only the reference") would mean diffing every binary's
convertoutput against every other binary's directly. Not built: byte-equality is transitive, so "A equivalent to the shared reference" and "B equivalent to the shared reference" already jointly establish "A equivalent to B" — the existing O(N) hub-spoke comparison carries the same information an O(N²) pairwise matrix would, without the redundantconvertinvocations. Documented explicitly in §17 and §20.1 rather than silently narrowed.Bug found along the way
§20.4 cited
§20.1.2for a specific CI-gate item, but §20.1 is a flat numbered list, not subdivided — the same class of citation drift the issue itself flags for the audit's stale§20.1.3. Corrected to "§20.1 item 2".Verification
Local, pre-merge:
go build/vet/testclean,gofmt -l .clean,golangci-lint run0 issues,gofusa check0 errors/PASS,gofusa trace -req-coverage 100100%/100%.CI caught a real gap on first push: the 90% coverage gate failed (89.9%) — 3 genuinely-uncovered branches from the new code (malformed
--baselineJSON, and the markdown/text regression-rendering blocks). Fixed with 3 targeted tests, coverage back to 90.3%, pushed as a follow-up commit, CI green on re-run.Post-merge, independently re-verified from a fresh clone at
ba434c9: build/vet/gofmt/test clean (90.3% coverage confirmed),gofusa check0 errors/PASS,gofusa trace100%/100% (101/101 requirements), and end-to-end — built the binary, ranrelay interop --format jsonagainst itself, fed the output back as--baseline(clean round-trip, no false regressions), andrelay conform --manifestreports 18 total requirements with Requirement 18 correctlyNOT_OBSERVABLE/implementation's own CI.CI green on the
mainmerge commit (DCO, Lint, Docker build, CodeQL, Security, Test, Self-conformance, go-FuSa full safety lifecycle — all pass).