Repository navigation
[INTEROP-04] Internal relay interop is repeatedly relied upon in CI as if it were external interop, with non-gating || true wrappers #149
Description
Finding. relay interop is RELAY's own cross-protocol / golden-vector CLI check — an internal
self-consistency tool, not third-party interoperability testing. Several ports (rust-CAN, rust-LIN,
rust-RCP reportedly among them) run it in CI wrapped in || true plus a brittle grep scrape of its
output, which silently tolerates both output-format drift and genuinely non-matching results, meaning
the step can report green while validating nothing.
Recommendation. Remove the || true wrapper and assert on the command's actual exit code instead
of grep-scraping its stdout; and, ecosystem-wide, label these CI steps "internal conformance" rather
than "interop" so a reader of the CI config doesn't overstate the coverage they provide. This overlaps
THEME-I (CI gates that don't gate, filed separately in this batch) for the || true half of the
finding, and THEME-K for the mislabeling half.
Filed from the 2026-07-29 ecosystem audit register; independently re-verified against current HEAD before filing.
Activity
SoundMatt commentedon Aug 21, 2026
rust-RCP's half of this finding fixed and merged: rust-RCP#171.
Replaced
output=$(relay interop ... 2>&1) || true+ a brittle grep of stdout with a plainrelay interop --strict --protocol RCP target/release/rust-rcpinvocation whose real exit code gates the step directly — matching go-RCP's existing clean pattern for the same check. Also renamed the job fromrelay-interop/"RELAY interop" torelay-cli-conformance/"RELAY CLI conformance" per the naming convention indocs/INTEROP-ARCHITECTURE.md— this step only checks self-consistency against RELAY's own embedded golden vector, not a third-party reference implementation.Worth noting: the first push actually broke CI outright —
relay-conformancecollided with an already-existing, differently-scoped job of the same ID in the same workflow file (a Rust unit-test job, "RELAY spec v2.0 unit conformance").yaml.safe_loadsilently accepted the duplicate key (YAML spec says last-one-wins) so my own pre-push check missed it; GitHub Actions' schema validation correctly rejected the file. Renamed torelay-cli-conformanceto avoid the collision, re-verified with a proper duplicate-key detector, and CI came back fully green (18/18 checks) on the corrected push.Both fixes independently re-verified post-merge from a fresh clone at
84a541f:cargo build/test/fmtclean, no duplicate job IDs, andrelay interop --strict --protocol RCPmanually run against a freshly-built binary confirmsRESULT: PASS/ exit 0.rust-LIN's identical
|| truepattern is still open — same fix needed there, not yet done.SoundMatt commentedon Aug 21, 2026
Both halves fixed and merged, per
docs/INTEROP-ARCHITECTURE.md(THEME-K, #125):|| true+ grep-scrape with real exit-code gating onrelay interop --strict --protocol RCP; also renamed the job (relay-interop→relay-cli-conformance) per the naming-discipline convention, since it only checks self-consistency against RELAY's own golden vector, not a third-party reference implementation.relay interop --strict --protocol LIN. No rename needed there; the surrounding job was already correctly named "conformance".Both fixes verified locally (build/test/fmt clean) and mutation-tested against a deliberately broken binary before push — confirmed
RESULT: FAIL/ exit 1, which the previous|| trueform would have silently swallowed. Both independently re-verified post-merge from fresh clones.One real bug found and fixed along the way (rust-RCP#171's first push): renaming the job to
relay-conformancecollided with an already-existing, differently-scoped job of the same ID in the same workflow file —yaml.safe_loadsilently accepted the duplicate key (YAML spec: last-one-wins) so my own pre-push check missed it, but GitHub Actions' schema validation correctly rejected the file outright. Renamed torelay-cli-conformanceinstead; rust-LIN's fix was checked for the same class of collision before pushing and had none (5 unique job IDs).