Skip to content

fix: stop the gate reporting a leak when no benchmark exists - #21

Merged
Freshair129 merged 1 commit into
mainfrom
fix/gate-leak-reason
Sep 27, 2026
Merged

Freshair129 merged 1 commit into
mainfrom
fix/gate-leak-reason

Conversation

@Freshair129

Copy link
Copy Markdown
Owner

Summary

When a batch has no worker receipt, the Stage 17 quality gate gave this security reason:

retrieval benchmark reported a cross-tenant leak.

That was wrong. The benchmark was missing, and no leak had been measured. The check was crossTenantLeaks !== 0, which is true when the value is undefined. This was noted in the corpus v2 report.

The reason text (packages/gks-core/src/pipeline.mjs) now distinguishes the two cases:

Receipt benchmark Security reason
missing cross-tenant isolation is unproven: the retrieval benchmark is missing.
crossTenantLeaks: N, where N > 0 retrieval benchmark reported N cross-tenant leak(s).
crossTenantLeaks: 0 none (unchanged)

What stays the same:

  • The security dimension is still a critical FAIL in both cases, so the gate still blocks publication when isolation cannot be proven.
  • verdict, allowPublication and critical are unchanged in every case. Only the reason string changes.
  • A receipt always carries a validated benchmark, so the missing-benchmark case only happens when there is no receipt at all.

Compatibility: a consumer that matches the old reason string exactly will no longer match. Nothing in this repository does.

Golden corpus

This PR is the first behaviour change since the C0.4 corpus became replayable (#20). Before re-baselining, npm run check:corpus failed in exactly one case, C0.4-GENESISRAG17-RECEIPTS, at the security reason. After re-baselining, the diff of expected/*.json is only this reason text, in two places: the gate response and the Stage 17 evidence row that stores the verdict. The registry hash for that case and the manifest's recordedFrom are updated to match.

Test plan

  • tests/contract/pipeline-genesisrag17.test.mjs:
    • the missing-receipt test now asserts that the security dimension is exactly { FAIL, critical, ["cross-tenant isolation is unproven: …"] };
    • a new test uses a benchmark with crossTenantLeaks: 2. It asserts the verdict is FAIL, the security reason names 2 leaks, and the retrieval dimension still flags it.
  • npm test: vitest 289 passed and 2 skipped (the MSP suites need MSP_REPO_ROOT); security 12/12.
  • npm run check:c0 gives PASS 24 / NOT_RUN 1. check:baseline holds, since pipeline.mjs is not a hashed input. check:corpus replays all 24 cases.
  • CI on this PR

🤖 Generated with Claude Code

With no worker receipt, the Stage 17 gate's security dimension said
"retrieval benchmark reported a cross-tenant leak". The benchmark was
missing, not leaking: `undefined !== 0` held. The reason now distinguishes
the two cases:

- missing benchmark: "cross-tenant isolation is unproven: the retrieval
  benchmark is missing."
- counted leaks: "retrieval benchmark reported N cross-tenant leak(s)."

Both remain a critical FAIL, so the gate stays fail-closed. Only the
reason text changes.

The C0.4 golden corpus caught the change in exactly one case,
GENESISRAG17-RECEIPTS. It is re-baselined; the transcript diff is this
reason text only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Freshair129
Freshair129 merged commit 96a28a0 into main Sep 27, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant