Repository navigation
test-parity: audit known_failures.json — every entry needs an issue # and date #797
Description
Activity
- added a commit that references this issue
on Jul 29, 2026 Concrete instance of the provenance problem this issue tracks, found during #7265's CI triage.
zlib_4917_levelis documented intest-parity/known_failures.json(#6847) — "ext-zlib provider archive with a feature-stripped stdlib rebuild… on a COLD object cache (warm /tmp caches mask it)… Not a parity regression" — but it is not ingap_snapshot.json, which is the file the conformance-smoke shard gate actually reads.So the shard reported it as
pass -> compile_fail, flagged "REGRESSION — expected to pass", and cost an agent real time chasing a failure that was already understood and written down. Warm caches on other PRs mask it, which is why it appears intermittent and PR-specific.Two files describe the same thing and only one is consulted by the gate. That is the failure mode: a known failure recorded in the file humans read, absent from the file the machine reads, so the documentation cannot prevent the rediscovery it exists to prevent.
Worth considering as part of this issue's scope:
- Make one file authoritative, or generate the gate's input from the documented one so they cannot drift.
- If they must stay separate, add a check that every
known_failures.jsonentry has a correspondinggap_snapshot.jsonentry (and flag orphans in both directions).
Related observation from the same run: several
node_fail -> parity_failentries are the oracle reclassifying tests it could not run when the snapshot was taken — CLAUDE.md's silent-drop hazard firing in reverse. One such entry differed between two PRs' runs on the same base, which is the signature of an environment-dependent oracle rather than a code change.
Walk
test-parity/known_failures.json. Each entry must reference an open issue and the date it was skip-listed. Close stale entries; file issues for orphans.Part of #793.