fix(mutation-audit): write the in-flight record outside the scanned tree, and measure the floor before attributing anything - #46
Conversation
…ree, and measure the floor before attributing anything The audit's crash-recovery record lived in the repo root as `.mutation-audit-inflight.json`, carrying the FULL pristine text of the file being mutated. For `src/no-cloud.ts` that text is the denylist itself, and this suite scans this repo with its own scanner — which does not consult `.gitignore`. So the tool that validates every guard here was planting evidence against the repo while it measured it. MEASURED on a pristine fc84401 worktree with NO mutation applied: bun test tests/conformance.test.ts tests/cli.test.ts 96 pass / 0 fail rc=0 same, with the record copied in 92 pass / 4 fail rc=1 The baseline is taken before the record exists, so every mutation ran against a +4 false-failure floor the baseline never saw, and a mutation nothing genuinely catches still read `caught` on those four alone. Demonstrated end to end: a comment-only insertion in `src/no-cloud.ts` — a semantic no-op — was scored `caught 790/4` by this script, failing set exactly those four, while the same edit hand-applied to a clean tree gave 794 pass / 7 skip / 0 fail rc=0. TWO CHANGES, and the second is the load-bearing one. 1. The record moves to the system temp dir, keyed on the absolute repo root so recovery finds it on the next run in this tree and two worktrees never share one. Location, not an ignore list: an ignore list has to be repeated in every scanner and the scanner that forgets is always the next one. `assertOutsideScannedTree` refuses to run if the record would land inside the tree, so a `TMPDIR` pointing into the repo — or an edit moving the record back "so you can see it" — stops the run instead of restoring the floor. 2. A FLOOR PROBE runs before any mutation: the record in place, nothing mutated, the suite must read exactly the pristine baseline. One probe per distinct file the run will mutate, because the payload IS that file's text. On any mismatch the run ABORTS rather than attributing failures to a mutation, and a mismatch is never retried — retrying and taking the second answer is how an intermittent floor becomes a green audit. This is what makes the next contaminating artifact, whatever it is, impossible to mistake for a kill. Recovery is preserved and strengthened: the record is staged and renamed rather than written in place, since recovery must parse it before it can restore, and a crash mid-write left unrecoverable truncated JSON. The pre-relocation repo-root path is still recovered from once and deleted, so a run killed by the older script does not strand a mutated tracked file. No mutation entry is added or removed: `--anchors` is 89/89 rc=0 before and after.
Re-derived on
|
fc84401 |
7c06192 |
|
|---|---|---|
| baseline | 794 pass / 7 skip / 0 fail | 796 pass / 7 skip / 0 fail |
floor probe src/no-cloud.ts |
clean 794/0 | clean 796/0 |
floor probe src/source-text.ts |
clean 794/0 | clean 796/0 |
attrib |
29/29 caught rc=0 | 29/29 caught rc=0 |
| floor-caught mutations | 0 | 0 |
| pinned by exactly one test | 6 | the same 6 |
--anchors |
89/89 rc=0 | 89/89 rc=0 — #45 staled no anchor |
| no-op mutation | SURVIVED 794/0 rc=1 |
SURVIVED 796/0 rc=1 |
typecheck rc=0, repo-conformance rc=0, CI verify completed/success (polled at
repos/hasna/contracts/commits/714954b/check-runs, not gh pr checks).
#45 widened four failing sets, and none of the six thin margins
Its two new tests appear in exactly four mutations' failing sets:
| mutation | width | new test(s) |
|---|---|---|
M72-attrib-recognised-by-content |
37 → 39 | CALLER pin + DENYLIST ARRAY pin |
M73-attrib-is-per-occurrence-not-per-file |
7 → 9 | CALLER pin + DENYLIST ARRAY pin |
M91-attrib-blanks-only-the-compared-record-leaves |
18 → 19 | CALLER pin |
M88-attrib-record-key-count-is-bounded |
2 → 3 | CALLER pin |
The six one-test mutations (M94, M95, M79b, M79e, M83b, M85) are unchanged
at width 1 on both bases. #45 did not thicken any thin margin; it thickened four that
were already thick.
FINDING: the retired M93 is no longer a no-op on 7c06192
M93 ("an array match blanks its elements, never the enclosing array's span") was
retired from the list with a recorded measurement: applied together with M94 so
the blankConstantSpans throw could not do the catching, the whole suite passed
except M94's own test. Its earlier "caught" was parasitic.
Re-measured by the same method on 7c06192 — both edits hand-applied, needle counts
asserted changed and replacements asserted present:
794 pass / 7 skip / 2 fail rc=1
✗ source-text: … > blankConstantSpans REFUSES a span whose bytes are not the constant it claims
✗ source-text: … > withoutInlinedDeclarations blanks only the compared values in the DENYLIST ARRAY too
The second is #45's new array pin. Attribution is measured, not assumed: M94 alone
fails exactly one test (795/1, blankConstantSpans REFUSES …) and never the array
test, so the array failure belongs to M93.
So M93's retirement rationale — "no attacker-reachable consequence, no independent
code to remove" — is outdated as of 7c06192. Re-added today it would read caught,
on a genuine independent test rather than on the throw. Deliberately not re-added
here: the file's own rule is "do not re-add without a to: that actually loosens
something", and that judgement belongs with the #45 mutation entry that is now safe
to add — both for whoever owns d0410198.
On the type-system reconciliation (todos 6c52da74)
Checked, because a verdict that rests on tsc rather than on a test would be exactly
the kind of soft catch this PR exists to expose. Zero of the 29 is a type-system
kill: no mutation's failing set consists only of build/pack tests. Six mutations
include a build/pack test, but each also fails behavioural tests, and all six
one-test mutations are pinned by source-text: behavioural tests. The un-export of
blankSpans therefore does not account for any caught verdict in this set — consistent
with the finding that the un-export raises the cost of the edit without closing the hole.
Worth stating for the audit's own limits: bun test transpiles without typechecking, so
the only route by which a type-level defence registers here is the bun run build
inside tests/published-package-security.test.ts. A rule defended solely by the type
system would read as a survivor, not as caught.
Marker-parse soundness, since FORCE_COLOR=3 is set
The audit spawns the suite with FORCE_COLOR=0/NO_COLOR=1 and parses ^\(fail\).
Verified rather than assumed: across both 29-mutation runs, the reported fail count
equals the number of named failing tests for every mutation — 0 discrepancies in 58
readings. No "N fail" was ever reported with zero names. (Outside the audit, in this
FORCE_COLOR=3 shell, bun emits ✗ and zero (fail) — which is why the audit forces
its own colour env and strips ANSI anyway.)
Housekeeping
The stray pnpm-lock.yaml that regenerates in contracts worktrees during full-suite
runs never appeared in this one and is in no commit on this branch (git log --all -- pnpm-lock.yaml empty). Note for the record: because it would be present during the
baseline as well as the probe, the floor probe would not flag it — it is not a floor
in the "+N over baseline" sense, and it is already covered by the pre-existing refusal
to audit against a red baseline.
package.json remains at 0.8.3, unbumped and unpublished (npm still shows 0.8.2).
refs/heads/main was not pushed to. PRs #41/#42/#44 untouched.
1 participant
The defect
scripts/mutation-audit.tswrote its crash-recovery record to.mutation-audit-inflight.jsonin the repo root, and the record carries thefull pristine text of the file being mutated. For
src/no-cloud.tsthat text isthe denylist itself. This suite scans this repo with its own scanner, and the scanner
does not consult
.gitignore— so the tool that validates every guard here wasplanting evidence against the repo while it measured it.
Measured on a pristine
fc84401worktree with NO mutation applied (recordexistence and size asserted before scanning: 59745 bytes):
bun test tests/conformance.test.ts tests/cli.test.tsThe four are always the same:
repo conformance kit > open-contracts passes conformance against itselfrepo conformance kit > manifest, surface, hosting, safety, bins, and no_cloud_guard checks runcontracts CLI > runs repo conformance with stable capability check idscontracts CLI > scans this package without treating scanner declarations as runtime edgesThe baseline is taken before the record exists, so every mutation then ran
against a +4 false-failure floor the baseline never saw — and a mutation that
nothing genuinely catches still read
caughton those four alone.The manufactured kill, demonstrated rather than argued
A temporary mutation whose edit is a comment-only insertion in
src/no-cloud.ts— a semantic no-op:What changed
1. The record moves out of the scanned tree — to
$TMPDIR/hasna-mutation-audit/<worktree>-<sha256(repoRoot)[0:16]>.inflight.json,keyed on the absolute repo root so recovery finds it on the next run in this tree and
two worktrees never share one (the pid guard arbitrates concurrent runs against the
same tree; it cannot arbitrate two trees, where one run's recovery would restore the
other's file mid-mutation).
Location, not a per-scanner ignore list: an ignore list has to be repeated in
repo conformance, the no-cloud gate, the artifact scan, the hostname guard, and
whatever is added next — and the scanner that forgets is always the next one.
assertOutsideScannedTreerefuses to run (rc=2) if the record would land inside thetree, so a
TMPDIRpointing into the repo, or an edit that moves the record back "soyou can see it", stops the run instead of quietly restoring the floor. Verified:
Discoverability, which the repo-root location bought, is replaced by printing the
record's absolute path at the top of every run.
2. A floor probe, which is the load-bearing change. Before any mutation, the
record is written with its real pristine payload, nothing is mutated, the suite
runs, and the run aborts (rc=2) unless it reads exactly the pristine baseline.
One probe per distinct file the run will mutate, because the payload IS that file's
text — a probe over
src/no-cloud.tssays nothing about what a record holdingsrc/source-text.tswould do. A mismatch is never retried; only a suite thatgave no reading at all (0/0) is, matching the existing precedent. Retrying a mismatch
and taking the second answer is how an intermittent floor becomes a green audit.
Relocation fixes the known instance. The probe is what makes the next contaminating
artifact — a leftover build output, a temp file a scanner does reach, an env var the
harness sets — impossible to mistake for a kill.
3. The resume path is preserved and strengthened. The record is staged and
renamed rather than written in place: recovery must parse it before it can restore,
so a crash mid-write previously left truncated JSON and an unrecoverable mutation.
The pre-relocation repo-root path is still recovered from once and then deleted, so a
run killed by the older script does not strand a mutated tracked file. Verified end
to end by planting a legacy record naming a payload that is not the file's text:
Re-derived figures
attribis still 29/29 caught, rc=0, and NO mutation was floor-caught — 0 of the29 has a failing set that is a subset of the four. Both floor probes read
clean 794/0. The cited figure needed no correction, and the merged PRs that cite itstand.
--anchorsis 89/89 present, rc=0 — unchanged, before and after. No mutationentry was added or removed by this PR.
What the floor was actually hiding is the margin, not the count. Six of the 29
are pinned by exactly one test each:
M94-attrib-blanking-rechecks-the-span-against-the-textblankConstantSpans REFUSES a span whose bytes are not the constant it claimsM95-attrib-compares-bytes-not-the-decoded-valueA LITERAL THAT NEEDS AN ESCAPE TO SPELL THE CONSTANT IS REFUSEDM79b-attrib-call-argument-is-not-a-declarationinlineDataRegions refuses a collection consumed on the spot or passed to a callM79e-attrib-a-colon-must-introduce-a-record-valuean argument is not a stored constant WHEREVER it sits in the listM83b-attrib-an-unreadable-text-is-attributed-nothinga text the lexer cannot read yields no regions at allM85-attrib-a-constant-is-only-string-literalsinlineDataRegions reads arrays, records and nesting, and refuses anything elsePre-fix each of those read
789/5. A reader would take that for five independenttests binding the rule; it is one, and deleting that one test turns the mutation into
a survivor. That is what the +4 concealed, and it is now legible from the audit's own
output without re-running anything.
The reason the count survived: the floor was additive, not substitutive. Seven of
the 29 genuinely fail at least one of the four floor tests on their own merits, and
the other twenty-two carried the four as pure padding on top of a genuine failing
set.
Not re-derived here, and stated as a gap: the full 89-mutation run
(
bun scripts/mutation-audit.tswith no filter) was not re-measured — onlyattrib(29) and
--anchors(89). The two cited figures are covered; a whole-list caughtfigure is not, and the same floor applied to it.
Baselines on this branch
The tree is byte-clean after the full 29-mutation run — the restore path is intact,
and no record is left in the repo root or in the temp dir.
Scope notes
package.jsonstays at 0.8.3, unbumped and unpublished. Nothing is published.main(fc84401) is untouched; this is PR-first and not self-merged..gitignoreentry is replaced with a comment recording where the recordwent and why ignoring it was never sufficient.
d0410198: a mutation entry pinning the call-site rule wasdeliberately omitted from test(no-cloud): pin byte-confined blanking at the CALL SITE, not just the helpers #45 on the grounds that "a mutation whose green reading
survives deleting its own test is a booby trap." With the floor gone and the probe
in place, that entry is now safe to add. It is deliberately not added here.
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.