Skip to content

ci(mutation): cut the heavy files into range shards, per-file break threshold - #211

Merged
dlamarre-dev merged 5 commits into
mainfrom
ci/mutation-range-shards
Sep 27, 2026
Merged

dlamarre-dev merged 5 commits into
mainfrom
ci/mutation-range-shards

Conversation

@dlamarre-dev

@dlamarre-dev dlamarre-dev commented Sep 26, 2026 •

Copy link
Copy Markdown
Owner

Why

The nightly mutation run of 26 September was cancelled on the stego shard at the 120-minute job limit, at 158 of 159 mutants. It was already close to the limit before that:

night stego crypto
20 Sep (forced) 51 min, 358 mutants 63 min, 463 mutants
24 Sep (incremental) 116 min, 179 re-tested 66 min, 127 re-tested
26 Sep (incremental) cancelled at 120, 158/159 106 min, 148 re-tested

(The 25 September failure was a different problem, the --json progress throttle, and #206 already fixed it.)

The mutant counts barely moved. The tests got slower: the gallery round trips re-encode every cover (SPEC §9.8) and price every carrier with UERD (§9.3.1). On its own, src/api/node/gallery.test.ts takes about 6 minutes locally. A surviving mutant runs every test that covers it, and a timeout waits 1.5× that long, so the cost per mutant went up about fivefold. Tonight's Sunday run is forced and would need 4 to 5 hours for stego.ts (479 mutants now) or crypto.ts.

What

  • scripts/mutation-shards.ts: cuts files into line ranges (--mutate file:start-end). Each cut is named after a top-level declaration, not a line number. Stryker only mutates nodes that sit entirely inside the range, so a cut inside a function would silently drop the mutants that span it.
  • plan job: resolves the anchors for the current commit and feeds the matrix (fromJSON).
  • tests/mutation/shards.test.ts (runs in npm test on every PR): uses Stryker's own instrumenter to check that each file's ranges produce every mutant exactly once, and that the shards cover exactly the mutate list in stryker.config.mjs. I checked that it fails (476 ≠ 477) with a cut inside gateKek.
  • Summary: sums the rows per file and gets the expected shard count from the plan.
  • @stryker-mutator/instrumenter added as a devDependency, the same version as core (already installed as a transitive dependency; grouped by Dependabot with @stryker-mutator/*).

Shards: crypto ×4, stego ×5, vault ×2, reed-solomon ×2, codes (access, gf256, erasure) ×1 = 14. I placed the cuts using a cost model built from the stored reports. The test selection does not change, so the score still measures the same thing.

  • Per-file break threshold: each shard runs with STRYKER_NO_BREAK, and the summary job fails if a file, summed across its shards, falls below 75. Without this, crypto-4 (randomIntBelow, 17 mutants, 76.47%) would fail the night on a single extra survivor.
  • dryRunTimeoutMinutes 20 → 30 (dry runs measured at 9 to 15 min on the runner), and timeout-minutes 120 → 180.

Verification

  • tsc, eslint, prettier, notices:check, claims:check and the new test are all green locally. The test fails as expected with a cut inside gateKek.
  • First cold run with 11 shards (36264393968): 49 to 106 min per shard, with crypto-1 cancelled at 323/329 and stego-3 at 162/181 at 120 min. Timeouts set the floor: 9 to 21 per shard, about 15 to 20 min each. So I split those two ranges again (13 shards) and raised the limit to 180.
  • Second cold run, 13 shards (36271471541): ✅ all passed, 15 to 109 min, except stego-1 at 135 (103 on the previous run, same code). I split it again at StreamReader (6 of its 12 timeouts), which makes 14 shards.
    • That run also exposed a counting bug: Stryker carries every stored incremental result into its report, even outside --mutate, so reed-solomon.ts was counted twice through codes (main's old cache) and some crypto ranges overlapped. Fix: the summary keeps only the mutants wholly inside each shard's planned range (MUTATION_SHARDS) and reports how many it dropped, and the cache moves to stryker-v3. Replayed on that run's artifacts: 88.91%; crypto 477, reed-solomon 177, stego 479, vault 448 (+1 RuntimeError), matching the instrumenter.
  • Third cold run, 14 shards (36279181759): in progress.

🤖 Generated with Claude Code

dlamarre-dev and others added 3 commits September 26, 2026 14:57
The nightly of 26 September was cancelled on the stego shard at the
120-minute limit, 158 of 159 mutants in; on the 24th the same shard took
116 minutes, and crypto took 106 for 148 mutants on the 26th. The gallery
round trips got slow enough (SPEC 9.8 re-encode, then UERD costs) to raise
the cost of every mutant they cover about fivefold, so a forced Sunday run
of stego.ts (now 479 mutants) or crypto.ts would need four to five hours.

Stryker mutates a line range with `--mutate file:start-end`, but only the
nodes wholly inside it: a cut inside a function drops every mutant that
spans it without a word. So the cuts are named top-level declarations,
resolved per commit by scripts/mutation-shards.ts in a new `plan` job that
feeds the matrix, and tests/mutation/shards.test.ts asks Stryker's own
instrumenter on every PR whether the ranges still produce every mutant of
each file exactly once (verified to fail on a cut inside gateKek).

crypto, stego: 3 shards each; vault, reed-solomon: 2 each; access, gf256
and erasure stay together. The summary sums rows per file and takes the
expected shard count from the plan. The test selection is unchanged, so
the score measures the same thing it did.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The first range-sharded run put crypto-3 (randomIntBelow, 17 mutants) at
76.47%: one more survivor in a 17-mutant range would have failed the night
with nothing wrong in crypto.ts. Each shard now runs with STRYKER_NO_BREAK
and the summary job fails when any file, summed across its shards, is below
the threshold stryker.config.mjs sets. A local run still breaks in Stryker.

Also raises dryRunTimeoutMinutes from 20 to 30: that run measured crypto's
dry run at 14 minutes on the runner, and stego.ts takes 16 locally.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The first cold run with eleven shards (36264393968, on this branch) took
49 to 106 minutes per shard and lost crypto-1 at 323/329 and stego-3 at
162/181 to the 120-minute limit. Every shard has a floor the ranges cannot
lower: a 9-15 minute dry run, then 9 to 21 timeouts at 15-20 minutes each,
four at a time. Those two ranges are cut again (before encryptBytes and
extractBytesStegoJpeg), and the limit goes to 180 so the shards that land
near 105 keep some room for runner variance.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@dlamarre-dev dlamarre-dev changed the title ci(mutation): cut the heavy files into eleven range shards ci(mutation): cut the heavy files into range shards, per-file break threshold Sep 26, 2026
dlamarre-dev and others added 2 commits September 26, 2026 19:21
…treamReader

The thirteen-shard cold run passed (36271471541), but its summary counted
reed-solomon.ts twice (354 of 177) and crypto.ts as 624 of 477. Stryker's
incremental mode carries every stored result into the new report, in or out
of this run's --mutate: `codes` restored the old four-file `codes` entry
through the restore-keys prefix, and a crypto range restored the entry of a
range with other bounds. The summary now keeps only the mutants wholly
inside each shard's planned range (the plan is passed as MUTATION_SHARDS)
and says how many it left out; the cache generation goes to v3 so the old
layout is never restored. Replayed on that run's artifacts: 477, 177, 479
and 448 + 1 RuntimeError, matching the instrumenter.

stego-1 took 135 minutes against 103 the run before, and six of its twelve
timeouts sit in StreamReader's eighteen lines, so that class is a shard of
its own: fourteen shards.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The third branch mutation run failed stego-4's dry run on
`expected 'IMG_2026.jpg' not to match /PXL|2026/`. Names are drawn
uniformly from IMG_0000 to IMG_9999, so ten draws hit 2026 about once in
a thousand runs, and a failed dry-run test costs a mutation shard its whole
night. The exact-shape assertion beside it already excludes every part of
the cover name `PXL_20260921_143000`.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@dlamarre-dev
dlamarre-dev merged commit b1881eb into main Sep 27, 2026
17 checks passed
@dlamarre-dev
dlamarre-dev deleted the ci/mutation-range-shards branch September 27, 2026 01:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant