Repository navigation
ci(mutation): cut the heavy files into range shards, per-file break threshold - #211
Merged
Merged
Conversation
The nightly of 26 September was cancelled on the stego shard at the 120-minute limit, 158 of 159 mutants in; on the 24th the same shard took 116 minutes, and crypto took 106 for 148 mutants on the 26th. The gallery round trips got slow enough (SPEC 9.8 re-encode, then UERD costs) to raise the cost of every mutant they cover about fivefold, so a forced Sunday run of stego.ts (now 479 mutants) or crypto.ts would need four to five hours. Stryker mutates a line range with `--mutate file:start-end`, but only the nodes wholly inside it: a cut inside a function drops every mutant that spans it without a word. So the cuts are named top-level declarations, resolved per commit by scripts/mutation-shards.ts in a new `plan` job that feeds the matrix, and tests/mutation/shards.test.ts asks Stryker's own instrumenter on every PR whether the ranges still produce every mutant of each file exactly once (verified to fail on a cut inside gateKek). crypto, stego: 3 shards each; vault, reed-solomon: 2 each; access, gf256 and erasure stay together. The summary sums rows per file and takes the expected shard count from the plan. The test selection is unchanged, so the score measures the same thing it did. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The first range-sharded run put crypto-3 (randomIntBelow, 17 mutants) at 76.47%: one more survivor in a 17-mutant range would have failed the night with nothing wrong in crypto.ts. Each shard now runs with STRYKER_NO_BREAK and the summary job fails when any file, summed across its shards, is below the threshold stryker.config.mjs sets. A local run still breaks in Stryker. Also raises dryRunTimeoutMinutes from 20 to 30: that run measured crypto's dry run at 14 minutes on the runner, and stego.ts takes 16 locally. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The first cold run with eleven shards (36264393968, on this branch) took 49 to 106 minutes per shard and lost crypto-1 at 323/329 and stego-3 at 162/181 to the 120-minute limit. Every shard has a floor the ranges cannot lower: a 9-15 minute dry run, then 9 to 21 timeouts at 15-20 minutes each, four at a time. Those two ranges are cut again (before encryptBytes and extractBytesStegoJpeg), and the limit goes to 180 so the shards that land near 105 keep some room for runner variance. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…treamReader The thirteen-shard cold run passed (36271471541), but its summary counted reed-solomon.ts twice (354 of 177) and crypto.ts as 624 of 477. Stryker's incremental mode carries every stored result into the new report, in or out of this run's --mutate: `codes` restored the old four-file `codes` entry through the restore-keys prefix, and a crypto range restored the entry of a range with other bounds. The summary now keeps only the mutants wholly inside each shard's planned range (the plan is passed as MUTATION_SHARDS) and says how many it left out; the cache generation goes to v3 so the old layout is never restored. Replayed on that run's artifacts: 477, 177, 479 and 448 + 1 RuntimeError, matching the instrumenter. stego-1 took 135 minutes against 103 the run before, and six of its twelve timeouts sit in StreamReader's eighteen lines, so that class is a shard of its own: fourteen shards. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The third branch mutation run failed stego-4's dry run on `expected 'IMG_2026.jpg' not to match /PXL|2026/`. Names are drawn uniformly from IMG_0000 to IMG_9999, so ten draws hit 2026 about once in a thousand runs, and a failed dry-run test costs a mutation shard its whole night. The exact-shape assertion beside it already excludes every part of the cover name `PXL_20260921_143000`. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The nightly mutation run of 26 September was cancelled on the
stegoshard at the 120-minute job limit, at 158 of 159 mutants. It was already close to the limit before that:(The 25 September failure was a different problem, the
--jsonprogress throttle, and #206 already fixed it.)The mutant counts barely moved. The tests got slower: the gallery round trips re-encode every cover (SPEC §9.8) and price every carrier with UERD (§9.3.1). On its own,
src/api/node/gallery.test.tstakes about 6 minutes locally. A surviving mutant runs every test that covers it, and a timeout waits 1.5× that long, so the cost per mutant went up about fivefold. Tonight's Sunday run is forced and would need 4 to 5 hours forstego.ts(479 mutants now) orcrypto.ts.What
scripts/mutation-shards.ts: cuts files into line ranges (--mutate file:start-end). Each cut is named after a top-level declaration, not a line number. Stryker only mutates nodes that sit entirely inside the range, so a cut inside a function would silently drop the mutants that span it.planjob: resolves the anchors for the current commit and feeds the matrix (fromJSON).tests/mutation/shards.test.ts(runs innpm teston every PR): uses Stryker's own instrumenter to check that each file's ranges produce every mutant exactly once, and that the shards cover exactly themutatelist instryker.config.mjs. I checked that it fails (476 ≠ 477) with a cut insidegateKek.@stryker-mutator/instrumenteradded as a devDependency, the same version as core (already installed as a transitive dependency; grouped by Dependabot with@stryker-mutator/*).Shards: crypto ×4, stego ×5, vault ×2, reed-solomon ×2, codes (access, gf256, erasure) ×1 = 14. I placed the cuts using a cost model built from the stored reports. The test selection does not change, so the score still measures the same thing.
STRYKER_NO_BREAK, and the summary job fails if a file, summed across its shards, falls below 75. Without this, crypto-4 (randomIntBelow, 17 mutants, 76.47%) would fail the night on a single extra survivor.dryRunTimeoutMinutes20 → 30 (dry runs measured at 9 to 15 min on the runner), andtimeout-minutes120 → 180.Verification
tsc,eslint,prettier,notices:check,claims:checkand the new test are all green locally. The test fails as expected with a cut insidegateKek.StreamReader(6 of its 12 timeouts), which makes 14 shards.--mutate, soreed-solomon.tswas counted twice throughcodes(main's old cache) and some crypto ranges overlapped. Fix: the summary keeps only the mutants wholly inside each shard's planned range (MUTATION_SHARDS) and reports how many it dropped, and the cache moves tostryker-v3. Replayed on that run's artifacts: 88.91%; crypto 477, reed-solomon 177, stego 479, vault 448 (+1 RuntimeError), matching the instrumenter.🤖 Generated with Claude Code