[quality] mutation.yml: nightly reuses 100% of the incremental file, so the report has been frozen since 2026-10-28 — pass --force on the nightly step

Finding

Since the three-shard nightly first seeded its incremental files (run 40636419300, 2026-10-28), every nightly Mutation (advisory) run has reused 100 % of the previous night's results and has stopped measuring anything. The report it posts is frozen at the 2026-10-28 state of the suite.

Evidence — nightly run 41275775616 at de2f950 (which contains the survivor-pinning tests merged in #428 and #429):

INFO ProjectReader   Found 1 of 71 file(s) to be mutated using incremental report with 795 mutant(s), and 1 test(s).
INFO IncrementalDiffer Incremental report:
        Mutants:        0 files changed (+0 -0)
        Result:         795 of 795 mutant result(s) are reused.

(cli shard job 124757539855; the rest shard reused 803 of 809, re-running only the 6 mutants whose source lines moved in #427.)

The shard still reports src/cli.ts 121 survived / 84.78 % and src/watcher.ts 68 survived / 80.12 % — the identical figures from 2026-10-28 — even though #428 and #429 added tests that kill ~24 of those exact mutants. Per-shard wall time confirms nothing runs:

nightly run	cli	rest	learner
40636419300 (2026-10-28, seeded)	144 min	175 min	88 min
40821869628 (10-29)	0	0	0
41006932561 (10-30)	0	4	0
41171362594 (10-31)	0	0	0
41275775616 (11-01)	0	1	0

Why. stryker.config.json uses testRunner: "command" with coverageAnalysis: "off". In that mode Stryker sees exactly one pseudo-test ("All tests") and has no test-file list, so its IncrementalDiffer only invalidates mutants whose source text changed. Changes to src/*.test.ts are invisible to it: a newly added test that kills a survivor never flips that mutant's cached Survived result. Pull-request runs do not catch this either, by design (#400): Select files to mutate filters .test.ts out, so a test-only PR skips mutation entirely (#429's PR run: all three shards "No changed src files in shard …; skipping").

Reproduced locally on the smallest shard member (src/types.ts, 6 mutants) at de2f950:

npx stryker run --mutate src/types.ts --incrementalFile /tmp/inc.json → 4 killed, 2 survived (types.ts:58 and :59 StringLiteral — same as the nightly).
Add a test asserting CATEGORY_LABELS values (test(index): pin CATEGORY_LABELS display strings #430). Re-run the same command → 0 files changed (+0 -0), 6 of 6 mutant result(s) are reused, still 2 survived.
Same command with --force → 0 of 6 mutant result(s) are reused, 6 killed, 0 survived.

Net effect: the advisory gate accepted in #331 reports a stale number; test work that improves the mutation score (#428, #429, #430) is invisible, and a regression that lets a previously-killed mutant survive would be invisible too (the cached Killed is never re-checked). The incremental file is still valuable for PR runs (that is what #347/#400 built it for) — only the nightly must not consume it.

Recommendation

Make the nightly re-run every mutant of its shard and rewrite the incremental file, so PR runs restore a fresh baseline. One token in .github/workflows/mutation.yml.

Replace lines 88–94:

      - name: Stryker (nightly, whole shard, 5 h 30 min budget)
        if: github.event_name != 'pull_request'
        timeout-minutes: 330
        continue-on-error: true
        env:
          FILES: ${{ steps.files.outputs.files }}
        run: npx stryker run --mutate "$FILES" --incrementalFile "$INCREMENTAL_FILE"

with:

      - name: Stryker (nightly, whole shard, 5 h 30 min budget)
        if: github.event_name != 'pull_request'
        timeout-minutes: 330
        continue-on-error: true
        env:
          FILES: ${{ steps.files.outputs.files }}
        # --force: the command test runner cannot see test-file changes, so
        # without it the nightly reuses last night's results verbatim and a new
        # test that kills a survivor never changes the report. The nightly is
        # the seed PR runs restore; it must always be a full measurement.
        run: npx stryker run --mutate "$FILES" --incrementalFile "$INCREMENTAL_FILE" --force

and in the header comment (lines 3–5) append after "runs restore (#347, #400)": — the nightly passes --force so that seed is remeasured every night.

--force is Stryker's documented switch for exactly this ("Run all mutants, even if --incremental is provided and an incremental file exists. Can be used to force a rebuild of the incremental file."), and it still writes the incremental file, so PR-run behaviour is unchanged. Budget: the only real measurement so far (run 40636419300) took 144 / 175 / 88 min per shard, well inside the 330-min step budget that #400 sized for a cold run.

Decision taken by hive: put --force on the nightly command line rather than "force": true in stryker.config.json, because the config value would also apply to pull-request runs, which must keep reusing results to fit their 20-minute budget (#346, #400).

Who can land this

The change is confined to .github/workflows/mutation.yml. This agent's GitHub App token is minted at the contributor tier without the workflows permission, so any push touching .github/workflows/** is rejected server-side — no PR from this lane can contain it. It needs a human maintainer or an ISSUES_PRS_MERGE agent to apply the replacement text above. The test that proved the mechanism is in #430 and needs nothing from this issue.

nightly Stryker step passes --force
next nightly IncrementalDiffer line reads 0 of N mutant result(s) are reused and the cli/watcher survivor counts drop below 121/68

Priority

Impact: medium (the gate still "passes" but measures nothing; every mutation-score datum since 2026-10-28 is stale)
Effort: low (one flag)

Filed by quality agent (ACMM L4/L6 — full mode)

🐝 Hive Agent: quality | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88

needs-human: the fix is confined to .github/workflows/mutation.yml (append --force to the nightly npx stryker run line at L94); the hive App token has no workflows permission, so a maintainer must apply it.

🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

What to reply

/hive approve: go with the recommendation
/hive decision <your instructions>: go ahead, following your instructions
/hive help: show these again

If what this issue waited on has been done (a merged fix, or an action taken outside this repository), a maintainer can reply /fixed to close it as completed; Hive clears its park labels.

Only a maintainer with write access can un-park; everything else is ignored. Full command reference: https://github.com/hivecommons/hive/blob/v5/src/docs/maintainer-commands.md.

/hive decision Approve the minimal nightly-only --force fix and matching explanatory comment. Keep PR incremental behavior, advisory status, thresholds and existing time budgets unchanged; do not set force globally in Stryker config. The nightly must actually rerun mutants and write the refreshed incremental baseline; verify the next completed nightly reports zero reused mutant results before calling the issue fixed. A workflow-capable maintainer must carry the normal reviewed PR if this lane cannot push workflows; do not request broader App permissions or bypass the capability restriction. The policy decision is resolved now; any remaining blocker is delivery capability or the next real nightly result, not a need for further direction. Owner-delegated decision from the November 2 request.

@clubanderson

Un-parked by @clubanderson. Decision: Approve the minimal nightly-only --force fix and matching explanatory comment. Keep PR incremental behavior, advisory status, thresholds and existing time budgets unchanged; do not set force globally in Stryker config. The nightly must actually rerun mutants and write the refreshed incremental baseline; verify the next completed nightly reports zero reused mutant results before calling the issue fixed. A workflow-capable maintainer must carry the normal reviewed PR if this lane cannot push workflows; do not request broader App permissions or bypass the capability restriction. The policy decision is resolved now; any remaining blocker is delivery capability or the next real nightly result, not a need for further direction. Owner-delegated decision from the November 2 request.

stage/045-needs-decision is cleared when present and approved-direction is applied; any hold label is left exactly as it was. needs-human cleared: your command is the human input it waited for, so the issue returns to the agent queue; add hold to keep agents off it.

Recorded from the hive dashboard.

Direction is resolved and now recorded by Hive: #431 (comment). The remaining block is delivery capability, not another policy decision: a workflow-capable maintainer must carry the normal reviewed nightly-only --force patch because the Hive App cannot push workflow changes. Keep this issue open through the next real nightly, which must report zero reused mutant results. No broader App permissions, bypass, or claim of completion is authorized. Applying blocked preserves this external dependency without re-asking the approved direction; remove it only when the authorized delivery path is available.

Folded finding. This was filed as a separate issue, but 1 open agent-filed issue (plus 35 closed as completed in the last 14 days) already cite src/cli.ts, so it was folded into the oldest of them instead of opening another. Treat this issue as the tracker for that component: prefer one structural fix that covers every folded variant over a PR per variant.

[telemetry] --json-log emits no event for clean checks or check duration

Finding

rationguard has no metrics/tracing stack and no configured telemetry backend, so this is a recommendation only (no exporter proposed). The only observability surface is --json-log (src/cli.ts:179), which emits events through onEvent.

Gaps in that stream:

src/checker.ts:109-112 emits only per-match events. A clean check (no match) emits nothing, so a consumer cannot compute a match rate or tell "ran clean" from "did not run".
No duration or input-size signal per check, so regex cost regressions are invisible.
The sighting_record_failed event (src/learner.ts:473) is bounded, which is good; keep that pattern.

Proposal

Add one bounded check_complete event to the existing RationguardEvent union: {event:'check_complete', matches:<int>, duration_ms:<int>, input_bytes:<int>}. All fields are numeric, so there are no unbounded labels. Never include matched text or the file path. Additive, so existing event names stay unchanged. A test should assert it is emitted for both clean and matching input.

Not proposed: any exporter, endpoint or off-box data flow.

— hive: agent=telemetry backend=copilot model=claude-sonnet-5.5 copilot=1.0.88

Navigation Menu

Repository navigation

[quality] mutation.yml: nightly reuses 100% of the incremental file, so the report has been frozen since 2026-10-28 — pass --force on the nightly step #431

Finding

Recommendation

Who can land this

Priority

[telemetry] --json-log emits no event for clean checks or check duration

Finding

Proposal

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[quality] mutation.yml: nightly reuses 100% of the incremental file, so the report has been frozen since 2026-10-28 — pass --force on the nightly step #431

Description

Finding

Recommendation

Who can land this

Priority

Activity

hivecommons-hive commented on Nov 1, 2026

hivecommons-hive commented on Nov 1, 2026

clubanderson commented on Nov 2, 2026

hivecommons-hive commented on Nov 2, 2026

clubanderson commented on Nov 2, 2026

hivecommons-hive commented on Nov 3, 2026

[telemetry] --json-log emits no event for clean checks or check duration

Finding

Proposal

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions