[ENHANCEMENT] Post surviving mutants as a PR comment #2287

Problem (one or two sentences)

The mutation test job finds surviving mutants, but only as warning annotations. The CodeRabbit review does not see them, so reviewers can miss weak tests.

Context (who is affected and when)

The mutation-diff job in .github/workflows/mutation-testing.yml passes when mutants survive. It writes survivors to the job summary and to warning annotations. We start CodeRabbit with a label after CI passes. On PR #2243 the job found a survivor at src/core/task/Task.ts:610. The CodeRabbit review did not mention it. Contributors and maintainers see the same gap on every PR that changes src/, packages/ or webview-ui/src/.

Desired behavior (conceptual, not technical)

Each PR gets one comment that lists the surviving mutants. The comment updates on each new CI run. CodeRabbit can read the comment when the label starts the review.

Constraints / preferences (optional)

• The job stays advisory. Survivors must not fail the job.
• Fork PRs must work. The pull_request token is read-only for forks, so the job that posts the comment needs a separate trusted workflow.
• The trusted workflow must treat the report as untrusted data. It must not run code from the PR.
• Give only the comment job pull-requests: write.

Request checklist

• I searched the existing Issues and Discussions for duplicates
• This describes a specific problem with clear context and impact

Acceptance criteria (optional)

Given a PR with at least one surviving mutant
When the mutation-diff job completes
Then the PR has one comment that contains a hidden marker, the list of survivors with file:line, and the resolve guidance from the job summary
And a later CI run on the same PR updates that comment and does not add a second one

Given a PR with no surviving mutants
When the mutation-diff job completes
Then an existing comment changes to say that no mutants survived
And the workflow does not add a comment if none exists

Given a PR from a fork
When the mutation-diff job completes
Then the comment still appears
And no step runs code from the PR with a write token

But the mutation job must not fail because of survivors
And the comment must not contain content that comes from PR text without escaping

Proposed approach (optional)

1. In scripts/stryker-diff.mjs, write the text from formatSummary to a file under reports/mutation/. The job already uploads that folder.
2. Add a workflow_run workflow that starts when Changed-code mutation testing completes. Give the comment job pull-requests: write and actions: read.
3. The comment job downloads the artifact. It checks the file size and the marker. It finds the PR number from the workflow_run event data, not from the artifact.
4. Use the same list, create and update pattern as .github/workflows/label-pr-review-state.yml (listComments, createComment, updateComment). Use a new marker, for example <!-- zoo-code-mutation-report -->.
5. Do not use pull_request_target with a checkout of the PR code.

Trade-offs / risks (optional)

• We did not confirm that CodeRabbit reads an edited comment. We also did not confirm that it reads a comment from github-actions. Test both on a PR with a known survivor. If it ignores edits, delete the old comment and post a new one on each run (recreate in sticky-comment actions).
• Another option is to make survivors fail the job. That changes an advisory gate into a blocking gate. A maintainer must decide that.
• A third option is to raise timeout_ms for github-checks in .coderabbit.yaml. We start CodeRabbit after CI passes, so this does not help.
• The workflow_run event always uses the workflow file on the default branch. Changes to the new workflow cannot be tested on a PR before merge.

Related

• PR fix(anthropic): keep tool_result blocks contiguous by grouping image blocks separately #2243 (survivor at Task.ts:610, fixed in a later commit)
• CodeRabbit docs: https://docs.coderabbit.ai/tools/github-checks
• GitHub fork workflow rules: https://github.blog/2020-08-03-github-actions-improvements-for-fork-and-pull-request-workflows