Skip to content

feat: add a label when the review is ready for a human - #97

Merged
brentvatne merged 4 commits into
mainfrom
tsapeta/ready-for-human-label
Oct 27, 2026
Merged

brentvatne merged 4 commits into
mainfrom
tsapeta/ready-for-human-label

Conversation

@tsapeta

@tsapeta tsapeta commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Why

Maintainers want to see at a glance which PRs the AI review has cleared for a human, without opening each review comment. A label is filterable in PR lists and searches.

Change

  • A new option, review.readyForHumanLabel (off by default), names the label. The config check rejects a label that is also, in any case, review.label, review.skipLabel or a <label>:<agent> trigger.
  • After the review comment is posted, syncReadyLabel adds the label when the review covers the whole PR and the worst decision is "Ready for human review", with or without comments. A review doesn't cover the whole PR when a pass didn't finish, no agent's paths matched the changed files, a changed file matched no scope, or the run was partial: an explicit agent subset (--agents, /review <agents>), a --scopes run, or a review saved with --agents and posted by ecr post-review. A noise-only review stays ready.
  • A review that isn't ready, a /skip-review that ecr ci acts on, or a run that deletes the review comment removes the label. A partial run can remove it but never adds it.
  • The label's own event history decides what's ours: the label is removed only when the newest event for it is our account's labeled, and it isn't added again when the newest event is another account's unlabeled. No new state is stored in the comment. "Our account" is the token's login, so with a personal token that person's own removals count as ours, under the default GITHUB_TOKEN other workflows' label changes do too, and local and CI runs can't remove each other's labels. When the history is longer than the 30-page cap, the label is left alone.
  • Label names match case-insensitively, as GitHub treats them, and the removal uses the PR's spelling.
  • ecr ci, ecr post-review and ecr review --pr --post pass the option, and the deferred-review config fingerprint includes it. Per-scope comment mode has no single decision for the PR, so it sets no label and logs that.
  • /dismiss keeps the stored decision, so it doesn't change the label. Label failures only log, since the comment is already posted.
  • The README holds the contract, including these identity limits, that a label added before a push stays until the next review posts, and that it must never be a merge gate, since the decision comes from model output on untrusted PR content. templates/config.jsonc, the schema comment and LLP 0008 point to it.

Test Plan

bun test (900 pass), typecheck, lint, fmt:check and llp:check pass. src/__tests__/ready-label.test.ts covers the readiness rule (partial coverage, no reviewer applied, noise-only) and the config check (including case variants), and drives the reporter through a fake gh:

  • a ready review adds the label, and an existing label is left alone
  • a review that requests changes removes the bot's label, and a failed review keeps a person's label
  • a label a person removed isn't added again, and one the bot removed is; a removal in another case is matched, and a label spelled differently from the config is removed with the PR's spelling
  • the newest event wins across two pages of events, and a history at the page cap changes nothing
  • with no known own login, the label is never removed
  • clear() and postSkipNote() remove the bot's label
  • a partial run adds nothing and removes the bot's label; unmatched files mean not ready
  • no label calls when the option is off, and a label API failure doesn't fail the report
  • an aggregate where one scope requests changes removes the label

src/__tests__/deferred-review.test.ts covers the artifact's partialReview field and that an artifact without it still parses.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

🤖 AI code review

Decision: Ready for human review

Overall PR risk: Medium. This change modifies existing report functions to add label syncing after comment operations. Label ownership depends on issue event history; incorrect login resolution could leave stale labels or skip removals. Previously saved review artifacts fail the fingerprint check when the new option is configured. The feature is off by default, label failures only log, and removing the config key disables the feature.

🔎 Reviewer summaries (3)
  • consistency: I checked the new review.readyForHumanLabel option, the reporter label sync, and the CLI wiring against sibling patterns. The option, the logs, the paging, and the tests follow the repo conventions. I found no consistency problems.
  • correctness: I reviewed the ready-for-human label in the GitHub reporter, the ci, review and post-review commands, the config schema and the deferred artifact. I found no correctness defect. Partial runs, break-glass and clear paths only remove the label, and label errors only log.
  • security: I reviewed the new ready-for-human label path for security issues. The label sync runs gh through execFile with argument arrays, URL-encodes the DELETE path, and sends no token to logs. I found no exploitable issue.

🔗 Review setup:

  • This PR changes code the reviewer prompts cite (src/tests, src/config/schema.ts, templates/config.jsonc). Check that the guidance quoting it is still correct.

No findings.


This review is advisory — it never blocks a merge and never auto-approves.

@tsapeta
tsapeta marked this pull request as ready for review October 26, 2026 18:38
@tsapeta
tsapeta requested a review from brentvatne October 26, 2026 18:38
…al coverage

- GitHub label names are unique regardless of case. A config spelled differently
  from the repo's label never saw it as present, so it was never removed, and a
  person's removal was not matched. The presence check, the event match and the
  config check now ignore case, and DELETE uses the PR's spelling.
- The aggregate was labelled ready with changed files no scope matched, or on a
  --scopes run with no result for the other scopes. reportAggregate now takes
  coverageComplete from ci.ts and checks unmatched files.
- README, LLP 0008 and the template describe what a GitHub App token actually
  does, and that /skip-review removes the label only under ecr ci.
- An explicit agent subset (`--agents`, `/review <agents>`), a `--scopes` run,
  and a review saved with `--agents` and posted by `ecr post-review` never add
  the label; they can still remove the bot's own. This replaces the
  coverageComplete check, which only a --scopes run could fail.
- A review where no agent's paths matched approves without a coverage note; it
  is now not ready. A noise-only review stays ready.
- When the event history reaches the page cap, the label is left alone, since
  the newest event may not have been read.
- The README holds the contract, including the login caveats, instead of "a
  person's add or removal always wins"; the schema and template point to it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants