Skip to content

feat: let an agent mention configured people when a change needs a human review - #90

Merged
brentvatne merged 9 commits into
mainfrom
tsapeta/agent-mentions
Oct 23, 2026
Merged

brentvatne merged 9 commits into
mainfrom
tsapeta/agent-mentions

Conversation

@tsapeta

@tsapeta tsapeta commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #89.

Why

The docs agent in expo/expo can tell when docs or doc comments change in a way that needs a human review, but it has no way to bring in the docs owner. The agent already runs only on its paths, so it should be able to ask for a person when it decides one is needed.

Change

  • A new agent front-matter key mention takes a list of GitHub logins or org/team slugs (an optional leading @ is dropped). It's the only source of whom a review notifies, and CI loads agent files from the base commit, so a PR can't change it.
  • An invalid value (not a list, empty, or an entry that isn't a login or team, checked with MENTION_TARGET_RE) is a setup warning and the agent mentions no one, the same policy as paths.
  • Reviewer passes can return humanReview: one sentence on why a person should review the change, parsed like summary. The engine keeps the first reason per agent.
  • buildMentions turns a reason into a mention only for an agent that ran and has a mention list. attachMentions attaches an engine-owned mentions field and strips any coordinator-written value.
  • The comment shows a visible line under the overall summary, like 📣 <logins>: the docs reviewer asks for your review. <reason>. The reason goes through plainModelLine, and the renderer re-checks every login because a re-render reads the state back from the comment body.
  • The bot edits its comment in place on re-reviews. GitHub's docs say that an edit which adds a mention also notifies that person, so a mention still notifies when it first appears on a later re-review.
  • The shared.md template documents humanReview. README, templates/config.jsonc, LLP 0002, LLP 0005 and LLP 0006 document mention.
  • After a release, expo/expo can add mention to its docs agent.

Test Plan

bun test (810 pass), typecheck, lint, fmt:check, llp:check and ecr ref-check all pass.

The new tests fail before the change and pass after. They cover:

  • config loading of mention logins and teams
  • warnings for invalid values
  • humanReview fail-soft parsing
  • buildMentions and the coordinator boundary
  • the visible line and filtering of an invalid login
  • the state round trip and the aggregate comment

I also ran ecr review locally with a scratch docs agent (mention set to octocat, and a prompt asking for humanReview on substantive docs changes) on this branch's README changes. It returned a mention for octocat with the model's reason. I haven't run it against a live PR, so the GitHub notification itself is untested.

@tsapeta
tsapeta added this pull request to stack #91 October 22, 2026 13:41
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

🤖 AI code review

Decision: Ready for human review

Overall PR risk: Medium. The change modifies comment rendering and adds a mentioned state field for agents with mention declarations. Model text with @ or HTML references may render differently, invalid or missing state could cause repeat notifications, and prompt-injected humanReview might trigger unexpected mentions. Blast radius is limited to comment rendering with straightforward rollback.

🔎 Reviewer summaries (3)
  • consistency: Checked the new mention front-matter key, the humanReview output field, and the mention rendering against sibling patterns. They follow the paths, agentSummaries, and template conventions. No consistency findings.
  • correctness: I checked the new mention/humanReview flow: config parse, engine attach, state round-trip, and every render path. Model text cannot choose whom to notify. Every re-render passes the prior mention state. I found no defects.
  • security: I reviewed the new mention feature for security. Only the base-commit agent config supplies logins. The renderer checks each login again and breaks every @​ in model text. I found no exploit path.

🔗 Review setup:

  • This PR changes code the reviewer prompts cite (src, src/tests, src/config/load.ts, src/config/schema.ts, src/core/render.ts, and 2 more). Check that the guidance quoting it is still correct.

No findings.


This review is advisory — it never blocks a merge and never auto-approves.

@tsapeta
tsapeta force-pushed the tsapeta/agent-mentions branch from 220aadf to 585a605 Compare October 22, 2026 13:47
Each reviewer pass may return a short `summary` of what it reviewed and concluded. The
engine keeps one row per roster agent (its summary, or why it did not run: `paths` or
the router) in an engine-owned `agentSummaries` field, and the comment lists them in a
collapsed section under the overall summary. Summaries are model text, so the reporter
flattens them to one line, escapes HTML, and breaks `@` mentions.
…man review

An agent's front matter can list GitHub logins or teams under `mention`. When the agent
sets `humanReview` with a reason, the review comment shows a visible line that mentions
those people. Only the trusted config names people; the model writes only the reason,
and the reporter breaks every `@` in it. An invalid `mention` is a setup warning.
@tsapeta
tsapeta force-pushed the tsapeta/agent-mentions branch from 585a605 to 397311c Compare October 22, 2026 14:10
@tsapeta
tsapeta marked this pull request as ready for review October 22, 2026 14:13
@tsapeta
tsapeta requested a review from brentvatne October 22, 2026 14:13
brentvatne and others added 4 commits October 22, 2026 14:30
…hen the comment is too big

- plainModelLine let `_@login_` (rendered as `<em>@login</em>`) and
  character references (`&#64;login`, `&commat;login`, `@&#111;ctocat`)
  through, and GitHub decodes references before it looks for mentions.
  Escape every reference's `&`, and break each `@` that doesn't follow an
  ASCII letter or digit, which keeps email addresses whole.
- Only the aggregate comment checked MAX_COMMENT_CHARS. Escaping and the
  base64 state copy make summaries several times their byte budget, and a
  single comment over GitHub's limit failed to post, so the finished review
  was replaced by the failure notice. Over the limit, leave the summaries out
  with the same note the aggregate uses.
- Summary parts lived in a plain object, so an agent named `constructor` or
  `toString` crashed the whole run; they are now a Map. Parts are kept in
  diff order rather than pass-completion order, and a `cross-cutting` agent
  is listed once.
- Clipping no longer leaves half an emoji, and an unknown "not run" reason
  from an old or newer engine's state reads "not run".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Escaping every reference made model text like `&lt;T&gt;` or `&amp;` show
up literally. Only a reference that decodes to `@` itself can start a
mention: one after the `@` spells a login character, but that `@` is broken
like any other.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…lded away

- Titles, rationales, suggestions, source titles, the summary and the
  requalification reason went through stripStateMarkers only, so model text
  could still notify anyone (a title citing `@param` or `@types/node` pings a
  user or team). They now go through the same mention breaking as the
  summaries, in the single, aggregate and inline bodies.
- In the aggregate comment, a scope with no findings kept its fold closed, so
  the person a request notified landed on a collapsed section. A mention line
  now opens it.
- humanReview is recomputed every run, and GitHub notifies when an edit adds
  a mention, so a request that dropped out and came back notified again. The
  comment state now remembers who was notified; a returning handle renders
  without notifying.
- MENTION_TARGET_RE now follows GitHub's rules: single inner hyphens, at most
  39 characters, an optional EMU `_shortcode` suffix, and team slugs. One
  invalid entry no longer drops the agent's whole `mention` list: it is left
  out with a warning naming it.
- Document that team mentions posted with GITHUB_TOKEN are untested.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Resolved so the one filter, breakMentions, carries #89's rules (`@`
character references escaped, every `@` not after a letter or digit
broken) for both the summary rows and the multi-line model text #90 added.
humanReviewReasons and buildMentions take a Map, like the summary parts, so
an agent named `constructor` can't read an inherited property.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@brentvatne

Copy link
Copy Markdown
Member

Review notes from a multi-model review (Codex, Grok and Claude, each finding then re-checked against the code). Fixes are in 7b2eab6, pushed to this branch, and merge 514dada, which also merges #89's fixes so the mention filter stays one function.

1. Model text could still mention anyone (holds). The summary, and each finding's title, rationale, suggestion, source titles and requalification reason, went through stripStateMarkers only, in the single, aggregate and inline bodies. So "only the trusted config names people" didn't hold: a title citing @param or @types/node pings a user or a team.

  • Fix: all of them go through modelText (stripStateMarkers + breakMentions), with feat: add per-agent reviewer summaries to the review comment #89's rules: escape @ character references, and break each @ that doesn't follow a letter or digit. Lines and formatting are kept.
  • Trade-off: @ inside code in model text gets an invisible zero-width space too (@MainActor, @expo/...), so copying it from the comment carries the character. A backtick doesn't prove a code span (an HTML attribute can hold one), so a code-aware skip would reopen the bypass.

2. In the aggregate comment, a request was hidden when its scope had no findings (holds). The fold only opened for findings, requalified findings or replies. Fix: a mention line opens it.

3. Repeat notifications (holds). GitHub notifies when an edit adds a mention, and humanReview is recomputed every run (and steerable by the PR content), so a request that dropped out and came back notified again.

  • Fix: the comment state's new mentioned: { live, notified }. A handle stays live while it stays; one that was notified, left and came back renders as @​login; a new handle still notifies.
  • Design call for you: this means a genuinely new request for the same person later on the PR won't ping them. The alternative is to let every return notify. clear() re-creating a comment starts over, which notifies again. That's rare.

4. MENTION_TARGET_RE didn't match GitHub's rules (holds). It rejected EMU logins like name_shortcode, accepted -lead, trail- and dou--ble, and one bad entry made the loader drop the agent's whole mention list.

  • Fix: logins are letters, digits and single inner hyphens, at most 39 characters, with an optional _shortcode, and team slugs follow GitHub's rules. A bad entry is left out with a warning naming it, and the rest still load.

5. Team mentions posted with GITHUB_TOKEN (unverified). GitHub notifies only people with read access, and for an org repo, org members. github-actions[bot] isn't one, so @expo/docs may not notify. Now documented as untested; it needs a live check before expo/expo relies on it.

Also: humanReviewReasons and buildMentions take a Map, the same constructor exposure #89 fixed for summaries. The /dismiss and reply credits (by @login) still mention the person who acted; that predates this PR, and the LLP no longer says "only".

Checks: typecheck, bun test (825 pass), lint, format and llp:check pass. Each new test fails with its fix reverted.

brentvatne and others added 2 commits October 22, 2026 15:19
The capture in `(^|[^A-Za-z0-9])@` consumed the first `@` of `@@login`, so
the second never matched and `@​@login` still notified. A lookbehind
breaks each one. Found by the repo's own AI review.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@brentvatne

Copy link
Copy Markdown
Member

Follow-up: 6a476c4 merges #89's @@login fix (ce95adc) into breakMentions, so the multi-line model text gets it too. The multi-line test now covers @@run-user.

Base automatically changed from tsapeta/agent-summaries to main October 23, 2026 01:33
Resolves the conflicts with #86 and #87: both import lists, the inline heading (knownSeverity + modelText), buildMentions/attachMentions beside the new policy doc, both sets of config-load tests, and a duplicate renderInlineCommentBody import.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@brentvatne
brentvatne merged commit 124c2bf into main Oct 23, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants