Repository navigation
feat(automations): Block an Event automation when its run is blocked - #2095
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![sentry-warden[bot]](https://avatars.githubusercontent.com/in/2755838?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 95400ec. Configure here.
This was referenced Oct 9, 2026
dcramer
marked this pull request as ready for review
October 9, 2026 17:20
Sentry Evalsjunior-behavioral
Base commit junior-guardian
Base commit junior-integration
Base commit junior-router
Base commit |
|
This PR is too large for Bugbot to review. It changes 30,922 lines and 3,955,497 characters. Split the change into smaller pull requests to get a review. |
dcramer
added a commit
that referenced
this pull request
Oct 9, 2026
…result (#2052) Scheduled automation and Event automation runs used the chat Turn contract. The runtime delivered the final assistant text, the model had to emit `[[NO_REPLY]]` to stay silent, and the interactive rules told an unattended run to ask questions. This is the root cause of most of #2014, #1741, and part of #554. Automation runs now have their own mode, and each run ends with one declared result. Only a declared message is posted. **What changes** - **One switch.** `buildDispatchRoutingContext` sets `dispatch.declaresResult` for an automation Source. The prompt, the tools, the result check, and Slack Delivery read that field. Watches and other plugin dispatches keep the chat Turn contract. - **Automation prompt.** `<automation-run>` rules replace the task, conversation, and Slack action rules. The stored instruction is the job. Nobody can answer a question during the run. The run checks every condition in the instruction before an action with side effects. - **Declared result.** `finishAutomationRun` is registered only for these runs. It must be the only tool call in its message, and the run stops after it. - `send_message`: the runtime posts the declared message to the stored outcomes. The tool does not offer this result when the automation has no message outcome. - `no_action`: the dispatch completes and nothing is posted. - `misconfigured`: the dispatch is recorded as blocked with the declared reason. - A message that is only `[[NO_REPLY]]` is saved as `no_action`, so instructions written before this change stay silent. - **Who receives the message.** The run context lists each stored outcome and says if it is the creator's direct message or a channel. - **Who is mentioned.** The task input line for the creator now says that "me" and "my" mean the creator, to mention them in the message, and not to mention them otherwise. - **A run without a result.** The model gets one reminder. A second stop without a result fails the dispatch. - **A result is final.** A resumed slice returns a saved result and does not call the model again. **Behavior changes to accept** - A failed run posts nothing to its outcomes. Before, the channel got a failure reply. The failure shows on the dispatch, in the execution history, and as the last run status on the dashboard. - The model can end a run as `misconfigured`. For a Scheduled automation, the heartbeat already blocks the automation when its dispatch is blocked. For an Event automation, only that dispatch is blocked, and the automation runs again on the next event. - A person who replies in the thread of a posted message continues the run's Conversation with a normal chat Turn. That Turn sees the history of the run. **Not in this PR** - A `blocked` status for Event automations (#2095), and a direct message that tells the creator about a block (#2096). These were in the first version of this PR. They do not depend on this PR. - Tools such as `sendFiles` still post to the automation's Destination, not to its outcomes. - Repeated run failures do not notify the creator. **Review focus** - `automation-result.ts` owns the result contract. `finishedRunReply` decides what a finished run posts, for first runs in `turn.ts` and resumed runs in `resume.ts`. - The reminder loop and the early return for a saved result are in `agent/index.ts`. Unit tests cover their rules. No end-to-end test covers the reminder, because a real model cannot be made to skip the tool in a reliable way. - The agent test fixture now takes a person's reply in the thread of an automation's first post. One eval uses it to check the follow-up reply. The Slack mock now keeps a top-level post as the root of its thread. - The creator line changes the same text as #2033. The new prompt and tool change the model requests of the automation evals, so CI records them again. The chat prompt and the chat tools do not change. Refs #2051 <!-- junior-request-attribution:start --> via **David Cramer**. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:slack%3AC0B595QDZLL%3A1791321334.446739 --> -- [View Junior Session](https://junior-prod.sentry.dev/conversations/slack%3AC0B595QDZLL%3A1791321334.446739) [[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AC0B595QDZLL%3A1791321334.446739/?project=4510944073809921) <!-- junior-session-footer:end --> --------- Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com> Co-authored-by: David Cramer <david@sentry.io> Co-authored-by: David Cramer <dcramer@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
A run that could not get the authorization or credentials that it needs recorded a blocked dispatch, but its Event automation stayed active and ran again on each event. A blocked dispatch now sets the Event automation to blocked with the reason. A blocked automation does not match new events until its creator resumes it. Store the reason in a new status_reason column. A pause keeps the reason, so a paused automation whose run blocked returns to blocked on resume. The dashboard, the Automations API, and the event automation tools show the status and the reason. updateEventAutomation now takes a status, so a person can pause or resume an Event automation from chat. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The integration evals recorded these model responses in https://github.com/getsentry/junior/actions/runs/37961677828. The evals of this commit replay them in strict mode. Eval-Suite: integration
The new status option of updateEventAutomation needed only write access to the Destination, so another person in the channel could pause or resume a blocked Event automation. The Automation settings allow this for the creator only. The tool now has the same rule. Clear the block reason when a deleted Event automation is created again. A reason from before the delete made a later pause and resume return the new automation to blocked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The behavioral evals recorded these model responses in https://github.com/getsentry/junior/actions/runs/37968690273. The evals of this commit replay them in strict mode. Eval-Suite: behavioral
The integration evals recorded these model responses in https://github.com/getsentry/junior/actions/runs/37968690351. The evals of this commit replay them in strict mode. Eval-Suite: integration
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dcramer
force-pushed
the
feat/event-automation-blocked
branch
from
October 9, 2026 18:32
82c6bdb to
4e362a6
Compare
The integration evals recorded these model responses in https://github.com/getsentry/junior/actions/runs/37973966960. The evals of this commit replay them in strict mode. Eval-Suite: integration
The behavioral evals recorded these model responses in https://github.com/getsentry/junior/actions/runs/37973966950. The evals of this commit replay them in strict mode. Eval-Suite: behavioral
dcramer
added a commit
that referenced
this pull request
Oct 10, 2026
#2096) A blocked Scheduled automation or Event automation stops in silence. The reason shows only on the dashboard and in the automation tools. The creator now gets one direct message with the reason and a link to resume the automation. **What changes** - `automations/blocked-notice.ts` sends the notice. It goes only after the automation is stored as blocked, so the dashboard matches the message: - Event automations: when `blockEventAutomation` stores a new block. A redelivered block sends nothing. - Scheduled automations: when the heartbeat stores the block (`advanceScheduledAutomationAfterRun` now returns whether it did). If the creator changed the schedule during the run, the automation stays active and no notice goes out. - The notice is best-effort. The automation is already blocked when it is sent, and a failed send logs `automation.blocked_notice.failed`. - A missing-authorization reason now names the fix. On creator credentials it reads "This run needs a connected `<provider>` account." A run on system credentials cannot use a connected account, so its reason says to switch to creator credentials and connect one. - The notice and the creator direct-message outcome now share `openSlackDirectMessage`. **The result guidance of `finishAutomationRun` changes too** For a job that needs a provider Junior does not have, an Event automation run ended as `no_action` in 7 of 8 samples. Its reason said that the result "could not be verified", and the `no_action` guidance named exactly that. The automation then never became blocked, and no notice went out. The guidance now says that `no_action` is for a run that worked with nothing to post, or for a temporary failure, and that a missing provider, tool, account, or access is `misconfigured`. The same case then ended as `misconfigured` in 8 of 8 samples. One integration eval covers the chain: the run ends as `misconfigured`, the only post is the blocked notice, and it is in the creator's direct message. This changes the tool description of every automation run, so CI records the automation evals again. **Review focus** - The notice uses the same Slack token resolution as other background posts. A binding of dispatch work to one workspace installation belongs in a separate change. - The two reason texts have no test of their own. The blocking and heartbeat tests cover when a notice is sent and when it is not. - The heartbeat also blocks a Scheduled automation before dispatch, when it cannot build the dispatch metadata or create the dispatch. That path sends the notice too. This is the last of three parts of the first version of #2052. The other two, #2052 and #2095, are merged. Refs #2051 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A run that cannot get the authorization or credentials that it needs records a blocked dispatch. Its Event automation stayed active and ran again on each matching event, with the same result. A blocked dispatch now sets the Event automation to
blockedwith the reason, and a blocked automation does not match new events until its creator resumes it. Scheduled automations already work this way.What changes
markDispatchBlockedblocks the Event automation of a dispatch before it marks the dispatch, so a retry after a failed write still blocks it.status_reasoncolumn (migration0049_event_automation_status_reason).blocked.blockedand the reason. A blocked automation counts as "needs attention".updateEventAutomationtakesstatus: "active" | "paused", so a person can pause or resume an Event automation from chat, asslackScheduleUpdateAutomationdoes for Scheduled automations.Review focus
persistBlockedDispatchTurn.updateEventAutomationis in the tool list of each chat turn, so CI records the chat eval model requests again.This is the second of three parts of the first version of #2052.
Refs #2051
🤖 Generated with Claude Code