Repository navigation
Wrap up passes that run out of time instead of discarding them - #4
Merged
Merged
Conversation
A Goose pass still working 90 seconds before its timeout is stopped and its session resumed for one turn that asks for the findings it has so far. A pass that ends without a JSON answer gets the same one extra turn. Before, a panelist that hit its timeout lost all of its work, and a coordinator reply without JSON failed the whole pull request. Passes now record a named session in their private Goose home so they can be resumed. The ReviewBench judge reserve drops from 2.5 to 1.5 minutes, which gives the coordinator the extra minute.
A panelist's final JSON is around 20,000 characters and took 85 seconds to write at medium or low effort alike, so a 90-second wrap-up often ran out.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes reviews finish inside ReviewBench's 15-minute limit without losing a lens to a slow pass.
Problem. A panelist that hit its 7-minute timeout was killed, and all of its work was thrown away. In a 25-pull-request run, 8 reviews lost a panelist this way. Separately, a coordinator reply with no JSON object failed the pull request, and a single pull request that never succeeds fails a whole ReviewBench run.
Change.
GooseRunnernow holds back the last two minutes of each pass (GooseConfig.WrapUp):To make this possible, passes record a named session in their private Goose home instead of using
--no-session. WithSharedHome, sessions go to the caller's home as before. A pass whose timeout is shorter than twiceWrapUpgets no wrap-up.The window is two minutes because a panelist's final answer runs about 20,000 characters and took around 85 seconds to write at both medium and low effort. A 90-second window ran out on 7 of 11 wrap-ups under load.
The ReviewBench adapter's judge reserve drops from 2.5 to 1.5 minutes, since the judge never needed more than a minute. The coordinator gets that minute instead. In the worst case the stages still add up to 14 minutes: 7 for the panel, 5.5 for the coordinator, and 1.5 for the judge.
Verification
Runagainst a fakegoosefor a pass that outlives its soft deadline and for a pass that answers without JSON. It checks that the wrap-up resumes the same session with the right instruction and finishes within the timeout. The test fails when the wrap-up is disabled.try-agent.shnatively, with real Goose 1.43 and Databricks models, on the 8 test-set pull requests that lost a panelist in the earlier evaluation. All 8 ran at once.