Repository navigation
Don't stop on look-alike pages; catch answers that admit they missed - #48
Merged
Merged
Conversation
…issed
Found by running each scenario 3 times: mail_relocation failed every run in
~4s. The trace's new decision records showed why: Jev opened the "Relocation
packages" newsletter (0.42 vs searching 0.35), chose "done" while only 45%
sure the task was complete, and the answer ("this is a newsletter... the email
you mean doesn't appear on this page") passed the check.
- Jev choosing "done" with goal_done < 0.6 gets the LLM advisor's second look
(done, or the next step); with no consults left it finishes as before.
- The answer check asks a third question, admits_missing: does the answer say
the main thing wasn't found or is elsewhere? The score is min(claims,
answers, 1 - admits_missing).
- Eval: +2 cases (that real miss; a good answer with a minor caveat). Before
26/30, after 30/30; admits_missing <= 0.06 on good answers, 0.92 on the miss.
Realistic suite x2 in Edge: 22/22, median 10.4s (mail_relocation 10.5s, 11.0s).
Everyday suite x3 in Edge before this: 37/39, median 7.2s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Running every scenario 3 times in the user's Edge (to measure run-to-run variance) turned up a regression.
mail_relocationfailed all 3 runs in about 4s.The per-decision trace records added in #45 show why:
What changed
goal_done) is under 0.6, the LLM advisor takes a second look. It either confirms done or picks the next step. With no second looks left, Jev finishes as before, and the answer check still judges the result.admits_missing): "Does the answer itself say the main thing wasn't found or is elsewhere?" The score is nowmin(claims_supported, answers_task, 1 − admits_missing).Results
admits_missingstays ≤ 0.06 on good answers and hits 0.92 on the miss.mail_relocation10.5s and 11.0s).npm test: 77/77.npm run extension:smoke: 11/11.npm run buildpasses.🤖 Generated with Claude Code