Repository navigation
fix: invalidate resumed tasks when experience prefix changes - #21
rudycelekli wants to merge 1 commit into
Conversation
- With feedback on, record each mode's input library fingerprint (the same check and key as google-research#21) so a lesson changed by a note re-runs the later tasks of its domain. - Mark the namespace "feedback": true in config.json the first time the flag is used, refuse a later run there without it, record feedback_dir in the sweep record, and warn in harness2-report. - Put the exception for the notes inside the answer-free and evidence-boundary rules instead of overriding them in a later block. Drop the wrong sentence about shuffled ids (they are stable across reruns). A lesson still has to rest on the edit rows. - Read the whole feedback folder before the first rollout: non-UTF-8 files and grader-output markers stop the run, unmatched files are listed. Match file names exactly, also on case-insensitive file systems. - Share one constant for the layout line, fix the module docstring, and rewrite the docs section accordingly.
|
#36 records library_fingerprint for each mode in _run_one_mode (computed before the mode runs, from the same mode_lib.context_dir()) and checks it in _resume, re-running with "the library it read has changed" when it differs. That's the same check this PR adds as input_library_fingerprint. To confirm, I ran this PR's tests/test_library_resume.py unchanged against #36's branch (d338a59) and both tests pass:
The one difference I found: for a task-only run with no library, this PR skips the check. #36 compares against fingerprint(None) ("base"), so a summary written before the upgrade re-runs once. The two PRs also conflict in schedule.py, so it probably makes sense to keep one of them. If #36 is the one kept, I'd be glad to bring your regression test into it with credit, since it covers the recovered-selector case more directly than mine does. Happy to go the other way if you or the maintainers prefer. |
|
@rudycelekli update: restating what our PRs solve here and adding some details The problem both PRs fix
How both fix it
Each PR passes the other's library test
1. Behaviour difference: runs with no experience library
2. Code conflict:
Proposal
EDIT: adding visual just in case |
…oogle-research#21) GitHub's Ubuntu runners have no zsh, so the completion test failed there on /bin/zsh. It now finds bash and zsh on PATH and only checks the generated script's text for a shell that is missing. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
When an early task has a transient selector failure, the stream still runs later tasks with no experience from that task. A later retry can recover the early selection and add its experience, but downstream summaries currently skip without checking which input library produced them. The resumed stream therefore combines recovered prefix evidence with results adapted against an older prefix.
Record each mode’s input experience-library fingerprint and compare it before resuming its selection. A changed prefix reruns the dependent task; an unchanged prefix still resumes. Existing learning summaries without this provenance conservatively rerun once; task-only runs without a library retain legacy resume behavior.
Validation uses actual offline DemoAdapter/DemoProposer execution: a transient first-task selector failure allows the second task to finish with no library; recovery must rerun the second task and supply the recovered experience. Before the fix it incorrectly skips; after the fix the recovered evidence exists in its new proposer context and the next unchanged rerun skips both tasks. All 7 native tests pass, including existing byte-identical repair/resume controls. Ruff whole check, touched formatting, and diff checks pass. OS network fence; no paid models or benchmark services.