[jacob-adm]

Summary

On main, resuming a run skips every finished task, even if the selector/experience prompts or the experience library it read have since changed, so results made under the old inputs are kept. This PR records prompts_digest and library_fingerprint in each task's summary.json and re-runs the task on resume when either no longer matches; unchanged tasks still skip, and finished rollouts are still reused.

Resume flow, before and after:

flowchart TB
  subgraph B["Before (main)"]
    direction LR
    B1[resume a finished task] --> B2{adopted run, output<br/>and harness intact?}
    B2 -- no --> B3[re-run task]
    B2 -- yes --> B4[skip, reuse result]
    B5[prompts or library it read<br/>changed since the run] -. not checked .-> B4
  end
  subgraph A["After (this PR)"]
    direction LR
    A1[resume a finished task] --> A2{adopted run, output<br/>and harness intact?}
    A2 -- no --> A3[re-run task]
    A2 -- yes --> A4{library entries it read<br/>unchanged?}
    A4 -- no --> A3
    A4 -- yes --> A5{selection/experience<br/>prompts unchanged?}
    A5 -- no --> A3
    A5 -- yes --> A6[skip, reuse result]
    A3 -. new library entry .-> A7[later tasks in the domain<br/>see a changed library]
    A7 -.-> A4
  end
  B ~~~ A

Loading

Why

A resumed run refuses to mix models or task lists: _check_identity saves them, with METHOD_VERSION, in config.json and stops on a mismatch (run_sweep.py:209-235). Prompt edits are not part of that identity unless METHOD_VERSION is bumped, which invalidates the whole run directory. This PR re-runs only the tasks whose prompts or library changed, the same way main already re-runs a task whose adopted harness changed (schedule.py:409-411).

Fix

• Each mode records two values in <task>/summary.json, checked on resume. A mismatch re-runs the task; finished rollouts are still reused through their harness fingerprints.
  • prompts_digest: a hash of the selector and experience prompts and the candidate file headers (including the transcript preamble when candidates are shown as transcripts), built by the same helpers that write them (_select_mission(), _experience_mission(), _candidate_header()), with task data replaced by placeholders. At k = 1 only the experience prompt is hashed.
  • library_fingerprint: a fingerprint of the library entries the mode read, leaving out the task's own entry (written after it runs) and INDEX.md (rebuilt from the entries, so only its fixed wording goes unchecked).
• Apart from the current templates, the digest uses only inputs saved at run time on the selection: n_candidates, trivial, pool_kind, evidence, each candidate's asset_names, and experience_prompted (set where the experience prompt is sent).
• If computing the digest fails, a warning is printed and an empty digest is stored; the selection, library entry and task status are kept, and the task re-runs on the next resume. If the digest fails again on that resume, the prompt check is skipped with a warning instead of paying for another re-run that would fail the same way.

Effect on existing runs

• No change: tasks skip as before.
• Summaries written before this PR re-run once.
• A library shared with another run through root_for(tag=...) changes whenever that run writes to it, so its tasks re-run on resume.
• A re-run early in a domain re-runs the later tasks that read its library entry.
• Prompts that write harnesses (propose, aggregate, compose_round) are out of scope.

Related

• #21 adds the same library check (input_library_fingerprint); only one is needed. See the comparison on fix: invalidate resumed tasks when experience prefix changes #21.
• #6: once both are merged, feedback needs passing through the two helpers. I'll do it if this PR merges second.
• #74 also edits harness2/core/selection.py, so the two conflict there. I'll rebase if this PR merges second.
• #76 (task inputs) is complementary and merges cleanly.
• #82 merges cleanly, but its test_failed_rerun_clears_stale_selections_in_summary then expects ["partial", "skipped"]; with this PR the second task re-runs because the first task's library entry is gone, so it becomes ["partial", "partial"]. Whichever merges second updates that expectation.

Validation

$ python3 -m unittest discover -s tests
Ran 12 tests ... OK           # prompt edits at k=1/k=2, asset files, pre-PR summaries, deleted selection/,
                              # run_task -> run_domain resume, library change, failing digest,
                              # run_task(library=) resume, per-candidate digest text, experience_prompted
$ python3 scripts/check_provenance.py
PASS: 47 release asset checksums
$ ruff check harness2 tests
All checks passed!
$ git diff --check origin/main...HEAD   # clean

Python 3.12 and 3.13, at d297fbd.

Tooling

• Google Antigravity IDE 2.24.0 (1000335682), Gemini 4 Argon High: discovery, planning, implementation
• Claude Opus 5.5 (Claude Code, iTerm2 3.7.6): review, and the fixes made after review
• Local agent harness with canonical doctrine + PromptOps

[jacob-adm]

@rux001 @cuizz11 This is ready for review when you have a moment. If another route is considered, understood.

On main, if you edit the selector or experience prompts and then resume a run, every task that already finished is skipped and keeps the output chosen under the old prompts. This PR makes those tasks re-run when the prompts change, and keeps reusing them when nothing changed. The description covers the details and test coverage.

#32 and #37 don't depend on this PR and can be reviewed and merged in any order.

[jacob-adm]

I did a zero-cost check on whether this moves benchmark scores on a before/after run: I captured every prompt sent to the model on main vs this branch (d338a59) with the offline demo adapter, across k=1/2/3 with and without deliverable assets, and all 184 prompts are byte-identical (including the 24 experience prompts) once you ignore call order. So on a fresh run the model sees the exact same input on both, and the true score effect is zero. A paid before/after benchmark might just measure run-to-run noise, so I don't think it's personally worth the spend. In this context, I think the benchmark benefit is resumed runs. Scores could get more honest, since main can quietly mix tasks done under old and new prompts, which understates a prompt edit's effect by roughly the fraction already finished.

Concern is resume gets pricier: a re-run at position j invalidates everything after it in that domain through library_fingerprint, so one early transient failure re-runs nearly the whole tail. On top of that, pre-PR summaries all re-run once on first resume. Worth spelling out that cascade in EXPERIMENTS.md, and maybe adding an opt-out flag for people resuming after flaky failures. One gap: the demo doesn't exercise deliverable_assets_reference_only=True, though reading the code it takes the same path as main.

[jacob-adm]

@rux001 @cuizz11

EDIT 20261021: fixed once the tests moved into separate files so the three now merge in any order without conflict

gentle ping: #32, #36 and #37 are each mergeable against current main. Among themselves they conflict only in tests/test_workflow.py (keep both tests), so any merge order works and I'll rebase the rest after the first lands.

[jacob-adm]

Full Update (36)

• Cascade: now documented in docs/EXPERIMENTS.md. When a task re-runs, the later tasks in its domain re-run too, because they read the library it writes. This is intended. If an earlier task failed and now succeeds, its lesson enters the library, and the later tasks never saw it.
• One-time re-run: summaries written before this change have no prompts_digest or library_fingerprint, so they re-run once on the first resume after merging. This only affects sweeps that span the merge.

Why not avoid that one-time re-run:

• Trust old summaries when the fields are missing. Nothing records which prompts they were produced under, so this keeps exactly the stale results this PR is meant to catch.
• Check them against the saved prompts in _audit/. The selection prompt is saved under a label every task shares (select.<mode><k>), so a file can't be tied to one task. The experience prompt includes an evidence layout the summary doesn't record, so it can't be re-rendered exactly. A match would prove nothing.
• Bump config.METHOD_VERSION. run_sweep._check_identity then refuses the whole run directory, so every task re-runs in a new one. That costs at least as much, and every later prompt edit would need another manual bump.

On the opt-out flag I mentioned: I'd scope it to old summaries only. A summary missing these fields would be accepted, with today's digest and fingerprint written into it and one line logged per task, and checked normally from then on. I'd leave the cascade on, since turning it off keeps results built on a library that has since changed. I left the flag out so the default stays correct, and can add it if you want it.

[jacob-adm]

@rux001 @cuizz11

here's an unofficial report from some local testing. for information purposes and this info has had no human review outside of myself.

Merge test of the open PRs against main (1cfe254): what combines cleanly and what conflicts

I tried merging the open PRs together locally to see which ones combine cleanly, in case it helps when reviews start. I'm the author of #32, #36 and #37, so weigh what I say about those accordingly.

Procedure

• Base: main at 1cfe254, with PR heads fetched on 2026-10-23.
• Environment: Python 3.14.8, git 2.50.1, macOS.
• Test run: python -m unittest discover -s tests after every merge. Nine tests skip because they need tools this machine doesn't have.
• Merging: each PR is merged with git merge --no-ff onto the build so far, in a fixed order. That order was security fixes first, then fixes for wrong scores or results, then everything else, with ties broken by PR number.
  • If a merge conflicts, it's aborted and the PR is recorded as conflicting.
  • If tests fail after a merge, it's reset and the PR is recorded as failing.
• Swap test: for each PR that was held back, I reverted the merged PR(s) it collides with, merged it in their place, and ran the tests again.
• Pair check: for each failure with no text conflict, I merged just the two PRs involved onto main and ran the tests.
• Not tried:
  • feature PRs (new substrates, auth modes, CLIs, translations), to keep the test to fixes;
  • PRs whose CLA check fails;
  • PRs that address the same issue as an earlier open PR;
  • PRs that change code without tests;
  • Actualizar las plataformas de modelos de financiación al español #126 (an unrelated change to FUNDING.yml).

The merge order is my choice, and it decides which side of each conflict merged first. A different order would give a different set.

Result: 51 of 71 PRs merged together with every test passing

#37, #79, #32, #7, #127, #10, #12, #17, #24, #28, #88, #89, #110, #122, #9, #11, #23, #25, #30, #35, #58, #70, #8, #19, #21, #48, #53, #54, #76, #92, #105, #132, #134, #14, #16, #18, #20, #26, #49, #67, #131, #15, #29, #34, #47, #75, #87, #104, #119, #103, #1

Text conflicts; tests pass once swapped in (12)

In most of these pairs the two PRs fix different things and touch the same lines, so a rebase should let both in.

PR	Conflicts with
#118	#79, #47
#27	#24
#33	#32
#13	#127
#22	#23
#31	#30
#45	#21, #132
#97	#8
#36	#21
#101	#132
#68	#53
#90	#37

The one real either/or is #36 and #21: both add the same library check on resume, and #36 also checks the selection/experience prompts. They differ in one case: a task resumed directly with run_task(library=...) and no changes re-runs on #21 (['ok', 'ok']) and skips on #36 and main (['ok', 'skipped']). #21 records the fingerprint before the task writes its own library entry, then compares against the library with that entry included. Resuming through run_domain works on both. Only one of the two is needed.

Tests fail even after swapping (4)

Each needs rework against what's already merged:

• fix(proposer): reject incomplete audit cache entries #69: its proposer-cache tests fail without fix(proposer): isolate cached replies and harnesses by channel configuration #48, which also changes the cache.
• fix(core): reject unparseable trajectory sentinels in context, selection, and resume #72: its artifact-name tests fail; it overlaps fix: preserve distinct artifact renderings in proposer context #14.
• fix(report): count improver spend replayed from the audit cache #74: its spend-replay tests fail even with fix: stop passing API keys and host tokens to agent subprocesses #37 and Retry runs after improvement proposer failures #110 taken out.
• fix(schedule): clear stale mode selections when a task re-run fails #82: fails test_failed_rerun_clears_stale_selections_in_summary.

No text conflict, but a pair fails its tests

Confirmed on main with just the two PRs:

Pair	Failing test
#12 + #86	test_missing_n_passed_is_ungraded_not_a_crash
#23 + #78	test_cli_propagates_executor_failure_and_preserves_success
#134 + #102	test_missing_timeout_is_reported_for_lab
#21 + #82	test_failed_rerun_clears_stale_selections_in_summary

Other

• Fails its own tests on main: fix(jobbench): isolate rollout installs, temp files and workspaces #112 (test_jobbench_rollout_isolation, 3 errors).
• Same issue as an earlier open PR:
  • core: keep a failed mode stage spend in the summary #130 and fix(schedule): a failed mode keeps its paid stage records #101 (both fix harness2-report drops a mode's improver spend when the mode fails after its stages ran #98)
  • benchmarks: do not treat a timeout-killed JobBench rollout as finished #133 and fix(jobbench): note in the trajectory when a rollout exits non-zero #122 (both fix JobBench keeps a rollout killed at its time limit as a finished run, with no trace in its trajectory and no exit status in its log #121)
  • judge: treat a null all_pass / n_passed as absent, not as a bad score #135 and fix(judge): treat an explicit null grade as an absent one #86
  • Ao/harness2 1/getting started #63 bundles feat(examples): add inspectable local ledger exercises #56, docs: add a progressive getting-started guide #58, docs(examples): explain local ledger commands and evidence #60 and docs: explain when to use Harness2 and link first steps #62
• CLA check failing: fix(setup): let --dry-run preview Codex setup without an endpoint #51, fix(bridge): refuse to start without a model to serve #52, feat(lab): add a drex backend to read_doc.py #107, fix(lab): judge with Claude on Anthropic's API when HARNESS2_JUDGE_MODEL is anthropic/<id> #108, Add DeepSeek Harness executor for JobBench and LAB #115, fix(lab): route openai-compatible judges through Chat Completions #120, jobbench: stop requiring GNU timeout for OpenCode rollouts #128, core: let a deployer pin harness files against proposed edits #129

Tests passing only shows that the PRs combine. It doesn't show each fix is correct. I'm happy to share the script or re-run it against a newer main.

_{Tooling: Claude Opus 5.5 (Claude Code) ran the merges and tests.}