[OUAO-FRANK]

Closes #65

What

Task inputs are now part of a rollout's reuse identity, at both cache layers the report names:

• runmark.inputs_fingerprint(task_prompt, files_dir) digests the model-visible task text plus the source material, walked in sorted order and hashed by relative path + bytes, reusing the existing _ignored exclusions (__pycache__, .pyc, .DS_Store). The adapters already strip the answer key and the grading criteria from task_prompt, so those never enter the digest.
• mark_complete(..., inputs=) records it next to the harness fingerprint; is_complete(..., inputs=) and why_not(..., inputs=) compare it. LabAdapter.execute and JobBenchAdapter.execute pass it, so a rollout whose task changed is re-rolled with a log line naming the reason instead of being returned.
• schedule._resume compares the summary's recorded digest with the current task and re-runs, which also rebuilds that task's library entries (Library.add is keyed by task_id).

Contract

Consistent invalidation rather than a new hard error: a mismatch means "not reusable", exactly how a harness-fingerprint mismatch has always been handled, so the existing re-roll path does the work and no new failure mode is introduced.

Both checks are opt-in per caller, and a marker or summary written before the field existed keeps its previous meaning — _inputs_match treats a missing inputs key as "no opinion" — so upgrading does not invalidate every in-flight run. Bumping runmark.VERSION would have done that; I deliberately did not.

WorkBuddyAdapter is unchanged on purpose: its task_files is the whole task directory including the answer key, so its digest needs an explicit leak_globs exclusion before it can join. Left as a follow-up rather than guessed at here.

cli.run_sweep._check_identity is also untouched: adding a key to that dict would make every existing config.json mismatch on upgrade. It is a namespace guard, not a reuse gate — the reuse decision is per task, in _resume, which is where the fix belongs.

How checked

New file tests/test_task_input_identity.py, 13 tests, all passing:

python -m unittest tests.test_task_input_identity   # Ran 13 tests ... OK

The three RolloutReuseTests drive LabAdapter.execute with the executor replaced by a deterministic function that writes the current document (no model call, no downloaded benchmark, no shell) — the report's standalone reproduction. At the base commit test_changed_source_files_re_roll_the_rollout and test_changed_instructions_re_roll_the_rollout fail (executor calls: 1, stale answer), and test_unchanged_inputs_reuse_the_rollout passes, so the change re-rolls on changed inputs without over-invalidating unchanged ones. The other ten tests cover the digest, the marker, and _resume; they error at base only because the API they exercise does not exist yet.

Running the report's standalone LAB snippet by hand at head:

[lab-cx] re-rolling lab-cx-agg1/example__task-1/base1: task inputs 5dae5cd936c2a54b != c1da96d03d4c4a89 -- the task prompt or its source files changed since this rollout finished
prompt has v2: True
current source: version-two
cached output: version-two
executor calls: 2

The same snippet at base gives cached output: version-one / executor calls: 1, i.e. the reported symptom.

ruff check . passes and ruff format --check is clean over the touched files.

Platform caveat, stated plainly: the report's scheduler reproduction could not be re-run end to end on this machine. tests.test_workflow and schedule._write_summary need POSIX fcntl and a shell-driven demo adapter (pyproject.toml declares Operating System :: POSIX), so on my Windows host the existing suite is 4 failures / 1 error at the base commit and at head, with the identical failure names — no new breakage, but also no green baseline to compare against. I verified the summary layer through schedule._resume directly instead, which needs neither fcntl nor a subprocess. A Linux CI run of python -m unittest discover -s tests is the check I could not perform and would like from a maintainer.

The cost is one hash pass over a task's source files per rollout and per task summary — small next to the rollout itself, but it is a real cost on JobBench.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[kvishalv]

I'm following this because of #5, which proposes a --bench local adapter for people who run Harness² on their own task folders. In that setup, people edit task.md and the source files between runs all the time, so the gap in #65 would be the normal case there, not an edge case.

I read the diff with that in mind. The check in schedule._resume lives in core, so it covers every adapter, including new ones. The rollout check works differently. It only runs when an adapter passes inputs= to is_complete() and mark_complete(), and this PR does that for LAB and JobBench. WorkBuddyAdapter._reusable() still calls RM.is_complete(trial, harness_dir=harness_dir) without it, so I think a changed WorkBuddy task would re-run but could get its old trial back from execute(). Was leaving WorkBuddy out on purpose, maybe because its inputs come from the container image?

My other question is about adapters that don't exist yet. Since inputs=None turns the comparison off, a new adapter that forgets the argument gets the old behaviour without a warning. Would you consider making inputs a required keyword, so an adapter has to pass None on purpose to opt out? I'd like the local adapter in #5 to be unable to get this wrong by accident.

[jacob-adm]

Heads-up on overlap with #36: both PRs add a digest to the task summary and check it in schedule._resume. They cover different things #36 digests the selection/experience prompt templates (task data replaced by placeholders), this PR digests the task's own prompt and files so the two are complementary and both are needed. The only conflict is textual, in _resume and TaskResult.to_dict; whichever lands second rebases. No change needed here otherwise.

[OUAO-FRANK]

Thanks for the careful read — both points are exactly the kind of thing I want settled before this lands.

On WorkBuddy: yes, leaving it out was deliberate, not an oversight. WorkBuddyAdapter._reusable() keys reuse on the container image the run was executed against; the task's prompt and files live inside that image, not in the adapter's inputs= slot. So the "task inputs changed but the rollout still comes back" failure mode from #65 does not apply the same way — there is nothing extra for the adapter to hash that the image identity does not already cover. If that changes (for example, a WorkBuddy task that mounts files from outside the image), then it should thread inputs= through like LAB and JobBench do.

On making inputs a required keyword: I see the attraction, but it would be a broader API change than this fix. is_complete() / mark_complete() are called by several existing adapters, and inputs=None is currently the meaningful "no extra inputs, keep the old identity" default. Turning it required would force every adapter to spell that out and would read as a new contract, not a fix to #65. I'd rather keep this PR scoped and raise that as a follow-up if you want it, because it is a real ergonomics question for the #5 local adapter.

If a maintainer prefers the required-keyword direction, I'm happy to do it in a follow-up rather than widen this diff.

Net: no code change from me on either point — flagging so the decision is explicit rather than silent.

[jacob-adm]

Correction: I test merged #36 and this PR (git merge-tree) and they combine cleanly. No rebase needed in either order. The two checks are still complementary as described above.

[RanaPriyansh]

@OUAO-FRANK I verified base 1cfe254e8d25b15f37048c3730134686abe60335 and head 653b1311fbd1c323ce06d62bd84e1d7ee5e64949 on macOS/POSIX with Python 3.12.13 and 3.13.13.

• The existing python -m unittest -v tests.test_workflow passes all five tests on both revisions and interpreters.
• At head, python -m unittest discover -s tests -v discovers and passes all 18 tests on both interpreters.
• Your three RolloutReuseTests, loaded unchanged against each revision, reproduce the two intended baseline failures. The unchanged-input control passes. All three pass at head.
• The actual schedule.run_task reproduction in Resume and rollout caches silently reuse outputs after task inputs change #65 returns the stale 85.15 with zero new executions on base. At head it reruns and returns 3.00.

The scheduler reproductions used Python 3.12.13.

I found one remaining scheduler integration case. After a task's inputs change, _resume() correctly rejects the old summary. However, _write_summary() merges its old modes and selections into the new result without checking their input identity. A rerun of only agg therefore writes the new top-level inputs digest while retaining the old seq result. A subsequent seq run accepts that digest and skips execution.

This standalone reproduction uses the real scheduler and demo executor, without model calls:

import tempfile
from pathlib import Path
from harness2.core import schedule
from harness2.demo import DemoAdapter, DemoProposer

with tempfile.TemporaryDirectory() as temporary:
    adapter = DemoAdapter(Path(temporary))
    proposer = DemoProposer(adapter.spec.channels["proposer"])
    kwargs = dict(adapter=adapter, prop=proposer,
                  runs_root=adapter.spec.runs_root)
    schedule.run_task("ledger-1", 1, modes=("agg", "seq"), **kwargs)
    source = adapter.task_files("ledger-1") / "ledger.csv"
    source.write_text("amount\n1.00\n2.00\n")
    agg = schedule.run_task("ledger-1", 1, modes=("agg",), **kwargs)
    before = adapter.execution_count
    seq = schedule.run_task("ledger-1", 1, modes=("seq",), **kwargs)
    output = adapter.deliverable_text(
        Path(seq.selections["seq"]["chosen_run_dir"])
    ).strip()
    print(agg.status, seq.status, adapter.execution_count - before, output)

Actual at head: ok skipped 0 Total: 85.15. The seq result should instead be recomputed from the changed ledger, whose total is 3.00. With unchanged inputs, both modes correctly skip without additional executions.

The summary merge needs to prevent modes from a different input identity from becoming reusable under the new digest. This transition also needs an end-to-end regression. The existing summary tests call _resume() directly and do not exercise _write_summary().

These are local offline checks on macOS, not Linux or hosted CI validation. I have not changed your branch.

Codex used.

[OUAO-FRANK]

@RanaPriyansh thanks for the thorough repro — that's a real gap, and you're right that the existing tests only exercised _resume(), not the merge.

Fixed in ad6e747. _write_summary() now detects an input-identity change and carries nothing over from the old summary in that case:

inputs_changed = (
    res.inputs is not None
    and old.get("inputs") is not None
    and old.get("inputs") != res.inputs
)
if not inputs_changed:
    # merge old modes / selections / errors as before ...
# else: the previous summary answered a different question, so its
# mode records and selections must not be re-used under the new digest.

So the agg-only rerun after a ledger change now writes the new digest with only agg, and the following seq run sees the new digest without a stale seq record and recomputes. Added SummaryInputTests.test_write_summary_drops_modes_from_an_older_input_identity, which writes a full agg+seq summary under the old inputs, rewrites agg under new inputs, and asserts seq is gone from both modes and selections. It fails on 653b131 and passes on ad6e747.

python -m unittest tests.test_task_input_identity → 14 tests OK (run locally with an fcntl shim for the flock on this Windows box; the repo CI runs Linux).