Resume and rollout caches silently reuse outputs after task inputs change #65

Problem

Task instructions and source files are not part of the saved experiment/rollout identity. If those contents change while the task IDs, models, namespace and harness stay the same, resume can silently return an answer to the old task. Both the task-summary cache and the production LAB rollout cache accept the obsolete output.

Reproduced against upstream 1cfe254e8d25b15f37048c3730134686abe60335, on Linux / Python 3.12.15. No model calls or credentials are needed.

Scheduler reproduction

import tempfile
from pathlib import Path
from harness2.core import schedule
from harness2.demo import DemoAdapter, DemoProposer

with tempfile.TemporaryDirectory() as temporary:
    adapter = DemoAdapter(Path(temporary))
    proposer = DemoProposer(adapter.spec.channels["proposer"])
    kwargs = dict(adapter=adapter, prop=proposer,
                  runs_root=adapter.spec.runs_root, modes=("agg",))
    first = schedule.run_task("ledger-1", 1, **kwargs)
    source = adapter.task_files("ledger-1") / "ledger.csv"
    source.write_text("amount\n1.00\n2.00\n")
    count = adapter.execution_count
    resumed = schedule.run_task("ledger-1", 1, **kwargs)
    output = adapter.deliverable_text(Path(resumed.selections["agg"]["chosen_run_dir"]))
    print(first.status, resumed.status, output.strip(), adapter.execution_count - count)

Actual: ok skipped Total: 85.15 0. The changed ledger totals 3.00, but there are no new executions and the old answer is reused.

The lower-level rollout cache has the same gap

The LAB example below uses the real adapter/cache/marker/integrity logic. Only the external executor is replaced with a deterministic function that writes the current source into an output file. It does not run a downloaded benchmark or a model.

import json
import tempfile
from pathlib import Path
from unittest.mock import patch
from harness2 import config
from harness2.benchmarks.lab import LabAdapter

with tempfile.TemporaryDirectory() as temporary:
    root = Path(temporary)
    lab = root / "lab"
    task_id = "example/task-1"
    task = lab / "tasks" / task_id
    (task / "documents").mkdir(parents=True)
    task_json = task / "task.json"
    document = task / "documents/source.txt"
    task_json.write_text(json.dumps({"instructions": "Use version one."}))
    document.write_text("version-one")
    with patch.object(config, "RUNS", root / "runs"), patch.object(
        config, "lab_repo", return_value=lab
    ):
        adapter = LabAdapter(substrate="cx")
        run_id = "lab-cx-agg1/example__task-1/base1"

        def runner(task_id, install, run_ids, timeout):
            out = adapter._run_dir(run_ids[0])
            (out / "output").mkdir(parents=True)
            (out / "output/answer.txt").write_text(document.read_text())
            (out / "metrics.json").write_text("{}")
            return [out]

        with patch.object(adapter.substrate, "execute", side_effect=runner) as executor:
            adapter.execute(task_id, None, run_id)
            task_json.write_text(json.dumps({"instructions": "Use version two."}))
            document.write_text("version-two")
            output = adapter.execute(task_id, None, run_id)
            print(adapter.task_prompt(task_id))
            print("current source:", document.read_text())
            print("cached output:", (output / "output/answer.txt").read_text())
            print("executor calls:", executor.call_count)

Why it happens

• cli.run_sweep._check_identity saves task IDs/order and configuration, but no task-content or dataset-revision identity.
• schedule._resume validates the chosen harness/output, without comparing the current task prompt or source material.
• runmark.is_complete compares the harness fingerprint. LabAdapter.execute adds output integrity checks but no input-content comparison. Consequently, invalidating/deleting a task summary alone would still leave this obsolete rollout reusable.

Impact and expected behavior

A corrected task document or instruction, or a refreshed dataset retaining the same task IDs, can leave an experiment reporting completion for the previous inputs. The JobBench dataset is resolved at download time rather than pinned to a commit, as documented in the reproducibility guide; unchanged task IDs alone are not proof of unchanged data. I have not run a live dataset-refresh experiment or measured benchmark-score effects.

Expected behavior is either a clear refusal to resume under changed inputs (requiring a new run directory) or consistent invalidation at both summary and rollout layers. Silently retaining the old answer should not be treated as successful resume.

Possible scope

Introduce a stable identity for the model-visible task prompt and source material, using the existing answer-key exclusions rather than hashing grading criteria into agent context. Validate it before task-summary reuse and before adapter rollout reuse. A dataset revision can supplement this identity, but may not detect locally corrected inputs. Cover unchanged inputs, modified instructions, added/removed/changed source files, legacy markers, and lower-level cache reuse.

If the desired contract is that task contents are immutable within a run directory, a fail-fast identity check would keep this smaller than automatic recomputation. If automatic recomputation is preferred, downstream experience-library dependencies need to be invalidated too.

Related PRs #21 and #36 address experience-prefix and selector/experience-template changes. This report concerns the task's own instructions/source files and adapter rollout identity. It is also separate from #48's proposer channel-configuration cache and our #54's concurrent library ownership fix. No implementation is included here; agreeing on the immutable-input versus recomputation contract first seems useful.