[RoldSI]

Follow-up to #49, which is merged. That PR added stop_on_success as a Controller argument but never added it to ExperimentMeta, so it reached neither identity_hash() nor experiment_block().

Why this is a blocker, not a tidiness fix

The flag's whole purpose is the opt-out: stop_on_success=False is what an experiment measuring attack reliability needs (how many of N attempts succeed, not whether any did). That mode is currently a no-op.

Two Controllers differing only in this flag produce the same identity_hash(), so they resolve to the same experiment directory. _status_of maps a won task to status "success", and "success" is in _KEPT_STATUSES, so plan_resume keeps every task from the prior tree. A reliability run therefore resumes the stop-on-win directory and executes zero runs, returning the stop-on-win record verbatim. Nothing warns, because from resume's point of view every task is already complete.

Reproduced: pass 1 (stop_on_success=True, max_runs_per_task=3) writes a directory with one run at stop_reason="success". Pass 2 (stop_on_success=False, same labels) resolves to that same directory, len(runs) == 0, and reports the earlier run's n_runs=1.

Keyed only when disabled

identity_hash() hashes a dict, so adding a key changes the digest even at the default value. The first version of this branch keyed it unconditionally, which re-keyed every results tree ever written: the same meta hashed ef26fece before and 5b18e4e4 after, so dirname() resolves somewhere new, plan_resume sees a fresh experiment, and finished work is recomputed. The certified 843-hour tree would have been unresumable.

Keying it only when False gets the property the flag needs while leaving every default run on its historical hash:

if not self.stop_on_success:
    identity["stop_on_success"] = False

Verified: default and explicit-True both hash ef26fece (unchanged from main); False hashes 0f2b58f6.

The regression test whose absence let that through

The suite had five tests asserting a field changes the hash and three asserting one does not, but none asserting what the hash is — so the tree-breaking version was 596/596 green.

test_default_identity_hash_is_frozen now pins ef26fece, and its docstring says not to update the constant when it fails: key the new field conditionally instead.

Also

• stop_on_success is recorded in experiment_block(), so a stored tree says which regime produced its run counts. Without it, two trees with very different mean runs-per-task are indistinguishable after the fact.
• reference/results.md's documented identity list corrected; it omitted the field.

Checks

598 passed (594 before, plus 4 new) · ruff check/format clean · mypy clean on 25 source files.

[RoldSI]

Superseded by the branch that removes the flag entirely: with stop-on-success as the only behaviour there is no second regime to distinguish, so the identity key this PR added (and then had to make conditional to avoid re-keying every existing results tree) is unnecessary. See the replacement PR.