Outbox: let an effect that failed be attempted again

[nishanthvonteddu]

A side effect that failed did not happen. Running it again is not a repeat — it is the first time it happens.

What breaks

ActionOutbox.execute() caches a failed operation as terminal:

if record["status"] == "failed":
    raise RuntimeError(record["error"])

Every later call for that key re-raises the stored error. The outbox exists to stop one side effect being applied twice; refusing a failure forever solves a different problem and creates one. A transient fault — a full disk, a rate limit, a misconfigured path — becomes permanent, because the receipt outlives its cause and repairing the cause changes nothing. The affected node can never complete on any retry or resume.

How to see it break

Point S16_SANDBOX_ROOT at a path that does not exist and run anything that calls write_file. Correct the setting, then retry the same node: it still fails, with the stale error.

I hit this for real. Four write_file receipts in ~/.s16code/outbox/ were left holding OSError: [Errno 30] Read-only file system and stayed unrunnable after the setting was fixed.

The fix

failed falls through to a fresh attempt, carrying an attempt counter and the previous_error so a retry that keeps failing for the same reason stays legible without a log search.

completed and started are unchanged: a receipt is still reused, and a crash left in flight is still parked for reconciliation rather than guessed at.

Test

test_outbox_retries_an_effect_that_failed fails before this change and passes after.

The shipped test covered completed and started but not failed, which is why this survived.