Skip to content

Outbox: make an effect a crash left uncertain findable - #30

Open
nishanthvonteddu wants to merge 1 commit into
theschoolofai:mainfrom
nishanthvonteddu:fix/outbox-reconcile-unreachable
Open

nishanthvonteddu wants to merge 1 commit into
theschoolofai:mainfrom
nishanthvonteddu:fix/outbox-reconcile-unreachable

Conversation

@nishanthvonteddu

Copy link
Copy Markdown

A run parked by a crash mid-side-effect waits on an event nothing ever sends, and nothing records that it is stuck.

What breaks

execute() meets a record left started — the process died between dispatching an effect and learning whether it landed — and parks:

if record["status"] == "started":
    return Deferred(key, "outbox.reconcile", {"uncertain": True})

Refusing to guess is right. Repeating an effect that may already have happened is worse than waiting.

But "outbox.reconcile" appears exactly once in the package — where it is created. No producer emits it. The only way out is an operator completing the handle by hand through /completions, and they cannot:

  • nothing lists the stuck keys
  • started carries no timestamp, so an effect in flight looks identical to one abandoned weeks ago
  • nothing says the recovery path exists

The run waits forever and no surface admits it.

The fix

Not a reconciler — the answer genuinely requires a person checking whether the effect landed. What was missing is the means to act:

  • started records carry started_at
  • uncertain() lists parked effects, oldest first
  • resolve(key, receipt) closes one with what the person found, stored in the shape execute returns so the next call does not re-park on an answered question
  • the module docstring states the contract, which a reader previously had no way to discover

Test

test_outbox_can_list_what_a_crash_left_uncertain — parks an effect, finds it via uncertain(), resolves it, confirms the receipt is reused afterwards. Fails before, passes after.

`execute()` parks a run on `Deferred(key, "outbox.reconcile")` when it
meets a record left `started` by a process that died mid-dispatch. Not
guessing is right: repeating an effect that may already have landed is
worse than waiting.

But `"outbox.reconcile"` appears exactly once in the package -- where it
is created. Nothing emits it. The only way out is an operator completing
the handle by hand, and they cannot, because nothing lists the stuck keys,
nothing timestamps them, and nothing says the recovery path exists. The
run waits forever and no surface admits it.

`started` records now carry `started_at`, so an effect in flight can be
told from one abandoned weeks ago. `uncertain()` lists them oldest first,
and `resolve()` closes one with what the person found out -- stored in the
shape `execute` returns, so the next call does not re-park on a question
that has already been answered.

The module docstring now states the contract, which was the part a reader
had no way to discover.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant