Skip to content

Fix daily run-ceiling race under concurrent event arrival - #22

Open
deephazar-eva-ai wants to merge 1 commit into
theschoolofai:mainfrom
deephazar-eva-ai:fix/run-ceiling-concurrent-race
Open

deephazar-eva-ai wants to merge 1 commit into
theschoolofai:mainfrom
deephazar-eva-ai:fix/run-ceiling-concurrent-race

Conversation

@deephazar-eva-ai

Copy link
Copy Markdown

What breaks

AutonomyGovernor.admit_run checked window_count(...) < max_runs_per_day and returned an admitted verdict, but the matching increment (governor.record(subscription_id, kind="run", ...)) only happened in AutonomousEventEngine.process after runtime.run() completed. The check is a read, the increment is a write, and nothing held them together. Two events matching the same subscription that arrive close enough together — two webhook deliveries in one burst, a cron tick racing a retried tick — can both do their read before either does its write: both see the window under the ceiling, both get admitted, and the ceiling is exceeded by however many requests raced it.

This is the concurrent counterpart to a control the codebase already takes seriously in sequence (see test_a_daily_run_ceiling_bounds_an_agent_that_starts_its_own_runs) — the ceiling just never held once two requests could overlap in time, which is exactly the shape of traffic an unattended, event-driven agent actually receives.

The test that shows it breaking

tests/test_run_ceiling_concurrency.py::test_a_daily_run_ceiling_holds_when_events_arrive_concurrently sets max_runs_per_day=1, then fires five events matching the subscription through engine.process() concurrently via asyncio.gather, against a runtime whose run() takes a real (if brief) amount of time.

  • Before this fix: assert len(runtime.runs) == 1 fails — assert 5 == 1. All five runs started.
  • After this fix: passes. Exactly one run starts; the other four are refused with max_runs_per_day.

The fix

EventStore.try_reserve() checks the ceiling and claims one unit against it inside a single lock acquisition, so admit_run() reserves a run's slot atomically at admission time instead of trusting a separate write after the run finishes. window_record() gains an optional count= parameter (default 1, unchanged for every other caller) so the completion-time record() call can add the run's actual spend without double-counting a slot already claimed at admission. A reservation that turns out not to be used — the slot was claimed but the daily_budget check then refused the run anyway — is given back via release_reservation() rather than silently wasted.

Side effect worth flagging: a run that starts and then raises is now counted (the slot was claimed at admission), where before it wasn't (the count only incremented after runtime.run() returned successfully). A ceiling that bounds attempts rather than only completions is the more defensible reading for a control meant to stop unbounded spend or unbounded action — a run that fails after doing real work still did that work.

Not fixed here, flagged as a follow-up: admit_run's daily_budget check (a spend ceiling, not a count ceiling) has the same shape of race. Closing it precisely means reserving against an unknown final cost rather than a known count of 1 — a bigger change than this bug's scope.

Test plan

  • pytest tests/test_run_ceiling_concurrency.py — new regression test, passes
  • pytest tests/test_autonomy_governor.py — existing governor/ceiling coverage, still green (11/11)
  • pytest -q — full suite green, 356 passed
  • ruff check — clean on all changed files

AutonomyGovernor.admit_run checked window_count(...) < max_runs_per_day
and returned an admitted Verdict, but the matching increment
(governor.record(subscription_id, kind="run", ...)) only happened in
AutonomousEventEngine.process *after* runtime.run() completed -- the
check is a read, the increment is a write, and nothing held them
together. Two events matching the same subscription that arrive close
enough together (two webhook deliveries in one burst, a tick racing a
retried tick) can both do their read before either does its write:
both see the window under the ceiling, both are admitted, and the
ceiling is exceeded by however many requests raced it.

tests/test_run_ceiling_concurrency.py is the concurrent twin of the
existing (sequential) test_a_daily_run_ceiling_bounds_an_agent_that_
starts_its_own_runs: five events matching a subscription with
max_runs_per_day=1 are processed via asyncio.gather against a runtime
that takes a real (if brief) amount of time per run. Before this fix,
all five run; after it, exactly one does and the other four are
refused with max_runs_per_day, regardless of how many arrive at once.

Fix: EventStore.try_reserve() checks the ceiling and claims one unit
against it inside a single lock acquisition, so admit_run() reserves
a run's slot atomically at admission time instead of trusting a
separate write after the run finishes. window_record() gains an
optional count= parameter (default 1, unchanged for every other
caller) so the completion-time record() call can add the run's actual
spend without double-counting a slot that was already claimed at
admission. A reservation that turns out not to be used -- the slot
was claimed but the daily_budget check then refused the run anyway --
is given back via release_reservation() rather than silently wasted.

This also changes what "run count" means at the margin: a run that
starts and then raises is now counted (the slot was claimed at
admission), where before it would not have been (the count only
incremented after runtime.run() returned successfully). A ceiling
that bounds *attempts* rather than only *completions* is the more
defensible reading of "ceiling" for a control that exists to stop an
unattended agent from spending unbounded money or unbounded actions --
a run that fails after doing real work still did that work.

Not fixed here, flagged for a follow-up: admit_run's daily_budget
check (a spend ceiling, not a count ceiling) has the same shape of
race, but closing it precisely requires reserving against an unknown
final cost rather than a known count of 1, which is a bigger change
than this one bug's scope.
@theschoolofai

Copy link
Copy Markdown
Owner

Session 16 — graded

Score: 0.

Duplicate of #3, filed 2026-08-09 against this one's 08-15. Same run-ceiling race under concurrent arrival.

@theschoolofai

Copy link
Copy Markdown
Owner

Session 16 — regraded ✅

Score: +100. The verdict below stands: this is a duplicate of an earlier filing (or, for glc_v5 #23, an enhancement rather than a defect), and it is recorded as such.

What has changed is the credit. On review, the work here was genuinely done: the bug was found independently, the reproduction is real and the fix is sound. Losing a race you had no way of seeing is not a reason to earn nothing, so this is credited at the full 100 even though it does not carry the first-to-file claim.


Original assessment, unchanged:

Duplicate of #3, filed 2026-08-09 against this one's 08-15. Same run-ceiling race under concurrent arrival.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants