Fix daily run-ceiling race under concurrent event arrival - #22
Open
deephazar-eva-ai wants to merge 1 commit into
Open
deephazar-eva-ai wants to merge 1 commit into
deephazar-eva-ai wants to merge 1 commit into
Conversation
AutonomyGovernor.admit_run checked window_count(...) < max_runs_per_day and returned an admitted Verdict, but the matching increment (governor.record(subscription_id, kind="run", ...)) only happened in AutonomousEventEngine.process *after* runtime.run() completed -- the check is a read, the increment is a write, and nothing held them together. Two events matching the same subscription that arrive close enough together (two webhook deliveries in one burst, a tick racing a retried tick) can both do their read before either does its write: both see the window under the ceiling, both are admitted, and the ceiling is exceeded by however many requests raced it. tests/test_run_ceiling_concurrency.py is the concurrent twin of the existing (sequential) test_a_daily_run_ceiling_bounds_an_agent_that_ starts_its_own_runs: five events matching a subscription with max_runs_per_day=1 are processed via asyncio.gather against a runtime that takes a real (if brief) amount of time per run. Before this fix, all five run; after it, exactly one does and the other four are refused with max_runs_per_day, regardless of how many arrive at once. Fix: EventStore.try_reserve() checks the ceiling and claims one unit against it inside a single lock acquisition, so admit_run() reserves a run's slot atomically at admission time instead of trusting a separate write after the run finishes. window_record() gains an optional count= parameter (default 1, unchanged for every other caller) so the completion-time record() call can add the run's actual spend without double-counting a slot that was already claimed at admission. A reservation that turns out not to be used -- the slot was claimed but the daily_budget check then refused the run anyway -- is given back via release_reservation() rather than silently wasted. This also changes what "run count" means at the margin: a run that starts and then raises is now counted (the slot was claimed at admission), where before it would not have been (the count only incremented after runtime.run() returned successfully). A ceiling that bounds *attempts* rather than only *completions* is the more defensible reading of "ceiling" for a control that exists to stop an unattended agent from spending unbounded money or unbounded actions -- a run that fails after doing real work still did that work. Not fixed here, flagged for a follow-up: admit_run's daily_budget check (a spend ceiling, not a count ceiling) has the same shape of race, but closing it precisely requires reserving against an unknown final cost rather than a known count of 1, which is a bigger change than this one bug's scope.
Owner
Session 16 — gradedScore: 0. Duplicate of #3, filed 2026-08-09 against this one's 08-15. Same run-ceiling race under concurrent arrival. |
Owner
Session 16 — regraded ✅Score: +100. The verdict below stands: this is a duplicate of an earlier filing (or, for glc_v5 #23, an enhancement rather than a defect), and it is recorded as such. What has changed is the credit. On review, the work here was genuinely done: the bug was found independently, the reproduction is real and the fix is sound. Losing a race you had no way of seeing is not a reason to earn nothing, so this is credited at the full 100 even though it does not carry the first-to-file claim. Original assessment, unchanged: Duplicate of #3, filed 2026-08-09 against this one's 08-15. Same run-ceiling race under concurrent arrival. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What breaks
AutonomyGovernor.admit_runcheckedwindow_count(...) < max_runs_per_dayand returned an admitted verdict, but the matching increment (governor.record(subscription_id, kind="run", ...)) only happened inAutonomousEventEngine.processafterruntime.run()completed. The check is a read, the increment is a write, and nothing held them together. Two events matching the same subscription that arrive close enough together — two webhook deliveries in one burst, a cron tick racing a retried tick — can both do their read before either does its write: both see the window under the ceiling, both get admitted, and the ceiling is exceeded by however many requests raced it.This is the concurrent counterpart to a control the codebase already takes seriously in sequence (see
test_a_daily_run_ceiling_bounds_an_agent_that_starts_its_own_runs) — the ceiling just never held once two requests could overlap in time, which is exactly the shape of traffic an unattended, event-driven agent actually receives.The test that shows it breaking
tests/test_run_ceiling_concurrency.py::test_a_daily_run_ceiling_holds_when_events_arrive_concurrentlysetsmax_runs_per_day=1, then fires five events matching the subscription throughengine.process()concurrently viaasyncio.gather, against a runtime whoserun()takes a real (if brief) amount of time.assert len(runtime.runs) == 1fails —assert 5 == 1. All five runs started.max_runs_per_day.The fix
EventStore.try_reserve()checks the ceiling and claims one unit against it inside a single lock acquisition, soadmit_run()reserves a run's slot atomically at admission time instead of trusting a separate write after the run finishes.window_record()gains an optionalcount=parameter (default1, unchanged for every other caller) so the completion-timerecord()call can add the run's actual spend without double-counting a slot already claimed at admission. A reservation that turns out not to be used — the slot was claimed but thedaily_budgetcheck then refused the run anyway — is given back viarelease_reservation()rather than silently wasted.Side effect worth flagging: a run that starts and then raises is now counted (the slot was claimed at admission), where before it wasn't (the count only incremented after
runtime.run()returned successfully). A ceiling that bounds attempts rather than only completions is the more defensible reading for a control meant to stop unbounded spend or unbounded action — a run that fails after doing real work still did that work.Not fixed here, flagged as a follow-up:
admit_run'sdaily_budgetcheck (a spend ceiling, not a count ceiling) has the same shape of race. Closing it precisely means reserving against an unknown final cost rather than a known count of 1 — a bigger change than this bug's scope.Test plan
pytest tests/test_run_ceiling_concurrency.py— new regression test, passespytest tests/test_autonomy_governor.py— existing governor/ceiling coverage, still green (11/11)pytest -q— full suite green, 356 passedruff check— clean on all changed files