[FrankD412]

Why

Stacked on #1602. Base is fdinatale/aip-2293-integration-flake. Retarget to main once #1602 merges.

The per-root trace idle watchdog in ReplayBarrierCoordinator treats a replay tree as idle whenever in_flight == 0. Some grouped timers have already started but haven't reached observe_issued() yet. For example, a timer can be waiting on acquire_prefill_slot() or send_credit() in credit/issuer.py. Those timers aren't counted. If the watchdog fires while one is in that state, it moves the tree's later timers forward and collapses the recorded replay spacing.

The watchdog gets there by two paths:

• The zero-delay rearm in submit(), added in 80662af (feat(agentx): port AgentX harness support from SemiAnalysisAI fork #1413). A second timer lands on a barrier-blocked turn, and the rearm's guard is in_flight == 0 and idle_cap_expired.
• _on_root_timer_done(), added in fix(timing): keep the trace idle cap past timers that issue nothing #1602. A second timer returns without issuing a request, and the cap is applied again.

coderabbit and dynamo-review-agent flagged the first path on #1602:

• fix(timing): keep the trace idle cap past timers that issue nothing #1602 (review)
• fix(timing): keep the trace idle cap past timers that issue nothing #1602 (review)

Repro

Three timers in one root fire at 1.00 s (A), 1.05 s and 1.40 s (E), so the recorded gap from A to E is 0.40 s. A waits on an event that stands in for a prefill slot. Both trees gave the same result on 10 of 10 runs:

Second timer	main (98d8e53)	#1602 (ad2b581)	This PR
Barrier-blocked (submit() rearm)	0.050 s	0.050 s	0.402 s
Issues nothing (_on_root_timer_done)	0.402 s	0.050 s	0.402 s

What

• src/aiperf/common/loop_scheduler.py:
  • Records the group of each running task started through execute_async(group_id=...). That covers fired grouped timers and zero-delay grouped schedules. The record is removed in _done_callback and cleared by cancel_all_running().
  • running_count_for_group() counts a group's running tasks. It is the counterpart of pending_count_for_group().
• src/aiperf/timing/replay_dependencies.py: _enforce_root_idle_cap() shifts nothing while the root still has a running grouped task, and leaves idle_cap_expired set. One check covers both paths. The cap still gets applied later:
  • If the running timer issues a request, observe_issued() clears the cap.
  • If it returns without issuing, _on_root_timer_done() applies the cap again. The scheduler has already stopped counting the task by the time the observer runs.

Why not the suggested guard

The reviews suggested adding running_count_for_group(root_id) == 0 to the rearm in submit(). That doesn't work, because the task calling submit() is itself a running grouped task, so the count never reaches zero. On this base, both test_idle_cap_waits_for_started_timer_that_has_not_issued cases (barrier_blocked and issues_nothing) still fail with that guard in place. test_idle_watchdog_retries_when_advanced_turn_is_barrier_blocked still passes with it, because #1602's completion observer re-enforces the cap; without that observer, it fails because the cap stalls. Excluding the calling task fixes submit(), but the _on_root_timer_done() path from #1602 never goes through submit(), so it stays broken.

Verification

• 6 new tests. All 6 fail without the source change and pass with it. test_idle_cap_resumes_when_started_timer_returns_without_issuing now holds one grouped timer while a second returns, so the cap is re-enforced and must defer:
  • test_idle_cap_waits_for_started_timer_that_has_not_issued[barrier_blocked|issues_nothing]
  • test_idle_cap_resumes_when_started_timer_returns_without_issuing
  • test_running_count_for_group_counts_started_unfinished_tasks
  • test_running_count_for_group_excludes_task_in_group_observer
  • test_cancel_all_running_clears_group_running_counts
• Existing tests that use a MagicMock scheduler now set running_count_for_group.return_value = 0. A bare MagicMock attribute is truthy, so without this the cap would never apply in those tests.
• Local runs, Python 3.13:
  • tests/unit/timing/test_replay_barrier_coordinator.py and tests/unit/common/test_loop_scheduler.py: 71 passed, on each of 10 runs.
  • tests/unit/{timing,common,credit} and the overlap-barrier component test, with -n 8: 5061 passed, 1 failed. The failure was test_bootstrap_parent_death::test_parent_death_signal_real_kill_reaps_child. That test doesn't use this code, and it passes when run alone, both with and without this change.
  • -m integration Weka idle-cap, agentic_replay drops post-child delay on parent resume #1231 and spawn-join tests: 7 passed. test_runtime_idle_cap_handles_independent_stream_clock_drift passed 5 of 5.
  • -m component_integration, filtered to replay, barrier, agentic and idle tests: 26 passed, 3 skipped.
• pre-commit run --from-ref ad2b5816 --to-ref HEAD: all hooks passed.
• Python 3.11, 3.12 and Windows haven't been run yet. That's left to CI.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

Try out this PR

Quick install:

pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@2f7a11a51df4e31b34d02aa4b68f96e2d789e0ff

Recommended with virtual environment (using uv):

uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@2f7a11a51df4e31b34d02aa4b68f96e2d789e0ff

---

Last updated for commit: 2f7a11a • Browse code

[ajcasagrande]

I reviewed this PR at 90fc62a11 against fdinatale/aip-2293-integration-flake (merge-base 3c024b39d). Checking running tasks in shared enforcement covers both watchdog entry paths without collapsing replay spacing. The remaining feedback concerns test coverage and one statement in the PR description; neither blocks merging.

Verified

• Scheduler/coordinator and adjacent agentic-replay tests passed: 149 passed.
• Real mock-server integration checks passed: 2 passed, covering independent-stream drift and system-idle advancement without bypassing the child join.
• Independent probes confirmed that a deferred cap resumes after non-issuing work finishes (0.0001 s) and clears when work issues. Python 3.11/3.12 and Windows were not run locally.

Not addressable to a diff line

Low: The description's “Why not the suggested guard” names a test that passes with the submit-only guard on this stacked base: test_idle_watchdog_retries_when_advanced_turn_is_barrier_blocked. The completion observer at replay_dependencies.py:193–216 rescues it; disabling that observer reproduces the stated stall. Qualify that explanation as applying without #1602's observer, or cite the two new overlapping-timer regressions, which do fail with the suggested alternative. This is description-only, not a source change.

_{🤖 Keeping watch until every timer clocks out.}

[ajcasagrande]

Low: This test never exercises deferral: the initial watchdog fires before slow_noop starts, and nothing retries enforcement while it runs; replacing enforcement with the complete base implementation still yields 1 passed. The autouse fixture also turns this sleep(0.05) into sleep(0) (tests/unit/conftest.py:150–164). Add a second grouped no-op whose completion triggers enforcement while slow_noop is held, assert the distant timer remains pending, then release it and check advancement.

_{🤖 The timing agent needs a deferred appointment.}

[FrankD412]

Good catch, thanks. Fixed in d8d797d along the lines you suggested: a second grouped no-op at 1.05 s returns while slow_noop is still held, so the observer re-enforces the expired cap and has to defer. The test asserts the far timer is still pending and hasn't fired, then releases slow_noop and checks that the far timer fires within 0.2 s. The sleep(0.05) is gone. With the running-task guard removed from _enforce_root_idle_cap, the test now fails (pending_count_for_group is 0, expected 1).

I also updated the "Why not the suggested guard" section of the description: the barrier-blocked test only fails without #1602's observer, and the two test_idle_cap_waits_for_started_timer_that_has_not_issued cases are now cited as the ones that fail with the submit-only guard.

[ajcasagrande]

I reviewed this PR at d7521164d against fdinatale/aip-2293-integration-flake (merge-base 3c024b39d). The shared enforcement guard preserves replay spacing across both entry paths, and the strengthened regression now exercises deferral while another grouped timer runs. No correctness blocker remains; the existing helper-docstring nit is tracked below.

Verified

• Scheduler/coordinator tests passed: 71 passed, on Python 3.13/Linux.
• Real mock-server integration checks passed: 2 passed, covering independent-stream drift and system-idle advancement without bypassing the child join.
• Restoring base enforcement in memory collapsed both overlap-test gaps to approximately 0.0503 s; HEAD passes their ≥0.35 s requirement. A separate asyncio probe confirmed issuance clears the expired cap and leaves another root's deadline unchanged.
• Python 3.11/3.12 and Windows were not run locally.

Resolved while reviewing

• The deferral-test and description corrections check out: the test holds one timer while another returns, and the submit-only alternative still fails both overlap cases.
• The revised scheduler-test docstrings explain the ownership window and observer ordering, as the author stated.

Raised by others

• The non-blocking helper-docstring nit remains in its existing thread; no duplicate comment here.

_{🤖 On watch until the last grouped operative clocks out.}

[ajcasagrande]

I reviewed this PR at d7521164d against fdinatale/aip-2293-integration-flake (merge-base 3c024b39d). No correctness blocker remains; the only additional feedback is the wording nit below. The earlier review on this HEAD remains applicable.

Verified

• Scheduler/coordinator tests passed with locked dependencies: 71 passed, Python 3.13/Linux; the real mock-server integration checks also passed: 2 passed.
• Restoring base enforcement in memory reduced the overlap gaps to 0.05050 s and 0.05043 s, below the tests' 0.35 s minimum; HEAD preserves the required spacing.
• Python 3.11/3.12 and Windows were not tested locally. Initial timing-test attempts encountered clock/import-order failures; the controlled run passed all 71.

Raised by others

• The existing helper-docstring nit stays in its original thread.

_{🤖 The timing agent keeps every operative on the clock.}

[ajcasagrande]

I reviewed this PR at 2f7a11a51 against fdinatale/aip-2293-integration-flake (merge-base 3c024b39d). The shared enforcement guard preserves grouped-timer spacing across both watchdog entry paths while allowing the expired cap to resume after non-issuing work finishes. Nothing blocks merging.

Verified

• Scheduler/coordinator tests passed: 71 passed, Python 3.13.12/Linux; the broader timing/common/credit suite passed 5,062 tests.
• Real mock-server integration checks passed: 2 passed, covering independent-stream drift and system-idle advancement without bypassing child completion.
• Restoring base enforcement in memory collapsed the overlap gaps to 0.05052 s and 0.05050 s, below the tests’ 0.35 s minimum; HEAD passes. An asyncio probe confirmed issuance clears the deferred cap without moving either root’s pending deadline.
• Python 3.11/3.12 and Windows were not tested locally.

Resolved while reviewing

• The deferral regression now holds one grouped timer while another finishes; restoring base enforcement fails its pending-timer assertion.
• The wording correction and unused-helper removal are present in 5613a84e7.
• The redundant trailing comment is gone in 2f7a11a51.

_{🤖 Keeping grouped operatives on their recorded schedule.}