Fleet stats support

[yusufozturk]

Support for VirtualMetric Agent statistics

Summary by CodeRabbit

• New Features

  • Added optional restart-and-verify support for the stats scenario, including exact checks for decoded metric counters after a mid-run restart.
  • Fleet simulation now validates reported stats more precisely, including per-bucket counter values.

• Bug Fixes

  • Improved config-apply detection by waiting for a fleet acknowledgement instead of relying on log messages.
  • Stats validation now fails on counter mismatches and reports deadline-related discrepancies more clearly.

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 40 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 9aa2c568-8fd6-4065-8a22-ccfd1631828d

📥 Commits

Reviewing files that changed from the base of the PR and between f60e91a and eb3f1e1.

📒 Files selected for processing (2)

• internal/config/case.go
• internal/runner/runner.go

Walkthrough

Adds RestartMidRun and ExpectStats fields to FleetConfig, introduces a fleetStatBucket type with a statField() helper, adds a fleetWaitStats() polling function for exact-match stat assertions, replaces log-grep config-applied detection with fleet-link rep.config acknowledgements, and wires fleetWaitStats() into the "stats" scenario.

Fleet Stats Validation & Config-Applied Detection

Layer / File(s)	Summary
FleetConfig new fields internal/config/case.go	Adds RestartMidRun bool and ExpectStats map[string]map[string]int64 to FleetConfig for the stats scenario.
fleetStatBucket type and statField helper internal/runner/runner.go	Introduces fleetStatBucket struct with exec/event/byte counters, updates fleetStatus.directors stats map to use it, and adds statField() to look up a counter by director ID, bucket name, and field name.
fleetWaitStats polling helper internal/runner/runner.go	Adds fleetWaitStats() that polls the simulator status until all expected decoded stat counters are exactly equal; returns an error on deadline expiry or immediately on over-counts.
Config-applied detection via fleet-link acknowledgement internal/runner/runner.go	Replaces log-grep detection of "director applied config" with fleetWaitCount waiting for rep.config replies and inspecting the payload for "executed":true.
Stats scenario deep validation internal/runner/runner.go	In the "stats" scenario, calls fleetWaitStats() when fc.ExpectStats is non-empty and prints decoded record/frame counts on success.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• VirtualMetric/PipeBench#29: Established the FleetConfig/runner fleet correctness infrastructure that this PR directly extends with RestartMidRun, ExpectStats, and the new stat validation path.

Poem

🐇 A bucket of stats, a restart mid-hop,
The director acknowledged — no grep needed to stop.
I poll and I wait till the counters align,
Each frame and each record confirmed to be fine.
The fleet hops along on a perfectly straight line! 🎉

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and matches the main change: adding fleet stats support.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch dev-agent-stats

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/runner/runner.go`:
- Around line 3940-3964: Restrict the RestartMidRun flow in runner.go so it only
runs for the stats scenario, since fc.RestartMidRun is documented as stats-only
but this block currently executes before the scenario switch. Update the
RestartMidRun handling to check fc.Scenario against "stats" before calling
orch.StopServices, orch.UpServices, and the reconnect/config redelivery logic,
or otherwise reject unsupported fc.RestartMidRun + scenario combinations up
front with a clear failure.
- Around line 3902-3928: The config delivery flow is not failing the run when
`rep.config` is missing or returns without `executed:true`, which can produce
false success later; update the director config check in `runner.go` to use the
same hard-fail behavior as the existing `config_update` path. In the code around
`fleetWaitCount`, `fleetSimStatus`, and the restart/setup branch that uses
`fleetSimSend`, treat any missing acknowledgment or non-executed reply as a
fatal setup failure instead of printing a success line, so both delivery paths
stop immediately when the delivered config was not applied.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 8cf513c9-9fb6-4e0a-ab44-085fb982855b

📥 Commits

Reviewing files that changed from the base of the PR and between 3c78e70 and f60e91a.

📒 Files selected for processing (2)

• internal/config/case.go
• internal/runner/runner.go

[cloudflare-workers-and-pages]

Deploying pipebench with    Cloudflare Pages

Latest commit:	eb3f1e1
Status:	✅  Deploy successful!
Preview URL:	https://db646c8a.pipebench.pages.dev
Branch Preview URL:	https://dev-agent-stats.pipebench.pages.dev

View logs