Skip to content

feat(stats): linger an empty group broadcast before unannouncing it - #4871

Merged
kixelated merged 3 commits into
mainfrom
quest/m0/stats-linger
Oct 5, 2026
Merged

kixelated merged 3 commits into
mainfrom
quest/m0/stats-linger

Conversation

@kixelated

@kixelated kixelated commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

At depth >= 1 a group's stats broadcast (<prefix>/<group>/node/<node>) is unannounced on the first drain its group has no traffic or session entries, and re-announced when one returns. On moq.pro's fleet (2026-09-29) one customer's node stats broadcasts ended on 31 nodes at once every 2 to 4 minutes and returned 46 to 112 s later, churning announces across the mesh.

Quest: quest/m0/stats-linger.md.

Approach

  • moq_stats::produce::Config::linger (default 5 minutes, with_linger): a group broadcast stays announced until it has been empty for the linger. A returning entry re-arms it. Zero keeps today's behavior (unannounce on the first empty drain). Depth 0 is unchanged.
  • While a group lingers empty, its tracks read {}: no live gauges, no session presence.
  • Totals: main still has the per-path maps (docs(quest): stats totals and per-broadcast tracks replace the per-path maps #4846 only planned the stats split), so a path that leaves drops out of frames as it does today, and its return reads as a restart from zero. A reader diffing frames counts the same totals as it would across an unannounce and re-announce. The producer-wide group allocator already keeps group numbers increasing across the return.
  • The linger is measured on the publish task's monotonic clock at each drain; tests run on paused tokio time.
  • Relay: stats.linger / --stats-linger / MOQ_STATS_LINGER (humantime, e.g. 5m), using the same skip-plus-_arg pattern as cache.duration. Unset keeps the producer default.

Impact

  • moq-stats: new public field produce::Config::linger and Config::with_linger. Config is #[non_exhaustive], so this is additive. Behavior change: at depth >= 1, an empty group broadcast now stays announced for 5 minutes by default instead of unannouncing at once.
  • moq-relay: new stats::Config::linger: Option<Duration> and the stats.linger / --stats-linger / MOQ_STATS_LINGER setting.
  • Wire: none. Track names and frame shapes are unchanged.
  • Docs: doc/concept/stats.md, doc/bin/relay/config.md.

Decisions

Confirmed by the maintainer (2026-10-05):

  • Default linger stays 5 minutes (observed returns were 46 to 112 s).
  • The release backport drops the path from frames while its group lingers empty, matching main, rather than carrying its last totals forward.

Alternatives

  • Carry a path's last totals forward while its group lingers, so a return continues its counters. Rejected for main: the stats split replaces the per-path maps with per-group totals that continue under one epoch, and dropping the path matches today's semantics for a path that leaves a live group.
  • Count drains instead of time. Rejected: the ticker delays missed ticks, so a drain count drifts from the configured duration.

Follow-ups

  • release backport (moq.pro tracks release). release's produce.rs differs from main's only by the wall-clock group seed (fix(stats): resume past cached groups after reclaim or restart (backport) #4810), so this should cherry-pick cleanly with the same drop-from-frames semantics. The quest file stays on main, trimmed to that backport.
  • Stats epochs and the stats split pick up the linger as their idle-group window (quest/m0/broadcast-epoch/stats-split.md updated).

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

kixelated and others added 3 commits October 5, 2026 15:31
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A grouped stats broadcast (`<prefix>/<group>/node/<node>`) now stays
announced for `Config::linger` (default 5 minutes) after its group's last
traffic and session entries leave, so viewer churn no longer unannounces and
re-announces it across the mesh. While it lingers empty its tracks read `{}`;
a returning entry re-arms the linger. The relay exposes it as
`stats.linger` / `--stats-linger` / `MOQ_STATS_LINGER`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Outcome (quest start, unattended):

  • Implemented on main; just check passes locally (419 + 143 Rust tests, the new linger tests among them). The regression test fails without the linger.
  • Quest premise note: docs(quest): stats totals and per-broadcast tracks replace the per-path maps #4846 only planned the stats split, so main still has the per-path maps. The linger therefore uses the "path drops out of frames while the group lingers" semantics on main too, the same choice the quest left open for release.
  • Open decisions for the maintainer:
    1. Default linger: 5 minutes (recommended; observed returns were 46 to 112 s, and the cost of an idle announced group is small). Alternative: 2 minutes.
    2. release backport totals: drop the path from frames (recommended; same as main, cherry-picks cleanly) vs carry totals forward.
  • The quest file stays on main, trimmed to the release backport, which is left for a separate PR.

(Written by Claude Opus 5.5)

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: c8a4060

No actionable correctness findings in the full diff and surrounding producer/configuration code.

Direction looks sound: monotonic, per-group idle timing resets when either traffic or sessions return, preserves depth-0 lifetime, and keeps zero-linger behavior (rs/moq-stats/src/produce.rs:359–381, 851–860). Lingering groups still flush empty frames and prune per-path state, so a returning path follows the existing reset semantics; the plain/compressed tracks share that path. The relay knob follows the existing duration adapter and precedence pattern (rs/moq-relay/src/stats.rs:81–114, 127–141).

Verification limits: static review only; I did not run the added paused-time tests or configuration tests. The author's local check result is reported, not independently reproduced; the latest fetched Check and Platform runs were queued. The release backport remains separate.

@coderabbitai

coderabbitai Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: f0a1de52-bb82-434f-8868-d0566809a8fc
📥 Commits

Reviewing files that changed from the base of the PR and between 5b673b6 and c8a4060.

📒 Files selected for processing (9)
  • doc/bin/relay/config.md
  • doc/concept/stats.md
  • quest/m0/README.md
  • quest/m0/broadcast-epoch/stats-split.md
  • quest/m0/stats-linger.md
  • rs/moq-relay/src/config.rs
  • rs/moq-relay/src/settings.rs
  • rs/moq-relay/src/stats.rs
  • rs/moq-stats/src/produce.rs

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 0 remain after this review.


Walkthrough

The stats producer now keeps empty grouped broadcasts announced for a configurable linger period, which defaults to five minutes. Returning entries reuse the broadcast if they return before expiry. Relay configuration supports TOML, environment, and CLI values, with the CLI value taking precedence. Tests cover linger expiry, returning entries, zero linger, and configuration overrides. Documentation and quest notes describe the behavior and backport status.

Priority: ⬆️ High

Merge Risk: ⚪ Minimal · up to c8a40

The linger behavior is ready to merge after normal checks; no actionable risk remains identified.

Security Architecture Review

Security architecture risk: 🔵 Low · up to c8a40

Authorization and cleanup ownership remain intact. The main risk is increased resource retention when grouped stats are enabled and a permitted client can generate many distinct groups. Stats remain disabled by default, depth zero is unaffected, and setting linger to zero restores immediate empty-group removal.

Retained concerns

  • Medium · security · inferred: When grants and grouping depth let a principal generate many distinct group keys, short-lived authorized activity now leaves each group's broadcast, tracks and periodic processing alive for the linger window. The producer's time-only cleanup increases retained-state amplification relative to immediate removal. This can increase resource pressure on the shared relay without requiring permission to read stats. Effective cardinality and exhaustion thresholds remain deployment-dependent; fixed group prefixes, expiry and zero linger are important countercontrols.
Security review details

Security Blast Radius

  • inferred — The directly supported resource exposure is the relay's shared stats producer and origin. Existing cluster distribution can carry retained announcements onward, but fleet-wide exhaustion or cross-tenant data access is not established. Independent attackable cardinality depends on grant prefixes and configured grouping depth.

Security Findings and Attack Paths

  • inferred — The conditional availability path is permitted activity across distinct grouping prefixes, followed by registry-driven broadcast allocation and delayed cleanup. Repeating short-lived activity can accumulate idle groups across the linger window. This is a resource-retention concern, not a verified anonymous exploit or privilege escalation.

Trust Boundaries and Controls

  • observed — Admission rejects sessions without the requested scoped role. Publishing still checks permitted paths and path-depth bounds. These controls predate the PR and remain unchanged; prolonging an announcement does not widen those grants.

Resilience and Maintainability Implications

  • observed — Empty groups expire on subsequent drains, and shutdown closes remaining broadcasts. Existing limits bound requested track pairs and parked requests per group, with unused requested pairs reclaimed. These limits do not bound total group cardinality.

Hardening Proposals

  • proposed — Consider an explicit count or memory budget for lingering groups where grants permit variable grouping prefixes, with deterministic idle-group eviction. This would bound retention amplification without treating per-group track quotas as a global safeguard.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 73.08% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 26 functions across 4 files. (5 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: adding a linger period before an empty grouped stats broadcast is unannounced.
Description check ✅ Passed The description explains the problem, implementation, configuration options, behavior, and scope of the changeset.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 73.08% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 26 functions across 4 files. (5 skipped: 5 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches
✨ Simplify code
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
🧰 Additional context used
📚 Code guidelines (3)
rs/AGENTS.md — auto-discovered
rs/moq-relay/AGENTS.md — auto-discovered
doc/bin/relay/config.md — auto-discovered

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@kixelated

Copy link
Copy Markdown
Collaborator Author

Merge summary:

  • Grouped stats broadcasts (depth >= 1) stay announced for produce::Config::linger (default 5 minutes) after their group empties; zero keeps the old unannounce-on-first-empty-drain behavior. Relay knob: stats.linger / --stats-linger / MOQ_STATS_LINGER.
  • Decisions confirmed by the maintainer: keep the 5 minute default; the release backport drops the path from frames while its group lingers empty (no totals carried forward).
  • Reviews: the automated review on c8a4060 and CodeRabbit found nothing actionable. CI is green (Check, Test, Quest, macOS, Windows).
  • quest/m0/stats-linger.md stays on main until the release backport merges; that backport PR follows.

(Written by Claude Opus 5.5)

@kixelated
kixelated merged commit a52ee7e into main Oct 5, 2026
7 checks passed
@kixelated
kixelated deleted the quest/m0/stats-linger branch October 5, 2026 23:28
kixelated added a commit that referenced this pull request Oct 6, 2026
feat(stats): linger an empty group broadcast before unannouncing it (backport #4871)
This was referenced Oct 6, 2026
shermerL pushed a commit to shermerL/moq that referenced this pull request Oct 7, 2026
…backport moq-dev#4871)

A grouped stats broadcast (`<prefix>/<group>/node/<node>`) now stays
announced for `Config::linger` (default 5 minutes) after its group's last
traffic and session entries leave, so viewer churn no longer unannounces and
re-announces it across the mesh. While it lingers empty its tracks read `{}`;
a returning entry re-arms the linger. The relay exposes it as
`stats.linger` / `--stats-linger` / `MOQ_STATS_LINGER`.

Cherry-pick of moq-dev#4871 (a52ee7e) onto `release`; the announce test helper
uses `release`'s announce update API.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant