Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion quest/m0/broadcast-epoch/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,4 +89,5 @@ This README owns:
- [GStreamer](/quest/m0/broadcast-epoch/gst.md) - moqsink publishes each run under a fresh epoch
- [Remove `--hop`](/quest/m0/broadcast-epoch/hop-removal.md) - `moq` takes an optional `--epoch` instead of `--hop`, a plain publisher declares a random Hop ID, and the per-session hop stamp is gone
- [Bounded stats aggregate](/quest/m0/broadcast-epoch/stats-aggregate-bound.md) - the stats aggregator folds departed nodes into a retired total, so epoch churn stops growing its memory
- [Stats epochs](/quest/m0/broadcast-epoch/stats-epoch.md) - moq-stats publishes each node under its own epoch, so a restarted node never stalls its viewers
- [Stats epochs](/quest/m0/broadcast-epoch/stats-epoch.md) - moq-stats publishes each group announcement under its own epoch, so neither a restarted node nor a returning idle group stalls its viewers
- [Stats totals and per-broadcast tracks](/quest/m0/broadcast-epoch/stats-split.md) - the same release retires the per-path stats maps for totals and per-broadcast tracks (decided 2026-10-05)
3 changes: 3 additions & 0 deletions quest/m0/broadcast-epoch/stats-aggregate-bound.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@ Decided (2026-10-02): grace-window fold.
only a same-instance return after a long outage), or whether some bounded
per-path baseline reconciles it. Test exact totals across that return either
way, not just monotonicity.
- With an epoch per group announcement, a stats group returning from idle is
always a new path, so the grace re-arm covers only reconnects of a still
announced path, and the double-count above needs a same-epoch return.
- Rejected: folding on depart double-counts a node that reconnects with its
counters intact. TTL eviction without a fold makes merged totals regress.
- Timers follow the repo rule: time decisions use `max(wall, pts)`. Tests mock
Expand Down
31 changes: 22 additions & 9 deletions quest/m0/broadcast-epoch/stats-epoch.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,33 +2,46 @@

## Goal

Each stats producer publishes under its broadcast epoch,
`<prefix>[/<group>]/node/<node>/@<epoch>`, so a restarted node never reuses a
broadcast name or the group numbers a relay cached under it.
Each stats group broadcast publishes under its own epoch,
`<prefix>[/<group>]/node/<node>/@<epoch>`, minted each time the group is
announced, so neither a restarted node nor a group returning from idle reuses
a broadcast name or the group numbers a relay cached under it.

## Plan

- Use the shared `moq_net::Epoch`, not a stats-local UUID (decided
2026-10-02). One epoch per `Producer`, shared by all its group broadcasts.
2026-10-02).
- One epoch per group announcement (decided 2026-10-05, replacing 2026-10-02's
one epoch per `Producer`). A group that goes idle unannounces after its
linger and drops its totals; when it returns it announces under a new epoch
counted from zero. Why: a producer never holds an idle group's state, so its
memory is bounded by active and lingering groups, and readers see every
reset as a new path rather than detecting one. The cost: a reader that
misses every frame across the linger loses that epoch's tail; billing
under-bills by that tail, consistent with the 0-bill baseline.
At depth 0 the single broadcast never unannounces, so its epoch still lasts
the producer's life.
- [#4739](https://github.com/moq-dev/moq/pull/4739)'s original commits are
prior art: path syntax, `Producer::epoch()`, `parse_node_path`, the
prior art: path syntax, `parse_node_path`, the
aggregator's restarted-epoch handling, and the tests. The PR landed only the
producer-wide group allocator.
- Decide what an unset node becomes once the epoch follows it (#4739 used
`local`).
- The aggregator treats a new epoch as a new node: its counters add to the
retired total from the [bounded aggregate](/quest/m0/broadcast-epoch/stats-aggregate-bound.md)
instead of regressing the merged total.
- MoQ Pro wants to share the same epoch with its storage stats, so expose it
from `Producer`.
- Expose each group broadcast's epoch (a per-group accessor replacing
#4739's `Producer::epoch()`) so a consumer can tell epochs apart.
MoQ Pro's VOD `storage.json` mints its own epoch rather than sharing a
producer-wide one, which no longer exists.
- `release` carries a temporary seed instead
([#4810](https://github.com/moq-dev/moq/pull/4810)): a producer's first group
number is wall-clock microseconds, so a restarted node numbers above its
previous run. Epochs replace it, so it never reaches `main`: release-to-main
back-merges keep `main`'s `rs/moq-stats` and `doc/concept/stats.md`. Drop the
seed and its `doc/concept/stats.md` sentence from `release` when this ships
there. MoQ Pro's VOD `storage.json` seeds the same way and moves to the
shared epoch with it.
there. MoQ Pro's VOD `storage.json` seeds the same way and moves to its
own epoch with it.
- demo/web stats keys include the epoch. Update `doc/concept/stats.md`,
`doc/bin/relay/config.md`, and the relay stats config docs.

Expand Down
96 changes: 96 additions & 0 deletions quest/m0/broadcast-epoch/stats-split.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
# [L] Stats totals and per-broadcast tracks

## Goal

Each node's stats broadcast publishes per-group cumulative totals that are
never pruned while the group is announced, plus one stats track per broadcast, served only when requested.
The per-path map tracks are gone. While a group stays announced, a reader
that misses a frame loses nothing: the next totals frame still carries
everything since the epoch began.

## Plan

Decided 2026-10-05 (planned from moq-dev/moq.pro#2202):

- **Why.** Every counter is cumulative per entry, and an entry is pruned right
after the frame carrying its closing readout. A broadcast that starts and
ends inside a group a reader missed shows in no later frame, so a lagging
reader undercounts. The maps also grow with every live broadcast, and
readers like billing want only per-project sums.
- **Totals.** Per group broadcast (`<prefix>[/<group>]/node/<node>/@<epoch>`),
per tier and role, cumulative within the [stats epoch](/quest/m0/broadcast-epoch/stats-epoch.md).
Every group announcement, and so every restart, starts a new epoch counted
from zero; nothing is serialized to disk. `Registry`'s unpruned lifetime totals
(`rs/moq-net/src/stats.rs`) are node-wide, per tier and role only, so they
can't be published as-is: keep the totals per group, folding an entry into
its group's total when it is pruned, or two projects sharing a tier would
Comment thread
coderabbitai[bot] marked this conversation as resolved.
merge.
- **Idle groups** (decided 2026-10-05). Today a group broadcast disappears
the moment it has no entries (`rs/moq-stats/src/produce.rs`). Instead it
stays announced for the stats linger (`quest/m0/stats-linger.md`, an m0
quest planned in [#4843](https://github.com/moq-dev/moq/pull/4843); list it
under Related once it lands, not Required, so it does not gate the release)
after its last entry ends, so a group that returns within the linger
continues its totals. A zero linger is valid: a returning group then always
takes a new epoch. After the linger it unannounces and drops its
totals; a return announces under a new [epoch](/quest/m0/broadcast-epoch/stats-epoch.md)
counted from zero. Totals are cumulative and a lingering group's frames all
repeat its final totals, so a reader that misses every frame across the
linger loses that epoch's tail; billing under-bills by that tail,
consistent with the 0-bill baseline. Document that. Memory is bounded by active and
lingering groups. Test two groups sharing a tier, a group returning within
the linger (same epoch, totals continue), and one returning after it (new
epoch from zero).
- **Per-broadcast tracks.** A reader that wants one broadcast subscribes to
its track; nothing is produced for a broadcast no one requests. The track
carries that broadcast's cumulative counters and finishes after its closing
readout. A request after the closing readout gets the zeroed track below,
so a zeroed per-broadcast track is not authoritative; totals carry the
real sums. Pick the track naming while implementing, but it must be
injective: tiers and broadcast paths both contain `/`, so no plain
concatenation of the two parses back to one pair.
- A broadcast's counters start at zero. A request for a broadcast with no
entry (never seen, or already pruned) is held open with zeroed counters
until one appears, as an unrecorded tier is today, and is reclaimed when
its last consumer leaves.
- The tier-sized caps (`MAX_REQUESTED_TRACKS` is 64) are too small for a page
of visible broadcasts. Give per-broadcast tracks their own cap per group
broadcast, sized (or configurable) for every reader the aggregator fans in,
not one page. Refuse requests beyond it with a typed error the reader
retries, rather than a parked track that reads as zero. Tier requests park
instead (`MAX_PARKED_REQUESTS`) because a collector treats a refusal as
final; per-broadcast readers must not.
- **Sessions** (decided 2026-10-05). `sessions.json` is keyed by auth root
and loses pruned roots the same way, so fold per-tier `Presence` into the
totals (session counts) and serve per-root detail as a requested track,
the same model as per-broadcast tracks.
- **Retire the map tracks** (`publisher.json`, `subscriber.json`,
`sessions.json`, and their `.json.z` siblings) in the same release. Decide
while implementing whether totals and per-broadcast tracks keep `.json.z`
siblings.
- Ship in the same breaking release as stats epochs, so consumers take one
stats path and wire change, not two.
- Update the aggregator, `doc/concept/stats.md`, and `demo/web/src/stats.ts`
(which reads the three maps) with the change. The
aggregator merges totals across nodes, and merges one broadcast's track
across nodes when a reader requests it, so a multi-node consumer serves the
same format it reads.

Public API: `moq-stats` producer and consumer types. Wire: stats track names
and payloads.

MoQ Pro adopts it when it pins the release: billing reads totals, its
Broadcasts page subscribes per visible broadcast, its `announced` probe
reads totals only, and its customer stats feed serves this format summed
across nodes.

## Required

- [Stats epochs](/quest/m0/broadcast-epoch/stats-epoch.md) - totals are cumulative within an epoch

## Related

- [Bounded stats aggregate](/quest/m0/broadcast-epoch/stats-aggregate-bound.md) - retired
nodes fold into a bounded total; per-epoch totals feed it
- [Media stats](/quest/m1/stats/README.md) - publisher and viewer media stats
stay hang tracks, separate from the relay's stats
Loading