diff --git a/quest/m0/broadcast-epoch/README.md b/quest/m0/broadcast-epoch/README.md index 141a6f8ff5..7f968003ee 100644 --- a/quest/m0/broadcast-epoch/README.md +++ b/quest/m0/broadcast-epoch/README.md @@ -89,4 +89,5 @@ This README owns: - [GStreamer](/quest/m0/broadcast-epoch/gst.md) - moqsink publishes each run under a fresh epoch - [Remove `--hop`](/quest/m0/broadcast-epoch/hop-removal.md) - `moq` takes an optional `--epoch` instead of `--hop`, a plain publisher declares a random Hop ID, and the per-session hop stamp is gone - [Bounded stats aggregate](/quest/m0/broadcast-epoch/stats-aggregate-bound.md) - the stats aggregator folds departed nodes into a retired total, so epoch churn stops growing its memory -- [Stats epochs](/quest/m0/broadcast-epoch/stats-epoch.md) - moq-stats publishes each node under its own epoch, so a restarted node never stalls its viewers +- [Stats epochs](/quest/m0/broadcast-epoch/stats-epoch.md) - moq-stats publishes each group announcement under its own epoch, so neither a restarted node nor a returning idle group stalls its viewers +- [Stats totals and per-broadcast tracks](/quest/m0/broadcast-epoch/stats-split.md) - the same release retires the per-path stats maps for totals and per-broadcast tracks (decided 2026-10-05) diff --git a/quest/m0/broadcast-epoch/stats-aggregate-bound.md b/quest/m0/broadcast-epoch/stats-aggregate-bound.md index 7fe85310c7..95b3019084 100644 --- a/quest/m0/broadcast-epoch/stats-aggregate-bound.md +++ b/quest/m0/broadcast-epoch/stats-aggregate-bound.md @@ -26,6 +26,9 @@ Decided (2026-10-02): grace-window fold. only a same-instance return after a long outage), or whether some bounded per-path baseline reconciles it. Test exact totals across that return either way, not just monotonicity. +- With an epoch per group announcement, a stats group returning from idle is + always a new path, so the grace re-arm covers only reconnects of a still + announced path, and the double-count above needs a same-epoch return. - Rejected: folding on depart double-counts a node that reconnects with its counters intact. TTL eviction without a fold makes merged totals regress. - Timers follow the repo rule: time decisions use `max(wall, pts)`. Tests mock diff --git a/quest/m0/broadcast-epoch/stats-epoch.md b/quest/m0/broadcast-epoch/stats-epoch.md index d9236e330e..38300cc73b 100644 --- a/quest/m0/broadcast-epoch/stats-epoch.md +++ b/quest/m0/broadcast-epoch/stats-epoch.md @@ -2,16 +2,27 @@ ## Goal -Each stats producer publishes under its broadcast epoch, -`[/]/node//@`, so a restarted node never reuses a -broadcast name or the group numbers a relay cached under it. +Each stats group broadcast publishes under its own epoch, +`[/]/node//@`, minted each time the group is +announced, so neither a restarted node nor a group returning from idle reuses +a broadcast name or the group numbers a relay cached under it. ## Plan - Use the shared `moq_net::Epoch`, not a stats-local UUID (decided - 2026-10-02). One epoch per `Producer`, shared by all its group broadcasts. + 2026-10-02). +- One epoch per group announcement (decided 2026-10-05, replacing 2026-10-02's + one epoch per `Producer`). A group that goes idle unannounces after its + linger and drops its totals; when it returns it announces under a new epoch + counted from zero. Why: a producer never holds an idle group's state, so its + memory is bounded by active and lingering groups, and readers see every + reset as a new path rather than detecting one. The cost: a reader that + misses every frame across the linger loses that epoch's tail; billing + under-bills by that tail, consistent with the 0-bill baseline. + At depth 0 the single broadcast never unannounces, so its epoch still lasts + the producer's life. - [#4739](https://github.com/moq-dev/moq/pull/4739)'s original commits are - prior art: path syntax, `Producer::epoch()`, `parse_node_path`, the + prior art: path syntax, `parse_node_path`, the aggregator's restarted-epoch handling, and the tests. The PR landed only the producer-wide group allocator. - Decide what an unset node becomes once the epoch follows it (#4739 used @@ -19,16 +30,18 @@ broadcast name or the group numbers a relay cached under it. - The aggregator treats a new epoch as a new node: its counters add to the retired total from the [bounded aggregate](/quest/m0/broadcast-epoch/stats-aggregate-bound.md) instead of regressing the merged total. -- MoQ Pro wants to share the same epoch with its storage stats, so expose it - from `Producer`. +- Expose each group broadcast's epoch (a per-group accessor replacing + #4739's `Producer::epoch()`) so a consumer can tell epochs apart. + MoQ Pro's VOD `storage.json` mints its own epoch rather than sharing a + producer-wide one, which no longer exists. - `release` carries a temporary seed instead ([#4810](https://github.com/moq-dev/moq/pull/4810)): a producer's first group number is wall-clock microseconds, so a restarted node numbers above its previous run. Epochs replace it, so it never reaches `main`: release-to-main back-merges keep `main`'s `rs/moq-stats` and `doc/concept/stats.md`. Drop the seed and its `doc/concept/stats.md` sentence from `release` when this ships - there. MoQ Pro's VOD `storage.json` seeds the same way and moves to the - shared epoch with it. + there. MoQ Pro's VOD `storage.json` seeds the same way and moves to its + own epoch with it. - demo/web stats keys include the epoch. Update `doc/concept/stats.md`, `doc/bin/relay/config.md`, and the relay stats config docs. diff --git a/quest/m0/broadcast-epoch/stats-split.md b/quest/m0/broadcast-epoch/stats-split.md new file mode 100644 index 0000000000..d133da227f --- /dev/null +++ b/quest/m0/broadcast-epoch/stats-split.md @@ -0,0 +1,96 @@ +# [L] Stats totals and per-broadcast tracks + +## Goal + +Each node's stats broadcast publishes per-group cumulative totals that are +never pruned while the group is announced, plus one stats track per broadcast, served only when requested. +The per-path map tracks are gone. While a group stays announced, a reader +that misses a frame loses nothing: the next totals frame still carries +everything since the epoch began. + +## Plan + +Decided 2026-10-05 (planned from moq-dev/moq.pro#2202): + +- **Why.** Every counter is cumulative per entry, and an entry is pruned right + after the frame carrying its closing readout. A broadcast that starts and + ends inside a group a reader missed shows in no later frame, so a lagging + reader undercounts. The maps also grow with every live broadcast, and + readers like billing want only per-project sums. +- **Totals.** Per group broadcast (`[/]/node//@`), + per tier and role, cumulative within the [stats epoch](/quest/m0/broadcast-epoch/stats-epoch.md). + Every group announcement, and so every restart, starts a new epoch counted + from zero; nothing is serialized to disk. `Registry`'s unpruned lifetime totals + (`rs/moq-net/src/stats.rs`) are node-wide, per tier and role only, so they + can't be published as-is: keep the totals per group, folding an entry into + its group's total when it is pruned, or two projects sharing a tier would + merge. +- **Idle groups** (decided 2026-10-05). Today a group broadcast disappears + the moment it has no entries (`rs/moq-stats/src/produce.rs`). Instead it + stays announced for the stats linger (`quest/m0/stats-linger.md`, an m0 + quest planned in [#4843](https://github.com/moq-dev/moq/pull/4843); list it + under Related once it lands, not Required, so it does not gate the release) + after its last entry ends, so a group that returns within the linger + continues its totals. A zero linger is valid: a returning group then always + takes a new epoch. After the linger it unannounces and drops its + totals; a return announces under a new [epoch](/quest/m0/broadcast-epoch/stats-epoch.md) + counted from zero. Totals are cumulative and a lingering group's frames all + repeat its final totals, so a reader that misses every frame across the + linger loses that epoch's tail; billing under-bills by that tail, + consistent with the 0-bill baseline. Document that. Memory is bounded by active and + lingering groups. Test two groups sharing a tier, a group returning within + the linger (same epoch, totals continue), and one returning after it (new + epoch from zero). +- **Per-broadcast tracks.** A reader that wants one broadcast subscribes to + its track; nothing is produced for a broadcast no one requests. The track + carries that broadcast's cumulative counters and finishes after its closing + readout. A request after the closing readout gets the zeroed track below, + so a zeroed per-broadcast track is not authoritative; totals carry the + real sums. Pick the track naming while implementing, but it must be + injective: tiers and broadcast paths both contain `/`, so no plain + concatenation of the two parses back to one pair. +- A broadcast's counters start at zero. A request for a broadcast with no + entry (never seen, or already pruned) is held open with zeroed counters + until one appears, as an unrecorded tier is today, and is reclaimed when + its last consumer leaves. +- The tier-sized caps (`MAX_REQUESTED_TRACKS` is 64) are too small for a page + of visible broadcasts. Give per-broadcast tracks their own cap per group + broadcast, sized (or configurable) for every reader the aggregator fans in, + not one page. Refuse requests beyond it with a typed error the reader + retries, rather than a parked track that reads as zero. Tier requests park + instead (`MAX_PARKED_REQUESTS`) because a collector treats a refusal as + final; per-broadcast readers must not. +- **Sessions** (decided 2026-10-05). `sessions.json` is keyed by auth root + and loses pruned roots the same way, so fold per-tier `Presence` into the + totals (session counts) and serve per-root detail as a requested track, + the same model as per-broadcast tracks. +- **Retire the map tracks** (`publisher.json`, `subscriber.json`, + `sessions.json`, and their `.json.z` siblings) in the same release. Decide + while implementing whether totals and per-broadcast tracks keep `.json.z` + siblings. +- Ship in the same breaking release as stats epochs, so consumers take one + stats path and wire change, not two. +- Update the aggregator, `doc/concept/stats.md`, and `demo/web/src/stats.ts` + (which reads the three maps) with the change. The + aggregator merges totals across nodes, and merges one broadcast's track + across nodes when a reader requests it, so a multi-node consumer serves the + same format it reads. + +Public API: `moq-stats` producer and consumer types. Wire: stats track names +and payloads. + +MoQ Pro adopts it when it pins the release: billing reads totals, its +Broadcasts page subscribes per visible broadcast, its `announced` probe +reads totals only, and its customer stats feed serves this format summed +across nodes. + +## Required + +- [Stats epochs](/quest/m0/broadcast-epoch/stats-epoch.md) - totals are cumulative within an epoch + +## Related + +- [Bounded stats aggregate](/quest/m0/broadcast-epoch/stats-aggregate-bound.md) - retired + nodes fold into a bounded total; per-epoch totals feed it +- [Media stats](/quest/m1/stats/README.md) - publisher and viewer media stats + stay hang tracks, separate from the relay's stats