Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/bin/relay/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,7 @@ prefix = ".stats" # Broadcasts appear under <prefix>/node/<no
interval = 1 # Seconds between snapshots.
node = "sjc/1" # Disambiguates relays sharing a cluster.
depth = 1 # Also bucket by the first N path segments (per tenant).
linger = "5m" # Keep an empty group's broadcast announced this long. Default.
```

Each node publishes `publisher.json`, `subscriber.json`, and `sessions.json`
Expand Down
8 changes: 6 additions & 2 deletions doc/concept/stats.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,12 @@ generate more stats.
group segment literally named `node` is ambiguous; don't use one.

At depth 0 the broadcast stays announced for the producer's life. At depth
1 or more, a group's broadcast is announced while that group has entries and
unannounced once it has none. Group numbers keep increasing across recreated
1 or more, a group's broadcast is announced while that group has entries, and
for a linger (five minutes by default) after its last one leaves. A group that
returns within the linger keeps its broadcast, so viewer churn doesn't
unannounce and re-announce it across the mesh; while it lingers empty, its
tracks hold `{}`. Once the linger elapses with the group still empty, the
broadcast is unannounced. Group numbers keep increasing across recreated
tracks and group broadcasts for the producer's life; they may have gaps. A
recreated compressed track starts a new group with a full snapshot, never a
delta whose compression state belonged to its previous writer.
Expand Down
2 changes: 1 addition & 1 deletion quest/m0/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ a published `@moq/watch` break.
- [noq reassembly cap](/quest/m0/noq-reassembly-cap.md) - iroh's upstream noq carries quinn's stream reassembly cap, once n0 releases it
- [qmux credit](/quest/m0/qmux-credit.md) - qmux returns connection credit for dropped and stopped streams and delivers its close frame, on both lines
- [Shared fronts](/quest/m0/shared-fronts.md) - viewer sessions share a front, so fronts scale with peers, not viewers, and viewer churn no longer leaks fronts
- [Stats linger](/quest/m0/stats-linger.md) - a grouped stats broadcast stays announced for a linger after its last session, so viewer churn stops re-announcing it across the mesh, backported to `release`
- [Stats linger](/quest/m0/stats-linger.md) - landed on `main`; the `release` backport remains, so moq.pro's grouped stats broadcasts stop re-announcing on viewer churn
- [Wildcard](/quest/m0/wildcard/README.md) - a relay resolves subscriptions against advertised prefixes, a service claims the prefix it could serve and refuses the rest instead of enumerating broadcasts, and the browser player treats a covering claim as availability
- [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - every first-party publisher that can restart mints a fresh `@<uuidv7>` epoch, viewers follow the newest live one, and bare names still resolve on every version
- [Audio jitter target](/quest/m0/audio-jitter-target/README.md) - the audio playout target is a measured estimate of arrival timing in both languages, not a round-trip guess
Expand Down
13 changes: 6 additions & 7 deletions quest/m0/broadcast-epoch/stats-split.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,11 @@ Decided 2026-10-05 (planned from moq-dev/moq.pro#2202):
can't be published as-is: keep the totals per group, folding an entry into
its group's total when it is pruned, or two projects sharing a tier would
merge.
- **Idle groups** (decided 2026-10-05). Today a group broadcast disappears
the moment it has no entries (`rs/moq-stats/src/produce.rs`). Instead it
stays announced for the stats linger (`quest/m0/stats-linger.md`, an m0
quest planned in [#4843](https://github.com/moq-dev/moq/pull/4843); list it
under Related once it lands, not Required, so it does not gate the release)
after its last entry ends, so a group that returns within the linger
continues its totals. A zero linger is valid: a returning group then always
- **Idle groups** (decided 2026-10-05). A group broadcast already stays
announced for the stats linger (`produce::Config::linger`, landed in
[#4871](https://github.com/moq-dev/moq/pull/4871)) after its last entry
ends. Today a path that left drops out of frames while the group lingers;
with totals, a group that returns within the linger continues its totals. A zero linger is valid: a returning group then always
takes a new epoch. After the linger it unannounces and drops its
totals; a return announces under a new [epoch](/quest/m0/broadcast-epoch/stats-epoch.md)
counted from zero. Totals are cumulative and a lingering group's frames all
Expand Down Expand Up @@ -90,6 +88,7 @@ across nodes.

## Related

- [Stats linger](/quest/m0/stats-linger.md) - the idle-group window these totals continue across
- [Bounded stats aggregate](/quest/m0/broadcast-epoch/stats-aggregate-bound.md) - retired
nodes fold into a bounded total; per-epoch totals feed it
- [Media stats](/quest/m1/stats/README.md) - publisher and viewer media stats
Expand Down
65 changes: 15 additions & 50 deletions quest/m0/stats-linger.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# [S] Stats broadcasts linger
# [XS] Stats broadcasts linger

## Goal

Expand All @@ -10,55 +10,20 @@ them, are unchanged.

## Plan

Decided 2026-10-05 in moq.pro's quest audit: approved as recommended. The
earlier plan was in the closed
[#4052](https://github.com/moq-dev/moq/pull/4052) beside tree-routed
announces; the linger is independent of that and lands alone. It sits in m0
as a standalone quest, not a release gate, because moq.pro's m0 stats-linger
quest waits on a `release` commit carrying it (decided 2026-10-05).

Data from moq.pro's live fleet: on 2026-09-29 one customer's node stats
broadcasts ended on 31 nodes at once every 2 to 4 minutes and returned 46 to
112 s later, and the fleet served about 148,000 `.stats` subscriptions against
7 media ones in that hour. Size the linger from those gaps, not the one minute
first proposed, and make it configurable.

- `moq-stats`'s producer unpublishes a group broadcast on the drain where its
group has no traffic or session rows (`publish` in
`rs/moq-stats/src/produce.rs`). Keep it, and the epoch and group sequence it
publishes under ([stats epochs](/quest/m0/broadcast-epoch/stats-epoch.md)),
until the linger elapses with the group still empty; a row returning
within the linger re-arms it. While it lingers empty, its live gauges and
session presence read zero, so a reader sees no stale live counters.
- Under one epoch totals never go backwards or count twice. On `main`, #4846
(stats split, idle groups, decided 2026-10-05) settles it: a group
returning within the linger keeps its epoch and its totals continue, and
the per-path maps are gone. The `release` backport keeps today's per-path
map, which drops a path's entry and totals when it leaves (`flush` in
`produce.rs`); there, either the path drops out of frames while the group
lingers so its return reads as a restart, or the producer carries its last
totals forward. Pick while building the backport and record why.
- Once the linger elapses, the group broadcast unannounces. A later return
re-announces it under a new epoch, with its group sequence and totals
counted from zero (decided 2026-10-05). This reverses stats epochs' one-epoch-per-producer
rule; [#4846](https://github.com/moq-dev/moq/pull/4846) writes that reversal
into the epoch line, so this quest only follows it.
- Time decisions use `max(wall, pts)`; tests mock time.
- Depth 0 already lives for the producer's life and is unchanged.

moq.pro tracks the `release` branch, so once this lands on `main` it is
backported to `release` as an additive PR. The backport is rewritten against
`release`'s `produce.rs`, which has no epochs: a returning group starts a new
broadcast from the wall-clock group seed of #4810 instead.

Test: a session closes and reopens within the linger with no unannounce,
and no path's total goes backwards or counts twice across the return; while
the group lingers empty, live gauges and presence are zero; the group unannounces after the linger, and
a return after that announces a new epoch counted from zero; the reported
totals match a run without the linger.

Public API: a linger knob on the stats producer config, surfaced in the relay
as `stats.linger` / `--stats-linger` beside `--stats-enabled`. Wire: none.
Landed on `main` in [#4871](https://github.com/moq-dev/moq/pull/4871):
`moq_stats::produce::Config::linger` (default 5 minutes), surfaced in the
relay as `stats.linger` / `--stats-linger`. While a group lingers empty its
tracks read `{}`, and a path that left drops out of frames, so its return
reads as a restart.

Remaining: backport #4871 to `release` as an additive cherry-pick PR, since
moq.pro tracks `release` and its m0 stats-linger quest waits on a `release`
commit carrying it (decided 2026-10-05). `release`'s `produce.rs` differs
from `main`'s only by the wall-clock group seed of #4810, which keeps group
numbers increasing across a return. Both branches still have the per-path
maps, so the backport keeps #4871's choice: the path drops out of frames while the
group lingers, rather than the producer carrying its last totals forward.
Delete this quest once the backport merges.

## Related

Expand Down
12 changes: 11 additions & 1 deletion rs/moq-relay/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -559,14 +559,15 @@ root = ["ca.pem"]
/// Bare defaults loaded from TOML survive when the CLI does not mention them.
#[test]
fn cli_does_not_clobber_toml_stats_enabled() {
let _env = EnvGuard::clear(&["MOQ_STATS_ENABLED", "MOQ_STATS_DEPTH"]);
let _env = EnvGuard::clear(&["MOQ_STATS_ENABLED", "MOQ_STATS_DEPTH", "MOQ_STATS_LINGER"]);

let toml = r#"
[stats]
enabled = true
interval = 5
node = "localhost"
depth = 2
linger = "2m"
"#;
let dir = std::env::temp_dir().join("moq-relay-config-test");
std::fs::create_dir_all(&dir).unwrap();
Expand All @@ -583,6 +584,15 @@ depth = 2
assert_eq!(config.stats.interval, 5);
assert_eq!(config.stats.node.as_deref(), Some("localhost"));
assert_eq!(config.stats.depth, 2);
assert_eq!(config.stats.linger(), Some(std::time::Duration::from_secs(120)));

let args = vec![
std::ffi::OsString::from("moq-relay"),
std::ffi::OsString::from(&path),
std::ffi::OsString::from("--stats-linger=30s"),
];
let config = Config::parse_and_merge(args).expect("config load");
assert_eq!(config.stats.linger(), Some(std::time::Duration::from_secs(30)));
}

/// Bare runtime defaults loaded from TOML survive when the CLI omits them.
Expand Down
3 changes: 3 additions & 0 deletions rs/moq-relay/src/settings.rs
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,9 @@ struct Stats {

#[usage(env = "MOQ_STATS_DEPTH", cli("--stats-depth"))]
depth: Option<u64>,

#[usage(env = "MOQ_STATS_LINGER", cli("--stats-linger"))]
linger: Option<String>,
}

#[derive(usage::Config)]
Expand Down
28 changes: 26 additions & 2 deletions rs/moq-relay/src/stats.rs
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,19 @@ pub struct Config {
setting = "stats.depth"
)]
pub depth: usize,

/// How long a group's stats broadcast stays announced after the group's
/// last session and traffic leave, e.g. "5m" or "30s". Defaults to 5
/// minutes. A group that returns within it keeps its broadcast, so viewer
/// churn doesn't unannounce and re-announce it across the mesh. Only applies
/// at `depth` 1 or more. See [`moq_stats::produce::Config::linger`].
#[usage(skip)]
#[serde(with = "crate::duration::serde_option")]
pub linger: Option<Duration>,

#[usage(long = "stats-linger", env = "MOQ_STATS_LINGER", setting = "stats.linger")]
#[serde(default, rename = "__cli_linger", skip_serializing_if = "Option::is_none")]
pub(crate) linger_arg: Option<crate::duration::Duration>,
}

impl Default for Config {
Expand All @@ -87,11 +100,18 @@ impl Default for Config {
interval: 1,
node: None,
depth: 0,
linger: None,
linger_arg: None,
}
}
}

impl Config {
/// The linger after command-line overrides; `None` keeps the producer's default.
pub(crate) fn linger(&self) -> Option<Duration> {
self.linger_arg.map(crate::duration::Duration::into_std).or(self.linger)
}

/// Build a [`moq_stats::Producer`] from this config, publishing on `origin`.
///
/// Returns a no-op producer when [`Self::enabled`] is false, so the relay can
Expand All @@ -107,13 +127,17 @@ impl Config {
let interval = Duration::from_secs(self.interval.max(1));
let node = self.node.clone().map(PathOwned::from);
let depth = self.depth;
tracing::info!(prefix, interval_secs = interval.as_secs(), node = ?node, depth, "stats publishing enabled");
let config = moq_stats::produce::Config::new()
let linger = self.linger();
tracing::info!(prefix, interval_secs = interval.as_secs(), node = ?node, depth, ?linger, "stats publishing enabled");
let mut config = moq_stats::produce::Config::new()
.with_origin(origin)
.with_prefix(prefix)
.with_interval(interval)
.with_node(node)
.with_depth(depth);
if let Some(linger) = linger {
config = config.with_linger(linger);
}
moq_stats::Producer::new(config)
}
}
Loading
Loading