Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions doc/bin/relay/cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,57 @@ hop from the relay it connects to, fresh for each connection, followed by a 0.
Its reconnect is therefore a new publisher downstream, a reprice on the same
connection stays in place, and the 0 keeps it ranked as anonymous.

## Edge and core roles

Set `cluster.role = "edge"` for relays that serve end users, or `"core"`
for the backbone. Leave it unset during migration to preserve the legacy mesh.
A core accepts only sessions whose auth grant sets `peer = true`, including
HTTP fetches and announcements. A verified certificate alone is not membership.
Edges refuse inbound peer sessions; they dial their region's cores themselves.

An edge's peer entries must declare `role = "core"` and use `tls://`.
An edge advertises only routes ingested locally, so a route received from one
core cannot reach another core through that edge. Cores may dial explicit cores
and legacy peers, but refuse peers declared as edges. A role-aware relay refuses
LAN discovery because its advertisements do not declare peer roles.

```toml
[cluster]
role = "edge"
connect = [
{ url = "tls://core-a.example:4444/", role = "core", token = "CLUSTER_JWT" },
{ url = "tls://core-b.example:4444/", role = "core", token = "CLUSTER_JWT" },
]
```

On each core, enable encrypted qmux on its TCP listener:

```toml
[listen.tcp]
bind = "0.0.0.0:4444"
tls = true

[listen.tls]
cert = "core.pem"
key = "core.key"

[cluster]
role = "core"
```

The `tls://` scheme requires an explicit port and uses `connect.tls` trust,
certificate, and hostname settings. It speaks qmux directly over TLS on TCP.
Its listener reports the TCP transport to the auth server, as both secure and
plaintext WebSocket listeners report WebSocket. Use peer grants authenticated
with cluster credentials. The TLS TCP listener refuses `listen.tls.root` because
qmux's TLS accept API does not expose the verified client certificate to the
relay's auth request. Use a separate QUIC listener for mTLS-authenticated peers.

Generate peer lists so each edge dials all of its region's cores, cores do not
link within a region, and inter-region cores dial only peers with lower names.
The relay validates role compatibility and follows the configured links; it
has no region inventory to generate those lists itself.

## Topology

List the peers each relay dials. That's the whole topology: a relay dials only
Expand Down
6 changes: 4 additions & 2 deletions doc/bin/relay/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,9 @@ key = "key.pem"
generate = ["localhost"] # Or: a self-signed cert for development.
root = ["peer-ca.pem"] # Optional: CAs for client certs (mTLS), reported to the auth server.

[listen.tcp] # Plaintext qmux over TCP for trusted local workers.
[listen.tcp] # Qmux over TCP. Plaintext by default.
bind = "127.0.0.1:4444"
# tls = true # tls:// on this port, using listen.tls certificates.

[listen.unix] # Plaintext qmux over a Unix socket, gated by peer credentials.
bind = "/run/moq/internal.sock"
Expand Down Expand Up @@ -139,7 +140,8 @@ See [Authentication](/bin/relay/auth).

```toml
[cluster]
connect = ["https://us-east.example.com/?cost=10"] # Peers to dial. ?cost prices the link, or use {url, cost, egress, token} objects.
# role = "edge" # Or "core"; unset keeps legacy mesh behavior.
connect = ["https://us-east.example.com/?cost=10"] # Peers to dial. ?cost prices the link, or use {url, role, cost, egress, token} objects.
node = "https://us-west.example.com/" # This relay's own URL.
connect_api = "https://api.example.com/peers" # Or fetch the peer list (JSON array of URLs and/or objects) live.
token = "cluster.jwt" # JWT for dials without an inline ?jwt=.
Expand Down
39 changes: 39 additions & 0 deletions rs/moq-net/benches/origin.rs
Original file line number Diff line number Diff line change
Expand Up @@ -477,9 +477,48 @@ fn bench_handoff(c: &mut Criterion) {
group.finish();
}

/// An edge's core-facing announce cursors export local ingress and exclude
/// core ingress. Sweep route count and core-facing cursors independently.
fn bench_edge_split_horizon(c: &mut Criterion) {
let mut group = c.benchmark_group("origin/edge_split_horizon");
for (publishers, cores) in [(100, 2), (1_000, 2), (1_000, 20)] {
for peer in [false, true] {
let label = if peer { "core_ingress" } else { "local_ingress" };
let id = BenchmarkId::new(label, format!("{publishers}p_{cores}c"));
group.bench_function(id, |b| {
let (producer, _driver) = origin::Producer::new(origin::Config::default());
let ingress = if peer {
producer.clone().peer()
} else {
producer.clone()
};
let _routes: Vec<_> = (0..publishers)
.map(|i| ingress.publish(format!("room/{i}"), origin::Route::default()).unwrap())
.collect();
let mut cursors: Vec<_> = (0..cores).map(|_| producer.consume().local().announced()).collect();
for cursor in &mut cursors {
while cursor.next().now_or_never().flatten().is_some() {}
}
b.iter(|| {
let broadcast = ingress.publish("room/incoming", origin::Route::default()).unwrap();
for cursor in &mut cursors {
assert_eq!(cursor.next().now_or_never().flatten().is_some(), !peer);
}
drop(broadcast);
for cursor in &mut cursors {
assert_eq!(cursor.next().now_or_never().flatten().is_some(), !peer);
}
});
});
}
}
group.finish();
}

criterion_group!(
benches,
bench_announce,
bench_edge_split_horizon,
bench_announce_mounted,
bench_announce_fleet,
bench_announce_duplicate,
Expand Down
Loading
Loading