Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 12 additions & 6 deletions doc/bin/relay/auth.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,10 @@ refuses the grant),
`expires`
(optional unix seconds; the session closes then), `revalidate` (optional
seconds until the relay asks again), `tier` (optional label handed to
[stats](/bin/relay/config#stats)), and `peer` (optional; `true` marks another
[stats](/bin/relay/config#stats)), `peer` (optional; `true` marks another
relay, so what it announces counts as entering the cluster elsewhere, not as
ingest here). A 2xx with a grant admits. A 401 or 403 refuses.
ingest here), and `upstream` (optional; `true` marks that peer as an
[upstream link](/bin/relay/cluster#upstream-links), and requires `peer`). A 2xx with a grant admits. A 401 or 403 refuses.
Anything else at connect, a timeout, a 5xx, or an unparseable body, refuses and
logs an error; nothing is admitted because the server was down. A grant that
names nothing refuses, and one with `revalidate` but no `expires` is refused
Expand All @@ -66,7 +67,7 @@ clock, so later polls, outages, and wall-clock adjustments do not restart it.
**Revalidate and outage.** On the cadence the relay POSTs `revalidate` with the
same request. A grant applies: a changed `root` or `mounts`, or one that no longer covers
what the session holds closes it with `Unauthorized` (the live session is not
resized in place), and so does a flipped `peer`; a changed `tier` keeps the
resized in place), and so does a flipped `peer` or `upstream`; a changed `tier` keeps the
session and moves its stats: its presence counts under the new tier from then
on, as does each group and subscription it starts afterwards, while one already
in flight finishes where it began. A 401 or 403 closes the session now, as does a 2xx
Expand Down Expand Up @@ -384,7 +385,12 @@ bind = "/run/moq/internal.sock"
allow.uid = [1001]
```

Bind TCP to loopback or a private interface; it carries no peer identity. The
Unix socket is created mode `0666`, so gate it with a restrictive parent
directory or an explicit allowlist.
Bind plaintext TCP to loopback or a private interface; it carries no peer
identity. The Unix socket is created mode `0666`, so gate it with a restrictive
parent directory or an explicit allowlist.
These are native-only paths for gateways and stats publishers on the same host.

`listen.tcp.tls = true` serves the TCP listener over TLS with the listen
certificate instead, for `tls://` dials such as [cluster links](/bin/relay/cluster#tls-links)
that need neither QUIC nor a WebSocket. It still carries no peer identity: it
asks for no client certificate, so a peer on it presents a token, not mTLS.
54 changes: 53 additions & 1 deletion doc/bin/relay/cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,58 @@ For a full mesh, list every other relay, or serve the list from `connect_api`.
A session carries both directions, so one dial per pair is enough; listing a
pair on both sides opens a redundant second session.

## Upstream links

Mark a link `upstream` and this relay never offers it a route learned on
another upstream link. Everything else still transits, and clients still see
every route the relay knows. An edge marks its region's cores upstream, so it
reaches every core and every core reaches its clients, but it never carries
traffic from one core to another. A drone marks its CDN uplink upstream, so its
mesh reaches the CDN and the CDN reaches the mesh, but the drone never carries
traffic between CDN relays.

```toml
# edge.toml
[cluster]
connect = [
{ url = "https://core-a.example.com/", upstream = true },
{ url = "https://core-b.example.com/", upstream = true },
]
```

The mark belongs to the link, whichever side dialed. A peer this relay dials
is upstream when its `connect` or `connect_api` entry says so. A peer that
dials in is upstream when its [grant](/bin/relay/auth#the-contract) sets
`"upstream": true` beside `"peer": true`, so an edge whose auth server grants
that to core certificates treats a core that dials it as upstream too. A
relay that predates the mark treats every link as transit, so a cluster
migrates one region at a time.

Which relay dials which is still the peer list's job: there are no roles and
no topology check, so whatever generates the list applies the layout's rules.
Hiding a core from clients is admission, not routing: its auth refuses any
session that is not a cluster peer.

## TLS links

A link inside a region is rarely congested, so it can skip QUIC: a `tls://`
peer URL dials qmux over TLS on TCP, verified with the same `connect.tls`
settings as any other dial. The accepting relay serves it from its TCP
listener with TLS on. That listener asks for no client certificate, so a peer
on it authenticates with a token (`cluster.token`, `token`, or `?jwt=`), not
mTLS.

```toml
# core.toml
[listen.tcp]
bind = "[::]:4443"
tls = true

# edge.toml
[cluster]
connect = [{ url = "tls://core-a.internal:4443/", upstream = true }]
```

## Link costs

Add `?cost=N` to a peer URL to route by price instead of hop count. An unpriced
Expand All @@ -101,7 +153,7 @@ connect = ["https://sibling.same-dc/?cost=0", "https://us-east.example.com/?cost
```

The same policy reads as an object, which is the only form that accepts
`egress` and `token`. A bare URL stays valid, and an object whose `url` still
`egress`, `token`, and [`upstream`](#upstream-links). A bare URL stays valid, and an object whose `url` still
carries `?cost=` or `?jwt=` alongside those fields is rejected rather than
given a precedence a migration could silently get wrong.

Expand Down
3 changes: 2 additions & 1 deletion doc/bin/relay/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ root = ["peer-ca.pem"] # Optional: CAs for client certs (mTLS), re

[listen.tcp] # Plaintext qmux over TCP for trusted local workers.
bind = "127.0.0.1:4444"
# tls = true # Or: qmux over TLS (tls://) with the listen certificate, no client certs.

[listen.unix] # Plaintext qmux over a Unix socket, gated by peer credentials.
bind = "/run/moq/internal.sock"
Expand Down Expand Up @@ -151,7 +152,7 @@ See [Authentication](/bin/relay/auth).

```toml
[cluster]
connect = ["https://us-east.example.com/?cost=10"] # Peers to dial. ?cost prices the link, or use {url, cost, egress, token} objects.
connect = ["https://us-east.example.com/?cost=10"] # Peers to dial. ?cost prices the link, or use {url, cost, egress, token, upstream} objects.
node = "https://us-west.example.com/" # This relay's own URL.
connect_api = "https://api.example.com/peers" # Or fetch the peer list (JSON array of URLs and/or objects) live.
token = "cluster.jwt" # JWT for dials without an inline ?jwt=.
Expand Down
2 changes: 1 addition & 1 deletion doc/lib/rs/moq-auth.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Everything a party needs to ask for or answer an authorization on
answers the relay, in a service that mints tokens for clients, or in your own
accept loop that decides in process.

- **Request and grant**: `Request` is the JSON a relay POSTs per session event (`connect`, `revalidate`, `end`) with everything it knows: id, node, transport, addresses, SNI and ALPN, the raw path and query, the moq-transport SETUP `token` (its Token Type and bytes, base64url on the wire), the declared role, and the verified certificate facts. `Grant` is the answer: `publish` and `subscribe` pattern unions, an optional `root` alias, `mounts` that read a subtree from elsewhere on the origin, `expires`, `revalidate`, `tier`, and `peer`, which marks the session as a cluster peer so the routes it announces report `Source::Peer`. `Grant::validate` refuses a grant that names nothing, asks to be revalidated without a bound or at no interval, or has already expired; expiry is exact, with no grace for clock skew.
- **Request and grant**: `Request` is the JSON a relay POSTs per session event (`connect`, `revalidate`, `end`) with everything it knows: id, node, transport, addresses, SNI and ALPN, the raw path and query, the moq-transport SETUP `token` (its Token Type and bytes, base64url on the wire), the declared role, and the verified certificate facts. `Grant` is the answer: `publish` and `subscribe` pattern unions, an optional `root` alias, `mounts` that read a subtree from elsewhere on the origin, `expires`, `revalidate`, `tier`, `peer`, which marks the session as a cluster peer so the routes it announces report `Source::Peer`, and `upstream`, which marks that peer an [upstream link](/bin/relay/cluster#upstream-links). `Grant::validate` refuses a grant that names nothing, marks a non-peer upstream, asks to be revalidated without a bound or at no interval, or has already expired; expiry is exact, with no grace for clock skew.
- **Lease**: `lease::Producer` and `lease::Consumer` are the handle a session holds for its grant. The consumer reads the current grant, waits for a change, and learns why the lease ended; the producer updates and revokes. Either side's terminal call returns the reason the lease actually ended with, so whichever got there first is what both report. `Consumer::fixed` is a grant nobody drives. `Consumer::revalidate` nudges a re-check now and the producer observes it via `poll_revalidate` or `revalidate_requested`, which is how the relay's session push lands. Whoever runs the accept loop builds the producer, so an embedder decides in process with no trait and no HTTP. Enforcing `expires` is the holder's job; the `Client` driver also revokes at expiry so its `end` event goes out. `Grant::deadline()` (feature `tokio`, also enabled by `client` and `serve`) snapshots expiry on Tokio's clock. Call it once per accepted grant and retain the deadline; an expiry already past is now.
- **Client**: `Client::new(url, tls)` and `Client::connect(request)` drive a lease against an auth server over `https://`, `unix://`, or loopback `http://`: revalidate on cadence with jittered backoff through an outage until `expires`, revoke on a 401/403 or an invalid grant, and POST `end` with the reason, duration, and byte totals the session reported through `lease::Consumer::close` when it ended. Dropping the consumer reports zero bytes. `end.reason` is `dropped`, `expired`, `refused`, `invalid`, `narrowed`, `shutdown`, or the session's own classification.
- **Server**: `serve::Policy` and `serve::Server` (feature `serve`) are the reference auth server behind `moq auth serve`: a JWT from the `jwt` query or a type-0 SETUP token, verified against a key file, a `{kid}.jwk` directory, or a JWK Set file (a token of another type, or two different ones, is refused), an explicit grant for verified certificates, the anonymous permissions, a tier, the revalidation cadence, a default `expires`, and live session caps per token and per remote address. A token is authorized at the dialed path with `Claims::authorize`; residuals become the grant. `Server::router` is an axum `POST /` you can mount in your own service.
Expand Down
2 changes: 1 addition & 1 deletion doc/lib/rs/moq-net.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ above ([hang](/lib/rs/hang)); relays and CDNs implement only this.
- **Groups** are written frame by frame and delivered on independent streams. Old groups are cached for fetch-by-sequence; stale groups are skipped per the subscriber's budget.
- **Track ends**: `finish()` ends a track at its live edge, while `finish_at(n)` declares the exclusive end ahead of it and still accepts the groups below. A subscriber awaits it with `finished()`. A remote track ends only once every group below its end has arrived or was dropped; one reset before its header arrived is skipped after the subscription's max age on moq-lite (one second without one), or after one second on IETF.
- **Datagrams** send a single small frame unreliably on moq-lite 05+ and moq-transport.
- **Routes** record the relay hops and a cost, which is what the relay [cluster](/bin/relay/cluster) routes on. A hop of 0 marks the chain anonymous: `Route::is_anonymous()` is true, and that route ranks below every fully identified one. `Route::source()` says where a delivered route entered: `Source::Local`, or `Source::Peer(hop)` when a handle marked `origin::Producer::peer()` announced it. `origin::Consumer::local()` sees only the local ones.
- **Routes** record the relay hops and a cost, which is what the relay [cluster](/bin/relay/cluster) routes on. A hop of 0 marks the chain anonymous: `Route::is_anonymous()` is true, and that route ranks below every fully identified one. `Route::source()` says where a delivered route entered: `Source::Local`, or `Source::Peer(hop)` when a handle marked `origin::Producer::peer()` announced it. `origin::Consumer::local()` sees only the local ones. A handle marked `origin::Producer::upstream()` announces as a peer, and a consumer taken from it never sees a route another upstream handle announced, so a relay never transits between two upstream links.
- **Stats** counters per broadcast and session, drained by [`moq-stats`](https://docs.rs/moq-stats).

It runs over anything implementing `web_transport_trait::poll::Session`: noq, the
Expand Down
5 changes: 5 additions & 0 deletions js/auth/src/contract.ts
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,8 @@ export const GrantSchema = z
tier: z.optional(z.string()),
/** The session is a cluster peer (another relay): what it announces entered the cluster elsewhere. */
peer: z.optional(z.boolean()),
/** The peer is upstream: the relay never offers it a route learned from another upstream. Requires `peer`. */
upstream: z.optional(z.boolean()),
})
.check(
z.refine((grant) => (grant.publish?.length ?? 0) > 0 || (grant.subscribe?.length ?? 0) > 0, {
Expand All @@ -145,5 +147,8 @@ export const GrantSchema = z
z.refine((grant) => grant.revalidate === undefined || grant.expires !== undefined, {
message: "a grant that asks to be revalidated must expire",
}),
z.refine((grant) => !grant.upstream || grant.peer === true, {
message: "an upstream grant must be a peer",
}),
);
export type Grant = z.infer<typeof GrantSchema>;
3 changes: 2 additions & 1 deletion js/auth/src/interop.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -261,7 +261,7 @@ describe("Rust-serialized contract", () => {
const RUST_INVALID_END =
'{"id":"00ff","event":"end","reason":"invalid","duration":1.5,"bytes":{"sent":10,"received":20},"node":"relay-1","transport":"websocket","remote":"203.0.113.9:4433","path":"/demo/room","query":"jwt=abc"}';
const RUST_GRANT =
'{"publish":["alice/**"],"subscribe":["**"],"root":"pid/room","expires":4102444800,"revalidate":60,"tier":"websocket","peer":true}';
'{"publish":["alice/**"],"subscribe":["**"],"root":"pid/room","expires":4102444800,"revalidate":60,"tier":"websocket","peer":true,"upstream":true}';

test("parse a request a Rust relay sent", () => {
const request = RequestSchema.parse(JSON.parse(RUST_REQUEST));
Expand Down Expand Up @@ -298,6 +298,7 @@ describe("Rust-serialized contract", () => {
revalidate: 60,
tier: "websocket",
peer: true,
upstream: true,
});
expect(JSON.stringify(grant)).toBe(RUST_GRANT);
});
Expand Down
6 changes: 3 additions & 3 deletions quest/m1/cluster-routing/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,9 @@ vector the 2026-09-30 cache-tiers audit kept:
routes learned on another upstream link, and a route learned upstream keeps
a down-only mark across other links so a mesh with two uplinks cannot leak
CDN routes back into the CDN. An edge marks its core links upstream; a
Starlink drone marks its CDN link upstream. See
[Upstream links](/quest/m1/cluster-routing/transit.md).
Starlink drone marks its CDN link upstream. The per-link mark landed on
today's path vector (`doc/bin/relay/cluster.md`); the down-only mark lands
with [Routes and announces](/quest/m1/cluster-routing/routes.md).
- **One node id space, no cluster ids.** A customer cluster is nodes behind
upstream or trusted links. Loop detection by cluster id would drop a
partitioned swarm's traffic to itself through the CDN (BGP's partitioned-AS
Expand Down Expand Up @@ -127,7 +128,6 @@ Once every child has landed:

## Required

- [Upstream links](/quest/m1/cluster-routing/transit.md) - a link marked upstream never receives routes learned on another upstream link, which builds edge tiers and drone uplinks from one rule
- [Selection](/quest/m1/cluster-routing/selection.md) - a broadcast under overlapping prefixes routes to one origin deterministically, and same-path origins are one source
- [Multi-CDN endpoints](/quest/m1/cluster-routing/multi-cdn.md) - an endpoint holds sessions to several CDNs, uses its preferred one, and fails over to the next
- [Simulate the split](/quest/m1/cluster-routing/sim.md) - moq.pro's simulator compares the route layer with path vector before the wire is written
Expand Down
2 changes: 1 addition & 1 deletion quest/m1/cluster-routing/routes.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ Decided 2026-10-01 (see the [line's decisions](/quest/m1/cluster-routing/README.
No per-announce seqno.
- Down-only bit: set on a route learned on an upstream link, kept across
other links, and a route carrying it is never sent on an upstream link
([Upstream links](/quest/m1/cluster-routing/transit.md)).
(extending the `upstream` link mark in `doc/bin/relay/cluster.md`).
- A plain client with one link advertises a ROUTE for itself and the
ANNOUNCEs it publishes; its node id is scoped to its session (shared
identity across sessions is [Route trust](/quest/m3/route-trust.md)).
Expand Down
1 change: 0 additions & 1 deletion quest/m1/cluster-routing/selection.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,5 +59,4 @@ no timestamp rewind.

## Required

- [Upstream links](/quest/m1/cluster-routing/transit.md) - the edges and cores this spreads over
- [Wildcard](/quest/m0/wildcard/README.md) - the longest-prefix rule and pool spread this builds on
Loading
Loading