diff --git a/demo/web/src/meet.ts b/demo/web/src/meet.ts index ff5cdf1aaa..6a161b20e1 100644 --- a/demo/web/src/meet.ts +++ b/demo/web/src/meet.ts @@ -140,7 +140,7 @@ function join(): void { tile("local", name, localCanvas, true); localPreview = new Publish.Preview.Renderer({ canvas: localCanvas, - frame: local.cameraCapture.out.frame, + frames: local.cameraCapture.out.frames, display: local.cameraCapture.out.display, flip: true, }); diff --git a/doc/bin/relay/index.md b/doc/bin/relay/index.md index 46f403016e..580494c03d 100644 --- a/doc/bin/relay/index.md +++ b/doc/bin/relay/index.md @@ -47,6 +47,19 @@ Every option is also a `--flag` or `MOQ_*` environment variable, and [`demo/relay/`](https://github.com/moq-dev/moq/tree/main/demo/relay) has working configs for development, production, and a cluster. +## Embedding + +`Relay::load` is the embedder API: it binds listeners, resolves auth, and +builds the cluster. Mount extra routes on `web.routes()` and publish from +application workers on `cluster.origin`. `rs/moq-relay/tests/embed.rs` is +the small public example (custom `/app` plus an origin worker). + +`Relay` is `#[non_exhaustive]`. Destructure with `..`. Dropping `workers` +or `uring` releases the QUIC port while the rest still compiles; keep those +fields if `runtime.workers` is set. An owning runner for that hazard is +the relay-embedding quest on `dev`. Embedding is not a +[moq-dev/smoke](https://github.com/moq-dev/smoke) client. + ## Operate | Task | Guide | diff --git a/doc/lib/js/hang.md b/doc/lib/js/hang.md index 2c7fb3c541..04eb7da717 100644 --- a/doc/lib/js/hang.md +++ b/doc/lib/js/hang.md @@ -22,3 +22,23 @@ import * as Container from "@moq/hang/container"; Most apps never import it directly; the elements and `Broadcast` classes in the watch and publish packages do. Reach for it when hand-rolling a catalog or building a custom player. + +## Migrating + +A hang catalog is a JSON **Snapshot**, not one full root per frame. Read it +with `Json.Snapshot.Consumer` and `Catalog.RootSchema`, the way +`@moq/watch` does: + +```ts +const track = broadcast.track(Catalog.TRACK).subscribe({ priority: Catalog.PRIORITY.catalog }); +const catalog = new Json.Snapshot.Consumer({ + track, + schema: Catalog.RootSchema, +}); +const root = await catalog.next(); +``` + +Frame 0 of a group is a full catalog; later frames are RFC 7396 merge patches. +`JSON.parse` plus `RootSchema` on every frame is the old consumer and rejects +the deltas. Compressed catalogs use `Catalog.TRACK_COMPRESSED` and +`compression: true` on the same consumer. diff --git a/doc/lib/js/index.md b/doc/lib/js/index.md index 96c51cba85..505395938b 100644 --- a/doc/lib/js/index.md +++ b/doc/lib/js/index.md @@ -57,7 +57,11 @@ React and Solid adapters for the reactive state. Below the elements, `Watch.Broadcast` and `Publish.Broadcast` are the same pipelines without DOM, and `@moq/net` is the protocol itself. Examples: [`js/net/examples/`](https://github.com/moq-dev/moq/tree/main/js/net/examples) -covers connecting, publishing, subscribing, and discovery. +covers connecting, publishing, subscribing, and discovery. The reconnecting +`Connection` handle, catalog Snapshot reads, and stats Snapshot versus Window +are covered by [moq-dev/smoke](https://github.com/moq-dev/smoke)'s from-dev +channel (`./dev.sh`). See also the [net](/lib/js/net#migrating) and +[hang](/lib/js/hang#migrating) migration notes. ## Browser support diff --git a/doc/lib/js/net.md b/doc/lib/js/net.md index 82fcb73744..ae8207fdbc 100644 --- a/doc/lib/js/net.md +++ b/doc/lib/js/net.md @@ -50,3 +50,28 @@ Examples in [`js/net/examples/`](https://github.com/moq-dev/moq/tree/main/js/net/examples). Runs in the browser and, over WebSocket, in Node, Bun, and Deno; see [server-side](/lib/js/#server-side). + +The unpublished `dev` surface is proven by the from-dev channel in +[moq-dev/smoke](https://github.com/moq-dev/smoke) (`./dev.sh`), not by an +in-tree packaged fixture. + +## Migrating + +- **`Connection.Reload` / `Connection.Shared` are gone.** `new Connection({ url })` + is the reconnecting handle: one origin and one reconnect loop per relay URL. + `Connection.connect` is still the one-shot session. `closed` settles when + *this handle* is released, not when a session drops. The failure that stopped + retrying the current URL is `error`; `url.set(next)` recovers the same handle + (credential refresh). +- **Do not call `consume` on the reconnecting handle.** `Established.consume(path)` + stays on a one-shot session. A `Connection` exposes `origin`; resolve with + `origin.request(path)` (swaps on a republish) and discover with + `connection.announced(prefix)`. Announce events carry `pattern`, not `path`. +- **`broadcast.track(name).subscribe(opts)` is the public read.** + `broadcast.subscribe(name)` is the wire-layer helper. Hang catalog reads go + through the track handle, then `ordered()` when a codec needs sequence order. + `ordered()` takes the subscription over: `recvGroup` throws afterwards. +- **JSON is three modes**, in [`@moq/json`](https://www.npmjs.com/package/@moq/json): + `Snapshot` (lossy latest-value, merge-patch deltas), `Stream` (lossless + append-log), `Window` (retained range). Pick the mode; do not parse every + frame as a full document. Live stats are Snapshot. Billing rollups are Window. diff --git a/doc/lib/rs/index.md b/doc/lib/rs/index.md index 201cc02d81..dfe7720cd2 100644 --- a/doc/lib/rs/index.md +++ b/doc/lib/rs/index.md @@ -34,6 +34,14 @@ The reference implementation. Every crate is on ## Quick start `moq-tokio` configures the endpoint; `moq-net` does the protocol. +`moq-native` is a tombstone: replace `moq-native` with `moq-tokio` in +`Cargo.toml` and `moq_native` with `moq_tokio` in source. There is no +compatibility shim. + +A hang catalog or live stats track is `moq_json::snapshot`; a retained +rollup is `moq_json::window`. Subscribe with +`broadcast.track(name)?.subscribe(...)`, then the JSON consumer, not a +generic frame reader. ```rust // The Origin is the local hub: the session fills it with remote broadcasts diff --git a/js/json/README.md b/js/json/README.md index d7c6ed8f37..bb46cb8132 100644 --- a/js/json/README.md +++ b/js/json/README.md @@ -7,12 +7,13 @@ [![npm version](https://img.shields.io/npm/v/@moq/json)](https://www.npmjs.com/package/@moq/json) [![TypeScript](https://img.shields.io/badge/TypeScript-ready-blue.svg)](https://www.typescriptlang.org/) -JSON publishing over [Media over QUIC](https://moq.dev/) tracks, in two modes: +JSON publishing over [Media over QUIC](https://moq.dev/) tracks, in three modes: -- **`Snapshot`**: lossy. One JSON value updated over time; a consumer only gets the most recent value. Intermediate updates are collapsed and older groups are dropped. +- **`Snapshot`**: lossy. One JSON value updated over time; a consumer only gets the most recent value. Intermediate updates are collapsed and older groups are dropped. Hang catalogs and live stats use this. - **`Stream`**: lossless. An ordered append-log of self-contained records; every record is preserved and delivered in order, nothing is ever superseded. +- **`Window`**: a bounded run of records, appended to the back and dropped from the front. A late reader is restated what is still retained (billing rollups). -Pick `Snapshot` when consumers care about "what is the value now" (a catalog, a status document) and `Stream` when they care about every record (an event log, a media timeline). +Pick `Snapshot` when consumers care about "what is the value now", `Stream` when they care about every record of an unbounded log, and `Window` when old records retire. Do not parse every Snapshot frame as a full document: frame 0 is a snapshot and later frames are RFC 7396 merge patches. ## Quick Start diff --git a/quest/m1/README.md b/quest/m1/README.md index bf8a7394a3..c0d9cc2d25 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -28,7 +28,6 @@ does not require it. - [FFI frame cursor](/quest/m1/api-ffi-frame-cursor.md) - empty groups and cancelled reads do not become false EOF or lost frames - [Subscription bounds](/quest/m1/api-subscription-bounds.md) - local and requested ranges use consistent exclusive ends - [Publisher finish borrows](/quest/m1/api-finish-borrow.md) - finish borrows the handle so abort can still run after a clean end -- [External API proof](/quest/m1/api-release-proof.md) - packaged callers exercise real moq.pro use cases and record each audit finding's disposition - [Monotonic timeline](/quest/m1/monotonic-timeline.md) - a marker group of one empty frame declares a break and moves the live edge; producers refuse a rewind; consumers jump the playhead on an unproven hole and drop rewind detection - [Anonymous rank](/quest/m1/anonymous-route-rank.md) - moq-net: a route through an anonymous hop ranks below every identified route at any cost, and hop 0 travels the chain to say so - [A/V clock](/quest/m1/plan-av-clock.md) - the audio playhead drives Sync.reference while audio plays, through per-track sync handles diff --git a/quest/m1/api-finish-borrow.md b/quest/m1/api-finish-borrow.md index 60d87c4bd3..e1944ae0a8 100644 --- a/quest/m1/api-finish-borrow.md +++ b/quest/m1/api-finish-borrow.md @@ -62,4 +62,4 @@ then `just test smoke-full` because `moq-ffi` behavior changes. ## Related -- [External API proof](/quest/m1/api-release-proof.md) - records this disposition +- [Merge dev](/quest/m1/merge-dev.md) - records this disposition diff --git a/quest/m1/api-release-proof.md b/quest/m1/api-release-proof.md deleted file mode 100644 index 33c749277d..0000000000 --- a/quest/m1/api-release-proof.md +++ /dev/null @@ -1,79 +0,0 @@ -# [L] Prove the dev API through packaged external consumers - -## Goal - -Before publishing the dev API, representative external browser and native -callers build and exercise the intended contracts, and every finding in the -2026-09-12 audit has an explicit fix or deferral decision. - -## Plan - -The audit inspected dev `e2350b39a6ce9bd0734841fc4b4ce399ee195562`, fetched -from origin on 2026-09-12, against main and the actual consumer at -`/home/kixelated/work/moq.pro`. Consumer code was read without modification; -it targets an older API. Compile migrations alone are not upstream defects. - -Coverage included Rust origin/announce, sessions, track subscription/control, -group/frame ownership, timestamp invariants, bandwidth handles, JSON/binary -publishers, mux/hang boundaries, browser net/signals and JSON/binary wrappers, -watch/publish integration, shared FFI with C and native language wrappers, -and relay embedding. This is a source/API audit, not an exhaustive behavioral -proof of every exported symbol, platform backend, codec, or protocol version. - -New findings and recommendations, ordered by consequence: - -| Finding | Evidence status | Quest | -|---|---|---| -| FFI pending reads serialize independent datagram/group lanes | Reproduced; fixed: independent group and datagram lanes | completed | -| Shared/private connections disagree on credential-refresh recovery and terminal state | Reproduced; fixed: one URL-recovery contract, `closed` is handle disposal | completed | -| Relay embedding can discard newly added socket owners without a compile error | Source-traced; real edge embedding pattern | [Embedding](/quest/m1/api-relay-embedding.md) | -| FFI first-frame convenience treats empty groups as EOF and loses an acquired group on cancellation | Source-traced | [Frame cursor](/quest/m1/api-ffi-frame-cursor.md) | -| JSON/binary readers hide the subscription cleanup handle | Abandoned: finish must be `&mut` so abort can follow | deferred | -| Local inclusive ends cannot express the empty exclusive range | Existing C adapter documents the mismatch | [Bounds](/quest/m1/api-subscription-bounds.md) | -| Typed Getter input can be rejected solely for lacking an internal brand | Fixed: getter() reuses any conforming Getter | Fixed | -| JSON edit guard logs failed implicit publication | Fixed: `modify` refuses a closed track, a failed drop aborts it | completed | -| Terminal publisher methods inconsistently retain the caller's handle | Signature comparison; finish must borrow so abort can follow | [Finish borrows](/quest/m1/api-finish-borrow.md) | - -Recommend resolving behavioral failures and published contract choices before -merge. Cosmetic consistency can be deferred explicitly if the maintainer -accepts the later breaking change. The table is a disposition checklist, not -an automatic declaration that every proposed quest must be implemented first. -Record each disposition and its fixing revision in the merge proof. - -Build a small public fixture against packaged exports, outside workspace -hoisting and path aliases, modelling: - -- Connection, announcements, credential refresh, and publication replacement. - moq.pro `app/src/lib/live.svelte.ts:75,113` is the reference use case. -- Stats snapshots versus retained rollup groups; choose the intended mode - instead of mechanically replacing a removed read helper. -- Catalog-only reading with full snapshots followed by deltas. The old - consumer `app/src/lib/broadcastCatalog.svelte.ts:46` assumes every frame is - a full catalog. Use Json.Snapshot.Consumer and the catalog schema as the - upstream watch path does (`js/watch/src/broadcast.ts:304`); generic JSON - reads plus schema validation do not reconstruct merge patches. -- Embedded relay with custom routes and application workers, based on - moq.pro `rs/edge/src/main.rs:88,317`, through the settled owning API. - -Do not copy the private application into the public repo. Include dependency -deduplication in packaged validation: moq.pro already tests physical package -identity (`app/test/deps.test.ts`); the module-local Connection pool and private -net hooks deserve a real two-copy test. Duplicate-copy failure remains an -unverified hypothesis until reproduced, not an established audit defect. - -Update migration docs beside the affected APIs, including Connection exports, -broadcast.track(...).subscribe(...), Ordered readers, JSON modes, and the -moq_native to moq_tokio transition. Do not reintroduce obsolete wrappers to -avoid migration. Run the fixture in CI, record package versions and exact -revisions, and let moq.pro's separately owned pin/release process adopt the -proved surface. This quest does not bump packages or deploy the consumer. - -Public API: no additional API change beyond the fixing quests; fixture and -documentation prove the chosen surface. Wire: no new format. Use existing -browser/native integration recipes and cross-language smoke as appropriate. - -## Related - -- [Merge dev](/quest/m1/merge-dev.md) - records the final release and interop proof -- [C ABI parity](/quest/m2/2152-libmoq-c-abi-catch-up-with-the-moq-ffi-surface.md) - already owns C request/server omissions -- [C fetch](https://github.com/moq-dev/moq/blob/e2350b39a6ce9bd0734841fc4b4ce399ee195562/quest/m2/libmoq-fetch.md) - already owns the missing C fetch entry point diff --git a/quest/m1/merge-dev.md b/quest/m1/merge-dev.md index b5607aebc6..9139b8d945 100644 --- a/quest/m1/merge-dev.md +++ b/quest/m1/merge-dev.md @@ -45,10 +45,28 @@ merge for it. The rest of the archive line, wildcard resolution, and every additive quest that builds on dev-only code start on main afterwards from [m2](/quest/m2/README.md). +The 2026-09-12 external API audit is closed. Proof is +[moq-dev/smoke](https://github.com/moq-dev/smoke) `./dev.sh` (unpublished +`dev` checkout, not crates.io/npm latest) plus +`rs/moq-relay/tests/embed.rs` for custom routes and origin workers. +Embedding ownership of `workers`/`uring` stays on +[Relay embedding](/quest/m1/api-relay-embedding.md). + +| Finding | Disposition | Revision | +|---|---|---| +| FFI pending reads serialize independent datagram/group lanes | Fixed: independent group and datagram lanes | `1f7b2b45b` (#3651) | +| Shared/private connections disagree on credential-refresh recovery and terminal state | Fixed: one URL-recovery contract, `closed` is handle disposal | `b5a289a05` (#3636) | +| Relay embedding can discard newly added socket owners without a compile error | Deferred to [Embedding](/quest/m1/api-relay-embedding.md); `tests/embed.rs` covers routes and origin workers on the current load API | open | +| FFI first-frame convenience treats empty groups as EOF and loses an acquired group on cancellation | Deferred to [Frame cursor](/quest/m1/api-ffi-frame-cursor.md) | open | +| JSON/binary readers hide the subscription cleanup handle | Abandoned: finish must be `&mut` so abort can follow | `a8dbf886d` (#3637) | +| Local inclusive ends cannot express the empty exclusive range | Deferred to [Bounds](/quest/m1/api-subscription-bounds.md) | open | +| Typed Getter input can be rejected solely for lacking an internal brand | Fixed: `getter()` reuses any conforming Getter | `26b505995` (#3639) | +| JSON edit guard logs failed implicit publication | Fixed: `modify` refuses a closed track, a failed drop aborts it | `ff45019fc` (#3644) | +| Terminal publisher methods inconsistently retain the caller's handle | Deferred to [Finish borrows](/quest/m1/api-finish-borrow.md) | open | + ## Required - [m0](/quest/m0/README.md) - every release blocker lands or is punted before the merge -- [External API proof](/quest/m1/api-release-proof.md) - the packaged consumer fixture and explicit fix/deferral decisions must be recorded before merge - [Monotonic timeline](/quest/m1/monotonic-timeline.md) - so a shed marker still jumps the playhead on a timestamp hole (#3291) - [Wildcard docs](/quest/m1/wildcard-docs.md) - the release that follows ships pattern advertisements, so their docs ship in it diff --git a/rs/moq-relay/tests/embed.rs b/rs/moq-relay/tests/embed.rs new file mode 100644 index 0000000000..0fd30e1e8c --- /dev/null +++ b/rs/moq-relay/tests/embed.rs @@ -0,0 +1,150 @@ +//! An application embeds the relay: custom HTTP routes plus a worker that +//! publishes into [`Cluster::origin`]. +//! +//! Smoke cannot host this. It installs published clients, not the relay crate. +//! The owning runner that keeps `workers`/`uring` from being dropped is +//! [quest/m1/api-relay-embedding.md](../../../quest/m1/api-relay-embedding.md); +//! this test uses the current `Relay::load` pieces with `runtime.workers` unset, +//! so the `..` remainder does not hold bound QUIC sockets. + +use std::{net::TcpListener, time::Duration}; + +use axum::routing::get; +use moq_relay::{Config, PublicConfig, Relay}; +use moq_tokio::moq_net::{self, Hop, Timestamp}; + +const TIMEOUT: Duration = Duration::from_secs(10); + +fn free_tcp_port() -> u16 { + let probe = TcpListener::bind("127.0.0.1:0").expect("bind probe"); + let port = probe.local_addr().expect("local addr").port(); + drop(probe); + port +} + +fn client() -> moq_tokio::Client { + let mut config = moq_tokio::connect::Config::default(); + config.tls.insecure = Some(true); + config.once = Some(true); + config.websocket.delay = Duration::ZERO.into(); + config.bind = Some("127.0.0.1:0".parse().expect("parse bind")); + config.init(Default::default()).expect("client init") +} + +#[tokio::test] +async fn embedder_custom_route_and_origin_worker() { + let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); + + let port = free_tcp_port(); + let mut config = Config::default(); + config.listen.bind = Some("127.0.0.1:0".to_string()); + config.listen.tls.generate = vec!["localhost".into()]; + config.web.ws = true; + config.web.http.listen = Some(format!("127.0.0.1:{port}").parse().expect("parse listen")); + #[allow(deprecated)] + { + config.auth.public = Some(PublicConfig::Simple(vec![String::new()])); + } + + let Relay { + web, + cluster, + shutdown: _, + shutdown_trigger, + .. + } = Relay::load(config).await.expect("load relay"); + + let app = web.routes().route("/app", get(|| async { "app\n" })); + let (server_result_tx, mut server_result_rx) = tokio::sync::oneshot::channel(); + let web_handle = tokio::spawn(async move { + let _ = server_result_tx.send(web.serve(app).await); + }); + + let deadline = std::time::Instant::now() + Duration::from_secs(5); + loop { + if tokio::net::TcpStream::connect(("127.0.0.1", port)).await.is_ok() { + break; + } + match server_result_rx.try_recv() { + Ok(Ok(())) => panic!("web server exited before listening"), + Ok(Err(err)) => panic!("web server failed before listening: {err:#}"), + Err(tokio::sync::oneshot::error::TryRecvError::Empty) => {} + Err(tokio::sync::oneshot::error::TryRecvError::Closed) => { + panic!("web server task ended before listening") + } + } + if std::time::Instant::now() >= deadline { + panic!("http listener never became ready on port {port}"); + } + tokio::time::sleep(Duration::from_millis(25)).await; + } + + let body = reqwest::get(format!("http://127.0.0.1:{port}/app")) + .await + .expect("fetch /app") + .text() + .await + .expect("read /app"); + assert_eq!(body, "app\n"); + + let mut broadcast = cluster.origin.create_broadcast("app.worker").expect("create broadcast"); + broadcast.announce(Default::default()).expect("announce"); + let mut track = broadcast.create_track("ping", None).expect("create track"); + let mut group = track.append_group().expect("append group"); + group + .write_frame(Timestamp::ZERO, b"pong".as_ref()) + .expect("write frame"); + group.finish().expect("finish group"); + + let url: url::Url = format!("ws://127.0.0.1:{port}/").parse().expect("parse url"); + let sub_origin = moq_tokio::origin::spawn(Hop::random()); + let sub_consumer = sub_origin.consume(); + let mut announcements = sub_consumer.announced(); + let session = tokio::time::timeout(TIMEOUT, client().with_subscriber(sub_origin).connect(url).established()) + .await + .expect("subscriber connect timeout") + .expect("subscriber connect failed"); + + let update = tokio::time::timeout(TIMEOUT, announcements.next()) + .await + .expect("announcement timeout") + .expect("origin closed"); + assert!(update.active, "expected announce, got retraction"); + let path = moq_net::Path::new(update.pattern.as_prefix().expect("prefix announcement")).to_owned(); + assert_eq!(path.as_str(), "app.worker"); + let bc = sub_consumer + .request_broadcast(&path) + .await + .expect("announced broadcast resolves"); + let mut track_sub = bc.track("ping").unwrap().subscribe(None).await.expect("subscribe"); + let mut group_sub = tokio::time::timeout(TIMEOUT, track_sub.recv_group()) + .await + .expect("recv_group timeout") + .expect("recv_group failed") + .expect("track closed prematurely"); + let frame = tokio::time::timeout(TIMEOUT, group_sub.read_frame()) + .await + .expect("read_frame timeout") + .expect("read_frame failed") + .expect("group closed prematurely"); + assert_eq!(&frame.payload[..], b"pong"); + + drop(session); + drop(track); + drop(broadcast); + shutdown_trigger.start(); + web_handle.abort(); + let _ = web_handle.await; + drop(cluster); + + let deadline = std::time::Instant::now() + Duration::from_secs(2); + loop { + if TcpListener::bind(("127.0.0.1", port)).is_ok() { + break; + } + if std::time::Instant::now() >= deadline { + panic!("stopping the embedder should release the HTTP listener"); + } + tokio::time::sleep(Duration::from_millis(25)).await; + } +} diff --git a/test/smoke/README.md b/test/smoke/README.md index acc4cc2de9..db543432b5 100644 --- a/test/smoke/README.md +++ b/test/smoke/README.md @@ -10,6 +10,12 @@ from the workspace source (`cargo`, `bun`, `uv`, `cc`) to catch *interop* regressions before anything is published. No apt/brew/npm/PyPI, and no distribution-mechanism matrix. +The unpublished `dev` API (reconnecting `Connection`, catalog Snapshot +deltas, stats Snapshot versus Window) is proven by smoke's **from-dev** +channel (`./dev.sh` in that repo), which consumes this checkout or git +`dev` instead of npm/crates.io latest. Embedded relay routes live in +`rs/moq-relay/tests/embed.rs`, not in smoke. + It stands up a `moq-relay`, then for each publisher language publishes an H.264 broadcast and confirms every subscriber sees data flowing before the timeout. Most subscribers check for a non-empty frame. The browser additionally verifies