From 8e8538391080b3cedc0434bea6b60535cf781322 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 17 Sep 2026 17:01:20 +0200 Subject: [PATCH 1/3] quest: plan LiveKit parity, the data convention, and SIP extensions Planned against a LiveKit survey (Sept 2026): the room SDK landed in #3634, so the client shim moves back to m2 on top of @moq/room, and the WebRTC bridge evaluation is abandoned because LiveKit Egress already pushes RTMP and SRT into gateways and a LiveKit-protocol server is the whole SFU. New: LiveKit Agents adapters (Python, then Node) over the framework's own custom AudioInput/AudioOutput seam, a data convention (a `data` catalog section plus one request/response shape) that teleop's robot primitive now requires instead of a private CatalogExt section, active speaker on rooms, SIP DTMF and transfer, and a C++ SDK over libmoq. Co-Authored-By: Claude Fable 5.1 --- quest/m2/README.md | 5 +++ quest/m2/data-convention.md | 56 +++++++++++++++++++++++++++++++ quest/m2/livekit-agents-node.md | 24 +++++++++++++ quest/m2/livekit-agents-python.md | 48 ++++++++++++++++++++++++++ quest/m2/livekit-shim.md | 40 ++++++++++++++++++++++ quest/m2/room-active-speaker.md | 18 ++++++++++ quest/m3/README.md | 5 +-- quest/m3/cpp-sdk.md | 25 ++++++++++++++ quest/m3/livekit-shim.md | 36 -------------------- quest/m3/livekit-webrtc-bridge.md | 27 --------------- quest/m3/sip-dtmf.md | 25 ++++++++++++++ quest/m3/sip-stack.md | 5 +-- quest/m3/sip-transfer.md | 28 ++++++++++++++++ quest/m3/teleop/robot.md | 15 ++++++--- 14 files changed, 285 insertions(+), 72 deletions(-) create mode 100644 quest/m2/data-convention.md create mode 100644 quest/m2/livekit-agents-node.md create mode 100644 quest/m2/livekit-agents-python.md create mode 100644 quest/m2/livekit-shim.md create mode 100644 quest/m2/room-active-speaker.md create mode 100644 quest/m3/cpp-sdk.md delete mode 100644 quest/m3/livekit-shim.md delete mode 100644 quest/m3/livekit-webrtc-bridge.md create mode 100644 quest/m3/sip-dtmf.md create mode 100644 quest/m3/sip-transfer.md diff --git a/quest/m2/README.md b/quest/m2/README.md index 6424bd4a59..9edabd630f 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -119,6 +119,11 @@ before format-specific metadata. Unrelated areas can proceed in parallel. - [JS track end](/quest/m2/2318-js-net-remaining-capability-gaps-vs-rs-moq-net-setup-role.md) - js/net declares a track end ahead of the live edge and observes the publisher's SUBSCRIBE_END - [E2EE](/quest/m2/e2ee/README.md) - TypeScript and Rust peers interoperate over encrypted broadcasts no relay can decrypt - [Processor](/quest/m2/processor/README.md) - a customer-run worker publishes an on-demand contribution with scoped access +- [LiveKit Agents adapter (Python)](/quest/m2/livekit-agents-python.md) - a `livekit-agents` voice session runs against a MoQ relay through a MoQ audio input and output and an announce-prefix runner +- [LiveKit client shim](/quest/m2/livekit-shim.md) - a drop-in `livekit-client` facade over `@moq/room`, media and core events first +- [Data convention](/quest/m2/data-convention.md) - a `data` catalog section and one request/response shape, so rooms, teleop, and voice stop hand-rolling reply channels +- [Active speaker](/quest/m2/room-active-speaker.md) - audio level and speaking signals on every room member +- [LiveKit Agents adapter (Node)](/quest/m2/livekit-agents-node.md) - the Python adapter ported to `@livekit/agents` - [#3056](/quest/m2/3056-watch-video-decoder-captures-the-rewind-generation-at.md) - watch: the video decoder resets on a declared discontinuity - [#933](/quest/m2/933-video-rotation-metadata-not-propagated-from-mobile-camera.md) - the catalog rotation follows the live camera's orientation - [#2075](/quest/m2/2075-mirror-catalog-reservation-gating-in-moq-hang-js-hang.md) - @moq/publish gates the first catalog snapshot until every reserved track is described diff --git a/quest/m2/data-convention.md b/quest/m2/data-convention.md new file mode 100644 index 0000000000..537decde00 --- /dev/null +++ b/quest/m2/data-convention.md @@ -0,0 +1,56 @@ +# [L] Data convention + +## Goal + +One convention for application data beside media, so rooms, teleoperation, +voice agents, and any app stop hand-rolling it: a `data` section in the hang +catalog that makes a JSON or binary track discoverable and typed, and one +request/response shape with correlation ids carried over those tracks. +Implemented in `js/hang`, `rs/hang`, and `moq-json`, adopted by `@moq/room` +chat, and written into `draft-lcurley-moq-hang`. + +Today the catalog root is audio and video only, `moq-json` carries snapshot, +stream, and window modes with no way to advertise them, and three consumers +each invented a reply channel: the Pipecat transport's fixed-name +`transcript.json.z` track, teleop's proposed `rpc` tracks, and pronto's +`status` echoing a command `sequence`. + +## Plan + +- Catalog: a `data` map in the root beside `video` and `audio`, keyed by + track name, each entry carrying `mode` (`snapshot`, `stream`, `window`, or + `datagram`), a free-form `schema` identifier the app owns, and the track + `Info` the reader needs (`timescale`, `priority`, `latency_max`). Rust + keeps `CatalogExt` for app-private sections; this is the shared, + browser-visible one. +- Request/response: a caller publishes a `stream`-mode track of + `{id, method, params}` and the callee answers on its own `stream`-mode + track of `{id, ok, result}` or `{id, error}`. The callee advertises its + response track in its catalog; the caller is discovered from an announce + prefix, the direction teleop and Voice already use (operator publishes, + robot subscribes; `request/` and `response/` subtrees). Timeouts are the + caller's, and a lagged reader is told so; recovery after a reconnect + belongs to the application, never to a retry loop in the library. +- Reliability is what `moq-net` gives: a group is one QUIC stream, so frames + in it are ordered and exactly once for a reader that keeps up, and + `MAX_GROUP_CACHE` bounds how far behind one may fall. Say that in the docs + instead of promising delivery. +- Non-goals: byte-stream file transfer, delivery receipts, per-participant + addressing (a path and a scoped token already do that), and a cross-host + timebase (teleop's correlation quest). +- Adopt it in-tree in the same change: `@moq/room` `Chat` declares its + window track in the section, and the JS and Rust examples + (`rs/moq-native/examples/chat.rs`, `rs/moq-json/examples/telemetry.rs`) + move onto it. The draft gains the section and the request/response shape. +- Tests: catalog round trip in both languages, a request answered across a + local relay, a lagged caller surfacing the error, and an unknown `mode` + refused at parse. + +## Related + +- [Robot teleoperation primitive](/quest/m3/teleop/robot.md) - the `rpc`, + `telemetry`, and `command` tracks are the first non-room consumer +- [LiveKit client shim](/quest/m2/livekit-shim.md) - `publishData`, + streams, and RPC map onto this +- [Catalog track identity](/quest/m3/catalog-tracks.md) - whatever it + decides about immutable definitions applies to `data` entries too diff --git a/quest/m2/livekit-agents-node.md b/quest/m2/livekit-agents-node.md new file mode 100644 index 0000000000..0ea7dee0d0 --- /dev/null +++ b/quest/m2/livekit-agents-node.md @@ -0,0 +1,24 @@ +# [M] LiveKit Agents adapter (Node) + +## Goal + +The Python adapter's design ported to `@livekit/agents`: an `AudioInput` +(`ReadableStream`) and `AudioOutput` (`captureFrame`, `flush`, +`clearBuffer`) over `@moq/net` and `@moq/hang`, plus the same announce-prefix +runner, published from `js/` as `@moq/livekit-agents`. Done when the stock +Node voice-agent example answers a browser publisher through a local relay. + +## Plan + +- `AgentSession.start({agent})` builds `RoomIO` only when a room is passed + (`agent_session.ts`), and their tests assign `session.input.audio` + directly, so the shape is the same as Python. `@livekit/rtc-node` stays a + dependency for `AudioFrame`. +- Reuse the Python adapter's track naming, barge-in replacement, RTVI turn + mapping, and tests one for one, so a browser publisher is agnostic to which + runtime answers. + +## Required + +- [LiveKit Agents adapter (Python)](/quest/m2/livekit-agents-python.md) - + settles the design this ports diff --git a/quest/m2/livekit-agents-python.md b/quest/m2/livekit-agents-python.md new file mode 100644 index 0000000000..12c8facd2d --- /dev/null +++ b/quest/m2/livekit-agents-python.md @@ -0,0 +1,48 @@ +# [M] LiveKit Agents adapter (Python) + +## Goal + +A `livekit-agents` voice agent runs against a MoQ relay instead of a LiveKit +room, with its STT, LLM, TTS, turn detection, and plugins unchanged: a +`moq-livekit-agents` package in `py/` supplies an `AudioInput` and +`AudioOutput` over MoQ and a runner that starts one `AgentSession` per +announced request broadcast and publishes its response. Done when a stock +LiveKit voice-agent example answers a browser publisher through a local relay +with only the I/O wiring changed. + +## Plan + +- The framework already runs without a room. `AgentSession.start(agent)` + with `session.input.audio` and `session.output.audio` assigned is how its + own test harness (`tests/fake_io.py`) and `console` mode + (`cli/tcp_console.py`) work, and `get_job_context(required=False)` keeps a + session independent of a Worker. Subclass `voice/io.py`'s `AudioInput` + (async iterator of `rtc.AudioFrame`) and `AudioOutput` (`capture_frame`, + `flush`, `clear_buffer`, and `on_playback_finished` when the track drains). +- Media the way Pipecat's MoQ transport does it (`pipecat.transports.moq`): + the `moq` package encodes and decodes Opus inside the FFI, the agent's + audio is one hang audio track replaced by a fresh one on barge-in so + catalog-aware players follow the successor, and frames cross as int16 at + the session's sample rate. `livekit-rtc` stays a dependency for the frame + type; no WebRTC connection is ever opened. +- The runner replaces dispatch: consume an announce prefix, start a session + per `request/` broadcast, publish at `response/`, end the session on + unannounce. The disjoint subtrees are the loop guard. A scoped token + (subscribe `request/**`, publish `response/**`) is the whole credential; + there is no registration and no dispatch API. +- Client turn signals ride the same RTVI JSON track the Pipecat transport + uses, mapped onto `session.interrupt()` and `session.commit_user_turn()`, + so one browser publisher works with either framework's agent. +- The README states what `RoomIO` provides and this does not: participant + attributes, `lk.chat` text input, transcription forwarding, `SessionHost`, + and RPC. Text and transcript surfaces follow the + [data convention](/quest/m2/data-convention.md) when it lands. +- Tests: a fake `AgentSession` I/O round trip over a local relay, barge-in + replacing the output track, and unannounce ending the session. Pin + `livekit-agents` to the 1.8 line and record the `io.py` surface it relies + on, since that module is not covered by their stability promise. + +## Related + +- [LiveKit Agents adapter (Node)](/quest/m2/livekit-agents-node.md) +- [LiveKit client shim](/quest/m2/livekit-shim.md) - the client-side half diff --git a/quest/m2/livekit-shim.md b/quest/m2/livekit-shim.md new file mode 100644 index 0000000000..3a1a154501 --- /dev/null +++ b/quest/m2/livekit-shim.md @@ -0,0 +1,40 @@ +# [L] LiveKit client shim + +## Goal + +A drop-in `livekit-client`-compatible JS package (e.g. `@moq/livekit`) that +runs a multi-participant room entirely over MoQ. The v1 surface is media plus +core events: Room connect/disconnect, local and remote participants, +camera/mic/screenshare publish, auto-subscribe, and the TrackSubscribed event +family; data surfaces (publishData, streams, RPC) are stubbed. The token slot +takes an ordinary moq-auth token, no LiveKit JWT parsing. Done when an +off-the-shelf LiveKit JS sample runs against a MoQ relay with only the import +and the connect URL/token changed. + +## Plan + +- The shim is a LiveKit-API facade over `@moq/room` (`js/room`, landed in + #3634): the room is a path prefix in the connection URL and token root, + participants are discovered from the announce stream, identity is the path + before `camera.hang` and `screen.hang`, and a screenshare's + announce/unannounce is its lifecycle. The shim groups the two broadcasts + per identity into one RemoteParticipant and maps catalog entries to + TrackPublications. +- Build on `@moq/publish` and `@moq/watch`. LiveKit quality hints + (setVideoQuality, adaptive settings) map to the receiver-driven pixel + target, or no-op gracefully. +- v1 is identity-only: `participant.identity` comes from the path and muted + state derives from catalog track presence. Names and coarse state come + from `@moq/room`'s `hang/*.json` metadata in a follow-up, not a rival + scheme; `ActiveSpeakersChanged` waits on + [active speaker](/quest/m2/room-active-speaker.md). +- Tokens come from `@moq/room` `claims()`: publish under `/**` + only, so participants cannot publish at each other's paths. +- `publishData`, text and byte streams, and RPC throw a clear + not-implemented error in v1; they map onto the + [data convention](/quest/m2/data-convention.md) afterwards. + +## Related + +- [LiveKit Agents adapter (Python)](/quest/m2/livekit-agents-python.md) - + the agent-side half of the same migration diff --git a/quest/m2/room-active-speaker.md b/quest/m2/room-active-speaker.md new file mode 100644 index 0000000000..4dbc59dbab --- /dev/null +++ b/quest/m2/room-active-speaker.md @@ -0,0 +1,18 @@ +# [S] Active speaker + +## Goal + +`@moq/room` and `moq-room` expose who is talking: every `Remote` and the +`Local` carry an audio level signal and a debounced `speaking` boolean, and +the `Room` exposes the ordered active-speaker set, so a grid can highlight +the speaker and the LiveKit shim can emit `ActiveSpeakersChanged`. + +## Plan + +- Measure on the decoded audio the member already plays (`js/watch`'s + audio path in `Remote`; the capture path in `Local`), not on the wire, so + a muted member never reads as speaking and nothing new crosses the relay. +- Level is a smoothed RMS in dBFS; `speaking` uses an attack and release + threshold with a short hold, exposed through the existing `@moq/signals` + idiom. Rust mirrors the same numbers in `moq-room`. +- The demo at `demo/web/src/meet.html` highlights the loudest tile as proof. diff --git a/quest/m3/README.md b/quest/m3/README.md index 4f316d313b..1327411827 100644 --- a/quest/m3/README.md +++ b/quest/m3/README.md @@ -23,8 +23,8 @@ worth settling now. Deferral does not by itself abandon a feature. - [iOS capture](/quest/m3/mobile-capture-ios.md) - camera and screen capture if the mobile ownership decision selects Rust - [Android capture](/quest/m3/mobile-capture-android.md) - NDK/JNI capture and codec backends if the mobile ownership decision selects Rust - [Mobile completion](/quest/m3/mobile-completion.md) - verify the selected native/mobile path before closing #700 +- [C++ SDK](/quest/m3/cpp-sdk.md) - an idiomatic wrapper over libmoq's C ABI, with the OBS plugin as its first consumer - [Linux OBS GPU input](/quest/m3/obs-linux-gpu.md) - publish OBS compositor frames without CPU readback on a validated Linux graphics/encoder combination -- [LiveKit client shim](/quest/m3/livekit-shim.md) - a media compatibility facade over the room SDK - [QUIC capacity probing](/quest/m3/quic-probe.md) - compare useful early retransmissions with padding and no probing - [Embedded video](/quest/m3/video-embedded.md) - EGL import in the renderer, so moq-video presents on a Pi @@ -42,7 +42,8 @@ worth settling now. Deferral does not by itself abandon a feature. - [#1838](/quest/m3/1838-tr-101-290-monitoring-requirements-broadcast-contribution.md) - TR 101 290 monitoring: requirements (broadcast/contribution health metrics) - [Teleoperation](/quest/m3/teleop/README.md) - MoQ carries robot video down and control up on one session as a library capability - [SIP media stack](/quest/m3/sip-stack.md) - terminate one inbound SIP audio call leg and expose it as Opus frames +- [SIP DTMF](/quest/m3/sip-dtmf.md) - RFC 4733 keypad events become a data track beside the caller's audio +- [SIP transfer](/quest/m3/sip-transfer.md) - blind transfer over REFER and warm transfer over a second leg, driven through the data convention - [Carrier voice](/quest/m3/carrier-voice/README.md) - determine whether MoQ should be the call fabric for programmable carrier voice -- [LiveKit WebRTC bridge](/quest/m3/livekit-webrtc-bridge.md) - a go/no-go verdict, backed by a spike, on per-track LiveKit-to-MoQ bridging - [Vision worker](/quest/m3/processor-vision.md) - a documented customer-run vision worker proves the processor contract - [Common Access Tokens](/quest/m3/cat/README.md) - a moq-transport client presents a CAT in SETUP and `moq auth serve` admits it with the scope its `moqt` claim names diff --git a/quest/m3/cpp-sdk.md b/quest/m3/cpp-sdk.md new file mode 100644 index 0000000000..f849a3c975 --- /dev/null +++ b/quest/m3/cpp-sdk.md @@ -0,0 +1,25 @@ +# [M] C++ SDK + +## Goal + +C++ developers (robotics tooling, engines, native apps) get a thin, +idiomatic wrapper over `libmoq`'s C ABI: RAII handles, `std::span` payloads, +callbacks or futures for announce and subscribe, and hang audio, video, and +data tracks, installed as a CMake package beside `moq.pc`. The OBS plugin +(`cpp/obs`) moves onto it as the first consumer, and the wrapper's surface +tracks the Go, Swift, and Kotlin bindings one for one. + +## Plan + +- Header-first in `cpp/moq`, generated where the C ABI already carries the + shape (cbindgen output in `rs/libmoq`), handwritten where ownership or + callbacks need C++ semantics. C++20, no exceptions across the boundary. +- Ship through `libmoq`'s existing release pipeline (`build.sh`, CMake + config), with a smoke that publishes and watches a clock through a local + relay on Linux, macOS, and Windows. +- Unity is deliberately not a target; a C# binding is a separate decision. + +## Required + +- [#2152](/quest/m2/2152-libmoq-c-abi-catch-up-with-the-moq-ffi-surface.md) - + the C ABI has to carry sessions before a wrapper is worth publishing diff --git a/quest/m3/livekit-shim.md b/quest/m3/livekit-shim.md deleted file mode 100644 index 1e1253f46b..0000000000 --- a/quest/m3/livekit-shim.md +++ /dev/null @@ -1,36 +0,0 @@ -# [L] LiveKit client shim - -## Goal - -A drop-in `livekit-client`-compatible JS package (e.g. `@moq/livekit`) that -runs a multi-participant room entirely over MoQ. The v1 surface is media plus -core events: Room connect/disconnect, local and remote participants, -camera/mic/screenshare publish, auto-subscribe, and the TrackSubscribed event -family; data surfaces (publishData, streams, RPC) are stubbed. The token slot -takes an ordinary moq-auth token, no LiveKit JWT parsing. Done when an -off-the-shelf LiveKit JS sample runs against a MoQ relay with only the import -and the connect URL/token changed. - -## Plan - -- The shim is a LiveKit-API facade over `@moq/room`, which carries hang.live's convention: the room is a path prefix in the connection URL and token root, - participants are discovered from the bare announce stream, identity is the - next path segment, and each participant publishes `/camera` - (camera + mic, hd/sd renditions) and `/screen` (screenshare, - whose announce/unannounce is the screenshare lifecycle). The shim groups - the two paths per identity into one RemoteParticipant and maps catalog - entries to TrackPublications. -- Build on `@moq/publish` and `@moq/watch`. LiveKit quality hints - (setVideoQuality, adaptive settings) map to the receiver-driven pixel - target, or no-op gracefully. -- v1 is identity-only: `participant.identity` comes from the path and muted - state derives from catalog track presence. Names and coarse state are a - follow-up wired to the room SDK's `hang/*.json` metadata, not a rival - scheme. -- Recommend tokens scoped to `put: /` so participants cannot - publish at each other's paths (hang.live grants `put` on the whole room - subtree today). - -## Related - -- [WebRTC bridge evaluation](/quest/m3/livekit-webrtc-bridge.md) diff --git a/quest/m3/livekit-webrtc-bridge.md b/quest/m3/livekit-webrtc-bridge.md deleted file mode 100644 index a36d29916d..0000000000 --- a/quest/m3/livekit-webrtc-bridge.md +++ /dev/null @@ -1,27 +0,0 @@ -# [M] LiveKit WebRTC bridge evaluation - -## Goal - -A go/no-go verdict, backed by a working spike, on bridging LiveKit rooms to -MoQ per-track over WebRTC using LiveKit's Rust SDK: a native participant -joins the room and republishes its tracks into MoQ, optionally the reverse. -The verdict decides whether this becomes a maintained gateway. - -## Plan - -- The zero-code paths already exist and bound the value: LiveKit Egress - pushes RTMP/SRT into the existing gateways today (composited and - transcoded), and `moq export rtc --connect` publishes WHIP into LiveKit - Ingress. LiveKit has no WHIP/WHEP egress, so per-track LiveKit to MoQ - requires joining the room as a participant via livekit rust-sdks - (Apache 2.0, no Chrome/GStreamer). -- Spike it in-tree: a rust-sdks participant subscribes to all tracks and - republishes via moq-net + moq-mux, with codec passthrough where possible - (Opus/VP8/VP9; H.264 through the Annex-B importer). -- Assess dependency weight of the webrtc stack, per-track fidelity versus the - Egress paths, simulcast handling, and where a real gateway would live (edge - versus standalone). - -## Related - -- [LiveKit client shim](/quest/m3/livekit-shim.md) diff --git a/quest/m3/sip-dtmf.md b/quest/m3/sip-dtmf.md new file mode 100644 index 0000000000..dc027c7f6a --- /dev/null +++ b/quest/m3/sip-dtmf.md @@ -0,0 +1,25 @@ +# [M] SIP DTMF + +## Goal + +`moq-sip` surfaces the caller's keypad: RFC 4733 telephone-events on the +negotiated RTP leg become a `stream`-mode data track beside the caller's +audio, declared in the catalog under the data convention, and the embedder +can send digits back on an outbound leg. In-band tone detection stays out. + +## Plan + +- Negotiate `telephone-event/8000` in the SDP answer beside Opus and G.711; + decode event, end bit, and duration into one JSON frame per digit + (`{digit, duration_ms}`) with the RTP timestamp mapped onto the audio + clock so a transcript can interleave them. +- Sending: the embedder writes the same frames and the crate emits the RTP + events with the standard three end packets. +- Test against a softphone and a PBX that sends both RFC 4733 and no DTMF + at all (the track exists but stays empty). + +## Required + +- [SIP media stack](/quest/m3/sip-stack.md) - the leg this extends +- [Data convention](/quest/m2/data-convention.md) - the track shape and + its catalog entry diff --git a/quest/m3/sip-stack.md b/quest/m3/sip-stack.md index 422399ce35..45c786adb9 100644 --- a/quest/m3/sip-stack.md +++ b/quest/m3/sip-stack.md @@ -11,8 +11,7 @@ consumable by an embedder. ## Plan -- Open with an evaluation, spike-backed like the - [WebRTC bridge verdict](/quest/m3/livekit-webrtc-bridge.md): drive a real +- Open with a spike-backed evaluation: drive a real inbound call through the candidate full Rust SIP stacks (ezk-sip, rvoip, and whatever else is current) and adopt the one that holds up. Fallback if none do: an existing parser crate (e.g. rsip) for message/SDP syntax plus a @@ -31,3 +30,5 @@ consumable by an embedder. so every embedder gets answer-with-silence for free. - The SIP edge gateway and line provisioning that consume this crate as an inbound-call product are moq.pro (downstream) work. +- [SIP DTMF](/quest/m3/sip-dtmf.md) and [SIP transfer](/quest/m3/sip-transfer.md) - + the extensions that lift the v1 boundaries above diff --git a/quest/m3/sip-transfer.md b/quest/m3/sip-transfer.md new file mode 100644 index 0000000000..de682df423 --- /dev/null +++ b/quest/m3/sip-transfer.md @@ -0,0 +1,28 @@ +# [M] SIP transfer + +## Goal + +`moq-sip` moves a live call: a blind transfer sends REFER for a target URI +and follows the NOTIFY subscription to completion, and a warm transfer holds +the caller, originates a second leg to the target, and bridges the two legs +when the embedder says so. The embedder drives both through the same +request/response track the data convention defines, so an agent or an app +can transfer without SIP-shaped code. + +## Plan + +- Blind: REFER with `Refer-To`, handle the implicit subscription and the + `sipfrag` NOTIFYs, and end the original dialog on success. Refuse REFER + from the far end (405) in v1. +- Warm: reuse the origination quest's outbound leg; hold is re-INVITE with + `a=sendonly`; bridging is the embedder swapping which leg's Opus feeds + which, not a media mixer. +- Test blind against a PBX that accepts REFER and one that rejects it, and + warm end to end with a softphone as the target. + +## Required + +- [SIP media stack](/quest/m3/sip-stack.md) +- [SIP call origination](/quest/m3/carrier-voice/sip-originate.md) - warm + transfer needs the outbound leg +- [Data convention](/quest/m2/data-convention.md) - the control channel diff --git a/quest/m3/teleop/robot.md b/quest/m3/teleop/robot.md index a4c98c9e0a..8803bdb5d7 100644 --- a/quest/m3/teleop/robot.md +++ b/quest/m3/teleop/robot.md @@ -61,11 +61,12 @@ The framing is where the guarantee lives, not the subscription flags: ### Contents -- The catalog section, through `moq-mux`'s `CatalogExt` and - `RenditionConfig`. hang stays media-only; the `CatalogExt` doc example is - already a `telemetry` section (`rs/moq-mux/src/catalog/tracks.rs`, with a - `gps` rendition in its tests), so this is the designed seam and needs no hang - schema change. +- The catalog section, through the shared + [data convention](/quest/m2/data-convention.md): `telemetry`, `command`, + and `rpc` are `data` entries the browser package can discover, which + supersedes the earlier plan of an app-private `CatalogExt` section + (`rs/moq-mux/src/catalog/tracks.rs`), and the `rpc` tracks are its + request/response shape rather than a teleop-only one. - Announce-prefix fan-in, generalised from `rs/moq-boy/src/input.rs`. - The two delivery classes, as `moq-json`'s snapshot and stream modes with the group structure and `Info::latency_max` each one needs. @@ -81,6 +82,10 @@ Port `moq-boy` onto the crate in the same change, as the no-arbitration case. It is the only existing consumer, and if the abstraction cannot express crowd control then it is the wrong abstraction. +## Required + +- [Data convention](/quest/m2/data-convention.md) - the catalog section and rpc shape the crate adopts + ## Related - [arbitration](/quest/m3/teleop/arbitration.md) - which controller is obeyed From cb014003696307f5b995bd6f4fa1e3a0a4112b0b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 17 Sep 2026 17:09:36 +0200 Subject: [PATCH 2/3] quest: rescope the data convention onto the json and binary sections Codex: the catalog already carries json and binary sections (#3109), so the convention is the window mode, the request/response envelope, and adoption, not a second root section. Active speaker measures before the playback mute gate, and the shim throws on quality hints it cannot honor. Co-Authored-By: Claude Fable 5.1 --- quest/m2/README.md | 2 +- quest/m2/data-convention.md | 82 ++++++++++++++++----------------- quest/m2/livekit-shim.md | 9 ++-- quest/m2/room-active-speaker.md | 12 +++-- quest/m3/teleop/robot.md | 14 +++--- 5 files changed, 63 insertions(+), 56 deletions(-) diff --git a/quest/m2/README.md b/quest/m2/README.md index 9edabd630f..1e816442ca 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -121,7 +121,7 @@ before format-specific metadata. Unrelated areas can proceed in parallel. - [Processor](/quest/m2/processor/README.md) - a customer-run worker publishes an on-demand contribution with scoped access - [LiveKit Agents adapter (Python)](/quest/m2/livekit-agents-python.md) - a `livekit-agents` voice session runs against a MoQ relay through a MoQ audio input and output and an announce-prefix runner - [LiveKit client shim](/quest/m2/livekit-shim.md) - a drop-in `livekit-client` facade over `@moq/room`, media and core events first -- [Data convention](/quest/m2/data-convention.md) - a `data` catalog section and one request/response shape, so rooms, teleop, and voice stop hand-rolling reply channels +- [Data convention](/quest/m2/data-convention.md) - one request/response shape over the catalog's `json` and `binary` sections, plus the `window` mode, so rooms, teleop, and voice stop hand-rolling reply channels - [Active speaker](/quest/m2/room-active-speaker.md) - audio level and speaking signals on every room member - [LiveKit Agents adapter (Node)](/quest/m2/livekit-agents-node.md) - the Python adapter ported to `@livekit/agents` - [#3056](/quest/m2/3056-watch-video-decoder-captures-the-rewind-generation-at.md) - watch: the video decoder resets on a declared discontinuity diff --git a/quest/m2/data-convention.md b/quest/m2/data-convention.md index 537decde00..a99ca76376 100644 --- a/quest/m2/data-convention.md +++ b/quest/m2/data-convention.md @@ -1,50 +1,50 @@ -# [L] Data convention +# [M] Data convention ## Goal -One convention for application data beside media, so rooms, teleoperation, -voice agents, and any app stop hand-rolling it: a `data` section in the hang -catalog that makes a JSON or binary track discoverable and typed, and one -request/response shape with correlation ids carried over those tracks. -Implemented in `js/hang`, `rs/hang`, and `moq-json`, adopted by `@moq/room` -chat, and written into `draft-lcurley-moq-hang`. +One request/response convention over the catalog's existing `json` and +`binary` data sections (#3109), so rooms, teleoperation, voice agents, and +any app stop hand-rolling reply channels: a caller's `stream`-mode track of +`{id, method, params}` answered on the callee's `stream`-mode track of +`{id, ok, result}` or `{id, error}`, with the callee advertised in its +catalog and the caller discovered from an announce prefix. The `window` +mode moq-json already implements joins the catalog's known modes, every +in-tree data track declares itself, and the hang draft carries all of it. -Today the catalog root is audio and video only, `moq-json` carries snapshot, -stream, and window modes with no way to advertise them, and three consumers -each invented a reply channel: the Pipecat transport's fixed-name -`transcript.json.z` track, teleop's proposed `rpc` tracks, and pronto's -`status` echoing a command `sequence`. +Today three consumers each invented a reply channel: the Pipecat +transport's fixed-name `transcript.json.z` track, teleop's proposed `rpc` +tracks, and pronto's `status` echoing a command `sequence`. `@moq/room`'s +chat window is not in the catalog at all. ## Plan -- Catalog: a `data` map in the root beside `video` and `audio`, keyed by - track name, each entry carrying `mode` (`snapshot`, `stream`, `window`, or - `datagram`), a free-form `schema` identifier the app owns, and the track - `Info` the reader needs (`timescale`, `priority`, `latency_max`). Rust - keeps `CatalogExt` for app-private sections; this is the shared, - browser-visible one. -- Request/response: a caller publishes a `stream`-mode track of - `{id, method, params}` and the callee answers on its own `stream`-mode - track of `{id, ok, result}` or `{id, error}`. The callee advertises its - response track in its catalog; the caller is discovered from an announce - prefix, the direction teleop and Voice already use (operator publishes, - robot subscribes; `request/` and `response/` subtrees). Timeouts are the - caller's, and a lagged reader is told so; recovery after a reconnect - belongs to the application, never to a retry loop in the library. -- Reliability is what `moq-net` gives: a group is one QUIC stream, so frames - in it are ordered and exactly once for a reader that keeps up, and - `MAX_GROUP_CACHE` bounds how far behind one may fall. Say that in the docs - instead of promising delivery. -- Non-goals: byte-stream file transfer, delivery receipts, per-participant - addressing (a path and a scoped token already do that), and a cross-host - timebase (teleop's correlation quest). -- Adopt it in-tree in the same change: `@moq/room` `Chat` declares its - window track in the section, and the JS and Rust examples - (`rs/moq-native/examples/chat.rs`, `rs/moq-json/examples/telemetry.rs`) - move onto it. The draft gains the section and the request/response shape. -- Tests: catalog round trip in both languages, a request answered across a - local relay, a lagged caller surfacing the error, and an unknown `mode` - refused at parse. +- Modes: add `window` (the retained run moq-json's `js/json/src/window` + writes) to `KnownMode` in `js/hang/src/catalog/mode.ts` and + `rs/hang/src/catalog/mode.rs`, so a chat window is readable by a generic + consumer. Datagram delivery is a track `Info` property, not a mode, and + stays out of the catalog. +- Request/response: `moq-json` gains typed `Request`/`Response` producers + and consumers over its stream mode in JS and Rust: the caller keeps the + correlation id and its own timeout, the callee answers each id once, and + a lagged reader surfaces the error rather than retrying. Discovery is the + direction teleop and Voice already use: the callee's response track is a + `json` entry with a `schema` naming the method set, and callers are found + under an announce prefix (operator publishes, robot subscribes; + `request/` and `response/` subtrees). +- Reliability is what `moq-net` gives (a group is one QUIC stream; a reader + that falls past `MAX_GROUP_CACHE` is told so). The docs say that instead + of promising delivery. Non-goals: byte-stream file transfer, delivery + receipts, per-participant addressing beyond a path and a scoped token, + and a cross-host timebase. +- Adopt in the same change: `@moq/room` `Chat` declares its `window` track + in the `json` section; `rs/moq-native/examples/chat.rs` and + `rs/moq-json/examples/telemetry.rs` declare theirs; the draft gains the + `window` mode and the request/response shape. A catalog without data + sections parses unchanged, which the existing `deserialize_section` + leniency already guarantees. +- Tests: `window` round trip in both languages, a request answered across a + local relay, a lagged caller surfacing the error, and an unknown mode + still passing through verbatim. ## Related @@ -53,4 +53,4 @@ each invented a reply channel: the Pipecat transport's fixed-name - [LiveKit client shim](/quest/m2/livekit-shim.md) - `publishData`, streams, and RPC map onto this - [Catalog track identity](/quest/m3/catalog-tracks.md) - whatever it - decides about immutable definitions applies to `data` entries too + decides about immutable definitions applies to data entries too diff --git a/quest/m2/livekit-shim.md b/quest/m2/livekit-shim.md index 3a1a154501..6be912da7f 100644 --- a/quest/m2/livekit-shim.md +++ b/quest/m2/livekit-shim.md @@ -20,9 +20,12 @@ and the connect URL/token changed. announce/unannounce is its lifecycle. The shim groups the two broadcasts per identity into one RemoteParticipant and maps catalog entries to TrackPublications. -- Build on `@moq/publish` and `@moq/watch`. LiveKit quality hints - (setVideoQuality, adaptive settings) map to the receiver-driven pixel - target, or no-op gracefully. +- Build on `@moq/publish` and `@moq/watch`. LiveKit quality hints map to + the receiver-driven pixel target where they can (`setVideoQuality` picks + the rendition, `adaptiveStream` follows the rendered size); a hint the + target cannot express (`setVideoFPS`, per-layer bitrate caps) throws a + clear unsupported error rather than no-oping, so an app never believes a + limit is active. - v1 is identity-only: `participant.identity` comes from the path and muted state derives from catalog track presence. Names and coarse state come from `@moq/room`'s `hang/*.json` metadata in a follow-up, not a rival diff --git a/quest/m2/room-active-speaker.md b/quest/m2/room-active-speaker.md index 4dbc59dbab..7a6095c730 100644 --- a/quest/m2/room-active-speaker.md +++ b/quest/m2/room-active-speaker.md @@ -9,10 +9,14 @@ the speaker and the LiveKit shim can emit `ActiveSpeakersChanged`. ## Plan -- Measure on the decoded audio the member already plays (`js/watch`'s - audio path in `Remote`; the capture path in `Local`), not on the wire, so - a muted member never reads as speaking and nothing new crosses the relay. +- Measure on decoded audio, independent of the member's playback mute: + `Member.muted` defaults to true and gates only the output today, so the + level analysis runs on the decoded frames before that gate (or keeps the + decoder subscribed while the sink is silent), and a muted-for-me member + still reads as speaking. Measuring on the wire is not an option since the + relay never decodes. - Level is a smoothed RMS in dBFS; `speaking` uses an attack and release threshold with a short hold, exposed through the existing `@moq/signals` idiom. Rust mirrors the same numbers in `moq-room`. -- The demo at `demo/web/src/meet.html` highlights the loudest tile as proof. +- The demo at `demo/web/src/meet.html` highlights the loudest tile as proof, + with every remote left muted. diff --git a/quest/m3/teleop/robot.md b/quest/m3/teleop/robot.md index 8803bdb5d7..44c1bd1627 100644 --- a/quest/m3/teleop/robot.md +++ b/quest/m3/teleop/robot.md @@ -61,12 +61,12 @@ The framing is where the guarantee lives, not the subscription flags: ### Contents -- The catalog section, through the shared - [data convention](/quest/m2/data-convention.md): `telemetry`, `command`, - and `rpc` are `data` entries the browser package can discover, which - supersedes the earlier plan of an app-private `CatalogExt` section - (`rs/moq-mux/src/catalog/tracks.rs`), and the `rpc` tracks are its - request/response shape rather than a teleop-only one. +- The catalog section: `telemetry`, `command`, and `rpc` are entries in + the catalog's `json` section (#3109), which the browser package can + discover, superseding the earlier plan of an app-private `CatalogExt` + section (`rs/moq-mux/src/catalog/tracks.rs`); the `rpc` tracks are the + shared [data convention](/quest/m2/data-convention.md)'s request/response + shape rather than a teleop-only one. - Announce-prefix fan-in, generalised from `rs/moq-boy/src/input.rs`. - The two delivery classes, as `moq-json`'s snapshot and stream modes with the group structure and `Info::latency_max` each one needs. @@ -84,7 +84,7 @@ control then it is the wrong abstraction. ## Required -- [Data convention](/quest/m2/data-convention.md) - the catalog section and rpc shape the crate adopts +- [Data convention](/quest/m2/data-convention.md) - the request/response shape the `rpc` tracks adopt ## Related From 1ebeba4fa336bb4bf767f38681008936e019278f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 17 Sep 2026 17:25:11 +0200 Subject: [PATCH 3/3] quest: window is JSON-only, native chat adopts, doc/concept, shim data Codex round two: split mode support by section so a binary window entry never reads as supported, declare moq_room's native chat too, update doc/concept with the hang catalog change, scope active speaker to @moq/room with a moq-audio meter for native, and give the shim's data surfaces their own quest. Co-Authored-By: Claude Fable 5.1 --- quest/m2/README.md | 3 ++- quest/m2/data-convention.md | 31 +++++++++++++++++++------------ quest/m2/livekit-shim-data.md | 26 ++++++++++++++++++++++++++ quest/m2/livekit-shim.md | 4 ++-- quest/m2/room-active-speaker.md | 15 ++++++++++----- 5 files changed, 59 insertions(+), 20 deletions(-) create mode 100644 quest/m2/livekit-shim-data.md diff --git a/quest/m2/README.md b/quest/m2/README.md index 1e816442ca..aaf63ea5af 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -121,8 +121,9 @@ before format-specific metadata. Unrelated areas can proceed in parallel. - [Processor](/quest/m2/processor/README.md) - a customer-run worker publishes an on-demand contribution with scoped access - [LiveKit Agents adapter (Python)](/quest/m2/livekit-agents-python.md) - a `livekit-agents` voice session runs against a MoQ relay through a MoQ audio input and output and an announce-prefix runner - [LiveKit client shim](/quest/m2/livekit-shim.md) - a drop-in `livekit-client` facade over `@moq/room`, media and core events first +- [LiveKit shim data surfaces](/quest/m2/livekit-shim-data.md) - publishData, text streams, and RPC on the data convention - [Data convention](/quest/m2/data-convention.md) - one request/response shape over the catalog's `json` and `binary` sections, plus the `window` mode, so rooms, teleop, and voice stop hand-rolling reply channels -- [Active speaker](/quest/m2/room-active-speaker.md) - audio level and speaking signals on every room member +- [Active speaker](/quest/m2/room-active-speaker.md) - audio level and speaking signals on every `@moq/room` member, and a `moq-audio` meter for native apps - [LiveKit Agents adapter (Node)](/quest/m2/livekit-agents-node.md) - the Python adapter ported to `@livekit/agents` - [#3056](/quest/m2/3056-watch-video-decoder-captures-the-rewind-generation-at.md) - watch: the video decoder resets on a declared discontinuity - [#933](/quest/m2/933-video-rotation-metadata-not-propagated-from-mobile-camera.md) - the catalog rotation follows the live camera's orientation diff --git a/quest/m2/data-convention.md b/quest/m2/data-convention.md index a99ca76376..4e826e0bbf 100644 --- a/quest/m2/data-convention.md +++ b/quest/m2/data-convention.md @@ -19,10 +19,14 @@ chat window is not in the catalog at all. ## Plan - Modes: add `window` (the retained run moq-json's `js/json/src/window` - writes) to `KnownMode` in `js/hang/src/catalog/mode.ts` and - `rs/hang/src/catalog/mode.rs`, so a chat window is readable by a generic - consumer. Datagram delivery is a track `Info` property, not a mode, and - stays out of the catalog. + writes) as a JSON-only known mode. `KnownMode` in + `js/hang/src/catalog/mode.ts` and `Mode` in `rs/hang/src/catalog/mode.rs` + are shared by both sections today, and `@moq/binary` and `moq-binary` + implement only snapshot and stream, so support detection splits by + section (`jsonModeSupported` and `binaryModeSupported`, or per-section + known sets) and a binary `window` entry stays unreadable rather than + appearing supported. Datagram delivery is a track `Info` property, not a + mode, and stays out of the catalog. - Request/response: `moq-json` gains typed `Request`/`Response` producers and consumers over its stream mode in JS and Rust: the caller keeps the correlation id and its own timeout, the callee answers each id once, and @@ -36,12 +40,15 @@ chat window is not in the catalog at all. of promising delivery. Non-goals: byte-stream file transfer, delivery receipts, per-participant addressing beyond a path and a scoped token, and a cross-host timebase. -- Adopt in the same change: `@moq/room` `Chat` declares its `window` track - in the `json` section; `rs/moq-native/examples/chat.rs` and - `rs/moq-json/examples/telemetry.rs` declare theirs; the draft gains the - `window` mode and the request/response shape. A catalog without data - sections parses unchanged, which the existing `deserialize_section` - leniency already guarantees. +- Adopt in the same change: `@moq/room` `Chat` and its native twin + `moq_room::chat::Publisher` (`rs/moq-room/src/chat.rs`) declare their + `window` track in the `json` section, so Rust and JavaScript room + catalogs agree; `rs/moq-native/examples/chat.rs` and + `rs/moq-json/examples/telemetry.rs` declare theirs. The draft gains the + `window` mode and the request/response shape, and `doc/concept`'s + catalog page gains both, per the cross-package checklist for an `rs/hang` + catalog change. A catalog without data sections parses unchanged, which + the existing `deserialize_section` leniency already guarantees. - Tests: `window` round trip in both languages, a request answered across a local relay, a lagged caller surfacing the error, and an unknown mode still passing through verbatim. @@ -50,7 +57,7 @@ chat window is not in the catalog at all. - [Robot teleoperation primitive](/quest/m3/teleop/robot.md) - the `rpc`, `telemetry`, and `command` tracks are the first non-room consumer -- [LiveKit client shim](/quest/m2/livekit-shim.md) - `publishData`, - streams, and RPC map onto this +- [LiveKit shim data surfaces](/quest/m2/livekit-shim-data.md) - + `publishData`, streams, and RPC map onto this - [Catalog track identity](/quest/m3/catalog-tracks.md) - whatever it decides about immutable definitions applies to data entries too diff --git a/quest/m2/livekit-shim-data.md b/quest/m2/livekit-shim-data.md new file mode 100644 index 0000000000..d9929db590 --- /dev/null +++ b/quest/m2/livekit-shim-data.md @@ -0,0 +1,26 @@ +# [M] LiveKit shim data surfaces + +## Goal + +`publishData`, text streams, and RPC in the LiveKit client shim work instead +of throwing: reliable `publishData` is a `stream`-mode `binary` track and +lossy a `snapshot`-mode one, text streams are `stream`-mode `json` tracks +keyed by topic, and RPC is the data convention's request/response pair +addressed by the callee's identity path. Byte streams with progress stay +unimplemented and throw. Done when a LiveKit sample that uses each of the +three runs against a relay unchanged. + +## Plan + +- Every track the shim publishes is declared in the participant's + `camera.hang` catalog, so a plain `@moq/room` client can read a LiveKit + app's data too. The README maps each LiveKit method to its track and its + delivery guarantee. +- Tests: a reliable and a lossy `publishData` round trip, a topic stream + read by a late joiner (only what the stream retains), and an RPC with a + timeout and an error reply. + +## Required + +- [LiveKit client shim](/quest/m2/livekit-shim.md) +- [Data convention](/quest/m2/data-convention.md) diff --git a/quest/m2/livekit-shim.md b/quest/m2/livekit-shim.md index 6be912da7f..8426713838 100644 --- a/quest/m2/livekit-shim.md +++ b/quest/m2/livekit-shim.md @@ -34,8 +34,8 @@ and the connect URL/token changed. - Tokens come from `@moq/room` `claims()`: publish under `/**` only, so participants cannot publish at each other's paths. - `publishData`, text and byte streams, and RPC throw a clear - not-implemented error in v1; they map onto the - [data convention](/quest/m2/data-convention.md) afterwards. + not-implemented error in v1; [shim data surfaces](/quest/m2/livekit-shim-data.md) + maps them onto the data convention afterwards. ## Related diff --git a/quest/m2/room-active-speaker.md b/quest/m2/room-active-speaker.md index 7a6095c730..23df3bd2da 100644 --- a/quest/m2/room-active-speaker.md +++ b/quest/m2/room-active-speaker.md @@ -2,10 +2,14 @@ ## Goal -`@moq/room` and `moq-room` expose who is talking: every `Remote` and the -`Local` carry an audio level signal and a debounced `speaking` boolean, and -the `Room` exposes the ordered active-speaker set, so a grid can highlight -the speaker and the LiveKit shim can emit `ActiveSpeakersChanged`. +`@moq/room` exposes who is talking: every `Remote` and the `Local` carry an +audio level signal and a debounced `speaking` boolean, and the `Room` +exposes the ordered active-speaker set, so a grid can highlight the speaker +and the LiveKit shim can emit `ActiveSpeakersChanged`. `moq-room` is +media-free (announce events and chat; decoded audio lives in `moq-audio` +and the application), so the native half is a level meter in `moq-audio` +over decoded PCM with the same thresholds, which a native app wires itself; +no room media layer is added. ## Plan @@ -17,6 +21,7 @@ the speaker and the LiveKit shim can emit `ActiveSpeakersChanged`. relay never decodes. - Level is a smoothed RMS in dBFS; `speaking` uses an attack and release threshold with a short hold, exposed through the existing `@moq/signals` - idiom. Rust mirrors the same numbers in `moq-room`. + idiom. The `moq-audio` meter uses the same numbers and a test vector + shared with the JS one. - The demo at `demo/web/src/meet.html` highlights the loudest tile as proof, with every remote left muted.