quest: plan LiveKit parity, the data convention, and SIP extensions #3737
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| # [M] Data convention | ||
|
|
||
| ## Goal | ||
|
|
||
| One request/response convention over the catalog's existing `json` and | ||
| `binary` data sections (#3109), so rooms, teleoperation, voice agents, and | ||
| any app stop hand-rolling reply channels: a caller's `stream`-mode track of | ||
| `{id, method, params}` answered on the callee's `stream`-mode track of | ||
| `{id, ok, result}` or `{id, error}`, with the callee advertised in its | ||
| catalog and the caller discovered from an announce prefix. The `window` | ||
| mode moq-json already implements joins the catalog's known modes, every | ||
| in-tree data track declares itself, and the hang draft carries all of it. | ||
|
|
||
| Today three consumers each invented a reply channel: the Pipecat | ||
| transport's fixed-name `transcript.json.z` track, teleop's proposed `rpc` | ||
| tracks, and pronto's `status` echoing a command `sequence`. `@moq/room`'s | ||
| chat window is not in the catalog at all. | ||
|
|
||
| ## Plan | ||
|
|
||
| - Modes: add `window` (the retained run moq-json's `js/json/src/window` | ||
| writes) as a JSON-only known mode. `KnownMode` in | ||
| `js/hang/src/catalog/mode.ts` and `Mode` in `rs/hang/src/catalog/mode.rs` | ||
| are shared by both sections today, and `@moq/binary` and `moq-binary` | ||
| implement only snapshot and stream, so support detection splits by | ||
| section (`jsonModeSupported` and `binaryModeSupported`, or per-section | ||
| known sets) and a binary `window` entry stays unreadable rather than | ||
| appearing supported. Datagram delivery is a track `Info` property, not a | ||
| mode, and stays out of the catalog. | ||
| - Request/response: `moq-json` gains typed `Request`/`Response` producers | ||
| and consumers over its stream mode in JS and Rust: the caller keeps the | ||
| correlation id and its own timeout, the callee answers each id once, and | ||
| a lagged reader surfaces the error rather than retrying. Discovery is the | ||
| direction teleop and Voice already use: the callee's response track is a | ||
| `json` entry with a `schema` naming the method set, and callers are found | ||
| under an announce prefix (operator publishes, robot subscribes; | ||
| `request/` and `response/` subtrees). | ||
| - Reliability is what `moq-net` gives (a group is one QUIC stream; a reader | ||
| that falls past `MAX_GROUP_CACHE` is told so). The docs say that instead | ||
| of promising delivery. Non-goals: byte-stream file transfer, delivery | ||
| receipts, per-participant addressing beyond a path and a scoped token, | ||
| and a cross-host timebase. | ||
| - Adopt in the same change: `@moq/room` `Chat` and its native twin | ||
| `moq_room::chat::Publisher` (`rs/moq-room/src/chat.rs`) declare their | ||
| `window` track in the `json` section, so Rust and JavaScript room | ||
| catalogs agree; `rs/moq-native/examples/chat.rs` and | ||
| `rs/moq-json/examples/telemetry.rs` declare theirs. The draft gains the | ||
| `window` mode and the request/response shape, and `doc/concept`'s | ||
| catalog page gains both, per the cross-package checklist for an `rs/hang` | ||
| catalog change. A catalog without data sections parses unchanged, which | ||
| the existing `deserialize_section` leniency already guarantees. | ||
| - Tests: `window` round trip in both languages, a request answered across a | ||
| local relay, a lagged caller surfacing the error, and an unknown mode | ||
| still passing through verbatim. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Robot teleoperation primitive](/quest/m3/teleop/robot.md) - the `rpc`, | ||
| `telemetry`, and `command` tracks are the first non-room consumer | ||
| - [LiveKit shim data surfaces](/quest/m2/livekit-shim-data.md) - | ||
| `publishData`, streams, and RPC map onto this | ||
| - [Catalog track identity](/quest/m3/catalog-tracks.md) - whatever it | ||
| decides about immutable definitions applies to data entries too |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| # [M] LiveKit Agents adapter (Node) | ||
|
|
||
| ## Goal | ||
|
|
||
| The Python adapter's design ported to `@livekit/agents`: an `AudioInput` | ||
| (`ReadableStream<AudioFrame>`) and `AudioOutput` (`captureFrame`, `flush`, | ||
| `clearBuffer`) over `@moq/net` and `@moq/hang`, plus the same announce-prefix | ||
| runner, published from `js/` as `@moq/livekit-agents`. Done when the stock | ||
| Node voice-agent example answers a browser publisher through a local relay. | ||
|
|
||
| ## Plan | ||
|
|
||
| - `AgentSession.start({agent})` builds `RoomIO` only when a room is passed | ||
| (`agent_session.ts`), and their tests assign `session.input.audio` | ||
| directly, so the shape is the same as Python. `@livekit/rtc-node` stays a | ||
| dependency for `AudioFrame`. | ||
| - Reuse the Python adapter's track naming, barge-in replacement, RTVI turn | ||
| mapping, and tests one for one, so a browser publisher is agnostic to which | ||
| runtime answers. | ||
|
|
||
| ## Required | ||
|
|
||
| - [LiveKit Agents adapter (Python)](/quest/m2/livekit-agents-python.md) - | ||
| settles the design this ports |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| # [M] LiveKit Agents adapter (Python) | ||
|
|
||
| ## Goal | ||
|
|
||
| A `livekit-agents` voice agent runs against a MoQ relay instead of a LiveKit | ||
| room, with its STT, LLM, TTS, turn detection, and plugins unchanged: a | ||
| `moq-livekit-agents` package in `py/` supplies an `AudioInput` and | ||
| `AudioOutput` over MoQ and a runner that starts one `AgentSession` per | ||
| announced request broadcast and publishes its response. Done when a stock | ||
| LiveKit voice-agent example answers a browser publisher through a local relay | ||
| with only the I/O wiring changed. | ||
|
|
||
| ## Plan | ||
|
|
||
| - The framework already runs without a room. `AgentSession.start(agent)` | ||
| with `session.input.audio` and `session.output.audio` assigned is how its | ||
| own test harness (`tests/fake_io.py`) and `console` mode | ||
| (`cli/tcp_console.py`) work, and `get_job_context(required=False)` keeps a | ||
| session independent of a Worker. Subclass `voice/io.py`'s `AudioInput` | ||
| (async iterator of `rtc.AudioFrame`) and `AudioOutput` (`capture_frame`, | ||
| `flush`, `clear_buffer`, and `on_playback_finished` when the track drains). | ||
| - Media the way Pipecat's MoQ transport does it (`pipecat.transports.moq`): | ||
| the `moq` package encodes and decodes Opus inside the FFI, the agent's | ||
| audio is one hang audio track replaced by a fresh one on barge-in so | ||
| catalog-aware players follow the successor, and frames cross as int16 at | ||
| the session's sample rate. `livekit-rtc` stays a dependency for the frame | ||
| type; no WebRTC connection is ever opened. | ||
| - The runner replaces dispatch: consume an announce prefix, start a session | ||
| per `request/<x>` broadcast, publish at `response/<x>`, end the session on | ||
| unannounce. The disjoint subtrees are the loop guard. A scoped token | ||
| (subscribe `request/**`, publish `response/**`) is the whole credential; | ||
| there is no registration and no dispatch API. | ||
| - Client turn signals ride the same RTVI JSON track the Pipecat transport | ||
| uses, mapped onto `session.interrupt()` and `session.commit_user_turn()`, | ||
| so one browser publisher works with either framework's agent. | ||
| - The README states what `RoomIO` provides and this does not: participant | ||
| attributes, `lk.chat` text input, transcription forwarding, `SessionHost`, | ||
| and RPC. Text and transcript surfaces follow the | ||
| [data convention](/quest/m2/data-convention.md) when it lands. | ||
| - Tests: a fake `AgentSession` I/O round trip over a local relay, barge-in | ||
| replacing the output track, and unannounce ending the session. Pin | ||
| `livekit-agents` to the 1.8 line and record the `io.py` surface it relies | ||
| on, since that module is not covered by their stability promise. | ||
|
|
||
| ## Related | ||
|
|
||
| - [LiveKit Agents adapter (Node)](/quest/m2/livekit-agents-node.md) | ||
| - [LiveKit client shim](/quest/m2/livekit-shim.md) - the client-side half |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| # [M] LiveKit shim data surfaces | ||
|
|
||
| ## Goal | ||
|
|
||
| `publishData`, text streams, and RPC in the LiveKit client shim work instead | ||
| of throwing: reliable `publishData` is a `stream`-mode `binary` track and | ||
| lossy a `snapshot`-mode one, text streams are `stream`-mode `json` tracks | ||
| keyed by topic, and RPC is the data convention's request/response pair | ||
| addressed by the callee's identity path. Byte streams with progress stay | ||
| unimplemented and throw. Done when a LiveKit sample that uses each of the | ||
| three runs against a relay unchanged. | ||
|
|
||
| ## Plan | ||
|
|
||
| - Every track the shim publishes is declared in the participant's | ||
| `camera.hang` catalog, so a plain `@moq/room` client can read a LiveKit | ||
| app's data too. The README maps each LiveKit method to its track and its | ||
| delivery guarantee. | ||
| - Tests: a reliable and a lossy `publishData` round trip, a topic stream | ||
| read by a late joiner (only what the stream retains), and an RPC with a | ||
| timeout and an error reply. | ||
|
|
||
| ## Required | ||
|
|
||
| - [LiveKit client shim](/quest/m2/livekit-shim.md) | ||
| - [Data convention](/quest/m2/data-convention.md) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| # [L] LiveKit client shim | ||
|
|
||
| ## Goal | ||
|
|
||
| A drop-in `livekit-client`-compatible JS package (e.g. `@moq/livekit`) that | ||
| runs a multi-participant room entirely over MoQ. The v1 surface is media plus | ||
| core events: Room connect/disconnect, local and remote participants, | ||
| camera/mic/screenshare publish, auto-subscribe, and the TrackSubscribed event | ||
| family; data surfaces (publishData, streams, RPC) are stubbed. The token slot | ||
| takes an ordinary moq-auth token, no LiveKit JWT parsing. Done when an | ||
| off-the-shelf LiveKit JS sample runs against a MoQ relay with only the import | ||
| and the connect URL/token changed. | ||
|
|
||
| ## Plan | ||
|
|
||
| - The shim is a LiveKit-API facade over `@moq/room` (`js/room`, landed in | ||
| #3634): the room is a path prefix in the connection URL and token root, | ||
| participants are discovered from the announce stream, identity is the path | ||
| before `camera.hang` and `screen.hang`, and a screenshare's | ||
| announce/unannounce is its lifecycle. The shim groups the two broadcasts | ||
| per identity into one RemoteParticipant and maps catalog entries to | ||
| TrackPublications. | ||
| - Build on `@moq/publish` and `@moq/watch`. LiveKit quality hints map to | ||
| the receiver-driven pixel target where they can (`setVideoQuality` picks | ||
| the rendition, `adaptiveStream` follows the rendered size); a hint the | ||
| target cannot express (`setVideoFPS`, per-layer bitrate caps) throws a | ||
| clear unsupported error rather than no-oping, so an app never believes a | ||
| limit is active. | ||
| - v1 is identity-only: `participant.identity` comes from the path and muted | ||
| state derives from catalog track presence. Names and coarse state come | ||
| from `@moq/room`'s `hang/*.json` metadata in a follow-up, not a rival | ||
| scheme; `ActiveSpeakersChanged` waits on | ||
| [active speaker](/quest/m2/room-active-speaker.md). | ||
| - Tokens come from `@moq/room` `claims()`: publish under `<identity>/**` | ||
| only, so participants cannot publish at each other's paths. | ||
| - `publishData`, text and byte streams, and RPC throw a clear | ||
| not-implemented error in v1; [shim data surfaces](/quest/m2/livekit-shim-data.md) | ||
| maps them onto the data convention afterwards. | ||
|
|
||
| ## Related | ||
|
|
||
| - [LiveKit Agents adapter (Python)](/quest/m2/livekit-agents-python.md) - | ||
| the agent-side half of the same migration |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| # [S] Active speaker | ||
|
|
||
| ## Goal | ||
|
|
||
| `@moq/room` exposes who is talking: every `Remote` and the `Local` carry an | ||
| audio level signal and a debounced `speaking` boolean, and the `Room` | ||
| exposes the ordered active-speaker set, so a grid can highlight the speaker | ||
| and the LiveKit shim can emit `ActiveSpeakersChanged`. `moq-room` is | ||
| media-free (announce events and chat; decoded audio lives in `moq-audio` | ||
| and the application), so the native half is a level meter in `moq-audio` | ||
| over decoded PCM with the same thresholds, which a native app wires itself; | ||
| no room media layer is added. | ||
|
|
||
| ## Plan | ||
|
|
||
| - Measure on decoded audio, independent of the member's playback mute: | ||
| `Member.muted` defaults to true and gates only the output today, so the | ||
| level analysis runs on the decoded frames before that gate (or keeps the | ||
|
Comment on lines
+16
to
+18
There was a problem hiding this comment.
This plan only measures decoded frames and discusses Useful? React with 👍 / 👎. |
||
| decoder subscribed while the sink is silent), and a muted-for-me member | ||
| still reads as speaking. Measuring on the wire is not an option since the | ||
| relay never decodes. | ||
| - Level is a smoothed RMS in dBFS; `speaking` uses an attack and release | ||
| threshold with a short hold, exposed through the existing `@moq/signals` | ||
| idiom. The `moq-audio` meter uses the same numbers and a test vector | ||
| shared with the JS one. | ||
| - The demo at `demo/web/src/meet.html` highlights the loudest tile as proof, | ||
| with every remote left muted. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| # [M] C++ SDK | ||
|
|
||
| ## Goal | ||
|
|
||
| C++ developers (robotics tooling, engines, native apps) get a thin, | ||
| idiomatic wrapper over `libmoq`'s C ABI: RAII handles, `std::span` payloads, | ||
| callbacks or futures for announce and subscribe, and hang audio, video, and | ||
| data tracks, installed as a CMake package beside `moq.pc`. The OBS plugin | ||
| (`cpp/obs`) moves onto it as the first consumer, and the wrapper's surface | ||
| tracks the Go, Swift, and Kotlin bindings one for one. | ||
|
|
||
| ## Plan | ||
|
|
||
| - Header-first in `cpp/moq`, generated where the C ABI already carries the | ||
| shape (cbindgen output in `rs/libmoq`), handwritten where ownership or | ||
| callbacks need C++ semantics. C++20, no exceptions across the boundary. | ||
| - Ship through `libmoq`'s existing release pipeline (`build.sh`, CMake | ||
| config), with a smoke that publishes and watches a clock through a local | ||
| relay on Linux, macOS, and Windows. | ||
| - Unity is deliberately not a target; a C# binding is a separate decision. | ||
|
|
||
| ## Required | ||
|
|
||
| - [#2152](/quest/m2/2152-libmoq-c-abi-catch-up-with-the-moq-ffi-surface.md) - | ||
| the C ABI has to carry sessions before a wrapper is worth publishing | ||
|
Comment on lines
+24
to
+25
There was a problem hiding this comment.
Completing only #2152 cannot provide the promised one-for-one Swift/Kotlin surface: that quest explicitly excludes dynamic track serving and moves it to Useful? React with 👍 / 👎. |
||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When a caller supplies LiveKit's
topicordestinationIdentitiesoptions, mapping only the payload onto a binary snapshot/stream leaves nowhere to carry those fields. Receivers therefore cannot reproduce the original topic or filter addressed packets, so targeted data can be delivered to unintended room clients and compatible samples will observe incorrect events. Define an envelope or track-naming scheme that preserves this metadata, or explicitly reject these options.AGENTS.md reference: AGENTS.md:L57-L60
Useful? React with 👍 / 👎.