feat(net)!: datagrams are unfetchable and bounded by the group range

doc/concept/moq-lite.md

@@ -385,6 +385,12 @@ datagram: unreliable, unordered, under about 1200 bytes, and never
 retransmitted. It suits real-time audio and sensor data. There is no stream
 fallback, so a datagram that doesn't fit isn't delivered that way.
 
+Nothing caches a datagram. A subscription gets the ones inside its group
+range, and a new one may get the few still in the publisher's short send
+buffer, but `FETCH` never returns one. A publisher that knows the sequence was
+a datagram refuses with `NOT_FETCHABLE` (moq-lite-07), otherwise with
+`NOT_FOUND`. Use a group for anything a late joiner needs.
+
 ## What moq-lite leaves out
 
 Compared with moq-transport: no request IDs (a stream per request instead), no

doc/concept/standard.md

@@ -86,7 +86,10 @@ tracks waiting for a group that never arrives.
 A moq-lite datagram is a single-frame group, so on moq-transport it travels
 as an `OBJECT_DATAGRAM` at object 0 whose Group ID is the sequence, and a relay
 forwards it without renumbering. A datagram carrying any other Object ID, or a
-status other than Normal, is dropped. JavaScript does not yet carry datagrams
+status other than Normal, is dropped. Datagrams are never fetchable: a fetch
+object flagged as a datagram fails only that fetch, as `NotFetchable`, and a
+relay answers its own downstream FETCH with `DOES_NOT_EXIST`, or `NOT_FETCHABLE`
+to a moq-lite-07 peer. JavaScript does not yet carry datagrams
 on moq-transport.
 
 A client may present one credential in its `SETUP` with the `AUTHORIZATION

drafts/draft-lcurley-moq-lite.md

@@ -330,6 +330,8 @@ Sent when resetting a stream (RESET_STREAM), or when refusing to receive one (ST
 | ------- | ------------- | ----------- |
 |  0x39  | TIMESTAMP_MISMATCH | A frame's timestamp does not match its track's timescale. |
 | ------- | ------------- | ----------- |
+|  0x3A  | NOT_FETCHABLE | The FETCH named a group delivered only as a datagram, which is never cached (see [Datagrams](#datagrams)). It has no moq-transport value; a bridge refuses the FETCH with DOES_NOT_EXIST. |
+| ------- | ------------- | ----------- |
 
 Note that CANCELLED is 0x1, not 0x0: a stream reset with 0x0 is an INTERNAL_ERROR, not a routine cancellation.
 An endpoint terminating a stream because the session is ending SHOULD use SESSION_CLOSED rather than the session's own code, since the two spaces are disjoint.
@@ -658,7 +660,11 @@ The subscriber MAY cache the error and potentially retry later.
 QUIC datagrams provide unreliable, unordered delivery for latency-sensitive content that does not need retransmission.
 
 A publisher MAY transmit a Group consisting of exactly one Frame as a single QUIC datagram, in addition to (or instead of) opening a Group Stream, based on application hints, group size, and network conditions; a multi-frame Group is delivered via a Group Stream only.
-A datagram-delivered group is not cached or retransmitted; a publisher SHOULD only send a datagram if the congestion controller can transmit it immediately.
+A publisher sends a datagram to a subscription only when its Group Sequence is inside the subscription's range, where a non-zero `Frame Start` excludes the start group (see [Positions](#positions)).
+A datagram is never cached or retransmitted, and FETCH never returns one.
+A publisher MAY send a new subscription the datagrams still in a short send buffer, so a subscriber cannot assume a datagram was sent after it subscribed.
+A publisher that knows a FETCH names a group delivered only as a datagram resets the stream with NOT_FETCHABLE; otherwise it answers as for a group that does not exist, with NOT_FOUND.
+A publisher SHOULD only send a datagram if the congestion controller can transmit it immediately.
 There is no separate subscription for datagram delivery: datagrams are routed to existing subscriptions via the Subscribe ID, and a subscriber receiving the same group via both a stream and a datagram MUST deduplicate by group sequence.
 
 Each datagram body has the following encoding (note: there is no message length prefix; the QUIC datagram boundary delimits the payload):
@@ -1382,6 +1388,8 @@ The `Message Length` describes the payload size on the wire.
 
 ## moq-lite-07
 
+- A subscription's range bounds datagrams like groups, and FETCH never returns a datagram.
+- Assigned 0x3A NOT_FETCHABLE in the stream error table: a FETCH for a group delivered only as a datagram.
 - The subscriber FINs its Subscribe Stream after settling its tail; graceful session close waits for that FIN or reset.
 - A refusal is not retried at another route of the same prefix either.
 - Made TRACK_INFO Publisher Max Age optional, encoded as milliseconds plus one with zero meaning no limit.

js/net/src/track.test.ts

@@ -156,6 +156,22 @@ test("appendDatagram delivers to a subscriber", async () => {
 	expect(got && dec.decode(got.payload)).toBe("hello");
 });
 
+// The in-range datagrams, sent between the dropped ones, show the reader is live, so the
+// range is what dropped the others.
+test("the group range bounds datagrams", async () => {
+	const producer = new TrackProducer("test");
+	const track = producer.subscribe({ groups: { start: { included: 5 } } });
+	track.setGroups({ end: { excluded: 7 } });
+	for (const sequence of [4, 5, 7, 6]) {
+		producer.insertDatagram(sequence, Timestamp.fromMillis(0), enc.encode("x"));
+	}
+	producer.close();
+
+	expect((await track.recvDatagram())?.sequence).toBe(5);
+	expect((await track.recvDatagram())?.sequence).toBe(6);
+	expect(await track.recvDatagram()).toBeUndefined();
+});
+
 test("datagram buffers drop oldest at capacity without delaying active subscribers", async () => {
 	const producer = new TrackProducer("test");
 	const slow = producer.subscribe();

js/net/src/track.ts

@@ -1430,8 +1430,9 @@ export class Subscriber {
 	 * Receive the next datagram in arrival order.
 	 *
 	 * Datagrams are a separate best-effort channel from groups (see
-	 * {@link Producer.appendDatagram}); they share only the sequence namespace. A consumer
-	 * that falls too far behind silently loses the oldest datagrams. Read this alongside
+	 * {@link Producer.appendDatagram}); they share only the sequence namespace. Those outside
+	 * the group range are skipped. A consumer that falls too far behind silently loses the
+	 * oldest datagrams. Read this alongside
 	 * {@link recvGroup} (e.g. in a separate loop) to receive both channels concurrently.
 	 * The two cursors are independent: a datagram never moves the group cursor.
 	 */
@@ -1446,8 +1447,11 @@ export class Subscriber {
 		for (;;) {
 			const datagrams = this.#state.datagrams.peek();
 
-			if (datagrams.length > 0) {
-				return datagrams.shift();
+			// The group range bounds datagrams too, judged when each is read. One outside it is
+			// gone for good: nothing holds it for a later raise, unlike a group past the cap.
+			const { start, end } = this.#cursor.peek();
+			for (let datagram = datagrams.shift(); datagram; datagram = datagrams.shift()) {
+				if (datagram.sequence >= start && (end === undefined || datagram.sequence < end)) return datagram;
 			}
 
 			const closed = this.#state.closed.peek();

quest/m1/README.md

@@ -80,7 +80,6 @@ blocks. The quests that gated m0 lines moved under them.
 - [Fetch without SUBSCRIBE](/quest/m1/ietf-fetch-only.md) - a relay fetches from an IETF upstream without subscribing, finished tracks included, with End of Track always reported
 - [IETF object gaps](/quest/m1/ietf-object-gaps.md) - a gapped object ID is refused loudly like a subgroup in Rust and JS, and an overflowing one closes the session in Rust
 - [JS IETF datagrams](/quest/m1/js-ietf-datagram.md) - `@moq/net` sends and receives datagram groups over moq-transport, like Rust
-- [Datagrams are live-only](/quest/m1/datagram-unfetchable.md) - no FETCH, replay to a new subscriber, or cache fill ever returns a datagram group
 - [JavaScript FETCH](/quest/m1/js-fetch.md) - browser publishers answer IETF FETCH through the JS ranges request surface
 - [Bounded HLS playlists](/quest/m1/hls-bounded.md) - `moq-hls` renders a capped sliding window for a fresh viewer, with a stable media sequence, gap slots, and no video before a sync point
 - [JS request window](/quest/m1/js-request-window.md) - @moq/net grants MAX_REQUEST_ID as requests close, so a peer never stalls after ~21k requests

quest/m1/js-ietf-datagram.md

@@ -15,7 +15,3 @@ same way Rust does, and close the session on a malformed Type.
 
 The integration test `ietf does not deliver datagrams` flips to delivery on
 every supported draft, and `just test interop --all` covers both directions.
-
-## Related
-
-- [Datagrams are live-only](/quest/m1/datagram-unfetchable.md) - the subscribe range for datagrams, settled on both protocols

rs/moq-c/src/error.rs

@@ -393,7 +393,8 @@ fn stream_kind(err: &moq_net::StreamError) -> moq_protocol_kind {
 		moq_net::StreamError::GoingAway => MOQ_PROTOCOL_KIND_GOING_AWAY,
 		moq_net::StreamError::TooFarBehind => MOQ_PROTOCOL_KIND_TOO_FAR_BEHIND,
 		moq_net::StreamError::MalformedTrack => MOQ_PROTOCOL_KIND_MALFORMED_TRACK,
-		moq_net::StreamError::NotFound => MOQ_PROTOCOL_KIND_NOT_FOUND,
+		// A datagram reached by a FETCH is, to a binding, a miss like any other.
+		moq_net::StreamError::NotFound | moq_net::StreamError::NotFetchable => MOQ_PROTOCOL_KIND_NOT_FOUND,
 		moq_net::StreamError::Unroutable => MOQ_PROTOCOL_KIND_UNROUTABLE,
 		moq_net::StreamError::Old => MOQ_PROTOCOL_KIND_OLD,
 		moq_net::StreamError::Evicted => MOQ_PROTOCOL_KIND_EVICTED,

rs/moq-ffi/src/consumer.rs

@@ -302,7 +302,7 @@ impl MoqBroadcastConsumer {
 
 fn map_fetch_error(err: moq_net::Error) -> MoqError {
 	match err {
-		moq_net::Error::NotFound => MoqError::NotFound,
+		moq_net::Error::NotFound | moq_net::Error::NotFetchable => MoqError::NotFound,
 		moq_net::Error::Unsupported | moq_net::Error::Version => MoqError::Unsupported,
 		err => err.into(),
 	}

rs/moq-ffi/src/error.rs

@@ -136,7 +136,8 @@ fn stream_kind(err: &moq_net::StreamError) -> MoqProtocolKind {
 		moq_net::StreamError::GoingAway => MoqProtocolKind::GoingAway,
 		moq_net::StreamError::TooFarBehind => MoqProtocolKind::TooFarBehind,
 		moq_net::StreamError::MalformedTrack => MoqProtocolKind::MalformedTrack,
-		moq_net::StreamError::NotFound => MoqProtocolKind::NotFound,
+		// A datagram reached by a FETCH is, to a binding, a miss like any other.
+		moq_net::StreamError::NotFound | moq_net::StreamError::NotFetchable => MoqProtocolKind::NotFound,
 		moq_net::StreamError::Unroutable => MoqProtocolKind::Unroutable,
 		moq_net::StreamError::Old => MoqProtocolKind::Old,
 		moq_net::StreamError::Evicted => MoqProtocolKind::Evicted,
@@ -280,7 +281,7 @@ impl From<moq_net::Error> for MoqError {
 	fn from(err: moq_net::Error) -> Self {
 		match err {
 			moq_net::Error::Transport(message) => Self::Transport(message),
-			moq_net::Error::NotFound => Self::NotFound,
+			moq_net::Error::NotFound | moq_net::Error::NotFetchable => Self::NotFound,
 			moq_net::Error::Closed | moq_net::Error::GoingAway | moq_net::Error::SessionClosed => Self::Closed,
 			moq_net::Error::Cancel => Self::Cancelled,
 			moq_net::Error::Unauthorized => Self::Unauthorized,

rs/moq-hls/src/export/rendition.rs

@@ -1047,10 +1047,14 @@ fn is_cache_miss(err: &moq_net::Error) -> bool {
 	matches!(
 		err,
 		moq_net::Error::NotFound
+			| moq_net::Error::NotFetchable
 			| moq_net::Error::Old
 			| moq_net::Error::Evicted
 			| moq_net::Error::Stream(
-				moq_net::StreamError::NotFound | moq_net::StreamError::Old | moq_net::StreamError::Evicted
+				moq_net::StreamError::NotFound
+					| moq_net::StreamError::NotFetchable
+					| moq_net::StreamError::Old
+					| moq_net::StreamError::Evicted
 			)
 	)
 }
@@ -1100,7 +1104,12 @@ mod tests {
 
 	#[test]
 	fn a_cache_miss_that_crossed_a_session_is_still_a_cache_miss() {
-		for local in [moq_net::Error::NotFound, moq_net::Error::Old, moq_net::Error::Evicted] {
+		for local in [
+			moq_net::Error::NotFound,
+			moq_net::Error::NotFetchable,
+			moq_net::Error::Old,
+			moq_net::Error::Evicted,
+		] {
 			assert!(is_cache_miss(&local), "{local:?} locally");
 			let lite = over_lite(&local);
 			assert!(is_cache_miss(&lite), "{local:?} over lite ({lite:?})");

rs/moq-net/src/coding/codes.rs

@@ -7,8 +7,8 @@ use crate::{Error, StreamError, ietf, lite};
 /// Every [`Reader`](super::Reader) and [`Writer`](super::Writer) carries the negotiated
 /// version, which is what picks the registry here. The two wires draw on the same names,
 /// but they do not agree on every value: moq-lite's are fixed by
-/// [`StreamError::to_code`], while moq-transport's moved across the drafts we negotiate
-/// (see the `ietf::error` module). Sending a code from the wrong table is silent, because the
+/// [`StreamError::to_code`] (a later version adds codes but never moves one), while
+/// moq-transport's moved across the drafts we negotiate (see the `ietf::error` module). Sending a code from the wrong table is silent, because the
 /// number is valid in both.
 ///
 /// Encoding and decoding live on one trait so a peer cannot be told a code from one table
@@ -38,15 +38,21 @@ pub trait StreamCodes {
 	}
 }
 
-/// moq-lite's registry, specified by draft-lcurley-moq-lite (Error Codes) and identical
-/// across the versions we negotiate.
+/// moq-lite's registry, specified by draft-lcurley-moq-lite (Error Codes). A code a version
+/// lacks is sent there as the closest one it has, and not read back there.
 impl StreamCodes for lite::Version {
 	fn encode_stream_code(&self, err: &StreamError) -> u32 {
-		err.to_code()
+		match err {
+			StreamError::NotFetchable if !self.has_not_fetchable() => StreamError::NotFound.to_code(),
+			err => err.to_code(),
+		}
 	}
 
 	fn decode_stream_code(&self, code: u32) -> StreamError {
-		StreamError::from_code(code)
+		match StreamError::from_code(code) {
+			StreamError::NotFetchable if !self.has_not_fetchable() => StreamError::Unknown(code),
+			err => err,
+		}
 	}
 }
 
@@ -110,6 +116,32 @@ mod tests {
 		);
 	}
 
+	/// NOT_FETCHABLE is lite-07's: an earlier peer is told NOT_FOUND, the miss it would
+	/// have been told before, and is not credited with a code its version never assigned.
+	#[test]
+	fn not_fetchable_is_lite_07s() {
+		for version in [lite::Version::Lite05, lite::Version::Lite06] {
+			assert_eq!(
+				version.encode_stream_code(&StreamError::NotFetchable),
+				StreamError::NotFound.to_code(),
+				"{version}"
+			);
+			assert_eq!(
+				version.decode_stream_code(0x3a),
+				StreamError::Unknown(0x3a),
+				"{version}"
+			);
+		}
+
+		let version = lite::Version::Lite07;
+		assert_eq!(version.encode_stream_code(&StreamError::NotFetchable), 0x3a);
+		assert_eq!(version.decode_stream_code(0x3a), StreamError::NotFetchable);
+		assert_eq!(
+			version.decode_stream_code(StreamError::NotFound.to_code()),
+			StreamError::NotFound
+		);
+	}
+
 	/// A dropped [`Writer`](super::Writer) resets with a cancellation, and `Drop` cannot
 	/// carry the bound that would reach the negotiated registry. It does not need one only
 	/// while every registry we speak agrees on the value, so pin that.