diff --git a/quest/m0/README.md b/quest/m0/README.md index 0f104ffe7e..935edeae68 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -30,6 +30,7 @@ Published API or wire breaks still land on dev; each quest's Plan says so. ## Required +- [Packaged relay starts](/quest/m0/relay-systemd-unit.md) - the `.deb` and `.rpm` relay service starts instead of crash-looping on `--file` - [Skip unchanged announce updates](/quest/m0/announce-update-dedupe.md) - a publisher sends an announce update only when the wire route changed - [Wildcard](/quest/m0/wildcard/README.md) - a relay resolves subscriptions against advertised prefixes, a service claims the prefix it could serve and refuses the rest instead of enumerating broadcasts, and the browser player treats a covering claim as availability - [Audio quality harness](/quest/m0/audio-quality-harness/README.md) - a browser playout latency regression fails a nightly run instead of arriving as a bug report, and its recorder supplies the jitter target's replay traces diff --git a/quest/m0/relay-systemd-unit.md b/quest/m0/relay-systemd-unit.md new file mode 100644 index 0000000000..37a7e51a19 --- /dev/null +++ b/quest/m0/relay-systemd-unit.md @@ -0,0 +1,18 @@ +# [XS] Packaged relay starts + +## Goal + +The `.deb` and `.rpm` relay service starts. Today +`packaging/moq-relay/moq-relay.service` runs `moq-relay --file +/etc/moq-relay/relay.toml`, but the config path is positional and the relay +rejects unknown flags, so every packaged install crash-loops. + +## Plan + +`ExecStart=/usr/bin/moq-relay /etc/moq-relay/relay.toml`. Add the reporter's +unit test in the relay crate that parses the unit's `ExecStart` with the +relay's own `Cli`, so the unit and the parser can't drift again. + +## Closes + +- [#4343](https://github.com/moq-dev/moq/issues/4343) - close this issue when the quest finishes diff --git a/quest/m1/README.md b/quest/m1/README.md index 7e397ee3cd..be62e0315b 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -33,12 +33,18 @@ transport, benchmark tooling); worktrees isolate commits, not semantics. - [Live in apps](/quest/m1/announce-live-apps.md) - the demo and `@moq/room` show "no broadcasts" from the `live` marker, which waits for the first session on page load - [Watch refusal](/quest/m1/watch-refusal.md) - `` shows an origin refusal as an error instead of sitting offline - [kio waiter overflow](/quest/m1/kio-waiter-lost.md) - a retained `Waiter` past 8 lists stops adding a duplicate entry to lists it already recorded +- [moqsink recoverable errors](/quest/m1/moqsink-keyframe-latch.md) - a leading delta or a timestamp rewind drops frames until a keyframe instead of invalidating a moqsink pad - [Capture re-anchor](/quest/m1/capture-reanchor.md) - a repeating or restarting device clock never rewinds native capture during a fast backlog drain - [Splice edge cases](/quest/m1/splice-edges.md) - an unstamped successor, a pruned segment's boundary group, and a warm head during a takeover are each handled correctly - [Resumed groups](/quest/m1/resume-latest.md) - a half-delivered group ends once the new copy is past it, so a group-only reader never parks after a mid-group failover - [Track tail hardening](/quest/m1/track-tail-hardening.md) - Rust and JS wait out a track's tail by the same rules, with the known hang, count, truncation, grace, and memory holes closed - [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) - every stream group in a lite subscription arrives or is dropped by name, and lite-07 drops its stream count for it +- [FIN wait expiry](/quest/m1/fin-wait-expiry.md) - a group awaiting its FIN ack still expires and follows priority updates on lite and IETF +- [Cross-relay bursts](/quest/m1/cross-relay-bursts.md) - bursty small-group tracks cross two relays without lost groups, unanswered FETCHes, or stalls - [Session death parity](/quest/m1/session-death.md) - a local close ends tracks cleanly in both languages, and JS group readers see the session's error on session death +- [Watch video guards](/quest/m1/watch-video-guards.md) - promoting a video track holds the last picture, and an older group never reaches the codec between live deltas +- [Watch decoder recovery](/quest/m1/watch-decoder-recovery.md) - one malformed packet rebuilds the audio or video decoder instead of ending playback +- [Watch audio under CSP](/quest/m1/watch-worklet-file.md) - production builds ship the audio worklet as a file, so `script-src 'self'` pages play audio - [moqsrc stop](/quest/m1/moqsrc-stop.md) - moqsrc's stop blocks until its session ends, without deadlocking on a blocked pad push - [More tests under load](/quest/m1/test-flakes-2.md) - the second round of load-only failures, fixed at the cause - [Auth outage clock](/quest/m1/auth-outage-clock.md) - the relay and moq-auth outage tests run on a paused clock again and assert both bounds of `expires` @@ -50,8 +56,11 @@ transport, benchmark tooling); worktrees isolate commits, not semantics. - [Accept-side flags](/quest/m1/cli-given-flags.md) - dial-only and local verbs refuse every `--listen-*` flag instead of ignoring it - [#2075](/quest/m1/2075-mirror-catalog-reservation-gating-in-moq-hang-js-hang.md) - @moq/publish gates the first catalog snapshot until every reserved track is described - [Full codec string](/quest/m1/publish-codec-string.md) - browser-published video carries the encoder's full RFC 6381 codec string, so native players decode it +- [TS program selection](/quest/m1/ts-programs.md) - `import ts` refuses a multi-program stream unless `--program ` picks one or publishes each - [TS export jitter](/quest/m1/ts-export-jitter.md) - the video reorder bound follows later catalogs and the declared reorder depth, so a late B-frame never reorders TS output; an undeclared stream can reorder once per new maximum depth - [TS import shared shift](/quest/m1/ts-import-shared-shift.md) - unflagged loop wraps move audio and video by one shift, so A/V sync holds across wraps +- [ffmpeg muxdelay](/quest/m1/ts-publish-muxdelay.md) - the documented MPEG-TS publish line adds `-muxdelay 0`, so quiet audio is not clumped +- [Same-hop importers](/quest/m1/hop-aligned-import.md) - importers sharing a `--hop` and fed one stream publish identical groups and timestamps, so failover survives - [PipeWire duplicate cameras](/quest/m1/pipewire-dup-cameras.md) - a webcam lists once with PipeWire enabled - [Catalog wall clock](/quest/m1/catalog-wall-clock.md) - `Clock::wall_clock` keeps the catalog's full precision instead of truncating to milliseconds - [Capture control](/quest/m1/capture-control.md) - on dev, `encode::Capture` replaces `CaptureOptions` without a `clock` field (it reads the catalog's), an unsupported `cut()` errors, and dropping the last `Control` cancels in-flight opens diff --git a/quest/m1/cluster-routing.md b/quest/m1/cluster-routing.md index 983b880eb6..3d52174867 100644 --- a/quest/m1/cluster-routing.md +++ b/quest/m1/cluster-routing.md @@ -122,3 +122,4 @@ make MoQ's common case. - [Skip unchanged announce updates](/quest/m0/announce-update-dedupe.md) - cuts duplicate updates on today's routing - [Redundant ingest](/quest/m2/redundant-ingest.md) - builds on the `--hop` failover this must keep or replace - [Routing cost domains](/quest/m2/routing-cost-domains.md) - cost across the cluster boundaries this keeps path vector +- [Cross-relay delivery under bursts](/quest/m1/cross-relay-bursts.md) - its #4349 report also shows closed broadcasts announced for up to 229 s and flapping between Retracted and Announced across nodes, evidence for per-incarnation seqnos diff --git a/quest/m1/cross-relay-bursts.md b/quest/m1/cross-relay-bursts.md new file mode 100644 index 0000000000..25607b1be8 --- /dev/null +++ b/quest/m1/cross-relay-bursts.md @@ -0,0 +1,34 @@ +# [L] Cross-relay delivery under bursts + +## Goal + +Bursty small-group tracks cross two relays without losing groups or stalling. +On cdn.moq.pro, publishers and a subscriber on different nodes saw groups +never arrive (FETCH for them unanswered for 2 s while the publisher stayed +connected), a subscription opened early deliver only a burst's tail, and +return tracks stall up to 31 s with `Stream(Old)`. The same workload on one +node, or one self-hosted relay, loses nothing. + +## Plan + +Reproduce first, on a local two-relay cluster with the reporter's harness +(offered in the issue) and the relay build cdn.moq.pro ran. Then fix what the +repro shows. Suspects: `Old` expiry or newest-first dropping on the relay hop, +and FETCH not forwarded or answered upstream. Re-measure the stalls after +[FIN wait expiry](/quest/m1/fin-wait-expiry.md), a candidate cause. Keep the +repro as a regression test in the cluster tests. + +The issue's stale and flapping announcements after a clean close are +cluster-routing evidence, not this quest's scope. + +## Required + +- [FIN wait expiry](/quest/m1/fin-wait-expiry.md) - removes one candidate cause of the stalls before measuring + +## Closes + +- [#4349](https://github.com/moq-dev/moq/issues/4349) - close this issue when the quest finishes + +## Related + +- [Cluster routing](/quest/m1/cluster-routing.md) - owns the stale and flapping announcements from the same report diff --git a/quest/m1/fin-wait-expiry.md b/quest/m1/fin-wait-expiry.md new file mode 100644 index 0000000000..eb163fa6fe --- /dev/null +++ b/quest/m1/fin-wait-expiry.md @@ -0,0 +1,31 @@ +# [S] A group awaiting its FIN ack still expires + +## Goal + +A group whose FIN is written but not yet acknowledged still expires with +`Old` and still follows priority updates, on lite and IETF. Today the +`Closed` arm only awaits `writer.poll_close`, so a stale group keeps its +queued bytes and its old send order, which can stall newer groups behind it. + +## Plan + +The issue ships four failing tests, a `with_fin_gate` test hook, and a fix: + +- lite: poll priority above the state match, not only in `Serve`. +- lite and IETF: in `Closed`, poll `poll_expired_while_pending` and abort with + `Old`. +- moq-tokio: `set_priority` in `Closing` fires the existing interrupt so the + new order applies, instead of only storing it. + +Each priority update rebuilds the `closed()` watch; check the publisher +benchmarks for a regression. The flaky +`subscription_end_integrity::a_subscription_cut_by_the_publisher_disconnecting_does_not_end_clean` +the reporter hit is tracked in [More tests under load](/quest/m1/test-flakes-2.md). + +## Closes + +- [#4332](https://github.com/moq-dev/moq/issues/4332) - close this issue when the quest finishes + +## Related + +- [Starvation](/quest/m1/qos/starvation.md) - its frontier relies on FIN acks, which can now end in `Old` diff --git a/quest/m1/hop-aligned-import.md b/quest/m1/hop-aligned-import.md new file mode 100644 index 0000000000..6f6f68b6c6 --- /dev/null +++ b/quest/m1/hop-aligned-import.md @@ -0,0 +1,45 @@ +# [L] Same-hop importers publish identical tracks + +## Goal + +Two importers sharing a `--hop` and fed the same encoded stream publish +identical tracks, so a relay can fail over between them. Today two +`moq import ts` fed one forked stream break that contract twice: a standby +refuses every track until it has parsed its PMT, which aborts every +subscriber the moment it connects (#4352), and each numbers its groups from +its own counter, so after a failover `export ts` sees a timestamp rewind and +exits (#4354). + +## Plan + +Decided: same-hop publishers MUST publish the same broadcasts and tracks. +Keep `--hop`, and make every container importer (ts, fmp4, flv, mkv, and the +SRT, RTMP, and HLS gateways that reuse them) meet the contract when fed one +encoded stream. Capture is out: two encoders never align. + +- A group's sequence derives from its keyframe's media timestamp (PTS in TS), + not a per-process counter. Decide how both processes agree across a + timestamp wrap (TS PTS wraps every 26.5 h) when they started on opposite + sides of it. +- Frame timestamps derive from the input alone too: any re-anchor shift must + come from the stream, not process start or wall clock. +- An importer announces only once it knows its tracks, so it never refuses a + track the incumbent serves. +- Docs (`doc/bin/cli.md` "Redundant publishers", the `--hop` doc comment) + narrow the contract to one encoded stream fed to each publisher. + +Tests: per importer, two instances started at different offsets into the same +input produce the same group sequences and timestamps. End to end: the +issues' 1+1 setup (one relay, two same-hop `import ts`, two `export ts`) +survives the standby joining and the incumbent stopping. + +## Closes + +- [#4352](https://github.com/moq-dev/moq/issues/4352) - close this issue when the quest finishes +- [#4354](https://github.com/moq-dev/moq/issues/4354) - close this issue when the quest finishes + +## Related + +- [Redundant ingest](/quest/m2/redundant-ingest.md) - splicing across first hops and two encoders, which this does not attempt +- [TS import shared shift](/quest/m1/ts-import-shared-shift.md) - the TS re-anchor shift that must stay input-derived +- [Broadcast epochs](/quest/m1/broadcast-epoch/README.md) - a redundant pair shares one epoch diff --git a/quest/m1/moqsink-keyframe-latch.md b/quest/m1/moqsink-keyframe-latch.md new file mode 100644 index 0000000000..5cb8205681 --- /dev/null +++ b/quest/m1/moqsink-keyframe-latch.md @@ -0,0 +1,34 @@ +# [S] moqsink drops frames on recoverable importer errors + +## Goal + +A `moqsink` pad survives the importer errors moq-mux documents as +recoverable: it drops frames until the next keyframe instead of calling +`self.fail()` and losing the rendition for good. Two paths reach that +failure in `Media::write` (`rs/moq-gst/src/sink/pad.rs`) today: + +- A new pad starts unlatched (`keyframe: false`), so a first buffer that is a + delta frame fails it. +- A `TimestampRewind` with no signalled break (an rtspsrc jitterbuffer + re-anchoring) has no guard at all. + +## Plan + +The header-only buffer after a break was fixed in +https://github.com/moq-dev/moq/pull/4356: the latch stays armed after a +successful decode. Reuse that latch here. + +Decided: a video pad starts latched. On `TimestampRewind`, set the latch and +drop until a keyframe lands above the producer's live edge, losing up to one +GOP per re-anchor. Mapping a rewound timeline forward instead belongs to +[#3021](/quest/m1/3021-moq-gst-anchor-generated-media-timelines-to-wall-clock.md). +The reporter offered the guard they run in production. + +Regression tests beside `video_pause_drops_deltas_until_the_next_keyframe`: + +- A new pad's first buffer is a delta: it drops. +- A rewind without a break: deltas drop until a keyframe above the edge. + +## Closes + +- [#4366](https://github.com/moq-dev/moq/issues/4366) - close this issue when the quest finishes diff --git a/quest/m1/test-flakes-2.md b/quest/m1/test-flakes-2.md index c1742598ca..8cc9502a74 100644 --- a/quest/m1/test-flakes-2.md +++ b/quest/m1/test-flakes-2.md @@ -22,6 +22,10 @@ retry. [#4084](https://github.com/moq-dev/moq/pull/4084) removed from its sibling after [#4055](https://github.com/moq-dev/moq/pull/4055) papered over it with a retry. +- moq-tokio + `subscription_end_integrity::a_subscription_cut_by_the_publisher_disconnecting_does_not_end_clean` + ends `Ok(None)` with 10 of 20 frames in 3 of 8 full-suite runs on a clean + tree, and passes alone (#4332). - `just test media` late join failed once after [#4181](https://github.com/moq-dev/moq/pull/4181): "joined at frame 111, 16 frames behind 127", against a budget of one GOP (15). diff --git a/quest/m1/tooling/justfiles.md b/quest/m1/tooling/justfiles.md index 22597e1461..429dedb432 100644 --- a/quest/m1/tooling/justfiles.md +++ b/quest/m1/tooling/justfiles.md @@ -86,8 +86,17 @@ wanted), `_shell`, `_flake` as its one-liner, `clean`, the rs `package` and `fuzz` bodies, and the OBS `compile`, `_includes`, `_unit`, `test`, `check`, and `preset` bodies. +Arguments reach tools intact: `just rs test`, `check-test`, and +`capture-test` interpolate `{{ args }}` today, so a quoted nextest filterset +becomes a shell syntax error (#4342). Recipes that forward arguments use +`[positional-arguments]` and `"$@"`. No self-test for it, per the rule above. + Docs: `doc/setup/dev.md`, `CONTRIBUTING.md`, `test/README.md`, and the `AGENTS.md` mentions of `just wasm` follow the survivors. Verify with `just check`, `just ci test`, and `just check --all`, and confirm every recipe name check.yml, cache.yml, nightly.yml, interop.yml, wasm.yml, obs.yml, swift.yml, and release-*.yml invoke still resolves. + +## Closes + +- [#4342](https://github.com/moq-dev/moq/issues/4342) - close this issue when the quest finishes diff --git a/quest/m1/ts-programs.md b/quest/m1/ts-programs.md new file mode 100644 index 0000000000..14e17a964c --- /dev/null +++ b/quest/m1/ts-programs.md @@ -0,0 +1,37 @@ +# [L] moq import ts: select programs + +## Goal + +A multi-program transport stream is never silently merged. Today `import ts` +keeps the first program's identity, adds every PMT's streams to one broadcast +on one clock, and loses content; `export ts` rebuilds one program. After +this quest `import ts` refuses a multi-program input by default, naming the +programs, and `--program` chooses what to import. + +## Plan + +Decided: + +- Default: error before publishing when the initial PAT lists more than one + non-zero program, naming each program and pointing at `--program`. A later + PAT that adds a program ends the import with the same error; media already + published stays. +- `--program ` imports program `n` only. +- `--program all` publishes one broadcast per program, each with its own clock + and catalog. The catalog suffix stays last so discovery and format detection + still see it: `--broadcast event.hang` publishes `event/1.hang` and + `event/2.hang`. +- `export ts` stays one program per broadcast. + +Tests: a synthetic two-program input with far-apart clocks fails naming both +programs; `--program 2` publishes only program 2's streams on program 2's +clock; `--program all` publishes both broadcasts, each timed correctly. Update +`doc/bin/cli.md` and the `--help` text. + +## Closes + +- [#4353](https://github.com/moq-dev/moq/issues/4353) - close this issue when the quest finishes + +## Related + +- [TS import shared shift](/quest/m1/ts-import-shared-shift.md) - its per-program shift assumes the single program this settles diff --git a/quest/m1/ts-publish-muxdelay.md b/quest/m1/ts-publish-muxdelay.md new file mode 100644 index 0000000000..688904da4d --- /dev/null +++ b/quest/m1/ts-publish-muxdelay.md @@ -0,0 +1,27 @@ +# [XS] Documented ffmpeg line flushes audio + +## Goal + +The documented `ffmpeg ... -f mpegts -pes_payload_size 0 -` publish line stops +holding small audio frames in ffmpeg's muxer for up to 0.35 s. Those clumps +raise the importer's catalog `jitter` for the life of the track, and watch +adds about 230 ms of playout delay to match. + +## Plan + +Decided: add `-muxdelay 0` everywhere the line appears +(`demo/pub/justfile`, `doc/bin/cli.md`, `doc/concept/standard.md`, +`rs/moq-cli/README.md`). `import ts` needs no PCR lead. The reporter's patch +is ready. Verify by re-measuring the catalog jitter on a quiet-audio source; +there is no automated test for a docs line. + +The importer's jitter estimate keeping its maximum forever is a separate +problem; the player side is covered by the audio jitter target. + +## Closes + +- [#4347](https://github.com/moq-dev/moq/issues/4347) - close this issue when the quest finishes + +## Related + +- [Audio jitter target](/quest/m0/audio-jitter-target/README.md) - a decaying playout estimate softens a transient burst diff --git a/quest/m1/watch-decoder-recovery.md b/quest/m1/watch-decoder-recovery.md new file mode 100644 index 0000000000..f4e208676d --- /dev/null +++ b/quest/m1/watch-decoder-recovery.md @@ -0,0 +1,28 @@ +# [M] Watch decoders recover from an error + +## Goal + +One malformed packet no longer ends audio or video for the rest of a +subscription. Today a WebCodecs error closes the decoder, both audio loops +`break` (added in #2415 to stop an `InvalidStateError` loop), and video calls +`effect.close()`; nothing rebuilds the decoder, and playback silently stops. + +## Plan + +Decided: rebuild inside the loop, for audio and video alike. On a decoder +error, rebuild through one shared `build()` helper, skip to the next keyframe, +and give up after three rebuilds in a row with no output. A rebuilt legacy +audio decoder re-runs its warmup priming. Cover the legacy and CMAF loops. + +Tests: a corrupt packet mid-stream is followed by decoded output from the next +keyframe; a stream of only corrupt packets gives up after three rebuilds +instead of spinning. + +## Closes + +- [#4324](https://github.com/moq-dev/moq/issues/4324) - close this issue when the quest finishes + +## Related + +- [Audio warmup](/quest/m1/audio-warmup.md) - the same audio loops and priming +- [Watch video guards](/quest/m1/watch-video-guards.md) - keeps bad input from reaching the video codec diff --git a/quest/m1/watch-video-guards.md b/quest/m1/watch-video-guards.md new file mode 100644 index 0000000000..765b652fc7 --- /dev/null +++ b/quest/m1/watch-video-guards.md @@ -0,0 +1,37 @@ +# [S] Watch video holds its picture and its order + +## Goal + +Two `js/watch/src/video/decoder.ts` defects stop reaching the viewer: + +- Promoting a video track when none is active (after pause, scrolling back + into view, or a hidden rendition returning) holds the last picture instead + of painting black for a round trip plus a keyframe. +- An older finished group accepted behind the live group is never fed to the + codec between live deltas. + +## Plan + +Decided: + +- `#runActive` ignores the new track's initial `undefined` frame; clearing + stays with `#clearCurrentFrame` and `close()`. The held timestamp stays + with the held frame, so the outputs describe what is on screen. +- The guard lives in both video decode loops (legacy and CMAF): skip a frame + whose group is older than the newest group already fed, and reset on a + discontinuity. The consumer keeps serving older groups, since audio's + timestamp-indexed ring needs them. Fix the consumer comment claiming video + drops a late frame at render; that only holds after decode. + +Tests: a promote from nothing keeps the previous frame and timestamp; an older +group arriving after a newer one never reaches the decoder. + +## Closes + +- [#4338](https://github.com/moq-dev/moq/issues/4338) - close this issue when the quest finishes +- [#4339](https://github.com/moq-dev/moq/issues/4339) - close this issue when the quest finishes + +## Related + +- [#3056](/quest/m1/3056-watch-video-decoder-captures-the-rewind-generation-at.md) - another fix in the same decoder +- [Watch decoder recovery](/quest/m1/watch-decoder-recovery.md) - a bad feed today ends video for the subscription diff --git a/quest/m1/watch-worklet-file.md b/quest/m1/watch-worklet-file.md new file mode 100644 index 0000000000..b6a778364a --- /dev/null +++ b/quest/m1/watch-worklet-file.md @@ -0,0 +1,29 @@ +# [S] Watch audio under a strict CSP + +## Goal + +A page with CSP `script-src 'self'` plays watch audio. Today +`js/common/vite-plugin-worklet.ts` always exports the render worklet as a +`blob:` URL, so `addModule` is refused and audio never starts, with only a +console "spawn error". + +## Plan + +Decided: in `vite build`, emit the worklet as its own file and export +`new URL(".js", import.meta.url)`; keep the blob URL only in +`vite serve`. Vite, webpack 5, and esbuild (with a plugin) resolve that +pattern inside a dependency. This applies to the publish capture worklet too +and deletes the runtime blob code. Document the consumer-bundler requirement. +The publish capture worker (`?worker&inline`) is out of scope. + +Verify in a browser under `script-src 'self'` (Chromium and Firefox) and +from a Vite consumer app. + +## Closes + +- [#4323](https://github.com/moq-dev/moq/issues/4323) - close this issue when the quest finishes + +## Related + +- [JS bundle trims](/quest/m1/js-bundle-trims.md) - minifies worklets through the same plugin +- [Plan: watch worker](/quest/m1/plan-watch-worker.md) - notes the capture worker's `worker-src blob:` need diff --git a/quest/m2/redundant-ingest.md b/quest/m2/redundant-ingest.md index 96ce3c0644..241fe5bb33 100644 --- a/quest/m2/redundant-ingest.md +++ b/quest/m2/redundant-ingest.md @@ -25,5 +25,6 @@ decision, with a quest for the chosen mechanism. ## Related +- [Same-hop importers](/quest/m1/hop-aligned-import.md) - identical tracks from one encoded stream under one first hop, the case this generalizes - [Cluster routing](/quest/m1/cluster-routing.md) - decides what replaces first-hop failover inside a cluster - [Broadcast epochs](/quest/m1/broadcast-epoch/README.md) - explicit epochs are what a redundant pair would share