diff --git a/.github/ISSUE_SPECS/1106-spectrum-hop-probe-waits-on-the-clock.md b/.github/ISSUE_SPECS/1106-spectrum-hop-probe-waits-on-the-clock.md index 340443678..de10b1e2f 100644 --- a/.github/ISSUE_SPECS/1106-spectrum-hop-probe-waits-on-the-clock.md +++ b/.github/ISSUE_SPECS/1106-spectrum-hop-probe-waits-on-the-clock.md @@ -47,3 +47,40 @@ Authorized paths: `hosts/host-web/qualification/sdk-response-entry.ts`, any sibl 2. A planted slow start (for example a 500 ms delay before the first pump) still passes, while the same delay with the old attempt bound fails. Record both as PR evidence. 3. A probe that never publishes still fails with its typed message within the deadline. + +## Decision record: hop probe wait (batch 8) + +Shipped. `runConfiguredSpectrumHopQualification` waits through `waitForCount`: it polls `pump()` +and then sleeps 5 ms, and stops at a 10 s `performance.now()` deadline. There is no poll count. +The failure still names what it saw: the windows seen with their captured samples, the elapsed +time, the poll count, the notification count and the context state. +The only sibling bounded by an attempt count was `submitBlocks` in the collection probe (200 +retries of 5 ms on source backpressure). It now stops at the probe's 10 s readiness budget. + +Departures from the text, and why: +- The wait starts at the click, not when `resume()` resolves. Firefox resolves `resume()` only + when its audio has started, so a wait that began after the resolve could not observe a slow start. +- The wait ends on two windows one hop (1,024 samples) apart, not on the first two windows. A + late start leaves the first reads behind the audio and the capture ring drops windows (WebKit + published windows 7,168 samples apart under the planted delay). The hop is the distance between + two consecutive publications, so a hop bug still times out and fails; the row reports that pair. +- The slow start is planted by holding the native `startSpectrumStream` for 750 ms after the + click. The context is running and every `pump()` returns at once with nothing published. A + `resume()` outside the click handler is not possible (the `resume-clicks` gate), and a worklet + that stalls the audio thread did not work: it blocks `pump()` too, so the old loop survived it. + +Gates (`run.mjs`, row `sdk-spectrum-hop-wait`): the first publication is at least 650 ms after the +click; the wait ended before its deadline with a bounded poll count; a wait that can never be +satisfied (250 ms deadline, nothing published) ends at its own deadline and polls no faster than +its timer. Self-test mutations: `sdk-spectrum-hop-wait-fast-start`, `-early`, `-spin`, +`-never-unbounded`. + +Measured (first publication / second publication after the click, polls): Chromium 813 / 836 ms, +162 polls; Firefox 836 / 889 ms, 150 polls; WebKit 1289 / 1344 ms, 165 polls. The never-publishing +wait ended at 254 ms (deadline 250 ms) with 35 to 50 polls. + +Red evidence: under the same 750 ms plant, the old 512-poll loop (restored temporarily after the +resume, then removed) fails in Chromium (`published 0 windows` after 512 polls in 13 ms) and WebKit +(512 polls in 10 ms). It passes in Firefox, whose `resume()` resolves only after the delay. With a +3 s plant and a 1 s deadline the new probe fails in bounded time with its typed message (`published +0 windows in 1001 ms (deadline 1000 ms, 197 polls ...)`). diff --git a/.github/ISSUE_SPECS/1290-replace-the-running-browser-session-in-the-rust-host.md b/.github/ISSUE_SPECS/1290-replace-the-running-browser-session-in-the-rust-host.md index a8accd7bd..76efee23f 100644 --- a/.github/ISSUE_SPECS/1290-replace-the-running-browser-session-in-the-rust-host.md +++ b/.github/ISSUE_SPECS/1290-replace-the-running-browser-session-in-the-rust-host.md @@ -415,3 +415,157 @@ Gates -> tests: `structural_apply::the_catalogs_describe_the_successor_from_the_ successor's full one-slot queue, leaves the rendering plan unarmed and arms the successor's compressor at the adoption block); `meter_windows_a_strip_count_change_leaves_unpublished_are_counted`; `a_rebuild_whose_tables_cannot_reach_the_host_is_refused_before_the_commit` (nothing committed). + +## Amendment (controls coordinator, batch 12, 2026-10-15): (d+e) + +Checked against `ea336c039`. + +- **Where things are.** #1381b pairs the meter consumers and the observation readers in the swap + block (`RenderCompanions::adopt_predecessor`, `hosts/host-web/src/lib.rs:5190`). `adopt_shape` + (`:1562`) moves the meter pending windows and the tap state. A swap that carries a meter keeps + `meter_generation`. What is not done yet: + - The published frame does not grow or shrink with the plan. `meter_frame` is sized at boot + (`:1450`, `:6054`), and `poll_meters` publishes nothing when the strip count differs (`:3411`). + The worklet cuts fixed views and builds a fixed `meterMessage` at boot from the header's counts + (`hosts/host-web/web/miso-engine-v1-audio-worklet.js:728-747`). + - The successor's spectrum capture is not paired (#1395 is open). + - #1381's gate 3, the frame with a changed strip count (`track_count` 3, the shifted strip's word, + `master_track_plus_one` 3), was left to (d) by #1381b. +- **D7.** The batch 10 amendment still holds: meters and taps pair in the swap block, and spectrum + pairs at promotion through #1395's `SpectrumService` method. +- **Meter frame growth, options.** `process()` must not cut a view or allocate (#1562a record). + - (1) Size the frame and the worklet's views for a boot maximum of strips. The `3(T + S) + 3` + layout is packed by count, so the view offsets still move with `T`. This needs a new layout at + capacity offsets, which changes the ABI rows and `sdk/src/core/boundary.ts`, and every post + clones the capacity. + - (2) The frame goes with the plan's tables. It is allocated in `build_plan_shape` beside the + per-meter scratch and exchanged in `adopt_shape` (a pointer exchange). The message handler that + commits the rebuild also cuts the successor's views and builds its `meterMessage` in advance. + `process()` changes to them by reference at the first poll after adoption. + `rebindMemoryViews` also cuts the pending views again. The layout and the ABI rows do not + change, except for an export that gives the pending frame pointer, if one is necessary. + - **Recommendation: (2).** It is the #1290a `PlanShape` pattern, and it keeps exact sizes. +- **A handle shift must not lose a carried window.** The #1381b fold records this: a snapshot + published before the swap keeps the old meter handle. When the handles shift (an added strip + sorts first), the poll's handle check drops the snapshot and counts it as loss. Gate 7 polls after + every block, so gate 7 does not see this. (d) matches a carried pending snapshot by its carried + slot, and adds a gate: gate 7's script with a poll every second block shows no counted loss. +- **Split, so lane W does not wait for lane K:** + - **(d1) meters.** Option (2), #1381's gate 3 with a changed strip count, the handle-shift gate, + and gate 7's meter part. This has no lane K dependency. + - **(d2) spectrum.** Gate 7's spectrum part: at promotion, take the successor's capture out of + the adopted plan's attachment and call #1395's method with the pairing list kept beside the + `PendingShape`. Dependencies: #1395a and #1395b. +- **(e) needs no code.** Gate 9's native part is already met by (a): + - (a)'s gates run on one thread, so the apply runs between render calls, as the message handler + runs it. + - `blocking_rebuilds() == 1` is asserted at `hosts/host-web/src/tests.rs:6150` and + `hosts/host-web/tests/structural_apply_realtime.rs:393`. + - The browser legs are #1295's. + - Record (e) as met when (d2) closes. +- **Restarted and removed effect owners.** A subscription on an owner that restarts or is removed + is #1584 D6a and D6b, not (d). (d)'s gate 7 has no effect. +- **No edit under `crates/engine/src/realtime`.** +- **Order in lane W:** (d1), #1584 D6b, (d2), #1562b-W. The four slices edit `hosts/host-web/src/lib.rs` + and `src/tests.rs` in that order. (d1) and #1562b-W also edit the worklet JS and the ABI layout + mirrors, in that order. + +## Ruling (root, 2026-10-15): meter frame growth, (d1) + +The meter frame travels with the `PlanShape`; its views are cut in the message handler that +commits (control side, counted on a non-isolated page), and `process()` switches to them by +reference at the swap block. `process()` never cuts a view or allocates. Gate: a strip-count change +publishes a correctly shaped frame on the first block after adoption, with zero render +allocations. (d1) lands in batch 12a; (d2) in batch 12b after #1395. + +## Decision record: (d1) meters (batch 12, lane W) + +**Shipped.** A strip-count change no longer leaves the browser publishing nothing. The meter frame and +its header travel with the `PlanShape` (root's ruling); the page cuts its views over the successor's +frame in the message handler that commits, and `process()` switches to them by reference. + +- **Frame and header in the shape.** `PlanShape` owns `meter_frame` (`3(T + S) + 3` words, built by + `build_plan_shape`), `meter_header` (counts, master index and frame address from the plan; the poll's + stream fields at run time) and `meter_carried_from`. `ReadyOwnership` lost its own frame and header. + `adopt_shape` copies the stream fields (`windows`, `first/end_sample`, `sequence`, `master_gr_present`, + `reserved[0..1]`) into the adopted header: the same stream continues in a frame of the new size. A + pointer exchange and scalar writes; nothing allocates in the swap block. +- **`WebMeterHeader` 72 -> 80 bytes, nothing moved.** `reserved_pad` (68) became `retired`; appended + `frame_pointer` (72) and `successor_pointer` (76). A commit writes the newest candidate's header + address into the rendering plan's `successor_pointer` (`run_command`); the adopting block marks the + header it leaves `retired = 1`, names the adopted header in its `successor_pointer`, and points the + adopted header at any candidate still waiting. ABI mirrors regenerated (`abi_layout.rs`, + `sdk/assets`, `sdk/src/generated/abi.ts`, `check-abi-layout-v1.py`, its self-test fixture, the + direct oracle, `docs/BUILTINS_AND_METERING_V1.md`). +- **Worklet.** The views are per meter set (`meterSet`, `pendingMeterSet`), cut by `cutMeterSet` in + `syncMeterViews`, which every message handler runs last (and `rebindMemoryViews` re-cuts both sets + on a memory growth). `postMeterFrame` reads the active header's `retired` word; if set it switches + `meterSet = pendingMeterSet` (a reference assignment) before the poll; a retired header with no set + cut for the header it names is sticky `RESULT_INTERNAL` (cannot happen in single mode). A lease + released across a swap is caught by `syncMeterViews`, which compares the host's current header + address with the page's. The pinned post spelling in `check-web-audioworklet.sh` and its mutation in + `test-web-audioworklet.sh` follow (`this.port.postMessage(set.message)`: the set's message is + allocated with its views, in the handler). +- **Handle shift.** A meter's handle is its position, so a strip added or removed in front shifts the + survivors' handles. The hook records both handles per carried meter (`CarriedReader`), the shape keeps + the predecessor's in `meter_carried_from[successor]`, and the poll accepts a snapshot named by either. + A queue is one strip's, so the second handle names the same strip (by carry identity, not position). + Not kept: a window two swaps old still names a handle the table no longer holds and is dropped with a + count; it needs a second swap between two polls. +- **Counted loss kept.** A window the predecessor closed before an added strip existed cannot be + described by the successor's frame (that strip has no data for it): it is dropped with a count by the + poll that finds it, as before. A removed strip's window is published without it (gate below). +- **Charging.** The frame was an uncharged allocation at boot. It is now a row of the shape + (`ShapeRows::frame_bytes`, `carried_bytes`): the boot adds it to the bridge rows, a replacement to its + plan row. `observation_staging_resource_is_exact_for_the_taps_off_and_on_modes` gained the frame + term. Boot high-water growth 20,447,232 bytes (before: 20,447,232); `check-browser-expected-resources` + green with no re-pin. + +**Departures.** None from root's ruling. Mechanical edits outside lane W's own files, listed for the +coordinator: `tools/realtime-policy/render-thread-exports.json` and `src/policy.rs` (the handler now +calls `meter_header_ptr` after construction: one table row and one named unwrapped export), +`scripts/check-web-audioworklet.sh` and `scripts/test-web-audioworklet.sh` (the pinned post), the ABI +mirrors named above. + +**Gates -> tests.** +- Root's gate (a strip-count change publishes a correctly shaped frame on the first block after + adoption, zero render allocations): `tests/structural_apply_realtime.rs` + `a_strip_count_change_publishes_the_successors_frame_on_the_first_block_after_adoption` (render and + `meter_poll` under the render-locked counters, adoption block included; every frame after it equals a + fresh boot of the committed model for the same window); `structural_apply:: + a_strip_count_change_publishes_the_successors_frame_from_the_adoption_block` (the same in the lib, with + the retired/successor words). +- #1381 gate 3 (changed strip count, `track_count` 10, shifted words): the two tests above. +- Handle shift: `a_window_closed_before_a_handle_shift_is_published_by_its_carried_strip` (remove the + first strip, poll every second block so the window is polled after the swap: published, sequence + continues, no counted loss; red when the carried-from acceptance is disabled, checked). + `an_aligned_poll_every_second_block_loses_nothing_across_an_added_strip` is gate 7's cadence variant. +- Counted loss: `windows_an_added_strip_cannot_describe_are_counted_and_the_next_frame_is_the_successors` + (replaces `meter_windows_a_strip_count_change_leaves_unpublished_are_counted`); + `meter_delivery_carries_through_a_swap_that_keeps_the_strips` (first half of the old + `..._never_publishes_a_misshaped_frame`). +- Worklet: three blocks in `scripts/test-web-audioworklet.mjs` (views cut in the handler and switched by + reference at the adopting block, sequence continues; lease released across a swap; retired header with + no pending set is sticky). + +**Limits found.** (1) The shipped module cannot run a rebuild until #1562b, so the real-wasm pointer +chain (`retired`, `successor_pointer`) is covered by the hermetic worklet fake and, on native, by +equality with `address_word` (which is `0` for a 64-bit address); the chromium leg proves the boot +`frame_pointer` path with real addresses. (2) The SDK's `meterProjection` still checks a frame against +its own session map, so after a structural apply it refuses the successor's frame until the SDK map +follows the apply (#1296's); the engine side is complete. + +## Fold (batch 12a review) + +- **Pointer checks at full width.** The native check of `successor_pointer` compared two + `address_word` results, and `address_word` is `0` for any address above 4 GiB, so on a 64-bit + host both sides were `0` and the check was always true (the "Limits found" note above recorded + the weakness and left it). `PlanShape` now keeps the full-width address beside the header word + (`name_successor` sets both where the commit and `adopt_shape` set the word; test builds only), + and the gate `a_strip_count_change_publishes_the_successors_frame_from_the_adoption_block` + compares that address with the candidate's and the adopted plan's header addresses + (`AudioWorkletEngineHost::meter_successor_address`). Quick mutation, reverted: with the + address not recorded the gate fails (`assertion left != right`, tests.rs:6777). +- **Doc fixes.** `miso_engine_web_v1_meter_header_ptr` no longer says "stable" (the address changes + at each swap, each plan carrying its own header); the browser oracle's header size comment says + 80 bytes. diff --git a/.github/ISSUE_SPECS/1395-carry-spectrum-capture-state-across-a-plan-swap.md b/.github/ISSUE_SPECS/1395-carry-spectrum-capture-state-across-a-plan-swap.md index 332e7dfa7..8817f7748 100644 --- a/.github/ISSUE_SPECS/1395-carry-spectrum-capture-state-across-a-plan-swap.md +++ b/.github/ISSUE_SPECS/1395-carry-spectrum-capture-state-across-a-plan-swap.md @@ -199,3 +199,148 @@ Checked against `12d4a4f9d`. - **Split (plan v2).** #1395a: the single capture (gates 1, 3's single part, 4); #1395b: the collection (gate 2, 3's collection part). - **Dependencies.** #1327a (the table, the record and the section). #1401 is closed. + +## Amendment (controls coordinator, batch 12, 2026-10-15) + +Checked against `ea336c039`. + +- **No engine edit.** #1327a declared `ObserverKind::Spectrum` (`crates/engine/src/realtime/plan.rs:548-555`). + This slice edits nothing under `crates/engine/src/realtime`. +- **The observer table is generic.** `GraphRuntimeObserver::carry_kind` returns + `Option` (default `None`, `crates/graph/src/lib.rs:2887`). `GraphObserverRow` and + `GraphObserverMove` carry their kind (`:3519`, `:3533`), and the move section writes the kind of + each move into the record. So D2 is: `SpectrumCaptureObserver` (`spectrum.rs:2497`) answers + `Some(Spectrum)`, and host-core gives spectrum rows keyed by D1, as `MeterInventoryRow` and + `resolve_meter_carry` do for meters (`crates/host-core/src/prepare.rs:3152`). A graph edit is still + "only if needed". The spectrum observer's binding handle is positional + (`SPECTRUM_OBSERVER_HANDLE - index`, `spectrum.rs:31`). So when a collection is reordered, the + observer swapped in must take its new binding's handle (`adopt_binding_handle`), as the meter + observer does. +- **D3 and D5 as amended in batch 10 still hold.** Pairing happens on the control side when the + adopted candidate is promoted, as one `SpectrumService` method (`spectrum_service.rs:585`) that + uses D3's two functions. No `pair_carried` exists yet. Add `crates/host-core/src/spectrum_service.rs` + to the authorized paths (the batch 10 amendment requires it, but the list was not changed). +- **Where the browser's successor capture is.** The boot plan's capture is moved into + `SpectrumService` (`take_spectrum_capture`, `hosts/host-web/src/lib.rs:5139`, called at `:5932`). + A successor's capture stays in the successor's `RenderCompanions` (`:5055`). #1290 (d) takes it + out at promotion, in the service step after the adoption block, and calls this slice's method. + Between adoption and promotion, a carried entry is served by the capture that the service + already holds, because its observer was swapped. An entry that is not carried reads as pending + until promotion. +- **A refused hand-over.** `RealtimePlanOwner::carry_mismatch_count` + (`crates/engine/src/realtime/plan_exchange.rs:719`) is readable at promotion only when the + control and render halves are one instance (single mode, the only browser mode today). The C ABI's + #1593 and the browser's Worker mode (#1332) must publish it to the control side. Record this as + a rule for those slices. This slice does not change it. +- **Anchors.** The batch 10 anchors in `spectrum.rs` are unchanged. The successor entries are now at + `prepare.rs:1339` (single) and `:1371` (collection). +- **#1618** lands after #1395b, as root ruled. Its settle is part of the same `SpectrumService` + method: an entry that is not carried, or all entries after a refused hand-over. So + `spectrum_service.rs` is edited in this order: #1395a, #1395b, #1618. +- **Order in lane K.** #1395a, then #1395b, after #1584 D6a. Both slices edit + `crates/host-core/src/prepare.rs` and `crates/host-core/tests/successor_swap.rs`. + +## Decision record: #1395a, the single capture (batch 8) + +**Shipped.** A single spectrum capture carries across a plan swap, and the control side keeps +reading it. + +- *Swap block (D1, D2).* `SpectrumCaptureObserver` answers `carry_kind() = Some(Spectrum)` + (`spectrum.rs`). `prepare.rs` gives each single capture an inventory row keyed by + `(target, channels)` (`SpectrumInventoryRow`), and `resolve_spectrum_carry` joins the rows with the + predecessor's by key, never by entry or handle, and only at equal sample rate and quantum (both + are now in `PlanStateInventory`). The moves go into the one observer section with the meters' + (`install_observer_carry`), and a carry program is requested when a capture may carry + (`spectrum_may_carry`). No engine, graph or `latest_cell` edit. The observer keeps no binding + handle, so `adopt_binding_handle` has nothing to do (a collection's reorder in #1395b needs no + handle work either). +- *Pairing at promotion (D3, D5, batch-10 amendment).* The pairing list is built at preparation + from the same join (`PlanStateInventory::spectrum_pairing`, `SpectrumPair`), so it names exactly + the observers the swap block moves. `SpectrumCapture::pair_carried(&mut self, predecessor, + pairs)` swaps the two captures only when the list names entry 0 of both and their target and + channels are equal. `SpectrumService::adopt_successor(successor, pairing, carried)` applies it + when the adoption carried, installs the successor's fresh captures otherwise (a refused + hand-over), and returns the displaced captures so the caller drops them off the audio thread. + The service is allocation- and free-free here. No host pairs in the swap block. +- *Departures.* (1) `pair_carried` takes the control-side pairing list, not the swap block's + `ObserverCarry` slice (the record exists and a test proves it equals the list). (2) The + inventory carries only single captures' rows: a collection has none, so it never moves an + observer; #1395b adds its rows and the collection arm of `adopt_successor`. (3) An in-flight + arm or stream on a capture that does not carry is not settled by `adopt_successor`; that is + #1618's, in this method. (4) `browser_successor.rs` gate 2 read the successor's own capture; that + observer is now carried away, so the test pairs the captures first (a mechanical update). + +**Gates -> tests** (`crates/host-core/tests/successor_swap.rs`, `mod spectrum_carry`): +gate 1 `a_continuous_capture_carries_and_its_reader_keeps_receiving` (status, window fields and +planes equal the reference's block by block, epoch 1, no gap; record == pairing list); +`a_refused_hand_over_installs_the_fresh_capture` (the oracle can fail); +gate 3 `an_armed_one_shot_completes_across_the_swap` (a window that spans the swap, equal to the +reference's) and `a_changed_key_starts_fresh`; gate 4 +`the_swap_block_and_the_promotion_allocate_and_free_nothing` (zero allocator calls in every +block and in `adopt_successor`). Unit tests: `spectrum::tests::pair_carried_swaps_only_a_named_pair_of_one_key`, +`prepare::tests::a_spectrum_capture_carries_by_key_at_one_shape`. Gate 2 (collection) is #1395b. + +## Decision record: #1395b, the collection (batch 8) + +**Shipped.** A collection's entries carry across a plan swap, matched by `(target, channels)`, and +the control side keeps each entry's reader. + +- *Swap block (D1, D2).* `prepare.rs` gives every collection entry an inventory row (`entry` is its + index, its location is looked up by the positional handle `observer_handle(entry)`), and + `resolve_spectrum_carry` joins the rows by key in any order, at equal rate and quantum, into the + same observer section and the same pairing list as the single capture. No engine, graph or + `latest_cell` edit. A collection's reorder needs no handle work: the observer keeps no binding + handle (#1395a). +- *Pairing at promotion (D3).* `SpectrumCaptureCollection::pair_carried(&mut self, predecessor, + pairs) -> bool` swaps each named pair of captures (successor index, predecessor index). It + checks the whole list first (index range, equal key per pair, successor entries strictly + increasing; two pairs on one predecessor entry cannot pass, because a collection admits no + duplicate key); a fault pairs nothing, like an empty list. The selection follows its entry: the + successor selects the entry's new index and keeps `selection_epoch`; when the predecessor's + selected entry did not pair, the successor has no selection and its epoch is the predecessor's + plus one; a predecessor with no selection leaves none and its epoch. It runs for an empty list + too. `SpectrumService::adopt_successor` calls it for a collection with the list when the + adoption carried and with an empty list after a refused hand-over, so a refused hand-over also + advances the epoch. Allocation-free, drops nothing; the caller drops the displaced collection. +- *Departures.* (1) `PlanStateInventory` records whether its captures are a collection + (`spectrum_collection`). A single capture and a collection never carry into each other: the + service holds one kind and pairs inside it, so a plan whose capture kind changed starts fresh. + The spec does not say; no host changes kind across a swap today. (2) As in #1395a, + `pair_carried` takes the control-side `SpectrumPair` list, not the swap block's record (a test + proves the two are equal). (3) The O(n) validation pass uses the key-uniqueness argument above, + not a set. (4) Not settled here, and left to #1618: the stream or arm that the service holds when + the selected entry did not carry. After a lost selection the service still has `stream_active`; + a later `select_stream` then arms a one-shot (a collection restarts a cadence only from a + selected entry), so #1618's settle must reset the service's stream state in the same method. + +**Gates -> tests.** `crates/host-core/tests/successor_swap.rs`, `mod spectrum_carry`: gate 2 +`a_reordered_collection_pairs_by_key_and_its_reader_keeps_receiving` (B lists the entries in the +other order; pairing `[(0,1),(1,0)]` equals the swap block's record; the stream on `t` reads window +for window with no `Failed`/`Gap`/`Warming`, epoch 1, no sequence gap, a window across the swap; +the selection epoch equals the reference's; a `select_stream` to the second entry (which carried +idle) reads windows equal to the reference's, bit for bit; the stop that this switch makes reaches +the rendering observer); +gate 3 collection part `a_collection_entry_with_a_changed_key_starts_fresh` (`u` Left to Right: one +pair, one record, the changed entry reads its own plane like the reference); +`a_refused_hand_over_installs_the_fresh_collection_without_a_selection` (no selection, epoch 2, the +displaced collection keeps its selection); gate 4 +`a_collection_swap_block_and_promotion_allocate_and_free_nothing` (zero allocator calls in every +block and in `adopt_successor`, two entries moved). Unit tests: +`spectrum::tests::collection_pair_carried_follows_keys_and_the_selection` (selection index and +epoch, a stop through the successor's collection leaves the rendering observer's queue empty after +a window was published without it, the other entry arms its own observer, lost selection, three +list faults), `prepare::tests::a_spectrum_collection_carries_by_key_in_any_order`. The harness +now takes either request kind (`Spec`), and the host's script (`Script`) is replaced by the +successor's at the swap. All #1395a tests are unchanged and green. + +**Commands run.** `cargo test --locked -p host-core --all-features`, `-p graph`, `-p control-plane +--features test-support`, `-p host-web --features host-web/test-support`; clippy `-D warnings` on +host-core with no features, default and all features (all targets); the host-core loom step +(`spectrum_handoff_loom`); `realtime-policy`; `check-bench-policy.sh`; `cargo fmt --all --check`; +`check-cross-targets.sh`; `RUSTDOCFLAGS='-D warnings' cargo doc -p host-core --no-deps +--all-features`. All pass. + +## Fold (batch 12a review) + +`PlanStateInventory::spectrum_count` had no caller and no gate that it would defend (the capture +rows are read through `spectrum_carry_pairs`, which the carry gates pin), so it is deleted. diff --git a/.github/ISSUE_SPECS/1562-admit-browser-live-edits-in-the-worker-through-the-committed-model.md b/.github/ISSUE_SPECS/1562-admit-browser-live-edits-in-the-worker-through-the-committed-model.md index dcc8f9916..e5cabdd84 100644 --- a/.github/ISSUE_SPECS/1562-admit-browser-live-edits-in-the-worker-through-the-committed-model.md +++ b/.github/ISSUE_SPECS/1562-admit-browser-live-edits-in-the-worker-through-the-committed-model.md @@ -449,3 +449,39 @@ Checked against `12d4a4f9d`. #1562b-W lands first, then #1562b-K, with the dependencies in the batch-10 amendment (#1290 d+e, #1280a, #1280c, #1281, #1282). + +## Amendment (controls coordinator, batch 12, 2026-10-15): #1562b + +Checked against `ea336c039`. + +- **Dependencies at this head.** Landed: #1562a, #1381b, #1290 (a), #1280a, #1280c, #1281, #1282, + #1283, #1284, #1327a, #1327b and #1622. Still open: #1290 (d1) and (d2) (see #1290's batch 12 + amendment; (e) needs no code). Added: #1584 D6a and D6b. Without them, the first browser rebuild + that can be reached loses an acknowledged subscription on a restarted effect owner, and ends a + removed owner's subscription silently. #1630 (the input-section carry) is still not a blocker, + because the C ABI behaves the same. +- **Anchors moved.** `REBUILD_PENDING` is at `hosts/host-web/src/lib.rs:5260`. The + `WebRuntimePreparer` override is at `:5301`, `allow_rebuild` at `:5278`, and + `structural_command_for_test` at `:2107`. The core hook is still `compile.rs:676`, and it is + called at `crates/control-plane/src/control.rs:1260`. Gate 11's test is + `hosts/host-web/tests/control_admission.rs:476`. +- **#1562b-W, additions to the batch 10 list:** + - Delete `structural_command_for_test` and the `allow_rebuild` switch with D9. Their tests + (#1290a, #1381b, #1584 D6b) then drive the rebuild through `control_command`. Delete each + test that #1562b-W supersedes in the same slice. + - The structural part of gate 3 ("a structural transaction keeps the solo"), which #1562a's + record deferred. The batch 10 list does not have it. + - Meters and observation follow the new strip order after a rebuild that adds or removes a strip + (batch 8). This includes `meterProjection`'s fixed strip IDs (`sdk/src/browser/measurement.ts:208`, + `:302`), which is lane W's. + - The structure epoch in the meter header. `WebMeterHeader` (`hosts/host-web/src/lib.rs:937`) has + a `reserved_pad` word that is always zero, or the header can grow at its end. The ABI layout rows + and the regenerated `sdk/src/generated` copies change with it. +- **#1562b-K, additions:** + - `sdk/src/core/observation-subscriptions.ts` addresses a tap by strip and effect index. After a + rebuild, it maps its managed subscriptions to the new control list order. It ends a managed + subscription on #1584's `OBSERVATION_ENDED` event and reports this to its subscribers. + - It deletes the core hook only after #1562b-W has deleted the browser's override, so the two + lanes compile at every merge. +- **Frozen decisions.** D1-D6 and the batch 10 split hold. +- **No edit under `crates/engine/src/realtime`.** diff --git a/.github/ISSUE_SPECS/1584-subscribe-to-an-effects-observation-tap-with-a-transaction-edit.md b/.github/ISSUE_SPECS/1584-subscribe-to-an-effects-observation-tap-with-a-transaction-edit.md index 2a8b2a048..743c47f77 100644 --- a/.github/ISSUE_SPECS/1584-subscribe-to-an-effects-observation-tap-with-a-transaction-edit.md +++ b/.github/ISSUE_SPECS/1584-subscribe-to-an-effects-observation-tap-with-a-transaction-edit.md @@ -209,3 +209,327 @@ Commands: `cargo test --locked -p protocol --all-features`, `cargo test --locked ## Supersedes and shrinks - None. The SDK's own sharing goes in #1590. + +## Amendment (controls coordinator, batch 12, 2026-10-15) + +Checked against `ea336c039`. + +### The defect D6 closes first (root, batch 12) + +- **What is dropped today.** A subscribe that the host acknowledged is lost at a swap when its effect + owner does not carry. The carry drain applies pending `Observe` records only for effect moves: its + second loop iterates `moves` (`crates/graph/src/runtime.rs:5443-5470`). A record in the queue of a + restarted owner (a prepared change, such as link mode) or a removed owner is never applied and + never counted, and the successor's tap is prepared unarmed (#1327b record, "Departures"; #1327 + batch 11 fold). A carried owner keeps its subscription (#1280b drain, then #1327b's + `Runtime::carry_taps`, `:5494`). +- **Which host can reach it.** Only the browser acknowledges a subscribe today (kinds 7 and 8, + through `SessionState::push_effect_observe`, `crates/control-plane/src/control.rs:2244`, on the + newest plan's producer). The C ABI has no subscribe path until D1-D5 and D7. The browser rebuild + path is test-support only until #1562b-W, so D6 must land before #1562b-W. +- **A candidate that a later rebuild supersedes** cannot hold a subscribe now: a rebuild while a + candidate waits returns `BACKPRESSURE` (`control.rs:1292`). The slice that makes supersession + reachable (#1290 (b) or #1310) owns that gate. + +### Frozen decisions that do not hold as written + +- **D3, where the table lives.** #1559 shipped the monitor overlay as `controls::MonitorOverlay` + (`crates/controls/src/effective.rs:28`), held at `SessionState.monitor` (`control.rs:403`), and not + on `ProtocolController`. The monitor edits are `controls::MonitorWrite` (`effective.rs:103`), not + `MonitorEdit`. `host_core::SuccessorBase` already passes the overlay to every successor preparation + as `MonitorBase { committed, next, ramp }` (`crates/host-core/src/prepare.rs:1004`, `:1034`). + - Option 1, as filed: `ObservationBindings` on `ProtocolController`, and a new `armed_taps` + argument through `RuntimePreparer::prepare` and every host-core successor entry. That gives two + owners of monitor state and two paths into the preparation. + - Option 2: the bindings are a field of `controls::MonitorOverlay`. `MonitorOverlay::after` applies + the observe writes in transaction order and drops each binding whose target the next model does + not have, and `MonitorChange` returns those ended keys. Host-core arms the taps of + `MonitorBase.next` when it prepares a successor. No preparation signature changes. + - **Recommendation: option 2.** D3's sharing rule (refs, `observation.window_conflict`, + `observation.not_subscribed`) does not change. Only its owner changes. +- **D6, `armed_taps`.** With option 2, `ArmedTap` and the `armed_taps` argument are not needed: + host-core resolves `MonitorBase.next`'s bindings against the successor's model. +- **D6, "opens its first window at its first rendered block".** At preparation the adoption sample + is not known. `ObservationLane::arm` takes the window's `first_sample` + (`crates/effect-contract/src/live.rs:1630`), and `accumulate` keeps it (`:1754`). So the slice + adds a prepared arm to `ObservationLane`: the tap is armed, and `accumulate` takes the block's + `first_sample` as the window start on its first block. `swap_state` moves this state with the tap + table, so a carried tap replaces the prepared arm completely. Do not queue a record into the + successor's own queue instead: a carried tap would then be re-armed by its first block, and its + window would restart. +- **D6, the removed owner.** As filed, the binding goes silently. Root needs a typed end. Options: + - (A) A reliable event, `OBSERVATION_ENDED { event_sequence, revision, strip_id, section, tap_id }`, + one for each ended binding, queued with the transaction's `SESSION_COMMITTED`. The room for + all of these events is checked before commit (`required_events`, + `crates/protocol/src/controller.rs:1897-1901`). If there is not sufficient room, the transaction + is refused with `BACKPRESSURE` and nothing changes. The end takes effect at the revision's + watermark. All listeners see it on both hosts (the browser drains the lane since #1570b). + - (B) A list of ended keys in the transaction reply. Only the sender sees it, and other + subscribers stay silent. + - (C) Refuse a transaction that removes a subscribed target (`observation.subscribed`) until every + subscriber disarms. This is honest, but a mixer cannot remove an insert while a meter shows it, + and shared refs make this worse. + - **Recommendation: (A).** A binding also ends when the next model keeps the owner but its tap ID + does not resolve to a `Resident` tap (an insert ID reused for another effect type). +- **Records left in a predecessor queue that did not move.** With D6, these records do not need a + counter. The overlay is the record of what was acknowledged, and the successor is prepared from + it. The queue is not that record. This closes the #1327 batch 11 fold item. +- **D7a.** Holds. Until #1589, the browser's kinds 7 and 8 write the overlay's bindings through + `push_effect_observe` (absolute set and clear, refs 1). The `Observe` edit is refused on the + browser (`ProviderFeatures::observation` false), so one host never has two writers of one + observation queue. +- **D7, the capacity request moved.** Since #1400 the C ABI asks for its live request in + `CapiPreparer` (`crates/capi/src/runtime/mod.rs:97`, `HostLiveControlRequest::default()`, + `observation_taps` 0). `prepare_runtime` (`crates/control-plane/src/compile.rs:689`) keeps the + strip, effect and route producers and drops the other handles (`:713-719`). D7's edits go there. + Add `crates/capi/src/runtime/mod.rs` to the authorized paths. + +### The batch 12 slice: #1584 D6a (lane K), then D6b (lane W) + +- **D6a scope.** Option 2's bindings on `controls::MonitorOverlay`, with the ended keys from `after`. + `push_effect_observe` writes the bindings after a successful push. Host-core arms the bindings of + `MonitorBase.next` in each successor preparation that attaches observation, with + `attach_effect_observation` (`crates/effect-compiler/src/prepare.rs:1776`) and the prepared arm on + `ObservationLane`. `OBSERVATION_ENDED` (option A) is added to the protocol wire, schema, corpus, + and `docs/CONTROL_PROTOCOL_{REGISTRY,SEMANTICS}.md`, with its room check. D6a does not include + D1-D5, D7 or D8's C ABI parts. The C ABI keeps `observation_taps` 0, so its overlay has no + bindings yet. +- **D6a gates** (control-plane test-support, with a test preparer that asks for observation + capacity; and host-core `successor_swap.rs`): + 1. Restarted owner, subscribe already applied: compressor insert `c1` on `t1`, gain-reduction tap + armed with window 4, 6 blocks rendered. Commit a prepared-only change of `c1` (link mode). After + adoption, the successor's tap is armed with window 4, its first window opens at the adoption + block's first sample, and every window equals a fresh plan of the committed model with the tap + armed at that block. + 2. Restarted owner, subscribe still pending (root's case): the same, with the arm admitted after + the last block before the swap, so that it is still in the predecessor's queue at the swap + block. The result is the same as gate 1. + 3. Pending disarm on a restarted owner: the tap was armed before, and the disarm is still in the + queue at the swap. The successor's tap is unarmed and publishes nothing. + 4. Removed owner: commit the removal of `c1`, then a variant that removes `t1`, then a variant + that reuses `c1`'s ID for an effect without that tap. The binding is gone. The reliable lane + holds `SESSION_COMMITTED` and one `OBSERVATION_ENDED` with the committing revision and the + stable key. The successor has no such tap. + 5. Room before commit: the reliable lane has room for `SESSION_COMMITTED` but not for the ended + event. The removal is refused with `BACKPRESSURE`, and the model, revision, overlay and plan do + not change. After a drain, the same request commits. + 6. Carried owner: #1327b's `observation_carry` tests stay green. A carried tap that the overlay + arms continues its window sequence with no restart: the prepared arm never wins over the + swapped state. + 7. Realtime: zero allocations and frees on the swap block and the successor's first block. The + `ObservationLane` change stays inside its `REALTIME_POLICY` region, and `realtime-policy` + passes. + - `crates/host-core/tests/effect_observation.rs:534` (`a_plan_replacement_drops_every_subscription`, + E13) is superseded. Rewrite it: a replacement prepared with the overlay keeps the subscription, + and a replacement prepared with `MonitorBase::NONE` starts unarmed. +- **D6a authorized paths:** `crates/controls/src/effective.rs`, + `crates/control-plane/src/control.rs`, `crates/host-core/src/prepare.rs`, + `crates/effect-compiler/src/prepare.rs` (`attach_effect_observation` only), + `crates/effect-contract/src/live.rs` (`ObservationLane` only), + `crates/protocol/src/{message_wire.rs,schema.rs,controller.rs,lib.rs}`, + `crates/conformance/src/protocol_corpus.rs`, `fuzz/corpus/complete-schema-manifest.md`, the + host-core and control-plane tests above, and `docs/CONTROL_PROTOCOL_{REGISTRY,SEMANTICS}.md`. +- **D6b (lane W, after D6a).** `build_plan_shape` (`hosts/host-web/src/lib.rs:5437`) sets + `observation_armed` for the bindings of `MonitorBase.next`, with arm sample 0 and the commanded bit + clear: a fresh tap has no stale window. Gates through `structural_command_for_test`: + - After a restart of a subscribed compressor, the meter frame's gain-reduction word and the + observation read return windows from the adoption block on. + - After an insert removal, there is no slot for the removed insert, and the browser's event drain + returns `OBSERVATION_ENDED`. + - `tests/structural_apply_realtime.rs` stays at zero render-locked allocations. +- **The SDK's reaction** (`sdk/src/core/observation-subscriptions.ts` ends a managed subscription + on `OBSERVATION_ENDED`) goes to #1562b-K. +- **No edit under `crates/engine/src/realtime`.** +- **Remainder of #1584.** D1-D5, D7 and D8 (the C ABI `Observe` edit) are a later slice. It writes + the same bindings through `MonitorWrite`, and gate 5's removal case then also checks + `observation.not_subscribed` after the end. + +### Anchors moved + +- `hosts/host-web/src/lib.rs`: kinds 7 and 8 are at `:754` and `:756`, `into_observe_record` at + `:3944`, and the admission push is `push_observe` at `:1759`. +- `crates/effect-contract/src/live.rs`: `Observe` `:126`, `apply_observe` `:1471`, + `ObservationLane` `:1528`, `arm` `:1630`, `disarm_all` `:1670`. +- `crates/effect-compiler/src/prepare.rs`: `try_push_observe` `:782`, `observe_available_capacity` + `:801`. +- `crates/protocol/src/controller.rs`: `ProviderFeatures` `:109`, `telemetry_configuration` + `:663`, `commit_prepared_structural` `:2150`. +- `crates/host-core/tests/effect_observation.rs`: the replacement test is at `:534`, and + `scalar_reference_windows` at `:587`. +- The capi race loop is at `crates/capi/tests/resource_lifecycle.rs:3350`. +- `sdk/src/core/observation-subscriptions.ts`: `ObservationSubscriptionOwner` `:477`, `refs` + `:212`, the conflict `:570`. +- `SessionState::command` is at `control.rs:1208`, and `miso_engine_v1_submit_command` at + `crates/capi/src/ffi.rs:680`. +- `host_core::launch_maximum_observation_taps` does not exist yet (D7 adds it). + +## Ruling (root, 2026-10-15): batch 12 + +1. **D3, where the bindings live.** Neither as filed nor inside `MonitorOverlay`: solo and + subscriptions are different things. The bindings get their own type (for example + `ObservationBindings`), owned by `SessionState` next to the monitor overlay, and they travel to + every successor preparation in `SuccessorBase` the same way the monitor does (a field, not a new + preparer argument). Its `after()` drops the bindings whose target is gone and returns their keys. + `ProtocolController` does not own them. D6's `ArmedTap`/`armed_taps` are replaced by this. +2. **D6, a removed owner.** A reliable `OBSERVATION_ENDED` event (revision and stable key), + room-checked with `SESSION_COMMITTED` before the commit; no room is `BACKPRESSURE` and nothing + changes. The same on both hosts, with a parity row. +3. **Order.** D6a (core, lane K) lands in batch 12a; D6b (browser arm bits, lane W) in batch 12b. + +## Decision record: #1584 D6a (batch 8, lane K, batch 12) + +What shipped, on `codex/controls-batch-12k`: + +- **The bindings type** (`crates/controls/src/observation.rs`): `ObservationBindings` (a sorted table of + `TapKey { strip, section: TapSection::{Console, Inserts}, tap_id }` to the window as asked for), + apart from `MonitorOverlay`. `after(next_model)` drops the bindings whose target the model lacks and + returns their keys (`ObservationChange { bindings, ended }`); a target is the strip, the console slot + or insert, and a resident tap with that ID (`effect_rows::resident_observation_tap`). The window is + stored as asked (`0` = the plan's meter window, which each plan resolves for itself); refs and the + conflict codes (D3) belong to the slice that writes the table from the core's `Observe` edit. +- **Owner and carry**: `SessionState.observation` beside `monitor`; `SuccessorBase.observation: + &ObservationBindings` (field, no new preparer argument; `host_core::NO_OBSERVATION` for a host + without subscriptions). The rebuild arm computes `after()` from the prospective model, passes the + result as the base's field and writes it with the commit, after the last fallible step. +- **Prepared arm**: `ObservationLane::arm_prepared(tap, window)` (`start_pending`); `accumulate` takes + the first accumulated block's `first_sample` as the window start and clears the flag; `arm` and + `disarm_all` clear it; `swap_state` moves it with the tap, so a carried tap's state replaces it. No + record is queued. `attach_effect_observation` takes `&[ObservationArm]` (strip, console flag, ID, tap + ID, window) and arms by tap ID; host-core builds the arms from the base's bindings. +- **`OBSERVATION_ENDED` (`8011`)**: reliable event, `observation_ended` schema (sequence 1, strip 2, + section 3 = rack code inserts `2` or console `5`, effect/slot ID 4, tap ID 5; revision in the event + header), advertised with `8001` (`ProviderFeatures::observation_events`, in `ALL`). The key is text, so + the queue slot holds a cell index and the controller keeps `slots + 1` prepared cells (an event popped + for egress keeps its cell until delivered). The host attaches the ends to the prepared token with + `ProtocolController::attach_observation_ended`, which reserves the room with the token's own + `SESSION_COMMITTED` and refuses with `NoRoom`; the control plane returns `BACKPRESSURE` with the token + dropped, before anything is prepared or published. The commit emits the ends after `SESSION_COMMITTED` + at the committing revision, in key order, sequences following the commit's. One control plane serves + both hosts, so the event is the same on both; the wire parity row is `event.observation_ended` in the + conformance corpus (and the capabilities row in its capabilities frame). +- **`push_effect_observe`** records the pushed state per observation slot (`observation_pending`, sized + when a plan is published) and the next `service()` folds it into the table. Departure from "writes the + bindings after a successful push": the browser admits a subscription inside `process()` + (`a_bus_observation_admits_and_renders_without_allocating`, #1213 gate 7), and a binding owns its key's + text, so the write cannot allocate there. The push is still the acknowledgement; the fold runs before + any rebuild reads the table (`command()` services first) and before the successor's slots replace the + pending ones. `observation_bindings()` takes `&mut self` for the same reason. +- **Resource row**: `capi_resources` charges the cell store (`Option` x slots + 1). + +Gates (D6a's seven) and tests: + +1. Restarted owner, subscribe applied: `observation_carry::a_restarted_effects_subscription_survives_the_rebuild_at_eight_lanes` + (pending=false), and E13 rewritten as `a_plan_replacement_keeps_the_subscriptions_the_host_armed` + (`effect_observation.rs`; the replacement changes the slot's link mode so nothing carries). The + reference is a record applied at the adoption block in the successor's own queue: same state and input, + windows equal field for field, first window at `SWAP_SAMPLE`. +2. Pending subscribe: the same test with pending=true (the record waits in the predecessor's queue). +3. Pending disarm: `a_pending_disarm_leaves_the_restarted_tap_unarmed_at_eight_lanes`. +4. Removed owner: `a_removed_effect_ends_its_binding_with_an_event` (insert removed, strip removed, ID + reused for `miso.delay`: one `OBSERVATION_ENDED`, committing revision, stable key, sequence after the + commit's, successor has no tap); `controls/tests/observation.rs` (`after()` incl. console slot and an + unknown tap); host-core `a_stale_binding_arms_nothing`. +5. No room: `no_room_for_the_end_refuses_the_removal_and_changes_nothing` (model, revision, table, plan + epoch, lane unchanged; the same request commits after a drain); protocol + `the_ends_need_room_with_the_commit_and_a_dropped_token_releases_it` and + `attached_observation_ends_follow_their_commit_in_order`. +6. Carried owner: `a_carried_tap_is_not_restarted_by_the_prepared_arm` (NodeToNode, BankToBank, + NodeToBank; streams equal the empty-table run, sequences 1, 2, 3); the #1327b tests are green. +7. Realtime: the swap-block allocator and render-audit counters are `((0,0,0),(0,0))` in gates 1, 2 and + 6; `realtime-policy` passes (141 regions); the browser admission allocation gate stays at zero. + +Re-pins: `COMPLETE_SCHEMA_HASH` `0x4b21_d9e6_f520_afd9` to `0x33e4_b1ce_bcbf_d902` (reason: #1584 D6a adds +the event `8011`, so the corpus gains `event.observation_ended`, 46 frames, and the capabilities frame +lists `8011`), with the history line, the parity script's self-test strings, the manifest and the +conformance record; two C ABI capabilities vectors in `capi/src/runtime/tests.rs` (the capabilities +response lists `8011`; same length). No rendered bit moved: the audit's `pcm_digest` is +`bf7dcedd1d92621f`. + +Departures and limits: + +- The reliable lane of both hosts has two slots (`protocol_queue_config`), so a transaction can end at + most one binding (the commit takes the other). A transaction that ends more can never fit and is + `BACKPRESSURE` forever; the app disarms first. Successor: size the lane for the largest ended set, or + carry the ends in one event. +- The C ABI keeps `observation_taps` 0, so its table is empty; the Observe edit (D1-D5, D7, D8) writes the + same table through the core. +- The capi race test and `tools/audit` do not cover the new event (not in D6a's paths). + +## Fold (root ruling A, batch 12) + +Root ruling A replaces D6a's one event per ended binding with one `OBSERVATION_ENDED` per commit. Commit +"Fold #1584 D6a: one OBSERVATION_ENDED per commit carries every ended key (root ruling A)". + +- **Wire**: `OBSERVATION_ENDED` (`8011`) is the sequence (field 1) and one repeated `key` message (field 2, + mandatory, at least one); a key holds strip ID 1, section 2 (inserts `2`, console `5`), effect or slot ID 3 and + tap ID 4. Keys are strictly ascending (strip, console before inserts, effect ID, tap ID), so the encoder and the + decoder both refuse an empty list, an out-of-order list and a repeated key. Revision stays in the event header. + Typed `ObservationEnded { event_sequence, keys: Vec }`; schema modules `observation_tap_key` + and `observation_ended`; codecs, typed frame, registry and semantics docs follow. +- **Room**: `ProtocolController::attach_observation_ended` reserves ONE reliable slot beside the token's + `SESSION_COMMITTED` whatever the number of keys, and returns the event frame's length. `BACKPRESSURE` now means + only that the lane holds unread events. The commit moves the prepared event into its cell: it allocates + nothing (the keys are built and sized when the transaction prepares, off the commit). +- **Size at preparation**: the event is encoded for size by `attach` against the endpoint's codec, whose frame limit + is the control frame. A frame over that limit (or an unencodable list) is `AttachObservationEndedError::Unencodable`, + and the control plane refuses the transaction as `CompileRejected(resource limit)` and not as backpressure: a retry + could never succeed. `capi_resources` charges the heap a cell can own, one key per observation slot of the plan + (`observation_ended_heap_bytes`, from the producers' `observe_capacity`, their strip and effect ID text and the key + struct), for every cell (slots + 1) and for the token. +- **Gates**: `a_strip_removed_with_many_subscribed_inserts_gives_one_event_with_every_key` (control plane: N = 1, 2, + 7 and 40 subscribed inserts of a removed strip commit, one event, N keys in key order; the lane has two slots); + `no_room_for_the_end_refuses_the_removal_and_changes_nothing` (gate 5, now stated as unread events only); + protocol `attached_observation_ends_follow_their_commit_in_one_event`, + `the_end_needs_room_with_the_commit_and_a_dropped_token_releases_it` and + `observation_ended_round_trips_every_key_and_refuses_anything_else`; the corpus row `event.observation_ended` now + carries two keys (console, then insert) and is the cross-host wire parity row (one control plane serves both + hosts; the native and Wasm corpus runs pin the same bytes). +- **Re-pin**: `COMPLETE_SCHEMA_HASH` `0x33e4_b1ce_bcbf_d902` to `0xcc9a_7bcc_502c_4936` (history line "#1584 D6a + fold: one OBSERVATION_ENDED per commit"), the parity script's self-test strings, the fuzz manifest and the + conformance record. The capabilities frame is unchanged, so the C ABI vectors stay. No rendered bit moved + (audit `pcm_digest` `bf7dcedd1d92621f`, checked below). +- **Limit that remains**: one event must fit one control frame. A commit that ends so many keys that their event + exceeds `maximum_control_frame_bytes` (hundreds of keys at the browser's 16 KiB) is refused as a resource limit; + the app disarms some first. Successor candidate: cap the table at admission by the keys one frame holds. +- **Charge made exact (capi)**: at D6a (97233c465) `cargo test -p capi` failed `capi_retained_bytes_charge_every_byte_the_compile_retains` + (parametric-eq-nine-track: observed 165710, charged 165670) and `tiny_control_frame_still_accounts_three_provider_counters_exactly`, + both 40 bytes over the charge: the session's `observation_offsets` table (`(producers + 1) x u32`, here 10 x 4) and its + `observation_pending` slots were allocated and never charged. `capi_resources` now charges both + (`ObservationResources { producers, slots, ended_heap_bytes }`), beside the fold's ended-event heap; the whole + `cargo test -p capi` passes (132 passed). Audit capi `pcm_digest` `bf7dcedd1d92621f`, allocations 0. + +## Fold (batch 12a review) + +- **Ended-event charge is a frame bound.** The `OBSERVATION_ENDED` cells were charged as the + active plan's producers' key heap times the cell count. After a removal the successor (fewer + producers) was charged while its cells could still hold the predecessor's keys. The charge is + now `ended_event_ceiling(maximum_control_frame_bytes)`: an event must fit one control frame, so + its text is at most the frame and its key count at most the frame over the smallest encoded key + (measured by encoding one key and two, so it follows the wire), times `size_of::()`. + The token row uses the same ceiling. `capi_resources` also charges the `ObservationBindings` + table (capacity at most twice its length, each binding with the text of two IDs at 127 bytes) + and the transient copy `after()` makes (each binding is kept or ended, so at most `slots` + together, plus the ended protocol keys), per plan, so both the active plan and a candidate + carry them. `ObservationResources` lost `ended_heap_bytes`. +- **Measured (capi, x86-64, control frame 4,096):** `capi_retained_bytes` of the reference + session 165,758 -> 186,614; the budget ceiling is 186,614 plus 10 %, rounded to 64: 205,312. + `capi_retained_bytes_charge_every_byte_the_compile_retains` and the tiny-frame test stay exact: + they add `control_plane::observation_reserve_bytes` (test-support) to the observed side beside + the send producers' bound, since nothing holds those bytes until a tap is armed. The two peak + tests add the token's ceiling to the prepared-protocol term. +- **No allocation after the commit.** The pending-subscription slots are built before the + protocol commit (`observation_slots` reserves with `try_reserve_exact` from the candidate's + producers; a failure is a `CompileRejected(Allocation)` refusal with nothing changed) and the + commit only assigns them. `ObservationBindings::after` returns `Result` and reserves both + lists exactly before it copies, as does the ended-key list; the text clones of a key remain + ordinary `String` allocations (before the commit, in a refusable step, but not fallible). +- **Over-frame refusal gate.** `an_end_event_over_one_control_frame_refuses_the_removal_and_changes_nothing` + (control-plane, frame 1,024 bytes, 60 subscribed inserts on the removed strip): the removal is + `CompileRejected(Resource(Limit))` and the revision, canonical session, bindings, plan epoch and + event lane are as they were; a strip with two subscribed inserts commits under the same limit. + Red when the refusal is turned into backpressure (checked, reverted). +- **Dead API.** `ObservationLane::is_prepared_arm_pending` had no caller (its doc said gates use + it) and is deleted. +- **Internal invariant spelled accurately.** A reliable `OBSERVATION_ENDED` slot whose cell is + empty is now `EventEgressError::ObservationEndedCellMissing` (was `DiagnosticStorageFull`); + behaviour (an internal error) is unchanged. diff --git a/.github/ISSUE_SPECS/1618-settle-a-one-shot-spectrum-capture-across-a-plan-swap.md b/.github/ISSUE_SPECS/1618-settle-a-one-shot-spectrum-capture-across-a-plan-swap.md index 9802189f8..7efc82fa6 100644 --- a/.github/ISSUE_SPECS/1618-settle-a-one-shot-spectrum-capture-across-a-plan-swap.md +++ b/.github/ISSUE_SPECS/1618-settle-a-one-shot-spectrum-capture-across-a-plan-swap.md @@ -62,3 +62,49 @@ A carried spectrum observer keeps its one-shot state across a swap (#1395 gate 3 rescoped to the captures that do not carry (no successor observer at the same tap, or a refused hand-over): such a capture settles to `Invalid` with its generation (option A), and a new arm completes on the successor. It lands after #1395. + +## Decision record: #1618, the settle (batch 8) + +**Shipped (option A, as ruled).** `SpectrumService::adopt_successor(successor, pairing, carried, +analysis)` settles what the caller had on a capture that did not carry; a carried capture is +untouched (#1395a gate 3 stays green). + +- *One-shot.* `SpectrumCapture::settle_unpaired(&mut self, predecessor)` (`spectrum.rs`): when the + predecessor's word is in any phase but `IDLE` (armed, capturing, held with or without a queued + result, complete, invalid) and its mode is one-shot, the successor's fresh capture takes + `(predecessor generation, INVALID)` by one atomic exchange. The reader's next `read` answers + `CaptureInvalid` once (the existing `INVALID` arm of `try_read_record`, which also drains a queued + stale record and parks the word `IDLE`), and a new arm is generation `g + 1`, so no record of the + old arm can pass for it. The arm stays the caller's until that read (`caller_armed` is kept). + The successor's observer moves the word only from `ARMED` or `CAPTURING`, so the exchange is safe + while that plan renders; no render, loom-model or `latest_cell` edit (the #1609 loom model and + race test are unchanged and green). +- *Stream.* A new private `end_stream_state` (shared with `stop_stream`) runs when the stream did + not carry: `stream_active` false, cadence, smoothing and window gone, metadata `Stopped`, and the + caller's `analysis` ends with it. This is #1395b's note: after a lost selection `select_stream` + now answers `StreamInactive` instead of arming a one-shot, and the host selects and starts again. +- *Collection.* When the selected entry did not pair, the successor has no selection (#1395b), so + the caller's arm is released (`caller_armed` false) and a read answers `NotArmed`; the selection + epoch tells the caller to select again. A single capture and a collection never pair, so a kind + change also drops the arm and the stream. +- *Departures.* (1) `adopt_successor` takes `analysis: Option<&mut SpectrumAnalysis>` (the owner of + the stream's history ends with the stream); its only caller is the test harness, updated. (2) A + lost collection selection reads `NotArmed`, not `CaptureInvalid`: no capture exists to hold a + word. (3) The gate sentence "a later `select_stream` restarts the cadence" is met as + `select` + `start_stream` restarting it: with the stream ended, `select_stream` is a typed + `StreamInactive` (it can only restart a running stream). + +**Gates -> tests.** `crates/host-core/tests/successor_swap.rs`, `mod spectrum_carry`: gate 1 +`a_one_shot_that_does_not_carry_settles_invalid_and_a_new_arm_completes` (changed channel mask: one +`CaptureInvalid` at the promotion, none before or after; the new arm completes at a later block, +window equal to a fresh arm on the same plan bit for bit) and +`a_one_shot_after_a_refused_hand_over_settles_invalid`; no false settle +`an_idle_capture_that_does_not_carry_settles_nothing`; stream +`a_lost_selection_ends_the_stream_and_a_new_start_restarts_the_cadence`, +`a_refused_hand_over_installs_the_fresh_capture` (now also `!stream_active`), +`a_one_shot_on_a_lost_selection_is_released`. Gate 2 `spectrum::tests:: +a_window_held_behind_a_stale_record_settles_invalid_when_the_plan_swaps` (held window behind a +stale record, through the service: `CaptureInvalid`, then `NotArmed`, then a new arm completes with +the new observer's window). Gate 3: `spectrum_handoff_loom`, the #1609 race test and gate 4 of +#1395 (`..._allocate_and_free_nothing`, zero calls in `adopt_successor`) stay green. With the +settle disabled the gate 1 tests and the gate 2 test go red. diff --git a/.github/ISSUE_SPECS/1674-pair-carried-meter-and-observation-readers-on-the-c-abi.md b/.github/ISSUE_SPECS/1674-pair-carried-meter-and-observation-readers-on-the-c-abi.md new file mode 100644 index 000000000..a3d53d1aa --- /dev/null +++ b/.github/ISSUE_SPECS/1674-pair-carried-meter-and-observation-readers-on-the-c-abi.md @@ -0,0 +1,82 @@ +# Pair carried meter and observation readers on the C ABI + +Successor found by the batch-12 spec check, decision 17 +(`docs/rulings/one-control-abstraction-2026-10-13.md`). Code anchors verified on `ea336c039`. +Issue number: assigned at filing (written here as #1674). Feature sprint mode (owner, 2026-10-13): +one implementer, normal tests, a batch review. + +## Product outcome + +A C ABI app reads a strip's meter and an effect's observation tap across a structural edit with no +restart. When the swap block carries a meter or a tap, the app's reader follows the state that +continues on the successor. The reader does not stay on the retiring plan. + +## Context + +- The carry moves the observer state, producer included, into the successor in the swap block + (#1327a meters, #1327b taps). Then it writes `ObserverCarry` records + (`crates/engine/src/realtime/plan.rs:548-560`). A host whose readers live in the plan's + attachment pairs them in the same block through `HostAttachment::adopt_predecessor`, which + `PreparedRenderPlan::carry_from` calls (`plan.rs:1237`). The browser does this + (`RenderCompanions`, `hosts/host-web/src/lib.rs:5190`, #1381b) with + `host_core::pair_carried_meters` and `pair_carried_observations` + (`crates/host-core/src/prepare.rs:3293`, `:3263`), both allocation-free and inside realtime-policy + regions. +- The C ABI has no reader of either kind today. `CapiPreparer` asks for no meters and no observation + capacity (`crates/capi/src/runtime/mod.rs:97`), and `prepare_runtime` drops the handles it does + not keep (`crates/control-plane/src/compile.rs:713-719`). No `HostAttachment` exists outside + host-web and the tests. +- #1581 D6 keeps the meter consumers in `PreparedRuntime` (read by #1587), and #1584 D7 keeps the + observation handles there (read by #1586). Those fields are on the control side, and the swap + block cannot reach them. After a carry, they would read the retiring plan's producers, which + then go silent. + +## Decisions frozen for this slice + +- **D1. The readers live in the plan's attachment.** The C ABI's plan carries one attachment that + holds the meter consumers and the observation handles, in host-core's order. The giving issue's + `PreparedRuntime` fields move into this attachment. The C ABI reads them through the plan owner's + `active_attachment_mut` (`crates/control-plane/src/plan.rs:244`), on the thread that reads them + today. +- **D2. The hook.** The attachment implements `HostAttachment::adopt_predecessor`. It downcasts the + predecessor through `as_any_mut` and calls `pair_carried_meters` and `pair_carried_observations` + with the record. Another attachment type, or an empty record, pairs nothing. +- **D3. Realtime.** The hook makes no allocation, takes no lock and makes no syscall. Its + bookkeeping is sized at preparation and charged to the plan's retained rows. +- **D4. No engine edit.** `adopt_predecessor` and the record exist (#1327a). + +## Objective gates + +1. **A meter window continues.** On the C ABI, meters on, add a muted track whose ID sorts first + (a rebuild). The strip that was meter 0 and is now meter 1 continues its window sequence through + the reader that the app already holds, with no counted loss. Its first window after the swap + equals the reference window field by field. +2. **An acknowledged subscription continues.** With a compressor tap subscribed and its window + open, the same rebuild keeps the window sequence on the app's reader, and the next window + equals the window that the predecessor would have published. +3. **Realtime.** The swap block, the hook included, makes zero allocations and frees. + `target/release/audit capi` reports 0 allocations, locks and syscalls. +4. Commands: `cargo test --locked -p capi`, `cargo test --locked -p control-plane --features + test-support`, `cargo build --locked --release -p audit -p capi && target/release/audit capi`, + `cargo run --locked --release -q -p realtime-policy`, clippy, `cargo fmt --all -- --check`. + +## Test value + +- Gates 1 and 2 turn red if a carried meter or tap leaves the C ABI's reader on the retiring plan. +- Gate 3 turns red if the pairing allocates or frees on the render thread. + +## Authorized paths + +- `crates/capi/src/runtime/**` (the attachment and its tests), `crates/control-plane/src/{compile.rs,control.rs,plan.rs}` + (the attachment at preparation and its read path), `crates/capi/tests/resource_lifecycle.rs`, + and this spec's decision record. + +## Dependencies + +- Whichever issue first gives the C ABI meter or observation readers: #1581 (D6, the meter + consumers) or #1584 (D7, the observation handles). The second of them adds its readers to the + same attachment, and its gate (1 or 2) moves to it. Lane K. + +## Size + +Half a day. diff --git a/crates/capi/src/runtime/tests.rs b/crates/capi/src/runtime/tests.rs index 17976427b..f6fbff00c 100644 --- a/crates/capi/src/runtime/tests.rs +++ b/crates/capi/src/runtime/tests.rs @@ -90,7 +90,7 @@ const ALL_COMMAND_RESPONSE_VECTORS: [&str; 10] = [ // #1627 re-pin: `AUTOMATION_ENQUEUE` is retired. The capabilities record drops fields 9, 12, // 19 and 20 (23 fields, was 27; the length 0x188, was 0x1c8) and the flags are 0x7fdf as // before; the earlier #1315 re-pin already listed no command `0x0006` and cleared bit 5. - "4d49534f43544c000100000030000200010000008801000001000000000000002a00000000000000170000000000000001000201020000000100000000000000020002010200000000000000000000000300020102000000010000000000000004000201020000000000000000000000050004010800000000100000000000000600030104000000000800000000000007000401080000000010000000000000080001010100000004000000000000000a0004010800000001000000000000000b0004010800000000100000000000000d0004010800000001000000000000000e0004010800000002000000000000000f0004010800000001000000000000001000040108000000100000000000000011000401080000000020000000000000120004010800000000100000000000001500020102000000000100000000000016000201020000000001000000000000170002010200000000010000000000001800030104000000000800000000000019000c01160000000100020003000700080009000a000b000c000d000e0000001a000c010a000000018010802080218030800000000000001b00040108000000df7f000000000000", + "4d49534f43544c000100000030000200010000008801000001000000000000002a00000000000000170000000000000001000201020000000100000000000000020002010200000000000000000000000300020102000000010000000000000004000201020000000000000000000000050004010800000000100000000000000600030104000000000800000000000007000401080000000010000000000000080001010100000004000000000000000a0004010800000001000000000000000b0004010800000000100000000000000d0004010800000001000000000000000e0004010800000002000000000000000f0004010800000001000000000000001000040108000000100000000000000011000401080000000020000000000000120004010800000000100000000000001500020102000000000100000000000016000201020000000001000000000000170002010200000000010000000000001800030104000000000800000000000019000c01160000000100020003000700080009000a000b000c000d000e0000001a000c010c000000018010801180208021803080000000001b00040108000000df7f000000000000", // #338 re-pin: canonical JSON snapshot bytes were 16,712 (0x4148), beginning with `{`. // #1093 re-pin: 13,729 (0x35a1) after the fixture moved to decision 12's console shape. // #1216 re-pin: 13,918 (0x365e), route field `mute` added, value false (21 bytes, 9 routes). @@ -2147,7 +2147,7 @@ fn all_five_event_families_cross_c_dequeue_with_exact_oracle_bytes() { "4d49534f43544c000100000030000200030000002000000004000000000000002b0000000000000002000000000000000100030104000000010000000000000002000101010000000200000000000000", "4d49534f43544c000100000030000200090000005800000005000000000000002b00000000000000060000000000000001000d010400000001000000000000000200030104000000010000000000000003000d0100000000040003010400000000000000000000000500080101000000000000000000000006000101010000000100000000000000", "4d49534f43544c000100000030000200090000005800000006000000000000002b00000000000000060000000000000001000d010800000001000000020000000200030104000000010000000000000003000d0100000000040003010400000000000000000000000500080101000000000000000000000006000101010000000100000000000000", - "4d49534f43544c000100000030000200010000008801000007000000000000002b00000000000000170000000000000001000201020000000100000000000000020002010200000000000000000000000300020102000000010000000000000004000201020000000000000000000000050004010800000000100000000000000600030104000000000800000000000007000401080000000010000000000000080001010100000004000000000000000a0004010800000001000000000000000b0004010800000000100000000000000d0004010800000001000000000000000e0004010800000002000000000000000f0004010800000001000000000000001000040108000000100000000000000011000401080000000020000000000000120004010800000000100000000000001500020102000000000100000000000016000201020000000001000000000000170002010200000000010000000000001800030104000000000800000000000019000c01160000000100020003000700080009000a000b000c000d000e0000001a000c010a000000018010802080218030800000000000001b00040108000000df7f000000000000", + "4d49534f43544c000100000030000200010000008801000007000000000000002b00000000000000170000000000000001000201020000000100000000000000020002010200000000000000000000000300020102000000010000000000000004000201020000000000000000000000050004010800000000100000000000000600030104000000000800000000000007000401080000000010000000000000080001010100000004000000000000000a0004010800000001000000000000000b0004010800000000100000000000000d0004010800000001000000000000000e0004010800000002000000000000000f0004010800000001000000000000001000040108000000100000000000000011000401080000000020000000000000120004010800000000100000000000001500020102000000000100000000000016000201020000000001000000000000170002010200000000010000000000001800030104000000000800000000000019000c01160000000100020003000700080009000a000b000c000d000e0000001a000c010c000000018010801180208021803080000000001b00040108000000df7f000000000000", "4d49534f43544c000100000030000200090000005800000008000000000000002b00000000000000060000000000000001000d01000000000200030104000000000000000000000003000d01040000000600000000000000040003010400000001000000000000000500080101000000000000000000000006000101010000000100000000000000", ]; const EVENTS: [&str; 6] = [ diff --git a/crates/capi/tests/resource_lifecycle.rs b/crates/capi/tests/resource_lifecycle.rs index ade9200fc..4cc5cc44e 100644 --- a/crates/capi/tests/resource_lifecycle.rs +++ b/crates/capi/tests/resource_lifecycle.rs @@ -700,9 +700,15 @@ impl HostHalf { + effects + self.prepared.report.route_control_resources.total_bytes + self.prepared.report.inventory_retained_bytes; + // The token also holds the commit's one `OBSERVATION_ENDED` event (#1584 D6a), which the + // control plane bounds by the frame it must fit. + let (_, ended_event) = + control_plane::observation_reserve_bytes(compile_limits.maximum_control_frame_bytes, 0) + .expect("the observation reserve"); let prepared_protocol = compile_limits.maximum_control_frame_bytes + size_of::() as u64 + compile_limits.maximum_control_frame_bytes + + ended_event + self.catalog_charge(); (epoch, prepared_protocol) } @@ -754,6 +760,11 @@ struct CompileObservation { owners: HostOwners, /// `strip_control_table_bytes` at the compiled session's strip count and strip ID bytes. strip_table_charge: u64, + /// The observation rows that are bounds, not allocations (#1584 D6a): the cells' event heap, + /// the binding table and its transient copies, from `control_plane::observation_reserve_bytes` + /// at the compile's control frame and observation slots. Nothing holds them until a host arms + /// a tap, so they are on the observed side as the send producers' upper bound is. + observation_reserve: u64, } fn observe_compile(document: &str, compile_limits: &CompileLimits) -> CompileObservation { @@ -789,6 +800,14 @@ fn observe_compile(document: &str, compile_limits: &CompileLimits) -> CompileObs }); let strip_table_charge = host_core::strip_control_table_bytes(strip_count, strip_id_bytes) .expect("the strip table's charge"); + let (observation_reserve, _) = control_plane::observation_reserve_bytes( + compile_limits.maximum_control_frame_bytes, + effects + .iter() + .map(|producer| producer.observe_capacity()) + .sum(), + ) + .expect("the observation reserve"); let host_core::PreparedHost { plan, sources, @@ -826,6 +845,7 @@ fn observe_compile(document: &str, compile_limits: &CompileLimits) -> CompileObs host_live, owners, strip_table_charge, + observation_reserve, } } @@ -869,12 +889,14 @@ impl CompileObservation { + self.owners.sources + self.owners.catalog + self.owners.inventory + + self.observation_reserve + decode_field_rounding, self.report.capi_retained_bytes, "{label}: capi's own allocations + the effect producers + the strip producers + the \ source producers + the parameter catalog + the plan state inventory + the \ decode-field rounding, all observed (left; the send producers' row is the graph \ - function's total), against `capi_retained_bytes` (right)" + function's total, and the observation reserve is the control plane's bound), against \ + `capi_retained_bytes` (right)" ); } @@ -1187,6 +1209,12 @@ impl Budget { /// 113,601 -> 107,553: 6,048 each, x86-64). `capi_retained_bytes` charges capi's own allocations /// and stays 165,166. The graph ceilings stand until the next `aarch64-debug` print tightens them /// (AArch64 runs only in CI); the rows only shrank, so they still bound them. +/// +/// The batch 12a review fold charges the observation reserve (#1584 D6a) as a bound that does not +/// depend on the plan's producers: the cells' `OBSERVATION_ENDED` heap at what one control frame +/// can hold (4,096 bytes here), the binding table and its transient copies. `capi_retained_bytes` +/// measured 165,758 before and 186,614 after (x86-64; the rows do not depend on the lane width), +/// so its ceiling is 186,614 plus 10 %, rounded up to 64: 205,275.4 -> 205,312. const REFERENCE_BUDGETS: [Budget; 19] = [ Budget { row: "graph_session_plus_plan_bytes", @@ -1293,8 +1321,8 @@ const REFERENCE_BUDGETS: [Budget; 19] = [ Budget { row: "capi_retained_bytes", value: |report| report.capi_retained_bytes, - eight_lanes: 181_696, - four_lanes: 181_696, + eight_lanes: 205_312, + four_lanes: 205_312, }, Budget { row: "largest_named_allocation_bytes", @@ -1775,6 +1803,7 @@ fn double_live_oracle_drives_exact_and_one_below_c_caps() { revision: 1, edit_ramps: &session::EditRamps::new(), monitor: host_core::MonitorBase::NONE, + observation: &host_core::NO_OBSERVATION, }, &host_core::LiveValuesSuperseded::default(), ) diff --git a/crates/conformance/src/main.rs b/crates/conformance/src/main.rs index 9f9b15d4a..31ce99eeb 100644 --- a/crates/conformance/src/main.rs +++ b/crates/conformance/src/main.rs @@ -28,7 +28,7 @@ fn main() -> ExitCode { fn complete_schema_hash() -> u64 { let codec = ProtocolCodec::default(); let corpus = complete_schema_corpus(); - assert_eq!(corpus.len(), 45); + assert_eq!(corpus.len(), 46); let mut hash = 0xcbf2_9ce4_8422_2325_u64; for frame in &corpus { hash = hash_bytes(hash, frame.name.as_bytes()); diff --git a/crates/conformance/src/protocol_corpus.rs b/crates/conformance/src/protocol_corpus.rs index 25821a20e..2d6b9f457 100644 --- a/crates/conformance/src/protocol_corpus.rs +++ b/crates/conformance/src/protocol_corpus.rs @@ -860,7 +860,14 @@ pub enum ConformanceDecoder { /// Issue #1628 (draft 21b) repinned it from `cfa58ab7b7bbb7c4`: the event `AUTOMATION_CANCELED` /// (`0x8002`) is retired and never reallocated, so the corpus loses `event.automation_canceled` /// and the capabilities response lists no event `0x8002`. The frame count is 45. -pub const COMPLETE_SCHEMA_HASH: u64 = 0x4b21_d9e6_f520_afd9; +/// Issue #1584 (D6a) repinned it from `4b21d9e6f520afd9`: the reliable event `OBSERVATION_ENDED` +/// (`0x8011`) is added, so the corpus gains `event.observation_ended` (the frame count is 46 and the +/// events are six) and the capabilities response lists event `0x8011`. +/// #1584 D6a fold: one OBSERVATION_ENDED per commit. Root ruling A repinned it from +/// `33e4b1cebcbfd902`: the event carries a repeated `key` message (the wire shape is the sequence +/// and one nested key per ended tap), so `event.observation_ended` carries two keys. The frame +/// count stays 46. +pub const COMPLETE_SCHEMA_HASH: u64 = 0xcc9a_7bcc_502c_4936; /// Build every command, successful response, registered non-OK status, event, and all-opcode /// session transaction using only public typed encoder entry points. @@ -1027,7 +1034,7 @@ pub fn complete_schema_corpus() -> Vec { maximum_telemetry_handles: 256, maximum_transaction_edits: 64, supported_commands: &[1, 2, 3, 7, 8, 9, 10, 11, 12, 13, 14], - supported_events: &[0x8001, 0x8010, 0x8020, 0x8021, 0x8030], + supported_events: &[0x8001, 0x8010, 0x8011, 0x8020, 0x8021, 0x8030], flags: CapabilityFlags::B4_BASE, }; macro_rules! success { @@ -1229,6 +1236,27 @@ pub fn complete_schema_corpus() -> Vec { origin_request_id: None }) ); + event!( + "event.observation_ended", + EventPayload::ObservationEnded(&ObservationEnded { + event_sequence: 2, + // Two keys of one commit: a console slot, then an insert (console sorts first). + keys: vec![ + ObservationTapKey { + strip_id: "vocal".to_owned(), + section: ObservationSection::Console, + effect_id: "glue".to_owned(), + tap_id: 3, + }, + ObservationTapKey { + strip_id: "vocal".to_owned(), + section: ObservationSection::Inserts, + effect_id: "comp".to_owned(), + tap_id: 1, + }, + ], + }) + ); event!( "event.meter_batch", EventPayload::MeterBatch(MeterBatch { diff --git a/crates/conformance/tests/conformance_corpus.rs b/crates/conformance/tests/conformance_corpus.rs index 8751b9504..a47b51b96 100644 --- a/crates/conformance/tests/conformance_corpus.rs +++ b/crates/conformance/tests/conformance_corpus.rs @@ -7,7 +7,7 @@ use protocol::{DecodeScratch, ProtocolCodec}; fn frozen_corpus_bytes_and_typed_decoders_are_unchanged() { let codec = ProtocolCodec::default(); let corpus = complete_schema_corpus(); - assert_eq!(corpus.len(), 45); + assert_eq!(corpus.len(), 46); let mut hash = 0xcbf2_9ce4_8422_2325_u64; for frame in &corpus { hash = hash_bytes(hash, frame.name.as_bytes()); diff --git a/crates/conformance/tests/mutation_million.rs b/crates/conformance/tests/mutation_million.rs index 0f3c3d314..1d0454dbe 100644 --- a/crates/conformance/tests/mutation_million.rs +++ b/crates/conformance/tests/mutation_million.rs @@ -28,8 +28,8 @@ fn deterministic_mutations_cover_complete_schema_closed_dispatch(mutation_runs: let seeds = complete_schema_corpus(); assert_eq!( seeds.len(), - 45, - "the complete-schema corpus: commands, successes, non-OK statuses and five events" + 46, + "the complete-schema corpus: commands, successes, non-OK statuses and six events" ); assert!(seeds.iter().any(|seed| { seed.name == "command.session_transaction_apply" diff --git a/crates/control-plane/src/compile.rs b/crates/control-plane/src/compile.rs index 2afffe516..e4e7e7075 100644 --- a/crates/control-plane/src/compile.rs +++ b/crates/control-plane/src/compile.rs @@ -194,11 +194,15 @@ pub(crate) fn protocol_queue_config( ) .ok_or_else(|| failure(ResourceFault::Limit))?, reliable_response_slots: one, - reliable_event_slots: NonZeroUsize::new(2).expect("two is nonzero"), + reliable_event_slots: NonZeroUsize::new(RELIABLE_EVENT_SLOTS).expect("two is nonzero"), telemetry_slots: one, }) } +/// The reliable event queue's slots: the transaction's `SESSION_COMMITTED` and its one +/// `OBSERVATION_ENDED` (#1584 D6a). +const RELIABLE_EVENT_SLOTS: usize = 2; + #[allow(clippy::too_many_arguments)] pub(crate) fn capi_resources( limits: ControlLimits, @@ -209,6 +213,7 @@ pub(crate) fn capi_resources( inventory_bytes: u64, effect_controls: host_core::EffectControlResources, route_controls: host_core::RouteControlResources, + observation: ObservationResources, ) -> Result { let queue_config = protocol_queue_config(limits)?; let queue = ProtocolQueues::resource_report_for_config(queue_config) @@ -278,6 +283,12 @@ pub(crate) fn capi_resources( effect_row, route_row, ]; + let ObservationReserve { + ended_event, + cell_heap, + bindings_table, + bindings_transient, + } = observation_reserve(limits.maximum_control_frame_bytes, observation.slots)?; let maximum_configuration_items = usize::try_from(limits.maximum_control_frame_bytes) .map_err(|_| failure(ResourceFault::Platform))? / size_of::(); @@ -290,6 +301,27 @@ pub(crate) fn capi_resources( checked_layout::>(1)?, checked_layout::>>(1)?, checked_layout::(2)?, + // #1584 D6a: the observation-ended cells, one more than the reliable queue's slots (an + // event popped for egress keeps its cell until its frame is delivered), and the heap the + // events in them own. A cell can hold the keys of any earlier plan's subscriptions, so + // the ceiling is not a function of this plan's producers (a successor that removed + // producers still holds their keys): an event must fit one control frame, which bounds + // its key count and its text (`ended_event_ceiling`). The heap is allocated when a + // transaction prepares, never at the commit. + checked_layout::>( + queue_config.reliable_event_slots.get() + 1, + )?, + checked_byte_layout(cell_heap)?, + // #1584 D6a: the binding table the session keeps for the newest plan and the transient + // copies `ObservationBindings::after` makes of it while a rebuild prepares: at most one + // binding per observation slot, so at most `slots` bindings kept or ended, and as many + // protocol keys for the ended ones. A binding owns the text of its two IDs. + checked_byte_layout(bindings_table)?, + checked_byte_layout(bindings_transient)?, + // #1584 D6a: the newest plan's pending-subscription slots and the first slot of each + // producer (one entry past the last). + checked_layout::(observation.slots)?, + checked_layout::(observation.producers + 1)?, checked_layout::(RENDER_DIAGNOSTIC_SLOTS)?, checked_layout::(RENDER_DIAGNOSTIC_CODE.len() * RENDER_DIAGNOSTIC_SLOTS)?, // ProtocolController and SessionControlProvider each retain their own telemetry config. @@ -317,6 +349,8 @@ pub(crate) fn capi_resources( checked_byte_layout(limits.maximum_control_frame_bytes)?, checked_layout::(1)?, checked_byte_layout(limits.maximum_control_frame_bytes)?, + // The token holds the commit's one `OBSERVATION_ENDED` event until the commit moves it. + checked_byte_layout(ended_event)?, ]; let epoch_retained = checked_sum(&epoch_rows)?; let fixed_allocations = checked_sum(&fixed_allocation_rows)? @@ -365,6 +399,7 @@ pub(crate) fn prepared_capi_resources( inventory: &PlanStateInventory, effect_controls: host_core::EffectControlResources, route_controls: host_core::RouteControlResources, + observation: ObservationResources, limits: ControlLimits, ) -> Result { let source_id_bytes = @@ -403,9 +438,175 @@ pub(crate) fn prepared_capi_resources( inventory.retained_bytes(), effect_controls, route_controls, + observation, ) } +/// What a plan's observation subscriptions retain (#1584 D6a). +#[derive(Clone, Copy)] +pub(crate) struct ObservationResources { + /// Observation slots of the plan: one per declared tap of every producer that has a queue. + /// The binding table holds at most one binding per slot. + pub(crate) slots: usize, + /// Effect producers of the plan. + pub(crate) producers: usize, +} + +/// The observation rows of `effects`, the newest plan's producers. +pub(crate) fn observation_resources( + effects: &[host_core::EffectControlProducer], +) -> Result { + let mut slots = 0_usize; + for producer in effects { + slots = slots + .checked_add(producer.observe_capacity()) + .ok_or_else(|| failure(ResourceFault::Arithmetic))?; + } + Ok(ObservationResources { + producers: effects.len(), + slots, + }) +} + +/// The longest a stable ID is (`[a-z][a-z0-9._-]{0,126}`), which bounds the text of a key. +const STABLE_ID_MAXIMUM_BYTES: usize = 127; + +/// The smallest encoded size of one key of an `OBSERVATION_ENDED` event: two one-character IDs, +/// the section and the tap ID with their field headers, measured as what a second key adds to the +/// frame. It bounds how many keys fit one frame. +fn minimum_ended_key_bytes() -> Result { + let codec = protocol::ProtocolCodec::default(); + let size = |ids: &[&str]| -> Option { + let event = protocol::ObservationEnded { + event_sequence: 0, + keys: ids + .iter() + .map(|id| protocol::ObservationTapKey { + strip_id: (*id).to_owned(), + section: protocol::ObservationSection::Console, + effect_id: (*id).to_owned(), + tap_id: 0, + }) + .collect(), + }; + match codec.encode_event_frame_into( + &protocol::TypedEventFrame { + revision: protocol::SessionRevision(1), + payload: protocol::EventPayload::ObservationEnded(&event), + }, + &mut [], + ) { + Err(protocol::EncodeError::OutputTooSmall { required }) => Some(required), + Err(_) | Ok(_) => None, + } + }; + let (one, two) = (size(&["a"]), size(&["a", "b"])); + one.zip(two) + .and_then(|(one, two)| two.checked_sub(one)) + .filter(|bytes| *bytes > 0) + .and_then(|bytes| u64::try_from(bytes).ok()) + .ok_or_else(|| failure(ResourceFault::Arithmetic)) +} + +/// The observation rows that are bounds, not allocations: nothing holds them until a host arms a +/// tap, a transaction ends one, or an event waits for egress. +struct ObservationReserve { + /// The heap one event can own ([`ended_event_ceiling`]); the token holds one. + ended_event: u64, + /// That heap in every cell, one more cell than the reliable queue has slots. + cell_heap: u64, + /// The binding table's capacity bound. + bindings_table: u64, + /// `ObservationBindings::after`'s transient copies. + bindings_transient: u64, +} + +fn observation_reserve( + frame_bytes: u64, + slots: usize, +) -> Result { + let ended_event = ended_event_ceiling(frame_bytes)?; + let cell_heap = ended_event + .checked_mul( + u64::try_from(RELIABLE_EVENT_SLOTS + 1) + .map_err(|_| failure(ResourceFault::Platform))?, + ) + .ok_or_else(|| failure(ResourceFault::Arithmetic))?; + let (bindings_table, bindings_transient) = observation_binding_rows(slots)?; + Ok(ObservationReserve { + ended_event, + cell_heap, + bindings_table, + bindings_transient, + }) +} + +/// What the active row charges for observation subscriptions beyond the allocations a compile +/// makes, as `(active, token)` bytes: the cells' event heap, the binding table and its transient +/// copies, and the one event the prepared token holds (#1584 D6a). The adapters' resource tests +/// add the first to the allocations they observe, as they add the send producers' upper bound. +/// +/// # Errors +/// +/// The arithmetic overflowed. +#[cfg(feature = "test-support")] +pub fn observation_reserve_bytes( + frame_bytes: u64, + slots: usize, +) -> Result<(u64, u64), CompileFailure> { + let reserve = observation_reserve(frame_bytes, slots)?; + Ok(( + checked_sum(&[ + reserve.cell_heap, + reserve.bindings_table, + reserve.bindings_transient, + ])?, + reserve.ended_event, + )) +} + +/// The heap one `OBSERVATION_ENDED` event can own, bounded by the control frame it must fit: its +/// text is encoded in the frame, so it is at most the frame; its key count is at most the frame +/// over the smallest key, and each key is a `protocol::ObservationTapKey` (#1584 D6a). The bound +/// does not depend on any plan's producers: a cell can hold the keys of an earlier plan. +pub(crate) fn ended_event_ceiling(frame_bytes: u64) -> Result { + let keys = frame_bytes / minimum_ended_key_bytes()?; + u64::try_from(size_of::()) + .ok() + .and_then(|key| key.checked_mul(keys)) + .and_then(|table| table.checked_add(frame_bytes)) + .ok_or_else(|| failure(ResourceFault::Arithmetic)) +} + +/// What the binding table and a rebuild's transient copies of it hold for a plan with `slots` +/// observation slots, as `(table, transient)` bytes (#1584 D6a). The table's `Vec` grows by +/// insertion, so its capacity is at most twice its length (and at least four entries). The +/// transient is what `ObservationBindings::after` builds from the table: each binding goes to the +/// kept table or the ended list, so at most `slots` of them together, and the protocol keys the +/// ended ones are copied into. Every binding and key owns the text of two IDs. +fn observation_binding_rows(slots: usize) -> Result<(u64, u64), CompileFailure> { + let arithmetic = || failure(ResourceFault::Arithmetic); + let text = 2 * STABLE_ID_MAXIMUM_BYTES; + let binding = size_of::() + .checked_add(text) + .ok_or_else(arithmetic)?; + let key = size_of::() + .checked_add(text) + .ok_or_else(arithmetic)?; + let table = if slots == 0 { 0 } else { slots.max(4) } + .checked_mul(2) + .and_then(|capacity| capacity.checked_mul(binding)) + .ok_or_else(arithmetic)?; + let transient = slots + .checked_mul(binding) + .and_then(|bytes| bytes.checked_add(slots.checked_mul(key)?)) + .ok_or_else(arithmetic)?; + Ok(( + u64::try_from(table).map_err(|_| failure(ResourceFault::Platform))?, + u64::try_from(transient).map_err(|_| failure(ResourceFault::Platform))?, + )) +} + pub(crate) fn controller_retained_capacity( limits: ControlLimits, ) -> Result { @@ -734,6 +935,7 @@ pub(crate) fn prepare_runtime( &prepared.inventory, prepared.report.effect_control_resources, prepared.report.route_control_resources, + observation_resources(&effects)?, limits, )?; if capi.active_retained > limits.maximum_control_retained_bytes @@ -952,35 +1154,35 @@ where .try_reserve_exact(decode_field_count) .map_err(|_| failure(ResourceFault::Allocation))?; decode_fields.resize(decode_field_count, 0); + let mut session = SessionState { + controller: ObservedController::new(controller), + providers: ProviderEpoch::current(sources, inventory, strips, effects, capi, revision_gate), + pending_providers, + retired_providers, + publisher, + retirer, + limits, + preparer, + live_values_superseded, + decode_fields: decode_fields.into_boxed_slice(), + response_scratch: boxed_zeroed(limits.maximum_control_frame_bytes)?, + shared: Arc::clone(&shared), + observed_render_sequence: 0, + counter_snapshot_sequence: 0, + render_diagnostics: prepare_render_diagnostic_slots()?, + render_diagnostic_head: 0, + render_diagnostic_len: 0, + protocol_reliable_pending: false, + monitor: controls::MonitorOverlay::new(), + observation: controls::ObservationBindings::new(), + observation_pending: Box::default(), + observation_offsets: Box::default(), + }; + (session.observation_offsets, session.observation_pending) = + crate::control::observation_slots(&session.providers.effects) + .map_err(|_| failure(ResourceFault::Allocation))?; Ok(CompiledChildren { - session: SessionState { - controller: ObservedController::new(controller), - providers: ProviderEpoch::current( - sources, - inventory, - strips, - effects, - capi, - revision_gate, - ), - pending_providers, - retired_providers, - publisher, - retirer, - limits, - preparer, - live_values_superseded, - decode_fields: decode_fields.into_boxed_slice(), - response_scratch: boxed_zeroed(limits.maximum_control_frame_bytes)?, - shared: Arc::clone(&shared), - observed_render_sequence: 0, - counter_snapshot_sequence: 0, - render_diagnostics: prepare_render_diagnostic_slots()?, - render_diagnostic_head: 0, - render_diagnostic_len: 0, - protocol_reliable_pending: false, - monitor: controls::MonitorOverlay::new(), - }, + session, plan: PlanState::new(owner, shared), }) } diff --git a/crates/control-plane/src/control.rs b/crates/control-plane/src/control.rs index 458663d87..910fd1f94 100644 --- a/crates/control-plane/src/control.rs +++ b/crates/control-plane/src/control.rs @@ -218,7 +218,7 @@ pub struct TestOwnerCounters { pub reservation_committed: u64, } -/// One effect producer's row of [`SessionControlSnapshot::effect_cells`]: `(epoch, strip ID, +/// One effect producer's row of `SessionControlSnapshot::effect_cells`: `(epoch, strip ID, /// effect instance ID, writes per cell)`. #[cfg(feature = "test-support")] pub type EffectCellRow = (u64, Box, Box, Vec); @@ -405,6 +405,94 @@ pub struct SessionState { /// catalog shows it (D11), and it is written only after every fallible step of the /// transaction that sets it has passed (D8). pub(crate) monitor: controls::MonitorOverlay, + /// The observation taps the host has armed (#1584 D6a), kept beside the monitor overlay and + /// apart from it: solo and subscriptions are different things. Host-local, never saved and + /// never in a snapshot; empty at boot. Written after a successful push to a tap's queue, and + /// by a rebuild with the commit, which drops each binding whose target the transaction removed + /// and tells the subscribers. Every successor is prepared from it, so a subscription the host + /// acknowledged survives a rebuild that restarts its effect. + pub(crate) observation: controls::ObservationBindings, + /// The subscription state each observation slot of the newest plan was last pushed, waiting + /// to be folded into [`Self::observation`] (#1584 D6a). The browser admits a subscription + /// inside `process()`, where nothing may allocate, and a binding owns its key's text, so the + /// push records the absolute state here, in storage sized when the plan is published, and the + /// next service step folds it into the table. A rebuild folds first: it only starts from + /// [`Self::command`], which services. + pub(crate) observation_pending: Box<[PendingArm]>, + /// For each position of the newest plan's effect producers, the first of its slots in + /// `observation_pending`, and one entry past the last: a producer's slots are + /// `offsets[position]..offsets[position + 1]`, one per declared tap in declaration order. + pub(crate) observation_offsets: Box<[u32]>, +} + +/// A plan's observation slot offsets and its pending-subscription storage. +pub(crate) type ObservationSlots = (Box<[u32]>, Box<[PendingArm]>); + +/// The pending-subscription storage for `producers`, the newest plan's: the first slot of each +/// producer (and one past the last), and one slot per declared tap (#1584 D6a). Built when a plan +/// is prepared, fallibly and before anything commits; the session takes it after the commit and +/// after the fold, so nothing pending is lost to a change of positions. +pub(crate) fn observation_slots( + producers: &[host_core::EffectControlProducer], +) -> Result { + let mut offsets = Vec::new(); + offsets.try_reserve_exact(producers.len() + 1)?; + let mut total = 0_u32; + for producer in producers { + offsets.push(total); + total = + total.saturating_add(u32::try_from(producer.observe_capacity()).unwrap_or(u32::MAX)); + } + offsets.push(total); + let mut pending = Vec::new(); + pending.try_reserve_exact(total as usize)?; + pending.resize(total as usize, PendingArm::None); + Ok((offsets.into_boxed_slice(), pending.into_boxed_slice())) +} + +/// What the host last pushed to one observation slot (#1584 D6a). +#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)] +pub(crate) enum PendingArm { + /// Nothing since the last fold. + #[default] + None, + /// Armed, with this window length in blocks (`0` is the plan's meter window). + Arm(u32), + /// Disarmed. + Disarm, +} + +/// The stable key of the tap `tap_index` of the effect `producer` addresses, or `None` for an index +/// the effect does not declare. +fn tap_key_of( + producer: &host_core::EffectControlProducer, + tap_index: u32, +) -> Option { + let tap = producer.descriptor.observations.get(tap_index as usize)?; + let effect = session::StableId::parse(&producer.effect_id)?; + let section = match producer.address.rack { + host_core::LiveEffectRack::Console => controls::TapSection::Console(effect), + host_core::LiveEffectRack::Inserts => controls::TapSection::Inserts(effect), + }; + Some(controls::TapKey { + strip: session::StableId::parse(&producer.track_id)?, + section, + tap_id: tap.id.0, + }) +} + +/// The wire key of an ended binding. +fn ended_key(key: &controls::TapKey) -> protocol::ObservationTapKey { + protocol::ObservationTapKey { + strip_id: key.strip.as_str().to_owned(), + section: if key.section.is_console() { + protocol::ObservationSection::Console + } else { + protocol::ObservationSection::Inserts + }, + effect_id: key.section.id().as_str().to_owned(), + tap_id: key.tap_id, + } } pub(crate) struct ObservedController { @@ -703,6 +791,7 @@ pub(crate) fn map_event_egress_error(error: EventEgressError) -> EventError { EventEgressError::ReliableQueueFull(_) => EventError::Backpressure, EventEgressError::Disabled | EventEgressError::DiagnosticStorageFull + | EventEgressError::ObservationEndedCellMissing | EventEgressError::Encode(_) => EventError::Internal, } } @@ -1183,6 +1272,7 @@ where /// acknowledges nothing and adds no queue. The only things it discards are plans render has /// already displaced; a committed revision's plan is never among them. pub fn service(&mut self) -> Result<(), CommandError> { + self.fold_observation_pending(); self.synchronize_plan_epochs()?; self.refresh_provider_counters(); self.collect_render_activity(); @@ -1248,7 +1338,7 @@ where // #1257 D1: a value-only delta of track faders, mutes, pans and live effect // parameters (#1264) rides the newest plan's live lanes; every other delta takes // the rebuild below, unchanged. - let prepared = match self.commit_live(prepared) { + let mut prepared = match self.commit_live(prepared) { LiveCommit::Done(result) => return result, LiveCommit::Rebuild(prepared) => prepared, // `commit_live` materializes an in-place token before it returns. @@ -1320,6 +1410,52 @@ where ); // Issue #1273 D1: the candidate succeeds the newest plan, so every source the // transaction leaves unchanged is prepared to carry that plan's ring. + // #1584 D6a: the subscriptions the successor is prepared with, and those the + // transaction ends. The room for their one `OBSERVATION_ENDED` event (every ended + // key in it) is reserved with the token's `SESSION_COMMITTED`, before anything is + // prepared, published or queued: a lane holding unread events is retryable + // backpressure and nothing has changed (the acked-batch rule: a subscription is + // never dropped without its event). An event over one control frame can never + // be sent, so it is a resource refusal, not backpressure. + let observation_change = self + .observation + .after( + prepared + .get() + .prospective_session() + .compiled() + .ok_or(CommandError::Internal)? + .normalized_model(), + ) + .map_err(|_| { + CommandError::CompileRejected(failure(ResourceFault::Allocation)) + })?; + if !observation_change.ended.is_empty() { + let mut ended = Vec::new(); + ended + .try_reserve_exact(observation_change.ended.len()) + .map_err(|_| { + CommandError::CompileRejected(failure(ResourceFault::Allocation)) + })?; + ended.extend(observation_change.ended.iter().map(ended_key)); + match self + .controller + .attach_observation_ended(prepared.get_mut(), ended) + { + Ok(_) => {} + Err(protocol::AttachObservationEndedError::NoRoom(_)) => { + drop(prepared); + return Err(CommandError::Backpressure); + } + Err(protocol::AttachObservationEndedError::Unencodable) => { + drop(prepared); + return Err(CommandError::CompileRejected(failure( + ResourceFault::Limit, + ))); + } + Err(_) => return Err(CommandError::Internal), + } + } let prepared_runtime = prepare_runtime::( &*self.preparer, prepared @@ -1340,6 +1476,7 @@ where next: &monitor_change.overlay, ramp: monitor_change.ramp, }, + observation: &observation_change.bindings, }), &self.live_values_superseded, ) @@ -1470,12 +1607,29 @@ where drop(prepared); return Err(CommandError::Internal); } + // #1584 D6a: the newest plan's pending-subscription storage is built here, with + // the other fallible steps, so no allocation follows the commit; the commit then + // only assigns it. A refusal drops what was built and changes nothing. + let slots = match observation_slots(&candidate_provider.effects) { + Ok(slots) => slots, + Err(_) => { + drop(reports); + drop(reservation); + drop(candidate_provider); + drop(prepared); + return Err(CommandError::CompileRejected(failure( + ResourceFault::Allocation, + ))); + } + }; // #1313 D2: a transaction that takes this arm is a rebuild, whatever else it holds. let committed = prepared .commit(&mut self.controller, protocol::EditPath::Rebuild) .map_err(|_| CommandError::Internal)?; // #1559 D8: the overlay is written after the last fallible step, with the commit. self.monitor = monitor_change.overlay; + // #1584 D6a: and the subscriptions the successor was prepared with. + self.observation = observation_change.bindings; // Issue #1273 D2: the protocol commit was the last fallible step, so the // persisting producers move now and never move back. Infallible. // The newest epoch, spelled by field: the reservation borrows the publisher. @@ -1500,6 +1654,9 @@ where ); candidate_provider.revision_gate.publish(revision); self.pending_providers.push(candidate_provider); + // #1584 D6a: the newest plan's slots, built above. The fold ran when this command + // was serviced and nothing has been pushed since (one thread), so none is lost. + (self.observation_offsets, self.observation_pending) = slots; reports.push((epoch, A::Row::from(resources))); reservation.commit(); #[cfg(feature = "test-support")] @@ -2248,10 +2405,80 @@ where armed: bool, window_blocks: u32, ) -> Option> { - self.newest_providers_mut() + let producer = self + .pending_providers + .last_mut() + .unwrap_or(&mut self.providers) .effects - .get_mut(position) - .map(|producer| producer.try_push_observe(tap_index, armed, window_blocks)) + .get_mut(position)?; + let pushed = producer.try_push_observe(tap_index, armed, window_blocks); + if pushed.is_ok() { + // #1584 D6a: the push is the acknowledgement, so the state every later successor is + // prepared from is recorded with it, after the only fallible step and without + // allocating: this runs inside the browser's `process()`. A record the lane refused + // records nothing. The next service step folds it into the table. + let slots = self + .observation_offsets + .get(position) + .copied() + .zip(self.observation_offsets.get(position + 1).copied()); + if let Some((first, end)) = slots + && let Some(slot) = first.checked_add(tap_index).filter(|slot| *slot < end) + && let Some(pending) = self.observation_pending.get_mut(slot as usize) + { + *pending = if armed { + PendingArm::Arm(window_blocks) + } else { + PendingArm::Disarm + }; + } + } + Some(pushed) + } + + /// Folds what the host pushed to the observation slots into the table of armed taps, in slot + /// order, and clears it. Control thread: it allocates for a new binding's key. + pub(crate) fn fold_observation_pending(&mut self) { + if self + .observation_pending + .iter() + .all(|arm| *arm == PendingArm::None) + { + return; + } + let producers = &self + .pending_providers + .last() + .unwrap_or(&self.providers) + .effects; + for (position, producer) in producers.iter().enumerate() { + let (Some(&first), Some(&end)) = ( + self.observation_offsets.get(position), + self.observation_offsets.get(position + 1), + ) else { + break; + }; + for slot in first..end { + let arm = core::mem::take(&mut self.observation_pending[slot as usize]); + let Some(key) = tap_key_of(producer, slot - first) else { + continue; + }; + match arm { + PendingArm::None => {} + PendingArm::Arm(window) => self.observation.set(key, window), + PendingArm::Disarm => { + self.observation.clear(&key); + } + } + } + } + } + + /// The taps the host has armed (#1584 D6a), for a test or an adapter that reports them. + #[must_use] + pub fn observation_bindings(&mut self) -> &controls::ObservationBindings { + self.fold_observation_pending(); + &self.observation } /// Feeds a source of the newest committed session. A source that persists across a pending diff --git a/crates/control-plane/src/lib.rs b/crates/control-plane/src/lib.rs index 983269537..5d5c1683d 100644 --- a/crates/control-plane/src/lib.rs +++ b/crates/control-plane/src/lib.rs @@ -70,8 +70,8 @@ pub use plan::{PlanQueries, PlanState}; #[cfg(feature = "test-support")] pub use compile::{ - CapiResources, CompiledModelAdmission, LiveCatalogAdmission, LiveEpochResources, prepare_caps, - validate_live_peak, + CapiResources, CompiledModelAdmission, LiveCatalogAdmission, LiveEpochResources, + observation_reserve_bytes, prepare_caps, validate_live_peak, }; #[cfg(feature = "test-support")] pub use control::{ diff --git a/crates/control-plane/tests/observation_bindings.rs b/crates/control-plane/tests/observation_bindings.rs new file mode 100644 index 000000000..8e87346cf --- /dev/null +++ b/crates/control-plane/tests/observation_bindings.rs @@ -0,0 +1,656 @@ +//! #1584 D6a: the subscriptions a host acknowledged survive every rebuild, and a rebuild that +//! removes their effect tells the subscribers. +//! +//! The session holds the table of armed taps next to the monitor overlay. A push to a tap's queue +//! writes it; a rebuild prepares its successor from it and drops the bindings whose target the +//! transaction removed, emitting ONE reliable `OBSERVATION_ENDED` per commit, carrying every ended +//! key, after the transaction's `SESSION_COMMITTED`, the room for it reserved before the commit. +#![cfg(feature = "test-support")] + +use control_plane::{ + CommandError, CompileFailure, CompiledChildren, ControlAdapter, ControlLimits, EventLane, + PlanResources, PlanState, ResourceFault, RuntimePreparer, SessionState, compile_children, +}; +use controls::{TapKey, TapSection}; +use host_core::{ + CompiledSession, HostLiveControlHandles, HostLiveControlRequest, HostLiveLanes, + HostPrepareCaps, LiveEffectAddress, LiveValuesSuperseded, PrepareDiagnostics, PreparedHost, + SuccessorBase, prepare_host_runtime_with_live_lanes, + prepare_host_runtime_with_live_lanes_successor, +}; +use protocol::{ + CommandPayload, DecodeScratch, DecodedEventPayload, ExpectedRevision, ObservationSection, + ProtocolCodec, RequestId, SessionEdit, SessionRevision, TypedCommandFrame, +}; +use session::{ + ControlAddress, Effect, EffectIdentity, EffectQuality, Lane, LinkMode, Owner, RackName, + Section, SidechainDeclaration, StableId, +}; + +const SESSION: &str = + include_str!("../../../fixtures/session/v1/compressor-dynamic-bank-observation.json"); +const WINDOW_BLOCKS: u32 = 4; + +struct Adapter; + +impl ControlAdapter for Adapter { + type Row = PlanResources; + const ADAPTER_ALLOCATIONS: &'static [u64] = &[]; +} + +/// A preparer that asks for observation capacity, as a host with subscriptions does. +struct ObservingPreparer; + +impl RuntimePreparer for ObservingPreparer { + fn prepare( + &self, + compiled: &CompiledSession, + caps: &HostPrepareCaps, + lanes: HostLiveLanes, + successor: Option>, + superseded: &LiveValuesSuperseded, + ) -> Result<(PreparedHost, HostLiveControlHandles), PrepareDiagnostics> { + let live = HostLiveControlRequest { + observation_taps: 4, + ..HostLiveControlRequest::default() + }; + match successor { + Some(base) => prepare_host_runtime_with_live_lanes_successor( + compiled, caps, &live, lanes, base, superseded, + ), + None => prepare_host_runtime_with_live_lanes(compiled, caps, &live, lanes, superseded), + } + } +} + +fn limits() -> ControlLimits { + ControlLimits { + source_ring_frames: 1_024, + maximum_automation_spans_per_block: 128, + maximum_document_bytes: 1_000_000, + maximum_diagnostic_bytes: 4_096, + maximum_tracks: 100, + maximum_sources: 100, + maximum_routes: 100, + maximum_effects: 100, + maximum_graph_session_plus_plan_bytes: 1_000_000_000, + maximum_source_total_bytes: 100_000_000, + maximum_source_overhead_bytes: 100_000_000, + maximum_effect_state_bytes: 1_000_000_000, + maximum_effect_scratch_bytes: 1_000_000_000, + maximum_builtin_retained_bytes: 1_000_000_000, + maximum_control_retained_bytes: 100_000_000, + maximum_named_allocation_bytes: 1_000_000_000, + maximum_meter_streams: 1, + maximum_meter_items: 1, + maximum_meter_bytes: 1, + maximum_control_frame_bytes: 65_536, + maximum_replay_bytes: 262_144, + maximum_replay_entries: 16, + maximum_submixes: 0, + maximum_vcas: 0, + maximum_vca_reach_pairs: u64::MAX, + maximum_replacement_peak_bytes: u64::MAX, + } +} + +struct Rig { + session: SessionState, + plan: PlanState, + next_request: u64, + block: u64, +} + +impl Rig { + fn new() -> Self { + Self::with_limits(limits()) + } + + fn with_limits(limits: ControlLimits) -> Self { + let CompiledChildren { session, plan } = + compile_children::(SESSION, limits, Box::new(ObservingPreparer)) + .expect("boot compile"); + Self { + session, + plan, + next_request: 1, + block: 0, + } + } + + /// Render one block, so the plan drains the observation queues. + fn render(&mut self) { + let mut samples = [0.0_f32; 256]; + let output = engine::realtime::PlanarBufferMut::try_new(&mut samples, 2, 128, 128) + .expect("output planes"); + self.plan + .render(self.block * 128, output) + .expect("a rendered block"); + self.block += 1; + } + + fn revision(&self) -> u64 { + self.session.test_controller().session().revision().0 + } + + fn frame(&self, request: u64, edits: &[SessionEdit]) -> Vec { + let codec = ProtocolCodec::default(); + let frame = TypedCommandFrame { + request_id: RequestId::new(request).expect("nonzero request"), + expected_revision: ExpectedRevision::Exact(SessionRevision(self.revision())), + payload: CommandPayload::SessionTransactionApply(edits), + }; + let mut bytes = vec![0_u8; codec.limits().max_frame_bytes]; + let length = codec + .encode_command_frame_into(&frame, &mut bytes) + .expect("typed command"); + bytes.truncate(length); + bytes + } + + /// Submit `edits` as a new request. + fn apply(&mut self, edits: &[SessionEdit]) -> Result { + let request = self.next_request; + self.next_request += 1; + let bytes = self.frame(request, edits); + self.session.command(&bytes, 65_536) + } + + /// Arm tap 0 (gain reduction) of `strip`'s `comp` insert, as the browser's kind 7 does. + fn subscribe(&mut self, strip: &str) { + let position = self.position(strip); + self.session + .push_effect_observe(position, 0, true, WINDOW_BLOCKS) + .expect("a producer") + .expect("room in the queue"); + } + + fn unsubscribe(&mut self, strip: &str) { + let position = self.position(strip); + self.session + .push_effect_observe(position, 0, false, 0) + .expect("a producer") + .expect("room in the queue"); + } + + fn position(&self, strip: &str) -> usize { + self.session + .effect_producers() + .iter() + .position(|producer| { + &*producer.track_id == strip && producer.address == LiveEffectAddress::insert(0) + }) + .expect("the strip's insert has a producer") + } + + /// Every reliable event queued, decoded: `(revision, payload)`. + fn drain(&mut self) -> Vec<(u64, DecodedEventPayload<'static>)> { + let codec = ProtocolCodec::default(); + let mut events = Vec::new(); + while let Some(length) = self + .session + .dequeue_event(EventLane::Reliable, 65_536) + .expect("event") + { + let bytes = self.session.event_response(length).to_vec(); + let mut fields = vec![0_u16; 4_096]; + let mut scratch = DecodeScratch::new(&mut fields); + let decoded = codec + .decode_typed_event(&bytes, &mut scratch) + .expect("an event frame"); + let revision = decoded.header.revision.0; + // Only the owned payloads this test reads are kept. + let payload = match decoded.payload { + DecodedEventPayload::SessionCommitted(value) => { + DecodedEventPayload::SessionCommitted(value) + } + DecodedEventPayload::ObservationEnded(value) => { + DecodedEventPayload::ObservationEnded(value) + } + _ => continue, + }; + events.push((revision, payload)); + } + events + } +} + +fn key(strip: &str) -> TapKey { + TapKey { + strip: StableId::parse(strip).expect("strip"), + section: TapSection::Inserts(StableId::parse("comp").expect("insert")), + tap_id: 1, + } +} + +fn insert_address(strip: &str, control: u32) -> ControlAddress { + ControlAddress { + owner: Owner::Strip(StableId::parse(strip).expect("strip")), + section: Section::Inserts(StableId::parse("comp").expect("insert")), + control, + lane: Lane::Both, + } +} + +fn remove_insert(strip: &str) -> SessionEdit { + SessionEdit::RemoveTrackEffect { + track_id: StableId::parse(strip).expect("strip"), + rack_name: RackName::Inserts, + effect_id: StableId::parse("comp").expect("insert"), + } +} + +/// An insert `comp` that is a delay: the ID of the compressor, reused for an effect with no taps. +fn delay_as_comp(strip: &str) -> SessionEdit { + SessionEdit::PutTrackEffect { + track_id: StableId::parse(strip).expect("strip"), + rack_name: RackName::Inserts, + final_position: 0, + effect: Effect { + id: StableId::parse("comp").expect("insert"), + identity: EffectIdentity::Native { + effect_id: StableId::parse("miso.delay").expect("delay"), + }, + quality: EffectQuality::Normal, + bypass: false, + link_mode: LinkMode::DualMono, + params: Vec::new(), + sidechain: SidechainDeclaration::None, + }, + } +} + +fn ended(events: &[(u64, DecodedEventPayload<'static>)]) -> Vec<(u64, protocol::ObservationEnded)> { + events + .iter() + .filter_map(|(revision, payload)| match payload { + DecodedEventPayload::ObservationEnded(value) => Some((*revision, value.clone())), + _ => None, + }) + .collect() +} + +fn committed(events: &[(u64, DecodedEventPayload<'static>)]) -> usize { + events + .iter() + .filter(|(_, payload)| matches!(payload, DecodedEventPayload::SessionCommitted(_))) + .count() +} + +/// The push is the acknowledgement: it writes the table with it, a disarm clears it, and a push +/// the lane refuses writes nothing. +/// +/// Red if the table is written before the push succeeds, ignores a disarm, or keys a tap by +/// anything but its strip, section and tap ID. +#[test] +fn a_push_writes_the_table_and_a_refused_push_writes_nothing() { + let mut rig = Rig::new(); + assert!(rig.session.observation_bindings().is_empty()); + rig.subscribe("comp1"); + rig.subscribe("comp3"); + let bindings = rig.session.observation_bindings(); + assert_eq!(bindings.len(), 2); + assert_eq!(bindings.window_of(&key("comp1")), Some(WINDOW_BLOCKS)); + // The queue holds one slot per declared tap, so a disarm before any render is refused, and + // the binding stays: the table never says less than the lane was told. + let position = rig.position("comp1"); + let refused = rig + .session + .push_effect_observe(position, 0, false, 0) + .expect("a producer"); + assert!(refused.is_err(), "the queue has no room for the disarm"); + assert_eq!(rig.session.observation_bindings().len(), 2); + // After a render drained it, the disarm is accepted and clears the binding. + rig.render(); + rig.unsubscribe("comp1"); + assert_eq!(rig.session.observation_bindings().len(), 1); + assert_eq!( + rig.session.observation_bindings().window_of(&key("comp1")), + None + ); + assert_eq!( + rig.session.observation_bindings().window_of(&key("comp3")), + Some(WINDOW_BLOCKS) + ); +} + +/// A subscription survives a rebuild that restarts its effect (here a link-mode change): the +/// table is unchanged and the transaction ends nothing, so no `OBSERVATION_ENDED` follows the +/// `SESSION_COMMITTED`. (That the restarted tap is armed and opens at the adoption block is +/// host-core's gate.) +/// +/// Red if a rebuild empties the table or ends a binding whose effect is still there. +#[test] +fn a_restart_keeps_the_binding_and_ends_nothing() { + let mut rig = Rig::new(); + rig.subscribe("comp1"); + let revision = rig.revision(); + rig.apply(&[SessionEdit::set_control( + insert_address("comp1", controls::LINK_MODE_CONTROL), + effect_contract::LinkMode::Average as u32 as f32, + None, + )]) + .expect("a rebuild"); + assert_eq!(rig.revision(), revision + 1); + assert_eq!(rig.session.observation_bindings().len(), 1); + assert_eq!( + rig.session.observation_bindings().window_of(&key("comp1")), + Some(WINDOW_BLOCKS) + ); + let events = rig.drain(); + assert_eq!(committed(&events), 1); + assert!(ended(&events).is_empty()); +} + +/// Each way an effect leaves ends its binding with one reliable event after the transaction's +/// `SESSION_COMMITTED`, at the committing revision, naming the stable key; a binding of another +/// strip stays. +/// +/// Red if a removed effect keeps its binding silently (a subscriber waits for windows that never +/// come), if the event names the wrong revision or key, or if a surviving binding is ended. +#[test] +fn a_removed_effect_ends_its_binding_with_an_event() { + let cases: [(&str, Vec); 3] = [ + ("an insert removed", vec![remove_insert("comp1")]), + ( + "a strip removed", + vec![ + SessionEdit::RemoveRoute { + route_id: StableId::parse("comp1-main").expect("route"), + }, + SessionEdit::RemoveTrack { + track_id: StableId::parse("comp1").expect("strip"), + }, + ], + ), + ( + "an ID reused for an effect without the tap", + vec![delay_as_comp("comp1")], + ), + ]; + for (what, edits) in cases { + let mut rig = Rig::new(); + rig.subscribe("comp1"); + rig.subscribe("comp2"); + let before = rig.revision(); + rig.apply(&edits) + .unwrap_or_else(|error| panic!("{what}: {error:?}")); + let revision = rig.revision(); + assert_eq!(revision, before + 1, "{what}"); + assert_eq!(rig.session.observation_bindings().len(), 1, "{what}"); + assert!( + rig.session + .effect_producers() + .iter() + .filter(|producer| &*producer.track_id == "comp1") + .all(|producer| producer.observe_capacity() == 0), + "{what}: the successor has no tap for the ended binding" + ); + assert_eq!( + rig.session.observation_bindings().window_of(&key("comp2")), + Some(WINDOW_BLOCKS), + "{what}: the other strip's binding stays" + ); + let events = rig.drain(); + assert_eq!( + events + .iter() + .map(|(_, payload)| matches!(payload, DecodedEventPayload::SessionCommitted(_))) + .collect::>(), + vec![true, false], + "{what}: SESSION_COMMITTED, then the end" + ); + let ended = ended(&events); + assert_eq!(ended.len(), 1, "{what}"); + let (event_revision, event) = &ended[0]; + assert_eq!(*event_revision, revision, "{what}: the committing revision"); + assert_eq!(event.keys.len(), 1, "{what}"); + assert_eq!(event.keys[0].strip_id, "comp1", "{what}"); + assert_eq!(event.keys[0].section, ObservationSection::Inserts, "{what}"); + assert_eq!(event.keys[0].effect_id, "comp", "{what}"); + assert_eq!(event.keys[0].tap_id, 1, "{what}"); + // Sequences are endpoint-monotonic: the end follows the commit's by one. + let DecodedEventPayload::SessionCommitted(commit) = &events[0].1 else { + panic!("{what}: first event"); + }; + assert_eq!(event.event_sequence, commit.event_sequence + 1, "{what}"); + } +} + +/// The room for the end event is checked with the commit's own event before anything changes (the +/// acked-batch rule): a lane that holds an unread event, leaving room for `SESSION_COMMITTED` and +/// not for the end, refuses the transaction as backpressure, and the model, the revision, the table +/// and the lane are as they were; after a drain the same request commits. Backpressure shows only +/// while the lane holds unread events: the empty lane takes the same removal. +/// +/// Red if the commit goes ahead and drops the end, or if a refusal leaves the table emptied or +/// the revision moved. +#[test] +fn no_room_for_the_end_refuses_the_removal_and_changes_nothing() { + let mut rig = Rig::new(); + rig.subscribe("comp1"); + // One committed event unread in the two-slot lane: one slot is free, the removal needs two. + rig.apply(&[SessionEdit::SetSessionId { + session_id: StableId::parse("another-session").expect("ID"), + }]) + .expect("a model-only commit"); + let revision = rig.revision(); + let canonical = rig.session.test_transaction_snapshot().canonical; + let bindings = rig.session.observation_bindings().clone(); + let epoch = rig.session.newest_plan_epoch(); + let request = rig.next_request; + let bytes = rig.frame(request, &[remove_insert("comp1")]); + rig.next_request += 1; + let refused = rig.session.command(&bytes, 65_536); + assert!( + matches!(refused, Err(CommandError::Backpressure)), + "{refused:?}" + ); + assert_eq!(rig.revision(), revision); + assert_eq!(rig.session.test_transaction_snapshot().canonical, canonical); + assert_eq!(rig.session.observation_bindings(), &bindings); + assert_eq!( + rig.session.newest_plan_epoch(), + epoch, + "no plan was published" + ); + let events = rig.drain(); + assert_eq!(committed(&events), 1, "only the earlier commit's event"); + assert!(ended(&events).is_empty()); + + // The lane is empty: the same request now commits, with its end. + let bytes = rig.frame(request, &[remove_insert("comp1")]); + rig.session + .command(&bytes, 65_536) + .expect("the same removal commits after a drain"); + assert_eq!(rig.revision(), revision + 1); + assert!(rig.session.observation_bindings().is_empty()); + let events = rig.drain(); + assert_eq!(committed(&events), 1); + assert_eq!(ended(&events).len(), 1); +} + +/// Root ruling A: a transaction that removes a strip with N subscribed inserts commits whatever N is +/// (the lane has two slots), and gives exactly one `OBSERVATION_ENDED` after its +/// `SESSION_COMMITTED`, at the committing revision, with the N stable keys in key order. +/// +/// Red if the keys are split across events (the second no longer fits the two-slot lane and the +/// removal is refused as backpressure forever), if the event holds fewer or more keys than the +/// commit ended, or if the keys are out of key order. +#[test] +fn a_strip_removed_with_many_subscribed_inserts_gives_one_event_with_every_key() { + fn put_compressor(strip: &str, id: &str) -> SessionEdit { + SessionEdit::PutTrackEffect { + track_id: StableId::parse(strip).expect("strip"), + rack_name: RackName::Inserts, + final_position: 1, + effect: Effect { + id: StableId::parse(id).expect("insert"), + identity: EffectIdentity::Native { + effect_id: StableId::parse("miso.compressor").expect("compressor"), + }, + quality: EffectQuality::Normal, + bypass: false, + link_mode: LinkMode::DualMono, + params: Vec::new(), + sidechain: SidechainDeclaration::None, + }, + } + } + for count in [1_u32, 2, 7, 40] { + let mut rig = Rig::new(); + let ids: Vec = (0..count).map(|index| format!("ins{index:02}")).collect(); + let puts: Vec = ids.iter().map(|id| put_compressor("comp1", id)).collect(); + rig.apply(&puts).expect("the inserts are added"); + // The plan with the inserts is adopted at a block, so the next rebuild has room. + rig.render(); + rig.render(); + rig.drain(); + for id in &ids { + let position = rig + .session + .effect_producers() + .iter() + .position(|producer| &*producer.track_id == "comp1" && &*producer.effect_id == id) + .expect("the insert has a producer"); + rig.session + .push_effect_observe(position, 0, true, WINDOW_BLOCKS) + .expect("a producer") + .expect("room in the queue"); + } + assert_eq!(rig.session.observation_bindings().len(), ids.len()); + let before = rig.revision(); + rig.apply(&[ + SessionEdit::RemoveRoute { + route_id: StableId::parse("comp1-main").expect("route"), + }, + SessionEdit::RemoveTrack { + track_id: StableId::parse("comp1").expect("strip"), + }, + ]) + .unwrap_or_else(|error| panic!("{count} subscribed inserts: {error:?}")); + assert_eq!(rig.revision(), before + 1, "{count}"); + assert!(rig.session.observation_bindings().is_empty(), "{count}"); + let events = rig.drain(); + assert_eq!(events.len(), 2, "{count}: SESSION_COMMITTED and one end"); + let ended = ended(&events); + assert_eq!(ended.len(), 1, "{count}: exactly one event"); + let (revision, event) = &ended[0]; + assert_eq!(*revision, before + 1, "{count}"); + let got: Vec<(&str, &str, u32)> = event + .keys + .iter() + .map(|key| (&*key.strip_id, &*key.effect_id, key.tap_id)) + .collect(); + let want: Vec<(&str, &str, u32)> = ids.iter().map(|id| ("comp1", id.as_str(), 1)).collect(); + assert_eq!(got, want, "{count}: every key, in key order"); + assert!( + event + .keys + .iter() + .all(|key| key.section == ObservationSection::Inserts), + "{count}" + ); + } +} + +/// An `OBSERVATION_ENDED` event that cannot fit one control frame can never be sent, so the removal +/// that would end that many subscriptions is refused as a resource limit (not retryable +/// backpressure) before anything changes: the revision, the canonical session, the bindings, the +/// plan epoch and the event lane are as they were. A strip with two subscribed inserts (an end +/// that fits) commits under the same small limit. +/// +/// Red if the commit goes ahead and drops the end (an acked removal whose subscribers never hear), +/// if the refusal is backpressure (the host would retry forever), or if the refusal leaves the +/// table emptied, a plan published or an event queued. +#[test] +fn an_end_event_over_one_control_frame_refuses_the_removal_and_changes_nothing() { + fn put_compressor(strip: &str, id: &str) -> SessionEdit { + SessionEdit::PutTrackEffect { + track_id: StableId::parse(strip).expect("strip"), + rack_name: RackName::Inserts, + final_position: 1, + effect: Effect { + id: StableId::parse(id).expect("insert"), + identity: EffectIdentity::Native { + effect_id: StableId::parse("miso.compressor").expect("compressor"), + }, + quality: EffectQuality::Normal, + bypass: false, + link_mode: LinkMode::DualMono, + params: Vec::new(), + sidechain: SidechainDeclaration::None, + }, + } + } + let remove_strip = || { + vec![ + SessionEdit::RemoveRoute { + route_id: StableId::parse("comp1-main").expect("route"), + }, + SessionEdit::RemoveTrack { + track_id: StableId::parse("comp1").expect("strip"), + }, + ] + }; + let mut small = limits(); + small.maximum_control_frame_bytes = 1_024; + // The strip with `count` subscribed inserts, under the small frame. + let build = |count: u32| { + let mut rig = Rig::with_limits(small); + let ids: Vec = (0..count).map(|index| format!("ins{index:02}")).collect(); + // One insert per request: the small frame holds one put, not sixty. + for id in &ids { + rig.apply(&[put_compressor("comp1", id)]) + .expect("the insert is added"); + rig.render(); + rig.render(); + rig.drain(); + let position = rig + .session + .effect_producers() + .iter() + .position(|producer| &*producer.track_id == "comp1" && &*producer.effect_id == id) + .expect("the insert has a producer"); + rig.session + .push_effect_observe(position, 0, true, WINDOW_BLOCKS) + .expect("a producer") + .expect("room in the queue"); + } + assert_eq!(rig.session.observation_bindings().len(), ids.len()); + rig + }; + + // A few keys fit the frame: the same removal commits under the same limit. + let mut fits = build(2); + let before = fits.revision(); + fits.apply(&remove_strip()) + .expect("two ended keys fit one frame"); + assert_eq!(fits.revision(), before + 1); + assert_eq!(ended(&fits.drain()).len(), 1); + + let mut rig = build(60); + let revision = rig.revision(); + let canonical = rig.session.test_transaction_snapshot().canonical; + let bindings = rig.session.observation_bindings().clone(); + let epoch = rig.session.newest_plan_epoch(); + let refused = rig.apply(&remove_strip()); + assert!( + matches!( + refused, + Err(CommandError::CompileRejected(CompileFailure::Resource( + ResourceFault::Limit + ))) + ), + "{refused:?}" + ); + assert_eq!(rig.revision(), revision); + assert_eq!(rig.session.test_transaction_snapshot().canonical, canonical); + assert_eq!(rig.session.observation_bindings(), &bindings); + assert_eq!( + rig.session.newest_plan_epoch(), + epoch, + "no plan was published" + ); + assert!(rig.drain().is_empty(), "no event was queued"); +} diff --git a/crates/controls/src/effect_rows.rs b/crates/controls/src/effect_rows.rs index cfc2fba0f..e12e2da28 100644 --- a/crates/controls/src/effect_rows.rs +++ b/crates/controls/src/effect_rows.rs @@ -12,9 +12,9 @@ use effect_compiler::{launch_native_effect_registry, lowers_session_bypass}; use effect_contract::control::PreparedReason; use effect_contract::{ - AutomationRate, EffectDescriptor, LinkMode as ContractLinkMode, ParameterChannelPolicy, - ParameterDescriptor, ParameterDomain, ParameterLattice, ParameterMapping, ParameterUnit, - normalize_zero, + AutomationRate, EffectDescriptor, LinkMode as ContractLinkMode, ObservationCost, + ObservationDescriptor, ParameterChannelPolicy, ParameterDescriptor, ParameterDomain, + ParameterLattice, ParameterMapping, ParameterUnit, normalize_zero, }; use session::{ ConsoleEntry, Effect, EffectIdentity, EffectParam, LinkMode, ParameterChannel, @@ -436,6 +436,23 @@ fn identity_of<'a>( } } +/// The declared observation tap `tap_id` of the effect at `section` of strip `id`, if the strip, +/// the instance and the tap exist and the tap is [`ObservationCost::Resident`] (a computed tap has +/// no subscription path). `None` for a `cid` effect, an unknown effect and a section that is not +/// an effect's. +pub(crate) fn resident_observation_tap( + model: &SessionModel, + id: &StableId, + section: &Section, + tap_id: u32, +) -> Option<&'static ObservationDescriptor> { + let descriptor = identity_of(model, id, section).and_then(contract)?; + descriptor + .observations + .iter() + .find(|tap| tap.id.0 == tap_id && tap.cost == ObservationCost::Resident) +} + /// The mutable instance at `section` of strip `id`. fn instance_mut<'a>( model: &'a mut SessionModel, diff --git a/crates/controls/src/lib.rs b/crates/controls/src/lib.rs index bc89ecdfe..83ee1b379 100644 --- a/crates/controls/src/lib.rs +++ b/crates/controls/src/lib.rs @@ -34,6 +34,7 @@ mod lane; mod live_lanes; mod live_ramps; mod mask; +mod observation; mod owner_edits; mod registry; mod route; @@ -60,6 +61,9 @@ pub use error::ControlError; pub use live_lanes::{LiveLanes, live_lanes}; pub use live_ramps::LiveRamps; pub use mask::mask_values; +pub use observation::{ + ObservationBinding, ObservationBindings, ObservationChange, TapKey, TapSection, +}; pub use owner_edits::{OwnerEditError, OwnerEdits}; pub use registry::{ ControlRow, check_monitor, derived_rows_in, derived_rows_with, get, list, owner_count, diff --git a/crates/controls/src/observation.rs b/crates/controls/src/observation.rs new file mode 100644 index 000000000..5a3198c90 --- /dev/null +++ b/crates/controls/src/observation.rs @@ -0,0 +1,189 @@ +//! The host's observation subscriptions (#1584): which effect taps a host armed and with which +//! window, kept apart from the monitor overlay (solo and subscriptions are different things). +//! +//! A subscription is host-local state. It is never saved and never in a snapshot, and it needs no +//! revision. A rebuild must not lose it: the host prepares each successor plan from the bindings +//! the transaction leaves ([`ObservationBindings::after`]), so a tap the host acknowledged is armed +//! in the successor from its first rendered block. A binding whose target the next model no longer +//! has ends, and [`ObservationChange::ended`] names it, so the host tells its subscribers. + +use std::collections::TryReserveError; + +use session::{Section, SessionModel, StableId}; + +use crate::effect_rows::resident_observation_tap; +use crate::registry::strip_index; + +/// The effect section a tap belongs to: a console slot or an insert of one strip. +#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)] +pub enum TapSection { + /// A session console slot, by its slot ID. + Console(StableId), + /// One of the strip's inserts, by its effect ID. + Inserts(StableId), +} + +impl TapSection { + /// The ID of the console slot or the insert. + #[must_use] + pub fn id(&self) -> &StableId { + match self { + Self::Console(id) | Self::Inserts(id) => id, + } + } + + /// The control section this names. + #[must_use] + pub fn section(&self) -> Section { + match self { + Self::Console(id) => Section::Console(id.clone()), + Self::Inserts(id) => Section::Inserts(id.clone()), + } + } + + /// Whether this is a console slot. + #[must_use] + pub const fn is_console(&self) -> bool { + matches!(self, Self::Console(_)) + } +} + +/// The stable key of one observation tap: the strip, the effect section and the effect-local tap +/// ID. It names no plan position, so it survives every rebuild while the effect exists. +#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)] +pub struct TapKey { + /// The strip (a track or a submix). + pub strip: StableId, + /// The console slot or insert. + pub section: TapSection, + /// The effect-local tap ID (`ObservationDescriptor::id`). + pub tap_id: u32, +} + +/// One armed tap and the window it was armed with. +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct ObservationBinding { + /// The tap. + pub key: TapKey, + /// The window length in blocks as it was asked for; `0` is the plan's meter window, which each + /// plan resolves for itself. + pub window_blocks: u32, +} + +/// The taps a host has armed, sorted by key. +/// +/// In this slice a binding is an absolute set or clear (the browser's observe kinds 7 and 8); a +/// shared arm with reference counts belongs to the transaction edit that arms through the core. +#[derive(Clone, Debug, Default, Eq, PartialEq)] +pub struct ObservationBindings { + /// Sorted by key, with no duplicate key. + bindings: Vec, +} + +/// What [`ObservationBindings::after`] returns. +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct ObservationChange { + /// The bindings whose target the next model still has. + pub bindings: ObservationBindings, + /// The keys of the bindings the change ended, in key order: the strip, the insert or the slot + /// is gone, or the effect behind the ID has no resident tap with that ID. + pub ended: Vec, +} + +impl ObservationBindings { + /// No binding. `const`, so a host may keep one in a `static`. + #[must_use] + pub const fn new() -> Self { + Self { + bindings: Vec::new(), + } + } + + /// The bindings, sorted by key. + #[must_use] + pub fn bindings(&self) -> &[ObservationBinding] { + &self.bindings + } + + /// How many taps are armed. + #[must_use] + pub fn len(&self) -> usize { + self.bindings.len() + } + + /// Whether no tap is armed. + #[must_use] + pub fn is_empty(&self) -> bool { + self.bindings.is_empty() + } + + /// The window `key` is armed with, or `None` when it is not armed. + #[must_use] + pub fn window_of(&self, key: &TapKey) -> Option { + self.search(key) + .ok() + .map(|index| self.bindings[index].window_blocks) + } + + /// Arm `key` with `window_blocks`, replacing any earlier window. + pub fn set(&mut self, key: TapKey, window_blocks: u32) { + match self.search(&key) { + Ok(index) => self.bindings[index].window_blocks = window_blocks, + Err(index) => self + .bindings + .insert(index, ObservationBinding { key, window_blocks }), + } + } + + /// Disarm `key`. Returns whether it was armed. + pub fn clear(&mut self, key: &TapKey) -> bool { + match self.search(key) { + Ok(index) => { + self.bindings.remove(index); + true + } + Err(_) => false, + } + } + + fn search(&self, key: &TapKey) -> Result { + self.bindings + .binary_search_by(|binding| binding.key.cmp(key)) + } + + /// Whether the next model still has the target of `key`: the strip, the section's instance + /// and a resident tap of that ID. + #[must_use] + pub fn has_target(model: &SessionModel, key: &TapKey) -> bool { + strip_index(model, &key.strip).is_some() + && resident_observation_tap(model, &key.strip, &key.section.section(), key.tap_id) + .is_some() + } + + /// The bindings on a session whose strips and effects are `next`'s: each binding whose target + /// `next` no longer has is dropped and its key returned in [`ObservationChange::ended`]. + /// + /// Both lists are reserved exactly before anything is copied, so a refusal leaves `self` + /// untouched and a caller can refuse the transaction before it commits anything. + /// + /// # Errors + /// + /// The allocation failed. + pub fn after(&self, next: &SessionModel) -> Result { + let mut kept = Vec::new(); + kept.try_reserve_exact(self.bindings.len())?; + let mut ended = Vec::new(); + ended.try_reserve_exact(self.bindings.len())?; + for binding in &self.bindings { + if Self::has_target(next, &binding.key) { + kept.push(binding.clone()); + } else { + ended.push(binding.key.clone()); + } + } + Ok(ObservationChange { + bindings: Self { bindings: kept }, + ended, + }) + } +} diff --git a/crates/controls/tests/observation.rs b/crates/controls/tests/observation.rs new file mode 100644 index 000000000..1bc497352 --- /dev/null +++ b/crates/controls/tests/observation.rs @@ -0,0 +1,154 @@ +//! The host's observation subscriptions (#1584 D6a): the table a rebuild is prepared from, and the +//! bindings a transaction ends. + +mod common; + +use common::{effects_fixture, id, insert_id}; +use controls::{ObservationBindings, TapKey, TapSection}; +use session::{EffectIdentity, SessionModel}; + +/// The compressor's gain-reduction tap, effect-local ID 1. +const GAIN_REDUCTION: u32 = 1; + +fn insert_key(strip: &str, tap_id: u32) -> TapKey { + TapKey { + strip: id(strip), + section: TapSection::Inserts(insert_id("miso.compressor")), + tap_id, + } +} + +fn console_key(strip: &str) -> TapKey { + TapKey { + strip: id(strip), + section: TapSection::Console(id("glue")), + tap_id: GAIN_REDUCTION, + } +} + +fn bound(keys: &[(TapKey, u32)]) -> ObservationBindings { + let mut bindings = ObservationBindings::new(); + for (key, window) in keys { + bindings.set(key.clone(), *window); + } + bindings +} + +fn strip_mut<'a>(model: &'a mut SessionModel, name: &str) -> &'a mut session::Track { + model + .tracks + .iter_mut() + .find(|track| track.id.as_str() == name) + .expect("a track") +} + +/// The table is sorted, an absolute set replaces the window, and a clear reports whether it held. +/// +/// Red if a binding is duplicated, a set keeps the older window, or a clear of an unarmed tap +/// claims it held one. +#[test] +fn the_table_sets_and_clears_by_key() { + let mut bindings = ObservationBindings::new(); + assert!(bindings.is_empty()); + bindings.set(insert_key("vocal", GAIN_REDUCTION), 4); + bindings.set(console_key("guitar"), 8); + bindings.set(insert_key("guitar", GAIN_REDUCTION), 0); + assert_eq!(bindings.len(), 3); + assert!( + bindings + .bindings() + .windows(2) + .all(|pair| pair[0].key < pair[1].key), + "sorted by key, no duplicate" + ); + bindings.set(insert_key("vocal", GAIN_REDUCTION), 16); + assert_eq!(bindings.len(), 3); + assert_eq!( + bindings.window_of(&insert_key("vocal", GAIN_REDUCTION)), + Some(16) + ); + assert!(bindings.clear(&insert_key("vocal", GAIN_REDUCTION))); + assert!(!bindings.clear(&insert_key("vocal", GAIN_REDUCTION))); + assert_eq!( + bindings.window_of(&insert_key("vocal", GAIN_REDUCTION)), + None + ); + assert_eq!(bindings.len(), 2); +} + +/// A transaction that leaves every target in place ends nothing; each way a target can go ends +/// exactly its bindings and names them: the strip removed, the insert removed, the insert's ID +/// reused for an effect without the tap, the console slot removed, a tap ID the effect has not. +/// +/// Red if a binding outlives its strip, its insert or its slot (a subscription with no tap), +/// if an ID reused for another effect keeps the old binding, or if a surviving binding is dropped. +#[test] +fn a_binding_ends_when_its_target_goes() { + let model = effects_fixture(); + let bindings = bound(&[ + (insert_key("vocal", GAIN_REDUCTION), 4), + (insert_key("guitar", GAIN_REDUCTION), 8), + (console_key("vocal"), 4), + ]); + let unchanged = bindings.after(&model).expect("room"); + assert_eq!(unchanged.bindings, bindings); + assert!(unchanged.ended.is_empty()); + + // A strip removed: its bindings end, the others stay. + let mut without_guitar = model.clone(); + without_guitar + .tracks + .retain(|track| track.id.as_str() != "guitar"); + let change = bindings.after(&without_guitar).expect("room"); + assert_eq!(change.ended, vec![insert_key("guitar", GAIN_REDUCTION)]); + assert_eq!(change.bindings.len(), 2); + + // An insert removed. + let mut without_insert = model.clone(); + strip_mut(&mut without_insert, "vocal") + .inserts + .effects + .retain(|effect| effect.id != insert_id("miso.compressor")); + let change = bindings.after(&without_insert).expect("room"); + assert_eq!(change.ended, vec![insert_key("vocal", GAIN_REDUCTION)]); + assert_eq!( + change + .bindings + .window_of(&insert_key("guitar", GAIN_REDUCTION)), + Some(8) + ); + + // The insert's ID reused for an effect without that tap. + let mut reused = model.clone(); + strip_mut(&mut reused, "guitar") + .inserts + .effects + .iter_mut() + .find(|effect| effect.id == insert_id("miso.compressor")) + .expect("the compressor insert") + .identity = EffectIdentity::Native { + effect_id: id("miso.delay"), + }; + let change = bindings.after(&reused).expect("room"); + assert_eq!(change.ended, vec![insert_key("guitar", GAIN_REDUCTION)]); + + // A console slot removed from the session (and every strip's entry for it). + let mut without_slot = model.clone(); + without_slot.console.pre_insert.clear(); + for strip in &mut without_slot.tracks { + strip.console.retain(|entry| entry.slot != id("glue")); + } + for strip in &mut without_slot.submixes { + strip.console.retain(|entry| entry.slot != id("glue")); + } + let change = bindings.after(&without_slot).expect("room"); + assert_eq!(change.ended, vec![console_key("vocal")]); + + // A tap the effect does not declare ends too. + let unknown = bound(&[(insert_key("vocal", 99), 4)]); + assert_eq!( + unknown.after(&model).expect("room").ended, + vec![insert_key("vocal", 99)] + ); + assert!(unknown.after(&model).expect("room").bindings.is_empty()); +} diff --git a/crates/effect-compiler/src/prepare.rs b/crates/effect-compiler/src/prepare.rs index 8a0c7a85f..2566d93eb 100644 --- a/crates/effect-compiler/src/prepare.rs +++ b/crates/effect-compiler/src/prepare.rs @@ -1751,6 +1751,26 @@ pub struct EffectObservationHandle { pub readers: Box<[ObservationReader]>, } +/// One observation tap a successor plan is prepared with armed (#1584 D6a): the host acknowledged +/// the subscription before the rebuild, and the rebuild must not drop it. +/// +/// An instance is addressed by its strip, its section (a console slot or an insert) and its +/// session ID, and the tap by its effect-local ID. Prefer building these from the host's +/// subscription table; an arm whose instance or tap the plan does not have is ignored. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub struct ObservationArm<'a> { + /// The strip (a track or a submix) the instance belongs to. + pub strip_id: &'a str, + /// Whether the instance is a console slot (`true`) or an insert (`false`). + pub console: bool, + /// The console slot ID or the insert's effect ID. + pub effect_id: &'a str, + /// The effect-local tap ID ([`effect_contract::ObservationTapId`]). + pub tap_id: u32, + /// The window length in blocks; `0` is the plan's meter window. + pub window_blocks: u32, +} + /// Attach observation capacity to every prepared effect that declares at least one tap. /// /// # Level 1 of the two-level zero (issue #143 D3) @@ -1777,8 +1797,19 @@ pub fn attach_effect_observation( prepared: &mut EffectPreparedSession, maximum_taps: u32, window_blocks: u32, + arms: &[ObservationArm<'_>], ) -> Result, EffectDiagnosticSet> { let declared = declared_live_addresses(&prepared.session); + // The arms by instance, so each entry finds its own in one lookup. + // (strip, console, instance ID) -> (tap ID, window blocks) of each armed tap. + type ArmedTaps<'a> = BTreeMap<(&'a str, bool, &'a str), Vec<(u32, u32)>>; + let mut armed = ArmedTaps::new(); + for arm in arms { + armed + .entry((arm.strip_id, arm.console, arm.effect_id)) + .or_default() + .push((arm.tap_id, arm.window_blocks)); + } let mut handles = Vec::new(); let mut diagnostics = Vec::new(); for entry in &mut prepared.entries { @@ -1812,7 +1843,8 @@ pub fn attach_effect_observation( publishers.push(publisher); readers.push(reader); } - let Some(lane) = ObservationLane::new(descriptor.observations, publishers, window_blocks) + let Some(mut lane) = + ObservationLane::new(descriptor.observations, publishers, window_blocks) else { diagnostics.push(EffectDiagnostic { code: "effect.observation.prepare", @@ -1820,6 +1852,22 @@ pub fn attach_effect_observation( }); continue; }; + // #1584 D6a: a tap the host acknowledged a subscription for is armed here, to open its + // first window at its first rendered block. An arm for a tap the effect does not declare + // is ignored: the host drops such a binding before it reaches here. + let console = !matches!(entry.rack, EffectRack::Dynamic); + if let Some(taps) = armed.get(&(entry.track_id.as_str(), console, entry.effect_id.as_str())) + { + for &(tap_id, window) in taps { + if let Some(index) = descriptor + .observations + .iter() + .position(|tap| tap.id.0 == tap_id) + { + lane.arm_prepared(index as u32, window); + } + } + } handles.push(EffectObservationHandle { track_id: entry.track_id.as_str().into(), address, diff --git a/crates/effect-contract/src/live.rs b/crates/effect-contract/src/live.rs index 2dbc27753..2326b9c77 100644 --- a/crates/effect-contract/src/live.rs +++ b/crates/effect-contract/src/live.rs @@ -1505,6 +1505,12 @@ struct ObservationTap { /// The declared fold, copied once at bind so the render path never walks the descriptor. fold: ObservationFold, armed: bool, + /// Armed at preparation ([`ObservationLane::arm_prepared`]) and not yet accumulated: the + /// window's first sample is the first sample of the first block that folds into it, which is + /// not known before the plan renders. Cleared by that block, by [`ObservationLane::arm`] and by + /// [`ObservationLane::disarm_all`]. Part of the tap's state, so [`ObservationLane::swap_state`] + /// moves it with the rest and a carried tap's own state replaces it whole. + start_pending: bool, /// Blocks per published window; never zero once armed. window_blocks: u32, blocks: u32, @@ -1562,6 +1568,7 @@ impl ObservationLane { publisher, fold: descriptor.fold, armed: false, + start_pending: false, window_blocks: default_window_blocks, blocks: 0, first_sample: 0, @@ -1649,6 +1656,7 @@ impl ObservationLane { _ => {} } tap.armed = armed; + tap.start_pending = false; tap.window_blocks = if window_blocks == 0 { default } else { @@ -1662,6 +1670,31 @@ impl ObservationLane { true } + /// Arm `tap_index` at preparation, before the plan has rendered a block (#1584 D6a). + /// + /// A successor plan is prepared from the monitor state the host acknowledged, so a subscription + /// survives a rebuild that restarts its effect. The adoption sample is not known here, so the + /// tap's first window opens at the first sample of the first block that accumulates into it + /// (`accumulate` takes it), exactly as if a record had armed it at that block. No record is + /// queued: a queued record would re-arm a *carried* tap at its first block and restart its + /// window. A carried tap exchanges its whole state with the predecessor's at the swap + /// ([`Self::swap_state`]), so this arm never wins over a carried tap's own. + /// + /// A zero `window_blocks` means the lane's default, as in [`Self::arm`]. Returns `false` for an + /// index this lane has no tap for. Off the render thread, before the plan is published. + pub fn arm_prepared(&mut self, tap_index: u32, window_blocks: u32) -> bool { + if !self.arm(tap_index, true, window_blocks, 0) { + return false; + } + if let Some(tap) = usize::try_from(tap_index) + .ok() + .and_then(|index| self.taps.get_mut(index)) + { + tap.start_pending = true; + } + true + } + /// Release every subscription without touching the published windows (issue #143 D7). /// /// A plan replacement drops subscriptions because the plan they addressed is gone; the app @@ -1671,6 +1704,7 @@ impl ObservationLane { self.armed_count = 0; for tap in self.taps.iter_mut() { tap.armed = false; + tap.start_pending = false; tap.blocks = 0; tap.peak_left = 0.0; tap.peak_right = 0.0; @@ -1751,9 +1785,16 @@ impl ObservationLane { if !tap.armed { return; } - // `first_sample` is **not** taken from the block. It was set when the tap was armed and it - // is set again to `end_sample` when a window closes, so consecutive windows tile with no - // gap as a property of this type rather than of whatever the caller happens to pass. + // A tap armed at preparation opens its first window here, at the first block it folds + // (#1584 D6a). Every other window's start is set below. + if tap.start_pending { + tap.start_pending = false; + tap.first_sample = first_sample; + } + // `first_sample` is otherwise **not** taken from the block. It was set when the tap was + // armed and it is set again to `end_sample` when a window closes, so consecutive windows + // tile with no gap as a property of this type rather than of whatever the caller happens + // to pass. tap.end_sample = first_sample.saturating_add(frames); match tap.fold { // `max(|x|)` is what turns an effect's own negative-for-reduction convention into the diff --git a/crates/host-core/src/lib.rs b/crates/host-core/src/lib.rs index f91ac1aaf..67b2e3d80 100644 --- a/crates/host-core/src/lib.rs +++ b/crates/host-core/src/lib.rs @@ -145,12 +145,13 @@ pub use live_delta::{take_fast_classification_count, take_lowered_owner_count}; pub use prepare::{ HostLiveControlHandles, HostLiveControlRequest, HostLiveLanes, HostMeterRequest, HostPrepareCaps, HostPrepareReport, HostShapePolicy, LiveRampRow, LiveRamps, - MAXIMUM_VCA_REACH_PAIRS, MAXIMUM_VCAS_BOUNDED, MonitorBase, PlanStateInventory, PreparedHost, - SOURCE_STALL_TOLERANCE_MS, SourceInventoryRow, SuccessorBase, check_counts, compile_host_model, - compile_host_session, count_effects, default_source_ring_frames, effect_producer_position, - pair_carried_meters, pair_carried_observations, parse_host_session, prepare_host_runtime, - prepare_host_runtime_between_render_calls, prepare_host_runtime_successor, - prepare_host_runtime_with_live_controls, prepare_host_runtime_with_live_controls_and_spectrum, + MAXIMUM_VCA_REACH_PAIRS, MAXIMUM_VCAS_BOUNDED, MonitorBase, NO_OBSERVATION, PlanStateInventory, + PreparedHost, SOURCE_STALL_TOLERANCE_MS, SourceInventoryRow, SuccessorBase, check_counts, + compile_host_model, compile_host_session, count_effects, default_source_ring_frames, + effect_producer_position, pair_carried_meters, pair_carried_observations, parse_host_session, + prepare_host_runtime, prepare_host_runtime_between_render_calls, + prepare_host_runtime_successor, prepare_host_runtime_with_live_controls, + prepare_host_runtime_with_live_controls_and_spectrum, prepare_host_runtime_with_live_controls_and_spectrum_collection, prepare_host_runtime_with_live_controls_and_spectrum_collection_successor, prepare_host_runtime_with_live_controls_and_spectrum_successor, @@ -190,8 +191,8 @@ pub use spectrum::{ SpectrumCaptureError, SpectrumCaptureReadError, SpectrumCaptureRequest, SpectrumCaptureResources, SpectrumChannels, SpectrumContinuousCaptureError, SpectrumContinuousReadError, SpectrumContinuousWindow, SpectrumHop, SpectrumOutput, - SpectrumPrepareError, SpectrumSmoothingConfig, SpectrumSmoothingConfigError, SpectrumTarget, - SpectrumTargetRef, SpectrumWindow, spectrum_analysis_history_resources, + SpectrumPair, SpectrumPrepareError, SpectrumSmoothingConfig, SpectrumSmoothingConfigError, + SpectrumTarget, SpectrumTargetRef, SpectrumWindow, spectrum_analysis_history_resources, spectrum_capture_collection_resources, spectrum_capture_resources, spectrum_capture_resources_for, }; diff --git a/crates/host-core/src/prepare.rs b/crates/host-core/src/prepare.rs index 792f5b92f..a58afd983 100644 --- a/crates/host-core/src/prepare.rs +++ b/crates/host-core/src/prepare.rs @@ -709,6 +709,19 @@ pub struct PlanStateInventory { /// observation readers ([`HostLiveControlHandles::effect_observations`]), so a row's `entry` /// is its position (#1327 D1). observations: Box<[ObservationInventoryRow]>, + /// The session rate and render quantum the plan runs at: a spectrum capture carries only + /// between plans of equal shape (#1395 D1). + sample_rate_hz: u32, + quantum_frames: u32, + /// One row per spectrum capture, in the order of the plan's captures, so a row's `entry` is + /// its position (#1395 D1): a single capture's is 0, a collection's is its entry index. + spectrum: Box<[SpectrumInventoryRow]>, + /// Whether the captures are a collection (#1395b). A single capture and a collection never + /// carry into each other: the control side holds one or the other, so a plan whose capture + /// kind changed starts its captures fresh. + spectrum_collection: bool, + /// The captures this plan takes over from its predecessor (#1395 D3), built at preparation. + spectrum_pairing: Box<[crate::spectrum::SpectrumPair]>, /// One row per compensation line of the plan (#1283 D1), sorted by edge ID. delay_lines: Box<[graph::GraphDelayLine]>, /// One row per strip input alignment line of the plan (#1284 D1), sorted by strip ID. @@ -777,6 +790,22 @@ pub(crate) struct ObservationInventoryRow { pub(crate) taps: usize, } +/// One spectrum capture of a prepared plan, owner key `(target, channels)`, as its successor's +/// preparation sees it (#1395 D1). The observer's binding handle is positional, so it is not part +/// of the key. +#[derive(Clone, Debug, Eq, PartialEq)] +pub(crate) struct SpectrumInventoryRow { + /// The boundary the capture observes. + pub(crate) target: crate::spectrum::SpectrumTarget, + /// The planes the capture copies. + pub(crate) channels: crate::spectrum::SpectrumChannels, + /// The capture's index in the plan's captures. + pub(crate) entry: usize, + /// Where the capture's observer sits in this plan. Meaningful only to this plan's successor's + /// carry program, which names it as the observer to take. + pub(crate) location: graph::GraphObserverLocation, +} + /// A meter's [`MeterConfig`] as plain comparable data: equal exactly when the configurations are /// equal word for word (#1327 D1), which `MeterConfig` itself cannot say because it holds a float. #[derive(Clone, Copy, Debug, Eq, Ord, PartialEq, PartialOrd)] @@ -880,6 +909,20 @@ impl PlanStateInventory { self.observations.len() } + /// The captures this plan takes over from the plan it was prepared to displace (#1395 D1, + /// D3): the pairs the control side applies at promotion with + /// [`SpectrumService::adopt_successor`](crate::SpectrumService::adopt_successor), when the + /// adoption carried. Built at preparation from each capture's target and channels, never from + /// positions; it names exactly the observers the plan's swap block moves, so the swap block + /// pairs nothing itself. Empty for a plan prepared without a predecessor, one whose + /// predecessor has no equal capture, one of another sample rate or quantum, and one whose capture + /// kind (single or collection) differs from its predecessor's. A collection's pairs are in the + /// successor's entry order. + #[must_use] + pub fn spectrum_pairing(&self) -> &[crate::spectrum::SpectrumPair] { + &self.spectrum_pairing + } + /// Compensation line rows, one per delayed edge of the plan (#1283). #[must_use] pub fn delay_line_count(&self) -> usize { @@ -951,6 +994,11 @@ impl PlanStateInventory { .map(|row| row.id.len() + row.effect_id.len()) .sum::() + self.meters.iter().map(|row| row.id.len()).sum::() + + self + .spectrum + .iter() + .map(|row| row.target.id().len()) + .sum::() + self .delay_lines .iter() @@ -974,6 +1022,8 @@ impl PlanStateInventory { + core::mem::size_of_val::<[EffectLaneInventoryRow]>(&self.effect_lanes) + core::mem::size_of_val::<[MeterInventoryRow]>(&self.meters) + core::mem::size_of_val::<[ObservationInventoryRow]>(&self.observations) + + core::mem::size_of_val::<[SpectrumInventoryRow]>(&self.spectrum) + + core::mem::size_of_val::<[crate::spectrum::SpectrumPair]>(&self.spectrum_pairing) + core::mem::size_of_val::<[graph::GraphDelayLine]>(&self.delay_lines) + core::mem::size_of_val::<[graph::GraphStripDelayLine]>(&self.strip_delays) + core::mem::size_of_val::<[graph::GraphLiveRoute]>(&self.live_routes) @@ -1021,8 +1071,18 @@ pub struct SuccessorBase<'a> { /// The monitor overlay of the predecessor's base and of this successor (decision 17, C8, /// #1559 D9). [`MonitorBase::NONE`] for a host without a solo. pub monitor: MonitorBase<'a>, + /// The taps the host has armed, as the successor renders under them (#1584 D6a): the + /// predecessor's bindings after the transaction, so a binding whose effect the transaction + /// removed is already gone. Each successor tap named here is armed at preparation and opens + /// its first window at its first rendered block; a tap that carries from the predecessor + /// takes the predecessor's own state instead, and is never restarted by this. + /// [`NO_OBSERVATION`] for a host without subscriptions. + pub observation: &'a controls::ObservationBindings, } +/// The bindings that arm nothing, for a [`SuccessorBase`] of a host without subscriptions. +pub static NO_OBSERVATION: controls::ObservationBindings = controls::ObservationBindings::new(); + /// The host-local monitor values (solo) a successor is prepared under (decision 17, C8, #1559). /// /// The successor bakes `controls::effective(model, next)`, so it renders the solo from its first @@ -2022,11 +2082,39 @@ fn prepare_host_runtime_with_live_controls_policy_and_spectrum( ), _ => false, }; + // Issue #1395 D1: a spectrum capture carries when its target and channels are the + // predecessor's and the two plans run at one rate and quantum; as for meters, this only decides + // that a program is needed. + let spectrum_may_carry = match (successor, &spectrum_request) { + (Some(base), Some(SpectrumPreparationRequest::Single(request))) => { + base.inventory.sample_rate_hz == compiled.sample_rate().0 + && base.inventory.quantum_frames == compiled.quantum().0 + && !base.inventory.spectrum_collection + && base + .inventory + .spectrum + .iter() + .any(|row| row.target == request.target && row.channels == request.channels) + } + (Some(base), Some(SpectrumPreparationRequest::Collection(request))) => { + base.inventory.sample_rate_hz == compiled.sample_rate().0 + && base.inventory.quantum_frames == compiled.quantum().0 + && base.inventory.spectrum_collection + && request.entries.iter().any(|entry| { + base.inventory + .spectrum + .iter() + .any(|row| row.target == entry.target && row.channels == entry.channels) + }) + } + _ => false, + }; let carry_program = match successor { Some(base) if !carried_sources.is_empty() || !carried_inputs.is_empty() || meters_may_carry + || spectrum_may_carry || strip_joins.iter().any(StripJoin::may_carry) || chain_joins.iter().any(ChainJoin::may_carry) => { @@ -2178,8 +2266,23 @@ fn prepare_host_runtime_with_live_controls_policy_and_spectrum( let _ = taps; return Err(shape("host.observation.live_controls")); } - taps => attach_effect_observation(&mut effects, taps, observation_window_blocks) - .map_err(effect_diagnostics)?, + taps => { + // #1584 D6a: the taps the host armed before this rebuild are armed in the successor. + let arms: Vec> = successor + .map(|base| base.observation.bindings()) + .unwrap_or_default() + .iter() + .map(|binding| effect_compiler::ObservationArm { + strip_id: binding.key.strip.as_str(), + console: binding.key.section.is_console(), + effect_id: binding.key.section.id().as_str(), + tap_id: binding.key.tap_id, + window_blocks: binding.window_blocks, + }) + .collect(); + attach_effect_observation(&mut effects, taps, observation_window_blocks, &arms) + .map_err(effect_diagnostics)? + } }; // Issue #137 D1/D2: the live-control requests are derived here, once, from the canonical strip @@ -2654,14 +2757,14 @@ fn prepare_host_runtime_with_live_controls_policy_and_spectrum( } // Issue #1327 D1: this plan's own meters, in reader order: the inventory its successor joins // with, and the rows the join below looks this plan's observers up in. + let observer_rows = + graph::observer_rows(&mut plan).ok_or_else(|| graph_failure("host.inventory.identity"))?; let meter_rows: Vec = { - let locations: BTreeMap = - graph::observer_rows(&mut plan) - .ok_or_else(|| graph_failure("host.inventory.identity"))? - .into_iter() - .filter(|row| row.kind == engine::realtime::ObserverKind::Meter) - .map(|row| (row.handle, row.location)) - .collect(); + let locations: BTreeMap = observer_rows + .iter() + .filter(|row| row.kind == engine::realtime::ObserverKind::Meter) + .map(|row| (row.handle, row.location)) + .collect(); let requested: BTreeMap = meter_requests .iter() .map(|item| (item.request.handle.0.get(), item)) @@ -2692,6 +2795,56 @@ fn prepare_host_runtime_with_live_controls_policy_and_spectrum( Some(base) => resolve_meter_carry(&base.inventory.meters, &meter_rows), None => MeterCarry::default(), }; + // Issue #1395 D1: this plan's own spectrum captures, in capture order: the inventory its + // successor joins with, and the rows the join below looks this plan's observers up in. A + // single capture is entry 0; a collection's entries are its entries in order. + let spectrum_locations: BTreeMap = observer_rows + .iter() + .filter(|row| row.kind == engine::realtime::ObserverKind::Spectrum) + .map(|row| (row.handle, row.location)) + .collect(); + let spectrum_row = |entry: usize, + target: &crate::spectrum::SpectrumTarget, + channels: crate::spectrum::SpectrumChannels| + -> Result { + let location = crate::spectrum::observer_handle(entry) + .and_then(|handle| spectrum_locations.get(&handle).copied()) + .ok_or_else(|| graph_failure("host.inventory.identity"))?; + Ok(SpectrumInventoryRow { + target: target.clone(), + channels, + entry, + location, + }) + }; + let spectrum_rows: Vec = match &spectrum_capture { + Some(PreparedSpectrumCapture::Single(capture)) => { + vec![spectrum_row(0, capture.target(), capture.channels())?] + } + Some(PreparedSpectrumCapture::Collection(collection)) => (0..collection.len()) + .map(|entry| { + let (target, channels) = collection + .entry(entry) + .ok_or_else(|| graph_failure("host.inventory.identity"))?; + spectrum_row(entry, target, channels) + }) + .collect::>()?, + None => Vec::new(), + }; + let spectrum_collection = matches!( + &spectrum_capture, + Some(PreparedSpectrumCapture::Collection(_)) + ); + let spectrum_carry = match successor { + Some(base) => resolve_spectrum_carry( + base.inventory, + &spectrum_rows, + spectrum_collection, + compiled.sample_rate().0, + compiled.quantum().0, + ), + None => SpectrumCarry::default(), + }; // Issue #1277 D1: this plan's own fader and matrix lanes, sorted by strip ID and owner: the // inventory its successor joins with, and the table the join below looks this plan's lanes up // in. @@ -2796,7 +2949,7 @@ fn prepare_host_runtime_with_live_controls_policy_and_spectrum( None => Vec::new(), }; let meter_carry_bytes = graph::observer_carry_retained_bytes( - meter_carry.moves.len(), + meter_carry.moves.len() + spectrum_carry.moves.len(), meter_carry.joins.len(), observation_carry.len(), ) @@ -2871,8 +3024,14 @@ fn prepare_host_runtime_with_live_controls_policy_and_spectrum( graph::install_effect_carry(&mut plan, effect_carry.moves, effect_carry_scratch_bytes) .map_err(|_| graph_failure("host.carry.install"))?; } - if !meter_carry.moves.is_empty() || !meter_carry.joins.is_empty() { - graph::install_observer_carry(&mut plan, meter_carry.moves, meter_carry.joins) + if !meter_carry.moves.is_empty() + || !meter_carry.joins.is_empty() + || !spectrum_carry.moves.is_empty() + { + // One section holds every kind's moves (#1327 D2, #1395 D2); the meters' come first. + let mut moves = meter_carry.moves; + moves.extend(spectrum_carry.moves); + graph::install_observer_carry(&mut plan, moves, meter_carry.joins) .map_err(|_| graph_failure("host.carry.install"))?; } if !observation_carry.is_empty() { @@ -2936,6 +3095,11 @@ fn prepare_host_runtime_with_live_controls_policy_and_spectrum( effect_lanes: effect_lanes.into_boxed_slice(), meters: meter_rows.into_boxed_slice(), observations: observation_rows.into_boxed_slice(), + sample_rate_hz: compiled.sample_rate().0, + quantum_frames: compiled.quantum().0, + spectrum: spectrum_rows.into_boxed_slice(), + spectrum_collection, + spectrum_pairing: spectrum_carry.pairing.into_boxed_slice(), delay_lines: delay_lines.into_boxed_slice(), strip_delays: strip_delays.into_boxed_slice(), live_routes: live_routes.into_boxed_slice(), @@ -3196,6 +3360,65 @@ fn resolve_meter_carry(predecessor: &[MeterInventoryRow], own: &[MeterInventoryR carry } +/// What a successor's preparation takes from the spectrum captures of its predecessor (#1395). +#[derive(Default)] +struct SpectrumCarry { + /// The observer moves the successor's swap block runs. + moves: Vec, + /// The same pairs as the control side applies them at promotion (D3): the swap block's record + /// and this list name one set of entries. + pairing: Vec, +} + +/// Join the successor's spectrum rows `own` with the predecessor's (#1395 D1). +/// +/// A capture carries when its target and channels equal a predecessor capture's, both plans run at +/// one sample rate and quantum, and both hold a single capture or both a collection (the control +/// side holds one kind and pairs within it); nothing else matters, and nothing in it is a live value, so +/// there is no retarget. Matching is by key, never by entry or handle. Every other capture is +/// fresh. +fn resolve_spectrum_carry( + predecessor: &PlanStateInventory, + own: &[SpectrumInventoryRow], + own_is_collection: bool, + sample_rate_hz: u32, + quantum_frames: u32, +) -> SpectrumCarry { + let mut carry = SpectrumCarry::default(); + if predecessor.sample_rate_hz != sample_rate_hz + || predecessor.quantum_frames != quantum_frames + || predecessor.spectrum_collection != own_is_collection + { + return carry; + } + let mut taken = vec![false; predecessor.spectrum.len()]; + for row in own { + let found = predecessor + .spectrum + .iter() + .enumerate() + .find(|(index, before)| { + !taken[*index] && before.target == row.target && before.channels == row.channels + }); + let Some((index, before)) = found else { + continue; + }; + taken[index] = true; + carry.moves.push(graph::GraphObserverMove { + kind: engine::realtime::ObserverKind::Spectrum, + successor: row.location, + predecessor: before.location, + successor_entry: row.entry, + predecessor_entry: before.entry, + }); + carry.pairing.push(crate::spectrum::SpectrumPair { + successor: row.entry, + predecessor: before.entry, + }); + } + carry +} + /// Join the successor's effect observation rows `own` with the predecessor's (#1327 D1): a tap /// set carries when its effect instance carried (`moves` names the successor instance's site) and /// both instances declare as many taps, with a row on each side. Every other instance's taps start @@ -5352,6 +5575,11 @@ mod tests { effect_lanes: Box::new([row]), meters: Box::default(), observations: Box::default(), + sample_rate_hz: 48_000, + quantum_frames: 128, + spectrum: Box::default(), + spectrum_collection: false, + spectrum_pairing: Box::default(), delay_lines: Box::default(), strip_delays: Box::default(), live_routes: Box::default(), @@ -5359,6 +5587,131 @@ mod tests { } } + fn spectrum_row( + target: crate::spectrum::SpectrumTarget, + channels: crate::spectrum::SpectrumChannels, + entry: usize, + unit: u32, + ) -> super::SpectrumInventoryRow { + super::SpectrumInventoryRow { + target, + channels, + entry, + location: graph::GraphObserverLocation { + unit, + member: graph::GraphObserverLocation::NO_MEMBER, + index: 0, + }, + } + } + + /// #1395 D1: a capture carries on an equal target and channel mask at an equal rate and + /// quantum, found by key and not by location; any other difference starts it fresh. + #[test] + fn a_spectrum_capture_carries_by_key_at_one_shape() { + use crate::spectrum::{SpectrumChannels, SpectrumTarget}; + let target = || SpectrumTarget::TrackPostMatrix("eq0".into()); + let mut before = inventory_with(lane_row(1, layout())); + before.spectrum = Box::new([spectrum_row(target(), SpectrumChannels::Stereo, 0, 7)]); + let own = [spectrum_row(target(), SpectrumChannels::Stereo, 0, 2)]; + let carry = super::resolve_spectrum_carry(&before, &own, false, 48_000, 128); + assert_eq!(carry.moves.len(), 1); + assert_eq!( + carry.moves[0].predecessor.unit, 7, + "named by its own location" + ); + assert_eq!(carry.moves[0].successor.unit, 2); + assert_eq!(carry.pairing.len(), 1); + for (label, own, rate, quantum) in [ + ( + "channels", + [spectrum_row(target(), SpectrumChannels::Left, 0, 2)], + 48_000, + 128, + ), + ( + "target", + [spectrum_row( + SpectrumTarget::TrackPostInputBuiltins("eq0".into()), + SpectrumChannels::Stereo, + 0, + 2, + )], + 48_000, + 128, + ), + ( + "rate", + [spectrum_row(target(), SpectrumChannels::Stereo, 0, 2)], + 96_000, + 128, + ), + ( + "quantum", + [spectrum_row(target(), SpectrumChannels::Stereo, 0, 2)], + 48_000, + 256, + ), + ] { + let carry = super::resolve_spectrum_carry(&before, &own, false, rate, quantum); + assert!( + carry.moves.is_empty() && carry.pairing.is_empty(), + "a changed {label} starts fresh" + ); + } + } + + /// #1395b D1: a collection's entries are matched by key in any order, an entry whose key is + /// absent from the predecessor is fresh, and a single capture and a collection never carry + /// into each other. + #[test] + fn a_spectrum_collection_carries_by_key_in_any_order() { + use crate::spectrum::{SpectrumChannels, SpectrumTarget}; + let t = || SpectrumTarget::TrackPostMatrix("t".into()); + let u = || SpectrumTarget::TrackPostInputBuiltins("u".into()); + let mut before = inventory_with(lane_row(1, layout())); + before.spectrum_collection = true; + before.spectrum = Box::new([ + spectrum_row(t(), SpectrumChannels::Stereo, 0, 10), + spectrum_row(u(), SpectrumChannels::Left, 1, 11), + ]); + // Reordered, with a changed key and an added entry. + let own = [ + spectrum_row(u(), SpectrumChannels::Left, 0, 20), + spectrum_row(t(), SpectrumChannels::Stereo, 1, 21), + spectrum_row(u(), SpectrumChannels::Right, 2, 22), + ]; + let carry = super::resolve_spectrum_carry(&before, &own, true, 48_000, 128); + let pairs: Vec<(usize, usize)> = carry + .pairing + .iter() + .map(|pair| (pair.successor, pair.predecessor)) + .collect(); + assert_eq!(pairs, vec![(0, 1), (1, 0)], "by key, not by position"); + let moves: Vec<(u32, u32)> = carry + .moves + .iter() + .map(|step| (step.successor.unit, step.predecessor.unit)) + .collect(); + assert_eq!( + moves, + vec![(20, 11), (21, 10)], + "named by their own locations" + ); + let none = super::resolve_spectrum_carry(&before, &own, false, 48_000, 128); + assert!( + none.moves.is_empty() && none.pairing.is_empty(), + "a collection does not carry into a single capture" + ); + let mut single = inventory_with(lane_row(1, layout())); + single.spectrum = Box::new([spectrum_row(t(), SpectrumChannels::Stereo, 0, 10)]); + let none = super::resolve_spectrum_carry(&single, &own, true, 48_000, 128); + assert!( + none.moves.is_empty() && none.pairing.is_empty(), + "a single capture does not carry into a collection" + ); + } + /// D1: the slot's model values are equal and both plans bank it, but the predecessor's lane has /// another prepared layout (state sizes, state layout version, prepared bypass or latency). /// The lane is not moved and its strip restarts. A join that trusts the existence of two rows diff --git a/crates/host-core/src/spectrum.rs b/crates/host-core/src/spectrum.rs index 1b07d2429..6dec96183 100644 --- a/crates/host-core/src/spectrum.rs +++ b/crates/host-core/src/spectrum.rs @@ -13,7 +13,8 @@ use std::sync::{ }; use engine::realtime::{ - Consumer, Producer, QueueGeneration, RenderError, bounded_spsc, bounded_spsc_retained_payload, + Consumer, ObserverKind, Producer, QueueGeneration, RenderError, bounded_spsc, + bounded_spsc_retained_payload, }; use graph::{ GraphNodeId, GraphNodeObserverBinding, GraphObservationBlock, GraphObservationValidity, @@ -228,7 +229,7 @@ impl SpectrumTarget { } } - fn id(&self) -> &str { + pub(crate) fn id(&self) -> &str { match self { Self::TrackPostInputBuiltins(track_id) | Self::TrackPostMatrix(track_id) @@ -957,6 +958,81 @@ impl SpectrumCapture { pub const fn channels(&self) -> SpectrumChannels { self.channels } + + /// Take over the capture of the plan this one's plan displaced (#1395 D3), for a swap in which + /// the observers exchanged places. + /// + /// `pairs` is the successor's pairing list ([`PlanStateInventory::spectrum_pairing`]): when it + /// names this single capture as the continuation of `predecessor` (entry 0 of both), the two + /// captures exchange places and `true` is returned. `self` then holds the consumer, the three + /// shared words and the read counters that belong to the observer that now renders in the + /// successor's plan, so a reader that held `predecessor` keeps receiving with no `Failed` + /// epoch, no `Gap` and no restart; `predecessor` holds the successor's fresh capture, whose + /// observer sits in the displaced plan. Anything else changes nothing and returns `false`: an + /// empty list, and captures whose target or channels differ (the key of the carry, so a list + /// that names such a pair is a fault that must not pair readers). + /// + /// Allocation-free and drops nothing: the caller drops the displaced capture where it chooses. + /// + /// [`PlanStateInventory::spectrum_pairing`]: crate::PlanStateInventory::spectrum_pairing + pub fn pair_carried(&mut self, predecessor: &mut Self, pairs: &[SpectrumPair]) -> bool { + let named = pairs.contains(&SpectrumPair { + successor: 0, + predecessor: 0, + }); + if !named || self.target != predecessor.target || self.channels != predecessor.channels { + return false; + } + core::mem::swap(self, predecessor); + true + } + + /// Settle the one-shot capture of a displaced plan that did not carry (#1618, option A): when + /// `predecessor` was armed, capturing, or held a window or a result, `self` (the successor's + /// fresh capture) starts in `INVALID` at the predecessor's arm generation, so the reader that + /// armed it reads [`SpectrumCaptureReadError::Invalid`] once and can arm again. Returns whether + /// it settled anything. + /// + /// The window the predecessor held, and the result it queued, lived in an observer that no + /// longer renders, so no later block can complete the arm. A predecessor with no one-shot in + /// flight (idle, or running a stream) leaves `self` untouched. The generation carries over, so + /// the next arm is `g + 1` and a record of an earlier arm can never be mistaken for its + /// window. + /// + /// Call it only for a capture that [`Self::pair_carried`] did not exchange: a carried capture + /// keeps its one-shot state. Control-side, allocation-free and free-free. It is one atomic + /// exchange on a word that the successor's observer treats as "no window" (that observer moves + /// the word only from `ARMED` or `CAPTURING`), so it is safe while the successor's plan + /// renders. + pub fn settle_unpaired(&mut self, predecessor: &Self) -> bool { + if predecessor.mode.load(Ordering::Acquire) != ONE_SHOT_MODE { + return false; + } + let word = predecessor.state.load(Ordering::Acquire); + if word_phase(word) == IDLE { + return false; + } + self.state + .swap(state_word(word_generation(word), INVALID), Ordering::AcqRel); + true + } +} + +/// One spectrum capture a successor takes from its predecessor (#1395 D1): the successor's +/// capture at entry `successor` continues the predecessor's at entry `predecessor`. An entry is a +/// capture's index in its plan's captures (a single capture is entry 0). +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub struct SpectrumPair { + /// The entry in the successor's captures. + pub successor: usize, + /// The entry in the predecessor's captures. + pub predecessor: usize, +} + +/// The binding handle of the observer of capture `index`: handles are positional, so they are not +/// keys (#1395 D1). +pub(crate) fn observer_handle(index: usize) -> Option { + SPECTRUM_OBSERVER_HANDLE.checked_sub(u64::try_from(index).ok()?) } /// Control-side ownership of several explicitly prepared spectrum captures. @@ -991,6 +1067,70 @@ impl SpectrumCaptureCollection { self.captures.is_empty() } + /// Take over the captures of the collection the plan of this one displaced (#1395 D3), for a + /// swap in which the observers exchanged places. Returns whether any capture paired. + /// + /// `pairs` is the successor's pairing list ([`PlanStateInventory::spectrum_pairing`]): the + /// capture at `successor` continues the one at `predecessor`, and the two exchange places, so + /// each paired entry of `self` holds the consumer, the three shared words and the read counters + /// of the observer that now renders in the successor's plan. A reader that held the + /// predecessor's entry keeps receiving, whatever index the entry has now; `predecessor` is left + /// holding the successor's fresh captures. The list is checked whole before anything moves: an + /// index out of range, a pair whose target or channels differ (the carry's key), or successor + /// entries not strictly increasing make it a fault that pairs nothing, as an empty list does. + /// (Two pairs cannot then name one predecessor entry: both successor entries would carry its + /// key, and a collection admits no duplicate key.) + /// + /// The selection follows the entry. When the predecessor's selected entry paired, `self` + /// selects that entry's new index and keeps the predecessor's `selection_epoch`. When it did + /// not, `self` has no selection and its epoch is the predecessor's plus one, so a reader of the + /// epoch sees the loss; the host's next `select` arms an entry as it does today. A predecessor + /// with no selection leaves `self` with none and with the predecessor's epoch. This runs for an + /// empty list too (a refused hand-over carries nothing). It does not settle the arm or the + /// stream the service held on a lost selection: [`crate::SpectrumService::adopt_successor`] does. + /// + /// Allocation-free and drops nothing: the caller drops the displaced collection where it + /// chooses. + /// + /// [`PlanStateInventory::spectrum_pairing`]: crate::PlanStateInventory::spectrum_pairing + pub fn pair_carried(&mut self, predecessor: &mut Self, pairs: &[SpectrumPair]) -> bool { + let mut previous: Option = None; + let sound = pairs.iter().all(|pair| { + let increasing = previous.is_none_or(|before| pair.successor > before); + previous = Some(pair.successor); + increasing + && match ( + self.captures.get(pair.successor), + predecessor.captures.get(pair.predecessor), + ) { + (Some(next), Some(before)) => { + next.target == before.target && next.channels == before.channels + } + _ => false, + } + }); + let pairs = if sound { pairs } else { &[] }; + let mut selected = None; + for pair in pairs { + core::mem::swap( + &mut self.captures[pair.successor], + &mut predecessor.captures[pair.predecessor], + ); + if predecessor.selected == Some(pair.predecessor) { + selected = Some(pair.successor); + } + } + self.selected = selected; + self.selection_epoch = if selected.is_some() || predecessor.selected.is_none() { + predecessor.selection_epoch + } else { + // At the top of the range `select` refuses with `EpochOverflow`, so no identity is + // ever reused. + predecessor.selection_epoch.saturating_add(1) + }; + !pairs.is_empty() + } + /// Return the accepted entry at `index`: its stable target identity and its mask. /// /// The target is borrowed, so this accessor clones and allocates nothing. The browser calls @@ -2543,6 +2683,13 @@ impl GraphRuntimeObserver for SpectrumCaptureObserver { fn block_end(&mut self) { SpectrumCaptureObserver::block_end(self); } + + /// A capture's whole observer, its window history, producer and shared words included, is + /// state a swap carries (#1395 D2). It keeps no binding handle, so a hand-over needs no + /// `adopt_binding_handle`. + fn carry_kind(&self) -> Option { + Some(ObserverKind::Spectrum) + } } /// Caller-owned output planes for one analyzed spectrum. @@ -3133,6 +3280,196 @@ mod tests { (observer, capture) } + /// #1395 D3: only a named pair of captures of one key exchange places; the successor's capture + /// then holds the state the predecessor's held, and nothing else swaps. + #[test] + fn pair_carried_swaps_only_a_named_pair_of_one_key() { + let pair = [super::SpectrumPair { + successor: 0, + predecessor: 0, + }]; + let started = |channels| { + let (_observer, mut capture) = continuous_pair(channels); + capture.start_continuous(48_000, 128).expect("start"); + capture + }; + // A named pair of one key: the successor's fresh capture takes the running stream. + let (_observer, mut next) = continuous_pair(SpectrumChannels::Stereo); + let mut before = started(SpectrumChannels::Stereo); + assert!(next.pair_carried(&mut before, &pair)); + assert_eq!(next.stream_epoch(), Some(1)); + assert_eq!(before.stream_epoch(), None); + // An empty list names nothing. + let (_observer, mut next) = continuous_pair(SpectrumChannels::Stereo); + let mut before = started(SpectrumChannels::Stereo); + assert!(!next.pair_carried(&mut before, &[])); + assert_eq!(next.stream_epoch(), None); + assert_eq!(before.stream_epoch(), Some(1)); + // A list that names captures of another channel mask is a fault, never a pairing. + let (_observer, mut next) = continuous_pair(SpectrumChannels::Left); + let mut before = started(SpectrumChannels::Stereo); + assert!(!next.pair_carried(&mut before, &pair)); + assert_eq!(before.channels(), SpectrumChannels::Stereo); + assert_eq!(before.stream_epoch(), Some(1)); + } + + /// Three captures of distinct keys with their observers, as a collection holds them. + fn collection_of( + keys: &[(&str, SpectrumChannels)], + ) -> (Vec, SpectrumCaptureCollection) { + let mut observers = Vec::new(); + let mut captures = Vec::new(); + for (id, channels) in keys { + let (observer, mut capture) = continuous_pair(*channels); + capture.target = SpectrumTarget::TrackPostMatrix((*id).into()); + observers.push(observer); + captures.push(capture); + } + (observers, SpectrumCaptureCollection::new(captures)) + } + + /// Run `frames` frames of the ramp through `observer` in quanta of 128, from `from_sample`. + fn feed_blocks(observer: &mut SpectrumCaptureObserver, from_sample: usize, frames: usize) { + for first_sample in (from_sample..from_sample + frames).step_by(128) { + let (left, right) = ramp_block(first_sample, 128); + observer.capture( + &left, + &right, + first_sample as u64, + super::GraphObservationValidity::CLEAR, + ); + } + } + + /// #1395b D3: a reordered collection pairs by key. The selection follows its entry to the new + /// index with its epoch, a stop reaches the observer that renders (the mode word moved with + /// the capture), the other entry arms its own observer, a lost selection advances the epoch, + /// and a list that names a pair of two keys is a fault that pairs nothing. + #[test] + fn collection_pair_carried_follows_keys_and_the_selection() { + use super::SpectrumPair; + let a_keys = [ + ("t", SpectrumChannels::Stereo), + ("u", SpectrumChannels::Left), + ]; + let b_keys = [ + ("u", SpectrumChannels::Left), + ("t", SpectrumChannels::Stereo), + ]; + let pairs = [ + SpectrumPair { + successor: 0, + predecessor: 1, + }, + SpectrumPair { + successor: 1, + predecessor: 0, + }, + ]; + // A selects `t` and streams; B lists the entries the other way round. + let (mut a_observers, mut a) = collection_of(&a_keys); + let (mut b_observers, mut b) = collection_of(&b_keys); + a.select( + SpectrumTarget::TrackPostMatrix("t".into()).as_ref(), + SpectrumChannels::Stereo, + ) + .expect("select t"); + a.start_continuous(48_000, 128).expect("start"); + feed_blocks(&mut a_observers[0], 0, 1024); + // The swap block: the observers of the keys exchange places. + core::mem::swap(&mut a_observers[0], &mut b_observers[1]); + core::mem::swap(&mut a_observers[1], &mut b_observers[0]); + assert!(b.pair_carried(&mut a, &pairs)); + assert_eq!(b.selected_index(), Some(1), "t is entry 1 now"); + assert_eq!(b.selection_epoch(), 1, "the epoch carried"); + assert_eq!(b.stream_epoch(), Some(1)); + // The stream goes on from the history the observer kept: the window completes after the + // 1024 frames already fed plus 1024 more, and carries the first epoch. + feed_blocks(&mut b_observers[1], 1024, 1024); + let window = b.try_read_continuous().expect("the carried window"); + assert_eq!(window.stream_epoch, 1); + assert_eq!(window.first_sample, 0); + // The observer still publishes at its hop (2048 frames here), so an empty queue after a stop is a stop. + feed_blocks(&mut b_observers[1], 2048, 2048); + assert_eq!(b.captures[1].consumer.available_at_entry(), 1); + b.try_read_continuous().expect("the next window"); + // A stop through the successor's collection reaches the observer that renders. + b.stop_continuous(); + feed_blocks(&mut b_observers[1], 4096, 4096); + assert_eq!( + b.captures[1].consumer.available_at_entry(), + 0, + "no window is published after the stop" + ); + // `u` was idle and carried idle: selecting it arms its own observer. + b.select( + SpectrumTarget::TrackPostMatrix("u".into()).as_ref(), + SpectrumChannels::Left, + ) + .expect("select u"); + assert_eq!(b.selected_index(), Some(0)); + assert_eq!(b.selection_epoch(), 2); + feed_blocks(&mut b_observers[0], 0, SPECTRUM_WINDOW_FRAMES); + b_observers[0].block_end(); + assert!(b.try_read().is_ok(), "u's observer completes its window"); + + // A lost selection: the selected key does not pair, so there is none and the epoch moves. + let (_o, mut a) = collection_of(&a_keys); + let (_p, mut b) = collection_of(&[("t", SpectrumChannels::Left), a_keys[1]]); + a.select( + SpectrumTarget::TrackPostMatrix("t".into()).as_ref(), + SpectrumChannels::Stereo, + ) + .expect("select t"); + let only_u = [SpectrumPair { + successor: 1, + predecessor: 1, + }]; + assert!(b.pair_carried(&mut a, &only_u)); + assert_eq!(b.selected_index(), None); + assert_eq!(b.selection_epoch(), 2); + assert_eq!(a.selected_index(), Some(0), "the predecessor keeps its own"); + // A predecessor with no selection leaves none and its epoch. + let (_o, mut a) = collection_of(&a_keys); + let (_p, mut b) = collection_of(&a_keys); + assert!(!b.pair_carried(&mut a, &[])); + assert_eq!((b.selected_index(), b.selection_epoch()), (None, 0)); + + // A pair across two keys, an index out of range and an out-of-order list pair nothing. + for fault in [ + vec![SpectrumPair { + successor: 0, + predecessor: 1, + }], + vec![SpectrumPair { + successor: 2, + predecessor: 0, + }], + vec![ + SpectrumPair { + successor: 1, + predecessor: 1, + }, + SpectrumPair { + successor: 0, + predecessor: 0, + }, + ], + ] { + let (_o, mut a) = collection_of(&a_keys); + let (_p, mut b) = collection_of(&a_keys); + a.select( + SpectrumTarget::TrackPostMatrix("t".into()).as_ref(), + SpectrumChannels::Stereo, + ) + .expect("select t"); + assert!(!b.pair_carried(&mut a, &fault), "{fault:?}"); + assert_eq!(a.selected_index(), Some(0), "{fault:?}"); + assert_eq!(b.selected_index(), None, "{fault:?}"); + assert_eq!(b.selection_epoch(), 2, "the selection is lost: {fault:?}"); + } + } + fn ramp_block(first_sample: usize, frames: usize) -> (Vec, Vec) { let left = (0..frames) .map(|offset| (first_sample + offset) as f32) @@ -3899,6 +4236,69 @@ mod tests { ); } + /// #1618 gate 2: a window held behind a stale record when the plan swaps, on a capture that + /// does not carry. The held window died with its observer, so no block end will ever publish + /// it: the service settles the arm, the read answers `CaptureInvalid` (not `Pending` forever), + /// the generation carries over, and a new arm completes on the successor's observer, whose + /// window a record of the old generation cannot be mistaken for. + #[test] + fn a_window_held_behind_a_stale_record_settles_invalid_when_the_plan_swaps() { + use crate::{PreparedSpectrumCaptures, SpectrumService, SpectrumServiceError}; + + let (mut observer, capture) = one_shot_pair(SpectrumChannels::Left); + let mut service = + SpectrumService::new(PreparedSpectrumCaptures::Single(capture), 48_000, 128, None); + service.arm().expect("first arm"); + service.cancel().expect("cancel"); + service.arm().expect("an arm after a cancel is never Busy"); + let stale = super::SpectrumCapturedRecord { + window: window(0.25), + stream_epoch: 0, + sequence: 0, + dropped_captures: 0, + arm_generation: 1, + }; + assert!(observer.producer.try_push(stale).is_ok()); + feed_window(&mut observer, 8_192, 16); + observer.block_end(); + assert!( + service + .read(SpectrumChannels::Left) + .expect("pending") + .is_none(), + "the window is held behind the stale record" + ); + + // The plan swaps; the successor's capture is fresh and its observer is its own. + let (mut next_observer, next) = one_shot_pair(SpectrumChannels::Left); + let displaced = + service.adopt_successor(PreparedSpectrumCaptures::Single(next), &[], true, None); + assert!( + matches!( + service.read(SpectrumChannels::Left), + Err(SpectrumServiceError::CaptureInvalid) + ), + "settled, not pending forever" + ); + // The refusal is read once; the generation is the old arm's, so the next arm is its + // successor and completes on the new observer. + assert!(matches!( + service.read(SpectrumChannels::Left), + Err(SpectrumServiceError::NotArmed) + )); + service.arm().expect("a new arm after the refusal"); + feed_window(&mut next_observer, 16_384, 16); + // The displaced observer keeps its held window; its later block ends reach nobody. + observer.block_end(); + next_observer.block_end(); + let snapshot = service + .read(SpectrumChannels::Left) + .expect("read") + .expect("the new arm completes on the successor"); + assert_eq!(snapshot.window.first_sample, 16_384); + drop(displaced); + } + /// Gate 1 (issue #1609): a cancel races the completion of the window it cancels. /// /// Per seed, a render thread drives the real observer through whole-window blocks, calling diff --git a/crates/host-core/src/spectrum_service.rs b/crates/host-core/src/spectrum_service.rs index f96dc9c58..a6af3f0f3 100644 --- a/crates/host-core/src/spectrum_service.rs +++ b/crates/host-core/src/spectrum_service.rs @@ -24,8 +24,8 @@ use crate::spectrum::{ SpectrumCaptureCollectionRequest, SpectrumCaptureCollectionSelectionError, SpectrumCaptureReadError, SpectrumCaptureRequest, SpectrumChannels, SpectrumContinuousCaptureError, SpectrumContinuousReadError, SpectrumContinuousWindow, - SpectrumHop, SpectrumOutput, SpectrumSmoothingConfig, SpectrumTarget, SpectrumTargetRef, - SpectrumWindow, + SpectrumHop, SpectrumOutput, SpectrumPair, SpectrumSmoothingConfig, SpectrumTarget, + SpectrumTargetRef, SpectrumWindow, }; /// Spectrum capture target at a track's `post_input` tap, after its input section. @@ -652,6 +652,88 @@ impl SpectrumService { self.hop } + /// Install the captures of a successor plan whose adoption block has run (#1395 D3, D5): the + /// service's captures become the successor's, and the displaced ones come back for the caller + /// to drop where it chooses. + /// + /// Control-side work, done at promotion and never in the swap block. `pairing` is the + /// successor's [`PlanStateInventory::spectrum_pairing`](crate::PlanStateInventory::spectrum_pairing), + /// and `carried` says whether the adoption carried (`CarryOutcome::Carried`); a refused + /// hand-over (`PredecessorMismatch`) moved no observer, so every successor capture is fresh. + /// A capture the pairing names keeps the reader the service already holds: that capture's + /// observer moved into the successor's plan, so its consumer, its shared mode word and its + /// read counters stay valid, and the stream epoch, the sequence and an armed one-shot carry on + /// unchanged (#1395 gate 3). Until this call, the service's captures are the predecessor's, + /// which is correct for a carried capture and reads as pending for the others. + /// + /// A collection pairs entry by entry (#1395b): each entry the pairing names keeps its reader, + /// whatever index it has in the successor, and the selection follows its entry with its epoch + /// ([`SpectrumCaptureCollection::pair_carried`]). When the selected entry did not carry, or the + /// hand-over was refused, the successor has no selection and its epoch is the predecessor's + /// plus one. A single capture and a collection never pair with each other (their keys never + /// join), so such a promotion installs the successor's fresh captures. + /// + /// A capture that does not carry is replaced by the successor's fresh one, and what the + /// caller had on the old one is settled here (#1618), so nothing waits on an observer that no + /// longer renders: + /// + /// - A one-shot arm, capturing or held (with or without a result queued), settles to + /// [`SpectrumServiceError::CaptureInvalid`] with its arm generation: the successor's capture + /// starts `INVALID` at the predecessor's generation ([`SpectrumCapture::settle_unpaired`]), + /// the caller's next [`Self::read`] returns the refusal once, and a new arm then completes on + /// the successor. The arm stays the caller's until that read. + /// - A collection whose selected entry did not carry has no selection (see above), so an arm + /// on it ends with the selection: the caller's arm is released and a read answers + /// [`SpectrumServiceError::NotArmed`]; the selection epoch tells the caller to select again. + /// - A running stream ends: [`Self::stream_active`] is `false`, the cadence, the smoothing and + /// the window at hand are gone, the metadata is `Stopped`, and `analysis` (the history's + /// owner, when the caller has one) ends with it. The caller starts a stream anew with + /// [`Self::start_stream`] (after [`Self::select`] on a collection); [`Self::select_stream`] + /// answers [`SpectrumServiceError::StreamInactive`] until then. + /// + /// A carried capture is untouched: its one-shot state and its stream run on (#1395 gate 3). + /// A caller that owns an analysis passes it, or the analysis keeps facts of a stream that + /// ended. + /// + /// Allocation-free and free-free: nothing is dropped here. + #[must_use = "the displaced captures hold a reader and a shared state; drop them off the audio thread"] + pub fn adopt_successor( + &mut self, + mut successor: PreparedSpectrumCaptures, + pairing: &[SpectrumPair], + carried: bool, + analysis: Option<&mut SpectrumAnalysis>, + ) -> PreparedSpectrumCaptures { + // `kept`: the capture the caller used carried, so its arm and stream run on. + // `settled`: the arm stands as an `INVALID` result for the caller's read. + let (kept, settled) = match (&mut successor, &mut self.captures) { + (PreparedSpectrumCaptures::Single(next), PreparedSpectrumCaptures::Single(before)) => { + if carried && next.pair_carried(before, pairing) { + (true, false) + } else { + (false, next.settle_unpaired(before)) + } + } + // A collection also runs without a carry: the selection the service held is gone with + // the predecessor's captures, and the successor's epoch must say so. + ( + PreparedSpectrumCaptures::Collection(next), + PreparedSpectrumCaptures::Collection(before), + ) => { + next.pair_carried(before, if carried { pairing } else { &[] }); + (next.selected_index().is_some(), false) + } + _ => (false, false), + }; + if !kept { + self.caller_armed &= settled; + if self.stream_active { + self.end_stream_state(analysis); + } + } + core::mem::replace(&mut self.captures, successor) + } + // REALTIME_POLICY_BEGIN: arm, read, cancel, select and the stream calls run inside render-locked exports. /// Arm the selected capture for its next complete window. A success makes the arm the /// caller's. @@ -1151,8 +1233,13 @@ impl SpectrumService { /// Stop the stream, discard its queued window and make the metadata `Stopped`. Allowed /// without a running stream. With `analysis`, the analysis of the stream ends at the same /// call (its history stays, the next stream resets it) and the metadata carries its facts. - pub fn stop_stream(&mut self, mut analysis: Option<&mut SpectrumAnalysis>) { + pub fn stop_stream(&mut self, analysis: Option<&mut SpectrumAnalysis>) { self.captures.stop_continuous(); + self.end_stream_state(analysis); + } + + /// End the stream's state in the service and in its analysis; the capture is the caller's. + fn end_stream_state(&mut self, mut analysis: Option<&mut SpectrumAnalysis>) { self.stream_active = false; self.stream_cadence = None; self.stream_smoothing = None; diff --git a/crates/host-core/tests/browser_successor.rs b/crates/host-core/tests/browser_successor.rs index 685383340..d97f09f01 100644 --- a/crates/host-core/tests/browser_successor.rs +++ b/crates/host-core/tests/browser_successor.rs @@ -37,7 +37,7 @@ use host_core::{ LiveValuesSuperseded, PlanStateInventory, PrepareDiagnostics, PreparedHost, SPECTRUM_WINDOW_FRAMES, SourceSubmission, SpectrumCapture, SpectrumCaptureCollection, SpectrumCaptureCollectionEntry, SpectrumCaptureCollectionRequest, SpectrumCaptureRequest, - SpectrumChannels, SpectrumTarget, SuccessorBase, compile_host_session, + SpectrumChannels, SpectrumPair, SpectrumTarget, SuccessorBase, compile_host_session, prepare_host_runtime_with_live_controls_and_spectrum, prepare_host_runtime_with_live_controls_and_spectrum_collection, prepare_host_runtime_with_live_controls_and_spectrum_collection_successor, @@ -181,7 +181,8 @@ fn live_controls(branch: Branch) -> HostLiveControlRequest { #[allow(dead_code)] // The handles stay alive with the plan; only the captures are read. enum Kept { None(HostLiveControlHandles), - Single(HostLiveControlHandles, SpectrumCapture), + /// The plan's handles, its capture and the pairs it takes over from its predecessor (#1395). + Single(HostLiveControlHandles, SpectrumCapture, Vec), Collection(HostLiveControlHandles, SpectrumCaptureCollection), } @@ -207,7 +208,8 @@ fn prepare( lanes, &superseded, )?; - (host, Kept::Single(handles, capture)) + let pairing = host.inventory.spectrum_pairing().to_vec(); + (host, Kept::Single(handles, capture, pairing)) } (Branch::Single, Some(base)) => { let (host, handles, capture) = @@ -220,7 +222,8 @@ fn prepare( &superseded, base, )?; - (host, Kept::Single(handles, capture)) + let pairing = host.inventory.spectrum_pairing().to_vec(); + (host, Kept::Single(handles, capture, pairing)) } (Branch::Collection, None) => { let (host, handles, capture) = @@ -284,6 +287,7 @@ fn base<'a>( revision: SUCCESSOR_REVISION, edit_ramps, monitor: host_core::MonitorBase::NONE, + observation: &host_core::NO_OBSERVATION, } } @@ -359,16 +363,21 @@ fn every_branch_swaps_in_without_a_gap() { } } -/// Gate 2. Red if the successor's capture is not wired to its plan's graph taps. +/// Gate 2. Red if the successor's plan does not run the capture's observer at its graph tap. #[test] fn the_successors_capture_completes_a_window() { let a = Session::compile(session_a()); let b = Session::compile(session_b()); - let (mut swapped, _a_kept, b_kept) = run(Branch::Single, &a, &b, Successor::Carry); - let Kept::Single(_handles, mut capture) = b_kept else { + let (mut swapped, a_kept, b_kept) = run(Branch::Single, &a, &b, Successor::Carry); + let (Kept::Single(_, mut previous, _), Kept::Single(_handles, mut capture, pairing)) = + (a_kept, b_kept) + else { panic!("a single capture"); }; - capture.arm().expect("arm the successor's capture"); + // The swap carried the predecessor's observer into this plan: the capture that reads it is + // the predecessor's, which the control side pairs at promotion (#1395 D3). + assert!(capture.pair_carried(&mut previous, &pairing)); + capture.arm().expect("arm the paired capture"); let rate = swapped.successor.sample_rate_hz; let plane = vec![0.25_f32; QUANTUM]; let window_blocks = SPECTRUM_WINDOW_FRAMES / QUANTUM; diff --git a/crates/host-core/tests/effect_observation.rs b/crates/host-core/tests/effect_observation.rs index e6b440828..7dba54952 100644 --- a/crates/host-core/tests/effect_observation.rs +++ b/crates/host-core/tests/effect_observation.rs @@ -527,12 +527,107 @@ fn the_first_window_at_or_after_applied_at_sample_reflects_the_command() { assert!(quiet.first_sample < applied_at); } -/// **E13** (D7): a plan replacement drops every subscription, and re-subscribing works. +/// A plan of `compiled` with live controls and observation capacity: fresh with no predecessor, or +/// the successor of `first`, swapped in at `first`'s next block, under the taps `bindings` names. +/// `committed` is the model `first` was prepared from. +fn prepare_successor( + compiled: &host_core::CompiledSession, + first: Option<(&mut Session, &session::SessionModel)>, + bindings: &controls::ObservationBindings, +) -> Session { + let request = live_controls(Leg::AllArmed); + let (mut prepared, handles) = match &first { + Some((first, committed)) => host_core::prepare_host_runtime_with_live_controls_successor( + compiled, + &caps(), + &request, + host_core::SuccessorBase { + inventory: &first.prepared.inventory, + committed, + revision: 2, + edit_ramps: &session::EditRamps::new(), + monitor: host_core::MonitorBase::NONE, + observation: bindings, + }, + ), + None => host_core::prepare_host_runtime_with_live_controls(compiled, &caps(), &request), + } + .unwrap_or_else(|failure| panic!("prepare: {failure:?}")); + let gate = prepared + .plan + .take_revision_gate() + .expect("the plan's gate handle"); + let mut block = 0; + if let Some((first, _)) = first { + assert_eq!( + prepared + .sources + .adopt_persisting(&mut first.prepared.sources), + 1 + ); + // The swap happens at the successor's first block: the exchange runs inside it. + assert_eq!( + prepared + .plan + .adopt_predecessor_plan(&mut first.prepared.plan), + engine::realtime::CarryOutcome::Carried + ); + block = first.block; + } + Session { + prepared, + gate, + handles, + block, + } +} + +/// The table of every observed tap 0 of `session`, each armed with [`WINDOW_BLOCKS`]. +fn bindings_of_all(session: &Session) -> controls::ObservationBindings { + let mut bindings = controls::ObservationBindings::new(); + for handle in &session.handles.effect_observations { + let effect = session::StableId::parse(&handle.effect_id).expect("a stable effect ID"); + let section = match handle.address.rack { + LiveEffectRack::Console => controls::TapSection::Console(effect), + LiveEffectRack::Inserts => controls::TapSection::Inserts(effect), + }; + bindings.set( + controls::TapKey { + strip: session::StableId::parse(&handle.track_id).expect("a stable strip ID"), + section, + tap_id: handle.descriptor.observations[0].id.0, + }, + WINDOW_BLOCKS, + ); + } + assert_eq!(bindings.len(), TRACKS); + bindings +} + +/// **E13** (D7, rewritten by #1584 D6a): a replacement plan prepared with the taps the host armed +/// keeps every subscription, and one prepared with none starts unarmed. +/// +/// The kept subscription opens its first window at the replacement's first block, so its windows +/// equal those of a fresh plan whose record arms the same tap at that block. A replacement prepared +/// from an empty table has the capacity, carries nothing and publishes nothing until the app +/// subscribes again, which works. /// -/// Red mutation: carry armed flags across the replacement -> the absence assertion fails. +/// Red mutation: prepare the replacement's taps unarmed whatever the table holds -> the kept +/// replacement publishes nothing. Carry armed flags into a replacement with an empty table -> the +/// unarmed assertions fail. #[test] -fn a_plan_replacement_drops_every_subscription() { - let mut first = prepare(Leg::AllArmed); +fn a_plan_replacement_keeps_the_subscriptions_the_host_armed() { + // The replacement changes the console slot's link mode, a prepared setup: every lane restarts, + // so nothing carries and the taps are the plan's own. + let committed = compile_host_session(SESSION, &caps()).expect("compile"); + let changed_document = SESSION.replacen( + "\"link_mode\": \"dual_mono\"", + "\"link_mode\": \"average\"", + 1, + ); + assert_ne!(changed_document, SESSION); + let changed = compile_host_session(&changed_document, &caps()).expect("compile"); + let mut first = prepare_successor(&committed, None, &controls::ObservationBindings::new()); subscribe_all(&mut first, true, WINDOW_BLOCKS); let _ = render(&mut first, WINDOW_BLOCKS as usize * 2); assert_eq!( @@ -542,33 +637,91 @@ fn a_plan_replacement_drops_every_subscription() { ); let before = reader(&first.handles, 3).read().expect("a window"); assert!(before.left > 0.0); + let bindings = bindings_of_all(&first); - // The structural edit: a replacement plan, prepared from the same session. Subscriptions are - // per-plan, so the new plan's lanes exist and are **unarmed** -- there is nowhere for an old - // subscription to have been carried to. - let mut second = prepare(Leg::AllArmed); + // Prepared with the table: every tap is armed before the plan renders, and no record was queued. + let mut kept = prepare_successor( + &changed, + Some((&mut first, committed.normalized_model())), + &bindings, + ); + assert_eq!( + kept.prepared.plan.observation_binding_counts(), + [TRACKS as u64, TRACKS as u64, TRACKS as u64], + "the replacement is armed at preparation" + ); + for producer in &kept.handles.effect_controls { + assert_eq!( + producer.observe_available_capacity(), + producer.observe_capacity(), + "the arm queued no record: a carried tap would be restarted by one" + ); + } + let opens_at = (kept.block * QUANTUM) as u64; + let _ = render(&mut kept, WINDOW_BLOCKS as usize); assert_eq!( - second.prepared.plan.observation_binding_counts(), + kept.prepared.plan.observation_binding_counts()[2], + TRACKS as u64, + "and it stays armed through the swap block" + ); + + // The reference: a fresh plan of the same session whose record arms the tap at its first + // block. The replacement's effects restarted, so both run from rest on the same input. + let mut reference = prepare_successor(&changed, None, &controls::ObservationBindings::new()); + subscribe_all(&mut reference, true, WINDOW_BLOCKS); + let _ = render(&mut reference, WINDOW_BLOCKS as usize); + for track in 0..TRACKS { + let window = reader(&kept.handles, track) + .read() + .unwrap_or_else(|| panic!("track {track} publishes on the replacement")); + let expected = reader(&reference.handles, track) + .read() + .expect("the reference publishes"); + assert_eq!( + window.first_sample, opens_at, + "the window opens at the replacement's first block" + ); + assert_eq!( + (window.sequence, window.blocks, window.left, window.right), + ( + expected.sequence, + expected.blocks, + expected.left, + expected.right + ), + "track {track}: the prepared arm equals a record at the first block" + ); + } + + // Prepared with an empty table, as `MonitorBase::NONE` and no subscription leave it: the + // capacity is there, and no subscription is. + let mut bare = prepare_successor( + &committed, + Some((&mut kept, changed.normalized_model())), + &controls::ObservationBindings::new(), + ); + assert_eq!( + bare.prepared.plan.observation_binding_counts(), [TRACKS as u64, TRACKS as u64, 0], - "the replacement plan carries capacity and no subscription" + "the replacement carries capacity and no subscription" ); - let _ = render(&mut second, WINDOW_BLOCKS as usize * 2); + let _ = render(&mut bare, WINDOW_BLOCKS as usize * 2); for track in 0..TRACKS { assert_eq!( - reader(&second.handles, track).read(), + reader(&bare.handles, track).read(), None, "track {track} published nothing against an unsubscribed replacement plan" ); } // Re-subscribing against the new plan works, and acks a fresh map: the sequence restarts at 1. - subscribe_all(&mut second, true, WINDOW_BLOCKS); + let resubscribed_at = subscribe_all(&mut bare, true, WINDOW_BLOCKS); // Exactly one window, so the sequence number is checked rather than merely bounded. - let _ = render(&mut second, WINDOW_BLOCKS as usize); - let after = reader(&second.handles, 3).read().expect("a window"); + let _ = render(&mut bare, WINDOW_BLOCKS as usize); + let after = reader(&bare.handles, 3).read().expect("a window"); assert_eq!(after.sequence, 1, "the replacement plan starts a fresh map"); - assert!( - after.first_sample >= before.end_sample, + assert_eq!( + after.first_sample, resubscribed_at, "and a fresh window, opened where the subscription was applied" ); assert!(after.left > 0.0, "and it publishes again"); diff --git a/crates/host-core/tests/successor_swap.rs b/crates/host-core/tests/successor_swap.rs index e7ceed7eb..66aceab6c 100644 --- a/crates/host-core/tests/successor_swap.rs +++ b/crates/host-core/tests/successor_swap.rs @@ -1015,6 +1015,25 @@ impl LivePlan { predecessor: &host_core::PlanStateInventory, committed: &SessionModel, backend: Backend, + ) -> Self { + Self::successor_observed( + session, + request, + predecessor, + committed, + &host_core::NO_OBSERVATION, + backend, + ) + } + + /// A successor prepared with the taps the host armed (#1584 D6a). + fn successor_observed( + session: &Session, + request: &host_core::HostLiveControlRequest, + predecessor: &host_core::PlanStateInventory, + committed: &SessionModel, + observation: &controls::ObservationBindings, + backend: Backend, ) -> Self { let (prepared, handles) = host_core::test_only_prepare_host_runtime_with_live_controls_successor_on( @@ -1027,6 +1046,7 @@ impl LivePlan { revision: successor::SUCCESSOR_REVISION, edit_ramps: &EditRamps::new(), monitor: host_core::MonitorBase::NONE, + observation, }, backend, ) @@ -2646,6 +2666,7 @@ impl CellPlan { revision: successor::SUCCESSOR_REVISION, edit_ramps, monitor: host_core::MonitorBase::NONE, + observation: &host_core::NO_OBSERVATION, }, backend, form, @@ -3908,6 +3929,7 @@ fn a_successor_revision_its_gate_cannot_stamp_is_refused() { revision, edit_ramps: &EditRamps::new(), monitor: host_core::MonitorBase::NONE, + observation: &host_core::NO_OBSERVATION, }, backend, Form::Split, @@ -3931,6 +3953,7 @@ fn a_successor_revision_its_gate_cannot_stamp_is_refused() { revision: accepted, edit_ramps: &EditRamps::new(), monitor: host_core::MonitorBase::NONE, + observation: &host_core::NO_OBSERVATION, }, backend, Form::Split, @@ -3960,6 +3983,7 @@ fn successor_handle( revision: successor::SUCCESSOR_REVISION, edit_ramps: &EditRamps::new(), monitor: host_core::MonitorBase::NONE, + observation: &host_core::NO_OBSERVATION, }, backend, ) @@ -7138,6 +7162,7 @@ mod meter_carry { revision: SUCCESSOR_REVISION, edit_ramps: &session::EditRamps::new(), monitor: MonitorBase::NONE, + observation: &host_core::NO_OBSERVATION, }), backend, ); @@ -7571,10 +7596,9 @@ mod observation_carry { continue; }; let stream = streams.entry(handle.track_id.to_string()).or_default(); - if stream - .last() - .is_none_or(|last| last.sequence != window.sequence) - { + // A window is new when its sequence or its start differs: a restarted tap's sequence + // begins at 1 again, after the predecessor's. + if stream.last().is_none_or(|last| *last != window) { stream.push(window); } } @@ -7592,6 +7616,36 @@ mod observation_carry { } } + /// Disarm the same tap, as an app does. + fn unsubscribe(plan: &mut LivePlan, strips: &[String]) { + for strip in strips { + plan.handles + .effect_control_mut(strip, LiveEffectAddress::insert(0)) + .expect("a control channel for the insert") + .try_push_observe(0, false, WINDOW_BLOCKS) + .expect("room in the queue"); + } + } + + /// The bindings of `strips`' compressor tap 1 (the gain-reduction tap), each armed with + /// [`WINDOW_BLOCKS`]: what the host's table holds after the subscribes above. + fn bindings_of(strips: &[String]) -> controls::ObservationBindings { + let mut bindings = controls::ObservationBindings::new(); + for strip in strips { + bindings.set(tap_key(strip), WINDOW_BLOCKS); + } + bindings + } + + /// The stable key of `strip`'s compressor insert `comp`, gain-reduction tap. + fn tap_key(strip: &str) -> controls::TapKey { + controls::TapKey { + strip: id(strip), + section: controls::TapSection::Inserts(id("comp")), + tap_id: 1, + } + } + /// Attach `plan`'s observation readers to it, as the browser does. fn attach(plan: &mut LivePlan) { let observations = core::mem::take(&mut plan.handles.effect_observations); @@ -7636,20 +7690,57 @@ mod observation_carry { /// prepared as its successor and swapped in at [`SWAP_BLOCK`], rendered to [`BLOCKS`]. B /// subscribes to nothing: whatever its taps hold at the swap is what the carry brought. fn swapped(backend: Backend, a: &Session, b: &Session, subscribed: &[String]) -> Swapped { + swapped_with( + backend, + a, + b, + &Swap { + early: subscribed, + ..Swap::default() + }, + ) + } + + /// What a swapped run subscribes, when, and what the host's table holds at the swap. + #[derive(Default)] + struct Swap<'a> { + /// Subscribes acknowledged before A's first block. + early: &'a [String], + /// Subscribes acknowledged after A's last block: still in A's queue at the swap block. + late: &'a [String], + /// Disarms acknowledged after A's last block, likewise. + late_disarm: &'a [String], + /// Subscribes pushed into B's own queue before its first block: the reference for a tap + /// armed at preparation. + successor_records: &'a [String], + /// The taps the host's table holds (#1584 D6a); none by default. + bindings: Option<&'a controls::ObservationBindings>, + } + + fn swapped_with(backend: Backend, a: &Session, b: &Session, swap: &Swap<'_>) -> Swapped { use bench_support::alloc as bench_alloc; let feed = bursty_feed(SOURCE, 5); let request = observed_request(); let mut streams = Streams::new(); let mut plan = LivePlan::fresh_with(a, &request, backend); - subscribe(&mut plan, subscribed); + subscribe(&mut plan, swap.early); attach(&mut plan); for block in 0..SWAP_BLOCK { plan.submit(&feed, QUANTUM, block); plan.render(QUANTUM); read_all(&mut plan, &mut streams); } - let mut successor = - LivePlan::successor_with(b, &request, &plan.prepared.inventory, &a.model, backend); + subscribe(&mut plan, swap.late); + unsubscribe(&mut plan, swap.late_disarm); + let mut successor = LivePlan::successor_observed( + b, + &request, + &plan.prepared.inventory, + &a.model, + swap.bindings.unwrap_or(&host_core::NO_OBSERVATION), + backend, + ); + subscribe(&mut successor, swap.successor_records); attach(&mut successor); assert_eq!( successor @@ -7929,6 +8020,279 @@ mod observation_carry { a_restarted_effect_starts_its_tap_fresh(Backend::Simd4); } + /// A compressor insert whose link mode changed restarts, with a base that holds it and a + /// successor that is the base plus one track. + fn restarted_pair(backend: Backend) -> (Session, Session) { + let base = compressor_insert_session(bank_width(backend) - 2); + let mut changed = with_muted_track(base.clone()); + track_mut(&mut changed, "eq1").inserts.effects[0].link_mode = LinkMode::Average; + (Session::compile(base), Session::compile(changed)) + } + + /// #1584 D6a gates 1 and 2: a subscription the host acknowledged survives a rebuild that + /// restarts its effect, whether the record was applied before the swap or still waits in the + /// predecessor's queue. The restarted tap is armed at preparation and opens its first window at + /// the adoption block's first sample, and every window equals the one a record in the + /// successor's own queue, applied at that block, produces from the same state and input. + /// + /// Red mutation: prepare the successor's taps unarmed (drop the arm in + /// `attach_effect_observation`) -> eq1 publishes no window after the swap, and a record that + /// waited in the predecessor's queue is lost with it. + fn a_restarted_effects_subscription_survives_the_rebuild(backend: Backend, pending: bool) { + let what = format!("{backend:?} pending={pending}"); + let (a, b) = restarted_pair(backend); + let all = strips_of(&a); + let restarted = vec!["eq1".to_owned()]; + let others: Vec = all.iter().filter(|s| *s != "eq1").cloned().collect(); + // The reference: B's own queue holds the record for eq1 at its first block. + let reference = swapped_with( + backend, + &a, + &b, + &Swap { + early: &others, + successor_records: &restarted, + ..Swap::default() + }, + ); + // The run: the host's table holds eq1's tap (and the carried ones). + let bindings = bindings_of(&all); + let run = if pending { + swapped_with( + backend, + &a, + &b, + &Swap { + early: &others, + late: &restarted, + bindings: Some(&bindings), + ..Swap::default() + }, + ) + } else { + swapped_with( + backend, + &a, + &b, + &Swap { + early: &all, + bindings: Some(&bindings), + ..Swap::default() + }, + ) + }; + assert_eq!(run.carry, CarryOutcome::Carried, "{what}"); + assert_eq!(run.restarted, vec![Box::::from("eq1")], "{what}"); + assert_eq!( + run.armed[0], + all.len() as u64, + "{what}: the restarted effect's tap is armed on the successor, with the carried ones" + ); + let eq1 = run.streams["eq1"] + .iter() + .find(|window| window.first_sample >= SWAP_SAMPLE) + .copied() + .expect("eq1 publishes after the swap"); + assert_eq!( + eq1.first_sample, SWAP_SAMPLE, + "{what}: the first window opens at the adoption block" + ); + assert_eq!(eq1.sequence, 1, "{what}: a fresh tap starts its sequence"); + assert!( + eq1.left > 0.0 || eq1.right > 0.0, + "{what}: the compressor reduces gain, or the reading is vacuous" + ); + for strip in &all { + let want = reference.streams[strip] + .iter() + .filter(|window| window.first_sample >= SWAP_SAMPLE || strip != "eq1"); + let got = run.streams[strip] + .iter() + .filter(|window| window.first_sample >= SWAP_SAMPLE || strip != "eq1"); + assert_eq!( + got.collect::>(), + want.collect::>(), + "{what} {strip}: the windows equal a record applied at the adoption block" + ); + } + assert_eq!( + run.allocator, + ((0, 0, 0), (0, 0)), + "{what}: allocator and render audit over the swap block" + ); + } + + #[cfg(target_feature = "avx2")] + #[test] + fn a_restarted_effects_subscription_survives_the_rebuild_at_eight_lanes() { + for pending in [false, true] { + a_restarted_effects_subscription_survives_the_rebuild(Backend::Simd8, pending); + } + } + + #[test] + #[cfg_attr( + not(any(target_feature = "neon", target_feature = "simd128")), + ignore = "a console effect bank binds only at the width its build executes, and four \ + lanes is the NEON and simd128 width (#1017)" + )] + fn a_restarted_effects_subscription_survives_the_rebuild_at_four_lanes() { + for pending in [false, true] { + a_restarted_effects_subscription_survives_the_rebuild(Backend::Simd4, pending); + } + } + + /// #1584 D6a gate 3: a disarm still waiting in the predecessor's queue at the swap leaves the + /// restarted effect's successor tap unarmed: the host's table no longer holds it. + /// + /// Red mutation: arm every tap the predecessor had armed -> eq1's successor tap is armed and + /// publishes after the swap. + fn a_pending_disarm_leaves_the_restarted_tap_unarmed(backend: Backend) { + let what = format!("{backend:?}"); + let (a, b) = restarted_pair(backend); + let all = strips_of(&a); + let restarted = vec!["eq1".to_owned()]; + let others: Vec = all.iter().filter(|s| *s != "eq1").cloned().collect(); + let bindings = bindings_of(&others); + let run = swapped_with( + backend, + &a, + &b, + &Swap { + early: &all, + late_disarm: &restarted, + bindings: Some(&bindings), + ..Swap::default() + }, + ); + assert_eq!(run.restarted, vec![Box::::from("eq1")], "{what}"); + assert_eq!( + run.armed[0], + others.len() as u64, + "{what}: the disarmed tap starts unarmed" + ); + assert!( + run.streams["eq1"] + .iter() + .all(|window| window.end_sample <= SWAP_SAMPLE), + "{what}: eq1 publishes nothing after the swap" + ); + } + + #[cfg(target_feature = "avx2")] + #[test] + fn a_pending_disarm_leaves_the_restarted_tap_unarmed_at_eight_lanes() { + a_pending_disarm_leaves_the_restarted_tap_unarmed(Backend::Simd8); + } + + #[test] + #[cfg_attr( + not(any(target_feature = "neon", target_feature = "simd128")), + ignore = "a console effect bank binds only at the width its build executes, and four \ + lanes is the NEON and simd128 width (#1017)" + )] + fn a_pending_disarm_leaves_the_restarted_tap_unarmed_at_four_lanes() { + a_pending_disarm_leaves_the_restarted_tap_unarmed(Backend::Simd4); + } + + /// #1584 D6a gate 6: the table arms every carried tap too, and the prepared arm never wins + /// over the state a carried tap brings: the windows continue (sequences 1, 2, 3, the one open + /// at the swap included) exactly as with an empty table. + /// + /// Red mutation: let `arm_prepared` survive `swap_state` (restart the window at the first + /// block) -> the window spanning the swap restarts and the sequence breaks. + fn a_carried_tap_is_not_restarted_by_the_prepared_arm(backend: Backend, form: Form) { + let what = format!("{backend:?} {form:?}"); + let (a, b) = pair(backend, form); + let subscribed = strips_of(&a); + let kept: Vec = subscribed + .iter() + .filter(|strip| strips_of(&b).contains(strip)) + .cloned() + .collect(); + let bindings = bindings_of(&kept); + let plain = swapped(backend, &a, &b, &subscribed); + let armed = swapped_with( + backend, + &a, + &b, + &Swap { + early: &subscribed, + bindings: Some(&bindings), + ..Swap::default() + }, + ); + assert_eq!(armed.carry, CarryOutcome::Carried, "{what}"); + assert_eq!(armed.restarted, Vec::>::new(), "{what}"); + assert_eq!(armed.streams, plain.streams, "{what}: no tap restarted"); + // The retiring predecessor holds whatever the successor's lanes held before the exchange. + assert_eq!(armed.armed[0], plain.armed[0], "{what}"); + for strip in &kept { + assert_eq!( + armed.streams[strip] + .iter() + .map(|window| window.sequence) + .collect::>(), + vec![1, 2, 3], + "{what} {strip}: no gap and no restart" + ); + } + assert_eq!(armed.allocator, ((0, 0, 0), (0, 0)), "{what}"); + } + + #[cfg(target_feature = "avx2")] + #[test] + fn a_carried_tap_is_not_restarted_by_the_prepared_arm_at_eight_lanes() { + for form in [Form::NodeToNode, Form::BankToBank, Form::NodeToBank] { + a_carried_tap_is_not_restarted_by_the_prepared_arm(Backend::Simd8, form); + } + } + + #[test] + #[cfg_attr( + not(any(target_feature = "neon", target_feature = "simd128")), + ignore = "a console effect bank binds only at the width its build executes, and four \ + lanes is the NEON and simd128 width (#1017)" + )] + fn a_carried_tap_is_not_restarted_by_the_prepared_arm_at_four_lanes() { + for form in [Form::NodeToNode, Form::BankToBank, Form::NodeToBank] { + a_carried_tap_is_not_restarted_by_the_prepared_arm(Backend::Simd4, form); + } + } + + /// #1584 D6a gate 4, the plan half: a table that still names a removed insert, a removed + /// strip or an ID reused for an effect without the tap arms nothing and fails nothing. (The + /// control plane drops such a binding and ends it with an event; this is the plan's own + /// guard.) + #[test] + fn a_stale_binding_arms_nothing() { + let backend = Backend::current(); + let base = compressor_insert_session(bank_width(backend) - 2); + let mut stripped = with_muted_track(base.clone()); + track_mut(&mut stripped, "eq1").inserts.effects.clear(); + let (a, b) = (Session::compile(base), Session::compile(stripped)); + let all = strips_of(&a); + let mut bindings = bindings_of(&all); + // A strip B does not have, and an insert B does not have on a strip it does. + bindings.set(tap_key("gone"), WINDOW_BLOCKS); + let run = swapped_with( + backend, + &a, + &b, + &Swap { + early: &all, + bindings: Some(&bindings), + ..Swap::default() + }, + ); + assert_eq!(run.carry, CarryOutcome::Carried); + assert_eq!( + run.armed[0], + all.len() as u64 - 1, + "every tap B still has is armed; eq1's insert is gone" + ); + } + /// Gate 5 for the taps: the swap block -- carry, the attachment hook's pairing and the first /// render -- allocates and frees nothing, counted rather than aborted. #[test] @@ -8604,3 +8968,1168 @@ fn a_carried_send_is_retargeted_to_the_successors_gain_at_eight_lanes() { fn a_carried_send_is_retargeted_to_the_successors_gain_at_four_lanes() { a_carried_send_is_retargeted_to_the_successors_gain(Backend::Simd4); } + +// --------------------------------------------------------------------------------------------- +// #1395a: a single spectrum capture keeps its history across a plan swap. +// +// A capture whose target and channels are the predecessor's keeps its analysis history, its open +// window, its stream epoch, its sequence and its one-shot state: the observer moves into the +// successor's plan in the swap block, and the control side keeps the reader that is connected to +// it, pairing it at promotion from the list the successor's preparation built. The oracle is the +// post-edit session prepared fresh with the same capture, started at the same block and fed from +// frame 0. + +mod spectrum_carry { + use engine::realtime::{HostAttachment, ObserverCarry, ObserverKind, PlanAdoption}; + use host_core::{ + HostLiveControlRequest, HostLiveLanes, LiveValuesSuperseded, MonitorBase, PreparedHost, + PreparedSpectrumCaptures, SpectrumCaptureCollectionEntry, SpectrumCaptureCollectionRequest, + SpectrumCaptureRequest, SpectrumChannels, SpectrumContinuousWindow, SpectrumHop, + SpectrumPair, SpectrumService, SpectrumServiceError, SpectrumSmoothingConfig, + SpectrumSnapshot, SpectrumStreamStatus, SpectrumTarget, SuccessorBase, + prepare_host_runtime_with_live_controls_and_spectrum, + prepare_host_runtime_with_live_controls_and_spectrum_collection, + prepare_host_runtime_with_live_controls_and_spectrum_collection_successor, + prepare_host_runtime_with_live_controls_and_spectrum_successor, + spectrum_capture_collection_resources, spectrum_capture_resources_for, + }; + + use super::successor::{self, Block, SUCCESSOR_REVISION}; + use super::*; + + /// Blocks per run: 8192 frames, four windows of the 2048-frame capture and then some. + const TOTAL: usize = 64; + /// The swap block: sample 5120, inside the window that opened at 4608 (hop 512). + const SWAP: usize = 40; + const SWAP_SAMPLE: u64 = (SWAP * QUANTUM) as u64; + /// A one-shot is armed at block 30 and completes 16 blocks later, past the swap. + const ARM: usize = 30; + const TRACK: &str = "eq0"; + + fn request(target: SpectrumTarget, channels: SpectrumChannels) -> SpectrumCaptureRequest { + SpectrumCaptureRequest { + maximum_capture_bytes: spectrum_capture_resources_for(&target).retained_bytes, + target, + channels, + } + } + + fn stereo_post_matrix() -> SpectrumCaptureRequest { + request( + SpectrumTarget::TrackPostMatrix(TRACK.into()), + SpectrumChannels::Stereo, + ) + } + + /// What the host does to the service of a collection: which entry it selects before the stream + /// starts, and which entry it switches the running stream to, at which block. + #[derive(Clone, Default)] + struct Script { + select: Option<(SpectrumTarget, SpectrumChannels)>, + switch: Option<(usize, SpectrumTarget, SpectrumChannels)>, + } + + /// A capture request of either kind, prepared fresh or as a successor. + trait Spec { + fn prepare( + &self, + session: &Session, + base: Option>, + ) -> (PreparedHost, PreparedSpectrumCaptures); + + fn script(&self) -> Script { + Script::default() + } + } + + impl Spec for SpectrumCaptureRequest { + fn prepare( + &self, + session: &Session, + base: Option>, + ) -> (PreparedHost, PreparedSpectrumCaptures) { + let live = HostLiveControlRequest::default(); + let superseded = LiveValuesSuperseded::default(); + let (host, _handles, capture) = match base { + None => prepare_host_runtime_with_live_controls_and_spectrum( + &session.compiled, + &caps(), + &live, + self, + HostLiveLanes::ALL, + &superseded, + ), + Some(base) => prepare_host_runtime_with_live_controls_and_spectrum_successor( + &session.compiled, + &caps(), + &live, + self, + HostLiveLanes::ALL, + &superseded, + base, + ), + } + .unwrap_or_else(|failure| panic!("prepare: {failure:?}")); + (host, capture.into()) + } + } + + /// A collection request with the host's script for it. + struct Many { + request: SpectrumCaptureCollectionRequest, + script: Script, + } + + impl Spec for Many { + fn prepare( + &self, + session: &Session, + base: Option>, + ) -> (PreparedHost, PreparedSpectrumCaptures) { + let live = HostLiveControlRequest::default(); + let superseded = LiveValuesSuperseded::default(); + let (host, _handles, capture) = match base { + None => prepare_host_runtime_with_live_controls_and_spectrum_collection( + &session.compiled, + &caps(), + &live, + &self.request, + HostLiveLanes::ALL, + &superseded, + ), + Some(base) => { + prepare_host_runtime_with_live_controls_and_spectrum_collection_successor( + &session.compiled, + &caps(), + &live, + &self.request, + HostLiveLanes::ALL, + &superseded, + base, + ) + } + } + .unwrap_or_else(|failure| panic!("prepare: {failure:?}")); + (host, capture.into()) + } + + fn script(&self) -> Script { + self.script.clone() + } + } + + /// The test's host attachment: it records the carry record the swap block hands it, within the + /// capacity reserved at construction. + struct Rig { + carries: Vec, + } + + impl Rig { + fn new() -> Box { + Box::new(Self { + carries: Vec::with_capacity(16), + }) + } + } + + impl HostAttachment for Rig { + fn as_any_mut(&mut self) -> &mut dyn core::any::Any { + self + } + + fn adopt_predecessor( + &mut self, + predecessor: &mut dyn HostAttachment, + carries: &[ObserverCarry], + ) { + if predecessor.as_any_mut().downcast_mut::().is_some() { + self.carries.extend_from_slice(carries); + } + } + } + + /// What one read of the stream returned. + #[derive(Clone, Copy)] + struct Read { + status: SpectrumStreamStatus, + window: Option, + } + + #[derive(Clone, Copy, Debug)] + enum Drive { + /// A continuous stream started before block 0 and read after every block. + Stream, + /// A one-shot armed before the given block and read after every block from there on. + OneShot(usize), + /// As `OneShot`, but a refused read is recorded, and after `CaptureInvalid` the host arms + /// again at once (#1618). + OneShotRearm(usize), + } + + /// The host's reader of one run: the service and what it read. + struct Driver { + service: SpectrumService, + drive: Drive, + script: Script, + reads: Vec, + shot: Option, + /// The refusals the one-shot's reads answered, with the block after which each came. + refused: Vec<(usize, SpectrumServiceError)>, + /// The block after which the one-shot completed. + shot_at: Option, + } + + impl Driver { + fn new( + captures: PreparedSpectrumCaptures, + rate: u32, + drive: Drive, + script: Script, + ) -> Self { + let mut service = SpectrumService::new( + captures, + rate, + QUANTUM as u32, + Some(SpectrumHop::new(512).expect("hop")), + ); + if let Some((target, channels)) = &script.select { + service + .select(target.as_ref(), *channels) + .expect("select the first entry"); + } + if matches!(drive, Drive::Stream) { + service + .start_stream(SpectrumSmoothingConfig::default(), None) + .expect("start the stream"); + } + Self { + service, + drive, + script, + reads: Vec::new(), + shot: None, + refused: Vec::new(), + shot_at: None, + } + } + + fn before(&mut self, block: usize) { + if let Some((at, target, channels)) = &self.script.switch + && *at == block + { + self.service + .select_stream(target.as_ref(), *channels, None, None) + .expect("switch the stream to the other entry"); + } + if matches!(self.drive, Drive::OneShot(at) | Drive::OneShotRearm(at) if at == block) { + if matches!(self.drive, Drive::OneShotRearm(_)) { + // A collection's select armed its entry for nobody: the host's own arm + // replaces that one. + let _ = self.service.cancel(); + } + self.service.arm().expect("arm"); + } + } + + fn after(&mut self, block: usize) { + match self.drive { + Drive::Stream => { + self.service.read_stream(None).expect("read the stream"); + self.reads.push(Read { + status: self.service.stream_metadata().status, + window: self.service.stream_window().copied(), + }); + } + Drive::OneShot(at) if block >= at && self.shot.is_none() => { + self.shot = self + .service + .read(SpectrumChannels::Stereo) + .expect("read the one-shot"); + } + Drive::OneShotRearm(at) if block >= at && self.shot.is_none() => { + // Without a selection the read refuses before it looks at the mask. + let channels = self.service.channels().unwrap_or(SpectrumChannels::Stereo); + match self.service.read(channels) { + Ok(Some(snapshot)) => { + self.shot = Some(snapshot); + self.shot_at = Some(block); + } + Ok(None) => {} + Err(error) => { + self.refused.push((block, error)); + if error == SpectrumServiceError::CaptureInvalid { + self.service.arm().expect("arm again after the refusal"); + } + } + } + } + Drive::OneShot(_) | Drive::OneShotRearm(_) => {} + } + } + } + + /// How the host promotes the successor after its adoption block. + #[derive(Clone, Copy, Debug, Eq, PartialEq)] + enum Promote { + /// As the engine reported the block: `carried` is `CarryOutcome::Carried`. + AsReported, + /// As after a refused hand-over: `carried` is false although the list names a pair. + Refused, + } + + struct Run { + blocks: Vec, + driver: Driver, + /// The spectrum entries of the carry record the swap block wrote. + record: Vec, + /// The successor's pairing list. + pairing: Vec, + /// `(allocations, reallocations, deallocations)` of the promotion call. + adopt_counts: (u64, u64, u64), + /// The captures the promotion displaced, dropped last. + displaced: PreparedSpectrumCaptures, + _exchange: ( + engine::realtime::PlanPublisher, + engine::realtime::PlanRetirer, + ), + } + + /// `a` rendered to [`SWAP`], `b` prepared as its successor and swapped in, promoted after its + /// adoption block, rendered to [`TOTAL`]. + fn swapped( + a: &Session, + a_request: &impl Spec, + b: &Session, + b_request: &impl Spec, + drive: Drive, + promote: Promote, + ) -> Run { + let backend = Backend::current(); + let _ = backend; + let feed = Feed::with_blocks(SOURCE, 1, TOTAL); + let (mut a_prepared, a_capture) = a_request.prepare(a, None); + a_prepared.plan.set_host_attachment(Rig::new()); + let rate = a_prepared.report.sample_rate_hz; + let PreparedHost { + plan, + sources: mut a_sources, + inventory, + .. + } = a_prepared; + let (mut publisher, mut owner, retirer) = successor::exchange(plan); + let mut driver = Driver::new(a_capture, rate, drive, a_request.script()); + let mut blocks = Vec::new(); + feed.submit(&mut a_sources, rate, 0); + for block in 0..SWAP { + feed.submit(&mut a_sources, rate, block + 1); + driver.before(block); + blocks.push(successor::render(&mut owner)); + driver.after(block); + } + let (mut b_prepared, b_capture) = b_request.prepare( + b, + Some(SuccessorBase { + inventory: &inventory, + committed: &a.model, + revision: SUCCESSOR_REVISION, + edit_ramps: &session::EditRamps::new(), + monitor: MonitorBase::NONE, + observation: &host_core::NO_OBSERVATION, + }), + ); + b_prepared.sources.adopt_persisting(&mut a_sources); + let pairing = b_prepared.inventory.spectrum_pairing().to_vec(); + b_prepared.plan.set_host_attachment(Rig::new()); + let PreparedHost { + mut plan, + sources: mut b_sources, + .. + } = b_prepared; + plan.take_revision_gate() + .expect("the successor's handle") + .publish(SUCCESSOR_REVISION); + publisher + .reserve_replacement(plan, PlanAdoption::Next) + .unwrap_or_else(|_| panic!("reserve the successor")) + .commit(); + // From the swap on the host acts on the successor's collection, in the successor's terms. + driver.script = b_request.script(); + let mut b_capture = Some(b_capture); + let mut adopt_counts = (0, 0, 0); + let mut displaced = None; + for block in SWAP..TOTAL { + if block + 1 < TOTAL { + feed.submit(&mut b_sources, rate, block + 1); + } + driver.before(block); + blocks.push(successor::render(&mut owner)); + if block == SWAP { + let carried = + blocks[SWAP].carry == CarryOutcome::Carried && promote == Promote::AsReported; + let mark = bench_support::alloc::current_thread_counters(); + displaced = Some(driver.service.adopt_successor( + b_capture.take().expect("the successor's capture"), + &pairing, + carried, + None, + )); + let delta = bench_support::alloc::current_thread_delta_since(mark); + adopt_counts = (delta.allocations, delta.reallocations, delta.deallocations); + } + driver.after(block); + } + let record = owner + .active_attachment_mut::() + .expect("attachment") + .carries + .iter() + .copied() + .filter(|carry| carry.kind == ObserverKind::Spectrum) + .collect(); + Run { + blocks, + driver, + record, + pairing, + adopt_counts, + displaced: displaced.expect("promoted"), + _exchange: (publisher, retirer), + } + } + + /// `b` prepared fresh with `request` and read as `drive` from frame 0. + fn reference(b: &Session, request: &impl Spec, drive: Drive) -> Driver { + let feed = Feed::with_blocks(SOURCE, 1, TOTAL); + let (prepared, capture) = request.prepare(b, None); + let rate = prepared.report.sample_rate_hz; + let PreparedHost { + plan, mut sources, .. + } = prepared; + let (_publisher, mut owner, _retirer) = successor::exchange(plan); + let mut driver = Driver::new(capture, rate, drive, request.script()); + feed.submit(&mut sources, rate, 0); + for block in 0..TOTAL { + if block + 1 < TOTAL { + feed.submit(&mut sources, rate, block + 1); + } + driver.before(block); + successor::render(&mut owner); + driver.after(block); + } + driver + } + + /// A: `eq0` and `eq1`. B adds a muted track that sorts first, so every observer of the + /// successor sits at another place than the predecessor's. + fn sessions() -> (Session, Session) { + let a = Session::compile(session_a()); + let mut b_model = session_a(); + add_track(&mut b_model, "eq0", MUTED_TRACK, SOURCE, true); + (a, Session::compile(b_model)) + } + + fn assert_same_window( + got: &SpectrumContinuousWindow, + want: &SpectrumContinuousWindow, + label: &str, + ) { + assert_eq!(got.first_sample, want.first_sample, "{label}: first_sample"); + assert_eq!(got.channels, want.channels, "{label}: channels"); + assert_eq!(got.source_underrun, want.source_underrun, "{label}"); + assert_eq!(got.stream_epoch, want.stream_epoch, "{label}: epoch"); + assert_eq!(got.sequence, want.sequence, "{label}: sequence"); + assert_eq!(got.dropped_captures, want.dropped_captures, "{label}"); + assert!( + got.left + .iter() + .zip(&want.left) + .all(|(a, b)| a.to_bits() == b.to_bits()) + && got + .right + .iter() + .zip(&want.right) + .all(|(a, b)| a.to_bits() == b.to_bits()), + "{label}: the planes equal the reference's bit for bit" + ); + } + + /// Gate 1: a continuous capture keeps reading, window for window, across the swap. + #[test] + fn a_continuous_capture_carries_and_its_reader_keeps_receiving() { + let (a, b) = sessions(); + let request = stereo_post_matrix(); + let expected = reference(&b, &request, Drive::Stream); + let run = swapped( + &a, + &request, + &b, + &request, + Drive::Stream, + Promote::AsReported, + ); + assert_eq!(run.blocks[SWAP].carry, CarryOutcome::Carried); + assert_eq!(run.blocks[SWAP].swap, SwapOutcome::Applied); + assert_eq!( + run.pairing, + vec![SpectrumPair { + successor: 0, + predecessor: 0 + }] + ); + assert_eq!( + run.record, + vec![ObserverCarry { + kind: ObserverKind::Spectrum, + successor: 0, + predecessor: 0 + }], + "the swap block moved exactly the observers the control side's list pairs" + ); + + let reads = &run.driver.reads; + assert_eq!(reads.len(), TOTAL); + let first_ready = reads + .iter() + .position(|read| read.status == SpectrumStreamStatus::Ready) + .expect("the stream produces before the swap"); + assert!(first_ready < SWAP, "a window is complete before the swap"); + let mut next_sequence = 0; + let mut spanning = false; + let mut after_swap = 0; + for (block, (got, want)) in reads.iter().zip(&expected.reads).enumerate() { + assert_eq!(got.status, want.status, "block {block}: status"); + if block > first_ready { + assert!( + matches!( + got.status, + SpectrumStreamStatus::Ready | SpectrumStreamStatus::Pending + ), + "block {block}: no Failed, Gap or Warming after the stream first produced, got {:?}", + got.status + ); + } + match (&got.window, &want.window) { + (Some(got), Some(want)) => { + assert_same_window(got, want, &format!("block {block}")); + assert_eq!(got.sequence, next_sequence, "block {block}: no gap"); + assert_eq!(got.stream_epoch, 1, "block {block}: the epoch never moved"); + next_sequence += 1; + spanning |= got.first_sample < SWAP_SAMPLE + && got.end_sample().expect("end") > SWAP_SAMPLE; + after_swap += usize::from(block >= SWAP); + } + (None, None) => {} + _ => panic!("block {block}: a window on one side only"), + } + } + assert!( + spanning, + "a window spans the swap, or the carry is not tested" + ); + assert!(after_swap >= 3, "the reader keeps receiving after the swap"); + } + + /// Without the pairing the host reads the predecessor's capture, whose observer sits in the + /// displaced plan: the oracle above can fail. + #[test] + fn a_refused_hand_over_installs_the_fresh_capture() { + let (a, b) = sessions(); + let request = stereo_post_matrix(); + let run = swapped(&a, &request, &b, &request, Drive::Stream, Promote::Refused); + assert_eq!(run.blocks[SWAP].carry, CarryOutcome::Carried); + // The service reads the successor's own capture now. Its observer sits in a plan that + // carried its predecessor's observer away, so it renders nothing: the stream the service + // still believes it runs has no epoch and no window. + let after = &run.driver.reads[SWAP + 1..]; + assert!( + after.iter().all(|read| read.window.is_none()), + "the fresh capture has no stream: the carried observer is not its producer" + ); + assert_eq!( + after.last().expect("read").status, + SpectrumStreamStatus::Inactive + ); + // The capture that was displaced is the predecessor's, still connected to the observer + // that renders on, and still on its stream (the host drops it off the audio thread). + let PreparedSpectrumCaptures::Single(old) = &run.displaced else { + panic!("a single capture"); + }; + assert_eq!(old.stream_epoch(), Some(1)); + // The stream the service ran on the old capture ended with it (#1618): the host starts + // one anew on the successor's capture. + let mut run = run; + assert!(!run.driver.service.stream_active()); + assert_eq!(run.driver.service.stream_cadence(), None); + run.driver + .service + .start_stream(SpectrumSmoothingConfig::default(), None) + .expect("a stream starts on the fresh capture"); + } + + /// Gate 3, one-shot: a capture armed before the swap and carried completes with a window that + /// began before the swap block. + #[test] + fn an_armed_one_shot_completes_across_the_swap() { + let (a, b) = sessions(); + let request = stereo_post_matrix(); + let expected = reference(&b, &request, Drive::OneShot(ARM)); + let run = swapped( + &a, + &request, + &b, + &request, + Drive::OneShot(ARM), + Promote::AsReported, + ); + let got = run.driver.shot.expect("the carried one-shot completes"); + let want = expected.shot.expect("the reference completes"); + assert!( + got.window.first_sample < SWAP_SAMPLE + && got.window.end_sample().expect("end") > SWAP_SAMPLE, + "the window spans the swap block" + ); + assert_eq!(got.window.first_sample, want.window.first_sample); + assert_eq!(got.window.channels, want.window.channels); + assert!( + got.window + .left + .iter() + .zip(&want.window.left) + .chain(got.window.right.iter().zip(&want.window.right)) + .all(|(a, b)| a.to_bits() == b.to_bits()), + "the planes equal the reference's bit for bit" + ); + } + + /// Gate 3, key: a changed channel mask carries nothing, the list is empty, and the fresh + /// capture starts its own stream. + #[test] + fn a_changed_key_starts_fresh() { + let (a, b) = sessions(); + let a_request = stereo_post_matrix(); + let b_request = request( + SpectrumTarget::TrackPostMatrix(TRACK.into()), + SpectrumChannels::Left, + ); + let mut run = swapped( + &a, + &a_request, + &b, + &b_request, + Drive::Stream, + Promote::AsReported, + ); + assert!( + run.pairing.is_empty(), + "no pair across a changed channel mask" + ); + assert!(run.record.is_empty(), "the swap block moved no capture"); + // The host restarts the stream it had on the old capture, on the successor's. + run.driver.service.stop_stream(None); + run.driver + .service + .start_stream(SpectrumSmoothingConfig::default(), None) + .expect("restart"); + // The successor's plan renders on past TOTAL: not available here, so the proof is the + // service's own: it reports the fresh capture's mask, not the carried one's. + assert_eq!(run.driver.service.channels(), Some(SpectrumChannels::Left)); + assert_eq!( + run.driver.service.stream_epoch(), + Some(1), + "a fresh epoch, not the predecessor's" + ); + } + + /// #1618 gate 1: a one-shot armed on a capture that does not carry (its channel mask changed) + /// would wait for an observer that no longer renders. The service settles it: the read after + /// the promotion answers `CaptureInvalid` once, and the arm that follows completes on the + /// successor with the window a fresh arm there gives. + #[test] + fn a_one_shot_that_does_not_carry_settles_invalid_and_a_new_arm_completes() { + let (a, b) = sessions(); + let a_request = stereo_post_matrix(); + let b_request = request( + SpectrumTarget::TrackPostMatrix(TRACK.into()), + SpectrumChannels::Left, + ); + let run = swapped( + &a, + &a_request, + &b, + &b_request, + Drive::OneShotRearm(ARM), + Promote::AsReported, + ); + assert!(run.pairing.is_empty(), "the changed key carries nothing"); + assert_eq!( + run.driver.refused, + vec![(SWAP, SpectrumServiceError::CaptureInvalid)], + "pending until the promotion, one refusal at it, none after" + ); + let got = run + .driver + .shot + .expect("the new arm completes on the successor"); + let shot_at = run.driver.shot_at.expect("completed"); + assert!(shot_at > SWAP && shot_at < TOTAL, "completed at {shot_at}"); + assert!( + got.window.first_sample > SWAP_SAMPLE, + "a window of the successor, armed after the promotion" + ); + assert_eq!(got.window.channels, SpectrumChannels::Left); + // The window equals what a fresh arm on the same plan gives at the same block. + let expected = reference(&b, &b_request, Drive::OneShotRearm(SWAP + 1)); + let want = expected.shot.expect("the reference completes"); + assert_eq!(got.window.first_sample, want.window.first_sample); + assert!( + got.window + .left + .iter() + .zip(&want.window.left) + .all(|(a, b)| a.to_bits() == b.to_bits()), + "the plane equals the reference's bit for bit" + ); + } + + /// #1618 gate 1, refused hand-over: the list names a pair but the adoption did not carry, so + /// the capture is settled the same way. (The engine did carry the observer here, so the + /// re-armed fresh capture has no producer and stays pending: the point is the refusal.) + #[test] + fn a_one_shot_after_a_refused_hand_over_settles_invalid() { + let (a, b) = sessions(); + let request = stereo_post_matrix(); + let run = swapped( + &a, + &request, + &b, + &request, + Drive::OneShotRearm(ARM), + Promote::Refused, + ); + assert_eq!( + run.driver.refused, + vec![(SWAP, SpectrumServiceError::CaptureInvalid)] + ); + assert!(run.driver.shot.is_none()); + } + + /// A one-shot that completed and was read before the swap leaves nothing to settle: the next + /// read after the promotion is not a refusal, and a stream may start on the fresh capture. + #[test] + fn an_idle_capture_that_does_not_carry_settles_nothing() { + let (a, b) = sessions(); + let a_request = stereo_post_matrix(); + let b_request = request( + SpectrumTarget::TrackPostMatrix(TRACK.into()), + SpectrumChannels::Left, + ); + let mut run = swapped( + &a, + &a_request, + &b, + &b_request, + Drive::OneShotRearm(0), + Promote::AsReported, + ); + // Armed at block 0: it completed at block 15, long before the swap. + assert!( + run.driver.refused.is_empty(), + "nothing in flight at the swap" + ); + run.driver + .service + .start_stream(SpectrumSmoothingConfig::default(), None) + .expect("the fresh capture is idle"); + } + + /// Gate 4: the swap block and the promotion allocate and free nothing. + #[test] + fn the_swap_block_and_the_promotion_allocate_and_free_nothing() { + use bench_support::alloc::{Mode, assert_installed, mode, set_mode}; + struct RestoreMode(Mode); + impl Drop for RestoreMode { + fn drop(&mut self) { + set_mode(self.0); + } + } + assert_installed(); + let _restore = RestoreMode(mode()); + set_mode(Mode::Count); + engine::realtime::audit::warm_up(); + let (a, b) = sessions(); + let request = stereo_post_matrix(); + let run = swapped( + &a, + &request, + &b, + &request, + Drive::Stream, + Promote::AsReported, + ); + assert_eq!(run.blocks[SWAP].carry, CarryOutcome::Carried); + assert!( + !run.record.is_empty(), + "a capture carried through the measured block" + ); + for (index, block) in run.blocks.iter().enumerate().skip(1) { + assert_eq!(block.allocator, (0, 0, 0), "block {index}: allocator calls"); + assert_eq!(block.audit, (0, 0), "block {index}: render audit"); + } + assert_eq!( + run.adopt_counts, + (0, 0, 0), + "the promotion allocates and frees nothing" + ); + } + + // ----------------------------------------------------------------------------------------- + // #1395b: a collection of captures carries entry by entry, matched by key. + + /// The block at which the host switches the running stream to the other entry. + const SWITCH: usize = 44; + + fn entry(target: SpectrumTarget, channels: SpectrumChannels) -> SpectrumCaptureCollectionEntry { + SpectrumCaptureCollectionEntry { target, channels } + } + + /// Track `eq0` after its matrix, both channels: the entry the stream runs on. + fn t_stereo() -> SpectrumCaptureCollectionEntry { + entry( + SpectrumTarget::TrackPostMatrix(TRACK.into()), + SpectrumChannels::Stereo, + ) + } + + /// Track `eq1` after its input builtins, one channel. + fn u_entry(channels: SpectrumChannels) -> SpectrumCaptureCollectionEntry { + entry( + SpectrumTarget::TrackPostInputBuiltins("eq1".into()), + channels, + ) + } + + fn many( + entries: Vec, + switch: Option, + ) -> Many { + let maximum_capture_bytes = spectrum_capture_collection_resources(&entries) + .expect("collection resources") + .retained_bytes; + Many { + request: SpectrumCaptureCollectionRequest { + entries, + maximum_capture_bytes, + }, + script: Script { + select: Some((t_stereo().target, t_stereo().channels)), + switch: switch.map(|entry| (SWITCH, entry.target, entry.channels)), + }, + } + } + + fn sorted(mut pairs: Vec) -> Vec<(usize, usize)> { + pairs.sort_by_key(|pair| pair.successor); + pairs + .into_iter() + .map(|pair| (pair.successor, pair.predecessor)) + .collect() + } + + /// The reads of `got` equal `want`'s block for block from `from` on, windows bit for bit. + fn assert_reads_equal(got: &Driver, want: &Driver, from: usize) -> usize { + let mut windows = 0; + for (block, (got, want)) in got.reads.iter().zip(&want.reads).enumerate().skip(from) { + assert_eq!(got.status, want.status, "block {block}: status"); + match (&got.window, &want.window) { + (Some(got), Some(want)) => { + assert_same_window(got, want, &format!("block {block}")); + windows += 1; + } + (None, None) => {} + _ => panic!("block {block}: a window on one side only"), + } + } + windows + } + + /// The stream on `t` kept receiving across the swap, as in gate 1: no `Failed`, `Gap` or + /// `Warming` after it first produced, one epoch, consecutive sequences, a window across the + /// swap, and windows after it. + fn assert_stream_kept_receiving(driver: &Driver) { + let reads = &driver.reads[..SWITCH]; + let first_ready = reads + .iter() + .position(|read| read.status == SpectrumStreamStatus::Ready) + .expect("the stream produces before the swap"); + assert!(first_ready < SWAP); + let mut next_sequence = 0; + let mut spanning = false; + let mut after_swap = 0; + for (block, read) in reads.iter().enumerate().skip(first_ready) { + assert!( + matches!( + read.status, + SpectrumStreamStatus::Ready | SpectrumStreamStatus::Pending + ), + "block {block}: no Failed, Gap or Warming, got {:?}", + read.status + ); + if let Some(window) = &read.window { + assert_eq!(window.sequence, next_sequence, "block {block}: no gap"); + assert_eq!( + window.stream_epoch, 1, + "block {block}: the epoch never moved" + ); + next_sequence += 1; + spanning |= window.first_sample < SWAP_SAMPLE + && window.end_sample().expect("end") > SWAP_SAMPLE; + after_swap += usize::from(block >= SWAP); + } + } + assert!( + spanning, + "a window spans the swap, or the carry is not tested" + ); + assert!(after_swap >= 1, "the reader keeps receiving after the swap"); + } + + /// Gate 2: B lists the entries in the other order. Each pairs by key, the selection follows + /// its entry to the new index with its epoch, the stream reads on, and a switch to the other + /// entry (which also carried, idle) starts it as the reference does. + #[test] + fn a_reordered_collection_pairs_by_key_and_its_reader_keeps_receiving() { + let (a, b) = sessions(); + let a_request = many(vec![t_stereo(), u_entry(SpectrumChannels::Left)], None); + let b_request = many( + vec![u_entry(SpectrumChannels::Left), t_stereo()], + Some(u_entry(SpectrumChannels::Left)), + ); + let expected = reference(&b, &b_request, Drive::Stream); + let run = swapped( + &a, + &a_request, + &b, + &b_request, + Drive::Stream, + Promote::AsReported, + ); + assert_eq!(run.blocks[SWAP].carry, CarryOutcome::Carried); + assert_eq!( + sorted(run.pairing.clone()), + vec![(0, 1), (1, 0)], + "paired by key, not by position" + ); + assert_eq!( + sorted( + run.record + .iter() + .map(|carry| SpectrumPair { + successor: carry.successor, + predecessor: carry.predecessor + }) + .collect() + ), + vec![(0, 1), (1, 0)], + "the swap block moved exactly the observers the control side's list pairs" + ); + assert_stream_kept_receiving(&run.driver); + let windows = assert_reads_equal(&run.driver, &expected, 0); + assert!(windows >= 8, "windows on both entries compared: {windows}"); + let switched = run.driver.reads[SWITCH..] + .iter() + .filter_map(|read| read.window.as_ref()) + .collect::>(); + assert!( + !switched.is_empty() + && switched + .iter() + .all(|window| window.channels == SpectrumChannels::Left), + "the second entry produced its own windows after the switch" + ); + // The selection moved with its entry: still `t`, with the epoch it had. + assert_eq!( + run.driver.service.selection_epoch(), + expected.service.selection_epoch() + ); + } + + /// Gate 3, collection: one entry's channels changed. That entry starts fresh, the other + /// carries as in gate 1. + #[test] + fn a_collection_entry_with_a_changed_key_starts_fresh() { + let (a, b) = sessions(); + let a_request = many(vec![t_stereo(), u_entry(SpectrumChannels::Left)], None); + let b_request = many( + vec![t_stereo(), u_entry(SpectrumChannels::Right)], + Some(u_entry(SpectrumChannels::Right)), + ); + let expected = reference(&b, &b_request, Drive::Stream); + let run = swapped( + &a, + &a_request, + &b, + &b_request, + Drive::Stream, + Promote::AsReported, + ); + assert_eq!(sorted(run.pairing.clone()), vec![(0, 0)]); + assert_eq!(run.record.len(), 1, "only the unchanged key moved"); + assert_stream_kept_receiving(&run.driver); + assert!(assert_reads_equal(&run.driver, &expected, 0) >= 8); + assert!( + run.driver.reads[SWITCH..] + .iter() + .filter_map(|read| read.window.as_ref()) + .all(|window| window.channels == SpectrumChannels::Right), + "the changed entry reads its own channel, from the plane it copies" + ); + } + + /// A refused hand-over moved no observer: the successor's fresh collection is installed, + /// without a selection, and its epoch says the selection was lost. + #[test] + fn a_refused_hand_over_installs_the_fresh_collection_without_a_selection() { + let (a, b) = sessions(); + let a_request = many(vec![t_stereo(), u_entry(SpectrumChannels::Left)], None); + let b_request = many(vec![u_entry(SpectrumChannels::Left), t_stereo()], None); + let run = swapped( + &a, + &a_request, + &b, + &b_request, + Drive::Stream, + Promote::Refused, + ); + assert_eq!(run.driver.service.target(), None, "no selection"); + assert_eq!( + run.driver.service.selection_epoch(), + 2, + "the epoch the predecessor's selection reached, plus one" + ); + assert!( + run.driver.reads[SWAP + 1..] + .iter() + .all(|read| read.window.is_none()), + "the fresh collection has no stream" + ); + // The displaced collection is the predecessor's, still on its selection. + let PreparedSpectrumCaptures::Collection(old) = &run.displaced else { + panic!("a collection"); + }; + assert_eq!(old.selected_index(), Some(0)); + assert_eq!(old.selection_epoch(), 1); + } + + /// #1618 / #1395b: the selected entry did not carry, so the service holds no stream any more. + /// `select_stream` answers `StreamInactive`; a plain select and a new start run the cadence + /// again on the entry (it does not arm a one-shot in the stream's place). + #[test] + fn a_lost_selection_ends_the_stream_and_a_new_start_restarts_the_cadence() { + let (a, b) = sessions(); + let a_request = many(vec![t_stereo(), u_entry(SpectrumChannels::Left)], None); + let b_request = many(vec![u_entry(SpectrumChannels::Left), t_stereo()], None); + let mut run = swapped( + &a, + &a_request, + &b, + &b_request, + Drive::Stream, + Promote::Refused, + ); + let service = &mut run.driver.service; + assert!(!service.stream_active(), "no stream without a selection"); + assert_eq!(service.stream_cadence(), None); + assert_eq!( + service + .select_stream( + t_stereo().target.as_ref(), + SpectrumChannels::Stereo, + None, + None + ) + .expect_err("nothing to restart"), + SpectrumServiceError::StreamInactive + ); + service + .select(t_stereo().target.as_ref(), SpectrumChannels::Stereo) + .expect("select the entry"); + service + .start_stream(SpectrumSmoothingConfig::default(), None) + .expect("the cadence starts again"); + assert!(service.stream_active()); + assert!(service.stream_cadence().is_some()); + assert_eq!( + service.stream_metadata().status, + SpectrumStreamStatus::Warming + ); + } + + /// #1618: a one-shot armed on the selected entry of a collection whose selection did not carry + /// ends with the selection. The read answers `NotArmed`, the caller's arm is released (a + /// select and a stream start are not `Busy`), and the epoch says the selection was lost. + #[test] + fn a_one_shot_on_a_lost_selection_is_released() { + let (a, b) = sessions(); + let a_request = many(vec![t_stereo(), u_entry(SpectrumChannels::Left)], None); + let b_request = many(vec![u_entry(SpectrumChannels::Left), t_stereo()], None); + let mut run = swapped( + &a, + &a_request, + &b, + &b_request, + Drive::OneShotRearm(ARM), + Promote::Refused, + ); + assert_eq!( + run.driver.refused.first(), + Some(&(SWAP, SpectrumServiceError::NotArmed)) + ); + assert!(run.driver.shot.is_none()); + let service = &mut run.driver.service; + assert_eq!(service.selection_epoch(), 2); + service + .select(t_stereo().target.as_ref(), SpectrumChannels::Stereo) + .expect("the arm is released: select is not Busy"); + service + .start_stream(SpectrumSmoothingConfig::default(), None) + .expect("and neither is a stream start"); + } + + /// Gate 4, collection: the swap block and the promotion allocate and free nothing. + #[test] + fn a_collection_swap_block_and_promotion_allocate_and_free_nothing() { + use bench_support::alloc::{Mode, assert_installed, mode, set_mode}; + struct RestoreMode(Mode); + impl Drop for RestoreMode { + fn drop(&mut self) { + set_mode(self.0); + } + } + assert_installed(); + let _restore = RestoreMode(mode()); + set_mode(Mode::Count); + engine::realtime::audit::warm_up(); + let (a, b) = sessions(); + let a_request = many(vec![t_stereo(), u_entry(SpectrumChannels::Left)], None); + let b_request = many(vec![u_entry(SpectrumChannels::Left), t_stereo()], None); + let run = swapped( + &a, + &a_request, + &b, + &b_request, + Drive::Stream, + Promote::AsReported, + ); + assert_eq!(run.blocks[SWAP].carry, CarryOutcome::Carried); + assert_eq!(run.record.len(), 2, "both captures carried"); + for (index, block) in run.blocks.iter().enumerate().skip(1) { + assert_eq!(block.allocator, (0, 0, 0), "block {index}: allocator calls"); + assert_eq!(block.audit, (0, 0), "block {index}: render audit"); + } + assert_eq!( + run.adopt_counts, + (0, 0, 0), + "the promotion allocates and frees nothing" + ); + } +} diff --git a/crates/host-core/tests/support/successor.rs b/crates/host-core/tests/support/successor.rs index e215e53ee..eab3cb6a0 100644 --- a/crates/host-core/tests/support/successor.rs +++ b/crates/host-core/tests/support/successor.rs @@ -102,6 +102,7 @@ impl Session { revision: SUCCESSOR_REVISION, edit_ramps: &session::EditRamps::new(), monitor: host_core::MonitorBase::NONE, + observation: &host_core::NO_OBSERVATION, }, backend, ) diff --git a/crates/protocol/src/controller.rs b/crates/protocol/src/controller.rs index a8b4ed0c7..a7cf0aed9 100644 --- a/crates/protocol/src/controller.rs +++ b/crates/protocol/src/controller.rs @@ -23,11 +23,11 @@ use crate::{ CounterTelemetryRecord, CounterValue, CountersRequest, DecodeError, DecodeScratch, DecodedCommandPayload, Diagnostic, DiagnosticEvent, DiagnosticsPage, DiagnosticsRequest, EditPath, EncodeError, EventPayload, ExpectedRevision, MessageId, MeterBatch, MeterRecord, - NonOkResponse, PreparedSessionTransaction, ProtocolCodec, ProtocolQueues, QueueReport, - ReliablePayload, ReliableSlot, RequestId, ResponseGetRequest, ResponseResult, SampleTime, - SessionCommitted, SessionEdit, SessionRevision, SessionSnapshot, SessionStore, - SessionStoreError, StatusCode, SuccessResponsePayload, TelemetryConfiguration, TelemetryKey, - TelemetryRecord, TransactionApplied, TransportSetRequest, TransportSnapshot, + NonOkResponse, ObservationEnded, ObservationTapKey, PreparedSessionTransaction, ProtocolCodec, + ProtocolQueues, QueueReport, ReliablePayload, ReliableSlot, RequestId, ResponseGetRequest, + ResponseResult, SampleTime, SessionCommitted, SessionEdit, SessionRevision, SessionSnapshot, + SessionStore, SessionStoreError, StatusCode, SuccessResponsePayload, TelemetryConfiguration, + TelemetryKey, TelemetryRecord, TransactionApplied, TransportSetRequest, TransportSnapshot, TransportStateEvent, TypedEventFrame, TypedNonOkResponseFrame, TypedSuccessResponseFrame, }; #[cfg(any(test, feature = "test-support"))] @@ -117,6 +117,9 @@ pub struct ProviderFeatures { pub diagnostics: bool, pub session_events: bool, pub transport_events: bool, + /// `OBSERVATION_ENDED` (#1584): a host that keeps observation subscriptions tells its + /// subscribers when one ends. It rides the session family, so it needs `session_events`. + pub observation_events: bool, } impl ProviderFeatures { @@ -130,6 +133,7 @@ impl ProviderFeatures { diagnostics: true, session_events: true, transport_events: true, + observation_events: true, }; /// Disable every optional provider and event family. pub const NONE: Self = Self { @@ -141,6 +145,7 @@ impl ProviderFeatures { diagnostics: false, session_events: false, transport_events: false, + observation_events: false, }; } @@ -1024,7 +1029,7 @@ pub struct ControllerResponse { struct CapabilitySet { commands: [u16; 12], command_len: u8, - events: [u16; 5], + events: [u16; 6], event_len: u8, flags: CapabilityFlags, } @@ -1189,6 +1194,12 @@ pub struct PreparedStructuralCommand { event_reservations: Option, event_sequence: u64, applied_operations: u32, + /// The one `OBSERVATION_ENDED` event this commit emits after its `SESSION_COMMITTED`, carrying + /// every ended key, with its sequence (#1584 D6a fold). `None` unless the host attached keys. + observation_ended: Option, + /// The reliable-event room held for `observation_ended`, checked and reserved before the + /// host commits anything. + ended_reservations: Option, armed: bool, } @@ -1259,6 +1270,30 @@ pub enum PreparedCommandCommitError { StaleGeneration, } +/// Why [`ProtocolController::attach_observation_ended`] attached nothing. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum AttachObservationEndedError { + /// The token is not this controller's current one. + Token(PreparedCommandCommitError), + /// The reliable event queue has no room for the token's `SESSION_COMMITTED` and the event + /// together: it holds unread events. Nothing changed; a drain makes room. + NoRoom(QueueReport), + /// The event cannot be encoded: its frame is over the endpoint's frame limit, a key is not a + /// stable ID, or the keys are not in strictly ascending order. No drain helps, so the host + /// refuses the transaction for good and never as backpressure. + Unencodable, + /// A token of monitor writes only commits no revision and so emits no event. + MonitorOnly, +} + +impl fmt::Display for AttachObservationEndedError { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(formatter, "{self:?}") + } +} + +impl std::error::Error for AttachObservationEndedError {} + /// Bounded event-egress outcome. The caller always owns output; short output leaves the exact /// pending event intact for a later retry and no partial frame is written. #[derive(Clone, Copy, Debug, Eq, PartialEq)] @@ -1267,6 +1302,10 @@ pub enum EventEgressError { Disabled, /// Prepared reliable diagnostic storage has no free slot. DiagnosticStorageFull, + /// A reliable `OBSERVATION_ENDED` slot names a cell that holds no event. An internal + /// invariant: the commit fills the cell before it queues the slot, and the cell stays until + /// the slot's frame is delivered. Reported, never expected. + ObservationEndedCellMissing, /// A reliable queue rejected the original event without dropping or coalescing it. ReliableQueueFull(QueueReport), /// The typed caller-buffer encoder rejected the requested event/output. @@ -1343,6 +1382,11 @@ pub struct ProtocolController { next_reliable_event_sequence: u64, telemetry_configuration: TelemetryConfiguration, diagnostic_event_slots: Box<[RetainedDiagnosticSlot]>, + /// Prepared storage for the keys of queued `OBSERVATION_ENDED` events (#1584 D6a): one cell + /// more than the reliable queue has slots, because one popped event stays pending in + /// `pending_reliable_event` until its frame reaches the caller. A cell is taken by a commit + /// whose room the queue reserved, so a commit never finds none. + observation_ended_cells: Box<[Option]>, pending_reliable_event: Option, pending_meter_record: Option, pending_counter_record: Option, @@ -1367,6 +1411,14 @@ impl ProtocolController

{ .try_reserve_exact(diagnostic_slots) .map_err(|_| ControllerResourceAllocationError)?; diagnostics.resize_with(diagnostic_slots, || RetainedDiagnosticSlot::Empty); + let ended_cells = diagnostic_slots + .checked_add(1) + .ok_or(ControllerResourceAllocationError)?; + let mut observation_ended_cells = Vec::new(); + observation_ended_cells + .try_reserve_exact(ended_cells) + .map_err(|_| ControllerResourceAllocationError)?; + observation_ended_cells.resize_with(ended_cells, || None); Ok(Self { session, queues, @@ -1377,6 +1429,7 @@ impl ProtocolController

{ next_reliable_event_sequence: 1, telemetry_configuration: retained_telemetry_configuration(capacity)?, diagnostic_event_slots: diagnostics.into_boxed_slice(), + observation_ended_cells: observation_ended_cells.into_boxed_slice(), pending_reliable_event: None, pending_meter_record: None, pending_counter_record: None, @@ -1825,6 +1878,8 @@ impl ProtocolController

{ event_reservations: Some(event_reservations), event_sequence: plan.event_sequence, applied_operations: plan.applied_operations, + observation_ended: None, + ended_reservations: None, armed: true, }) } @@ -2134,6 +2189,68 @@ impl ProtocolController

{ Ok(()) } + /// Reserve room for the one `OBSERVATION_ENDED` event that `prepared`'s commit emits after its + /// `SESSION_COMMITTED`, carrying every key in `ended` (#1584 D6a fold, the acked-batch + /// question), and return the event frame's encoded length. + /// + /// The host knows which subscriptions the transaction ends, the controller does not, so the + /// host calls this after it prepares the token and before it commits anything. `ended` is in + /// strictly ascending key order (the wire's: strip, console before inserts, effect ID, tap + /// ID). The event is encoded for size now, off the commit: a frame over the endpoint's limit is + /// [`AttachObservationEndedError::Unencodable`]. The room is checked together with the + /// `SESSION_COMMITTED` the token already holds, one slot for the event whatever the number of + /// keys: a queue holding unread events that leave less room refuses here, with nothing + /// changed, and the host refuses the transaction as backpressure. After an `Ok` the commit + /// cannot fail for room and allocates nothing. A second call replaces the first. An empty list + /// attaches nothing and returns 0. + /// + /// # Errors + /// + /// [`AttachObservationEndedError`]: the token is another controller's or stale, it is a token of + /// monitor writes (which emits no event), the queue has no room, or the event cannot be + /// encoded. + pub fn attach_observation_ended( + &mut self, + prepared: &mut PreparedStructuralCommand, + ended: Vec, + ) -> Result { + self.check_prepared_structural(prepared) + .map_err(AttachObservationEndedError::Token)?; + // Drop an earlier attachment first, so its room counts as free for this one. + prepared.observation_ended = None; + prepared.ended_reservations = None; + if ended.is_empty() { + return Ok(0); + } + if prepared.prospective_session().is_monitor_only() { + return Err(AttachObservationEndedError::MonitorOnly); + } + let event = ObservationEnded { + // `event_sequence` is the `SESSION_COMMITTED` sequence; this event follows it. + event_sequence: prepared.event_sequence.saturating_add(1), + keys: ended, + }; + // The size of the frame, which an empty output reports without writing one. + let revision = prepared.prospective_session().revision(); + let required = match self.codec.encode_event_frame_into( + &TypedEventFrame { + revision, + payload: EventPayload::ObservationEnded(&event), + }, + &mut [], + ) { + Err(EncodeError::OutputTooSmall { required }) => required, + Err(_) | Ok(_) => return Err(AttachObservationEndedError::Unencodable), + }; + let reservations = self + .queues + .reserve_reliable_events(1) + .map_err(AttachObservationEndedError::NoRoom)?; + prepared.observation_ended = Some(event); + prepared.ended_reservations = Some(reservations); + Ok(required) + } + /// Validate token affinity/currentness, then apply every prepared controller mutation without /// another fallible semantic or capacity decision. /// @@ -2193,6 +2310,29 @@ impl ProtocolController

{ ), ); self.next_reliable_event_sequence = prepared.event_sequence.saturating_add(1); + // #1584 D6a fold: the one event of the subscriptions this commit ended follows it. Its + // room was reserved before the host committed, and the event was built and sized + // then, so this moves it and allocates nothing. + if let Some(event) = prepared.observation_ended.take() { + let mut reservations = prepared + .ended_reservations + .take() + .expect("an attached observation-ended event holds its reserved room"); + let cell = self + .observation_ended_cells + .iter() + .position(Option::is_none) + .expect("a reserved reliable slot has a free observation-ended cell"); + self.next_reliable_event_sequence = event.event_sequence.saturating_add(1); + self.observation_ended_cells[cell] = Some(event); + self.queues.commit_reserved_reliable_event( + &mut reservations, + ReliableSlot::observation_ended( + commit.revision, + u32::try_from(cell).expect("cell count is a queue length"), + ), + ); + } } let response = prepared .response @@ -2792,6 +2932,12 @@ impl ProtocolController

{ { *storage = RetainedDiagnosticSlot::Empty; } + if let ReliablePayload::ObservationEnded { cell } = slot.payload + && let Ok(index) = usize::try_from(cell) + && let Some(storage) = self.observation_ended_cells.get_mut(index) + { + *storage = None; + } self.pending_reliable_event = None; } result.map(Some) @@ -2887,6 +3033,20 @@ impl ProtocolController

{ }, output, ), + ReliablePayload::ObservationEnded { cell } => { + let event = usize::try_from(cell) + .ok() + .and_then(|index| self.observation_ended_cells.get(index)) + .and_then(Option::as_ref) + .ok_or(EventEgressError::ObservationEndedCellMissing)?; + self.codec.encode_event_frame_into( + &TypedEventFrame { + revision: slot.revision, + payload: EventPayload::ObservationEnded(event), + }, + output, + ) + } ReliablePayload::Diagnostic { diagnostic_slot } => { let index = usize::try_from(diagnostic_slot) .map_err(|_| EventEgressError::DiagnosticStorageFull)?; @@ -3485,13 +3645,14 @@ impl ProtocolController

{ command_len += 1; } } - let candidates = [0x8001_u16, 0x8010, 0x8020, 0x8021, 0x8030]; - let mut events = [0_u16; 5]; + let candidates = [0x8001_u16, 0x8010, 0x8011, 0x8020, 0x8021, 0x8030]; + let mut events = [0_u16; 6]; let mut event_len = 0_usize; for id in candidates { let enabled = match id { 0x8001 => session_transactions, 0x8010 => features.transport && features.transport_events, + 0x8011 => session_transactions && features.observation_events, 0x8020 => features.meters, 0x8021 => features.counters, 0x8030 => features.diagnostics, diff --git a/crates/protocol/src/controller/tests.rs b/crates/protocol/src/controller/tests.rs index b69ceeee5..569d04ffb 100644 --- a/crates/protocol/src/controller/tests.rs +++ b/crates/protocol/src/controller/tests.rs @@ -198,6 +198,16 @@ fn controller_over( model: session::SessionModel, replay_entries: usize, current_sample: SampleTime, +) -> ProtocolController { + controller_with_event_slots(model, replay_entries, current_sample, 1) +} + +/// [`controller_over`] with `event_slots` slots in the reliable event queue. +fn controller_with_event_slots( + model: session::SessionModel, + replay_entries: usize, + current_sample: SampleTime, + event_slots: usize, ) -> ProtocolController { let session = SessionStore::new( model, @@ -215,7 +225,7 @@ fn controller_over( control_command_slots: NonZeroUsize::new(1).expect("one"), control_command_bytes: NonZeroUsize::new(64).expect("bytes"), reliable_response_slots: NonZeroUsize::new(1).expect("response"), - reliable_event_slots: NonZeroUsize::new(1).expect("event"), + reliable_event_slots: NonZeroUsize::new(event_slots).expect("event"), telemetry_slots: NonZeroUsize::new(1).expect("telemetry"), }) .expect("queues"); @@ -1122,6 +1132,256 @@ fn structural_prepare_is_invisible_until_commit_and_byte_identical_to_one_call() assert_eq!(endpoint.replay().len(), 1); } +fn ended_key(strip: &str, tap_id: u32) -> crate::ObservationTapKey { + crate::ObservationTapKey { + strip_id: strip.to_owned(), + section: crate::ObservationSection::Inserts, + effect_id: "comp".to_owned(), + tap_id, + } +} + +/// The reliable events of `endpoint`, decoded. +fn drain_events( + endpoint: &mut ProtocolController, +) -> Vec<(u64, crate::DecodedEventPayload<'static>)> { + let codec = ProtocolCodec::default(); + let mut events = Vec::new(); + loop { + let mut frame = vec![0_u8; 1024]; + let Some(length) = endpoint + .dequeue_reliable_event_frame_into(&mut frame) + .expect("an event") + else { + return events; + }; + let mut fields = [0_u16; 64]; + let decoded = codec + .decode_typed_event(&frame[..length], &mut DecodeScratch::new(&mut fields)) + .expect("a decodable event"); + let payload = match decoded.payload { + crate::DecodedEventPayload::SessionCommitted(value) => { + crate::DecodedEventPayload::SessionCommitted(value) + } + crate::DecodedEventPayload::ObservationEnded(value) => { + crate::DecodedEventPayload::ObservationEnded(value) + } + _ => panic!("an unexpected event"), + }; + events.push((decoded.header.revision.0, payload)); + } +} + +/// #1584 D6a fold (root ruling A): the keys a host attaches to a prepared transaction travel in +/// ONE event, which the commit emits right after `SESSION_COMMITTED`, at the committing revision, +/// with the next endpoint sequence, whatever the number of keys and the number of lane slots. +/// +/// Red if the ends are split into several events (the second and later one fails to fit a +/// two-slot lane), precede the commit event, carry the previous revision, drop or reorder a key, +/// or skip or repeat a sequence. +#[test] +fn attached_observation_ends_follow_their_commit_in_one_event() { + let transaction = [SessionEdit::SetSessionId { + session_id: session::StableId::parse("ending-session").expect("ID"), + }]; + let input = full_command( + 1, + ExpectedRevision::Exact(SessionRevision(7)), + crate::CommandPayload::SessionTransactionApply(&transaction), + ); + // Two slots: the commit's event and the one event of the ends, however many keys it holds. + let mut endpoint = controller_with_event_slots( + parse_session_json(EXAMPLE).expect("fixture"), + 8, + SampleTime(0), + 2, + ); + let mut prepared = match endpoint + .prepare_command_frame(&input, &mut DecodeScratch::new(&mut [0_u16; 1024]), 2048) + .expect("prepare") + { + PreparedCommandFrame::Structural(prepared) => *prepared, + PreparedCommandFrame::Immediate(_) => panic!("valid transaction was immediate"), + }; + let keys: Vec<_> = (0..20_u32).map(|tap| ended_key("a", tap)).collect(); + let frame_bytes = endpoint + .attach_observation_ended(&mut prepared, keys.clone()) + .expect("room for the commit and one event of twenty keys"); + endpoint + .commit_prepared_structural(prepared, EditPath::Rebuild) + .expect("commit"); + let mut frame = vec![0_u8; 2048]; + let mut events = Vec::new(); + while let Some(length) = endpoint + .dequeue_reliable_event_frame_into(&mut frame) + .expect("an event") + { + events.push(frame[..length].to_vec()); + } + assert_eq!(events.len(), 2, "the commit and one event"); + assert_eq!(events[1].len(), frame_bytes, "the size attach reported"); + let mut fields = [0_u16; 256]; + let codec = ProtocolCodec::default(); + let committed = codec + .decode_typed_event(&events[0], &mut DecodeScratch::new(&mut fields)) + .expect("commit event"); + let crate::DecodedEventPayload::SessionCommitted(commit) = committed.payload else { + panic!("the commit event comes first"); + }; + assert_eq!(committed.header.revision.0, 8); + let ended = codec + .decode_typed_event(&events[1], &mut DecodeScratch::new(&mut fields)) + .expect("end event"); + assert_eq!(ended.header.revision.0, 8, "the committing revision"); + let crate::DecodedEventPayload::ObservationEnded(ended) = ended.payload else { + panic!("one end event follows the commit"); + }; + assert_eq!(ended.keys, keys); + assert_eq!(ended.event_sequence, commit.event_sequence + 1); + // The next commit's sequence continues after the end. + let next = full_command( + 2, + ExpectedRevision::Exact(SessionRevision(8)), + crate::CommandPayload::SessionTransactionApply(&[SessionEdit::SetSessionId { + session_id: session::StableId::parse("again-session").expect("ID"), + }]), + ); + let mut output = [0_u8; 2048]; + endpoint + .process_command_frame_into( + &next, + &mut DecodeScratch::new(&mut [0_u16; 1024]), + &mut output, + ) + .expect("next commit"); + let events = drain_events(&mut endpoint); + let crate::DecodedEventPayload::SessionCommitted(second) = &events[0].1 else { + panic!("commit event"); + }; + assert_eq!(second.event_sequence, commit.event_sequence + 2); +} + +/// #1584 D6a fold, the acked-batch question: the room for the one end event is checked together +/// with the commit's own event, and only unread events take it away. A lane holding an unread event +/// refuses the attachment and changes nothing; a token dropped after a successful attachment +/// releases the room it held; an event no frame can carry is refused for good, not as no room. +/// +/// Red if an attachment succeeds that the lane cannot honour (the commit would drop the end after +/// the ack), if the number of keys changes the room needed, if a refusal leaves a half +/// reservation, if a dropped token keeps the room, or if an unencodable event reports no room (a +/// retry that can never succeed). +#[test] +fn the_end_needs_room_with_the_commit_and_a_dropped_token_releases_it() { + let transaction = [SessionEdit::SetSessionId { + session_id: session::StableId::parse("ending-session").expect("ID"), + }]; + let input = full_command( + 1, + ExpectedRevision::Exact(SessionRevision(7)), + crate::CommandPayload::SessionTransactionApply(&transaction), + ); + // Two slots: the commit and the end fit; an unread event leaves too little. + let mut endpoint = controller_with_event_slots( + parse_session_json(EXAMPLE).expect("fixture"), + 8, + SampleTime(0), + 2, + ); + let prepare = |endpoint: &mut ProtocolController| match endpoint + .prepare_command_frame(&input, &mut DecodeScratch::new(&mut [0_u16; 1024]), 2048) + .expect("prepare") + { + PreparedCommandFrame::Structural(prepared) => *prepared, + PreparedCommandFrame::Immediate(_) => panic!("valid transaction was immediate"), + }; + // Empty lane: any number of keys fits, in one event. + let mut token = prepare(&mut endpoint); + let many: Vec<_> = (0..100_u32).map(|tap| ended_key("a", tap)).collect(); + endpoint + .attach_observation_ended(&mut token, many) + .expect("a hundred keys need one slot"); + // A second attachment replaces the first, so its room counts as free. + endpoint + .attach_observation_ended(&mut token, vec![ended_key("c", 1), ended_key("d", 1)]) + .expect("a replacement fits too"); + drop(token); + // The dropped token released everything: a fresh one can reserve the whole lane again. + let mut token = prepare(&mut endpoint); + endpoint + .attach_observation_ended(&mut token, vec![ended_key("a", 1)]) + .expect("the room came back"); + endpoint + .commit_prepared_structural(token, EditPath::Rebuild) + .expect("commit"); + assert_eq!(drain_events(&mut endpoint).len(), 2); + + // One unread event in the lane: the commit takes the other slot and the end has none. + let next = full_command( + 2, + ExpectedRevision::Exact(SessionRevision(8)), + crate::CommandPayload::SessionTransactionApply(&[SessionEdit::SetSessionId { + session_id: session::StableId::parse("again-session").expect("ID"), + }]), + ); + let mut output = [0_u8; 2048]; + endpoint + .process_command_frame_into( + &next, + &mut DecodeScratch::new(&mut [0_u16; 1024]), + &mut output, + ) + .expect("a commit that leaves an unread event"); + let again = full_command( + 3, + ExpectedRevision::Exact(SessionRevision(9)), + crate::CommandPayload::SessionTransactionApply(&transaction), + ); + let mut token = match endpoint + .prepare_command_frame(&again, &mut DecodeScratch::new(&mut [0_u16; 1024]), 2048) + .expect("prepare") + { + PreparedCommandFrame::Structural(prepared) => *prepared, + PreparedCommandFrame::Immediate(_) => panic!("valid transaction was immediate"), + }; + assert!(matches!( + endpoint.attach_observation_ended(&mut token, vec![ended_key("a", 1)]), + Err(crate::AttachObservationEndedError::NoRoom(_)) + )); + // Out of order keys cannot be encoded: refused for good, with the lane untouched. + drop(token); + drain_events(&mut endpoint); + let mut token = prepare_at(&mut endpoint, 9, &transaction); + assert!(matches!( + endpoint.attach_observation_ended(&mut token, vec![ended_key("b", 1), ended_key("a", 1)]), + Err(crate::AttachObservationEndedError::Unencodable) + )); + // So is a list no frame can hold: the controller's frame limit is the codec's. + let oversized: Vec<_> = (0..200_000_u32).map(|tap| ended_key("a", tap)).collect(); + assert!(matches!( + endpoint.attach_observation_ended(&mut token, oversized), + Err(crate::AttachObservationEndedError::Unencodable) + )); +} + +fn prepare_at( + endpoint: &mut ProtocolController, + revision: u64, + transaction: &[SessionEdit], +) -> PreparedStructuralCommand { + let input = full_command( + 7, + ExpectedRevision::Exact(SessionRevision(revision)), + crate::CommandPayload::SessionTransactionApply(transaction), + ); + match endpoint + .prepare_command_frame(&input, &mut DecodeScratch::new(&mut [0_u16; 1024]), 2048) + .expect("prepare") + { + PreparedCommandFrame::Structural(prepared) => *prepared, + PreparedCommandFrame::Immediate(_) => panic!("valid transaction was immediate"), + } +} + fn transaction_edit_path(frame: &[u8]) -> EditPath { let mut fields = [0_u16; 64]; match ProtocolCodec::default() diff --git a/crates/protocol/src/lib.rs b/crates/protocol/src/lib.rs index d3fa2b492..13cf885af 100644 --- a/crates/protocol/src/lib.rs +++ b/crates/protocol/src/lib.rs @@ -25,12 +25,13 @@ pub use message_wire::{ ControlStatePage, ControlStateRecord, ControlTaper, CounterId, CounterSnapshot, CounterSnapshotRef, CounterValue, CountersRequest, DecodedCapabilities, DecodedMeterBatch, Diagnostic, DiagnosticEvent, DiagnosticSeverity, DiagnosticsPage, DiagnosticsRequest, EditPath, - EnumChoice, MeterBatch, MeterComponent, MeterRecord, NonOkResponse, ParameterChannel, - ParameterRack, ParameterUnit, PathSegment, RESPONSE_BYPASS_CONTROL, ResponseConfiguration, - ResponseCurves, ResponseGetRequest, ResponseGrid, ResponseOverride, ResponseResult, - ResponseTarget, SessionCommitted, SessionSnapshot, SessionSnapshotRequest, - TelemetryConfiguration, TransactionApplied, TransportSetRequest, TransportSnapshot, - TransportState, TransportStateEvent, response_result_encoded_len, + EnumChoice, MeterBatch, MeterComponent, MeterRecord, NonOkResponse, ObservationEnded, + ObservationSection, ObservationTapKey, ParameterChannel, ParameterRack, ParameterUnit, + PathSegment, RESPONSE_BYPASS_CONTROL, ResponseConfiguration, ResponseCurves, + ResponseGetRequest, ResponseGrid, ResponseOverride, ResponseResult, ResponseTarget, + SessionCommitted, SessionSnapshot, SessionSnapshotRequest, TelemetryConfiguration, + TransactionApplied, TransportSetRequest, TransportSnapshot, TransportState, + TransportStateEvent, response_result_encoded_len, }; /// The control core's address types, to name the address of a [`SessionEdit::SetControl`]. pub mod control_core { @@ -79,8 +80,8 @@ pub const MAXIMUM_TELEMETRY_PAGE_RECORDS: usize = 256; #[cfg(any(test, feature = "test-support"))] pub use controller::MockProvider; pub use controller::{ - CommandFrameProcessError, CommittedCommandFrame, ControlCommand, ControlProvider, - ControllerRequest, ControllerResourceAllocationError, ControllerResponse, + AttachObservationEndedError, CommandFrameProcessError, CommittedCommandFrame, ControlCommand, + ControlProvider, ControllerRequest, ControllerResourceAllocationError, ControllerResponse, ControllerRetainedCapacity, EventEgressError, ParameterProviderError, PreparedCommandCommitError, PreparedCommandFrame, PreparedImmediateCommandFrame, PreparedStructuralCommand, ProtocolController, ProtocolControllerConfig, ProviderFeatures, diff --git a/crates/protocol/src/message_wire.rs b/crates/protocol/src/message_wire.rs index e7052cc86..887df63b2 100644 --- a/crates/protocol/src/message_wire.rs +++ b/crates/protocol/src/message_wire.rs @@ -886,6 +886,75 @@ pub struct TransportStateEvent { pub origin_request_id: Option, } +/// The effect section an [`ObservationEnded`] names: the `RackName` codes of the two sections that +/// hold an effect (inserts `2`, console `5`). +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +#[repr(u8)] +pub enum ObservationSection { + /// One of the strip's inserts. + Inserts = 2, + /// A session console slot. + Console = 5, +} + +impl ObservationSection { + /// The section's stable wire code. + #[must_use] + pub const fn code(self) -> u8 { + self as u8 + } + + /// The section a wire code names, or `None` for any other. + #[must_use] + pub const fn from_code(code: u8) -> Option { + match code { + 2 => Some(Self::Inserts), + 5 => Some(Self::Console), + _ => None, + } + } +} + +/// The stable key of one observation tap, as the wire names it: the strip, the section, the +/// insert's effect ID or the console slot ID, and the effect-local tap ID. +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct ObservationTapKey { + /// The strip (a track or a submix) the tap belongs to. + pub strip_id: String, + /// Whether the tap's effect is an insert or a console slot. + pub section: ObservationSection, + /// The insert's effect ID or the console slot ID. + pub effect_id: String, + /// The effect-local tap ID. + pub tap_id: u32, +} + +impl ObservationTapKey { + /// The order keys travel in on the wire: the strip, then console slots before inserts, then + /// the effect ID, then the tap ID. It is the order of the host's binding table. + fn wire_order(&self) -> (&str, bool, &str, u32) { + ( + &self.strip_id, + self.section == ObservationSection::Inserts, + &self.effect_id, + self.tap_id, + ) + } +} + +/// Typed reliable `OBSERVATION_ENDED` event payload (#1584 D6a, folded to one event per commit): the +/// taps a host had armed no longer exist, because a transaction removed their strips, inserts or +/// console slots, or reused the IDs for effects without those taps. The event's revision is the +/// revision that removed them, and `keys` is every tap that commit ended. +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct ObservationEnded { + /// Endpoint-monotonic reliable event sequence. + pub event_sequence: u64, + /// The taps that ended: at least one, in strictly ascending wire order (the strip, console + /// slots before inserts, the effect ID, the tap ID), so no key repeats. + pub keys: Vec, +} + /// Frozen meter component registry for one fixed telemetry record. #[derive(Clone, Copy, Debug, Eq, PartialEq)] #[repr(u16)] @@ -1685,6 +1754,80 @@ impl ProtocolCodec { }) } + /// Return the exact caller-buffer size of a reliable typed observation-ended event. + pub fn encoded_observation_ended_len( + &self, + value: &ObservationEnded, + ) -> Result { + let mut sink = CountSink::new(self.limits()); + write_observation_ended(&mut sink, value)?; + checked_sink_len(self, &mut sink) + } + + /// Encode a reliable typed observation-ended event payload into caller output. + pub fn encode_observation_ended( + &self, + value: &ObservationEnded, + output: &mut [u8], + ) -> Result { + let required = self.encoded_observation_ended_len(value)?; + if output.len() < required { + return Err(EncodeError::OutputTooSmall { required }); + } + let mut writer = SliceSink::new(output, self.limits()); + write_observation_ended(&mut writer, value)?; + checked_writer_len(&writer, required) + } + + /// Strictly decode a reliable typed observation-ended event payload. Every ID must be a stable + /// ID, every section an effect section, and the keys at least one and strictly ascending. + pub fn decode_observation_ended( + &self, + payload: &[u8], + count: u32, + ) -> Result { + let message = Message::top_level(payload, count, self.limits())? + .schema_spec(&schema::observation_ended::SPEC)?; + let raw_keys = values_spec!(message, schema::observation_ended::KEY)?; + let mut keys = Vec::with_capacity(raw_keys.count()); + for raw in values_spec!(message, schema::observation_ended::KEY)? { + let fields = Message::nested_at_depth(raw, self.limits(), 1)?; + let fields = fields.schema_spec(&schema::observation_tap_key::SPEC)?; + let id = |spec: schema::FieldSpec| -> Result { + let text = read_string(self, one_spec!(fields, spec)?)?; + session::StableId::parse(text).ok_or(DecodeError::InvalidField)?; + Ok(text.to_owned()) + }; + let key = ObservationTapKey { + strip_id: id(schema::observation_tap_key::STRIP_ID)?, + section: ObservationSection::from_code(read_u8(one_spec!( + fields, + schema::observation_tap_key::SECTION + )?)?) + .ok_or(DecodeError::InvalidField)?, + effect_id: id(schema::observation_tap_key::EFFECT_ID)?, + tap_id: read_u32(one_spec!(fields, schema::observation_tap_key::TAP_ID)?)?, + }; + if keys + .last() + .is_some_and(|last: &ObservationTapKey| last.wire_order() >= key.wire_order()) + { + return Err(DecodeError::InvalidField); + } + keys.push(key); + } + if keys.is_empty() { + return Err(DecodeError::InvalidField); + } + Ok(ObservationEnded { + event_sequence: read_u64(one_spec!( + message, + schema::observation_ended::EVENT_SEQUENCE + )?)?, + keys, + }) + } + /// Return the exact caller-buffer bytes required for one fixed-record meter batch. pub fn encoded_meter_batch_len(&self, value: MeterBatch<'_>) -> Result { let mut sink = CountSink::new(self.limits()); @@ -2482,6 +2625,59 @@ pub(crate) fn write_transport_state_event( Ok(()) } +pub(crate) fn write_observation_ended( + sink: &mut dyn Sink, + value: &ObservationEnded, +) -> Result<(), EncodeError> { + if value.keys.is_empty() + || value.keys.iter().any(|key| { + session::StableId::parse(&key.strip_id).is_none() + || session::StableId::parse(&key.effect_id).is_none() + }) + || !value + .keys + .windows(2) + .all(|pair| pair[0].wire_order() < pair[1].wire_order()) + { + return Err(EncodeError::InvalidField); + } + sink.check_field_count( + schema::observation_ended::SPEC + .field_count(&[(schema::observation_ended::KEY, value.keys.len())])?, + )?; + write_spec!( + sink, + schema::observation_ended::EVENT_SEQUENCE, + &value.event_sequence.to_le_bytes() + )?; + for key in &value.keys { + sink.nested_spec(schema::observation_ended::KEY, &mut |sink| { + sink.message_header(schema::observation_tap_key::SPEC.field_count(&[])?)?; + write_spec!( + sink, + schema::observation_tap_key::STRIP_ID, + key.strip_id.as_bytes() + )?; + write_spec!( + sink, + schema::observation_tap_key::SECTION, + &[key.section.code()] + )?; + write_spec!( + sink, + schema::observation_tap_key::EFFECT_ID, + key.effect_id.as_bytes() + )?; + write_spec!( + sink, + schema::observation_tap_key::TAP_ID, + &key.tap_id.to_le_bytes() + ) + })?; + } + Ok(()) +} + pub(crate) fn write_meter_batch( sink: &mut dyn Sink, value: MeterBatch<'_>, @@ -3862,6 +4058,8 @@ fn check_capabilities_invariants(value: CapabilityInvariantView<'_>) -> Result<( || ((value.flags.0 & (1 << 13) != 0) != (has_command(0x0008) && has_event(0x8010))) || ((value.flags.0 & (1 << 14) != 0) != has_command(0x000e)) || (has_command(0x0003) != has_event(0x8001)) + // `OBSERVATION_ENDED` rides the session family (#1584): it follows a commit. + || (has_event(0x8011) && !session_family) || (has_command(0x000c) != has_command(0x000d)) || (has_command(0x0008) != has_event(0x8010)) || (has_command(0x000a) != has_event(0x8021)) @@ -3874,7 +4072,7 @@ fn check_capabilities_invariants(value: CapabilityInvariantView<'_>) -> Result<( fn allocated_id(id: u16, event: bool) -> bool { if event { // `0x8002` is retired: never allocated again, never advertised (#1628). - matches!(id, 0x8001 | 0x8010 | 0x8020 | 0x8021 | 0x8030) + matches!(id, 0x8001 | 0x8010 | 0x8011 | 0x8020 | 0x8021 | 0x8030) && !crate::wire::is_retired_message_id(id) } else { // `0x0004`, `0x0005` and `0x0006` are retired: never allocated again, never advertised. diff --git a/crates/protocol/src/message_wire/tests.rs b/crates/protocol/src/message_wire/tests.rs index c9ce6c827..63f8565ef 100644 --- a/crates/protocol/src/message_wire/tests.rs +++ b/crates/protocol/src/message_wire/tests.rs @@ -2040,6 +2040,154 @@ fn b3a_transport_goldens_truncation_and_direct_codec_are_strict() { ); } +fn ended_key( + strip: &str, + section: ObservationSection, + effect: &str, + tap: u32, +) -> ObservationTapKey { + ObservationTapKey { + strip_id: strip.to_owned(), + section, + effect_id: effect.to_owned(), + tap_id: tap, + } +} + +/// #1584 (root ruling A): `OBSERVATION_ENDED` carries every ended key of one commit and round-trips +/// them exactly, every truncation fails, and the codec refuses on both sides an ID that is not a +/// stable ID, a section that holds no effect, no key, and keys out of strict ascending order +/// (console slots before inserts, so a duplicate is out of order too). +/// +/// Red if a key loses a field or its place in the round trip, a builtins or own section is taken +/// for an effect section, an empty event decodes, or a list the decoder cannot read as one ordered +/// set is encodable. +#[test] +fn observation_ended_round_trips_every_key_and_refuses_anything_else() { + use ObservationSection::{Console, Inserts}; + let codec = codec(); + let event = ObservationEnded { + event_sequence: 7, + keys: vec![ + ended_key("bass", Inserts, "comp", 1), + ended_key("vocal", Console, "glue", 3), + ended_key("vocal", Inserts, "comp", 1), + ended_key("vocal", Inserts, "comp", 2), + ended_key("vocal", Inserts, "eq", 1), + ], + }; + let mut bytes = vec![0_u8; 1024]; + let length = codec + .encode_observation_ended(&event, &mut bytes) + .expect("encode"); + assert_eq!( + length, + codec.encoded_observation_ended_len(&event).expect("length") + ); + bytes.truncate(length); + // The field count is the sequence and one key message per key. + let count = 1 + event.keys.len() as u32; + assert_eq!( + codec.decode_observation_ended(&bytes, count), + Ok(event.clone()) + ); + for end in 0..bytes.len() { + assert!( + codec + .decode_observation_ended(&bytes[..end], count) + .is_err(), + "{end}" + ); + } + // A single key is the smallest event; the section is on the wire. + let one = |section| ObservationEnded { + event_sequence: 7, + keys: vec![ended_key("vocal", section, "glue", 3)], + }; + let mut other = vec![0_u8; 256]; + let mut encoded = Vec::new(); + for section in [Console, Inserts] { + let written = codec + .encode_observation_ended(&one(section), &mut other) + .expect("encode"); + assert_eq!( + codec.decode_observation_ended(&other[..written], 2), + Ok(one(section)) + ); + encoded.push(other[..written].to_vec()); + } + assert_ne!(encoded[0], encoded[1]); + + // Not a stable ID, no key, or out of order: encode refuses it. + for bad in ["", "Vocal", "9lives", "has space"] { + let mut invalid = one(Console); + invalid.keys[0].strip_id = bad.to_owned(); + assert_eq!( + codec.encode_observation_ended(&invalid, &mut other), + Err(EncodeError::InvalidField), + "{bad:?}" + ); + } + let mut empty = one(Console); + empty.keys.clear(); + assert_eq!( + codec.encode_observation_ended(&empty, &mut other), + Err(EncodeError::InvalidField) + ); + for keys in [ + // Inserts before a console slot of the same strip. + vec![ + ended_key("vocal", Inserts, "comp", 1), + ended_key("vocal", Console, "glue", 3), + ], + // A repeated key. + vec![ + ended_key("vocal", Console, "glue", 3), + ended_key("vocal", Console, "glue", 3), + ], + // A lower tap after a higher one. + vec![ + ended_key("vocal", Inserts, "comp", 2), + ended_key("vocal", Inserts, "comp", 1), + ], + ] { + assert_eq!( + codec.encode_observation_ended( + &ObservationEnded { + event_sequence: 7, + keys + }, + &mut other + ), + Err(EncodeError::InvalidField) + ); + } + + // Decode refuses what encode would: the section byte of the first key is 2 or 5, anything else + // names no effect. (A key message's section field is its second field, id 2, type U8.) + let written = codec + .encode_observation_ended(&one(Console), &mut other) + .expect("encode"); + let mut section_values = Vec::new(); + for (index, window) in other[..written].windows(2).enumerate() { + // Field header: id (u16 LE) then the wire type byte; a U8 section is `2, 0, 1`. + if window == [2, 0] && other[..written].get(index + 2) == Some(&1) { + section_values.push(index + 8); + } + } + let value = *section_values.last().expect("the section field header"); + assert_eq!(other[value], 5, "console"); + for code in [0_u8, 1, 3, 4, 6] { + let mut forged = other[..written].to_vec(); + forged[value] = code; + assert_eq!( + codec.decode_observation_ended(&forged, 2), + Err(DecodeError::InvalidField), + "section {code}" + ); + } +} + /// Issue #1351 gate 4: every registered counter ID, the newest `TRANSITION_REPREPARE_REFUSALS` /// included, round-trips through the telemetry configuration; the next ID and the retired IDs /// 2 and 11 to 14 (#1629) are refused. diff --git a/crates/protocol/src/queue.rs b/crates/protocol/src/queue.rs index 755e120cf..685a98033 100644 --- a/crates/protocol/src/queue.rs +++ b/crates/protocol/src/queue.rs @@ -70,6 +70,13 @@ pub enum ReliablePayload { /// Originating transport-set command, when any. origin_request_id: Option, }, + /// A reference to a reliable `OBSERVATION_ENDED` event (#1584 D6a). The event's key is + /// variable-length text, so it stays in prepared endpoint storage until its complete event + /// frame has reached caller output, as a diagnostic does. + ObservationEnded { + /// Index into the endpoint's bounded observation-ended store. + cell: u32, + }, /// A bounded controller-owned typed diagnostic reference. The diagnostic itself remains in /// prepared endpoint storage until its complete event frame has reached caller output. Diagnostic { @@ -114,6 +121,18 @@ impl ReliableSlot { } } + /// Construct a reliable `OBSERVATION_ENDED` event referencing prepared storage `cell`, with + /// zero outer-header request ID. `revision` is the revision that removed the tap. + #[must_use] + pub const fn observation_ended(revision: SessionRevision, cell: u32) -> Self { + Self { + header: ReliableHeader::Event, + revision, + message_id: MessageId::ObservationEnded, + payload: ReliablePayload::ObservationEnded { cell }, + } + } + /// Construct a typed reliable `TRANSPORT_STATE` event with zero outer-header request ID. #[must_use] pub const fn transport_state( diff --git a/crates/protocol/src/schema.rs b/crates/protocol/src/schema.rs index 0eb488f9a..40a8ebd21 100644 --- a/crates/protocol/src/schema.rs +++ b/crates/protocol/src/schema.rs @@ -628,6 +628,34 @@ pub(crate) mod transport_state_event { }; } +/// One ended tap's stable key inside `OBSERVATION_ENDED` (#1584 D6a fold). +pub(crate) mod observation_tap_key { + use super::*; + pub(crate) const STRIP_ID: FieldSpec = FieldSpec::req(1, Wire::Utf8); + /// The section: inserts `2` or console `5`, the `RackName` codes. + pub(crate) const SECTION: FieldSpec = FieldSpec::req(2, Wire::U8); + /// The insert's effect ID or the console slot ID. + pub(crate) const EFFECT_ID: FieldSpec = FieldSpec::req(3, Wire::Utf8); + pub(crate) const TAP_ID: FieldSpec = FieldSpec::req(4, Wire::U32); + pub(crate) static SPEC: MessageSpec = MessageSpec { + name: "ObservationTapKey", + fields: &[STRIP_ID, SECTION, EFFECT_ID, TAP_ID], + }; +} + +/// `OBSERVATION_ENDED` (#1584): the stable keys of every tap whose subscription one commit ended, +/// one repeated `key` message each, at least one, in strictly ascending key order. The revision +/// that removed the taps' effects is the event header's. +pub(crate) mod observation_ended { + use super::*; + pub(crate) const EVENT_SEQUENCE: FieldSpec = FieldSpec::req(1, Wire::U64); + pub(crate) const KEY: FieldSpec = FieldSpec::msg(2, true, true, &observation_tap_key::SPEC); + pub(crate) static SPEC: MessageSpec = MessageSpec { + name: "ObservationEnded", + fields: &[EVENT_SEQUENCE, KEY], + }; +} + pub(crate) mod meter_batch { use super::*; pub(crate) const OBSERVED_SAMPLE: FieldSpec = FieldSpec::req(1, Wire::U64); @@ -1772,6 +1800,8 @@ mod tests { &transport_set::SPEC, &transport_snapshot::SPEC, &transport_state_event::SPEC, + &observation_tap_key::SPEC, + &observation_ended::SPEC, &meter_batch::SPEC, &telemetry_configuration::SPEC, &counters_request::SPEC, diff --git a/crates/protocol/src/typed_frame.rs b/crates/protocol/src/typed_frame.rs index 7cbe99c38..8b66a4b46 100644 --- a/crates/protocol/src/typed_frame.rs +++ b/crates/protocol/src/typed_frame.rs @@ -10,10 +10,10 @@ use crate::{ CounterSnapshot, CounterSnapshotRef, CountersRequest, DecodeError, DecodeScratch, DecodedCapabilities, DecodedMeterBatch, Diagnostic, DiagnosticEvent, DiagnosticsPage, DiagnosticsRequest, EncodeError, EventHeader, FrameKind, MessageId, MeterBatch, NonOkResponse, - OUTER_HEADER_BYTES, ProtocolCodec, RequestId, ResponseGetRequest, ResponseHeader, - ResponseResult, SessionCommitted, SessionEdit, SessionRevision, SessionSnapshot, - SessionSnapshotRequest, StatusCode, TelemetryConfiguration, TransactionApplied, - TransportSetRequest, TransportSnapshot, TransportStateEvent, + OUTER_HEADER_BYTES, ObservationEnded, ProtocolCodec, RequestId, ResponseGetRequest, + ResponseHeader, ResponseResult, SessionCommitted, SessionEdit, SessionRevision, + SessionSnapshot, SessionSnapshotRequest, StatusCode, TelemetryConfiguration, + TransactionApplied, TransportSetRequest, TransportSnapshot, TransportStateEvent, }; #[cfg(test)] @@ -99,6 +99,7 @@ pub struct TypedNonOkResponseFrame<'a> { pub enum EventPayload<'a> { SessionCommitted(SessionCommitted), TransportState(TransportStateEvent), + ObservationEnded(&'a ObservationEnded), MeterBatch(MeterBatch<'a>), CounterSnapshot(CounterSnapshotRef<'a>), Diagnostic(&'a Diagnostic), @@ -168,6 +169,7 @@ pub enum DecodedTypedResponseFrame<'a> { pub enum DecodedEventPayload<'a> { SessionCommitted(SessionCommitted), TransportState(TransportStateEvent), + ObservationEnded(ObservationEnded), MeterBatch(DecodedMeterBatch<'a>), CounterSnapshot(CounterSnapshot), Diagnostic(DiagnosticEvent), @@ -221,6 +223,7 @@ impl<'a> EventPayload<'a> { match self { Self::SessionCommitted(_) => MessageId::SessionCommitted, Self::TransportState(_) => MessageId::TransportState, + Self::ObservationEnded(_) => MessageId::ObservationEnded, Self::MeterBatch(_) => MessageId::MeterBatch, Self::CounterSnapshot(_) => MessageId::CounterSnapshot, Self::Diagnostic(_) => MessageId::Diagnostic, @@ -529,6 +532,9 @@ impl ProtocolCodec { MessageId::TransportState => DecodedEventPayload::TransportState( self.decode_transport_state_event(decoded.payload, header.tlv_count)?, ), + MessageId::ObservationEnded => DecodedEventPayload::ObservationEnded( + self.decode_observation_ended(decoded.payload, header.tlv_count)?, + ), MessageId::MeterBatch => DecodedEventPayload::MeterBatch( self.decode_meter_batch(decoded.payload, header.tlv_count)?, ), @@ -728,6 +734,9 @@ fn write_event_payload( EventPayload::TransportState(value) => { crate::message_wire::write_transport_state_event(sink, *value) } + EventPayload::ObservationEnded(value) => { + crate::message_wire::write_observation_ended(sink, value) + } EventPayload::MeterBatch(value) => crate::message_wire::write_meter_batch(sink, *value), EventPayload::CounterSnapshot(value) => { crate::message_wire::write_counter_snapshot(sink, *value) diff --git a/crates/protocol/src/wire.rs b/crates/protocol/src/wire.rs index 1d0e6c2f7..f07da3b32 100644 --- a/crates/protocol/src/wire.rs +++ b/crates/protocol/src/wire.rs @@ -122,6 +122,9 @@ pub enum MessageId { SessionCommitted = 0x8001, /// `TRANSPORT_STATE`. TransportState = 0x8010, + /// `OBSERVATION_ENDED` (#1584): a host-local observation subscription ended because its + /// effect left the session. + ObservationEnded = 0x8011, /// `METER_BATCH`. MeterBatch = 0x8020, /// `COUNTER_SNAPSHOT`. @@ -178,6 +181,7 @@ impl MessageId { 0x000e => Self::ResponseGet, 0x8001 => Self::SessionCommitted, 0x8010 => Self::TransportState, + 0x8011 => Self::ObservationEnded, 0x8020 => Self::MeterBatch, 0x8021 => Self::CounterSnapshot, 0x8030 => Self::Diagnostic, diff --git a/docs/BUILTINS_AND_METERING_V1.md b/docs/BUILTINS_AND_METERING_V1.md index b2226d1e4..a09a24d91 100644 --- a/docs/BUILTINS_AND_METERING_V1.md +++ b/docs/BUILTINS_AND_METERING_V1.md @@ -133,10 +133,21 @@ The frame is `3 * (trackCount + submixCount) + 3` `f32` words (issue #1209): one strip -- the tracks, then the submix strips in canonical submix order -- the master pair, then one **non-negative decibel magnitude** per strip in the same order and the designated master's. With no submixes every word is where it was before submixes existed. The sample window and the submix count -ride a fixed 72-byte `WebMeterHeader` structure (`submix_count` at offset 64), because a `u64` does +ride a fixed 80-byte `WebMeterHeader` structure (`submix_count` at offset 64), because a `u64` does not survive an `f32` and splitting one across two lanes would put a decoding rule in the app that nothing could check. With meters on, host-web meters every strip at `PostMatrix`. +The frame and its header belong to the plan (issue #1290d1): a plan with another strip count, after +a structural apply, publishes a frame sized for its own strips. The header names its frame +(`frame_pointer`, offset 72) and, from the moment a command commits a replacement plan until the +block that adopts it, the replacement's header (`successor_pointer`, offset 76); the header the +adopting block leaves behind reads `retired` (offset 68) as `1` and names the adopted one. A page +cuts its views over the successor's frame in the message handler that committed the plan, and the +render path only switches to them, so no view is made and nothing is allocated in `process()`. The +stream (`sequence`, generation, loss count) continues in the adopted plan's header; a window the +predecessor closed before the swap is published by the strip it was carried to, and one the added +strip cannot describe is counted lost. + The `miso.meter.v1` message keeps its track fields unchanged -- `peaks` is the `2T + 2` track and master peaks, `trackGrDb` the `T` track magnitudes, `masterGrDb` the master's or `null` -- and appends `submixCount`, `submixPeaks` (`[bus0 L, bus0 R, ..]`, `2S` words) and `submixGrDb` (`S` diff --git a/docs/CONTROL_PROTOCOL_CONFORMANCE.md b/docs/CONTROL_PROTOCOL_CONFORMANCE.md index 12bb138e1..d7a9ccd07 100644 --- a/docs/CONTROL_PROTOCOL_CONFORMANCE.md +++ b/docs/CONTROL_PROTOCOL_CONFORMANCE.md @@ -1,6 +1,6 @@ # MISO control protocol conformance record -This manifest records Issue 005 non-benchmark evidence and its limits. The normative corpus is `complete_schema_corpus()`: 45 canonical frames covering 11 commands (including one transaction with all 35 edit opcodes), 13 success responses, 16 non-OK statuses including typed backpressure, and five events. Its FNV-1a-64 label-and-byte hash is `4b21d9e6f520afd9` (`COMPLETE_SCHEMA_HASH`; re-pinned for the retirement of `AUTOMATION_CANCELED` in #1628: the event `8002` is refused and never reallocated, so the corpus loses `event.automation_canceled` and the capabilities response lists no event `8002`; re-pinned for the retirement of `AUTOMATION_ENQUEUE` in #1627: `0006`, its success response, statuses 15 and 16 and capability fields 9, 12, 19 and 20 are refused and never reallocated, so the corpus loses the command frame, its response and the two status frames and the capabilities response has 23 fields; re-pinned for `RESPONSE_GET` in #1573: `000e`, one command frame and two response frames (a curve reply and a lone tag), and the capability registry lists `000e` with bit 14; re-pinned for the transaction response of the monitor edit path in the batch 4 review fold: the corpus gains `response.session_transaction_monitor`, `edit_path` 4; re-pinned for the retirement of the thirteen per-kind value ops in #1561: opcodes `0008`, `0203`, `020a`, `020b`, `020d`-`0210`, `0504`-`0507` and `0702` are refused and never reallocated, 35 edits; re-pinned for `TransactionApplied` field 2, `edit_path`, in #1313; re-pinned for `SetControl` in #1553: opcode `0800` appended, 48 edits; re-pinned for the control list in #1551: `CONTROL_LIST_GET` and `CONTROL_STATE_GET` (`000c`, `000d`) replace the retired `0004` and `0005`, with a list page of every kind of row and the state page of its control rows, re-pinned for the BLAKE3 source-identity spelling in #787, for decision 12's track encoding in #1093: field 7 carries the inserts, the retired fields 6 and 8 are gone, and field 11 carries the console entries, for decision 12's console edits in #1094: `SetConsole` `0007` and `SetTrackConsole` `0211`, for the submix strip in #1199: submix fields 2, 4, 5, 6, for the submix console in #1202: submix field 3 repeats the console entry, and for the bus tap in #1203: a tag-2 route source carries a required tap, and the corpus's `SetRouteSource` value is a tapped submix source, and for the route switch in #1216: route field `mute` (field 6) and opcode `0506` (`SetRouteMute`), and for the send's follow in #1218: route field `follows_mute` (field 7) and opcode `0507` (`SetRouteFollowsMute`), and for VCA groups in #1241: the VCA message (`id` 1, `fader` 2, repeated `members` 3) and opcodes `0700`-`0702` (`UpsertVca`, `RemoveVca`, `SetVcaFader`), and for the per-edit ramp length in #1394: #1394 appends optional `ramp_samples` to the live-value edits, and the corpus's `SetTrackFader` carries one, and for the session's control smoothing in #1365: #1365 appends `SetControlSmoothing` `0008`, and the corpus's transaction carries one). Decision 12's retired and refused codes each have a wire row in `retired_code_rows()`, and so do the retired message IDs `0004`, `0005`, `0006` and `8002` (#1551, #1627, #1628; `0006` is a frame with the payload the command had, `8002` an event ID under a command header: the decoder's `UNSUPPORTED_MESSAGE`, with no response frame because a message ID that decodes to no message has no safe request identity): rack codes 1 and 3 in every rack-addressed edit and in an automation target, track fields 6 and 8 in both flag forms, each structural or declaration edit addressed at `console`, each of the thirteen value-op codes retired by #1561 (an unallocated opcode, refused at decode), and a `SetTrackConsole` in a transaction with no `SetConsole` before it that changes the console (`session.edit.value_op_retired`); an endpoint over `console_session_fixture()` answers each with its status and commits nothing. Auditable seed frames are checked in under [`fuzz/corpus`](../fuzz/corpus/README.md). +This manifest records Issue 005 non-benchmark evidence and its limits. The normative corpus is `complete_schema_corpus()`: 46 canonical frames covering 11 commands (including one transaction with all 35 edit opcodes), 13 success responses, 16 non-OK statuses including typed backpressure, and six events. Its FNV-1a-64 label-and-byte hash is `cc9a7bcc502c4936` (`COMPLETE_SCHEMA_HASH`; re-pinned for the fold of `OBSERVATION_ENDED` to one event per commit in #1584 D6a (root ruling A): the event carries a repeated key message, and the corpus frame carries two keys; re-pinned for `OBSERVATION_ENDED` in #1584: the corpus gains `event.observation_ended` (`8011`) and the capabilities response lists event `8011`; re-pinned for the retirement of `AUTOMATION_CANCELED` in #1628: the event `8002` is refused and never reallocated, so the corpus loses `event.automation_canceled` and the capabilities response lists no event `8002`; re-pinned for the retirement of `AUTOMATION_ENQUEUE` in #1627: `0006`, its success response, statuses 15 and 16 and capability fields 9, 12, 19 and 20 are refused and never reallocated, so the corpus loses the command frame, its response and the two status frames and the capabilities response has 23 fields; re-pinned for `RESPONSE_GET` in #1573: `000e`, one command frame and two response frames (a curve reply and a lone tag), and the capability registry lists `000e` with bit 14; re-pinned for the transaction response of the monitor edit path in the batch 4 review fold: the corpus gains `response.session_transaction_monitor`, `edit_path` 4; re-pinned for the retirement of the thirteen per-kind value ops in #1561: opcodes `0008`, `0203`, `020a`, `020b`, `020d`-`0210`, `0504`-`0507` and `0702` are refused and never reallocated, 35 edits; re-pinned for `TransactionApplied` field 2, `edit_path`, in #1313; re-pinned for `SetControl` in #1553: opcode `0800` appended, 48 edits; re-pinned for the control list in #1551: `CONTROL_LIST_GET` and `CONTROL_STATE_GET` (`000c`, `000d`) replace the retired `0004` and `0005`, with a list page of every kind of row and the state page of its control rows, re-pinned for the BLAKE3 source-identity spelling in #787, for decision 12's track encoding in #1093: field 7 carries the inserts, the retired fields 6 and 8 are gone, and field 11 carries the console entries, for decision 12's console edits in #1094: `SetConsole` `0007` and `SetTrackConsole` `0211`, for the submix strip in #1199: submix fields 2, 4, 5, 6, for the submix console in #1202: submix field 3 repeats the console entry, and for the bus tap in #1203: a tag-2 route source carries a required tap, and the corpus's `SetRouteSource` value is a tapped submix source, and for the route switch in #1216: route field `mute` (field 6) and opcode `0506` (`SetRouteMute`), and for the send's follow in #1218: route field `follows_mute` (field 7) and opcode `0507` (`SetRouteFollowsMute`), and for VCA groups in #1241: the VCA message (`id` 1, `fader` 2, repeated `members` 3) and opcodes `0700`-`0702` (`UpsertVca`, `RemoveVca`, `SetVcaFader`), and for the per-edit ramp length in #1394: #1394 appends optional `ramp_samples` to the live-value edits, and the corpus's `SetTrackFader` carries one, and for the session's control smoothing in #1365: #1365 appends `SetControlSmoothing` `0008`, and the corpus's transaction carries one). Decision 12's retired and refused codes each have a wire row in `retired_code_rows()`, and so do the retired message IDs `0004`, `0005`, `0006` and `8002` (#1551, #1627, #1628; `0006` is a frame with the payload the command had, `8002` an event ID under a command header: the decoder's `UNSUPPORTED_MESSAGE`, with no response frame because a message ID that decodes to no message has no safe request identity): rack codes 1 and 3 in every rack-addressed edit and in an automation target, track fields 6 and 8 in both flag forms, each structural or declaration edit addressed at `console`, each of the thirteen value-op codes retired by #1561 (an unallocated opcode, refused at decode), and a `SetTrackConsole` in a transaction with no `SetConsole` before it that changes the console (`session.edit.value_op_retired`); an endpoint over `console_session_fixture()` answers each with its status and commits nothing. Auditable seed frames are checked in under [`fuzz/corpus`](../fuzz/corpus/README.md). | Evidence | Recorded result | Limitation | | --- | --- | --- | diff --git a/docs/CONTROL_PROTOCOL_REGISTRY.md b/docs/CONTROL_PROTOCOL_REGISTRY.md index 1b1929bdc..279f60708 100644 --- a/docs/CONTROL_PROTOCOL_REGISTRY.md +++ b/docs/CONTROL_PROTOCOL_REGISTRY.md @@ -6,7 +6,7 @@ This freezes the semantic registry referenced by [the wire specification](CONTRO | Commands and echoed response IDs | Events | | --- | --- | -| `0001` capabilities get; `0002` session snapshot get; `0003` session transaction apply; `0007` transport get; `0008` transport set; `0009` telemetry configure; `000a` counters get; `000b` diagnostics get; `000c` control list get; `000d` control state get; `000e` response get | `8001` session committed; `8010` transport state; `8020` meter batch; `8021` counter snapshot; `8030` diagnostic | +| `0001` capabilities get; `0002` session snapshot get; `0003` session transaction apply; `0007` transport get; `0008` transport set; `0009` telemetry configure; `000a` counters get; `000b` diagnostics get; `000c` control list get; `000d` control state get; `000e` response get | `8001` session committed; `8010` transport state; `8011` observation ended; `8020` meter batch; `8021` counter snapshot; `8030` diagnostic | Message IDs `0004` (parameter metadata get) and `0005` (parameter state get) are **retired** by decision 17 (C4, #1551): the decoder refuses them as `UNSUPPORTED_MESSAGE`, no capability registry lists them, and they are never reallocated (`RETIRED_MESSAGE_IDS`; a conformance row pins each). `CONTROL_LIST_GET` and `CONTROL_STATE_GET` replace them. @@ -91,7 +91,7 @@ A control row's own rules: all numbers are finite and `minimum <= maximum`; choi ## Events -Reliable `8001` contains sequence, origin request ID, previous revision, and operation count. Reliable `8010` contains sequence, state, position, effective sample, and optional origin request ID. Lossy `8020` contains observed sample plus the fixed 16-byte meter array. Lossy `8021` contains observed sample plus ascending counter values. Reliable `8030` contains one diagnostic with provider sequence. Events have header request ID zero; reliable events have endpoint-monotonic payload sequence. +Reliable `8001` contains sequence, origin request ID, previous revision, and operation count. Reliable `8010` contains sequence, state, position, effective sample, and optional origin request ID. Reliable `8011` (`OBSERVATION_ENDED`, #1584) is the one event of a commit that ended host-local observation subscriptions, and it carries every ended key of that commit. It contains sequence `1:U64 R` and one repeated key message `2:MESSAGE R` per ended tap, at least one. A key message holds strip ID `1:UTF8 R`, section `2:U8 R` (inserts `2` or console `5`, the rack codes), the insert's effect ID or the console slot ID `3:UTF8 R`, and the effect-local tap ID `4:U32 R`. The keys are in strictly ascending order: the strip ID, then console slots before inserts, then the effect ID, then the tap ID; so a key never repeats. The event header's revision is the revision that ended the taps. Both IDs are stable IDs, and any other section code, an empty key list or a key out of order is `INVALID_FIELD`. It is advertised with `8001`, in the session family, when the endpoint keeps observation subscriptions. Lossy `8020` contains observed sample plus the fixed 16-byte meter array. Lossy `8021` contains observed sample plus ascending counter values. Reliable `8030` contains one diagnostic with provider sequence. Events have header request ID zero; reliable events have endpoint-monotonic payload sequence. A stored automation segment applies on `[start,end)` and holds its end value until replaced; the session schema states this hold rule (`SESSION_SCHEMA_V1.md`, #1626). diff --git a/docs/CONTROL_PROTOCOL_SEMANTICS.md b/docs/CONTROL_PROTOCOL_SEMANTICS.md index 87e1dc1b9..7e0c4e33f 100644 --- a/docs/CONTROL_PROTOCOL_SEMANTICS.md +++ b/docs/CONTROL_PROTOCOL_SEMANTICS.md @@ -22,6 +22,8 @@ There is one way to make a change at a sample time: a stored automation edit (`S Changing revision or locating transport cancels nothing. The event `AUTOMATION_CANCELED` (`8002`) is **retired** with the command that filled the queue it reported on (#1628; `AUTOMATION_ENQUEUE` is retired, #1627, so no batch is ever accepted and none can be dropped), and a frame under `8002` is refused at decode like any retired ID. A transaction commit reserves only its `SESSION_COMMITTED`; a transport set reserves only its `TRANSPORT_STATE`. `SESSION_COMMITTED`, transport state, and diagnostics are reliable; meter batches and periodic counter snapshots are the only lossy events. A reliable event has a monotonic sequence and queue capacity is reserved before its related commit. +**Observation subscriptions across a rebuild (#1584).** A host that keeps observation subscriptions (the browser's observe kinds on the engine-wide control plane; the C ABI's own edit follows) holds a table of armed taps beside its monitor overlay and apart from it: host-local, never saved, never in a snapshot, with no revision. A subscription is acknowledged when its record is queued on the newest plan, and the table is written with that acknowledgement. Every plan the host prepares after a rebuild is prepared from the table, so a tap whose effect restarts (a prepared change such as the link mode) is armed in the successor and opens its first window at the successor's first rendered block, as a record applied at that block would; a tap whose effect carries keeps its own window and sequence, and the prepared arm never restarts it. No acknowledged subscription is dropped at a swap. A rebuild that removes the effect (the insert, the strip or the console slot is gone, or the ID now names an effect without that resident tap) ends the binding: the commit queues one reliable `OBSERVATION_ENDED` (`8011`) after the transaction's `SESSION_COMMITTED`, at the committing revision, carrying the stable key of every binding that commit ended, in key order, however many there are. The room for this one event is reserved with the commit's own before anything is prepared or published; if the reliable lane holds unread events that leave no room for both, the transaction is refused as `BACKPRESSURE` with the model, the revision, the table and the plan unchanged, and the same request commits after the lane drains. The number of keys never changes the room needed. An event whose frame would exceed one control frame can never be sent, so that transaction is refused as a resource limit, not as backpressure: the app disarms some of those subscriptions first. Both hosts run the one control plane, so the browser and the C ABI emit the same event for the same transaction. + ## Queues and telemetry Control commands, responses, and reliable events reject before related state changes when full and preserve the original item. Telemetry may replace only the latest same `(revision, handle/component)` value, incrementing `telemetry_coalesced`; a new key rejected at capacity increments `telemetry_dropped`. Counters are non-resetting and saturate at `u64::MAX`. diff --git a/fuzz/corpus/complete-schema-manifest.md b/fuzz/corpus/complete-schema-manifest.md index 0e2fcc720..a1ea271b2 100644 --- a/fuzz/corpus/complete-schema-manifest.md +++ b/fuzz/corpus/complete-schema-manifest.md @@ -1,13 +1,13 @@ # Issue-005 complete typed BTLV corpus -The shared `complete_schema_corpus()` fixture has 45 canonical frames: all 11 commands (the +The shared `complete_schema_corpus()` fixture has 46 canonical frames: all 11 commands (the session transaction contains all 35 allocated `SessionEditOpcode` values), all 13 successful -responses, all 16 registered non-OK statuses (including typed `BACKPRESSURE`), and all five event +responses, all 16 registered non-OK statuses (including typed `BACKPRESSURE`), and all six event schemas. Optional/boundary values are represented by the transaction's nested fixture, optional transport position, empty valid pages, and the typed backpressure variant. The canonical sequence is FNV-1a-64 over each stable frame label followed by its frame bytes: -`4b21d9e6f520afd9`, pinned once as `COMPLETE_SCHEMA_HASH`. Native mutation, each typed fuzz +`cc9a7bcc502c4936`, pinned once as `COMPLETE_SCHEMA_HASH`. Native mutation, each typed fuzz decoder, and scalar/simd128 Wasm execution all consume this same public fixture source. Issue #787 re-pinned this value after the transaction's two same-length source-identity spellings @@ -66,6 +66,11 @@ Issue #1628 (draft 21b) re-pinned it from `cfa58ab7b7bbb7c4`: the event `AUTOMAT (`0x8002`) is retired and never reallocated, so the corpus loses `event.automation_canceled` (the frame count is 45 and the events are five), and the capabilities response lists no event `0x8002`. `retired_code_rows()` gains a row for `0x8002`. +Issue #1584 (D6a) re-pinned it from `4b21d9e6f520afd9`: the reliable event `OBSERVATION_ENDED` (`0x8011`) +is added, so the corpus gains `event.observation_ended` (the frame count is 46 and the events are six), +and the capabilities response lists event `0x8011`. +#1584 D6a fold: one OBSERVATION_ENDED per commit (root ruling A) re-pinned it from `33e4b1cebcbfd902`: the +event carries a repeated key message and the corpus frame carries two keys (the frame count stays 46). The value stood at `88a8ee6a6d9e4acc` here until #274. It was correct until `b454b230`, and the two re-pins that followed (`b454b230`, then #241's `04d291dd`) did not reach this file or the diff --git a/hosts/host-web/qualification/run.mjs b/hosts/host-web/qualification/run.mjs index 0296bf6a5..f7480e9c8 100644 --- a/hosts/host-web/qualification/run.mjs +++ b/hosts/host-web/qualification/run.mjs @@ -31,6 +31,10 @@ const MUTATIONS = [ "sdk-observation-window-reversed", "sdk-observation-window-malformed", "sdk-response", "sdk-observation", "sdk-spectrum", "sdk-spectrum-hop-missing", "sdk-spectrum-hop-wrong", "sdk-spectrum-recovery-loss", + // Issue #1106: a hop probe whose audio start was not delayed, one that never waited, one that + // spun, and a never-publishing wait that outlived its deadline. + "sdk-spectrum-hop-wait-fast-start", "sdk-spectrum-hop-wait-early", "sdk-spectrum-hop-wait-spin", + "sdk-spectrum-hop-wait-never-unbounded", "sdk-spectrum-recovery-unavailable", "sdk-spectrum-recovery-identity", // Issue #1097 gate 4: a live bypass that reached no instance, or the wrong one. "sdk-live-bypass-console", "sdk-live-bypass-insert", "sdk-live-bypass-address", @@ -812,6 +816,25 @@ function validateSdkResponse(browserName, response) { && configuredHop?.resultSmoothingMs === 37.5 && configuredHop?.ownedArrays === true, "actual browser H1024 did not preserve Worklet metadata, Worker results, overlap, smoothing, and ownership"); + // Issue #1106: the probe waits on its deadline, not a poll count. Its audio starts late by a + // planted delay, so both windows arrive only after the audio start (a publication + // before that means the delay was not in effect); a wait that can never be satisfied ends at + // its own deadline and polls no faster than its timer. + const hopWait = configuredHop?.wait; + const neverPublishes = hopWait?.neverPublishes; + gate(browserName, "sdk-spectrum-hop-wait", hopWait?.startDelayMs >= 500 + && hopWait?.deadlineMs === 10_000 + && hopWait?.pollMs > 0 + && hopWait?.firstPublicationAfterMs >= hopWait?.startDelayMs - 100 + && hopWait?.secondPublicationAfterMs >= hopWait.firstPublicationAfterMs + && hopWait?.elapsedMs < hopWait.deadlineMs + && hopWait?.polls <= 2 * Math.ceil(hopWait.elapsedMs / hopWait.pollMs) + 2 + && neverPublishes?.count === 0 + && neverPublishes?.elapsedMs >= neverPublishes.deadlineMs + && neverPublishes?.elapsedMs < neverPublishes.deadlineMs + 2_000 + && neverPublishes?.polls >= 1 + && neverPublishes?.polls <= 2 * Math.ceil(neverPublishes.deadlineMs / hopWait.pollMs) + 2, + `the hop probe's wait did not outlast a planted slow audio start or was not bounded by its deadline: ${JSON.stringify(hopWait)}`); } function mutate(result, mutation) { @@ -848,6 +871,18 @@ function mutate(result, mutation) { if (mutation === "sdk-spectrum-hop-wrong") { copy.sdkResponse.spectrum.continuous.first.hopFrames = 2_048; } + if (mutation === "sdk-spectrum-hop-wait-fast-start") { + copy.sdkResponse.spectrum.configuredHop.wait.startDelayMs = 0; + } + if (mutation === "sdk-spectrum-hop-wait-early") { + copy.sdkResponse.spectrum.configuredHop.wait.firstPublicationAfterMs = 1; + } + if (mutation === "sdk-spectrum-hop-wait-spin") { + copy.sdkResponse.spectrum.configuredHop.wait.polls = 100_000; + } + if (mutation === "sdk-spectrum-hop-wait-never-unbounded") { + copy.sdkResponse.spectrum.configuredHop.wait.neverPublishes.elapsedMs = 100; + } if (mutation === "sdk-spectrum-recovery-loss") { copy.sdkResponse.spectrum.continuous.recoveryDelivery.nativeMissedWindows = "0"; copy.sdkResponse.spectrum.continuous.recoveryDelivery.skippedPublications = "0"; @@ -910,6 +945,7 @@ function mutationProofs(browserName, result) { (error) => error instanceof Error && error.message.startsWith( mutation.startsWith("resume-clicks-") ? `${browserName}: resume-clicks:` : mutation.startsWith("sdk-observation-wait-") ? `${browserName}: sdk-observation-wait:` + : mutation.startsWith("sdk-spectrum-hop-wait-") ? `${browserName}: sdk-spectrum-hop-wait:` : `${browserName}:`, ), `${browserName}: ${mutation}: red mutation escaped its gate`, diff --git a/hosts/host-web/qualification/sdk-response-entry.ts b/hosts/host-web/qualification/sdk-response-entry.ts index b91e459ce..21adafa8f 100644 --- a/hosts/host-web/qualification/sdk-response-entry.ts +++ b/hosts/host-web/qualification/sdk-response-entry.ts @@ -618,6 +618,29 @@ async function createContinuousSpectrumBrowser( return browser; } +/** + * Wait until `read()` reports `required` items, bounded by `deadlineMs` of wall-clock time and + * never by a poll count (#1106). `pump()` can return at once, so each poll is followed by a + * `pollMs` timer: the loop yields to the event loop and the audio clock instead of spinning, and + * a slow audio start only costs time. Returns what it saw so a failure can name it. + */ +async function waitForCount(options: { + readonly read: () => number; + readonly pump: () => Promise; + readonly required: number; + readonly deadlineMs: number; + readonly pollMs: number; +}) { + const began = performance.now(); + let polls = 0; + while (options.read() < options.required && performance.now() - began < options.deadlineMs) { + await options.pump(); + polls += 1; + await new Promise((resolve) => globalThis.setTimeout(resolve, options.pollMs)); + } + return { count: options.read(), polls, elapsedMs: performance.now() - began }; +} + function streamMetadataFields(metadata) { if (metadata === undefined) return undefined; return { @@ -921,6 +944,12 @@ async function runContinuousSpectrumQualification(): Promise> { const query = { target: { kind: "output" as const, outputId: "main-out" }, @@ -934,7 +963,13 @@ async function runConfiguredSpectrumHopQualification(): Promise void; + const streamStartGate = new Promise((resolve) => { openStreamStart = resolve; }); browser.host.startSpectrumStream = async (...args) => { + await streamStartGate; const reply = await nativeStartSpectrumStream(...args); nativeStarts.push(reply.metadata); return reply; @@ -948,6 +983,7 @@ async function runConfiguredSpectrumHopQualification(): Promise { notifications.push(notification); @@ -958,14 +994,17 @@ async function runConfiguredSpectrumHopQualification(): Promise { subscription = subscribed.handle; }); + // Settled by the await after the wait; a failure before it must not become an unhandled rejection. + subscribing.catch(() => undefined); await submitSpectrumSource(browser, frames); const context = browser.context as AudioContext; const resumeButton = document.createElement("button"); @@ -974,6 +1013,8 @@ async function runConfiguredSpectrumHopQualification(): Promise | undefined; let resumeGesture = false; + let resumeCall!: Promise; + let clickedAt = 0; try { await new Promise((resolve, reject) => { resumeTimer = setTimeout(() => reject(new Error( @@ -981,23 +1022,61 @@ async function runConfiguredSpectrumHopQualification(): Promise { resumeGesture = true; - void context.resume().then(resolve, reject); + clickedAt = performance.now(); + globalThis.setTimeout(openStreamStart, HOP_PROBE_START_DELAY_MS); + resumeCall = context.resume(); + resumeCall.catch(reject); + resolve(); }, { once: true }); }); } finally { - if (resumeTimer !== undefined) clearTimeout(resumeTimer); resumeButton.remove(); } - const deadline = performance.now() + 10_000; - for (let attempt = 0; attempt < 512 && publications.length < 2; attempt += 1) { - await subscription.pump(); - if (performance.now() >= deadline) break; + // The wait starts at the click, not when `resume()` resolves: a browser may resolve it before or + // after its first render. No window can publish for `HOP_PROBE_START_DELAY_MS`, and the wait + // must outlast that by its deadline, never by a poll count. It ends on two windows one hop + // apart: a late start can leave the first reads behind the audio and drop windows, and the hop + // is the distance between two consecutive publications. + const consecutive = () => { + for (let index = 0; index + 1 < publications.length; index += 1) { + const a = publications[index].metadata; + const b = publications[index + 1].metadata; + if (a.captureEpoch === b.captureEpoch && b.capturedSample - a.capturedSample === BigInt(hopFrames)) { + return index; + } + } + return -1; + }; + let waited; + try { + waited = await waitForCount({ + read: () => (consecutive() >= 0 ? 2 : Math.min(1, publications.length)), + pump: async () => { await subscription?.pump(); }, + required: 2, + deadlineMs: HOP_PROBE_DEADLINE_MS, + pollMs: HOP_PROBE_POLL_MS, + }); + await resumeCall; + } finally { + if (resumeTimer !== undefined) clearTimeout(resumeTimer); } - if (publications.length < 2) { - throw new Error(`H1024 browser probe published ${publications.length} windows`); + await subscribing; + const pairIndex = consecutive(); + if (pairIndex < 0) { + throw new Error(`H1024 browser probe published ${publications.length} windows ` + + `[${publications.map(({ metadata }) => metadata.capturedSample).join(", ")}], none one hop apart, in ${waited.elapsedMs.toFixed(0)} ms ` + + `(deadline ${HOP_PROBE_DEADLINE_MS} ms, ${waited.polls} polls, audio start delayed ${HOP_PROBE_START_DELAY_MS} ms, ` + + `${notifications.length} notifications, context ${context.state})`); } - const first = publications[0]; - const second = publications[1]; + const neverPublishes = await waitForCount({ + read: () => 0, + pump: async () => undefined, + required: 1, + deadlineMs: HOP_PROBE_NEVER_DEADLINE_MS, + pollMs: HOP_PROBE_POLL_MS, + }); + const first = publications[pairIndex]; + const second = publications[pairIndex + 1]; const firstResult = first.result; const secondResult = second.result; const firstFrequencies = firstResult.frequenciesHz.slice(); @@ -1017,9 +1096,21 @@ async function runConfiguredSpectrumHopQualification(): Promise notification.metadata.hopFrames), notificationSmoothingMs: notifications.map((notification) => notification.metadata.smoothingMs), - publications: publications.slice(0, 2).map(({ metadata, result }) => ({ + publications: [first, second].map(({ metadata, result }) => ({ metadata: streamMetadataFields(metadata), result: { capturedSample: result.capturedSample.toString(), @@ -1044,6 +1135,7 @@ async function runConfiguredSpectrumHopQualification(): Promise subscription?.close()); } } @@ -1191,7 +1283,8 @@ async function runSpectrumCollectionQualification(): Promise { for (let block = firstBlock; block < lastBlock; block += 1) { let acknowledgement; - for (let retry = 0; retry < 200; retry += 1) { + const submitDeadline = performance.now() + readinessTimeoutMs; + for (;;) { acknowledgement = await browser.host.submitSource({ sourceId: "live-control-source", generation: 1n, @@ -1205,6 +1298,7 @@ async function runSpectrumCollectionQualification(): Promise= submitDeadline) break; await new Promise((resolve) => globalThis.setTimeout(resolve, 5)); } if (acknowledgement?.result !== 0) { diff --git a/hosts/host-web/src/ffi.rs b/hosts/host-web/src/ffi.rs index a400c205f..d434f4b5e 100644 --- a/hosts/host-web/src/ffi.rs +++ b/hosts/host-web/src/ffi.rs @@ -3462,7 +3462,9 @@ pub extern "C" fn miso_engine_web_v1_meter_poll(handle: u32) -> u32 { render_locked(|| with_host_mut(handle, 0, AudioWorkletEngineHost::poll_meters)) } -/// Return the stable meter-header address, or zero for an invalid handle (issue #143 D5). +/// Return the address of the header that describes the plan now rendering, or zero for an invalid +/// handle (issue #143 D5). The address changes at a plan swap (issue #1290d1: each plan carries its +/// own header), so a page reads it again after a swap or follows `successor_pointer`. /// /// The `f32` meter frame carries numbers a meter draws; the window those numbers describe is a /// pair of absolute sample counts, which an `f32` cannot hold. They ride this fixed structure, read @@ -4062,6 +4064,15 @@ pub mod native_staging { super::with_host(handle, None, |host| Some(*host.watermark())) } + /// The meter header and frame the host `handle` names last published, as the worklet reads + /// them through `miso_engine_web_v1_meter_header_ptr` and the header's frame pointer (#1290d1); + /// `None` for an unknown handle. + pub fn meter_frame(handle: u32) -> Option<(crate::WebMeterHeader, Vec)> { + super::with_host(handle, None, |host| { + Some((*host.meter_header(), host.meter_frame().to_vec())) + }) + } + /// Stage live-control command records, as the worklet writes them through /// `miso_engine_web_v1_buffer_ptr(handle, BUFFER_COMMAND)`. pub fn command_records(handle: u32, records: &[u8]) -> bool { diff --git a/hosts/host-web/src/lib.rs b/hosts/host-web/src/lib.rs index c50817d0f..b2cec073e 100644 --- a/hosts/host-web/src/lib.rs +++ b/hosts/host-web/src/lib.rs @@ -961,8 +961,21 @@ pub struct WebMeterHeader { /// Submix strips the frame carries after the tracks (issue #1209 D2), in canonical submix /// order. Appended in place, so every earlier offset is unmoved. pub submix_count: u32, - /// Always zero; pads the structure to its eight-byte alignment. - pub reserved_pad: u32, + /// `0` while this header describes the plan that renders; `1` once a swap block replaced it + /// (issue #1290d1). The plan's meter frame travels with its header, so the worklet's views + /// over the frame are the plan's: a header that reads `1` is the page's signal to switch to + /// the views it cut over [`Self::successor_pointer`]'s header and frame, in the block that + /// adopted the plan. The retired header stays alive until the service step after that block. + pub retired: u32, + /// Linear-memory address of the `3(T + S) + 3` word `f32` frame this header describes, or `0` + /// for a handle with no compiled session (issue #1290d1). + pub frame_pointer: u32, + /// Linear-memory address of the header that replaces this one: the newest committed + /// replacement plan's, from its commit until its adoption block, and the adopted plan's once + /// [`Self::retired`] is set. `0` when no replacement waits. A page cuts the views over that + /// header and its frame in the message handler that commits the plan, so the render path + /// switches views by reference and never cuts one (issue #1290d1). + pub successor_pointer: u32, } /// `WebMeterHeader::reserved[1]` validity bits. The high 32 bits carry a saturating loss count. @@ -999,7 +1012,9 @@ const fn empty_meter_header() -> WebMeterHeader { master_gr_present: 0, reserved: [0; 2], submix_count: 0, - reserved_pad: 0, + retired: 0, + frame_pointer: 0, + successor_pointer: 0, } } @@ -1285,8 +1300,11 @@ struct EffectInstance { /// [`RenderCompanions`]. The host swaps the pair at the block that adopts the plan, a pointer /// exchange that allocates and frees nothing. /// -/// The published meter frame and header are not here: the worklet binds fixed views over them at -/// boot, so they keep the boot's size (#1290d owns their growth). +/// The published meter frame and its header are here too (#1290d1): the frame is `3(T + S) + 3` +/// words for this plan's strips, so a plan with another strip count publishes a frame of its own +/// size. They travel with the tables, in the same pointer exchange, and the page cuts its views +/// over the successor's frame in the message handler that commits it ([`WebMeterHeader::retired`] +/// and [`WebMeterHeader::successor_pointer`] tell the render path which views to use). struct PlanShape { /// Canonical normalized track order: the leading segment of the strip order that /// `track_index` addresses (issue #1213 D1). @@ -1368,6 +1386,30 @@ struct PlanShape { /// entry before it pops, so delivery resets do not touch it. Allocated at preparation, sized /// by the meter count. meter_remaining: Box<[usize]>, + /// Per meter, the handle its carried predecessor published under, or `0` (#1290d1). The swap + /// block pairs a meter with the predecessor's at a strip identity, and a strip added or + /// removed in front shifts the handles (a handle is a position), so a window the predecessor + /// closed before the swap and the poll finds after it names the predecessor's handle. The poll + /// accepts it for this meter, for the plan's life: a meter's queue is its strip's alone, so + /// the second handle names the same strip. A window two swaps old names a handle this table + /// no longer holds and is dropped with a count. + meter_carried_from: Box<[u64]>, + /// The published frame, `3(T + S) + 3` words (issue #1209 D1): `[track0 L, track0 R, .., trackN + /// L, trackN R, bus0 L, bus0 R, .., busM L, busM R, master L, master R, track0 GR, .., trackN + /// GR, bus0 GR, .., busM GR, master GR]`, one peak pair and one gain-reduction word per strip. + /// With `S = 0` every word is where it always was. + meter_frame: Box<[f32]>, + /// The header of [`Self::meter_frame`]: the plan's strip counts and designated master, and the + /// window the poll published. The swap block copies the stream's fields from the retiring + /// plan's header into the adopted plan's. + meter_header: WebMeterHeader, + /// The full-width address of the header [`WebMeterHeader::successor_pointer`] names, or `0`. + /// The header word is a `u32` linear-memory address, which reads `0` for an address above + /// 4 GiB (a native test host), so a gate that compares the word with the same conversion + /// would pass vacuously. This is the pointer itself, set beside the word by + /// [`Self::name_successor`], and a gate compares it with `core::ptr::eq`. + #[cfg(any(test, feature = "test-support"))] + successor_address: usize, /// Whether each strip contributed a gain-reduction reading to the most recent poll. Allocated /// at compilation, one entry per strip; it is what makes `masterGrDb` absent rather than zero. observation_present: Box<[bool]>, @@ -1443,13 +1485,6 @@ struct ReadyOwnership { /// collection. It is declared before the prepared host so consumers are released before the /// plan's producers. spectrum: Option, - /// `[track0 L, track0 R, .., trackN L, trackN R, bus0 L, bus0 R, .., busM L, busM R, master L, - /// master R, track0 GR, .., trackN GR, bus0 GR, .., busM GR, master GR]` -- `3(T + S) + 3` - /// words, one peak pair and one gain-reduction word per strip (issue #1209 D1). With `S = 0` - /// every word is where it always was. - meter_frame: Box<[f32]>, - /// The window and shape the `f32` frame cannot carry (issue #143 D5). - meter_header: WebMeterHeader, /// Master peaks folded over the rendered block while the lease is on. master_peak: [f32; 2], /// Half-open span covered by `master_peak`; retained across empty polls until a track span is @@ -1533,6 +1568,12 @@ impl ReadyOwnership { shape: Box::new(shape), adopted: false, }); + // The rendering plan's header names the newest replacement's, so the page cuts its + // views over the successor's frame in the handler that committed it (#1290d1). + if let Some(entry) = self.pending_shapes.last() { + self.shape + .name_successor(core::ptr::from_ref(&entry.shape.meter_header)); + } self.blocking_rebuilds = self.blocking_rebuilds.saturating_add(1); } result @@ -1554,11 +1595,16 @@ impl ReadyOwnership { /// executors carried, which this hand-over follows with the per-reader state the tables hold. /// A swap that carried a meter keeps the meter delivery running, because the carried meters /// keep their window grid and the host's master windows are continuous; one that carried none - /// restarts it, because its meters open their first window at the adoption block. The poll - /// finds the carried pending windows where it left them only while the meter handles are - /// unchanged: a swap that removes one strip and adds another shifts handles, so a carried - /// pending snapshot fails the poll's handle check, is dropped, and is added to the meter loss - /// count. + /// restarts it, because its meters open their first window at the adoption block. A swap that + /// adds or removes a strip in front shifts the meters' handles (a handle is a position): each + /// carried meter remembers the handle its predecessor published under + /// ([`PlanShape::meter_carried_from`]), so a window closed before the swap, pending or still + /// queued, is accepted by the strip it was carried to, not dropped. + /// + /// The meter frame travels with the tables (#1290d1): the adopted plan's frame and header are + /// the published ones from this block on. The retiring header is marked + /// ([`WebMeterHeader::retired`]) and names the adopted one, so the page's render path switches + /// to the views it cut over them in advance. Pointer exchanges and scalar writes only. fn adopt_shape(&mut self, epoch: u64, carried: &[CarriedReader]) { self.shape_epoch = epoch; let meters_carried = carried @@ -1582,7 +1628,22 @@ impl ReadyOwnership { entry.adopted = true; taken = true; // `self.shape` is the adopted plan's and `entry.shape` the retired one's. The - // pending track snapshots and the tap state follow their readers. + // stream the page reads continues in the adopted plan's header (#1290d1): its + // window, sequence, generation and loss are the stream's, its strip counts and + // frame are the plan's. The retired header says so and names the adopted one, so + // the page's render path switches to the views it cut in advance. + let stream = entry.shape.meter_header; + entry.shape.meter_header.retired = 1; + entry + .shape + .name_successor(core::ptr::from_ref(&self.shape.meter_header)); + self.shape.meter_header.windows = stream.windows; + self.shape.meter_header.first_sample = stream.first_sample; + self.shape.meter_header.end_sample = stream.end_sample; + self.shape.meter_header.sequence = stream.sequence; + self.shape.meter_header.master_gr_present = stream.master_gr_present; + self.shape.meter_header.reserved = stream.reserved; + // The pending track snapshots and the tap state follow their readers. for reader in carried { let (Ok(successor), Ok(predecessor)) = ( usize::try_from(reader.successor), @@ -1598,6 +1659,15 @@ impl ReadyOwnership { ) { core::mem::swap(into, from); } + // The windows the predecessor closed under its handle and the poll + // finds after this block belong to this strip (#1290d1). + if let Some(from) = self.shape.meter_carried_from.get_mut(successor) { + *from = if reader.predecessor_handle == reader.successor_handle { + 0 + } else { + reader.predecessor_handle + }; + } } ObserverKind::Observation => { self.shape @@ -1611,6 +1681,16 @@ impl ReadyOwnership { } } if taken { + // A replacement committed after this one waits behind it (#1290d1). + let waiting = self + .pending_shapes + .iter() + .rev() + .find(|entry| !entry.adopted) + .map_or(core::ptr::null(), |entry| { + core::ptr::from_ref(&entry.shape.meter_header) + }); + self.shape.name_successor(waiting); if meters_carried { // The delivery epoch is the stream's, not the plan's: the page sees the same // generation, and the poll finds the carried windows where it left them. @@ -1681,15 +1761,15 @@ impl ReadyOwnership { if clear_publication { // Keep the callback-reachable reset free of slice-fill's bounds-panic owner in the // shipped Wasm call graph. The iterator walks the already prepared frame exactly once. - for slot in &mut self.meter_frame { + for slot in &mut self.shape.meter_frame { *slot = 0.0; } - self.meter_header.sequence = 0; - self.meter_header.windows = 0; - self.meter_header.first_sample = 0; - self.meter_header.end_sample = 0; - self.meter_header.master_gr_present = 0; - self.meter_header.reserved[1] = 0; + self.shape.meter_header.sequence = 0; + self.shape.meter_header.windows = 0; + self.shape.meter_header.first_sample = 0; + self.shape.meter_header.end_sample = 0; + self.shape.meter_header.master_gr_present = 0; + self.shape.meter_header.reserved[1] = 0; self.meter_windows = 0; } } @@ -2461,7 +2541,9 @@ impl AudioWorkletEngineHost { /// `masterGrDb` is `null` when the designated strip has none. #[must_use] pub fn meter_frame(&self) -> &[f32] { - self.ready.as_ref().map_or(&[], |ready| &ready.meter_frame) + self.ready + .as_ref() + .map_or(&[], |ready| &ready.shape.meter_frame) } /// The sample window and shape the `f32` frame cannot carry (issue #143 D5). @@ -2469,7 +2551,18 @@ impl AudioWorkletEngineHost { pub fn meter_header(&self) -> &WebMeterHeader { self.ready .as_ref() - .map_or(&EMPTY_METER_HEADER, |ready| &ready.meter_header) + .map_or(&EMPTY_METER_HEADER, |ready| &ready.shape.meter_header) + } + + /// The full-width address of the header the rendering plan's header names as its successor, + /// or `0` when none waits (a gate compares it with the candidate's or the adopted header's + /// address; the header's own `u32` word reads `0` above 4 GiB on a native host). + #[cfg(any(test, feature = "test-support"))] + #[must_use] + pub fn meter_successor_address(&self) -> usize { + self.ready + .as_ref() + .map_or(0, |ready| ready.shape.successor_address) } /// Armed taps, in the dense effect-slot order. Off-ABI introspection for the tests. @@ -2952,7 +3045,7 @@ impl AudioWorkletEngineHost { if let Some(ready) = self.ready.as_mut() { ready.meter_generation = ready.meter_generation.saturating_add(1); ready.reset_meter_delivery(true); - ready.meter_header.reserved[0] = ready.meter_generation; + ready.shape.meter_header.reserved[0] = ready.meter_generation; } self.meter_activation_sample = self.next_meter_boundary(self.status.next_absolute_sample); self.record(RESULT_OK) @@ -3398,24 +3491,6 @@ impl AudioWorkletEngineHost { if meter_count == 0 { return 0; } - // #1290a: the published frame keeps the boot's size (the worklet binds fixed views over - // it), so a plan whose strip count differs cannot be described by it. Publish nothing - // rather than a frame whose words name other strips; #1290d grows the frame with the plan. - let frame_words = ready - .shape - .tracks - .len() - .saturating_add(ready.shape.submixes.len()) - .saturating_mul(3) - .saturating_add(3); - if ready.meter_frame.len() != frame_words { - // Each interval the render side closes in this state is a window that is not - // published: dropped with a count, which the next frame that is published reports. - while ready.pop_master_window().is_some() { - ready.meter_loss_count = ready.meter_loss_count.saturating_add(1); - } - return 0; - } // The render path is the sole producer of this fixed-capacity ring and increments the // count only when the configured master interval closes. An empty ring therefore proves // that no coherent track/master publication can exist. Preserve pending/public bytes and @@ -3479,11 +3554,16 @@ impl AudioWorkletEngineHost { .enumerate() .all(|(index, entry)| { let Some(snapshot) = entry else { return false }; - companions - .meters - .get(index) - .is_some_and(|meter| snapshot.handle == meter.handle) - && snapshot.reset_generation == first.reset_generation + // A window the predecessor closed before a swap names the predecessor's handle + // for the strip the meter was carried from (#1290d1). + companions.meters.get(index).is_some_and(|meter| { + snapshot.handle == meter.handle + || ready + .shape + .meter_carried_from + .get(index) + .is_some_and(|from| *from != 0 && snapshot.handle.0.get() == *from) + }) && snapshot.reset_generation == first.reset_generation && snapshot.window_sequence == first.window_sequence && snapshot.start_sample == first.start_sample && snapshot.end_sample == first.end_sample @@ -3549,9 +3629,9 @@ impl AudioWorkletEngineHost { self.meter_activation_sample = reset_activation_sample; return 0; } - if ready.meter_header.sequence > 0 - && ready.meter_header.reserved[0] == ready.meter_generation - && first.start_sample != ready.meter_header.end_sample + if ready.shape.meter_header.sequence > 0 + && ready.shape.meter_header.reserved[0] == ready.meter_generation + && first.start_sample != ready.shape.meter_header.end_sample { ready.meter_loss_count = ready.meter_loss_count.saturating_add(1); } @@ -3628,16 +3708,16 @@ impl AudioWorkletEngineHost { ); ready.meter_snapshot_drops_seen = snapshot.cumulative_dropped_snapshots; } - if let Some(slot) = ready.meter_frame.get_mut(index * 2) { + if let Some(slot) = ready.shape.meter_frame.get_mut(index * 2) { *slot = snapshot.left.sample_peak; } - if let Some(slot) = ready.meter_frame.get_mut(index * 2 + 1) { + if let Some(slot) = ready.shape.meter_frame.get_mut(index * 2 + 1) { *slot = snapshot.right.sample_peak; } ready.meter_snapshot_generation = Some(snapshot.reset_generation); } for (plane, peak) in master_peak.iter().enumerate() { - if let Some(slot) = ready.meter_frame.get_mut(master + plane) { + if let Some(slot) = ready.shape.meter_frame.get_mut(master + plane) { *slot = *peak; } } @@ -3649,7 +3729,7 @@ impl AudioWorkletEngineHost { // reduction word) becomes decibels once per closed window, never per sample and never on a // lane kernel. let gain_base = strips * 2 + 2; - for slot in ready.meter_frame.iter_mut().skip(gain_base) { + for slot in ready.shape.meter_frame.iter_mut().skip(gain_base) { *slot = 0.0; } for slot in ready.shape.observation_present.iter_mut() { @@ -3719,7 +3799,7 @@ impl AudioWorkletEngineHost { let value = observed_decibels(*tap, magnitude); // Several armed taps on one track fold max-magnitude into the one positional slot // the frame carries for that track, on the control plane. - if let Some(slot) = ready.meter_frame.get_mut(gain_base + track as usize) + if let Some(slot) = ready.shape.meter_frame.get_mut(gain_base + track as usize) && value > *slot { *slot = value; @@ -3744,16 +3824,20 @@ impl AudioWorkletEngineHost { }; if master_present && let Some(track) = ready.shape.master_track - && let Some(value) = ready.meter_frame.get(gain_base + track as usize).copied() - && let Some(slot) = ready.meter_frame.get_mut(gain_base + strips) + && let Some(value) = ready + .shape + .meter_frame + .get(gain_base + track as usize) + .copied() + && let Some(slot) = ready.shape.meter_frame.get_mut(gain_base + strips) { *slot = value; } - ready.meter_header.windows = windows; - ready.meter_header.sequence = ready.meter_header.sequence.saturating_add(1); - ready.meter_header.master_gr_present = u32::from(master_present); - ready.meter_header.first_sample = folded_start; - ready.meter_header.end_sample = folded_end; + ready.shape.meter_header.windows = windows; + ready.shape.meter_header.sequence = ready.shape.meter_header.sequence.saturating_add(1); + ready.shape.meter_header.master_gr_present = u32::from(master_present); + ready.shape.meter_header.first_sample = folded_start; + ready.shape.meter_header.end_sample = folded_end; let mut validity = METER_VALID_COMPLETE; if master_valid { validity |= METER_VALID_MASTER; @@ -3764,8 +3848,8 @@ impl AudioWorkletEngineHost { if observed_any { validity |= METER_VALID_GAIN_REDUCTION; } - ready.meter_header.reserved[0] = ready.meter_generation; - ready.meter_header.reserved[1] = + ready.shape.meter_header.reserved[0] = ready.meter_generation; + ready.shape.meter_header.reserved[1] = validity | (ready.meter_loss_count.min(u64::from(u32::MAX)) << METER_LOSS_SHIFT); ready.meter_windows = ready.meter_windows.saturating_add(u64::from(windows)); windows @@ -5074,6 +5158,10 @@ struct CarriedReader { kind: ObserverKind, successor: u32, predecessor: u32, + /// For a meter, the handles the two positions' meters publish under (`0` for any other kind): + /// the poll accepts the predecessor's for windows it closed before the swap (#1290d1). + successor_handle: u64, + predecessor_handle: u64, } impl CarriedReader { @@ -5082,6 +5170,8 @@ impl CarriedReader { kind: ObserverKind::Meter, successor: 0, predecessor: 0, + successor_handle: 0, + predecessor_handle: 0, }; } @@ -5168,8 +5258,6 @@ impl Drop for RenderCompanions { struct BootRecord { /// The engine-owned report of the boot plan, after the attachment's rows were charged. report: Option, - /// The designated master strip, echoed back by preparation after validation. - master_track: Option, /// The session's `live_values_superseded` count, shared with every cell reader of its plans. superseded: Option, /// The class of the preparation refusal, so a refused boot keeps its result code. @@ -5239,10 +5327,24 @@ impl HostAttachment for RenderCompanions { let Some(entry) = self.carried.get_mut(written) else { break; }; + let (successor_handle, predecessor_handle) = match carry.kind { + ObserverKind::Meter => ( + self.meters + .get(carry.successor) + .map_or(0, |meter| meter.handle.0.get()), + previous + .meters + .get(carry.predecessor) + .map_or(0, |meter| meter.handle.0.get()), + ), + _ => (0, 0), + }; *entry = CarriedReader { kind: carry.kind, successor, predecessor, + successor_handle, + predecessor_handle, }; written += 1; } @@ -5413,7 +5515,6 @@ impl WebRuntimePreparer { })?; if boot { record.report = Some(host.report); - record.master_track = handles.master_track; record.superseded = Some(handles.live_values_superseded.clone()); } record.shape = Some(shape); @@ -5554,6 +5655,16 @@ fn build_plan_shape( .try_reserve_exact(observations.len()) .map_err(|_| "web.resource.allocation")?; let observation_present = vec![false; strip_count]; + // The plan's published frame and the header that describes it (#1290d1). The frame's address + // is the heap block's, so it stays put as the tables move. + let meter_frame = zero_meter_frame(strip_count)?; + let mut meter_header = empty_meter_header(); + meter_header.track_count = + u32::try_from(track_count).map_err(|_| "web.live_controls.effects")?; + meter_header.submix_count = + u32::try_from(submix_count).map_err(|_| "web.live_controls.effects")?; + meter_header.master_track_plus_one = master_track.map_or(0, |track| track.saturating_add(1)); + meter_header.frame_pointer = address_word(meter_frame.as_ptr()); for (reader, handle) in observations.iter().enumerate() { let Some(strip) = strip_index(&tracks, &submixes, &handle.track_id) else { return Err("web.live_controls.observation"); @@ -5607,10 +5718,53 @@ fn build_plan_shape( .map(|_| 0) .collect::>() .into_boxed_slice(), + meter_carried_from: (0..meter_count) + .map(|_| 0) + .collect::>() + .into_boxed_slice(), + meter_frame, + meter_header, + #[cfg(any(test, feature = "test-support"))] + successor_address: 0, observation_present: observation_present.into_boxed_slice(), }) } +impl PlanShape { + /// Name `successor` as the header that replaces this plan's (null for none): the `u32` word + /// the page reads, and beside it the full-width pointer a gate compares. + #[inline] + fn name_successor(&mut self, successor: *const WebMeterHeader) { + self.meter_header.successor_pointer = address_word(successor); + #[cfg(any(test, feature = "test-support"))] + { + self.successor_address = successor.addr(); + } + } +} + +/// `3(T + S) + 3` words: two peak lanes and one gain-reduction magnitude per strip, then the +/// master's (issue #1209 D1). With `S = 0` it is the `3T + 3` frame every reader already knows. +fn zero_meter_frame(strip_count: usize) -> Result, &'static str> { + let count = strip_count + .checked_mul(3) + .and_then(|value| value.checked_add(3)) + .ok_or("web.resource.arithmetic")?; + let mut value = Vec::new(); + value + .try_reserve_exact(count) + .map_err(|_| "web.resource.allocation")?; + value.resize(count, 0.0); + Ok(value.into_boxed_slice()) +} + +/// The linear-memory address a header names a structure by: the pointer's address as a `u32`, or +/// `0` where the address does not fit (a native test host; the shipped `wasm32` module's always +/// does). The same conversion the exports use. +fn address_word(pointer: *const T) -> u32 { + u32::try_from(pointer.addr()).unwrap_or(0) +} + /// `count` zero words, or the allocation refusal. fn zero_u32(count: usize) -> Result, &'static str> { let mut value = Vec::new(); @@ -5634,6 +5788,10 @@ struct ShapeRows { pending_bytes: u64, /// The per-meter remaining counts. remaining_bytes: u64, + /// The per-meter carried-from handles. + carried_bytes: u64, + /// The published meter frame (#1290d1). + frame_bytes: u64, } impl PlanShape { @@ -5744,12 +5902,20 @@ impl PlanShape { let remaining_bytes = u64::try_from(self.meter_remaining.len()) .ok()? .checked_mul(size_of::() as u64)?; + let carried_bytes = u64::try_from(self.meter_carried_from.len()) + .ok()? + .checked_mul(size_of::() as u64)?; + let frame_bytes = u64::try_from(self.meter_frame.len()) + .ok()? + .checked_mul(size_of::() as u64)?; Some(ShapeRows { arm_table_bytes, arm_sample_bytes, arm_largest: arm_sample_largest.max(arm_table_bytes), pending_bytes, remaining_bytes, + carried_bytes, + frame_bytes, }) } @@ -5762,11 +5928,15 @@ impl PlanShape { .checked_add(rows.arm_table_bytes)? .checked_add(rows.arm_sample_bytes)? .checked_add(rows.pending_bytes)? - .checked_add(rows.remaining_bytes)?; + .checked_add(rows.remaining_bytes)? + .checked_add(rows.carried_bytes)? + .checked_add(rows.frame_bytes)?; let largest = shell .max(rows.arm_largest) .max(rows.pending_bytes) - .max(rows.remaining_bytes); + .max(rows.remaining_bytes) + .max(rows.carried_bytes) + .max(rows.frame_bytes); Some((total, largest)) } } @@ -5856,7 +6026,7 @@ fn compile_ready( mut plan, } = control_plane::compile_children::(document, limits, Box::new(preparer)) .map_err(|failure| boot_refusal(&failure, &record))?; - let (engine, master_track, live_values_superseded, shape) = { + let (engine, live_values_superseded, shape) = { let mut record = record .lock() .map_err(|_| fixed_diagnostic("web.internal.boot_record"))?; @@ -5865,9 +6035,7 @@ fn compile_ready( record.superseded.take(), record.shape.take(), ) { - (Some(engine), Some(superseded), Some(shape)) => { - (engine, record.master_track, superseded, shape) - } + (Some(engine), Some(superseded), Some(shape)) => (engine, superseded, shape), _ => return Err(fixed_diagnostic("web.internal.boot_record").into()), } }; @@ -5922,10 +6090,6 @@ fn compile_ready( // #1290a: the tables were built by the boot plan's own preparation (`build_plan_shape`), the // same function every replacement plan's preparation runs. The companions' spectrum capture is // the one thing moved out here, to the spectrum service. - let model = session.normalized_model(); - let track_count = model.tracks.len(); - let submix_count = model.submixes.len(); - let strip_count = track_count + submix_count; let companions = plan .active_attachment_mut::() .ok_or_else(|| fixed_diagnostic("web.internal.companions"))?; @@ -5952,24 +6116,22 @@ fn compile_ready( report.largest_named_allocation_bytes = report .largest_named_allocation_bytes .max(report.largest_bridge_allocation_bytes); - let mut meter_header = empty_meter_header(); - meter_header.track_count = - u32::try_from(track_count).map_err(|_| fixed_diagnostic("web.live_controls.effects"))?; - meter_header.submix_count = - u32::try_from(submix_count).map_err(|_| fixed_diagnostic("web.live_controls.effects"))?; - meter_header.master_track_plus_one = master_track.map_or(0, |track| track.saturating_add(1)); let master_capacity = companions .meters() .first() .map_or(0, |meter| meter.consumer.capacity()); let pending_meter_bytes = rows.pending_bytes; let remaining_meter_bytes = rows.remaining_bytes; + let carried_meter_bytes = rows.carried_bytes; + let frame_meter_bytes = rows.frame_bytes; let master_meter_bytes = u64::try_from(master_capacity) .ok() .and_then(|count| count.checked_mul(size_of::>() as u64)) .ok_or_else(|| fixed_diagnostic("web.resource.arithmetic"))?; let meter_delivery_bytes = pending_meter_bytes .checked_add(remaining_meter_bytes) + .and_then(|bytes| bytes.checked_add(carried_meter_bytes)) + .and_then(|bytes| bytes.checked_add(frame_meter_bytes)) .and_then(|bytes| bytes.checked_add(master_meter_bytes)) .ok_or_else(|| fixed_diagnostic("web.resource.arithmetic"))?; report.bridge_metadata_bytes = report @@ -5984,6 +6146,8 @@ fn compile_ready( .largest_bridge_allocation_bytes .max(pending_meter_bytes) .max(remaining_meter_bytes) + .max(carried_meter_bytes) + .max(frame_meter_bytes) .max(master_meter_bytes); report.largest_named_allocation_bytes = report .largest_named_allocation_bytes @@ -6040,7 +6204,6 @@ fn compile_ready( pending }, blocking_rebuilds: 0, - meter_header, command_decoded: boxed_observe_staging(decoded_count)?, has_in_flight_commands: false, spectrum: spectrum_capture.map(|captures| { @@ -6051,7 +6214,6 @@ fn compile_ready( spectrum_hop, ) }), - meter_frame: boxed_zero_meter_frame(strip_count)?, master_peak: [0.0, 0.0], master_start_sample: None, master_end_sample: 0, @@ -6101,21 +6263,6 @@ fn live_control_request( }) } -/// `3(T + S) + 3` words: two peak lanes and one gain-reduction magnitude per strip, then the -/// master's (issue #1209 D1). With `S = 0` it is the `3T + 3` frame every reader already knows. -fn boxed_zero_meter_frame(strip_count: usize) -> Result, Vec> { - let count = strip_count - .checked_mul(3) - .and_then(|value| value.checked_add(3)) - .ok_or_else(|| fixed_diagnostic("web.resource.arithmetic"))?; - let mut value = Vec::new(); - value - .try_reserve_exact(count) - .map_err(|_| fixed_diagnostic("web.resource.allocation"))?; - value.resize(count, 0.0); - Ok(value.into_boxed_slice()) -} - /// The decode staging of the observation records: `count` entries, none when the boot asked for no /// observation taps. fn boxed_observe_staging(count: usize) -> Result, Vec> { diff --git a/hosts/host-web/src/tests.rs b/hosts/host-web/src/tests.rs index 08b22c52f..fb6ad703e 100644 --- a/hosts/host-web/src/tests.rs +++ b/hosts/host-web/src/tests.rs @@ -284,8 +284,10 @@ fn frozen_layouts_and_values_are_exact() { assert_eq!(MAXIMUM_OBSERVATION_TAPS, 16); // The meter header is a new fixed structure, not a change to an existing one. // Issue #1209 D2 appended `submix_count` and its pad: 64 bytes became 72, nothing moved. - assert_eq!(size_of::(), 72); - assert_eq!(METER_HEADER_BYTES, 72); + // Issue #1290d1 turned the pad into `retired` and appended the frame and successor pointers: + // 72 bytes became 80, nothing moved. + assert_eq!(size_of::(), 80); + assert_eq!(METER_HEADER_BYTES, 80); assert_eq!(offset_of!(WebMeterHeader, track_count), 8); assert_eq!(offset_of!(WebMeterHeader, windows), 12); assert_eq!(offset_of!(WebMeterHeader, first_sample), 16); @@ -295,7 +297,9 @@ fn frozen_layouts_and_values_are_exact() { assert_eq!(offset_of!(WebMeterHeader, master_gr_present), 44); assert_eq!(offset_of!(WebMeterHeader, reserved), 48); assert_eq!(offset_of!(WebMeterHeader, submix_count), 64); - assert_eq!(offset_of!(WebMeterHeader, reserved_pad), 68); + assert_eq!(offset_of!(WebMeterHeader, retired), 68); + assert_eq!(offset_of!(WebMeterHeader, frame_pointer), 72); + assert_eq!(offset_of!(WebMeterHeader, successor_pointer), 76); assert_eq!(offset_of!(WebCommandReport, result), 8); assert_eq!(offset_of!(WebCommandReport, rejected_index), 16); assert_eq!(offset_of!(WebCommandReport, applied_at_sample), 24); @@ -918,17 +922,23 @@ fn observation_staging_resource_is_exact_for_the_taps_off_and_on_modes() { + host_core::LiveValuesSuperseded::retained_bytes(), "{name}: the control plane's row holds the rows an independent preparation reports" ); + // The published meter frame is part of the plan's tables, charged with them (#1290d1): + // `3(T + S) + 3` words, held whether or not metering is on. + let model = compiled.normalized_model(); + let frame_bytes = (3 * (model.tracks.len() + model.submixes.len()) as u64 + 3) * 4; let expected_bridge_metadata = projection .report .bridge_metadata_bytes - .checked_add(control_retained) + .checked_add(frame_bytes) + .and_then(|bytes| bytes.checked_add(control_retained)) .and_then(|bytes| bytes.checked_add(session_model_bytes)) .and_then(|bytes| bytes.checked_add(decoded_bytes)) .expect("bridge metadata arithmetic"); let expected_bridge_retained = projection .report .bridge_retained_bytes - .checked_add(control_retained) + .checked_add(frame_bytes) + .and_then(|bytes| bytes.checked_add(control_retained)) .and_then(|bytes| bytes.checked_add(session_model_bytes)) .and_then(|bytes| bytes.checked_add(decoded_bytes)) .expect("bridge retained arithmetic"); @@ -3387,10 +3397,11 @@ fn the_meter_frame_carries_a_peak_pair_and_a_gain_word_per_submix() { let mut host = bus_meter_host(QUANTUM, true); let frame = bus_meter_frame(&mut host, BLOCKS); let header = *host.meter_header(); - assert_eq!(header.struct_size, 72); + assert_eq!(header.struct_size, 80); assert_eq!(header.track_count, 3); assert_eq!(header.submix_count, 2); - assert_eq!(header.reserved_pad, 0); + assert_eq!(header.retired, 0); + assert_eq!(header.successor_pointer, 0); let (tracks, strips) = (3_usize, 5_usize); assert_eq!(frame.len(), 3 * strips + 3, "3(T + S) + 3 words"); @@ -5850,6 +5861,7 @@ fn web_runtime_preparer_attaches_the_companions_on_every_branch() { revision: 1, edit_ramps: &edit_ramps, monitor: host_core::MonitorBase::NONE, + observation: &host_core::NO_OBSERVATION, }; let (mut successor, successor_handles) = preparer .prepare(&compiled, &caps, lanes, Some(base), &superseded) @@ -6645,22 +6657,48 @@ mod structural_apply { windows } - /// The meter delivery across a swap (#1381b): a swap that keeps the strip count carries the - /// predecessor's meters, so the delivery epoch the page sees does not change and the windows - /// follow each other without a gap, and a swap that changes the count publishes nothing rather - /// than a frame whose words name other strips (the worklet's views over the frame keep the - /// boot's size, #1290d). + /// One published meter frame: the header and the frame's words. + type Published = (WebMeterHeader, Vec); + + /// Drive `blocks` more blocks with the lease on, polling after the blocks whose zero-based + /// index leaves `phase` modulo `every`; every frame published, in order. + fn poll_cadence(rig: &mut Rig, blocks: u64, every: u64, phase: u64) -> Vec { + let mut frames = Vec::new(); + for _ in 0..blocks { + let index = rig.block; + rig.step(); + if index % every == phase && rig.host.poll_meters() > 0 { + frames.push((*rig.host.meter_header(), rig.host.meter_frame().to_vec())); + } + } + frames + } + + /// The frames a fresh boot of `model` publishes over `blocks` blocks, polled after each, keyed + /// by the window's first sample: what the swapped host must publish for the same window. + fn fresh_windows(model: &SessionModel, blocks: u64) -> Vec { + let mut rig = Rig::boot(model, options(), None); + assert_eq!(rig.host.set_meter_lease(true), RESULT_OK); + poll_cadence(&mut rig, blocks, 1, 0) + } + + fn window_of(frames: &[Published], first_sample: u64) -> &Published { + frames + .iter() + .find(|(header, _)| header.first_sample == first_sample) + .unwrap_or_else(|| panic!("no window opens at sample {first_sample}")) + } + + /// The meter delivery across a swap that keeps the strips (#1381b): the predecessor's meters + /// carry, so the delivery epoch the page sees does not change and the windows follow each + /// other without a gap. /// /// Test value: red if the swap block's hook does not pair the meter consumers (the successor's /// meters would sit on observers the retiring plan owns and no window would ever fold again), - /// if the swap restarts the delivery epoch or discards the windows waiting for a poll, or if - /// the poll reads past the boot-sized frame or the successor's meters after a swap that adds a - /// strip. + /// or if the swap restarts the delivery epoch or discards the windows waiting for a poll. #[test] - fn meter_delivery_carries_through_a_swap_and_never_publishes_a_misshaped_frame() { + fn meter_delivery_carries_through_a_swap_that_keeps_the_strips() { let model = gate_model(); - - // A swap that keeps the strips: an unused source is added. let mut source = model.sources[0].clone(); source.id = StableId::parse("a-unused").expect("source ID"); let mut rig = Rig::boot(&model, options(), None); @@ -6695,13 +6733,31 @@ mod structural_apply { published.reserved[1], "no window was lost to the swap" ); + } + + /// A swap that adds a strip publishes the successor's frame (#1290d1): the frame travels with + /// the plan, so the window the adoption block closes is described by a `3(T + S) + 3` frame for + /// the plan's own strips, bit for bit what a fresh boot of the committed model publishes for + /// that window, in the same stream (generation, sequence, no counted loss). + /// + /// Test value: red if the frame keeps the boot's size (the poll would publish nothing, or a + /// frame whose words name other strips), if the header keeps the predecessor's strip counts, + /// if the swap restarts the stream, or if the added strip's words land at the wrong index. + #[test] + fn a_strip_count_change_publishes_the_successors_frame_from_the_adoption_block() { + let model = gate_model(); + let committed = with_muted_track(&model, ADDED); + // Windows close after the odd blocks. The transaction runs before block 5, so the block + // that adopts the successor is the one that closes the window opening at sample 512. + let apply_before = 5; + let reference = fresh_windows(&committed, 8); - // A swap that adds a strip. let mut rig = Rig::boot(&model, options(), None); assert_eq!(rig.host.set_meter_lease(true), RESULT_OK); - assert!(poll_blocks(&mut rig, 6) > 0); - let frame = rig.host.meter_frame().to_vec(); - let header = *rig.host.meter_header(); + let before = poll_cadence(&mut rig, apply_before, 1, 0); + let last = before.last().expect("the boot plan published windows"); + assert_eq!(last.0.track_count, 9); + assert_eq!(last.1.len(), 3 * 9 + 3); let base = rig.host.committed_revision(); assert_eq!( rig.apply(&add_muted_track(&model, ADDED), 1), @@ -6710,10 +6766,218 @@ mod structural_apply { revision: base + 1 } ); - assert_eq!(poll_blocks(&mut rig, 8), 0, "no frame describes ten strips"); - assert_eq!(rig.host.live_control_tracks().len(), 10); - assert_eq!(rig.host.meter_frame(), &frame[..], "the frame is untouched"); - assert_eq!(rig.host.meter_header().sequence, header.sequence); + // The candidate's header is named by the one that renders, before the adoption block. + { + let ready = rig.host.ready.as_ref().expect("ready"); + let waiting = ready.pending_shapes.last().expect("the candidate's tables"); + assert_eq!(waiting.shape.meter_header.track_count, 10); + assert_eq!(waiting.shape.meter_frame.len(), 3 * 10 + 3); + // Full width: the `u32` word reads 0 for an address above 4 GiB on a native host, so + // the check is on the pointer the commit set beside it. + assert_ne!(rig.host.meter_successor_address(), 0); + assert_eq!( + rig.host.meter_successor_address(), + core::ptr::from_ref(&waiting.shape.meter_header).addr(), + "the rendering plan names the newest candidate's header" + ); + assert_eq!(ready.shape.meter_header.retired, 0); + } + let after = poll_cadence(&mut rig, 1, 1, 0); + assert_eq!(rig.epoch(), 1); + assert_eq!( + after.len(), + 1, + "the first block after adoption publishes: the successor's frame is not too small" + ); + let (header, frame) = &after[0]; + assert_eq!(header.track_count, 10); + assert_eq!(header.submix_count, 0); + assert_eq!(header.master_track_plus_one, 0); + assert_eq!( + frame.len(), + 3 * 10 + 3, + "a frame sized for the successor's strips" + ); + assert_eq!(header.first_sample, 512); + assert_eq!( + header.reserved[0], last.0.reserved[0], + "the stream's generation is the plan's predecessor's" + ); + assert_eq!( + header.sequence, + last.0.sequence + 1, + "the sequence continues" + ); + assert_eq!(header.reserved[1] >> 32, 0, "no window was counted lost"); + let (expected_header, expected) = window_of(&reference, 512); + assert_eq!(header.reserved[1], expected_header.reserved[1]); + assert_eq!( + frame.iter().map(|word| word.to_bits()).collect::>(), + expected + .iter() + .map(|word| word.to_bits()) + .collect::>(), + "the window is what a fresh boot of the committed model publishes" + ); + assert!( + frame.iter().any(|word| *word != 0.0), + "the frame is not silence" + ); + + // The header the swap retired says so and names the one that replaced it; the adopted + // header carries no successor (nothing waits). + let ready = rig.host.ready.as_ref().expect("ready"); + let retired = ready + .pending_shapes + .iter() + .find(|entry| entry.adopted) + .expect("the retired tables wait for the service step"); + assert_eq!(retired.shape.meter_header.retired, 1); + assert_eq!(retired.shape.meter_header.track_count, 9); + assert_ne!(retired.shape.successor_address, 0); + assert_eq!( + retired.shape.successor_address, + core::ptr::from_ref(&ready.shape.meter_header).addr(), + "the retired header names the adopted one" + ); + assert_ne!( + retired.shape.successor_address, + core::ptr::from_ref(&retired.shape.meter_header).addr() + ); + assert_eq!(ready.shape.meter_header.retired, 0); + assert_eq!(ready.shape.meter_header.successor_pointer, 0); + assert_eq!( + rig.host.meter_successor_address(), + 0, + "the adopted header names no successor" + ); + + // And the stream goes on in the successor's frame. + let later = poll_cadence(&mut rig, 3, 1, 0); + assert!(!later.is_empty()); + for (header, frame) in &later { + assert_eq!(header.track_count, 10); + assert_eq!(frame.len(), 33); + let (_, expected) = window_of(&reference, header.first_sample); + assert_eq!(frame, expected); + } + } + + /// A window the predecessor closed before the swap and the poll finds after it belongs to the + /// strip it was carried to (#1290d1). Removing the first strip shifts every survivor's meter + /// handle down by one, so the windows queued under the old handles must still be accepted: + /// published in the successor's smaller frame with the stream's sequence, nothing counted. + /// + /// Test value: red if the poll matches a queued or pending window by the meter's current + /// handle only (the whole window is dropped and counted, nine times), or if the carried strips + /// are matched by position instead of identity (the frame would name other strips' peaks). + #[test] + fn a_window_closed_before_a_handle_shift_is_published_by_its_carried_strip() { + let model = gate_model(); + let with_added = with_muted_track(&model, ADDED); + let reference = fresh_windows(&model, 10); + + let mut rig = Rig::boot(&with_added, options(), None); + assert_eq!(rig.host.set_meter_lease(true), RESULT_OK); + // Windows close after the odd blocks; polling after the even blocks leaves each window + // queued for one block. The transaction runs before block 6, which adopts the successor + // and is polled: the window that closed after block 5 is polled after the swap. + let before = poll_cadence(&mut rig, 6, 2, 0); + let last = before.last().expect("the boot plan published windows"); + assert_eq!(last.0.track_count, 10); + assert_eq!(last.0.first_sample, 256, "the window of blocks 2 and 3"); + let base = rig.host.committed_revision(); + assert_eq!( + rig.apply( + &[ + SessionEdit::RemoveRoute { + route_id: StableId::parse(&format!("{ADDED}-main")).expect("route ID"), + }, + SessionEdit::RemoveTrack { + track_id: StableId::parse(ADDED).expect("track ID"), + }, + ], + 1 + ), + Outcome::Applied { + path: EditPath::Rebuild, + revision: base + 1 + } + ); + let after = poll_cadence(&mut rig, 4, 2, 0); + assert_eq!(rig.epoch(), 1); + assert_eq!( + after.len(), + 2, + "the windows of blocks 4-5 and 6-7 are published" + ); + let (header, frame) = &after[0]; + assert_eq!( + header.first_sample, 512, + "the window the predecessor closed" + ); + assert_eq!(header.track_count, 9); + assert_eq!(frame.len(), 3 * 9 + 3); + assert_eq!(header.sequence, last.0.sequence + 1); + assert_eq!(header.reserved[0], last.0.reserved[0]); + assert_eq!( + header.reserved[1] >> 32, + 0, + "no window was counted lost to the handle shift" + ); + let (_, expected) = window_of(&reference, 512); + assert_eq!( + frame.iter().map(|word| word.to_bits()).collect::>(), + expected + .iter() + .map(|word| word.to_bits()) + .collect::>(), + "each survivor's peaks are at its own index of the smaller frame" + ); + assert!(frame.iter().any(|word| *word != 0.0)); + let (next, next_frame) = &after[1]; + assert_eq!(next.sequence, header.sequence + 1); + assert_eq!(next.reserved[1] >> 32, 0); + assert_eq!(next_frame, &window_of(&reference, 768).1); + } + + /// The same stream across a swap that adds a strip in front, polled every second block with + /// the polls aligned to the windows (#1290d1, gate 7): nothing is counted lost, the sequence + /// continues, and every window is the fresh boot's. + /// + /// Test value: red if the swap restarts or loses a window the aligned cadence leaves nothing + /// to lose (the stream's generation, sequence or loss count moving across the swap). + #[test] + fn an_aligned_poll_every_second_block_loses_nothing_across_an_added_strip() { + let model = gate_model(); + let committed = with_muted_track(&model, ADDED); + let reference = fresh_windows(&committed, 12); + + let mut rig = Rig::boot(&model, options(), None); + assert_eq!(rig.host.set_meter_lease(true), RESULT_OK); + let before = poll_cadence(&mut rig, 6, 2, 1); + let last = before.last().expect("the boot plan published windows"); + let base = rig.host.committed_revision(); + assert!(matches!( + rig.apply(&add_muted_track(&model, ADDED), 1), + Outcome::Applied { + path: EditPath::Rebuild, + .. + } + )); + let after = poll_cadence(&mut rig, 6, 2, 1); + assert_eq!(rig.epoch(), 1); + assert_eq!(rig.host.committed_revision(), base + 1); + assert_eq!(after.len(), 3); + let mut sequence = last.0.sequence; + for (header, frame) in &after { + sequence += 1; + assert_eq!(header.sequence, sequence); + assert_eq!(header.reserved[0], last.0.reserved[0]); + assert_eq!(header.reserved[1] >> 32, 0, "no counted loss"); + assert_eq!(header.track_count, 10); + assert_eq!(frame, &window_of(&reference, header.first_sample).1); + } } /// An acknowledged subscription survives a swap (#1381b): the tap's armed bit and arm sample @@ -6829,19 +7093,22 @@ mod structural_apply { rig.host.ready.as_ref().expect("ready").meter_loss_count } - /// Telemetry is dropped only with a count. A swap that changes the strip count publishes no - /// frame (the carried meters keep the delivery state, #1381b), so every window it leaves - /// unpublished is counted when the poll finds it unpublishable and added to the loss count - /// the next published frame reports, and the swap does not zero what was counted before it. + /// Telemetry is dropped only with a count. The windows the predecessor closed before a swap + /// that adds a strip and nobody polled before it cannot be described by the successor's frame + /// (the added strip did not exist to measure them): the poll that finds them drops them with a + /// count, the swap does not zero what was counted before it, the next frame published is the + /// successor's, and it reports the loss. /// - /// Test value: red if the unpublished windows of a strip-count change vanish without a count - /// (a page would see a silent gap), or if the swap zeroes the loss count. + /// Test value: red if the windows the successor cannot describe vanish without a count (a page + /// would see a silent gap), if the swap zeroes the loss count, or if a frame is published + /// that mixes the two plans' strips. #[test] - fn meter_windows_a_strip_count_change_leaves_unpublished_are_counted() { + fn windows_an_added_strip_cannot_describe_are_counted_and_the_next_frame_is_the_successors() { let model = gate_model(); let mut rig = Rig::boot(&model, options(), None); assert_eq!(rig.host.set_meter_lease(true), RESULT_OK); assert!(poll_blocks(&mut rig, 6) > 0); + let published = *rig.host.meter_header(); // Windows rendered and not polled when the swap comes, and a loss counted before it. for _ in 0..4 { rig.step(); @@ -6860,31 +7127,35 @@ mod structural_apply { ); rig.step(); assert_eq!(rig.epoch(), 1); - let after_swap = meter_loss(&rig); + assert_eq!(meter_loss(&rig), 5, "the swap keeps the count"); assert_eq!( - after_swap, 5, - "the swap keeps the count; the {waiting} windows waiting are counted by the poll" + rig.host.meter_header().sequence, + published.sequence, + "nothing was published meanwhile" ); - // Every window closed afterwards is polled, found unpublishable and counted. - let windows_before = rig.host.ready.as_ref().expect("ready").master_count as u64; - let mut counted = after_swap; + let mut first = None; for _ in 0..8 { rig.step(); - let closed = rig.host.ready.as_ref().expect("ready").master_count as u64; - assert_eq!(rig.host.poll_meters(), 0); - let ready = rig.host.ready.as_ref().expect("ready"); - assert_eq!( - ready.master_count, 0, - "the poll drains what it cannot publish" - ); - assert_eq!(meter_loss(&rig), counted + closed); - counted += closed; + if rig.host.poll_meters() > 0 { + first = Some((*rig.host.meter_header(), rig.host.meter_frame().to_vec())); + break; + } } + let (header, frame) = first.expect("the successor's window is published"); + assert_eq!(header.track_count, 10); + assert_eq!(frame.len(), 33); + assert!( + header.first_sample >= 10 * 128, + "a window from the adoption block on, not one of the predecessor's" + ); + assert!(header.reserved[1] & METER_VALID_LOSS != 0); assert!( - counted > after_swap + windows_before, - "windows were closed and counted" + header.reserved[1] >> 32 >= 5 + waiting, + "the windows left behind are counted ({} for {waiting} waiting and 5 before)", + header.reserved[1] >> 32 ); + assert_eq!(header.sequence, published.sequence + 1); } /// Run the production control entry on a record the preparer cannot leave its tables in: the diff --git a/hosts/host-web/tests/browser-v1/direct-oracle.mjs b/hosts/host-web/tests/browser-v1/direct-oracle.mjs index 5732ccc67..9607eca0c 100644 --- a/hosts/host-web/tests/browser-v1/direct-oracle.mjs +++ b/hosts/host-web/tests/browser-v1/direct-oracle.mjs @@ -22,8 +22,9 @@ const COMMAND_RECORD_BYTES = ABI_LAYOUT.commandRecord.bytes; const COMMAND_OBSERVE_SUBSCRIBE = 7; const COMMAND_OBSERVE_UNSUBSCRIBE = 8; const BUFFER_METER_FRAME = 7; -// Issue #1209 D2: 72 since `submix_count` and its pad were appended at 64 and 68. -const METER_HEADER_BYTES = 72; +// 80 bytes: `submix_count` at 64 and `retired` at 68 (issue #1209 D2, #1290d1), then +// `frame_pointer` at 72 and `successor_pointer` at 76 (issue #1290d1). +const METER_HEADER_BYTES = 80; const OBSERVATION_WINDOW_BLOCKS = 2; const RESOURCE_NAMES = [ "optionsBytes", "statusBytes", "sessionDocumentBytes", "diagnosticBytes", "idStagingBytes", @@ -400,6 +401,10 @@ async function runObservationTimeline(modulePath, sessionDocument, sourceId, tap assert.equal(header.getUint32(4, true), ABI_VERSION); assert.equal(header.getUint32(8, true), 1); assert.equal(header.getUint32(64, true), 0, "no submix strips"); + // Issue #1290d1: the header names its frame, is not retired, and no replacement waits. + assert.equal(header.getUint32(68, true), 0, "the header is not retired"); + assert.equal(header.getUint32(72, true), framePointer, "the header names its frame"); + assert.equal(header.getUint32(76, true), 0, "no replacement plan waits"); assert.equal(header.getUint32(40, true), taps === 0n ? 0 : 1); const frame = new Float32Array(memoryBuffer, framePointer, 6); assert.equal(exports.miso_engine_web_v1_meter_lease(handle, 1), 0); diff --git a/hosts/host-web/tests/structural_apply_realtime.rs b/hosts/host-web/tests/structural_apply_realtime.rs index f5018f417..3eb69a530 100644 --- a/hosts/host-web/tests/structural_apply_realtime.rs +++ b/hosts/host-web/tests/structural_apply_realtime.rs @@ -413,3 +413,147 @@ fn a_live_edit_in_flight_at_the_adoption_block_costs_no_render_call_an_allocatio assert_eq!(swapped.drops_at_first_service, 1); assert_eq!(swapped.blocking_rebuilds, 1); } + +/// What a metered script published. +struct Metered { + /// Every frame the polls published, in order: the header and the frame's words. + frames: Vec<(host_web::WebMeterHeader, Vec)>, + /// Render-locked allocator calls over every render call and every meter poll. + render_allocations: u32, + /// The frames the polls after the block `ADOPTION_WITH_WINDOW` published. + published_at_adoption: usize, +} + +/// Windows close after the odd blocks (a two-block period), so a transaction before block 5 is +/// adopted by the block that closes the window opening at sample 512. +const ADOPTION_WITH_WINDOW: usize = 5; + +fn run_metered(structural: bool) -> Metered { + std::thread::spawn(move || { + let base = model(); + let handle = boot(&if structural { + base.clone() + } else { + with_muted_track(&base) + }); + assert_eq!( + host_web::miso_engine_web_v1_meter_lease(handle, 1), + RESULT_OK + ); + let mut run = Metered { + frames: Vec::new(), + render_allocations: 0, + published_at_adoption: 0, + }; + for block in 0..BLOCKS { + feed(handle, block); + if structural && block == ADOPTION_WITH_WINDOW { + let revision = host_web::miso_engine_web_v1_control_revision(handle); + native_staging::structural_command( + handle, + &frame(&add_muted_track(&base), 1, revision), + ) + .expect("a live handle") + .expect("the structural transaction publishes"); + } + let allocations = host_web::miso_engine_web_v1_render_allocation_count(); + assert_eq!( + host_web::miso_engine_web_v1_render(handle, QUANTUM_FRAMES), + RESULT_OK, + "block {block}: the render" + ); + let windows = host_web::miso_engine_web_v1_meter_poll(handle); + run.render_allocations += + host_web::miso_engine_web_v1_render_allocation_count() - allocations; + if windows > 0 { + run.frames + .push(native_staging::meter_frame(handle).expect("a handle")); + if block == ADOPTION_WITH_WINDOW { + run.published_at_adoption += 1; + } + } + if structural && block == ADOPTION_WITH_WINDOW { + assert_eq!(host_web::miso_engine_web_v1_service(handle), RESULT_OK); + } + } + assert_eq!(host_web::miso_engine_web_v1_dispose(handle), RESULT_OK); + run + }) + .join() + .expect("the run completes") +} + +/// #1290d1 gate: a strip-count change publishes a correctly shaped frame on the first block after +/// adoption, with zero render allocations. The frame travels with the plan's tables, so the +/// adoption block exchanges a pointer and the poll that follows writes into the successor's frame. +/// +/// Test value: red if the swap block or the poll after it allocates or frees (a frame cut or +/// grown in render), if the first poll after adoption publishes nothing or a frame of the +/// predecessor's size, or if the window is not what a fresh boot of the committed model publishes +/// for it. +#[test] +fn a_strip_count_change_publishes_the_successors_frame_on_the_first_block_after_adoption() { + let _turn = TURN.lock().expect("the turn"); + let fresh = run_metered(false); + let swapped = run_metered(true); + + assert_eq!(fresh.render_allocations, 0); + assert_eq!( + swapped.render_allocations, 0, + "a render call or a poll allocated or freed" + ); + assert_eq!( + swapped.published_at_adoption, 1, + "the poll after the adoption block publishes" + ); + let sizes = |run: &Metered| { + run.frames + .iter() + .map(|(header, frame)| (header.track_count, frame.len())) + .collect::>() + }; + let published = sizes(&swapped); + assert_eq!( + published.first(), + Some(&(9, 3 * 9 + 3)), + "the boot plan's frames" + ); + let adopted = published + .iter() + .position(|shape| *shape == (10, 3 * 10 + 3)) + .expect("the successor's frames follow"); + assert!( + published[adopted..] + .iter() + .all(|shape| *shape == (10, 3 * 10 + 3)), + "every frame after the adoption block is the successor's" + ); + let (adoption_header, adoption_frame) = &swapped.frames[adopted]; + assert_eq!( + adoption_header.first_sample, + (ADOPTION_WITH_WINDOW as u64 - 1) * u64::from(QUANTUM_FRAMES) + ); + assert_eq!( + adoption_header.sequence, + swapped.frames[adopted - 1].0.sequence + 1, + "the stream continues" + ); + assert_eq!(adoption_header.reserved[1] >> 32, 0, "nothing counted lost"); + for (header, frame) in &swapped.frames[adopted..] { + let (_, reference) = fresh + .frames + .iter() + .find(|(candidate, _)| candidate.first_sample == header.first_sample) + .expect("the fresh boot published the same window"); + assert_eq!( + frame.iter().map(|word| word.to_bits()).collect::>(), + reference + .iter() + .map(|word| word.to_bits()) + .collect::>(), + "the window opening at {} equals a fresh boot of the committed model", + header.first_sample + ); + } + assert!(adoption_frame.iter().any(|word| *word != 0.0)); +} diff --git a/hosts/host-web/web/miso-engine-v1-audio-worklet-host.d.ts b/hosts/host-web/web/miso-engine-v1-audio-worklet-host.d.ts index 194711154..7536e5077 100644 --- a/hosts/host-web/web/miso-engine-v1-audio-worklet-host.d.ts +++ b/hosts/host-web/web/miso-engine-v1-audio-worklet-host.d.ts @@ -99,7 +99,11 @@ // construction whose four arrays are a `Float32Array` of `2 * trackCount + 2` peaks, one of // `trackCount` gain-reduction magnitudes, and the buses' `2 * submixCount` peaks and // `submixCount` magnitudes, all allocated at construction and overwritten in place; nothing is -// transferred, so the caller keeps no ownership obligation. The single allocation left is the +// transferred, so the caller keeps no ownership obligation. The counts are the plan's: a structural +// apply that adds or removes a strip gives its successor a frame of its own size, whose body and +// views are made in the message handler that committed it (never in the render callback), and the +// first window the successor closes is published in it, in the same stream (`sequence`, +// `generation` and `lossCount` continue). The single allocation left is the // structured clone `postMessage` performs, which is `4 * (3 * (trackCount + submixCount) + 2)` // bytes plus a handful of small numbers -- 392 bytes for a 32-track console with no buses. That cost has not been separately benchmarked in a browser; the telemetry // lease measures the render export, not the post around it. @@ -506,6 +510,8 @@ export interface MisoMeterFrame { readonly lossCount: number; /// Complete windows folded into this frame; normally `1`. readonly windows: number; + /// The tracks of the plan that rendered the window. A structural apply that adds or removes a + /// track changes it between two frames of one stream; every array below is sized for it. readonly trackCount: number; /// `[track0 L, track0 R, .., trackN L, trackN R, master L, master R]` peak magnitudes. readonly peaks: Float32Array; diff --git a/hosts/host-web/web/miso-engine-v1-audio-worklet.js b/hosts/host-web/web/miso-engine-v1-audio-worklet.js index 062542f38..26cda062c 100644 --- a/hosts/host-web/web/miso-engine-v1-audio-worklet.js +++ b/hosts/host-web/web/miso-engine-v1-audio-worklet.js @@ -1,8 +1,13 @@ const ABI_VERSION = 0x00010000; const BOOT_OPTIONS_BYTES = 64; // Issue #143 D5: the fixed structure carrying the sample window an `f32` frame cannot hold. -// Issue #1209 D2 appended `submix_count` at 64 and its pad at 68. -const METER_HEADER_BYTES = 72; +// Issue #1209 D2 appended `submix_count` at 64 and its pad at 68. Issue #1290d1 made the pad +// `retired` and appended the frame and successor pointers: the frame travels with the plan, so the +// header names its own frame and, while a replacement plan waits, the replacement's header. +const METER_HEADER_BYTES = 80; +const METER_HEADER_RETIRED = 68; +const METER_HEADER_FRAME_POINTER = 72; +const METER_HEADER_SUCCESSOR_POINTER = 76; const RESULT_OK = 0; const RESULT_INVALID_ARGUMENT = 1; const RESULT_UNSUPPORTED = 7; @@ -265,6 +270,13 @@ class MisoEngineAudioWorkletProcessor extends AudioWorkletProcessor { // the memory moved before a handler re-bound; the status reports it. this.viewSpecs = []; this.staleViewQuanta = 0; + // Issue #1290d1: the meter frame travels with the plan, so the views over it do too. The + // active set is the views over the frame the host publishes into; the pending set was cut, in + // the message handler that committed a replacement plan, over that plan's header and frame. + // `process()` switches from the one to the other by reference when the header says the plan + // was replaced, and never cuts a view. + this.meterSet = null; + this.pendingMeterSet = null; this.clock = renderClock(); // Issue #1381a D6, `single` mode: the instance's control work runs here, in the message // handler, outside `process()`. The service step follows every message, the main realm's @@ -281,6 +293,9 @@ class MisoEngineAudioWorkletProcessor extends AudioWorkletProcessor { this.drainControlEvents(requestId); } finally { this.rebindMemoryViews(); + // A command may have committed a replacement plan, whose frame is sized for its strips: + // its views are cut here, before any quantum can adopt it (#1290d1). + this.syncMeterViews(); } }; try { @@ -456,6 +471,128 @@ class MisoEngineAudioWorkletProcessor extends AudioWorkletProcessor { for (const spec of this.viewSpecs) { this[spec.field] = new spec.Constructor(buffer, spec.pointer, spec.length); } + if (this.meterSet !== null) this.cutMeterViews(this.meterSet, buffer); + if (this.pendingMeterSet !== null) this.cutMeterViews(this.pendingMeterSet, buffer); + } + + /// Issue #1290d1: cut the views of one meter set over `buffer` and remember nothing else; the + /// set's pointers and counts are its identity. Control side only. + cutMeterViews(set, buffer) { + const strips = set.trackCount + set.submixCount; + set.header = new DataView(buffer, set.headerPointer, METER_HEADER_BYTES); + set.frame = new Float32Array(buffer, set.framePointer, strips * 3 + 3); + // Fixed views over the one buffer, built here and never again: the frozen render-callback + // policy forbids `subarray` inside it, and rightly -- a per-block view is a per-block + // allocation. With buses the master pair no longer follows the tracks, so the message's + // `2T + 2` peaks are assembled from the track view and the master view. + set.trackPeaks = new Float32Array(buffer, set.framePointer, set.trackCount * 2); + set.submixPeaks = new Float32Array( + buffer, set.framePointer + set.trackCount * 2 * 4, set.submixCount * 2); + set.masterPeaks = new Float32Array(buffer, set.framePointer + strips * 2 * 4, 2); + set.trackGain = new Float32Array( + buffer, set.framePointer + (strips * 2 + 2) * 4, set.trackCount); + set.submixGain = new Float32Array( + buffer, set.framePointer + (strips * 2 + 2 + set.trackCount) * 4, set.submixCount); + } + + /// Issue #1290d1: the meter set for the header at `headerPointer`: the views over the header and + /// over the frame the header names, and the message that frame is posted in. `null` for a header + /// or frame the module's memory cannot hold. The strip counts are the plan's, so a plan with + /// another strip count gets a set of its own. Control side only; allocates. + cutMeterSet(headerPointer) { + const buffer = this.memoryBuffer; + if (!u32(headerPointer) || headerPointer === 0 + || headerPointer + METER_HEADER_BYTES > buffer.byteLength) return null; + const header = new DataView(buffer, headerPointer, METER_HEADER_BYTES); + // The header owns the strip counts; the strip order is the core's control list (the owner + // header rows), which the frame's sections follow. + if (header.getUint32(0, true) !== METER_HEADER_BYTES + || header.getUint32(4, true) !== ABI_VERSION) return null; + const trackCount = header.getUint32(8, true); + const submixCount = header.getUint32(64, true); + const framePointer = header.getUint32(METER_HEADER_FRAME_POINTER, true); + // Issue #1209 D1: the frame is `3(T + S) + 3` words -- a peak pair per strip (the tracks, + // then the submixes), the master pair, then one non-negative gain-reduction magnitude per + // strip in the same order and the master's. + const words = (trackCount + submixCount) * 3 + 3; + if (framePointer === 0 || framePointer % 4 !== 0 + || framePointer + words * 4 > buffer.byteLength) return null; + const set = { + headerPointer, + framePointer, + trackCount, + submixCount, + masterGainIndex: words - 1, + header, + frame: null, + trackPeaks: null, + submixPeaks: null, + masterPeaks: null, + trackGain: null, + submixGain: null, + message: { + tag: "miso.meter.v1", + sequence: 0, + generation: 0n, + validity: 0, + lossCount: 0, + windows: 0, + trackCount, + // The frozen `2T + 2` track and master peaks: an existing reader indexes it exactly as + // before, whatever the submix count. + peaks: new Float32Array(trackCount * 2 + 2), + // Issue #143: one non-negative decibel magnitude per track. Positional and always finite; + // `0` deliberately conflates "not reducing" with "no observed effect", because the array + // is read without null checks and the distinction lives in the subscription map. + trackGrDb: new Float32Array(trackCount), + masterGrDb: null, + firstSample: 0n, + endSample: 0n, + // Issue #1209 D4: the buses, appended. `submixPeaks` is `[bus0 L, bus0 R, ..]`, and + // `submixGrDb` one non-negative decibel magnitude per bus, in canonical submix order. + submixCount, + submixPeaks: new Float32Array(submixCount * 2), + submixGrDb: new Float32Array(submixCount), + }, + }; + this.cutMeterViews(set, buffer); + return set; + } + + /// Issue #1290d1: bring the page's meter sets in line with the host's, in a message handler. The + /// active set follows the header the host publishes through (a swap the render path did not + /// switch for, a lease released across it, is picked up here), and the pending set follows the + /// header the active one names as its successor: the replacement plan a command just committed, + /// whose views are cut now so the quantum that adopts it only switches to them. A header or + /// frame the module's memory cannot hold is a protocol fault, sticky. + syncMeterViews() { + if (!this.metersAttached || this.disposed || !this.ready || this.stickyResult !== RESULT_OK + || this.handle === 0) return; + this.rebindMemoryViews(); + const current = this.exports.miso_engine_web_v1_meter_header_ptr(this.handle); + let active = this.meterSet; + if (active === null || active.headerPointer !== current) { + active = this.cutMeterSet(current); + if (active === null) { + this.sticky(RESULT_INTERNAL, this.lastRequestId); + return; + } + this.meterSet = active; + this.pendingMeterSet = null; + } + const successor = active.header.getUint32(METER_HEADER_SUCCESSOR_POINTER, true); + if (successor === 0) { + this.pendingMeterSet = null; + return; + } + if (this.pendingMeterSet === null || this.pendingMeterSet.headerPointer !== successor) { + const pending = this.cutMeterSet(successor); + if (pending === null) { + this.sticky(RESULT_INTERNAL, this.lastRequestId); + return; + } + this.pendingMeterSet = pending; + } } stageSpectrumRequest(options) { @@ -712,62 +849,14 @@ class MisoEngineAudioWorkletProcessor extends AudioWorkletProcessor { this.metersAttached = init.options.liveControlMeterBlocks !== 0n; this.observationAttached = init.options.liveControlObservationTaps !== 0n; if (this.metersAttached) { - // Issue #1209 D1: the frame is `3(T + S) + 3` words -- a peak pair per strip (the tracks, - // then the submixes), the master pair, then one non-negative gain-reduction magnitude per - // strip in the same order and the master's. The header says how many submixes it holds. const headerPointer = this.exports.miso_engine_web_v1_meter_header_ptr(this.handle); if (!u32(headerPointer) || headerPointer === 0) return false; - this.bindView("meterHeaderView", DataView, headerPointer, METER_HEADER_BYTES); - if (this.meterHeaderView.getUint32(0, true) !== METER_HEADER_BYTES - || this.meterHeaderView.getUint32(4, true) !== ABI_VERSION) { - return false; - } - // The header owns the strip counts; the strip order is the core's control list (the - // owner header rows), which the frame's sections follow. The capacity check below rejects - // a header whose counts do not match the frame the engine sized. - this.trackCount = this.meterHeaderView.getUint32(8, true); - this.submixCount = this.meterHeaderView.getUint32(64, true); - const strips = this.trackCount + this.submixCount; - if (!u32(framePointer) || framePointer === 0 - || frameCapacity !== (strips * 3 + 3) * 4) return false; - this.bindView("meterView", Float32Array, framePointer, frameCapacity / 4); - // Fixed views over the one buffer, built here and never again: the frozen render-callback - // policy forbids `subarray` inside it, and rightly -- a per-block view is a per-block - // allocation. With buses the master pair no longer follows the tracks, so the message's - // `2T + 2` peaks are assembled from the track view and the master view. - this.bindView("meterTrackPeakView", Float32Array, framePointer, this.trackCount * 2); - this.bindView("meterSubmixPeakView", Float32Array, - framePointer + this.trackCount * 2 * 4, this.submixCount * 2); - this.bindView("meterMasterPeakView", Float32Array, framePointer + strips * 2 * 4, 2); - this.bindView("meterGainView", Float32Array, - framePointer + (strips * 2 + 2) * 4, this.trackCount); - this.bindView("meterSubmixGainView", Float32Array, - framePointer + (strips * 2 + 2 + this.trackCount) * 4, this.submixCount); - this.meterMasterGainIndex = strips * 3 + 2; - this.meterMessage = { - tag: "miso.meter.v1", - sequence: 0, - generation: 0n, - validity: 0, - lossCount: 0, - windows: 0, - trackCount: this.trackCount, - // The frozen `2T + 2` track and master peaks: an existing reader indexes it exactly as - // before, whatever the submix count. - peaks: new Float32Array(this.trackCount * 2 + 2), - // Issue #143: one non-negative decibel magnitude per track. Positional and always finite; - // `0` deliberately conflates "not reducing" with "no observed effect", because the array - // is read without null checks and the distinction lives in the subscription map. - trackGrDb: new Float32Array(this.trackCount), - masterGrDb: null, - firstSample: 0n, - endSample: 0n, - // Issue #1209 D4: the buses, appended. `submixPeaks` is `[bus0 L, bus0 R, ..]`, and - // `submixGrDb` one non-negative decibel magnitude per bus, in canonical submix order. - submixCount: this.submixCount, - submixPeaks: new Float32Array(this.submixCount * 2), - submixGrDb: new Float32Array(this.submixCount), - }; + const set = this.cutMeterSet(headerPointer); + // The buffer exports and the header describe one frame: the capacity check rejects a header + // whose counts do not match the frame the engine sized. + if (set === null || !u32(framePointer) || framePointer !== set.framePointer + || frameCapacity !== (set.trackCount + set.submixCount) * 3 * 4 + 12) return false; + this.meterSet = set; } this.telemetryMessage = { tag: "miso.telemetry.v1", @@ -1684,31 +1773,52 @@ class MisoEngineAudioWorkletProcessor extends AudioWorkletProcessor { /// allocation, and it is the one this ABI exists to pay: one flat numeric payload per window /// instead of one per block. postMeterFrame() { + let set = this.meterSet; + // A host booted without meters has no frame; the lease is then a no-op the host refuses. + if (set === null) return; + // Issue #1290d1: a block that adopted a replacement plan published its frame and header in + // the plan's own, and marked the header it left. Switch to the views the handler that + // committed the plan cut over them: a reference assignment, no view is made here. A header + // that says it was replaced, without the set cut for the header it names, is a fault. + if (set.header.getUint32(METER_HEADER_RETIRED, true) !== 0) { + const next = this.pendingMeterSet; + if (next === null + || next.headerPointer !== set.header.getUint32(METER_HEADER_SUCCESSOR_POINTER, true)) { + this.stickyResult = RESULT_INTERNAL; + this.ready = false; + return; + } + this.meterSet = next; + this.pendingMeterSet = null; + set = next; + } const windows = this.exports.miso_engine_web_v1_meter_poll(this.handle); if (windows === 0) return; - const metadata = this.meterHeaderView.getBigUint64(56, true); + const header = set.header; + const metadata = header.getBigUint64(56, true); // The Rust adapter only publishes complete track/master spans. Keep this guard at the // transport boundary so a future invalid publication cannot reach legacy numeric consumers. if ((metadata & 1n) === 0n || (metadata & 2n) === 0n) return; this.meterSequence += 1; - this.meterMessage.sequence = this.meterSequence; - this.meterMessage.generation = this.meterHeaderView.getBigUint64(48, true); - this.meterMessage.validity = Number(metadata & 0xffffffffn); - this.meterMessage.lossCount = Number(metadata >> 32n); - this.meterMessage.windows = windows; - this.meterMessage.peaks.set(this.meterTrackPeakView); - this.meterMessage.peaks.set(this.meterMasterPeakView, this.trackCount * 2); - this.meterMessage.submixPeaks.set(this.meterSubmixPeakView); + const message = set.message; + message.sequence = this.meterSequence; + message.generation = header.getBigUint64(48, true); + message.validity = Number(metadata & 0xffffffffn); + message.lossCount = Number(metadata >> 32n); + message.windows = windows; + message.peaks.set(set.trackPeaks); + message.peaks.set(set.masterPeaks, set.trackCount * 2); + message.submixPeaks.set(set.submixPeaks); // Issue #143 D5: the gain-reduction section rides the same post. There is no second message // and no second clock -- the pinned-occurrence rule does not move. - this.meterMessage.trackGrDb.set(this.meterGainView); - this.meterMessage.submixGrDb.set(this.meterSubmixGainView); - this.meterMessage.masterGrDb = this.meterHeaderView.getUint32(44, true) === 1 - ? this.meterView[this.meterMasterGainIndex] + message.trackGrDb.set(set.trackGain); + message.submixGrDb.set(set.submixGain); + message.masterGrDb = header.getUint32(44, true) === 1 + ? set.frame[set.masterGainIndex] : null; - this.meterMessage.firstSample = this.meterHeaderView.getBigUint64(16, true); - this.meterMessage.endSample = this.meterHeaderView.getBigUint64(24, true); - this.port.postMessage(this.meterMessage); + message.firstSample = header.getBigUint64(16, true); + message.endSample = header.getBigUint64(24, true); + this.port.postMessage(set.message); } /// Fold one render-time window and post it (issue #137 D3). diff --git a/scripts/check-abi-layout-v1.py b/scripts/check-abi-layout-v1.py index d9dadf28b..23e0026c3 100644 --- a/scripts/check-abi-layout-v1.py +++ b/scripts/check-abi-layout-v1.py @@ -226,7 +226,7 @@ "bootOptions": 64, "status": 80, "resourceReport": 224, - "meterHeader": 72, + "meterHeader": 80, "commandReport": 48, "planWatermark": 96, "observationSelection": 32, diff --git a/scripts/check-protocol-wasm-parity.sh b/scripts/check-protocol-wasm-parity.sh index 38f677ba9..514e04c99 100755 --- a/scripts/check-protocol-wasm-parity.sh +++ b/scripts/check-protocol-wasm-parity.sh @@ -169,14 +169,14 @@ PY ) local -a files=("$PIN_FILE" "$PIN_FILE" "$GUEST_FILE") local -a searches=( - 'COMPLETE_SCHEMA_HASH: u64 = 0x4b21_d9e6_f520_afd9;' - 'COMPLETE_SCHEMA_HASH: u64 = 0x4b21_d9e6_f520_afd9;' - 'assert_eq!(corpus.len(), 45);' + 'COMPLETE_SCHEMA_HASH: u64 = 0xcc9a_7bcc_502c_4936;' + 'COMPLETE_SCHEMA_HASH: u64 = 0xcc9a_7bcc_502c_4936;' + 'assert_eq!(corpus.len(), 46);' ) local -a replacements=( 'COMPLETE_SCHEMA_HASH: u64 = 0x88a8_ee6a_6d9e_4acc;' 'COMPLETE_SCHEMA_HASH: u64 = 0x0bad_0bad_0bad_0bad;' - 'assert_eq!(corpus.len(), 44);' + 'assert_eq!(corpus.len(), 45);' ) local index for index in "${!labels[@]}"; do diff --git a/scripts/check-web-audioworklet.sh b/scripts/check-web-audioworklet.sh index 784d6aa0c..653a176ea 100755 --- a/scripts/check-web-audioworklet.sh +++ b/scripts/check-web-audioworklet.sh @@ -14,8 +14,11 @@ process_policy_re='BigInt|new[[:space:]]|subarray|memory\.grow|WebAssembly|fetch # exactly this way, each posting a body preallocated at construction, and nothing else. A third # post, a post of a freshly built object, or a post that is not one of these two lines fails the # gate. `new ` remains banned in the body, so the frames cannot be allocated here either. +# Issue #1290d1: the meter body is the active meter set's, preallocated with its views in the +# message handler that committed the plan (a replacement plan's frame is sized for its strips), so +# the pinned spelling names the set. process_policy_posts=( - 'this.port.postMessage(this.meterMessage);' + 'this.port.postMessage(set.message);' 'this.port.postMessage(frame);' ) check_process_policy() { diff --git a/scripts/fixtures/abi-layout-v1-self-test.json b/scripts/fixtures/abi-layout-v1-self-test.json index f645f26b9..b8736305a 100644 --- a/scripts/fixtures/abi-layout-v1-self-test.json +++ b/scripts/fixtures/abi-layout-v1-self-test.json @@ -177,7 +177,7 @@ ] }, "meterHeader": { - "bytes": 72, + "bytes": 80, "fields": [ { "name": "structSize", "offset": 0, "type": "u32" }, { "name": "abiVersion", "offset": 4, "type": "u32" }, @@ -190,7 +190,9 @@ { "name": "masterGrPresent", "offset": 44, "type": "u32" }, { "name": "reserved", "offset": 48, "type": "u64[2]" }, { "name": "submixCount", "offset": 64, "type": "u32" }, - { "name": "reservedPad", "offset": 68, "type": "u32" } + { "name": "retired", "offset": 68, "type": "u32" }, + { "name": "framePointer", "offset": 72, "type": "u32" }, + { "name": "successorPointer", "offset": 76, "type": "u32" } ] }, "commandReport": { diff --git a/scripts/test-web-audioworklet.mjs b/scripts/test-web-audioworklet.mjs index 243726c21..1f405efd4 100644 --- a/scripts/test-web-audioworklet.mjs +++ b/scripts/test-web-audioworklet.mjs @@ -2606,17 +2606,41 @@ function createFakeExports(quantum, backend = 1, liveControlsAttached = true, su const submixCount = submixIds.length; const strips = trackIds.length + submixCount; const meterFrameFloats = strips * 3 + 3; - const meterHeader = new DataView(memory.buffer, meterHeaderPointer, 72); - meterHeader.setUint32(0, 72, true); - meterHeader.setUint32(64, submixCount, true); - meterHeader.setUint32(4, 0x00010000, true); - meterHeader.setUint32(8, trackIds.length, true); - meterHeader.setUint32(40, 1, true); - meterHeader.setUint32(44, 1, true); - meterHeader.setBigUint64(16, 512n, true); - meterHeader.setBigUint64(24, 768n, true); - meterHeader.setBigUint64(48, 1n, true); - meterHeader.setBigUint64(56, 0xbn, true); + // Issue #1290d1: the header is 80 bytes and names its frame (offset 72); `retired` (68) and + // `successor` (76) are zero until a replacement plan is committed and adopted. The replacement + // plan's own header and frame, sized for one more track, sit beside it: `commitMeterSuccessor` + // names it in the active header, as the host does when a command commits a rebuild, and + // `adoptOnNextRender` makes the next render call model the swap block (the old header is marked + // retired and names the new one, whose stream fields are the old header's). + const successorHeaderPointer = 41300; + const successorFramePointer = 44000; + const meterHeader = new DataView(memory.buffer, meterHeaderPointer, 80); + const successorHeader = new DataView(memory.buffer, successorHeaderPointer, 80); + for (const [header, tracks, framePointer] of [ + [meterHeader, trackIds.length, meterFramePointer], + [successorHeader, trackIds.length + 1, successorFramePointer], + ]) { + header.setUint32(0, 80, true); + header.setUint32(64, submixCount, true); + header.setUint32(4, 0x00010000, true); + header.setUint32(8, tracks, true); + header.setUint32(40, 1, true); + header.setUint32(44, 1, true); + header.setBigUint64(16, 512n, true); + header.setBigUint64(24, 768n, true); + header.setBigUint64(48, 1n, true); + header.setBigUint64(56, 0xbn, true); + header.setUint32(72, framePointer, true); + } + let meterAdopted = false; + const adoptMeterPlan = () => { + meterHeader.setUint32(68, 1, true); + meterHeader.setUint32(76, successorHeaderPointer, true); + for (let offset = 12; offset < 64; offset += 4) { + successorHeader.setUint32(offset, meterHeader.getUint32(offset, true), true); + } + meterAdopted = true; + }; let report = new DataView(memory.buffer, reportPointer, 48); report.setUint32(0, 48, true); report.setUint32(4, 0x00010000, true); @@ -2691,6 +2715,10 @@ function createFakeExports(quantum, backend = 1, liveControlsAttached = true, su if (actualFrames !== quantum) return 9; new Float32Array(memory.buffer, 8192, quantum).fill(0.25); new Float32Array(memory.buffer, 8192 + quantum * 4, quantum).fill(-0.25); + if (calls.adoptOnNextRender) { + calls.adoptOnNextRender = false; + adoptMeterPlan(); + } const quanta = status.getBigUint64(40, true) + 1n; status.setBigUint64(40, quanta, true); status.setBigUint64(32, quanta * BigInt(quantum), true); @@ -2720,7 +2748,8 @@ function createFakeExports(quantum, backend = 1, liveControlsAttached = true, su calls.meterLease.push(enabled); return 0; }, - miso_engine_web_v1_meter_header_ptr: () => meterHeaderPointer, + miso_engine_web_v1_meter_header_ptr: () => ( + meterAdopted ? successorHeaderPointer : meterHeaderPointer), miso_engine_web_v1_meter_poll: () => { if (calls.meterWindows === 0) return 0; calls.meterWindows -= 1; @@ -2729,14 +2758,19 @@ function createFakeExports(quantum, backend = 1, liveControlsAttached = true, su // hidden by a uniform fill. Issue #1209: so do the submix sections and the master pair, // which follows the submix peaks, so a worklet that reads the master after the tracks or // a bus from a track's slot is visible too. - const frame = new Float32Array(memory.buffer, meterFramePointer, meterFrameFloats); - const tracks = trackIds.length; + const tracks = trackIds.length + (meterAdopted ? 1 : 0); + const stripsNow = tracks + submixCount; + const frame = new Float32Array( + memory.buffer, + meterAdopted ? successorFramePointer : meterFramePointer, + stripsNow * 3 + 3, + ); frame.fill(0.5, 0, tracks * 2); - frame.fill(0.75, tracks * 2, strips * 2); - frame.fill(0.625, strips * 2, strips * 2 + 2); - frame.fill(6.5, strips * 2 + 2, strips * 2 + 2 + tracks); - frame.fill(2.25, strips * 2 + 2 + tracks, strips * 3 + 2); - frame[strips * 3 + 2] = 7.5; + frame.fill(0.75, tracks * 2, stripsNow * 2); + frame.fill(0.625, stripsNow * 2, stripsNow * 2 + 2); + frame.fill(6.5, stripsNow * 2 + 2, stripsNow * 2 + 2 + tracks); + frame.fill(2.25, stripsNow * 2 + 2 + tracks, stripsNow * 3 + 2); + frame[stripsNow * 3 + 2] = 7.5; return 1; }, miso_engine_web_v1_source_count: () => sourceRows.length, @@ -2826,6 +2860,8 @@ function createFakeExports(quantum, backend = 1, liveControlsAttached = true, su }; return { exports, calls, trackIds, submixIds, sourceRows, meterFrameFloats, meterHeader, growMemory, + successorHeader, commitMeterSuccessor: () => meterHeader.setUint32(76, successorHeaderPointer, true), + memory, }; } @@ -3774,7 +3810,18 @@ async function testProcessor() { `${spec.field} is re-bound to the grown memory`, ); } - assert(processor.viewSpecs.length >= 12, "every persistent view is registered"); + // The meter set's views are the plan's, not registered by name (#1290d1): the active set + // is re-cut with the rest. + for (const field of [ + "header", "frame", "trackPeaks", "submixPeaks", "masterPeaks", "trackGain", "submixGain", + ]) { + assert.equal( + processor.meterSet[field].buffer, + fake.exports.memory.buffer, + `the meter set's ${field} is re-bound to the grown memory`, + ); + } + assert(processor.viewSpecs.length >= 5, "every persistent view is registered"); const left = new Float32Array(64).fill(1); const right = new Float32Array(64).fill(1); assert.equal(processor.process([], [[left, right]]), true); @@ -4198,6 +4245,95 @@ async function testProcessor() { assert.deepEqual([...frame.submixGrDb], Array(S).fill(2.25), `S = ${S}: bus gain reduction`); } + { + // Issue #1290d1: the meter frame travels with the plan. A command that commits a + // replacement plan names the plan's header in the active one, and the handler cuts the + // views over that header and its frame (control side, allocating). The block that adopts the + // plan marks the header it left retired, and `process()` switches to the views by reference: + // it cuts none, so the very objects the handler made are the ones it posts from. + const { processor, fake } = makeProcessor(); + const left = new Float32Array(64); + const right = new Float32Array(64); + processor.receive({ tag: "miso.meters.v1", requestId: 1, enabled: true }); + fake.calls.meterWindows = 1; + assert.equal(processor.process([], [[left, right]]), true); + assert.equal(processor.port.posts.at(-1).message.trackCount, 2); + assert.equal(processor.pendingMeterSet, null, "nothing waits"); + + fake.commitMeterSuccessor(); + processor.port.onmessage({ data: { tag: "miso.service.v1" } }); + const pending = processor.pendingMeterSet; + assert.notEqual(pending, null, "the handler cut the successor's views"); + assert.equal(pending.trackCount, 3); + assert.equal(pending.frame.length, (3 + 1) * 3 + 3); + const active = processor.meterSet; + assert.equal(active.trackCount, 2, "the plan that renders keeps its views"); + + // A block that does not adopt keeps the active set, whatever waits. + fake.calls.meterWindows = 1; + assert.equal(processor.process([], [[left, right]]), true); + assert.equal(processor.meterSet, active); + assert.equal(processor.port.posts.at(-1).message.trackCount, 2); + + // The adopting block: the frame posted is the successor's, sized for its strips. + fake.calls.adoptOnNextRender = true; + fake.calls.meterWindows = 1; + const posts = processor.port.posts.length; + assert.equal(processor.process([], [[left, right]]), true); + assert.equal(processor.port.posts.length, posts + 1, "the first block after adoption posts"); + assert.equal(processor.meterSet, pending, "switched by reference to the set cut in advance"); + assert.equal(processor.pendingMeterSet, null); + const frame = processor.port.posts.at(-1).message; + assert.equal(frame.trackCount, 3); + assert.deepEqual([...frame.peaks], [...Array(6).fill(0.5), 0.625, 0.625]); + assert.deepEqual([...frame.trackGrDb], [6.5, 6.5, 6.5]); + assert.deepEqual([...frame.submixPeaks], [0.75, 0.75]); + assert.deepEqual([...frame.submixGrDb], [2.25]); + assert.equal(frame.masterGrDb, 7.5); + assert.equal(frame.sequence, 3, "the page's sequence continues across the swap"); + + // A memory growth re-cuts both sets, so a later swap still reads live memory. + fake.growMemory(); + processor.port.onmessage({ data: { tag: "miso.service.v1" } }); + assert.equal(processor.meterSet.frame.buffer, fake.exports.memory.buffer); + assert.equal(processor.meterSet.header.buffer, fake.exports.memory.buffer); + } + + { + // The lease released across a swap: nothing switched the views, and the next handler finds + // the host's header is not the one the page holds and cuts the new plan's. + const { processor, fake } = makeProcessor(); + const left = new Float32Array(64); + const right = new Float32Array(64); + fake.commitMeterSuccessor(); + processor.port.onmessage({ data: { tag: "miso.service.v1" } }); + fake.calls.adoptOnNextRender = true; + assert.equal(processor.process([], [[left, right]]), true); + assert.equal(processor.meterSet.trackCount, 2, "no lease, no switch"); + processor.port.onmessage({ data: { tag: "miso.meters.v1", requestId: 1, enabled: true } }); + assert.equal(processor.meterSet.trackCount, 3); + assert.equal(processor.pendingMeterSet, null); + fake.calls.meterWindows = 1; + assert.equal(processor.process([], [[left, right]]), true); + assert.equal(processor.port.posts.at(-1).message.trackCount, 3); + } + + { + // A header that says it was replaced with no set cut for its successor is a protocol + // fault: sticky and silent, never a frame read through views over a retired plan. + const { processor, fake } = makeProcessor(); + const left = new Float32Array(64); + const right = new Float32Array(64); + processor.receive({ tag: "miso.meters.v1", requestId: 1, enabled: true }); + fake.calls.adoptOnNextRender = true; + fake.calls.meterWindows = 1; + const posts = processor.port.posts.length; + assert.equal(processor.process([], [[left, right]]), true); + assert.equal(processor.port.posts.length, posts); + assert.equal(processor.stickyResult, 255); + assert.equal(processor.ready, false); + } + { // Issue #137 D3: a full telemetry window posts exactly one frame, and the frame is honest // about the resolution of the clock it actually found. The real process clock made the diff --git a/scripts/test-web-audioworklet.sh b/scripts/test-web-audioworklet.sh index b1dc10994..61aae249e 100755 --- a/scripts/test-web-audioworklet.sh +++ b/scripts/test-web-audioworklet.sh @@ -165,7 +165,7 @@ echo "web AudioWorklet transitive process-policy mutations passed" # "a pinned post disappears" and "the clock leaves its one pinned site". live_control_mutations=( # A pinned post is renamed: the occurrence count still matches, the pinned line does not. - 's/this\.port\.postMessage(this\.meterMessage);/this.port.postMessage(this.telemetryMessage);/' + 's/this\.port\.postMessage(set\.message);/this.port.postMessage(this.telemetryMessage);/' # The telemetry post is dropped from the window: the count no longer matches. 's/this\.port\.postMessage(frame);/frame.sequence += 0;/' # The lease guard is removed from the meter call site: the frame is posted unconditionally. diff --git a/sdk/assets/miso-engine-v1-abi-layout.json b/sdk/assets/miso-engine-v1-abi-layout.json index f645f26b9..b8736305a 100644 --- a/sdk/assets/miso-engine-v1-abi-layout.json +++ b/sdk/assets/miso-engine-v1-abi-layout.json @@ -177,7 +177,7 @@ ] }, "meterHeader": { - "bytes": 72, + "bytes": 80, "fields": [ { "name": "structSize", "offset": 0, "type": "u32" }, { "name": "abiVersion", "offset": 4, "type": "u32" }, @@ -190,7 +190,9 @@ { "name": "masterGrPresent", "offset": 44, "type": "u32" }, { "name": "reserved", "offset": 48, "type": "u64[2]" }, { "name": "submixCount", "offset": 64, "type": "u32" }, - { "name": "reservedPad", "offset": 68, "type": "u32" } + { "name": "retired", "offset": 68, "type": "u32" }, + { "name": "framePointer", "offset": 72, "type": "u32" }, + { "name": "successorPointer", "offset": 76, "type": "u32" } ] }, "commandReport": { diff --git a/sdk/src/browser/shipped-host.d.ts b/sdk/src/browser/shipped-host.d.ts index 194711154..7536e5077 100644 --- a/sdk/src/browser/shipped-host.d.ts +++ b/sdk/src/browser/shipped-host.d.ts @@ -99,7 +99,11 @@ // construction whose four arrays are a `Float32Array` of `2 * trackCount + 2` peaks, one of // `trackCount` gain-reduction magnitudes, and the buses' `2 * submixCount` peaks and // `submixCount` magnitudes, all allocated at construction and overwritten in place; nothing is -// transferred, so the caller keeps no ownership obligation. The single allocation left is the +// transferred, so the caller keeps no ownership obligation. The counts are the plan's: a structural +// apply that adds or removes a strip gives its successor a frame of its own size, whose body and +// views are made in the message handler that committed it (never in the render callback), and the +// first window the successor closes is published in it, in the same stream (`sequence`, +// `generation` and `lossCount` continue). The single allocation left is the // structured clone `postMessage` performs, which is `4 * (3 * (trackCount + submixCount) + 2)` // bytes plus a handful of small numbers -- 392 bytes for a 32-track console with no buses. That cost has not been separately benchmarked in a browser; the telemetry // lease measures the render export, not the post around it. @@ -506,6 +510,8 @@ export interface MisoMeterFrame { readonly lossCount: number; /// Complete windows folded into this frame; normally `1`. readonly windows: number; + /// The tracks of the plan that rendered the window. A structural apply that adds or removes a + /// track changes it between two frames of one stream; every array below is sized for it. readonly trackCount: number; /// `[track0 L, track0 R, .., trackN L, trackN R, master L, master R]` peak magnitudes. readonly peaks: Float32Array; diff --git a/sdk/src/generated/abi.ts b/sdk/src/generated/abi.ts index bb5db3d14..da213f3ab 100644 --- a/sdk/src/generated/abi.ts +++ b/sdk/src/generated/abi.ts @@ -416,7 +416,7 @@ export const ABI_LAYOUT = deepFreeze( ] }, "meterHeader": { - "bytes": 72, + "bytes": 80, "fields": [ { "name": "structSize", @@ -474,9 +474,19 @@ export const ABI_LAYOUT = deepFreeze( "type": "u32" }, { - "name": "reservedPad", + "name": "retired", "offset": 68, "type": "u32" + }, + { + "name": "framePointer", + "offset": 72, + "type": "u32" + }, + { + "name": "successorPointer", + "offset": 76, + "type": "u32" } ] }, diff --git a/tools/console-workload/src/lib.rs b/tools/console-workload/src/lib.rs index 890d5cf5b..7fe2dc427 100644 --- a/tools/console-workload/src/lib.rs +++ b/tools/console-workload/src/lib.rs @@ -1369,7 +1369,7 @@ impl SessionRuntime { let observations = if config.observation == ObservationArm::Absent { Vec::new() } else { - attach_effect_observation(&mut effects, MAXIMUM_OBSERVATION_TAPS, WINDOW_BLOCKS) + attach_effect_observation(&mut effects, MAXIMUM_OBSERVATION_TAPS, WINDOW_BLOCKS, &[]) .expect("effect observation capacity") }; let input = workload.input(); diff --git a/tools/parameter-metadata/src/abi_layout.rs b/tools/parameter-metadata/src/abi_layout.rs index 249bb3674..66d53d1ae 100644 --- a/tools/parameter-metadata/src/abi_layout.rs +++ b/tools/parameter-metadata/src/abi_layout.rs @@ -468,7 +468,7 @@ fn resource_report_fields() -> [Field; 28] { ] } -fn meter_header_fields() -> [Field; 12] { +fn meter_header_fields() -> [Field; 14] { [ ("structSize", offset_of!(WebMeterHeader, struct_size), "u32"), ("abiVersion", offset_of!(WebMeterHeader, abi_version), "u32"), @@ -497,9 +497,15 @@ fn meter_header_fields() -> [Field; 12] { offset_of!(WebMeterHeader, submix_count), "u32", ), + ("retired", offset_of!(WebMeterHeader, retired), "u32"), ( - "reservedPad", - offset_of!(WebMeterHeader, reserved_pad), + "framePointer", + offset_of!(WebMeterHeader, frame_pointer), + "u32", + ), + ( + "successorPointer", + offset_of!(WebMeterHeader, successor_pointer), "u32", ), ] diff --git a/tools/realtime-policy/render-thread-exports.json b/tools/realtime-policy/render-thread-exports.json index 40703d959..e86a726aa 100644 --- a/tools/realtime-policy/render-thread-exports.json +++ b/tools/realtime-policy/render-thread-exports.json @@ -38,6 +38,7 @@ {"name": "control_response_ptr", "callers": [{"worklet": "engine", "handler": "readControlCommand"}], "wrapped": false, "reason": "single-mode control work (#1562a D1): the address of the last control response, read in the message handler"}, {"name": "control_response_bytes", "callers": [{"worklet": "engine", "handler": "readControlCommand"}], "wrapped": false, "reason": "single-mode control work (#1562a D1): the length of the last control response, read in the message handler"}, {"name": "control_revision", "callers": [{"worklet": "engine", "handler": "readControlCommand"}], "wrapped": false, "reason": "single-mode control work (#1562a, carry-forward 2): the committed revision the SDK's queue seeds from, read in the message handler"}, + {"name": "meter_header_ptr", "callers": [{"worklet": "engine", "handler": "syncMeterViews"}], "wrapped": false, "reason": "single-mode control work (#1290d1): the address of the header the host publishes the meter frame through, read in the message handler to cut the views over a replacement plan's frame, never in process() or a render-locked window"}, {"name": "control_allocation_count", "callers": [{"worklet": "engine", "handler": "receive"}], "wrapped": false, "reason": "the reader of the control allocation count, which the status reports as singleModeControlAllocations"} ] } diff --git a/tools/realtime-policy/src/policy.rs b/tools/realtime-policy/src/policy.rs index 0b1f5238e..0698bd166 100644 --- a/tools/realtime-policy/src/policy.rs +++ b/tools/realtime-policy/src/policy.rs @@ -22,7 +22,12 @@ const RENDER_THREAD_EXPORTS: &str = include_str!("../render-thread-exports.json" /// Issue #1570b adds `control_event` (D1), the event entry: the worklet's message handler dequeues /// both event lanes through it after each message, and `SessionState::dequeue_event` runs the /// service step and encodes the frame, so it is control work of the same kind. -const UNWRAPPED_EXPORTS: [&str; 10] = [ +/// +/// Issue #1290d1 adds `meter_header_ptr`: the worklet's message handler reads the header the host +/// publishes the meter frame through, after each message, to cut the views over a replacement +/// plan's frame in advance. It reads one address and allocates nothing; it is never inside +/// `process()` or a render-locked window. +const UNWRAPPED_EXPORTS: [&str; 11] = [ "dispose", "render_allocation_count", "service", @@ -33,6 +38,7 @@ const UNWRAPPED_EXPORTS: [&str; 10] = [ "control_response_ptr", "control_response_bytes", "control_revision", + "meter_header_ptr", ]; /// One entry of the control-side allowlist (#1443 D3): a function whose `try_pop` calls run on a