From 810692b2a090940771f473a1fd5adbbe5e3df612 Mon Sep 17 00:00:00 2001 From: TATP-233 Date: Thu, 19 Nov 2026 23:33:54 +0800 Subject: [PATCH 1/2] feat(env): publish task-registered custom step-output carriers through the persistent-step-buffer ring Issue #2628: reward/terminated/truncated already publish through the persistent-step-buffer ring, but task transition snapshots had no first-class carrier, so downstream tasks defended against overwrites with per-step detach().clone() and could not rely on fixed addresses. TorchEnv.register_step_carrier registers a task-owned device tensor as a step-output carrier. The ring allocates per-slot buffers with empty_like at the first publish and copies the working carrier into the current slot at every step boundary, so consumers hold the published zero-copy view for published_buffer_depth further steps. Registration is cold-path and fails closed (after the first publish, without the fixed-shape chain, duplicate/empty names, non-tensor or wrong-device carriers). TorchEnvState gains a carriers field; depth 0 publishes the working carriers themselves. The sync-free autoreset user guide documents the step-output lifetime contract, including which outputs remain per-step fresh (info, final_observation). --- docs/sphinx/source/changelog.md | 13 ++ .../1-training/8-sync_free_autoreset.md | 45 +++++ src/unilab/base/torch_env.py | 82 ++++++++- tests/envs/test_step_output_carriers.py | 158 ++++++++++++++++++ 4 files changed, 294 insertions(+), 4 deletions(-) create mode 100644 tests/envs/test_step_output_carriers.py diff --git a/docs/sphinx/source/changelog.md b/docs/sphinx/source/changelog.md index 152c9a3c3..5446c741b 100644 --- a/docs/sphinx/source/changelog.md +++ b/docs/sphinx/source/changelog.md @@ -17,6 +17,19 @@ PyPI 版本变更与未发布变更;发布日期采用 PyPI 上传日期。完 ### Added / 新增 +- `TorchEnv.register_step_carrier` publishes task-owned step-output tensors + (for example transition snapshots) through the persistent-step-buffer + publication ring: consumers get address-stable, zero-copy slot views that + stay valid for one further step instead of per-step defensive clones. The + sync-free autoreset user guide documents the step-output lifetime contract, + including which outputs remain per-step fresh + ([#2628](https://github.com/Motphys/UniLab/issues/2628)). + `TorchEnv.register_step_carrier` 将任务自有的 step 输出张量(如 + transition 快照)通过 persistent step buffer 发布环发布:消费者获得地址 + 稳定、可零拷贝持有一步的槽位视图,替代逐步防御性 clone。sync-free + autoreset 用户指南补充了 step 输出生命周期契约(含哪些输出仍逐步重新 + 分配)说明 + ([#2628](https://github.com/Motphys/UniLab/issues/2628))。 - The documentation site is now versioned with sphinx-multiversion: the `main` track plus the `x.y.0` release of every minor line (v1.0.0, v1.1.0, v1.2.0, v1.3.0, v1.4.0, v1.5.0 at time of writing) are published with a diff --git a/docs/sphinx/source/en/2-user_guide/1-training/8-sync_free_autoreset.md b/docs/sphinx/source/en/2-user_guide/1-training/8-sync_free_autoreset.md index 77ad993ad..98c6de8e6 100644 --- a/docs/sphinx/source/en/2-user_guide/1-training/8-sync_free_autoreset.md +++ b/docs/sphinx/source/en/2-user_guide/1-training/8-sync_free_autoreset.md @@ -112,6 +112,51 @@ lower bound, must be declared with re-applies these transitions after every replay. Undeclared host state keeps the value it had at capture time. +## Step-output lifetime and custom carriers + +With `persistent_step_buffers` (implied by `autoreset_cuda_graph`) the step +outputs are zero-copy views into pre-allocated storage — no per-step output +allocation: + +- `state.obs`, `state.reward`, `state.terminated` and `state.truncated` are + views into the current publication-ring slot. A published state stays valid + for one further step: the ring holds two persistent slots, which the + off-policy double-buffer contract needs (a published `obs_t` stays stable + while the next step writes `obs_tp1`). +- `state.carriers` holds task-registered custom carriers (below); the same + ring semantics apply to them. +- `state.info` is host-side state shared by reference, not ring storage: + `info["steps"]` is a device tensor, but `info["timing"]`, `info["log"]` and + other entries are replaced or mutated every step. Read them before the next + `step` returns. +- `state.final_observation` aliases the env's single working scratch. It is + valid until the next reset segment runs (every step on the fixed-shape + path); clone it if it must be kept longer. + +### Registering a custom step-output carrier + +Tasks that materialize per-step data — transition snapshots, custom metric +tensors — can publish it through the same ring instead of defending against +overwrites with per-step `detach().clone()`: + +```python +snapshot = torch.zeros((env.num_envs, snapshot_dim), device=env.device) +env.register_step_carrier("transition", snapshot) # cold path: before the first reset() + +# ... a term or manager updates `snapshot` in place during update_state ... + +state = env.step(actions) +state.carriers["transition"] # ring-slot view; stable for one further step +``` + +`register_step_carrier` fails closed: it requires the fixed-shape chain +(`env.autoreset_cuda_graph`), must be called before the first `reset` (the +ring is allocated there), rejects duplicate or empty names and carriers that +are not tensors on the env device. Ring slots are allocated with +`torch.empty_like` at the first publish, so every address involved is fixed +from then on — the address-stability precondition for capturing the +step-output publication in a CUDA graph (issue #2628). + ## Opting terms into `ResetSelection` A `ResetSelection` (`src/unilab/base/reset_selection.py`) is a persistent diff --git a/src/unilab/base/torch_env.py b/src/unilab/base/torch_env.py index 0c74ff3e4..d99fd39eb 100644 --- a/src/unilab/base/torch_env.py +++ b/src/unilab/base/torch_env.py @@ -51,7 +51,10 @@ class TorchEnvState: Terminal pre-reset observations are represented only by ``final_observation``; ``info`` never carries duplicate - terminal-observation entries. + terminal-observation entries. ``carriers`` holds task-registered custom + step-output tensors (see ``TorchEnv.register_step_carrier``); like + ``obs``/``reward``, the published state's entries are zero-copy views + into persistent ring storage. """ obs: dict[str, torch.Tensor] @@ -60,6 +63,7 @@ class TorchEnvState: truncated: torch.Tensor info: dict[str, Any] final_observation: dict[str, torch.Tensor] | None = None + carriers: dict[str, torch.Tensor] = dataclasses.field(default_factory=dict) def replace(self, **updates: Any) -> "TorchEnvState": """Return a copy of this transition with ``updates`` applied.""" @@ -245,6 +249,10 @@ def bounded_int(name: str, default: int, minimum: int) -> int: self._publication_ring: list[TorchEnvState] = [] self._publication_cursor = 0 self._published_state: TorchEnvState | None = None + # Task-owned custom step-output carriers (issue #2628), published + # through the same ring as obs/reward/done. Registration closes at + # the first publish; the mapping then backs ``TorchEnvState.carriers``. + self._step_carriers: dict[str, torch.Tensor] = {} @property def device(self) -> torch.device: @@ -307,9 +315,10 @@ def state(self) -> Optional[TorchEnvState]: """The latest published transition. With ``persistent_step_buffers`` this is a publication slot whose - tensors stay unchanged for ``published_buffer_depth - 1`` further - steps (or, at depth 0, the working carriers that the next step - overwrites in place). + tensors — observations, reward, terminated, truncated and the + registered custom ``carriers`` — stay unchanged for + ``published_buffer_depth - 1`` further steps (or, at depth 0, the + working carriers that the next step overwrites in place). """ if self._published_state is not None: return self._published_state @@ -358,6 +367,65 @@ def record_host_transition(self, name: str, transition: Callable[[], None]) -> N return capture.record_host_transition(f"env.{name}", transition) + def register_step_carrier(self, name: str, carrier: torch.Tensor) -> None: + """Register a task-owned tensor as a persistent step-output carrier. + + The carrier is a device tensor the task updates in place during + ``update_state`` — for example a transition snapshot or a custom + per-step metric tensor. With ``persistent_step_buffers`` (implied by + ``autoreset_cuda_graph``) the env copies it, with ``copy_``, into the + current publication-ring slot at every step boundary, so consumers of + the published state can hold the slot's tensor for + ``published_buffer_depth`` further steps without a defensive clone. + Ring slots are pre-allocated with ``torch.empty_like``, so every + address is fixed after the first publish — the address-stability + precondition for capturing the step-output publication in a CUDA + graph (issue #2628). Consumers read the published values through + ``TorchEnvState.carriers``. + + Registration is a cold-path call. It must happen before the first + ``reset``/``step`` (which allocates the publication ring), and it + requires the fixed-shape chain. Fails closed on a duplicate or empty + name, a non-tensor carrier, a carrier that is not on the env device, + a carrier registered after the first publish, or when persistent + step buffers are disabled. + + Args: + name: Unique non-empty name under which the carrier is published + in ``TorchEnvState.carriers``. + carrier: Tensor on the env device. The task keeps ownership and + updates it in place; dtype and shape are captured by the + ring slots with ``torch.empty_like``. + + Raises: + RuntimeError: If the publication ring is already allocated (the + first ``reset``/``step`` has run). + ValueError: On an empty or duplicate name, a carrier on the wrong + device, or when persistent step buffers are disabled. + TypeError: If ``carrier`` is not a ``torch.Tensor``. + """ + if self._state is not None: + raise RuntimeError( + "register_step_carrier must be called before the first reset: " + "the publication ring is already allocated" + ) + if not self._persistent_step_buffers: + raise ValueError( + "register_step_carrier requires persistent step buffers " + "(enable env.autoreset_cuda_graph)" + ) + if not isinstance(name, str) or not name: + raise ValueError(f"step carrier name must be a non-empty string, got {name!r}") + if name in self._step_carriers: + raise ValueError(f"step carrier {name!r} is already registered") + if not isinstance(carrier, torch.Tensor): + raise TypeError(f"step carrier {name!r} must be a torch.Tensor") + if carrier.device != self._device: + raise ValueError( + f"step carrier {name!r} must be on {self._device}, got {carrier.device}" + ) + self._step_carriers[name] = carrier + def reset_selection_blockers(self) -> list[str]: """Reset participants that cannot take a ``ResetSelection``.""" return [f"{type(self).__qualname__} does not implement a ResetSelection reset"] @@ -408,6 +476,7 @@ def init_state(self) -> TorchEnvState: terminated=torch.ones((self._num_envs,), dtype=torch.bool, device=self._device), truncated=torch.zeros((self._num_envs,), dtype=torch.bool, device=self._device), info={"steps": initial_steps}, + carriers=self._step_carriers, ) self._last_initial_episode_steps = initial_steps.clone() if self._masked_autoreset: @@ -1019,6 +1088,9 @@ def _allocate_publication_ring(self) -> None: terminated=torch.empty_like(state.terminated), truncated=torch.empty_like(state.truncated), info=state.info, + carriers={ + name: torch.empty_like(value) for name, value in self._step_carriers.items() + }, ) for _ in range(self._published_buffer_depth) ] @@ -1037,6 +1109,8 @@ def _publish_step_state(self) -> None: slot.reward.copy_(state.reward) slot.terminated.copy_(state.terminated) slot.truncated.copy_(state.truncated) + for name, value in state.carriers.items(): + slot.carriers[name].copy_(value) slot.info = state.info slot.final_observation = state.final_observation self._published_state = slot diff --git a/tests/envs/test_step_output_carriers.py b/tests/envs/test_step_output_carriers.py new file mode 100644 index 000000000..a1f76c533 --- /dev/null +++ b/tests/envs/test_step_output_carriers.py @@ -0,0 +1,158 @@ +"""Custom persistent step-output carriers (issue #2628). + +Tasks that materialize per-step data (transition snapshots, custom metric +tensors) register a fixed-address carrier with ``TorchEnv.register_step_carrier``; +the env publishes it through the same publication ring as obs/reward/done, so +consumers hold the published slot view without a defensive per-step clone. +""" + +from __future__ import annotations + +import pytest +import torch + +from tests.envs.test_fixed_shape_autoreset import CUDA, DEVICES, NUM_ENVS, _make_env +from unilab.base.torch_env import TorchEnvState + + +def _zeros(device: str) -> torch.Tensor: + return torch.zeros((NUM_ENVS, 1), dtype=torch.float32, device=device) + + +# --------------------------------------------------------------------------- +# Registration contract +# --------------------------------------------------------------------------- + + +def test_register_step_carrier_requires_persistent_step_buffers() -> None: + env, _ = _make_env("cpu") + carrier = torch.zeros((NUM_ENVS, 2), dtype=torch.float32, device="cpu") + with pytest.raises(ValueError, match="persistent step buffers"): + env.register_step_carrier("transition", carrier) + + +def test_register_step_carrier_validates_name_tensor_and_device() -> None: + env, _ = _make_env("cpu", masked_autoreset=True, persistent_step_buffers=True) + carrier = torch.zeros((NUM_ENVS, 2), dtype=torch.float32, device="cpu") + with pytest.raises(ValueError, match="non-empty string"): + env.register_step_carrier("", carrier) + with pytest.raises(TypeError, match="must be a torch.Tensor"): + env.register_step_carrier("bad", carrier.tolist()) # type: ignore[arg-type] + with pytest.raises(ValueError, match="must be on"): + env.register_step_carrier("bad", torch.zeros((NUM_ENVS, 2), device="meta")) + env.register_step_carrier("transition", carrier) + with pytest.raises(ValueError, match="already registered"): + env.register_step_carrier("transition", torch.zeros_like(carrier)) + + +def test_register_step_carrier_fails_closed_after_the_first_publish() -> None: + env, _ = _make_env("cpu", masked_autoreset=True, persistent_step_buffers=True) + env.reset() + carrier = torch.zeros((NUM_ENVS, 2), dtype=torch.float32, device="cpu") + with pytest.raises(RuntimeError, match="before the first reset"): + env.register_step_carrier("transition", carrier) + + +# --------------------------------------------------------------------------- +# Ring publication semantics +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("device", DEVICES) +def test_registered_carriers_publish_through_the_ring(device: str) -> None: + env, _ = _make_env( + device, masked_autoreset=True, persistent_step_buffers=True, published_buffer_depth=2 + ) + working = torch.zeros((NUM_ENVS, 3), dtype=torch.float32, device=device) + env.register_step_carrier("transition", working) + env.reset() + + published: list[tuple[TorchEnvState, torch.Tensor]] = [] + for step in range(4): + # Simulate the task writing the carrier in place during update_state. + working.fill_(float(step + 1)) + state = env.step(_zeros(device)) + published.append((state, state.carriers["transition"].clone())) + + # The slot objects alternate with a two-deep ring. + assert published[0][0] is published[2][0] + assert published[1][0] is published[3][0] + assert published[0][0] is not published[1][0] + # The ring slot tensor is storage the env owns, not the working carrier. + latest = env.state + assert latest is not None + assert latest.carriers["transition"].data_ptr() != working.data_ptr() + # Only the freshest two slots still hold their publish-time content; older + # slots were rotated and overwritten. + for step, (state, values) in enumerate(published[-2:], start=len(published) - 2): + torch.testing.assert_close( + state.carriers["transition"], torch.full_like(values, float(step + 1)) + ) + + # The off-policy contract: the latest published view survives one further + # step (the other ring slot is published) and is rotated onto at the + # second. + held_view = latest.carriers["transition"] + held_values = held_view.clone() + working.fill_(99.0) + env.step(_zeros(device)) + torch.testing.assert_close(held_view, held_values) + working.fill_(100.0) + env.step(_zeros(device)) + torch.testing.assert_close(held_view, torch.full_like(held_values, 100.0)) + + +@pytest.mark.parametrize("device", DEVICES) +def test_unregistered_env_publishes_empty_carriers(device: str) -> None: + env, _ = _make_env( + device, masked_autoreset=True, persistent_step_buffers=True, published_buffer_depth=2 + ) + env.reset() + state = env.step(_zeros(device)) + assert state.carriers == {} + + +def test_registered_carriers_are_zero_copy_views_at_ring_depth_zero() -> None: + env, _ = _make_env( + "cpu", masked_autoreset=True, persistent_step_buffers=True, published_buffer_depth=0 + ) + working = torch.zeros((NUM_ENVS, 3), dtype=torch.float32, device="cpu") + env.register_step_carrier("transition", working) + env.reset() + working.fill_(7.0) + state = env.step(_zeros("cpu")) + # Depth 0 publishes the working carriers themselves: no ring, no copies. + assert state.carriers["transition"] is working + torch.testing.assert_close(state.carriers["transition"], torch.full((NUM_ENVS, 3), 7.0)) + + +@pytest.mark.skipif(not CUDA, reason="requires CUDA") +def test_registered_carriers_publish_under_autoreset_graph_replay() -> None: + env, _ = _make_env("cuda", autoreset_cuda_graph=True) + working = torch.zeros((NUM_ENVS, 2), dtype=torch.float32, device="cuda") + env.register_step_carrier("transition", working) + env.reset() + + published: list[tuple[TorchEnvState, torch.Tensor]] = [] + for step in range(6): + working.fill_(float(step + 1)) + state = env.step(_zeros("cuda")) + published.append((state, state.carriers["transition"].clone())) + + graph = env.autoreset_graph + assert graph is not None and graph.captured + # The freshest two slots hold their publish-time content across the graph + # replays; the captured autoreset segment does not touch the carriers. + for step, (state, values) in enumerate(published[-2:], start=len(published) - 2): + torch.testing.assert_close( + state.carriers["transition"], torch.full_like(values, float(step + 1)) + ) + + # A held view stays valid across one further replayed step. + latest = env.state + assert latest is not None + held_view = latest.carriers["transition"] + held_values = held_view.clone() + working.fill_(42.0) + env.step(_zeros("cuda")) + torch.testing.assert_close(held_view, held_values) From 13fe73cb641c45e5b9bf483550cb592ce2a78498 Mon Sep 17 00:00:00 2001 From: TATP-233 Date: Thu, 19 Nov 2026 23:37:32 +0800 Subject: [PATCH 2/2] docs(changelog): link PR 2631 for custom step-output carriers --- docs/sphinx/source/changelog.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/sphinx/source/changelog.md b/docs/sphinx/source/changelog.md index 5446c741b..f457a91e0 100644 --- a/docs/sphinx/source/changelog.md +++ b/docs/sphinx/source/changelog.md @@ -23,13 +23,15 @@ PyPI 版本变更与未发布变更;发布日期采用 PyPI 上传日期。完 stay valid for one further step instead of per-step defensive clones. The sync-free autoreset user guide documents the step-output lifetime contract, including which outputs remain per-step fresh - ([#2628](https://github.com/Motphys/UniLab/issues/2628)). + ([#2631](https://github.com/Motphys/UniLab/pull/2631), + [#2628](https://github.com/Motphys/UniLab/issues/2628)). `TorchEnv.register_step_carrier` 将任务自有的 step 输出张量(如 transition 快照)通过 persistent step buffer 发布环发布:消费者获得地址 稳定、可零拷贝持有一步的槽位视图,替代逐步防御性 clone。sync-free autoreset 用户指南补充了 step 输出生命周期契约(含哪些输出仍逐步重新 分配)说明 - ([#2628](https://github.com/Motphys/UniLab/issues/2628))。 + ([#2631](https://github.com/Motphys/UniLab/pull/2631), + [#2628](https://github.com/Motphys/UniLab/issues/2628))。 - The documentation site is now versioned with sphinx-multiversion: the `main` track plus the `x.y.0` release of every minor line (v1.0.0, v1.1.0, v1.2.0, v1.3.0, v1.4.0, v1.5.0 at time of writing) are published with a