Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions docs/sphinx/source/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,21 @@ PyPI 版本变更与未发布变更;发布日期采用 PyPI 上传日期。完

### Added / 新增

- `TorchEnv.register_step_carrier` publishes task-owned step-output tensors
(for example transition snapshots) through the persistent-step-buffer
publication ring: consumers get address-stable, zero-copy slot views that
stay valid for one further step instead of per-step defensive clones. The
sync-free autoreset user guide documents the step-output lifetime contract,
including which outputs remain per-step fresh
([#2631](https://github.com/Motphys/UniLab/pull/2631),
[#2628](https://github.com/Motphys/UniLab/issues/2628)).
`TorchEnv.register_step_carrier` 将任务自有的 step 输出张量(如
transition 快照)通过 persistent step buffer 发布环发布:消费者获得地址
稳定、可零拷贝持有一步的槽位视图,替代逐步防御性 clone。sync-free
autoreset 用户指南补充了 step 输出生命周期契约(含哪些输出仍逐步重新
分配)说明
([#2631](https://github.com/Motphys/UniLab/pull/2631),
[#2628](https://github.com/Motphys/UniLab/issues/2628))。
- Built-in reset-mode model-field DR terms now accept a ``ResetSelection``,
so tasks using them (for example G1 motion tracking with `foot_friction`)
can enable fixed-shape autoreset (`env.autoreset_cuda_graph`): the masked
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,51 @@ lower bound, must be declared with
re-applies these transitions after every replay. Undeclared host state keeps the
value it had at capture time.

## Step-output lifetime and custom carriers

With `persistent_step_buffers` (implied by `autoreset_cuda_graph`) the step
outputs are zero-copy views into pre-allocated storage — no per-step output
allocation:

- `state.obs`, `state.reward`, `state.terminated` and `state.truncated` are
views into the current publication-ring slot. A published state stays valid
for one further step: the ring holds two persistent slots, which the
off-policy double-buffer contract needs (a published `obs_t` stays stable
while the next step writes `obs_tp1`).
- `state.carriers` holds task-registered custom carriers (below); the same
ring semantics apply to them.
- `state.info` is host-side state shared by reference, not ring storage:
`info["steps"]` is a device tensor, but `info["timing"]`, `info["log"]` and
other entries are replaced or mutated every step. Read them before the next
`step` returns.
- `state.final_observation` aliases the env's single working scratch. It is
valid until the next reset segment runs (every step on the fixed-shape
path); clone it if it must be kept longer.

### Registering a custom step-output carrier

Tasks that materialize per-step data — transition snapshots, custom metric
tensors — can publish it through the same ring instead of defending against
overwrites with per-step `detach().clone()`:

```python
snapshot = torch.zeros((env.num_envs, snapshot_dim), device=env.device)
env.register_step_carrier("transition", snapshot) # cold path: before the first reset()

# ... a term or manager updates `snapshot` in place during update_state ...

state = env.step(actions)
state.carriers["transition"] # ring-slot view; stable for one further step
```

`register_step_carrier` fails closed: it requires the fixed-shape chain
(`env.autoreset_cuda_graph`), must be called before the first `reset` (the
ring is allocated there), rejects duplicate or empty names and carriers that
are not tensors on the env device. Ring slots are allocated with
`torch.empty_like` at the first publish, so every address involved is fixed
from then on — the address-stability precondition for capturing the
step-output publication in a CUDA graph (issue #2628).

## Opting terms into `ResetSelection`

A `ResetSelection` (`src/unilab/base/reset_selection.py`) is a persistent
Expand Down
82 changes: 78 additions & 4 deletions src/unilab/base/torch_env.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,10 @@ class TorchEnvState:

Terminal pre-reset observations are represented only by
``final_observation``; ``info`` never carries duplicate
terminal-observation entries.
terminal-observation entries. ``carriers`` holds task-registered custom
step-output tensors (see ``TorchEnv.register_step_carrier``); like
``obs``/``reward``, the published state's entries are zero-copy views
into persistent ring storage.
"""

obs: dict[str, torch.Tensor]
Expand All @@ -60,6 +63,7 @@ class TorchEnvState:
truncated: torch.Tensor
info: dict[str, Any]
final_observation: dict[str, torch.Tensor] | None = None
carriers: dict[str, torch.Tensor] = dataclasses.field(default_factory=dict)

def replace(self, **updates: Any) -> "TorchEnvState":
"""Return a copy of this transition with ``updates`` applied."""
Expand Down Expand Up @@ -245,6 +249,10 @@ def bounded_int(name: str, default: int, minimum: int) -> int:
self._publication_ring: list[TorchEnvState] = []
self._publication_cursor = 0
self._published_state: TorchEnvState | None = None
# Task-owned custom step-output carriers (issue #2628), published
# through the same ring as obs/reward/done. Registration closes at
# the first publish; the mapping then backs ``TorchEnvState.carriers``.
self._step_carriers: dict[str, torch.Tensor] = {}

@property
def device(self) -> torch.device:
Expand Down Expand Up @@ -307,9 +315,10 @@ def state(self) -> Optional[TorchEnvState]:
"""The latest published transition.

With ``persistent_step_buffers`` this is a publication slot whose
tensors stay unchanged for ``published_buffer_depth - 1`` further
steps (or, at depth 0, the working carriers that the next step
overwrites in place).
tensors — observations, reward, terminated, truncated and the
registered custom ``carriers`` — stay unchanged for
``published_buffer_depth - 1`` further steps (or, at depth 0, the
working carriers that the next step overwrites in place).
"""
if self._published_state is not None:
return self._published_state
Expand Down Expand Up @@ -358,6 +367,65 @@ def record_host_transition(self, name: str, transition: Callable[[], None]) -> N
return
capture.record_host_transition(f"env.{name}", transition)

def register_step_carrier(self, name: str, carrier: torch.Tensor) -> None:
"""Register a task-owned tensor as a persistent step-output carrier.

The carrier is a device tensor the task updates in place during
``update_state`` — for example a transition snapshot or a custom
per-step metric tensor. With ``persistent_step_buffers`` (implied by
``autoreset_cuda_graph``) the env copies it, with ``copy_``, into the
current publication-ring slot at every step boundary, so consumers of
the published state can hold the slot's tensor for
``published_buffer_depth`` further steps without a defensive clone.
Ring slots are pre-allocated with ``torch.empty_like``, so every
address is fixed after the first publish — the address-stability
precondition for capturing the step-output publication in a CUDA
graph (issue #2628). Consumers read the published values through
``TorchEnvState.carriers``.

Registration is a cold-path call. It must happen before the first
``reset``/``step`` (which allocates the publication ring), and it
requires the fixed-shape chain. Fails closed on a duplicate or empty
name, a non-tensor carrier, a carrier that is not on the env device,
a carrier registered after the first publish, or when persistent
step buffers are disabled.

Args:
name: Unique non-empty name under which the carrier is published
in ``TorchEnvState.carriers``.
carrier: Tensor on the env device. The task keeps ownership and
updates it in place; dtype and shape are captured by the
ring slots with ``torch.empty_like``.

Raises:
RuntimeError: If the publication ring is already allocated (the
first ``reset``/``step`` has run).
ValueError: On an empty or duplicate name, a carrier on the wrong
device, or when persistent step buffers are disabled.
TypeError: If ``carrier`` is not a ``torch.Tensor``.
"""
if self._state is not None:
raise RuntimeError(
"register_step_carrier must be called before the first reset: "
"the publication ring is already allocated"
)
if not self._persistent_step_buffers:
raise ValueError(
"register_step_carrier requires persistent step buffers "
"(enable env.autoreset_cuda_graph)"
)
if not isinstance(name, str) or not name:
raise ValueError(f"step carrier name must be a non-empty string, got {name!r}")
if name in self._step_carriers:
raise ValueError(f"step carrier {name!r} is already registered")
if not isinstance(carrier, torch.Tensor):
raise TypeError(f"step carrier {name!r} must be a torch.Tensor")
if carrier.device != self._device:
raise ValueError(
f"step carrier {name!r} must be on {self._device}, got {carrier.device}"
)
self._step_carriers[name] = carrier

def reset_selection_blockers(self) -> list[str]:
"""Reset participants that cannot take a ``ResetSelection``."""
return [f"{type(self).__qualname__} does not implement a ResetSelection reset"]
Expand Down Expand Up @@ -408,6 +476,7 @@ def init_state(self) -> TorchEnvState:
terminated=torch.ones((self._num_envs,), dtype=torch.bool, device=self._device),
truncated=torch.zeros((self._num_envs,), dtype=torch.bool, device=self._device),
info={"steps": initial_steps},
carriers=self._step_carriers,
)
self._last_initial_episode_steps = initial_steps.clone()
if self._masked_autoreset:
Expand Down Expand Up @@ -1019,6 +1088,9 @@ def _allocate_publication_ring(self) -> None:
terminated=torch.empty_like(state.terminated),
truncated=torch.empty_like(state.truncated),
info=state.info,
carriers={
name: torch.empty_like(value) for name, value in self._step_carriers.items()
},
)
for _ in range(self._published_buffer_depth)
]
Expand All @@ -1037,6 +1109,8 @@ def _publish_step_state(self) -> None:
slot.reward.copy_(state.reward)
slot.terminated.copy_(state.terminated)
slot.truncated.copy_(state.truncated)
for name, value in state.carriers.items():
slot.carriers[name].copy_(value)
slot.info = state.info
slot.final_observation = state.final_observation
self._published_state = slot
Expand Down
158 changes: 158 additions & 0 deletions tests/envs/test_step_output_carriers.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
"""Custom persistent step-output carriers (issue #2628).

Tasks that materialize per-step data (transition snapshots, custom metric
tensors) register a fixed-address carrier with ``TorchEnv.register_step_carrier``;
the env publishes it through the same publication ring as obs/reward/done, so
consumers hold the published slot view without a defensive per-step clone.
"""

from __future__ import annotations

import pytest
import torch

from tests.envs.test_fixed_shape_autoreset import CUDA, DEVICES, NUM_ENVS, _make_env
from unilab.base.torch_env import TorchEnvState


def _zeros(device: str) -> torch.Tensor:
return torch.zeros((NUM_ENVS, 1), dtype=torch.float32, device=device)


# ---------------------------------------------------------------------------
# Registration contract
# ---------------------------------------------------------------------------


def test_register_step_carrier_requires_persistent_step_buffers() -> None:
env, _ = _make_env("cpu")
carrier = torch.zeros((NUM_ENVS, 2), dtype=torch.float32, device="cpu")
with pytest.raises(ValueError, match="persistent step buffers"):
env.register_step_carrier("transition", carrier)


def test_register_step_carrier_validates_name_tensor_and_device() -> None:
env, _ = _make_env("cpu", masked_autoreset=True, persistent_step_buffers=True)
carrier = torch.zeros((NUM_ENVS, 2), dtype=torch.float32, device="cpu")
with pytest.raises(ValueError, match="non-empty string"):
env.register_step_carrier("", carrier)
with pytest.raises(TypeError, match="must be a torch.Tensor"):
env.register_step_carrier("bad", carrier.tolist()) # type: ignore[arg-type]
with pytest.raises(ValueError, match="must be on"):
env.register_step_carrier("bad", torch.zeros((NUM_ENVS, 2), device="meta"))
env.register_step_carrier("transition", carrier)
with pytest.raises(ValueError, match="already registered"):
env.register_step_carrier("transition", torch.zeros_like(carrier))


def test_register_step_carrier_fails_closed_after_the_first_publish() -> None:
env, _ = _make_env("cpu", masked_autoreset=True, persistent_step_buffers=True)
env.reset()
carrier = torch.zeros((NUM_ENVS, 2), dtype=torch.float32, device="cpu")
with pytest.raises(RuntimeError, match="before the first reset"):
env.register_step_carrier("transition", carrier)


# ---------------------------------------------------------------------------
# Ring publication semantics
# ---------------------------------------------------------------------------


@pytest.mark.parametrize("device", DEVICES)
def test_registered_carriers_publish_through_the_ring(device: str) -> None:
env, _ = _make_env(
device, masked_autoreset=True, persistent_step_buffers=True, published_buffer_depth=2
)
working = torch.zeros((NUM_ENVS, 3), dtype=torch.float32, device=device)
env.register_step_carrier("transition", working)
env.reset()

published: list[tuple[TorchEnvState, torch.Tensor]] = []
for step in range(4):
# Simulate the task writing the carrier in place during update_state.
working.fill_(float(step + 1))
state = env.step(_zeros(device))
published.append((state, state.carriers["transition"].clone()))

# The slot objects alternate with a two-deep ring.
assert published[0][0] is published[2][0]
assert published[1][0] is published[3][0]
assert published[0][0] is not published[1][0]
# The ring slot tensor is storage the env owns, not the working carrier.
latest = env.state
assert latest is not None
assert latest.carriers["transition"].data_ptr() != working.data_ptr()
# Only the freshest two slots still hold their publish-time content; older
# slots were rotated and overwritten.
for step, (state, values) in enumerate(published[-2:], start=len(published) - 2):
torch.testing.assert_close(
state.carriers["transition"], torch.full_like(values, float(step + 1))
)

# The off-policy contract: the latest published view survives one further
# step (the other ring slot is published) and is rotated onto at the
# second.
held_view = latest.carriers["transition"]
held_values = held_view.clone()
working.fill_(99.0)
env.step(_zeros(device))
torch.testing.assert_close(held_view, held_values)
working.fill_(100.0)
env.step(_zeros(device))
torch.testing.assert_close(held_view, torch.full_like(held_values, 100.0))


@pytest.mark.parametrize("device", DEVICES)
def test_unregistered_env_publishes_empty_carriers(device: str) -> None:
env, _ = _make_env(
device, masked_autoreset=True, persistent_step_buffers=True, published_buffer_depth=2
)
env.reset()
state = env.step(_zeros(device))
assert state.carriers == {}


def test_registered_carriers_are_zero_copy_views_at_ring_depth_zero() -> None:
env, _ = _make_env(
"cpu", masked_autoreset=True, persistent_step_buffers=True, published_buffer_depth=0
)
working = torch.zeros((NUM_ENVS, 3), dtype=torch.float32, device="cpu")
env.register_step_carrier("transition", working)
env.reset()
working.fill_(7.0)
state = env.step(_zeros("cpu"))
# Depth 0 publishes the working carriers themselves: no ring, no copies.
assert state.carriers["transition"] is working
torch.testing.assert_close(state.carriers["transition"], torch.full((NUM_ENVS, 3), 7.0))


@pytest.mark.skipif(not CUDA, reason="requires CUDA")
def test_registered_carriers_publish_under_autoreset_graph_replay() -> None:
env, _ = _make_env("cuda", autoreset_cuda_graph=True)
working = torch.zeros((NUM_ENVS, 2), dtype=torch.float32, device="cuda")
env.register_step_carrier("transition", working)
env.reset()

published: list[tuple[TorchEnvState, torch.Tensor]] = []
for step in range(6):
working.fill_(float(step + 1))
state = env.step(_zeros("cuda"))
published.append((state, state.carriers["transition"].clone()))

graph = env.autoreset_graph
assert graph is not None and graph.captured
# The freshest two slots hold their publish-time content across the graph
# replays; the captured autoreset segment does not touch the carriers.
for step, (state, values) in enumerate(published[-2:], start=len(published) - 2):
torch.testing.assert_close(
state.carriers["transition"], torch.full_like(values, float(step + 1))
)

# A held view stays valid across one further replayed step.
latest = env.state
assert latest is not None
held_view = latest.carriers["transition"]
held_values = held_view.clone()
working.fill_(42.0)
env.step(_zeros("cuda"))
torch.testing.assert_close(held_view, held_values)
Loading