Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 82 additions & 0 deletions docs/sphinx/source/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,70 @@ PyPI 版本变更与未发布变更;发布日期采用 PyPI 上传日期。完

### Changed / 变更

- Collapse the fixed-shape autoreset switch chain into the single public
switch `env.autoreset_cuda_graph`: enabling it now implies masked autoreset,
persistent step buffers and deferred finite validation. The intermediate
switches (`env.masked_autoreset`, `env.persistent_step_buffers`,
`env.defer_finite_validation` / `training.defer_finite_validation`),
`env.published_buffer_depth` and `env.autoreset_graph_warmup` are removed
from the public config surface (they are derived or internal constants);
supplying a deleted key fails closed with a clear config error. Behavior of
the graph capture protocol, `ResetSelection` negotiation and the eager
fallback path is unchanged
([#2617](https://github.com/Motphys/UniLab/issues/2617))。
收敛 fixed-shape autoreset 开关链为单一公共开关 `env.autoreset_cuda_graph`:
启用它即隐含 masked autoreset、持久 step buffer 与推迟有限值校验。中间层
开关(`env.masked_autoreset`、`env.persistent_step_buffers`、
`env.defer_finite_validation` / `training.defer_finite_validation`)、
`env.published_buffer_depth` 与 `env.autoreset_graph_warmup` 从公共配置面
删除(改为内部派生值或常量);传入已删除的键会以明确的配置错误 fail
closed。graph 捕获协议、`ResetSelection` 协商与 eager fallback 路径行为
不变([#2617](https://github.com/Motphys/UniLab/issues/2617))。

- Collapse the three deferred finite-validation switches into the single
env-owned validation strictness concept: `env.finite_validation_mode`
(`"strict"` or `"deferred"`, default `"deferred"`) together with
`env.finite_validation_interval`. The env's deferred finite validation, the
NaN guard defer mode (an attached guard follows the env), and the MJWarp
backend options are all derived from it (strict keeps the backend's
immediate per-call operand check; deferred latches its fail-closed checks
into device verdicts published at the env poll). The removed
`training.finite_validation_interval` pass-through,
`training.nan_guard.defer_validation` and the backend switches
`env.mjwarp_validate_tensor_finite` / `env.mjwarp_defer_tensor_validation`
fail closed with a clear config error naming the replacement. Judgment
semantics and the verdict carriers are unchanged
([#2618](https://github.com/Motphys/UniLab/issues/2618))。
收敛三处"推迟有限性校验"开关为 env 持有的单一校验严格度概念:
`env.finite_validation_mode`(`"strict"` 或 `"deferred"`,默认
`"deferred"`)与 `env.finite_validation_interval`。env 的推迟有限性校验、
NaN guard 的 defer 模式(挂载的 guard 跟随 env)与 MJWarp 后端选项均由其
派生(strict 保持后端逐调用算子检查立即生效;deferred 将 fail-closed 检查
锁存为设备 verdict,由 env 在轮询边界发布)。删除的
`training.finite_validation_interval` 透传、`training.nan_guard.defer_validation`
与后端开关 `env.mjwarp_validate_tensor_finite` /
`env.mjwarp_defer_tensor_validation` 会以指明替代键的明确配置错误 fail
closed。判定语义与 verdict 载体不变
([#2618](https://github.com/Motphys/UniLab/issues/2618))。
- `env.autoreset_cuda_graph` now defaults to `true`: task configs that do not
mention the switch run the fixed-shape autoreset path (masked full-width
reset, persistent step buffers, deferred finite validation, and — on CUDA —
autoreset segment CUDA graph capture). Unsupported combinations keep failing
closed at startup with the switch or backend named (a non-CUDA env device,
reset participants that do not accept a `ResetSelection`, or a backend
without the `external_graph_capture` capability); there is no silent eager
fallback. Set `env.autoreset_cuda_graph=false` explicitly to opt out — the
escape hatch for debugging or when a graph-related problem is suspected
([#2621](https://github.com/Motphys/UniLab/issues/2621))。
`env.autoreset_cuda_graph` 默认值翻转为 `true`:任务配置不写该开关时即运行
fixed-shape autoreset 路径(masked 全宽 reset、持久 step buffer、推迟有限值
校验,CUDA 下捕获 autoreset 段 CUDA graph)。不支持的组合仍在启动时按名 fail
closed(非 CUDA env 设备、不接受 `ResetSelection` 的 reset 参与者、或未声明
`external_graph_capture` 能力的后端);不做静默 eager 回退。显式设置
`env.autoreset_cuda_graph=false` 即可退出——用于调试或疑似 graph 相关问题时的
回退逃生口
([#2621](https://github.com/Motphys/UniLab/issues/2621))。

- High-traffic public interfaces — the Manager protocols in
`unilab.managers.types`, `TorchEnv`, the reset-state transaction, the MDP
observation/action/curriculum/ray terms, and the PPO/APPO/SAC/FlashSAC
Expand Down Expand Up @@ -126,6 +190,24 @@ PyPI 版本变更与未发布变更;发布日期采用 PyPI 上传日期。完
路径。
([#2610](https://github.com/Motphys/UniLab/issues/2610)).

### Removed / 移除

- The public reset selector no longer accepts a Python `slice`: the slice
branch is removed from `resolve_reset_env_ids` / `resolve_reset_mask` /
`ResetMaskBuffer.resolve` and from the `fill_rows_` / `assign_rows_`
signatures. Passing a slice now raises `TypeError`; pass explicit row
indices instead (e.g. `torch.arange(num_envs)[slice]`) or `None` for all
rows. The remaining forms are `None`, a 1-D integer row-index tensor, and
the internal `ResetSelection` protocol; no deprecation alias is kept
([#2619](https://github.com/Motphys/UniLab/issues/2619)).
公共 reset 选择器不再接受 Python `slice`:`resolve_reset_env_ids` /
`resolve_reset_mask` / `ResetMaskBuffer.resolve` 的 slice 分支与
`fill_rows_` / `assign_rows_` 签名中的 slice 形式均已删除。传入 slice
现在抛 `TypeError`;请改用显式行索引(如 `torch.arange(num_envs)[slice]`)
或用 `None` 表示全部行。保留形式为 `None`、一维整数行索引张量与内部
`ResetSelection` 协议;不留 deprecation 别名
([#2619](https://github.com/Motphys/UniLab/issues/2619))。

## 1.5.11 (2026-11-16)

### Added / 新增
Expand Down
2 changes: 1 addition & 1 deletion docs/sphinx/source/en/2-user_guide/1-training/0-index.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ Reproduce and operate the scoped single-GPU G1 FlashSAC/MJWarp long soak.
:::{grid-item-card} Sync-free autoreset
:link: 8-sync_free_autoreset
:link-type: doc
Opt-in fixed-shape autoreset, reset CUDA graph, and `ResetSelection` terms.
Default-on fixed-shape autoreset, reset CUDA graph, and `ResetSelection` terms.
:::

::::
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,10 @@ mask on the host (`any()`), builds `env_ids` with `nonzero`, and launches reset
kernels whose shape depends on the number of reset rows. Each of these steps
synchronizes the host with the GPU.

An opt-in fixed-shape path removes these synchronizations, and on CUDA it can
replay the whole reset segment as one CUDA graph. Every switch defaults to
`False`, so tasks that do not opt in keep the selected-row behavior unchanged.
A fixed-shape path removes these synchronizations, and on CUDA it can
replay the whole reset segment as one CUDA graph. The path is controlled by a
single public switch, `env.autoreset_cuda_graph`, which defaults to `true`;
tasks that opt out with an explicit `false` keep the selected-row behavior.

Implementation: `src/unilab/base/torch_env.py`,
`src/unilab/base/reset_selection.py`, `src/unilab/base/autoreset_graph.py`,
Expand All @@ -21,44 +22,55 @@ Implementation: `src/unilab/base/torch_env.py`,
`ResetSelection` (see "Opting terms into `ResetSelection`" below).
The G1 motion-tracking terms (`MotionCommand`, `JointPositionAction`, the
motion body rewards and the anchor/body height terminations) already opt in.
- `training.defer_finite_validation=true`, the default in every shipped
algorithm config.
- For the CUDA graph: a CUDA env device, and a backend that declares the
`external_graph_capture` capability and implements
`torch_graph_capture_scope`. MJWarp does both from `unisim-core` 2.7.2, the
pinned version.

## Configuration flags

All flags are fields of `ManagerBasedRlEnvCfg` (or of the base `EnvCfg` for the
backend flag). They live in the `env` section of the owner YAML. They are not
present in the shipped owner YAMLs, so set them with `++` from the command line:
`env.autoreset_cuda_graph` is the single public switch and defaults to
`true`. It implies the whole fixed-shape chain; the intermediate switches
(`masked_autoreset`, `persistent_step_buffers`), the publication ring depth and
the graph warmup are internal — derived from this switch or owned as constants
by `TorchEnv` — and are no longer configuration keys. Supplying a deleted key
fails closed with a config error. The fixed-shape chain also requires the
deferred finite validation mode: `finite_validation_mode='deferred'` is the
default, and `autoreset_cuda_graph=true` with `finite_validation_mode='strict'`
fails closed at env construction. Set the switch to `false` explicitly to opt
out; that escape hatch is the fallback for debugging or when a graph-related
problem is suspected (the run then uses the selected-row eager path).

```bash
uv run train --algo flashsac --task g1_motion_tracking --sim mjwarp \
++env.masked_autoreset=true \
++env.persistent_step_buffers=true \
++env.autoreset_cuda_graph=true \
++env.mjwarp_defer_tensor_validation=true
uv run train --algo flashsac --task g1_motion_tracking --sim mjwarp
```

| Flag | Default | Effect |
| --- | --- | --- |
| `masked_autoreset` | `False` | Runs the autoreset on every step as a full-width `reset(ResetSelection)` under the device done mask, with no `any()` gate and no `nonzero`. Requires `defer_finite_validation`. |
| `persistent_step_buffers` | `False` | Allocates the observation, reward, terminated and truncated carriers once and updates them with `copy_`. Consumers receive one of `published_buffer_depth` persistent publication slots. |
| `published_buffer_depth` | `2` | Size of the publication ring. A published state stays valid for `published_buffer_depth - 1` further steps. `0` publishes the working carriers themselves, which the next step overwrites. |
| `autoreset_cuda_graph` | `False` | Captures the masked autoreset segment, including the backend reset, into one `torch.cuda.graph` and replays it afterwards. Requires `masked_autoreset`, `persistent_step_buffers`, a CUDA device and a backend with external graph capture. |
| `autoreset_graph_warmup` | `2` | Eager executions before capture, so that lazily allocated buffers exist outside the graph. |
| `autoreset_cuda_graph` | `True` | Runs the autoreset on every step as a full-width `reset(ResetSelection)` under the device done mask (no `any()` gate, no `nonzero`), keeps the observation, reward, terminated and truncated carriers persistent, defers every fail-closed finite check to the poll boundary, and — on CUDA — captures the masked autoreset segment, including the backend reset, into one `torch.cuda.graph` and replays it afterwards. Requires a CUDA device, a backend with external graph capture, and `finite_validation_mode='deferred'`. |
| `autoreset_graph_capture_mode` | `"thread_local"` | `torch.cuda.graph` capture error mode: `"global"`, `"thread_local"` or `"relaxed"`. |
| `mjwarp_defer_tensor_validation` | `False` | Latches MJWarp's fail-closed operand and reset-mask checks into device verdicts instead of reading each one on the host. The env publishes them at its deferred-validation poll. Inside a graph capture MJWarp always defers. The flag also removes the host reads from the eager steps outside the captured segment. UniLab forwards it only when it is `True`. |

Invalid combinations fail when the config is validated: for example,
`autoreset_cuda_graph` without `masked_autoreset` and `persistent_step_buffers`,
or `masked_autoreset` without `defer_finite_validation`.
The MJWarp backend options follow the env's single validation strictness
(`env.finite_validation_mode`, issue #2618): deferred mode latches MJWarp's
fail-closed operand and reset-mask checks into device verdicts (the removed
`mjwarp_defer_tensor_validation` switch) that the env publishes at its
deferred-validation poll, and also removes the host reads from the eager steps
outside the captured segment; strict mode keeps MJWarp's per-call operand check
immediate (`mjwarp_validate_tensor_finite`). Inside a graph capture MJWarp
always defers.

A published state stays valid for one further step (the ring holds two
persistent slots, which the off-policy double-buffer contract needs); the
capture happens after two eager warmup executions so that lazily allocated
buffers exist outside the graph.

Invalid combinations fail when the environment is constructed: for example,
`autoreset_cuda_graph` on a non-CUDA device, or a backend without the
`external_graph_capture` capability.

## Behavior notes

**RNG draw order.** With `masked_autoreset` the reset lifecycle runs on every
**RNG draw order.** On the fixed-shape path the reset lifecycle runs on every
step; an all-false mask leaves the state unchanged. Full-width samplers, such as
the `MotionCommand` resample or the reset-event interval times, therefore draw
from the Manager RNG on steps where no row is done. With the same seed,
Expand All @@ -73,15 +85,14 @@ on done rows.
latched on the device. This covers finite checks, `TorchEnv.require` verdicts,
reset-mask containment and MJWarp/packed-reset verdicts.
`poll_finite_validation` reads all of them together every
`training.finite_validation_interval` steps (32 in the shipped configs). A
`env.finite_validation_interval` steps (32 in the shipped configs). A
failure is therefore raised at the next poll, up to `interval - 1` steps after
the faulty step. Packed-reset failures name the poll step
(`first detected at poll boundary, step N`). To localize a fault, rerun with
`training.finite_validation_interval=1`; the masked path still requires
`training.defer_finite_validation=true`.
`env.finite_validation_interval=1` so every step is a poll boundary.

**Stale diagnostic counters.** In masked mode the env has no host-side reset row
count or per-phase split. Consequences:
**Stale diagnostic counters.** On the fixed-shape path the env has no
host-side reset row count or per-phase split. Consequences:

- The per-reset detail keys in `info["timing"]`
(`RESET_DONE_DETAIL_TIMING_KEYS` in `src/unilab/base/backend_timing.py`, for
Expand Down Expand Up @@ -120,7 +131,7 @@ support and handling the selection as a masked full-width write.
calls a factory `values(n)` that returns `n` rows (`num_envs` rows under a
selection).

Both accept row ids, `None`, a slice or a selection.
Both accept row ids, `None`, or a selection.
- Do not read the mask on the host (`bool(mask.any())`, `.item()`,
`nonzero`) inside `reset`; that reintroduces the synchronization.

Expand Down Expand Up @@ -157,7 +168,7 @@ Some managers impose extra rules:

### Debugging `reset_selection_blockers`

With `masked_autoreset`, `init_state` calls
With the fixed-shape path, `init_state` calls
`ManagerBasedRlEnv.reset_selection_blockers()`. It collects one entry per
unsupported participant from every manager and raises:

Expand All @@ -169,13 +180,13 @@ ResetSelection; unsupported: EventManager: <term name> (<function name>); ...
Each entry has the form `<Manager>: <term name> (<function or class name>)`. A
manager that cannot take a selection for a structural reason reports a sentence
instead. One example is `TerminationManager: episode counts are read on the
host unless defer_finite_validation is enabled`.
host unless finite_validation_mode='deferred'`.

To list blockers without starting a training run, construct the env with the
flag off and call the method:

```python
env.reset_selection_blockers() # [] when the task can enable masked_autoreset
env.reset_selection_blockers() # [] when the task can enable the fixed-shape path
```

Then convert, replace or remove each named term. Reset-randomization events are
Expand Down
3 changes: 1 addition & 2 deletions docs/sphinx/source/en/2-user_guide/2-algorithms/1-ppo.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,8 +147,7 @@ uv run train --algo ppo --task go2_joystick_flat --sim mujoco \
| `training.tf32_matmul` | str | ieee | Learner FP32 matmul backend; `tf32` opts into tensor cores with automatic IEEE fallback |
| `training.nan_guard.enabled` | bool | true | Detect non-finite tensors and dump diagnostics |
| `training.nan_guard.check_interval` | int | 32 | Env steps between NaN checks |
| `training.defer_finite_validation` | bool | true | Batch finite checks into a periodic cycle |
| `training.finite_validation_interval` | int | 32 | Steps per batched finite-check cycle |
| `env.finite_validation_interval` | int | 32 | Steps per batched finite-check poll cycle (env-owned validation strictness) |
| `training.collector_tensor_device` | str | auto | Device of collector-side tensors; `auto` follows the learner |
| `training.logger` | str | tensorboard | Scalar logging backend (`tensorboard` or `wandb`) |
| `training.num_timesteps` | int / null | null | Global sample budget; overrides the derived iteration count when set |
Expand Down
3 changes: 1 addition & 2 deletions docs/sphinx/source/en/2-user_guide/2-algorithms/2-appo.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,8 +138,7 @@ uv run train --algo appo --task go2_joystick_flat --sim mujoco \
| `training.tf32_matmul` | str | ieee | Learner FP32 matmul backend; `tf32` opts into tensor cores with automatic IEEE fallback |
| `training.nan_guard.enabled` | bool | true | Detect non-finite tensors and dump diagnostics |
| `training.nan_guard.check_interval` | int | 32 | Env steps between NaN checks |
| `training.defer_finite_validation` | bool | true | Batch finite checks into a periodic cycle |
| `training.finite_validation_interval` | int | 32 | Steps per batched finite-check cycle |
| `env.finite_validation_interval` | int | 32 | Steps per batched finite-check poll cycle (env-owned validation strictness) |
| `training.collector_tensor_device` | str | auto | Device of collector-side tensors; `auto` follows the learner |
| `training.logger` | str | tensorboard | Scalar logging backend (`tensorboard` or `wandb`) |

Expand Down
3 changes: 1 addition & 2 deletions docs/sphinx/source/en/2-user_guide/2-algorithms/3-sac.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,8 +118,7 @@ uv run train --algo sac --task g1_walk_flat --sim mujoco \
| `training.dp_gradient_wire_dtype` | str | float32 | Gradient wire dtype for multi-GPU averaging; `bfloat16` compresses |
| `training.nan_guard.enabled` | bool | true | Detect non-finite tensors and dump diagnostics |
| `training.nan_guard.check_interval` | int | 32 | Env steps between NaN checks |
| `training.defer_finite_validation` | bool | true | Batch finite checks into a periodic cycle |
| `training.finite_validation_interval` | int | 32 | Steps per batched finite-check cycle |
| `env.finite_validation_interval` | int | 32 | Steps per batched finite-check poll cycle (env-owned validation strictness) |
| `training.collector_tensor_device` | str | auto | Device of collector-side tensors; `auto` follows the learner |
| `training.env_steps_per_sync` | int | 1 | Env steps per collection sync |
| `training.inference_slot_capacity` | int | 1 | In-flight learner-inference requests; 1 keeps inference serialized |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -151,8 +151,7 @@ uv run train --algo flashsac --task g1_walk_flat --sim mujoco \
| `training.dp_gradient_wire_dtype` | str | float32 | Gradient wire dtype for multi-GPU averaging; `bfloat16` compresses |
| `training.nan_guard.enabled` | bool | true | Detect non-finite tensors and dump diagnostics |
| `training.nan_guard.check_interval` | int | 32 | Env steps between NaN checks |
| `training.defer_finite_validation` | bool | true | Batch finite checks into a periodic cycle |
| `training.finite_validation_interval` | int | 32 | Steps per batched finite-check cycle |
| `env.finite_validation_interval` | int | 32 | Steps per batched finite-check poll cycle (env-owned validation strictness) |
| `training.collector_tensor_device` | str | auto | Device of collector-side tensors; `auto` follows the learner |
| `training.env_steps_per_sync` | int | 1 | Env steps per collection sync |
| `training.inference_slot_capacity` | int | 1 | In-flight learner-inference requests; 1 keeps inference serialized |
Expand Down
Loading
Loading