[Bug Report] In-step resets compute observations before forward kinematics: Newton ray-cast sensors report the pre-reset pose

[bbjyzzwwy]

Describe the bug

On the Newton backend, observations computed after an in-step reset (timeout / termination / manual reset handled inside ManagerBasedRLEnv.step()) are built from pre-reset forward kinematics. Any ray-cast sensor attached to an articulation body (RayCaster, RayCasterCamera, and subclasses) publishes its pose — and therefore casts its rays / renders its depth image — from the body transform of the previous episode's final pose. The first observation of every episode is sampled metres away from where the robot actually is.

The direct reset() / reset_to() paths are fine; only the two reset branches inside step() are affected. DirectRLEnv.step() has the same gap.

Mechanism (commit 3c0694c):

• ManagerBasedRLEnv.step() (source/isaaclab/isaaclab/envs/manager_based_rl_env.py, ~L270 and ~L282): after self._reset_idx(reset_env_ids) and after self._reset_idx(manual_reset_ids) it goes straight to command_manager.compute() → observation_manager.compute(). There is no scene.write_data_to_sim() / sim.forward() in between.
• ManagerBasedEnv.reset() and reset_to() (source/isaaclab/isaaclab/envs/manager_based_env.py) do call self.scene.write_data_to_sim(); self.sim.forward() right after _reset_idx, with the comment # update articulation kinematics — so the requirement is known, and step() just misses it.
• The asset data layer protects itself: ArticulationData._ensure_fk_fresh() (source/isaaclab_newton/isaaclab_newton/assets/articulation/articulation_data.py, ~L131) lazily calls SimulationManager.forward() when a pose-dependent property is read after a manual write. That is why robot.data.* looks fresh after the reset.
• The sensor pose path does not: _NewtonRayCasterPoseMixin.get_world_poses() (source/isaaclab_newton/isaaclab_newton/sensors/ray_caster/newton_raycast_sensor.py, ~L168) reads the body transforms straight out of NewtonManager.get_state_0() with a warp kernel, no lazy-FK guard. NewtonManager.forward() (source/isaaclab_newton/isaaclab_newton/physics/newton_manager.py, ~L577) documents that body poses are only recomputed from joint coordinates by forward() or the next step.

So within the reset step the articulation data and the sensors disagree about where the robot is, and the observation the policy gets as the first observation of the new episode mixes post-reset proprioception with a pre-reset height scan / depth image.

Steps to reproduce

Stock task, stock sensor, no Isaac Sim needed. A 1 s episode forces a timeout reset inside step() every 50 control steps; the reset event jitters the spawn by ±1 m so the pre- and post-reset poses are far apart. After each reset step we compare height_scanner.data.pos_w (xy) with the robot base (the scanner's offset is z-only, so the normal value is 0).

repro_stale_reset.py

"""Minimal repro: in-step resets leave ray-cast sensor poses at the PRE-reset body transforms.

Stock task, Newton backend, no Isaac Sim. A 1 s episode forces a timeout reset inside
`ManagerBasedRLEnv.step()` every 50 control steps; we compare the height_scanner's
published pose against the robot base right after each in-step reset.

Run:  python repro_stale_reset.py            (stock behaviour -> hundreds of mm stale)
      python repro_stale_reset.py --patched  (forward() after _reset_idx -> fixed)
"""
import argparse
import sys

import numpy as np


def main() -> None:
    from isaaclab.app import add_launcher_args
    from isaaclab_tasks.utils import setup_preset_cli

    parser = argparse.ArgumentParser()
    parser.add_argument("--patched", action="store_true")
    parser.add_argument("--steps", type=int, default=160)
    add_launcher_args(parser)
    args, remaining = setup_preset_cli(parser, agent_library="rsl_rl")
    sys.argv = [sys.argv[0]] + remaining

    import gymnasium as gym
    import torch
    from isaaclab.app import launch_simulation
    from isaaclab.envs import ManagerBasedRLEnv
    from isaaclab_newton.physics import NewtonCfg
    from isaaclab_tasks.utils import resolve_task_config

    task_id = "Isaac-Velocity-Rough-G1"
    env_cfg, _ = resolve_task_config(task_id, "rsl_rl_cfg_entry_point", play_mode=True)
    if env_cfg.sim.physics is None:
        env_cfg.sim.physics = NewtonCfg()
    env_cfg.scene.num_envs = 1
    env_cfg.curriculum = None
    env_cfg.episode_length_s = 1.0  # timeout reset inside step() every 50 control steps
    env_cfg.terminations.base_contact = None
    # Large reset jitter so the pre- and post-reset base poses are far apart.
    env_cfg.events.reset_base.params["pose_range"]["x"] = (-1.0, 1.0)
    env_cfg.events.reset_base.params["pose_range"]["y"] = (-1.0, 1.0)

    if args.patched:
        orig = ManagerBasedRLEnv._reset_idx

        def _reset_idx(self, env_ids):
            orig(self, env_ids)
            # What reset()/reset_to() already do after _reset_idx, and step() does not.
            self.scene.write_data_to_sim()
            self.sim.forward()

        ManagerBasedRLEnv._reset_idx = _reset_idx

    with launch_simulation(env_cfg, args):
        env = gym.make(task_id, cfg=env_cfg).unwrapped
        robot = env.scene["robot"]
        scanner = env.scene["height_scanner"]
        env.reset()
        t = lambda v: v.torch if hasattr(v, "torch") else v
        action = torch.zeros((1, env.action_space.shape[-1]), device=env.device)
        with torch.inference_mode():
            for k in range(args.steps):
                _, _, terminated, truncated, _ = env.step(action)
                if bool((terminated | truncated).any()):
                    base = t(robot.data.root_link_pos_w)[0, :2].cpu().numpy()
                    scan = t(scanner.data.pos_w)[0, :2].cpu().numpy()
                    print(f"step {k + 1:>4} (in-step reset): |height_scanner.pos_w - base| = "
                          f"{np.linalg.norm(scan - base) * 1000:8.1f} mm", flush=True)
        env.close()


if __name__ == "__main__":
    main()

$ python repro_stale_reset.py
step   50 (in-step reset): |height_scanner.pos_w - base| =   1320.9 mm
step  100 (in-step reset): |height_scanner.pos_w - base| =   2244.5 mm
step  150 (in-step reset): |height_scanner.pos_w - base| =   1647.2 mm

$ python repro_stale_reset.py --patched     # forward() after _reset_idx, as reset() does
step   50 (in-step reset): |height_scanner.pos_w - base| =      0.0 mm
step  100 (in-step reset): |height_scanner.pos_w - base| =      0.0 mm
step  150 (in-step reset): |height_scanner.pos_w - base| =      0.0 mm

The --patched variant monkeypatches ManagerBasedRLEnv._reset_idx to append self.scene.write_data_to_sim(); self.sim.forward() — i.e. exactly what reset() already does.

System Info

• Commit: 3c0694c (2026-08-16)
• Isaac Sim Version: not installed (kit-less Newton backend; newton 1.5.0, mujoco-warp 3.11.0, warp-lang 1.16.0)
• OS: Ubuntu 22.04.5, kernel 6.8.0
• GPU: NVIDIA GeForce RTX 5090 D
• CUDA: 13.0 (driver 580.142); torch 2.11.0+cu128; Python 3.12

Additional context

• Impact on the stock tasks: every core/velocity task attaches height_scanner = RayCasterCfg(prim_path="{ENV_REGEX_NS}/Robot/base", ...), so on Newton the first height scan of every training episode is taken around the previous episode's final position. For a per-step-updating RayCaster only that one observation is wrong; for a ray-cast camera with a refresh schedule and a history/backfill buffer the stale frame is copied into the whole history stack and persists until the next scheduled refresh (in our 10 Hz depth-camera setup that is up to ~0.8 s of policy input per episode).
• How we found it: the same policy walked down stairs markedly worse in our Isaac Lab port than in the mjlab original, and the divergence traced to episode starts. mjlab's step() calls sim.forward() (and re-senses) between _reset_idx and the observation compute, which is the behaviour the patch above restores.
• Related but distinct: [Bug Report] reset() returns stale camera images when fabric is enabled - num_rerenders_on_reset re-renders the old transforms #6394 reports stale RTX camera images from reset() with fabric enabled (render-side transforms). This report is about the Newton backend, ray-cast sensors, and the in-step reset path; reset() itself is correct here.
• Suggested fix (either): call self.scene.write_data_to_sim(); self.sim.forward() after both _reset_idx calls in ManagerBasedRLEnv.step() (and DirectRLEnv.step()), mirroring reset(); or give _NewtonRayCasterPoseMixin.get_world_poses() the same lazy-FK guard the asset data layer uses.

Checklist

• I have checked that there is no similar issue in the repo (required)
• I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

• After an in-step reset, ray-cast sensor poses (RayCaster.data.pos_w, RayCasterCamera.data.pos_w) match the post-reset body transforms in the observation returned by that step() call (the repro prints 0.0 mm without the monkeypatch).
• Same for DirectRLEnv.step().
• A regression test covering "sensor pose after in-step reset" on the Newton backend.

[bbjyzzwwy]

Re-ran the repro on today's develop head (155f31a, after the actuator-ownership refactor in #6839): still reproduces. step() is unchanged, NewtonManager.get_state_0() has no lazy-FK guard, and _ensure_fk_fresh still only protects ArticulationData.

$ python repro_stale_reset.py            # 155f31a72
step   50 (in-step reset): |height_scanner.pos_w - base| =   1575.8 mm
step  100 (in-step reset): |height_scanner.pos_w - base| =    916.1 mm
step  150 (in-step reset): |height_scanner.pos_w - base| =   1154.1 mm

$ python repro_stale_reset.py --patched
step   50 (in-step reset): |height_scanner.pos_w - base| =      0.0 mm
step  100 (in-step reset): |height_scanner.pos_w - base| =      0.0 mm
step  150 (in-step reset): |height_scanner.pos_w - base| =      0.0 mm

[RandomOakForest]

Thank you for posting this and following up. The team will review this soon.