Skip to content

fix(rng): drop eager host validation in ManagerRng.choice to unblock sync-free hot paths - #2625

Merged
TATP-233 merged 1 commit into
mainfrom
fix/issue-2624-rng-choice-host-sync
Nov 19, 2026
Merged

TATP-233 merged 1 commit into
mainfrom
fix/issue-2624-rng-choice-host-sync

Conversation

@TATP-233

Copy link
Copy Markdown
Collaborator

Summary

  • Removed the eager per-call probability validation (torch.isfinite(p).all() / (p < 0).any() wrapped in bool(...)) from ManagerRng.choice (src/unilab/managers/rng.py). The two bool(...) calls were device→host reads, so every choice(n, p=<cuda tensor>) call forced a synchronization that breaks sync-free hot paths: fixed-shape autoreset under torch.cuda.set_sync_debug_mode("error") hard-fails every step, and the call cannot be captured into a CUDA graph.
  • The check was redundant: torch.multinomial itself fails fast on invalid probabilities — a clean RuntimeError on CPU and a device-side assert on CUDA (both reproduced experimentally). The eager per-call pattern also contradicted the repository's deferred finite-validation boundary (poll_finite_validation), and its p.numel() == 0 branch was unreachable dead code. tests/managers/test_rng.py only asserts metadata checks (type/device/shape), so no test coverage is lost.
  • choice is now sync-free. The method docstring documents the caller contract: p must be finite, non-negative, with a positive sum; invalid p fails inside torch.multinomial (CPU: clean RuntimeError; CUDA: device-side assert with a poisoned context).
  • User-facing behavior change: invalid probability tables now surface as RuntimeError from torch.multinomial instead of the previous ValueError, and only when the caller actually passes invalid input.

Linked Work

Validation

  • make test-all passed on the final local head before this PR was created or updated
  • Additional task-specific validation listed below

Commands actually run:

uv run pytest tests/managers/test_rng.py -q            # 15 passed
# CUDA sync-free reproduction of the issue scenario (RTX host):
torch.cuda.set_sync_debug_mode("error"); rng.choice(3, (4096,), p=<cuda>)  # PASS, previously raised
make check                                             # ruff/mypy/pyright clean (1 pre-existing pyright warning in untouched src/unilab/base/__init__.py)
make test-all                                          # exit 0

Impact

  • Backend impact: none
  • Platform impact: both (CPU error type for invalid input changes from ValueError to torch's RuntimeError)
  • Training effect expected: no (no in-repo production caller of ManagerRng.choice; downstream DR terms using it become eligible for sync-free fixed-shape autoreset and CUDA graph capture)

Artifacts

  • W&B: n/a
  • benchmark result: n/a
  • video / screenshot: n/a
  • ONNX / checkpoint: n/a

Checklist

  • Added or updated tests where needed (existing tests cover all retained checks; the removed check had no test)
  • Updated docs if behavior or workflow changed (changelog entry + method docstring)
  • Linked the driving issue
  • Noted any follow-up work explicitly (none)

…sync-free hot paths

The per-call torch.isfinite / p < 0 check in ManagerRng.choice forced a
device-to-host read on every invocation, breaking sync-free step paths
(fixed-shape autoreset with sync-debug mode, CUDA graph capture).

The check was redundant: torch.multinomial itself fails fast on invalid
probabilities (clean RuntimeError on CPU, device-side assert on CUDA), and
the eager per-call pattern contradicted the repository's deferred
finite-validation boundary. Existing tests only assert metadata checks
(type/device/shape), so no coverage is lost.

choice is now sync-free; the method docstring documents the caller
contract (finite, non-negative probabilities with a positive sum).

Closes #2624
@TATP-233
TATP-233 merged commit bc78ce2 into main Nov 19, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant