Repository navigation
npu 910b4: fully_async island sets trainer.device=npu on Ascend - #229
Merged
Merged
Conversation
fully_async_ppo_trainer.yaml defaults trainer.device=cuda; on an NPU node the resource pools then ask Ray for GPU and never schedule (S20 ModelArts 2x910B4). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found on a ModelArts 2x910B4 notebook (S20 B). fully_async_ppo_trainer.yaml defaults trainer.device=cuda; on an NPU node the resource pools ask Ray for GPU and never schedule. The island now appends trainer.device=npu when device_family() is ascend.
Known gap, not fixed here: PR #227's RAY_EXPERIMENTAL_NOSET_ASCEND_RT_VISIBLE_DEVICES default did not reach FullyAsyncRollouter (verl still saw no NPU and refused vllm 0.16). Exporting the variable in the shell before launch worked; root cause not found yet.
On-card: yeto fully_async 2 cards, 20 rounds, exit 0, 21/21 publication checks VERIFIED, 12.7 s/round vs 23.4 s single-card sync. Tests: test_npu_910b4_prep + verl adapter/fully_async tests 97 passed.
🤖 Generated with Claude Code