feat: odc adapt#864
Merged
Merged
Conversation
…pad deadlock Multi-node (n_pes > local_world_size) now defaults ODC_GDA_DEFER_REDUCE=1 so variable-length (nopad) packing no longer deadlocks on the per-microbatch collective barrier across nodes. Single-node behavior unchanged. Adds a multi-node deployment section (HCA_LIST and other cluster-specific env) to the rocSHMEM runtime README.
Strip all NVIDIA/NVSHMEM/CUDA-only fallback code paths, symbol names and docs from the committed ODC sources so it is a pure ROCm implementation. - Delete the NVIDIA (`nvshmem.core` / `cuda.core`) branches in utils.py, shmem_triton.py, setup.py; ROCm (mori / rocshmem) is the only path. - Rename ODC's own nvshmem-prefixed symbols to neutral names and update all call sites: init_nvshmem->init_shmem, nvshmem_create_tensor-> shmem_create_tensor, get_nvshmem_handle->get_shmem_handle, set_nvshmem_flag->set_shmem_flag, nvshmem_*_kernel->shmem_*_kernel, NVSHMEM_EXTERN_LIBS->SHMEM_EXTERN_LIBS, LIB_NVSHMEM_PATH->LIB_SHMEM_PATH, nvshmem_triton.py->shmem_triton.py. - Delete tensor_ipc.cu (CUDA); keep tensor_ipc.hip and build it directly. - Remove NVIDIA/NVSHMEM docstrings, comments and README/pyproject notes. torch device APIs (torch.cuda / at::kCUDA / torch.version.hip) are the HIP-mapped names on ROCm and are intentionally preserved.
…VICE_REDUCE), watcher kept as default Add a gated, reversible single-node reduce-scatter-accumulate path that drops the ReductionWatcher subprocess + tensor_ipc (HIP GPU-IPC). Enabled only when ODC_SINGLE_DEVICE_REDUCE=1 (default 0 = watcher path, byte-for-byte unchanged). Mechanism: owner-side PULL + on-chip fp32 sum over same-node XGMI peer views (rs_ptr / mori peer tensors, same machinery gather + the watcher push use). Each PE stages its locally pre-accumulated grad into a symmetric fp32 buffer; after a rendezvous barrier, PE r XGMI-reads every same-node peer's segment destined for its shard and sums on-chip. No cross-rank writes -> no device atomics -> no MI300X write-visibility hazard. Like the multi-node GDA path, the per-microbatch call only pre-accumulates locally (no collective) and the single barriered reduce runs per-group at get_accumulation, so nopad variable micro-batch counts cannot mismatch a collective barrier (deadlock-free). Watcher / tensor_ipc code is fully retained; multi-node GDA/RO paths untouched. Validated single-node 8xMI300X (1.5B, gbs16 odc_pad, 50 iter): loss matches watcher within 9e-4, 0 nan, throughput within ~0.4%.
Delete the never-productionized rocSHMEM Reverse-Offload (RO) host-driven cross-node backend and switch the multi-node GPU-direct (GDA) path off the MPI/mpirun bootstrap onto the same unique-id-over-socket bootstrap the single-node backend uses, so the whole stack launches with plain torchrun. RO removal: - delete rocshmem_runtime/ro_backend/rs_host_ro.cpp and the `ro` variant in build_rocshmem_backend.sh (no more USE_RO=ON lib / librs_host_ro.so). - _rocshmem_backend.py: drop _ro_enabled/ODC_ROCSHMEM_RO, ro_enabled(), the rs_putmem/rs_getmem/rs_int_p/rs_quiet/rs_fence/is_remote host forwards, and the RO .so resolution branch. - scatter_accumulate.py / gather.py: drop the cross-node RO put/get branches (cross-node now always goes through GDA) and _ro_active(); clean RO comments. GDA uid bootstrap (no MPI): - _rocshmem_backend.py init(): GDA now uses rs_get_uid (rank0) -> dist.broadcast_object_list -> rs_init_uid (ROCSHMEM_INIT_WITH_UNIQUEID), identical to the single-node path; default ROCSHMEM_BOOTSTRAP_SOCKET_IFNAME=eth0. - drop _ensure_mpi_initialized()/libmpi load and rs_init_mpi() usage; remove the now-dead rs_init_mpi() from rs_host_gda.cpp. (-lmpi stays as a LINK-time dep only: rocSHMEM's librocshmem.a references MPI symbols unconditionally.) Launch -> torchrun + Slurm: - run_pretrain.sh: remove the PRIMUS_LAUNCHER=mpi special-case (always torchrun); the per-run pip-skip is now the launcher-agnostic PRIMUS_SKIP_PIP=1 switch. Multi-node correctness path is unchanged: DEFER reduce-scatter (watcher-free, device-side GDA) is still the code default for n_pes > local_world_size.
Make the single-node reduce-scatter-accumulate use the device-side owner-pull
path unconditionally (previously gated behind ODC_SINGLE_DEVICE_REDUCE=1) and
remove the host-side ReductionWatcher subprocess entirely. Cross-node reduce
always goes through the rocSHMEM GDA path; single-node goes through the
on-chip XGMI pull-sum. Both DEFER the collective reduce to once-per-group at
get_accumulation (matched barrier count across ranks -> nopad-safe).
- scatter_accumulate.py: delete ReductionWatcher / server_loop / DistLock /
start_reduction_watcher / call_watcher / the same-node + cross-node handshake
Triton kernels / ClientContext / ServerContext / register / infer_output_shape
and the ODC_OFFICIAL_PUSH/SETTLE_DEFER watcher knobs. Keep the validated GDA
and single-device methods byte-for-byte; rewrite the dispatch, get_accumulation
and sync around the two device paths. stop() is now a no-op (no subprocess).
- tensor_ipc is only used by the removed watcher path, so drop it: delete
csrc/tensor_ipc/{binding.cpp,tensor_ipc.hip}, its import/exports in
primitives/__init__.py, the CUDAExtension in setup.py (odc is now pure
Python), and the build_rocshmem_backend.sh tensor_ipc stage/variant.
- Refresh README / primitive docstrings that referenced the watcher/tensor_ipc.
Validated single-node 8xMI300X: device path loss matches the old watcher within
9e-4 with 0 nan (see the ODC_SINGLE_DEVICE_REDUCE introduction commit).
Co-authored-by: Cursor <cursoragent@cursor.com>
Reformat the rocSHMEM host bindings (rs_host.cpp, rs_host_gda.cpp) to 4-space indentation to align with the Primus turbo C++ style, and add a scoped odc/.clang-format (Google base, IndentWidth 4) so the style is reproducible. No functional change. Co-authored-by: Cursor <cursoragent@cursor.com>
…/odc/ Move the whole ODC project under primus/core/ (via git mv, history preserved) instead of a top-level sibling of primus/, and fix every path reference to the old odc_rocm_dev/ root: - run_odc.sh: fix the project-root derivation for the two-levels-deeper location (scripts -> rocshmem_runtime -> odc -> core -> primus -> root) and rename ODC_ROCM_DEV -> ODC_ROOT (PYTHONPATH now primus/core/odc[/odc_early]). - build_rocshmem_backend.sh, sitecustomize.py, _rocshmem_backend.py, rocshmem_runtime/README.md: repoint hardcoded odc_rocm_dev/ paths. - root .gitignore + odc_torch_fsdp2_patches.py / lb_mini_packing.py comments: repoint odc_rocm_dev/ references. The importable package is still top-level `odc` (installed via setup.py / PYTHONPATH), so `import odc` is unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>
The device / GDA reduce paths run no host-side subprocess, so ReductionService.stop() is now a no-op. Reword the after_train teardown patch's description/docstring/log accordingly (the hook is retained as a harmless single teardown call site). Co-authored-by: Cursor <cursoragent@cursor.com>
Ignore the underscore-prefixed MI355X ablation yamls (_ab200*, _dev200*, _m18*, _sdr*) so one-off experiment sweep configs are never committed, following the existing "ignore ODC experiment artifacts" precedent. Co-authored-by: Cursor <cursoragent@cursor.com>
Collapse imports / f-string per black --line-length=110 and isort --profile black on the device-reduce refactor and the teardown patch. No code change. Co-authored-by: Cursor <cursoragent@cursor.com>
The code-lint CI job failed because isort (pinned 5.13.2, --profile black) reorders imports differently than the local run used during the refactor. After relocating the package to primus/core/odc/odc/, isort at the repo root no longer auto-detects `odc` as first-party (no known_first_party config), so it groups `import odc` / `from odc...` with third-party imports and alphabetizes accordingly. Re-run the pinned hook to match CI; import-order only, no logic changes.
Add a [tool.isort] section to pyproject.toml (profile=black, matching the existing pre-commit `--profile black` arg) and declare `odc` as known_first_party. `odc` lives under primus/core/odc/odc/, so a clean CI checkout has no top-level `odc` package and isort classified `import odc` as third-party; declaring it first-party makes local and CI import grouping deterministic and identical. Re-run isort (5.13.2) to move the `odc` imports into the first-party group in the affected files.
Replace machine-specific absolute paths in the ODC examples/runtime with
neutral, env-overridable values:
- qwen14B-odc-dn.yaml: pretrained_checkpoint now ${PRETRAINED_CKPT:null}
- run_odc.sh: HF_HOME default /workspace/hf_cache (matches run_pretrain.sh)
- sitecustomize.py: docstring PYTHONPATH example uses <PRIMUS_ROOT>
Add SPDX-style license headers to ODC sources by provenance:
- AMD-authored files (.py/.sh use #, .cpp use //) get the AMD copyright
header: rocSHMEM runtime host bindings/scripts, build script, the ODC
early-init shim, the rocSHMEM P2P backend, and the Megatron ODC patches.
- Files adapted from sail-sg/odc (MIT per upstream package metadata; upstream
ships no LICENSE file or per-file headers) get an upstream MIT attribution
plus an AMD modifications notice: odc/__init__, odc/fsdp/{fsdp1,fsdp2},
odc/primitives/{__init__,gather,scatter_accumulate,shmem_triton,utils}.
Turn the two ODC MI355X example configs into neutral, shareable references: - Replace the ablation A/B comparison header comments with a one-line purpose. - deepseek1.5B-odc-lbmini.yaml: profile false; exp_name deepseek1.5B-odc-lbmini; wandb_project Primus_ODC_DeepSeek1.5B. - qwen14B-odc-dn.yaml: fix all stale "1.5B" references (header, exp_name, wandb_project, cost-model comment) to the 14B dual-node setup. Training hyperparameters are unchanged.
Satisfy the repo-root pre-commit isort/black config on ODC files touched by this cleanup: split long import lists and add the first-party import blank line so `pre-commit run --all-files` is green. No functional change.
Clean up the ODC FSDP patches flagged by github-code-quality: - fsdp1.py: drop commented-out reduce_scatter_tensor / all_reduce / hybrid- shard blocks, disabled low-precision-shard bodies, and leftover debug prints. - fsdp2.py: drop commented-out reduce_scatter_comm / all_gather_copy_in / partial-reduce / dead-return blocks and debug prints; keep the HSDP-defer rationale as a concise design note. - scatter_accumulate.py: retire the stale "watcher" wording (the reduce paths run no host-polling subprocess) since that mechanism was removed. Only comments were removed; no executable logic changed.
…es .so Replace the in-tree ctypes loader (_resolve_host_lib / _load_lib of librs_host5.so / librs_host_gda.so) with a primus_turbo import: the rocSHMEM host/GDA ops now live in primus_turbo.pytorch._C as the odc_rocshmem_host and odc_rocshmem_gda pybind submodules. ODC_ROCSHMEM_GDA selects the GDA submodule. Adapt the pybind ABI vs the old extern-C ctypes ABI: - rs_get_uid() returns bytes directly (no char* out-param buffer). - rs_init_uid(rank, nranks, uid_bytes) takes the uid as bytes. - gda_gather / gda_gather_async take the peer list as a Python list (std::vector<int>); drop the ctypes array + explicit count argument. - Submodule presence (hasattr gda_gather) replaces the old .so symbol probe. The rocSHMEM-independent helpers (_from_blob / _wrap / _nbytes / peer-delta affine resolution) are unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>
…pendency The rocSHMEM ops are now provided by primus_turbo, so the in-tree ctypes build is dead: - Remove build_rocshmem_backend.sh (built librs_host5.so / librs_host_gda.so). - setup.py: drop the "built from source by build_rocshmem_backend.sh" note; the rocSHMEM backend now comes from primus_turbo. Add a PRIMUS_TURBO_COMMIT TODO to pin the exact Turbo merge commit once the ODC-ops PR merges. - run_odc.sh: stop exporting ODC_ROCSHMEM_LIB (unused); add optional PRIMUS_TURBO_PATH prepended to PYTHONPATH so `import primus_turbo` (with the ODC rocSHMEM ops) resolves. - README / rocshmem_runtime README: document the migration. rocshmem_runtime/ is retained as the rocSHMEM library source primus_turbo compiles against (ROCSHMEM_HOME -> rocshmem_gda/). PRIMUS_TURBO_COMMIT is intentionally left as a TODO (cannot be filled until the Primus-Turbo ODC-ops PR is merged). Co-authored-by: Cursor <cursoragent@cursor.com>
… Primus-Turbo) - Flatten the odc/odc double directory: move the odc package (__init__/primitives/fsdp) up to primus/core/odc/; run_odc.sh now puts primus/core/ on PYTHONPATH so `import odc` is unchanged. - Remove the in-tree rocSHMEM binding sources (host_bindings/rs_host.cpp, gda_backend/rs_host_gda.cpp): the ops now come from Primus-Turbo (primus_turbo.pytorch._C.odc_rocshmem_host / _gda). Keep the ctypes escape hatch in _rocshmem_backend.py (ODC_ROCSHMEM_LIB -> external librs_host_gda.so). - Drop the odc setup.py/pyproject/.clang-format (pure-Python in-tree module with no compiled extension; consumed via PYTHONPATH). - Rewrite rocshmem_runtime/README.md and fix README/.gitignore references to the removed build_rocshmem_backend.sh / in-tree .so build.
main #851 removed primus/modules and relocated module_utils to primus/core/utils/module_utils.py. Update all remaining importers: - diffusion backend (diffusion_adapter, diffusion_pretrain_trainer): #851 left these importing the deleted primus.modules, breaking pytest collection of tests/unit_tests/backends/diffusion/ (ModuleNotFoundError: No module named 'primus.modules') -> the "Run Primus Core Tests" failure; - megatron/core/utils; - ODC patches (odc_torch_fsdp2, odc_lb_mini, fused_linear_ce) + lb_mini_dataset: adapt this branch's code to the new module_utils path after merging main.
…ation isort (--profile black) requires module_utils to sort before yaml_utils. Follow-up to the primus.modules -> primus.core.utils.module_utils migration; fixes the code-lint (pre-commit) CI failure on PR #864.
botaohu001
added a commit
that referenced
this pull request
Jul 9, 2026
…DP2 refactor (#808) PR #808 (FSDP2 fp32/bf16 optimizers + fp8 all-gather) replaced Megatron's FSDP2 wrapper with PrimusTorchFullyShardedDataParallel (installed as megatron.training.training.torch_FSDP). ODC still hooked the stock megatron.core...TorchFullyShardedDataParallel.__init__, which is no longer instantiated, so odc_init/_ensure_odc_ready never ran, reduction_service stayed None, and odc_nopad crashed at iter2 with NoneType.clear_accumulations. Hook the class the trainer actually uses, with an ImportError fallback to the stock class. Verified: single-node 1.5B odc_nopad runs 100/100 (lm loss 9.970).
…gle-node speed) PR #856's distributed_init_patches.py injects device_id into init_process_group (gated on use_torch_fsdp2), which on MI300X eagerly creates the world + ~26 Megatron sub-group RCCL communicators. Their resident streams/DMA queues serialize ODC's rocSHMEM P2P scatter copy streams onto the critical path (profiled: cross-stream overlap 120ms -> 2.4ms, ~+128ms/step single-node 1.5B; odc_nopad speedup 1.14x -> 1.05x). nccl_pad is unaffected (it uses these RCCL comms as its native reduce-scatter). ODC exchanges gradients over rocSHMEM, not RCCL, so it never needs the eager-RCCL device_id path. Gate the patch off under ODC_ENABLE=1. Verified on MI300X: odc_nopad single-node gbs16 2310 -> 2178 ms/step (back to FAST), nccl_pad unchanged, 0 crash / 0 nan. Safe here: the MI355X deadlock this patch guards does not trigger on MI300X. Co-authored-by: Cursor <cursoragent@cursor.com>
…-lint) Collapse the ODC gate predicate onto a single line per black (line-length 110); fixes the pre-commit (code-lint) failure on the prior fix commit. No functional change.
…al-node speed) #808 added an FSDP2 forward all-gather prefetch (set_modules_to_forward_prefetch) on top of the existing backward prefetch. ODC uses a serial single-stream rocSHMEM/GDA transport (overlap_factor=1.00x), so eagerly prefetching the next layer's all-gather cannot overlap and is pure overhead: profiled +3.4~3.9s/step in the rocSHMEM GDA gather_kernel and +26.6GB max reserved memory, i.e. the ~5-6% dual-node 14B odc_nopad regression vs #858 (which only did backward prefetch). Auto-gate the forward prefetch OFF whenever ODC is active (ODC_ENABLE=1), mirroring the #856 eager-RCCL device_id gate in distributed_init_patches.py. Backward prefetch is NOT the culprit and is kept (A/B: disabling backward alone recovers nothing; disabling forward alone recovers everything: 75996ms -> 70953ms). Manual escape hatches ODC_DISABLE_FWD_PREFETCH / ODC_DISABLE_BWD_PREFETCH are retained for debugging. Non-ODC FSDP2 runs are unaffected. Co-authored-by: Cursor <cursoragent@cursor.com>
Maintainer requirement: Primus must not use environment variables to control formal internal feature logic, and debug/A-B-only env code must not be merged. - Add config items to trainer_base.yaml: enable_odc (master switch, default false), odc_phase (default 2), odc_grad_spike_threshold (default 1000.0). - distributed_init_patches: gate the #856 device_id skip on enable_odc config instead of the ODC_ENABLE env var. - odc_torch_fsdp2_patches: read enable_odc / odc_phase / odc_grad_spike_threshold from config (get_args); drop the ODC_DEBUG_GRAD debug logging. The #808 forward all-gather prefetch skip now lives here (config-driven, runs only when ODC is on) via _disable_forward_prefetch; backward prefetch is kept. - torch_fully_sharded_data_parallel: revert the env-based prefetch gate (ODC_ENABLE / ODC_DISABLE_FWD_PREFETCH / ODC_DISABLE_BWD_PREFETCH). The generic FSDP2 wrapper is back to stock; the ODC-specific skip is driven by config in the ODC patch. Behaviour equivalent to the validated fix: ODC on => forward prefetch skipped, backward prefetch kept. Co-authored-by: Cursor <cursoragent@cursor.com>
…B knobs - odc_lb_mini_patches: _lb_mini_enabled now requires the enable_odc_lb_mini and enable_odc config items (drops the ODC_LB_MINI / ODC_ENABLE / LB_MINI_FORCE_DATA env fallbacks). - Delete the debug/A-B study env knobs from the LB-Mini iterator builder: LB_MINI_FILTER_MAXLEN, LB_MINI_FILTER_MINLEN (length-variance study), LB_MINI_SAME_MICRO (aligned-baseline A/B; LB-Mini now always runs decoupled), LB_MINI_PACKING (hardcoded to "kk"), and the LB_MINI_MAX_TOKEN env override of the lb_mini_max_token_len config. - Add sft_dataset_split config item (replaces the SFT_DATASET_SPLIT env var). - Update comments/docstrings to reference the config items. Co-authored-by: Cursor <cursoragent@cursor.com>
…le configs) - sitecustomize (odc_early load-order shim): drop the ODC_ENABLE gate. This shim only runs when odc_early is on PYTHONPATH, which the ODC launcher adds solely for ODC runs, so its presence is already the signal; whether ODC trains is decided by the enable_odc config. This is a launch-time load-order bootstrap, not feature logic. - run_odc.sh: stop exporting the converted feature switches (ODC_ENABLE, ODC_PHASE, ODC_LB_MINI, LB_MINI_SAME_MICRO via pad/nopad, LB_MINI_PACKING). Keep genuine infra env (MORI_SHMEM_HEAP_SIZE, sockets, TRITON_CACHE_DIR, ...). The pad/nopad positional arg is retained but is now a no-op. - Example configs (qwen14B-odc-dn, deepseek1.5B-odc-lbmini): set enable_odc=true, odc_phase=2, enable_odc_lb_mini=true so ODC is turned on via config, not env. Co-authored-by: Cursor <cursoragent@cursor.com>
Introduce primus/core/odc/runtime_config.py: a stdlib-only leaf holding an OdcRuntimeConfig singleton (defaults reproduce the previous ODC_* env defaults). odc/__init__ now exposes get/set_runtime_config eagerly and imports the heavy primitives lazily (PEP 562 __getattr__), so `import odc` is cheap and does NOT trigger the primitives' import-time backend selection. The ODC integration patch (patch_odc_torch_fsdp2) calls set_runtime_config from the odc_* trainer-config items BEFORE the primitives are first imported, so even the import-time backend selector (odc_p2p_backend) reads the populated config. set_runtime_config also bridges odc_gda_pipe back to PRIMUS_TURBO_ODC_GDA_PIPE (the one knob the Turbo C++ kernel reads via getenv). Add the flat config items to trainer_base.yaml: odc_p2p_backend, odc_mori_init, odc_max_buffer_size, odc_rocshmem_gda, odc_rocshmem_lib, odc_gda_rs_blocks, odc_gda_pipe, odc_gda_defer_reduce, odc_gda_warmup_mode, odc_gda_stride_bytes. Co-authored-by: Cursor <cursoragent@cursor.com>
…/A-B env Convert the ODC comm-kernel env vars to the runtime config (default-preserving): ODC_P2P_BACKEND->odc_p2p_backend, ODC_MORI_INIT->odc_mori_init, ODC_MAX_BUFFER_SIZE->odc_max_buffer_size, ODC_ROCSHMEM_GDA->odc_rocshmem_gda, ODC_ROCSHMEM_LIB->odc_rocshmem_lib, ODC_GDA_RS_BLOCKS->odc_gda_rs_blocks, PRIMUS_TURBO_ODC_GDA_PIPE->odc_gda_pipe, ODC_GDA_DEFER_REDUCE->odc_gda_defer_reduce, ODC_GDA_WARMUP_MODE->odc_gda_warmup_mode, ODC_GDA_STRIDE_BYTES->odc_gda_stride_bytes. Delete debug-only knobs (ODC_GDA_PROFILE, ODC_GDA_VERIFY) and the default-off A/B/experimental branches (ODC_OFFICIAL_PUSH, ODC_GDA_OVERLAP, ODC_GDA_BUCKET, ODC_GDA_STAGE_STREAMSYNC, ODC_GDA_WARMUP_TINY, ODC_GDA_GATHER_ASYNC, ODC_GDA_GATHER_BARRIER), keeping ONLY the validated default path so multi-node behaviour is unchanged. The odc_gda_warmup_mode config still selects strided/full/hdp/fence/hdpfence. No os.environ reads remain in the primitives except genuine infra (MORI_*/ROCSHMEM_* bootstrap, LOCAL_WORLD_SIZE). Co-authored-by: Cursor <cursoragent@cursor.com>
Make enable_odc_lb_mini an orthogonal, standalone config capability so
LB-Mini variable-length KK-balanced DATA can be served independent of the
enable_odc communication switch.
_lb_mini_enabled no longer requires enable_odc; it is gated on
enable_odc_lb_mini + use_torch_fsdp2 alone. A new _lb_mini_aligned derives
the micro-batch alignment mode from enable_odc:
* enable_odc=true -> DECOUPLED (same_micro_num=False): ranks may run
different micro-batch counts; requires ODC point-to-point comm. Unchanged
full LB-Mini behavior.
* enable_odc=false -> ALIGNED (same_micro_num=True, all_reduce MAX): the SAME
variable-length data is served, but uniform per-rank micro-batch counts keep
standard FSDP2 + RCCL collectives in lockstep -> a fair "same data" nccl
baseline with NO ODC. This is the maintainer-compliant, config-driven
replacement for the removed LB_MINI_FORCE_DATA A/B env.
Both the dataloader patch (variable-length data) and the schedule patch
(rank-local num_microbatches) are installed in either mode. The schedule patch
is comm-agnostic (only sets num_microbatches); under ALIGNED mode the counts are
identical across ranks, so it is NCCL/RCCL-safe and is NOT an ODC-specific
reduction. Installing the dataloader patch WITHOUT the schedule patch would be
incoherent (stock schedule pulls a fixed count while the iterator plans a
possibly-different per-rank count -> drift), so both are kept.
Co-authored-by: Cursor <cursoragent@cursor.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
|
|
||
|
|
||
| __all__ = [ | ||
| "init_shmem", |
|
|
||
| __all__ = [ | ||
| "init_shmem", | ||
| "SymmBufferRegistry", |
| __all__ = [ | ||
| "init_shmem", | ||
| "SymmBufferRegistry", | ||
| "ReductionService", |
| "init_shmem", | ||
| "SymmBufferRegistry", | ||
| "ReductionService", | ||
| "GatherService", |
| "SymmBufferRegistry", | ||
| "ReductionService", | ||
| "GatherService", | ||
| "finalize_distributed", |
Collaborator
There was a problem hiding this comment.
Why are these being ignored?
Xiaoming-AMD
requested changes
Jul 16, 2026
Address review "why are these being ignored": the block ignored personal dev scratch that does not belong in the shared repo .gitignore — the ODC module's local tests/ (stale NVSHMEM), docs/, examples/ (probe/repro scripts), internal *_REPORT.md notes, and one-off experiment sweep yamls. Those are moved to a local .git/info/exclude instead. Keep only genuine generated output dirs (odc_logs/, odc_run/); .cursor/ comes from main. Co-authored-by: Cursor <cursoragent@cursor.com>
Follow-up to the .gitignore trim: run_odc.sh writes logs to $HOME/odc_logs (outside the repo) and odc_run/ is unreferenced, so the repo .gitignore needs no ODC entries. .gitignore now matches main (PR adds nothing to it). Co-authored-by: Cursor <cursoragent@cursor.com>
Set odc_p2p_backend=rocshmem (and odc_rocshmem_gda=true for the dual-node 14B) in the two ODC example configs so they run the validated backend out of the box (trainer_base defaults to mori, whose multi-node path has a known bug). No other hyperparameters changed. Co-authored-by: Cursor <cursoragent@cursor.com>
Xiaoming-AMD
approved these changes
Jul 16, 2026
| torch.distributed.barrier(group=pg) | ||
| self.dispatched_tasks += 1 | ||
|
|
||
| def scatter_accumulate(self, key, input_tensor, pg: dist.ProcessGroup): |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adapts the On-Demand Communication (ODC) framework to the AMD ROCm stack (MI300X) inside Primus. ODC reduces gradient-communication synchronization from per-microbatch to per-minibatch and overlaps single-sided push/pull communication with the backward pass, enabling LB-Mini load balancing (variable microbatch counts per rank) on top of a rocSHMEM-based point-to-point backend.
The rocSHMEM point-to-point ops are now provided by Primus-Turbo (primus_turbo.pytorch._C.odc_rocshmem_host / odc_rocshmem_gda) and consumed by _rocshmem_backend.py, instead of being built in-tree as ctypes .sos — ODC is therefore a pure-Python in-tree module now (the in-tree rocSHMEM sources and build_rocshmem_backend.sh are removed). This depends on Primus-Turbo PR #409. Validated to match the pure-Primus baseline on both single-node (host / XGMI IPC) and multi-node (GDA) runs.