Repository navigation
[FEAT] Allocate CUDA device memory in a locality domain - #242
Conversation
CUDA 13.5 exposes locality domains: on GPUs such as Rubin the SMs and device memory are split into domains, and an SM reads memory of its own domain faster. Let TIRx place tensors in a domain: - Memory scopes `global.locality_domain.<k>` (k < 8). `Empty` serves them from a per-(device, domain) memory pool created with CU_MEM_LOCATION_TYPE_DEVICE_LOCALITY_DOMAIN; the plain `global` path is unchanged, and cudaFree releases both kinds. So `runtime.empty`, `tensor` and `copyto` take `mem_scope="global.locality_domain.<k>"` as they are. - `runtime.GetCudaLocalityDomainCount`, `...MultiprocessorCount` and `runtime.GetCudaTensorLocalityDomain`, wrapped by the new `tvm_tirx.backend.cuda.locality` module (`locality_domain_count`, `locality_domain_sm_count`, `locality_domain_of`, `locality_domain_scope`). The locality-domain calls use the driver API: the CUDA runtime loaded at run time can predate 13.5 (e.g. a PyTorch wheel's), and then neither knows the attributes nor fills `cudaPointerAttributes::localityDomainOrdinal`. They are guarded by CUDA_VERSION at build time and by the driver version at run time; without support a device reports one domain and the scopes raise a clear error. Tests (VR200, 2 domains): placement reported back per domain, data round trips including copyto, out-of-range domains, freed memory returning to the device, use as the first CUDA call of a process, kernels and DLPack/torch on localized tensors, and a kernel that writes the half living in the domain each CTA runs on. On VR200, reading 2 x 4 GiB with each domain's CTAs reading their own domain's buffer gives 8.98 TB/s, against 7.57 TB/s from cudaMalloc buffers and 5.98 TB/s when every domain reads the other domain's buffer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CTAs read the buffer of the locality domain they run in through a per-domain chunk queue, then steal from the other domain's queue, since the hardware decides where CTAs land. An XOR checksum and the queue counters verify that every chunk is read exactly once, with a full grid and with a single CTA.
| # software distributed under the License is distributed on an | ||
| # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| # KIND, either express or implied. See the License for the | ||
| # specific language governing permissions and limitations |
| namespace refl = tvm::ffi::reflection; | ||
| RegisterLocalityDomainScopes(); | ||
| refl::GlobalDef() | ||
| .def("runtime.GetCudaLocalityDomainCount", GetCudaLocalityDomainCount) |
There was a problem hiding this comment.
runtime.cuda.GetLocalityDomainCount
Address review feedback and align the API with torch.cuda.green_contexts: - Python module tvm_tirx.backend.cuda.locality -> tvm_tirx.runtime.locality_domain, with get_num_locality_domains, is_localization_supported, get_num_sms_per_locality_domain, get_locality_domain_id and locality_domain_scope. Device arguments accept a Device or a device id. - tvm-ffi functions are registered as runtime.cuda.<Name>, named after the Python function they back (e.g. runtime.cuda.GetNumLocalityDomains). - get_locality_domain_id returns None, not -1, for memory that is not localized. - Host-side support is gated on CUDA 13.4, where the driver APIs first appear. - Docs page and tests move to runtime.
|
Updated in 10a0cbd: the module is now On why the module is |
CI builds with CUDA 13.1, where global.locality_domain.0 raised instead of behaving like global as documented. Domain 0 on a device that reports a single locality domain, including any device without CUDA 13.4 support, is now a plain cudaMalloc allocation. The scope registration test is skipped when TVM-TIRx is built without the CUDA runtime.
| /*hStream=*/nullptr)); | ||
| // Like cudaMalloc, the allocation is usable from any stream once this returns. | ||
| CUDA_DRIVER_CALL(cuStreamSynchronize(nullptr)); | ||
| TVM_FFI_ICHECK_EQ(ret % kAllocAlignment, 0U); |
| GetLocalityDomainPool(dev.device_id, domain), | ||
| /*hStream=*/nullptr)); | ||
| // Like cudaMalloc, the allocation is usable from any stream once this returns. | ||
| CUDA_DRIVER_CALL(cuStreamSynchronize(nullptr)); |
There was a problem hiding this comment.
this is not needed, since we allocate and it synchronizes to default stream, for other non-blocking stream, we need a sync from default to yhat one
| * at its default of 0: freed memory goes back to the device at the next | ||
| * synchronization instead of staying reserved by the pool. | ||
| */ | ||
| CUmemoryPool GetLocalityDomainPool(int device_id, int domain) { |
There was a problem hiding this comment.
consider refactor to a class with Global()->GetLocalityDomainPool, and mutex as member of the class
| } | ||
|
|
||
| /*! \brief The domain k of `global.locality_domain.<k>`, or nullopt for any other scope. */ | ||
| std::optional<int> LocalityDomainOf(const MemoryScope& scope) { |
There was a problem hiding this comment.
avoid use string manipulation, instead use enum attribute, which can store integer value and being lookedup by the enum
There was a problem hiding this comment.
what do you mean here
| constexpr const char* kLocalityDomainScopePrefix = "global.locality_domain."; | ||
|
|
||
| void RegisterLocalityDomainScopes() { | ||
| static const bool registered = [] { |
There was a problem hiding this comment.
static is not useful here as static init block is already called
| // The pool calls below use the driver API, which needs a current context; make | ||
| // the runtime create the device's primary context and bind it to this thread, | ||
| // as it would for its own first call (the allocation may be the first CUDA call). | ||
| TVM_FFI_CHECK_CUDA_ERROR(cudaFree(nullptr)); |
There was a problem hiding this comment.
update to
// Ensure a current context even when the guard did not switch devices.
TVM_FFI_CHECK_CUDA_ERROR(cudaSetDevice(dev.device_id));
| # KIND, either express or implied. See the License for the | ||
| # specific language governing permissions and limitations | ||
| # under the License. | ||
| """CUDA locality domains: per-domain device memory and topology queries. |
There was a problem hiding this comment.
move to backend.cuda.runtime.locality_domain.py
tqchen
left a comment
There was a problem hiding this comment.
Summary of the requested cleanup:
- Move the Python module to
tvm_tirx.backend.cuda.runtime.locality_domain. Add a privatebackend/cuda/runtime/_ffi_api.pyusingtvm_ffi.init_ffi_api("runtime.cuda", __name__), and use those bindings from the Python wrappers. - Keep the current query names:
get_num_locality_domains,is_localization_supported,get_num_sms_per_locality_domain, andget_locality_domain_id, with the correspondingruntime.cuda.GetNumLocalityDomains,IsLocalizationSupported,GetNumSMsPerLocalityDomain, andGetLocalityDomainIdregistrations. Prefer existing PyTorch names where the operations match. This supersedes my earlier suggestion to rename the count query toGetLocalityDomainCount. The tensor query follows thelocality_domain_idterminology; the verified PyTorch properties refer to compute placement. - Update the documentation alongside the module move: API-reference page location/title,
automodule/currentmodule, autosummary and toctree links, namespace listings, and import examples. Update test imports as well. - Put pool management in a separate
LocalityDomainPoolManagerwithGlobal()and a member mutex. Keep the newer locality-domain feature isolated from the generalCUDADeviceAPIstate. Usestd::vector<std::vector<CUmemoryPool>>, indexed by device and domain, withnullptrmeaning not yet created. Validate IDs before indexing; protect resize, lookup, and creation with the mutex; return handles by value. Preserve process-lifetime ownership to avoid CUDA cleanup during static destruction. - Store the locality-domain integer as a memory-scope enum attribute and look it up by enum identity, instead of parsing the scope name. Absence of the attribute means it is not a locality-domain scope. Validate both lower and upper bounds against the device topology.
- Remove the redundant function-local static registration lambda: the registration already runs in a static initialization block. Remove the redundant post-allocation alignment assertion.
- Replace the context-initialization
cudaFree(nullptr)idiom with an explicitcudaSetDevice(dev.device_id)after the device guard, so the current context is initialized even when the guard does not switch devices. This relies on a loaded CUDA runtime >=12.0; verify that against the supported runtime floor. KeepcuInit(0)in the driver-attribute query path so queries work as the first CUDA operation. - Remove the unconditional
cuStreamSynchronize(nullptr)after allocation under the default-stream ordering contract. Same-stream use and legacy-default-stream dependencies with ordinary blocking streams already provide ordering. Consumers on nonblocking streams must explicitly establish a dependency from the allocation/default stream; document and exercise this case.
Additional correctness checks before landing:
- Confirm the destruction contract for pool-backed tensors.
cudaFreeof asynchronously allocated memory does not implicitly wait for outstanding accesses; allocation readiness alone does not establish safe deallocation. Cover tensor lifetime after asynchronous use. - Gate tests that read
%locality_domain_idon the required PTX/toolkit support, not only GPU architecture and domain count: allocation support starts at CUDA 13.4, while the PR states that this register needs PTX 9.5/CUDA 13.5. - Make the fallback documentation consistent: domain zero falls back to ordinary global allocation when locality support is unavailable, and its queried domain can be
None.
The scope-helper rename and removal of the public maximum-domain constant were discussed as alternatives, but are not requested changes in this summary. Please preserve existing commits and add any follow-ups as new commits on top.
- Python module tvm_tirx.runtime.locality_domain -> tvm_tirx.backend.cuda.runtime.locality_domain, calling the runtime.cuda.* functions through a private _ffi_api module; docs and tests move with it. - Locality-domain scopes carry their domain as the memory-scope enum attribute cuda.locality_domain_id instead of being parsed from the name. - Pools live in a LocalityDomainPoolManager, a vector indexed by device and domain behind a member mutex, still never destroyed. - Drop the redundant static registration guard and alignment check; make the context current with cudaSetDevice instead of cudaFree(nullptr). - Free pool memory with cuMemFreeAsync on the legacy default stream: unlike for cudaMalloc memory, cuMemFree does not wait for kernels still using it. Test dropping a tensor right after launching a kernel on it. - Skip the kernels that read %locality_domain_id unless nvcc is 13.5+, and document that domain 0 falls back to plain memory without domain support.
- Free localized memory like global memory: synchronize the device, then cudaFree. cudaFree of pool memory does not wait for kernels still using it, and cuMemFreeAsync on the legacy default stream did not cover nonblocking streams. - Make a context current before cuPointerGetAttribute in GetLocalityDomainId, which may run on a thread that never used CUDA. - Query the locality domain count once per device, through cuDeviceGet. - Tests: run free_returns_memory under the GPU lock; parse two-digit archs.
Empty validates a locality-domain scope and, on a device with a single domain, allocates it like global. AllocLocalityDomainSpace and FreeLocalityDomainSpace now only handle devices with several domains.
|
@tqchen this is ready for review |
CUDA 13.4 exposes locality domains. On GPUs such as Rubin, the SMs and device memory are split into domains, and an SM reads memory in its own domain faster than memory in another domain. This PR lets TIRx place tensors in a chosen domain and lets users query the domain topology.
User-facing API
torch.cuda.green_contexts.get_num_locality_domains,is_localization_supported, and thelocality_domain_idof green contexts).T.cuda.mov_sreg(32, "locality_domain_id")returns the domain the CTA runs on (PTX ISA 9.5). A kernel can use it to work on the data stored in its own domain.Implementation
global.locality_domain.<k>is registered for k < 8 in the CUDA device API, with k stored as the scope'scuda.locality_domain_idenum attribute.Emptynow has a third path for these scopes. It allocates from a per-(device, domain) memory pool that is created on first use withCU_MEM_LOCATION_TYPE_DEVICE_LOCALITY_DOMAIN. The plainglobalpath is unchanged. Pool memory is freed by synchronizing the device and then callingcudaFree, becausecudaFreewaits for running kernels only oncudaMallocmemory, not on pool memory. This makes freeing as safe as forglobal, on any stream. On a device that reports a single domain (including any CUDA older than 13.4),global.locality_domain.0is a plain allocation. Freed memory goes back to the device: the release threshold stays at its default of 0.runtime.cuda.:GetNumLocalityDomains,IsLocalizationSupported,GetNumSMsPerLocalityDomainandGetLocalityDomainId. They are bound throughtvm_tirx.backend.cuda.runtime._ffi_api; the newtvm_tirx.backend.cuda.runtime.locality_domainmodule has an API reference page.cudaPointerAttributes::localityDomainOrdinal; we saw garbage ordinals until switching to the driver API.CUDA_VERSION >= 13040at build time and by the driver version at run time. CUDA 13.4 is where these driver APIs first appear (as in PyTorch); reading%locality_domain_idin a kernel still needs PTX ISA 9.5 (CUDA 13.5). Without support, a device reports one domain and the scopes raise a clear error.Testing (VR200 / sm_107a, CUDA 13.5, 2 domains)
Tested with CUDA 13.5; the 13.4 gate was checked against the 13.4 headers but not run on a 13.4 driver.
tests/python/backend/cuda/test_locality_domain.pycovers:tensor,copyfromandcopyto;Existing tests are unaffected.
tests/python/backendandtests/python/runtimepass, excepttest_fp4.py::test_e2m1_dequantize, which fails the same way without this change (a ShuffleObj codegen check).Benchmark
The kernel is the work-queue kernel from the tests, written in TIRx and scaled up to 2 x 4 GiB, with a 512-thread CTA and 1 MiB chunks. It loads with
ld.global.cs.v4.u32, fully unrolled within a chunk, and launches 4 CTAs per SM. Each domain's CTAs drain their own buffer's queue and then help the other domain. Times are CUDA-event medians of 20 runs on VR200. An XOR checksum of every word confirms that all the data is read.cudaMallocbuffersPlacing data in the reading SMs' domain is about 22% faster than a plain allocation and 2.3x faster than reading across domains. Stealing from the other domain's queue costs about 2% (17.0 TB/s without it). Without stealing, though, the kernel would be wrong whenever a domain receives no CTAs.
Not included
rubin_dsmem_localitycluster scheduling policy, which concerns DSMEM placement within a cluster rather than memory locality.🤖 Generated with Claude Code