Skip to content

[FEAT] Allocate CUDA device memory in a locality domain - #242

Merged
tqchen merged 11 commits into
apache:mainfrom
jinhongyii:locality-domain
Nov 14, 2026
Merged

tqchen merged 11 commits into
apache:mainfrom
jinhongyii:locality-domain

Conversation

@jinhongyii

@jinhongyii jinhongyii commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

CUDA 13.4 exposes locality domains. On GPUs such as Rubin, the SMs and device memory are split into domains, and an SM reads memory in its own domain faster than memory in another domain. This PR lets TIRx place tensors in a chosen domain and lets users query the domain topology.

User-facing API

from tvm_tirx.backend.cuda.runtime import locality_domain

dev = tvm_tirx.cuda(0)                                   # or just the device id, 0
locality_domain.get_num_locality_domains(dev)            # 2 on VR200; 1 on devices without domains
locality_domain.is_localization_supported(dev)           # True when there are 2+ domains
locality_domain.get_num_sms_per_locality_domain(dev)     # SMs guaranteed to each domain

x = tvm_tirx.runtime.empty((m,), "float32", dev, mem_scope="global.locality_domain.1")
y = tvm_tirx.runtime.tensor(arr, dev, mem_scope=locality_domain.locality_domain_scope(0))
z = w.copyto(dev, mem_scope=locality_domain.locality_domain_scope(1))
locality_domain.get_locality_domain_id(x)                # 1; None for memory that is not localized
  • The query names follow PyTorch (torch.cuda.green_contexts.get_num_locality_domains, is_localization_supported, and the locality_domain_id of green contexts).
  • Inside a kernel, T.cuda.mov_sreg(32, "locality_domain_id") returns the domain the CTA runs on (PTX ISA 9.5). A kernel can use it to work on the data stored in its own domain.
  • Every domain's memory is accessible from every SM; only the access speed differs.
  • Localized tensors are ordinary CUDA tensors: they can be passed to kernels and shared through DLPack (e.g. with torch).

Implementation

  • Memory scopes: global.locality_domain.<k> is registered for k < 8 in the CUDA device API, with k stored as the scope's cuda.locality_domain_id enum attribute. Empty now has a third path for these scopes. It allocates from a per-(device, domain) memory pool that is created on first use with CU_MEM_LOCATION_TYPE_DEVICE_LOCALITY_DOMAIN. The plain global path is unchanged. Pool memory is freed by synchronizing the device and then calling cudaFree, because cudaFree waits for running kernels only on cudaMalloc memory, not on pool memory. This makes freeing as safe as for global, on any stream. On a device that reports a single domain (including any CUDA older than 13.4), global.locality_domain.0 is a plain allocation. Freed memory goes back to the device: the release threshold stays at its default of 0.
  • New FFI functions: each Python query wraps the tvm-ffi function of the same name under runtime.cuda.: GetNumLocalityDomains, IsLocalizationSupported, GetNumSMsPerLocalityDomain and GetLocalityDomainId. They are bound through tvm_tirx.backend.cuda.runtime._ffi_api; the new tvm_tirx.backend.cuda.runtime.locality_domain module has an API reference page.
  • Driver API instead of the runtime API: the CUDA runtime loaded at run time can predate 13.4, for example the one shipped in a PyTorch wheel. Such a runtime knows neither the new attributes nor cudaPointerAttributes::localityDomainOrdinal; we saw garbage ordinals until switching to the driver API.
  • Version guards: the code is guarded by CUDA_VERSION >= 13040 at build time and by the driver version at run time. CUDA 13.4 is where these driver APIs first appear (as in PyTorch); reading %locality_domain_id in a kernel still needs PTX ISA 9.5 (CUDA 13.5). Without support, a device reports one domain and the scopes raise a clear error.

Testing (VR200 / sm_107a, CUDA 13.5, 2 domains)

Tested with CUDA 13.5; the 13.4 gate was checked against the 13.4 headers but not run on a 13.4 driver.

tests/python/backend/cuda/test_locality_domain.py covers:

  • each scope reports its own domain back, and plain allocations report None;
  • data round trips through tensor, copyfrom and copyto;
  • the queries accept a device or a device id and reject non-CUDA devices;
  • out-of-range domains are rejected with the device's actual domain count;
  • freed memory returns to the device, and a tensor can be dropped right after launching a kernel on it;
  • a localized allocation works as the very first CUDA call of a process (this is a regression test for a missing current context found while benchmarking);
  • kernels and DLPack/torch work on localized tensors;
  • a domain-aware kernel writes only the half that lives in the domain its CTAs run on;
  • a persistent work-queue kernel (the one benchmarked below) reads every chunk exactly once. It is checked by an XOR checksum and by the queue counters, both with a full grid and with a single CTA, where all of the other domain's work has to be stolen.

Existing tests are unaffected. tests/python/backend and tests/python/runtime pass, except test_fp4.py::test_e2m1_dequantize, which fails the same way without this change (a ShuffleObj codegen check).

Benchmark

The kernel is the work-queue kernel from the tests, written in TIRx and scaled up to 2 x 4 GiB, with a 512-thread CTA and 1 MiB chunks. It loads with ld.global.cs.v4.u32, fully unrolled within a chunk, and launches 4 CTAs per SM. Each domain's CTAs drain their own buffer's queue and then help the other domain. Times are CUDA-event medians of 20 runs on VR200. An XOR checksum of every word confirms that all the data is read.

buffers read bandwidth
each domain's CTAs read the buffer in their own domain 16.6 TB/s
plain cudaMalloc buffers 13.0-13.7 TB/s
each domain's CTAs read the buffer in the other domain 7.1 TB/s

Placing data in the reading SMs' domain is about 22% faster than a plain allocation and 2.3x faster than reading across domains. Stealing from the other domain's queue costs about 2% (17.0 TB/s without it). Without stealing, though, the kernel would be wrong whenever a domain receives no CTAs.

Not included

  • Partitioning SMs by domain through green contexts.
  • The rubin_dsmem_locality cluster scheduling policy, which concerns DSMEM placement within a cluster rather than memory locality.

🤖 Generated with Claude Code

jinhongyii and others added 2 commits November 12, 2026 01:24
CUDA 13.5 exposes locality domains: on GPUs such as Rubin the SMs and device
memory are split into domains, and an SM reads memory of its own domain
faster. Let TIRx place tensors in a domain:

- Memory scopes `global.locality_domain.<k>` (k < 8). `Empty` serves them from
  a per-(device, domain) memory pool created with
  CU_MEM_LOCATION_TYPE_DEVICE_LOCALITY_DOMAIN; the plain `global` path is
  unchanged, and cudaFree releases both kinds. So `runtime.empty`, `tensor`
  and `copyto` take `mem_scope="global.locality_domain.<k>"` as they are.
- `runtime.GetCudaLocalityDomainCount`, `...MultiprocessorCount` and
  `runtime.GetCudaTensorLocalityDomain`, wrapped by the new
  `tvm_tirx.backend.cuda.locality` module (`locality_domain_count`,
  `locality_domain_sm_count`, `locality_domain_of`, `locality_domain_scope`).

The locality-domain calls use the driver API: the CUDA runtime loaded at run
time can predate 13.5 (e.g. a PyTorch wheel's), and then neither knows the
attributes nor fills `cudaPointerAttributes::localityDomainOrdinal`. They are
guarded by CUDA_VERSION at build time and by the driver version at run time;
without support a device reports one domain and the scopes raise a clear error.

Tests (VR200, 2 domains): placement reported back per domain, data round trips
including copyto, out-of-range domains, freed memory returning to the device,
use as the first CUDA call of a process, kernels and DLPack/torch on localized
tensors, and a kernel that writes the half living in the domain each CTA runs
on. On VR200, reading 2 x 4 GiB with each domain's CTAs reading their own
domain's buffer gives 8.98 TB/s, against 7.57 TB/s from cudaMalloc buffers and
5.98 TB/s when every domain reads the other domain's buffer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CTAs read the buffer of the locality domain they run in through a per-domain
chunk queue, then steal from the other domain's queue, since the hardware
decides where CTAs land. An XOR checksum and the queue counters verify that
every chunk is read exactly once, with a full grid and with a single CTA.
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

locality_domain.py

namespace refl = tvm::ffi::reflection;
RegisterLocalityDomainScopes();
refl::GlobalDef()
.def("runtime.GetCudaLocalityDomainCount", GetCudaLocalityDomainCount)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

runtime.cuda.GetLocalityDomainCount

Address review feedback and align the API with torch.cuda.green_contexts:

- Python module tvm_tirx.backend.cuda.locality -> tvm_tirx.runtime.locality_domain,
  with get_num_locality_domains, is_localization_supported,
  get_num_sms_per_locality_domain, get_locality_domain_id and
  locality_domain_scope. Device arguments accept a Device or a device id.
- tvm-ffi functions are registered as runtime.cuda.<Name>, named after the
  Python function they back (e.g. runtime.cuda.GetNumLocalityDomains).
- get_locality_domain_id returns None, not -1, for memory that is not localized.
- Host-side support is gated on CUDA 13.4, where the driver APIs first appear.
- Docs page and tests move to runtime.
@jinhongyii

Copy link
Copy Markdown
Contributor Author

Updated in 10a0cbd: the module is now tvm_tirx/runtime/locality_domain.py, and the FFI functions are registered as runtime.cuda.<Name>, each named after the Python function it backs (get_num_locality_domains -> runtime.cuda.GetNumLocalityDomains, etc.). The query names follow torch.cuda.green_contexts.

On why the module is tvm_tirx.runtime.locality_domain rather than tvm_tirx.runtime.cuda.locality_domain: tvm_tirx.runtime.cuda is already the device constructor (def cuda(dev_id=0) in runtime/_tensor.py, re-exported as tvm_tirx.cuda), so it can't also be a subpackage without either breaking tvm_tirx.cuda(0) or turning it into a callable module. Since there are no other CUDA-specific runtime modules yet, I kept it as a flat runtime submodule. The CUDA-specific part is still visible in the FFI names (runtime.cuda.*). Happy to move it if you'd prefer a different layout.

@jinhongyii

Copy link
Copy Markdown
Contributor Author

@tqchen

CI builds with CUDA 13.1, where global.locality_domain.0 raised instead of
behaving like global as documented. Domain 0 on a device that reports a
single locality domain, including any device without CUDA 13.4 support, is
now a plain cudaMalloc allocation. The scope registration test is skipped
when TVM-TIRx is built without the CUDA runtime.

@tqchen tqchen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comments for further cleanup

/*hStream=*/nullptr));
// Like cudaMalloc, the allocation is usable from any stream once this returns.
CUDA_DRIVER_CALL(cuStreamSynchronize(nullptr));
TVM_FFI_ICHECK_EQ(ret % kAllocAlignment, 0U);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this check is overly defensive

GetLocalityDomainPool(dev.device_id, domain),
/*hStream=*/nullptr));
// Like cudaMalloc, the allocation is usable from any stream once this returns.
CUDA_DRIVER_CALL(cuStreamSynchronize(nullptr));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not needed, since we allocate and it synchronizes to default stream, for other non-blocking stream, we need a sync from default to yhat one

* at its default of 0: freed memory goes back to the device at the next
* synchronization instead of staying reserved by the pool.
*/
CUmemoryPool GetLocalityDomainPool(int device_id, int domain) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider refactor to a class with Global()->GetLocalityDomainPool, and mutex as member of the class

}

/*! \brief The domain k of `global.locality_domain.<k>`, or nullopt for any other scope. */
std::optional<int> LocalityDomainOf(const MemoryScope& scope) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

avoid use string manipulation, instead use enum attribute, which can store integer value and being lookedup by the enum

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what do you mean here

constexpr const char* kLocalityDomainScopePrefix = "global.locality_domain.";

void RegisterLocalityDomainScopes() {
static const bool registered = [] {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

static is not useful here as static init block is already called

// The pool calls below use the driver API, which needs a current context; make
// the runtime create the device's primary context and bind it to this thread,
// as it would for its own first call (the allocation may be the first CUDA call).
TVM_FFI_CHECK_CUDA_ERROR(cudaFree(nullptr));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update to

// Ensure a current context even when the guard did not switch devices.
TVM_FFI_CHECK_CUDA_ERROR(cudaSetDevice(dev.device_id));

# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""CUDA locality domains: per-domain device memory and topology queries.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move to backend.cuda.runtime.locality_domain.py

@tqchen tqchen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary of the requested cleanup:

  1. Move the Python module to tvm_tirx.backend.cuda.runtime.locality_domain. Add a private backend/cuda/runtime/_ffi_api.py using tvm_ffi.init_ffi_api("runtime.cuda", __name__), and use those bindings from the Python wrappers.
  2. Keep the current query names: get_num_locality_domains, is_localization_supported, get_num_sms_per_locality_domain, and get_locality_domain_id, with the corresponding runtime.cuda.GetNumLocalityDomains, IsLocalizationSupported, GetNumSMsPerLocalityDomain, and GetLocalityDomainId registrations. Prefer existing PyTorch names where the operations match. This supersedes my earlier suggestion to rename the count query to GetLocalityDomainCount. The tensor query follows the locality_domain_id terminology; the verified PyTorch properties refer to compute placement.
  3. Update the documentation alongside the module move: API-reference page location/title, automodule/currentmodule, autosummary and toctree links, namespace listings, and import examples. Update test imports as well.
  4. Put pool management in a separate LocalityDomainPoolManager with Global() and a member mutex. Keep the newer locality-domain feature isolated from the general CUDADeviceAPI state. Use std::vector<std::vector<CUmemoryPool>>, indexed by device and domain, with nullptr meaning not yet created. Validate IDs before indexing; protect resize, lookup, and creation with the mutex; return handles by value. Preserve process-lifetime ownership to avoid CUDA cleanup during static destruction.
  5. Store the locality-domain integer as a memory-scope enum attribute and look it up by enum identity, instead of parsing the scope name. Absence of the attribute means it is not a locality-domain scope. Validate both lower and upper bounds against the device topology.
  6. Remove the redundant function-local static registration lambda: the registration already runs in a static initialization block. Remove the redundant post-allocation alignment assertion.
  7. Replace the context-initialization cudaFree(nullptr) idiom with an explicit cudaSetDevice(dev.device_id) after the device guard, so the current context is initialized even when the guard does not switch devices. This relies on a loaded CUDA runtime >=12.0; verify that against the supported runtime floor. Keep cuInit(0) in the driver-attribute query path so queries work as the first CUDA operation.
  8. Remove the unconditional cuStreamSynchronize(nullptr) after allocation under the default-stream ordering contract. Same-stream use and legacy-default-stream dependencies with ordinary blocking streams already provide ordering. Consumers on nonblocking streams must explicitly establish a dependency from the allocation/default stream; document and exercise this case.

Additional correctness checks before landing:

  • Confirm the destruction contract for pool-backed tensors. cudaFree of asynchronously allocated memory does not implicitly wait for outstanding accesses; allocation readiness alone does not establish safe deallocation. Cover tensor lifetime after asynchronous use.
  • Gate tests that read %locality_domain_id on the required PTX/toolkit support, not only GPU architecture and domain count: allocation support starts at CUDA 13.4, while the PR states that this register needs PTX 9.5/CUDA 13.5.
  • Make the fallback documentation consistent: domain zero falls back to ordinary global allocation when locality support is unavailable, and its queried domain can be None.

The scope-helper rename and removal of the public maximum-domain constant were discussed as alternatives, but are not requested changes in this summary. Please preserve existing commits and add any follow-ups as new commits on top.

- Python module tvm_tirx.runtime.locality_domain ->
  tvm_tirx.backend.cuda.runtime.locality_domain, calling the runtime.cuda.*
  functions through a private _ffi_api module; docs and tests move with it.
- Locality-domain scopes carry their domain as the memory-scope enum
  attribute cuda.locality_domain_id instead of being parsed from the name.
- Pools live in a LocalityDomainPoolManager, a vector indexed by device and
  domain behind a member mutex, still never destroyed.
- Drop the redundant static registration guard and alignment check; make the
  context current with cudaSetDevice instead of cudaFree(nullptr).
- Free pool memory with cuMemFreeAsync on the legacy default stream: unlike
  for cudaMalloc memory, cuMemFree does not wait for kernels still using it.
  Test dropping a tensor right after launching a kernel on it.
- Skip the kernels that read %locality_domain_id unless nvcc is 13.5+, and
  document that domain 0 falls back to plain memory without domain support.
- Free localized memory like global memory: synchronize the device, then
  cudaFree. cudaFree of pool memory does not wait for kernels still using it,
  and cuMemFreeAsync on the legacy default stream did not cover nonblocking
  streams.
- Make a context current before cuPointerGetAttribute in GetLocalityDomainId,
  which may run on a thread that never used CUDA.
- Query the locality domain count once per device, through cuDeviceGet.
- Tests: run free_returns_memory under the GPU lock; parse two-digit archs.
Empty validates a locality-domain scope and, on a device with a single
domain, allocates it like global. AllocLocalityDomainSpace and
FreeLocalityDomainSpace now only handle devices with several domains.
@jinhongyii

Copy link
Copy Markdown
Contributor Author

@tqchen this is ready for review

@tqchen
tqchen merged commit 6569e2a into apache:main Nov 14, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants