mem-ruby,configs: add a configurable GPU L3 to GPU_VIPER - #9
Open
Basemism wants to merge 2 commits into
Open
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
There are correctness issues in --numa-high-bit handling for start_index_bit, and the L3Exclusive eviction path can drop dirty data (plus a confirmed argparse default type hazard now affecting new TCC bank usage).
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR adds an optional, configurable memory-side GPU L3 (Infinity Cache–like) behavior to the MOESI_AMD_Base GPU directory path used by GPU_VIPER, and extends GPU_VIPER configurability for directory/TCC/L3 banking and address-bit placement.
Changes:
- Add GPU L3 retention vs. consume behavior (inclusive vs. experimental exclusive) in
MOESI_AMD_Base-dir.sm, plus L3 fill behavior on relevant completion paths. - Add GPU_VIPER CLI options to enable GPU L3 and exclusive mode, and make TCC/L3 per-slice bank counts independently configurable.
- Adjust L3 indexing bit placement intent by explicitly setting
start_index_bitin GPU_VIPER directory construction.
File summaries
| File | Description |
|---|---|
| src/mem/ruby/protocol/MOESI_AMD_Base-dir.sm | Implements GPU L3 retention/fill policies and adds L3Exclusive handling hooks in the directory SLICC machine. |
| configs/ruby/GPU_VIPER.py | Adds GPU L3 config flags, separates bank-count configurability, and sets L3 start_index_bit during directory construction. |
Review details
Suppressed comments (3)
configs/ruby/GPU_VIPER.py:605
options.numa_high_bitchanges which bits are used for directory interleaving (intlvHighBit=numa_bit). SettingL3CacheMemory.start_index_bittoblock_size_bits + dir_bitscan overlap with/underlap the actual directory-selection bits when--numa-high-bitis provided. To ensure the L3 set-index bits are always above the directory-selection bits, derive this fromnuma_bit(the current interleave high bit) instead.
dir_cntrl = DirCntrl(noTCCdir=True, TCC_select_num_bits=TCC_bits)
dir_cntrl.create(options, dir_ranges, ruby_system, system)
dir_cntrl.L3CacheMemory.start_index_bit = block_size_bits + dir_bits
dir_cntrl.number_of_TBEs = options.num_tbes
configs/ruby/GPU_VIPER.py:675
- Same issue as the CPU-side directories:
start_index_bitshould be derived from the interleave high bit (numa_bit) so the L3 set-index bits remain above the directory-selection bits regardless of how the address is interleaved.
dir_cntrl.create(options, [addr_range], ruby_system, system,
num_dirs=options.dgpu_num_dirs)
dir_cntrl.L3CacheMemory.start_index_bit = block_size_bits + dir_bits
dir_cntrl.number_of_TBEs = options.num_tbes
src/mem/ruby/protocol/MOESI_AMD_Base-dir.sm:1297
- Same eviction-path issue as above: the silent-drop path should be limited to the non-exclusive GPU L3 policy. In
L3Exclusivemode, evicted victim-cache entries may be dirty and need a writeback.
if (L3CacheMemory.cacheAvail(address) == false) {
Addr victim := L3CacheMemory.cacheProbe(address);
if (GPUonly) {
// GPU L3 (Infinity Cache) is read-only — evicted entries are
// always clean, so silent drop is correct.
- Files reviewed: 2/2 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
1246
to
+1250
| if (L3CacheMemory.cacheAvail(address) == false) { | ||
| Addr victim := L3CacheMemory.cacheProbe(address); | ||
| CacheEntry victim_entry := static_cast(CacheEntry, "pointer", | ||
| L3CacheMemory.lookup(victim)); | ||
| enqueue(memQueue_out, MemoryMsg, to_memory_controller_latency) { | ||
| out_msg.addr := victim; | ||
| out_msg.Type := MemoryRequestType:MEMORY_WB; | ||
| out_msg.Sender := machineID; | ||
| out_msg.MessageSize := MessageSizeType:Writeback_Data; | ||
| out_msg.DataBlk := victim_entry.DataBlk; | ||
| if (GPUonly) { | ||
| // GPU L3 (Infinity Cache) is read-only — evicted entries are | ||
| // always clean, so silent drop is correct. |
Author
There was a problem hiding this comment.
VicDirty runs d_writeDataToMemory before al_allocateL3Block. The dirty data reaches memory before entering L3, so the resulting L3 copy is clean. Silent GPU-L3 eviction is safe in both policies.
Comment on lines
297
to
300
| self.size = MemorySize(options.tcc_size) | ||
| self.dataArrayBanks = ( | ||
| 256 / options.num_tccs | ||
| ) # number of data banks | ||
| self.tagArrayBanks = 256 / options.num_tccs # number of tag banks | ||
| self.dataArrayBanks = options.tcc_num_banks # number of data banks | ||
| self.tagArrayBanks = options.tcc_num_banks # number of tag banks | ||
| self.size.value = self.size.value / options.num_tccs |
Author
There was a problem hiding this comment.
arparse applies int so default becomes int 16
Comment on lines
1229
to
1234
| action(al_allocateL3Block, "al", desc="allocate the L3 block on WB") { | ||
| // GPU memory-side policy: write dirty data through to memory, then retain a | ||
| // clean copy in L3. Experimental exclusive policy uses victim-cache fills. | ||
| // CPU: original behavior — always fill. | ||
| if (true) { | ||
| peek(requestNetwork_in, CPURequestMsg) { |
Author
There was a problem hiding this comment.
Don't know how I missed this. Removed the wrapper.
Add a configurable GPU L3 cache to the GPU-only MOESI_AMD_Base directory and support both inclusive and exclusive allocation policies. Keep the feature disabled by default so existing GPU_VIPER configurations retain their current behavior. Handle L3 hits, fills, writebacks, atomics, DMA reads, and probe responses without leaving stale copies behind. Preserve inclusive L3 entries on ordinary GPU and DMA reads, invalidate or update an L3 entry when a dirty TCC probe supplies newer data, and retain the exclusive mode as an explicit option. Also compute directory and block index widths before processing --numa-high-bit so configurations using an explicit NUMA bit can construct the L3 cache safely.
Use the existing --tcc-num-banks option when constructing each TCC instead of deriving its internal bank count from the number of TCC controllers. Add --l3-num-banks to configure the data and tag banks in each directory's L3 slice. The default of 16 preserves the cache object's prior per-slice bank count while making the modeled organization explicit.
Basemism
force-pushed
the
staging-basem/gpu-l3
branch
from
September 2, 2026 14:24
1eaa94d to
16742eb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add an optional memory-side GPU L3 to the GPU directories in
MOESI_AMD_Base. The feature is disabled by default through --use-gpu-l3The default inclusive policy:
This policy is inferred from the Infinity Cache organization described on page 10 of the AMD CDNA 3 Architecture white paper (https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/white-papers/amd-cdna-3-white-paper.pdf)
--l3-exclusive retains an experimental victim-cache policy for configurations that need an exclusive hierarchy.
Also make TCC bank counts and per-slice L3 bank counts independently configurable through
--tcc-num-banksand--l3-num-banks. Place the L3 set-index bits above the directory-selection bits and fix construction when--numa-high-bitis supplied.