Add additive L2 stat for NVIDIA-comparable write hits (GAME CHANGER!!!)

[AndrewVu23]

Summary

Adds a stats-only WRITE_ALLOCATED counter so an NVIDIA-comparable L2 write-hit estimate (HIT + WRITE_ALLOCATED) can be correlated. If this works as assumed this might be a game changer.

Motivation / root cause

• On streaming-write ubenchmarks the sim reports ~0% L2 write-hit while HW reports ~100%, giving a negative correlation (−0.05 to −0.12). Also on multiple tests L2 write-hit undercounts in sim compared to hw run (off by A LOT !). However, on some other tests the sim counter matches the hw counters exactly. This leads me to believe that there is another stats we did not account in while calculating L2 write-hit.

• After some testing, I've come to the conclustion that a write to a never-read line is a MISS in wr_miss_wa_lazy_fetch_on_read, (the config that we used for testing) (all four write-allocate handlers return MISS), but the line is allocated. In this case NVIDIA counts it as a HIT (totally assumed here).

Changes

• gpu-cache.h: new cache_request_status::WRITE_ALLOCATED (stats-only, never returned from access())
• gpu-cache.cc: string entry; exclude from TOTAL_ACCESS; increment in process_tag_probe when a write miss allocates under a write-allocate policy.

Evidence (before / after)

Before: L2 Write Hits correlation: −0.05 (streaming ubenchs)
strict HIT under-counts (srad_v2 sim 4128 vs HW 24576).

• We can see how off the sim is in the correlators.

After: HIT + WRITE_ALLOCATED correlation 0.999 on 10 rodinia apps.

Test plan

• ./format-code.sh clean
• grep "GLOBAL_ACC_W\]\[WRITE_ALLOCATED\]" appears in sim stdout
• HIT / MISS / TOTAL_ACCESS byte-identical to a pre-change build (additive check)

Compatibility / backward compatibility

Fully additive

Limitations / known gaps

• THIS IS TOTALLY AN ASSUMPTION BASED ON MY OWN OBSERVATION. HIT + WRITE_ALLOCATED is an estimate, not an exact reproduction of NVIDIA's (undocumented) definition. Validated across two suites: rodinia_2.0-ft (mostly correct) and ISPASS-2009 (4/6 reasonable with 2 hard outliers: RAY 16× under, STO 3× over).
• Also worth noting that DRAM Read & L2 Read looks extremely ugly in these tests (not caused by my counter of course). I will look more into this in the future

Related

accel-sim/accel-sim-framework#518 (L2 seems to be a big problem with the tuner)
Sometimes L2 hit rate can exceed 100%, which is a problem documented by NVIDIA https://forums.developer.nvidia.com/t/l2-hit-rate-more-than-100/244570

[AndrewVu23]

Table for ISPASS-2009

app	HW wr-hit	HIT+WA	ratio	note
LPS	130000	130000	1.00	strict was 0 -> counter fully fixes a streaming write
NN	13039	13111	1.01	✅
BFS	334987	346768	1.04	✅
NQU	234	256	1.09	✅
RAY	276064	16384	0.06	sim 16× under
STO	7856	24576	3.13	sim 3× over

My next plan:

• Check what other stats RAY and STO might be reading (that other apps don't have ofc) -> another counter(s)?
• Run against other testsuites too

[AndrewVu23]

Just cracked another case. My theory was correct. It was another counter. A LOCAL_ACC one (again, totally my assumption).

FIX: accel-sim/accel-sim-framework@7f83680 (Note that everything I discussed below will be from the commit above)

NVIDIA's lts_t_sectors_srcunit_tex_op_write* counters are all-source they count every L1TEX->L2 write regardless of address space while the sim side read GLOBAL_ACC_W only. That's why I needed to add LOCAL_ACC_W from L2 cache stats too (in correl_mapping.py).

Instead of TOTAL_ACCESS like in GLOBAL_ACC, we have to divide LOCAL_ACC into different sectors, then add them together to find TOTAL_ACCESS. It is because in gpu_cache.cc:

for (unsigned type = 0; type < NUM_MEM_ACCESS_TYPE; ++type) {
    if (total_access[type] > 0)
        fprintf(fout, "\t%s[%s][%s] = %u\n", m_cache_name.c_str(),
             mem_access_type_str((enum mem_access_type)type), "TOTAL_ACCESS",
                total_access[type]);
}}}

only prints TOTAL_ACCESS when non-zero -> drop apps that don't have local traffic.

I found this bug when checking the run files of ispass-2009-RAY (which has local traffic) and ispass-2009-BFS (same test suite but no local traffic). This is what I saw that confirms the print theory:

RAY:

Total_core_cache_stats_breakdown[GLOBAL_ACC_R][TOTAL_ACCESS] = 8192
Total_core_cache_stats_breakdown[LOCAL_ACC_R][TOTAL_ACCESS] = 62960
Total_core_cache_stats_breakdown[GLOBAL_ACC_W][TOTAL_ACCESS] = 16384
Total_core_cache_stats_breakdown[LOCAL_ACC_W][TOTAL_ACCESS] = 263532

BFS:

Total_core_cache_stats_breakdown[GLOBAL_ACC_R][TOTAL_ACCESS] = 180
Total_core_cache_stats_breakdown[GLOBAL_ACC_W][TOTAL_ACCESS] = 21
// No stats for [LOCAL_ACC_W][TOTAL_ACCESS]

And when we add GLOBAL_ACC_W & LOCAL_ACC_W of RAY together, the counts become near identical between sim & hw.

I reran this with rodinia_2.0-ft first to ensure that it doesn't mess up anything, and as expected, since rodinia has no register spilling, the graphs stay the same.

And the results for ISPASS were surprising:
Before:

After:

This fixed not only 1, but 2 tests (which is LIB, the job I terminated early during the first run to eat lunch (so that's why it doesn't appear in the table above)) from ISPASS. More than I expected.

The reason why STO failed is because of many reasons:
STO

L2_cache_stats_breakdown[GLOBAL_ACC_R][TOTAL_ACCESS] = 6904
L2_cache_stats_breakdown[GLOBAL_ACC_W][TOTAL_ACCESS] = 24576
// No stats for [LOCAL_ACC_W][TOTAL_ACCESS]

Since there was no spilling in STO, the results may have something to do with the write policy. And since L1 stats between sim & hw match perfectly, the errors have to be the path from L1 -> L2. However, there's something interesting:

Total_core_cache_stats_breakdown[GLOBAL_ACC_R][TOTAL_ACCESS] = 96768
Total_core_cache_stats_breakdown[GLOBAL_ACC_W][TOTAL_ACCESS] = 24576

This is L1 stats. The [GLOBAL_ACC_W][TOTAL_ACCESS] matches perfectly between L1 & L2. But L2 between sim and hw don't match. So there's definitely something wrong.

Another Issue (Modeling Gap)

I then investigated the kernels of STO and did the math (cust.h & sha1_kernel.cu):
STO use byte-granularity writes (cust.h)

output = 49,152 threads × 4 B = 196,608 B = 1,536 lines = 6,144 sectors
L1 MISS 1,536 -> exactly one per 128 B line
L1 SECTOR_MISS 4,608 -> the other 3 sectors of each line
L1 HIT 18,432 = 3 × 6,144 redundant re-touches
L1 TOTAL 24,576 = 4 × 6,144 -> matches HW exactly

And since we are using WRITE_THROUGH, we call send_write_request() on every hit -> the number matches between L1 & L2. But that's where it gets messy.

If we look at ispass-2009-BFS,

Total_core_cache_stats_breakdown[GLOBAL_ACC_R][TOTAL_ACCESS] = 1176164
Total_core_cache_stats_breakdown[GLOBAL_ACC_W][TOTAL_ACCESS] = 346688
....
L2_cache_stats_breakdown[GLOBAL_ACC_R][TOTAL_ACCESS] = 548389
L2_cache_stats_breakdown[GLOBAL_ACC_W][TOTAL_ACCESS] = 346688

GLOBAL_ACC_W for L1 & L2 are the same, and it actually matches hw for both L1 & L2. Why does HW allow write-through on BFS but not STO? Does it have anything to do with the modeling (different policies)? Or is there something about STO workload?

For now it's out of scope, but I'll look into those 2 apps further

Related

https://forums.developer.nvidia.com/t/whats-the-meaning-of-performance-counter-lts-t-sectors-srcunit-ltcfabric/228015

For the beautiful dashboard: accel-sim/accel-sim-framework#549 (please include this PR I love this dashboard)

[JRPan]

Yea I assume you are on Ampere? The ltcfabric is not that important. On Hopper and Blackwell it's more important. But I think this counter is just busted lol. I'm using total L2 access lts__sectors minus the srcunit_tex one to get cross partition/chiplet access.

If you want this to be merged you need get this finished this week. We are planning on a new release soon. If you can get this ready I can try to merge this before the new release. Otherwise the memory subsystem changed a lot in the new release.

[JRPan]

Also I don't think RTX Ampere has partitioned L2? I might be wrong tho.

[AndrewVu23]

This week? I got this coach.

To be fair I put the ltcfabric ref just to refer to the kernel profiling (couldn't find tex_op_write* ref back then). I didn't plan to use that.

Also thank you for the L2 partition insight. Just checked and there is none lol.

[AndrewVu23]

Summary

Added a merge buffer between L1 -> L2 path which fixed some apps but failed to reproduce the expected results on the ubench. Scrapped since a new architectural change doesn't fit for a new official upcoming release.

Motivation / root cause

• Let start by declaring HWKeep: HWKeep = lts__t_sectors_srcunit_tex_op_write / l1tex__t_sectors_pipe_lsu_mem_global_op_st
• Since the policy is write through, HWKeep should approximately be 1.000
• However, this is not the case for some apps, especially in STO & l1_lat (in GPU_Microbenchmark) where the sim counter overcounts the L2 write access.
  • l1_lat: measures L1 hit latency. Single-threaded.
  • STO: each thread writes 4 consecutive bytes as four separate 1-byte stores -> 8 threads/sector -> per warp all 4 stores instruction hit the same 4 sectors.
• At first, I hypothesized that it might have to deal with the write policy, but it didn't seem to be the answer. backprop rules out a write-back L1: 92.8% of its store sectors hit in L1 and it still forwards 94.7% of them to L2.
• A better explanation would be there is a merger in between the L1 -> L2 path, where consecutive writes to the same sector are combined.
• To test this on the hardware level, I created another ubench l1_write_merge.cpp (can't upload .cu) where there are 2 loop nestings:
  • 1st loop: Fill a sector before moving on -> merge behavior
  • 2nd loop: Sweep each sector -> revisit later
  • The result: same 75% L1 hit, 4x different L2 write traffic (on 1st loop HWKeep = 0.25; 2nd loop HWKeep = 1)

Changes

• Added a merge buffer: L1 -> buffer -> FIFO -> L2. This buffer has #n entries, where the requests will only generate if the entry got evicted. For example:

  Request 0 -> sector 0 -> store in entry 0
  Request 1 -> sector 0 -> merge with request 0
  Request 2 -> sector != 0 -> a separate entry
  ....
  Request n -> sector n -> pushes entry 0 out of the table -> request goes down to FIFO before sending out to L2.
  

  • In the actual prototype, I send the first request to sector 0 right away to the FIFO and drop the requests to that same sector (same count btw) -> easier to implement & correct for L2 write sector (not functionally correct tho)
  • Why no merge in the FIFO? That's the question I initially had. The problem is that l1_lat is single-threaded, meaning any request to FIFO will be drained almost straightaway -> no merging behavior. But the hw run shows that merging happens.

• Added a new ubench

Evidence (before / after)

To my surprise, the merger actually fixes most of the issue. Across 32 apps (ispass-2009, rodinia 2.0-ft, GPU_Microbenchmark) with a 4-entry buffer:

metric	before	after
mean L2 write error	19.10%	2.33%
l1_lat	+299%	-0.1%
STO	+221%	-18.0%

19 apps stay bit-identical, and the worst collateral change anywhere is hotspot at -0.2%. So nothing regressed.

• However, the ubench refuses to fit, and that's a big issue:
  • The new ubench has each thread fill a 32 B sector with four 8 B stores, across CHUNKS sectors, in two loop orders:
    • fill-then-move: for c { for k } -> finish a sector, then move to the next -> row-major
    • sweep-then-revisit: for k { for c } -> write slot 0 of every sector, then slot 1... -> column-major
  • At CHUNKS = 1 there is only one row. Both orders write the same four cells of that one row, in the same address sequence. The only difference is that the compiler unrolls one into back-to-back stores and leaves loop-control instructions between the other's. Hardware still gives HWkeep 0.250 vs 1.000 on that pair.
  • So no buffer keyed on sector addresses can reproduce both; it sees an identical address stream and must return the same answer.
  • Measured, running the ubench through the simulator:

ordering	CHUNKS	HW	4 entries	32 entries	112 entries
fill-then-move	1	0.250	1.000	0.250	0.250
sweep-then-revisit	1	1.000	1.000	0.250	0.250

• Welp. The results make sense on the apps, but don't make sense on the ubench. At CHUNK=1 both loops behave the same -> buffer must give the same answer; instead it didn't -> hardware's decision must've depended on something that isn't in the address stream.

Conclusion

You know what? L2 counters might be busted 🥀😭. I think that hardware distinguishes two instruction streams that differ only by loop-control instructions between the same stores. The cache can't observe that, so this likely belongs in the LSU or access-generation path rather than in wr_hit_wt.

I do think that there's a solution to this. However, due to the potential architectural changes I'd have to implement + time constraint + the memory subsystem is going to change drastically in the new release, I'm gonna stop here with this merging implementation.

But apart from this new architectural fix, I would say the previous PRs are mergable (the PR above this where we use WRITE_ALLOCATED + other 3060 config PRs).

Also can you check out this parser issue: #138? I think it's pretty significant when running in PTX mode.

(also don't forget my beautiful dashboard too)

[JRPan]

please run the formatter.

[AndrewVu23]

my bad i just fixed it

[AndrewVu23]

You probably might want to check out accel-sim/accel-sim-framework@7f83680 too since it contains the LOCAL_ACC_W fix I mentioned in the 3rd comment.