GLM DSA sparse kernel exceeds sm_120 Triton shared-memory limit after merging #408 #2

Environment: RTX 5080 16 GB (sm_120), AMD Ryzen 9 9950X, 123 GB RAM, Ubuntu 24.04 (kernel 7.1.9-xanmod), driver 610.57.04, CUDA 13.3, torch 2.11.0+cu130, triton 3.6.0, FreeToken 0.1.2 editable. Checkpoint: nvidia/Qwen3.8-Flash-Next-NVFP4.

Branch: 5080deploy @ 150c95d = main e0886cc + upstream FlashML-org#408 (nvfp4 KV, head 1c7bb42).

With the server stopped, pytest tests/ -m "not slow" gives 1879 passed, 191 skipped, 2 failed:

• tests/models/test_glm_dsa.py::test_sparse_kernel_equals_dense_when_all_selected
• tests/models/test_glm_dsa.py::test_splitk_matches_single_program

Both fail with:

triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 102400, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.

tests/models/test_glm_dsa.py on plain main e0886cc, same card: 7 passed. So the change comes from FlashML-org#408's edits to python/freetoken/kernel/triton/glm_dsa_sparse.py (nvfp4/fp8 KV branches added to the kernel). The kernel no longer fits sm_120's per-kernel shared-memory limit at the tests' block sizes.

It does not affect Qwen3.8-Flash-Next serving (QSA backend). GLM-5.x DSA serving on this card was not tried.

Related

• Upstream: feat(kvcache): add nvfp4 kv quantization FlashML-org/FreeToken#408 (introduces the kernel/triton/glm_dsa_sparse.py changes, head 1c7bb429); feat(models): support deepseek v4.1 with native fp8-fp4 kv storage FlashML-org/FreeToken#460 carries the same kernel change.
• Fork PRs: 5080deploy 1/9: merge upstream #408, nvfp4 KV cache quantization #9 (merge of feat(kvcache): add nvfp4 kv quantization FlashML-org/FreeToken#408 into 5080deploy), 5080deploy 9/9: merge upstream #389 (+#385), load-time FP8 dense projections #17 (top of the stack, where the full suite still shows these two failures).