Repository navigation
Conversation
bjf-frz
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Required human contributor answers
The author's public answers in #160, “Required human contributor answers” remain the intent for this follow-up; this PR improves that merged operator rather than introducing a new framework or changing its arithmetic contract.
1. What outcome are you trying to achieve?
Reuse the author's answer in #160, question 1: develop native CUDA operators with a complete, reusable build, integration and validation path. This follow-up improves the first operator's supported misaligned-input path.
2. What is the highest-leverage variable for that outcome?
Reuse #160, question 2: activation memory movement, kernel launches and efficient parallel computation while preserving the numerical contract. Here the changed variable is device latency for independently shifted contiguous inputs.
3. How will this work move that variable rather than create busywork?
Reuse #160, question 3: vectorized fused computation, preserved rounding, numerical checks and matched performance measurements. The concrete extension is safe aligned reads for misaligned inputs, with a measured scalar fallback where reconstruction costs more.
4. Who is the customer or user, and what do they actually need?
Reuse #160, question 4: the author developing native CUDA operators needs a complete implementation/build/validation path that can support subsequent operators.
PR necessity and bugfix evidence
The author explicitly requested a PR for this follow-up. It requires native load/dispatch changes; an existing configuration cannot enable vector processing of these inputs. This is a measured performance optimization, not a claim that the previous scalar implementation returned incorrect results.
Purpose
The existing SiLU-and-Mul operator uses scalar accesses when contiguous gate or up views have independently misaligned addresses. For a sufficiently large input this PR loads aligned 16-byte input vectors and reconstructs the requested logical vectors in registers. Input-view boundaries use scalar loads, with no out-of-view read. The aligned and misaligned paths share the vector compute and 16-byte output-store loop; output elements before the first aligned vector and after the last complete vector are scalar. No input packing or additional kernel launch is introduced.
Small misaligned inputs retain the original scalar path. The cutoff uses the input GPU's actual SM count and 16 bytes per vector:
That is 8 elements for BF16/FP16 and 4 for FP32. On the 148-SM B300, the thresholds are 48,496,640 and 24,248,320 elements, respectively. The 160 reference blocks/SM determine the cutoff; the launched grid is calculated separately and capped. Aligned inputs retain the direct vector path. This cutoff reflects the requested B300-based heuristic, not a demonstrated optimum on other hardware.
The operator preserves the existing arithmetic contract: compute SiLU in FP32, round it to the input dtype, then multiply by up. It also retains the existing fresh output, contiguous-input, stream, validation, and empty-input behavior.
Test Plan
vllm-rlt Version / Commit: base
989921ac1d3897c5c43e27ec8cddbc91fa953194, tested PR head21c706e646ae350ab20844ec24bd6dc63ed6e27b.pre-commit run --all-filespassed Ruff lint, Ruff formatting and CPU regression tests.torch.equal; signed zero is not distinguished). This is not an exhaustive numerical proof.fe71d62; Compute Sanitizer memcheck passed 24 cases with 0 errors. The subsequent source difference to the current head is the enum name and documentation only; current-head targeted tests are reported separately above.Test Result: performance
Measured on one idle B300, with no other compute process detected on that device during the run. The other GPUs were in use. These are operator-level measurements, not model throughput. Each point reports the median of 5 shuffled rounds of unprofiled CUDA Graph timing, 32 calls/graph, 30 graph replays per round (8 at the threshold); each function has 10 warmups. Clocks were not locked. Inputs use fixed seed 42, gate sampled from normal(std=3), up from normal(std=1). Gate/up are separate contiguous allocations; offsets in the table are in elements. The GPU results are limited to this hardware and workload.
The three code versions are:
21c706e, the restored output-prefix implementation.989921a, compiled from the same SiLU source in a separate operator namespace; misaligned inputs use scalar access.2b678ac739166bd0356883048673bbd1c34a3b0e, compiled directly from its originalcsrc/activation_kernels.cu. Its interface consumes one packed[gate | up]tensor, so it is shown in two timing modes:torch.cat((gate, up), dim=-1), output allocation, and the vLLM kernel are inside the timed graph. CUDA Graph timings measure device work; host-side allocation overhead is not included. This column reflects adapting separate inputs to the vLLM interface.Representative latency, in microseconds (lower is better):
All rows above are BF16. For the 120 common-shape measurements (BF16/FP16/FP32, 1–1024 rows, widths 5632/5633, aligned and independently shifted inputs), the PR/main latency ratio has a median of approximately 1.00; the misaligned inputs are below the cutoff and use the scalar path. For the 60 aligned cases, the observed main/PR speedup ranges from 0.960× to 1.015×. Small differences at these microsecond times are workload-specific; no blanket aligned-path speedup is claimed.
The cutoff is easier to see with fixed width 5632 and neighboring row counts. In each dtype the first row is just below the cutoff and the second is just above it:
Immediately above the cutoff, the PR is 1.80× faster for BF16, 1.87× for FP16, and 1.26× for FP32 than main on these shapes. Below it, the PR and main remain on the scalar path and have nearly identical latencies. For the above-cutoff examples, the PR's five-round ranges were 64.171–64.220 µs (BF16), 63.244–63.292 µs (FP16), and 53.564–53.592 µs (FP32); main's corresponding ranges were 115.800–115.840, 118.436–118.484, and 67.688–67.720 µs. These are repetitions of one benchmark run, not independent multi-run confidence intervals.
The vLLM packed column can be faster: at BF16
[8611, 5632]it takes 51.240 µs versus the PR's 64.196 µs, while adapting separate inputs withcattakes 218.988 µs. The packed layout is different, so that comparison does not isolate the cost of independent misalignment. For width 5633 the local vLLM dispatch falls back to scalar because the hidden dimension does not divide its vector width; the RLT operator handles the separately allocated tensors as flat contiguous arrays. Short ops-only and full profiles confirm the scalar and vector branches and show a separatecatkernel for the final column; profiled timings are not used as latency evidence.All 126 measured cases (CUDA Graph median, µs)
BF16
FP16
FP32
The benchmark reuses the repository's
benchmarks/kernels/silu_and_mul.pytiming and profiling helpers. The standalone comparison script, all five timing samples including eager timings, native build metadata, profiles, and GPU process-monitor log are retained in the local ignoredartifacts/silu-unified-20261103/directory; they are not committed. The displayed table includes all measured shapes.AI assistance was used for implementation, measurements and author self-review.