jfostier · Oct 27, 2026
@@ -0,0 +1,105 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-FileCopyrightText: Copyright (c) 2026 Jan Fostier
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# PR #613: reproducible block-FP8 rotation evidence
+
+This capsule addresses CodeRabbit's request for actual before/after collector timings. It contains two distinct experiments. The fresh comparison measures the exact latency passed to `log_perf` with the collector's normal settings. The historical experiment compares profiled kernels in fixed-length graphs with the same kernels inside a real model graph. These measurements answer different questions.
+
+The controlled kernel experiment supports the cache mechanism. The fresh comparison does **not** establish that the production collector's published latencies are more accurate. Most published row latencies decrease with the adaptive policy, so the earlier kernel-duration benefit must not be generalized to the production timing path without qualification.
+
+## Fresh direct comparison
+
+- Base: `cd6efc5e3cac59d7a6d63c32e32210077d6c7d58` (single-copy block-FP8).
+- PR: `64489fe6d79008fdffa474571404c74dc2597520` (adaptive count).
+- Hardware/runtime: NVIDIA H100 NVL, 60 MiB L2, vLLM 0.24.0, PyTorch 2.11.0+cu130. Full installed-package inventory, runtime source hashes, device details, clocks and power-limit observations are in `direct/*-environment.json` and `direct/*-gpu-after.json`.
+- Unmodified production timing arguments: three warmups, six timed replays, one repetition per graph, CUDA graphs required, power measurement disabled. Graph operation count follows the actual allocation count; it is not equalized between arms.
+- Three fresh processes per arm, seven collector calls per shape/start: **504 published latencies** across 12 shapes. Arm order alternates across starts; paired arms share seeded shape order and initial inputs. The protocol was saved before launch. All six processes exited zero.
+- Only the publication sink is replaced with JSONL. The real timing helper is wrapped to observe settings, allocations and results; no allocation count, kernel, warmup count or graph length is overridden. Each worker imports the entire collector package extracted from its exact Git revision. The timing helper and vLLM utility sources match across arms.
+- Statistics: median within a start, then median across three starts. All observations are retained. There are zero missing/invalid/duplicate observations and zero samples flagged by the predeclared rule (>15% from the within-start median). These three starts do not establish a population confidence interval.
+
+| M | N x K | Copies | Base (us) | PR (us) | Change |
+| ---: | --- | ---: | ---: | ---: | ---: |
+| 1 | 6144 x 2560 | 6 | 11.061 | 10.482 | -5.24% |
+| 1 | 2560 x 4096 | 6 | 11.771 | 11.024 | -6.34% |
+| 1 | 19456 x 2560 | 3 | 25.867 | 24.466 | -5.42% |
+| 1 | 2560 x 9728 | 6 | 19.381 | 18.664 | -3.70% |
+| 16 | 6144 x 2560 | 6 | 10.677 | 10.621 | -0.52% |
+| 16 | 2560 x 4096 | 6 | 11.643 | 11.078 | -4.85% |
+| 16 | 19456 x 2560 | 3 | 26.235 | 25.013 | -4.66% |
+| 16 | 2560 x 9728 | 6 | 19.504 | 18.935 | -2.92% |
+| 64 | 6144 x 2560 | 6 | 9.771 | 10.346 | +5.89% |
+| 64 | 19456 x 2560 | 3 | 25.552 | 24.578 | -3.81% |
+| 1 | 8448 x 8192 | 2 | 30.416 | 29.675 | -2.44% |
+| 16 | 16384 x 8192 | 1 | 50.128 | 50.197 | +0.14% |
+
+`analysis/direct.csv` and `analysis/summary.json` contain the full-precision results and per-start medians. `direct/*-measurements.jsonl` contains every observed published and loop latency, allocation count, distinct-storage count, graph status, seed and selected kernel label. Dispatch labels match between arms at every shape.
+
+These are complete collector row latencies, including activation quantization, GEMM and graph/event/submission effects. Changing the number of allocations also changes graph length. A decrease here can coexist with an increase in the isolated GEMM kernel duration. This experiment cannot attribute the difference uniquely to caching or graph overhead. The one-copy control above twice L2 changes by only about +0.14%; the M64 QKV point increases by about +5.89%. Both are retained alongside the decreases.
+
+## Historical model-kernel comparison
+
+`historical/` preserves the prior fixed-72-operation sweep: 201 isolated cases and six model cases across three independent process starts, with **207 raw compressed CUDA traces** in `cuda-traces.tar.gz`. The complete shape/copy grid includes the eight Qwen projection shapes, counts 1/2/3/4/6/8/12/24, and a 96 MiB negative-control matrix at counts 1/2/4.
+
+Reference: Qwen/Qwen3-4B-FP8 at immutable revision `96b30dc13593a244a5e59e84687309f53c375cfa`, TP=1, real checkpoint weights and prefilled KV. Model cases are batch/context 1/128 and 16/512. Captured logits exactly match the runtime reference at all six observations.
+
+The CPU analysis reads the raw traces, checks their hashes, verifies complete shape/copy coverage, requires 180 model and 360 isolated kernel observations per shape/start, and matches exact CUDA kernel names, grids and block dimensions. It selects this PR's capped copy counts from the sweep and recomputes the following values:
+
+| Measurement | Single-copy MAPE | PR-count MAPE | Shapes improved |
+| --- | ---: | ---: | ---: |
+| GEMM kernel duration | 9.067856% | 1.518637% | 8/8 |
+| Quantization plus GEMM active interval union | 8.023906% | 2.275138% | 8/8 |
+
+MAPE is `mean(abs(isolated / model - 1)) * 100`, with equal weight for each of the eight shape points. The median is taken within each independent start, then across starts. No kernel or timing observation is excluded. Full per-shape values and per-start medians are regenerated into `analysis/historical.csv` and `analysis/summary.json`.
+
+This reproduces the earlier **controlled kernel** claim; it is not new model validation of the shorter production graphs. No L2 hit-rate counters were collected. The two-L2 target remains a heuristic, and tiny matrices can be limited by the six-copy cap.
+
+The earlier uncapped-policy full-forward experiment remains negative: decode MAPE goes from 5.823069% to 10.686789%. `historical/full-forward-comparison.json` contains the recorded comparison points, and the analysis checks that arithmetic. These are historical derived points, not a new forward run or a result for the capped PR policy. No improved overall prediction accuracy is claimed.
+
+## Reproduce
+
+Run from the repository root. CPU reanalysis needs only the Python standard library and makes no GitHub or GPU calls:
+
+```bash
+capsule=benchmarks/evidence/vllm-fp8-block-rotation-pr613-20261027
+python3 "$capsule/analyze.py" --output /tmp/pr613-reanalysis
+cd "$capsule"
+sha256sum -c SHA256SUMS
+```
+
+The analyzer fails on missing, duplicated or nonfinite/nonpositive direct measurements, unexpected allocation counts, broken normalization, runtime/source drift, invalid model geometry/parity, incomplete raw trace coverage or kernel-identity mismatches. Outlier flags are diagnostic; observations are never dropped. The integrity checks were also exercised with deliberate missing, duplicate, negative, wrong-kernel and outlier mutations; see `analysis/validation.json`.
+
+For a fresh GPU comparison, use Python 3.12 with the pinned runtime above and a CUDA development toolkit available to DeepGEMM. Both immutable source revisions must be available in the local Git object database (the original PR head is intentionally pinned even if the PR is amended later). If needed, fetch it with `git fetch https://github.com/jfostier/aisimulate.git 64489fe6d79008fdffa474571404c74dc2597520`. The runner refuses a non-idle GPU or an existing output directory.
+
+The exact campaign command on this host was:
+
+```bash
+module load CUDA
+export PATH=/tmp/aisimulate-forward-ab/venv/bin:$PATH
+/tmp/aisimulate-forward-ab/venv/bin/python   benchmarks/evidence/vllm-fp8-block-rotation-pr613-20261027/run.py   --output benchmarks/evidence/vllm-fp8-block-rotation-pr613-20261027/direct
+```
+
+To reproduce without overwriting the archived observations, use a new `--output`, such as `/tmp/pr613-fresh`, then analyze it with `analyze.py --direct /tmp/pr613-fresh --output /tmp/pr613-fresh-analysis`. Successful collection checks include graph replay, distinct weight storage, six actual timed runs and the unmodified published-latency normalization.
+
+`direct/runner-source.tar.gz` preserves the exact executed runner bytes, matching the SHA-256 recorded by all workers. The top-level executable adds the repository-required NVIDIA SPDX copyright line; the analyzer verifies that its Python AST is identical to the executed snapshot. `historical/collection-sources.tar.gz` preserves the prior collection scripts and pre-launch protocol byte-for-byte; their frozen hashes are checked during analysis. The historical scripts retain their original `/tmp/aisimulate-rotation-case` paths and require the pinned model checkpoint there; they are provenance inputs, not part of the fresh direct comparison. No framework source is vendored or modified.
+
+## Other validation and limits
+
+`memory-checks/` retains the earlier real-collector smoke logs and original harness: six-copy failure versus one-copy success under a 26 GiB allocator quota for a 1 GiB matrix, a 66 MiB/two-copy case, and small six-copy FlashInfer/DeepGEMM cases. The six-copy failure is a guard against an unconditional-rotation variant, not a failure of the single-copy base collector. These checks are separate from the fresh paired campaign.
+
+The evidence covers H100 NVL/vLLM 0.24.0. Blackwell, other runtime versions and NVFP4 rotation remain unverified. Unit tests use synthetic timing to verify control flow, whereas this capsule contains GPU measurements. No packaged profiles, model coefficients or production collector code are changed by this evidence update.
+
+## Local checks
+
+The existing 22 focused collector tests pass on the unchanged production source in `64489fe6`. The application-wide conftest is excluded because this collector environment does not have the AISimulate application package installed. This is not a full-suite or hosted-CI result.
+
+```bash
+/tmp/aisimulate-forward-ab/venv/bin/python -m pytest -c python/aisimulate/pytest.ini --confcutdir=python/aisimulate/tests/unit/collector python/aisimulate/tests/unit/collector/test_vllm_gemm_graph_timing.py -m unit
+/tmp/aisimulate-forward-ab/venv/bin/python -m ruff check --config python/aisimulate/pyproject.toml benchmarks/evidence/vllm-fp8-block-rotation-pr613-20261027/{run,analyze,validate}.py python/aisimulate/collector/vllm/collect_gemm.py python/aisimulate/tests/unit/collector/test_vllm_gemm_graph_timing.py
+/tmp/aisimulate-forward-ab/venv/bin/python -m ruff format --check --config python/aisimulate/pyproject.toml benchmarks/evidence/vllm-fp8-block-rotation-pr613-20261027/{run,analyze,validate}.py python/aisimulate/collector/vllm/collect_gemm.py python/aisimulate/tests/unit/collector/test_vllm_gemm_graph_timing.py
+python3 benchmarks/evidence/vllm-fp8-block-rotation-pr613-20261027/validate.py
+```
+
+Results: 22 tests passed; Ruff lint/format passed on five files; six evidence-reader validation scenarios passed. The runner's and analyzer's numerical behavior were not changed to obtain a preferred benchmark direction. SHA256SUMS covers all packaged source, raw measurements, logs, archived traces, provenance and computed reports; it excludes itself and Python bytecode caches.
@@ -0,0 +1,58 @@
+0c020e06d817ff60921f6c8e7bc6abe8fb29b874996fdea00b7b1d933a497971  README.md
+85f32a15ab9676042602c576017fce9a99482fdaed4389be2166dd3b2cfc033b  analysis/direct.csv
+5fdcc94e6b9d6fba4b9878d2f3709292e81bf7197ca44468cbc79a5834b0cf83  analysis/historical.csv
+f24b69d6343745b26440c290b6443b4fd95c3181db32b9e0f253e3dfe92360e2  analysis/summary.json
+ec225f5cdb6b6cd7a9548f927d4407d342aec3a79bc0b200a3e52cb904988f7c  analysis/validation.json
+39aac89898c50cba669466f727e427fc178453d69288f0d57d71c71530e8f505  analyze.py
+87456ce12cae1196db32df2a7aea06a0528ca334d0a08bb04deb6bc9d6903876  direct/base-0-environment.json
+d8a31aacda6a0fc891fc1b5b52d4d1de06c5b978e46e60f95070cb4f8f0a9b5a  direct/base-0-gpu-after.json
+94404bbfccffd32de030b4f3106e7f989a49a587e72ed3ea63ea7d544f1d44b6  direct/base-0-measurements.jsonl
+d0f71b76c5d79cadfd89694bb2990e3d5a792c7706c9bdd7a20492b3bc97db31  direct/base-0.log
+7b910c46c398beddfd4814d3e96981960882f7f910595f1664947d1793e88219  direct/base-1-environment.json
+b142f14e80b97ad9b73c9f06c5ec64dc450e1e798d515f26e65f76882cbbc8ba  direct/base-1-gpu-after.json
+db008ea9282da0bdd7db06e54be75470db36bea77ea0ccc8136259b130cb9620  direct/base-1-measurements.jsonl
+1b40ecfbb5fd68b64e40b587256c399e0f44a8743ab1ee0da853457cc5048405  direct/base-1.log
+ecd688a083b086d7857f6e005d303ca59c71dbfa913b028aca2453da65b6a8cc  direct/base-2-environment.json
+5644632cf0f2ecfc10db0ecd2090925f7cf36168788c4345cbf7b459bfa928d2  direct/base-2-gpu-after.json
+3be50d48ede2c0119fb75537e6eca4c1ca9d7cf0578f7aac829ddbcd46d49933  direct/base-2-measurements.jsonl
+9280cab17af4c6f8bbff5f52607f5b0a53e91a0964b7e9bd029dd77c12c60d76  direct/base-2.log
+140ff1f2159fde18f92cf146d3e5895dffdd3f0832d279be811d31391bf5b995  direct/head-0-environment.json
+b660e8412c3ddcb460f5a254290e501e12d6736f2ce041045ffceabec5479e17  direct/head-0-gpu-after.json
+ae226caf189f4587c5141fbae01fb10bc97addbf3c7d32b5be0ba2e46313dd35  direct/head-0-measurements.jsonl
+7a2c9ece8f0f7f82951febab746af7e879f83bf5512fd3f188c41e247dc68b45  direct/head-0.log
+4ce84408681c78c08dd16b2200d5e27c0fd55d216820c64a0ed44ebe07219cbd  direct/head-1-environment.json
+96254119e95803f2ff4c34ce598afb037fb19dd0a8a7608c967320287799be76  direct/head-1-gpu-after.json
+48fc29595440a271dbb51be370d797a1b7321d7ce9f24bf6d9c2b1bc007e1f1b  direct/head-1-measurements.jsonl
+cd4793977cd2bf0a6fb2627de5d05cdd372acd1dec839439cf023b9319d56f45  direct/head-1.log
+f0445c577ae0bbe1b85141a02809235d62f1e0c33b453d1c5f06f36e8ae39d39  direct/head-2-environment.json
+01bf0aec06ec6fcee224031e36e5abd62d28793ae6e141add52c650003415b83  direct/head-2-gpu-after.json
+40a5561991b8245a1b53d59c8bac1e3375d3de4977695ea5fb0a6a2b6c289719  direct/head-2-measurements.jsonl
+136ef11a79ad0dcb707fd9e28e83366c768759d7fd64a0964b99383325f81137  direct/head-2.log
+f24b770162959e1c1794b2129b75f7fdb36ef8fffc1b21af8668b87b552502d6  direct/invocation.json
+ab3278352336125638a606036b400df73180cf60f3d3c176eb08630e6a5e1f84  direct/jobs.json
+7121239a74b76c01da99e5bdd7834e0489876c7105694e0b11f20d616d011c1e  direct/protocol.json
+97e314ab5451d46a7e2637acc1070a6bbf3aa8455ee0c81f3d3352cde31a21f1  direct/runner-source.tar.gz
+e0eb9d9d7beeccb117bd230400f675d25255c6ca5af14b02c658e4d5486a2520  direct/system.json
+d3ca7c0ebd49af8735be4903f2b552bcd47635c1bbb777788fd547635990e698  historical/analysis-contract.json
+334ed5525629de69177233f68254e1483eaec2fb8015eb7c019489e4f1bfb959  historical/collection-sources.tar.gz
+d892c85e7d10f760bc6d689c687055fd29c3c9d6d7ae54bff502e0dd2e479639  historical/cuda-traces.tar.gz
+b261c7ed81a65bc62580fb42586cf0415943a5fe1e2dacc0608c5ba341cca8cf  historical/environment.json
+a6e3608e9caa7d106c2a418997e36db6cc2ba1f53a82764ab5c16ed6293137fb  historical/forward-0.jsonl
+a03d9c2af569cb4a9573b577026452ab5bde3e0f9661454072295d1113dcd437  historical/forward-1.jsonl
+df3e6a9bc7fe2c3751ba8dfe612ba82255b694d5c1b1b29eb07dd58c027555e7  historical/forward-2.jsonl
+ff3e8bd5cfbb355883e00541564586dd79c9e8d89787e7bb60674d552c6f510c  historical/freeze.json
+efd93c64609a70e492c92697bb616a82ef769d06b96e5729d85717e27fa6906e  historical/full-forward-comparison.json
+ccb029a07a733724bcff03d736ddcee77744b3fcdf9b0be4cfebae195677f5e4  historical/ops-0.jsonl
+4737eb2df46225514866e85a5e88104901877d0b062bb92f1377b0d253688a3d  historical/ops-1.jsonl
+542501337d35718b6c7c39d6c5e26c0b5eca7072e8ce62fdcdb6cfc3e8bc2870  historical/ops-2.jsonl
+b40f1fe454a37597bb13609fa412020a6f20364ab9ad214a32c25e9dff4d776d  historical/protocol.json
+511b21e9c142cf545b592c52f219f5202e6746093f6699af4facf3fd314d568a  historical/qualification.json
+b6760bb1497756b4f34a1cd802174fc789fac94d8de136371e9b5889d1dc570a  memory-checks/after-one-copy.log
+a2990ef04e8869bb99ffc77f2a9a0d0b69213f7f30844ac3c0bffcb2398263ee  memory-checks/before-six-copy.log
+d210a45e4cdbfd204fc913878fac0c843053682bca3fcf12cb7f33f54c592be7  memory-checks/near-l2-two-copy.log
+ae4cbb20a0e1b34641a9bec102f63030eada7161d7c2a7741d0b5405aa37c78a  memory-checks/runner-source.tar.gz
+8a73366eb6e1cf57e9f1ea417fd98018dd74e2b4bf29b64643c6de65e0ad34d9  memory-checks/small-deepgemm.log
+05e0a22fae5a6961c53862548c34f98f930743d94b230d140c30fd167785ba72  memory-checks/small-flashinfer.log
+c81d7a3196f726f0f09f59150a13c3b4fd3749c674e2ec152edd4ee6b7449a30  protocol.json
+622becf092c731717852d0ae4e022f79f5924296d7c967f3c2df394f1869517f  run.py
+8609a58a00fb55c4c82445fa97ae582a006678d6f27a8e5fd2dc5cff62596a18  validate.py
@@ -0,0 +1,13 @@
+m,n,k,base_start_medians_ms,base_median_ms,base_copies,head_start_medians_ms,head_median_ms,head_copies,kernel_source,change_pct
+1,6144,2560,"[0.011034666250149408, 0.011183999478816986, 0.011061333119869232]",0.011061333119869232,1,"[0.010497777826256223, 0.010458666417333813, 0.010481778118345471]",0.010481778118345471,6,FlashInferFp8BlockScaledMMKernel,-5.239467930702846
+1,2560,4096,"[0.011770666887362799, 0.011781333635250727, 0.011674666156371435]",0.011770666887362799,1,"[0.010966222319338055, 0.011067555182509951, 0.011023999916182624]",0.011023999916182624,6,FlashInferFp8BlockScaledMMKernel,-6.343455118773345
+1,19456,2560,"[0.025957333544890087, 0.02586666742960612, 0.025637333591779072]",0.02586666742960612,1,"[0.024504888388845656, 0.02446577780776554, 0.024453333682484094]",0.02446577780776554,3,FlashInferFp8BlockScaledMMKernel,-5.415810233973817
+1,2560,9728,"[0.01950399950146675, 0.019381333142518997, 0.019354666272799175]",0.019381333142518997,1,"[0.018637332651350234, 0.018776888648668926, 0.018664000762833487]",0.018664000762833487,6,FlashInferFp8BlockScaledMMKernel,-3.701150867232228
+16,6144,2560,"[0.010677333921194077, 0.010656000425418219, 0.01073066641887029]",0.010677333921194077,1,"[0.010603555374675326, 0.010631110933091907, 0.010621333287821876]",0.010621333287821876,6,FlashInferFp8BlockScaledMMKernel,-0.5244814275316534
+16,2560,4096,"[0.011642667154471079, 0.011711999773979187, 0.011594666788975397]",0.011642667154471079,1,"[0.011078221930397881, 0.011061333119869232, 0.011095110740926532]",0.011078221930397881,6,FlashInferFp8BlockScaledMMKernel,-4.84807490057324
+16,19456,2560,"[0.026234666506449383, 0.026373334228992462, 0.02611733227968216]",0.026234666506449383,1,"[0.025052444802390203, 0.02499377727508545, 0.025013332565625507]",0.025013332565625507,3,FlashInferFp8BlockScaledMMKernel,-4.6554201118723215
+16,2560,9728,"[0.019498666127522785, 0.01950399950146675, 0.01978133370478948]",0.01950399950146675,1,"[0.018935110833909776, 0.018965333700180054, 0.018929777873886954]",0.018935110833909776,6,FlashInferFp8BlockScaledMMKernel,-2.916779543160841
+64,6144,2560,"[0.009712000067035357, 0.009861333295702934, 0.009770666559537252]",0.009770666559537252,1,"[0.01034577770365609, 0.010328888893127441, 0.01038044442733129]",0.01034577770365609,6,DeepGemmFp8BlockScaledMMKernel,5.886099383439358
+64,19456,2560,"[0.02555199960867564, 0.025600001215934753, 0.025424001117547352]",0.02555199960867564,1,"[0.02457777824666765, 0.024458666642506916, 0.024638222323523626]",0.02457777824666765,3,DeepGemmFp8BlockScaledMMKernel,-3.8127010681278106
+1,8448,8192,"[0.030415999392668407, 0.03046400099992752, 0.03019733230272929]",0.030415999392668407,1,"[0.029674666623274486, 0.029487999776999157, 0.029853334029515583]",0.029674666623274486,2,FlashInferFp8BlockScaledMMKernel,-2.4373118891257484
+16,16384,8192,"[0.050154666105906166, 0.05012799799442291, 0.05004266897837321]",0.05012799799442291,1,"[0.05048533280690511, 0.050111999114354454, 0.050197333097457886]",0.050197333097457886,1,FlashInferFp8BlockScaledMMKernel,0.13831612234480595