Repository navigation
Conversation
…LLM 1.3.0rc29 Main already ships the 2026-10 campaign tables for these runtimes (ai-dynamo#517), but its collector still defaults to vLLM 0.24.0 / SGLang 0.5.14 / TRT-LLM 1.3.0rc20. This lands the collector that produced those tables. It is the collector half of ai-dynamo#228; the op-probe harness follows separately. - Runtime pins and audits: framework_manifest defaults move to vLLM 0.30.0, SGLang 0.5.21 and TRT-LLM 1.3.0rc29. Per-op __compat__ spans keep main's GLM-5.3-Flash pins (vLLM 0.31.0, SGLang 0.5.20). The SGLang gemm fork for 0.5.20 applies to exactly 0.5.20, so the 0.5.21 default keeps its own path. - Op ports to the new runtimes (sglang runtime_compat, vLLM attention metadata, TRT-LLM GDN / DSV4 / MoE), new kernel_source label translations and the regenerated op_backend_facts registry. - vLLM and SGLang MoE run FlashInfer's autotuner before timing and log the chosen tactic and fallback reasons per case (the fix behind the SM100 NVFP4 MoE data). - Collection cost: vLLM MLA/DSA modules sweep a whole seq_len or isl x prefix series per build; DSA modules are measured at the per-rank heads their TP shards produce; graph-timed modules run fewer eager warmups; TRT-LLM reuses GEMM modules and attention layers within a worker and recycles the MoE worker every N cases; SGLang MoE builds its layer once per token sweep; per-phase timing lines; the stall watchdog counts a worker heartbeat as progress. - The retired DSV4 hca_attn / csa_attn sidecar ops are removed end to end (collector, Rust enum, data tables, reuse manifest). - Generator: a TRT-LLM 1.3.0rc29 template, the SGLang 0.5.20 template update, hardware and model facts from the op probes, and token budgets rounded up to a multiple of 64 on every backend (the Dynamo 1.2.0 golden contract now expects 4096 / 3584). - Nine bundled model configs (two with hf_quant_config) and their THIRD_PARTY_NOTICES entries. - benchmarks/evidence/collector/op-probe-2026-10: the probe findings and results the collector and generator comments cite as provenance. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
The nine bundled configs (eleven files) were added on 2026-09-25 so the op-probe harness could render its model roster offline; no collector case, collection run or shipped table uses them. They move to the harness PR together with what exists only for them: their DefaultHFModels entries, the Qwen3_5ForCausalLM -> QWEN35 mapping and flat-config handling, the THIRD_PARTY_NOTICES sections (both copies), the NeoHorse SDK test and the merge-time entries in the packaged-config inventory tests. The generator facts that cite these checkpoints stay: they match by model path or quantization layout and load no bundled config. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
… move to 0.31.0
The owner policy of 2026-09-20 (version history lives in git, not in
side-by-side files or version ranges) was recorded only in a commit message
and four vLLM file headers, so later ports widened __compat__ ranges again.
It is now a binding rule ("Version contract" in
.claude/rules/collector/layer_permissions.md) and a test: every stock vLLM,
SGLang and TRT-LLM registry module declares exactly `==` the runtime the
manifest resolves for its op. aic-auto-collect, aic-collector-op-development
and docs/perf-model/collector/upgrade.md follow the rule.
- vLLM: gemm, moe, kda and mhc move to 0.31.0 through manifest family pins
(owner decision): GLM-5.3-Flash already collects them there, so the
families upgrade instead of carrying 0.30.0 and 0.31.0 side by side.
Kimi-K3 KDA moves with its family. Every other vLLM CUDA collector pins
0.30.0. DeepSeek-V4 mHC is not admitted on 0.31.0: the architecture-free
mhc_module_perf view its consumer reads would merge GLM-5.3-Flash rows at
(hc_mult 4, hidden 4096); its 0.24.0-0.30.0 tables stay.
- SGLang pins 0.5.21 and TRT-LLM 1.3.0rc29 in every collector.
- Qwen3.8-2.4T model pins (vLLM 0.27.1, SGLang 0.5.17) fold into the
defaults per the pin-consolidation rule; the new tables already carry the
Qwen3.8 MoE and GDN shapes.
- The SGLang 0.5.20 GLM-5.3-Flash registry lane retires (owner decision):
model pin, KDA/mHC architecture gates and the collect_gemm_0520 fork.
Its tables stay under 0.5.20; re-collect from main c540ff9. The
standalone GLM attention runner keeps its own exact pins.
- SGLang KDA raises KdaRuntimeNotAuditedError (not KeyError) for an
architecture with no audited release.
Older collectors and data are untouched: data stays under its version key,
code is `git log -- <file>` away.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
Every bundled MLA/DSA FP8 checkpoint is block-scaled (fp8_block): the DeepSeek-V3/R1/V3.2/V4, Kimi-K2 and GLM-5.x FP8 configs all carry weight_block_size, and the NVIDIA ones are ModelOpt NVFP4. The only consumer of the per-tensor `fp8` rows in the vLLM MLA/DSA module tables was the prediction gate's forced per-tensor override (removed separately), so the two `gemm_type: fp8` combos leave mla_module.yaml, as the fictitious MoE fp8 cases did in AIC-1782 Task V2. collect_mla_module.py drops its per-tensor Fp8Config branch and the `--gemm-type fp8` choice. Shipped tables keep their rows. Collection harness shard plans that filter mla/dsa module shards by gemm type need their owner to drop the fp8 filter; this PR does not touch the harness. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
…eport to a data PR The evidence bundle keeps the 20 findings collector and generator code cite and, in each results file, _meta plus the models the citing comments name (5,275 -> ~1,080 lines); the complete archive lands with the op-probe harness. benchmarks/reports/perf-data/perf-data-reuse-analysis.md goes back to main's copy: main never regenerated it after the ai-dynamo#517 data, so a rebuild here carried ~2,200 lines of ai-dynamo#517 churn for a 2-line change of this PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
`pytest -m unit` (used for local runs so far) deselects the unmarked generator tests; CI runs tests/unit and tests/golden unfiltered. - extra_engine_args.1.3.0rc29.yaml.j2 was copied from rc23 before ai-dynamo#626 and missed its rule: cache_transceiver_config renders when a backend is set, max_tokens_in_buffer only when given (test_sweeper_request, rc29 trace candidate). - The Dynamo 1.2.0 Slurm goldens follow the 64-aligned token budgets, as the release-1.2.0 contract already does: vLLM 0.20.1 2140 -> 2176, SGLang 0.5.11 1628 -> 1664, TRT-LLM 1.3.0rc14 decode max_num_tokens 32 -> 64. - test_collect_mla_module_native_quant's fake helper lacked PhaseClock and render_phase_totals, which sglang/collect_mla_module.py now imports; it passed only when another test had loaded the module first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
Removes about 50 test functions (~1,000 lines) that would not catch a wrong row or a wrong kernel: duplicates of the one-release-per-file check (per-file __compat__ / admitted-release tests), tombstone tests for retired ops, log/print-format pins no tool parses, positive controls, exact code-string and docstring needles, and the test of dead code. Same-branch parametrizations are trimmed and near-duplicates merged. - collector/vllm/utils.kv_block_size had no callers (superseded by framework_kv_block_size); it goes with its test. Gap left open: framework_kv_block_size itself has no unit test. - The strict pin test names its one exception, collect_dsv41_module's pre-existing source-build pin from ai-dynamo#160 (never declared in the manifest), instead of accepting every `@` pin. - The no-eager-fallback test flags any allow_graph_fail other than literal False (a wrapper forwarding its own keyword-only False default is allowed), replacing five per-site call-count pins. - The mHC audited-release overlap guard also checks the live 0.5.21. - test_vllm_attn_kv_cache_layout needs real torch: registered as an optional collection skip in .github/application-test-inventory.json with the same reason text as the other three torch-only collector tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
Collector runtime: - The post-failure context probe skips XPU devices; every XPU failure used to recycle its worker. - WorkerStallTimeout records carry the case's task_params and group, so --cases-from can replay a stalled case. - Resume finalization counts only cases that passed inside the open event (_done & _attempted), not the cumulative ledger. - The DSV4 subprocess wait beats the worker heartbeat every 60 s; the 1800 s stall watchdog would otherwise fail a healthy hour-long sweep. - collect.py refuses AIS_SM/AIC_SM: it is the case-inventory stand-in for a missing GPU, not a collection setting. SGLang: - The single-rank parallel override is back on run_attention_torch; it had slid onto default_attention_backend. No measured change: the offline runtime publishes TP/CP/DP 1. - The FlashInfer GDN prefill lane gathers the state rows inside the timed call, as FlashInferGDNKernel.extend does (gdn_flashinfer.py:523-541). - KDA decode drops the source probe and the packed lane: at the pinned 0.5.21 a lower-bound layer always takes the plain lane (kda_backend.py:737-742), and Kimi-K3 always has one. - mHC and MLA modules reserve a free NCCL port (free_local_port) instead of a random guess. vLLM: - AIS_KDA_DECODE_PATHS rejects unknown or unrunnable selections instead of finishing "ok" with zero rows. - The AIS_DSA_LEGACY_MODULE A/B switch (a 0.29 comparison) and the AIS_KV_BLOCK_SIZE override are removed; neither may change rows unseen. - gpt-oss MXFP4 imports GptOssMxfp4Config unconditionally (mxfp4.py:110 @0.31.0). - The fp8_block n/k % 16 guard applies from SM90, where the cutlass block tier exists; SM89 selects Marlin first. Re-cited at 0.31.0. Generator and tools: - trtllm renders load the bundled quantization only for MoE models whose hardware moe_backend is quantization-keyed. - collect_campaign counts a failed shard whose GPU slot is reused at once. - test_error_classification imports collect without a global torch mock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
The op-probe harness (collector/opharness) boots the real serving framework per model, records which kernels and backends it deploys, and grades the collector's op paths against those records. It produced the serving evidence behind the vLLM 0.30.0, SGLang 0.5.21 and TensorRT-LLM 1.3.0rc29 collector upgrade. Contents: the components (probe driver, in-container probes, dummy checkpoints, path diff, lane-guard grading, case inventory, executor smoke, workflow checker), per-SM kernel taxonomies, the gate capture scripts for the current pins, the upgrade/onboarding/new-op workflows, the probe targets, the agent skill, and 134 unit tests. Intermediate results stay out of git (owner decision 2026-10-30). The harness writes them to its git-ignored results/ directory, and evidence_bundle.py now packs that tree into each campaign's evidence bundle next to the raw probes. One lane-guard test reads the committed op-probe evidence instead of local results. Left out: results for the retired SGLang 0.5.16 / TRT-LLM rc23 pins and their verdict scripts, a one-off migration script, the stale TODO notes, and the bundled probe-roster model configs (the harness fetches configs from the Hub; the bundled copies were only an offline fallback). Alignment with the collector upgrade: targets.yaml and the upgrade workflow note that framework_manifest.yaml pins the vLLM gemm/moe/kda/mhc families to 0.31.0 ahead of the probed 0.30.0; the skill names framework_kv_block_size. All files carry SPDX headers and pass ruff check / ruff format. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
The collector upgrade committed a 20-finding excerpt that its code cites and promised the complete archive with the harness. findings.yaml now holds all 100 findings of the 2026-09/10 campaign (the 20 cited entries are unchanged). The README says where the uncommitted intermediate outputs that finding statements name are preserved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
|
…s cover every version Owner decisions (2026-10-30) on the CodeRabbit review of the guards: - layer_permissions.md: an evidence-backed FIXME(kernel-limit) guard is legal when it raises at run time, in the worker, before the expensive step; cites the framework source line or a reproducible failing run whose passing rows bound the envelope; and is re-verified on every version bump. Plan-time drops for kernel limits stay forbidden. - vLLM and TRT-LLM MLA modules: DSA per-rank heads 1/2/4 on SM100/103 are no longer dropped from the plan; run_mla_module_worker raises before the module is built (FlashInfer trtllm_fmha_kernel_launcher.cu:318 @0.6.18; TRT-LLM StoreSmemP.h:125 @1.3.0rc29). The cases show in every failure log again at no GPU cost. - TRT-LLM MoE int4_wo cites moe_resolution.py:665 @1.3.0rc29 and drops the version range (the file pins rc29); TRT-LLM SM90 FP8-KV decode names its evidence shard. - Governed exceptions: `versions: all` covers every version of the backend; an exception follows the deviation, and `expires` bounds it. The compute_scale, fp8_static and trtllm mHC entries use it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
…eat/opharness Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
…s skill ai-dynamo#649 (2c59403) removed AIS_KV_BLOCK_SIZE and AIS_DSA_LEGACY_MODULE from the vLLM collectors. The skill's environment table no longer lists them; AIS_DEBUG_IDXMETA stays (collect_mla_module still reads it). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
…f expiring An exception now follows the deviation (versions: all), so a calendar date bounded nothing that a reviewer or CI ever checked. Each standards exception carries the release it was last verified on; every bump of the covered tables' collector pin re-checks the deviation by source or run, as for a FIXME(kernel-limit), and either moves verified_on or deletes the entry. An entry behind the pin is stale. Re-verified on today's pins: - vLLM compute_scale: 0.30.0, the op's own pin (opharness gate 2026-09-27). - vLLM fp8_static: 0.31.0. FlashInfer still precedes Cutlass (kernels/linear/__init__.py:422-423) and still takes CC>=100 per-tensor scales (scaled_mm/flashinfer.py:50-62). - TRT-LLM mHC: 1.3.0rc29. DeepSeek-V4 serving still defaults to fused_hc (models/modeling_deepseekv4.py:292-297) while the collector times pre/post_mapping. Evidence waivers (design section 9) keep their expiry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
…eat/opharness Brings in the standards-exceptions rule change: entries carry verified_on (re-checked on every pin bump) instead of an expires date. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
…ises at rc29 The rule this branch wrote (plan-time kernel-limit drops forbidden; a guard raises in the worker with FIXME(kernel-limit) and a citation) did not yet hold in its own code. - SGLang attention: the flashinfer fp8_context_fmha case is no longer dropped at plan time. Both FP8-prefill refusals (flashinfer: no FP8 prefill compute; fa3/trtllm_mha: no BF16 prefill on an FP8 cache) run before the model runner and tensors are built, cited at 0.5.21 (flashinfer_backend.py:1833-1834, flashattention_backend.py:1322-1332, trtllm_mha_backend.py:1600-1609). - vLLM MSA: the decode-scorer limit message starts with FIXME(kernel-limit), so it classifies as kernel_limit instead of unexpected. - failure_handling.md describes both classifications. - Pre-existing limit raises carry the prefix where the cited source still holds at the pinned release, re-cited there: - TRT-LLM FlashInferAttention FP8 FMHA: attention/backends/flashinfer.py:2740-2746 @1.3.0rc29. - TRT-LLM MLA below SM90: attentionOp.cpp:3234/:3240 @1.3.0rc29 (stock and module collectors). - WideEP fp8_block on non-SM90: deep_gemm/compiler.cuh unchanged rc20 -> rc29. - Not prefixed: the TRT-LLM DSA module's SM120 refusal. rc29 pins DeepGEMM 8b1392b, whose attention.hpp dispatches SM120 for both indexer phases (sm120_mqa_logits :205-212, sm120_paged_mqa_logits :414-418). A FIXME(re-verify) note records it; an SM120 rc29 run decides it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
…eat/opharness Brings in queued-and-classified kernel limits: the sglang flashinfer fp8_context_fmha cases are planned again and raise FIXME(kernel-limit) in the worker; four more limit messages now carry the prefix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
…served it The op-probe sm89 vllm-0.30.0 entry (benchmarks/evidence/collector/ op-probe-2026-10/findings.yaml) records compute_scale as the same governed approximation as on sm90: torch.compile folds the activation quant, so the standalone row has no serving instance. The scope listed only h100_sxm and h200_sxm, leaving l40s outside it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
…eat/opharness Brings in l40s on the vLLM compute_scale standards exception's systems. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
Brings in 10 main commits. One conflicts: ai-dynamo#405 (SGLang DSA collector inputs and timing boundary), written against SGLang 0.5.14 @49e384ce, while this branch pins the SGLang collectors to exactly 0.5.21. Port of ai-dynamo#405 to 0.5.21 (checked against the v0.5.21 source): - Input relay: prepare_for_extend still stages prefill_input_ids_cpu, so resolve_forward_inputs is kept. The FutureMap is built as the scheduler builds it (spec_algorithm.create_future_map with the one-argument decide_needs_cpu_seq_lens and decide_needs_confidence_relay), and the decode handoff stashes a RelayPayload. - History seeding: act_quant moved to sglang.kernels.ops.attention.dsa. The writers and their signatures are unchanged. - Token check: num_token_non_padded_cpu is now global_num_token_non_padded_cpu. - ForwardBatch.init_new keeps the branch's return_hidden_states_before_norm keyword (required at 0.5.21). - Prefill graph: 0.5.21 serves DSA prefill under the "breakable" backend. The resolved config is reached through get_exec(), since ServerArgs.cuda_graph_config stays None, and the runner constructor changed. ai-dynamo#405's tc_piecewise adapter therefore does not run at 0.5.21. The collector keeps prefill graphs off (disable_prefill_cuda_graph=True) and times DSA prefill eagerly, as the 0.5.21 tables were measured. graph_token_bucket reads get_exec() and returns None when the graph is disabled; any enabled backend fails closed until the adapter is ported (FIXME(port) in dsa_prefill_graph.py lists the work). - One-token extend: it stays out (seq_len > 1). Without the prefill graph it runs eagerly, where the DSA indexer illegal-accessed (FIXME(re-verify)). - Citations moved from 49e384ce to v0.5.21 lines. - Also fixed: - The rope contiguity patch passes fused_set_kv_buffer_arg by keyword (positionally it bound to offsets). - The expert-location reset uses the 0.5.21 runtime_context resource, via runtime_compat.reset_expert_location_metadata. The phase clock and ai-dynamo#405's DSA inputs are combined in _run_prefill and _run_decode, with a "history" phase after metadata. Tests: - main's DSA tests now stub the 0.5.21 names. - The boundary tests drive the native-graph branch through a fake bucket. - New tests pin the real graph_token_bucket: disabled returns None, any enabled backend fails closed. - A one-token extend is rejected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
Blocking: - SGLang full-node ops took no `replay` keyword, so collect_sglang raised a TypeError after the pool ops and lost the summary and provenance. collect_sglang_fullnode_op now accepts replay and selects cases in the pool ops' order. A test checks every keyword collect.py passes. - TRT-LLM MoE gates allocate e_score_correction_bias with torch.empty, and only load_weights fills it, so leftover memory picked the experts. _serving_moe_gate zeroes it, also on the min-latency gate, and passes moe_backend upper-cased (TRTLLM keeps a bf16 bias). DeepSeek-V4 gets DeepseekV4Gate (scored layer) and MiniMax-M3 gets MiniMaxM3Gate, as rc29 serving builds them. - vLLM DSA page: serving sizes the cache block from the first non-SSM attention layer, the indexer cache, which accepts only 64 on CUDA. framework_kv_block_size takes that cache backend (current_platform._find_non_ssm_backend), so FLASHINFER_MLA_SPARSE runs at 64, not 32. vLLM DSA series: - The module is sized at the model's max_position_embeddings. The indexer buffers scale with max_model_len, so a series-max size bent batch scaling. - A failed point's traceback frames are cleared, so its memory does not stay pinned for the smaller points. - FIXME(gap) note: masked-MHA prefill needs model-level heads x TP, which a TP=1 per-rank module never reaches. Collector harness: - helper.run_measurement_subprocess beats the stall watchdog while a child's output grows, and on Linux kills the child with its worker (PR_SET_PDEATHSIG). The SGLang MLA/DSA and MSA collectors use it. - --cases-from refuses an output root that already holds results and never finalizes its CSV staging into parquet or collection_meta.yaml. - SGLang mHC/MLA cleanup resets the 0.5.21 expert-location resource (runtime_compat.reset_expert_location_metadata). - collect_campaign passes AISIM_COLLECTOR_REF (clean-tree HEAD, 40-64 hex) to every shard. Kernel-limit guards: - SGLang MoE: the FlashInfer CUTLASS BF16 %8, SM90 DSV4 FP4 %128 and Triton fp8_block %128 raises now cite their 0.5.21 / FlashInfer 0.6.18 source lines. The int4 group-size raise has no source backing, so it loses the prefix and carries FIXME(re-verify) with the kernels' real limits. - The TRT-LLM fp8 GEMM n/k%16 raise is deleted. The checked-in rc29 l40s and rtx_pro_6000 tables hold 120 rows each that it refused, and SM90+ fails naturally with a cheap TORCH_CHECK. - The SGLang GEMM gap note adds FlashInfer split-K for the 68 tuned bf16 shapes and the fp8_block DeepGEMM-to-Triton fallback for N%64 / K%128. Generator: - The rc29 template renders AUTO, the rc29 default, when no MoE backend is set. - hardware.yaml keeps WIDEEP for Blackwell, and apply_moe_backend drops it from TRT-LLM 1.3.0rc25, the first release without the literal. - 64-token rounding of caller-evaluated limits (preserve_engine_limits) logs the change. - The Sweeper (benchmark rule) renders the MiniMax-M3 sparse-attention algorithm/implementation keys. Other: - A test asserts every standards exception's verified_on equals the collector pin of its tables. - The GLM-5.3-Flash SGLang plan no longer schedules kda/mhc_module. - kernel_source_backends.yaml cites the retired 0520 file at c540ff9. - Stale hca_attn/csa_attn Rust comments are updated. - The PR244 audit excludes the retired hca table, and its manifest is regenerated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
…eat/opharness Brings in ai-dynamo#649's merge of upstream/main 4b153ef (SGLang DSA inputs and boundary ported to 0.5.21) and the round-3 review fixes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
…ix in the opharness - kernel_taxonomy_sm100/sm103.yaml ended in a blank line, which fails Python Static Checks (git diff --check). - probe_driver._fail_cause: since 2026-10-31 the rc29 template renders AUTO when no MoE backend is set and WIDEEP is no longer filled from rc25. The two rules still classify older records. - sgl_attn_ctx_gemma4_hd512.py imports torch before setting sys.argv, so ruff sorts its imports the same from the repo root and the package dir. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the op-probe harness (
python/aisimulate/collector/opharness/). It boots the real serving framework per model, records which kernels and backends serving deploys, and grades the collector's op paths against those records. It produced the serving evidence behind the vLLM 0.30.0 / SGLang 0.5.21 / TensorRT-LLM 1.3.0rc29 collector upgrade.Stacked on #649. This PR targets
main, so until #649 merges its diff also shows #649's commits. #649's branch is merged in, not rebased, as it moves. Review only these commits here:9a67cd332feat: add the op-probe harness for collector upgrades994e7c149docs: publish the complete op-probe findings archivebd189fba6docs: drop the vLLM A/B hooks feat: upgrade the op collector to vLLM 0.30.0, SGLang 0.5.21 and TRT-LLM 1.3.0rc29 #649 removed from the opharness skillHarness-only diff: tianhaox/aisimulate@feat/collector-runtime-upgrade...feat/opharness
This replaces the harness half of #228. #228 bundled the collector upgrade, the harness and its results into one 683-file PR. The collector half is #649, and this PR is the harness half without the intermediate results.
What is in
collector/opharness/components/components/kernel_taxonomy_<sm>.yamlcomponents/captures/collector/opharness/workflows/targets.yaml,configs/python/aisimulate/.claude/skills/opharness/tests/unit/collector/opharness/benchmarks/evidence/collector/op-probe-2026-10/findings.yamlWhat is deliberately out
results/directory, andevidence_bundle.pynow packs that tree into each campaign's evidence bundle. The 2026-10 set is preserved attianhaox/aisimulate@76bacff5b71db00be053a0f61697daf7622ef6e9on branchfeat/op-probe-harness; the evidence README records this.Alignment with #649
targets.yamland the upgrade workflow now say thatframework_manifest.yamlpins the vLLM gemm/moe/kda/mhc families to 0.31.0, ahead of the probed 0.30.0. No harness identity records exist for those families on 0.31.0 yet.AIS_KV_BLOCK_SIZEorAIS_DSA_LEGACY_MODULE, which feat: upgrade the op collector to vLLM 0.30.0, SGLang 0.5.21 and TRT-LLM 1.3.0rc29 #649 removed. Older findings that mention them are dated records and stay as written.collect.pynow refuses to collect withAIS_SMset. The case inventory is unaffected: its--plan-onlycall returns before that check, verified withAIS_SMexported on all three backends.targets.yamlmatches the manifest defaults for vLLM, SGLang and TRT-LLM after the merge.benchmarks/evidenceinstead of local results.Hygiene
ruff checkandruff format --checkpass on the whole harness and its tests. CI does not lintcollector/, but this keeps it at feat: upgrade the op collector to vLLM 0.30.0, SGLang 0.5.21 and TRT-LLM 1.3.0rc29 #649's bar.Path.read_text/write_textwith identical output. The two server log handles stay open for their subprocess and carry anoqawith that reason.Tests
tests/unit/collector+tests/goldentests/unit/collector4182 passed, 0 failed--strict), codeowners policy testsThe probe scripts run inside GPU framework containers and were not exercised on a GPU for this PR. Their edits are formatting, SPDX headers and the behavior-preserving file-handle changes above.
Follow-ups, not in this PR
framework_manifest.yamlinstead of a separate version list intargets.yaml.fused_sigmoid_gating_delta_rule_update_kernel, which sm89/sm120 label asfla_triton. So the KDA decode gate on those SMs only compares the causal_conv1d kernel and cannot see the packed-vs-fused recurrence change in feat: upgrade the op collector to vLLM 0.30.0, SGLang 0.5.21 and TRT-LLM 1.3.0rc29 #649. Label it from those machines' probe records.🤖 Generated with Claude Code