[TileRT] GLM-5.3 MI355X AgentX parallel KV inject experiments, reverted pending tile-ai/TileRT#66 / GLM-5.3 MI355X AgentX 并行 KV 注入实验（已回退，待 tile-ai/TileRT#66） - #3376
Merged
Conversation
…/ 新增 GLM-5.3 FP8 MI355X 的 PD 分离配方
Adds GLM-5.3 FP8 on MI355X as a prefill/decode-disaggregated TileRT recipe:
upstream vLLM ROCm prefill (TileRTConnector, kv_producer) + TileRT
decode_server + an OpenAI-compatible pd_router, framework=tilert, KV over
mooncake. Two scenarios, Single-turn 8k1k and AgentX agentic-coding, mirroring
the merged B200 glm5.1-fp8-b200-tilert entries on the amd_utils orchestration.
以 TileRT 的 prefill/decode 分离形态新增 MI355X 上的 GLM-5.3 FP8：上游 vLLM
ROCm 作 prefill + TileRT decode_server + 兼容 OpenAI 的 pd_router，KV 走
mooncake。两个场景：单轮 8k1k 与 AgentX 智能体编码，形态对齐已合并的 B200
glm5.1-fp8-b200-tilert，走 amd_utils 编排链。
IMAGES AND WHEEL / 镜像与轮子
decode ghcr.io/tile-ai/tilert-rocm-decode:0.1.6
prefill ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6
(the upstream vllm/vllm-openai-rocm release image plus the mooncake
ROCm transfer engine built from source)
Neither image carries the wheel; setup_deps.sh installs tilert==0.1.6 from
PyPI into both ranks at container start -- full on decode, --no-deps connector
plugin on prefill -- so vLLM runs exactly as shipped and no file inside either
image is modified.
TOPOLOGY / 拓扑
1 prefill node (TP8) + 1 decode node (TP8), conc-list [1] because TileRT decode
serves one sequence at a time. bf16 KV cache on both roles -- the ROCm vLLM
sparse-MLA backend has no fp8_ds_mla dtype. max-model-len 202752. MTP via
spec-decoding=mtp at draft depth 3. Prefill runs --block-size 64: the ROCm
backend supports [1,64] and vLLM picks 1, which makes the connector's KI plane
copy fail and MLA address the wrong rows.
GOLDEN AL / 黄金接受率
Nothing hard-codes an acceptance length. server_tilert.sh reads the committed
curve at launch, selecting it exactly as infx/srt_slurm/synthetic_acceptance.py
does (<model-prefix>_<method>.yaml, then [model][thinking mode][draft tokens]),
applies it to agentic throughput only, suppresses it under RUN_EVAL/EVAL_ONLY
because simulated acceptance would score ~0 on GSM8K, and fails the run rather
than falling back to real acceptance when the curve or draft length is missing.
接受率不写死在任何配方或配置里，由 server_tilert.sh 启动时从提交的曲线读取，
选择规则与上游 srt-slurm 连接器一致；仅作用于 agentic 吞吐，eval 下自动关闭，
曲线缺失时直接失败而非静默回落。
NOTE FOR REVIEWERS: golden_al_distribution/glm5.3_mtp.yaml is GLM-5.2's curve,
copied. No SPEED-Bench run exists for GLM-5.3 and the two share a base and
layer topology. Only the K=3 cell is carried over. The file is labelled
PROVISIONAL. Flagged rather than left to read as a measured 5.3 curve.
LOCAL VALIDATION / 本地验证
Run on 2x8 MI350X with these images and wheel, through the official entry
points:
GSM8K (RUN_EVAL + EVAL_ONLY, 1319 items) strict 0.9750, flexible 0.9735
AgentX 3600 s, conc 1 submission_valid true, coverage
99.8%/99.9%, 532 requests
Single-turn 8k1k, conc 1 TTFT p50 1010 ms
MI350X is not MI355X: same gfx950 ISA and memory system, but 1000 W vs 1400 W
and 2200 vs 2400 MHz. These show the recipe runs and is accurate; they are not
leaderboard results, which the sweep produces on real mi355x runners.
本地数据跑在 MI350X 上，不是 MI355X，仅用于证明配方能跑且结果正确，
不作为打榜成绩。
Matrix UTs 335 passed. perf-changelog is a pure tail append; the 8457 lines
inherited from main are byte-for-byte unchanged.
… / 修正 TileRT 配方的 Bash 规范与 MTP 草稿深度
Review follow-ups on the GLM-5.3 FP8 MI355X TileRT recipes.
- server_tilert.sh no longer invents defaults for caller-supplied
configuration. Every value now arrives from the recipe through job.slurm
and is validated with check_env_vars, mirroring server_atom.sh. The same
applies to the two recipes and to the tilert branch of setup_deps.sh.
- Draft depth is consistent. DECODE_MTP_SIZE=3 is the TileRT decode draft
depth, used by the weight converter and as the key the golden acceptance
curve is read at; PREFILL_SPEC_TOKENS=1 is the vLLM prefill rank's MTP
layer. Previously the converter and the curve used a hard-coded 3 while
the recipes exported an unused DECODE_MTP_SIZE=1.
- THINKING_MODE and MODEL_PREFIX are forwarded into the container and
validated, so the golden-acceptance selection no longer falls back to
built-in guesses.
- job.slurm drops MODEL_NAME="${MODEL_NAME:-None}", which masked a missing
model name for every engine, not just tilert.
- models_tilert.yaml keeps only model-owned engine environment; profile,
model_type and max_model_len moved to the recipes, which removes the
stale glm-5_2 model_type that the recipe already overrode.
- launch_mi355x-amds.sh and server.sh no longer carry unrelated comment and
blank-line deletions.
- MODELS.md/MODELS_zh.md and the two changelog entries point at the
upstream PR and carry the correct date.
以评审意见修正 GLM-5.3 FP8 MI355X 的 TileRT 配方：server_tilert.sh 不再为调用方
提供的配置自造默认值，全部改由配方经 job.slurm 传入并用 check_env_vars 校验，
与 server_atom.sh 保持一致；统一 MTP 草稿深度（decode 侧 DECODE_MTP_SIZE=3，
供权重转换与黄金接受率曲线取值，prefill 侧 PREFILL_SPEC_TOKENS=1）；将
THINKING_MODE 与 MODEL_PREFIX 传入容器并校验；移除 job.slurm 中会掩盖所有引擎
MODEL_NAME 缺失的兜底；models_tilert.yaml 只保留模型自有的引擎环境；撤销
launcher 与 server.sh 中与本次改动无关的删除；文档与 changelog 链接改为上游 PR。
…完整 1M max-model-len was 202752 on both roles. That is GLM-5.1's context window, inherited from the B200 glm5.1-fp8-b200-tilert recipe this one mirrors (perf-changelog entry for that recipe records "max-seq-len 202752"), not GLM-5.3's. GLM-5.3 shares GLM-5.2's base, and every in-tree GLM-5.2 recipe -- TRT-LLM and SGLang alike -- runs at 1048576. Both roles now launch at 1048576, and the AgentX trace filter follows it through MAX_MODEL_LEN. 原配方两侧均为 202752，这是 GLM-5.1 的上下文窗口，沿用自本配方对标的 B200 glm5.1-fp8-b200-tilert，并非 GLM-5.3 的。GLM-5.3 与 GLM-5.2 同底座，而树内所有 GLM-5.2 配方（TRT-LLM 与 SGLang）均使用 1048576。现两侧均以 1048576 启动， AgentX 轨迹过滤经 MAX_MODEL_LEN 随之生效。
…pool / 补齐 MI355X 上 TileRT 的 amd_utils 编排输入 The tilert arm of benchmarks/multi_node/runtime_settings.sh is the B200 srt-slurm profile. The MI355X recipes run through the shared amd_utils chain instead, whose submit.sh validates ROUTER_TYPE, PROXY_PING_PORT, HEADNODE_PORT and SERVER_PORT, and whose job.slurm and setup_deps.sh also require VLLM_ROUTER_IMAGE, SKIP_RDMA_CHECK, SKIP_GPU_SANITY, ENABLE_METRICS, PREFILL_ROUTER_POLICY, DECODE_ROUTER_POLICY, FLUSH_DRAIN_TIMEOUT, CLEAR_CACHE_BETWEEN_CONC, ROCM_PATH, UCX_HOME and RIXL_HOME. None of those were exported for FRAMEWORK=tilert, so submit.sh exited before sbatch and the recipe returned no job id. The AMD block is scoped by RUNNER_TYPE so the B200 TileRT lane is untouched. The fixed-seq-len recipe now sets KV_OFFLOADING=none itself: submit.sh and job.slurm validate it, and the matrix emits it only for agentic scenarios. launch_mi355x-amds.sh now fails when a recipe returns no job id. Its wait loop polled for slurm_job-.out while its liveness guard degenerated to `grep -q ""`, which matches any queued job of that user, so a failed submit span the loop until the job hit its time limit rather than reporting the error. benchmarks/multi_node/runtime_settings.sh 的 tilert 分支是 B200 的 srt-slurm 配置，而 MI355X 走的是共用 amd_utils 链：submit.sh 校验 ROUTER_TYPE、 PROXY_PING_PORT、HEADNODE_PORT、SERVER_PORT，job.slurm 与 setup_deps.sh 另需 VLLM_ROUTER_IMAGE、SKIP_RDMA_CHECK、SKIP_GPU_SANITY、ENABLE_METRICS、 PREFILL_ROUTER_POLICY、DECODE_ROUTER_POLICY、FLUSH_DRAIN_TIMEOUT、 CLEAR_CACHE_BETWEEN_CONC、ROCM_PATH、UCX_HOME、RIXL_HOME。FRAMEWORK=tilert 下这些一概未导出，导致 submit.sh 在 sbatch 之前退出、配方返回空 job id。AMD 专用 块以 RUNNER_TYPE 限定，不影响 B200。 定长配方自行设置 KV_OFFLOADING=none（矩阵仅在 agentic 场景下产出该字段）。 launch_mi355x-amds.sh 现在在配方未返回 job id 时直接失败：原等待循环会一直轮询 slurm_job-.out，而其存活判断退化为 `grep -q ""`，会匹配该用户任意排队作业，于是 提交失败不会报错，而是空转到作业超时。
Port the launcher fix from CrimsonDump/InferenceX commit 9e745f3. Set RDMAV_FORK_SAFE=1 before either role initializes verbs to avoid silent TCP fallback and degraded TTFT; TPOT is unaffected. Append performance changelog entries for 8k1k and AgentX. 在 prefill 和 decode 两侧初始化 verbs 前设置 RDMAV_FORK_SAFE=1，避免静默回退到 TCP 导致 TTFT 恶化；TPOT 不受影响。补充代码注释及 8k1k、AgentX 性能变更记录。
…位后重新触发 sweep The three earlier sweeps failed at the model-availability check because /it-share/data/GLM-5.3 did not exist on the MI355X cluster. zai-org/GLM-5.3 is now staged there: 155 files, 141 shards, 704 GiB, every file byte-exact against the Hugging Face manifest, world-readable for the runner account. config.json confirms fp8 (e4m3, weight_block_size 128x128) and max_position_embeddings 1048576, matching the recipe's precision and context. No code change. Re-applying the sweep label produced label events but no workflow run, and run-sweep.yml has no workflow_dispatch trigger, so this empty commit provides the synchronize event. 此前三次 sweep 均因 MI355X 集群上不存在 /it-share/data/GLM-5.3 而在模型可用性 检查处失败。现已完成权重落盘：155 个文件、141 个分片、704 GiB，逐文件字节数与 Hugging Face manifest 完全一致，并已对 runner 账号开放读取权限。config.json 确认为 fp8（e4m3，weight_block_size 128x128）、max_position_embeddings 为 1048576，与配方的精度与上下文设置一致。 无代码改动。重新添加 sweep 标签只产生了 label 事件而未触发工作流，且 run-sweep.yml 没有 workflow_dispatch 触发器，故以此空提交产生 synchronize 事件。
…合入分支 perf-changelog.yaml was the only conflict: both sides appended to the tail (base 8457 lines; main +59, this branch +35). Resolved so main's 8516 lines stay byte-identical and this branch's four entries move to the physical end, as the append-only rule requires. 1002 entries, no other file conflicted. perf-changelog.yaml 是唯一冲突：双方均为尾部追加（基线 8457 行，main +59， 本分支 +35）。按 append-only 规则解决：保持 main 的 8516 行逐字节不变，本分支 的四条entries移至文件物理末尾。共 1002 条，其余文件无冲突。
…ileRT 路径不再校验 MoRI 与 SGLang 环境变量 Sweep 35625047683 reached the container, installed tilert 0.1.6 and found mooncake, then failed in env.sh on eleven variables it never reads: MORI_IO_SQ_BACKOFF_TIMEOUT_US, MORI_IO_QP_MAX_SEND_WR, MORI_IO_QP_MAX_CQE, MORI_IO_QP_MAX_SGE, MORI_IO_TC_DISABLE, UCX_IB_GID_INDEX, MORI_APP_LOG_LEVEL, SGLANG_ROUTER_STDOUT_LOGS, TORCH_NCCL_BLOCKING_WAIT, NCCL_BLOCKING_WAIT and SGLANG_OPT_USE_AITER_INDEXER. Those are MoRI-IO queue-pair tuning, the UCX RoCE GID index, SGLang router logging and the SGLang decode cuda-graph NCCL workaround. server_tilert.sh references none of them; KV moves over mooncake and no SGLang router starts. env.sh validated them before it had even validated ENGINE, so the list is now scoped to the engines that consume it. Supplying values instead would mean inventing MoRI tuning for a transport this recipe never uses. Verified that ENGINE=sglang-disagg still fails on the same list. Also export INFMAX_CONTAINER_WORKSPACE=/workspace for the agentic replay, which runs in this container rather than a separate client: benchmark_lib's install_agentic_deps and build_replay_cmd both require it, the SGLang path sets it in its client container's env file, and launch_mi355x-amds.sh exports it only on the DSv4.1-Flash single-node branch. Without it the agentic lane would have failed at the next step. sweep 35625047683 已进入容器、装好 tilert 0.1.6 并找到 mooncake，随后在 env.sh 因十一个它从不读取的变量而失败。这些是 MoRI-IO 队列对调优、UCX RoCE GID 索引、 SGLang router 日志以及 SGLang decode cuda-graph 的 NCCL 规避项；server_tilert.sh 一概不引用，KV 走 mooncake，也不启动 SGLang router。env.sh 甚至在校验 ENGINE 之前就校验了它们，现已将该列表限定到真正消费它们的引擎；若改为提供取值，等于为 本配方根本不用的传输层编造 MoRI 调优。已验证 ENGINE=sglang-disagg 仍会因同一 列表失败。 同时为 agentic 回放导出 INFMAX_CONTAINER_WORKSPACE=/workspace：回放在本容器内 运行而非独立客户端容器，benchmark_lib 的 install_agentic_deps 与 build_replay_cmd 均要求该变量，而 SGLang 路径是在其客户端容器的 env 文件中设置、launcher 仅在 DSv4.1-Flash 单节点分支导出。否则 agentic lane 会在下一步失败。
…p8 MLA KV 以容纳 1M 上下文
Sweep 35626519997 failed on both lanes for the same reason, on opposite roles:
decode (agentic, g14) torch.OutOfMemoryError: tried to allocate 99.06 GiB
with 95.94 GiB free; 184.17 GiB of weights already
resident on a 287.98 GiB card
prefill (8k1k, g09) ValueError: 91.71 GiB KV cache is needed, which is
larger than the available KV cache memory (85.25 GiB)
bf16 MLA KV does not fit at 1048576 tokens on either rank. The recipe chose
bf16 because the ROCm sparse-MLA backend has no fp8_ds_mla, which is true but
does not imply bf16: ROCM_AITER_MLA_SPARSE lists fp8 and fp8_e4m3 in
supported_kv_cache_dtypes, and TileRT's MlaNsaProfile.configure() maps
fp8_ds_mla, fp8 and fp8_e4m3 onto the same mla_fp8 layout. Only fp8_ds_mla is
CUDA-only. Plain fp8 therefore gives both ranks the matching layout that
TileRT's --kv-cache-dtype requires, and halves the cache:
KV_BYTES_FP8 = 528 B/token against KV_BYTES_BF16 = 1024 B/token. Decode drops
to roughly 51 GB alongside its weights, prefill to roughly 46 GiB inside its
85.25 GiB budget.
Also take the container-creation barrier timeout from CONTAINER_BARRIER_TIMEOUT
(5400s) rather than hardcoding 1800s, matching how server_sglang.sh sources it.
This recipe pulls two ~32 GB images, one per rank; on a node caching neither,
the rank that comes up first waits out the whole timeout while its peer pulls.
The observed cold decode pull took about 25 minutes of the old 30-minute budget.
sweep 35626519997 两条 lane 因同一原因在相对的两侧失败：decode 侧在 287.98 GiB
卡上已驻留 184.17 GiB 权重、再分配 99.06 GiB KV 时 OOM；prefill 侧因需要
91.71 GiB KV 而可用 85.25 GiB 被 vLLM 拒绝。1048576 上下文下 bf16 MLA KV 两侧
均无法容纳。配方原先选 bf16 是因为 ROCm 稀疏 MLA 没有 fp8_ds_mla，此事为真但并
不意味着只能用 bf16：ROCM_AITER_MLA_SPARSE 的 supported_kv_cache_dtypes 含 fp8
与 fp8_e4m3，而 TileRT 的 MlaNsaProfile.configure() 将 fp8_ds_mla/fp8/fp8_e4m3
映射到同一 mla_fp8 布局，仅 fp8_ds_mla 为 CUDA 独有。因此普通 fp8 即可让两侧获得
TileRT 所要求的一致布局，并将 KV 减半（每 token 528 字节对 1024 字节）。
屏障超时同时改为取自 CONTAINER_BARRIER_TIMEOUT（5400 秒），与
server_sglang.sh 的做法一致：本配方每 rank 各拉取约 32 GB 镜像，在均未缓存的
节点上，先起来的一侧会在对端拉取期间耗尽原有 30 分钟预算（实测冷拉取约 25 分钟）。
…合入分支 Two conflicts from main's twelve new commits. setup_deps.sh: #3349 deleted install_transformers_glm5 and its dispatch arm as retired runtime support, while this branch had added the tilert arm next to it. Took main's deletion and kept only the tilert helpers and elif arm, leaving no else branch. Safe for this recipe: both images already load glm_moe_dsa, as the last sweep showed -- decode loaded weights and prefill reached KV allocation. perf-changelog.yaml: this branch appends 43 lines at the tail. Main's side also carries mid-file insertions from #3207 and #2912, older branches that merged later; those bytes are preserved exactly and this branch's five entries stay at the physical end. 1009 entries, parses clean. main 的十二个新提交带来两处冲突。setup_deps.sh：#3349 作为「移除已退役运行时 支持」删除了 install_transformers_glm5 及其分派分支，而本分支曾在其旁新增 tilert 分支；现采用 main 的删除，仅保留 tilert 辅助函数与 elif 分支，不留 else。 对本配方安全：两个镜像均已支持 glm_moe_dsa（上轮 sweep 中 decode 已加载权重、 prefill 已进入 KV 分配阶段）。perf-changelog.yaml：本分支在尾部追加 43 行； main 侧另含 #3207、#2912 这两个较早分支较晚合入而产生的文件中部插入，其字节 原样保留，本分支五条条目仍位于文件物理末尾。共 1009 条，解析正常。
…kip-sweep] / 恢复 bf16 布局，1M 上下文取决于 DRAM 侧 PD 缓冲 Reverts the fp8 KV change from 19bea0b. It cannot run: vLLM 0.24.0's ROCM_AITER_MLA_SPARSE writes a flat 576 B/token fp8 row with a per-layer scale, and TileRT 0.1.6 infers the layout from the tensor stride and accepts only 1152 B (bf16) or 656 B (fp8_ds_mla), raising "unexpected MLA cache stride 576 B/token" at register_kv_caches on every prefill rank. The original bf16 choice was the only layout-correct one on ROCm. PREFILL_KV_DTYPE is now an explicit bfloat16 (auto resolves the same on this model; float16 would pass the stride check and be misread). Memory at 1048576 tokens, per rank, verified against the tilert and vLLM sources and the MI355X logs: decode : 90.72 GiB weights + 93.25 GiB engine cache window + 99.06 GiB PD receive buffer (~283 of 287.98 GiB) prefill: 90.45 GiB weights + 40.3 GiB profiling + 91.71 GiB vLLM KV + 99.06 GiB PD staging buffer outside vLLM's budget (~321 GiB) The two PD buffers are dense in max_seq_len. Both must move to pinned host memory for this context to start, which is a TileRT change; the recipe is set up for that release (GPU_MEM_UTIL 0.85 so vLLM's 222.5 GiB fits inside its budget) and keeps 1048576. Until the wheel and images are bumped the sweep cannot pass, hence [skip-sweep]. Also: drop the invented TILERT_CONVERT_DEVICE default; pass --num-mtp "$DECODE_MTP_SIZE" to decode_server so converter, golden key and engine depth agree by data flow; make the conversion-cache check read tilert_meta.json num_mtp so a stale non-MTP cache is re-converted instead of failing after a 90 GiB weight load; validate CONTAINER_BARRIER_TIMEOUT in job.slurm's tilert branch; correct the earlier changelog text (router deps are installed on decode only, models_tilert.yaml holds prefill env, not profile defaults) and replace the wrong halving arithmetic. 撤销 19bea0b 的 fp8 KV 改动：vLLM 0.24.0 的 ROCM_AITER_MLA_SPARSE 写出每 token 576 字节的扁平 fp8 行，而 TileRT 0.1.6 依步长推断布局，只接受 1152（bf16）或 656（fp8_ds_mla）字节，会在 register_kv_caches 报 "unexpected MLA cache stride 576 B/token"。原作者的 bf16 是 ROCm 上唯一正确的布局。 1048576 上下文下每 rank 的内存（已对照源码与日志核实）：decode 侧 90.72 GiB 权重 + 93.25 GiB 引擎缓存窗口 + 99.06 GiB PD 接收缓冲（约 283/287.98 GiB）；prefill 侧 90.45 GiB 权重 + 40.3 GiB 分析开销 + 91.71 GiB vLLM KV + 位于 vLLM 预算之外 的 99.06 GiB PD 暂存缓冲（约 321 GiB）。两个 PD 缓冲按 max_seq_len 密集分配， 须移入锁页主机内存才能在该上下文启动，此为 TileRT 侧改动；配方已为该版本就位 （GPU_MEM_UTIL 0.85）并保留 1048576。wheel 与镜像升级前 sweep 无法通过，故 [skip-sweep]。 另：移除 TILERT_CONVERT_DEVICE 的自造默认；向 decode_server 显式传递 --num-mtp； 转换缓存检查读取 tilert_meta.json 的 num_mtp；job.slurm 的 tilert 分支校验 CONTAINER_BARRIER_TIMEOUT；修正此前 changelog 中的错误表述与减半算术。
…合入分支 perf-changelog.yaml only: main appended 15 lines (#3337), this branch appends at the tail; main's 8589 lines are byte-identical and this branch's entries follow them. configs/amd-master.yaml auto-merged. 仅 perf-changelog.yaml 冲突：main 追加 15 行（#3337），本分支尾部追加；main 的 8589 行逐字节不变，本分支条目位于其后。configs/amd-master.yaml 自动合并。
…ert 0.1.6 patch + waiver 3330 / PD 缓冲移入 2 MiB 大页锁页内存以支持 1M 上下文，附 tilert 0.1.6 补丁与豁免
TileRT 0.1.6 keeps its two PD buffers on the GPU, dense in max_seq_len:
99.06 GiB each at 1048576 tokens. With 90.72 GiB of weights and the 93.25 GiB
engine cache window the decode rank sits at ~283 of 287.98 GiB (node-marginal,
OOM on g14), and prefill rank 0 needs ~321 GiB (weights 90.45 + profiling
40.29 + vLLM KV 91.71 inside the budget + the 99.06 GiB staging buffer outside
it). No --kv-cache-dtype shrinks them on ROCm: TileRT accepts only the 1152 B
(bf16) or 656 B (fp8_ds_mla) stride and vLLM's ROCM_AITER_MLA_SPARSE has no
fp8_ds_mla.
setup_deps.sh now applies patches/tilert-0.1.6-pd-buffers-in-dram.patch to the
pip-installed tilert 0.1.6 on both ranks when TILERT_PD_BUFFER_DEVICE=cpu
(marker-gated, pinned to 0.1.6, checked before the wheel download). The
patch (5 pure-Python files under tilert/pd_vllm) adds decode_server
--pd-buffer-device and connector extra config tilert_pd_buffer_device, and
allocates both buffers with alloc_pinned_huge(): a 2 MiB-aligned mmap with
MADV_HUGEPAGE, populated via MADV_POPULATE_WRITE, verified from
/proc/self/smaps to be fully huge-page backed, hipHostRegister'ed, wrapped as
a pinned torch tensor; convert() moves each received plane to the device.
The verification matters: the Pensando ionic RDMA VFs cap 4 KiB-page
ibv_reg_mr at 3968 MiB per HCA (~2^20 entries), while 2 MiB-backed 100 GiB
regions register in one MR and took a cross-node mooncake write at 20.9 GiB/s
(measured on g09/g11). Registering in <4 GiB pieces does not help; the budget
is per HCA, not per MR.
server_tilert.sh validates TILERT_PD_BUFFER_DEVICE in {cpu,cuda} up front and
passes --pd-buffer-device only for cpu, so cuda runs the shipped wheel (which
has no such flag) unmodified. job.slurm forwards and validates the knob; both
recipes set cpu and GPU_MEM_UTIL 0.85 (222.5 GiB inside a 244.8 GiB budget).
vLLM is not patched. Engine-patch waiver: docs/waiver/3330.md; the upstream
TileRT link is pending.
TileRT 0.1.6 将两个按 max_seq_len 密集分配的 PD 缓冲放在 GPU 上，1048576 上下文下
各 99.06 GiB；连同 90.72 GiB 权重与 93.25 GiB 引擎缓存窗口，decode 侧约 283/287.98
GiB（节点边缘，g14 OOM），prefill rank 0 约需 321 GiB。ROCm 上无法用
--kv-cache-dtype 缩减：TileRT 只接受 1152（bf16）或 656（fp8_ds_mla）字节步长，而
vLLM 的 ROCM_AITER_MLA_SPARSE 没有 fp8_ds_mla。
现由 setup_deps.sh 在 TILERT_PD_BUFFER_DEVICE=cpu 时对两侧 pip 安装的 tilert 0.1.6
应用 patches/tilert-0.1.6-pd-buffers-in-dram.patch（有标记门控、固定 0.1.6，在下载
wheel 前检查）。补丁新增 decode_server --pd-buffer-device 与 connector 配置
tilert_pd_buffer_device，用 alloc_pinned_huge() 分配两个缓冲：2 MiB 对齐的 mmap +
MADV_HUGEPAGE，经 MADV_POPULATE_WRITE 预填充，依 /proc/self/smaps 校验全部为大页，
hipHostRegister 后包装为 pinned torch 张量；convert() 将接收到的各层数据搬到设备。
校验的必要性：ionic RDMA VF 对 4 KiB 页的 ibv_reg_mr 上限为每 HCA 3968 MiB（约 2^20
条页表项），而 2 MiB 大页的 100 GiB 区域可作单个 MR 注册，并以 20.9 GiB/s 接收跨节点
mooncake 写入（g09/g11 实测）；分成小于 4 GiB 的块无济于事，预算按 HCA 而非按 MR。
server_tilert.sh 预先校验 TILERT_PD_BUFFER_DEVICE 取值，仅在 cpu 时传递
--pd-buffer-device，cuda 下原样运行 wheel。job.slurm 转发并校验该变量；两份配方设为
cpu 并将 GPU_MEM_UTIL 设为 0.85。vLLM 未打补丁。引擎补丁豁免见 docs/waiver/3330.md，
上游 TileRT 链接待补。
…口 60 秒 sync.py --enable-port keeps the local port open for max(60, timeout // 2) seconds after the barrier passes, in the foreground, so a peer one poll behind still sees it. With the TileRT launcher's CONTAINER_BARRIER_TIMEOUT=5400 that became a 45-minute idle sleep on every rank: jobs 45373/45374 passed the barrier at 06:28 and sat in time.sleep until 07:13 with two nodes allocated (main thread in hrtimer_nanosleep, accept thread in inet_csk_accept). The old hardcoded 1800s hid a 15-minute version of the same sleep. sync.py gains --grace <seconds>; unset keeps the existing formula, so the SGLang/vLLM/ATOM arms are unchanged. server_tilert.sh passes --grace 60: both ranks pass within one 5 s poll of each other, and the prefill and decode stages have their own readiness waits (DECODE_WAIT, PREFILL_WAIT, ROUTER_WAIT). sync.py 的 --enable-port 在屏障通过后会在前台将本地端口保持 max(60, timeout // 2) 秒，以便晚一次轮询的对端仍能看到。TileRT 启动脚本的 CONTAINER_BARRIER_TIMEOUT=5400 使之变为每个 rank 空转 45 分钟：作业 45373/45374 于 06:28 通过屏障后一直 sleep 到 07:13，占用两个节点。原先硬编码的 1800 秒同样隐藏着 15 分钟的空转。 sync.py 新增 --grace <秒>；不传则沿用原公式，SGLang/vLLM/ATOM 各臂不受影响。 server_tilert.sh 传入 --grace 60：两侧在一次 5 秒轮询内相继通过，且后续 prefill/decode 阶段各有自己的就绪等待。
…经 torch.cuda.cudart 注册 PD 主机缓冲 Sweep 35693738597 job 45373: the decode rank died in alloc_pinned_huge with "neither libamdhip64 nor libcudart could be loaded". The patch dlopen'ed /opt/rocm/lib/libamdhip64.so or libamdhip64.so by name, which exists in the prefill image (ROCm 7.2.3) but not in tilert-rocm-decode:0.1.6, whose torch 2.12+rocm7.14 wheel ships only the versioned libamdhip64.so.7 and no /opt/rocm tree. alloc_pinned_huge now registers the region with torch.cuda.cudart() .cudaHostRegister(addr, total, 0), torch's own runtime binding, hipified on ROCm and present on CUDA builds; verified in both images (cudaError.success, tensor.is_pinned() True). The ctypes library search is gone. Waiver text updated to match. sweep 35693738597 的作业 45373：decode 侧在 alloc_pinned_huge 中因 "neither libamdhip64 nor libcudart could be loaded" 退出。补丁按名字 dlopen /opt/rocm/lib/libamdhip64.so 或 libamdhip64.so，prefill 镜像（ROCm 7.2.3）有，而 tilert-rocm-decode:0.1.6 的 torch 2.12+rocm7.14 wheel 只带版本化的 libamdhip64.so.7， 且无 /opt/rocm 目录。 现改为通过 torch.cuda.cudart().cudaHostRegister(addr, total, 0) 注册，即 torch 自身 的运行时绑定，ROCm 上已 hipify、CUDA 构建亦有；两镜像均验证通过 （cudaError.success，is_pinned 为 True）。移除 ctypes 库查找，豁免文本同步更新。
Add glm5.3* to the 1M-context trace-source pattern and append the required performance changelog entry. Explicit loader overrides remain unchanged. Validation: bash -n and git diff --check passed; 11 real-resolver behavioral cases passed with external dependency installation/download stubbed; changelog YAML, exact config key, and historical byte preservation checked. No GPU benchmark rerun. AI disclosure: GPT 6 Astra via Perplexity Computer implemented and validated this follow-up. No delegated agents. 中文：将 glm5.3* 加入 1M 上下文语料匹配分支，默认选择未截断的 062126 语料，并在性能变更记录末尾追加条目；显式 loader 覆盖保持不变。Shell 语法、差异检查、11 个真实解析函数行为用例及变更记录校验通过；未重新运行 GPU 基准。由 GPT 6 Astra 经 Perplexity Computer 实现和验证，未使用委派 agent。
…ic workspace before sourcing; merge origin/main / 结果目录与工作区分离、提前设置 agentic 工作区变量，并合入 origin/main Sweep 35704948491 brought both lanes up at 1048576 tokens with the PD buffers in DRAM (decode: "receive buffer ready: 99.06 GiB on cpu", "transport=mooncake ready", REQSTAT lines with 7252-token prompts converted and decoded; prefill vLLM ready at gpu-memory-utilization 0.85; router ready) and the 8k1k lane ran its benchmark to completion (16/16, median TTFT 888 ms, TPOT 1.89 ms). Both lanes then failed after the fact: - 8k1k: the launcher copied the result JSON into $GITHUB_WORKSPACE and its EXIT trap ran `sudo rm -rf "$BENCHMARK_LOGS_DIR"`. The tilert arm of benchmarks/multi_node/runtime_settings.sh is the B200 profile and sets BENCHMARK_LOGS_DIR="$GITHUB_WORKSPACE", so the trap deleted the checkout, results included. The AMD block now sets BENCHMARK_LOGS_DIR="$GITHUB_WORKSPACE/benchmark_logs", the convention runners/runtime_settings.sh uses for this pool, and launch_mi355x-amds.sh refuses a BENCHMARK_LOGS_DIR that is the checkout or contains it. - agentic: benchmark_lib.sh derives AGENTIC_DIR and AIPERF_DIR from INFMAX_CONTAINER_WORKSPACE when it is sourced; server_tilert.sh exported the variable later, inside run_agentic_replay, so the client bootstrap looked for /utils/agentic-benchmark/requirements.txt. The export now precedes the source line; resolve_trace_source and install_agentic_deps fail the replay instead of falling through (no errexit here); HF_HOME points at the node's /run_logs mount so the trace corpus is not re-downloaded per job. Merges origin/main (6c5eff0, #3360: priority classifier on Opus 5), no conflicts. sweep 35704948491 中两条 lane 均在 1048576 上下文下以 DRAM 中的 PD 缓冲成功启动， 8k1k 基准测试跑完（16/16，TTFT 中位数 888 ms，TPOT 1.89 ms），随后在事后失败： 8k1k 侧 launcher 将结果复制进 $GITHUB_WORKSPACE 后，EXIT trap 的 `rm -rf "$BENCHMARK_LOGS_DIR"` 因 tilert 分支沿用 B200 配置（值为 $GITHUB_WORKSPACE）而删除了整个检出目录；AMD 专用块现改为 $GITHUB_WORKSPACE/benchmark_logs，launcher 亦拒绝指向检出目录的取值。agentic 侧 benchmark_lib.sh 在 source 时由 INFMAX_CONTAINER_WORKSPACE 推导路径，而该变量此前在 之后才导出；现已提前，两个引导函数失败即返回，HF_HOME 指向节点 /run_logs 以免每次 重下语料。同时合入 origin/main（6c5eff0b8，#3360），无冲突。
…n tilert 0.1.6.post1 / GLM-5.3 改用未过滤 WEKA 语料，PD 缓冲改由 tilert 0.1.6.post1 分片
Two changes, both measured on a 2x8 MI350X pair at --max-model-len 1048576
with the bf16 wire layout.
1) resolve_trace_source() gives the unfiltered 062126 corpus to the 1M-context
families and the 256k-filtered variant to everyone else, but the pattern list
only spells glm5.2*, so glm5.3 falls through to the 256k branch. GLM-5.3 shares
GLM-5.2's base and context window, so it belongs with the 1M families. The run
log names the loader explicitly:
Loading traces via aiperf public-dataset:
semianalysis_cc_traces_weka_062126_256k
(semianalysisai/cc-traces-weka-062126-256k) [MODEL_PREFIX=glm5.3]
corpus ISL p50 p90 p95 max
...062126_256k (current) 69,813 170,341 -- 249,403
...062126 (this change) 291,511 412,030 441,187 476,041
The 256k corpus never replays the tail this recipe is sized for, and its
numbers are not comparable with the merged glm5.2 AMD entries, which do replay
the unfiltered corpus.
2) tilert 0.1.6.post1 shards both PD buffers by layer across the eight devices
(layer lid on device lid % 8) instead of placing each dense buffer on one card,
so the recipe no longer needs the DRAM patch or TILERT_PD_BUFFER_DEVICE=cpu.
Per card: 12.54 GiB instead of 99.06 GiB. convert() dequantises each layer on
the device that received it, so its transients (108.6 KiB/token, 82.9 GiB at
800k tokens) spread as well rather than staying on cuda:0 -- the ceiling the
DRAM route leaves in place. The KV path stays device-to-device: 81 GB in
751 ms = 108 GB/s, 54% of the 4x400 GbE line rate, against 20.9 GiB/s measured
for the host-to-host hop.
Measured with both changes, 3600 s AgentX at concurrency 1:
submission_valid True, TTFT and inter-token-latency coverage both 100%,
239 successful requests, 0 errors, TTFT p50 5,711.8 ms, ITL p50 2.18 ms.
Peak device memory 202.1 GiB per card on decode and 269.2 GiB on prefill, of
287.69 GiB. GPU_MEM_UTIL stays at 0.85.
一、resolve_trace_source() 把未过滤的 062126 语料给 1M 上下文家族、其余给 256k
过滤版，但模式串只写了 glm5.2*，于是 glm5.3 落进 256k 分支。GLM-5.3 与 GLM-5.2
同底座、同上下文窗口，应归入 1M 家族。256k 语料从不回放本配方所声明的长尾，且与
已合入的 glm5.2 AMD 条目不可比（对照见上表，日志原文见上）。
二、tilert 0.1.6.post1 把两个 PD 缓冲按层分片到 8 张卡（层 lid 落在卡 lid % 8），
不再把整块放在单卡上，因此本配方不再需要 DRAM 补丁与 TILERT_PD_BUFFER_DEVICE=cpu。
每卡 12.54 GiB 而非 99.06 GiB。convert() 在收到该层的那张卡上就地反量化，其中间
结果（108.6 KiB/token，800k 时 82.9 GiB）也随之摊开——这正是 DRAM 方案未解的那个
天花板。KV 通路保持设备到设备：81 GB / 751 ms = 108 GB/s，为 4x400 GbE 线速的
54%，对比主机到主机实测 20.9 GiB/s。
两项改动同时生效下实测（3600 秒 AgentX，并发 1）：submission_valid True、两项
覆盖率均 100%、239 条成功、0 错误、TTFT p50 5,711.8 ms、ITL p50 2.18 ms；decode
每卡峰值 202.1 GiB、prefill 269.2 GiB（卡容量 287.69 GiB）；GPU_MEM_UTIL 保持 0.85。
…he PD-DRAM patch, waiver 3330 and TILERT_PD_BUFFER_DEVICE / 交由 tilert 0.1.6.post1 分片 PD 缓冲，移除 PD-DRAM 补丁、豁免 3330 与 TILERT_PD_BUFFER_DEVICE
The previous commit moved the recipes to tilert==0.1.6.post1 and set
TILERT_PD_BUFFER_DEVICE=cuda. With the shipped post1 wheel the host-buffer
route is dead code, so this removes it rather than carrying an unused engine
patch:
- benchmarks/multi_node/amd_utils/patches/tilert-0.1.6-pd-buffers-in-dram.patch
and docs/waiver/3330.md are deleted; setup_deps.sh no longer installs `patch`
or applies anything to the pip-installed wheel, and the 0.1.6 version-pin
guard that protected the patch goes with it. Nothing in either image is
modified any more, so the engine-patch waiver has no subject.
- TILERT_PD_BUFFER_DEVICE is removed from both recipes, server_tilert.sh
(check_env_vars, the cpu|cuda case, the status line, the decode_server
--pd-buffer-device flag and the connector's tilert_pd_buffer_device extra
config key), job.slurm (check_env_vars and the -e forward) and setup_deps.sh.
The shipped decode_server has no such flag (strict parse_args), and
prefill_connector reads its extra config with .get(), so the key was inert.
- configs/amd-master.yaml: router metadata 0.1.6 -> 0.1.6.post1. pd_router.py
is one of the six files post1 changes: the fixed 600 s request timeouts to
vLLM and decode become TILERT_PD_HTTP_TIMEOUT_S (default 3600).
Verified against the two PyPI wheels rather than the release notes
(tilert-0.1.6 sha256 59276124..., tilert-0.1.6.post1 sha256 d4de62b0...,
uploaded 2026-09-22T08:14:17Z). post1 differs in tilert/pd_vllm/{transport,
receive_server, prefill_connector, pd_router}.py and profiles/{base,
mla_nsa}.py. Both PD buffers become nshards = min(TILERT_PD_SHARDS or 8,
torch.cuda.device_count()) per-device tensors of ceil(num_layers / nshards)
layers each, layer lid on shard lid % nshards; each shard is registered with
mooncake on its own device (location hip:<i % rails>), hello_layout carries
per-shard bases and rdma_plan writes layer by layer to the matching remote
shard. The glm5_2 profile has 79 layers, so the heaviest shard holds 10 of
them: 99.06 GiB x 10/79 = 12.54 GiB per card, the figure the recipe comments
quote. mla_nsa.convert() now walks the shard that received each layer and
synchronizes every device it touched.
Recipe comments and the GPU_MEM_UTIL derivation are reworded for the sharded
layout (the 12.54 GiB staging shard sits outside vLLM's budget next to the
~6.3 GiB non-torch baseline). No benchmark was rerun for this commit; the
sweep on this PR is its validation.
上一提交已将配方切到 tilert==0.1.6.post1 并设 TILERT_PD_BUFFER_DEVICE=cuda。对
正式发布的 post1 wheel 而言，主机内存缓冲那条路已是死代码，因此本提交将其整体移除，
而不是继续携带一个不再使用的引擎补丁：
- 删除 patches/tilert-0.1.6-pd-buffers-in-dram.patch 与 docs/waiver/3330.md；
setup_deps.sh 不再安装 `patch`、不再对 pip 安装的 wheel 做任何修改，为补丁服务的
0.1.6 版本锁定检查一并删除。两侧镜像内不再有任何被改动的文件，引擎补丁豁免失去对象。
- 从两份配方、server_tilert.sh（check_env_vars、cpu|cuda 分支、状态行、decode_server
的 --pd-buffer-device 参数、connector 的 tilert_pd_buffer_device 额外配置键）、
job.slurm（check_env_vars 与 -e 转发）和 setup_deps.sh 中移除 TILERT_PD_BUFFER_DEVICE。
正式 decode_server 没有该参数（parse_args 严格模式），prefill_connector 用 .get()
读取额外配置，因此该键此前实际无效。
- configs/amd-master.yaml：router 元数据 0.1.6 -> 0.1.6.post1。pd_router.py 是 post1
改动的六个文件之一：对 vLLM 与 decode 固定的 600 秒请求超时改为
TILERT_PD_HTTP_TIMEOUT_S（默认 3600）。
以上依据 PyPI 上两个 wheel 的实际差异核对，而非发布说明（tilert-0.1.6 sha256
59276124...，tilert-0.1.6.post1 sha256 d4de62b0...，上传于 2026-09-22T08:14:17Z）。
post1 改动 tilert/pd_vllm/{transport, receive_server, prefill_connector,
pd_router}.py 与 profiles/{base, mla_nsa}.py：两个 PD 缓冲改为 nshards =
min(TILERT_PD_SHARDS 或 8, torch.cuda.device_count()) 个逐卡张量，每个含
ceil(num_layers / nshards) 层，第 lid 层落在第 lid % nshards 个分片；各分片在所在
卡上向 mooncake 注册（location hip:<i % rails>），hello_layout 携带各分片基址，
rdma_plan 逐层写入对应的远端分片。glm5_2 profile 共 79 层，最重的分片含 10 层：
99.06 GiB x 10/79 = 12.54 GiB/卡，即配方注释所引数字。mla_nsa.convert() 改为在收到
该层的分片上处理并同步所有涉及的设备。
配方注释与 GPU_MEM_UTIL 的推导按分片布局改写（12.54 GiB 暂存分片与约 6.3 GiB 非
torch 基线同在 vLLM 预算之外）。本提交未重跑基准；本 PR 的 sweep 即其验证。
…d export PORT for lm-eval [skip-sweep] / AgentX 路径在 EVAL_ONLY/RUN_EVAL 下运行 GSM8K，并为 lm-eval 导出 PORT server_tilert.sh dispatched every TILERT_IS_AGENTIC=1 job to run_agentic_replay and ignored RUN_EVAL/EVAL_ONLY there. The eval rows the all-evals label generates for a conc-1 agentic config are eval-only, so such a job would have replayed AgentX traces for DURATION seconds, produced no results*.json and failed the template's eval-only check. The fixed-sequence path had a second gap: benchmark_lib's run_lm_eval does check_env_vars OPENAI_API_KEY PORT, and only run_agentic_replay exported PORT, so run_lm_eval_on_router would have exited at that check. - The prefill/router role now sends agentic EVAL_ONLY jobs through run_bench_and_eval (which already skips the throughput loop under EVAL_ONLY and runs the eval), and runs run_lm_eval_on_router after a replay when RUN_EVAL is set. - run_lm_eval_on_router exports PORT=$ROUTER_PORT. lm-eval names the model from MODEL_NAME, which start_prefill already registers with vLLM next to SERVED_MODEL_NAME, so no model-name change is needed; MODEL stays the local HF directory for compute_eval_context_length. - Artifact staging mirrors server_sglang.sh: rewrite_lm_eval_meta_env after the TP/EP/CONC exports, then append_lm_eval_summary unless run_eval already staged the agentic eval-only shape itself. start_decode already disables the simulated acceptance length under RUN_EVAL/EVAL_ONLY, so evals run with real MTP verification as on the SGLang path. [skip-sweep]: the changelog/MODELS commit that follows is the head the sweep should run on. server_tilert.sh 此前把所有 TILERT_IS_AGENTIC=1 的作业都交给 run_agentic_replay， 并在该路径忽略 RUN_EVAL/EVAL_ONLY。all-evals 标签为并发 1 的 agentic 配置生成的评测行 是 eval-only 的，这样的作业会回放 AgentX 轨迹 DURATION 秒、不产生 results*.json， 并在模板的 eval-only 检查处失败。固定序列路径还有另一处缺口：benchmark_lib 的 run_lm_eval 会 check_env_vars OPENAI_API_KEY PORT，而只有 run_agentic_replay 导出了 PORT，run_lm_eval_on_router 会在该检查处退出。 - prefill/router 角色现将 agentic 的 EVAL_ONLY 作业交给 run_bench_and_eval（它已在 EVAL_ONLY 下跳过吞吐循环并执行评测），并在设置 RUN_EVAL 时于回放结束后调用 run_lm_eval_on_router。 - run_lm_eval_on_router 导出 PORT=$ROUTER_PORT。lm-eval 以 MODEL_NAME 命名模型，而 start_prefill 已将其与 SERVED_MODEL_NAME 一并注册到 vLLM，无需改动模型名；MODEL 仍为本地 HF 目录，供 compute_eval_context_length 使用。 - 产物暂存对齐 server_sglang.sh：在 TP/EP/CONC 导出之后 rewrite_lm_eval_meta_env， 再 append_lm_eval_summary，除非 run_eval 已自行暂存 agentic eval-only 这一形态。 start_decode 已在 RUN_EVAL/EVAL_ONLY 下关闭模拟接受长度，因此评测与 SGLang 路径一样 使用真实 MTP 验证。[skip-sweep]：随后的变更记录/MODELS 提交才是 sweep 应运行的 head。
…post1 sharded-PD recipes (#3366) / 为 0.1.6.post1 分片 PD 配方追加变更记录并更新 MODELS.md 链接（#3366） Appends one perf-changelog.yaml entry covering glm5.3-fp8-mi355x-tilert and glm5.3-fp8-mi355x-tilert-agentic for this PR: tilert 0.1.6.post1's layer-sharded PD buffers in place of the PD-DRAM patch, the removal of the patch, waiver and TILERT_PD_BUFFER_DEVICE, the router metadata bump and the AgentX-path eval fix, with CrimsonDump's 2x8 MI350X measurements attributed. Byte-preserving tail append; historical entries (including the #3330 ones this branch carries) are untouched. MODELS.md / MODELS_zh.md point the GLM-5.3 row at #3366, the PR that would introduce the model if this branch merges instead of #3330. 为本 PR 在 perf-changelog.yaml 末尾追加一条覆盖 glm5.3-fp8-mi355x-tilert 与 glm5.3-fp8-mi355x-tilert-agentic 的记录：以 tilert 0.1.6.post1 的按层分片 PD 缓冲 取代 PD-DRAM 补丁，移除补丁、豁免与 TILERT_PD_BUFFER_DEVICE，升级 router 元数据， 并修复 AgentX 路径的评测；所引 2x8 MI350X 数据注明出自 CrimsonDump。按字节保留原 文件尾部追加；历史条目（含本分支携带的 #3330 条目）不作改动。MODELS.md / MODELS_zh.md 的 GLM-5.3 行改为指向 #3366，即若本分支而非 #3330 合入时引入该模型的 PR。
…x-tilert-sharded-pd # Conflicts: # configs/amd-master.yaml # perf-changelog.yaml
…ly / 移除 GLM-5.3 MI355X 8k1k 配置，仅保留 AgentX Remove the glm5.3-fp8-mi355x-tilert fixed-seq-len entry, its recipe benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh, and the fixed-sequence bench loop in server_tilert.sh, which now runs agentic-coding only: EVAL_ONLY waits for the router and runs GSM8K, otherwise the AgentX replay runs and RUN_EVAL follows it. The 8k1k changelog entries and MODELS rows are dropped; the entries that named both keys now name the agentic key only.
…ode [skip-sweep] tilert 0.1.6.post1 resets the decode sequence and copies the whole prompt's KV into all eight rank caches on every request, serially on the default streams. On #3366's AgentX run that inject step is 2.4 s at p50 and 4.3 s at p90, the largest part of TTFT, although 200 of 239 turns extend the previous request's prompt by a few hundred tokens. patches/tilert-0.1.6.post1-pd-prefix-reuse.patch, applied by setup_deps.sh on both ranks when the recipe sets TILERT_PD_PREFIX_REUSE=1: - glm5_rocm_engine.inject keeps the last prompt's ids and KV, and copies only the rows from the page before the first differing token onward. The kept rows are checked byte for byte against the fresh transfer on a sample of positions on every rank and layer; a mismatch falls back to a full copy. The index-key caches are always rewritten in full. - The KV copies fan out on one stream per device pair instead of the default streams, which torch fences on both devices per copy. - receive_server asks for the prompt ids in its hello and the prefill connector sends them when asked; decode_server hands them to inject.
Contributor
|
Thanks for the contribution!
中文感谢你的贡献！
|
…KV prefix reuse patch docs/waiver/3376.md documents the tilert 0.1.6.post1 PD prefix-reuse patch, states that the unpatched wheel runs this benchmark (#3366) so the patch is a performance change, and gives the removal plan. The appended perf-changelog entry covers glm5.3-fp8-mi355x-tilert-agentic.
Oseltamivir
marked this pull request as ready for review
September 23, 2026 03:29
Oseltamivir
requested review from
1am9trash,
Ankur-singh,
billishyahao,
chunfangamd,
seungrokj and
yctseng0211
as code owners
September 23, 2026 03:29
Contributor
|
Claude finished @Oseltamivir's task in 5m 7s —— View job Review of PR #3376
LGTM - no blocking issues found. Scope: I reviewed the two commits on top of #3366's head Not verified (for the author to watch in the first sweep): I could not download the |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Comment on lines
+16
to
+19
| # key = num_speculative_tokens (MTP level); value = golden AL | ||
| glm-5.3-fp8: | ||
| thinking_on: | ||
| 3: 2.99 |
Contributor
There was a problem hiding this comment.
🔴 Adds a GLM-5.3 golden AL entry (glm-5.3-fp8/thinking_on/3: 2.99) that is not measured on GLM-5.3 but copied from GLM-5.2, so the simulated-MTP-acceptance safeguard silently accepts it instead of failing. AGENTS.md says missing golden curves must fail before submission and forbids working around that with a guessed/borrowed acceptance length, but this file makes the lookup in server_tilert.sh's start_decode succeed with GLM-5.2's value. Fix: do not commit a stand-in AL value under the target model's key; either leave the model out of the golden curve file so the existing missing-curve check fails loudly, or gate the entry so it cannot be read as a measured GLM-5.3 value until a real SPEED-Bench run exists.
Why this was flagged
server_tilert.sh's start_decode (benchmarks/multi_node/amd_utils/server_tilert.sh) looks up golden_al_distribution/glm5.3_mtp.yaml keyed by MODEL_PREFIX=glm5.3, THINKING_MODE, DECODE_MTP_SIZE=3 and gets value 2.99, which is GLM-5.2's SPEED-Bench measurement (run 28058352479), not GLM-5.3's. TILERT_SIMULATE_ACC_LEN is then exported and used to drive simulated MTP acceptance for the whole AgentX sweep. AGENTS.md:75 says 'Missing golden curves or unmeasured draft lengths fail before submission... do not work around the error with a guessed or hard-coded acceptance length' -- this file does exactly that by supplying a borrowed value instead of leaving the entry absent, so the fail-before-submission check in start_decode never triggers. Any reported TTFT/ITL/throughput for glm5.3-fp8-mi355x-tilert-agentic reflects GLM-5.2's acceptance behavior, not GLM-5.3's, though the config's own comment already flags this and asks maintainers for a call.
Verification: normal, acknowledged in diff (the note holds factually but does not resolve the defect). The candidate is accurate: this PR commits golden_al_distribution/glm5.3_mtp.yaml whose only cell (glm-5.3-fp8 / thinking_on / 3: 2.99) is, by its own header (lines 1-14), "copied from glm5.2_mtp.yaml; NOT measured on GLM-5.3" — GLM-5.2's SPEED-Bench value from run 28058352479, i.e. a borrowed/guessed…
| export ROUTER_TYPE=tilert-pd-router ROUTER_PORT=30000 PROXY_PING_PORT=36367 | ||
| export HEADNODE_PORT=20000 SERVER_PORT=2584 PROXY_STREAM_IDLE_TIMEOUT=300 | ||
| export ENABLE_METRICS=0 PREFILL_ROUTER_POLICY=random DECODE_ROUTER_POLICY=random | ||
| export FLUSH_DRAIN_TIMEOUT=120 CLEAR_CACHE_BETWEEN_CONC=1 |
Contributor
There was a problem hiding this comment.
🟡 (optional) CLEAR_CACHE_BETWEEN_CONC=1, newly set for tilert at runtime_settings.sh:83 to keep concurrency points measured cold, does nothing on this path. server_tilert.sh's conc loop (server_tilert.sh:441-459) never reads that flag, and trace_replay.sh's clear_kv_caches (trace_replay.sh:44-46) no-ops because SERVER_FLUSH_URLS_CSV is only ever set by server_sglang.sh, so the one long-lived decode_server (started once, server_tilert.sh:489) keeps this PR's new self._resident_ids KV-reuse state live across conc boundaries. Fix: clear or restart decode KV-reuse state between concurrency points in server_tilert.sh whenever CLEAR_CACHE_BETWEEN_CONC=1, or stop setting it for a path that ignores it; today's conc-list:[1] hides the gap but any future multi-conc tilert recipe inherits it.
Why this was flagged
Trigger: any ENGINE=tilert recipe on RUNNER_TYPE mi355x-amds whose scenario's conc-list has more than one value (this PR's own glm5.3-fp8-mi355x-tilert-agentic entry in configs/amd-master.yaml uses conc-list:[1], so it doesn't hit this yet). runtime_settings.sh:83 exports CLEAR_CACHE_BETWEEN_CONC=1 for that path, matching the pre-existing sglang line at runtime_settings.sh:18. But server_tilert.sh's for-loop over BENCH_MAX_CONCURRENCY (server_tilert.sh:441-459) never consumes CLEAR_CACHE_BETWEEN_CONC or calls any flush. trace_replay.sh:44-46's clear_kv_caches short-circuits with only a WARN when SERVER_FLUSH_URLS_CSV is unset, and that variable is populated only in server_sglang.sh:1098-1099, never for tilert. Because start_decode (server_tilert.sh:489) runs once before the whole conc loop, the decode engine's new self._resident_ids state (patches/tilert-0.1.6.post1-pd-prefix-reuse.patch) persists into the next concurrency point's first requests, unlike the SGLang path which does get flushed between points. Data correctness is unaffected (the patch's byte-verify still falls back on…
Verification: nit. The mechanism is real and correctly diagnosed, but no shipped recipe triggers it, so it is not a merge blocker. CLEAR_CACHE_BETWEEN_CONC=1 added for the tilert MI355X path at runtime_settings.sh:83 is dead config on that path: - server_tilert.sh never references CLEAR_CACHE_BETWEEN_CONC, SERVER_FLUSH_URLS_CSV, clear_kv_caches, or trace_replay (grep of the file returns nothing). Its… | nit.…
…er, pack prompt ids The first AgentX run on this PR (sweep 35812357318, Slurm job 45562) applied the patch on both ranks and the per-pair copy streams cut the full inject copy from 10.5 us/token to 1.78 us/token (565 ms instead of 3278 ms at 314k tokens). But every reuse attempt with a long common prefix failed verification (29 of 29), so every turn fell back to a full copy, and sending the prompt ids as a JSON list raised the decode's wire wait from 100-190 ms to 282 ms at p50. - Verification now flags each rank's kv and pe cache of every layer separately; caches whose sample matches keep their prefix and the rest are copied in full. - Mismatches are logged by layer, rank and kind, with an all-zero check. For the first three reuse attempts the same check also runs before reset_sequence(), to tell whether the reset or the previous decode changed the kept rows. - The prompt ids travel as base64-packed int32 (prompt_ids_i32); the list form for repetition penalties is unchanged.
Contributor
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=35836899099 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=35836899099 |
…MTP layer, delta-encode prompt ids The v2 diagnostics (sweep 35817675288, Slurm job 45577) showed why the byte-exact check rejected reuse: on 221 of 246 continuing AgentX turns layer 0 matched on every rank while layers 1-78 differed, before and after reset_sequence(), and no kept row was zeroed except the MTP layer, which the reset clears. vLLM recomputes part of the prefix with different chunking, so the kept KV is the same tokens' KV with different rounding. On the other 25 turns every main layer matched exactly. - Reuse now requires the token ids to match (as before), layer 0 to match the fresh transfer byte for byte on every rank, and five sampled deeper layers on ranks 0 and 1 to be within a relative L2 error of 0.25 (TILERT_PD_REUSE_REL_TOL); otherwise every row is copied. The worst relative error is logged and added to REQSTAT. - The MTP layer is always copied in full. - Verification checks 36 caches instead of 1262 (it cost ~170 ms). - Prompt ids are sent as a suffix over the previous request's ids when the decode hello's hash shows the decode node still holds them.
…ager server_tilert.sh hardcoded --enforce-eager on the vLLM prefill. It is now the PREFILL_ENFORCE_EAGER recipe knob (0 or 1, validated, forwarded by job.slurm); the AgentX recipe sets 0 so vLLM uses torch.compile and CUDA graphs. On this PR's AgentX runs the prefill-side part of TTFT stayed at 1.8 s p50 / 4.4 s p90 after the decode inject dropped to 0.13 s p50. The TileRT prefix-reuse patch is unchanged from 865925a, so this run compares against sweep 35827481885.
…weep] Restore the tree of a0caa02 (#3366's head, the unpatched tilert==0.1.6.post1 recipe whose AgentX path produced sweep 35741321941): drop the pd_vllm patch, docs/waiver/3376.md, TILERT_PD_PREFIX_REUSE and PREFILL_ENFORCE_EAGER, and the two changelog entries this PR appended. The parallel KV inject that cut AgentX TTFT from 5.76 s to 3.20 s at p50 goes upstream to TileRT instead; it is re-tested here unpatched once a TileRT release carries it. Prefix reuse (25 of 246 turns bit-identical; the rest differed by 0.25-0.36 relative error) and dropping --enforce-eager (no TTFT change over the first 184 requests) are not pursued.
Collaborator
Author
|
/reuse-sweep-run 35741321941 |
Oseltamivir
changed the title
adibarra
added a commit
that referenced
this pull request
Sep 25, 2026
…mage to v0.5.20-cu130 / 将 dsv4-fp4-b200-sglang-agentic-hicache-mtp 的 SGLang 镜像更新至 v0.5.20-cu130 (#3334) * feat(dsv4): update B200 SGLang AgentX HiCache MTP image to v0.5.20-cu130 Update dsv4-fp4-b200-sglang-agentic-hicache-mtp from lmsysorg/sglang:v0.5.19-cu130 to lmsysorg/sglang:v0.5.20-cu130 (sglang v0.5.20, build commit 94602c9c). SGLang v0.5.20 removed the deprecated --cuda-graph-max-bs alias (sgl-project/sglang#38375), so the recipe passes the same value through --cuda-graph-max-bs-decode. Model, topology, DSpark settings, HiCache settings and all points are unchanged. 将 dsv4-fp4-b200-sglang-agentic-hicache-mtp 的镜像从 lmsysorg/sglang:v0.5.19-cu130 更新为 lmsysorg/sglang:v0.5.20-cu130（sglang v0.5.20，构建提交 94602c9c）。SGLang v0.5.20 移除了已弃用的 --cuda-graph-max-bs 别名（sgl-project/sglang#38375），因此配方改用 --cuda-graph-max-bs-decode 传入同一数值。模型、拓扑、DSpark 设置、HiCache 设置及所有测试点保持不变。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(changelog): record the B200 DeepSeek-V4-Pro SGLang AgentX image update Append the perf-changelog entry for dsv4-fp4-b200-sglang-agentic-hicache-mtp moving to lmsysorg/sglang:v0.5.20-cu130 (PR #3334). 为 dsv4-fp4-b200-sglang-agentic-hicache-mtp 更新至 lmsysorg/sglang:v0.5.20-cu130 追加 perf-changelog 条目（PR #3334）。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(ci): verify only newly submitted signoffs (#3340) 仅对新提交的签核自动运行验证，编辑清单后需手动重试，并修正文档及验证器提示。 * perf(amd): enable ATOM DSpark K6 with RCCL DEP / ATOM 全并发启用 DSpark K6 与 RCCL DEP (#2912) * perf(amd): switch DSV4 AgentX to native RCCL DEP 将 DeepSeek-V4-Pro MI355X ATOM AgentX 的 c48 及以上测试切换到本地验证过的原生 RCCL DEP 配置，并保持低并发 TP 测试不变。 同步固定的 post-merge ATOM 镜像、EP8 元数据、关闭 TBO/EPLB、真实 MTP 接受率以及本地验证过的路由和 AIPerf 参数。 * docs(perf): link DSV4 RCCL DEP pull request 将 DeepSeek-V4-Pro RCCL DEP 性能变更记录中的占位链接替换为实际的 InferenceX PR 链接。 * fix(amd): trim DSV4 AgentX DEP overrides 精简 DeepSeek-V4-Pro AgentX RCCL DEP 配置，移除与 CLI 或公共默认值重复的环境变量，并补齐本地验证使用的 3600 秒 warmup grace。 * fix(amd): restore DSV4 AgentX scheduling controls Restore the request-equivalent weight, prefill delayer, and decode interval requested for the AgentX run. Remove the newly introduced terminal MTP overrides while keeping the rest of the cleanup unchanged. * fix(amd): remove redundant AgentX timeout overrides Keep the 3600-second agentic warmup allowance, but rely on the server keep-alive setting and AIPerf default benchmark grace period. * fix(amd): keep DSV4 state checkpoints at 8K Remove the DEP-only 32K override so both TP and DEP retain the original 8192-token state checkpoint interval. * perf(amd): capture dense DSV4 DEP decode graphs 为 DSV4-Pro 的 DPA/DEP 路径补齐小 batch CUDA graph，避免非二次幂 decode batch 向上填充造成的 attention、MoE 和 RCCL 无效计算。保留 TP 路径默认行为，并为高并发配置保留大 batch graph。 * perf(amd): include upstream DSV4 DEP fixes in AgentX Pin the September 12 ATOM nightly containing the RCCL EP sentinel fix and AITER stage2 tuning fix. Record the server's installed sources and bundled tuning rows in the AgentX artifacts while retaining dense decode graphs, FP8 KV, and the current scheduler settings. 固定到包含 RCCL EP sentinel 与 AITER stage2 调优修复的 9 月 12 日 ATOM nightly；保存服务端实际源码版本及镜像内 tuning 行，保留 dense decode graph、FP8 KV 和现行调度设置。镜像包含 AITER#4159，但本配方不启用其 BF16 FlyDSL paged SWA 分支。 * fix(ci): provide exact PR context to priority classifier Supply immutable base/head revisions and the permitted diff command, document read-only tool use, and allow 16 turns for structured classification. 为优先级分类器提供确切的 Base/Head 版本及允许执行的 Diff 命令，明确只读工具用法，并将结构化分类的轮数上限调整为 16。 * perf(amd): enable DSpark K6 across the ATOM AgentX sweep 将 PR #2912 的十个 ATOM AgentX 性能点切换为固定版本 0813 checkpoint、 DSpark K6/q7 和 golden AL 3.77；C256 全量 GSM8K 保留真实验证。 修复 draft_model 路由与共享缓存挂载，统一服务与客户端 tokenizer， 增加 DSpark head、66 个分片及 tokenizer 检查并记录运行身份。 保留既有镜像优化、原生 RCCL DEP 和 dense FULL graph 配置。 * fix(amd): preserve inference mode in DSpark TP MoE callbacks 修复 DSpark q7 触发 AITER TP 通信融合 MoE 补齐路径时的 inference tensor 原地写入错误。仅对固定源码应用 InferenceMode callback 补丁，保留通信融合与 graph，保存补丁哈希，并覆盖原始失败、补齐、对齐和重复应用场景。 * fix(amd): use upstream ATOM inference-mode fix 将 MI355X ATOM AgentX 配方切换到包含 ROCm/ATOM#2233 的 pr2233-4f3a808 镜像，并移除运行时 AITER 源码补丁、对应测试和补丁清单记录。 * perf(amd): use BF16 KV through concurrency 16 将 MI355X ATOM AgentX 的 C1、C2、C4、C8 和 C16 任务切换为 BF16 KV cache；C48 及以上 DEP 任务继续使用 FP8 KV cache。 * perf(amd): pin DSv4 ATOM AgentX to official nightly_202609161445 Restore run-sweep.yml to the PR base and replace the temporary rocm/atom-dev:pr2233-4f3a808 image with official nightly_202609161445, which includes the merged ROCm/ATOM#2233 inference-mode fix. 将 run-sweep.yml 还原到 PR base，并把临时镜像 pr2233-4f3a808 换成 包含已合入 ATOM#2233 修复的官方 nightly_202609161445。 * fix: restore missing config-keys in perf-changelog merge The main merge left the first PR #2912 changelog entry without its - config-keys: header, so process_changelog could not parse the added YAML. 修复合并后第一条 2912 changelog 条目缺少 - config-keys: 导致的 YAML 解析失败。 * docs: drop unrelated CI procedure changes 将两份 CI 流程文档恢复为 main 版本，使其不再出现在本 PR 的变更中。 * docs: align CI procedures with PR base 将中英文 CI 流程文档恢复为 PR 当前基础版本，确保它们不再出现在本 PR 的变更中。 * docs(perf): correct DSv4-Pro ATOM AgentX changelog image to nightly_202609161445 Align the perf-changelog entry with the recipe's actual image (rocm/atom-dev:nightly_202609161445 in configs/amd-master.yaml), replacing the stale nightly_202609071454 reference and its now-mismatched digest and build-commit details. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(perf): align dsv4-fp4-mi355x-atom-agentic-mtp changelog with PR 2912 Rewrite the dsv4-fp4-mi355x-atom-agentic-mtp entry to match the final recipe: EAGLE MTP -> DSpark (draft_model) on the DeepSeek-V4-Pro-0813 checkpoint, NUM_SPEC_TOKENS 3 -> 6 (K6), SPEC_DECODE_AL 2.49 -> 3.77, and the RCCL DEP band on TP8/DPA8/EP8. Correct the state checkpoint interval to STATE_CHECKPOINT_INTERVAL_TOKENS=8192 (was mis-stated as 32768) and note eval verifies real DSpark drafts, not MTP. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(perf): align dsv4-fp4-mi355x-atom-agentic-mtp changelog with PR 2912 Rewrite the dsv4-fp4-mi355x-atom-agentic-mtp entry to match the final recipe: EAGLE MTP -> DSpark (draft_model) on the DeepSeek-V4-Pro-0813 checkpoint, NUM_SPEC_TOKENS 3 -> 6 (K6), SPEC_DECODE_AL 2.49 -> 3.77, and the RCCL DEP band on TP8/DPA8/EP8. Correct the state checkpoint interval to STATE_CHECKPOINT_INTERVAL_TOKENS=8192 (was mis-stated as 32768) and note eval verifies real DSpark drafts, not MTP. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: seungrokj <144636725+seungrokj@users.noreply.github.com> Co-authored-by: Chun Fang <chun.fang@amd.com> Co-authored-by: seungrokj <seungrok.jung@amd.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * docs: define draft precision as serve-the-draft-as-it-ships (#3348) Rewrite the Draft-model precision rule in CONTRIBUTING.md, its Chinese mirror, the checklist item, and verifier Check 13 so the baseline is the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling. Forbid only submission-side changes that lower draft precision below that default. Add the Qwen FP8 embedded MTP head and the SGLang DSpark wo_a FP8-to-BF16 load conversion as worked examples of what is compliant. 将 CONTRIBUTING.md 中的 Draft 模型精度规则、其中文镜像、审阅清单条目以及 验证器检查 13 重写为：基线是随所服务 checkpoint 一同发布的 draft，保持其 存储精度，并采用锁定上游镜像的默认加载处理。仅禁止提交方将 draft 精度降到 该默认值以下的改动。新增 Qwen FP8 内嵌 MTP head 与 SGLang DSpark wo_a FP8 转 BF16 加载转换两个合规示例。 * Fix MI300X AMD partition and match its eight-node runner pool (#3350) Submit to compute-0 and retire mi300x-amd_08 from both inventory groups. Benchmark commands and resource settings are unchanged. * Consolidate deprecated configs and remove retired runtime support (#3349) * chore: consolidate deprecated configs by vendor * style: separate deprecated configs with blank lines * chore: remove retired model routing and stale agent guidance * chore: archive retired server registry entries and fix deprecation guidance * chore: archive remaining retired disaggregated registry variants * chore: remove retired DSV4 fixed-sequence helper branches * docs: correct remaining GLM AgentX policy comment * Add B300 Qwen3.5 397B FP8 disaggregated AgentX recipes (#3268) * feat: add B300 Qwen3.5 FP8 disaggregated AgentX starter * chore: link B300 starter performance changelog * perf: colocate B300 Qwen3.5 prefill and decode workers * fix: match B300 Mamba cache flag to pinned SGLang * perf: use latest SGLang nightly for B300 Qwen3.5 FP8 * fix: use Dynamo compatibility with latest SGLang * fix: use intra-node NVLink for colocated B300 PD * perf: select measured B300 FP8 HiCache PD candidate * fix: propagate failed B300 Slurm runs after collecting artifacts * Prepare expanded measured B300 FP8 qualification curve * perf(b300): qualify measured concurrency 48 PD point * perf(b300): preserve existing aggregate publication scopes * style(b300): normalize final config newline * fix: require validated B300 Qwen3.5 AgentX power telemetry * fix: bind B300 power windows to matrix concurrency * perf: qualify B300 FP8 write-through PD at concurrency 96 * style: format B300 power contract fixture * fix: declare the B300 AgentX formal power window contract * docs: consolidate the B300 FP8 performance changelog * Avoid unverified host-memory pressure for B300 H298 recipes * fix: allow complete B300 AgentX runs on eligible nodes * perf: retain B300 FP8 candidates supported by canonical qualification * feat(models): acquire verified original BF16 MTP shards atomically * test(models): reject corrupted provenance and unexpected asset files * feat(b300): stage original BF16 MTP assets for Qwen FP8 recipes * feat(b300): prepare C48 disaggregation with 64 decode slots * perf(b300): add screened TP2 EP2 write-through candidate * perf(b300): select measured C24 decode capacity * perf: expand B300 FP8 disaggregated AgentX frontier * fix: allow full B300 TP2 sweep startup and drain time * perf: select B300 1P2D throughput frontier point * perf: select B300 1P2D C32 frontier point * perf: add B300 1P2D C64 frontier point * perf: refine B300 FP8 disaggregation frontier candidates * Refine B300 FP8 frontier with decode cache and DP attention * perf: refine B300 FP8 disaggregated frontier * perf(b300): qualify six FP8 disaggregated frontier candidates * perf(b300): qualify C4 latency endpoint and drop regressed C40 * perf(b300): select basic DCGM counters for Qwen AgentX * perf(b300): add higher-capacity C48 FP8 disaggregation candidate * perf: expand B300 FP8 sweep with screened C12 and C44 points * perf: add screened B300 FP8 C56 ReplaySSM capacity point * test: remove B300 SRT status test * test: remove MTP acquisition test * fix(b300): use the official FP8 embedded MTP head * fix(b300): disable inherited AgentX power collection * fix(b300): narrow node exclusion and simplify recipe metadata * [Klaud Cold] Update qwen3.5-fp8-h200-sglang-agentic-hicache-mtp SGLang image to nightly-dev-cu13-20260916-c9a8fba9 / 将 qwen3.5-fp8-h200-sglang-agentic-hicache-mtp 的 SGLang 镜像更新至 nightly-dev-cu13-20260916-c9a8fba9 (#3200) * perf: update qwen3.5-fp8-h200-sglang-agentic-hicache-mtp SGLang image to nightly-dev-cu13-20260916-c9a8fba9 Move the H200 Qwen3.5 FP8 AgentX HiCache MTP recipe from lmsysorg/sglang:nightly-dev-cu13-20260914-4358a161 to lmsysorg/sglang:nightly-dev-cu13-20260916-c9a8fba9 (sgl-project/sglang@c9a8fba9, index digest sha256:509c2742b9441fe9182ae3dd7dac8840ba48e03d2d44b7b24e80d428f117fd85). The 09-16 nightly is the first cu13 build carrying sgl-project/sglang#39516, the HiCache startup fallback for the pinned sglang-kernel 0.4.7 wheel that broke every registered host pool on the 09-15 build. Recipe script, MTP settings, HiCache arms and the concurrency grid are unchanged. 将 H200 Qwen3.5 FP8 AgentX HiCache MTP 配方的 SGLang 镜像从 lmsysorg/sglang:nightly-dev-cu13-20260914-4358a161 更新为 lmsysorg/sglang:nightly-dev-cu13-20260916-c9a8fba9（sgl-project/sglang@c9a8fba9）。 09-16 nightly 是首个包含 sgl-project/sglang#39516 的 cu13 构建，该修复为固定的 sglang-kernel 0.4.7 wheel 提供 HiCache 启动回退，09-15 构建中所有已注册主机池因此无法启动。 配方脚本、MTP 设置、HiCache 分支及并发网格保持不变。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * perf: add changelog entry for the qwen3.5-fp8-h200-sglang-agentic-hicache-mtp image bump Append the perf-changelog entry for moving qwen3.5-fp8-h200-sglang-agentic-hicache-mtp to lmsysorg/sglang:nightly-dev-cu13-20260916-c9a8fba9 after the targeted smoke run (35135979333) passed the c2 throughput point and the c24 gsm8k eval. 为 qwen3.5-fp8-h200-sglang-agentic-hicache-mtp 更新至 lmsysorg/sglang:nightly-dev-cu13-20260916-c9a8fba9 追加 perf-changelog 条目； 定向冒烟运行（35135979333）已通过 c2 吞吐点与 c24 gsm8k 评测。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: adibarra <93070681+adibarra@users.noreply.github.com> * [Klaud Cold] Update minimaxm3-fp8-mi300x-vllm-agentic-mtp vLLM ROCm image to v0.29.0 / 将 minimaxm3-fp8-mi300x-vllm-agentic-mtp 的 vLLM ROCm 镜像更新至 v0.29.0 (#3063) * feat: update minimaxm3-fp8-mi300x-vllm-agentic-mtp vLLM ROCm image to v0.29.0 Move the MI300X MiniMax-M3 MXFP8 AgentX EAGLE3 recipe from the unstable commit nightly vllm/vllm-openai-rocm:nightly-d9105ea8001e0a6d77a96327d17515bb5791fb36 to the v0.29.0 release image vllm/vllm-openai-rocm:v0.29.0 (digest sha256:e5e47f6aaab675c252c381f0dac237b31b10d87bb74d092b07fb4065efd7f5a1). Recipe, search space and evals are unchanged. 将 MI300X MiniMax-M3 MXFP8 AgentX EAGLE3 配方的镜像从不稳定的提交 nightly vllm/vllm-openai-rocm:nightly-d9105ea8001e0a6d77a96327d17515bb5791fb36 更新为 v0.29.0 发布镜像 vllm/vllm-openai-rocm:v0.29.0 （digest sha256:e5e47f6aaab675c252c381f0dac237b31b10d87bb74d092b07fb4065efd7f5a1）。 配方、搜索空间与评测保持不变。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: add perf-changelog entry for the minimaxm3-fp8-mi300x-vllm-agentic-mtp v0.29.0 image bump Record the vLLM ROCm image move from nightly-d9105ea8001e0a6d77a96327d17515bb5791fb36 to the v0.29.0 release image for PR #3063 after the smoke run passed. 为 PR #3063 记录 vLLM ROCm 镜像从 nightly-d9105ea8001e0a6d77a96327d17515bb5791fb36 更新至 v0.29.0 发布镜像的 perf-changelog 条目（冒烟运行已通过）。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: shorten PR 3063 performance changelog 缩短 PR 3063 的性能变更日志条目。 * docs: keep PR 3063 changelog concise 保持 PR 3063 性能变更日志简洁。 --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: adibarra <93070681+adibarra@users.noreply.github.com> * [Klaud Cold] Update minimaxm3-fp8-mi300x-vllm-agentic-mtp vLLM ROCm image to v0.29.0 / 将 minimaxm3-fp8-mi300x-vllm-agentic-mtp 的 vLLM ROCm 镜像更新至 v0.29.0 (#3063) * feat: update minimaxm3-fp8-mi300x-vllm-agentic-mtp vLLM ROCm image to v0.29.0 Move the MI300X MiniMax-M3 MXFP8 AgentX EAGLE3 recipe from the unstable commit nightly vllm/vllm-openai-rocm:nightly-d9105ea8001e0a6d77a96327d17515bb5791fb36 to the v0.29.0 release image vllm/vllm-openai-rocm:v0.29.0 (digest sha256:e5e47f6aaab675c252c381f0dac237b31b10d87bb74d092b07fb4065efd7f5a1). Recipe, search space and evals are unchanged. 将 MI300X MiniMax-M3 MXFP8 AgentX EAGLE3 配方的镜像从不稳定的提交 nightly vllm/vllm-openai-rocm:nightly-d9105ea8001e0a6d77a96327d17515bb5791fb36 更新为 v0.29.0 发布镜像 vllm/vllm-openai-rocm:v0.29.0 （digest sha256:e5e47f6aaab675c252c381f0dac237b31b10d87bb74d092b07fb4065efd7f5a1）。 配方、搜索空间与评测保持不变。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: add perf-changelog entry for the minimaxm3-fp8-mi300x-vllm-agentic-mtp v0.29.0 image bump Record the vLLM ROCm image move from nightly-d9105ea8001e0a6d77a96327d17515bb5791fb36 to the v0.29.0 release image for PR #3063 after the smoke run passed. 为 PR #3063 记录 vLLM ROCm 镜像从 nightly-d9105ea8001e0a6d77a96327d17515bb5791fb36 更新至 v0.29.0 发布镜像的 perf-changelog 条目（冒烟运行已通过）。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: shorten PR 3063 performance changelog 缩短 PR 3063 的性能变更日志条目。 * docs: keep PR 3063 changelog concise 保持 PR 3063 性能变更日志简洁。 --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: adibarra <93070681+adibarra@users.noreply.github.com> * [Klaud Cold] Update glm5.2-fp8-mi325x-sglang-agentic-mtp SGLang ROCm image to v0.5.19-rocm720-mi30x / 将 glm5.2-fp8-mi325x-sglang-agentic-mtp 的 SGLang ROCm 镜像更新至 v0.5.19-rocm720-mi30x (#2980) * chore: bump glm5.2-fp8-mi325x-sglang-agentic-mtp SGLang image to v0.5.19-rocm720-mi30x Update the GLM-5.2 FP8 MI325X SGLang AgentX MTP family from lmsysorg/sglang:v0.5.16-rocm720-mi30x to the v0.5.19 release image lmsysorg/sglang:v0.5.19-rocm720-mi30x (Docker Hub digest sha256:a0ffcdd013af6d79c9b7c4348c42b14a79a48f6c43dd3ca0235ec3802da45f75). Model, precision, topology, speculative settings, workloads and recipe script are unchanged. 将 GLM-5.2 FP8 MI325X SGLang AgentX MTP 配方的镜像从 lmsysorg/sglang:v0.5.16-rocm720-mi30x 更新为 v0.5.19 发布镜像 lmsysorg/sglang:v0.5.19-rocm720-mi30x。模型、精度、拓扑、投机解码参数、 工作负载与配方脚本保持不变。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore: add perf-changelog entry for glm5.2-fp8-mi325x-sglang-agentic-mtp image bump Append the changelog entry for updating the GLM-5.2 FP8 MI325X SGLang AgentX MTP image to lmsysorg/sglang:v0.5.19-rocm720-mi30x (PR #2980). 为 glm5.2-fp8-mi325x-sglang-agentic-mtp 镜像更新至 lmsysorg/sglang:v0.5.19-rocm720-mi30x 追加性能变更日志条目（PR #2980）。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: shorten PR 2980 performance changelog 缩短 PR 2980 的性能变更日志条目。 --------- Co-authored-by: Klaud-Cold <klaud-cold@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: adibarra <93070681+adibarra@users.noreply.github.com> * [Klaud Cold] Add MI325X TP4 and TP2 DeepSeek-V4.1-Flash vLLM AgentX arms with Engram host offload / 新增 Engram 主机卸载的 MI325X TP4 与 TP2 DeepSeek-V4.1-Flash vLLM AgentX 臂 (#3337) * [Klaud Cold] Add MI325X TP4 and TP2 DeepSeek-V4.1-Flash vLLM AgentX arms with Engram host offload / 新增 Engram 主机卸载的 MI325X TP4 与 TP2 DeepSeek-V4.1-Flash vLLM AgentX 臂 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Repin dsv41flash-mi325x-tp2-tp4-engram-offload to the ROCm 10.0 nightly channel / 将镜像重新固定到 ROCm 10.0 nightly 渠道 Match SemiAnalysisAI/InferenceX#3326, which moved MI355X to nightly-rocm100-3df4ae15 on the ROCm 10.0 channel. Same vLLM commit, different ROCm runtime. 与 SemiAnalysisAI/InferenceX#3326 保持一致：相同的 vLLM commit，ROCm 10.0 运行时。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: refresh PR #3337 for sweep reuse [skip-sweep] Sync with origin/main after the green sweep run 35637516947; the reuse gate authorizes that run on this head. 在绿色 sweep 运行 35637516947 之后与 origin/main 同步；reuse gate 在此 head 上授权该运行。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(ci): run the priority classifier on Opus 5 at low effort [skip-sweep] (#3360) * chore: bump srt-slurm to v2.18.0 and stream benchmark output (#3367) Move the utils/srt-slurm submodule from 984180e (#448) to v2.18.0 (2ac4eb1), 28 commits ahead, which adds benchmark.stream_output (NVIDIA/srt-slurm#483). Enable it for every srt-slurm recipe through setup_srt_slurm so benchmark.out is mirrored into the Actions job log while the client runs. The TileRT fork pin predates the option and is left unchanged. * Add DSV4 B200 Dynamo+SGLang AgentX configs / 添加 DSV4 B200 Dynamo+SGLang AgentX 配置 (#3257) * feat(dsv4): add B200 Dynamo SGLang AgentX configs * docs(perf): link DSV4 B200 AgentX PR * fix(dsv4): expand c256 decode KV headroom --------- Co-authored-by: hshrivastava-droid <hshrivastava@nvidia.com> Co-authored-by: Rohit Nagraj <rohitnagraj.99@gmail.com> * [Klaud Cold] Add MI300X TP4 DeepSeek-V4.1-Flash vLLM AgentX arm with Engram host offload / 新增 Engram 主机卸载的 MI300X TP4 DeepSeek-V4.1-Flash vLLM AgentX 臂 (#3336) * [Klaud Cold] Add MI300X TP4 and TP2 DeepSeek-V4.1-Flash vLLM AgentX arms with Engram host offload / 新增 Engram 主机卸载的 MI300X TP4 与 TP2 DeepSeek-V4.1-Flash vLLM AgentX 臂 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Repin dsv41flash-mi300x-tp2-tp4-engram-offload to the ROCm 10.0 nightly channel / 将镜像重新固定到 ROCm 10.0 nightly 渠道 Match SemiAnalysisAI/InferenceX#3326, which moved MI355X to nightly-rocm100-3df4ae15 on the ROCm 10.0 channel. Same vLLM commit, different ROCm runtime. 与 SemiAnalysisAI/InferenceX#3326 保持一致：相同的 vLLM commit，ROCm 10.0 运行时。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Drop the MI300X TP2 arm: measured infeasible on a 192 GB card / 移除 MI300X TP2 臂：在 192 GB 卡上实测不可行 Run 35671005506 reported "Available KV cache memory: -13.51 GiB" at TP2 concurrency 1 and the engine refused to start, even with the indexer buffer already halved to 4096 batched tokens. TP8 and TP4 are unaffected; TP4 passed at concurrency 1 and 32 in the same run. 运行 35671005506 在 TP2 并发 1 下报告 "Available KV cache memory: -13.51 GiB" 并拒绝启动，此时 indexer 缓冲区已减半至 4096。TP8 与 TP4 不受影响。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: refresh PR #3336 for sweep reuse [skip-sweep] Sync with origin/main after the green sweep run 35678619120; the reuse gate authorizes that run on this head. 在绿色 sweep 运行 35678619120 之后与 origin/main 同步；reuse gate 在此 head 上授权该运行。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: warn when sign-off lacks a reuse command (#3368) 中文：签署审核缺少授权复用命令时改为警告，同步裁定发布逻辑、双语文档和回归测试。 * [PowerX] retain healthy window measurements in audit sidecars / 在审计文件中保留健康窗口测量 (#3357) * fix: retain healthy power windows in audit sidecars 保留健康窗口的独立测量和逐点失败证据，保持包完整性检查、required-power 及发布判定不变。 * docs: clarify retained power diagnostics 精简中英文窗口保留说明，明确审计顶层逐 GPU 诊断字段不构成发布许可。 * [B200][SGLang][AgentX] Improve DeepSeek V4.1 Flash performance (#3346) * perf(b200): tune DeepSeek V4.1 host Engram layout on latest SGLang * chore(b200): link performance candidate to PR 3346 * perf(b200): add STP and retain native DSpark FP8 projections * fix(b200): import pinned images and handle strided draft inputs * perf(b200): compare GPU-resident Engram tables for STP * fix(b200): use shipped default DSpark implementation * perf(b200): service STP decodes between long prefills * perf(b200): probe TP2 DSpark with bounded prefill memory * perf(b200): schedule DSpark decode between long prefills * perf(b200): compare GPU-resident Engram for TP4 DSpark * fix(b200): release completed MXFP4 loader temporaries * fix(b200): mark loader patch utility executable * perf(b200): test more DSpark SWA prefix tails * fix(b200): avoid TP2 loader allocation fragmentation * perf(b200): compare DSpark prefill service at interval four * perf(b200): compare host DSpark with larger SWA retention * perf(b200): qualify TP2 DSpark across concurrency * perf(b200): test more TP2 SWA prefix retention * docs(b200): document TP2 loader memory patch waiver * perf(b200): select DSpark curve with concurrency-scaled SWA capacity * perf(b200): qualify stock TP2 loader with expandable allocator * Test stock B200 loader on latest SGLang nightly AI-assisted by gpt-6-astra with high reasoning. * Document latest-nightly stock B200 loader qualification * refactor: remove redundant B200 recipe startup checks * [B300][SGLang][AgentX] Improve DeepSeek V4.1 Flash performance (#3342) * feat(b300): add DeepSeek V4.1 SGLang nightly AgentX recipe * fix(b300): record recipe pull request provenance * feat(b300): add non-speculative SGLang AgentX arm * docs: consolidate performance changelog into one PR entry * fix(b300): preserve native DSpark FP8 draft projections * fix(b300): normalize strided native FP8 projection inputs * fix: restore default SGLang DSpark precision * perf(b300): expand DSpark parallelism and interleave decode * Launch B300 SGLang AgentX through an owned Slurm batch job * Match B300 batch PMI environment to non-MPI container step * Use supported PMIx container steps for B300 AgentX batch jobs * perf(b300): scope final AgentX sweep to DSpark candidates * Test B300 low-concurrency prefix retention and checkpoint prefetch * Scale B300 TP4 prefix retention within measured KV budget * Pin B300 to the September 22 SGLang nightly * Test larger B300 TP2 KV budget for high concurrency * Increase high-concurrency TP2 cache memory on B300 * Screen higher prefill duty for B300 TP2 DSpark * perf(amd): tune MiniMax-M3 ATOM AgentX memory, graphs, indexer CP, and offload / 调优 MiniMax-M3 ATOM AgentX 显存、图捕获、Indexer CP 与卸载 (#3189) * perf(minimaxm3-atom): raise util, capture the real batch sizes, add indexer CP and a TP2 offload curve Six changes to the MiniMax-M3 MXFP4 ATOM agentic recipe, all measured on 8xMI355X against the same cc-traces replay: - gpu-memory-utilization 0.90 -> 0.95. - Declare cudagraph capture sizes. The stock list pads a running batch of 9 to 16 and 17 to 32; this workload keeps only 22-41% of the offered concurrency in decode at any instant, so those are the batches that actually replay. - Enable ATOM_M3_INDEXER_CP on TP4 past CONC=24. Each rank scores all four index heads over 1/TP of the blocks instead of one head over all of them. - Restore EAGLE3 on the CONC=40/48 bands. The forward step is 37-41 ms with or without it, so dropping it handed back the whole acceptance-length multiplier. - Route CONC=20/25/30 to the chunk-256 + ATOM_SLRU offload tier and derive LMCACHE_LOOKUP_SERVER_WORKER_IDS from TP instead of hardcoding four ranks. - Spread TP2 offload ranks across NUMA nodes; pinning 256 GB/rank takes 45 min with both ranks on node 0 and 21 s with one per node. Matrix: drop TP2 c5, add the TP2 c20/25/30 offload curve. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(minimaxm3-atom): drop the dead per-band knobs The per-CONC case set MAX_NUM_SEQS, MAX_NUM_BATCHED_TOKENS and GPU_MEM_UTIL in all seven bands and two lines later overwrote all three unconditionally: 21 of its 49 assignments could never be read. What actually survived it was NUM_SPEC_TOKENS, SPEC_DECODE_AL and STATE_OFFLOAD_CPU_GIB, and after restoring EAGLE3 on 40/48 only CONC=56 differs on the first two. Collapsed to defaults plus three overrides; STATE_OFFLOAD_CPU_GIB keeps its exact per-band values. ATOM_ENABLE_REPLAYSSM is removed rather than collapsed: it is only read by gdn_attn, which gates the gated-delta-net path, and MiniMax-M3 is pure attention with no SSM layers, so the value never reached anything. Verified by replaying every point in the amd-master matrix (19 combinations of TP, CONC and offload) through a stubbed benchmark_lib and diffing the resulting server argv plus every exported env var: byte-identical apart from the removed variable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(minimaxm3-atom): one table per concurrency, and drop the tier M3 cannot use The hybrid offload tier is unreachable for this model. ATOM's M3 connector is PAGE-only -- the docstring says MiniMax-M3 has no recurrent state and no SLOT sidecar, and it raises if slot_regions is non-empty -- so OFFLOAD_STATE and the staging knobs behind STATE_OFFLOAD_CPU_GIB never applied. Its LMCACHE_CHUNK_SIZE is justified by "block-size(128) x dcp(8)", but this model's search space sets no dcp-size at all. Removed, along with the comment claiming M3 carries a per-request recurrent state. An unlisted concurrency now fails instead of silently taking a tier that does not fit. CONC=56 is removed for the same reason: nothing in the matrix runs it here (the 56 in this file's neighbourhood belongs to Kimi K3, which has its own script), and it was the only band that disabled the draft model. With it gone NUM_SPEC_TOKENS is constant, so the guard around SPEC_ARGS goes too. The remaining per-concurrency facts now live in one table. Previously the CP gate was a numeric comparison and the offload tier an enumeration, so adding a concurrency above 24 would silently enable indexer CP on a point nobody measured, while forgetting the tier failed loudly. Both are declared per band now: the loud failure is kept, the silent one becomes a missing speedup. Verified the same way as the previous cleanup: all 19 matrix points replayed through a stubbed benchmark_lib, server argv and every exported env var byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf(minimaxm3-atom): spread TP4 offload ranks across NUMA nodes, move to the rocm7 nightly Pinning 256 GB/rank with all four TP4 ranks on NUMA node 0 takes 27m08s (12:13:10 "NUMA mapping" -> 12:40:18 "Created backend: LocalCPUBackend"); the same pinning with TP2 split one rank per node takes 21 s. TP4 now selects 0,1,4,5 for the same reason TP2 selects 0,4. Only applied when the caller has not already pinned a device set, and only on the offload path -- a run without the CPU tier pins no host memory. The image moves to the rocm7 line of ATOM's nightly release (232f62e3), which carries the M3 dense-split work. Checked against the measured baseline rather than assumed: triton 3.7.0 and torch 2.10.0+rocm7.2.4.git3d3aa833 are identical to the image every number in this PR was measured on, which matters because the rocm10.0 line of the same build carries triton 3.8.0 and that version regresses this model's Gluon kernels. LMCache in the new image is 0.5.5rc3, not the 0.4.5 the matrix declared, so the two MiniMax-M3 offload rows are corrected; the other models keep their own versions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(perf): record MiniMax-M3 AgentX tuning sweep 记录 MiniMax-M3 ATOM AgentX 调优、矩阵变化、测试结果与已知 CPU DRAM 元数据限制，以便上游 PR 启动完整 AgentX sweep。 * docs(perf): link MiniMax-M3 tuning pull request 将 MiniMax-M3 ATOM AgentX 性能 changelog 的占位链接替换为上游 PR #3189。 * fix(ci): provide exact PR context to priority classifier Supply immutable base/head revisions and the permitted diff command, document read-only tool use, and allow 16 turns for structured classification. 为优先级分类器提供确切的 Base/Head 版本及允许执行的 Diff 命令，明确只读工具用法，并将结构化分类的轮数上限调整为 16。 * ci(agentx): run MiniMax-M3 throughput only 禁用本次 changelog 的自动 vendor eval，只保留 19 个 AgentX 吞吐任务，避免为每个并发额外展开评测。 * perf(minimaxm3-atom): extend indexer CP down to CONC=15 on TP4 The gate was CONC > 24, chosen from a kernel microbenchmark over uniform contexts. Three same-commit A/B pairs (1800 s per arm, TP4 + EAGLE3) put the real crossover below 15: CONC 15 interactivity p90 +3.7% ISL-normalised throughput +0.67% CONC 24 +12.3% +0.24% CONC 28 +16.7% +2.7% Monotone and never negative. Throughput is neutral everywhere (only CONC=28 clears the 0.6% noise floor) while interactivity rises with the running batch, which is what CP actually shortens. Prefix hit rates are identical within each pair (96.7/96.7, 97.5/97.5, 97.2/97.3), so the gain is not a caching artifact. CONC=20 is interpolated between the two measured points, not run on its own. Nothing below 15 is measured and the microbenchmark is negative at batch ~2, so the band stops there. CONC=20 appears in both search-space rows, so it is split out of the offload enumeration: TP4 gets CP without a tier, and the tp_size == 4 gate keeps CP off the TP2 offload curve. 25 and 30 are TP2-only and stay without a CP declaration, which would never take effect but would misreport the point. Verified with the 19-point argv+env replay: the only difference against the previous revision is ATOM_M3_INDEXER_CP="1" on tp4_c15, tp4_c20 and tp4_c24. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(perf): record expanded MiniMax-M3 indexer CP band 记录 TP4 C15/C20/C24 新增 indexer-only CP 的 A/B 证据，并说明 TP2 C20 仍由 TP-size gate 保持关闭。 * ci: restore default PR priority classifier Revert the 16-turn exact-diff prompt so this recipe PR does not change shared sweep CI. 还原默认的优先级分类器配置，避免本配方 PR 修改共享 sweep workflow。 Co-authored-by: Cursor <cursoragent@cursor.com> * perf(minimaxm3-atom): switch to official nightly_202609170640_minimax_m3 image Replace the developer-named ATOM nightly so this submission uses an official tag. 将 ATOM 镜像替换为不含开发者用户名的官方 tag nightly_202609170640_minimax_m3。 Co-authored-by: Cursor <cursoragent@cursor.com> * switch to 0917 nightly atom image * perf(minimaxm3): enable agentic eval by removing no-evals opt-out Drop `no-evals: true` from the minimaxm3-fp4-mi355x-atom-agentic-mtp changelog entry so plan generation no longer skips eval rows; the model's automatic minimax-vendor (minimax_m3_full) agentic eval then runs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(perf): correct MiniMax-M3 ATOM AgentX changelog image to nightly_202609171455 Align the perf-changelog entry with the recipe's actual image (rocm/atom-dev:nightly_202609171455 in configs/amd-master.yaml), replacing the stale nightly_202609170640_minimax_m3 reference in both the English and Chinese descriptions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(minimaxm3): size LMCache CPU pool per-rank from TOTAL_CPU_DRAM_GB/TP The cpu256 offload tier hardcoded LMCACHE_MAX_LOCAL_CPU_SIZE=256 per rank, independent of the matrix generator's TOTAL_CPU_DRAM_GB budget. Derive it as TOTAL_CPU_DRAM_GB / TP so the per-rank pool respects the declared budget and is TP-independent (node_DRAM * dram-utilization / 8). Bump the atom arm's dram-utilization 0.20 -> 0.687 so the offload arms land ~256 GB/rank. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update perf-changelog.yaml * docs(minimaxm3): expand ATOM AgentX changelog and record EAGLE3-GQA draft online-quant finding Update the minimaxm3-fp4-mi355x-atom-agentic-mtp changelog entry to reflect every change in SemiAnalysisAI/InferenceX#3189 and to record the EAGLE3-GQA draft online-quant verification: - Raise ATOM GPU memory utilization 0.90 -> 0.95 and declare a fine-grained CUDA graph ladder (capture sizes 1-64) sized for the 22-41%-of-CONC running batch, trimmed by ModelRunner to min(2*CONC, 8192). TP4 C24 A/B: throughput +9.2%, interactivity +50.7%, unchanged estimated graph memory. - Enable ATOM_M3_INDEXER_CP for TP4 C15/C20/C24/C28/C32/C40/C48; the TP-size gate (tp_size == sparse_num_index_heads == 4) keeps it off the TP2 C20 offload point. Restore EAGLE3 K3 (golden AL 2.78) at C40/C48. - Collapse per-band knobs into one case; add the TP2 LMCache DRAM-offload curve at C20/C25/C30 on the chunk-256 ATOM_SLRU tier; size LMCACHE_MAX_LOCAL_CPU_SIZE as TOTAL_CPU_DRAM_GB/TP; cover every rank in LMCACHE_LOOKUP_SERVER_WORKER_IDS. Remove TP2 C5 and the unreachable hybrid/state-offload and C56 branches. - NUMA-spread offload GPUs (0,4 for TP2; 0,1,4,5 for TP4), cutting TP2 host-memory pinning from 45 min to 21 s. Move to rocm/atom-dev:nightly_202609171455 and LMCache 0.5.5rc3+rocm7.2.4 on both offload arms. - Set dram-utilization to 0.687 so the recorded budget matches the real ~256 GB per-rank pinning (512 GB total TP2, 1,024 GB TP4), independent of TP, replacing the earlier 0.20 metadata that under-described the allocation. - Verified against ATOM commit b6e2a39433d73b586c366ca90e661f098eb463e3 that --online_quant_config ptpc_fp8 does not change the Inferact/MiniMax-M3-EAGLE3-GQA draft precision: eagle3_llama.py builds every draft linear (qkv_proj, o_proj, gate_up_proj, down_proj, fc) with no quant_config, so should_stream_online_quant short-circuits to False and OnlineQuantStreamer.maybe_create finds no candidates. The draft runs at native bf16; only the target MiniMax-M3 weights are online- quantized to ptpc_fp8, and exclude_layer applies to the target alone. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Wang Yiting <yitiwang@amd.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: billishyahao <bill.he@amd.com> Co-authored-by: seungrokj <144636725+seungrokj@users.noreply.github.com> Co-authored-by: seungrokj <seungrok.jung@amd.com> * feat(agentx): retune GLM5.2 FP4 MI355X ATOM recipe on _0921 docker (#3359) * feat(agentx): retune GLM5.2 FP4 MI355X ATOM recipe on _0921 docker Signed-off-by: zhuyuhua-v <yuhzhu@amd.com> * docs(glm5.2): note MTP block stays BF16 under --online_quant_config (PR #3359) Changelog for the GLM-5.2 FP4 MI355X ATOM AgentX recipe retune: - Track the updated GLM-5.2 standalone recipe (ROCm/ATOM PR 2345) on the MI355X ATOM AgentX config. - Bump the image from rocm/atom-dev:nightly_202609151450 to rocm/atom-dev:nightly_202609211553, which carries ROCm/ATOM PR 2345. - Drop the LMCache DRAM offload from the small-concurrency TP4 arm: conc [2, 4, 8, 10] moves from kv-offloading dram with the lmcache 0.4.5 backend to kv-offloading none, so those points serve off the native GPU prefix cache alone and no longer start an LMCache CPU tier. The TP4+DCP4 arm at conc [16, 24, 32, 40, 48] keeps LMCache at 256 GiB per rank (dram-utilization 0.171 unchanged), and the TP8 arm stays GPU-resident as before. - Add --index_cache_dtype fp4 and --block-size 64 to the ATOM server launch on every point. Neither flag was set before, so both took the ATOM default. - Drop ATOM_USE_FLYDSL_GATHER_KV_B_PROJ=0, which pinned gather_kv_b_proj to the Triton path on the DCP prefill-context and MTP verify paths; the updated recipe no longer sets it. - The MTP block data precision is not touched by --online_quant_config: the exclude_layer list adds "model.layers.78.*" so layer 78 (the GLM-5.2 MTP head) is skipped by online quantization and stays in native BF16, as it ships unquantized in amd/GLM-5.2-MXFP4. The expert excludes only reach layers 0-77, so without this exclude the whole MTP block would be online-quantized to ptpc_fp8 while the target's experts stay MXFP4. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Signed-off-by: zhuyuhua-v <yuhzhu@amd.com> Co-authored-by: seungrokj <144636725+seungrokj@users.noreply.github.com> Co-authored-by: seungrokj <seungrok.jung@amd.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * config(dsv41flash): sweep MI355X at TP=2 and TP=4 through concurrency 128 / MI355X DSv4.1-Flash 在 TP=2 与 TP=4 下扫描至并发 128 (#3326) * config(dsv41flash): sweep MI355X at TP=2 and TP=4 through concurrency 128 Add a TP=2 search space next to TP=4 for the MI355X DSv4.1-Flash AgentX entry, and extend both rows to concurrency 128. TP=2 became feasible once the Engram tables moved to host memory: at 47.2 GiB of device memory per rank they previously forced four GPUs per server, and two GPUs per server doubles the servers per node. The image moves to the ROCm 10 nightly and carries a placeholder tag until that build is published. The digest is pinned before the sweep runs, which is why the PR opens as a draft. 中文：为 MI355X DSv4.1-Flash AgentX 配置在 TP=4 之外新增 TP=2 搜索空间，并将两者 的并发扩展到 128。Engram 表已驻留主机内存，不再占用每 rank 47.2 GiB 设备内存 （此前这迫使每个服务器使用四张 GPU），因此 TP=2 可行，且每节点可容纳的服务器数 量翻倍。镜像切换到 ROCm 10 nightly，在该构建发布前暂用占位 tag；digest 固定后再 运行 sweep，因此本 PR 以 draft 形式提交。 Co-authored-by: Cursor <cursoragent@cursor.com> * config(dsv41flash): record the sweep PR link in the changelog 中文：在 changelog 中记录本次 sweep 的 PR 链接。 Co-authored-by: Cursor <cursoragent@cursor.com> * config(dsv41flash): let vLLM pick max_num_seqs for the MI355X AgentX recipe A fixed max_num_seqs of 128 caps in-flight sequences at the outer concurrency once the sweep reaches c128, leaving no headroom for AgentX subagent fan-out. Use the MI355X API-server default of 1024 instead, and derive the CUDA graph ceiling from max(2 * CONC, 128): 1024 through c64, 2048 at c128. Supersedes PR #3111, which carried this recipe change and was closed in favour of this sweep. 中文：扫描到 c128 时，固定的 max_num_seqs 128 会把在途序列数限制在外层并发上， 使 AgentX 子代理扇出没有余量。改用 MI355X API server 默认值 1024，并由 max(2 * CONC, 128) 推导 CUDA graph 上限：c64 及以下为 1024，c128 为 2048。 本次提交取代已关闭的 PR #3111（该 PR 原本包含此 recipe 改动）。 Co-authored-by: Cursor <cursoragent@cursor.com> * config(dsv41flash): keep the placeholder on the existing ROCm nightly channel Per #3215, the MI355X DSv4.1-Flash entry stays on the existing upstream ROCm nightly channel rather than moving to nightly-rocm100. Drop the rocm10 marker from the placeholder tag; the pin is filled in with the nightly this sweep actually runs on before dispatch. 中文：依据 #3215，MI355X DSv4.1-Flash 条目继续使用现有的上游 ROCm nightly 渠道， 不切换到 nightly-rocm100。占位 tag 去掉 rocm10 标记；派发前再填入本次 sweep 实际运行的 nightly。 Co-authored-by: Cursor <cursoragent@cursor.com> * Revert "config(dsv41flash): keep the placeholder on the existing ROCm nightly channel" This reverts commit 039194e50d92a80c847d770729e106237e2b36c4. The MI355X DSv4.1-Flash AgentX sweep runs on the ROCm 10 nightly after all, so restore the nightly-rocm100 placeholder. The pin is filled in with the published nightly before dispatch. 中文：还原提交 039194e50d92a80c847d770729e106237e2b36c4。 MI355X DSv4.1-Flash AgentX sweep 最终仍在 ROCm 10 nightly 上运行，因此恢复 nightly-rocm100 占位 tag。派发前再填入已发布的 nightly。 Co-authored-by: Cursor <cursoragent@cursor.com> * config(dsv41flash): pin the MI355X AgentX sweep to the ROCm 10 nightly Replace the placeholder with vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984, published 2026-09-21T05:50:03Z with digest sha256:eccb72b74b8c04ce7406d9212e200b129f58e24be642795a857f755a94fdb3a1. The tag is active and linux/amd64, and its manifest resolves from registry-1.docker.io at that digest. This sweep qualifies the new image; the previous green run on nightly-eed1f3d0 is not evidence for it. 将占位 tag 替换为 vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984， 该镜像于 2026-09-21T05:50:03Z 发布，digest 为 sha256:eccb72b74b8c04ce7406d9212e200b129f58e24be642795a857f755a94fdb3a1。 该 tag 状态为 active、架构为 linux/amd64，其 manifest 可从 registry-1.docker.io 按该 digest 解析。 本次 sweep 用于验证新镜像；此前在 nightly-eed1f3d0 上的绿色运行不能作为其证据。 Co-authored-by: Cursor <cursoragent@cursor.com> * config(dsv41flash): offload Engram to host memory at TP=2 on MI355X vllm-project/vllm#57491 widened the two ROCm is_cuda() gates to is_cuda_alike(), so the pinned nightly resolves an Engram config on gfx950 and offloads the tables to pinned host memory by default. Set cpu_offload explicitly per TP instead of taking that default. TP=2 needs the offload: the tables cost 47.2 GiB per rank at TP=4, so 94.4 GiB at TP=2, which does not fit beside half of the 511 GB checkpoint on a 288 GiB card. TP=4 keeps them resident so it stays comparable with the validated concurrency 1-32 run. vllm-project/vllm#57491 将 ROCm 上的两处 is_cuda() 判断放宽为 is_cuda_alike()，因此所固定的 nightly 在 gfx950 上会解析 Engram 配置， 并默认将表下放到锁页主机内存。这里按 TP 显式设置 cpu_offload，而不是 沿用该默认值。 TP=2 需要该下放：表在 TP=4 时每 rank 占用 47.2 GiB，TP=2 时为 94.4 GiB， 无法与 511 GB 检查点的一半同时放入 288 GiB 的单卡。TP=4 保持表常驻 GPU， 以便与已验证的并发 1-32 运行保持可比。 Co-authored-by: Cursor <cursoragent@cursor.com> * config(dsv41flash): disable SWA bounded replay on MI355X Every TP=2 and TP=4 point of run 35567570539 died with HSA_STATUS_ERROR_MEMORY_FAULT, taking the worker and then the engine core down and aborting AgentX warmup. The fault follows the first launch of _pad_replayed_slots_kernel in all 16 server logs, which is the first prefix hit carrying a replay start. vllm-project/vllm#56227 added SWA bounded replay, default on, after the eed1f3d0 pin and before nightly-rocm100-3df4ae153. It pads the replayed tokens' slots in the prefix-cacheable groups, but the window clamp it relies on landed in the FlashInfer and FlashMLA kernels; the ROCm sparse SWA path only gained the replay_start kwarg, and consumes it on the decode path while the fault is in prefill. Pass --no-swa-bounded-replay until ROCm clamps too. Prefix caching itself stays on. 运行 35567570539 的所有 TP=2 与 TP=4 数据点都以 HSA_STATUS_ERROR_MEMORY_FAULT 崩溃，先后带崩 worker 与 engine core， 并中止 AgentX warmup。16 份 server 日志中，该故障均紧随 _pad_replayed_slots_kernel 的首次启动，即首个带 replay start 的前缀命中。 vllm-project/vllm#56227 在 eed1f3d0 与 nightly-rocm100-3df4ae153 之间 引入了默认开启的 SWA bounded replay。它会填充被重放 token 在可前缀缓存 分组中的 slot，但其依赖的窗口钳制只落在 FlashInfer 与 FlashMLA 内核中； ROCm 稀疏 SWA 路径仅新增了 replay_start 参数，且只在 decode 路径消费它， 而故障发生在 prefill。在 ROCm 同样实现钳制之前传入 --no-swa-bounded-replay。前缀缓存本身保持开启。 Co-authored-by: Cursor <cursoragent@cursor.com> * config(dsv41flash): fix the MI355X KV split at high concurrency Concurrency 64 collapsed on both arms of run 35574132719 because the KV pool was too small to hold the AgentX working set. TP=2 fell to a 17.6% prefix cache hit rate, 187 s TTFT and 150 tok/s, against 94.8%, 1.3 s and 957 tok/s at c32. Two settings were spending device memory that the KV pool needed. The Engram tables stayed resident at TP=4. vllm-project/vllm#57491 widened the two ROCm is_cuda() gates to is_cuda_alike(), so gfx950 can now offload them to pinned host memory as every NVIDIA arm has since #2963. Measured here at TP=4 with 16384 batched tokens, resident tables leave 37.96 GiB of KV cache and 14.15x maximum concurrency at 1M context, against 84.54 GiB and 31.52x offloaded. The sparse-attention indexer and its companion per-rank buffers scale at roughly 4.4 MiB per batched token, so the upstream 16384 was the larger cost. Size it per TP instead, 4096 at TP=2 and 8192 at TP=4, and cap max_num_seqs at the shape graph capture already covers rather than the MI355X API-server default of 1024. TP=2 then keeps 79.34 GiB (39.44x) and TP=4 keeps 121.03 GiB (54.15x), against the B300 arm's 132.48 GiB (60.17x). B300 runs 8192 at TP=4 and the Blackwell TP=2 arms run 4096 (#3320, #3321). Sweep concurrency 64 and 128 to confirm the high end first. Concurrency 1-32 is backfilled once those land, since every point now runs the new memory split. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com> * config(dsv41flash): change the MI355X memory split only where KV ran short The previous commit applied one memory split per TP across every concurrency. Concurrency 1-32 never ran short of KV, so it paid for changes it did not need: offloaded Engram lookups go to pinned host memory over UVA, and a smaller prefill chunk costs TTFT. Scope each change to the points that were actually starved, which also leaves concurrency 1-32 and TP=4 c64 byte-identical to the settings run 35574132719 already measured on this image and flag set, so those points are combined with this sweep rather than re-run. Dividing measured KV tokens by concurrency gives the per-request budget and separates the failures cleanly. Every point that held had 232K or more; TP=2 c64 collapsed at 122K. TP=4 c1-c64 resident, 16384 14.83M tokens 232K and up TP=4 c128 offload, 8192 49.63M 388K TP=2 c1-c32 offload, 16384 7.84M 245K and up TP=2 c64 offload, 8192 24.68M 386K TP=2 c128 offload, 4096 33.34M 260K Probing the configurations directly also turned up two things. TP=2 c128 at 4096 with the API-server default of 1024 sequences faults during profiling with HSA_STATUS_ERROR_EXCEPTION at M=1024, N=129280, K=256, reproduced on two GPU pairs: DSpark verifies 1+5 tokens per sequence, so a decode batch of max_num_seqs needs six times that many token slots and 4096 leaves four. Cap max_num_seqs at the graph-capture shape wherever the chunk falls below that bound, which is TP=2 c128 alone. And the reason c128 never completed before is that at 16384 with capture at 2048, TP=2 c128 held 7.72 GiB, 3.02M tokens, 23.6K per request. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com> * config(dsv41flash): measure the full MI355X concurrency range on the new pin * docs(dsv41flash): retire the MI355X draft status and repoint GPU validation The MI355X section still described a draft recipe at TP4 concurrency 1-32 and cited run 34710937012 on nightly-eed1f3d0 as the pinned image. Both predate this branch: the arm now sweeps TP4 and TP2 at concurrency 1-128 on nightly-rocm100-3df4ae15. Lead with the current pin, keep the earlier run only as the superseded-pin note explaining why its points do not carry onto this image, and add the merged upstream recipe #1006 beside #968. MODELS.md and MODELS_zh.md carried a second DeepSeek-V4.1-Flash row holding the MI355X arm as pending GPU validation. Drop it; the active row already covers the model. --------- Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: functionstackx <47992694+functionstackx@users.noreply.github.com> Co-authored-by: Chun Fang <chun.fang@amd.com> * [AMD] Update GLM-5.2 MI355X image, HiCache capacity, and sweep / 更新 GLM-5.2 MI355X 镜像、HiCache 容量与 sweep (#3329) * perf: update GLM-5.2 ROCm image and HiCache path 将 GLM-5.2 ROCm 镜像更新到 20260920，切换 HiCache kernel/page_first，并调整并发矩阵。 * perf: split GLM-5.2 low and high concurrency points TP8/EP1 仅保留并发 1、2、4；TP4/EP4 HiCache 仅运行并发 12、14、16，并补充实际 PR 链接。 * Keep existing GLM-5.2 HiCache settings * Use default GLM-5.2 HiCache backend and layout * Add missing GLM-5.2 concurrency points * perf: increase GLM-5.2 HiCache capacity 将 GLM-5.2 TP4/EP4 HiCache 调整为每 rank 180 GB，并将 SA DRAM 利用率提高到 0.85。 * fix changelog --------- Co-authored-by: Chun Fang <chun.fang@amd.com> * fix(review): bind verifier verdicts to signoffs (#3371) 让清单编辑触发验证，并按签署资源隔离验证评论。 * [TileRT] GLM-5.3 MI355X AgentX parallel KV inject experiments, reverted pending tile-ai/TileRT#66 / GLM-5.3 MI355X AgentX 并行 KV 注入实验（已回退，待 tile-ai/TileRT#66） (#3376) * [GB300][SGLang][AgentX] Improve DeepSeek V4.1 Flash performance (#3344) * perf(gb300): add nightly SGLang V4.1 Flash DSpark candidates * chore(gb300): link performance changelog to PR 3344 * perf(gb300): add non-speculative SGLang AgentX candidates * chore(gb300): consolidate performance changelog entry * perf(gb300): resolve local SGLang checkpoint for Engram cache advice * perf(gb300): retain native FP8 DSpark draft projections * fix(gb300): make native draft FP8 inputs contiguous * perf(gb300): test GPU-resident Engram tables on TP4 * Use shipped nightly DSpark precision on GB300 * Interleave GB300 STP decode with long prefills * perf(gb300): prioritize DSpark across the AgentX curve * perf(gb300): test larger SWA prefix cache for DSpark * Test larger SWA cache on GB300 TP2 DSpark * Select DSpark and scale SWA retention for the GB300 sweep * perf(gb300): select qualified prefill capacity on latest nightly * chore(gb300): remove documentation changes from performance PR * [H200][SGLang][AgentX] Improve DeepSeek V4.1 Flash performance (#3341) * perf(h200): qualify DeepSeek V4.1 Flash on latest SGLang nightly * chore(h200): link performance qualification PR * perf(h200): enable huge pages for Engram host shards * perf(h200): add native non-speculative SGLang throughput arm * chore(h200): consolidate performance changelog entry * fix(sglang): retain native FP8 DSpark attention projections * style(sglang): sort native projection patch imports * chore(sglang): store context-free native projection patch * fix(h200): make native draft FP8 inputs contiguous * perf(h200): test GPU-resident Engram for non-speculative serving * perf(h200): interleave STP decode with long prefix prefill * fix: restore default SGLang DSpark precision * perf(h200): add TP4 host Engram throughput comparison * perf(h200): prioritize TP4 DSpark and prevent prefill starvation * perf(h200): tune block32 kernels and extend DSpark concurrency * perf(h200): compare native Marlin MoE with stock DSpark * perf(h200): retain DSpark after non-speculative comparisons * perf(h200): retain more SWA prefixes within the KV budget * perf(h200): select measured DSpark MoE backends by TP * perf: consolidate H200 DSpark topology and launch tuning * perf: qualify H200 DSpark with exact EP1 baseline topologies * perf: select complete H200 EP1 qualification grid * fix(h200): allow full high-concurrency AgentX warmup * [CI] select Python 3.12 for result processing / 固定结果处理 Python 3.12 (#3397) * fix: select Python 3.12 for result processing 中文：在分配 GPU 前准备独立的结果处理 Python，并在固定序列和 AgentX 路径中显式使用；配置缺失时仍保留失败产物。 * fix: use managed Python for H200 AgentX power results 在 H200 AgentX DCGM 结果处理路径中校验并使用受管 Python 解释器，保留失败产物；补充回归测试和中英文恢复说明。 * docs(review): explicitly ban FP8 NextN MoE flag (#3403) 明确禁止启用 SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE，并同步中英文贡献指南及审阅清单。 * [TileRT] Bump GLM-5.3 FP8 MI355X AgentX to tilert 0.1.6.post2 / GLM-5.3 FP8 MI355X AgentX 升级至 tilert 0.1.6.post2 (#3389) * [TileRT] Bump GLM-5.3 FP8 MI355X AgentX to tilert 0.1.6.post2 * Set changelog pr-link * [MI355X][SGLang][AgentX] Improve DeepSeek V4.1 Flash performance Add the native SGLang DeepSeek V4.1 Flash DSpark recipe for MI355X, with radix caching enabled and supported long-context memory limits. Validated by official sweep 35943469224: six canonical performance cells and six full GSM8K evaluations. Reuse this sweep for production publication. * [GB200][SGLang][AgentX] Improve DeepSeek V4.1 Flash performance Qualify the native SGLang DeepSeek V4.1 Flash DSpark recipe on GB200 with radix caching and full context. Official sweep 35876607563 passed sixteen canonical performance cells and sixteen full GSM8K evaluations across TP2/EP1 and TP4/EP1 at C1 through C128. Reuse the qualified sweep for production publication. * [H200][SGLang][AgentX] Qualify DSpark decoder SWA bounded replay (#3392) Enable decoder SWA bounded replay and pin the qualified SGLang nightly for H200 DSpark. Reuse full sweep 35893977246: 16 canonical performance cells and 16 complete GSM8K evaluations passed; final staging verified. * [H100][SGLang][AgentX] Improve DeepSeek V4.1 Flash performance (#3345) Restore the first fully qualified H100 SGLang recipe and pinned September 22 image, including TP8/EP8 and DP8 coverage. Reuse successful full sweep 35690159495. Preserve native context, prefix caching and shipped DSpark precision. * refactor: migrate fixed-sequence recipes to SRT-Slurm / 将定长配方迁移至 SRT-Slurm (#3352) * refactor: start native single-node SRT-Slurm migration 开始原生单节点 SRT-Slurm 迁移，添加 H200 SGLang 8k1k 并行候选配方并复用现有基准客户端；生产路由保持不变。 * docs: link single-node migration changelog to draft PR 为单节点迁移的 changelog 条目补充草稿 PR 链接。 * feat: wire native H200 SRT pilot into end-to-end workflow 将原生 H200 SRT 试点接入端到端工作流，显式校验配方、保留结果和功耗产物，并保持生产路由不变。 * test: cover native single-node job failures and artifacts 覆盖原生单节点作业的提交失败、Slurm 失败、产物保留及定向取消行为。 * fix: drop unused AIPerf inputs from SRT setup 删除 SRT 准备阶段未使用的 AIPerf 排空参数要求，使固定序列工作流能进入实际提交；新增缺省参数回归覆盖。 * fix: let native SRT stage the pilot container image 将配方指定镜像直接交给原生 SRT/Pyxis，移除试点对旧 squash 缓存就绪状态的依赖，保留模型预检查。 * fix: bootstrap native SRT binaries before pilot submission 在试点提交前执行原生 SRT 二进制准备，并覆盖准备失败时不提交作业的行为。 * feat: port H200 MTP and Qwen recipes to native SRT 将 H200 DeepSeek-R1 MTP 与 Qwen3.5 EP8 配方迁移到并行原生 SRT 试点，保留聊天模板及随并发变化的图捕获，并使用 4/16/64 逐点验证回归。 * docs: remove migration guide 移除独立迁移指南及入口链接，迁移范围与验证证据保留在 PR 中。 * feat: migrate fixed-sequence SGLang recipes to native SRT 将 H100、H200、B200 和 B300 的 21 个活跃 SGLang 定长配方及全部 181 个配置点切换到原生 SRT-Slurm。共享提交、eval 和产物处理，并保留逐点拓扑与推测解码设置。 * feat: migrate fixed-sequence TRT recipes to native SRT 将八个定长 TRT 配方迁移至原生 SRT，保留 63 个测试点的引擎参数、客户端和 eval token 预算。 * feat: convert remaining fixed-sequence recipes to native SRT 将剩余 AMD SGLang/ATOM 与 RTX 定长配方切换到原生 SRT 配置，保留测试点及服务参数；固定直接 ATOM 服务草稿依赖并扩展行为验证。 * fix: forward native Docker eval model identity 为 RTX 原生 Docker 路径显式传递 eval 模型名称，并验证实际启动器的命令、失败传播及容器清理。 * fix: apply AMD container options as native leaf overrides 按原生 SRT 语义逐项传递容器选项，增加映射行为回归测试并输出提交失败原因。 * refactor: reuse native job workspace for AMD scratch files 复用作业检出目录存放 AMD 原生运行时临时文件，由现有 runner 清理流程回收，移除额外 scratch-root 配置。 * fix: complete AMD native launch and terminal status handling 修复 AMD 原生启动命令的尾部换行，在 accounting 不可用时通过 Slurm 控制器验证终态，并等待启动失败的日志进程退出。 * refactor: require native recipes for the Slurm fixed-sequence cutover 移除 Docker 迁移支持和 50 个已被原生配方替代的定长 Bash 实现。Slurm 定长作业必须提供 SRT 配方，保留 AgentX、多节点和显式采集器的独立执行路径。 * refactor: make single-node fixed-sequence coverage SRT-only 归档仅支持 Docker 的 RTX 定长配置及脚本，移除无调用方的 runner 路由；保留历史 changelog 条目，但不为已归档的精确配置键生成作业。 * fix: allow client dependencies in container virtual environments 修复容器虚拟环境中的客户端依赖安装，移除不兼容的用户目录安装参数。 * fix: limit native single-node steps to serving GPUs 保留节点独占预留，同时将原生单节点服务和客户端步骤限制为配方指定的 GPU 数量，避免采集空闲设备功耗。 * fix: restore repository workdir for native container steps 原生容器步骤从已有仓库挂载目录启动，避免 Python 动态模块在根目录下导入失败，并保留调用方运行时覆盖。 * fix: stream AMD power samples without input buffering 逐行写出 AMD 功耗 CSV，避免 awk 输入缓冲在采样器关闭时丢失末尾数据；新增真实流式读取回归测试。 * fix: match registered H200 runner labels 将 H200 DGXC runner 清单对齐到实际注册的两位数字标签，恢复精确 smoke 选择和 sweep 调度。 * chore: pin srt-slurm submodule to main [skip ci] * refactor(srt): return single-node migration to NVIDIA upstream 将单节点迁移切回 NVIDIA 上游 srt-slurm。ATOM 使用原生 AToMesh 及单独固定的官方路由器镜像；TRT-LLM 使用原生模型名称转发。保留 worker 镜像、服务参数、并发范围与拓扑，不增加 Bash 分支。 * chore: bump srt-slurm submodule to v2.23.2 Move the shared SRT submodule from v2.22.1 (3cbc5dd) to upstream release v2.23.2 (8dace5f). Upstream delta is three additive changes: vllm-router multi-node hybrid-DP rank offsets, SGLang leader IP honoring the cluster network interface, and nsys/DSight coverage for SGLang and Dynamo. No recipe or connector changes required; utils/ and runners/ tests pass. * refactor(srt): check MI300X firmware in a setup hook Replace the MI300X container preamble with a host setup hook that fails on MEC firmware older than 177 instead of exporting HSA_NO_SCRATCH_RECLAIM in every container. * fix(srt): apply pending upstream srt-slurm patches at setup Post-eval steps dropped the recipe's srun_options, so ATOM evals on MI355X ran in a read-on…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What the experiments measured (MI355X, GLM-5.3 FP8, AgentX, conc 1)
--enforce-eager, sweep 35836899099injectcopy from 10.5 to 1.78 µs/token. It accounts for nearly all of the TTFT gain; this is what goes upstream.--enforce-eageron the vLLM prefill left TTFT unchanged (2.08 vs 2.18 s p50 over the same 184 requests). Dropped.AI model disclosure
e3d79eea9anda0caa02b9.claude-opus-5-5[1m](Claude Opus 5.5, 1M context) via Claude Code, operated by @Oseltamivir; the initial TTFT breakdown usedclaude-fable-5-1[1m]. Role: read the post1 wheel sources, wrote and iterated the patches, analysed the decodeREQSTATlogs and AIPerf exports, and wrote the text.Checklist
/use <run_id>(not applicable while parked)