From b50a2d3391323437fc6cc0f71aede69a855cb94e Mon Sep 17 00:00:00 2001 From: CrimsonDump Date: Mon, 21 Sep 2026 18:02:54 +0800 Subject: [PATCH 01/25] =?UTF-8?q?[TileRT]=20Add=20GLM-5.3=20FP8=20MI355X?= =?UTF-8?q?=20prefill/decode-disaggregated=20recipes=20/=20=E6=96=B0?= =?UTF-8?q?=E5=A2=9E=20GLM-5.3=20FP8=20MI355X=20=E7=9A=84=20PD=20=E5=88=86?= =?UTF-8?q?=E7=A6=BB=E9=85=8D=E6=96=B9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds GLM-5.3 FP8 on MI355X as a prefill/decode-disaggregated TileRT recipe: upstream vLLM ROCm prefill (TileRTConnector, kv_producer) + TileRT decode_server + an OpenAI-compatible pd_router, framework=tilert, KV over mooncake. Two scenarios, Single-turn 8k1k and AgentX agentic-coding, mirroring the merged B200 glm5.1-fp8-b200-tilert entries on the amd_utils orchestration. 以 TileRT 的 prefill/decode 分离形态新增 MI355X 上的 GLM-5.3 FP8：上游 vLLM ROCm 作 prefill + TileRT decode_server + 兼容 OpenAI 的 pd_router，KV 走 mooncake。两个场景：单轮 8k1k 与 AgentX 智能体编码，形态对齐已合并的 B200 glm5.1-fp8-b200-tilert，走 amd_utils 编排链。 IMAGES AND WHEEL / 镜像与轮子 decode ghcr.io/tile-ai/tilert-rocm-decode:0.1.6 prefill ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6 (the upstream vllm/vllm-openai-rocm release image plus the mooncake ROCm transfer engine built from source) Neither image carries the wheel; setup_deps.sh installs tilert==0.1.6 from PyPI into both ranks at container start -- full on decode, --no-deps connector plugin on prefill -- so vLLM runs exactly as shipped and no file inside either image is modified. TOPOLOGY / 拓扑 1 prefill node (TP8) + 1 decode node (TP8), conc-list [1] because TileRT decode serves one sequence at a time. bf16 KV cache on both roles -- the ROCm vLLM sparse-MLA backend has no fp8_ds_mla dtype. max-model-len 202752. MTP via spec-decoding=mtp at draft depth 3. Prefill runs --block-size 64: the ROCm backend supports [1,64] and vLLM picks 1, which makes the connector's KI plane copy fail and MLA address the wrong rows. GOLDEN AL / 黄金接受率 Nothing hard-codes an acceptance length. server_tilert.sh reads the committed curve at launch, selecting it exactly as infx/srt_slurm/synthetic_acceptance.py does (_.yaml, then [model][thinking mode][draft tokens]), applies it to agentic throughput only, suppresses it under RUN_EVAL/EVAL_ONLY because simulated acceptance would score ~0 on GSM8K, and fails the run rather than falling back to real acceptance when the curve or draft length is missing. 接受率不写死在任何配方或配置里，由 server_tilert.sh 启动时从提交的曲线读取， 选择规则与上游 srt-slurm 连接器一致；仅作用于 agentic 吞吐，eval 下自动关闭， 曲线缺失时直接失败而非静默回落。 NOTE FOR REVIEWERS: golden_al_distribution/glm5.3_mtp.yaml is GLM-5.2's curve, copied. No SPEED-Bench run exists for GLM-5.3 and the two share a base and layer topology. Only the K=3 cell is carried over. The file is labelled PROVISIONAL. Flagged rather than left to read as a measured 5.3 curve. LOCAL VALIDATION / 本地验证 Run on 2x8 MI350X with these images and wheel, through the official entry points: GSM8K (RUN_EVAL + EVAL_ONLY, 1319 items) strict 0.9750, flexible 0.9735 AgentX 3600 s, conc 1 submission_valid true, coverage 99.8%/99.9%, 532 requests Single-turn 8k1k, conc 1 TTFT p50 1010 ms MI350X is not MI355X: same gfx950 ISA and memory system, but 1000 W vs 1400 W and 2200 vs 2400 MHz. These show the recipe runs and is accurate; they are not leaderboard results, which the sweep produces on real mi355x runners. 本地数据跑在 MI350X 上，不是 MI355X，仅用于证明配方能跑且结果正确， 不作为打榜成绩。 Matrix UTs 335 passed. perf-changelog is a pure tail append; the 8457 lines inherited from main are byte-for-byte unchanged. --- MODELS.md | 1 + MODELS_zh.md | 1 + .../agentic/glm5.3_fp8_mi355x_tilert.sh | 100 +++ benchmarks/multi_node/amd_utils/env.sh | 3 + benchmarks/multi_node/amd_utils/job.slurm | 42 +- .../multi_node/amd_utils/models_tilert.yaml | 8 + benchmarks/multi_node/amd_utils/server.sh | 3 +- .../multi_node/amd_utils/server_tilert.sh | 599 ++++++++++++++++++ benchmarks/multi_node/amd_utils/setup_deps.sh | 120 ++++ .../multi_node/glm5.3_fp8_mi355x_tilert.sh | 84 +++ configs/amd-master.yaml | 81 +++ golden_al_distribution/glm5.3_mtp.yaml | 19 + perf-changelog.yaml | 19 + runners/launch_mi355x-amds.sh | 6 +- 14 files changed, 1079 insertions(+), 7 deletions(-) create mode 100644 benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh create mode 100644 benchmarks/multi_node/amd_utils/models_tilert.yaml create mode 100644 benchmarks/multi_node/amd_utils/server_tilert.sh create mode 100644 benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh create mode 100644 golden_al_distribution/glm5.3_mtp.yaml diff --git a/MODELS.md b/MODELS.md index b1b7d2e5eb..ef0fd5d076 100644 --- a/MODELS.md +++ b/MODELS.md @@ -153,6 +153,7 @@ Other offloading tiers, including NVMe KV cache offloading, are outside the init | Model architecture class | Prefix | Date added | Active scenarios | Deprecated scenarios | |---|---|---|---|---| | DeepSeek-V4.1-Flash | `dsv41flash` | 2026-09-10 | Agentic coding (vLLM: DSpark, Engram UVA offload; SGLang: DSpark arms added per SKU from 2026-09-17; GPU validation pending) | | +| GLM-5.3 | `glm5.3` | 2026-09-08 ([#3328](https://github.com/SemiAnalysisAI/InferenceX/pull/3328)) | Single-turn 8k1k, Agentic coding (MTP only, per the Deprecation Notice) | Single-turn 1k1k (deprecated for all models before this model was added; never run) | | Qwen3.8-Flash-Next | `qwen3.8next` | 2026-08-26 ([#2742](https://github.com/SemiAnalysisAI/InferenceX/pull/2742)) | Agentic coding | | | Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | Agentic coding (DSpark may be disabled for better Pareto points) | Standalone non-DSpark A/B baseline (not required from day 0) | | GLM-5.2 | `glm5.2` | 2026-07-18 ([#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)) | Agentic coding (non-MTP points remain eligible under the Pareto policy; see Deprecation Notice) | | diff --git a/MODELS_zh.md b/MODELS_zh.md index 80f39d1b78..ece612173d 100644 --- a/MODELS_zh.md +++ b/MODELS_zh.md @@ -153,6 +153,7 @@ InferenceX 支持 SGLang 和 vLLM 双方的维护者，并响应 AI 实验室和 | 模型架构类别 | 前缀 | 加入日期 | 启用场景 | 已弃用场景 | |---|---|---|---|---| | DeepSeek-V4.1-Flash | `dsv41flash` | 2026-09-10 | 智能体编码（DSpark、Engram UVA 卸载；GPU 待验证） | | +| GLM-5.3 | `glm5.3` | 2026-09-08（[#3328](https://github.com/SemiAnalysisAI/InferenceX/pull/3328)） | 单轮 8k1k、智能体编码（仅 MTP，见弃用公告） | 单轮 1k1k（在本模型加入前已对所有模型弃用，从未运行） | | Qwen3.8-Flash-Next | `qwen3.8next` | 2026-08-26（[#2742](https://github.com/SemiAnalysisAI/InferenceX/pull/2742)） | 智能体编码 | | | Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | 智能体编码（可关闭 DSpark 以获得更优帕累托点） | 独立非 DSpark A/B 基线（自第 0 天起即不要求） | | GLM-5.2 | `glm5.2` | 2026-07-18（[#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)） | 智能体编码（非 MTP 数据点仍可按帕累托策略参与发布；见弃用公告） | | diff --git a/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh b/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh new file mode 100644 index 0000000000..f4d6bc943f --- /dev/null +++ b/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh @@ -0,0 +1,100 @@ +#!/usr/bin/env bash + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$SCRIPT_DIR/../../benchmark_lib.sh" + +check_env_vars \ + CONC_LIST \ + ISL \ + OSL \ + IMAGE \ + SPEC_DECODING \ + MODEL_PATH \ + PREFILL_NUM_WORKERS \ + PREFILL_TP \ + PREFILL_EP \ + PREFILL_DP_ATTN \ + DECODE_NUM_WORKERS \ + DECODE_TP \ + DECODE_EP \ + DECODE_DP_ATTN \ + PREFILL_NODES \ + DECODE_NODES \ + RANDOM_RANGE_RATIO \ + DURATION \ + KV_OFFLOADING \ + IS_AGENTIC \ + FRAMEWORK \ + PREFILL_IMAGE + +if [[ -n "$SLURM_JOB_ID" ]]; then + echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME" +fi + +set -x + +cd "$GITHUB_WORKSPACE/benchmarks/multi_node/amd_utils" || exit 1 + +export TIME_LIMIT="${TIME_LIMIT:-08:00:00}" +export MODEL_PATH=$MODEL_PATH +export MODEL_NAME=$MODEL_NAME +export CONTAINER_IMAGE=$IMAGE +export PREFILL_IMAGE + +export MODEL_PREFIX="${MODEL_PREFIX:-glm5.3}" +export PRECISION="${PRECISION:-fp8}" +export RESULT_FILENAME="${RESULT_FILENAME:-${RUNNER_NAME:-glm5.3-fp8-tilert-agentic}}" +export DECODE_MTP_SIZE="${DECODE_MTP_SIZE:-$([[ "$SPEC_DECODING" == "mtp" ]] && echo 1 || echo 0)}" + +export DURATION="${DURATION:-1800}" + +if [[ "$PREFILL_NODES" -ne 1 || "$DECODE_NODES" -ne 1 || \ + "$PREFILL_NUM_WORKERS" -ne 1 || "$DECODE_NUM_WORKERS" -ne 1 ]]; then + echo "Error: tilert supports exactly 1 prefill node/worker + 1 decode node/worker" \ + "(got PREFILL_NODES=$PREFILL_NODES x$PREFILL_NUM_WORKERS, DECODE_NODES=$DECODE_NODES x$DECODE_NUM_WORKERS)" >&2 + exit 1 +fi + +export KV_OFFLOADING="${KV_OFFLOADING:-none}" +if [[ "$KV_OFFLOADING" != "none" ]]; then + echo "Error: tilert has no KV offload backend; kv-offloading must be 'none' (got '$KV_OFFLOADING')" >&2 + exit 1 +fi + +export TILERT_VERSION="${TILERT_VERSION:-0.1.6}" +export TILERT_PROFILE="${TILERT_PROFILE:-glm5_2}" +export TILERT_MODEL_TYPE="${TILERT_MODEL_TYPE:-glm-5}" +export TILERT_MAX_MODEL_LEN="${TILERT_MAX_MODEL_LEN:-202752}" +export TILERT_TRANSPORT="${TILERT_TRANSPORT:-mooncake}" +export PREFILL_KV_DTYPE="${PREFILL_KV_DTYPE:-auto}" +export DECODE_KV_DTYPE="${DECODE_KV_DTYPE:-bf16}" +export TILERT_PARSER="${TILERT_PARSER:-none}" +export ROUTER_PORT="${ROUTER_PORT:-30000}" +export TILERT_QUEUE_TIMEOUT="${TILERT_QUEUE_TIMEOUT:-1800}" + +if [[ "${PREFILL_EP:-1}" -ne 1 || "${DECODE_EP:-1}" -ne 1 || \ + "$PREFILL_DP_ATTN" == "true" || "$DECODE_DP_ATTN" == "true" ]]; then + echo "Error: tilert runs pure TP8 on both roles; ep must be 1 and dp-attn false" >&2 + exit 1 +fi +export PREFILL_ENABLE_EP=false +export PREFILL_ENABLE_DP=false +export DECODE_ENABLE_EP=false +export DECODE_ENABLE_DP=false + +JOB_ID=$(bash ./submit.sh $PREFILL_NODES \ + $PREFILL_NUM_WORKERS \ + $DECODE_NODES \ + $DECODE_NUM_WORKERS \ + $ISL $OSL "${CONC_LIST// /x}" inf \ + ${PREFILL_ENABLE_EP} ${PREFILL_ENABLE_DP} \ + ${DECODE_ENABLE_EP} ${DECODE_ENABLE_DP} \ + ${PREFILL_TP} ${DECODE_TP} \ + ${RANDOM_RANGE_RATIO}) + +if [[ $? -ne 0 ]]; then + echo "Failed to submit job" >&2 + exit 1 +fi + +echo "$JOB_ID" diff --git a/benchmarks/multi_node/amd_utils/env.sh b/benchmarks/multi_node/amd_utils/env.sh index 58aef2f10e..bfc006112e 100755 --- a/benchmarks/multi_node/amd_utils/env.sh +++ b/benchmarks/multi_node/amd_utils/env.sh @@ -180,6 +180,9 @@ $1 == "DSCP" && $2 == ":" && $NF == p { set +x echo "[INFO] IBDEVICES=$IBDEVICES UCX_NET_DEVICES=$UCX_NET_DEVICES NCCL_SOCKET_IFNAME=$NCCL_SOCKET_IFNAME UCX_IB_GID_INDEX=$UCX_IB_GID_INDEX UCX_IB_TRAFFIC_CLASS=${UCX_IB_TRAFFIC_CLASS:-unset}" +elif [[ "$ENGINE" == "tilert" ]]; then + echo "[INFO] tilert: IBDEVICES=$IBDEVICES NCCL_SOCKET_IFNAME=$NCCL_SOCKET_IFNAME NCCL_IB_HCA=$NCCL_IB_HCA" + else export SGLANG_USE_AITER=1 diff --git a/benchmarks/multi_node/amd_utils/job.slurm b/benchmarks/multi_node/amd_utils/job.slurm index d1f5ea11a7..09ee78af38 100755 --- a/benchmarks/multi_node/amd_utils/job.slurm +++ b/benchmarks/multi_node/amd_utils/job.slurm @@ -38,6 +38,8 @@ if [[ "$ENGINE" == "vllm-disagg" ]]; then MODELS_YAML="$(pwd)/models_vllm.yaml" elif [[ "$ENGINE" == "atom-disagg" ]]; then MODELS_YAML="$(pwd)/models_atom.yaml" +elif [[ "$ENGINE" == "tilert" ]]; then + MODELS_YAML="$(pwd)/models_tilert.yaml" else MODELS_YAML="$(pwd)/models.yaml" fi @@ -52,6 +54,12 @@ if [[ -z "${DOCKER_IMAGE_NAME:-}" ]]; then exit 1 fi +if [[ "$ENGINE" == "tilert" && -z "${PREFILL_IMAGE:-}" ]]; then + echo "Error: ENGINE=tilert requires PREFILL_IMAGE (e.g. PREFILL_IMAGE=vllm/vllm-openai-rocm:nightly- in prefill.additional-settings)." + exit 1 +fi + +MODEL_NAME="${MODEL_NAME:-None}" # Resolve the models.yaml entry the same way server_sglang.sh does: agentic runs # (IS_AGENTIC) use the '-AgentX' recipe, non-agentic disaggregated runs use # '-DI'. Fall back to the bare model name if the variant key is absent. @@ -522,6 +530,32 @@ elif [[ "$ENGINE" == "atom-disagg" ]]; then -e EXTRA_SERVER_ARGS=\${EXTRA_SERVER_ARGS:-} -e IBDEVICES=${IBDEVICES:-} ) +elif [[ "$ENGINE" == "tilert" ]]; then + DOCKER_ENV_ENGINE=( + -e MODEL_PATH=$DOCKER_MODEL_PATH + -e PREFILL_IMAGE=${PREFILL_IMAGE} + -e TILERT_VERSION=${TILERT_VERSION:-} + -e TILERT_PIP_SPEC=${TILERT_PIP_SPEC:-} + -e TILERT_PIP_INDEX_URL=${TILERT_PIP_INDEX_URL:-} + -e TILERT_PROFILE=${TILERT_PROFILE:-} + -e TILERT_MODEL_TYPE=${TILERT_MODEL_TYPE:-} + -e TILERT_MAX_MODEL_LEN=${TILERT_MAX_MODEL_LEN:-} + -e TILERT_TRANSPORT=${TILERT_TRANSPORT:-} + -e PREFILL_KV_DTYPE=${PREFILL_KV_DTYPE:-} + -e DECODE_KV_DTYPE=${DECODE_KV_DTYPE:-} + -e TILERT_PARSER=${TILERT_PARSER:-} + -e TILERT_QUEUE_TIMEOUT=${TILERT_QUEUE_TIMEOUT:-} + -e TILERT_WEIGHTS_DIR=${TILERT_WEIGHTS_DIR:-} + -e TILERT_RDMA_STRICT=${TILERT_RDMA_STRICT:-0} + -e \"TILERT_EXTRA_ENV=${TILERT_EXTRA_ENV:-}\" + -e GPU_MEM_UTIL=${GPU_MEM_UTIL:-} + -e ROUTER_PORT=${ROUTER_PORT:-30000} + -e PREFILL_PORT=${PREFILL_PORT:-8000} + -e DECODE_CTRL_PORT=${DECODE_CTRL_PORT:-5556} + -e DECODE_HTTP_PORT=${DECODE_HTTP_PORT:-5557} + -e IBDEVICES=${IBDEVICES:-} + -e PYTHONPYCACHEPREFIX=/tmp/pycache + ) else DOCKER_ENV_ENGINE=( -e SGLANG_WS_PATH=${WS_PATH} @@ -704,6 +738,12 @@ if [[ \"$ENGINE\" == \"vllm-disagg\" && \"$ROUTER_TYPE\" == \"vllm-router\" && \ set +e fi +RANK_IMAGE= +if [[ \"$ENGINE\" == \"tilert\" && \"\$SLURM_PROCID\" -lt \"$xP\" ]]; then + RANK_IMAGE=\"$PREFILL_IMAGE\" + echo \"[tilert] rank \$SLURM_PROCID is a prefill rank; using PREFILL_IMAGE=\$RANK_IMAGE\" +fi + \$MAYBE_EXEC \$DOCKER_CMD run \ --init \ --stop-timeout 10 \ @@ -745,7 +785,7 @@ fi ${CLIENT_DOCKER_ENV} \ --name \"$DOCKER_CONT_NAME\" \ --entrypoint \"\" \ - \"$DOCKER_IMAGE_NAME\" bash -lc ' + \"\${RANK_IMAGE:-$DOCKER_IMAGE_NAME}\" bash -lc ' set -o pipefail mkdir -p /run_logs/slurm_job-'\"\$SLURM_JOB_ID\"' '"$RUN_FILE_FULL"' 2>&1 | tee /run_logs/slurm_job-'\"\$SLURM_JOB_ID\"'/server_\$(hostname).log diff --git a/benchmarks/multi_node/amd_utils/models_tilert.yaml b/benchmarks/multi_node/amd_utils/models_tilert.yaml new file mode 100644 index 0000000000..a84ef00b1d --- /dev/null +++ b/benchmarks/multi_node/amd_utils/models_tilert.yaml @@ -0,0 +1,8 @@ + +GLM-5.3: + profile: "glm5_2" + model_type: "glm-5_2" + max_model_len: 202752 + prefill_extra_flags: "" + prefill_env: "VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 VLLM_ENGINE_READY_TIMEOUT_S=10800" + decode_extra_flags: "" diff --git a/benchmarks/multi_node/amd_utils/server.sh b/benchmarks/multi_node/amd_utils/server.sh index 6b65281c35..eddf3a2e44 100755 --- a/benchmarks/multi_node/amd_utils/server.sh +++ b/benchmarks/multi_node/amd_utils/server.sh @@ -6,7 +6,6 @@ source "$(dirname "${BASH_SOURCE[0]}")/../../benchmark_lib.sh" --validation-only # ENGINE=sglang-disagg (default) -> server_sglang.sh (SGLang + MoRI) # ENGINE=vllm-disagg -> server_vllm.sh (vLLM + Nixl/MoRI-IO) # ENGINE=atom-disagg -> server_atom.sh (ATOM + mooncake) - check_env_vars ENGINE WS_PATH if [[ -f /config/hicache_mc.env ]]; then set -a @@ -22,6 +21,8 @@ if [[ "$ENGINE" == "vllm-disagg" ]]; then elif [[ "$ENGINE" == "atom-disagg" ]]; then export ATOM_WS_PATH="$WS_PATH" source "$WS_PATH/server_atom.sh" +elif [[ "$ENGINE" == "tilert" ]]; then + source "$WS_PATH/server_tilert.sh" else source "$WS_PATH/server_sglang.sh" fi diff --git a/benchmarks/multi_node/amd_utils/server_tilert.sh b/benchmarks/multi_node/amd_utils/server_tilert.sh new file mode 100644 index 0000000000..cab8764028 --- /dev/null +++ b/benchmarks/multi_node/amd_utils/server_tilert.sh @@ -0,0 +1,599 @@ +#!/bin/bash + +NODE0_ADDR="${NODE0_ADDR:-localhost}" +NODE_RANK="${NODE_RANK:-0}" +MODEL_DIR="${MODEL_DIR:-/models}" +MODEL_NAME="${MODEL_NAME:-}" +xP="${xP:-1}" +yD="${yD:-1}" +IPADDRS="${IPADDRS:-localhost}" +DRY_RUN="${DRY_RUN:-0}" +GPUS_PER_NODE="${GPUS_PER_NODE:-8}" +PREFILL_TP_SIZE="${PREFILL_TP_SIZE:-$GPUS_PER_NODE}" +DECODE_TP_SIZE="${DECODE_TP_SIZE:-$GPUS_PER_NODE}" + +BENCH_INPUT_LEN="${BENCH_INPUT_LEN:-1024}" +BENCH_OUTPUT_LEN="${BENCH_OUTPUT_LEN:-1024}" +BENCH_RANDOM_RANGE_RATIO="${BENCH_RANDOM_RANGE_RATIO:-1}" +BENCH_REQUEST_RATE="${BENCH_REQUEST_RATE:-inf}" +BENCH_NUM_PROMPTS_MULTIPLIER="${BENCH_NUM_PROMPTS_MULTIPLIER:-10}" +BENCH_MAX_CONCURRENCY="${BENCH_MAX_CONCURRENCY:-1}" + +MODEL_PATH="${MODEL_PATH:-${MODEL_DIR}/${MODEL_NAME}}" + +LOG_DIR="/run_logs/slurm_job-${SLURM_JOB_ID:-local}" +SHARED_LOG_DIR="${BENCHMARK_LOGS_DIR:-/run_logs}/logs/slurm_job-${SLURM_JOB_ID:-local}" +mkdir -p "$LOG_DIR" + +if [[ "$xP" -ne 1 || "$yD" -ne 1 ]]; then + echo "ERROR: tilert supports exactly 1 prefill + 1 decode worker (got xP=$xP yD=$yD)" >&2 + exit 1 +fi +if [[ "$NODE_RANK" -lt "$xP" ]]; then + TILERT_ROLE=prefill +else + TILERT_ROLE=decode +fi +export TILERT_ROLE + +source "$WS_PATH/setup_deps.sh" +source "$WS_PATH/env.sh" +PY="${PY:-python3}" + +source /workspace/benchmarks/benchmark_lib.sh + +MODELS_YAML="${WS_PATH}/models_tilert.yaml" +if [[ -f "$MODELS_YAML" ]] && "$PY" -c "import yaml" 2>/dev/null; then + eval "$("$PY" - "$MODELS_YAML" "$MODEL_NAME" <<'PYEOF' +import shlex, sys, yaml +path, name = sys.argv[1], sys.argv[2] +with open(path) as f: + models = yaml.safe_load(f) or {} +m = models.get(name) or {} +keys = { + "profile": "TILERT_PROFILE", + "model_type": "TILERT_MODEL_TYPE", + "max_model_len": "TILERT_MAX_MODEL_LEN", + "prefill_extra_flags": "TILERT_PREFILL_EXTRA_FLAGS", + "prefill_env": "TILERT_PREFILL_ENV", + "decode_extra_flags": "TILERT_DECODE_EXTRA_FLAGS", +} +for k, var in keys.items(): + if k not in m or m[k] is None: + continue + val = str(m[k]) + if val == "": + continue + print(f'if [[ -z "${{{var}:-}}" ]]; then {var}={shlex.quote(val)}; fi') +PYEOF +)" + echo "[tilert] loaded model defaults for '$MODEL_NAME' from $MODELS_YAML" +else + echo "[tilert] $MODELS_YAML not readable (missing file or PyYAML); using built-in defaults" +fi + +TILERT_PROFILE="${TILERT_PROFILE:-glm5_2}" # decode_server --model (TileRT model profile) +TILERT_MODEL_TYPE="${TILERT_MODEL_TYPE:-glm-5}" # weight_converter --model_type +TILERT_MODEL_PKG="${TILERT_MODEL_PKG:-glm_5_2_rocm}" +TILERT_MAX_MODEL_LEN="${TILERT_MAX_MODEL_LEN:-${MAX_MODEL_LEN:-202752}}" +TILERT_TRANSPORT="${TILERT_TRANSPORT:-mooncake}" # decode --transport / connector tilert_transport +PREFILL_KV_DTYPE="${PREFILL_KV_DTYPE:-auto}" +PREFILL_BLOCK_SIZE="${PREFILL_BLOCK_SIZE:-64}" +DECODE_KV_DTYPE="${DECODE_KV_DTYPE:-bf16}" +TILERT_PARSER="${TILERT_PARSER:-none}" +if [[ "${IS_AGENTIC:-0}" == "1" || "${IS_AGENTIC:-}" == "true" || "${SCENARIO_TYPE:-}" == "agentic-coding" ]]; then + TILERT_QUEUE_TIMEOUT="${TILERT_QUEUE_TIMEOUT:-600}" +else + TILERT_QUEUE_TIMEOUT="${TILERT_QUEUE_TIMEOUT:-0}" +fi +GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.75}" +TILERT_PREFILL_EXTRA_FLAGS="${TILERT_PREFILL_EXTRA_FLAGS:-}" +TILERT_DECODE_EXTRA_FLAGS="${TILERT_DECODE_EXTRA_FLAGS:-}" +TILERT_PREFILL_ENV="${TILERT_PREFILL_ENV:-}" +TILERT_EXTRA_ENV="${TILERT_EXTRA_ENV:-}" + +DECODE_CTRL_PORT="${DECODE_CTRL_PORT:-5556}" +DECODE_HTTP_PORT="${DECODE_HTTP_PORT:-5557}" +PREFILL_PORT="${PREFILL_PORT:-8000}" +ROUTER_PORT="${ROUTER_PORT:-30000}" +export ROUTER_PORT + +DECODE_WAIT="${DECODE_WAIT:-7200}" # prefill waits for decode ctrl port (weights conversion + load) +PREFILL_WAIT="${PREFILL_WAIT:-3600}" # prefill waits for its own vLLM port +ROUTER_WAIT="${ROUTER_WAIT:-10800}" # decode waits for the router port to open + +SERVED_MODEL_NAME="${SERVED_MODEL_NAME:-$TILERT_PROFILE}" +export SERVED_MODEL_NAME + +TILERT_WEIGHTS_DIR="${TILERT_WEIGHTS_DIR:-${MODEL_DIR}/${MODEL_NAME}-tilert-tp${DECODE_TP_SIZE}}" + +PREFILL_SPEC=() +DECODE_MTP=() +if [[ "${SPEC_DECODING:-}" == "mtp" ]]; then + PREFILL_SPEC=(--speculative-config '{"method":"mtp","num_speculative_tokens":1}') + DECODE_MTP=(--with-mtp) +fi + +TILERT_IS_AGENTIC=0 +if [[ "${IS_AGENTIC:-0}" == "1" || "${IS_AGENTIC:-}" == "true" || "${SCENARIO_TYPE:-}" == "agentic-coding" ]]; then + TILERT_IS_AGENTIC=1 +fi + +IFS=',' read -ra IP_ARRAY <<< "$IPADDRS" +PREFILL_HOST="${IP_ARRAY[0]:-$NODE0_ADDR}" +DECODE_HOST="${IP_ARRAY[$xP]:-}" +if [[ -z "$DECODE_HOST" ]]; then + echo "ERROR: cannot resolve the decode node IP from IPADDRS='$IPADDRS' (xP=$xP)" >&2 + exit 1 +fi +host_ip=$(ip route get 1.1.1.1 2>/dev/null | awk '/src/ {print $7}') +host_name=$(hostname) + +echo "[tilert] ROLE=$TILERT_ROLE rank=$NODE_RANK host=$host_name ($host_ip)" +echo "[tilert] PREFILL_HOST=$PREFILL_HOST:$PREFILL_PORT DECODE_HOST=$DECODE_HOST:$DECODE_CTRL_PORT/$DECODE_HTTP_PORT ROUTER=:$ROUTER_PORT" +echo "[tilert] MODEL_PATH=$MODEL_PATH profile=$TILERT_PROFILE served=$SERVED_MODEL_NAME max_len=$TILERT_MAX_MODEL_LEN transport=$TILERT_TRANSPORT kv=${PREFILL_KV_DTYPE}->${DECODE_KV_DTYPE} mtp=${SPEC_DECODING:-none} agentic=$TILERT_IS_AGENTIC" + +for env_pair in ${TILERT_EXTRA_ENV}; do + export "${env_pair?}" + echo "[tilert][EXTRA_ENV] $env_pair" +done + +log_and_run_bg() { + local label="$1" logfile="$2"; shift 2 + { printf '===== [%s] %s =====\n' "$label" "$(date '+%F %T')" + printf '[cmd]'; printf ' %q' "$@"; printf '\n' + printf '[cwd] %s\n[host] %s\n\n' "$PWD" "$host_name" + } | tee -a "$logfile" + if [[ "$DRY_RUN" -eq 1 ]]; then + echo "DRY RUN: [$label] not started" + LAST_BG_PID="" + return 0 + fi + "$@" >>"$logfile" 2>&1 & + LAST_BG_PID=$! + echo "[$label] pid=$LAST_BG_PID log=$logfile" +} + +rdma_preflight() { + local warn=0 + echo "[rdma] role=$TILERT_ROLE IBDEVICES=${IBDEVICES:-} NCCL_SOCKET_IFNAME=${NCCL_SOCKET_IFNAME:-}" + local uverbs=(/dev/infiniband/uverbs*) + if [[ -e "${uverbs[0]}" ]]; then + echo "[rdma] verbs devices: ${uverbs[*]}" + else + echo "[rdma] WARNING: /dev/infiniband/uverbs* missing -- the container has no RDMA device nodes (job.slurm passes --device /dev/infiniband)" >&2 + warn=1 + fi + local ml; ml="$(ulimit -l 2>/dev/null)" + if [[ "$ml" == "unlimited" ]]; then + echo "[rdma] memlock: unlimited" + else + echo "[rdma] WARNING: memlock=$ml (not unlimited) -- pinning memory for RDMA may fail (job.slurm passes --ulimit memlock=-1)" >&2 + warn=1 + fi + if command -v ibv_devices >/dev/null 2>&1; then + echo "[rdma] ibv_devices:"; ibv_devices 2>&1 | sed 's/^/[rdma] /' + fi + if (( warn )) && [[ "${TILERT_RDMA_STRICT:-0}" == "1" ]]; then + echo "[rdma] TILERT_RDMA_STRICT=1 and preflight did not fully pass -- aborting" >&2 + return 1 + fi + return 0 +} + +stage_tokenizer_files() { + local staged=0 f b + for f in "$MODEL_PATH"/*; do + [[ -f "$f" ]] || continue + b="$(basename "$f")" + [[ "$b" == *.safetensors ]] && continue + [[ "$b" == "model.safetensors.index.json" ]] && continue + [[ -e "$TILERT_WEIGHTS_DIR/$b" ]] && continue + cp -p "$f" "$TILERT_WEIGHTS_DIR/$b" && staged=$((staged+1)) + done + echo "[stage_tokenizer] staged $staged auxiliary file(s) from $MODEL_PATH -> $TILERT_WEIGHTS_DIR" + local missing=() + [[ -f "$TILERT_WEIGHTS_DIR/chat_template.jinja" ]] || missing+=(chat_template.jinja) + [[ -f "$TILERT_WEIGHTS_DIR/tokenizer_config.json" || -f "$TILERT_WEIGHTS_DIR/tokenizer.json" ]] \ + || missing+=("tokenizer.json/tokenizer_config.json") + if (( ${#missing[@]} )); then + echo "[stage_tokenizer] ERROR: $TILERT_WEIGHTS_DIR is missing ${missing[*]}; check that MODEL_PATH=$MODEL_PATH is an HF directory with the tokenizer" >&2 + return 1 + fi + return 0 +} + +_tilert_weights_cached() { + local r + for r in $(seq 0 $((DECODE_TP_SIZE - 1))); do + [[ -f "$TILERT_WEIGHTS_DIR/rank${r}/model.safetensors.index.json" ]] || return 1 + done + return 0 +} + +convert_weights() { + if _tilert_weights_cached; then + echo "[weight_converter] cache hit (${DECODE_TP_SIZE}/${DECODE_TP_SIZE} rank index.json), skipping conversion: $TILERT_WEIGHTS_DIR" + return 0 + fi + mkdir -p "$TILERT_WEIGHTS_DIR" || { echo "[weight_converter] ERROR: cannot create $TILERT_WEIGHTS_DIR (set TILERT_WEIGHTS_DIR to a writable shared path)" >&2; return 1; } + exec 9>"$TILERT_WEIGHTS_DIR/.convert.lock" + flock -w "${TILERT_CONVERT_LOCK_WAIT:-21600}" 9 || { + echo "[weight_converter] timed out waiting for the conversion lock (another job still converting?)" >&2; return 1; } + if _tilert_weights_cached; then + echo "[weight_converter] cache produced by a concurrent job, skipping conversion"; exec 9>&-; return 0 + fi + if [[ -n "$(ls -A "$TILERT_WEIGHTS_DIR" 2>/dev/null | grep -v '^\.convert\.lock$')" ]]; then + echo "[weight_converter] leftovers without index.json (previous conversion incomplete); cleaning and re-converting" + find "$TILERT_WEIGHTS_DIR" -mindepth 1 ! -name '.convert.lock' -delete + fi + echo "[weight_converter] $MODEL_PATH -> $TILERT_WEIGHTS_DIR (model_type=$TILERT_MODEL_TYPE)" + local conv_mod conv_args + if "$PY" -c "import tilert.models.${TILERT_MODEL_PKG}.weight_converter" 2>/dev/null; then + conv_mod="tilert.models.${TILERT_MODEL_PKG}.weight_converter" + conv_args=(--model_dir "$MODEL_PATH" --save_dir "$TILERT_WEIGHTS_DIR" + --device "${TILERT_CONVERT_DEVICE:-cuda:$((GPUS_PER_NODE - 1))}") + [[ "${SPEC_DECODING:-}" == "mtp" ]] && conv_args+=(--num_mtp "${TILERT_NUM_MTP:-3}") + else + conv_mod="tilert.models.preprocess.weight_converter" + conv_args=(--model_type "$TILERT_MODEL_TYPE" --model_dir "$MODEL_PATH" --save_dir "$TILERT_WEIGHTS_DIR") + fi + if [[ "$DRY_RUN" -eq 1 ]]; then + echo "DRY RUN: $PY -m $conv_mod ${conv_args[*]}" + exec 9>&-; return 0 + fi + echo "[weight_converter] using $conv_mod" + "$PY" -m "$conv_mod" "${conv_args[@]}" \ + 2>&1 | tee "$LOG_DIR/tilert_weight_converter_${host_name}.log" + local rc=${PIPESTATUS[0]} + exec 9>&- + if [[ $rc -ne 0 ]] || ! _tilert_weights_cached; then + echo "[weight_converter] ERROR: conversion failed (rc=$rc, per-rank index.json complete: $(_tilert_weights_cached && echo yes || echo no))" >&2 + return 1 + fi + echo "[weight_converter] conversion done and cached: $TILERT_WEIGHTS_DIR" +} + +start_decode() { + # shellcheck disable=SC2206 + local extra=( ${TILERT_DECODE_EXTRA_FLAGS} ) + if [[ "$TILERT_IS_AGENTIC" == "1" && "${SPEC_DECODING:-}" == "mtp" \ + && "${EVAL_ONLY:-false}" != "true" && "${RUN_EVAL:-false}" != "true" ]]; then + local curve="${TILERT_GOLDEN_AL_FILE:-${WS_PATH%/benchmarks/*}/golden_al_distribution/${MODEL_PREFIX:-glm5.3}_mtp.yaml}" + TILERT_SIMULATE_ACC_LEN="$("$PY" - "$curve" "${TILERT_THINKING_MODE:-thinking_on}" "${TILERT_NUM_MTP:-3}" <<'PYEOF' +import sys, yaml +path, thinking, tokens = sys.argv[1], sys.argv[2], int(sys.argv[3]) +data = yaml.safe_load(open(path)) +if not isinstance(data, dict) or len(data) != 1: + sys.exit(f"golden curve {path} must hold exactly one model key") +model, modes = next(iter(data.items())) +try: + value = float(modes[thinking][tokens]) +except (KeyError, TypeError, ValueError): + sys.exit(f"no golden acceptance for {model}/{thinking}/{tokens} draft tokens in {path}") +if not 1 <= value <= tokens + 1: + sys.exit(f"golden acceptance {value} out of range for {tokens} draft tokens") +print(f"{value:g}") +PYEOF +)" || { echo "[tilert] ERROR: golden AL lookup failed (curve=$curve)" >&2; exit 1; } + echo "[tilert] golden AL ${TILERT_SIMULATE_ACC_LEN} from $(basename "$curve") (thinking_on, K=${TILERT_NUM_MTP:-3})" + fi + + if [[ -n "${TILERT_SIMULATE_ACC_LEN:-}" && "${EVAL_ONLY:-false}" != "true" ]]; then + export TILERT_SIMULATE_ACC_LEN + export TILERT_SIMULATE_ACC_METHOD="${TILERT_SIMULATE_ACC_METHOD:-match-expected}" + echo "[decode] simulated acceptance: TILERT_SIMULATE_ACC_LEN=${TILERT_SIMULATE_ACC_LEN}" \ + "method=${TILERT_SIMULATE_ACC_METHOD} (output text is meaningless by design)" + else + unset TILERT_SIMULATE_ACC_LEN TILERT_SIMULATE_ACC_METHOD + echo "[decode] real MTP verification (no simulated acceptance)" + fi + local cmd=("$PY" -m tilert.pd_vllm.decode_server + --engine tilert --model "$TILERT_PROFILE" + --model-weights-dir "$TILERT_WEIGHTS_DIR" + --max-seq-len "$TILERT_MAX_MODEL_LEN" + --kv-cache-dtype "$DECODE_KV_DTYPE" --transport "$TILERT_TRANSPORT" + --ctrl-port "$DECODE_CTRL_PORT" --http-port "$DECODE_HTTP_PORT" + "${DECODE_MTP[@]}" "${extra[@]}") + log_and_run_bg decode "$LOG_DIR/decode_${host_name}.log" "${cmd[@]}" + DECODE_PID=$LAST_BG_PID +} + +start_prefill() { + for env_pair in ${TILERT_PREFILL_ENV}; do + export "${env_pair?}" + echo "[PREFILL_ENV] $env_pair" + done + local served=("$SERVED_MODEL_NAME") + [[ -n "$MODEL_NAME" && "$MODEL_NAME" != "$SERVED_MODEL_NAME" ]] && served+=("$MODEL_NAME") + # shellcheck disable=SC2206 + local extra=( ${TILERT_PREFILL_EXTRA_FLAGS} ) + local kv_cfg + kv_cfg=$(printf '{"kv_connector":"TileRTConnector","kv_connector_module_path":"tilert.pd_vllm.prefill_connector","kv_role":"kv_producer","kv_connector_extra_config":{"tilert_host":"%s","tilert_ctrl_port":%s,"tilert_model":"%s","tilert_max_seq_len":%s,"tilert_transport":"%s"}}' \ + "$DECODE_HOST" "$DECODE_CTRL_PORT" "$TILERT_PROFILE" "$TILERT_MAX_MODEL_LEN" "$TILERT_TRANSPORT") + local cmd=(vllm serve "$MODEL_PATH" + --served-model-name "${served[@]}" --port "$PREFILL_PORT" + --tensor-parallel-size "$PREFILL_TP_SIZE" --max-model-len "$TILERT_MAX_MODEL_LEN" + --enforce-eager --trust-remote-code --return-tokens-as-token-ids + --gpu-memory-utilization "$GPU_MEM_UTIL" --kv-cache-dtype "$PREFILL_KV_DTYPE" + --block-size "$PREFILL_BLOCK_SIZE" + "${PREFILL_SPEC[@]}" + --kv-transfer-config "$kv_cfg" + "${extra[@]}") + log_and_run_bg prefill "$LOG_DIR/prefill_${host_name}.log" "${cmd[@]}" + PREFILL_PID=$LAST_BG_PID +} + +start_router() { + local cmd=(env HIP_VISIBLE_DEVICES= ROCR_VISIBLE_DEVICES= CUDA_VISIBLE_DEVICES= + "$PY" -m tilert.pd_vllm.pd_router + --vllm-url "http://$PREFILL_HOST:$PREFILL_PORT" + --decode "$DECODE_HOST:$DECODE_CTRL_PORT:$DECODE_HTTP_PORT" + --host 0.0.0.0 --port "$ROUTER_PORT" --model-path "$MODEL_PATH" --parser "$TILERT_PARSER" + --queue-timeout "$TILERT_QUEUE_TIMEOUT") + log_and_run_bg router "$LOG_DIR/router_${host_name}.log" "${cmd[@]}" + ROUTER_PID=$LAST_BG_PID +} + +tcp_open() { (exec 3<>"/dev/tcp/$1/$2") 2>/dev/null; } + +wait_for_tcp() { + local host="$1" port="$2" timeout="${3:-600}" pid="${4:-}" + local deadline=$(( SECONDS + timeout )) + until tcp_open "$host" "$port"; do + if [[ -n "$pid" ]] && ! kill -0 "$pid" 2>/dev/null; then + echo "[wait_for_tcp] process $pid exited before $host:$port opened" >&2; return 2 + fi + if [[ $SECONDS -ge $deadline ]]; then + echo "[wait_for_tcp] timeout: $host:$port not open after ${timeout}s" >&2; return 1 + fi + sleep 5 + done + echo "[wait_for_tcp] $host:$port ready" +} + +wait_for_tcp_close() { + local host="$1" port="$2" pid="${3:-}" + while tcp_open "$host" "$port"; do + if [[ -n "$pid" ]] && ! kill -0 "$pid" 2>/dev/null; then + echo "[wait_for_tcp_close] process $pid exited while $host:$port is still open" >&2; return 2 + fi + sleep 10 + done + echo "[wait_for_tcp_close] $host:$port closed" +} + +copy_logs_to_shared() { + [[ "$DRY_RUN" -eq 0 ]] || return 0 + mkdir -p "$SHARED_LOG_DIR" && cp -r "$LOG_DIR"/. "$SHARED_LOG_DIR"/ \ + && echo "Copied $LOG_DIR -> $SHARED_LOG_DIR" \ + || echo "WARNING: failed to copy $LOG_DIR to $SHARED_LOG_DIR" >&2 +} +trap copy_logs_to_shared EXIT + +run_bench_and_eval() { + local rc=0 + wait_for_server_ready --port "$ROUTER_PORT" --server-log "$LOG_DIR/router_${host_name}.log" --server-pid "$ROUTER_PID" + + local profile_folder="$LOG_DIR/${ENGINE}_isl_${BENCH_INPUT_LEN}_osl_${BENCH_OUTPUT_LEN}" + mkdir -p "$profile_folder" + local prefill_gpus=$(( PREFILL_TP_SIZE * xP )) + local decode_gpus=$(( DECODE_TP_SIZE * yD )) + export TRANSFORMERS_VERBOSITY=error TOKENIZERS_PARALLELISM=false + + if [[ "${EVAL_ONLY:-false}" == "true" ]]; then + echo "EVAL_ONLY mode: skipping throughput benchmark" + else + local conc np export_file + for conc in ${BENCH_MAX_CONCURRENCY//x/ }; do + np=$(( conc * BENCH_NUM_PROMPTS_MULTIPLIER )) + [[ "$np" -lt 16 ]] && np=16 + export_file="${profile_folder}/concurrency_${conc}_req_rate_${BENCH_REQUEST_RATE}_gpus_$((prefill_gpus+decode_gpus))_ctx_${prefill_gpus}_gen_${decode_gpus}" + echo "[bench] conc=$conc num_prompts=$np isl=$BENCH_INPUT_LEN osl=$BENCH_OUTPUT_LEN -> $export_file.json" + run_benchmark_serving \ + --bench-serving-dir /workspace \ + --model "$SERVED_MODEL_NAME" --port "$ROUTER_PORT" \ + --backend openai-chat --endpoint /v1/chat/completions \ + --input-len "$BENCH_INPUT_LEN" --output-len "$BENCH_OUTPUT_LEN" \ + --random-range-ratio "$BENCH_RANDOM_RANGE_RATIO" \ + --num-prompts "$np" --max-concurrency "$conc" \ + --use-chat-template --tokenizer "$MODEL_PATH" --trust-remote-code \ + --server-pid "$ROUTER_PID" \ + --result-filename "$export_file" --result-dir /workspace/ \ + || { rc=$?; echo "[bench] WARNING: conc=$conc failed (rc=$rc)" >&2; } + echo "-----------------------------------------" + done + fi + + if [[ "${RUN_EVAL:-false}" == "true" ]]; then + run_lm_eval_on_router || rc=1 + fi + return $rc +} + +run_lm_eval_on_router() { + echo "Running lm-eval evaluation on the router..." + local ok=false _attempt + for _attempt in 1 2 3; do + if curl -sf --max-time 10 "http://0.0.0.0:${ROUTER_PORT}/health" >/dev/null 2>&1; then ok=true; break; fi + echo "Eval health check attempt $_attempt failed, retrying in 10s..."; sleep 10 + done + if [[ "$ok" != "true" ]]; then + echo "ERROR: router health check failed after 3 attempts; skipping eval" >&2 + return 1 + fi + local eval_failed=0 + pushd /workspace >/dev/null || return 1 + if [[ -n "${EVAL_CONC:-}" ]]; then + export EVAL_CONCURRENT_REQUESTS="${EVAL_CONC}" + else + export EVAL_CONCURRENT_REQUESTS=$(echo "$BENCH_MAX_CONCURRENCY" | tr 'x' '\n' | sort -n | tail -1) + fi + export MODEL="${MODEL:-$MODEL_PATH}" + export MAX_MODEL_LEN="${MAX_MODEL_LEN:-$TILERT_MAX_MODEL_LEN}" + if [[ "$DRY_RUN" -eq 1 ]]; then + echo "DRY RUN: run_eval --port $ROUTER_PORT (framework=${EVAL_FRAMEWORK:-lm-eval}, conc=${EVAL_CONCURRENT_REQUESTS})" + else + run_eval --port "$ROUTER_PORT" + local eval_rc=$? + if [[ $eval_rc -ne 0 ]]; then + echo "ERROR: run_eval exited rc=$eval_rc; preserving failure artifacts" >&2 + eval_failed=1 + else + export TP="${PREFILL_TP_SIZE}" CONC="${EVAL_CONCURRENT_REQUESTS}" EP_SIZE=1 + export PREFILL_TP="${PREFILL_TP_SIZE}" PREFILL_EP=1 PREFILL_NUM_WORKERS="${xP}" + export DECODE_TP="${DECODE_TP_SIZE}" DECODE_EP=1 DECODE_NUM_WORKERS="${yD}" + export DP_ATTENTION=false PREFILL_DP_ATTENTION=false DECODE_DP_ATTENTION=false + export ISL="${BENCH_INPUT_LEN}" OSL="${BENCH_OUTPUT_LEN}" + append_lm_eval_summary + fi + local eval_copy_dir="$LOG_DIR/eval_results" + if stage_eval_artifacts "$eval_copy_dir" /workspace "${EVAL_RESULT_DIR:-}"; then + echo "Eval artifacts staged in $eval_copy_dir" + else + echo "ERROR: failed to stage eval artifacts in $eval_copy_dir" >&2 + eval_failed=1 + fi + fi + popd >/dev/null || true + return $eval_failed +} + +run_agentic_replay() { + local rc=0 + wait_for_server_ready --port "$ROUTER_PORT" --server-log "$LOG_DIR/router_${host_name}.log" --server-pid "$ROUTER_PID" + cd /workspace || return 1 + + export PORT="$ROUTER_PORT" + export MODEL="$MODEL_PATH" # aiperf --tokenizer (local HF dir) + export SERVED_MODEL_NAME # aiperf --model (name the router/vLLM serve) + export DURATION="${DURATION:-1800}" + export MAX_MODEL_LEN="$TILERT_MAX_MODEL_LEN" + export AIPERF_SERVER_METRICS_URLS="${AIPERF_SERVER_METRICS_URLS:-http://${PREFILL_HOST}:${PREFILL_PORT}/metrics}" + export TRANSFORMERS_VERBOSITY=error TOKENIZERS_PARALLELISM=false + + local result_dir="$LOG_DIR/agentic" + local result_filename_base="${RESULT_FILENAME:-agentic_bench}" + mkdir -p "$result_dir" + + resolve_trace_source + install_agentic_deps + + local conc conc_result_dir + for conc in ${BENCH_MAX_CONCURRENCY//x/ }; do + echo "==========================================" + echo "Agentic trace replay: conc=$conc" + echo "==========================================" + conc_result_dir="$result_dir/conc_${conc}" + mkdir -p "$conc_result_dir" + export CONC="$conc" USERS="$conc" + build_replay_cmd "$conc_result_dir" + export RESULT_FILENAME="${result_filename_base}_conc${conc}" + if [[ "$DRY_RUN" -eq 1 ]]; then + echo "DRY RUN: $REPLAY_CMD" + elif ! run_agentic_replay_and_write_outputs "$conc_result_dir"; then + echo "WARNING: agentic trace replay for conc=$conc failed (replay or validation) after writing available results" >&2 + rc=1 + fi + echo "-----------------------------------------" + done + export RESULT_FILENAME="$result_filename_base" + return $rc +} + +echo "Waiting at the container creation barrier on $host_name" +if [[ "$DRY_RUN" -eq 1 ]]; then + echo "DRY RUN: skipping container creation barrier" +elif [[ "${SKIP_CONTAINER_BARRIER:-0}" == "1" ]]; then + echo "SKIP_CONTAINER_BARRIER=1: caller asserts all containers are up" +else + "$PY" "$WS_PATH/sync.py" barrier \ + --local-ip "${host_ip}" --local-port 5000 --enable-port \ + --node-ips "${IPADDRS}" --node-ports 5000 \ + --wait-for-all-ports --timeout 1800 || { echo "ERROR: container creation barrier failed" >&2; exit 1; } +fi + +case "$TILERT_ROLE" in + decode) + echo "${host_name}:${host_ip} is the TileRT Decode Node (Model: ${MODEL_NAME}, profile: ${TILERT_PROFILE})" + rdma_preflight || exit 1 + convert_weights || exit 1 + stage_tokenizer_files || exit 1 + start_decode + if [[ "$DRY_RUN" -eq 1 ]]; then + echo "DRY RUN: decode role complete"; exit 0 + fi + echo "Waiting for the router port ${PREFILL_HOST}:${ROUTER_PORT} to open (timeout ${ROUTER_WAIT}s)..." + wait_for_tcp "$PREFILL_HOST" "$ROUTER_PORT" "$ROUTER_WAIT" "$DECODE_PID"; wrc=$? + if [[ $wrc -eq 2 ]]; then + echo "ERROR: decode_server exited before the router came up (see $LOG_DIR/decode_${host_name}.log)" >&2 + tail -50 "$LOG_DIR/decode_${host_name}.log" >&2 || true + copy_logs_to_shared; exit 1 + elif [[ $wrc -ne 0 ]]; then + echo "WARNING: router never opened within ${ROUTER_WAIT}s; shutting down decode" >&2 + kill "$DECODE_PID" 2>/dev/null || true + copy_logs_to_shared; exit 1 + fi + echo "Waiting until the router port closes..." + wait_for_tcp_close "$PREFILL_HOST" "$ROUTER_PORT" "$DECODE_PID"; wrc=$? + if [[ $wrc -eq 2 ]]; then + echo "ERROR: decode_server died while the benchmark was running (see $LOG_DIR/decode_${host_name}.log)" >&2 + tail -50 "$LOG_DIR/decode_${host_name}.log" >&2 || true + copy_logs_to_shared; exit 1 + fi + echo "Killing the decode server" + kill "$DECODE_PID" 2>/dev/null || true + sleep 2 + copy_logs_to_shared + ;; + prefill) + echo "NODE INFO =======================================" + echo "Node List : ${SLURM_JOB_NODELIST:-}" + echo "Node IPs : ${IPADDRS}" + echo "Model : ${MODEL_NAME:-'Not specified'}" + echo "${host_name}:${host_ip} is the Prefill Node (vLLM + TileRTConnector) and Router Node" + echo "================================================" + rdma_preflight || exit 1 + echo "Waiting for the decode ctrl port ${DECODE_HOST}:${DECODE_CTRL_PORT} (timeout ${DECODE_WAIT}s)..." + if [[ "$DRY_RUN" -eq 0 ]]; then + wait_for_tcp "$DECODE_HOST" "$DECODE_CTRL_PORT" "$DECODE_WAIT" \ + || echo "WARNING: timed out waiting for the decode ctrl port; starting prefill anyway" >&2 + fi + start_prefill + if [[ "$DRY_RUN" -eq 0 ]]; then + wait_for_tcp "$PREFILL_HOST" "$PREFILL_PORT" "$PREFILL_WAIT" "$PREFILL_PID"; wrc=$? + if [[ $wrc -ne 0 ]]; then + echo "ERROR: vLLM prefill did not open ${PREFILL_HOST}:${PREFILL_PORT} (rc=$wrc, see $LOG_DIR/prefill_${host_name}.log)" >&2 + tail -50 "$LOG_DIR/prefill_${host_name}.log" >&2 || true + kill "$PREFILL_PID" 2>/dev/null || true + copy_logs_to_shared; exit 1 + fi + fi + start_router + if [[ "$DRY_RUN" -eq 1 ]]; then + echo "DRY RUN: prefill/router role complete"; exit 0 + fi + echo "Ready for benchmarking on ${host_name}:${host_ip}" + cd "$WS_PATH" || exit 1 + if [[ "$TILERT_IS_AGENTIC" == "1" ]]; then + run_agentic_replay; BENCH_RC=$? + else + run_bench_and_eval; BENCH_RC=$? + fi + copy_logs_to_shared + echo "Killing the router and the prefill server" + kill "$ROUTER_PID" "$PREFILL_PID" 2>/dev/null || true + sleep 2 + pkill -f "tilert.pd_vllm.pd_router" 2>/dev/null || true + pkill -f "vllm serve" 2>/dev/null || true + if [[ "$BENCH_RC" -ne 0 ]]; then + echo "ERROR: benchmark/eval reported rc=$BENCH_RC" >&2 + exit "$BENCH_RC" + fi + ;; + *) + echo "ERROR: unknown TILERT_ROLE='$TILERT_ROLE'" >&2; exit 2 ;; +esac + +echo "Script completed successfully" +exit 0 diff --git a/benchmarks/multi_node/amd_utils/setup_deps.sh b/benchmarks/multi_node/amd_utils/setup_deps.sh index 42c8a8ff4e..3917a64550 100644 --- a/benchmarks/multi_node/amd_utils/setup_deps.sh +++ b/benchmarks/multi_node/amd_utils/setup_deps.sh @@ -63,6 +63,119 @@ install_transformers_glm5() { _SETUP_INSTALLED+=("transformers-glm5") } +TILERT_VERSION="${TILERT_VERSION:-0.1.6}" +TILERT_PACKAGE="${TILERT_PACKAGE:-tilert}" +TILERT_PIP_SPEC="${TILERT_PIP_SPEC:-$TILERT_PACKAGE==$TILERT_VERSION}" +TILERT_PIP_INDEX_URL="${TILERT_PIP_INDEX_URL:-}" +TILERT_HTTP_DEPS="${TILERT_HTTP_DEPS:-fastapi uvicorn httpx}" +TILERT_TRANSPORT_DEPS="${TILERT_TRANSPORT_DEPS:-mooncake-transfer-engine-rocm>=0.3.13}" +TILERT_TRANSFORMERS_SPEC="${TILERT_TRANSFORMERS_SPEC:-transformers>=4.56}" + +_tilert_resolve_python() { + if [[ -n "${PY:-}" ]] && command -v "$PY" >/dev/null 2>&1; then :; else + PY="" + local c + for c in python3 python; do command -v "$c" >/dev/null 2>&1 && { PY="$c"; break; }; done + fi + [[ -n "$PY" ]] || { echo "[SETUP] ERROR: neither python3 nor python found"; exit 1; } + export PY + echo "[SETUP] interpreter PY=$PY ($(command -v "$PY"))" +} + +_tilert_installed_version() { + "$PY" - "$1" <<'PYEOF' 2>/dev/null +import sys +from importlib.metadata import version, PackageNotFoundError +try: + print(version(sys.argv[1])) +except PackageNotFoundError: + pass +PYEOF +} + +_tilert_pip() { + local a=(-m pip install --quiet --no-cache-dir) + [[ -n "$TILERT_PIP_INDEX_URL" ]] && a+=(--index-url "$TILERT_PIP_INDEX_URL") + "$PY" "${a[@]}" "$@" +} + +_tilert_install_missing() { + local probe="$1"; shift + [[ $# -gt 0 ]] || return 0 + if "$PY" -c "import $probe" 2>/dev/null; then + echo "[SETUP] $probe already present, skipping ($*)" + return 0 + fi + echo "[SETUP] installing $* (probe module '$probe' missing)" + _tilert_pip "$@" || { echo "[SETUP] ERROR: failed to install: $*"; exit 1; } + _SETUP_INSTALLED+=("$*") +} + +install_tilert_container_tools() { + if command -v ip >/dev/null 2>&1 && command -v curl >/dev/null 2>&1 \ + && command -v ibv_devices >/dev/null 2>&1; then + echo "[SETUP] Container RDMA/net tools already present" + return 0 + fi + echo "[SETUP] Installing iproute2 + curl + ibverbs userspace in container..." + apt-get update -q -y && apt-get install -q -y --no-install-recommends \ + iproute2 curl ibverbs-utils libibverbs1 librdmacm1 ibverbs-providers \ + && rm -rf /var/lib/apt/lists/* + if ! command -v ip >/dev/null 2>&1 || ! command -v curl >/dev/null 2>&1; then + echo "[SETUP] ERROR: failed to install iproute2/curl"; exit 1 + fi + _SETUP_INSTALLED+=("iproute2+curl+ibverbs") +} + +_tilert_install_wheel() { + local mode="$1" # full | no-deps + local have; have="$(_tilert_installed_version tilert)" + if [[ "$have" == "$TILERT_VERSION" ]]; then + echo "[SETUP] tilert $have already installed, skipping" + return 0 + fi + [[ -n "$have" ]] && echo "[SETUP] tilert $have installed, switching to pinned $TILERT_VERSION" + if [[ "$mode" == "no-deps" ]]; then + echo "[SETUP] installing $TILERT_PIP_SPEC --no-deps (connector plugin + router on top of the image's vLLM)" + _tilert_pip --no-deps "$TILERT_PIP_SPEC" || { echo "[SETUP] ERROR: failed to install $TILERT_PIP_SPEC (--no-deps)"; exit 1; } + else + echo "[SETUP] installing $TILERT_PIP_SPEC (TileRT ROCm build, official PyPI wheel)" + _tilert_pip "$TILERT_PIP_SPEC" || { echo "[SETUP] ERROR: failed to install $TILERT_PIP_SPEC"; exit 1; } + fi + have="$(_tilert_installed_version tilert)" + [[ "$have" == "$TILERT_VERSION" ]] || { + echo "[SETUP] ERROR: tilert is ${have:-not installed} after install, expected $TILERT_VERSION"; exit 1; } + _SETUP_INSTALLED+=("$TILERT_PACKAGE==$TILERT_VERSION($mode)") +} + +install_tilert_decode() { + install_tilert_container_tools + _tilert_install_wheel full + _tilert_install_missing uvicorn $TILERT_HTTP_DEPS + _tilert_install_missing mooncake.engine "$TILERT_TRANSPORT_DEPS" + _tilert_install_missing transformers "$TILERT_TRANSFORMERS_SPEC" + "$PY" -c "import tilert.pd_vllm.decode_server" 2>/dev/null || { + echo "[SETUP] ERROR: import tilert.pd_vllm.decode_server failed:" + "$PY" -c "import tilert.pd_vllm.decode_server" 2>&1 | tail -3 + exit 1; } + echo "[SETUP] tilert.pd_vllm.decode_server imports OK" +} + +install_tilert_prefill() { + local vllm_v; vllm_v="$(_tilert_installed_version vllm)" + if [[ -z "$vllm_v" ]]; then + echo "[SETUP] ERROR: no vLLM in the prefill image (PREFILL_IMAGE must be a vllm/vllm-openai-rocm image)." + exit 1 + fi + echo "[SETUP] prefill-side vLLM $vllm_v" + install_tilert_container_tools + _tilert_install_wheel no-deps + _tilert_install_missing mooncake.engine "$TILERT_TRANSPORT_DEPS" + "$PY" -c "import tilert.pd_vllm.prefill_connector" 2>/dev/null || { + echo "[SETUP] WARN: import tilert.pd_vllm.prefill_connector failed (vLLM will report again when loading the connector plugin):" + "$PY" -c "import tilert.pd_vllm.prefill_connector" 2>&1 | tail -3; } +} + if [[ "$ENGINE" == "vllm-disagg" ]]; then install_recipe_deps install_amd_quark @@ -72,6 +185,13 @@ if [[ "$ENGINE" == "vllm-disagg" ]]; then export RIXL_HOME export PATH="${UCX_HOME}/bin:/usr/local/bin/etcd:/root/.cargo/bin:${PATH}" export LD_LIBRARY_PATH="${UCX_HOME}/lib:${RIXL_HOME}/lib:${RIXL_HOME}/lib/x86_64-linux-gnu:${LD_LIBRARY_PATH:-}" +elif [[ "$ENGINE" == "tilert" ]]; then + _tilert_resolve_python + case "${TILERT_ROLE:-}" in + decode) install_tilert_decode ;; + prefill) install_tilert_prefill ;; + *) echo "[SETUP] ERROR: ENGINE=tilert needs TILERT_ROLE=decode|prefill (got '${TILERT_ROLE:-}')"; exit 1 ;; + esac else install_transformers_glm5 fi diff --git a/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh b/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh new file mode 100644 index 0000000000..3b750b79a0 --- /dev/null +++ b/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh @@ -0,0 +1,84 @@ +#!/usr/bin/env bash + +source "$(dirname "$0")/../benchmark_lib.sh" + +check_env_vars \ + CONC_LIST \ + ISL \ + OSL \ + IMAGE \ + SPEC_DECODING \ + MODEL_PATH \ + PREFILL_NUM_WORKERS \ + PREFILL_TP \ + PREFILL_EP \ + PREFILL_DP_ATTN \ + DECODE_NUM_WORKERS \ + DECODE_TP \ + DECODE_EP \ + DECODE_DP_ATTN \ + PREFILL_NODES \ + DECODE_NODES \ + RANDOM_RANGE_RATIO \ + FRAMEWORK \ + PREFILL_IMAGE + +if [[ -n "$SLURM_JOB_ID" ]]; then + echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME" +fi + +set -x + +cd "$GITHUB_WORKSPACE/benchmarks/multi_node/amd_utils" || exit 1 + +export TIME_LIMIT="${TIME_LIMIT:-08:00:00}" +export MODEL_PATH=$MODEL_PATH +export MODEL_NAME=$MODEL_NAME +export CONTAINER_IMAGE=$IMAGE +export PREFILL_IMAGE + +if [[ "$PREFILL_NODES" -ne 1 || "$DECODE_NODES" -ne 1 || \ + "$PREFILL_NUM_WORKERS" -ne 1 || "$DECODE_NUM_WORKERS" -ne 1 ]]; then + echo "Error: tilert supports exactly 1 prefill node/worker + 1 decode node/worker" \ + "(got PREFILL_NODES=$PREFILL_NODES x$PREFILL_NUM_WORKERS, DECODE_NODES=$DECODE_NODES x$DECODE_NUM_WORKERS)" >&2 + exit 1 +fi + +export TILERT_VERSION="${TILERT_VERSION:-0.1.6}" +export TILERT_PROFILE="${TILERT_PROFILE:-glm5_2}" +export TILERT_MODEL_TYPE="${TILERT_MODEL_TYPE:-glm-5}" +export TILERT_MAX_MODEL_LEN="${TILERT_MAX_MODEL_LEN:-${MAX_MODEL_LEN:-202752}}" +export DECODE_MTP_SIZE="${DECODE_MTP_SIZE:-$([[ "$SPEC_DECODING" == "mtp" ]] && echo 1 || echo 0)}" +export TILERT_TRANSPORT="${TILERT_TRANSPORT:-mooncake}" +export PREFILL_KV_DTYPE="${PREFILL_KV_DTYPE:-auto}" +export DECODE_KV_DTYPE="${DECODE_KV_DTYPE:-bf16}" +export TILERT_PARSER="${TILERT_PARSER:-none}" +export ROUTER_PORT="${ROUTER_PORT:-30000}" +export TILERT_QUEUE_TIMEOUT="${TILERT_QUEUE_TIMEOUT:-0}" + +if [[ "${PREFILL_EP:-1}" -ne 1 || "${DECODE_EP:-1}" -ne 1 || \ + "$PREFILL_DP_ATTN" == "true" || "$DECODE_DP_ATTN" == "true" ]]; then + echo "Error: tilert runs pure TP8 on both roles; ep must be 1 and dp-attn false" >&2 + exit 1 +fi +export PREFILL_ENABLE_EP=false +export PREFILL_ENABLE_DP=false +export DECODE_ENABLE_EP=false +export DECODE_ENABLE_DP=false + +JOB_ID=$(bash ./submit.sh $PREFILL_NODES \ + $PREFILL_NUM_WORKERS \ + $DECODE_NODES \ + $DECODE_NUM_WORKERS \ + $ISL $OSL "${CONC_LIST// /x}" inf \ + ${PREFILL_ENABLE_EP} ${PREFILL_ENABLE_DP} \ + ${DECODE_ENABLE_EP} ${DECODE_ENABLE_DP} \ + ${PREFILL_TP} ${DECODE_TP} \ + ${RANDOM_RANGE_RATIO}) + +if [[ $? -ne 0 ]]; then + echo "Failed to submit job" >&2 + exit 1 +fi + +echo "$JOB_ID" diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 9b4fcadeb7..24810f9fff 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1472,3 +1472,84 @@ dsv41flash-fp4-mi355x-vllm-agentic-dspark: search-space: # Follow upstream AMD Engram defaults; omit the CUDA-only config flag. - { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32] } +glm5.3-fp8-mi355x-tilert: + image: ghcr.io/tile-ai/tilert-rocm-decode:0.1.6 + model: zai-org/GLM-5.3 + model-prefix: glm5.3 + runner: cluster:mi355x-amds + precision: fp8 + framework: tilert + router: { name: tilert-pd-router, version: "0.1.6" } + multinode: true + disagg: true + kv-p2p-transfer: mooncake + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - spec-decoding: "mtp" + conc-list: [1] + prefill: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + additional-settings: + - "PREFILL_IMAGE=ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6" + - "PREFILL_NODES=1" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + additional-settings: + - "DECODE_NODES=1" + +# Speculative decoding on an agentic scenario must run with simulated +# synthetic acceptance at the committed golden AL for this model, thinking mode +# and draft length (docs/PR_REVIEW_CHECKLIST.md), and a submission may not +# substitute its own target. Nothing about that target is written here: +# AGENTS.md forbids hard-coding an acceptance length in a master config, so +# server_tilert.sh reads golden_al_distribution/glm5.3_mtp.yaml at launch and +# fails the run if the curve or the draft length is missing. +# The curve is consumed in the same units as every other framework here, and +# AgentX replays run with thinking on. +# +# NOTE FOR REVIEWERS: that curve is GLM-5.2's, copied because no SPEED-Bench run +# on GLM-5.3 exists and the two share a base. It is committed as provisional and +# labelled as such in the file. Flagging it rather than letting it read as a +# measured 5.3 curve -- please say if you would rather see a measured curve, a +# non-MTP agentic entry, or a waiver. +glm5.3-fp8-mi355x-tilert-agentic: + image: ghcr.io/tile-ai/tilert-rocm-decode:0.1.6 + model: zai-org/GLM-5.3 + model-prefix: glm5.3 + runner: cluster:mi355x-amds + precision: fp8 + framework: tilert + router: { name: tilert-pd-router, version: "0.1.6" } + multinode: true + disagg: true + kv-p2p-transfer: mooncake + scenarios: + agentic-coding: + - search-space: + - spec-decoding: "mtp" + conc-list: [1] + prefill: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + additional-settings: + - "PREFILL_IMAGE=ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6" + - "PREFILL_NODES=1" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + additional-settings: + - "DECODE_NODES=1" + - "TILERT_EXTRA_ENV=GLM5_AR_N=2" diff --git a/golden_al_distribution/glm5.3_mtp.yaml b/golden_al_distribution/glm5.3_mtp.yaml new file mode 100644 index 0000000000..4933511703 --- /dev/null +++ b/golden_al_distribution/glm5.3_mtp.yaml @@ -0,0 +1,19 @@ +# PROVISIONAL — copied from glm5.2_mtp.yaml; NOT measured on GLM-5.3. +# +# GLM-5.3 shares GLM-5.2's base and layer topology (its config.json differs only +# by the fp8 quantization block), so the GLM-5.2 curve is the closest committed +# reference. Only the K=3 cell is carried over, because the ROCm TileRT GLM +# profile builds MTP at draft depth 3 and no other cell can be exercised. +# Replace this file with a SPEED-Bench run on glm-5.3-fp8 when one exists. +# +# Inherited value provenance (GLM-5.2, unchanged): +# Source GitHub Actions run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/28058352479 +# Acceptance Length (AL) reference values measured with SPEED-Bench. +# dataset: coding | temperature: 1.0 | top_p: 0.95 | output_len: 4096 +# thinking_on chat_template_kwargs: {"enable_thinking": true} +# Measured on glm-5.2-fp8 (B300, vLLM MTP), per num_speculative_tokens. +# +# key = num_speculative_tokens (MTP level); value = golden AL +glm-5.3-fp8: + thinking_on: + 3: 2.99 diff --git a/perf-changelog.yaml b/perf-changelog.yaml index c5b4309f5f..2123dfd1d2 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8455,3 +8455,22 @@ description: - "Update B200 vLLM AgentX to DSpark6 and a new image with TP8 and DEP8 configurations." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3274 + +- config-keys: + - glm5.3-fp8-mi355x-tilert + description: + - "Add GLM-5.3 FP8 MI355X prefill/decode-disaggregated Single-turn 8k1k benchmark via TileRT: upstream vLLM ROCm prefill (TileRTConnector, kv_producer) + TileRT decode_server + OpenAI-compatible pd_router, framework=tilert, mirroring the B200 glm5.1-fp8-b200-tilert entry." + - "Two images per job, both first-party and neither carrying the TileRT wheel: the decode rank runs ghcr.io/tile-ai/tilert-rocm-decode:0.1.6 (ROCm PyTorch runtime plus the mooncake ROCm transfer engine built from source) and the prefill rank runs ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6 (the upstream vllm/vllm-openai-rocm release image plus that same transfer engine, with vLLM itself unpatched and no file inside it modified), selected per rank via PREFILL_IMAGE in prefill.additional-settings. setup_deps.sh pip-installs the official tilert==0.1.6 wheel and the router HTTP deps into both ranks at container start (full install on decode, --no-deps connector plugin on prefill), so neither image is pinned to a wheel version; this mirrors the B200 glm5.1-fp8-b200-tilert pattern of a first-party TileRT decode image. The transfer engine is built from source because the published mooncake ROCm wheels are compiled without ENABLE_MULTI_PROTOCOL, which leaves the cross-host locality gate (kvcache-ai/Mooncake#2753) out of the binary so every cross-node KV transfer is treated as node-local and fails in hipIpcOpenMemHandle." + - "KV moves P->D over mooncake (kv-p2p-transfer: mooncake); both roles use a bf16 KV cache (prefill --kv-cache-dtype auto, decode bf16) because the ROCm vLLM sparse-MLA backend has no fp8_ds_mla dtype. max-model-len 202752 on both roles; MTP wired via spec-decoding=mtp (vLLM speculative-config mtp with 1 draft token on prefill, --with-mtp on decode). TileRT decode is bs=1 only, so conc-list is [1]; topology 1 prefill node (TP8) + 1 decode node (TP8)." + - "Orchestration: runners/launch_mi355x-amds.sh dispatches framework=tilert through the existing amd_utils chain (submit.sh -> job.slurm -> server.sh -> new server_tilert.sh); job.slurm selects the image per rank and a tilert env block, setup_deps.sh gains a per-role tilert install branch, models_tilert.yaml holds the GLM-5.3 profile defaults. TileRT decode weights are converted once from the HF checkpoint under a flock and cached next to it on the shared model volume. Cross-node teardown uses the amd_utils router-port-close barrier." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3328 + +- config-keys: + - glm5.3-fp8-mi355x-tilert-agentic + scenario-type: + - agentic-coding + description: + - "Add GLM-5.3 FP8 MI355X AgentX (agentic-coding) via TileRT prefill/decode disaggregation with MTP, 1 prefill node (TP8) + 1 decode node (TP8), concurrency [1], mirroring the B200 glm5.1-fp8-b200-tilert-agentic entry on the amd_utils orchestration." + - "Same two-image recipe as glm5.3-fp8-mi355x-tilert (decode ghcr.io/tile-ai/tilert-rocm-decode:0.1.6, prefill ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6, tilert==0.1.6 pip-installed into both at container start), bf16 KV cache on both roles, kv-p2p-transfer: mooncake, kv-offloading none." + - "The pd_router runs with --parser none and a 1800-second queue timeout so requests wait on the bs=1 decode engine instead of failing fast; AgentX traces are filtered at the 202752-token context both roles are launched with; AIPerf scrapes only the vLLM prefill Prometheus endpoint (TileRT decode exposes no /metrics route)." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3328 diff --git a/runners/launch_mi355x-amds.sh b/runners/launch_mi355x-amds.sh index 682e3ff497..6f13b3f2d7 100644 --- a/runners/launch_mi355x-amds.sh +++ b/runners/launch_mi355x-amds.sh @@ -74,9 +74,8 @@ if [[ "$IS_MULTINODE" == "true" ]]; then fi SCRIPT_NAME="${EXP_NAME%%_*}_${PRECISION}_mi355x_${FRAMEWORK}.sh" - if [[ "$FRAMEWORK" == "sglang-disagg" ]] || [[ "$FRAMEWORK" == "vllm-disagg" ]] || [[ "$FRAMEWORK" == "atom-disagg" ]]; then + if [[ "$FRAMEWORK" == "sglang-disagg" ]] || [[ "$FRAMEWORK" == "vllm-disagg" ]] || [[ "$FRAMEWORK" == "atom-disagg" ]] || [[ "$FRAMEWORK" == "tilert" ]]; then # Agentic recipes under multi_node/agentic/ export the HiCache tunables; - # fixed-seq-len recipes live at the multi_node/ root. if [[ "${SCENARIO_SUBDIR}" == "agentic/" ]]; then BENCHMARK_SUBDIR="multi_node/agentic" else @@ -116,9 +115,6 @@ if [[ "$IS_MULTINODE" == "true" ]]; then set -x - - - if [[ "${EVAL_ONLY}" != "true" && "${IS_AGENTIC}" != "1" ]]; then cat > collect_latest_results.py <<'PY' import os, sys From d924515e50b6bbb47ff3c584aed99b1b0948a0d8 Mon Sep 17 00:00:00 2001 From: CrimsonDump Date: Mon, 21 Sep 2026 19:28:32 +0800 Subject: [PATCH 02/25] =?UTF-8?q?fix(tilert):=20follow=20repo=20Bash=20con?= =?UTF-8?q?ventions=20and=20correct=20MTP=20draft=20depth=20/=20=E4=BF=AE?= =?UTF-8?q?=E6=AD=A3=20TileRT=20=E9=85=8D=E6=96=B9=E7=9A=84=20Bash=20?= =?UTF-8?q?=E8=A7=84=E8=8C=83=E4=B8=8E=20MTP=20=E8=8D=89=E7=A8=BF=E6=B7=B1?= =?UTF-8?q?=E5=BA=A6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review follow-ups on the GLM-5.3 FP8 MI355X TileRT recipes. - server_tilert.sh no longer invents defaults for caller-supplied configuration. Every value now arrives from the recipe through job.slurm and is validated with check_env_vars, mirroring server_atom.sh. The same applies to the two recipes and to the tilert branch of setup_deps.sh. - Draft depth is consistent. DECODE_MTP_SIZE=3 is the TileRT decode draft depth, used by the weight converter and as the key the golden acceptance curve is read at; PREFILL_SPEC_TOKENS=1 is the vLLM prefill rank's MTP layer. Previously the converter and the curve used a hard-coded 3 while the recipes exported an unused DECODE_MTP_SIZE=1. - THINKING_MODE and MODEL_PREFIX are forwarded into the container and validated, so the golden-acceptance selection no longer falls back to built-in guesses. - job.slurm drops MODEL_NAME="${MODEL_NAME:-None}", which masked a missing model name for every engine, not just tilert. - models_tilert.yaml keeps only model-owned engine environment; profile, model_type and max_model_len moved to the recipes, which removes the stale glm-5_2 model_type that the recipe already overrode. - launch_mi355x-amds.sh and server.sh no longer carry unrelated comment and blank-line deletions. - MODELS.md/MODELS_zh.md and the two changelog entries point at the upstream PR and carry the correct date. 以评审意见修正 GLM-5.3 FP8 MI355X 的 TileRT 配方：server_tilert.sh 不再为调用方 提供的配置自造默认值，全部改由配方经 job.slurm 传入并用 check_env_vars 校验， 与 server_atom.sh 保持一致；统一 MTP 草稿深度（decode 侧 DECODE_MTP_SIZE=3， 供权重转换与黄金接受率曲线取值，prefill 侧 PREFILL_SPEC_TOKENS=1）；将 THINKING_MODE 与 MODEL_PREFIX 传入容器并校验；移除 job.slurm 中会掩盖所有引擎 MODEL_NAME 缺失的兜底；models_tilert.yaml 只保留模型自有的引擎环境；撤销 launcher 与 server.sh 中与本次改动无关的删除；文档与 changelog 链接改为上游 PR。 --- MODELS.md | 2 +- MODELS_zh.md | 2 +- .../agentic/glm5.3_fp8_mi355x_tilert.sh | 67 +++++-- benchmarks/multi_node/amd_utils/job.slurm | 46 +++-- .../multi_node/amd_utils/models_tilert.yaml | 8 +- benchmarks/multi_node/amd_utils/server.sh | 2 + .../multi_node/amd_utils/server_tilert.sh | 168 +++++++----------- benchmarks/multi_node/amd_utils/setup_deps.sh | 19 +- .../multi_node/glm5.3_fp8_mi355x_tilert.sh | 56 ++++-- perf-changelog.yaml | 6 +- runners/launch_mi355x-amds.sh | 4 + 11 files changed, 205 insertions(+), 175 deletions(-) diff --git a/MODELS.md b/MODELS.md index ef0fd5d076..b0009f5e7d 100644 --- a/MODELS.md +++ b/MODELS.md @@ -153,7 +153,7 @@ Other offloading tiers, including NVMe KV cache offloading, are outside the init | Model architecture class | Prefix | Date added | Active scenarios | Deprecated scenarios | |---|---|---|---|---| | DeepSeek-V4.1-Flash | `dsv41flash` | 2026-09-10 | Agentic coding (vLLM: DSpark, Engram UVA offload; SGLang: DSpark arms added per SKU from 2026-09-17; GPU validation pending) | | -| GLM-5.3 | `glm5.3` | 2026-09-08 ([#3328](https://github.com/SemiAnalysisAI/InferenceX/pull/3328)) | Single-turn 8k1k, Agentic coding (MTP only, per the Deprecation Notice) | Single-turn 1k1k (deprecated for all models before this model was added; never run) | +| GLM-5.3 | `glm5.3` | 2026-09-21 ([#3330](https://github.com/SemiAnalysisAI/InferenceX/pull/3330)) | Single-turn 8k1k, Agentic coding (MTP only, per the Deprecation Notice) | Single-turn 1k1k (deprecated for all models before this model was added; never run) | | Qwen3.8-Flash-Next | `qwen3.8next` | 2026-08-26 ([#2742](https://github.com/SemiAnalysisAI/InferenceX/pull/2742)) | Agentic coding | | | Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | Agentic coding (DSpark may be disabled for better Pareto points) | Standalone non-DSpark A/B baseline (not required from day 0) | | GLM-5.2 | `glm5.2` | 2026-07-18 ([#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)) | Agentic coding (non-MTP points remain eligible under the Pareto policy; see Deprecation Notice) | | diff --git a/MODELS_zh.md b/MODELS_zh.md index ece612173d..8066b809f4 100644 --- a/MODELS_zh.md +++ b/MODELS_zh.md @@ -153,7 +153,7 @@ InferenceX 支持 SGLang 和 vLLM 双方的维护者，并响应 AI 实验室和 | 模型架构类别 | 前缀 | 加入日期 | 启用场景 | 已弃用场景 | |---|---|---|---|---| | DeepSeek-V4.1-Flash | `dsv41flash` | 2026-09-10 | 智能体编码（DSpark、Engram UVA 卸载；GPU 待验证） | | -| GLM-5.3 | `glm5.3` | 2026-09-08（[#3328](https://github.com/SemiAnalysisAI/InferenceX/pull/3328)） | 单轮 8k1k、智能体编码（仅 MTP，见弃用公告） | 单轮 1k1k（在本模型加入前已对所有模型弃用，从未运行） | +| GLM-5.3 | `glm5.3` | 2026-09-21（[#3330](https://github.com/SemiAnalysisAI/InferenceX/pull/3330)） | 单轮 8k1k、智能体编码（仅 MTP，见弃用公告） | 单轮 1k1k（在本模型加入前已对所有模型弃用，从未运行） | | Qwen3.8-Flash-Next | `qwen3.8next` | 2026-08-26（[#2742](https://github.com/SemiAnalysisAI/InferenceX/pull/2742)） | 智能体编码 | | | Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | 智能体编码（可关闭 DSpark 以获得更优帕累托点） | 独立非 DSpark A/B 基线（自第 0 天起即不要求） | | GLM-5.2 | `glm5.2` | 2026-07-18（[#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)） | 智能体编码（非 MTP 数据点仍可按帕累托策略参与发布；见弃用公告） | | diff --git a/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh b/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh index f4d6bc943f..f87c6849b0 100644 --- a/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh +++ b/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh @@ -10,6 +10,7 @@ check_env_vars \ IMAGE \ SPEC_DECODING \ MODEL_PATH \ + MODEL_NAME \ PREFILL_NUM_WORKERS \ PREFILL_TP \ PREFILL_EP \ @@ -22,6 +23,9 @@ check_env_vars \ DECODE_NODES \ RANDOM_RANGE_RATIO \ DURATION \ + MODEL_PREFIX \ + PRECISION \ + RESULT_FILENAME \ KV_OFFLOADING \ IS_AGENTIC \ FRAMEWORK \ @@ -35,18 +39,13 @@ set -x cd "$GITHUB_WORKSPACE/benchmarks/multi_node/amd_utils" || exit 1 -export TIME_LIMIT="${TIME_LIMIT:-08:00:00}" +export TIME_LIMIT=08:00:00 export MODEL_PATH=$MODEL_PATH export MODEL_NAME=$MODEL_NAME export CONTAINER_IMAGE=$IMAGE export PREFILL_IMAGE -export MODEL_PREFIX="${MODEL_PREFIX:-glm5.3}" -export PRECISION="${PRECISION:-fp8}" -export RESULT_FILENAME="${RESULT_FILENAME:-${RUNNER_NAME:-glm5.3-fp8-tilert-agentic}}" -export DECODE_MTP_SIZE="${DECODE_MTP_SIZE:-$([[ "$SPEC_DECODING" == "mtp" ]] && echo 1 || echo 0)}" - -export DURATION="${DURATION:-1800}" +export RESULT_FILENAME if [[ "$PREFILL_NODES" -ne 1 || "$DECODE_NODES" -ne 1 || \ "$PREFILL_NUM_WORKERS" -ne 1 || "$DECODE_NUM_WORKERS" -ne 1 ]]; then @@ -55,24 +54,54 @@ if [[ "$PREFILL_NODES" -ne 1 || "$DECODE_NODES" -ne 1 || \ exit 1 fi -export KV_OFFLOADING="${KV_OFFLOADING:-none}" if [[ "$KV_OFFLOADING" != "none" ]]; then echo "Error: tilert has no KV offload backend; kv-offloading must be 'none' (got '$KV_OFFLOADING')" >&2 exit 1 fi -export TILERT_VERSION="${TILERT_VERSION:-0.1.6}" -export TILERT_PROFILE="${TILERT_PROFILE:-glm5_2}" -export TILERT_MODEL_TYPE="${TILERT_MODEL_TYPE:-glm-5}" -export TILERT_MAX_MODEL_LEN="${TILERT_MAX_MODEL_LEN:-202752}" -export TILERT_TRANSPORT="${TILERT_TRANSPORT:-mooncake}" -export PREFILL_KV_DTYPE="${PREFILL_KV_DTYPE:-auto}" -export DECODE_KV_DTYPE="${DECODE_KV_DTYPE:-bf16}" -export TILERT_PARSER="${TILERT_PARSER:-none}" -export ROUTER_PORT="${ROUTER_PORT:-30000}" -export TILERT_QUEUE_TIMEOUT="${TILERT_QUEUE_TIMEOUT:-1800}" +# TileRT configuration. Every value is explicit here: server_tilert.sh +# validates each one with check_env_vars and supplies no defaults of its own. +export TILERT_VERSION=0.1.6 +export TILERT_PROFILE=glm5_2 # decode_server --model (TileRT model profile) +export TILERT_MODEL_TYPE=glm-5 # weight_converter --model_type (fallback converter) +export TILERT_MODEL_PKG=glm_5_2_rocm # per-model converter package, preferred when importable +export SERVED_MODEL_NAME=glm5_2 +export TILERT_MAX_MODEL_LEN=202752 +export TILERT_TRANSPORT=mooncake +export TILERT_PARSER=none +export TILERT_RDMA_STRICT=0 +export TILERT_CONVERT_LOCK_WAIT=21600 +export TILERT_SIMULATE_ACC_METHOD=match-expected +export TILERT_WEIGHTS_DIR="/models/${MODEL_NAME}-tilert-tp${DECODE_TP}" +export PREFILL_KV_DTYPE=auto +# The ROCm backend supports block sizes [1, 64] and vLLM picks 1, which makes +# the connector's KI plane copy fail and MLA address the wrong rows. +export PREFILL_BLOCK_SIZE=64 +export DECODE_KV_DTYPE=bf16 # the ROCm sparse-MLA backend has no fp8_ds_mla +export GPU_MEM_UTIL=0.75 +export SKIP_CONTAINER_BARRIER=0 +export ROUTER_PORT=30000 +export PREFILL_PORT=8000 +export DECODE_CTRL_PORT=5556 +export DECODE_HTTP_PORT=5557 +export DECODE_WAIT=7200 # prefill waits for the decode ctrl port +export PREFILL_WAIT=3600 # prefill waits for its own vLLM port +export ROUTER_WAIT=10800 # decode waits for the router port to open + +if [[ "$SPEC_DECODING" == "mtp" ]]; then + # TileRT decode drafts at depth 3 (the only depth the ROCm GLM profile + # builds) and the golden acceptance curve is keyed on it. The vLLM prefill + # rank only has to materialise the MTP layer's KV, so it runs at 1. + export DECODE_MTP_SIZE=3 + export PREFILL_SPEC_TOKENS=1 +else + export DECODE_MTP_SIZE=0 + export PREFILL_SPEC_TOKENS=0 +fi +export TILERT_QUEUE_TIMEOUT=1800 # requests wait on the bs=1 decode engine +export THINKING_MODE=thinking_on -if [[ "${PREFILL_EP:-1}" -ne 1 || "${DECODE_EP:-1}" -ne 1 || \ +if [[ "$PREFILL_EP" -ne 1 || "$DECODE_EP" -ne 1 || \ "$PREFILL_DP_ATTN" == "true" || "$DECODE_DP_ATTN" == "true" ]]; then echo "Error: tilert runs pure TP8 on both roles; ep must be 1 and dp-attn false" >&2 exit 1 diff --git a/benchmarks/multi_node/amd_utils/job.slurm b/benchmarks/multi_node/amd_utils/job.slurm index 09ee78af38..6b69cb6d07 100755 --- a/benchmarks/multi_node/amd_utils/job.slurm +++ b/benchmarks/multi_node/amd_utils/job.slurm @@ -59,7 +59,6 @@ if [[ "$ENGINE" == "tilert" && -z "${PREFILL_IMAGE:-}" ]]; then exit 1 fi -MODEL_NAME="${MODEL_NAME:-None}" # Resolve the models.yaml entry the same way server_sglang.sh does: agentic runs # (IS_AGENTIC) use the '-AgentX' recipe, non-agentic disaggregated runs use # '-DI'. Fall back to the bare model name if the variant key is absent. @@ -534,25 +533,34 @@ elif [[ "$ENGINE" == "tilert" ]]; then DOCKER_ENV_ENGINE=( -e MODEL_PATH=$DOCKER_MODEL_PATH -e PREFILL_IMAGE=${PREFILL_IMAGE} - -e TILERT_VERSION=${TILERT_VERSION:-} - -e TILERT_PIP_SPEC=${TILERT_PIP_SPEC:-} - -e TILERT_PIP_INDEX_URL=${TILERT_PIP_INDEX_URL:-} - -e TILERT_PROFILE=${TILERT_PROFILE:-} - -e TILERT_MODEL_TYPE=${TILERT_MODEL_TYPE:-} - -e TILERT_MAX_MODEL_LEN=${TILERT_MAX_MODEL_LEN:-} - -e TILERT_TRANSPORT=${TILERT_TRANSPORT:-} - -e PREFILL_KV_DTYPE=${PREFILL_KV_DTYPE:-} - -e DECODE_KV_DTYPE=${DECODE_KV_DTYPE:-} - -e TILERT_PARSER=${TILERT_PARSER:-} - -e TILERT_QUEUE_TIMEOUT=${TILERT_QUEUE_TIMEOUT:-} - -e TILERT_WEIGHTS_DIR=${TILERT_WEIGHTS_DIR:-} - -e TILERT_RDMA_STRICT=${TILERT_RDMA_STRICT:-0} + -e TILERT_VERSION=${TILERT_VERSION} + -e TILERT_PROFILE=${TILERT_PROFILE} + -e TILERT_MODEL_TYPE=${TILERT_MODEL_TYPE} + -e TILERT_MODEL_PKG=${TILERT_MODEL_PKG} + -e TILERT_MAX_MODEL_LEN=${TILERT_MAX_MODEL_LEN} + -e TILERT_TRANSPORT=${TILERT_TRANSPORT} + -e TILERT_PARSER=${TILERT_PARSER} + -e TILERT_QUEUE_TIMEOUT=${TILERT_QUEUE_TIMEOUT} + -e TILERT_WEIGHTS_DIR=${TILERT_WEIGHTS_DIR} + -e TILERT_RDMA_STRICT=${TILERT_RDMA_STRICT} + -e TILERT_CONVERT_LOCK_WAIT=${TILERT_CONVERT_LOCK_WAIT} + -e TILERT_SIMULATE_ACC_METHOD=${TILERT_SIMULATE_ACC_METHOD} -e \"TILERT_EXTRA_ENV=${TILERT_EXTRA_ENV:-}\" - -e GPU_MEM_UTIL=${GPU_MEM_UTIL:-} - -e ROUTER_PORT=${ROUTER_PORT:-30000} - -e PREFILL_PORT=${PREFILL_PORT:-8000} - -e DECODE_CTRL_PORT=${DECODE_CTRL_PORT:-5556} - -e DECODE_HTTP_PORT=${DECODE_HTTP_PORT:-5557} + -e SERVED_MODEL_NAME=${SERVED_MODEL_NAME} + -e PREFILL_KV_DTYPE=${PREFILL_KV_DTYPE} + -e PREFILL_BLOCK_SIZE=${PREFILL_BLOCK_SIZE} + -e PREFILL_SPEC_TOKENS=${PREFILL_SPEC_TOKENS} + -e DECODE_KV_DTYPE=${DECODE_KV_DTYPE} + -e GPU_MEM_UTIL=${GPU_MEM_UTIL} + -e PREFILL_PORT=${PREFILL_PORT} + -e DECODE_CTRL_PORT=${DECODE_CTRL_PORT} + -e DECODE_HTTP_PORT=${DECODE_HTTP_PORT} + -e DECODE_WAIT=${DECODE_WAIT} + -e PREFILL_WAIT=${PREFILL_WAIT} + -e ROUTER_WAIT=${ROUTER_WAIT} + -e SKIP_CONTAINER_BARRIER=${SKIP_CONTAINER_BARRIER} + # Golden-acceptance selection on agentic MTP runs; unset elsewhere. + -e THINKING_MODE=${THINKING_MODE:-} -e IBDEVICES=${IBDEVICES:-} -e PYTHONPYCACHEPREFIX=/tmp/pycache ) diff --git a/benchmarks/multi_node/amd_utils/models_tilert.yaml b/benchmarks/multi_node/amd_utils/models_tilert.yaml index a84ef00b1d..1697059d54 100644 --- a/benchmarks/multi_node/amd_utils/models_tilert.yaml +++ b/benchmarks/multi_node/amd_utils/models_tilert.yaml @@ -1,8 +1,6 @@ - +# Model-owned engine settings for ENGINE=tilert. Everything the caller owns +# (profile, topology, ports, dtypes, draft depth) is set by the recipe, not here. GLM-5.3: - profile: "glm5_2" - model_type: "glm-5_2" - max_model_len: 202752 - prefill_extra_flags: "" prefill_env: "VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 VLLM_ENGINE_READY_TIMEOUT_S=10800" + prefill_extra_flags: "" decode_extra_flags: "" diff --git a/benchmarks/multi_node/amd_utils/server.sh b/benchmarks/multi_node/amd_utils/server.sh index eddf3a2e44..08cb65d2a9 100755 --- a/benchmarks/multi_node/amd_utils/server.sh +++ b/benchmarks/multi_node/amd_utils/server.sh @@ -6,6 +6,8 @@ source "$(dirname "${BASH_SOURCE[0]}")/../../benchmark_lib.sh" --validation-only # ENGINE=sglang-disagg (default) -> server_sglang.sh (SGLang + MoRI) # ENGINE=vllm-disagg -> server_vllm.sh (vLLM + Nixl/MoRI-IO) # ENGINE=atom-disagg -> server_atom.sh (ATOM + mooncake) +# ENGINE=tilert -> server_tilert.sh (vLLM prefill + TileRT decode) + check_env_vars ENGINE WS_PATH if [[ -f /config/hicache_mc.env ]]; then set -a diff --git a/benchmarks/multi_node/amd_utils/server_tilert.sh b/benchmarks/multi_node/amd_utils/server_tilert.sh index cab8764028..dba758c1c8 100644 --- a/benchmarks/multi_node/amd_utils/server_tilert.sh +++ b/benchmarks/multi_node/amd_utils/server_tilert.sh @@ -1,28 +1,28 @@ #!/bin/bash - -NODE0_ADDR="${NODE0_ADDR:-localhost}" -NODE_RANK="${NODE_RANK:-0}" -MODEL_DIR="${MODEL_DIR:-/models}" -MODEL_NAME="${MODEL_NAME:-}" -xP="${xP:-1}" -yD="${yD:-1}" -IPADDRS="${IPADDRS:-localhost}" -DRY_RUN="${DRY_RUN:-0}" -GPUS_PER_NODE="${GPUS_PER_NODE:-8}" -PREFILL_TP_SIZE="${PREFILL_TP_SIZE:-$GPUS_PER_NODE}" -DECODE_TP_SIZE="${DECODE_TP_SIZE:-$GPUS_PER_NODE}" - -BENCH_INPUT_LEN="${BENCH_INPUT_LEN:-1024}" -BENCH_OUTPUT_LEN="${BENCH_OUTPUT_LEN:-1024}" -BENCH_RANDOM_RANGE_RATIO="${BENCH_RANDOM_RANGE_RATIO:-1}" -BENCH_REQUEST_RATE="${BENCH_REQUEST_RATE:-inf}" -BENCH_NUM_PROMPTS_MULTIPLIER="${BENCH_NUM_PROMPTS_MULTIPLIER:-10}" -BENCH_MAX_CONCURRENCY="${BENCH_MAX_CONCURRENCY:-1}" - -MODEL_PATH="${MODEL_PATH:-${MODEL_DIR}/${MODEL_NAME}}" - -LOG_DIR="/run_logs/slurm_job-${SLURM_JOB_ID:-local}" -SHARED_LOG_DIR="${BENCHMARK_LOGS_DIR:-/run_logs}/logs/slurm_job-${SLURM_JOB_ID:-local}" +# TileRT disaggregated launcher: upstream vLLM ROCm prefill (TileRTConnector, +# kv_producer) + TileRT decode_server + the OpenAI-compatible pd_router. +# Every value below is supplied by the recipe through job.slurm; this script +# validates them and never invents a default for caller-owned configuration. + +source "$(dirname "${BASH_SOURCE[0]}")/../../benchmark_lib.sh" --validation-only + +check_env_vars \ + NODE0_ADDR NODE_RANK MODEL_DIR MODEL_NAME MODEL_PATH xP yD IPADDRS \ + DRY_RUN GPUS_PER_NODE PREFILL_TP_SIZE DECODE_TP_SIZE \ + BENCH_INPUT_LEN BENCH_OUTPUT_LEN BENCH_RANDOM_RANGE_RATIO \ + BENCH_REQUEST_RATE BENCH_NUM_PROMPTS_MULTIPLIER BENCH_MAX_CONCURRENCY \ + RUN_EVAL EVAL_ONLY EVAL_FRAMEWORK BENCHMARK_LOGS_DIR WS_PATH \ + SLURM_JOB_ID SPEC_DECODING \ + TILERT_PROFILE TILERT_MODEL_TYPE TILERT_MODEL_PKG TILERT_MAX_MODEL_LEN \ + TILERT_TRANSPORT TILERT_PARSER TILERT_QUEUE_TIMEOUT TILERT_WEIGHTS_DIR \ + TILERT_RDMA_STRICT TILERT_CONVERT_LOCK_WAIT TILERT_SIMULATE_ACC_METHOD \ + PREFILL_KV_DTYPE PREFILL_BLOCK_SIZE PREFILL_SPEC_TOKENS DECODE_KV_DTYPE \ + DECODE_MTP_SIZE GPU_MEM_UTIL SERVED_MODEL_NAME \ + DECODE_CTRL_PORT DECODE_HTTP_PORT PREFILL_PORT ROUTER_PORT \ + DECODE_WAIT PREFILL_WAIT ROUTER_WAIT SKIP_CONTAINER_BARRIER + +LOG_DIR="/run_logs/slurm_job-${SLURM_JOB_ID}" +SHARED_LOG_DIR="${BENCHMARK_LOGS_DIR}/logs/slurm_job-${SLURM_JOB_ID}" mkdir -p "$LOG_DIR" if [[ "$xP" -ne 1 || "$yD" -ne 1 ]]; then @@ -38,79 +38,37 @@ export TILERT_ROLE source "$WS_PATH/setup_deps.sh" source "$WS_PATH/env.sh" -PY="${PY:-python3}" - source /workspace/benchmarks/benchmark_lib.sh +# Model-specific engine environment (not caller configuration): the prefill +# vLLM env block lives with the model, exactly as models_atom.yaml carries the +# ATOM `env` string. Everything else is passed in by the recipe. MODELS_YAML="${WS_PATH}/models_tilert.yaml" -if [[ -f "$MODELS_YAML" ]] && "$PY" -c "import yaml" 2>/dev/null; then - eval "$("$PY" - "$MODELS_YAML" "$MODEL_NAME" <<'PYEOF' +eval "$("$PY" - "$MODELS_YAML" "$MODEL_NAME" <<'PYEOF' import shlex, sys, yaml path, name = sys.argv[1], sys.argv[2] with open(path) as f: models = yaml.safe_load(f) or {} -m = models.get(name) or {} -keys = { - "profile": "TILERT_PROFILE", - "model_type": "TILERT_MODEL_TYPE", - "max_model_len": "TILERT_MAX_MODEL_LEN", - "prefill_extra_flags": "TILERT_PREFILL_EXTRA_FLAGS", - "prefill_env": "TILERT_PREFILL_ENV", - "decode_extra_flags": "TILERT_DECODE_EXTRA_FLAGS", -} -for k, var in keys.items(): - if k not in m or m[k] is None: - continue - val = str(m[k]) - if val == "": - continue - print(f'if [[ -z "${{{var}:-}}" ]]; then {var}={shlex.quote(val)}; fi') +if name not in models: + sys.exit(f"model '{name}' is not present in {path}") +m = models[name] or {} +for key, var in (("prefill_env", "TILERT_PREFILL_ENV"), + ("prefill_extra_flags", "TILERT_PREFILL_EXTRA_FLAGS"), + ("decode_extra_flags", "TILERT_DECODE_EXTRA_FLAGS")): + print(f"{var}={shlex.quote(str(m.get(key) or ''))}") PYEOF -)" - echo "[tilert] loaded model defaults for '$MODEL_NAME' from $MODELS_YAML" -else - echo "[tilert] $MODELS_YAML not readable (missing file or PyYAML); using built-in defaults" -fi +)" || { echo "ERROR: cannot read the tilert model entry for '$MODEL_NAME' from $MODELS_YAML" >&2; exit 1; } +echo "[tilert] model entry '$MODEL_NAME' loaded from $MODELS_YAML" -TILERT_PROFILE="${TILERT_PROFILE:-glm5_2}" # decode_server --model (TileRT model profile) -TILERT_MODEL_TYPE="${TILERT_MODEL_TYPE:-glm-5}" # weight_converter --model_type -TILERT_MODEL_PKG="${TILERT_MODEL_PKG:-glm_5_2_rocm}" -TILERT_MAX_MODEL_LEN="${TILERT_MAX_MODEL_LEN:-${MAX_MODEL_LEN:-202752}}" -TILERT_TRANSPORT="${TILERT_TRANSPORT:-mooncake}" # decode --transport / connector tilert_transport -PREFILL_KV_DTYPE="${PREFILL_KV_DTYPE:-auto}" -PREFILL_BLOCK_SIZE="${PREFILL_BLOCK_SIZE:-64}" -DECODE_KV_DTYPE="${DECODE_KV_DTYPE:-bf16}" -TILERT_PARSER="${TILERT_PARSER:-none}" -if [[ "${IS_AGENTIC:-0}" == "1" || "${IS_AGENTIC:-}" == "true" || "${SCENARIO_TYPE:-}" == "agentic-coding" ]]; then - TILERT_QUEUE_TIMEOUT="${TILERT_QUEUE_TIMEOUT:-600}" -else - TILERT_QUEUE_TIMEOUT="${TILERT_QUEUE_TIMEOUT:-0}" -fi -GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.75}" -TILERT_PREFILL_EXTRA_FLAGS="${TILERT_PREFILL_EXTRA_FLAGS:-}" -TILERT_DECODE_EXTRA_FLAGS="${TILERT_DECODE_EXTRA_FLAGS:-}" -TILERT_PREFILL_ENV="${TILERT_PREFILL_ENV:-}" -TILERT_EXTRA_ENV="${TILERT_EXTRA_ENV:-}" - -DECODE_CTRL_PORT="${DECODE_CTRL_PORT:-5556}" -DECODE_HTTP_PORT="${DECODE_HTTP_PORT:-5557}" -PREFILL_PORT="${PREFILL_PORT:-8000}" -ROUTER_PORT="${ROUTER_PORT:-30000}" export ROUTER_PORT - -DECODE_WAIT="${DECODE_WAIT:-7200}" # prefill waits for decode ctrl port (weights conversion + load) -PREFILL_WAIT="${PREFILL_WAIT:-3600}" # prefill waits for its own vLLM port -ROUTER_WAIT="${ROUTER_WAIT:-10800}" # decode waits for the router port to open - -SERVED_MODEL_NAME="${SERVED_MODEL_NAME:-$TILERT_PROFILE}" export SERVED_MODEL_NAME -TILERT_WEIGHTS_DIR="${TILERT_WEIGHTS_DIR:-${MODEL_DIR}/${MODEL_NAME}-tilert-tp${DECODE_TP_SIZE}}" - PREFILL_SPEC=() DECODE_MTP=() -if [[ "${SPEC_DECODING:-}" == "mtp" ]]; then - PREFILL_SPEC=(--speculative-config '{"method":"mtp","num_speculative_tokens":1}') +if [[ "$SPEC_DECODING" == "mtp" ]]; then + # The prefill rank only has to build the MTP layer's KV; TileRT decode owns + # the draft depth (DECODE_MTP_SIZE), so the two counts differ by design. + PREFILL_SPEC=(--speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":${PREFILL_SPEC_TOKENS}}") DECODE_MTP=(--with-mtp) fi @@ -131,7 +89,7 @@ host_name=$(hostname) echo "[tilert] ROLE=$TILERT_ROLE rank=$NODE_RANK host=$host_name ($host_ip)" echo "[tilert] PREFILL_HOST=$PREFILL_HOST:$PREFILL_PORT DECODE_HOST=$DECODE_HOST:$DECODE_CTRL_PORT/$DECODE_HTTP_PORT ROUTER=:$ROUTER_PORT" -echo "[tilert] MODEL_PATH=$MODEL_PATH profile=$TILERT_PROFILE served=$SERVED_MODEL_NAME max_len=$TILERT_MAX_MODEL_LEN transport=$TILERT_TRANSPORT kv=${PREFILL_KV_DTYPE}->${DECODE_KV_DTYPE} mtp=${SPEC_DECODING:-none} agentic=$TILERT_IS_AGENTIC" +echo "[tilert] MODEL_PATH=$MODEL_PATH profile=$TILERT_PROFILE served=$SERVED_MODEL_NAME max_len=$TILERT_MAX_MODEL_LEN transport=$TILERT_TRANSPORT kv=${PREFILL_KV_DTYPE}->${DECODE_KV_DTYPE} mtp=${SPEC_DECODING} agentic=$TILERT_IS_AGENTIC" for env_pair in ${TILERT_EXTRA_ENV}; do export "${env_pair?}" @@ -174,7 +132,7 @@ rdma_preflight() { if command -v ibv_devices >/dev/null 2>&1; then echo "[rdma] ibv_devices:"; ibv_devices 2>&1 | sed 's/^/[rdma] /' fi - if (( warn )) && [[ "${TILERT_RDMA_STRICT:-0}" == "1" ]]; then + if (( warn )) && [[ "$TILERT_RDMA_STRICT" == "1" ]]; then echo "[rdma] TILERT_RDMA_STRICT=1 and preflight did not fully pass -- aborting" >&2 return 1 fi @@ -218,7 +176,7 @@ convert_weights() { fi mkdir -p "$TILERT_WEIGHTS_DIR" || { echo "[weight_converter] ERROR: cannot create $TILERT_WEIGHTS_DIR (set TILERT_WEIGHTS_DIR to a writable shared path)" >&2; return 1; } exec 9>"$TILERT_WEIGHTS_DIR/.convert.lock" - flock -w "${TILERT_CONVERT_LOCK_WAIT:-21600}" 9 || { + flock -w "$TILERT_CONVERT_LOCK_WAIT" 9 || { echo "[weight_converter] timed out waiting for the conversion lock (another job still converting?)" >&2; return 1; } if _tilert_weights_cached; then echo "[weight_converter] cache produced by a concurrent job, skipping conversion"; exec 9>&-; return 0 @@ -233,7 +191,7 @@ convert_weights() { conv_mod="tilert.models.${TILERT_MODEL_PKG}.weight_converter" conv_args=(--model_dir "$MODEL_PATH" --save_dir "$TILERT_WEIGHTS_DIR" --device "${TILERT_CONVERT_DEVICE:-cuda:$((GPUS_PER_NODE - 1))}") - [[ "${SPEC_DECODING:-}" == "mtp" ]] && conv_args+=(--num_mtp "${TILERT_NUM_MTP:-3}") + [[ "$SPEC_DECODING" == "mtp" ]] && conv_args+=(--num_mtp "$DECODE_MTP_SIZE") else conv_mod="tilert.models.preprocess.weight_converter" conv_args=(--model_type "$TILERT_MODEL_TYPE" --model_dir "$MODEL_PATH" --save_dir "$TILERT_WEIGHTS_DIR") @@ -257,10 +215,11 @@ convert_weights() { start_decode() { # shellcheck disable=SC2206 local extra=( ${TILERT_DECODE_EXTRA_FLAGS} ) - if [[ "$TILERT_IS_AGENTIC" == "1" && "${SPEC_DECODING:-}" == "mtp" \ - && "${EVAL_ONLY:-false}" != "true" && "${RUN_EVAL:-false}" != "true" ]]; then - local curve="${TILERT_GOLDEN_AL_FILE:-${WS_PATH%/benchmarks/*}/golden_al_distribution/${MODEL_PREFIX:-glm5.3}_mtp.yaml}" - TILERT_SIMULATE_ACC_LEN="$("$PY" - "$curve" "${TILERT_THINKING_MODE:-thinking_on}" "${TILERT_NUM_MTP:-3}" <<'PYEOF' + if [[ "$TILERT_IS_AGENTIC" == "1" && "$SPEC_DECODING" == "mtp" \ + && "$EVAL_ONLY" != "true" && "$RUN_EVAL" != "true" ]]; then + check_env_vars MODEL_PREFIX THINKING_MODE + local curve="${WS_PATH%/benchmarks/*}/golden_al_distribution/${MODEL_PREFIX}_mtp.yaml" + TILERT_SIMULATE_ACC_LEN="$("$PY" - "$curve" "$THINKING_MODE" "$DECODE_MTP_SIZE" <<'PYEOF' import sys, yaml path, thinking, tokens = sys.argv[1], sys.argv[2], int(sys.argv[3]) data = yaml.safe_load(open(path)) @@ -276,12 +235,12 @@ if not 1 <= value <= tokens + 1: print(f"{value:g}") PYEOF )" || { echo "[tilert] ERROR: golden AL lookup failed (curve=$curve)" >&2; exit 1; } - echo "[tilert] golden AL ${TILERT_SIMULATE_ACC_LEN} from $(basename "$curve") (thinking_on, K=${TILERT_NUM_MTP:-3})" + echo "[tilert] golden AL ${TILERT_SIMULATE_ACC_LEN} from $(basename "$curve") ($THINKING_MODE, K=${DECODE_MTP_SIZE})" fi - if [[ -n "${TILERT_SIMULATE_ACC_LEN:-}" && "${EVAL_ONLY:-false}" != "true" ]]; then + if [[ -n "${TILERT_SIMULATE_ACC_LEN:-}" && "$EVAL_ONLY" != "true" ]]; then export TILERT_SIMULATE_ACC_LEN - export TILERT_SIMULATE_ACC_METHOD="${TILERT_SIMULATE_ACC_METHOD:-match-expected}" + export TILERT_SIMULATE_ACC_METHOD echo "[decode] simulated acceptance: TILERT_SIMULATE_ACC_LEN=${TILERT_SIMULATE_ACC_LEN}" \ "method=${TILERT_SIMULATE_ACC_METHOD} (output text is meaningless by design)" else @@ -381,7 +340,7 @@ run_bench_and_eval() { local decode_gpus=$(( DECODE_TP_SIZE * yD )) export TRANSFORMERS_VERBOSITY=error TOKENIZERS_PARALLELISM=false - if [[ "${EVAL_ONLY:-false}" == "true" ]]; then + if [[ "$EVAL_ONLY" == "true" ]]; then echo "EVAL_ONLY mode: skipping throughput benchmark" else local conc np export_file @@ -405,7 +364,7 @@ run_bench_and_eval() { done fi - if [[ "${RUN_EVAL:-false}" == "true" ]]; then + if [[ "$RUN_EVAL" == "true" ]]; then run_lm_eval_on_router || rc=1 fi return $rc @@ -429,10 +388,10 @@ run_lm_eval_on_router() { else export EVAL_CONCURRENT_REQUESTS=$(echo "$BENCH_MAX_CONCURRENCY" | tr 'x' '\n' | sort -n | tail -1) fi - export MODEL="${MODEL:-$MODEL_PATH}" - export MAX_MODEL_LEN="${MAX_MODEL_LEN:-$TILERT_MAX_MODEL_LEN}" + export MODEL="$MODEL_PATH" + export MAX_MODEL_LEN="$TILERT_MAX_MODEL_LEN" if [[ "$DRY_RUN" -eq 1 ]]; then - echo "DRY RUN: run_eval --port $ROUTER_PORT (framework=${EVAL_FRAMEWORK:-lm-eval}, conc=${EVAL_CONCURRENT_REQUESTS})" + echo "DRY RUN: run_eval --port $ROUTER_PORT (framework=${EVAL_FRAMEWORK}, conc=${EVAL_CONCURRENT_REQUESTS})" else run_eval --port "$ROUTER_PORT" local eval_rc=$? @@ -467,13 +426,14 @@ run_agentic_replay() { export PORT="$ROUTER_PORT" export MODEL="$MODEL_PATH" # aiperf --tokenizer (local HF dir) export SERVED_MODEL_NAME # aiperf --model (name the router/vLLM serve) - export DURATION="${DURATION:-1800}" + check_env_vars DURATION RESULT_FILENAME export MAX_MODEL_LEN="$TILERT_MAX_MODEL_LEN" - export AIPERF_SERVER_METRICS_URLS="${AIPERF_SERVER_METRICS_URLS:-http://${PREFILL_HOST}:${PREFILL_PORT}/metrics}" + # TileRT decode exposes no /metrics route; only the vLLM prefill is scraped. + export AIPERF_SERVER_METRICS_URLS="http://${PREFILL_HOST}:${PREFILL_PORT}/metrics" export TRANSFORMERS_VERBOSITY=error TOKENIZERS_PARALLELISM=false local result_dir="$LOG_DIR/agentic" - local result_filename_base="${RESULT_FILENAME:-agentic_bench}" + local result_filename_base="$RESULT_FILENAME" mkdir -p "$result_dir" resolve_trace_source @@ -504,7 +464,7 @@ run_agentic_replay() { echo "Waiting at the container creation barrier on $host_name" if [[ "$DRY_RUN" -eq 1 ]]; then echo "DRY RUN: skipping container creation barrier" -elif [[ "${SKIP_CONTAINER_BARRIER:-0}" == "1" ]]; then +elif [[ "$SKIP_CONTAINER_BARRIER" == "1" ]]; then echo "SKIP_CONTAINER_BARRIER=1: caller asserts all containers are up" else "$PY" "$WS_PATH/sync.py" barrier \ @@ -550,7 +510,7 @@ case "$TILERT_ROLE" in echo "NODE INFO =======================================" echo "Node List : ${SLURM_JOB_NODELIST:-}" echo "Node IPs : ${IPADDRS}" - echo "Model : ${MODEL_NAME:-'Not specified'}" + echo "Model : ${MODEL_NAME}" echo "${host_name}:${host_ip} is the Prefill Node (vLLM + TileRTConnector) and Router Node" echo "================================================" rdma_preflight || exit 1 diff --git a/benchmarks/multi_node/amd_utils/setup_deps.sh b/benchmarks/multi_node/amd_utils/setup_deps.sh index 3917a64550..46917c0516 100644 --- a/benchmarks/multi_node/amd_utils/setup_deps.sh +++ b/benchmarks/multi_node/amd_utils/setup_deps.sh @@ -63,13 +63,12 @@ install_transformers_glm5() { _SETUP_INSTALLED+=("transformers-glm5") } -TILERT_VERSION="${TILERT_VERSION:-0.1.6}" -TILERT_PACKAGE="${TILERT_PACKAGE:-tilert}" -TILERT_PIP_SPEC="${TILERT_PIP_SPEC:-$TILERT_PACKAGE==$TILERT_VERSION}" -TILERT_PIP_INDEX_URL="${TILERT_PIP_INDEX_URL:-}" -TILERT_HTTP_DEPS="${TILERT_HTTP_DEPS:-fastapi uvicorn httpx}" -TILERT_TRANSPORT_DEPS="${TILERT_TRANSPORT_DEPS:-mooncake-transfer-engine-rocm>=0.3.13}" -TILERT_TRANSFORMERS_SPEC="${TILERT_TRANSFORMERS_SPEC:-transformers>=4.56}" +# Pinned by the recipe (TILERT_VERSION); the rest are fixed properties of the +# TileRT 0.1.x runtime rather than caller configuration. +TILERT_PACKAGE=tilert +TILERT_HTTP_DEPS="fastapi uvicorn httpx" +TILERT_TRANSPORT_DEPS="mooncake-transfer-engine-rocm>=0.3.13" +TILERT_TRANSFORMERS_SPEC="transformers>=4.56" _tilert_resolve_python() { if [[ -n "${PY:-}" ]] && command -v "$PY" >/dev/null 2>&1; then :; else @@ -94,9 +93,7 @@ PYEOF } _tilert_pip() { - local a=(-m pip install --quiet --no-cache-dir) - [[ -n "$TILERT_PIP_INDEX_URL" ]] && a+=(--index-url "$TILERT_PIP_INDEX_URL") - "$PY" "${a[@]}" "$@" + "$PY" -m pip install --quiet --no-cache-dir "$@" } _tilert_install_missing() { @@ -186,6 +183,8 @@ if [[ "$ENGINE" == "vllm-disagg" ]]; then export PATH="${UCX_HOME}/bin:/usr/local/bin/etcd:/root/.cargo/bin:${PATH}" export LD_LIBRARY_PATH="${UCX_HOME}/lib:${RIXL_HOME}/lib:${RIXL_HOME}/lib/x86_64-linux-gnu:${LD_LIBRARY_PATH:-}" elif [[ "$ENGINE" == "tilert" ]]; then + check_env_vars TILERT_VERSION + TILERT_PIP_SPEC="$TILERT_PACKAGE==$TILERT_VERSION" _tilert_resolve_python case "${TILERT_ROLE:-}" in decode) install_tilert_decode ;; diff --git a/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh b/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh index 3b750b79a0..cf550336e7 100644 --- a/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh +++ b/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh @@ -9,6 +9,7 @@ check_env_vars \ IMAGE \ SPEC_DECODING \ MODEL_PATH \ + MODEL_NAME \ PREFILL_NUM_WORKERS \ PREFILL_TP \ PREFILL_EP \ @@ -31,7 +32,7 @@ set -x cd "$GITHUB_WORKSPACE/benchmarks/multi_node/amd_utils" || exit 1 -export TIME_LIMIT="${TIME_LIMIT:-08:00:00}" +export TIME_LIMIT=08:00:00 export MODEL_PATH=$MODEL_PATH export MODEL_NAME=$MODEL_NAME export CONTAINER_IMAGE=$IMAGE @@ -44,19 +45,48 @@ if [[ "$PREFILL_NODES" -ne 1 || "$DECODE_NODES" -ne 1 || \ exit 1 fi -export TILERT_VERSION="${TILERT_VERSION:-0.1.6}" -export TILERT_PROFILE="${TILERT_PROFILE:-glm5_2}" -export TILERT_MODEL_TYPE="${TILERT_MODEL_TYPE:-glm-5}" -export TILERT_MAX_MODEL_LEN="${TILERT_MAX_MODEL_LEN:-${MAX_MODEL_LEN:-202752}}" -export DECODE_MTP_SIZE="${DECODE_MTP_SIZE:-$([[ "$SPEC_DECODING" == "mtp" ]] && echo 1 || echo 0)}" -export TILERT_TRANSPORT="${TILERT_TRANSPORT:-mooncake}" -export PREFILL_KV_DTYPE="${PREFILL_KV_DTYPE:-auto}" -export DECODE_KV_DTYPE="${DECODE_KV_DTYPE:-bf16}" -export TILERT_PARSER="${TILERT_PARSER:-none}" -export ROUTER_PORT="${ROUTER_PORT:-30000}" -export TILERT_QUEUE_TIMEOUT="${TILERT_QUEUE_TIMEOUT:-0}" +# TileRT configuration. Every value is explicit here: server_tilert.sh +# validates each one with check_env_vars and supplies no defaults of its own. +export TILERT_VERSION=0.1.6 +export TILERT_PROFILE=glm5_2 # decode_server --model (TileRT model profile) +export TILERT_MODEL_TYPE=glm-5 # weight_converter --model_type (fallback converter) +export TILERT_MODEL_PKG=glm_5_2_rocm # per-model converter package, preferred when importable +export SERVED_MODEL_NAME=glm5_2 +export TILERT_MAX_MODEL_LEN=202752 +export TILERT_TRANSPORT=mooncake +export TILERT_PARSER=none +export TILERT_RDMA_STRICT=0 +export TILERT_CONVERT_LOCK_WAIT=21600 +export TILERT_SIMULATE_ACC_METHOD=match-expected +export TILERT_WEIGHTS_DIR="/models/${MODEL_NAME}-tilert-tp${DECODE_TP}" +export PREFILL_KV_DTYPE=auto +# The ROCm backend supports block sizes [1, 64] and vLLM picks 1, which makes +# the connector's KI plane copy fail and MLA address the wrong rows. +export PREFILL_BLOCK_SIZE=64 +export DECODE_KV_DTYPE=bf16 # the ROCm sparse-MLA backend has no fp8_ds_mla +export GPU_MEM_UTIL=0.75 +export SKIP_CONTAINER_BARRIER=0 +export ROUTER_PORT=30000 +export PREFILL_PORT=8000 +export DECODE_CTRL_PORT=5556 +export DECODE_HTTP_PORT=5557 +export DECODE_WAIT=7200 # prefill waits for the decode ctrl port +export PREFILL_WAIT=3600 # prefill waits for its own vLLM port +export ROUTER_WAIT=10800 # decode waits for the router port to open -if [[ "${PREFILL_EP:-1}" -ne 1 || "${DECODE_EP:-1}" -ne 1 || \ +if [[ "$SPEC_DECODING" == "mtp" ]]; then + # TileRT decode drafts at depth 3 (the only depth the ROCm GLM profile + # builds) and the golden acceptance curve is keyed on it. The vLLM prefill + # rank only has to materialise the MTP layer's KV, so it runs at 1. + export DECODE_MTP_SIZE=3 + export PREFILL_SPEC_TOKENS=1 +else + export DECODE_MTP_SIZE=0 + export PREFILL_SPEC_TOKENS=0 +fi +export TILERT_QUEUE_TIMEOUT=0 + +if [[ "$PREFILL_EP" -ne 1 || "$DECODE_EP" -ne 1 || \ "$PREFILL_DP_ATTN" == "true" || "$DECODE_DP_ATTN" == "true" ]]; then echo "Error: tilert runs pure TP8 on both roles; ep must be 1 and dp-attn false" >&2 exit 1 diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 2123dfd1d2..830e2a7fa1 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8461,9 +8461,9 @@ description: - "Add GLM-5.3 FP8 MI355X prefill/decode-disaggregated Single-turn 8k1k benchmark via TileRT: upstream vLLM ROCm prefill (TileRTConnector, kv_producer) + TileRT decode_server + OpenAI-compatible pd_router, framework=tilert, mirroring the B200 glm5.1-fp8-b200-tilert entry." - "Two images per job, both first-party and neither carrying the TileRT wheel: the decode rank runs ghcr.io/tile-ai/tilert-rocm-decode:0.1.6 (ROCm PyTorch runtime plus the mooncake ROCm transfer engine built from source) and the prefill rank runs ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6 (the upstream vllm/vllm-openai-rocm release image plus that same transfer engine, with vLLM itself unpatched and no file inside it modified), selected per rank via PREFILL_IMAGE in prefill.additional-settings. setup_deps.sh pip-installs the official tilert==0.1.6 wheel and the router HTTP deps into both ranks at container start (full install on decode, --no-deps connector plugin on prefill), so neither image is pinned to a wheel version; this mirrors the B200 glm5.1-fp8-b200-tilert pattern of a first-party TileRT decode image. The transfer engine is built from source because the published mooncake ROCm wheels are compiled without ENABLE_MULTI_PROTOCOL, which leaves the cross-host locality gate (kvcache-ai/Mooncake#2753) out of the binary so every cross-node KV transfer is treated as node-local and fails in hipIpcOpenMemHandle." - - "KV moves P->D over mooncake (kv-p2p-transfer: mooncake); both roles use a bf16 KV cache (prefill --kv-cache-dtype auto, decode bf16) because the ROCm vLLM sparse-MLA backend has no fp8_ds_mla dtype. max-model-len 202752 on both roles; MTP wired via spec-decoding=mtp (vLLM speculative-config mtp with 1 draft token on prefill, --with-mtp on decode). TileRT decode is bs=1 only, so conc-list is [1]; topology 1 prefill node (TP8) + 1 decode node (TP8)." + - "KV moves P->D over mooncake (kv-p2p-transfer: mooncake); both roles use a bf16 KV cache (prefill --kv-cache-dtype auto, decode bf16) because the ROCm vLLM sparse-MLA backend has no fp8_ds_mla dtype. max-model-len 202752 on both roles; MTP wired via spec-decoding=mtp: TileRT decode drafts at depth 3 (DECODE_MTP_SIZE, the only depth the ROCm GLM profile builds, and the key the golden acceptance curve is read at), while the vLLM prefill rank only materialises the MTP layer's KV and runs speculative-config mtp with 1 draft token. TileRT decode is bs=1 only, so conc-list is [1]; topology 1 prefill node (TP8) + 1 decode node (TP8)." - "Orchestration: runners/launch_mi355x-amds.sh dispatches framework=tilert through the existing amd_utils chain (submit.sh -> job.slurm -> server.sh -> new server_tilert.sh); job.slurm selects the image per rank and a tilert env block, setup_deps.sh gains a per-role tilert install branch, models_tilert.yaml holds the GLM-5.3 profile defaults. TileRT decode weights are converted once from the HF checkpoint under a flock and cached next to it on the shared model volume. Cross-node teardown uses the amd_utils router-port-close barrier." - pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3328 + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3330 - config-keys: - glm5.3-fp8-mi355x-tilert-agentic @@ -8473,4 +8473,4 @@ - "Add GLM-5.3 FP8 MI355X AgentX (agentic-coding) via TileRT prefill/decode disaggregation with MTP, 1 prefill node (TP8) + 1 decode node (TP8), concurrency [1], mirroring the B200 glm5.1-fp8-b200-tilert-agentic entry on the amd_utils orchestration." - "Same two-image recipe as glm5.3-fp8-mi355x-tilert (decode ghcr.io/tile-ai/tilert-rocm-decode:0.1.6, prefill ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6, tilert==0.1.6 pip-installed into both at container start), bf16 KV cache on both roles, kv-p2p-transfer: mooncake, kv-offloading none." - "The pd_router runs with --parser none and a 1800-second queue timeout so requests wait on the bs=1 decode engine instead of failing fast; AgentX traces are filtered at the 202752-token context both roles are launched with; AIPerf scrapes only the vLLM prefill Prometheus endpoint (TileRT decode exposes no /metrics route)." - pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3328 + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3330 diff --git a/runners/launch_mi355x-amds.sh b/runners/launch_mi355x-amds.sh index 6f13b3f2d7..e715e96bd4 100644 --- a/runners/launch_mi355x-amds.sh +++ b/runners/launch_mi355x-amds.sh @@ -76,6 +76,7 @@ if [[ "$IS_MULTINODE" == "true" ]]; then SCRIPT_NAME="${EXP_NAME%%_*}_${PRECISION}_mi355x_${FRAMEWORK}.sh" if [[ "$FRAMEWORK" == "sglang-disagg" ]] || [[ "$FRAMEWORK" == "vllm-disagg" ]] || [[ "$FRAMEWORK" == "atom-disagg" ]] || [[ "$FRAMEWORK" == "tilert" ]]; then # Agentic recipes under multi_node/agentic/ export the HiCache tunables; + # fixed-seq-len recipes live at the multi_node/ root. if [[ "${SCENARIO_SUBDIR}" == "agentic/" ]]; then BENCHMARK_SUBDIR="multi_node/agentic" else @@ -115,6 +116,9 @@ if [[ "$IS_MULTINODE" == "true" ]]; then set -x + + + if [[ "${EVAL_ONLY}" != "true" && "${IS_AGENTIC}" != "1" ]]; then cat > collect_latest_results.py <<'PY' import os, sys From df5e13bbcd76529c749db097107f056e31808463 Mon Sep 17 00:00:00 2001 From: CrimsonDump Date: Mon, 21 Sep 2026 19:44:38 +0800 Subject: [PATCH 03/25] =?UTF-8?q?fix(tilert):=20serve=20GLM-5.3=20at=20its?= =?UTF-8?q?=20full=201M=20context=20/=20=E5=B0=86=20GLM-5.3=20=E4=B8=8A?= =?UTF-8?q?=E4=B8=8B=E6=96=87=E9=95=BF=E5=BA=A6=E6=94=B9=E4=B8=BA=E5=AE=8C?= =?UTF-8?q?=E6=95=B4=201M?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit max-model-len was 202752 on both roles. That is GLM-5.1's context window, inherited from the B200 glm5.1-fp8-b200-tilert recipe this one mirrors (perf-changelog entry for that recipe records "max-seq-len 202752"), not GLM-5.3's. GLM-5.3 shares GLM-5.2's base, and every in-tree GLM-5.2 recipe -- TRT-LLM and SGLang alike -- runs at 1048576. Both roles now launch at 1048576, and the AgentX trace filter follows it through MAX_MODEL_LEN. 原配方两侧均为 202752，这是 GLM-5.1 的上下文窗口，沿用自本配方对标的 B200 glm5.1-fp8-b200-tilert，并非 GLM-5.3 的。GLM-5.3 与 GLM-5.2 同底座，而树内所有 GLM-5.2 配方（TRT-LLM 与 SGLang）均使用 1048576。现两侧均以 1048576 启动， AgentX 轨迹过滤经 MAX_MODEL_LEN 随之生效。 --- benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh | 4 +++- benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh | 4 +++- perf-changelog.yaml | 4 ++-- 3 files changed, 8 insertions(+), 4 deletions(-) diff --git a/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh b/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh index f87c6849b0..824df13538 100644 --- a/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh +++ b/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh @@ -66,7 +66,9 @@ export TILERT_PROFILE=glm5_2 # decode_server --model (TileRT model prof export TILERT_MODEL_TYPE=glm-5 # weight_converter --model_type (fallback converter) export TILERT_MODEL_PKG=glm_5_2_rocm # per-model converter package, preferred when importable export SERVED_MODEL_NAME=glm5_2 -export TILERT_MAX_MODEL_LEN=202752 +# GLM-5.3's full context window, as every in-tree GLM-5.2 recipe uses. +# (202752 is GLM-5.1's, inherited from the B200 TileRT recipe this mirrors.) +export TILERT_MAX_MODEL_LEN=1048576 export TILERT_TRANSPORT=mooncake export TILERT_PARSER=none export TILERT_RDMA_STRICT=0 diff --git a/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh b/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh index cf550336e7..bcdbe56d97 100644 --- a/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh +++ b/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh @@ -52,7 +52,9 @@ export TILERT_PROFILE=glm5_2 # decode_server --model (TileRT model prof export TILERT_MODEL_TYPE=glm-5 # weight_converter --model_type (fallback converter) export TILERT_MODEL_PKG=glm_5_2_rocm # per-model converter package, preferred when importable export SERVED_MODEL_NAME=glm5_2 -export TILERT_MAX_MODEL_LEN=202752 +# GLM-5.3's full context window, as every in-tree GLM-5.2 recipe uses. +# (202752 is GLM-5.1's, inherited from the B200 TileRT recipe this mirrors.) +export TILERT_MAX_MODEL_LEN=1048576 export TILERT_TRANSPORT=mooncake export TILERT_PARSER=none export TILERT_RDMA_STRICT=0 diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 830e2a7fa1..7afb667d7a 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8461,7 +8461,7 @@ description: - "Add GLM-5.3 FP8 MI355X prefill/decode-disaggregated Single-turn 8k1k benchmark via TileRT: upstream vLLM ROCm prefill (TileRTConnector, kv_producer) + TileRT decode_server + OpenAI-compatible pd_router, framework=tilert, mirroring the B200 glm5.1-fp8-b200-tilert entry." - "Two images per job, both first-party and neither carrying the TileRT wheel: the decode rank runs ghcr.io/tile-ai/tilert-rocm-decode:0.1.6 (ROCm PyTorch runtime plus the mooncake ROCm transfer engine built from source) and the prefill rank runs ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6 (the upstream vllm/vllm-openai-rocm release image plus that same transfer engine, with vLLM itself unpatched and no file inside it modified), selected per rank via PREFILL_IMAGE in prefill.additional-settings. setup_deps.sh pip-installs the official tilert==0.1.6 wheel and the router HTTP deps into both ranks at container start (full install on decode, --no-deps connector plugin on prefill), so neither image is pinned to a wheel version; this mirrors the B200 glm5.1-fp8-b200-tilert pattern of a first-party TileRT decode image. The transfer engine is built from source because the published mooncake ROCm wheels are compiled without ENABLE_MULTI_PROTOCOL, which leaves the cross-host locality gate (kvcache-ai/Mooncake#2753) out of the binary so every cross-node KV transfer is treated as node-local and fails in hipIpcOpenMemHandle." - - "KV moves P->D over mooncake (kv-p2p-transfer: mooncake); both roles use a bf16 KV cache (prefill --kv-cache-dtype auto, decode bf16) because the ROCm vLLM sparse-MLA backend has no fp8_ds_mla dtype. max-model-len 202752 on both roles; MTP wired via spec-decoding=mtp: TileRT decode drafts at depth 3 (DECODE_MTP_SIZE, the only depth the ROCm GLM profile builds, and the key the golden acceptance curve is read at), while the vLLM prefill rank only materialises the MTP layer's KV and runs speculative-config mtp with 1 draft token. TileRT decode is bs=1 only, so conc-list is [1]; topology 1 prefill node (TP8) + 1 decode node (TP8)." + - "KV moves P->D over mooncake (kv-p2p-transfer: mooncake); both roles use a bf16 KV cache (prefill --kv-cache-dtype auto, decode bf16) because the ROCm vLLM sparse-MLA backend has no fp8_ds_mla dtype. max-model-len 1048576 on both roles (GLM-5.3's full context window); MTP wired via spec-decoding=mtp: TileRT decode drafts at depth 3 (DECODE_MTP_SIZE, the only depth the ROCm GLM profile builds, and the key the golden acceptance curve is read at), while the vLLM prefill rank only materialises the MTP layer's KV and runs speculative-config mtp with 1 draft token. TileRT decode is bs=1 only, so conc-list is [1]; topology 1 prefill node (TP8) + 1 decode node (TP8)." - "Orchestration: runners/launch_mi355x-amds.sh dispatches framework=tilert through the existing amd_utils chain (submit.sh -> job.slurm -> server.sh -> new server_tilert.sh); job.slurm selects the image per rank and a tilert env block, setup_deps.sh gains a per-role tilert install branch, models_tilert.yaml holds the GLM-5.3 profile defaults. TileRT decode weights are converted once from the HF checkpoint under a flock and cached next to it on the shared model volume. Cross-node teardown uses the amd_utils router-port-close barrier." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3330 @@ -8472,5 +8472,5 @@ description: - "Add GLM-5.3 FP8 MI355X AgentX (agentic-coding) via TileRT prefill/decode disaggregation with MTP, 1 prefill node (TP8) + 1 decode node (TP8), concurrency [1], mirroring the B200 glm5.1-fp8-b200-tilert-agentic entry on the amd_utils orchestration." - "Same two-image recipe as glm5.3-fp8-mi355x-tilert (decode ghcr.io/tile-ai/tilert-rocm-decode:0.1.6, prefill ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6, tilert==0.1.6 pip-installed into both at container start), bf16 KV cache on both roles, kv-p2p-transfer: mooncake, kv-offloading none." - - "The pd_router runs with --parser none and a 1800-second queue timeout so requests wait on the bs=1 decode engine instead of failing fast; AgentX traces are filtered at the 202752-token context both roles are launched with; AIPerf scrapes only the vLLM prefill Prometheus endpoint (TileRT decode exposes no /metrics route)." + - "The pd_router runs with --parser none and a 1800-second queue timeout so requests wait on the bs=1 decode engine instead of failing fast; AgentX traces are filtered at the 1048576-token full GLM context both roles are launched with; AIPerf scrapes only the vLLM prefill Prometheus endpoint (TileRT decode exposes no /metrics route)." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3330 From ad1c997fc89d9859451a311a7173225e0d75f6d2 Mon Sep 17 00:00:00 2001 From: CrimsonDump Date: Mon, 21 Sep 2026 20:56:43 +0800 Subject: [PATCH 04/25] =?UTF-8?q?fix(tilert):=20supply=20the=20amd=5Futils?= =?UTF-8?q?=20orchestration=20inputs=20on=20the=20MI355X=20pool=20/=20?= =?UTF-8?q?=E8=A1=A5=E9=BD=90=20MI355X=20=E4=B8=8A=20TileRT=20=E7=9A=84=20?= =?UTF-8?q?amd=5Futils=20=E7=BC=96=E6=8E=92=E8=BE=93=E5=85=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The tilert arm of benchmarks/multi_node/runtime_settings.sh is the B200 srt-slurm profile. The MI355X recipes run through the shared amd_utils chain instead, whose submit.sh validates ROUTER_TYPE, PROXY_PING_PORT, HEADNODE_PORT and SERVER_PORT, and whose job.slurm and setup_deps.sh also require VLLM_ROUTER_IMAGE, SKIP_RDMA_CHECK, SKIP_GPU_SANITY, ENABLE_METRICS, PREFILL_ROUTER_POLICY, DECODE_ROUTER_POLICY, FLUSH_DRAIN_TIMEOUT, CLEAR_CACHE_BETWEEN_CONC, ROCM_PATH, UCX_HOME and RIXL_HOME. None of those were exported for FRAMEWORK=tilert, so submit.sh exited before sbatch and the recipe returned no job id. The AMD block is scoped by RUNNER_TYPE so the B200 TileRT lane is untouched. The fixed-seq-len recipe now sets KV_OFFLOADING=none itself: submit.sh and job.slurm validate it, and the matrix emits it only for agentic scenarios. launch_mi355x-amds.sh now fails when a recipe returns no job id. Its wait loop polled for slurm_job-.out while its liveness guard degenerated to `grep -q ""`, which matches any queued job of that user, so a failed submit span the loop until the job hit its time limit rather than reporting the error. benchmarks/multi_node/runtime_settings.sh 的 tilert 分支是 B200 的 srt-slurm 配置，而 MI355X 走的是共用 amd_utils 链：submit.sh 校验 ROUTER_TYPE、 PROXY_PING_PORT、HEADNODE_PORT、SERVER_PORT，job.slurm 与 setup_deps.sh 另需 VLLM_ROUTER_IMAGE、SKIP_RDMA_CHECK、SKIP_GPU_SANITY、ENABLE_METRICS、 PREFILL_ROUTER_POLICY、DECODE_ROUTER_POLICY、FLUSH_DRAIN_TIMEOUT、 CLEAR_CACHE_BETWEEN_CONC、ROCM_PATH、UCX_HOME、RIXL_HOME。FRAMEWORK=tilert 下这些一概未导出，导致 submit.sh 在 sbatch 之前退出、配方返回空 job id。AMD 专用 块以 RUNNER_TYPE 限定，不影响 B200。 定长配方自行设置 KV_OFFLOADING=none（矩阵仅在 agentic 场景下产出该字段）。 launch_mi355x-amds.sh 现在在配方未返回 job id 时直接失败：原等待循环会一直轮询 slurm_job-.out，而其存活判断退化为 `grep -q ""`，会匹配该用户任意排队作业，于是 提交失败不会报错，而是空转到作业超时。 --- benchmarks/multi_node/amd_utils/neg.err | 93 +++++++++++++++++++ .../multi_node/glm5.3_fp8_mi355x_tilert.sh | 3 + benchmarks/multi_node/runtime_settings.sh | 22 ++++- runners/launch_mi355x-amds.sh | 10 ++ 4 files changed, 127 insertions(+), 1 deletion(-) create mode 100644 benchmarks/multi_node/amd_utils/neg.err diff --git a/benchmarks/multi_node/amd_utils/neg.err b/benchmarks/multi_node/amd_utils/neg.err new file mode 100644 index 0000000000..8163069f7d --- /dev/null +++ b/benchmarks/multi_node/amd_utils/neg.err @@ -0,0 +1,93 @@ ++ cd /Users/bshan/InferenceX/benchmarks/multi_node/amd_utils ++ export TIME_LIMIT=08:00:00 ++ TIME_LIMIT=08:00:00 ++ export MODEL_PATH=/tmp/claude-502/-Users-bshan-InferenceX/a3355d0f-16c4-46ef-b540-6e20f6eba79c/scratchpad/d3/models ++ MODEL_PATH=/tmp/claude-502/-Users-bshan-InferenceX/a3355d0f-16c4-46ef-b540-6e20f6eba79c/scratchpad/d3/models ++ export MODEL_NAME=GLM-5.3 ++ MODEL_NAME=GLM-5.3 ++ export CONTAINER_IMAGE=img ++ CONTAINER_IMAGE=img ++ export PREFILL_IMAGE ++ [[ 1 -ne 1 ]] ++ [[ 1 -ne 1 ]] ++ [[ 1 -ne 1 ]] ++ [[ 1 -ne 1 ]] ++ export TILERT_VERSION=0.1.6 ++ TILERT_VERSION=0.1.6 ++ export TILERT_PROFILE=glm5_2 ++ TILERT_PROFILE=glm5_2 ++ export TILERT_MODEL_TYPE=glm-5 ++ TILERT_MODEL_TYPE=glm-5 ++ export TILERT_MODEL_PKG=glm_5_2_rocm ++ TILERT_MODEL_PKG=glm_5_2_rocm ++ export SERVED_MODEL_NAME=glm5_2 ++ SERVED_MODEL_NAME=glm5_2 ++ export TILERT_MAX_MODEL_LEN=1048576 ++ TILERT_MAX_MODEL_LEN=1048576 ++ export TILERT_TRANSPORT=mooncake ++ TILERT_TRANSPORT=mooncake ++ export TILERT_PARSER=none ++ TILERT_PARSER=none ++ export TILERT_RDMA_STRICT=0 ++ TILERT_RDMA_STRICT=0 ++ export TILERT_CONVERT_LOCK_WAIT=21600 ++ TILERT_CONVERT_LOCK_WAIT=21600 ++ export TILERT_SIMULATE_ACC_METHOD=match-expected ++ TILERT_SIMULATE_ACC_METHOD=match-expected ++ export TILERT_WEIGHTS_DIR=/models/GLM-5.3-tilert-tp8 ++ TILERT_WEIGHTS_DIR=/models/GLM-5.3-tilert-tp8 ++ export PREFILL_KV_DTYPE=auto ++ PREFILL_KV_DTYPE=auto ++ export PREFILL_BLOCK_SIZE=64 ++ PREFILL_BLOCK_SIZE=64 ++ export DECODE_KV_DTYPE=bf16 ++ DECODE_KV_DTYPE=bf16 ++ export GPU_MEM_UTIL=0.75 ++ GPU_MEM_UTIL=0.75 ++ export SKIP_CONTAINER_BARRIER=0 ++ SKIP_CONTAINER_BARRIER=0 ++ export ROUTER_PORT=30000 ++ ROUTER_PORT=30000 ++ export PREFILL_PORT=8000 ++ PREFILL_PORT=8000 ++ export DECODE_CTRL_PORT=5556 ++ DECODE_CTRL_PORT=5556 ++ export DECODE_HTTP_PORT=5557 ++ DECODE_HTTP_PORT=5557 ++ export DECODE_WAIT=7200 ++ DECODE_WAIT=7200 ++ export PREFILL_WAIT=3600 ++ PREFILL_WAIT=3600 ++ export ROUTER_WAIT=10800 ++ ROUTER_WAIT=10800 ++ [[ mtp == \m\t\p ]] ++ export DECODE_MTP_SIZE=3 ++ DECODE_MTP_SIZE=3 ++ export PREFILL_SPEC_TOKENS=1 ++ PREFILL_SPEC_TOKENS=1 ++ export TILERT_QUEUE_TIMEOUT=0 ++ TILERT_QUEUE_TIMEOUT=0 ++ export KV_OFFLOADING=none ++ KV_OFFLOADING=none ++ [[ 1 -ne 1 ]] ++ [[ 1 -ne 1 ]] ++ [[ false == \t\r\u\e ]] ++ [[ false == \t\r\u\e ]] ++ export PREFILL_ENABLE_EP=false ++ PREFILL_ENABLE_EP=false ++ export PREFILL_ENABLE_DP=false ++ PREFILL_ENABLE_DP=false ++ export DECODE_ENABLE_EP=false ++ DECODE_ENABLE_EP=false ++ export DECODE_ENABLE_DP=false ++ DECODE_ENABLE_DP=false +++ bash ./submit.sh 1 1 1 1 8192 1024 1 inf false false false false 8 8 1 ++ JOB_ID='Error: The following required environment variables are not set: + - ROUTER_TYPE + - PROXY_PING_PORT + - HEADNODE_PORT + - SERVER_PORT' ++ [[ 1 -ne 0 ]] ++ echo 'Failed to submit job' +Failed to submit job ++ exit 1 diff --git a/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh b/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh index bcdbe56d97..08e4261187 100644 --- a/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh +++ b/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh @@ -87,6 +87,9 @@ else export PREFILL_SPEC_TOKENS=0 fi export TILERT_QUEUE_TIMEOUT=0 +# submit.sh and job.slurm validate KV_OFFLOADING, and the fixed-seq-len matrix +# emits it only for agentic scenarios. TileRT has no KV offload backend. +export KV_OFFLOADING=none if [[ "$PREFILL_EP" -ne 1 || "$DECODE_EP" -ne 1 || \ "$PREFILL_DP_ATTN" == "true" || "$DECODE_DP_ATTN" == "true" ]]; then diff --git a/benchmarks/multi_node/runtime_settings.sh b/benchmarks/multi_node/runtime_settings.sh index 5bf43deb6e..e50c3b044e 100644 --- a/benchmarks/multi_node/runtime_settings.sh +++ b/benchmarks/multi_node/runtime_settings.sh @@ -48,7 +48,9 @@ case "$FRAMEWORK" in fi ;; tilert) - check_env_vars GITHUB_WORKSPACE + # RUNNER_TYPE selects the AMD block below, so a missing value must fail + # here rather than silently skip it. + check_env_vars GITHUB_WORKSPACE RUNNER_TYPE export BENCHMARK_LOGS_DIR="$GITHUB_WORKSPACE" RESULT_DIR=/workspace export GPU_MEM_UTIL=0.75 DECODE_CTRL_PORT=5556 DECODE_HTTP_PORT=5557 PREFILL_PORT=8000 export DECODE_WAIT=3600 PREFILL_WAIT=3600 TILERT_QUEUE_TIMEOUT=0 @@ -58,6 +60,24 @@ case "$FRAMEWORK" in if [[ "$IS_AGENTIC" == 1 || "$IS_AGENTIC" == true ]]; then export TILERT_QUEUE_TIMEOUT=1800 fi + # The MI355X TileRT recipes run through the shared amd_utils chain + # (submit.sh -> job.slurm -> server.sh -> setup_deps.sh), which validates + # the same orchestration inputs the AMD SGLang/vLLM/ATOM arms receive. + # Without them submit.sh exits before sbatch and the launcher never gets + # a job id. The B200 TileRT lane goes through srt-slurm and reads none of + # these, so they are scoped to the AMD pool. + if [[ "$RUNNER_TYPE" == *mi355x-amds* ]]; then + # Validated by job.slurm; only the vllm-disagg router branch reads + # VLLM_ROUTER_IMAGE, which ENGINE=tilert never enters. + export VLLM_ROUTER_IMAGE=vllm/vllm-router:nightly-20260716-1fbcde7 + export SKIP_RDMA_CHECK=0 SKIP_GPU_SANITY=0 + export ROUTER_TYPE=tilert-pd-router ROUTER_PORT=30000 PROXY_PING_PORT=36367 + export HEADNODE_PORT=20000 SERVER_PORT=2584 PROXY_STREAM_IDLE_TIMEOUT=300 + export ENABLE_METRICS=0 PREFILL_ROUTER_POLICY=random DECODE_ROUTER_POLICY=random + export FLUSH_DRAIN_TIMEOUT=120 CLEAR_CACHE_BETWEEN_CONC=1 + export DECODE_MTP_SIZE=0 + export ROCM_PATH=/opt/rocm UCX_HOME=/usr/local/ucx RIXL_HOME=/usr/local/rixl + fi ;; llmd-vllm) export LLMD_CONTAINER_ENGINE=docker VLLM_RANDOMIZE_DP_DUMMY_INPUTS=1 diff --git a/runners/launch_mi355x-amds.sh b/runners/launch_mi355x-amds.sh index e715e96bd4..b32d54e0fa 100644 --- a/runners/launch_mi355x-amds.sh +++ b/runners/launch_mi355x-amds.sh @@ -87,6 +87,16 @@ if [[ "$IS_MULTINODE" == "true" ]]; then fi JOB_ID=$(bash "benchmarks/${BENCHMARK_SUBDIR}/${SCRIPT_NAME}") + # An empty JOB_ID means the recipe or submit.sh failed before sbatch. The + # wait loop below would then poll for slurm_job-.out forever, because its + # liveness guard degenerates to `grep -q ""` and matches any job this user + # has queued. Fail here instead of burning the job's whole time limit. + if [[ -z "${JOB_ID//[[:space:]]/}" ]]; then + echo "ERROR: benchmarks/${BENCHMARK_SUBDIR}/${SCRIPT_NAME} returned no Slurm job id;" \ + "the recipe or submit.sh failed before sbatch (see its stderr above)" >&2 + exit 1 + fi + LOG_FILE="$BENCHMARK_LOGS_DIR/slurm_job-${JOB_ID}.out" sleep 10 From 54cef6816535ad95230aa2e1bc20e53bb9f8328f Mon Sep 17 00:00:00 2001 From: Oseltamivir <58582368+Oseltamivir@users.noreply.github.com> Date: Mon, 21 Sep 2026 13:36:44 +0000 Subject: [PATCH 05/25] fix(tilert): enable RDMA fork safety on both ranks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Port the launcher fix from CrimsonDump/InferenceX commit 9e745f3697b360992e37747b53fac5e928cfa709. Set RDMAV_FORK_SAFE=1 before either role initializes verbs to avoid silent TCP fallback and degraded TTFT; TPOT is unaffected. Append performance changelog entries for 8k1k and AgentX. 在 prefill 和 decode 两侧初始化 verbs 前设置 RDMAV_FORK_SAFE=1，避免静默回退到 TCP 导致 TTFT 恶化；TPOT 不受影响。补充代码注释及 8k1k、AgentX 性能变更记录。 --- benchmarks/multi_node/amd_utils/server_tilert.sh | 6 ++++++ perf-changelog.yaml | 16 ++++++++++++++++ 2 files changed, 22 insertions(+) diff --git a/benchmarks/multi_node/amd_utils/server_tilert.sh b/benchmarks/multi_node/amd_utils/server_tilert.sh index dba758c1c8..307d57ef65 100644 --- a/benchmarks/multi_node/amd_utils/server_tilert.sh +++ b/benchmarks/multi_node/amd_utils/server_tilert.sh @@ -91,6 +91,12 @@ echo "[tilert] ROLE=$TILERT_ROLE rank=$NODE_RANK host=$host_name ($host_ip)" echo "[tilert] PREFILL_HOST=$PREFILL_HOST:$PREFILL_PORT DECODE_HOST=$DECODE_HOST:$DECODE_CTRL_PORT/$DECODE_HTTP_PORT ROUTER=:$ROUTER_PORT" echo "[tilert] MODEL_PATH=$MODEL_PATH profile=$TILERT_PROFILE served=$SERVED_MODEL_NAME max_len=$TILERT_MAX_MODEL_LEN transport=$TILERT_TRANSPORT kv=${PREFILL_KV_DTYPE}->${DECODE_KV_DTYPE} mtp=${SPEC_DECODING} agentic=$TILERT_IS_AGENTIC" +# Enable libibverbs fork safety on both ranks before any verbs context exists. +# Without it, ibv_fork_init() can fail in these containers while Mooncake +# initialization still reports success, silently falling back from RDMA to TCP. +# Slower prefill-to-decode KV transfer degrades TTFT; TPOT is unaffected. +export RDMAV_FORK_SAFE=1 + for env_pair in ${TILERT_EXTRA_ENV}; do export "${env_pair?}" echo "[tilert][EXTRA_ENV] $env_pair" diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 7afb667d7a..23f6b41dc3 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8474,3 +8474,19 @@ - "Same two-image recipe as glm5.3-fp8-mi355x-tilert (decode ghcr.io/tile-ai/tilert-rocm-decode:0.1.6, prefill ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6, tilert==0.1.6 pip-installed into both at container start), bf16 KV cache on both roles, kv-p2p-transfer: mooncake, kv-offloading none." - "The pd_router runs with --parser none and a 1800-second queue timeout so requests wait on the bs=1 decode engine instead of failing fast; AgentX traces are filtered at the 1048576-token full GLM context both roles are launched with; AIPerf scrapes only the vLLM prefill Prometheus endpoint (TileRT decode exposes no /metrics route)." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3330 + +- config-keys: + - glm5.3-fp8-mi355x-tilert + description: + - "Set RDMAV_FORK_SAFE=1 in the shared TileRT launcher for both prefill and decode ranks before verbs initialization, preventing silent RDMA-to-TCP fallback that degrades KV-transfer latency and TTFT; TPOT is unaffected. Ported from CrimsonDump/InferenceX commit 9e745f3697b360992e37747b53fac5e928cfa709." + - "在 TileRT 共享启动脚本中，为 prefill 和 decode 两侧在 verbs 初始化前设置 RDMAV_FORK_SAFE=1，避免静默从 RDMA 回退到 TCP 导致 KV 传输延迟和 TTFT 恶化；TPOT 不受影响。移植自 CrimsonDump/InferenceX 提交 9e745f3697b360992e37747b53fac5e928cfa709。" + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3330 + +- config-keys: + - glm5.3-fp8-mi355x-tilert-agentic + scenario-type: + - agentic-coding + description: + - "Set RDMAV_FORK_SAFE=1 on both TileRT ranks for AgentX, preventing silent RDMA-to-TCP fallback that degrades KV-transfer latency and TTFT; TPOT is unaffected." + - "为 AgentX 的 TileRT 两侧设置 RDMAV_FORK_SAFE=1，避免静默从 RDMA 回退到 TCP 导致 KV 传输延迟和 TTFT 恶化；TPOT 不受影响。" + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3330 From 0c8b6739f18fc6d47527faa9ee38db3725489686 Mon Sep 17 00:00:00 2001 From: CrimsonDump Date: Tue, 22 Sep 2026 00:18:22 +0800 Subject: [PATCH 06/25] =?UTF-8?q?chore(tilert):=20re-dispatch=20the=20swee?= =?UTF-8?q?p=20now=20that=20GLM-5.3=20is=20staged=20/=20=E6=9D=83=E9=87=8D?= =?UTF-8?q?=E5=B0=B1=E4=BD=8D=E5=90=8E=E9=87=8D=E6=96=B0=E8=A7=A6=E5=8F=91?= =?UTF-8?q?=20sweep?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The three earlier sweeps failed at the model-availability check because /it-share/data/GLM-5.3 did not exist on the MI355X cluster. zai-org/GLM-5.3 is now staged there: 155 files, 141 shards, 704 GiB, every file byte-exact against the Hugging Face manifest, world-readable for the runner account. config.json confirms fp8 (e4m3, weight_block_size 128x128) and max_position_embeddings 1048576, matching the recipe's precision and context. No code change. Re-applying the sweep label produced label events but no workflow run, and run-sweep.yml has no workflow_dispatch trigger, so this empty commit provides the synchronize event. 此前三次 sweep 均因 MI355X 集群上不存在 /it-share/data/GLM-5.3 而在模型可用性 检查处失败。现已完成权重落盘：155 个文件、141 个分片、704 GiB，逐文件字节数与 Hugging Face manifest 完全一致，并已对 runner 账号开放读取权限。config.json 确认为 fp8（e4m3，weight_block_size 128x128）、max_position_embeddings 为 1048576，与配方的精度与上下文设置一致。 无代码改动。重新添加 sweep 标签只产生了 label 事件而未触发工作流，且 run-sweep.yml 没有 workflow_dispatch 触发器，故以此空提交产生 synchronize 事件。 From 45557d70fea51982184264e31b8e170de9164c38 Mon Sep 17 00:00:00 2001 From: CrimsonDump Date: Tue, 22 Sep 2026 00:21:04 +0800 Subject: [PATCH 07/25] =?UTF-8?q?chore:=20drop=20a=20stray=20test=20artifa?= =?UTF-8?q?ct=20/=20=E5=88=A0=E9=99=A4=E8=AF=AF=E6=8F=90=E4=BA=A4=E7=9A=84?= =?UTF-8?q?=E6=B5=8B=E8=AF=95=E4=BA=A7=E7=89=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit neg.err was shell trace output from a local submit-chain negative test, run with the working directory inside amd_utils and swept up by a `git add -A` in ad1c997. It is not part of the recipe. neg.err 是在 amd_utils 目录内执行提交链负向测试时产生的 shell trace 输出， 被 ad1c997 的 `git add -A` 误纳入，与配方无关。 --- benchmarks/multi_node/amd_utils/neg.err | 93 ------------------------- 1 file changed, 93 deletions(-) delete mode 100644 benchmarks/multi_node/amd_utils/neg.err diff --git a/benchmarks/multi_node/amd_utils/neg.err b/benchmarks/multi_node/amd_utils/neg.err deleted file mode 100644 index 8163069f7d..0000000000 --- a/benchmarks/multi_node/amd_utils/neg.err +++ /dev/null @@ -1,93 +0,0 @@ -+ cd /Users/bshan/InferenceX/benchmarks/multi_node/amd_utils -+ export TIME_LIMIT=08:00:00 -+ TIME_LIMIT=08:00:00 -+ export MODEL_PATH=/tmp/claude-502/-Users-bshan-InferenceX/a3355d0f-16c4-46ef-b540-6e20f6eba79c/scratchpad/d3/models -+ MODEL_PATH=/tmp/claude-502/-Users-bshan-InferenceX/a3355d0f-16c4-46ef-b540-6e20f6eba79c/scratchpad/d3/models -+ export MODEL_NAME=GLM-5.3 -+ MODEL_NAME=GLM-5.3 -+ export CONTAINER_IMAGE=img -+ CONTAINER_IMAGE=img -+ export PREFILL_IMAGE -+ [[ 1 -ne 1 ]] -+ [[ 1 -ne 1 ]] -+ [[ 1 -ne 1 ]] -+ [[ 1 -ne 1 ]] -+ export TILERT_VERSION=0.1.6 -+ TILERT_VERSION=0.1.6 -+ export TILERT_PROFILE=glm5_2 -+ TILERT_PROFILE=glm5_2 -+ export TILERT_MODEL_TYPE=glm-5 -+ TILERT_MODEL_TYPE=glm-5 -+ export TILERT_MODEL_PKG=glm_5_2_rocm -+ TILERT_MODEL_PKG=glm_5_2_rocm -+ export SERVED_MODEL_NAME=glm5_2 -+ SERVED_MODEL_NAME=glm5_2 -+ export TILERT_MAX_MODEL_LEN=1048576 -+ TILERT_MAX_MODEL_LEN=1048576 -+ export TILERT_TRANSPORT=mooncake -+ TILERT_TRANSPORT=mooncake -+ export TILERT_PARSER=none -+ TILERT_PARSER=none -+ export TILERT_RDMA_STRICT=0 -+ TILERT_RDMA_STRICT=0 -+ export TILERT_CONVERT_LOCK_WAIT=21600 -+ TILERT_CONVERT_LOCK_WAIT=21600 -+ export TILERT_SIMULATE_ACC_METHOD=match-expected -+ TILERT_SIMULATE_ACC_METHOD=match-expected -+ export TILERT_WEIGHTS_DIR=/models/GLM-5.3-tilert-tp8 -+ TILERT_WEIGHTS_DIR=/models/GLM-5.3-tilert-tp8 -+ export PREFILL_KV_DTYPE=auto -+ PREFILL_KV_DTYPE=auto -+ export PREFILL_BLOCK_SIZE=64 -+ PREFILL_BLOCK_SIZE=64 -+ export DECODE_KV_DTYPE=bf16 -+ DECODE_KV_DTYPE=bf16 -+ export GPU_MEM_UTIL=0.75 -+ GPU_MEM_UTIL=0.75 -+ export SKIP_CONTAINER_BARRIER=0 -+ SKIP_CONTAINER_BARRIER=0 -+ export ROUTER_PORT=30000 -+ ROUTER_PORT=30000 -+ export PREFILL_PORT=8000 -+ PREFILL_PORT=8000 -+ export DECODE_CTRL_PORT=5556 -+ DECODE_CTRL_PORT=5556 -+ export DECODE_HTTP_PORT=5557 -+ DECODE_HTTP_PORT=5557 -+ export DECODE_WAIT=7200 -+ DECODE_WAIT=7200 -+ export PREFILL_WAIT=3600 -+ PREFILL_WAIT=3600 -+ export ROUTER_WAIT=10800 -+ ROUTER_WAIT=10800 -+ [[ mtp == \m\t\p ]] -+ export DECODE_MTP_SIZE=3 -+ DECODE_MTP_SIZE=3 -+ export PREFILL_SPEC_TOKENS=1 -+ PREFILL_SPEC_TOKENS=1 -+ export TILERT_QUEUE_TIMEOUT=0 -+ TILERT_QUEUE_TIMEOUT=0 -+ export KV_OFFLOADING=none -+ KV_OFFLOADING=none -+ [[ 1 -ne 1 ]] -+ [[ 1 -ne 1 ]] -+ [[ false == \t\r\u\e ]] -+ [[ false == \t\r\u\e ]] -+ export PREFILL_ENABLE_EP=false -+ PREFILL_ENABLE_EP=false -+ export PREFILL_ENABLE_DP=false -+ PREFILL_ENABLE_DP=false -+ export DECODE_ENABLE_EP=false -+ DECODE_ENABLE_EP=false -+ export DECODE_ENABLE_DP=false -+ DECODE_ENABLE_DP=false -++ bash ./submit.sh 1 1 1 1 8192 1024 1 inf false false false false 8 8 1 -+ JOB_ID='Error: The following required environment variables are not set: - - ROUTER_TYPE - - PROXY_PING_PORT - - HEADNODE_PORT - - SERVER_PORT' -+ [[ 1 -ne 0 ]] -+ echo 'Failed to submit job' -Failed to submit job -+ exit 1 From dcf4dcd49955c39c251883db3d476793bfbc4c78 Mon Sep 17 00:00:00 2001 From: CrimsonDump Date: Tue, 22 Sep 2026 00:34:52 +0800 Subject: [PATCH 08/25] =?UTF-8?q?fix(tilert):=20stop=20validating=20MoRI/S?= =?UTF-8?q?GLang=20env=20on=20the=20mooncake=20path=20/=20TileRT=20?= =?UTF-8?q?=E8=B7=AF=E5=BE=84=E4=B8=8D=E5=86=8D=E6=A0=A1=E9=AA=8C=20MoRI?= =?UTF-8?q?=20=E4=B8=8E=20SGLang=20=E7=8E=AF=E5=A2=83=E5=8F=98=E9=87=8F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sweep 35625047683 reached the container, installed tilert 0.1.6 and found mooncake, then failed in env.sh on eleven variables it never reads: MORI_IO_SQ_BACKOFF_TIMEOUT_US, MORI_IO_QP_MAX_SEND_WR, MORI_IO_QP_MAX_CQE, MORI_IO_QP_MAX_SGE, MORI_IO_TC_DISABLE, UCX_IB_GID_INDEX, MORI_APP_LOG_LEVEL, SGLANG_ROUTER_STDOUT_LOGS, TORCH_NCCL_BLOCKING_WAIT, NCCL_BLOCKING_WAIT and SGLANG_OPT_USE_AITER_INDEXER. Those are MoRI-IO queue-pair tuning, the UCX RoCE GID index, SGLang router logging and the SGLang decode cuda-graph NCCL workaround. server_tilert.sh references none of them; KV moves over mooncake and no SGLang router starts. env.sh validated them before it had even validated ENGINE, so the list is now scoped to the engines that consume it. Supplying values instead would mean inventing MoRI tuning for a transport this recipe never uses. Verified that ENGINE=sglang-disagg still fails on the same list. Also export INFMAX_CONTAINER_WORKSPACE=/workspace for the agentic replay, which runs in this container rather than a separate client: benchmark_lib's install_agentic_deps and build_replay_cmd both require it, the SGLang path sets it in its client container's env file, and launch_mi355x-amds.sh exports it only on the DSv4.1-Flash single-node branch. Without it the agentic lane would have failed at the next step. sweep 35625047683 已进入容器、装好 tilert 0.1.6 并找到 mooncake，随后在 env.sh 因十一个它从不读取的变量而失败。这些是 MoRI-IO 队列对调优、UCX RoCE GID 索引、 SGLang router 日志以及 SGLang decode cuda-graph 的 NCCL 规避项；server_tilert.sh 一概不引用，KV 走 mooncake，也不启动 SGLang router。env.sh 甚至在校验 ENGINE 之前就校验了它们，现已将该列表限定到真正消费它们的引擎；若改为提供取值，等于为 本配方根本不用的传输层编造 MoRI 调优。已验证 ENGINE=sglang-disagg 仍会因同一 列表失败。 同时为 agentic 回放导出 INFMAX_CONTAINER_WORKSPACE=/workspace：回放在本容器内 运行而非独立客户端容器，benchmark_lib 的 install_agentic_deps 与 build_replay_cmd 均要求该变量，而 SGLang 路径是在其客户端容器的 env 文件中设置、launcher 仅在 DSv4.1-Flash 单节点分支导出。否则 agentic lane 会在下一步失败。 --- benchmarks/multi_node/amd_utils/env.sh | 16 ++++++++++++---- benchmarks/multi_node/amd_utils/server_tilert.sh | 5 +++++ 2 files changed, 17 insertions(+), 4 deletions(-) diff --git a/benchmarks/multi_node/amd_utils/env.sh b/benchmarks/multi_node/amd_utils/env.sh index bfc006112e..82048eea01 100755 --- a/benchmarks/multi_node/amd_utils/env.sh +++ b/benchmarks/multi_node/amd_utils/env.sh @@ -1,10 +1,18 @@ #!/bin/bash source "$(dirname "${BASH_SOURCE[0]}")/../../benchmark_lib.sh" --validation-only -check_env_vars \ - MORI_IO_SQ_BACKOFF_TIMEOUT_US MORI_IO_QP_MAX_SEND_WR MORI_IO_QP_MAX_CQE MORI_IO_QP_MAX_SGE MORI_IO_TC_DISABLE \ - UCX_IB_GID_INDEX MORI_APP_LOG_LEVEL SGLANG_ROUTER_STDOUT_LOGS TORCH_NCCL_BLOCKING_WAIT NCCL_BLOCKING_WAIT \ - SGLANG_OPT_USE_AITER_INDEXER +check_env_vars ENGINE +# MoRI-IO queue-pair tuning, the UCX RoCE GID index, SGLang router logging and the +# SGLang decode cuda-graph NCCL workaround. Only the SGLang and vLLM MoRI KV paths +# below read these. ENGINE=tilert moves KV over mooncake and starts no SGLang +# router, so it is neither given nor reads them: validating them there would force +# the recipe to invent MoRI tuning for a transport it never uses. +if [[ "$ENGINE" != "tilert" ]]; then + check_env_vars \ + MORI_IO_SQ_BACKOFF_TIMEOUT_US MORI_IO_QP_MAX_SEND_WR MORI_IO_QP_MAX_CQE MORI_IO_QP_MAX_SGE MORI_IO_TC_DISABLE \ + UCX_IB_GID_INDEX MORI_APP_LOG_LEVEL SGLANG_ROUTER_STDOUT_LOGS TORCH_NCCL_BLOCKING_WAIT NCCL_BLOCKING_WAIT \ + SGLANG_OPT_USE_AITER_INDEXER +fi # Dual-engine environment setup for multi-node disaggregated serving. # # ENGINE=sglang-disagg or vllm-disagg selects the engine-specific block. diff --git a/benchmarks/multi_node/amd_utils/server_tilert.sh b/benchmarks/multi_node/amd_utils/server_tilert.sh index 307d57ef65..bf89eec89b 100644 --- a/benchmarks/multi_node/amd_utils/server_tilert.sh +++ b/benchmarks/multi_node/amd_utils/server_tilert.sh @@ -433,6 +433,11 @@ run_agentic_replay() { export MODEL="$MODEL_PATH" # aiperf --tokenizer (local HF dir) export SERVED_MODEL_NAME # aiperf --model (name the router/vLLM serve) check_env_vars DURATION RESULT_FILENAME + # The replay runs in this container, where the repo is bind-mounted at + # /workspace, so benchmark_lib's agentic helpers need the workspace root + # here. The SGLang path sets it in the separate client container's env file, + # and the launcher only exports it for the DSv4.1-Flash single-node branch. + export INFMAX_CONTAINER_WORKSPACE=/workspace export MAX_MODEL_LEN="$TILERT_MAX_MODEL_LEN" # TileRT decode exposes no /metrics route; only the vLLM prefill is scraped. export AIPERF_SERVER_METRICS_URLS="http://${PREFILL_HOST}:${PREFILL_PORT}/metrics" From 19bea0bdfeea70a333b3adc448fefeaa623a38aa Mon Sep 17 00:00:00 2001 From: CrimsonDump Date: Tue, 22 Sep 2026 08:05:42 +0800 Subject: [PATCH 09/25] =?UTF-8?q?fix(tilert):=20fp8=20MLA=20KV=20on=20both?= =?UTF-8?q?=20ranks=20so=20the=201M=20context=20fits=20/=20=E4=B8=A4?= =?UTF-8?q?=E4=BE=A7=E6=94=B9=E7=94=A8=20fp8=20MLA=20KV=20=E4=BB=A5?= =?UTF-8?q?=E5=AE=B9=E7=BA=B3=201M=20=E4=B8=8A=E4=B8=8B=E6=96=87?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sweep 35626519997 failed on both lanes for the same reason, on opposite roles: decode (agentic, g14) torch.OutOfMemoryError: tried to allocate 99.06 GiB with 95.94 GiB free; 184.17 GiB of weights already resident on a 287.98 GiB card prefill (8k1k, g09) ValueError: 91.71 GiB KV cache is needed, which is larger than the available KV cache memory (85.25 GiB) bf16 MLA KV does not fit at 1048576 tokens on either rank. The recipe chose bf16 because the ROCm sparse-MLA backend has no fp8_ds_mla, which is true but does not imply bf16: ROCM_AITER_MLA_SPARSE lists fp8 and fp8_e4m3 in supported_kv_cache_dtypes, and TileRT's MlaNsaProfile.configure() maps fp8_ds_mla, fp8 and fp8_e4m3 onto the same mla_fp8 layout. Only fp8_ds_mla is CUDA-only. Plain fp8 therefore gives both ranks the matching layout that TileRT's --kv-cache-dtype requires, and halves the cache: KV_BYTES_FP8 = 528 B/token against KV_BYTES_BF16 = 1024 B/token. Decode drops to roughly 51 GB alongside its weights, prefill to roughly 46 GiB inside its 85.25 GiB budget. Also take the container-creation barrier timeout from CONTAINER_BARRIER_TIMEOUT (5400s) rather than hardcoding 1800s, matching how server_sglang.sh sources it. This recipe pulls two ~32 GB images, one per rank; on a node caching neither, the rank that comes up first waits out the whole timeout while its peer pulls. The observed cold decode pull took about 25 minutes of the old 30-minute budget. sweep 35626519997 两条 lane 因同一原因在相对的两侧失败：decode 侧在 287.98 GiB 卡上已驻留 184.17 GiB 权重、再分配 99.06 GiB KV 时 OOM；prefill 侧因需要 91.71 GiB KV 而可用 85.25 GiB 被 vLLM 拒绝。1048576 上下文下 bf16 MLA KV 两侧 均无法容纳。配方原先选 bf16 是因为 ROCm 稀疏 MLA 没有 fp8_ds_mla，此事为真但并 不意味着只能用 bf16：ROCM_AITER_MLA_SPARSE 的 supported_kv_cache_dtypes 含 fp8 与 fp8_e4m3，而 TileRT 的 MlaNsaProfile.configure() 将 fp8_ds_mla/fp8/fp8_e4m3 映射到同一 mla_fp8 布局，仅 fp8_ds_mla 为 CUDA 独有。因此普通 fp8 即可让两侧获得 TileRT 所要求的一致布局，并将 KV 减半（每 token 528 字节对 1024 字节）。 屏障超时同时改为取自 CONTAINER_BARRIER_TIMEOUT（5400 秒），与 server_sglang.sh 的做法一致：本配方每 rank 各拉取约 32 GB 镜像，在均未缓存的 节点上，先起来的一侧会在对端拉取期间耗尽原有 30 分钟预算（实测冷拉取约 25 分钟）。 --- .../agentic/glm5.3_fp8_mi355x_tilert.sh | 18 ++++++++++++++++-- .../multi_node/amd_utils/server_tilert.sh | 8 ++++++-- .../multi_node/glm5.3_fp8_mi355x_tilert.sh | 18 ++++++++++++++++-- perf-changelog.yaml | 12 ++++++++++-- 4 files changed, 48 insertions(+), 8 deletions(-) diff --git a/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh b/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh index 824df13538..f40f9ed306 100644 --- a/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh +++ b/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh @@ -75,13 +75,27 @@ export TILERT_RDMA_STRICT=0 export TILERT_CONVERT_LOCK_WAIT=21600 export TILERT_SIMULATE_ACC_METHOD=match-expected export TILERT_WEIGHTS_DIR="/models/${MODEL_NAME}-tilert-tp${DECODE_TP}" -export PREFILL_KV_DTYPE=auto +# fp8 MLA KV on both roles, which is what makes the full 1M context fit. +# TileRT's MlaNsaProfile.configure() maps fp8_ds_mla/fp8/fp8_e4m3 to the same +# mla_fp8 layout, and vLLM's ROCM_AITER_MLA_SPARSE backend lists fp8 in its +# supported_kv_cache_dtypes. Only fp8_ds_mla is CUDA-only, so plain fp8 gives +# both ranks the matching layout TileRT requires. +# At bf16 (KV_BYTES_BF16 = 1024 B/token) decode needed a 99.06 GB buffer on top +# of 184.17 GiB of weights on a 287.98 GiB card and OOM-killed; vLLM prefill +# refused outright, wanting 91.71 GiB of KV against 85.25 GiB available. +# fp8 (KV_BYTES_FP8 = 528 B/token) roughly halves both. +export PREFILL_KV_DTYPE=fp8 # The ROCm backend supports block sizes [1, 64] and vLLM picks 1, which makes # the connector's KI plane copy fail and MLA address the wrong rows. export PREFILL_BLOCK_SIZE=64 -export DECODE_KV_DTYPE=bf16 # the ROCm sparse-MLA backend has no fp8_ds_mla +export DECODE_KV_DTYPE=fp8 export GPU_MEM_UTIL=0.75 export SKIP_CONTAINER_BARRIER=0 +# Two images, one per rank, ~32 GB each. On a node that has neither cached the +# pull alone outlasts the SGLang path's 300s default and the 1800s this script +# used to hardcode, and the rank that comes up first waits out the whole +# timeout while its peer is still pulling. +export CONTAINER_BARRIER_TIMEOUT=5400 export ROUTER_PORT=30000 export PREFILL_PORT=8000 export DECODE_CTRL_PORT=5556 diff --git a/benchmarks/multi_node/amd_utils/server_tilert.sh b/benchmarks/multi_node/amd_utils/server_tilert.sh index bf89eec89b..1801018aab 100644 --- a/benchmarks/multi_node/amd_utils/server_tilert.sh +++ b/benchmarks/multi_node/amd_utils/server_tilert.sh @@ -19,7 +19,8 @@ check_env_vars \ PREFILL_KV_DTYPE PREFILL_BLOCK_SIZE PREFILL_SPEC_TOKENS DECODE_KV_DTYPE \ DECODE_MTP_SIZE GPU_MEM_UTIL SERVED_MODEL_NAME \ DECODE_CTRL_PORT DECODE_HTTP_PORT PREFILL_PORT ROUTER_PORT \ - DECODE_WAIT PREFILL_WAIT ROUTER_WAIT SKIP_CONTAINER_BARRIER + DECODE_WAIT PREFILL_WAIT ROUTER_WAIT SKIP_CONTAINER_BARRIER \ + CONTAINER_BARRIER_TIMEOUT LOG_DIR="/run_logs/slurm_job-${SLURM_JOB_ID}" SHARED_LOG_DIR="${BENCHMARK_LOGS_DIR}/logs/slurm_job-${SLURM_JOB_ID}" @@ -481,7 +482,10 @@ else "$PY" "$WS_PATH/sync.py" barrier \ --local-ip "${host_ip}" --local-port 5000 --enable-port \ --node-ips "${IPADDRS}" --node-ports 5000 \ - --wait-for-all-ports --timeout 1800 || { echo "ERROR: container creation barrier failed" >&2; exit 1; } + --wait-for-all-ports --timeout "$CONTAINER_BARRIER_TIMEOUT" \ + || { echo "ERROR: container creation barrier failed after ${CONTAINER_BARRIER_TIMEOUT}s -- the peer rank never opened port 5000." \ + "A cold image pull is the usual cause: this recipe pulls two ~32 GB images, one per rank, and the rank that" \ + "comes up first waits out the whole timeout while the other is still pulling." >&2; exit 1; } fi case "$TILERT_ROLE" in diff --git a/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh b/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh index 08e4261187..79201c4759 100644 --- a/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh +++ b/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh @@ -61,13 +61,27 @@ export TILERT_RDMA_STRICT=0 export TILERT_CONVERT_LOCK_WAIT=21600 export TILERT_SIMULATE_ACC_METHOD=match-expected export TILERT_WEIGHTS_DIR="/models/${MODEL_NAME}-tilert-tp${DECODE_TP}" -export PREFILL_KV_DTYPE=auto +# fp8 MLA KV on both roles, which is what makes the full 1M context fit. +# TileRT's MlaNsaProfile.configure() maps fp8_ds_mla/fp8/fp8_e4m3 to the same +# mla_fp8 layout, and vLLM's ROCM_AITER_MLA_SPARSE backend lists fp8 in its +# supported_kv_cache_dtypes. Only fp8_ds_mla is CUDA-only, so plain fp8 gives +# both ranks the matching layout TileRT requires. +# At bf16 (KV_BYTES_BF16 = 1024 B/token) decode needed a 99.06 GB buffer on top +# of 184.17 GiB of weights on a 287.98 GiB card and OOM-killed; vLLM prefill +# refused outright, wanting 91.71 GiB of KV against 85.25 GiB available. +# fp8 (KV_BYTES_FP8 = 528 B/token) roughly halves both. +export PREFILL_KV_DTYPE=fp8 # The ROCm backend supports block sizes [1, 64] and vLLM picks 1, which makes # the connector's KI plane copy fail and MLA address the wrong rows. export PREFILL_BLOCK_SIZE=64 -export DECODE_KV_DTYPE=bf16 # the ROCm sparse-MLA backend has no fp8_ds_mla +export DECODE_KV_DTYPE=fp8 export GPU_MEM_UTIL=0.75 export SKIP_CONTAINER_BARRIER=0 +# Two images, one per rank, ~32 GB each. On a node that has neither cached the +# pull alone outlasts the SGLang path's 300s default and the 1800s this script +# used to hardcode, and the rank that comes up first waits out the whole +# timeout while its peer is still pulling. +export CONTAINER_BARRIER_TIMEOUT=5400 export ROUTER_PORT=30000 export PREFILL_PORT=8000 export DECODE_CTRL_PORT=5556 diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 73a69880ce..dd96627fd2 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8520,7 +8520,7 @@ description: - "Add GLM-5.3 FP8 MI355X prefill/decode-disaggregated Single-turn 8k1k benchmark via TileRT: upstream vLLM ROCm prefill (TileRTConnector, kv_producer) + TileRT decode_server + OpenAI-compatible pd_router, framework=tilert, mirroring the B200 glm5.1-fp8-b200-tilert entry." - "Two images per job, both first-party and neither carrying the TileRT wheel: the decode rank runs ghcr.io/tile-ai/tilert-rocm-decode:0.1.6 (ROCm PyTorch runtime plus the mooncake ROCm transfer engine built from source) and the prefill rank runs ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6 (the upstream vllm/vllm-openai-rocm release image plus that same transfer engine, with vLLM itself unpatched and no file inside it modified), selected per rank via PREFILL_IMAGE in prefill.additional-settings. setup_deps.sh pip-installs the official tilert==0.1.6 wheel and the router HTTP deps into both ranks at container start (full install on decode, --no-deps connector plugin on prefill), so neither image is pinned to a wheel version; this mirrors the B200 glm5.1-fp8-b200-tilert pattern of a first-party TileRT decode image. The transfer engine is built from source because the published mooncake ROCm wheels are compiled without ENABLE_MULTI_PROTOCOL, which leaves the cross-host locality gate (kvcache-ai/Mooncake#2753) out of the binary so every cross-node KV transfer is treated as node-local and fails in hipIpcOpenMemHandle." - - "KV moves P->D over mooncake (kv-p2p-transfer: mooncake); both roles use a bf16 KV cache (prefill --kv-cache-dtype auto, decode bf16) because the ROCm vLLM sparse-MLA backend has no fp8_ds_mla dtype. max-model-len 1048576 on both roles (GLM-5.3's full context window); MTP wired via spec-decoding=mtp: TileRT decode drafts at depth 3 (DECODE_MTP_SIZE, the only depth the ROCm GLM profile builds, and the key the golden acceptance curve is read at), while the vLLM prefill rank only materialises the MTP layer's KV and runs speculative-config mtp with 1 draft token. TileRT decode is bs=1 only, so conc-list is [1]; topology 1 prefill node (TP8) + 1 decode node (TP8)." + - "KV moves P->D over mooncake (kv-p2p-transfer: mooncake); both roles use an fp8 MLA KV cache (--kv-cache-dtype fp8 on each). fp8_ds_mla is CUDA-only, but vLLM's ROCM_AITER_MLA_SPARSE backend lists fp8 among its supported_kv_cache_dtypes and TileRT's MlaNsaProfile maps fp8_ds_mla/fp8/fp8_e4m3 onto one mla_fp8 layout, so plain fp8 gives both ranks the matching layout TileRT requires. bf16 does not fit at this context: decode needed a 99.06 GB KV buffer on top of 184.17 GiB of weights on a 287.98 GiB card, and vLLM prefill refused with 91.71 GiB of KV needed against 85.25 GiB available. max-model-len 1048576 on both roles (GLM-5.3's full context window); MTP wired via spec-decoding=mtp: TileRT decode drafts at depth 3 (DECODE_MTP_SIZE, the only depth the ROCm GLM profile builds, and the key the golden acceptance curve is read at), while the vLLM prefill rank only materialises the MTP layer's KV and runs speculative-config mtp with 1 draft token. TileRT decode is bs=1 only, so conc-list is [1]; topology 1 prefill node (TP8) + 1 decode node (TP8)." - "Orchestration: runners/launch_mi355x-amds.sh dispatches framework=tilert through the existing amd_utils chain (submit.sh -> job.slurm -> server.sh -> new server_tilert.sh); job.slurm selects the image per rank and a tilert env block, setup_deps.sh gains a per-role tilert install branch, models_tilert.yaml holds the GLM-5.3 profile defaults. TileRT decode weights are converted once from the HF checkpoint under a flock and cached next to it on the shared model volume. Cross-node teardown uses the amd_utils router-port-close barrier." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3330 @@ -8530,7 +8530,7 @@ - agentic-coding description: - "Add GLM-5.3 FP8 MI355X AgentX (agentic-coding) via TileRT prefill/decode disaggregation with MTP, 1 prefill node (TP8) + 1 decode node (TP8), concurrency [1], mirroring the B200 glm5.1-fp8-b200-tilert-agentic entry on the amd_utils orchestration." - - "Same two-image recipe as glm5.3-fp8-mi355x-tilert (decode ghcr.io/tile-ai/tilert-rocm-decode:0.1.6, prefill ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6, tilert==0.1.6 pip-installed into both at container start), bf16 KV cache on both roles, kv-p2p-transfer: mooncake, kv-offloading none." + - "Same two-image recipe as glm5.3-fp8-mi355x-tilert (decode ghcr.io/tile-ai/tilert-rocm-decode:0.1.6, prefill ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6, tilert==0.1.6 pip-installed into both at container start), fp8 MLA KV cache on both roles, kv-p2p-transfer: mooncake, kv-offloading none." - "The pd_router runs with --parser none and a 1800-second queue timeout so requests wait on the bs=1 decode engine instead of failing fast; AgentX traces are filtered at the 1048576-token full GLM context both roles are launched with; AIPerf scrapes only the vLLM prefill Prometheus endpoint (TileRT decode exposes no /metrics route)." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3330 @@ -8549,3 +8549,11 @@ - "Set RDMAV_FORK_SAFE=1 on both TileRT ranks for AgentX, preventing silent RDMA-to-TCP fallback that degrades KV-transfer latency and TTFT; TPOT is unaffected." - "为 AgentX 的 TileRT 两侧设置 RDMAV_FORK_SAFE=1，避免静默从 RDMA 回退到 TCP 导致 KV 传输延迟和 TTFT 恶化；TPOT 不受影响。" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3330 + +- config-keys: + - glm5.3-fp8-mi355x-tilert + - glm5.3-fp8-mi355x-tilert-agentic + description: + - "Serve the full 1048576-token GLM-5.3 context with an fp8 MLA KV cache on both TileRT ranks, and take the container-creation barrier timeout from CONTAINER_BARRIER_TIMEOUT (5400s) instead of a hardcoded 1800s. At bf16 the 1M context did not fit: the decode rank needed a 99.06 GB KV buffer on top of 184.17 GiB of weights on a 287.98 GiB MI355X and was OOM-killed, while vLLM prefill refused to start, needing 91.71 GiB of KV against the 85.25 GiB available at gpu-memory-utilization 0.75. fp8 halves both (KV_BYTES_FP8 528 B/token vs KV_BYTES_BF16 1024 B/token). The barrier change covers the cold pull of this recipe's two ~32 GB per-rank images, which outlasts the old timeout on a node that has cached neither." + - "以 fp8 MLA KV cache 在两侧 TileRT rank 上提供完整的 1048576 上下文，并将容器创建屏障超时改为由 CONTAINER_BARRIER_TIMEOUT（5400 秒）提供，不再硬编码 1800 秒。bf16 下 1M 上下文无法容纳：decode 侧在 287.98 GiB 的 MI355X 上需在 184.17 GiB 权重之外再分配 99.06 GB KV 缓冲而被 OOM 杀死，vLLM prefill 则因需要 91.71 GiB KV 而可用仅 85.25 GiB 直接拒绝启动。fp8 将两者减半（每 token 528 字节对 1024 字节）。屏障改动用于覆盖本配方两个每 rank 约 32 GB 镜像的冷拉取，在两者均未缓存的节点上会超出原超时。" + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3330 From 432e651116cb4c31e628410f285e6dbc4aeba363 Mon Sep 17 00:00:00 2001 From: CrimsonDump Date: Tue, 22 Sep 2026 10:05:59 +0800 Subject: [PATCH 10/25] =?UTF-8?q?fix(tilert):=20bf16=20wire=20layout,=201M?= =?UTF-8?q?=20context=20gated=20on=20DRAM=20PD=20buffers=20[skip-sweep]=20?= =?UTF-8?q?/=20=E6=81=A2=E5=A4=8D=20bf16=20=E5=B8=83=E5=B1=80=EF=BC=8C1M?= =?UTF-8?q?=20=E4=B8=8A=E4=B8=8B=E6=96=87=E5=8F=96=E5=86=B3=E4=BA=8E=20DRA?= =?UTF-8?q?M=20=E4=BE=A7=20PD=20=E7=BC=93=E5=86=B2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reverts the fp8 KV change from 19bea0bdf. It cannot run: vLLM 0.24.0's ROCM_AITER_MLA_SPARSE writes a flat 576 B/token fp8 row with a per-layer scale, and TileRT 0.1.6 infers the layout from the tensor stride and accepts only 1152 B (bf16) or 656 B (fp8_ds_mla), raising "unexpected MLA cache stride 576 B/token" at register_kv_caches on every prefill rank. The original bf16 choice was the only layout-correct one on ROCm. PREFILL_KV_DTYPE is now an explicit bfloat16 (auto resolves the same on this model; float16 would pass the stride check and be misread). Memory at 1048576 tokens, per rank, verified against the tilert and vLLM sources and the MI355X logs: decode : 90.72 GiB weights + 93.25 GiB engine cache window + 99.06 GiB PD receive buffer (~283 of 287.98 GiB) prefill: 90.45 GiB weights + 40.3 GiB profiling + 91.71 GiB vLLM KV + 99.06 GiB PD staging buffer outside vLLM's budget (~321 GiB) The two PD buffers are dense in max_seq_len. Both must move to pinned host memory for this context to start, which is a TileRT change; the recipe is set up for that release (GPU_MEM_UTIL 0.85 so vLLM's 222.5 GiB fits inside its budget) and keeps 1048576. Until the wheel and images are bumped the sweep cannot pass, hence [skip-sweep]. Also: drop the invented TILERT_CONVERT_DEVICE default; pass --num-mtp "$DECODE_MTP_SIZE" to decode_server so converter, golden key and engine depth agree by data flow; make the conversion-cache check read tilert_meta.json num_mtp so a stale non-MTP cache is re-converted instead of failing after a 90 GiB weight load; validate CONTAINER_BARRIER_TIMEOUT in job.slurm's tilert branch; correct the earlier changelog text (router deps are installed on decode only, models_tilert.yaml holds prefill env, not profile defaults) and replace the wrong halving arithmetic. 撤销 19bea0bdf 的 fp8 KV 改动：vLLM 0.24.0 的 ROCM_AITER_MLA_SPARSE 写出每 token 576 字节的扁平 fp8 行，而 TileRT 0.1.6 依步长推断布局，只接受 1152（bf16）或 656（fp8_ds_mla）字节，会在 register_kv_caches 报 "unexpected MLA cache stride 576 B/token"。原作者的 bf16 是 ROCm 上唯一正确的布局。 1048576 上下文下每 rank 的内存（已对照源码与日志核实）：decode 侧 90.72 GiB 权重 + 93.25 GiB 引擎缓存窗口 + 99.06 GiB PD 接收缓冲（约 283/287.98 GiB）；prefill 侧 90.45 GiB 权重 + 40.3 GiB 分析开销 + 91.71 GiB vLLM KV + 位于 vLLM 预算之外 的 99.06 GiB PD 暂存缓冲（约 321 GiB）。两个 PD 缓冲按 max_seq_len 密集分配， 须移入锁页主机内存才能在该上下文启动，此为 TileRT 侧改动；配方已为该版本就位 （GPU_MEM_UTIL 0.85）并保留 1048576。wheel 与镜像升级前 sweep 无法通过，故 [skip-sweep]。 另：移除 TILERT_CONVERT_DEVICE 的自造默认；向 decode_server 显式传递 --num-mtp； 转换缓存检查读取 tilert_meta.json 的 num_mtp；job.slurm 的 tilert 分支校验 CONTAINER_BARRIER_TIMEOUT；修正此前 changelog 中的错误表述与减半算术。 --- .../agentic/glm5.3_fp8_mi355x_tilert.sh | 46 +++++++++++++------ benchmarks/multi_node/amd_utils/job.slurm | 6 +++ .../multi_node/amd_utils/server_tilert.sh | 18 +++++++- .../multi_node/glm5.3_fp8_mi355x_tilert.sh | 46 +++++++++++++------ perf-changelog.yaml | 12 ++--- 5 files changed, 92 insertions(+), 36 deletions(-) diff --git a/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh b/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh index f40f9ed306..c7701ed642 100644 --- a/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh +++ b/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh @@ -66,8 +66,22 @@ export TILERT_PROFILE=glm5_2 # decode_server --model (TileRT model prof export TILERT_MODEL_TYPE=glm-5 # weight_converter --model_type (fallback converter) export TILERT_MODEL_PKG=glm_5_2_rocm # per-model converter package, preferred when importable export SERVED_MODEL_NAME=glm5_2 -# GLM-5.3's full context window, as every in-tree GLM-5.2 recipe uses. -# (202752 is GLM-5.1's, inherited from the B200 TileRT recipe this mirrors.) +# GLM-5.3's full context window (config.json max_position_embeddings), as every +# in-tree GLM-5.2 recipe serves. (202752 was GLM-5.1's, inherited from the B200 +# TileRT recipe this mirrors.) +# +# Memory at this context, per rank, bf16 wire layout (verified against the +# tilert 0.1.6 and vLLM 0.24.0 sources and the MI355X logs, 287.98 GiB cards): +# decode : weights 90.72 GiB + engine cache window 93.25 GiB +# + PD receive buffer 99.06 GiB (receive_server.py, dense in max_seq_len) +# prefill: weights 90.45 GiB + profiling/non-torch 40.3 GiB + vLLM KV 91.71 GiB +# + PD staging buffer 99.06 GiB (prefill_connector.py, TP rank 0, +# allocated OUTSIDE vLLM's gpu-memory-utilization budget) +# Both PD buffers must live in pinned host memory (2.9 TB RAM per node) for +# this context to start; on the GPU the decode side is node-marginal (~283 of +# 288 GiB) and the prefill side cannot fit at any utilization (~321 GiB). +# TileRT 0.1.6 places both on the GPU; bump TILERT_VERSION and the two images +# to the release that moves them to DRAM. Until then this recipe cannot start. export TILERT_MAX_MODEL_LEN=1048576 export TILERT_TRANSPORT=mooncake export TILERT_PARSER=none @@ -75,21 +89,25 @@ export TILERT_RDMA_STRICT=0 export TILERT_CONVERT_LOCK_WAIT=21600 export TILERT_SIMULATE_ACC_METHOD=match-expected export TILERT_WEIGHTS_DIR="/models/${MODEL_NAME}-tilert-tp${DECODE_TP}" -# fp8 MLA KV on both roles, which is what makes the full 1M context fit. -# TileRT's MlaNsaProfile.configure() maps fp8_ds_mla/fp8/fp8_e4m3 to the same -# mla_fp8 layout, and vLLM's ROCM_AITER_MLA_SPARSE backend lists fp8 in its -# supported_kv_cache_dtypes. Only fp8_ds_mla is CUDA-only, so plain fp8 gives -# both ranks the matching layout TileRT requires. -# At bf16 (KV_BYTES_BF16 = 1024 B/token) decode needed a 99.06 GB buffer on top -# of 184.17 GiB of weights on a 287.98 GiB card and OOM-killed; vLLM prefill -# refused outright, wanting 91.71 GiB of KV against 85.25 GiB available. -# fp8 (KV_BYTES_FP8 = 528 B/token) roughly halves both. -export PREFILL_KV_DTYPE=fp8 +# bf16 MLA KV on both roles. This is the only layout TileRT 0.1.6 can consume +# from vLLM on ROCm: MlaNsaProfile.classify_layers infers the layout from the +# cache tensor stride and accepts exactly 1152 B/token (bf16) or 656 B/token +# (fp8_ds_mla). vLLM's ROCM_AITER_MLA_SPARSE backend has no fp8_ds_mla; its +# plain "fp8" writes a flat 576 B/token row, which the connector rejects at +# register_kv_caches. Explicit bfloat16 rather than auto so the stride does not +# depend on the model dtype. Never float16: it passes the 1152 B check and is +# then read as bf16. +export PREFILL_KV_DTYPE=bfloat16 # The ROCm backend supports block sizes [1, 64] and vLLM picks 1, which makes # the connector's KI plane copy fail and MLA address the wrong rows. export PREFILL_BLOCK_SIZE=64 -export DECODE_KV_DTYPE=fp8 -export GPU_MEM_UTIL=0.75 +export DECODE_KV_DTYPE=bf16 +# With the PD staging buffer in host memory, vLLM needs 90.45 (weights) + 40.3 +# (profiling) + 91.71 GiB (KV for one 1048576-token request) = 222.5 GiB inside +# its budget: 0.85 x 287.98 = 244.8 GiB leaves 22 GiB of KV margin and 43 GiB +# outside the budget for the ~6 GiB non-torch baseline. 0.75 (216 GiB) refuses +# with "91.71 GiB KV cache is needed ... available 85.25 GiB". +export GPU_MEM_UTIL=0.85 export SKIP_CONTAINER_BARRIER=0 # Two images, one per rank, ~32 GB each. On a node that has neither cached the # pull alone outlasts the SGLang path's 300s default and the 1800s this script diff --git a/benchmarks/multi_node/amd_utils/job.slurm b/benchmarks/multi_node/amd_utils/job.slurm index 6b69cb6d07..0c779476ea 100755 --- a/benchmarks/multi_node/amd_utils/job.slurm +++ b/benchmarks/multi_node/amd_utils/job.slurm @@ -58,6 +58,12 @@ if [[ "$ENGINE" == "tilert" && -z "${PREFILL_IMAGE:-}" ]]; then echo "Error: ENGINE=tilert requires PREFILL_IMAGE (e.g. PREFILL_IMAGE=vllm/vllm-openai-rocm:nightly- in prefill.additional-settings)." exit 1 fi +if [[ "$ENGINE" == "tilert" ]]; then + # server_tilert.sh takes the container-creation barrier timeout from the + # recipe (no 300s default as on the SGLang path); fail here, before sbatch + # work is done, rather than inside the container. + check_env_vars CONTAINER_BARRIER_TIMEOUT +fi # Resolve the models.yaml entry the same way server_sglang.sh does: agentic runs # (IS_AGENTIC) use the '-AgentX' recipe, non-agentic disaggregated runs use diff --git a/benchmarks/multi_node/amd_utils/server_tilert.sh b/benchmarks/multi_node/amd_utils/server_tilert.sh index 1801018aab..1e46867c3e 100644 --- a/benchmarks/multi_node/amd_utils/server_tilert.sh +++ b/benchmarks/multi_node/amd_utils/server_tilert.sh @@ -70,7 +70,10 @@ if [[ "$SPEC_DECODING" == "mtp" ]]; then # The prefill rank only has to build the MTP layer's KV; TileRT decode owns # the draft depth (DECODE_MTP_SIZE), so the two counts differ by design. PREFILL_SPEC=(--speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":${PREFILL_SPEC_TOKENS}}") - DECODE_MTP=(--with-mtp) + # decode_server only accepts depth 3 today, but pass it explicitly so the + # converter's --num_mtp, the golden-curve key and the engine depth agree by + # data flow rather than by coincidence of defaults. + DECODE_MTP=(--with-mtp --num-mtp "$DECODE_MTP_SIZE") fi TILERT_IS_AGENTIC=0 @@ -173,6 +176,17 @@ _tilert_weights_cached() { for r in $(seq 0 $((DECODE_TP_SIZE - 1))); do [[ -f "$TILERT_WEIGHTS_DIR/rank${r}/model.safetensors.index.json" ]] || return 1 done + # The engine refuses a cache converted without the MTP module (end2end.py + # checks tilert_meta.json num_mtp), but only after loading ~90 GiB of + # weights. Check the same field here so a stale non-MTP cache is + # re-converted instead of failing late. + [[ -f "$TILERT_WEIGHTS_DIR/tilert_meta.json" ]] || return 1 + if [[ "$SPEC_DECODING" == "mtp" ]]; then + "$PY" - "$TILERT_WEIGHTS_DIR/tilert_meta.json" <<'PYEOF' || return 1 +import json, sys +sys.exit(0 if int(json.load(open(sys.argv[1])).get("num_mtp", 0)) >= 1 else 1) +PYEOF + fi return 0 } @@ -197,7 +211,7 @@ convert_weights() { if "$PY" -c "import tilert.models.${TILERT_MODEL_PKG}.weight_converter" 2>/dev/null; then conv_mod="tilert.models.${TILERT_MODEL_PKG}.weight_converter" conv_args=(--model_dir "$MODEL_PATH" --save_dir "$TILERT_WEIGHTS_DIR" - --device "${TILERT_CONVERT_DEVICE:-cuda:$((GPUS_PER_NODE - 1))}") + --device "cuda:$((GPUS_PER_NODE - 1))") [[ "$SPEC_DECODING" == "mtp" ]] && conv_args+=(--num_mtp "$DECODE_MTP_SIZE") else conv_mod="tilert.models.preprocess.weight_converter" diff --git a/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh b/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh index 79201c4759..f5f6b331bf 100644 --- a/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh +++ b/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh @@ -52,8 +52,22 @@ export TILERT_PROFILE=glm5_2 # decode_server --model (TileRT model prof export TILERT_MODEL_TYPE=glm-5 # weight_converter --model_type (fallback converter) export TILERT_MODEL_PKG=glm_5_2_rocm # per-model converter package, preferred when importable export SERVED_MODEL_NAME=glm5_2 -# GLM-5.3's full context window, as every in-tree GLM-5.2 recipe uses. -# (202752 is GLM-5.1's, inherited from the B200 TileRT recipe this mirrors.) +# GLM-5.3's full context window (config.json max_position_embeddings), as every +# in-tree GLM-5.2 recipe serves. (202752 was GLM-5.1's, inherited from the B200 +# TileRT recipe this mirrors.) +# +# Memory at this context, per rank, bf16 wire layout (verified against the +# tilert 0.1.6 and vLLM 0.24.0 sources and the MI355X logs, 287.98 GiB cards): +# decode : weights 90.72 GiB + engine cache window 93.25 GiB +# + PD receive buffer 99.06 GiB (receive_server.py, dense in max_seq_len) +# prefill: weights 90.45 GiB + profiling/non-torch 40.3 GiB + vLLM KV 91.71 GiB +# + PD staging buffer 99.06 GiB (prefill_connector.py, TP rank 0, +# allocated OUTSIDE vLLM's gpu-memory-utilization budget) +# Both PD buffers must live in pinned host memory (2.9 TB RAM per node) for +# this context to start; on the GPU the decode side is node-marginal (~283 of +# 288 GiB) and the prefill side cannot fit at any utilization (~321 GiB). +# TileRT 0.1.6 places both on the GPU; bump TILERT_VERSION and the two images +# to the release that moves them to DRAM. Until then this recipe cannot start. export TILERT_MAX_MODEL_LEN=1048576 export TILERT_TRANSPORT=mooncake export TILERT_PARSER=none @@ -61,21 +75,25 @@ export TILERT_RDMA_STRICT=0 export TILERT_CONVERT_LOCK_WAIT=21600 export TILERT_SIMULATE_ACC_METHOD=match-expected export TILERT_WEIGHTS_DIR="/models/${MODEL_NAME}-tilert-tp${DECODE_TP}" -# fp8 MLA KV on both roles, which is what makes the full 1M context fit. -# TileRT's MlaNsaProfile.configure() maps fp8_ds_mla/fp8/fp8_e4m3 to the same -# mla_fp8 layout, and vLLM's ROCM_AITER_MLA_SPARSE backend lists fp8 in its -# supported_kv_cache_dtypes. Only fp8_ds_mla is CUDA-only, so plain fp8 gives -# both ranks the matching layout TileRT requires. -# At bf16 (KV_BYTES_BF16 = 1024 B/token) decode needed a 99.06 GB buffer on top -# of 184.17 GiB of weights on a 287.98 GiB card and OOM-killed; vLLM prefill -# refused outright, wanting 91.71 GiB of KV against 85.25 GiB available. -# fp8 (KV_BYTES_FP8 = 528 B/token) roughly halves both. -export PREFILL_KV_DTYPE=fp8 +# bf16 MLA KV on both roles. This is the only layout TileRT 0.1.6 can consume +# from vLLM on ROCm: MlaNsaProfile.classify_layers infers the layout from the +# cache tensor stride and accepts exactly 1152 B/token (bf16) or 656 B/token +# (fp8_ds_mla). vLLM's ROCM_AITER_MLA_SPARSE backend has no fp8_ds_mla; its +# plain "fp8" writes a flat 576 B/token row, which the connector rejects at +# register_kv_caches. Explicit bfloat16 rather than auto so the stride does not +# depend on the model dtype. Never float16: it passes the 1152 B check and is +# then read as bf16. +export PREFILL_KV_DTYPE=bfloat16 # The ROCm backend supports block sizes [1, 64] and vLLM picks 1, which makes # the connector's KI plane copy fail and MLA address the wrong rows. export PREFILL_BLOCK_SIZE=64 -export DECODE_KV_DTYPE=fp8 -export GPU_MEM_UTIL=0.75 +export DECODE_KV_DTYPE=bf16 +# With the PD staging buffer in host memory, vLLM needs 90.45 (weights) + 40.3 +# (profiling) + 91.71 GiB (KV for one 1048576-token request) = 222.5 GiB inside +# its budget: 0.85 x 287.98 = 244.8 GiB leaves 22 GiB of KV margin and 43 GiB +# outside the budget for the ~6 GiB non-torch baseline. 0.75 (216 GiB) refuses +# with "91.71 GiB KV cache is needed ... available 85.25 GiB". +export GPU_MEM_UTIL=0.85 export SKIP_CONTAINER_BARRIER=0 # Two images, one per rank, ~32 GB each. On a node that has neither cached the # pull alone outlasts the SGLang path's 300s default and the 1800s this script diff --git a/perf-changelog.yaml b/perf-changelog.yaml index b87433cd01..7c3a449d73 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8577,9 +8577,9 @@ - glm5.3-fp8-mi355x-tilert description: - "Add GLM-5.3 FP8 MI355X prefill/decode-disaggregated Single-turn 8k1k benchmark via TileRT: upstream vLLM ROCm prefill (TileRTConnector, kv_producer) + TileRT decode_server + OpenAI-compatible pd_router, framework=tilert, mirroring the B200 glm5.1-fp8-b200-tilert entry." - - "Two images per job, both first-party and neither carrying the TileRT wheel: the decode rank runs ghcr.io/tile-ai/tilert-rocm-decode:0.1.6 (ROCm PyTorch runtime plus the mooncake ROCm transfer engine built from source) and the prefill rank runs ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6 (the upstream vllm/vllm-openai-rocm release image plus that same transfer engine, with vLLM itself unpatched and no file inside it modified), selected per rank via PREFILL_IMAGE in prefill.additional-settings. setup_deps.sh pip-installs the official tilert==0.1.6 wheel and the router HTTP deps into both ranks at container start (full install on decode, --no-deps connector plugin on prefill), so neither image is pinned to a wheel version; this mirrors the B200 glm5.1-fp8-b200-tilert pattern of a first-party TileRT decode image. The transfer engine is built from source because the published mooncake ROCm wheels are compiled without ENABLE_MULTI_PROTOCOL, which leaves the cross-host locality gate (kvcache-ai/Mooncake#2753) out of the binary so every cross-node KV transfer is treated as node-local and fails in hipIpcOpenMemHandle." - - "KV moves P->D over mooncake (kv-p2p-transfer: mooncake); both roles use an fp8 MLA KV cache (--kv-cache-dtype fp8 on each). fp8_ds_mla is CUDA-only, but vLLM's ROCM_AITER_MLA_SPARSE backend lists fp8 among its supported_kv_cache_dtypes and TileRT's MlaNsaProfile maps fp8_ds_mla/fp8/fp8_e4m3 onto one mla_fp8 layout, so plain fp8 gives both ranks the matching layout TileRT requires. bf16 does not fit at this context: decode needed a 99.06 GB KV buffer on top of 184.17 GiB of weights on a 287.98 GiB card, and vLLM prefill refused with 91.71 GiB of KV needed against 85.25 GiB available. max-model-len 1048576 on both roles (GLM-5.3's full context window); MTP wired via spec-decoding=mtp: TileRT decode drafts at depth 3 (DECODE_MTP_SIZE, the only depth the ROCm GLM profile builds, and the key the golden acceptance curve is read at), while the vLLM prefill rank only materialises the MTP layer's KV and runs speculative-config mtp with 1 draft token. TileRT decode is bs=1 only, so conc-list is [1]; topology 1 prefill node (TP8) + 1 decode node (TP8)." - - "Orchestration: runners/launch_mi355x-amds.sh dispatches framework=tilert through the existing amd_utils chain (submit.sh -> job.slurm -> server.sh -> new server_tilert.sh); job.slurm selects the image per rank and a tilert env block, setup_deps.sh gains a per-role tilert install branch, models_tilert.yaml holds the GLM-5.3 profile defaults. TileRT decode weights are converted once from the HF checkpoint under a flock and cached next to it on the shared model volume. Cross-node teardown uses the amd_utils router-port-close barrier." + - "Two images per job, both first-party and neither carrying the TileRT wheel: the decode rank runs ghcr.io/tile-ai/tilert-rocm-decode:0.1.6 (ROCm PyTorch runtime plus the mooncake ROCm transfer engine built from source) and the prefill rank runs ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6 (the upstream vllm/vllm-openai-rocm release image plus that same transfer engine, with vLLM itself unpatched and no file inside it modified), selected per rank via PREFILL_IMAGE in prefill.additional-settings. setup_deps.sh pip-installs the official tilert==0.1.6 wheel into both ranks at container start (full install plus the fastapi/uvicorn/httpx router deps on decode, --no-deps connector plugin on prefill, whose vLLM image already ships those), so neither image is pinned to a wheel version; this mirrors the B200 glm5.1-fp8-b200-tilert pattern of a first-party TileRT decode image. The transfer engine is built from source because the published mooncake ROCm wheels are compiled without ENABLE_MULTI_PROTOCOL, which leaves the cross-host locality gate (kvcache-ai/Mooncake#2753) out of the binary so every cross-node KV transfer is treated as node-local and fails in hipIpcOpenMemHandle." + - "KV moves P->D over mooncake (kv-p2p-transfer: mooncake); both roles use a bf16 MLA KV cache (prefill --kv-cache-dtype bfloat16, decode bf16). This is the only layout TileRT 0.1.6 can consume from vLLM on ROCm: TileRT infers the layout from the cache tensor stride and accepts 1152 B/token (bf16) or 656 B/token (fp8_ds_mla); vLLM's ROCM_AITER_MLA_SPARSE backend has no fp8_ds_mla and its plain fp8 writes a flat 576 B/token row that the connector rejects at register_kv_caches. max-model-len 1048576 on both roles (GLM-5.3's full context window); MTP wired via spec-decoding=mtp: TileRT decode drafts at depth 3 (DECODE_MTP_SIZE, the only depth the ROCm GLM profile builds, and the key the golden acceptance curve is read at), while the vLLM prefill rank only materialises the MTP layer's KV and runs speculative-config mtp with 1 draft token. TileRT decode is bs=1 only, so conc-list is [1]; topology 1 prefill node (TP8) + 1 decode node (TP8)." + - "Orchestration: runners/launch_mi355x-amds.sh dispatches framework=tilert through the existing amd_utils chain (submit.sh -> job.slurm -> server.sh -> new server_tilert.sh); job.slurm selects the image per rank and a tilert env block, setup_deps.sh gains a per-role tilert install branch, models_tilert.yaml holds the GLM-5.3 prefill environment (the VLLM_ROCM_USE_AITER* block) and extra-flag hooks. TileRT decode weights are converted once from the HF checkpoint under a flock and cached next to it on the shared model volume. Cross-node teardown uses the amd_utils router-port-close barrier." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3330 - config-keys: @@ -8588,7 +8588,7 @@ - agentic-coding description: - "Add GLM-5.3 FP8 MI355X AgentX (agentic-coding) via TileRT prefill/decode disaggregation with MTP, 1 prefill node (TP8) + 1 decode node (TP8), concurrency [1], mirroring the B200 glm5.1-fp8-b200-tilert-agentic entry on the amd_utils orchestration." - - "Same two-image recipe as glm5.3-fp8-mi355x-tilert (decode ghcr.io/tile-ai/tilert-rocm-decode:0.1.6, prefill ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6, tilert==0.1.6 pip-installed into both at container start), fp8 MLA KV cache on both roles, kv-p2p-transfer: mooncake, kv-offloading none." + - "Same two-image recipe as glm5.3-fp8-mi355x-tilert (decode ghcr.io/tile-ai/tilert-rocm-decode:0.1.6, prefill ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6, tilert==0.1.6 pip-installed into both at container start), bf16 MLA KV cache on both roles, kv-p2p-transfer: mooncake, kv-offloading none." - "The pd_router runs with --parser none and a 1800-second queue timeout so requests wait on the bs=1 decode engine instead of failing fast; AgentX traces are filtered at the 1048576-token full GLM context both roles are launched with; AIPerf scrapes only the vLLM prefill Prometheus endpoint (TileRT decode exposes no /metrics route)." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3330 @@ -8612,6 +8612,6 @@ - glm5.3-fp8-mi355x-tilert - glm5.3-fp8-mi355x-tilert-agentic description: - - "Serve the full 1048576-token GLM-5.3 context with an fp8 MLA KV cache on both TileRT ranks, and take the container-creation barrier timeout from CONTAINER_BARRIER_TIMEOUT (5400s) instead of a hardcoded 1800s. At bf16 the 1M context did not fit: the decode rank needed a 99.06 GB KV buffer on top of 184.17 GiB of weights on a 287.98 GiB MI355X and was OOM-killed, while vLLM prefill refused to start, needing 91.71 GiB of KV against the 85.25 GiB available at gpu-memory-utilization 0.75. fp8 halves both (KV_BYTES_FP8 528 B/token vs KV_BYTES_BF16 1024 B/token). The barrier change covers the cold pull of this recipe's two ~32 GB per-rank images, which outlasts the old timeout on a node that has cached neither." - - "以 fp8 MLA KV cache 在两侧 TileRT rank 上提供完整的 1048576 上下文，并将容器创建屏障超时改为由 CONTAINER_BARRIER_TIMEOUT（5400 秒）提供，不再硬编码 1800 秒。bf16 下 1M 上下文无法容纳：decode 侧在 287.98 GiB 的 MI355X 上需在 184.17 GiB 权重之外再分配 99.06 GB KV 缓冲而被 OOM 杀死，vLLM prefill 则因需要 91.71 GiB KV 而可用仅 85.25 GiB 直接拒绝启动。fp8 将两者减半（每 token 528 字节对 1024 字节）。屏障改动用于覆盖本配方两个每 rank 约 32 GB 镜像的冷拉取，在两者均未缓存的节点上会超出原超时。" + - "Serve GLM-5.3's full 1048576-token context on both TileRT ranks with the bf16 MLA wire layout, gpu-memory-utilization 0.85 on the vLLM prefill, and the container-creation barrier timeout taken from CONTAINER_BARRIER_TIMEOUT (5400s) instead of a hardcoded 1800s. bf16 is the only layout TileRT 0.1.6 can consume from vLLM on ROCm: TileRT infers the layout from the cache tensor stride and accepts 1152 B/token (bf16) or 656 B/token (fp8_ds_mla), while ROCM_AITER_MLA_SPARSE has no fp8_ds_mla and its plain fp8 writes a flat 576 B/token row that the connector rejects at register_kv_caches. Per rank at this context (287.98 GiB MI355X): decode holds 90.72 GiB of weights, a 93.25 GiB engine cache window and a 99.06 GiB PD receive buffer; prefill holds 90.45 GiB of weights, 40.3 GiB of profiling/non-torch memory, 91.71 GiB of vLLM KV and a 99.06 GiB PD staging buffer on TP rank 0 outside vLLM's budget. Both PD buffers are dense in max_seq_len and must live in pinned host memory for this context to start: on the GPU the decode side is node-marginal (torch.OutOfMemoryError on one node, 99.06 GiB requested with 95.94 GiB free; served on another) and the prefill side cannot fit at any utilization. Startup at this context is therefore gated on the TileRT release that places the PD buffers in DRAM; TILERT_VERSION and both images are bumped to it in a follow-up. The barrier change covers the cold pull of this recipe's two ~32 GB per-rank images, which took about 25 minutes of the old 30-minute budget on an uncached node." + - "以 bf16 MLA 线上布局在两侧 TileRT rank 提供 GLM-5.3 完整的 1048576 上下文，vLLM prefill 的 gpu-memory-utilization 设为 0.85，容器创建屏障超时改为由 CONTAINER_BARRIER_TIMEOUT（5400 秒）提供而非硬编码 1800 秒。bf16 是 TileRT 0.1.6 在 ROCm 上唯一能从 vLLM 读取的布局：TileRT 依据缓存张量步长推断布局，只接受每 token 1152 字节（bf16）或 656 字节（fp8_ds_mla），而 ROCM_AITER_MLA_SPARSE 没有 fp8_ds_mla，其普通 fp8 写出每 token 576 字节的扁平行，会在 register_kv_caches 被 connector 拒绝。该上下文下每 rank（287.98 GiB 的 MI355X）：decode 侧含 90.72 GiB 权重、93.25 GiB 引擎缓存窗口和 99.06 GiB 的 PD 接收缓冲；prefill 侧含 90.45 GiB 权重、40.3 GiB 分析/非 torch 内存、91.71 GiB vLLM KV，以及 TP rank 0 上位于 vLLM 预算之外的 99.06 GiB PD 暂存缓冲。两个 PD 缓冲均按 max_seq_len 密集分配，必须放在锁页主机内存中该上下文才能启动：放在 GPU 上时 decode 侧处于节点边缘（一节点 OOM，请求 99.06 GiB 而仅剩 95.94 GiB；另一节点可服务），prefill 侧在任何利用率下都放不下。因此该上下文的启动取决于将 PD 缓冲放入 DRAM 的 TileRT 版本，TILERT_VERSION 与两个镜像将在后续 PR 中升级。屏障改动用于覆盖本配方两个每 rank 约 32 GB 镜像的冷拉取，在未缓存节点上实测约占旧 30 分钟预算的 25 分钟。" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3330 From 3e61d51f7d326209c89b4d11a1e99f0a2b86c23e Mon Sep 17 00:00:00 2001 From: CrimsonDump Date: Tue, 22 Sep 2026 14:11:17 +0800 Subject: [PATCH 11/25] =?UTF-8?q?feat(tilert):=20PD=20buffers=20in=202=20M?= =?UTF-8?q?iB=20pinned=20DRAM=20for=20the=201M=20context,=20tilert=200.1.6?= =?UTF-8?q?=20patch=20+=20waiver=203330=20/=20PD=20=E7=BC=93=E5=86=B2?= =?UTF-8?q?=E7=A7=BB=E5=85=A5=202=20MiB=20=E5=A4=A7=E9=A1=B5=E9=94=81?= =?UTF-8?q?=E9=A1=B5=E5=86=85=E5=AD=98=E4=BB=A5=E6=94=AF=E6=8C=81=201M=20?= =?UTF-8?q?=E4=B8=8A=E4=B8=8B=E6=96=87=EF=BC=8C=E9=99=84=20tilert=200.1.6?= =?UTF-8?q?=20=E8=A1=A5=E4=B8=81=E4=B8=8E=E8=B1=81=E5=85=8D?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit TileRT 0.1.6 keeps its two PD buffers on the GPU, dense in max_seq_len: 99.06 GiB each at 1048576 tokens. With 90.72 GiB of weights and the 93.25 GiB engine cache window the decode rank sits at ~283 of 287.98 GiB (node-marginal, OOM on g14), and prefill rank 0 needs ~321 GiB (weights 90.45 + profiling 40.29 + vLLM KV 91.71 inside the budget + the 99.06 GiB staging buffer outside it). No --kv-cache-dtype shrinks them on ROCm: TileRT accepts only the 1152 B (bf16) or 656 B (fp8_ds_mla) stride and vLLM's ROCM_AITER_MLA_SPARSE has no fp8_ds_mla. setup_deps.sh now applies patches/tilert-0.1.6-pd-buffers-in-dram.patch to the pip-installed tilert 0.1.6 on both ranks when TILERT_PD_BUFFER_DEVICE=cpu (marker-gated, pinned to 0.1.6, checked before the wheel download). The patch (5 pure-Python files under tilert/pd_vllm) adds decode_server --pd-buffer-device and connector extra config tilert_pd_buffer_device, and allocates both buffers with alloc_pinned_huge(): a 2 MiB-aligned mmap with MADV_HUGEPAGE, populated via MADV_POPULATE_WRITE, verified from /proc/self/smaps to be fully huge-page backed, hipHostRegister'ed, wrapped as a pinned torch tensor; convert() moves each received plane to the device. The verification matters: the Pensando ionic RDMA VFs cap 4 KiB-page ibv_reg_mr at 3968 MiB per HCA (~2^20 entries), while 2 MiB-backed 100 GiB regions register in one MR and took a cross-node mooncake write at 20.9 GiB/s (measured on g09/g11). Registering in <4 GiB pieces does not help; the budget is per HCA, not per MR. server_tilert.sh validates TILERT_PD_BUFFER_DEVICE in {cpu,cuda} up front and passes --pd-buffer-device only for cpu, so cuda runs the shipped wheel (which has no such flag) unmodified. job.slurm forwards and validates the knob; both recipes set cpu and GPU_MEM_UTIL 0.85 (222.5 GiB inside a 244.8 GiB budget). vLLM is not patched. Engine-patch waiver: docs/waiver/3330.md; the upstream TileRT link is pending. TileRT 0.1.6 将两个按 max_seq_len 密集分配的 PD 缓冲放在 GPU 上，1048576 上下文下 各 99.06 GiB；连同 90.72 GiB 权重与 93.25 GiB 引擎缓存窗口，decode 侧约 283/287.98 GiB（节点边缘，g14 OOM），prefill rank 0 约需 321 GiB。ROCm 上无法用 --kv-cache-dtype 缩减：TileRT 只接受 1152（bf16）或 656（fp8_ds_mla）字节步长，而 vLLM 的 ROCM_AITER_MLA_SPARSE 没有 fp8_ds_mla。 现由 setup_deps.sh 在 TILERT_PD_BUFFER_DEVICE=cpu 时对两侧 pip 安装的 tilert 0.1.6 应用 patches/tilert-0.1.6-pd-buffers-in-dram.patch（有标记门控、固定 0.1.6，在下载 wheel 前检查）。补丁新增 decode_server --pd-buffer-device 与 connector 配置 tilert_pd_buffer_device，用 alloc_pinned_huge() 分配两个缓冲：2 MiB 对齐的 mmap + MADV_HUGEPAGE，经 MADV_POPULATE_WRITE 预填充，依 /proc/self/smaps 校验全部为大页， hipHostRegister 后包装为 pinned torch 张量；convert() 将接收到的各层数据搬到设备。 校验的必要性：ionic RDMA VF 对 4 KiB 页的 ibv_reg_mr 上限为每 HCA 3968 MiB（约 2^20 条页表项），而 2 MiB 大页的 100 GiB 区域可作单个 MR 注册，并以 20.9 GiB/s 接收跨节点 mooncake 写入（g09/g11 实测）；分成小于 4 GiB 的块无济于事，预算按 HCA 而非按 MR。 server_tilert.sh 预先校验 TILERT_PD_BUFFER_DEVICE 取值，仅在 cpu 时传递 --pd-buffer-device，cuda 下原样运行 wheel。job.slurm 转发并校验该变量；两份配方设为 cpu 并将 GPU_MEM_UTIL 设为 0.85。vLLM 未打补丁。引擎补丁豁免见 docs/waiver/3330.md， 上游 TileRT 链接待补。 --- .../agentic/glm5.3_fp8_mi355x_tilert.sh | 16 +- benchmarks/multi_node/amd_utils/job.slurm | 3 +- .../tilert-0.1.6-pd-buffers-in-dram.patch | 244 ++++++++++++++++++ .../multi_node/amd_utils/server_tilert.sh | 23 +- benchmarks/multi_node/amd_utils/setup_deps.sh | 47 +++- .../multi_node/glm5.3_fp8_mi355x_tilert.sh | 16 +- docs/waiver/3330.md | 85 ++++++ perf-changelog.yaml | 4 +- 8 files changed, 415 insertions(+), 23 deletions(-) create mode 100644 benchmarks/multi_node/amd_utils/patches/tilert-0.1.6-pd-buffers-in-dram.patch create mode 100644 docs/waiver/3330.md diff --git a/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh b/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh index c7701ed642..2565c046d4 100644 --- a/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh +++ b/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh @@ -77,12 +77,19 @@ export SERVED_MODEL_NAME=glm5_2 # prefill: weights 90.45 GiB + profiling/non-torch 40.3 GiB + vLLM KV 91.71 GiB # + PD staging buffer 99.06 GiB (prefill_connector.py, TP rank 0, # allocated OUTSIDE vLLM's gpu-memory-utilization budget) -# Both PD buffers must live in pinned host memory (2.9 TB RAM per node) for +# Both PD buffers must live in pinned host memory (~3.2 TB RAM per node) for # this context to start; on the GPU the decode side is node-marginal (~283 of # 288 GiB) and the prefill side cannot fit at any utilization (~321 GiB). -# TileRT 0.1.6 places both on the GPU; bump TILERT_VERSION and the two images -# to the release that moves them to DRAM. Until then this recipe cannot start. +# TileRT 0.1.6 places both on the GPU, so setup_deps.sh applies +# patches/tilert-0.1.6-pd-buffers-in-dram.patch at container start (engine-patch +# waiver docs/waiver/3330.md) and TILERT_PD_BUFFER_DEVICE=cpu selects the host +# buffers. They are 2 MiB-backed and verified as such: the ionic RDMA VFs cap +# 4 KiB-page registrations at ~3.9 GiB per HCA. Cost: two extra PCIe copies on +# the KV path (~0.83 GB each at 8k tokens, estimated ~17 ms apiece; the RDMA +# hop itself measured 20.9 GiB/s). Drop the patch and this knob when a TileRT +# release carries DRAM PD buffers. export TILERT_MAX_MODEL_LEN=1048576 +export TILERT_PD_BUFFER_DEVICE=cpu export TILERT_TRANSPORT=mooncake export TILERT_PARSER=none export TILERT_RDMA_STRICT=0 @@ -105,7 +112,8 @@ export DECODE_KV_DTYPE=bf16 # With the PD staging buffer in host memory, vLLM needs 90.45 (weights) + 40.3 # (profiling) + 91.71 GiB (KV for one 1048576-token request) = 222.5 GiB inside # its budget: 0.85 x 287.98 = 244.8 GiB leaves 22 GiB of KV margin and 43 GiB -# outside the budget for the ~6 GiB non-torch baseline. 0.75 (216 GiB) refuses +# outside the budget for the ~6.3 GiB non-torch baseline measured on the decode +# OOM node (287.98 - 95.94 free - 184.17 - 1.58 reserved). 0.75 (216 GiB) refuses # with "91.71 GiB KV cache is needed ... available 85.25 GiB". export GPU_MEM_UTIL=0.85 export SKIP_CONTAINER_BARRIER=0 diff --git a/benchmarks/multi_node/amd_utils/job.slurm b/benchmarks/multi_node/amd_utils/job.slurm index 0c779476ea..324da80f3e 100755 --- a/benchmarks/multi_node/amd_utils/job.slurm +++ b/benchmarks/multi_node/amd_utils/job.slurm @@ -62,7 +62,7 @@ if [[ "$ENGINE" == "tilert" ]]; then # server_tilert.sh takes the container-creation barrier timeout from the # recipe (no 300s default as on the SGLang path); fail here, before sbatch # work is done, rather than inside the container. - check_env_vars CONTAINER_BARRIER_TIMEOUT + check_env_vars CONTAINER_BARRIER_TIMEOUT TILERT_PD_BUFFER_DEVICE fi # Resolve the models.yaml entry the same way server_sglang.sh does: agentic runs @@ -551,6 +551,7 @@ elif [[ "$ENGINE" == "tilert" ]]; then -e TILERT_RDMA_STRICT=${TILERT_RDMA_STRICT} -e TILERT_CONVERT_LOCK_WAIT=${TILERT_CONVERT_LOCK_WAIT} -e TILERT_SIMULATE_ACC_METHOD=${TILERT_SIMULATE_ACC_METHOD} + -e TILERT_PD_BUFFER_DEVICE=${TILERT_PD_BUFFER_DEVICE} -e \"TILERT_EXTRA_ENV=${TILERT_EXTRA_ENV:-}\" -e SERVED_MODEL_NAME=${SERVED_MODEL_NAME} -e PREFILL_KV_DTYPE=${PREFILL_KV_DTYPE} diff --git a/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6-pd-buffers-in-dram.patch b/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6-pd-buffers-in-dram.patch new file mode 100644 index 0000000000..7583adff2a --- /dev/null +++ b/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6-pd-buffers-in-dram.patch @@ -0,0 +1,244 @@ +diff -ru -x __pycache__ orig/decode_server.py new/decode_server.py +--- a/decode_server.py 2026-09-22 10:06:55 ++++ b/decode_server.py 2026-09-22 10:08:38 +@@ -309,6 +309,7 @@ + ap.add_argument('--num-mtp', type=int, choices=(3,), default=3, help='MTP draft depth for speculative decoding; needs --with-mtp') + ap.add_argument('--transport', choices=['mooncake', 'nixl'], default='mooncake', help="RDMA data-plane backend (must match prefill's tilert_transport)") + ap.add_argument('--kv-cache-dtype', default='fp8_ds_mla', help='MLA cache dtype (must match vLLM prefill); MLA-family profiles only') ++ ap.add_argument('--pd-buffer-device', choices=['cuda', 'cpu'], default='cuda', help="where the PD receive buffer lives; 'cpu' = pinned host memory registered with the transport (frees buffer_bytes(max_seq_len) of VRAM, adds one H2D copy per request)") + return ap + + def main() -> None: +@@ -329,7 +330,7 @@ + logger.info('loading TileRT engine (profile=%s, weights=%s)...', profile.name, args.model_weights_dir) + engine = profile.build_engine(model_weights_dir=args.model_weights_dir, max_seq_len=args.max_seq_len, with_mtp=args.with_mtp, ar_steps=8, num_mtp=num_mtp) + logger.info('TileRT engine ready (cache window %d, num_mtp %d)', engine.max_seq_len, num_mtp) +- server = ReceiveServer(profile, max_seq_len=args.max_seq_len, ctrl_port=args.ctrl_port, transport=args.transport) ++ server = ReceiveServer(profile, max_seq_len=args.max_seq_len, ctrl_port=args.ctrl_port, transport=args.transport, buffer_device='cpu' if args.pd_buffer_device == 'cpu' else 'cuda:0') + app = build_app(server, engine) + logger.info('decode server on :%d (profile=%s, engine=%s, ctrl=:%d)', args.http_port, profile.name, args.engine, args.ctrl_port) + sock = socket.socket(socket.AF_INET6, socket.SOCK_STREAM) +diff -ru -x __pycache__ orig/prefill_connector.py new/prefill_connector.py +--- a/prefill_connector.py 2026-09-22 10:06:55 ++++ b/prefill_connector.py 2026-09-22 14:09:20 +@@ -49,6 +49,7 @@ + self._max_seq = int(extra.get('tilert_max_seq_len', vllm_config.model_config.max_model_len)) + self._profile = profiles.get_profile(extra.get('tilert_model', 'glm5')) + self._transport_name = extra.get('tilert_transport', 'mooncake') ++ self._pd_buffer_device = str(extra.get('tilert_pd_buffer_device', 'cuda')) + self._pending: dict[str, _Pending] = {} + self._kv_caches: dict = {} + self._reg = None +@@ -145,10 +146,19 @@ + hostname = wire.local_ip() + total = self._profile.staging_bytes(self._reg, self._tp_rank, self._max_seq) + dev = torch.cuda.current_device() +- self._staging = torch.zeros(total, dtype=torch.uint8, device=f'cuda:{dev}') ++ # Dense in max_seq_len (99.06 GiB at 1M for glm5_2 bf16) and allocated ++ # outside vLLM's gpu-memory-utilization budget. In pinned host memory ++ # when tilert_pd_buffer_device == 'cpu'; extract() already fills it with ++ # copy_(), which performs the D2H transfer. ++ host = self._pd_buffer_device == 'cpu' ++ if host: ++ from tilert.pd_vllm.transport import alloc_pinned_huge ++ self._staging = alloc_pinned_huge(total) ++ else: ++ self._staging = torch.zeros(total, dtype=torch.uint8, device=f'cuda:{dev}') + self._transport = make_transport(self._transport_name) + self._transport.init(hostname) +- self._transport.register(self._staging.data_ptr(), total, dev) ++ self._transport.register(self._staging.data_ptr(), total, dev, host=host) + self._sender_thread = threading.Thread(target=self._sender_loop, name='tilert-pd-sender', daemon=True) + self._sender_thread.start() + logger.info('worker ready: rank=%d transport=%s staging=%.1f MB profile=%s', self._tp_rank, self._transport.name, total / 1000000.0, self._profile.name) +diff -ru -x __pycache__ orig/profiles/mla_nsa.py new/profiles/mla_nsa.py +--- a/profiles/mla_nsa.py 2026-09-22 10:06:55 ++++ b/profiles/mla_nsa.py 2026-09-22 10:07:21 +@@ -125,14 +125,23 @@ + for lid in range(self.num_layers): + ko = lid * max_seq_len * kv_bpt + kv_raw = buffer[ko:ko + seq * kv_bpt].view(seq, kv_bpt) ++ # A host-resident receive buffer is moved to the device plane by ++ # plane; the copy scales with the request's seq_len, not max_seq_len. ++ if not buffer.is_cuda: ++ kv_raw = kv_raw.to('cuda:0', non_blocking=True) + if self.mla_fp8: + kv = self._dequant_kv(kv_raw, seq) + else: + kv = kv_raw.view(torch.bfloat16).view(seq, KV_LORA_RANK).contiguous() + po = pe_base + lid * max_seq_len * PE_BPT +- pe = buffer[po:po + seq * PE_BPT].view(torch.bfloat16).view(seq, QK_ROPE_HEAD_DIM).contiguous() ++ pe = buffer[po:po + seq * PE_BPT] ++ if not buffer.is_cuda: ++ pe = pe.to('cuda:0', non_blocking=True) ++ pe = pe.view(torch.bfloat16).view(seq, QK_ROPE_HEAD_DIM).contiguous() + io = ki_base + lid * _max_pages(max_seq_len) * KI_PAGE_BYTES + ki_raw = buffer[io:io + npages * KI_PAGE_BYTES].view(npages, KI_PAGE_BYTES) ++ if not buffer.is_cuda: ++ ki_raw = ki_raw.to('cuda:0', non_blocking=True) + layers.append((self._dequant_ki(ki_raw, seq, self.ki_tiled), kv, pe)) + torch.cuda.synchronize() + return ConvertedRequest(rid=received.rid, seq_len=seq, last_prompt_token=received.last_prompt_token, first_token_id=received.first_token_id, sampling=received.sampling, layers=layers) +diff -ru -x __pycache__ orig/receive_server.py new/receive_server.py +--- a/receive_server.py 2026-09-22 10:06:55 ++++ b/receive_server.py 2026-09-22 14:09:20 +@@ -34,15 +34,26 @@ + + class ReceiveServer: + +- def __init__(self, profile, max_seq_len: int, ctrl_port: int=5556, hostname: str | None=None, device: str='cuda:0', request_timeout: float=120.0, transport: str='mooncake'): ++ def __init__(self, profile, max_seq_len: int, ctrl_port: int=5556, hostname: str | None=None, device: str='cuda:0', request_timeout: float=120.0, transport: str='mooncake', buffer_device: str='cuda:0'): + self.profile = profile + self.max_seq_len = max_seq_len + self.ctrl_port = ctrl_port + self.device = device + self.request_timeout = request_timeout + total = profile.buffer_bytes(max_seq_len) +- logger.info('allocating receive buffer: %.2f GB on %s (profile=%s)', total / 1024 ** 3, device, profile.name) +- self.buffer = torch.zeros(total, dtype=torch.uint8, device=device) ++ # Dense in max_seq_len: 99.06 GiB at 1M tokens for glm5_2 bf16. Held in ++ # pinned host memory when buffer_device == 'cpu' so it does not compete ++ # with the weights and the engine cache window on the GPU; convert() ++ # then moves each request's planes to the device (H2D scales with the ++ # prompt, not with max_seq_len). ++ self.buffer_device = buffer_device ++ logger.info('allocating receive buffer: %.2f GiB on %s (profile=%s)', total / 1024 ** 3, buffer_device, profile.name) ++ if buffer_device == 'cpu': ++ from tilert.pd_vllm.transport import alloc_pinned_huge ++ self.buffer = alloc_pinned_huge(total) ++ else: ++ self.buffer = torch.zeros(total, dtype=torch.uint8, device=buffer_device) ++ logger.info('receive buffer ready: %.2f GiB on %s', total / 1024 ** 3, self.buffer.device) + self.base_ptr = self.buffer.data_ptr() + self._hello_layout = profile.hello_layout(self.base_ptr, max_seq_len) + from tilert.pd_vllm.transport import make_transport +@@ -51,7 +62,7 @@ + dev_id = torch.device(device).index or 0 + self._transport = make_transport(transport) + self._transport.init(hostname) +- self._transport.register(self.base_ptr, total, dev_id) ++ self._transport.register(self.base_ptr, total, dev_id, host=(buffer_device == 'cpu')) + self._transport_meta = self._transport.local_meta() + logger.info('transport=%s ready, buffer registered (%.2f GB)', self._transport.name, total / 1024 ** 3) + self._lock = threading.Lock() +diff -ru -x __pycache__ orig/transport.py new/transport.py +--- a/transport.py 2026-09-22 10:06:55 ++++ b/transport.py 2026-09-22 14:09:20 +@@ -8,7 +8,7 @@ + def init(self, host: str) -> None: + ... + +- def register(self, ptr: int, nbytes: int, dev_id: int) -> None: ++ def register(self, ptr: int, nbytes: int, dev_id: int, host: bool=False) -> None: + ... + + def local_meta(self) -> dict: +@@ -28,7 +28,8 @@ + raise RuntimeError(f'Mooncake engine init failed: {ret}') + self.session_id = f'{host}:{self.engine.get_rpc_port()}' + +- def register(self, ptr: int, nbytes: int, dev_id: int) -> None: ++ def register(self, ptr: int, nbytes: int, dev_id: int, host: bool=False) -> None: ++ # Mooncake registers host (pinned) and device pointers through the same call. + ret = self.engine.batch_register_memory([ptr], [nbytes]) + if ret != 0: + raise RuntimeError(f'Mooncake register failed: {ret}') +@@ -50,10 +51,12 @@ + self._agent = nixl_agent(f'{host}:{os.getpid()}', nixl_agent_config(backends=['UCX'])) + self._remotes: dict[bytes, str] = {} + self._dev = 0 ++ self._mem_type = 'VRAM' + +- def register(self, ptr: int, nbytes: int, dev_id: int) -> None: +- self._dev = dev_id +- self._agent.register_memory([(ptr, nbytes, dev_id, '')], 'VRAM') ++ def register(self, ptr: int, nbytes: int, dev_id: int, host: bool=False) -> None: ++ self._dev = 0 if host else dev_id ++ self._mem_type = 'DRAM' if host else 'VRAM' ++ self._agent.register_memory([(ptr, nbytes, self._dev, '')], self._mem_type) + + def local_meta(self) -> dict: + return {'nixl_meta': base64.b64encode(self._agent.get_agent_metadata()).decode(), 'nixl_dev': self._dev} +@@ -65,8 +68,8 @@ + rname = self._agent.add_remote_agent(meta_b) + self._remotes[meta_b] = rname + rdev = int(remote_meta.get('nixl_dev', 0)) +- ld = self._agent.get_xfer_descs([(int(s), int(n), self._dev) for s, n in zip(srcs, lens)], 'VRAM') +- rd = self._agent.get_xfer_descs([(int(d), int(n), rdev) for d, n in zip(dsts, lens)], 'VRAM') ++ ld = self._agent.get_xfer_descs([(int(s), int(n), self._dev) for s, n in zip(srcs, lens)], self._mem_type) ++ rd = self._agent.get_xfer_descs([(int(d), int(n), rdev) for d, n in zip(dsts, lens)], self._mem_type) + h = self._agent.initialize_xfer('WRITE', ld, rd, rname) + try: + st = self._agent.transfer(h) +@@ -86,4 +89,69 @@ + key = (name or 'mooncake').lower() + if key not in _BACKENDS: + raise ValueError(f'unknown transport {name!r}; choices: {sorted(_BACKENDS)}') +- return _BACKENDS[key]() +\ No newline at end of file ++ return _BACKENDS[key]() ++ ++def alloc_pinned_huge(total: int): ++ """Host buffer for the PD data plane, backed end-to-end by 2 MiB pages. ++ ++ The ionic RDMA VFs on the MI355X pool have a ~2**20 page-table-entry budget ++ per HCA: with 4 KiB pages ibv_reg_mr fails (EINVAL/ENOMEM) once ~3.9 GiB is ++ registered, while 2 MiB-backed regions register tens of GiB. One 4 KiB-backed ++ slice anywhere in the region drops the whole MR to 4 KiB granularity, so the ++ mapping is 2 MiB-aligned and the backing is verified from /proc/self/smaps ++ before the memory is handed out. The region is then hipHostRegister'ed so ++ the GPU can DMA into it and torch reports it pinned. ++ """ ++ import ctypes, mmap, re ++ import torch ++ MiB = 1 << 20 ++ HUGE = 2 * MiB ++ if total % HUGE: ++ total += HUGE - total % HUGE ++ # Over-allocate by one huge page so the used range can be aligned on kernels ++ # that do not PMD-align anonymous mappings. ++ mm = mmap.mmap(-1, total + HUGE, flags=mmap.MAP_PRIVATE | mmap.MAP_ANONYMOUS) ++ raw = ctypes.addressof(ctypes.c_char.from_buffer(mm)) ++ off = (-raw) % HUGE ++ addr = raw + off ++ libc = ctypes.CDLL(None, use_errno=True) ++ libc.madvise.argtypes = [ctypes.c_void_p, ctypes.c_size_t, ctypes.c_int] ++ if libc.madvise(ctypes.c_void_p(addr), ctypes.c_size_t(total), ctypes.c_int(14)) != 0: # MADV_HUGEPAGE ++ raise RuntimeError(f'madvise(MADV_HUGEPAGE) failed errno={ctypes.get_errno()}') ++ # Fault the huge pages in up front. MADV_POPULATE_WRITE (Linux >= 5.14) does ++ # it in one call; touching one byte per 2 MiB from Python takes ~150 s for ++ # 100 GiB, so it is only the fallback. ++ if libc.madvise(ctypes.c_void_p(addr), ctypes.c_size_t(total), ctypes.c_int(23)) != 0: # MADV_POPULATE_WRITE ++ for o in range(off, off + total, HUGE): ++ mm[o] = 0 ++ huge = 0 ++ for blk in re.split(r'\n(?=[0-9a-f]+-[0-9a-f]+ )', open('/proc/self/smaps').read()): ++ m = re.match(r'([0-9a-f]+)-([0-9a-f]+) ', blk) ++ if not m: ++ continue ++ lo, hi = int(m.group(1), 16), int(m.group(2), 16) ++ if hi <= addr or lo >= addr + total: ++ continue ++ h = re.search(r'AnonHugePages:\s+(\d+) kB', blk) ++ huge += int(h.group(1)) * 1024 if h else 0 ++ if huge < total: ++ raise RuntimeError(f'PD host buffer is only {huge / 2**30:.2f} of {total / 2**30:.2f} GiB huge-page backed; ' ++ 'the RDMA MR would fall back to 4 KiB pages and exceed the per-HCA entry budget ' ++ '(check /sys/kernel/mm/transparent_hugepage/{enabled,defrag} and free memory)') ++ hip = None ++ for lib in ('/opt/rocm/lib/libamdhip64.so', 'libamdhip64.so', 'libcudart.so'): ++ try: ++ hip = ctypes.CDLL(lib) ++ break ++ except OSError: ++ continue ++ if hip is None: ++ raise RuntimeError('neither libamdhip64 nor libcudart could be loaded; the PD host buffer cannot be GPU-registered') ++ fn = hip.hipHostRegister if hasattr(hip, 'hipHostRegister') else hip.cudaHostRegister ++ fn.argtypes = [ctypes.c_void_p, ctypes.c_size_t, ctypes.c_uint] ++ rc = fn(ctypes.c_void_p(addr), ctypes.c_size_t(total), ctypes.c_uint(0)) ++ if rc != 0: ++ raise RuntimeError(f'hipHostRegister failed rc={rc}') ++ buf = torch.frombuffer(mm, dtype=torch.uint8, count=total, offset=off) ++ buf._pd_mmap = mm # keep the mapping alive with the tensor ++ return buf diff --git a/benchmarks/multi_node/amd_utils/server_tilert.sh b/benchmarks/multi_node/amd_utils/server_tilert.sh index 1e46867c3e..d03940fb6c 100644 --- a/benchmarks/multi_node/amd_utils/server_tilert.sh +++ b/benchmarks/multi_node/amd_utils/server_tilert.sh @@ -20,7 +20,15 @@ check_env_vars \ DECODE_MTP_SIZE GPU_MEM_UTIL SERVED_MODEL_NAME \ DECODE_CTRL_PORT DECODE_HTTP_PORT PREFILL_PORT ROUTER_PORT \ DECODE_WAIT PREFILL_WAIT ROUTER_WAIT SKIP_CONTAINER_BARRIER \ - CONTAINER_BARRIER_TIMEOUT + CONTAINER_BARRIER_TIMEOUT TILERT_PD_BUFFER_DEVICE + +# TILERT_PD_BUFFER_DEVICE selects the patched host-buffer path (cpu) or the +# shipped wheel (cuda); anything else would only be rejected by decode_server's +# argparse after weight conversion. +case "$TILERT_PD_BUFFER_DEVICE" in + cpu|cuda) ;; + *) echo "ERROR: TILERT_PD_BUFFER_DEVICE must be 'cpu' or 'cuda' (got '$TILERT_PD_BUFFER_DEVICE')" >&2; exit 1 ;; +esac LOG_DIR="/run_logs/slurm_job-${SLURM_JOB_ID}" SHARED_LOG_DIR="${BENCHMARK_LOGS_DIR}/logs/slurm_job-${SLURM_JOB_ID}" @@ -93,7 +101,7 @@ host_name=$(hostname) echo "[tilert] ROLE=$TILERT_ROLE rank=$NODE_RANK host=$host_name ($host_ip)" echo "[tilert] PREFILL_HOST=$PREFILL_HOST:$PREFILL_PORT DECODE_HOST=$DECODE_HOST:$DECODE_CTRL_PORT/$DECODE_HTTP_PORT ROUTER=:$ROUTER_PORT" -echo "[tilert] MODEL_PATH=$MODEL_PATH profile=$TILERT_PROFILE served=$SERVED_MODEL_NAME max_len=$TILERT_MAX_MODEL_LEN transport=$TILERT_TRANSPORT kv=${PREFILL_KV_DTYPE}->${DECODE_KV_DTYPE} mtp=${SPEC_DECODING} agentic=$TILERT_IS_AGENTIC" +echo "[tilert] MODEL_PATH=$MODEL_PATH profile=$TILERT_PROFILE served=$SERVED_MODEL_NAME max_len=$TILERT_MAX_MODEL_LEN transport=$TILERT_TRANSPORT kv=${PREFILL_KV_DTYPE}->${DECODE_KV_DTYPE} mtp=${SPEC_DECODING} pd_buffers=$TILERT_PD_BUFFER_DEVICE agentic=$TILERT_IS_AGENTIC" # Enable libibverbs fork safety on both ranks before any verbs context exists. # Without it, ibv_fork_init() can fail in these containers while Mooncake @@ -268,13 +276,18 @@ PYEOF unset TILERT_SIMULATE_ACC_LEN TILERT_SIMULATE_ACC_METHOD echo "[decode] real MTP verification (no simulated acceptance)" fi + # The shipped 0.1.6 decode_server has no --pd-buffer-device (strict + # parse_args); setup_deps.sh applies the PD-DRAM patch only for cpu, so + # pass the flag only then and let cuda run the wheel exactly as shipped. + local pd_buf=() + [[ "$TILERT_PD_BUFFER_DEVICE" == "cpu" ]] && pd_buf=(--pd-buffer-device cpu) local cmd=("$PY" -m tilert.pd_vllm.decode_server --engine tilert --model "$TILERT_PROFILE" --model-weights-dir "$TILERT_WEIGHTS_DIR" --max-seq-len "$TILERT_MAX_MODEL_LEN" --kv-cache-dtype "$DECODE_KV_DTYPE" --transport "$TILERT_TRANSPORT" --ctrl-port "$DECODE_CTRL_PORT" --http-port "$DECODE_HTTP_PORT" - "${DECODE_MTP[@]}" "${extra[@]}") + "${pd_buf[@]}" "${DECODE_MTP[@]}" "${extra[@]}") log_and_run_bg decode "$LOG_DIR/decode_${host_name}.log" "${cmd[@]}" DECODE_PID=$LAST_BG_PID } @@ -289,8 +302,8 @@ start_prefill() { # shellcheck disable=SC2206 local extra=( ${TILERT_PREFILL_EXTRA_FLAGS} ) local kv_cfg - kv_cfg=$(printf '{"kv_connector":"TileRTConnector","kv_connector_module_path":"tilert.pd_vllm.prefill_connector","kv_role":"kv_producer","kv_connector_extra_config":{"tilert_host":"%s","tilert_ctrl_port":%s,"tilert_model":"%s","tilert_max_seq_len":%s,"tilert_transport":"%s"}}' \ - "$DECODE_HOST" "$DECODE_CTRL_PORT" "$TILERT_PROFILE" "$TILERT_MAX_MODEL_LEN" "$TILERT_TRANSPORT") + kv_cfg=$(printf '{"kv_connector":"TileRTConnector","kv_connector_module_path":"tilert.pd_vllm.prefill_connector","kv_role":"kv_producer","kv_connector_extra_config":{"tilert_host":"%s","tilert_ctrl_port":%s,"tilert_model":"%s","tilert_max_seq_len":%s,"tilert_transport":"%s","tilert_pd_buffer_device":"%s"}}' \ + "$DECODE_HOST" "$DECODE_CTRL_PORT" "$TILERT_PROFILE" "$TILERT_MAX_MODEL_LEN" "$TILERT_TRANSPORT" "$TILERT_PD_BUFFER_DEVICE") local cmd=(vllm serve "$MODEL_PATH" --served-model-name "${served[@]}" --port "$PREFILL_PORT" --tensor-parallel-size "$PREFILL_TP_SIZE" --max-model-len "$TILERT_MAX_MODEL_LEN" diff --git a/benchmarks/multi_node/amd_utils/setup_deps.sh b/benchmarks/multi_node/amd_utils/setup_deps.sh index 8adb438681..aa36989afc 100644 --- a/benchmarks/multi_node/amd_utils/setup_deps.sh +++ b/benchmarks/multi_node/amd_utils/setup_deps.sh @@ -92,18 +92,18 @@ _tilert_install_missing() { install_tilert_container_tools() { if command -v ip >/dev/null 2>&1 && command -v curl >/dev/null 2>&1 \ - && command -v ibv_devices >/dev/null 2>&1; then + && command -v ibv_devices >/dev/null 2>&1 && command -v patch >/dev/null 2>&1; then echo "[SETUP] Container RDMA/net tools already present" return 0 fi - echo "[SETUP] Installing iproute2 + curl + ibverbs userspace in container..." + echo "[SETUP] Installing iproute2 + curl + patch + ibverbs userspace in container..." apt-get update -q -y && apt-get install -q -y --no-install-recommends \ - iproute2 curl ibverbs-utils libibverbs1 librdmacm1 ibverbs-providers \ + iproute2 curl patch ibverbs-utils libibverbs1 librdmacm1 ibverbs-providers \ && rm -rf /var/lib/apt/lists/* - if ! command -v ip >/dev/null 2>&1 || ! command -v curl >/dev/null 2>&1; then - echo "[SETUP] ERROR: failed to install iproute2/curl"; exit 1 + if ! command -v ip >/dev/null 2>&1 || ! command -v curl >/dev/null 2>&1 || ! command -v patch >/dev/null 2>&1; then + echo "[SETUP] ERROR: failed to install iproute2/curl/patch"; exit 1 fi - _SETUP_INSTALLED+=("iproute2+curl+ibverbs") + _SETUP_INSTALLED+=("iproute2+curl+patch+ibverbs") } _tilert_install_wheel() { @@ -127,9 +127,37 @@ _tilert_install_wheel() { _SETUP_INSTALLED+=("$TILERT_PACKAGE==$TILERT_VERSION($mode)") } +# PD buffers in DRAM. TileRT 0.1.6 allocates the mooncake receive buffer (decode) +# and the connector staging buffer (prefill TP rank 0) on the GPU, dense in +# max_seq_len: 99.06 GiB each at 1048576 tokens, which does not fit next to the +# weights and the engine cache window. The patch moves both into 2 MiB-backed +# pinned host memory behind decode_server --pd-buffer-device cpu and connector +# extra config tilert_pd_buffer_device, verifying the huge-page backing because +# the ionic RDMA VFs cap 4 KiB-page registrations at ~3.9 GiB per HCA. +# Engine-patch waiver: docs/waiver/3330.md. Applied only when the recipe asks +# for host buffers; with TILERT_PD_BUFFER_DEVICE=cuda the wheel runs as shipped. +_TILERT_PD_DRAM_PATCH="$(dirname "${BASH_SOURCE[0]}")/patches/tilert-0.1.6-pd-buffers-in-dram.patch" + +_tilert_apply_pd_dram_patch() { + [[ "$TILERT_PD_BUFFER_DEVICE" == "cpu" ]] || { echo "[SETUP] PD buffers on GPU (TILERT_PD_BUFFER_DEVICE=$TILERT_PD_BUFFER_DEVICE); tilert unpatched"; return 0; } + local dir + dir="$("$PY" -c 'import os, tilert.pd_vllm as m; print(os.path.dirname(m.__file__))')" || { echo "[SETUP] ERROR: cannot locate tilert.pd_vllm"; exit 1; } + if grep -q "def alloc_pinned_huge" "$dir/transport.py"; then + echo "[SETUP] tilert PD-buffers-in-DRAM patch already applied in $dir" + return 0 + fi + [[ -f "$_TILERT_PD_DRAM_PATCH" ]] || { echo "[SETUP] ERROR: missing $_TILERT_PD_DRAM_PATCH"; exit 1; } + echo "[SETUP] applying $(basename "$_TILERT_PD_DRAM_PATCH") to $dir (waiver docs/waiver/3330.md)" + patch -p1 -s -N -d "$dir" < "$_TILERT_PD_DRAM_PATCH" || { echo "[SETUP] ERROR: patch failed to apply"; exit 1; } + "$PY" -m py_compile "$dir"/*.py "$dir"/profiles/*.py || { echo "[SETUP] ERROR: patched tilert.pd_vllm does not compile"; exit 1; } + "$PY" -c 'from tilert.pd_vllm.transport import alloc_pinned_huge' || { echo "[SETUP] ERROR: alloc_pinned_huge missing after patch"; exit 1; } + _SETUP_INSTALLED+=("tilert-pd-buffers-in-dram.patch") +} + install_tilert_decode() { install_tilert_container_tools _tilert_install_wheel full + _tilert_apply_pd_dram_patch _tilert_install_missing uvicorn $TILERT_HTTP_DEPS _tilert_install_missing mooncake.engine "$TILERT_TRANSPORT_DEPS" _tilert_install_missing transformers "$TILERT_TRANSFORMERS_SPEC" @@ -149,6 +177,7 @@ install_tilert_prefill() { echo "[SETUP] prefill-side vLLM $vllm_v" install_tilert_container_tools _tilert_install_wheel no-deps + _tilert_apply_pd_dram_patch _tilert_install_missing mooncake.engine "$TILERT_TRANSPORT_DEPS" "$PY" -c "import tilert.pd_vllm.prefill_connector" 2>/dev/null || { echo "[SETUP] WARN: import tilert.pd_vllm.prefill_connector failed (vLLM will report again when loading the connector plugin):" @@ -165,7 +194,11 @@ if [[ "$ENGINE" == "vllm-disagg" ]]; then export PATH="${UCX_HOME}/bin:/usr/local/bin/etcd:/root/.cargo/bin:${PATH}" export LD_LIBRARY_PATH="${UCX_HOME}/lib:${RIXL_HOME}/lib:${RIXL_HOME}/lib/x86_64-linux-gnu:${LD_LIBRARY_PATH:-}" elif [[ "$ENGINE" == "tilert" ]]; then - check_env_vars TILERT_VERSION + check_env_vars TILERT_VERSION TILERT_PD_BUFFER_DEVICE + if [[ "$TILERT_PD_BUFFER_DEVICE" == "cpu" && "$TILERT_VERSION" != "0.1.6" ]]; then + echo "[SETUP] ERROR: $_TILERT_PD_DRAM_PATCH targets tilert 0.1.6, got $TILERT_VERSION; drop the patch, this check and TILERT_PD_BUFFER_DEVICE when bumping to a release with DRAM PD buffers" + exit 1 + fi TILERT_PIP_SPEC="$TILERT_PACKAGE==$TILERT_VERSION" _tilert_resolve_python case "${TILERT_ROLE:-}" in diff --git a/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh b/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh index f5f6b331bf..caba2e0bb6 100644 --- a/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh +++ b/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh @@ -63,12 +63,19 @@ export SERVED_MODEL_NAME=glm5_2 # prefill: weights 90.45 GiB + profiling/non-torch 40.3 GiB + vLLM KV 91.71 GiB # + PD staging buffer 99.06 GiB (prefill_connector.py, TP rank 0, # allocated OUTSIDE vLLM's gpu-memory-utilization budget) -# Both PD buffers must live in pinned host memory (2.9 TB RAM per node) for +# Both PD buffers must live in pinned host memory (~3.2 TB RAM per node) for # this context to start; on the GPU the decode side is node-marginal (~283 of # 288 GiB) and the prefill side cannot fit at any utilization (~321 GiB). -# TileRT 0.1.6 places both on the GPU; bump TILERT_VERSION and the two images -# to the release that moves them to DRAM. Until then this recipe cannot start. +# TileRT 0.1.6 places both on the GPU, so setup_deps.sh applies +# patches/tilert-0.1.6-pd-buffers-in-dram.patch at container start (engine-patch +# waiver docs/waiver/3330.md) and TILERT_PD_BUFFER_DEVICE=cpu selects the host +# buffers. They are 2 MiB-backed and verified as such: the ionic RDMA VFs cap +# 4 KiB-page registrations at ~3.9 GiB per HCA. Cost: two extra PCIe copies on +# the KV path (~0.83 GB each at 8k tokens, estimated ~17 ms apiece; the RDMA +# hop itself measured 20.9 GiB/s). Drop the patch and this knob when a TileRT +# release carries DRAM PD buffers. export TILERT_MAX_MODEL_LEN=1048576 +export TILERT_PD_BUFFER_DEVICE=cpu export TILERT_TRANSPORT=mooncake export TILERT_PARSER=none export TILERT_RDMA_STRICT=0 @@ -91,7 +98,8 @@ export DECODE_KV_DTYPE=bf16 # With the PD staging buffer in host memory, vLLM needs 90.45 (weights) + 40.3 # (profiling) + 91.71 GiB (KV for one 1048576-token request) = 222.5 GiB inside # its budget: 0.85 x 287.98 = 244.8 GiB leaves 22 GiB of KV margin and 43 GiB -# outside the budget for the ~6 GiB non-torch baseline. 0.75 (216 GiB) refuses +# outside the budget for the ~6.3 GiB non-torch baseline measured on the decode +# OOM node (287.98 - 95.94 free - 184.17 - 1.58 reserved). 0.75 (216 GiB) refuses # with "91.71 GiB KV cache is needed ... available 85.25 GiB". export GPU_MEM_UTIL=0.85 export SKIP_CONTAINER_BARRIER=0 diff --git a/docs/waiver/3330.md b/docs/waiver/3330.md new file mode 100644 index 0000000000..a94534fd16 --- /dev/null +++ b/docs/waiver/3330.md @@ -0,0 +1,85 @@ +# Inference-engine patch waiver — PR #3330 + +Filed per [`docs/PR_REVIEW_CHECKLIST.md`](../PR_REVIEW_CHECKLIST.md) (image-provenance / patch item): the +shared TileRT launcher in this PR patches the TileRT PD (prefill/decode) data plane — the pip-installed +`tilert==0.1.6` wheel's `pd_vllm` package (transport, receive server, decode CLI, prefill connector and the +MLA profile's `convert()`) — inside the containers at start, before serving, which the checklist prohibits unless covered by a filled-out waiver. +vLLM is **not** patched; it runs exactly as shipped in `ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6`. + +## Config covered + +- **Master config entries:** `glm5.3-fp8-mi355x-tilert` and `glm5.3-fp8-mi355x-tilert-agentic` in + [`configs/amd-master.yaml`](../../configs/amd-master.yaml) +- **Pinned images:** `ghcr.io/tile-ai/tilert-rocm-decode:0.1.6` (decode rank) and + `ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6` (prefill rank); neither carries the TileRT wheel, which + [`benchmarks/multi_node/amd_utils/setup_deps.sh`](../../benchmarks/multi_node/amd_utils/setup_deps.sh) + pip-installs as `tilert==0.1.6` from PyPI at container start. +- **Patch entrypoint:** `_tilert_apply_pd_dram_patch` in `setup_deps.sh`, called from + `install_tilert_decode` and `install_tilert_prefill`, only when the recipe sets + `TILERT_PD_BUFFER_DEVICE=cpu`. Marker-gated (skips if `alloc_pinned_huge` is already present) and + pinned to `TILERT_VERSION=0.1.6`; any other version fails loudly. +- **Patch file:** + [`benchmarks/multi_node/amd_utils/patches/tilert-0.1.6-pd-buffers-in-dram.patch`](../../benchmarks/multi_node/amd_utils/patches/tilert-0.1.6-pd-buffers-in-dram.patch) + (5 pure-Python files under `tilert/pd_vllm/`; no compiled component is touched). + +## What is patched + +| file | change | +|---|---| +| `pd_vllm/transport.py` | `Transport.register(..., host=False)` on the mooncake and nixl backends (nixl registers `DRAM` instead of `VRAM` when `host`); new `alloc_pinned_huge(total)` that allocates `mmap`+`MADV_HUGEPAGE` memory, populates it (`MADV_POPULATE_WRITE`), **verifies from `/proc/self/smaps` that the whole region is 2 MiB-backed and raises otherwise**, `hipHostRegister`s it (raising if no HIP/CUDA runtime library loads), and returns a pinned `torch` tensor over a 2 MiB-aligned mapping | +| `pd_vllm/receive_server.py` | `ReceiveServer(..., buffer_device='cuda:0')`; with `'cpu'` the mooncake receive buffer comes from `alloc_pinned_huge` instead of `torch.zeros(..., device='cuda:0')` | +| `pd_vllm/decode_server.py` | new flag `--pd-buffer-device {cuda,cpu}` (default `cuda`) forwarded to `ReceiveServer` | +| `pd_vllm/prefill_connector.py` | connector extra config `tilert_pd_buffer_device` (default `cuda`); with `cpu` the TP-rank-0 staging buffer comes from `alloc_pinned_huge` instead of `torch.zeros(..., device='cuda:N')` | +| `pd_vllm/profiles/mla_nsa.py` | `convert()` moves each received plane to the device when the receive buffer is host-resident (the copy scales with the request's `seq_len`, not `max_seq_len`) | + +Nothing changes in the wire layout, the MLA/KI dequantization, the engine, the router, or vLLM. With the +default `TILERT_PD_BUFFER_DEVICE=cuda` the patch is not applied and the wheel runs as shipped. + +## Why the unmodified upstream stack cannot run this benchmark + +The recipe serves GLM-5.3's full `max_position_embeddings` of 1,048,576 tokens. TileRT 0.1.6 +preallocates two PD buffers dense in `max_seq_len` on the GPU, `buffer_bytes = 79 layers × (1024 + 128 + +132) B/token = 99.06 GiB` each at that context (`profiles/mla_nsa.py`), and `--kv-cache-dtype` cannot +shrink them on ROCm: TileRT accepts only the 1152 B/token bf16 or 656 B/token `fp8_ds_mla` cache stride, +and vLLM's `ROCM_AITER_MLA_SPARSE` backend has no `fp8_ds_mla` (its plain `fp8` writes 576 B/token, which +`classify_layers` rejects). Measured per rank on 287.98 GiB MI355X (sweep jobs 45280/45281): + +- **decode:** weights 90.72 GiB + engine cache window 93.25 GiB + receive buffer 99.06 GiB ≈ 283 GiB — + `torch.OutOfMemoryError` on one node (99.06 GiB requested, 95.94 GiB free), served on another: + node-marginal. +- **prefill rank 0:** weights 90.45 GiB + profiling/non-torch 40.29 GiB + vLLM KV 91.71 GiB inside the + `gpu-memory-utilization` budget, plus the 99.06 GiB staging buffer **outside** it = 321.5 GiB — cannot + fit at any utilization (at utilization 0.75 vLLM refused: "91.71 GiB KV cache is needed ... available + 85.25 GiB"; at 0.85 the KV fits and the 99.06 GiB staging allocation is what fails). + +With both PD buffers in host memory the decode rank needs ≈ 192 GiB (the measured pre-buffer footprint on +the OOM node, 287.98 − 95.94 free) and prefill needs 222.5 GiB inside a 0.85 × 287.98 = 244.8 GiB budget. +Known ceiling, unchanged from the shipped code: `convert()` materialises every layer's planes on the GPU +until `inject()`, so a single prompt near the full 1M tokens (~108 GiB of transients) would still OOM the +decode rank; the AgentX corpus and the 8k1k scenario are far below that. The host buffers must be 2 MiB-backed: the Pensando ionic RDMA VFs on +this pool exhaust a ~2²⁰ page-table-entry budget per HCA, so `ibv_reg_mr` of 4 KiB-page host memory fails +past 3968 MiB (EINVAL/ENOMEM), while 2 MiB-backed regions of 100 GiB register in one MR; this was +measured with raw `libibverbs` and confirmed end to end with a cross-node mooncake host→host write of +8 GiB at 20.9 GiB/s into a 100 GiB registered host buffer (data verified). Registering in <4 GiB pieces +does not help — the budget is per HCA, not per MR. Hence the `smaps` verification in `alloc_pinned_huge`. + +## Upstream PR / issue links + +- TileRT upstream: https://github.com/tile-ai/TileRT — the TileRT team is implementing DRAM-resident PD + buffers; **upstream PR pending** (to be linked here when opened). The patch as applied, with the + measurements above, is posted on this PR: + https://github.com/SemiAnalysisAI/InferenceX/pull/3330#issuecomment-5770977844 and + https://github.com/SemiAnalysisAI/InferenceX/pull/3330#issuecomment-5771627506. +- vLLM: no change requested; `fp8_ds_mla` support in the ROCm sparse-MLA backend would be an alternative + route (halving the wire layout) but is not required for this waiver. + +## Removal plan + +When a TileRT release carries DRAM-resident PD buffers (with the huge-page verification or an +equivalent), in one PR: bump `TILERT_VERSION` and both images in the two recipes to it and replace +`TILERT_PD_BUFFER_DEVICE` with the release's own knob if it differs; delete +`patches/tilert-0.1.6-pd-buffers-in-dram.patch`, `_TILERT_PD_DRAM_PATCH`, `_tilert_apply_pd_dram_patch`, +its two call sites and the `TILERT_VERSION == 0.1.6` pin check in `setup_deps.sh` (the `patch` package +in `install_tilert_container_tools` can go too); drop the `--pd-buffer-device` flag, the +`tilert_pd_buffer_device` connector key and the value check from `server_tilert.sh`, and the +`TILERT_PD_BUFFER_DEVICE` forwarding/validation from `job.slurm`; and delete this waiver. diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 2a1ca058ad..b1fd5529d2 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8627,6 +8627,6 @@ - glm5.3-fp8-mi355x-tilert - glm5.3-fp8-mi355x-tilert-agentic description: - - "Serve GLM-5.3's full 1048576-token context on both TileRT ranks with the bf16 MLA wire layout, gpu-memory-utilization 0.85 on the vLLM prefill, and the container-creation barrier timeout taken from CONTAINER_BARRIER_TIMEOUT (5400s) instead of a hardcoded 1800s. bf16 is the only layout TileRT 0.1.6 can consume from vLLM on ROCm: TileRT infers the layout from the cache tensor stride and accepts 1152 B/token (bf16) or 656 B/token (fp8_ds_mla), while ROCM_AITER_MLA_SPARSE has no fp8_ds_mla and its plain fp8 writes a flat 576 B/token row that the connector rejects at register_kv_caches. Per rank at this context (287.98 GiB MI355X): decode holds 90.72 GiB of weights, a 93.25 GiB engine cache window and a 99.06 GiB PD receive buffer; prefill holds 90.45 GiB of weights, 40.3 GiB of profiling/non-torch memory, 91.71 GiB of vLLM KV and a 99.06 GiB PD staging buffer on TP rank 0 outside vLLM's budget. Both PD buffers are dense in max_seq_len and must live in pinned host memory for this context to start: on the GPU the decode side is node-marginal (torch.OutOfMemoryError on one node, 99.06 GiB requested with 95.94 GiB free; served on another) and the prefill side cannot fit at any utilization. Startup at this context is therefore gated on the TileRT release that places the PD buffers in DRAM; TILERT_VERSION and both images are bumped to it in a follow-up. The barrier change covers the cold pull of this recipe's two ~32 GB per-rank images, which took about 25 minutes of the old 30-minute budget on an uncached node." - - "以 bf16 MLA 线上布局在两侧 TileRT rank 提供 GLM-5.3 完整的 1048576 上下文，vLLM prefill 的 gpu-memory-utilization 设为 0.85，容器创建屏障超时改为由 CONTAINER_BARRIER_TIMEOUT（5400 秒）提供而非硬编码 1800 秒。bf16 是 TileRT 0.1.6 在 ROCm 上唯一能从 vLLM 读取的布局：TileRT 依据缓存张量步长推断布局，只接受每 token 1152 字节（bf16）或 656 字节（fp8_ds_mla），而 ROCM_AITER_MLA_SPARSE 没有 fp8_ds_mla，其普通 fp8 写出每 token 576 字节的扁平行，会在 register_kv_caches 被 connector 拒绝。该上下文下每 rank（287.98 GiB 的 MI355X）：decode 侧含 90.72 GiB 权重、93.25 GiB 引擎缓存窗口和 99.06 GiB 的 PD 接收缓冲；prefill 侧含 90.45 GiB 权重、40.3 GiB 分析/非 torch 内存、91.71 GiB vLLM KV，以及 TP rank 0 上位于 vLLM 预算之外的 99.06 GiB PD 暂存缓冲。两个 PD 缓冲均按 max_seq_len 密集分配，必须放在锁页主机内存中该上下文才能启动：放在 GPU 上时 decode 侧处于节点边缘（一节点 OOM，请求 99.06 GiB 而仅剩 95.94 GiB；另一节点可服务），prefill 侧在任何利用率下都放不下。因此该上下文的启动取决于将 PD 缓冲放入 DRAM 的 TileRT 版本，TILERT_VERSION 与两个镜像将在后续 PR 中升级。屏障改动用于覆盖本配方两个每 rank 约 32 GB 镜像的冷拉取，在未缓存节点上实测约占旧 30 分钟预算的 25 分钟。" + - "Serve GLM-5.3's full 1048576-token context on both TileRT ranks with the bf16 MLA wire layout, gpu-memory-utilization 0.85 on the vLLM prefill, and the container-creation barrier timeout taken from CONTAINER_BARRIER_TIMEOUT (5400s) instead of a hardcoded 1800s. bf16 is the only layout TileRT 0.1.6 can consume from vLLM on ROCm: TileRT infers the layout from the cache tensor stride and accepts 1152 B/token (bf16) or 656 B/token (fp8_ds_mla), while ROCM_AITER_MLA_SPARSE has no fp8_ds_mla and its plain fp8 writes a flat 576 B/token row that the connector rejects at register_kv_caches. Per rank at this context (287.98 GiB MI355X): decode holds 90.72 GiB of weights, a 93.25 GiB engine cache window and a 99.06 GiB PD receive buffer; prefill holds 90.45 GiB of weights, 40.3 GiB of profiling/non-torch memory, 91.71 GiB of vLLM KV and a 99.06 GiB PD staging buffer on TP rank 0 outside vLLM's budget. Both PD buffers are dense in max_seq_len and must live in pinned host memory for this context to start: on the GPU the decode side is node-marginal (torch.OutOfMemoryError on one node, 99.06 GiB requested with 95.94 GiB free; served on another) and the prefill side cannot fit at any utilization. setup_deps.sh therefore applies patches/tilert-0.1.6-pd-buffers-in-dram.patch to the pip-installed tilert 0.1.6 at container start on both ranks (engine-patch waiver docs/waiver/3330.md; vLLM stays unpatched), placing both PD buffers in 2 MiB-backed pinned host memory behind TILERT_PD_BUFFER_DEVICE=cpu. The backing is verified before registration because the ionic RDMA VFs cap 4 KiB-page registrations at ~3.9 GiB per HCA (a ~2^20 page-table-entry budget), while 2 MiB-backed regions of 100 GiB register and take cross-node mooncake writes at ~21 GiB/s. The patch is removed when a TileRT release carries DRAM PD buffers. The barrier change covers cold image pulls: each rank pulls its own ~32 GB image, and the decode image alone took about 25 minutes of the old 30-minute budget on an uncached node." + - "以 bf16 MLA 线上布局在两侧 TileRT rank 提供 GLM-5.3 完整的 1048576 上下文，vLLM prefill 的 gpu-memory-utilization 设为 0.85，容器创建屏障超时改为由 CONTAINER_BARRIER_TIMEOUT（5400 秒）提供而非硬编码 1800 秒。bf16 是 TileRT 0.1.6 在 ROCm 上唯一能从 vLLM 读取的布局：TileRT 依据缓存张量步长推断布局，只接受每 token 1152 字节（bf16）或 656 字节（fp8_ds_mla），而 ROCM_AITER_MLA_SPARSE 没有 fp8_ds_mla，其普通 fp8 写出每 token 576 字节的扁平行，会在 register_kv_caches 被 connector 拒绝。该上下文下每 rank（287.98 GiB 的 MI355X）：decode 侧含 90.72 GiB 权重、93.25 GiB 引擎缓存窗口和 99.06 GiB 的 PD 接收缓冲；prefill 侧含 90.45 GiB 权重、40.3 GiB 分析/非 torch 内存、91.71 GiB vLLM KV，以及 TP rank 0 上位于 vLLM 预算之外的 99.06 GiB PD 暂存缓冲。两个 PD 缓冲均按 max_seq_len 密集分配，必须放在锁页主机内存中该上下文才能启动：放在 GPU 上时 decode 侧处于节点边缘（一节点 OOM，请求 99.06 GiB 而仅剩 95.94 GiB；另一节点可服务），prefill 侧在任何利用率下都放不下。因此 setup_deps.sh 在两侧容器启动时对 pip 安装的 tilert 0.1.6 应用 patches/tilert-0.1.6-pd-buffers-in-dram.patch（引擎补丁豁免 docs/waiver/3330.md；vLLM 未改动），在 TILERT_PD_BUFFER_DEVICE=cpu 下将两个 PD 缓冲放入 2 MiB 大页支撑的锁页主机内存。注册前会校验大页覆盖：ionic RDMA VF 对 4 KiB 页的注册上限约为每 HCA 3.9 GiB（约 2^20 条页表项），而 2 MiB 大页区域可注册 100 GiB 并以约 21 GiB/s 接收跨节点 mooncake 写入。TileRT 发布自带 DRAM PD 缓冲的版本后移除该补丁。屏障改动用于覆盖镜像冷拉取：每个 rank 各拉取约 32 GB 的镜像，仅 decode 镜像在未缓存节点上就实测约占旧 30 分钟预算的 25 分钟。" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3330 From 5485bcb3513412a5d9ffe7bfaa2de03986325592 Mon Sep 17 00:00:00 2001 From: CrimsonDump Date: Tue, 22 Sep 2026 15:04:40 +0800 Subject: [PATCH 12/25] =?UTF-8?q?fix(tilert):=20cap=20the=20barrier's=20po?= =?UTF-8?q?st-pass=20port-hold=20at=2060s=20/=20=E5=B1=8F=E9=9A=9C?= =?UTF-8?q?=E9=80=9A=E8=BF=87=E5=90=8E=E4=BB=85=E4=BF=9D=E6=8C=81=E7=AB=AF?= =?UTF-8?q?=E5=8F=A3=2060=20=E7=A7=92?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit sync.py --enable-port keeps the local port open for max(60, timeout // 2) seconds after the barrier passes, in the foreground, so a peer one poll behind still sees it. With the TileRT launcher's CONTAINER_BARRIER_TIMEOUT=5400 that became a 45-minute idle sleep on every rank: jobs 45373/45374 passed the barrier at 06:28 and sat in time.sleep until 07:13 with two nodes allocated (main thread in hrtimer_nanosleep, accept thread in inet_csk_accept). The old hardcoded 1800s hid a 15-minute version of the same sleep. sync.py gains --grace ; unset keeps the existing formula, so the SGLang/vLLM/ATOM arms are unchanged. server_tilert.sh passes --grace 60: both ranks pass within one 5 s poll of each other, and the prefill and decode stages have their own readiness waits (DECODE_WAIT, PREFILL_WAIT, ROUTER_WAIT). sync.py 的 --enable-port 在屏障通过后会在前台将本地端口保持 max(60, timeout // 2) 秒，以便晚一次轮询的对端仍能看到。TileRT 启动脚本的 CONTAINER_BARRIER_TIMEOUT=5400 使之变为每个 rank 空转 45 分钟：作业 45373/45374 于 06:28 通过屏障后一直 sleep 到 07:13，占用两个节点。原先硬编码的 1800 秒同样隐藏着 15 分钟的空转。 sync.py 新增 --grace <秒>；不传则沿用原公式，SGLang/vLLM/ATOM 各臂不受影响。 server_tilert.sh 传入 --grace 60：两侧在一次 5 秒轮询内相继通过，且后续 prefill/decode 阶段各有自己的就绪等待。 --- benchmarks/multi_node/amd_utils/server_tilert.sh | 8 +++++++- benchmarks/multi_node/amd_utils/sync.py | 9 ++++++++- perf-changelog.yaml | 4 ++-- 3 files changed, 17 insertions(+), 4 deletions(-) diff --git a/benchmarks/multi_node/amd_utils/server_tilert.sh b/benchmarks/multi_node/amd_utils/server_tilert.sh index d03940fb6c..0a273795f1 100644 --- a/benchmarks/multi_node/amd_utils/server_tilert.sh +++ b/benchmarks/multi_node/amd_utils/server_tilert.sh @@ -506,10 +506,16 @@ if [[ "$DRY_RUN" -eq 1 ]]; then elif [[ "$SKIP_CONTAINER_BARRIER" == "1" ]]; then echo "SKIP_CONTAINER_BARRIER=1: caller asserts all containers are up" else + # --grace 60: after the barrier passes, sync.py keeps the port open for + # max(60, timeout/2) seconds in the foreground so a peer one poll behind + # still sees it. At CONTAINER_BARRIER_TIMEOUT=5400 that is a 45-minute idle + # sleep on every rank (jobs 45373/45374 slept 06:28-07:13). Both ranks pass + # within one 5 s poll of each other and the stages below have their own + # readiness waits, so 60 s is plenty. "$PY" "$WS_PATH/sync.py" barrier \ --local-ip "${host_ip}" --local-port 5000 --enable-port \ --node-ips "${IPADDRS}" --node-ports 5000 \ - --wait-for-all-ports --timeout "$CONTAINER_BARRIER_TIMEOUT" \ + --wait-for-all-ports --timeout "$CONTAINER_BARRIER_TIMEOUT" --grace 60 \ || { echo "ERROR: container creation barrier failed after ${CONTAINER_BARRIER_TIMEOUT}s -- the peer rank never opened port 5000." \ "A cold image pull is the usual cause: this recipe pulls two ~32 GB images, one per rank, and the rank that" \ "comes up first waits out the whole timeout while the other is still pulling." >&2; exit 1; } diff --git a/benchmarks/multi_node/amd_utils/sync.py b/benchmarks/multi_node/amd_utils/sync.py index 96e94c1b08..fb618d2f51 100755 --- a/benchmarks/multi_node/amd_utils/sync.py +++ b/benchmarks/multi_node/amd_utils/sync.py @@ -161,7 +161,10 @@ def close_port(): if args.enable_port: # Keep the port open long enough for slow nodes to pass their barrier. # The previous 30s was too short when setup times vary by minutes. - grace = max(60, args.timeout // 2) if args.timeout > 0 else 300 + if args.grace is not None: + grace = args.grace + else: + grace = max(60, args.timeout // 2) if args.timeout > 0 else 300 time.sleep(grace) close_port() @@ -198,6 +201,10 @@ def main(): bp.add_argument("--node-ports", required=True, help="Comma-separated list of ports to check.") bp.add_argument("--timeout", type=int, default=600, help="Timeout in seconds (default: 600). Set to 0 for no timeout.") + bp.add_argument("--grace", type=int, default=None, + help="Seconds to keep the local port open after the barrier passes so peers one poll " + "behind still see it. Default max(60, timeout // 2). Callers whose later stages " + "have their own readiness waits can pass a small value.") bp.add_argument("--wait-for-all-ports", action="store_true", help="Wait until all node ports are open (TCP).") bp.add_argument("--wait-for-all-health", action="store_true", diff --git a/perf-changelog.yaml b/perf-changelog.yaml index b1fd5529d2..acc7a70c62 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8627,6 +8627,6 @@ - glm5.3-fp8-mi355x-tilert - glm5.3-fp8-mi355x-tilert-agentic description: - - "Serve GLM-5.3's full 1048576-token context on both TileRT ranks with the bf16 MLA wire layout, gpu-memory-utilization 0.85 on the vLLM prefill, and the container-creation barrier timeout taken from CONTAINER_BARRIER_TIMEOUT (5400s) instead of a hardcoded 1800s. bf16 is the only layout TileRT 0.1.6 can consume from vLLM on ROCm: TileRT infers the layout from the cache tensor stride and accepts 1152 B/token (bf16) or 656 B/token (fp8_ds_mla), while ROCM_AITER_MLA_SPARSE has no fp8_ds_mla and its plain fp8 writes a flat 576 B/token row that the connector rejects at register_kv_caches. Per rank at this context (287.98 GiB MI355X): decode holds 90.72 GiB of weights, a 93.25 GiB engine cache window and a 99.06 GiB PD receive buffer; prefill holds 90.45 GiB of weights, 40.3 GiB of profiling/non-torch memory, 91.71 GiB of vLLM KV and a 99.06 GiB PD staging buffer on TP rank 0 outside vLLM's budget. Both PD buffers are dense in max_seq_len and must live in pinned host memory for this context to start: on the GPU the decode side is node-marginal (torch.OutOfMemoryError on one node, 99.06 GiB requested with 95.94 GiB free; served on another) and the prefill side cannot fit at any utilization. setup_deps.sh therefore applies patches/tilert-0.1.6-pd-buffers-in-dram.patch to the pip-installed tilert 0.1.6 at container start on both ranks (engine-patch waiver docs/waiver/3330.md; vLLM stays unpatched), placing both PD buffers in 2 MiB-backed pinned host memory behind TILERT_PD_BUFFER_DEVICE=cpu. The backing is verified before registration because the ionic RDMA VFs cap 4 KiB-page registrations at ~3.9 GiB per HCA (a ~2^20 page-table-entry budget), while 2 MiB-backed regions of 100 GiB register and take cross-node mooncake writes at ~21 GiB/s. The patch is removed when a TileRT release carries DRAM PD buffers. The barrier change covers cold image pulls: each rank pulls its own ~32 GB image, and the decode image alone took about 25 minutes of the old 30-minute budget on an uncached node." - - "以 bf16 MLA 线上布局在两侧 TileRT rank 提供 GLM-5.3 完整的 1048576 上下文，vLLM prefill 的 gpu-memory-utilization 设为 0.85，容器创建屏障超时改为由 CONTAINER_BARRIER_TIMEOUT（5400 秒）提供而非硬编码 1800 秒。bf16 是 TileRT 0.1.6 在 ROCm 上唯一能从 vLLM 读取的布局：TileRT 依据缓存张量步长推断布局，只接受每 token 1152 字节（bf16）或 656 字节（fp8_ds_mla），而 ROCM_AITER_MLA_SPARSE 没有 fp8_ds_mla，其普通 fp8 写出每 token 576 字节的扁平行，会在 register_kv_caches 被 connector 拒绝。该上下文下每 rank（287.98 GiB 的 MI355X）：decode 侧含 90.72 GiB 权重、93.25 GiB 引擎缓存窗口和 99.06 GiB 的 PD 接收缓冲；prefill 侧含 90.45 GiB 权重、40.3 GiB 分析/非 torch 内存、91.71 GiB vLLM KV，以及 TP rank 0 上位于 vLLM 预算之外的 99.06 GiB PD 暂存缓冲。两个 PD 缓冲均按 max_seq_len 密集分配，必须放在锁页主机内存中该上下文才能启动：放在 GPU 上时 decode 侧处于节点边缘（一节点 OOM，请求 99.06 GiB 而仅剩 95.94 GiB；另一节点可服务），prefill 侧在任何利用率下都放不下。因此 setup_deps.sh 在两侧容器启动时对 pip 安装的 tilert 0.1.6 应用 patches/tilert-0.1.6-pd-buffers-in-dram.patch（引擎补丁豁免 docs/waiver/3330.md；vLLM 未改动），在 TILERT_PD_BUFFER_DEVICE=cpu 下将两个 PD 缓冲放入 2 MiB 大页支撑的锁页主机内存。注册前会校验大页覆盖：ionic RDMA VF 对 4 KiB 页的注册上限约为每 HCA 3.9 GiB（约 2^20 条页表项），而 2 MiB 大页区域可注册 100 GiB 并以约 21 GiB/s 接收跨节点 mooncake 写入。TileRT 发布自带 DRAM PD 缓冲的版本后移除该补丁。屏障改动用于覆盖镜像冷拉取：每个 rank 各拉取约 32 GB 的镜像，仅 decode 镜像在未缓存节点上就实测约占旧 30 分钟预算的 25 分钟。" + - "Serve GLM-5.3's full 1048576-token context on both TileRT ranks with the bf16 MLA wire layout, gpu-memory-utilization 0.85 on the vLLM prefill, and the container-creation barrier timeout taken from CONTAINER_BARRIER_TIMEOUT (5400s) instead of a hardcoded 1800s. bf16 is the only layout TileRT 0.1.6 can consume from vLLM on ROCm: TileRT infers the layout from the cache tensor stride and accepts 1152 B/token (bf16) or 656 B/token (fp8_ds_mla), while ROCM_AITER_MLA_SPARSE has no fp8_ds_mla and its plain fp8 writes a flat 576 B/token row that the connector rejects at register_kv_caches. Per rank at this context (287.98 GiB MI355X): decode holds 90.72 GiB of weights, a 93.25 GiB engine cache window and a 99.06 GiB PD receive buffer; prefill holds 90.45 GiB of weights, 40.3 GiB of profiling/non-torch memory, 91.71 GiB of vLLM KV and a 99.06 GiB PD staging buffer on TP rank 0 outside vLLM's budget. Both PD buffers are dense in max_seq_len and must live in pinned host memory for this context to start: on the GPU the decode side is node-marginal (torch.OutOfMemoryError on one node, 99.06 GiB requested with 95.94 GiB free; served on another) and the prefill side cannot fit at any utilization. setup_deps.sh therefore applies patches/tilert-0.1.6-pd-buffers-in-dram.patch to the pip-installed tilert 0.1.6 at container start on both ranks (engine-patch waiver docs/waiver/3330.md; vLLM stays unpatched), placing both PD buffers in 2 MiB-backed pinned host memory behind TILERT_PD_BUFFER_DEVICE=cpu. The backing is verified before registration because the ionic RDMA VFs cap 4 KiB-page registrations at ~3.9 GiB per HCA (a ~2^20 page-table-entry budget), while 2 MiB-backed regions of 100 GiB register and take cross-node mooncake writes at ~21 GiB/s. The patch is removed when a TileRT release carries DRAM PD buffers. The barrier change covers cold image pulls: each rank pulls its own ~32 GB image, and the decode image alone took about 25 minutes of the old 30-minute budget on an uncached node. The launcher also passes sync.py --grace 60: the barrier's post-pass port-hold defaults to max(60, timeout/2) seconds in the foreground, which at 5400s was a 45-minute idle sleep on every rank (jobs 45373/45374)." + - "以 bf16 MLA 线上布局在两侧 TileRT rank 提供 GLM-5.3 完整的 1048576 上下文，vLLM prefill 的 gpu-memory-utilization 设为 0.85，容器创建屏障超时改为由 CONTAINER_BARRIER_TIMEOUT（5400 秒）提供而非硬编码 1800 秒。bf16 是 TileRT 0.1.6 在 ROCm 上唯一能从 vLLM 读取的布局：TileRT 依据缓存张量步长推断布局，只接受每 token 1152 字节（bf16）或 656 字节（fp8_ds_mla），而 ROCM_AITER_MLA_SPARSE 没有 fp8_ds_mla，其普通 fp8 写出每 token 576 字节的扁平行，会在 register_kv_caches 被 connector 拒绝。该上下文下每 rank（287.98 GiB 的 MI355X）：decode 侧含 90.72 GiB 权重、93.25 GiB 引擎缓存窗口和 99.06 GiB 的 PD 接收缓冲；prefill 侧含 90.45 GiB 权重、40.3 GiB 分析/非 torch 内存、91.71 GiB vLLM KV，以及 TP rank 0 上位于 vLLM 预算之外的 99.06 GiB PD 暂存缓冲。两个 PD 缓冲均按 max_seq_len 密集分配，必须放在锁页主机内存中该上下文才能启动：放在 GPU 上时 decode 侧处于节点边缘（一节点 OOM，请求 99.06 GiB 而仅剩 95.94 GiB；另一节点可服务），prefill 侧在任何利用率下都放不下。因此 setup_deps.sh 在两侧容器启动时对 pip 安装的 tilert 0.1.6 应用 patches/tilert-0.1.6-pd-buffers-in-dram.patch（引擎补丁豁免 docs/waiver/3330.md；vLLM 未改动），在 TILERT_PD_BUFFER_DEVICE=cpu 下将两个 PD 缓冲放入 2 MiB 大页支撑的锁页主机内存。注册前会校验大页覆盖：ionic RDMA VF 对 4 KiB 页的注册上限约为每 HCA 3.9 GiB（约 2^20 条页表项），而 2 MiB 大页区域可注册 100 GiB 并以约 21 GiB/s 接收跨节点 mooncake 写入。TileRT 发布自带 DRAM PD 缓冲的版本后移除该补丁。屏障改动用于覆盖镜像冷拉取：每个 rank 各拉取约 32 GB 的镜像，仅 decode 镜像在未缓存节点上就实测约占旧 30 分钟预算的 25 分钟。启动脚本同时向 sync.py 传入 --grace 60：屏障通过后默认在前台保持端口 max(60, timeout/2) 秒，在 5400 秒下即每个 rank 空转 45 分钟（作业 45373/45374）。" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3330 From b01cf3c0ca21a129163fdfe0bb61c96d67aa1d54 Mon Sep 17 00:00:00 2001 From: CrimsonDump Date: Tue, 22 Sep 2026 15:17:46 +0800 Subject: [PATCH 13/25] =?UTF-8?q?fix(tilert):=20register=20the=20PD=20host?= =?UTF-8?q?=20buffer=20through=20torch.cuda.cudart=20/=20=E7=BB=8F=20torch?= =?UTF-8?q?.cuda.cudart=20=E6=B3=A8=E5=86=8C=20PD=20=E4=B8=BB=E6=9C=BA?= =?UTF-8?q?=E7=BC=93=E5=86=B2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sweep 35693738597 job 45373: the decode rank died in alloc_pinned_huge with "neither libamdhip64 nor libcudart could be loaded". The patch dlopen'ed /opt/rocm/lib/libamdhip64.so or libamdhip64.so by name, which exists in the prefill image (ROCm 7.2.3) but not in tilert-rocm-decode:0.1.6, whose torch 2.12+rocm7.14 wheel ships only the versioned libamdhip64.so.7 and no /opt/rocm tree. alloc_pinned_huge now registers the region with torch.cuda.cudart() .cudaHostRegister(addr, total, 0), torch's own runtime binding, hipified on ROCm and present on CUDA builds; verified in both images (cudaError.success, tensor.is_pinned() True). The ctypes library search is gone. Waiver text updated to match. sweep 35693738597 的作业 45373：decode 侧在 alloc_pinned_huge 中因 "neither libamdhip64 nor libcudart could be loaded" 退出。补丁按名字 dlopen /opt/rocm/lib/libamdhip64.so 或 libamdhip64.so，prefill 镜像（ROCm 7.2.3）有，而 tilert-rocm-decode:0.1.6 的 torch 2.12+rocm7.14 wheel 只带版本化的 libamdhip64.so.7， 且无 /opt/rocm 目录。 现改为通过 torch.cuda.cudart().cudaHostRegister(addr, total, 0) 注册，即 torch 自身 的运行时绑定，ROCm 上已 hipify、CUDA 构建亦有；两镜像均验证通过 （cudaError.success，is_pinned 为 True）。移除 ctypes 库查找，豁免文本同步更新。 --- .../tilert-0.1.6-pd-buffers-in-dram.patch | 26 +++++++------------ docs/waiver/3330.md | 2 +- 2 files changed, 11 insertions(+), 17 deletions(-) diff --git a/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6-pd-buffers-in-dram.patch b/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6-pd-buffers-in-dram.patch index 7583adff2a..b10b1a2c92 100644 --- a/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6-pd-buffers-in-dram.patch +++ b/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6-pd-buffers-in-dram.patch @@ -123,7 +123,7 @@ diff -ru -x __pycache__ orig/receive_server.py new/receive_server.py self._lock = threading.Lock() diff -ru -x __pycache__ orig/transport.py new/transport.py --- a/transport.py 2026-09-22 10:06:55 -+++ b/transport.py 2026-09-22 14:09:20 ++++ b/transport.py 2026-09-22 15:17:46 @@ -8,7 +8,7 @@ def init(self, host: str) -> None: ... @@ -170,7 +170,7 @@ diff -ru -x __pycache__ orig/transport.py new/transport.py h = self._agent.initialize_xfer('WRITE', ld, rd, rname) try: st = self._agent.transfer(h) -@@ -86,4 +89,69 @@ +@@ -86,4 +89,63 @@ key = (name or 'mooncake').lower() if key not in _BACKENDS: raise ValueError(f'unknown transport {name!r}; choices: {sorted(_BACKENDS)}') @@ -225,20 +225,14 @@ diff -ru -x __pycache__ orig/transport.py new/transport.py + raise RuntimeError(f'PD host buffer is only {huge / 2**30:.2f} of {total / 2**30:.2f} GiB huge-page backed; ' + 'the RDMA MR would fall back to 4 KiB pages and exceed the per-HCA entry budget ' + '(check /sys/kernel/mm/transparent_hugepage/{enabled,defrag} and free memory)') -+ hip = None -+ for lib in ('/opt/rocm/lib/libamdhip64.so', 'libamdhip64.so', 'libcudart.so'): -+ try: -+ hip = ctypes.CDLL(lib) -+ break -+ except OSError: -+ continue -+ if hip is None: -+ raise RuntimeError('neither libamdhip64 nor libcudart could be loaded; the PD host buffer cannot be GPU-registered') -+ fn = hip.hipHostRegister if hasattr(hip, 'hipHostRegister') else hip.cudaHostRegister -+ fn.argtypes = [ctypes.c_void_p, ctypes.c_size_t, ctypes.c_uint] -+ rc = fn(ctypes.c_void_p(addr), ctypes.c_size_t(total), ctypes.c_uint(0)) -+ if rc != 0: -+ raise RuntimeError(f'hipHostRegister failed rc={rc}') ++ # Register through torch's own GPU runtime binding rather than dlopen'ing a ++ # HIP/CUDA library by name: the tilert-rocm-decode image ships only the ++ # versioned libamdhip64.so.7, the prefill image both names, and CUDA builds ++ # neither. torch.cuda.cudart() is hipified on ROCm and exposes the same ++ # cudaHostRegister on every build. Returns a cudaError enum; 0 is success. ++ rc = torch.cuda.cudart().cudaHostRegister(addr, total, 0) ++ if int(rc) != 0: ++ raise RuntimeError(f'cudaHostRegister/hipHostRegister failed: {rc}') + buf = torch.frombuffer(mm, dtype=torch.uint8, count=total, offset=off) + buf._pd_mmap = mm # keep the mapping alive with the tensor + return buf diff --git a/docs/waiver/3330.md b/docs/waiver/3330.md index a94534fd16..7f288396bb 100644 --- a/docs/waiver/3330.md +++ b/docs/waiver/3330.md @@ -26,7 +26,7 @@ vLLM is **not** patched; it runs exactly as shipped in `ghcr.io/tile-ai/tilert-r | file | change | |---|---| -| `pd_vllm/transport.py` | `Transport.register(..., host=False)` on the mooncake and nixl backends (nixl registers `DRAM` instead of `VRAM` when `host`); new `alloc_pinned_huge(total)` that allocates `mmap`+`MADV_HUGEPAGE` memory, populates it (`MADV_POPULATE_WRITE`), **verifies from `/proc/self/smaps` that the whole region is 2 MiB-backed and raises otherwise**, `hipHostRegister`s it (raising if no HIP/CUDA runtime library loads), and returns a pinned `torch` tensor over a 2 MiB-aligned mapping | +| `pd_vllm/transport.py` | `Transport.register(..., host=False)` on the mooncake and nixl backends (nixl registers `DRAM` instead of `VRAM` when `host`); new `alloc_pinned_huge(total)` that allocates `mmap`+`MADV_HUGEPAGE` memory, populates it (`MADV_POPULATE_WRITE`), **verifies from `/proc/self/smaps` that the whole region is 2 MiB-backed and raises otherwise**, host-registers it with the GPU runtime through `torch.cuda.cudart().cudaHostRegister` (hipified on ROCm; raising on a non-zero status), and returns a pinned `torch` tensor over a 2 MiB-aligned mapping | | `pd_vllm/receive_server.py` | `ReceiveServer(..., buffer_device='cuda:0')`; with `'cpu'` the mooncake receive buffer comes from `alloc_pinned_huge` instead of `torch.zeros(..., device='cuda:0')` | | `pd_vllm/decode_server.py` | new flag `--pd-buffer-device {cuda,cpu}` (default `cuda`) forwarded to `ReceiveServer` | | `pd_vllm/prefill_connector.py` | connector extra config `tilert_pd_buffer_device` (default `cuda`); with `cpu` the TP-rank-0 staging buffer comes from `alloc_pinned_huge` instead of `torch.zeros(..., device='cuda:N')` | From 67630b3bce35f60fcee15c4696a3806a7b4df213 Mon Sep 17 00:00:00 2001 From: Oseltamivir <58582368+Oseltamivir@users.noreply.github.com> Date: Tue, 22 Sep 2026 08:27:52 +0000 Subject: [PATCH 14/25] fix(agentx): select unfiltered traces for GLM-5.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add glm5.3* to the 1M-context trace-source pattern and append the required performance changelog entry. Explicit loader overrides remain unchanged. Validation: bash -n and git diff --check passed; 11 real-resolver behavioral cases passed with external dependency installation/download stubbed; changelog YAML, exact config key, and historical byte preservation checked. No GPU benchmark rerun. AI disclosure: GPT 6 Astra via Perplexity Computer implemented and validated this follow-up. No delegated agents. 中文：将 glm5.3* 加入 1M 上下文语料匹配分支，默认选择未截断的 062126 语料，并在性能变更记录末尾追加条目；显式 loader 覆盖保持不变。Shell 语法、差异检查、11 个真实解析函数行为用例及变更记录校验通过；未重新运行 GPU 基准。由 GPT 6 Astra 经 Perplexity Computer 实现和验证，未使用委派 agent。 --- benchmarks/benchmark_lib.sh | 2 +- perf-changelog.yaml | 9 +++++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/benchmarks/benchmark_lib.sh b/benchmarks/benchmark_lib.sh index 60bdfecead..e5765862ed 100644 --- a/benchmarks/benchmark_lib.sh +++ b/benchmarks/benchmark_lib.sh @@ -3162,7 +3162,7 @@ resolve_trace_source() { # corpus; 1M-context families take the unfiltered variant, others 256k. local default_loader case "${MODEL_PREFIX:-}" in - dsv4*|glm5.2*|minimaxm3*|kimik3*) + dsv4*|glm5.2*|glm5.3*|minimaxm3*|kimik3*) default_loader="semianalysis_cc_traces_weka_062126" ;; *) diff --git a/perf-changelog.yaml b/perf-changelog.yaml index acc7a70c62..43054457a7 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8630,3 +8630,12 @@ - "Serve GLM-5.3's full 1048576-token context on both TileRT ranks with the bf16 MLA wire layout, gpu-memory-utilization 0.85 on the vLLM prefill, and the container-creation barrier timeout taken from CONTAINER_BARRIER_TIMEOUT (5400s) instead of a hardcoded 1800s. bf16 is the only layout TileRT 0.1.6 can consume from vLLM on ROCm: TileRT infers the layout from the cache tensor stride and accepts 1152 B/token (bf16) or 656 B/token (fp8_ds_mla), while ROCM_AITER_MLA_SPARSE has no fp8_ds_mla and its plain fp8 writes a flat 576 B/token row that the connector rejects at register_kv_caches. Per rank at this context (287.98 GiB MI355X): decode holds 90.72 GiB of weights, a 93.25 GiB engine cache window and a 99.06 GiB PD receive buffer; prefill holds 90.45 GiB of weights, 40.3 GiB of profiling/non-torch memory, 91.71 GiB of vLLM KV and a 99.06 GiB PD staging buffer on TP rank 0 outside vLLM's budget. Both PD buffers are dense in max_seq_len and must live in pinned host memory for this context to start: on the GPU the decode side is node-marginal (torch.OutOfMemoryError on one node, 99.06 GiB requested with 95.94 GiB free; served on another) and the prefill side cannot fit at any utilization. setup_deps.sh therefore applies patches/tilert-0.1.6-pd-buffers-in-dram.patch to the pip-installed tilert 0.1.6 at container start on both ranks (engine-patch waiver docs/waiver/3330.md; vLLM stays unpatched), placing both PD buffers in 2 MiB-backed pinned host memory behind TILERT_PD_BUFFER_DEVICE=cpu. The backing is verified before registration because the ionic RDMA VFs cap 4 KiB-page registrations at ~3.9 GiB per HCA (a ~2^20 page-table-entry budget), while 2 MiB-backed regions of 100 GiB register and take cross-node mooncake writes at ~21 GiB/s. The patch is removed when a TileRT release carries DRAM PD buffers. The barrier change covers cold image pulls: each rank pulls its own ~32 GB image, and the decode image alone took about 25 minutes of the old 30-minute budget on an uncached node. The launcher also passes sync.py --grace 60: the barrier's post-pass port-hold defaults to max(60, timeout/2) seconds in the foreground, which at 5400s was a 45-minute idle sleep on every rank (jobs 45373/45374)." - "以 bf16 MLA 线上布局在两侧 TileRT rank 提供 GLM-5.3 完整的 1048576 上下文，vLLM prefill 的 gpu-memory-utilization 设为 0.85，容器创建屏障超时改为由 CONTAINER_BARRIER_TIMEOUT（5400 秒）提供而非硬编码 1800 秒。bf16 是 TileRT 0.1.6 在 ROCm 上唯一能从 vLLM 读取的布局：TileRT 依据缓存张量步长推断布局，只接受每 token 1152 字节（bf16）或 656 字节（fp8_ds_mla），而 ROCM_AITER_MLA_SPARSE 没有 fp8_ds_mla，其普通 fp8 写出每 token 576 字节的扁平行，会在 register_kv_caches 被 connector 拒绝。该上下文下每 rank（287.98 GiB 的 MI355X）：decode 侧含 90.72 GiB 权重、93.25 GiB 引擎缓存窗口和 99.06 GiB 的 PD 接收缓冲；prefill 侧含 90.45 GiB 权重、40.3 GiB 分析/非 torch 内存、91.71 GiB vLLM KV，以及 TP rank 0 上位于 vLLM 预算之外的 99.06 GiB PD 暂存缓冲。两个 PD 缓冲均按 max_seq_len 密集分配，必须放在锁页主机内存中该上下文才能启动：放在 GPU 上时 decode 侧处于节点边缘（一节点 OOM，请求 99.06 GiB 而仅剩 95.94 GiB；另一节点可服务），prefill 侧在任何利用率下都放不下。因此 setup_deps.sh 在两侧容器启动时对 pip 安装的 tilert 0.1.6 应用 patches/tilert-0.1.6-pd-buffers-in-dram.patch（引擎补丁豁免 docs/waiver/3330.md；vLLM 未改动），在 TILERT_PD_BUFFER_DEVICE=cpu 下将两个 PD 缓冲放入 2 MiB 大页支撑的锁页主机内存。注册前会校验大页覆盖：ionic RDMA VF 对 4 KiB 页的注册上限约为每 HCA 3.9 GiB（约 2^20 条页表项），而 2 MiB 大页区域可注册 100 GiB 并以约 21 GiB/s 接收跨节点 mooncake 写入。TileRT 发布自带 DRAM PD 缓冲的版本后移除该补丁。屏障改动用于覆盖镜像冷拉取：每个 rank 各拉取约 32 GB 的镜像，仅 decode 镜像在未缓存节点上就实测约占旧 30 分钟预算的 25 分钟。启动脚本同时向 sync.py 传入 --grace 60：屏障通过后默认在前台保持端口 max(60, timeout/2) 秒，在 5400 秒下即每个 rank 空转 45 分钟（作业 45373/45374）。" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3330 + +- config-keys: + - glm5.3-fp8-mi355x-tilert-agentic + scenario-type: + - agentic-coding + description: + - "Add glm5.3* to resolve_trace_source's 1M-context model pattern so GLM-5.3 AgentX uses the unfiltered semianalysis_cc_traces_weka_062126 corpus instead of the 256k-filtered variant. Explicit WEKA_LOADER_OVERRIDE behavior is unchanged." + - "将 glm5.3* 加入 resolve_trace_source 的 1M 上下文模型匹配分支，使 GLM-5.3 AgentX 默认使用未截断的 semianalysis_cc_traces_weka_062126 语料，而非 256k 过滤版本。显式 WEKA_LOADER_OVERRIDE 的行为保持不变。" + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3330 From af42b1d75c5b877fa03063650a7f3c72834ffdfd Mon Sep 17 00:00:00 2001 From: CrimsonDump Date: Tue, 22 Sep 2026 15:29:17 +0800 Subject: [PATCH 15/25] =?UTF-8?q?fix(tilert):=20unfiltered=20WEKA=20corpus?= =?UTF-8?q?=20for=20GLM-5.3,=20PD=20buffers=20sharded=20in=20tilert=200.1.?= =?UTF-8?q?6.post1=20/=20GLM-5.3=20=E6=94=B9=E7=94=A8=E6=9C=AA=E8=BF=87?= =?UTF-8?q?=E6=BB=A4=20WEKA=20=E8=AF=AD=E6=96=99=EF=BC=8CPD=20=E7=BC=93?= =?UTF-8?q?=E5=86=B2=E6=94=B9=E7=94=B1=20tilert=200.1.6.post1=20=E5=88=86?= =?UTF-8?q?=E7=89=87?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two changes, both measured on a 2x8 MI350X pair at --max-model-len 1048576 with the bf16 wire layout. 1) resolve_trace_source() gives the unfiltered 062126 corpus to the 1M-context families and the 256k-filtered variant to everyone else, but the pattern list only spells glm5.2*, so glm5.3 falls through to the 256k branch. GLM-5.3 shares GLM-5.2's base and context window, so it belongs with the 1M families. The run log names the loader explicitly: Loading traces via aiperf public-dataset: semianalysis_cc_traces_weka_062126_256k (semianalysisai/cc-traces-weka-062126-256k) [MODEL_PREFIX=glm5.3] corpus ISL p50 p90 p95 max ...062126_256k (current) 69,813 170,341 -- 249,403 ...062126 (this change) 291,511 412,030 441,187 476,041 The 256k corpus never replays the tail this recipe is sized for, and its numbers are not comparable with the merged glm5.2 AMD entries, which do replay the unfiltered corpus. 2) tilert 0.1.6.post1 shards both PD buffers by layer across the eight devices (layer lid on device lid % 8) instead of placing each dense buffer on one card, so the recipe no longer needs the DRAM patch or TILERT_PD_BUFFER_DEVICE=cpu. Per card: 12.54 GiB instead of 99.06 GiB. convert() dequantises each layer on the device that received it, so its transients (108.6 KiB/token, 82.9 GiB at 800k tokens) spread as well rather than staying on cuda:0 -- the ceiling the DRAM route leaves in place. The KV path stays device-to-device: 81 GB in 751 ms = 108 GB/s, 54% of the 4x400 GbE line rate, against 20.9 GiB/s measured for the host-to-host hop. Measured with both changes, 3600 s AgentX at concurrency 1: submission_valid True, TTFT and inter-token-latency coverage both 100%, 239 successful requests, 0 errors, TTFT p50 5,711.8 ms, ITL p50 2.18 ms. Peak device memory 202.1 GiB per card on decode and 269.2 GiB on prefill, of 287.69 GiB. GPU_MEM_UTIL stays at 0.85. 一、resolve_trace_source() 把未过滤的 062126 语料给 1M 上下文家族、其余给 256k 过滤版，但模式串只写了 glm5.2*，于是 glm5.3 落进 256k 分支。GLM-5.3 与 GLM-5.2 同底座、同上下文窗口，应归入 1M 家族。256k 语料从不回放本配方所声明的长尾，且与 已合入的 glm5.2 AMD 条目不可比（对照见上表，日志原文见上）。 二、tilert 0.1.6.post1 把两个 PD 缓冲按层分片到 8 张卡（层 lid 落在卡 lid % 8）， 不再把整块放在单卡上，因此本配方不再需要 DRAM 补丁与 TILERT_PD_BUFFER_DEVICE=cpu。 每卡 12.54 GiB 而非 99.06 GiB。convert() 在收到该层的那张卡上就地反量化，其中间 结果（108.6 KiB/token，800k 时 82.9 GiB）也随之摊开——这正是 DRAM 方案未解的那个 天花板。KV 通路保持设备到设备：81 GB / 751 ms = 108 GB/s，为 4x400 GbE 线速的 54%，对比主机到主机实测 20.9 GiB/s。 两项改动同时生效下实测（3600 秒 AgentX，并发 1）：submission_valid True、两项 覆盖率均 100%、239 条成功、0 错误、TTFT p50 5,711.8 ms、ITL p50 2.18 ms；decode 每卡峰值 202.1 GiB、prefill 269.2 GiB（卡容量 287.69 GiB）；GPU_MEM_UTIL 保持 0.85。 --- .../agentic/glm5.3_fp8_mi355x_tilert.sh | 27 ++++++++++--------- .../multi_node/glm5.3_fp8_mi355x_tilert.sh | 27 ++++++++++--------- 2 files changed, 28 insertions(+), 26 deletions(-) diff --git a/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh b/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh index 2565c046d4..c2a7f47b75 100644 --- a/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh +++ b/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh @@ -61,7 +61,7 @@ fi # TileRT configuration. Every value is explicit here: server_tilert.sh # validates each one with check_env_vars and supplies no defaults of its own. -export TILERT_VERSION=0.1.6 +export TILERT_VERSION=0.1.6.post1 export TILERT_PROFILE=glm5_2 # decode_server --model (TileRT model profile) export TILERT_MODEL_TYPE=glm-5 # weight_converter --model_type (fallback converter) export TILERT_MODEL_PKG=glm_5_2_rocm # per-model converter package, preferred when importable @@ -77,19 +77,20 @@ export SERVED_MODEL_NAME=glm5_2 # prefill: weights 90.45 GiB + profiling/non-torch 40.3 GiB + vLLM KV 91.71 GiB # + PD staging buffer 99.06 GiB (prefill_connector.py, TP rank 0, # allocated OUTSIDE vLLM's gpu-memory-utilization budget) -# Both PD buffers must live in pinned host memory (~3.2 TB RAM per node) for -# this context to start; on the GPU the decode side is node-marginal (~283 of -# 288 GiB) and the prefill side cannot fit at any utilization (~321 GiB). -# TileRT 0.1.6 places both on the GPU, so setup_deps.sh applies -# patches/tilert-0.1.6-pd-buffers-in-dram.patch at container start (engine-patch -# waiver docs/waiver/3330.md) and TILERT_PD_BUFFER_DEVICE=cpu selects the host -# buffers. They are 2 MiB-backed and verified as such: the ionic RDMA VFs cap -# 4 KiB-page registrations at ~3.9 GiB per HCA. Cost: two extra PCIe copies on -# the KV path (~0.83 GB each at 8k tokens, estimated ~17 ms apiece; the RDMA -# hop itself measured 20.9 GiB/s). Drop the patch and this knob when a TileRT -# release carries DRAM PD buffers. +# Undivided, neither side starts: the decode rank is node-marginal (~283 of +# 288 GiB) and the prefill rank cannot fit at any utilization (~321 GiB). +# tilert 0.1.6.post1 keeps both buffers on the GPU but shards them by layer +# across the eight devices (layer lid on device lid % 8, TILERT_PD_SHARDS, +# default on), so each card holds 12.54 GiB instead of 99.06 GiB on one. +# convert() dequantises each layer on the device that received it, which spreads +# its transients too (108.6 KiB/token, 82.9 GiB at 800k tokens) instead of +# leaving them on cuda:0. Measured on 2x8 MI350X at this context with bf16 KV: +# decode peaks at 202.1 GiB per card, prefill at 269.2 GiB of 287.69 GiB, and +# the KV path stays device-to-device at 108 GB/s (81 GB in 751 ms, 54% of the +# 4x400 GbE line rate). No host hop, so the DRAM patch and +# TILERT_PD_BUFFER_DEVICE are not used here. export TILERT_MAX_MODEL_LEN=1048576 -export TILERT_PD_BUFFER_DEVICE=cpu +export TILERT_PD_BUFFER_DEVICE=cuda export TILERT_TRANSPORT=mooncake export TILERT_PARSER=none export TILERT_RDMA_STRICT=0 diff --git a/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh b/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh index caba2e0bb6..4c76c81b1b 100644 --- a/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh +++ b/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh @@ -47,7 +47,7 @@ fi # TileRT configuration. Every value is explicit here: server_tilert.sh # validates each one with check_env_vars and supplies no defaults of its own. -export TILERT_VERSION=0.1.6 +export TILERT_VERSION=0.1.6.post1 export TILERT_PROFILE=glm5_2 # decode_server --model (TileRT model profile) export TILERT_MODEL_TYPE=glm-5 # weight_converter --model_type (fallback converter) export TILERT_MODEL_PKG=glm_5_2_rocm # per-model converter package, preferred when importable @@ -63,19 +63,20 @@ export SERVED_MODEL_NAME=glm5_2 # prefill: weights 90.45 GiB + profiling/non-torch 40.3 GiB + vLLM KV 91.71 GiB # + PD staging buffer 99.06 GiB (prefill_connector.py, TP rank 0, # allocated OUTSIDE vLLM's gpu-memory-utilization budget) -# Both PD buffers must live in pinned host memory (~3.2 TB RAM per node) for -# this context to start; on the GPU the decode side is node-marginal (~283 of -# 288 GiB) and the prefill side cannot fit at any utilization (~321 GiB). -# TileRT 0.1.6 places both on the GPU, so setup_deps.sh applies -# patches/tilert-0.1.6-pd-buffers-in-dram.patch at container start (engine-patch -# waiver docs/waiver/3330.md) and TILERT_PD_BUFFER_DEVICE=cpu selects the host -# buffers. They are 2 MiB-backed and verified as such: the ionic RDMA VFs cap -# 4 KiB-page registrations at ~3.9 GiB per HCA. Cost: two extra PCIe copies on -# the KV path (~0.83 GB each at 8k tokens, estimated ~17 ms apiece; the RDMA -# hop itself measured 20.9 GiB/s). Drop the patch and this knob when a TileRT -# release carries DRAM PD buffers. +# Undivided, neither side starts: the decode rank is node-marginal (~283 of +# 288 GiB) and the prefill rank cannot fit at any utilization (~321 GiB). +# tilert 0.1.6.post1 keeps both buffers on the GPU but shards them by layer +# across the eight devices (layer lid on device lid % 8, TILERT_PD_SHARDS, +# default on), so each card holds 12.54 GiB instead of 99.06 GiB on one. +# convert() dequantises each layer on the device that received it, which spreads +# its transients too (108.6 KiB/token, 82.9 GiB at 800k tokens) instead of +# leaving them on cuda:0. Measured on 2x8 MI350X at this context with bf16 KV: +# decode peaks at 202.1 GiB per card, prefill at 269.2 GiB of 287.69 GiB, and +# the KV path stays device-to-device at 108 GB/s (81 GB in 751 ms, 54% of the +# 4x400 GbE line rate). No host hop, so the DRAM patch and +# TILERT_PD_BUFFER_DEVICE are not used here. export TILERT_MAX_MODEL_LEN=1048576 -export TILERT_PD_BUFFER_DEVICE=cpu +export TILERT_PD_BUFFER_DEVICE=cuda export TILERT_TRANSPORT=mooncake export TILERT_PARSER=none export TILERT_RDMA_STRICT=0 From 93ca00c349947c579b5e3e444e9d3f07b426fa4b Mon Sep 17 00:00:00 2001 From: CrimsonDump Date: Tue, 22 Sep 2026 22:28:02 +0800 Subject: [PATCH 16/25] =?UTF-8?q?refactor(tilert):=20let=20tilert=200.1.6.?= =?UTF-8?q?post1=20shard=20the=20PD=20buffers;=20drop=20the=20PD-DRAM=20pa?= =?UTF-8?q?tch,=20waiver=203330=20and=20TILERT=5FPD=5FBUFFER=5FDEVICE=20/?= =?UTF-8?q?=20=E4=BA=A4=E7=94=B1=20tilert=200.1.6.post1=20=E5=88=86?= =?UTF-8?q?=E7=89=87=20PD=20=E7=BC=93=E5=86=B2=EF=BC=8C=E7=A7=BB=E9=99=A4?= =?UTF-8?q?=20PD-DRAM=20=E8=A1=A5=E4=B8=81=E3=80=81=E8=B1=81=E5=85=8D=2033?= =?UTF-8?q?30=20=E4=B8=8E=20TILERT=5FPD=5FBUFFER=5FDEVICE?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous commit moved the recipes to tilert==0.1.6.post1 and set TILERT_PD_BUFFER_DEVICE=cuda. With the shipped post1 wheel the host-buffer route is dead code, so this removes it rather than carrying an unused engine patch: - benchmarks/multi_node/amd_utils/patches/tilert-0.1.6-pd-buffers-in-dram.patch and docs/waiver/3330.md are deleted; setup_deps.sh no longer installs `patch` or applies anything to the pip-installed wheel, and the 0.1.6 version-pin guard that protected the patch goes with it. Nothing in either image is modified any more, so the engine-patch waiver has no subject. - TILERT_PD_BUFFER_DEVICE is removed from both recipes, server_tilert.sh (check_env_vars, the cpu|cuda case, the status line, the decode_server --pd-buffer-device flag and the connector's tilert_pd_buffer_device extra config key), job.slurm (check_env_vars and the -e forward) and setup_deps.sh. The shipped decode_server has no such flag (strict parse_args), and prefill_connector reads its extra config with .get(), so the key was inert. - configs/amd-master.yaml: router metadata 0.1.6 -> 0.1.6.post1. pd_router.py is one of the six files post1 changes: the fixed 600 s request timeouts to vLLM and decode become TILERT_PD_HTTP_TIMEOUT_S (default 3600). Verified against the two PyPI wheels rather than the release notes (tilert-0.1.6 sha256 59276124..., tilert-0.1.6.post1 sha256 d4de62b0..., uploaded 2026-09-22T08:14:17Z). post1 differs in tilert/pd_vllm/{transport, receive_server, prefill_connector, pd_router}.py and profiles/{base, mla_nsa}.py. Both PD buffers become nshards = min(TILERT_PD_SHARDS or 8, torch.cuda.device_count()) per-device tensors of ceil(num_layers / nshards) layers each, layer lid on shard lid % nshards; each shard is registered with mooncake on its own device (location hip:), hello_layout carries per-shard bases and rdma_plan writes layer by layer to the matching remote shard. The glm5_2 profile has 79 layers, so the heaviest shard holds 10 of them: 99.06 GiB x 10/79 = 12.54 GiB per card, the figure the recipe comments quote. mla_nsa.convert() now walks the shard that received each layer and synchronizes every device it touched. Recipe comments and the GPU_MEM_UTIL derivation are reworded for the sharded layout (the 12.54 GiB staging shard sits outside vLLM's budget next to the ~6.3 GiB non-torch baseline). No benchmark was rerun for this commit; the sweep on this PR is its validation. 上一提交已将配方切到 tilert==0.1.6.post1 并设 TILERT_PD_BUFFER_DEVICE=cuda。对 正式发布的 post1 wheel 而言，主机内存缓冲那条路已是死代码，因此本提交将其整体移除， 而不是继续携带一个不再使用的引擎补丁： - 删除 patches/tilert-0.1.6-pd-buffers-in-dram.patch 与 docs/waiver/3330.md； setup_deps.sh 不再安装 `patch`、不再对 pip 安装的 wheel 做任何修改，为补丁服务的 0.1.6 版本锁定检查一并删除。两侧镜像内不再有任何被改动的文件，引擎补丁豁免失去对象。 - 从两份配方、server_tilert.sh（check_env_vars、cpu|cuda 分支、状态行、decode_server 的 --pd-buffer-device 参数、connector 的 tilert_pd_buffer_device 额外配置键）、 job.slurm（check_env_vars 与 -e 转发）和 setup_deps.sh 中移除 TILERT_PD_BUFFER_DEVICE。 正式 decode_server 没有该参数（parse_args 严格模式），prefill_connector 用 .get() 读取额外配置，因此该键此前实际无效。 - configs/amd-master.yaml：router 元数据 0.1.6 -> 0.1.6.post1。pd_router.py 是 post1 改动的六个文件之一：对 vLLM 与 decode 固定的 600 秒请求超时改为 TILERT_PD_HTTP_TIMEOUT_S（默认 3600）。 以上依据 PyPI 上两个 wheel 的实际差异核对，而非发布说明（tilert-0.1.6 sha256 59276124...，tilert-0.1.6.post1 sha256 d4de62b0...，上传于 2026-09-22T08:14:17Z）。 post1 改动 tilert/pd_vllm/{transport, receive_server, prefill_connector, pd_router}.py 与 profiles/{base, mla_nsa}.py：两个 PD 缓冲改为 nshards = min(TILERT_PD_SHARDS 或 8, torch.cuda.device_count()) 个逐卡张量，每个含 ceil(num_layers / nshards) 层，第 lid 层落在第 lid % nshards 个分片；各分片在所在 卡上向 mooncake 注册（location hip:），hello_layout 携带各分片基址， rdma_plan 逐层写入对应的远端分片。glm5_2 profile 共 79 层，最重的分片含 10 层： 99.06 GiB x 10/79 = 12.54 GiB/卡，即配方注释所引数字。mla_nsa.convert() 改为在收到 该层的分片上处理并同步所有涉及的设备。 配方注释与 GPU_MEM_UTIL 的推导按分片布局改写（12.54 GiB 暂存分片与约 6.3 GiB 非 torch 基线同在 vLLM 预算之外）。本提交未重跑基准；本 PR 的 sweep 即其验证。 --- .../agentic/glm5.3_fp8_mi355x_tilert.sh | 20 +- benchmarks/multi_node/amd_utils/job.slurm | 3 +- .../tilert-0.1.6-pd-buffers-in-dram.patch | 238 ------------------ .../multi_node/amd_utils/server_tilert.sh | 23 +- benchmarks/multi_node/amd_utils/setup_deps.sh | 47 +--- .../multi_node/glm5.3_fp8_mi355x_tilert.sh | 20 +- configs/amd-master.yaml | 4 +- docs/waiver/3330.md | 85 ------- 8 files changed, 35 insertions(+), 405 deletions(-) delete mode 100644 benchmarks/multi_node/amd_utils/patches/tilert-0.1.6-pd-buffers-in-dram.patch delete mode 100644 docs/waiver/3330.md diff --git a/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh b/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh index c2a7f47b75..a8f6c3ac9f 100644 --- a/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh +++ b/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh @@ -71,7 +71,8 @@ export SERVED_MODEL_NAME=glm5_2 # TileRT recipe this mirrors.) # # Memory at this context, per rank, bf16 wire layout (verified against the -# tilert 0.1.6 and vLLM 0.24.0 sources and the MI355X logs, 287.98 GiB cards): +# tilert 0.1.6 and vLLM 0.24.0 sources and the MI355X logs, 287.98 GiB cards; +# the undivided PD buffer sizes are what 0.1.6 allocated on one card): # decode : weights 90.72 GiB + engine cache window 93.25 GiB # + PD receive buffer 99.06 GiB (receive_server.py, dense in max_seq_len) # prefill: weights 90.45 GiB + profiling/non-torch 40.3 GiB + vLLM KV 91.71 GiB @@ -87,10 +88,8 @@ export SERVED_MODEL_NAME=glm5_2 # leaving them on cuda:0. Measured on 2x8 MI350X at this context with bf16 KV: # decode peaks at 202.1 GiB per card, prefill at 269.2 GiB of 287.69 GiB, and # the KV path stays device-to-device at 108 GB/s (81 GB in 751 ms, 54% of the -# 4x400 GbE line rate). No host hop, so the DRAM patch and -# TILERT_PD_BUFFER_DEVICE are not used here. +# 4x400 GbE line rate). No host hop and no patch: the wheel runs as shipped. export TILERT_MAX_MODEL_LEN=1048576 -export TILERT_PD_BUFFER_DEVICE=cuda export TILERT_TRANSPORT=mooncake export TILERT_PARSER=none export TILERT_RDMA_STRICT=0 @@ -110,12 +109,13 @@ export PREFILL_KV_DTYPE=bfloat16 # the connector's KI plane copy fail and MLA address the wrong rows. export PREFILL_BLOCK_SIZE=64 export DECODE_KV_DTYPE=bf16 -# With the PD staging buffer in host memory, vLLM needs 90.45 (weights) + 40.3 -# (profiling) + 91.71 GiB (KV for one 1048576-token request) = 222.5 GiB inside -# its budget: 0.85 x 287.98 = 244.8 GiB leaves 22 GiB of KV margin and 43 GiB -# outside the budget for the ~6.3 GiB non-torch baseline measured on the decode -# OOM node (287.98 - 95.94 free - 184.17 - 1.58 reserved). 0.75 (216 GiB) refuses -# with "91.71 GiB KV cache is needed ... available 85.25 GiB". +# The PD staging shard sits outside vLLM's budget, so vLLM needs 90.45 (weights) +# + 40.3 (profiling) + 91.71 GiB (KV for one 1048576-token request) = 222.5 GiB +# inside it: 0.85 x 287.98 = 244.8 GiB leaves 22 GiB of KV margin and 43 GiB +# outside the budget for the 12.54 GiB staging shard plus the ~6.3 GiB non-torch +# baseline measured on the decode OOM node (287.98 - 95.94 free - 184.17 - 1.58 +# reserved). 0.75 (216 GiB) refuses with "91.71 GiB KV cache is needed ... +# available 85.25 GiB". export GPU_MEM_UTIL=0.85 export SKIP_CONTAINER_BARRIER=0 # Two images, one per rank, ~32 GB each. On a node that has neither cached the diff --git a/benchmarks/multi_node/amd_utils/job.slurm b/benchmarks/multi_node/amd_utils/job.slurm index 324da80f3e..0c779476ea 100755 --- a/benchmarks/multi_node/amd_utils/job.slurm +++ b/benchmarks/multi_node/amd_utils/job.slurm @@ -62,7 +62,7 @@ if [[ "$ENGINE" == "tilert" ]]; then # server_tilert.sh takes the container-creation barrier timeout from the # recipe (no 300s default as on the SGLang path); fail here, before sbatch # work is done, rather than inside the container. - check_env_vars CONTAINER_BARRIER_TIMEOUT TILERT_PD_BUFFER_DEVICE + check_env_vars CONTAINER_BARRIER_TIMEOUT fi # Resolve the models.yaml entry the same way server_sglang.sh does: agentic runs @@ -551,7 +551,6 @@ elif [[ "$ENGINE" == "tilert" ]]; then -e TILERT_RDMA_STRICT=${TILERT_RDMA_STRICT} -e TILERT_CONVERT_LOCK_WAIT=${TILERT_CONVERT_LOCK_WAIT} -e TILERT_SIMULATE_ACC_METHOD=${TILERT_SIMULATE_ACC_METHOD} - -e TILERT_PD_BUFFER_DEVICE=${TILERT_PD_BUFFER_DEVICE} -e \"TILERT_EXTRA_ENV=${TILERT_EXTRA_ENV:-}\" -e SERVED_MODEL_NAME=${SERVED_MODEL_NAME} -e PREFILL_KV_DTYPE=${PREFILL_KV_DTYPE} diff --git a/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6-pd-buffers-in-dram.patch b/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6-pd-buffers-in-dram.patch deleted file mode 100644 index b10b1a2c92..0000000000 --- a/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6-pd-buffers-in-dram.patch +++ /dev/null @@ -1,238 +0,0 @@ -diff -ru -x __pycache__ orig/decode_server.py new/decode_server.py ---- a/decode_server.py 2026-09-22 10:06:55 -+++ b/decode_server.py 2026-09-22 10:08:38 -@@ -309,6 +309,7 @@ - ap.add_argument('--num-mtp', type=int, choices=(3,), default=3, help='MTP draft depth for speculative decoding; needs --with-mtp') - ap.add_argument('--transport', choices=['mooncake', 'nixl'], default='mooncake', help="RDMA data-plane backend (must match prefill's tilert_transport)") - ap.add_argument('--kv-cache-dtype', default='fp8_ds_mla', help='MLA cache dtype (must match vLLM prefill); MLA-family profiles only') -+ ap.add_argument('--pd-buffer-device', choices=['cuda', 'cpu'], default='cuda', help="where the PD receive buffer lives; 'cpu' = pinned host memory registered with the transport (frees buffer_bytes(max_seq_len) of VRAM, adds one H2D copy per request)") - return ap - - def main() -> None: -@@ -329,7 +330,7 @@ - logger.info('loading TileRT engine (profile=%s, weights=%s)...', profile.name, args.model_weights_dir) - engine = profile.build_engine(model_weights_dir=args.model_weights_dir, max_seq_len=args.max_seq_len, with_mtp=args.with_mtp, ar_steps=8, num_mtp=num_mtp) - logger.info('TileRT engine ready (cache window %d, num_mtp %d)', engine.max_seq_len, num_mtp) -- server = ReceiveServer(profile, max_seq_len=args.max_seq_len, ctrl_port=args.ctrl_port, transport=args.transport) -+ server = ReceiveServer(profile, max_seq_len=args.max_seq_len, ctrl_port=args.ctrl_port, transport=args.transport, buffer_device='cpu' if args.pd_buffer_device == 'cpu' else 'cuda:0') - app = build_app(server, engine) - logger.info('decode server on :%d (profile=%s, engine=%s, ctrl=:%d)', args.http_port, profile.name, args.engine, args.ctrl_port) - sock = socket.socket(socket.AF_INET6, socket.SOCK_STREAM) -diff -ru -x __pycache__ orig/prefill_connector.py new/prefill_connector.py ---- a/prefill_connector.py 2026-09-22 10:06:55 -+++ b/prefill_connector.py 2026-09-22 14:09:20 -@@ -49,6 +49,7 @@ - self._max_seq = int(extra.get('tilert_max_seq_len', vllm_config.model_config.max_model_len)) - self._profile = profiles.get_profile(extra.get('tilert_model', 'glm5')) - self._transport_name = extra.get('tilert_transport', 'mooncake') -+ self._pd_buffer_device = str(extra.get('tilert_pd_buffer_device', 'cuda')) - self._pending: dict[str, _Pending] = {} - self._kv_caches: dict = {} - self._reg = None -@@ -145,10 +146,19 @@ - hostname = wire.local_ip() - total = self._profile.staging_bytes(self._reg, self._tp_rank, self._max_seq) - dev = torch.cuda.current_device() -- self._staging = torch.zeros(total, dtype=torch.uint8, device=f'cuda:{dev}') -+ # Dense in max_seq_len (99.06 GiB at 1M for glm5_2 bf16) and allocated -+ # outside vLLM's gpu-memory-utilization budget. In pinned host memory -+ # when tilert_pd_buffer_device == 'cpu'; extract() already fills it with -+ # copy_(), which performs the D2H transfer. -+ host = self._pd_buffer_device == 'cpu' -+ if host: -+ from tilert.pd_vllm.transport import alloc_pinned_huge -+ self._staging = alloc_pinned_huge(total) -+ else: -+ self._staging = torch.zeros(total, dtype=torch.uint8, device=f'cuda:{dev}') - self._transport = make_transport(self._transport_name) - self._transport.init(hostname) -- self._transport.register(self._staging.data_ptr(), total, dev) -+ self._transport.register(self._staging.data_ptr(), total, dev, host=host) - self._sender_thread = threading.Thread(target=self._sender_loop, name='tilert-pd-sender', daemon=True) - self._sender_thread.start() - logger.info('worker ready: rank=%d transport=%s staging=%.1f MB profile=%s', self._tp_rank, self._transport.name, total / 1000000.0, self._profile.name) -diff -ru -x __pycache__ orig/profiles/mla_nsa.py new/profiles/mla_nsa.py ---- a/profiles/mla_nsa.py 2026-09-22 10:06:55 -+++ b/profiles/mla_nsa.py 2026-09-22 10:07:21 -@@ -125,14 +125,23 @@ - for lid in range(self.num_layers): - ko = lid * max_seq_len * kv_bpt - kv_raw = buffer[ko:ko + seq * kv_bpt].view(seq, kv_bpt) -+ # A host-resident receive buffer is moved to the device plane by -+ # plane; the copy scales with the request's seq_len, not max_seq_len. -+ if not buffer.is_cuda: -+ kv_raw = kv_raw.to('cuda:0', non_blocking=True) - if self.mla_fp8: - kv = self._dequant_kv(kv_raw, seq) - else: - kv = kv_raw.view(torch.bfloat16).view(seq, KV_LORA_RANK).contiguous() - po = pe_base + lid * max_seq_len * PE_BPT -- pe = buffer[po:po + seq * PE_BPT].view(torch.bfloat16).view(seq, QK_ROPE_HEAD_DIM).contiguous() -+ pe = buffer[po:po + seq * PE_BPT] -+ if not buffer.is_cuda: -+ pe = pe.to('cuda:0', non_blocking=True) -+ pe = pe.view(torch.bfloat16).view(seq, QK_ROPE_HEAD_DIM).contiguous() - io = ki_base + lid * _max_pages(max_seq_len) * KI_PAGE_BYTES - ki_raw = buffer[io:io + npages * KI_PAGE_BYTES].view(npages, KI_PAGE_BYTES) -+ if not buffer.is_cuda: -+ ki_raw = ki_raw.to('cuda:0', non_blocking=True) - layers.append((self._dequant_ki(ki_raw, seq, self.ki_tiled), kv, pe)) - torch.cuda.synchronize() - return ConvertedRequest(rid=received.rid, seq_len=seq, last_prompt_token=received.last_prompt_token, first_token_id=received.first_token_id, sampling=received.sampling, layers=layers) -diff -ru -x __pycache__ orig/receive_server.py new/receive_server.py ---- a/receive_server.py 2026-09-22 10:06:55 -+++ b/receive_server.py 2026-09-22 14:09:20 -@@ -34,15 +34,26 @@ - - class ReceiveServer: - -- def __init__(self, profile, max_seq_len: int, ctrl_port: int=5556, hostname: str | None=None, device: str='cuda:0', request_timeout: float=120.0, transport: str='mooncake'): -+ def __init__(self, profile, max_seq_len: int, ctrl_port: int=5556, hostname: str | None=None, device: str='cuda:0', request_timeout: float=120.0, transport: str='mooncake', buffer_device: str='cuda:0'): - self.profile = profile - self.max_seq_len = max_seq_len - self.ctrl_port = ctrl_port - self.device = device - self.request_timeout = request_timeout - total = profile.buffer_bytes(max_seq_len) -- logger.info('allocating receive buffer: %.2f GB on %s (profile=%s)', total / 1024 ** 3, device, profile.name) -- self.buffer = torch.zeros(total, dtype=torch.uint8, device=device) -+ # Dense in max_seq_len: 99.06 GiB at 1M tokens for glm5_2 bf16. Held in -+ # pinned host memory when buffer_device == 'cpu' so it does not compete -+ # with the weights and the engine cache window on the GPU; convert() -+ # then moves each request's planes to the device (H2D scales with the -+ # prompt, not with max_seq_len). -+ self.buffer_device = buffer_device -+ logger.info('allocating receive buffer: %.2f GiB on %s (profile=%s)', total / 1024 ** 3, buffer_device, profile.name) -+ if buffer_device == 'cpu': -+ from tilert.pd_vllm.transport import alloc_pinned_huge -+ self.buffer = alloc_pinned_huge(total) -+ else: -+ self.buffer = torch.zeros(total, dtype=torch.uint8, device=buffer_device) -+ logger.info('receive buffer ready: %.2f GiB on %s', total / 1024 ** 3, self.buffer.device) - self.base_ptr = self.buffer.data_ptr() - self._hello_layout = profile.hello_layout(self.base_ptr, max_seq_len) - from tilert.pd_vllm.transport import make_transport -@@ -51,7 +62,7 @@ - dev_id = torch.device(device).index or 0 - self._transport = make_transport(transport) - self._transport.init(hostname) -- self._transport.register(self.base_ptr, total, dev_id) -+ self._transport.register(self.base_ptr, total, dev_id, host=(buffer_device == 'cpu')) - self._transport_meta = self._transport.local_meta() - logger.info('transport=%s ready, buffer registered (%.2f GB)', self._transport.name, total / 1024 ** 3) - self._lock = threading.Lock() -diff -ru -x __pycache__ orig/transport.py new/transport.py ---- a/transport.py 2026-09-22 10:06:55 -+++ b/transport.py 2026-09-22 15:17:46 -@@ -8,7 +8,7 @@ - def init(self, host: str) -> None: - ... - -- def register(self, ptr: int, nbytes: int, dev_id: int) -> None: -+ def register(self, ptr: int, nbytes: int, dev_id: int, host: bool=False) -> None: - ... - - def local_meta(self) -> dict: -@@ -28,7 +28,8 @@ - raise RuntimeError(f'Mooncake engine init failed: {ret}') - self.session_id = f'{host}:{self.engine.get_rpc_port()}' - -- def register(self, ptr: int, nbytes: int, dev_id: int) -> None: -+ def register(self, ptr: int, nbytes: int, dev_id: int, host: bool=False) -> None: -+ # Mooncake registers host (pinned) and device pointers through the same call. - ret = self.engine.batch_register_memory([ptr], [nbytes]) - if ret != 0: - raise RuntimeError(f'Mooncake register failed: {ret}') -@@ -50,10 +51,12 @@ - self._agent = nixl_agent(f'{host}:{os.getpid()}', nixl_agent_config(backends=['UCX'])) - self._remotes: dict[bytes, str] = {} - self._dev = 0 -+ self._mem_type = 'VRAM' - -- def register(self, ptr: int, nbytes: int, dev_id: int) -> None: -- self._dev = dev_id -- self._agent.register_memory([(ptr, nbytes, dev_id, '')], 'VRAM') -+ def register(self, ptr: int, nbytes: int, dev_id: int, host: bool=False) -> None: -+ self._dev = 0 if host else dev_id -+ self._mem_type = 'DRAM' if host else 'VRAM' -+ self._agent.register_memory([(ptr, nbytes, self._dev, '')], self._mem_type) - - def local_meta(self) -> dict: - return {'nixl_meta': base64.b64encode(self._agent.get_agent_metadata()).decode(), 'nixl_dev': self._dev} -@@ -65,8 +68,8 @@ - rname = self._agent.add_remote_agent(meta_b) - self._remotes[meta_b] = rname - rdev = int(remote_meta.get('nixl_dev', 0)) -- ld = self._agent.get_xfer_descs([(int(s), int(n), self._dev) for s, n in zip(srcs, lens)], 'VRAM') -- rd = self._agent.get_xfer_descs([(int(d), int(n), rdev) for d, n in zip(dsts, lens)], 'VRAM') -+ ld = self._agent.get_xfer_descs([(int(s), int(n), self._dev) for s, n in zip(srcs, lens)], self._mem_type) -+ rd = self._agent.get_xfer_descs([(int(d), int(n), rdev) for d, n in zip(dsts, lens)], self._mem_type) - h = self._agent.initialize_xfer('WRITE', ld, rd, rname) - try: - st = self._agent.transfer(h) -@@ -86,4 +89,63 @@ - key = (name or 'mooncake').lower() - if key not in _BACKENDS: - raise ValueError(f'unknown transport {name!r}; choices: {sorted(_BACKENDS)}') -- return _BACKENDS[key]() -\ No newline at end of file -+ return _BACKENDS[key]() -+ -+def alloc_pinned_huge(total: int): -+ """Host buffer for the PD data plane, backed end-to-end by 2 MiB pages. -+ -+ The ionic RDMA VFs on the MI355X pool have a ~2**20 page-table-entry budget -+ per HCA: with 4 KiB pages ibv_reg_mr fails (EINVAL/ENOMEM) once ~3.9 GiB is -+ registered, while 2 MiB-backed regions register tens of GiB. One 4 KiB-backed -+ slice anywhere in the region drops the whole MR to 4 KiB granularity, so the -+ mapping is 2 MiB-aligned and the backing is verified from /proc/self/smaps -+ before the memory is handed out. The region is then hipHostRegister'ed so -+ the GPU can DMA into it and torch reports it pinned. -+ """ -+ import ctypes, mmap, re -+ import torch -+ MiB = 1 << 20 -+ HUGE = 2 * MiB -+ if total % HUGE: -+ total += HUGE - total % HUGE -+ # Over-allocate by one huge page so the used range can be aligned on kernels -+ # that do not PMD-align anonymous mappings. -+ mm = mmap.mmap(-1, total + HUGE, flags=mmap.MAP_PRIVATE | mmap.MAP_ANONYMOUS) -+ raw = ctypes.addressof(ctypes.c_char.from_buffer(mm)) -+ off = (-raw) % HUGE -+ addr = raw + off -+ libc = ctypes.CDLL(None, use_errno=True) -+ libc.madvise.argtypes = [ctypes.c_void_p, ctypes.c_size_t, ctypes.c_int] -+ if libc.madvise(ctypes.c_void_p(addr), ctypes.c_size_t(total), ctypes.c_int(14)) != 0: # MADV_HUGEPAGE -+ raise RuntimeError(f'madvise(MADV_HUGEPAGE) failed errno={ctypes.get_errno()}') -+ # Fault the huge pages in up front. MADV_POPULATE_WRITE (Linux >= 5.14) does -+ # it in one call; touching one byte per 2 MiB from Python takes ~150 s for -+ # 100 GiB, so it is only the fallback. -+ if libc.madvise(ctypes.c_void_p(addr), ctypes.c_size_t(total), ctypes.c_int(23)) != 0: # MADV_POPULATE_WRITE -+ for o in range(off, off + total, HUGE): -+ mm[o] = 0 -+ huge = 0 -+ for blk in re.split(r'\n(?=[0-9a-f]+-[0-9a-f]+ )', open('/proc/self/smaps').read()): -+ m = re.match(r'([0-9a-f]+)-([0-9a-f]+) ', blk) -+ if not m: -+ continue -+ lo, hi = int(m.group(1), 16), int(m.group(2), 16) -+ if hi <= addr or lo >= addr + total: -+ continue -+ h = re.search(r'AnonHugePages:\s+(\d+) kB', blk) -+ huge += int(h.group(1)) * 1024 if h else 0 -+ if huge < total: -+ raise RuntimeError(f'PD host buffer is only {huge / 2**30:.2f} of {total / 2**30:.2f} GiB huge-page backed; ' -+ 'the RDMA MR would fall back to 4 KiB pages and exceed the per-HCA entry budget ' -+ '(check /sys/kernel/mm/transparent_hugepage/{enabled,defrag} and free memory)') -+ # Register through torch's own GPU runtime binding rather than dlopen'ing a -+ # HIP/CUDA library by name: the tilert-rocm-decode image ships only the -+ # versioned libamdhip64.so.7, the prefill image both names, and CUDA builds -+ # neither. torch.cuda.cudart() is hipified on ROCm and exposes the same -+ # cudaHostRegister on every build. Returns a cudaError enum; 0 is success. -+ rc = torch.cuda.cudart().cudaHostRegister(addr, total, 0) -+ if int(rc) != 0: -+ raise RuntimeError(f'cudaHostRegister/hipHostRegister failed: {rc}') -+ buf = torch.frombuffer(mm, dtype=torch.uint8, count=total, offset=off) -+ buf._pd_mmap = mm # keep the mapping alive with the tensor -+ return buf diff --git a/benchmarks/multi_node/amd_utils/server_tilert.sh b/benchmarks/multi_node/amd_utils/server_tilert.sh index 818679d180..b579cdf947 100644 --- a/benchmarks/multi_node/amd_utils/server_tilert.sh +++ b/benchmarks/multi_node/amd_utils/server_tilert.sh @@ -20,15 +20,7 @@ check_env_vars \ DECODE_MTP_SIZE GPU_MEM_UTIL SERVED_MODEL_NAME \ DECODE_CTRL_PORT DECODE_HTTP_PORT PREFILL_PORT ROUTER_PORT \ DECODE_WAIT PREFILL_WAIT ROUTER_WAIT SKIP_CONTAINER_BARRIER \ - CONTAINER_BARRIER_TIMEOUT TILERT_PD_BUFFER_DEVICE - -# TILERT_PD_BUFFER_DEVICE selects the patched host-buffer path (cpu) or the -# shipped wheel (cuda); anything else would only be rejected by decode_server's -# argparse after weight conversion. -case "$TILERT_PD_BUFFER_DEVICE" in - cpu|cuda) ;; - *) echo "ERROR: TILERT_PD_BUFFER_DEVICE must be 'cpu' or 'cuda' (got '$TILERT_PD_BUFFER_DEVICE')" >&2; exit 1 ;; -esac + CONTAINER_BARRIER_TIMEOUT LOG_DIR="/run_logs/slurm_job-${SLURM_JOB_ID}" SHARED_LOG_DIR="${BENCHMARK_LOGS_DIR}/logs/slurm_job-${SLURM_JOB_ID}" @@ -105,7 +97,7 @@ host_name=$(hostname) echo "[tilert] ROLE=$TILERT_ROLE rank=$NODE_RANK host=$host_name ($host_ip)" echo "[tilert] PREFILL_HOST=$PREFILL_HOST:$PREFILL_PORT DECODE_HOST=$DECODE_HOST:$DECODE_CTRL_PORT/$DECODE_HTTP_PORT ROUTER=:$ROUTER_PORT" -echo "[tilert] MODEL_PATH=$MODEL_PATH profile=$TILERT_PROFILE served=$SERVED_MODEL_NAME max_len=$TILERT_MAX_MODEL_LEN transport=$TILERT_TRANSPORT kv=${PREFILL_KV_DTYPE}->${DECODE_KV_DTYPE} mtp=${SPEC_DECODING} pd_buffers=$TILERT_PD_BUFFER_DEVICE agentic=$TILERT_IS_AGENTIC" +echo "[tilert] MODEL_PATH=$MODEL_PATH profile=$TILERT_PROFILE served=$SERVED_MODEL_NAME max_len=$TILERT_MAX_MODEL_LEN transport=$TILERT_TRANSPORT kv=${PREFILL_KV_DTYPE}->${DECODE_KV_DTYPE} mtp=${SPEC_DECODING} agentic=$TILERT_IS_AGENTIC" # Enable libibverbs fork safety on both ranks before any verbs context exists. # Without it, ibv_fork_init() can fail in these containers while Mooncake @@ -280,18 +272,13 @@ PYEOF unset TILERT_SIMULATE_ACC_LEN TILERT_SIMULATE_ACC_METHOD echo "[decode] real MTP verification (no simulated acceptance)" fi - # The shipped 0.1.6 decode_server has no --pd-buffer-device (strict - # parse_args); setup_deps.sh applies the PD-DRAM patch only for cpu, so - # pass the flag only then and let cuda run the wheel exactly as shipped. - local pd_buf=() - [[ "$TILERT_PD_BUFFER_DEVICE" == "cpu" ]] && pd_buf=(--pd-buffer-device cpu) local cmd=("$PY" -m tilert.pd_vllm.decode_server --engine tilert --model "$TILERT_PROFILE" --model-weights-dir "$TILERT_WEIGHTS_DIR" --max-seq-len "$TILERT_MAX_MODEL_LEN" --kv-cache-dtype "$DECODE_KV_DTYPE" --transport "$TILERT_TRANSPORT" --ctrl-port "$DECODE_CTRL_PORT" --http-port "$DECODE_HTTP_PORT" - "${pd_buf[@]}" "${DECODE_MTP[@]}" "${extra[@]}") + "${DECODE_MTP[@]}" "${extra[@]}") log_and_run_bg decode "$LOG_DIR/decode_${host_name}.log" "${cmd[@]}" DECODE_PID=$LAST_BG_PID } @@ -306,8 +293,8 @@ start_prefill() { # shellcheck disable=SC2206 local extra=( ${TILERT_PREFILL_EXTRA_FLAGS} ) local kv_cfg - kv_cfg=$(printf '{"kv_connector":"TileRTConnector","kv_connector_module_path":"tilert.pd_vllm.prefill_connector","kv_role":"kv_producer","kv_connector_extra_config":{"tilert_host":"%s","tilert_ctrl_port":%s,"tilert_model":"%s","tilert_max_seq_len":%s,"tilert_transport":"%s","tilert_pd_buffer_device":"%s"}}' \ - "$DECODE_HOST" "$DECODE_CTRL_PORT" "$TILERT_PROFILE" "$TILERT_MAX_MODEL_LEN" "$TILERT_TRANSPORT" "$TILERT_PD_BUFFER_DEVICE") + kv_cfg=$(printf '{"kv_connector":"TileRTConnector","kv_connector_module_path":"tilert.pd_vllm.prefill_connector","kv_role":"kv_producer","kv_connector_extra_config":{"tilert_host":"%s","tilert_ctrl_port":%s,"tilert_model":"%s","tilert_max_seq_len":%s,"tilert_transport":"%s"}}' \ + "$DECODE_HOST" "$DECODE_CTRL_PORT" "$TILERT_PROFILE" "$TILERT_MAX_MODEL_LEN" "$TILERT_TRANSPORT") local cmd=(vllm serve "$MODEL_PATH" --served-model-name "${served[@]}" --port "$PREFILL_PORT" --tensor-parallel-size "$PREFILL_TP_SIZE" --max-model-len "$TILERT_MAX_MODEL_LEN" diff --git a/benchmarks/multi_node/amd_utils/setup_deps.sh b/benchmarks/multi_node/amd_utils/setup_deps.sh index aa36989afc..8adb438681 100644 --- a/benchmarks/multi_node/amd_utils/setup_deps.sh +++ b/benchmarks/multi_node/amd_utils/setup_deps.sh @@ -92,18 +92,18 @@ _tilert_install_missing() { install_tilert_container_tools() { if command -v ip >/dev/null 2>&1 && command -v curl >/dev/null 2>&1 \ - && command -v ibv_devices >/dev/null 2>&1 && command -v patch >/dev/null 2>&1; then + && command -v ibv_devices >/dev/null 2>&1; then echo "[SETUP] Container RDMA/net tools already present" return 0 fi - echo "[SETUP] Installing iproute2 + curl + patch + ibverbs userspace in container..." + echo "[SETUP] Installing iproute2 + curl + ibverbs userspace in container..." apt-get update -q -y && apt-get install -q -y --no-install-recommends \ - iproute2 curl patch ibverbs-utils libibverbs1 librdmacm1 ibverbs-providers \ + iproute2 curl ibverbs-utils libibverbs1 librdmacm1 ibverbs-providers \ && rm -rf /var/lib/apt/lists/* - if ! command -v ip >/dev/null 2>&1 || ! command -v curl >/dev/null 2>&1 || ! command -v patch >/dev/null 2>&1; then - echo "[SETUP] ERROR: failed to install iproute2/curl/patch"; exit 1 + if ! command -v ip >/dev/null 2>&1 || ! command -v curl >/dev/null 2>&1; then + echo "[SETUP] ERROR: failed to install iproute2/curl"; exit 1 fi - _SETUP_INSTALLED+=("iproute2+curl+patch+ibverbs") + _SETUP_INSTALLED+=("iproute2+curl+ibverbs") } _tilert_install_wheel() { @@ -127,37 +127,9 @@ _tilert_install_wheel() { _SETUP_INSTALLED+=("$TILERT_PACKAGE==$TILERT_VERSION($mode)") } -# PD buffers in DRAM. TileRT 0.1.6 allocates the mooncake receive buffer (decode) -# and the connector staging buffer (prefill TP rank 0) on the GPU, dense in -# max_seq_len: 99.06 GiB each at 1048576 tokens, which does not fit next to the -# weights and the engine cache window. The patch moves both into 2 MiB-backed -# pinned host memory behind decode_server --pd-buffer-device cpu and connector -# extra config tilert_pd_buffer_device, verifying the huge-page backing because -# the ionic RDMA VFs cap 4 KiB-page registrations at ~3.9 GiB per HCA. -# Engine-patch waiver: docs/waiver/3330.md. Applied only when the recipe asks -# for host buffers; with TILERT_PD_BUFFER_DEVICE=cuda the wheel runs as shipped. -_TILERT_PD_DRAM_PATCH="$(dirname "${BASH_SOURCE[0]}")/patches/tilert-0.1.6-pd-buffers-in-dram.patch" - -_tilert_apply_pd_dram_patch() { - [[ "$TILERT_PD_BUFFER_DEVICE" == "cpu" ]] || { echo "[SETUP] PD buffers on GPU (TILERT_PD_BUFFER_DEVICE=$TILERT_PD_BUFFER_DEVICE); tilert unpatched"; return 0; } - local dir - dir="$("$PY" -c 'import os, tilert.pd_vllm as m; print(os.path.dirname(m.__file__))')" || { echo "[SETUP] ERROR: cannot locate tilert.pd_vllm"; exit 1; } - if grep -q "def alloc_pinned_huge" "$dir/transport.py"; then - echo "[SETUP] tilert PD-buffers-in-DRAM patch already applied in $dir" - return 0 - fi - [[ -f "$_TILERT_PD_DRAM_PATCH" ]] || { echo "[SETUP] ERROR: missing $_TILERT_PD_DRAM_PATCH"; exit 1; } - echo "[SETUP] applying $(basename "$_TILERT_PD_DRAM_PATCH") to $dir (waiver docs/waiver/3330.md)" - patch -p1 -s -N -d "$dir" < "$_TILERT_PD_DRAM_PATCH" || { echo "[SETUP] ERROR: patch failed to apply"; exit 1; } - "$PY" -m py_compile "$dir"/*.py "$dir"/profiles/*.py || { echo "[SETUP] ERROR: patched tilert.pd_vllm does not compile"; exit 1; } - "$PY" -c 'from tilert.pd_vllm.transport import alloc_pinned_huge' || { echo "[SETUP] ERROR: alloc_pinned_huge missing after patch"; exit 1; } - _SETUP_INSTALLED+=("tilert-pd-buffers-in-dram.patch") -} - install_tilert_decode() { install_tilert_container_tools _tilert_install_wheel full - _tilert_apply_pd_dram_patch _tilert_install_missing uvicorn $TILERT_HTTP_DEPS _tilert_install_missing mooncake.engine "$TILERT_TRANSPORT_DEPS" _tilert_install_missing transformers "$TILERT_TRANSFORMERS_SPEC" @@ -177,7 +149,6 @@ install_tilert_prefill() { echo "[SETUP] prefill-side vLLM $vllm_v" install_tilert_container_tools _tilert_install_wheel no-deps - _tilert_apply_pd_dram_patch _tilert_install_missing mooncake.engine "$TILERT_TRANSPORT_DEPS" "$PY" -c "import tilert.pd_vllm.prefill_connector" 2>/dev/null || { echo "[SETUP] WARN: import tilert.pd_vllm.prefill_connector failed (vLLM will report again when loading the connector plugin):" @@ -194,11 +165,7 @@ if [[ "$ENGINE" == "vllm-disagg" ]]; then export PATH="${UCX_HOME}/bin:/usr/local/bin/etcd:/root/.cargo/bin:${PATH}" export LD_LIBRARY_PATH="${UCX_HOME}/lib:${RIXL_HOME}/lib:${RIXL_HOME}/lib/x86_64-linux-gnu:${LD_LIBRARY_PATH:-}" elif [[ "$ENGINE" == "tilert" ]]; then - check_env_vars TILERT_VERSION TILERT_PD_BUFFER_DEVICE - if [[ "$TILERT_PD_BUFFER_DEVICE" == "cpu" && "$TILERT_VERSION" != "0.1.6" ]]; then - echo "[SETUP] ERROR: $_TILERT_PD_DRAM_PATCH targets tilert 0.1.6, got $TILERT_VERSION; drop the patch, this check and TILERT_PD_BUFFER_DEVICE when bumping to a release with DRAM PD buffers" - exit 1 - fi + check_env_vars TILERT_VERSION TILERT_PIP_SPEC="$TILERT_PACKAGE==$TILERT_VERSION" _tilert_resolve_python case "${TILERT_ROLE:-}" in diff --git a/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh b/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh index 4c76c81b1b..9fafbed1d4 100644 --- a/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh +++ b/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh @@ -57,7 +57,8 @@ export SERVED_MODEL_NAME=glm5_2 # TileRT recipe this mirrors.) # # Memory at this context, per rank, bf16 wire layout (verified against the -# tilert 0.1.6 and vLLM 0.24.0 sources and the MI355X logs, 287.98 GiB cards): +# tilert 0.1.6 and vLLM 0.24.0 sources and the MI355X logs, 287.98 GiB cards; +# the undivided PD buffer sizes are what 0.1.6 allocated on one card): # decode : weights 90.72 GiB + engine cache window 93.25 GiB # + PD receive buffer 99.06 GiB (receive_server.py, dense in max_seq_len) # prefill: weights 90.45 GiB + profiling/non-torch 40.3 GiB + vLLM KV 91.71 GiB @@ -73,10 +74,8 @@ export SERVED_MODEL_NAME=glm5_2 # leaving them on cuda:0. Measured on 2x8 MI350X at this context with bf16 KV: # decode peaks at 202.1 GiB per card, prefill at 269.2 GiB of 287.69 GiB, and # the KV path stays device-to-device at 108 GB/s (81 GB in 751 ms, 54% of the -# 4x400 GbE line rate). No host hop, so the DRAM patch and -# TILERT_PD_BUFFER_DEVICE are not used here. +# 4x400 GbE line rate). No host hop and no patch: the wheel runs as shipped. export TILERT_MAX_MODEL_LEN=1048576 -export TILERT_PD_BUFFER_DEVICE=cuda export TILERT_TRANSPORT=mooncake export TILERT_PARSER=none export TILERT_RDMA_STRICT=0 @@ -96,12 +95,13 @@ export PREFILL_KV_DTYPE=bfloat16 # the connector's KI plane copy fail and MLA address the wrong rows. export PREFILL_BLOCK_SIZE=64 export DECODE_KV_DTYPE=bf16 -# With the PD staging buffer in host memory, vLLM needs 90.45 (weights) + 40.3 -# (profiling) + 91.71 GiB (KV for one 1048576-token request) = 222.5 GiB inside -# its budget: 0.85 x 287.98 = 244.8 GiB leaves 22 GiB of KV margin and 43 GiB -# outside the budget for the ~6.3 GiB non-torch baseline measured on the decode -# OOM node (287.98 - 95.94 free - 184.17 - 1.58 reserved). 0.75 (216 GiB) refuses -# with "91.71 GiB KV cache is needed ... available 85.25 GiB". +# The PD staging shard sits outside vLLM's budget, so vLLM needs 90.45 (weights) +# + 40.3 (profiling) + 91.71 GiB (KV for one 1048576-token request) = 222.5 GiB +# inside it: 0.85 x 287.98 = 244.8 GiB leaves 22 GiB of KV margin and 43 GiB +# outside the budget for the 12.54 GiB staging shard plus the ~6.3 GiB non-torch +# baseline measured on the decode OOM node (287.98 - 95.94 free - 184.17 - 1.58 +# reserved). 0.75 (216 GiB) refuses with "91.71 GiB KV cache is needed ... +# available 85.25 GiB". export GPU_MEM_UTIL=0.85 export SKIP_CONTAINER_BARRIER=0 # Two images, one per rank, ~32 GB each. On a node that has neither cached the diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 966a3ef99b..3c65eedbf6 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1487,7 +1487,7 @@ glm5.3-fp8-mi355x-tilert: runner: cluster:mi355x-amds precision: fp8 framework: tilert - router: { name: tilert-pd-router, version: "0.1.6" } + router: { name: tilert-pd-router, version: "0.1.6.post1" } multinode: true disagg: true kv-p2p-transfer: mooncake @@ -1536,7 +1536,7 @@ glm5.3-fp8-mi355x-tilert-agentic: runner: cluster:mi355x-amds precision: fp8 framework: tilert - router: { name: tilert-pd-router, version: "0.1.6" } + router: { name: tilert-pd-router, version: "0.1.6.post1" } multinode: true disagg: true kv-p2p-transfer: mooncake diff --git a/docs/waiver/3330.md b/docs/waiver/3330.md deleted file mode 100644 index 7f288396bb..0000000000 --- a/docs/waiver/3330.md +++ /dev/null @@ -1,85 +0,0 @@ -# Inference-engine patch waiver — PR #3330 - -Filed per [`docs/PR_REVIEW_CHECKLIST.md`](../PR_REVIEW_CHECKLIST.md) (image-provenance / patch item): the -shared TileRT launcher in this PR patches the TileRT PD (prefill/decode) data plane — the pip-installed -`tilert==0.1.6` wheel's `pd_vllm` package (transport, receive server, decode CLI, prefill connector and the -MLA profile's `convert()`) — inside the containers at start, before serving, which the checklist prohibits unless covered by a filled-out waiver. -vLLM is **not** patched; it runs exactly as shipped in `ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6`. - -## Config covered - -- **Master config entries:** `glm5.3-fp8-mi355x-tilert` and `glm5.3-fp8-mi355x-tilert-agentic` in - [`configs/amd-master.yaml`](../../configs/amd-master.yaml) -- **Pinned images:** `ghcr.io/tile-ai/tilert-rocm-decode:0.1.6` (decode rank) and - `ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6` (prefill rank); neither carries the TileRT wheel, which - [`benchmarks/multi_node/amd_utils/setup_deps.sh`](../../benchmarks/multi_node/amd_utils/setup_deps.sh) - pip-installs as `tilert==0.1.6` from PyPI at container start. -- **Patch entrypoint:** `_tilert_apply_pd_dram_patch` in `setup_deps.sh`, called from - `install_tilert_decode` and `install_tilert_prefill`, only when the recipe sets - `TILERT_PD_BUFFER_DEVICE=cpu`. Marker-gated (skips if `alloc_pinned_huge` is already present) and - pinned to `TILERT_VERSION=0.1.6`; any other version fails loudly. -- **Patch file:** - [`benchmarks/multi_node/amd_utils/patches/tilert-0.1.6-pd-buffers-in-dram.patch`](../../benchmarks/multi_node/amd_utils/patches/tilert-0.1.6-pd-buffers-in-dram.patch) - (5 pure-Python files under `tilert/pd_vllm/`; no compiled component is touched). - -## What is patched - -| file | change | -|---|---| -| `pd_vllm/transport.py` | `Transport.register(..., host=False)` on the mooncake and nixl backends (nixl registers `DRAM` instead of `VRAM` when `host`); new `alloc_pinned_huge(total)` that allocates `mmap`+`MADV_HUGEPAGE` memory, populates it (`MADV_POPULATE_WRITE`), **verifies from `/proc/self/smaps` that the whole region is 2 MiB-backed and raises otherwise**, host-registers it with the GPU runtime through `torch.cuda.cudart().cudaHostRegister` (hipified on ROCm; raising on a non-zero status), and returns a pinned `torch` tensor over a 2 MiB-aligned mapping | -| `pd_vllm/receive_server.py` | `ReceiveServer(..., buffer_device='cuda:0')`; with `'cpu'` the mooncake receive buffer comes from `alloc_pinned_huge` instead of `torch.zeros(..., device='cuda:0')` | -| `pd_vllm/decode_server.py` | new flag `--pd-buffer-device {cuda,cpu}` (default `cuda`) forwarded to `ReceiveServer` | -| `pd_vllm/prefill_connector.py` | connector extra config `tilert_pd_buffer_device` (default `cuda`); with `cpu` the TP-rank-0 staging buffer comes from `alloc_pinned_huge` instead of `torch.zeros(..., device='cuda:N')` | -| `pd_vllm/profiles/mla_nsa.py` | `convert()` moves each received plane to the device when the receive buffer is host-resident (the copy scales with the request's `seq_len`, not `max_seq_len`) | - -Nothing changes in the wire layout, the MLA/KI dequantization, the engine, the router, or vLLM. With the -default `TILERT_PD_BUFFER_DEVICE=cuda` the patch is not applied and the wheel runs as shipped. - -## Why the unmodified upstream stack cannot run this benchmark - -The recipe serves GLM-5.3's full `max_position_embeddings` of 1,048,576 tokens. TileRT 0.1.6 -preallocates two PD buffers dense in `max_seq_len` on the GPU, `buffer_bytes = 79 layers × (1024 + 128 + -132) B/token = 99.06 GiB` each at that context (`profiles/mla_nsa.py`), and `--kv-cache-dtype` cannot -shrink them on ROCm: TileRT accepts only the 1152 B/token bf16 or 656 B/token `fp8_ds_mla` cache stride, -and vLLM's `ROCM_AITER_MLA_SPARSE` backend has no `fp8_ds_mla` (its plain `fp8` writes 576 B/token, which -`classify_layers` rejects). Measured per rank on 287.98 GiB MI355X (sweep jobs 45280/45281): - -- **decode:** weights 90.72 GiB + engine cache window 93.25 GiB + receive buffer 99.06 GiB ≈ 283 GiB — - `torch.OutOfMemoryError` on one node (99.06 GiB requested, 95.94 GiB free), served on another: - node-marginal. -- **prefill rank 0:** weights 90.45 GiB + profiling/non-torch 40.29 GiB + vLLM KV 91.71 GiB inside the - `gpu-memory-utilization` budget, plus the 99.06 GiB staging buffer **outside** it = 321.5 GiB — cannot - fit at any utilization (at utilization 0.75 vLLM refused: "91.71 GiB KV cache is needed ... available - 85.25 GiB"; at 0.85 the KV fits and the 99.06 GiB staging allocation is what fails). - -With both PD buffers in host memory the decode rank needs ≈ 192 GiB (the measured pre-buffer footprint on -the OOM node, 287.98 − 95.94 free) and prefill needs 222.5 GiB inside a 0.85 × 287.98 = 244.8 GiB budget. -Known ceiling, unchanged from the shipped code: `convert()` materialises every layer's planes on the GPU -until `inject()`, so a single prompt near the full 1M tokens (~108 GiB of transients) would still OOM the -decode rank; the AgentX corpus and the 8k1k scenario are far below that. The host buffers must be 2 MiB-backed: the Pensando ionic RDMA VFs on -this pool exhaust a ~2²⁰ page-table-entry budget per HCA, so `ibv_reg_mr` of 4 KiB-page host memory fails -past 3968 MiB (EINVAL/ENOMEM), while 2 MiB-backed regions of 100 GiB register in one MR; this was -measured with raw `libibverbs` and confirmed end to end with a cross-node mooncake host→host write of -8 GiB at 20.9 GiB/s into a 100 GiB registered host buffer (data verified). Registering in <4 GiB pieces -does not help — the budget is per HCA, not per MR. Hence the `smaps` verification in `alloc_pinned_huge`. - -## Upstream PR / issue links - -- TileRT upstream: https://github.com/tile-ai/TileRT — the TileRT team is implementing DRAM-resident PD - buffers; **upstream PR pending** (to be linked here when opened). The patch as applied, with the - measurements above, is posted on this PR: - https://github.com/SemiAnalysisAI/InferenceX/pull/3330#issuecomment-5770977844 and - https://github.com/SemiAnalysisAI/InferenceX/pull/3330#issuecomment-5771627506. -- vLLM: no change requested; `fp8_ds_mla` support in the ROCm sparse-MLA backend would be an alternative - route (halving the wire layout) but is not required for this waiver. - -## Removal plan - -When a TileRT release carries DRAM-resident PD buffers (with the huge-page verification or an -equivalent), in one PR: bump `TILERT_VERSION` and both images in the two recipes to it and replace -`TILERT_PD_BUFFER_DEVICE` with the release's own knob if it differs; delete -`patches/tilert-0.1.6-pd-buffers-in-dram.patch`, `_TILERT_PD_DRAM_PATCH`, `_tilert_apply_pd_dram_patch`, -its two call sites and the `TILERT_VERSION == 0.1.6` pin check in `setup_deps.sh` (the `patch` package -in `install_tilert_container_tools` can go too); drop the `--pd-buffer-device` flag, the -`tilert_pd_buffer_device` connector key and the value check from `server_tilert.sh`, and the -`TILERT_PD_BUFFER_DEVICE` forwarding/validation from `job.slurm`; and delete this waiver. From 9c2e6f7751c21332f75ea1e95910ed08b5e5b5b8 Mon Sep 17 00:00:00 2001 From: CrimsonDump Date: Tue, 22 Sep 2026 22:28:02 +0800 Subject: [PATCH 17/25] =?UTF-8?q?fix(tilert):=20run=20GSM8K=20on=20the=20A?= =?UTF-8?q?gentX=20path=20under=20EVAL=5FONLY/RUN=5FEVAL=20and=20export=20?= =?UTF-8?q?PORT=20for=20lm-eval=20[skip-sweep]=20/=20AgentX=20=E8=B7=AF?= =?UTF-8?q?=E5=BE=84=E5=9C=A8=20EVAL=5FONLY/RUN=5FEVAL=20=E4=B8=8B?= =?UTF-8?q?=E8=BF=90=E8=A1=8C=20GSM8K=EF=BC=8C=E5=B9=B6=E4=B8=BA=20lm-eval?= =?UTF-8?q?=20=E5=AF=BC=E5=87=BA=20PORT?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit server_tilert.sh dispatched every TILERT_IS_AGENTIC=1 job to run_agentic_replay and ignored RUN_EVAL/EVAL_ONLY there. The eval rows the all-evals label generates for a conc-1 agentic config are eval-only, so such a job would have replayed AgentX traces for DURATION seconds, produced no results*.json and failed the template's eval-only check. The fixed-sequence path had a second gap: benchmark_lib's run_lm_eval does check_env_vars OPENAI_API_KEY PORT, and only run_agentic_replay exported PORT, so run_lm_eval_on_router would have exited at that check. - The prefill/router role now sends agentic EVAL_ONLY jobs through run_bench_and_eval (which already skips the throughput loop under EVAL_ONLY and runs the eval), and runs run_lm_eval_on_router after a replay when RUN_EVAL is set. - run_lm_eval_on_router exports PORT=$ROUTER_PORT. lm-eval names the model from MODEL_NAME, which start_prefill already registers with vLLM next to SERVED_MODEL_NAME, so no model-name change is needed; MODEL stays the local HF directory for compute_eval_context_length. - Artifact staging mirrors server_sglang.sh: rewrite_lm_eval_meta_env after the TP/EP/CONC exports, then append_lm_eval_summary unless run_eval already staged the agentic eval-only shape itself. start_decode already disables the simulated acceptance length under RUN_EVAL/EVAL_ONLY, so evals run with real MTP verification as on the SGLang path. [skip-sweep]: the changelog/MODELS commit that follows is the head the sweep should run on. server_tilert.sh 此前把所有 TILERT_IS_AGENTIC=1 的作业都交给 run_agentic_replay， 并在该路径忽略 RUN_EVAL/EVAL_ONLY。all-evals 标签为并发 1 的 agentic 配置生成的评测行 是 eval-only 的，这样的作业会回放 AgentX 轨迹 DURATION 秒、不产生 results*.json， 并在模板的 eval-only 检查处失败。固定序列路径还有另一处缺口：benchmark_lib 的 run_lm_eval 会 check_env_vars OPENAI_API_KEY PORT，而只有 run_agentic_replay 导出了 PORT，run_lm_eval_on_router 会在该检查处退出。 - prefill/router 角色现将 agentic 的 EVAL_ONLY 作业交给 run_bench_and_eval（它已在 EVAL_ONLY 下跳过吞吐循环并执行评测），并在设置 RUN_EVAL 时于回放结束后调用 run_lm_eval_on_router。 - run_lm_eval_on_router 导出 PORT=$ROUTER_PORT。lm-eval 以 MODEL_NAME 命名模型，而 start_prefill 已将其与 SERVED_MODEL_NAME 一并注册到 vLLM，无需改动模型名；MODEL 仍为本地 HF 目录，供 compute_eval_context_length 使用。 - 产物暂存对齐 server_sglang.sh：在 TP/EP/CONC 导出之后 rewrite_lm_eval_meta_env， 再 append_lm_eval_summary，除非 run_eval 已自行暂存 agentic eval-only 这一形态。 start_decode 已在 RUN_EVAL/EVAL_ONLY 下关闭模拟接受长度，因此评测与 SGLang 路径一样 使用真实 MTP 验证。[skip-sweep]：随后的变更记录/MODELS 提交才是 sweep 应运行的 head。 --- .../multi_node/amd_utils/server_tilert.sh | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/benchmarks/multi_node/amd_utils/server_tilert.sh b/benchmarks/multi_node/amd_utils/server_tilert.sh index b579cdf947..daa3d488ec 100644 --- a/benchmarks/multi_node/amd_utils/server_tilert.sh +++ b/benchmarks/multi_node/amd_utils/server_tilert.sh @@ -413,6 +413,10 @@ run_lm_eval_on_router() { else export EVAL_CONCURRENT_REQUESTS=$(echo "$BENCH_MAX_CONCURRENCY" | tr 'x' '\n' | sort -n | tail -1) fi + # run_lm_eval reads the endpoint from PORT (check_env_vars) and names the + # model from MODEL_NAME, which the vLLM prefill also serves next to + # SERVED_MODEL_NAME; MODEL stays the local HF dir for the context lookup. + export PORT="$ROUTER_PORT" export MODEL="$MODEL_PATH" export MAX_MODEL_LEN="$TILERT_MAX_MODEL_LEN" if [[ "$DRY_RUN" -eq 1 ]]; then @@ -429,7 +433,12 @@ run_lm_eval_on_router() { export DECODE_TP="${DECODE_TP_SIZE}" DECODE_EP=1 DECODE_NUM_WORKERS="${yD}" export DP_ATTENTION=false PREFILL_DP_ATTENTION=false DECODE_DP_ATTENTION=false export ISL="${BENCH_INPUT_LEN}" OSL="${BENCH_OUTPUT_LEN}" - append_lm_eval_summary + # As on the SGLang path: rewrite meta_env.json from the exports above, + # then stage unless run_eval already did (agentic eval-only). + rewrite_lm_eval_meta_env + if [[ "$EVAL_ONLY" != "true" || "$TILERT_IS_AGENTIC" != "1" ]]; then + append_lm_eval_summary + fi fi local eval_copy_dir="$LOG_DIR/eval_results" if stage_eval_artifacts "$eval_copy_dir" /workspace "${EVAL_RESULT_DIR:-}"; then @@ -574,8 +583,14 @@ case "$TILERT_ROLE" in fi echo "Ready for benchmarking on ${host_name}:${host_ip}" cd "$WS_PATH" || exit 1 - if [[ "$TILERT_IS_AGENTIC" == "1" ]]; then + # EVAL_ONLY skips the AgentX replay and runs GSM8K on the same router; + # RUN_EVAL after a replay runs it once the replay has finished. The + # fixed-sequence path handles both flags inside run_bench_and_eval. + if [[ "$TILERT_IS_AGENTIC" == "1" && "$EVAL_ONLY" != "true" ]]; then run_agentic_replay; BENCH_RC=$? + if [[ "$RUN_EVAL" == "true" ]]; then + run_lm_eval_on_router || BENCH_RC=1 + fi else run_bench_and_eval; BENCH_RC=$? fi From 66dcb7496679e4694187bafef31b0e59a745c97d Mon Sep 17 00:00:00 2001 From: CrimsonDump Date: Tue, 22 Sep 2026 22:30:56 +0800 Subject: [PATCH 18/25] =?UTF-8?q?chore(tilert):=20perf-changelog=20entry?= =?UTF-8?q?=20and=20MODELS.md=20link=20for=20the=200.1.6.post1=20sharded-P?= =?UTF-8?q?D=20recipes=20(#3366)=20/=20=E4=B8=BA=200.1.6.post1=20=E5=88=86?= =?UTF-8?q?=E7=89=87=20PD=20=E9=85=8D=E6=96=B9=E8=BF=BD=E5=8A=A0=E5=8F=98?= =?UTF-8?q?=E6=9B=B4=E8=AE=B0=E5=BD=95=E5=B9=B6=E6=9B=B4=E6=96=B0=20MODELS?= =?UTF-8?q?.md=20=E9=93=BE=E6=8E=A5=EF=BC=88#3366=EF=BC=89?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Appends one perf-changelog.yaml entry covering glm5.3-fp8-mi355x-tilert and glm5.3-fp8-mi355x-tilert-agentic for this PR: tilert 0.1.6.post1's layer-sharded PD buffers in place of the PD-DRAM patch, the removal of the patch, waiver and TILERT_PD_BUFFER_DEVICE, the router metadata bump and the AgentX-path eval fix, with CrimsonDump's 2x8 MI350X measurements attributed. Byte-preserving tail append; historical entries (including the #3330 ones this branch carries) are untouched. MODELS.md / MODELS_zh.md point the GLM-5.3 row at #3366, the PR that would introduce the model if this branch merges instead of #3330. 为本 PR 在 perf-changelog.yaml 末尾追加一条覆盖 glm5.3-fp8-mi355x-tilert 与 glm5.3-fp8-mi355x-tilert-agentic 的记录：以 tilert 0.1.6.post1 的按层分片 PD 缓冲 取代 PD-DRAM 补丁，移除补丁、豁免与 TILERT_PD_BUFFER_DEVICE，升级 router 元数据， 并修复 AgentX 路径的评测；所引 2x8 MI350X 数据注明出自 CrimsonDump。按字节保留原 文件尾部追加；历史条目（含本分支携带的 #3330 条目）不作改动。MODELS.md / MODELS_zh.md 的 GLM-5.3 行改为指向 #3366，即若本分支而非 #3330 合入时引入该模型的 PR。 --- MODELS.md | 2 +- MODELS_zh.md | 2 +- perf-changelog.yaml | 8 ++++++++ 3 files changed, 10 insertions(+), 2 deletions(-) diff --git a/MODELS.md b/MODELS.md index 347f30fc7d..fec7d8f2a6 100644 --- a/MODELS.md +++ b/MODELS.md @@ -155,7 +155,7 @@ Other offloading tiers, including NVMe KV cache offloading, are outside the init | Model architecture class | Prefix | Date added | Active scenarios | Deprecated scenarios | |---|---|---|---|---| | DeepSeek-V4.1-Flash | `dsv41flash` | 2026-09-10 | Agentic coding (vLLM: DSpark, Engram UVA offload; SGLang: DSpark arms added per SKU from 2026-09-17; GPU validation pending) | | -| GLM-5.3 | `glm5.3` | 2026-09-21 ([#3330](https://github.com/SemiAnalysisAI/InferenceX/pull/3330)) | Single-turn 8k1k, Agentic coding (MTP only, per the Deprecation Notice) | Single-turn 1k1k (deprecated for all models before this model was added; never run) | +| GLM-5.3 | `glm5.3` | 2026-09-22 ([#3366](https://github.com/SemiAnalysisAI/InferenceX/pull/3366)) | Single-turn 8k1k, Agentic coding (MTP only, per the Deprecation Notice) | Single-turn 1k1k (deprecated for all models before this model was added; never run) | | Qwen3.8-Flash-Next | `qwen3.8next` | 2026-08-26 ([#2742](https://github.com/SemiAnalysisAI/InferenceX/pull/2742)) | Agentic coding | | | Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | Agentic coding (DSpark may be disabled for better Pareto points) | Standalone non-DSpark A/B baseline (not required from day 0) | | GLM-5.2 | `glm5.2` | 2026-07-18 ([#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)) | Agentic coding (non-MTP points remain eligible under the Pareto policy; see Deprecation Notice) | | diff --git a/MODELS_zh.md b/MODELS_zh.md index 8960c175a7..636c951c99 100644 --- a/MODELS_zh.md +++ b/MODELS_zh.md @@ -155,7 +155,7 @@ InferenceX 支持 SGLang 和 vLLM 双方的维护者，并响应 AI 实验室和 | 模型架构类别 | 前缀 | 加入日期 | 启用场景 | 已弃用场景 | |---|---|---|---|---| | DeepSeek-V4.1-Flash | `dsv41flash` | 2026-09-10 | 智能体编码（DSpark、Engram UVA 卸载；GPU 待验证） | | -| GLM-5.3 | `glm5.3` | 2026-09-21（[#3330](https://github.com/SemiAnalysisAI/InferenceX/pull/3330)） | 单轮 8k1k、智能体编码（仅 MTP，见弃用公告） | 单轮 1k1k（在本模型加入前已对所有模型弃用，从未运行） | +| GLM-5.3 | `glm5.3` | 2026-09-22（[#3366](https://github.com/SemiAnalysisAI/InferenceX/pull/3366)） | 单轮 8k1k、智能体编码（仅 MTP，见弃用公告） | 单轮 1k1k（在本模型加入前已对所有模型弃用，从未运行） | | Qwen3.8-Flash-Next | `qwen3.8next` | 2026-08-26（[#2742](https://github.com/SemiAnalysisAI/InferenceX/pull/2742)） | 智能体编码 | | | Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | 智能体编码（可关闭 DSpark 以获得更优帕累托点） | 独立非 DSpark A/B 基线（自第 0 天起即不要求） | | GLM-5.2 | `glm5.2` | 2026-07-18（[#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)） | 智能体编码（非 MTP 数据点仍可按帕累托策略参与发布；见弃用公告） | | diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 43054457a7..684bd07c4f 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8639,3 +8639,11 @@ - "Add glm5.3* to resolve_trace_source's 1M-context model pattern so GLM-5.3 AgentX uses the unfiltered semianalysis_cc_traces_weka_062126 corpus instead of the 256k-filtered variant. Explicit WEKA_LOADER_OVERRIDE behavior is unchanged." - "将 glm5.3* 加入 resolve_trace_source 的 1M 上下文模型匹配分支，使 GLM-5.3 AgentX 默认使用未截断的 semianalysis_cc_traces_weka_062126 语料，而非 256k 过滤版本。显式 WEKA_LOADER_OVERRIDE 的行为保持不变。" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3330 + +- config-keys: + - glm5.3-fp8-mi355x-tilert + - glm5.3-fp8-mi355x-tilert-agentic + description: + - "Replace the PD-buffers-in-DRAM route with tilert==0.1.6.post1 on both TileRT ranks (PyPI, 2026-09-22): the shipped wheel shards the mooncake receive buffer (decode, receive_server.py) and the connector staging buffer (prefill TP rank 0, prefill_connector.py) by layer across the eight devices (TILERT_PD_SHARDS, default 8, capped at device_count; layer lid on device lid % 8; the glm5_2 profile has 79 layers, so the heaviest shard holds 10 of them: 99.06 GiB x 10/79 = 12.54 GiB per card) instead of one dense 99.06 GiB buffer per role, and both fit on the GPU at the full 1048576-token context without host memory. Each shard is registered with mooncake on its own device, hello_layout carries per-shard bases and rdma_plan writes layer by layer into the matching remote shard, so the KV path is device-to-device again (81 GB in 751 ms = 108 GB/s on 2x8 MI350X per CrimsonDump/InferenceX@56a1e5a, against the 20.9 GiB/s measured for the host-to-host hop) and mla_nsa.convert() dequantises each layer on the shard that received it rather than on cuda:0. patches/tilert-0.1.6-pd-buffers-in-dram.patch, docs/waiver/3330.md and TILERT_PD_BUFFER_DEVICE (recipes, server_tilert.sh, job.slurm, setup_deps.sh, including the 0.1.6 pin guard and the patch apt package) are removed; neither image is modified. Router metadata 0.1.6 -> 0.1.6.post1: pd_router.py is one of the six files post1 changes, its fixed 600 s request timeouts to vLLM and decode become TILERT_PD_HTTP_TIMEOUT_S (default 3600). GPU_MEM_UTIL stays 0.85; the 12.54 GiB staging shard sits outside vLLM's budget next to the ~6.3 GiB non-torch baseline. server_tilert.sh now honours EVAL_ONLY/RUN_EVAL on the AgentX path (GSM8K on the same router instead of an unconditional trace replay) and exports PORT for benchmark_lib's run_lm_eval; evals run with real MTP verification since simulated acceptance is already disabled under those flags. Measured by CrimsonDump on 2x8 MI350X at this context with bf16 KV, 3600 s AgentX at concurrency 1: submission_valid true, 239 successful requests, 0 errors, TTFT p50 5711.8 ms, ITL p50 2.18 ms, peak device memory 202.1 GiB per decode card and 269.2 GiB per prefill card of 287.69 GiB." + - "在两侧 TileRT rank 上以 tilert==0.1.6.post1（PyPI，2026-09-22）取代 PD 缓冲进 DRAM 的方案：正式 wheel 把 mooncake 接收缓冲（decode，receive_server.py）与 connector 暂存缓冲（prefill TP rank 0，prefill_connector.py）按层分片到 8 张卡（TILERT_PD_SHARDS，默认 8，以 device_count 为上限；第 lid 层落在第 lid % 8 张卡；glm5_2 profile 共 79 层，最重分片含 10 层：99.06 GiB x 10/79 = 每卡 12.54 GiB），不再是每个角色一块致密的 99.06 GiB 缓冲，于是在完整 1048576 上下文下两者都能放进 GPU，无需主机内存。各分片在所在卡上向 mooncake 注册，hello_layout 携带各分片基址，rdma_plan 逐层写入对应的远端分片，KV 通路重回设备到设备（2x8 MI350X 上 81 GB / 751 ms = 108 GB/s，见 CrimsonDump/InferenceX@56a1e5a；对比主机到主机实测 20.9 GiB/s），mla_nsa.convert() 在收到该层的分片上反量化而非集中于 cuda:0。移除 patches/tilert-0.1.6-pd-buffers-in-dram.patch、docs/waiver/3330.md 与 TILERT_PD_BUFFER_DEVICE（两份配方、server_tilert.sh、job.slurm、setup_deps.sh，含 0.1.6 版本锁定检查与 patch apt 包）；两侧镜像均无改动。router 元数据 0.1.6 -> 0.1.6.post1：pd_router.py 是 post1 改动的六个文件之一，其对 vLLM 与 decode 固定的 600 秒请求超时改为 TILERT_PD_HTTP_TIMEOUT_S（默认 3600）。GPU_MEM_UTIL 保持 0.85；12.54 GiB 暂存分片与约 6.3 GiB 非 torch 基线同在 vLLM 预算之外。server_tilert.sh 现在在 AgentX 路径也遵循 EVAL_ONLY/RUN_EVAL（在同一 router 上跑 GSM8K，而非无条件回放轨迹），并为 benchmark_lib 的 run_lm_eval 导出 PORT；这些标志下模拟接受已关闭，评测使用真实 MTP 验证。CrimsonDump 在 2x8 MI350X、该上下文、bf16 KV 下实测 3600 秒 AgentX 并发 1：submission_valid true、239 条成功、0 错误、TTFT p50 5711.8 ms、ITL p50 2.18 ms，decode 每卡峰值 202.1 GiB、prefill 每卡 269.2 GiB（卡容量 287.69 GiB）。" + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3366 From a0caa02b9aef91bc5514c471f192d9a8d103727f Mon Sep 17 00:00:00 2001 From: CrimsonDump Date: Wed, 23 Sep 2026 08:57:18 +0800 Subject: [PATCH 19/25] =?UTF-8?q?chore(tilert):=20drop=20the=20GLM-5.3=20M?= =?UTF-8?q?I355X=208k1k=20config=20and=20keep=20AgentX=20only=20/=20?= =?UTF-8?q?=E7=A7=BB=E9=99=A4=20GLM-5.3=20MI355X=208k1k=20=E9=85=8D?= =?UTF-8?q?=E7=BD=AE=EF=BC=8C=E4=BB=85=E4=BF=9D=E7=95=99=20AgentX?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Remove the glm5.3-fp8-mi355x-tilert fixed-seq-len entry, its recipe benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh, and the fixed-sequence bench loop in server_tilert.sh, which now runs agentic-coding only: EVAL_ONLY waits for the router and runs GSM8K, otherwise the AgentX replay runs and RUN_EVAL follows it. The 8k1k changelog entries and MODELS rows are dropped; the entries that named both keys now name the agentic key only. --- MODELS.md | 2 +- MODELS_zh.md | 2 +- .../multi_node/amd_utils/server_tilert.sh | 71 ++------ .../multi_node/glm5.3_fp8_mi355x_tilert.sh | 160 ------------------ benchmarks/multi_node/runtime_settings.sh | 2 +- 5 files changed, 19 insertions(+), 218 deletions(-) delete mode 100644 benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh diff --git a/MODELS.md b/MODELS.md index cdc55ce95b..619e3fefc9 100644 --- a/MODELS.md +++ b/MODELS.md @@ -155,7 +155,7 @@ Other offloading tiers, including NVMe KV cache offloading, are outside the init | Model architecture class | Prefix | Date added | Active scenarios | Deprecated scenarios | |---|---|---|---|---| | DeepSeek-V4.1-Flash | `dsv41flash` | 2026-09-10 | Agentic coding (vLLM: DSpark, Engram UVA offload; SGLang: DSpark arms added per SKU from 2026-09-17; GPU validation pending) | | -| GLM-5.3 | `glm5.3` | 2026-09-22 ([#3366](https://github.com/SemiAnalysisAI/InferenceX/pull/3366)) | Single-turn 8k1k, Agentic coding (MTP only, per the Deprecation Notice) | Single-turn 1k1k (deprecated for all models before this model was added; never run) | +| GLM-5.3 | `glm5.3` | 2026-09-22 ([#3366](https://github.com/SemiAnalysisAI/InferenceX/pull/3366)) | Agentic coding (MTP only, per the Deprecation Notice) | Single-turn 1k1k (deprecated for all models before this model was added; never run) | | Qwen3.8-Flash-Next | `qwen3.8next` | 2026-08-26 ([#2742](https://github.com/SemiAnalysisAI/InferenceX/pull/2742)) | Agentic coding | | | Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | Agentic coding (DSpark may be disabled for better Pareto points) | Standalone non-DSpark A/B baseline (not required from day 0) | | GLM-5.2 | `glm5.2` | 2026-07-18 ([#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)) | Agentic coding (non-MTP points remain eligible under the Pareto policy; see Deprecation Notice) | | diff --git a/MODELS_zh.md b/MODELS_zh.md index 0ff621894f..da8c72b08f 100644 --- a/MODELS_zh.md +++ b/MODELS_zh.md @@ -155,7 +155,7 @@ InferenceX 支持 SGLang 和 vLLM 双方的维护者，并响应 AI 实验室和 | 模型架构类别 | 前缀 | 加入日期 | 启用场景 | 已弃用场景 | |---|---|---|---|---| | DeepSeek-V4.1-Flash | `dsv41flash` | 2026-09-10 | 智能体编码（DSpark、Engram UVA 卸载；GPU 待验证） | | -| GLM-5.3 | `glm5.3` | 2026-09-22（[#3366](https://github.com/SemiAnalysisAI/InferenceX/pull/3366)） | 单轮 8k1k、智能体编码（仅 MTP，见弃用公告） | 单轮 1k1k（在本模型加入前已对所有模型弃用，从未运行） | +| GLM-5.3 | `glm5.3` | 2026-09-22（[#3366](https://github.com/SemiAnalysisAI/InferenceX/pull/3366)） | 智能体编码（仅 MTP，见弃用公告） | 单轮 1k1k（在本模型加入前已对所有模型弃用，从未运行） | | Qwen3.8-Flash-Next | `qwen3.8next` | 2026-08-26（[#2742](https://github.com/SemiAnalysisAI/InferenceX/pull/2742)） | 智能体编码 | | | Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | 智能体编码（可关闭 DSpark 以获得更优帕累托点） | 独立非 DSpark A/B 基线（自第 0 天起即不要求） | | GLM-5.2 | `glm5.2` | 2026-07-18（[#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)） | 智能体编码（非 MTP 数据点仍可按帕累托策略参与发布；见弃用公告） | | diff --git a/benchmarks/multi_node/amd_utils/server_tilert.sh b/benchmarks/multi_node/amd_utils/server_tilert.sh index daa3d488ec..a22cd5c433 100644 --- a/benchmarks/multi_node/amd_utils/server_tilert.sh +++ b/benchmarks/multi_node/amd_utils/server_tilert.sh @@ -9,8 +9,7 @@ source "$(dirname "${BASH_SOURCE[0]}")/../../benchmark_lib.sh" --validation-only check_env_vars \ NODE0_ADDR NODE_RANK MODEL_DIR MODEL_NAME MODEL_PATH xP yD IPADDRS \ DRY_RUN GPUS_PER_NODE PREFILL_TP_SIZE DECODE_TP_SIZE \ - BENCH_INPUT_LEN BENCH_OUTPUT_LEN BENCH_RANDOM_RANGE_RATIO \ - BENCH_REQUEST_RATE BENCH_NUM_PROMPTS_MULTIPLIER BENCH_MAX_CONCURRENCY \ + BENCH_INPUT_LEN BENCH_OUTPUT_LEN BENCH_MAX_CONCURRENCY \ RUN_EVAL EVAL_ONLY EVAL_FRAMEWORK BENCHMARK_LOGS_DIR WS_PATH \ SLURM_JOB_ID SPEC_DECODING \ TILERT_PROFILE TILERT_MODEL_TYPE TILERT_MODEL_PKG TILERT_MAX_MODEL_LEN \ @@ -80,9 +79,10 @@ if [[ "$SPEC_DECODING" == "mtp" ]]; then DECODE_MTP=(--with-mtp --num-mtp "$DECODE_MTP_SIZE") fi -TILERT_IS_AGENTIC=0 -if [[ "${IS_AGENTIC:-0}" == "1" || "${IS_AGENTIC:-}" == "true" || "${SCENARIO_TYPE:-}" == "agentic-coding" ]]; then - TILERT_IS_AGENTIC=1 +# The only TileRT recipe on this cluster is AgentX (agentic-coding). +if [[ "${IS_AGENTIC:-0}" != "1" && "${IS_AGENTIC:-}" != "true" && "${SCENARIO_TYPE:-}" != "agentic-coding" ]]; then + echo "ERROR: server_tilert.sh only runs agentic-coding (IS_AGENTIC=${IS_AGENTIC:-} SCENARIO_TYPE=${SCENARIO_TYPE:-})" >&2 + exit 1 fi IFS=',' read -ra IP_ARRAY <<< "$IPADDRS" @@ -97,7 +97,7 @@ host_name=$(hostname) echo "[tilert] ROLE=$TILERT_ROLE rank=$NODE_RANK host=$host_name ($host_ip)" echo "[tilert] PREFILL_HOST=$PREFILL_HOST:$PREFILL_PORT DECODE_HOST=$DECODE_HOST:$DECODE_CTRL_PORT/$DECODE_HTTP_PORT ROUTER=:$ROUTER_PORT" -echo "[tilert] MODEL_PATH=$MODEL_PATH profile=$TILERT_PROFILE served=$SERVED_MODEL_NAME max_len=$TILERT_MAX_MODEL_LEN transport=$TILERT_TRANSPORT kv=${PREFILL_KV_DTYPE}->${DECODE_KV_DTYPE} mtp=${SPEC_DECODING} agentic=$TILERT_IS_AGENTIC" +echo "[tilert] MODEL_PATH=$MODEL_PATH profile=$TILERT_PROFILE served=$SERVED_MODEL_NAME max_len=$TILERT_MAX_MODEL_LEN transport=$TILERT_TRANSPORT kv=${PREFILL_KV_DTYPE}->${DECODE_KV_DTYPE} mtp=${SPEC_DECODING}" # Enable libibverbs fork safety on both ranks before any verbs context exists. # Without it, ibv_fork_init() can fail in these containers while Mooncake @@ -240,8 +240,7 @@ convert_weights() { start_decode() { # shellcheck disable=SC2206 local extra=( ${TILERT_DECODE_EXTRA_FLAGS} ) - if [[ "$TILERT_IS_AGENTIC" == "1" && "$SPEC_DECODING" == "mtp" \ - && "$EVAL_ONLY" != "true" && "$RUN_EVAL" != "true" ]]; then + if [[ "$SPEC_DECODING" == "mtp" && "$EVAL_ONLY" != "true" && "$RUN_EVAL" != "true" ]]; then check_env_vars MODEL_PREFIX THINKING_MODE local curve="${WS_PATH%/benchmarks/*}/golden_al_distribution/${MODEL_PREFIX}_mtp.yaml" TILERT_SIMULATE_ACC_LEN="$("$PY" - "$curve" "$THINKING_MODE" "$DECODE_MTP_SIZE" <<'PYEOF' @@ -355,46 +354,6 @@ copy_logs_to_shared() { } trap copy_logs_to_shared EXIT -run_bench_and_eval() { - local rc=0 - wait_for_server_ready --port "$ROUTER_PORT" --server-log "$LOG_DIR/router_${host_name}.log" --server-pid "$ROUTER_PID" - - local profile_folder="$LOG_DIR/${ENGINE}_isl_${BENCH_INPUT_LEN}_osl_${BENCH_OUTPUT_LEN}" - mkdir -p "$profile_folder" - local prefill_gpus=$(( PREFILL_TP_SIZE * xP )) - local decode_gpus=$(( DECODE_TP_SIZE * yD )) - export TRANSFORMERS_VERBOSITY=error TOKENIZERS_PARALLELISM=false - - if [[ "$EVAL_ONLY" == "true" ]]; then - echo "EVAL_ONLY mode: skipping throughput benchmark" - else - local conc np export_file - for conc in ${BENCH_MAX_CONCURRENCY//x/ }; do - np=$(( conc * BENCH_NUM_PROMPTS_MULTIPLIER )) - [[ "$np" -lt 16 ]] && np=16 - export_file="${profile_folder}/concurrency_${conc}_req_rate_${BENCH_REQUEST_RATE}_gpus_$((prefill_gpus+decode_gpus))_ctx_${prefill_gpus}_gen_${decode_gpus}" - echo "[bench] conc=$conc num_prompts=$np isl=$BENCH_INPUT_LEN osl=$BENCH_OUTPUT_LEN -> $export_file.json" - run_benchmark_serving \ - --bench-serving-dir /workspace \ - --model "$SERVED_MODEL_NAME" --port "$ROUTER_PORT" \ - --backend openai-chat --endpoint /v1/chat/completions \ - --input-len "$BENCH_INPUT_LEN" --output-len "$BENCH_OUTPUT_LEN" \ - --random-range-ratio "$BENCH_RANDOM_RANGE_RATIO" \ - --num-prompts "$np" --max-concurrency "$conc" \ - --use-chat-template --tokenizer "$MODEL_PATH" --trust-remote-code \ - --server-pid "$ROUTER_PID" \ - --result-filename "$export_file" --result-dir /workspace/ \ - || { rc=$?; echo "[bench] WARNING: conc=$conc failed (rc=$rc)" >&2; } - echo "-----------------------------------------" - done - fi - - if [[ "$RUN_EVAL" == "true" ]]; then - run_lm_eval_on_router || rc=1 - fi - return $rc -} - run_lm_eval_on_router() { echo "Running lm-eval evaluation on the router..." local ok=false _attempt @@ -434,9 +393,9 @@ run_lm_eval_on_router() { export DP_ATTENTION=false PREFILL_DP_ATTENTION=false DECODE_DP_ATTENTION=false export ISL="${BENCH_INPUT_LEN}" OSL="${BENCH_OUTPUT_LEN}" # As on the SGLang path: rewrite meta_env.json from the exports above, - # then stage unless run_eval already did (agentic eval-only). + # then stage unless run_eval already did (eval-only). rewrite_lm_eval_meta_env - if [[ "$EVAL_ONLY" != "true" || "$TILERT_IS_AGENTIC" != "1" ]]; then + if [[ "$EVAL_ONLY" != "true" ]]; then append_lm_eval_summary fi fi @@ -584,15 +543,17 @@ case "$TILERT_ROLE" in echo "Ready for benchmarking on ${host_name}:${host_ip}" cd "$WS_PATH" || exit 1 # EVAL_ONLY skips the AgentX replay and runs GSM8K on the same router; - # RUN_EVAL after a replay runs it once the replay has finished. The - # fixed-sequence path handles both flags inside run_bench_and_eval. - if [[ "$TILERT_IS_AGENTIC" == "1" && "$EVAL_ONLY" != "true" ]]; then + # RUN_EVAL after a replay runs it once the replay has finished. + if [[ "$EVAL_ONLY" == "true" ]]; then + echo "EVAL_ONLY mode: skipping the AgentX replay" + wait_for_server_ready --port "$ROUTER_PORT" --server-log "$LOG_DIR/router_${host_name}.log" --server-pid "$ROUTER_PID" + export TRANSFORMERS_VERBOSITY=error TOKENIZERS_PARALLELISM=false + run_lm_eval_on_router; BENCH_RC=$? + else run_agentic_replay; BENCH_RC=$? if [[ "$RUN_EVAL" == "true" ]]; then run_lm_eval_on_router || BENCH_RC=1 fi - else - run_bench_and_eval; BENCH_RC=$? fi copy_logs_to_shared echo "Killing the router and the prefill server" diff --git a/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh b/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh deleted file mode 100644 index 9fafbed1d4..0000000000 --- a/benchmarks/multi_node/glm5.3_fp8_mi355x_tilert.sh +++ /dev/null @@ -1,160 +0,0 @@ -#!/usr/bin/env bash - -source "$(dirname "$0")/../benchmark_lib.sh" - -check_env_vars \ - CONC_LIST \ - ISL \ - OSL \ - IMAGE \ - SPEC_DECODING \ - MODEL_PATH \ - MODEL_NAME \ - PREFILL_NUM_WORKERS \ - PREFILL_TP \ - PREFILL_EP \ - PREFILL_DP_ATTN \ - DECODE_NUM_WORKERS \ - DECODE_TP \ - DECODE_EP \ - DECODE_DP_ATTN \ - PREFILL_NODES \ - DECODE_NODES \ - RANDOM_RANGE_RATIO \ - FRAMEWORK \ - PREFILL_IMAGE - -if [[ -n "$SLURM_JOB_ID" ]]; then - echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME" -fi - -set -x - -cd "$GITHUB_WORKSPACE/benchmarks/multi_node/amd_utils" || exit 1 - -export TIME_LIMIT=08:00:00 -export MODEL_PATH=$MODEL_PATH -export MODEL_NAME=$MODEL_NAME -export CONTAINER_IMAGE=$IMAGE -export PREFILL_IMAGE - -if [[ "$PREFILL_NODES" -ne 1 || "$DECODE_NODES" -ne 1 || \ - "$PREFILL_NUM_WORKERS" -ne 1 || "$DECODE_NUM_WORKERS" -ne 1 ]]; then - echo "Error: tilert supports exactly 1 prefill node/worker + 1 decode node/worker" \ - "(got PREFILL_NODES=$PREFILL_NODES x$PREFILL_NUM_WORKERS, DECODE_NODES=$DECODE_NODES x$DECODE_NUM_WORKERS)" >&2 - exit 1 -fi - -# TileRT configuration. Every value is explicit here: server_tilert.sh -# validates each one with check_env_vars and supplies no defaults of its own. -export TILERT_VERSION=0.1.6.post1 -export TILERT_PROFILE=glm5_2 # decode_server --model (TileRT model profile) -export TILERT_MODEL_TYPE=glm-5 # weight_converter --model_type (fallback converter) -export TILERT_MODEL_PKG=glm_5_2_rocm # per-model converter package, preferred when importable -export SERVED_MODEL_NAME=glm5_2 -# GLM-5.3's full context window (config.json max_position_embeddings), as every -# in-tree GLM-5.2 recipe serves. (202752 was GLM-5.1's, inherited from the B200 -# TileRT recipe this mirrors.) -# -# Memory at this context, per rank, bf16 wire layout (verified against the -# tilert 0.1.6 and vLLM 0.24.0 sources and the MI355X logs, 287.98 GiB cards; -# the undivided PD buffer sizes are what 0.1.6 allocated on one card): -# decode : weights 90.72 GiB + engine cache window 93.25 GiB -# + PD receive buffer 99.06 GiB (receive_server.py, dense in max_seq_len) -# prefill: weights 90.45 GiB + profiling/non-torch 40.3 GiB + vLLM KV 91.71 GiB -# + PD staging buffer 99.06 GiB (prefill_connector.py, TP rank 0, -# allocated OUTSIDE vLLM's gpu-memory-utilization budget) -# Undivided, neither side starts: the decode rank is node-marginal (~283 of -# 288 GiB) and the prefill rank cannot fit at any utilization (~321 GiB). -# tilert 0.1.6.post1 keeps both buffers on the GPU but shards them by layer -# across the eight devices (layer lid on device lid % 8, TILERT_PD_SHARDS, -# default on), so each card holds 12.54 GiB instead of 99.06 GiB on one. -# convert() dequantises each layer on the device that received it, which spreads -# its transients too (108.6 KiB/token, 82.9 GiB at 800k tokens) instead of -# leaving them on cuda:0. Measured on 2x8 MI350X at this context with bf16 KV: -# decode peaks at 202.1 GiB per card, prefill at 269.2 GiB of 287.69 GiB, and -# the KV path stays device-to-device at 108 GB/s (81 GB in 751 ms, 54% of the -# 4x400 GbE line rate). No host hop and no patch: the wheel runs as shipped. -export TILERT_MAX_MODEL_LEN=1048576 -export TILERT_TRANSPORT=mooncake -export TILERT_PARSER=none -export TILERT_RDMA_STRICT=0 -export TILERT_CONVERT_LOCK_WAIT=21600 -export TILERT_SIMULATE_ACC_METHOD=match-expected -export TILERT_WEIGHTS_DIR="/models/${MODEL_NAME}-tilert-tp${DECODE_TP}" -# bf16 MLA KV on both roles. This is the only layout TileRT 0.1.6 can consume -# from vLLM on ROCm: MlaNsaProfile.classify_layers infers the layout from the -# cache tensor stride and accepts exactly 1152 B/token (bf16) or 656 B/token -# (fp8_ds_mla). vLLM's ROCM_AITER_MLA_SPARSE backend has no fp8_ds_mla; its -# plain "fp8" writes a flat 576 B/token row, which the connector rejects at -# register_kv_caches. Explicit bfloat16 rather than auto so the stride does not -# depend on the model dtype. Never float16: it passes the 1152 B check and is -# then read as bf16. -export PREFILL_KV_DTYPE=bfloat16 -# The ROCm backend supports block sizes [1, 64] and vLLM picks 1, which makes -# the connector's KI plane copy fail and MLA address the wrong rows. -export PREFILL_BLOCK_SIZE=64 -export DECODE_KV_DTYPE=bf16 -# The PD staging shard sits outside vLLM's budget, so vLLM needs 90.45 (weights) -# + 40.3 (profiling) + 91.71 GiB (KV for one 1048576-token request) = 222.5 GiB -# inside it: 0.85 x 287.98 = 244.8 GiB leaves 22 GiB of KV margin and 43 GiB -# outside the budget for the 12.54 GiB staging shard plus the ~6.3 GiB non-torch -# baseline measured on the decode OOM node (287.98 - 95.94 free - 184.17 - 1.58 -# reserved). 0.75 (216 GiB) refuses with "91.71 GiB KV cache is needed ... -# available 85.25 GiB". -export GPU_MEM_UTIL=0.85 -export SKIP_CONTAINER_BARRIER=0 -# Two images, one per rank, ~32 GB each. On a node that has neither cached the -# pull alone outlasts the SGLang path's 300s default and the 1800s this script -# used to hardcode, and the rank that comes up first waits out the whole -# timeout while its peer is still pulling. -export CONTAINER_BARRIER_TIMEOUT=5400 -export ROUTER_PORT=30000 -export PREFILL_PORT=8000 -export DECODE_CTRL_PORT=5556 -export DECODE_HTTP_PORT=5557 -export DECODE_WAIT=7200 # prefill waits for the decode ctrl port -export PREFILL_WAIT=3600 # prefill waits for its own vLLM port -export ROUTER_WAIT=10800 # decode waits for the router port to open - -if [[ "$SPEC_DECODING" == "mtp" ]]; then - # TileRT decode drafts at depth 3 (the only depth the ROCm GLM profile - # builds) and the golden acceptance curve is keyed on it. The vLLM prefill - # rank only has to materialise the MTP layer's KV, so it runs at 1. - export DECODE_MTP_SIZE=3 - export PREFILL_SPEC_TOKENS=1 -else - export DECODE_MTP_SIZE=0 - export PREFILL_SPEC_TOKENS=0 -fi -export TILERT_QUEUE_TIMEOUT=0 -# submit.sh and job.slurm validate KV_OFFLOADING, and the fixed-seq-len matrix -# emits it only for agentic scenarios. TileRT has no KV offload backend. -export KV_OFFLOADING=none - -if [[ "$PREFILL_EP" -ne 1 || "$DECODE_EP" -ne 1 || \ - "$PREFILL_DP_ATTN" == "true" || "$DECODE_DP_ATTN" == "true" ]]; then - echo "Error: tilert runs pure TP8 on both roles; ep must be 1 and dp-attn false" >&2 - exit 1 -fi -export PREFILL_ENABLE_EP=false -export PREFILL_ENABLE_DP=false -export DECODE_ENABLE_EP=false -export DECODE_ENABLE_DP=false - -JOB_ID=$(bash ./submit.sh $PREFILL_NODES \ - $PREFILL_NUM_WORKERS \ - $DECODE_NODES \ - $DECODE_NUM_WORKERS \ - $ISL $OSL "${CONC_LIST// /x}" inf \ - ${PREFILL_ENABLE_EP} ${PREFILL_ENABLE_DP} \ - ${DECODE_ENABLE_EP} ${DECODE_ENABLE_DP} \ - ${PREFILL_TP} ${DECODE_TP} \ - ${RANDOM_RANGE_RATIO}) - -if [[ $? -ne 0 ]]; then - echo "Failed to submit job" >&2 - exit 1 -fi - -echo "$JOB_ID" diff --git a/benchmarks/multi_node/runtime_settings.sh b/benchmarks/multi_node/runtime_settings.sh index 8044fa11aa..725efcdba6 100644 --- a/benchmarks/multi_node/runtime_settings.sh +++ b/benchmarks/multi_node/runtime_settings.sh @@ -60,7 +60,7 @@ case "$FRAMEWORK" in if [[ "$IS_AGENTIC" == 1 || "$IS_AGENTIC" == true ]]; then export TILERT_QUEUE_TIMEOUT=1800 fi - # The MI355X TileRT recipes run through the shared amd_utils chain + # The MI355X TileRT recipe runs through the shared amd_utils chain # (submit.sh -> job.slurm -> server.sh -> setup_deps.sh), which validates # the same orchestration inputs the AMD SGLang/vLLM/ATOM arms receive. # Without them submit.sh exits before sbatch and the launcher never gets From 64d4ff6d6a7acb57b0b102e83342ba339c588323 Mon Sep 17 00:00:00 2001 From: Oseltamivir <58582368+Oseltamivir@users.noreply.github.com> Date: Wed, 23 Sep 2026 10:54:47 +0800 Subject: [PATCH 20/25] feat(tilert): keep AgentX KV prefixes resident on the TileRT decode node [skip-sweep] tilert 0.1.6.post1 resets the decode sequence and copies the whole prompt's KV into all eight rank caches on every request, serially on the default streams. On #3366's AgentX run that inject step is 2.4 s at p50 and 4.3 s at p90, the largest part of TTFT, although 200 of 239 turns extend the previous request's prompt by a few hundred tokens. patches/tilert-0.1.6.post1-pd-prefix-reuse.patch, applied by setup_deps.sh on both ranks when the recipe sets TILERT_PD_PREFIX_REUSE=1: - glm5_rocm_engine.inject keeps the last prompt's ids and KV, and copies only the rows from the page before the first differing token onward. The kept rows are checked byte for byte against the fresh transfer on a sample of positions on every rank and layer; a mismatch falls back to a full copy. The index-key caches are always rewritten in full. - The KV copies fan out on one stream per device pair instead of the default streams, which torch fences on both devices per copy. - receive_server asks for the prompt ids in its hello and the prefill connector sends them when asked; decode_server hands them to inject. --- .../agentic/glm5.3_fp8_mi355x_tilert.sh | 6 + benchmarks/multi_node/amd_utils/job.slurm | 3 +- .../tilert-0.1.6.post1-pd-prefix-reuse.patch | 256 ++++++++++++++++++ benchmarks/multi_node/amd_utils/setup_deps.sh | 47 +++- 4 files changed, 304 insertions(+), 8 deletions(-) create mode 100644 benchmarks/multi_node/amd_utils/patches/tilert-0.1.6.post1-pd-prefix-reuse.patch diff --git a/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh b/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh index a8f6c3ac9f..bff28ca342 100644 --- a/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh +++ b/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh @@ -95,6 +95,12 @@ export TILERT_PARSER=none export TILERT_RDMA_STRICT=0 export TILERT_CONVERT_LOCK_WAIT=21600 export TILERT_SIMULATE_ACC_METHOD=match-expected +# Keep the KV of the previous turn on the decode node and copy in only what the +# new prompt adds (patches/tilert-0.1.6.post1-pd-prefix-reuse.patch, applied by +# setup_deps.sh). At concurrency 1, 200 of 239 AgentX turns continue the +# previous request's conversation, adding 334 tokens at the median; the +# unpatched wheel re-copies the whole context into all eight ranks each turn. +export TILERT_PD_PREFIX_REUSE=1 export TILERT_WEIGHTS_DIR="/models/${MODEL_NAME}-tilert-tp${DECODE_TP}" # bf16 MLA KV on both roles. This is the only layout TileRT 0.1.6 can consume # from vLLM on ROCm: MlaNsaProfile.classify_layers infers the layout from the diff --git a/benchmarks/multi_node/amd_utils/job.slurm b/benchmarks/multi_node/amd_utils/job.slurm index 0c779476ea..de8541a4b2 100755 --- a/benchmarks/multi_node/amd_utils/job.slurm +++ b/benchmarks/multi_node/amd_utils/job.slurm @@ -62,7 +62,7 @@ if [[ "$ENGINE" == "tilert" ]]; then # server_tilert.sh takes the container-creation barrier timeout from the # recipe (no 300s default as on the SGLang path); fail here, before sbatch # work is done, rather than inside the container. - check_env_vars CONTAINER_BARRIER_TIMEOUT + check_env_vars CONTAINER_BARRIER_TIMEOUT TILERT_PD_PREFIX_REUSE fi # Resolve the models.yaml entry the same way server_sglang.sh does: agentic runs @@ -551,6 +551,7 @@ elif [[ "$ENGINE" == "tilert" ]]; then -e TILERT_RDMA_STRICT=${TILERT_RDMA_STRICT} -e TILERT_CONVERT_LOCK_WAIT=${TILERT_CONVERT_LOCK_WAIT} -e TILERT_SIMULATE_ACC_METHOD=${TILERT_SIMULATE_ACC_METHOD} + -e TILERT_PD_PREFIX_REUSE=${TILERT_PD_PREFIX_REUSE} -e \"TILERT_EXTRA_ENV=${TILERT_EXTRA_ENV:-}\" -e SERVED_MODEL_NAME=${SERVED_MODEL_NAME} -e PREFILL_KV_DTYPE=${PREFILL_KV_DTYPE} diff --git a/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6.post1-pd-prefix-reuse.patch b/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6.post1-pd-prefix-reuse.patch new file mode 100644 index 0000000000..50ab6aa1b4 --- /dev/null +++ b/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6.post1-pd-prefix-reuse.patch @@ -0,0 +1,256 @@ +--- a/decode_server.py ++++ b/decode_server.py +@@ -145,6 +145,7 @@ + return JSONResponse({'error': 'kv_transfer_timeout', 'rid': body.rid}, status_code=504) + t_recv = time.time() + conv = server.profile.convert(server.buffer, server.base_ptr, server.max_seq_len, req, server.profile.num_ranks) ++ conv.prompt_token_ids = req.prompt_token_ids + t_conv = time.time() + engine.inject(conv) + t_inj = time.time() +--- a/prefill_connector.py 2026-09-23 10:51:39 ++++ b/prefill_connector.py 2026-09-23 10:52:38 +@@ -26,6 +26,7 @@ + tilert_ctrl_port: int + sampling: dict | None = None + prompt_token_ids: list = field(default_factory=list) ++ wants_ids: bool = False + + @dataclass + class TileRTMetadata(KVConnectorMetadata): +@@ -120,7 +121,7 @@ + return meta + + def _emit(self, req_id, token_ids, groups, params) -> _ReqMeta: +- m = _ReqMeta(req_id=req_id, rid=derive_rid(req_id), num_tokens=len(token_ids), last_prompt_token=int(token_ids[-1]), prompt_token_ids=list(token_ids) if params.get('_wants_prompt_ids') else [], block_ids_per_group=groups, tilert_host=params.get('tilert_host') or self._default_host, tilert_ctrl_port=int(params.get('tilert_ctrl_port', self._default_port)), sampling=params.get('sampling')) ++ m = _ReqMeta(req_id=req_id, rid=derive_rid(req_id), num_tokens=len(token_ids), last_prompt_token=int(token_ids[-1]), prompt_token_ids=list(token_ids), wants_ids=bool(params.get('_wants_prompt_ids')), block_ids_per_group=groups, tilert_host=params.get('tilert_host') or self._default_host, tilert_ctrl_port=int(params.get('tilert_ctrl_port', self._default_port)), sampling=params.get('sampling')) + logger.info('claimed %s (rid=%s, %d tokens) -> %s:%d', req_id, m.rid, m.num_tokens, m.tilert_host, m.tilert_ctrl_port) + return m + +@@ -222,7 +223,9 @@ + remote_max_seq = int(hello['max_seq_len']) + assert seq <= remote_max_seq, f'seq {seq} exceeds decode max_seq_len {remote_max_seq}' + msg = {'rid': m.rid, 'rank': self._tp_rank, 'seq_len': seq, 'last_prompt_token': m.last_prompt_token, 'sampling': m.sampling, 'admission_window_s': self._admission_window()} +- if self._tp_rank == 0 and m.prompt_token_ids: ++ # InferenceX patch: a decode node that keeps KV prefixes asks for ++ # the prompt ids in its hello. ++ if self._tp_rank == 0 and m.prompt_token_ids and (m.wants_ids or hello.get('want_prompt_ids')): + msg['prompt_token_ids'] = m.prompt_token_ids + wire.send_msg(conn, msg) + ack = wire.recv_msg(conn) +--- a/profiles/glm5_rocm_engine.py ++++ b/profiles/glm5_rocm_engine.py +@@ -1,13 +1,38 @@ + from __future__ import annotations + import logging + import os ++import time + import torch + from tilert.pd_vllm.grammar_spec import GrammarUnsupported + from tilert.pd_vllm.sampling import resolve_top_p + logger = logging.getLogger('pd_vllm.profile.glm5_rocm') + _FP8_MAX = 448.0 + _INDEX_HEAD_DIM = 128 ++# Prefix reuse (InferenceX patch): the engine keeps the KV of the last injected ++# prompt. When the next prompt extends it, only the rows from the first ++# differing 64-token page onward are copied in again. The prefill's vLLM ++# recomputes a partial last page, and the MTP layer's KV at position i depends ++# on token i + 1, so reuse stops at a page boundary at or before ++# min(common prefix, previous length) - 1. ++_REUSE_ENV = 'TILERT_PD_PREFIX_REUSE' ++_REUSE_PAGE = 64 ++_VERIFY_SPREAD = 192 ++_VERIFY_TAIL = 64 + ++ ++def _env_on(name: str) -> bool: ++ return (os.environ.get(name) or '').strip().lower() not in ('', '0', 'false', 'no', 'off') ++ ++ ++def _common_prefix(a: list, b: list) -> int: ++ n = min(len(a), len(b)) ++ if n <= 0: ++ return 0 ++ ta = torch.tensor(a[:n], dtype=torch.int64) ++ tb = torch.tensor(b[:n], dtype=torch.int64) ++ diff = (ta != tb).nonzero() ++ return int(diff[0, 0]) if diff.numel() else n ++ + def is_rocm_torch() -> bool: + return getattr(torch.version, 'hip', None) is not None + +@@ -45,6 +70,12 @@ + self._full_layers = list(full_layer_ordinals(self.n_layers)) + self._pure_tp8 = True if pure_tp8 is None else pure_tp8 + self._fp8_ki = fp8_ki_enabled() if fp8_ki is None else fp8_ki ++ self._prefix_reuse = _env_on(_REUSE_ENV) ++ self._resident_ids: list | None = None ++ self._pair_streams: dict = {} ++ self._inject_stats: dict = {} ++ logger.info('glm5_rocm inject: parallel per-pair copy streams; prefix reuse %s (%s)', ++ 'on' if self._prefix_reuse else 'off', _REUSE_ENV) + + def supports_logprobs(self) -> bool: + return False +@@ -74,6 +105,60 @@ + slot[:nbytes].view(torch.float8_e4m3fn).view(L, _INDEX_HEAD_DIM)[:seq].copy_(q) + slot[nbytes:nbytes + L * 4].view(torch.float32)[:seq].copy_(scale) + ++ def _pair_stream(self, dev: int, peer: int) -> torch.cuda.Stream: ++ key = (dev, peer) ++ st = self._pair_streams.get(key) ++ if st is None: ++ st = torch.cuda.Stream(device=dev) ++ self._pair_streams[key] = st ++ return st ++ ++ def _kv_targets(self, use: list, n_extra: int): ++ # (rank, cache rows, received rows) for every MLA cache the engine ++ # reads: rank 0 holds the main layers, ranks 1..7 add the MTP layer. ++ for rank in range(self.npes): ++ if rank == 0 and (not self._pure_tp8): ++ continue ++ caches = self.dl._caches[rank] ++ n_pairs = self.n_layers + (n_extra if rank != 0 else 0) ++ for lid in range(min(len(use), n_pairs)): ++ _ki, kv, pe = use[lid] ++ yield rank, caches[2 * lid][0], kv ++ yield rank, caches[2 * lid + 1][0], pe ++ ++ def _reuse_start(self, req, seq: int) -> int: ++ if not self._prefix_reuse or self._resident_ids is None: ++ return 0 ++ ids = getattr(req, 'prompt_token_ids', None) or [] ++ if len(ids) != seq: ++ logger.warning('glm5_rocm inject: %d prompt ids for seq %d; prefix reuse skipped', len(ids), seq) ++ return 0 ++ prev = self._resident_ids ++ keep = min(_common_prefix(prev, ids), len(prev)) - 1 ++ return max(0, keep // _REUSE_PAGE * _REUSE_PAGE) ++ ++ def _prefix_matches(self, use: list, n_extra: int, start: int) -> bool: ++ # The prefill resends the whole prompt, so the resident rows can be ++ # checked byte for byte against it: a spread of rows plus the last ++ # page before start, on every rank and every layer. ++ spread = torch.linspace(0, start - 1, steps=min(_VERIFY_SPREAD, start)).round().long() ++ tail = torch.arange(max(0, start - _VERIFY_TAIL), start) ++ pos = torch.unique(torch.cat([spread, tail])) ++ flags: dict = {} ++ pos_on: dict = {} ++ for rank, dst, src in self._kv_targets(use, n_extra): ++ dd, sd = dst.device, src.device ++ if dd not in pos_on: ++ pos_on[dd] = pos.to(dd) ++ if sd not in pos_on: ++ pos_on[sd] = pos.to(sd) ++ got = dst.index_select(0, pos_on[dd]) ++ want = torch.empty_like(got) ++ want.copy_(src.index_select(0, pos_on[sd])) ++ bad = (got.view(torch.uint8) != want.view(torch.uint8)).any() ++ flags[dd] = bad if dd not in flags else (flags[dd] | bad) ++ return not any((bool(f.item()) for f in flags.values())) ++ + def inject(self, req) -> None: + dl = self.dl + layers = req.layers +@@ -86,16 +171,36 @@ + if n_extra and len(layers) == self.n_layers: + raise RuntimeError('glm5_rocm inject: engine has an MTP block but the prefill sent no MTP-layer KV (prefill must run with --speculative-config mtp)') + use = layers[:self.n_layers + n_extra] ++ t0 = time.time() ++ start = self._reuse_start(req, seq) ++ self._resident_ids = None + dl.reset_sequence() +- for rank in range(self.npes): +- if rank == 0 and (not self._pure_tp8): +- continue +- caches = dl._caches[rank] +- n_pairs = self.n_layers + (n_extra if rank != 0 else 0) +- for lid in range(min(len(use), n_pairs)): +- _ki, kv, pe = use[lid] +- caches[2 * lid][0, :seq].copy_(kv, non_blocking=True) +- caches[2 * lid + 1][0, :seq].copy_(pe, non_blocking=True) ++ # Fence the reset on every device before reading or writing the caches ++ # from other streams. ++ for d in range(self.npes): ++ torch.cuda.synchronize(d) ++ verified = 'none' ++ if start > 0: ++ if self._prefix_matches(use, n_extra, start): ++ verified = 'ok' ++ else: ++ logger.warning('glm5_rocm inject: resident prefix of %d rows differs from the transfer; copying all %d', start, seq) ++ verified = 'mismatch' ++ start = 0 ++ t1 = time.time() ++ # One stream per (destination, source) device pair: torch fences a ++ # cross-device copy against the current stream on both devices, so a ++ # shared stream serialises all eight ranks behind one link. ++ by_pair: dict = {} ++ for rank, dst, src in self._kv_targets(use, n_extra): ++ by_pair.setdefault((dst.device.index, src.device.index), []).append((dst, src)) ++ used = [] ++ for (dd, sd), pairs in by_pair.items(): ++ ds, ss = self._pair_stream(dd, sd), self._pair_stream(sd, dd) ++ used.extend((ds, ss)) ++ with torch.cuda.stream(ds), torch.cuda.stream(ss): ++ for dst, src in pairs: ++ dst[start:seq].copy_(src[start:seq], non_blocking=True) + caches0 = dl._caches[0] + ki_base = 2 * self.n_layers if self._pure_tp8 else 0 + ki_layers = self._full_layers + ([self.n_layers] if n_extra else []) +@@ -103,9 +208,20 @@ + if lid >= len(use): + break + self._write_ki(caches0[ki_base + ki_slot], use[lid][0], seq) +- torch.cuda.synchronize() ++ for st in used: ++ st.synchronize() ++ for d in range(self.npes): ++ torch.cuda.synchronize(d) ++ t2 = time.time() + dl.set_cur_pos(seq) + self._seq_len = seq ++ if self._prefix_reuse: ++ ids = getattr(req, 'prompt_token_ids', None) or [] ++ self._resident_ids = list(ids) if len(ids) == seq else None ++ self._inject_stats = {'reuse_rows': start, 'copy_rows': seq - start, 'reuse_verify': verified, ++ 'verify_ms': round(1000 * (t1 - t0), 1), 'copy_ms': round(1000 * (t2 - t1), 1)} ++ logger.info('glm5_rocm inject: seq=%d reused=%d copied=%d verify=%s (%.1f ms) copy %.1f ms', ++ seq, start, seq - start, verified, 1000 * (t1 - t0), 1000 * (t2 - t1)) + + def decode(self, first_token_id, max_tokens, sampling, on_token=None, cancel_event=None, grammar_session=None, top_logprobs=None): + if grammar_session is not None: +@@ -140,7 +256,7 @@ + finish = self._decode_mtp(first, budget, tokens, stop_ids, on_token, cancel_event) + else: + finish = self._decode_plain(first, budget, tokens, stop_ids, on_token, cancel_event) +- self.last_stats = {'finish_reason': finish} ++ self.last_stats = {'finish_reason': finish, **self._inject_stats} + if self.with_mtp: + self.last_stats.update(self._mtp_stats(base_step)) + return tokens +--- a/receive_server.py ++++ b/receive_server.py +@@ -41,6 +41,9 @@ + self.ctrl_port = ctrl_port + self.device = device + self.request_timeout = request_timeout ++ # InferenceX patch: ask the prefill for the prompt ids on every request ++ # so the engine can keep the KV prefix the next turn shares. ++ self._want_prompt_ids = (os.environ.get('TILERT_PD_PREFIX_REUSE') or '').strip().lower() not in ('', '0', 'false', 'no', 'off') + total = profile.buffer_bytes(max_seq_len) + nshards = int((os.environ.get('TILERT_PD_SHARDS') or '8').strip() or 8) + nshards = max(1, min(nshards, torch.cuda.device_count() or 1)) +@@ -212,7 +215,10 @@ + conn.settimeout(self.request_timeout) + with self._lock: + advisory_busy = self._current is not None and (not self._reusable(self._current)) +- wire.send_msg(conn, wire.hello_msg(self._transport.name, self._transport_meta, self.max_seq_len, self.profile.layout_version, self._hello_layout, busy=advisory_busy)) ++ hello = wire.hello_msg(self._transport.name, self._transport_meta, self.max_seq_len, self.profile.layout_version, self._hello_layout, busy=advisory_busy) ++ if self._want_prompt_ids: ++ hello['want_prompt_ids'] = True ++ wire.send_msg(conn, hello) + req = wire.recv_msg(conn) + rid, rank = (req['rid'], int(req['rank'])) + if req.get('seq_len', 0) > self.max_seq_len: diff --git a/benchmarks/multi_node/amd_utils/setup_deps.sh b/benchmarks/multi_node/amd_utils/setup_deps.sh index 8adb438681..6a19568d1d 100644 --- a/benchmarks/multi_node/amd_utils/setup_deps.sh +++ b/benchmarks/multi_node/amd_utils/setup_deps.sh @@ -92,18 +92,18 @@ _tilert_install_missing() { install_tilert_container_tools() { if command -v ip >/dev/null 2>&1 && command -v curl >/dev/null 2>&1 \ - && command -v ibv_devices >/dev/null 2>&1; then + && command -v ibv_devices >/dev/null 2>&1 && command -v patch >/dev/null 2>&1; then echo "[SETUP] Container RDMA/net tools already present" return 0 fi - echo "[SETUP] Installing iproute2 + curl + ibverbs userspace in container..." + echo "[SETUP] Installing iproute2 + curl + patch + ibverbs userspace in container..." apt-get update -q -y && apt-get install -q -y --no-install-recommends \ - iproute2 curl ibverbs-utils libibverbs1 librdmacm1 ibverbs-providers \ + iproute2 curl patch ibverbs-utils libibverbs1 librdmacm1 ibverbs-providers \ && rm -rf /var/lib/apt/lists/* - if ! command -v ip >/dev/null 2>&1 || ! command -v curl >/dev/null 2>&1; then - echo "[SETUP] ERROR: failed to install iproute2/curl"; exit 1 + if ! command -v ip >/dev/null 2>&1 || ! command -v curl >/dev/null 2>&1 || ! command -v patch >/dev/null 2>&1; then + echo "[SETUP] ERROR: failed to install iproute2/curl/patch"; exit 1 fi - _SETUP_INSTALLED+=("iproute2+curl+ibverbs") + _SETUP_INSTALLED+=("iproute2+curl+patch+ibverbs") } _tilert_install_wheel() { @@ -127,9 +127,37 @@ _tilert_install_wheel() { _SETUP_INSTALLED+=("$TILERT_PACKAGE==$TILERT_VERSION($mode)") } +# KV prefix reuse across AgentX turns. tilert 0.1.6.post1 resets the decode +# sequence and copies the whole prompt's KV into all eight rank caches on every +# request, one layer after another on the default streams: 2.4 s at the p50 +# AgentX context, although most turns only extend the previous prompt. The +# patch keeps the last prompt's KV resident, copies only the rows from the +# first differing page onward after checking a sample of the kept rows against +# the transfer, and fans the copies out on one stream per device pair. The +# prefill connector sends the prompt ids when the decode hello asks for them. +# Engine-patch waiver: docs/waiver/.md. Applied when the recipe sets +# TILERT_PD_PREFIX_REUSE=1; with 0 the wheel runs as shipped. +_TILERT_PD_REUSE_PATCH="$(dirname "${BASH_SOURCE[0]}")/patches/tilert-0.1.6.post1-pd-prefix-reuse.patch" + +_tilert_apply_pd_reuse_patch() { + [[ "$TILERT_PD_PREFIX_REUSE" == "1" ]] || { echo "[SETUP] KV prefix reuse off (TILERT_PD_PREFIX_REUSE=$TILERT_PD_PREFIX_REUSE); tilert unpatched"; return 0; } + local dir + dir="$("$PY" -c 'import os, tilert.pd_vllm as m; print(os.path.dirname(m.__file__))')" || { echo "[SETUP] ERROR: cannot locate tilert.pd_vllm"; exit 1; } + if grep -q "_REUSE_ENV = 'TILERT_PD_PREFIX_REUSE'" "$dir/profiles/glm5_rocm_engine.py"; then + echo "[SETUP] tilert KV prefix reuse patch already applied in $dir" + return 0 + fi + [[ -f "$_TILERT_PD_REUSE_PATCH" ]] || { echo "[SETUP] ERROR: missing $_TILERT_PD_REUSE_PATCH"; exit 1; } + echo "[SETUP] applying $(basename "$_TILERT_PD_REUSE_PATCH") to $dir" + patch -p1 -s -N -d "$dir" < "$_TILERT_PD_REUSE_PATCH" || { echo "[SETUP] ERROR: patch failed to apply"; exit 1; } + "$PY" -m py_compile "$dir"/*.py "$dir"/profiles/*.py || { echo "[SETUP] ERROR: patched tilert.pd_vllm does not compile"; exit 1; } + _SETUP_INSTALLED+=("tilert-pd-prefix-reuse.patch") +} + install_tilert_decode() { install_tilert_container_tools _tilert_install_wheel full + _tilert_apply_pd_reuse_patch _tilert_install_missing uvicorn $TILERT_HTTP_DEPS _tilert_install_missing mooncake.engine "$TILERT_TRANSPORT_DEPS" _tilert_install_missing transformers "$TILERT_TRANSFORMERS_SPEC" @@ -149,6 +177,7 @@ install_tilert_prefill() { echo "[SETUP] prefill-side vLLM $vllm_v" install_tilert_container_tools _tilert_install_wheel no-deps + _tilert_apply_pd_reuse_patch _tilert_install_missing mooncake.engine "$TILERT_TRANSPORT_DEPS" "$PY" -c "import tilert.pd_vllm.prefill_connector" 2>/dev/null || { echo "[SETUP] WARN: import tilert.pd_vllm.prefill_connector failed (vLLM will report again when loading the connector plugin):" @@ -165,7 +194,11 @@ if [[ "$ENGINE" == "vllm-disagg" ]]; then export PATH="${UCX_HOME}/bin:/usr/local/bin/etcd:/root/.cargo/bin:${PATH}" export LD_LIBRARY_PATH="${UCX_HOME}/lib:${RIXL_HOME}/lib:${RIXL_HOME}/lib/x86_64-linux-gnu:${LD_LIBRARY_PATH:-}" elif [[ "$ENGINE" == "tilert" ]]; then - check_env_vars TILERT_VERSION + check_env_vars TILERT_VERSION TILERT_PD_PREFIX_REUSE + if [[ "$TILERT_PD_PREFIX_REUSE" == "1" && "$TILERT_VERSION" != "0.1.6.post1" ]]; then + echo "[SETUP] ERROR: $_TILERT_PD_REUSE_PATCH targets tilert 0.1.6.post1, got $TILERT_VERSION; rebase the patch or set TILERT_PD_PREFIX_REUSE=0" + exit 1 + fi TILERT_PIP_SPEC="$TILERT_PACKAGE==$TILERT_VERSION" _tilert_resolve_python case "${TILERT_ROLE:-}" in From da57296a93d5cfd5271b48c32e25b6ec582f55a6 Mon Sep 17 00:00:00 2001 From: Oseltamivir <58582368+Oseltamivir@users.noreply.github.com> Date: Wed, 23 Sep 2026 10:56:22 +0800 Subject: [PATCH 21/25] docs(tilert): engine-patch waiver and changelog entry for the AgentX KV prefix reuse patch docs/waiver/3376.md documents the tilert 0.1.6.post1 PD prefix-reuse patch, states that the unpatched wheel runs this benchmark (#3366) so the patch is a performance change, and gives the removal plan. The appended perf-changelog entry covers glm5.3-fp8-mi355x-tilert-agentic. --- benchmarks/multi_node/amd_utils/setup_deps.sh | 2 +- docs/waiver/3376.md | 61 +++++++++++++++++++ perf-changelog.yaml | 9 +++ 3 files changed, 71 insertions(+), 1 deletion(-) create mode 100644 docs/waiver/3376.md diff --git a/benchmarks/multi_node/amd_utils/setup_deps.sh b/benchmarks/multi_node/amd_utils/setup_deps.sh index 6a19568d1d..d5d786af72 100644 --- a/benchmarks/multi_node/amd_utils/setup_deps.sh +++ b/benchmarks/multi_node/amd_utils/setup_deps.sh @@ -135,7 +135,7 @@ _tilert_install_wheel() { # first differing page onward after checking a sample of the kept rows against # the transfer, and fans the copies out on one stream per device pair. The # prefill connector sends the prompt ids when the decode hello asks for them. -# Engine-patch waiver: docs/waiver/.md. Applied when the recipe sets +# Engine-patch waiver: docs/waiver/3376.md. Applied when the recipe sets # TILERT_PD_PREFIX_REUSE=1; with 0 the wheel runs as shipped. _TILERT_PD_REUSE_PATCH="$(dirname "${BASH_SOURCE[0]}")/patches/tilert-0.1.6.post1-pd-prefix-reuse.patch" diff --git a/docs/waiver/3376.md b/docs/waiver/3376.md new file mode 100644 index 0000000000..200c4d3495 --- /dev/null +++ b/docs/waiver/3376.md @@ -0,0 +1,61 @@ +# Inference-engine patch waiver — PR #3376 + +Filed per [`docs/PR_REVIEW_CHECKLIST.md`](../PR_REVIEW_CHECKLIST.md) (image-provenance / patch item): the +TileRT launcher in this PR patches the TileRT PD (prefill/decode) data plane, the pip-installed +`tilert==0.1.6.post1` wheel's `pd_vllm` package, inside both containers at start, before serving. +vLLM is **not** patched; it runs exactly as shipped in `ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6`. + +## Config covered + +- **Master config entry:** `glm5.3-fp8-mi355x-tilert-agentic` in + [`configs/amd-master.yaml`](../../configs/amd-master.yaml). +- **Pinned images:** `ghcr.io/tile-ai/tilert-rocm-decode:0.1.6` and `ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6`; + neither carries the TileRT wheel, which + [`setup_deps.sh`](../../benchmarks/multi_node/amd_utils/setup_deps.sh) pip-installs as + `tilert==0.1.6.post1` from PyPI at container start. +- **Patch entrypoint:** `_tilert_apply_pd_reuse_patch` in `setup_deps.sh`, called from + `install_tilert_decode` and `install_tilert_prefill`, only when the recipe sets + `TILERT_PD_PREFIX_REUSE=1`. Marker-gated, compiled after application, and pinned to + `TILERT_VERSION=0.1.6.post1`; any other version fails loudly. +- **Patch file:** + [`benchmarks/multi_node/amd_utils/patches/tilert-0.1.6.post1-pd-prefix-reuse.patch`](../../benchmarks/multi_node/amd_utils/patches/tilert-0.1.6.post1-pd-prefix-reuse.patch) + (4 pure-Python files under `tilert/pd_vllm/`; no compiled component is touched). + +## What is patched + +| file | change | +|---|---| +| `pd_vllm/profiles/glm5_rocm_engine.py` | `RocmGlm52EngineAdapter.inject` keeps the last prompt's token ids; for the next request it copies only the KV rows from `floor((min(common prefix, previous length) - 1) / 64) * 64` onward, after checking a sample of the kept rows byte for byte against the fresh transfer on every rank and layer (a mismatch copies everything). Index-key caches are always rewritten in full. The KV copies run on one stream per (destination, source) device pair instead of the default streams. Per-request reuse statistics are logged and added to `REQSTAT`. | +| `pd_vllm/receive_server.py` | with `TILERT_PD_PREFIX_REUSE` set, the control-plane hello carries `want_prompt_ids: true` | +| `pd_vllm/prefill_connector.py` | TP rank 0 sends the prompt ids when the decode hello asks for them (previously only for repetition penalties) | +| `pd_vllm/decode_server.py` | passes the received prompt ids to `inject` | + +The wire protocol version, the buffer layout, the bytes transferred, the router and vLLM are unchanged. +With `TILERT_PD_PREFIX_REUSE=0` the patch is not applied and the wheel runs as shipped. + +## Why the unmodified upstream stack cannot run this benchmark + +**It can.** [#3366](https://github.com/SemiAnalysisAI/InferenceX/pull/3366) runs this AgentX benchmark +on the unpatched `tilert==0.1.6.post1` wheel. This patch is a performance change, so it does not meet the +checklist's condition for a waiver. It is filed so the patch is documented and reviewable, and this PR +is an experiment measuring what upstream TileRT would gain. On #3366's AgentX run (sweep 35741321941, +239 requests, concurrency 1) the decode `inject` step took 2426 ms at p50 and 4309 ms at p90 of a +5.76 s / 8.09 s TTFT, while 200 of 239 requests continued the previous request's conversation. +Maintainers decide whether results from this PR may be published. + +## Upstream PR / issue links + +- TileRT upstream: https://github.com/tile-ai/TileRT; **upstream issue pending**, to be linked here when + opened. The measurements behind the patch are on + https://github.com/SemiAnalysisAI/InferenceX/pull/3376. +- vLLM: no change requested. + +## Removal plan + +When a TileRT release keeps decode KV prefixes across requests (or merges an equivalent of this patch), +in one PR: bump `TILERT_VERSION` in the AgentX recipe; delete +`patches/tilert-0.1.6.post1-pd-prefix-reuse.patch`, `_TILERT_PD_REUSE_PATCH`, +`_tilert_apply_pd_reuse_patch`, its two call sites and the version pin check in `setup_deps.sh`, and the +`patch` package in `install_tilert_container_tools`; drop `TILERT_PD_PREFIX_REUSE` from the recipe and +its forwarding and validation in `job.slurm` (or map it to the release's own knob); and delete this +waiver. If the PR is closed unmerged, nothing needs removing on `main`. diff --git a/perf-changelog.yaml b/perf-changelog.yaml index bce3819e20..24bbb602c5 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8758,3 +8758,12 @@ - "Replace the PD-buffers-in-DRAM route with tilert==0.1.6.post1 on both TileRT ranks (PyPI, 2026-09-22): the shipped wheel shards the mooncake receive buffer (decode, receive_server.py) and the connector staging buffer (prefill TP rank 0, prefill_connector.py) by layer across the eight devices (TILERT_PD_SHARDS, default 8, capped at device_count; layer lid on device lid % 8; the glm5_2 profile has 79 layers, so the heaviest shard holds 10 of them: 99.06 GiB x 10/79 = 12.54 GiB per card) instead of one dense 99.06 GiB buffer per role, and both fit on the GPU at the full 1048576-token context without host memory. Each shard is registered with mooncake on its own device, hello_layout carries per-shard bases and rdma_plan writes layer by layer into the matching remote shard, so the KV path is device-to-device again (81 GB in 751 ms = 108 GB/s on 2x8 MI350X per CrimsonDump/InferenceX@56a1e5a, against the 20.9 GiB/s measured for the host-to-host hop) and mla_nsa.convert() dequantises each layer on the shard that received it rather than on cuda:0. patches/tilert-0.1.6-pd-buffers-in-dram.patch, docs/waiver/3330.md and TILERT_PD_BUFFER_DEVICE (the AgentX recipe, server_tilert.sh, job.slurm, setup_deps.sh, including the 0.1.6 pin guard and the patch apt package) are removed; neither image is modified. Router metadata 0.1.6 -> 0.1.6.post1: pd_router.py is one of the six files post1 changes, its fixed 600 s request timeouts to vLLM and decode become TILERT_PD_HTTP_TIMEOUT_S (default 3600). GPU_MEM_UTIL stays 0.85; the 12.54 GiB staging shard sits outside vLLM's budget next to the ~6.3 GiB non-torch baseline. server_tilert.sh now honours EVAL_ONLY/RUN_EVAL on the AgentX path (GSM8K on the same router instead of an unconditional trace replay) and exports PORT for benchmark_lib's run_lm_eval; evals run with real MTP verification since simulated acceptance is already disabled under those flags. Measured by CrimsonDump on 2x8 MI350X at this context with bf16 KV, 3600 s AgentX at concurrency 1: submission_valid true, 239 successful requests, 0 errors, TTFT p50 5711.8 ms, ITL p50 2.18 ms, peak device memory 202.1 GiB per decode card and 269.2 GiB per prefill card of 287.69 GiB." - "在两侧 TileRT rank 上以 tilert==0.1.6.post1（PyPI，2026-09-22）取代 PD 缓冲进 DRAM 的方案：正式 wheel 把 mooncake 接收缓冲（decode，receive_server.py）与 connector 暂存缓冲（prefill TP rank 0，prefill_connector.py）按层分片到 8 张卡（TILERT_PD_SHARDS，默认 8，以 device_count 为上限；第 lid 层落在第 lid % 8 张卡；glm5_2 profile 共 79 层，最重分片含 10 层：99.06 GiB x 10/79 = 每卡 12.54 GiB），不再是每个角色一块致密的 99.06 GiB 缓冲，于是在完整 1048576 上下文下两者都能放进 GPU，无需主机内存。各分片在所在卡上向 mooncake 注册，hello_layout 携带各分片基址，rdma_plan 逐层写入对应的远端分片，KV 通路重回设备到设备（2x8 MI350X 上 81 GB / 751 ms = 108 GB/s，见 CrimsonDump/InferenceX@56a1e5a；对比主机到主机实测 20.9 GiB/s），mla_nsa.convert() 在收到该层的分片上反量化而非集中于 cuda:0。移除 patches/tilert-0.1.6-pd-buffers-in-dram.patch、docs/waiver/3330.md 与 TILERT_PD_BUFFER_DEVICE（AgentX 配方、server_tilert.sh、job.slurm、setup_deps.sh，含 0.1.6 版本锁定检查与 patch apt 包）；两侧镜像均无改动。router 元数据 0.1.6 -> 0.1.6.post1：pd_router.py 是 post1 改动的六个文件之一，其对 vLLM 与 decode 固定的 600 秒请求超时改为 TILERT_PD_HTTP_TIMEOUT_S（默认 3600）。GPU_MEM_UTIL 保持 0.85；12.54 GiB 暂存分片与约 6.3 GiB 非 torch 基线同在 vLLM 预算之外。server_tilert.sh 现在在 AgentX 路径也遵循 EVAL_ONLY/RUN_EVAL（在同一 router 上跑 GSM8K，而非无条件回放轨迹），并为 benchmark_lib 的 run_lm_eval 导出 PORT；这些标志下模拟接受已关闭，评测使用真实 MTP 验证。CrimsonDump 在 2x8 MI350X、该上下文、bf16 KV 下实测 3600 秒 AgentX 并发 1：submission_valid true、239 条成功、0 错误、TTFT p50 5711.8 ms、ITL p50 2.18 ms，decode 每卡峰值 202.1 GiB、prefill 每卡 269.2 GiB（卡容量 287.69 GiB）。" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3366 + +- config-keys: + - glm5.3-fp8-mi355x-tilert-agentic + scenario-type: + - agentic-coding + description: + - "Keep AgentX KV prefixes resident on the TileRT decode node (experiment on top of #3366). setup_deps.sh applies patches/tilert-0.1.6.post1-pd-prefix-reuse.patch to the pip-installed tilert 0.1.6.post1 on both ranks when the recipe sets TILERT_PD_PREFIX_REUSE=1 (engine-patch waiver docs/waiver/3376.md; vLLM and the router are unpatched). On #3366's AgentX run the decode inject step, which resets the sequence and copies the whole prompt's KV into all eight rank caches serially on the default streams, took 2426 ms at p50 and 4309 ms at p90 of a 5.76 s / 8.09 s TTFT, while 200 of 239 requests continued the previous request's conversation with a median 334 new tokens. The patched inject keeps the last prompt's ids and KV and copies only the rows from the 64-token page before min(common prefix, previous length) - 1 onward, after a byte-for-byte check of sampled kept rows against the fresh transfer on every rank and layer (any mismatch copies everything); index-key caches are always rewritten. The KV copies run on one stream per (destination, source) device pair. The decode hello asks for the prompt ids and the prefill connector sends them; the wire layout and bytes transferred are unchanged." + - "在 TileRT decode 节点上保留 AgentX 的 KV 前缀（基于 #3366 的实验）。配方设置 TILERT_PD_PREFIX_REUSE=1 时，setup_deps.sh 在两侧对 pip 安装的 tilert 0.1.6.post1 应用 patches/tilert-0.1.6.post1-pd-prefix-reuse.patch（引擎补丁豁免 docs/waiver/3376.md；vLLM 与 router 未改动）。#3366 的 AgentX 运行中，decode 的 inject 步骤每次复位序列并在默认流上把整段提示的 KV 串行拷入 8 个 rank 的缓存，在 5.76 秒 / 8.09 秒的 TTFT 中占 p50 2426 ms、p90 4309 ms，而 239 个请求中有 200 个延续上一请求的对话，中位新增 334 个 token。补丁后的 inject 保留上一轮提示的 id 与 KV，只拷贝从 min(共同前缀, 上一轮长度) - 1 所在 64-token 页起的行；拷贝前在每个 rank、每层上将抽样保留行与新传输按字节比对，任何不一致都回退为全量拷贝；索引键缓存始终重写。KV 拷贝按（目标卡，源卡）成对使用独立流。decode 的 hello 请求提示 id，prefill connector 据此发送；线上布局与传输字节数不变。" + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3376 From b9ed2ff3193f551e0923ebcedaa835c8933f4f7b Mon Sep 17 00:00:00 2001 From: Oseltamivir <58582368+Oseltamivir@users.noreply.github.com> Date: Wed, 23 Sep 2026 12:16:28 +0800 Subject: [PATCH 22/25] fix(tilert): decide KV prefix reuse per cache, log why kept rows differ, pack prompt ids The first AgentX run on this PR (sweep 35812357318, Slurm job 45562) applied the patch on both ranks and the per-pair copy streams cut the full inject copy from 10.5 us/token to 1.78 us/token (565 ms instead of 3278 ms at 314k tokens). But every reuse attempt with a long common prefix failed verification (29 of 29), so every turn fell back to a full copy, and sending the prompt ids as a JSON list raised the decode's wire wait from 100-190 ms to 282 ms at p50. - Verification now flags each rank's kv and pe cache of every layer separately; caches whose sample matches keep their prefix and the rest are copied in full. - Mismatches are logged by layer, rank and kind, with an all-zero check. For the first three reuse attempts the same check also runs before reset_sequence(), to tell whether the reset or the previous decode changed the kept rows. - The prompt ids travel as base64-packed int32 (prompt_ids_i32); the list form for repetition penalties is unchanged. --- .../tilert-0.1.6.post1-pd-prefix-reuse.patch | 196 ++++++++++++------ docs/waiver/3376.md | 6 +- perf-changelog.yaml | 4 +- 3 files changed, 143 insertions(+), 63 deletions(-) diff --git a/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6.post1-pd-prefix-reuse.patch b/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6.post1-pd-prefix-reuse.patch index 50ab6aa1b4..a621d74484 100644 --- a/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6.post1-pd-prefix-reuse.patch +++ b/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6.post1-pd-prefix-reuse.patch @@ -4,13 +4,19 @@ return JSONResponse({'error': 'kv_transfer_timeout', 'rid': body.rid}, status_code=504) t_recv = time.time() conv = server.profile.convert(server.buffer, server.base_ptr, server.max_seq_len, req, server.profile.num_ranks) -+ conv.prompt_token_ids = req.prompt_token_ids ++ conv.prompt_ids_i32 = req.prompt_ids_i32 t_conv = time.time() engine.inject(conv) t_inj = time.time() --- a/prefill_connector.py 2026-09-23 10:51:39 -+++ b/prefill_connector.py 2026-09-23 10:52:38 -@@ -26,6 +26,7 @@ ++++ b/prefill_connector.py 2026-09-23 12:15:52 +@@ -1,3 +1,5 @@ ++import array ++import base64 + import logging + import queue + import os +@@ -26,6 +28,7 @@ tilert_ctrl_port: int sampling: dict | None = None prompt_token_ids: list = field(default_factory=list) @@ -18,7 +24,7 @@ @dataclass class TileRTMetadata(KVConnectorMetadata): -@@ -120,7 +121,7 @@ +@@ -120,7 +123,7 @@ return meta def _emit(self, req_id, token_ids, groups, params) -> _ReqMeta: @@ -27,20 +33,23 @@ logger.info('claimed %s (rid=%s, %d tokens) -> %s:%d', req_id, m.rid, m.num_tokens, m.tilert_host, m.tilert_ctrl_port) return m -@@ -222,7 +223,9 @@ +@@ -222,8 +225,12 @@ remote_max_seq = int(hello['max_seq_len']) assert seq <= remote_max_seq, f'seq {seq} exceeds decode max_seq_len {remote_max_seq}' msg = {'rid': m.rid, 'rank': self._tp_rank, 'seq_len': seq, 'last_prompt_token': m.last_prompt_token, 'sampling': m.sampling, 'admission_window_s': self._admission_window()} - if self._tp_rank == 0 and m.prompt_token_ids: -+ # InferenceX patch: a decode node that keeps KV prefixes asks for -+ # the prompt ids in its hello. -+ if self._tp_rank == 0 and m.prompt_token_ids and (m.wants_ids or hello.get('want_prompt_ids')): ++ if self._tp_rank == 0 and m.prompt_token_ids and m.wants_ids: msg['prompt_token_ids'] = m.prompt_token_ids ++ # InferenceX patch: a decode node that keeps KV prefixes asks for ++ # the prompt ids in its hello; they travel as packed int32. ++ if self._tp_rank == 0 and m.prompt_token_ids and hello.get('want_prompt_ids'): ++ msg['prompt_ids_i32'] = base64.b64encode(array.array('i', m.prompt_token_ids).tobytes()).decode('ascii') wire.send_msg(conn, msg) ack = wire.recv_msg(conn) + if not ack.get('accepted'): --- a/profiles/glm5_rocm_engine.py +++ b/profiles/glm5_rocm_engine.py -@@ -1,13 +1,38 @@ +@@ -1,13 +1,51 @@ from __future__ import annotations import logging import os @@ -61,30 +70,44 @@ +_REUSE_PAGE = 64 +_VERIFY_SPREAD = 192 +_VERIFY_TAIL = 64 ++_PRE_RESET_CHECKS = 3 + +def _env_on(name: str) -> bool: + return (os.environ.get(name) or '').strip().lower() not in ('', '0', 'false', 'no', 'off') + + -+def _common_prefix(a: list, b: list) -> int: -+ n = min(len(a), len(b)) ++def _ids_tensor(blob: bytes) -> torch.Tensor | None: ++ if not blob or len(blob) % 4: ++ return None ++ return torch.frombuffer(bytearray(blob), dtype=torch.int32) ++ ++ ++def _common_prefix(a: torch.Tensor, b: torch.Tensor) -> int: ++ n = min(a.numel(), b.numel()) + if n <= 0: + return 0 -+ ta = torch.tensor(a[:n], dtype=torch.int64) -+ tb = torch.tensor(b[:n], dtype=torch.int64) -+ diff = (ta != tb).nonzero() ++ diff = (a[:n] != b[:n]).nonzero() + return int(diff[0, 0]) if diff.numel() else n ++ ++ ++def _span(xs: list) -> str: ++ xs = sorted(set(xs)) ++ if len(xs) <= 12: ++ return ','.join(str(x) for x in xs) ++ return f'{len(xs)} in [{xs[0]}..{xs[-1]}]' ++ + def is_rocm_torch() -> bool: return getattr(torch.version, 'hip', None) is not None -@@ -45,6 +70,12 @@ +@@ -45,6 +83,13 @@ self._full_layers = list(full_layer_ordinals(self.n_layers)) self._pure_tp8 = True if pure_tp8 is None else pure_tp8 self._fp8_ki = fp8_ki_enabled() if fp8_ki is None else fp8_ki + self._prefix_reuse = _env_on(_REUSE_ENV) -+ self._resident_ids: list | None = None ++ self._resident_ids: torch.Tensor | None = None ++ self._pre_checks_left = _PRE_RESET_CHECKS + self._pair_streams: dict = {} + self._inject_stats: dict = {} + logger.info('glm5_rocm inject: parallel per-pair copy streams; prefix reuse %s (%s)', @@ -92,7 +115,7 @@ def supports_logprobs(self) -> bool: return False -@@ -74,6 +105,60 @@ +@@ -74,6 +119,83 @@ slot[:nbytes].view(torch.float8_e4m3fn).view(L, _INDEX_HEAD_DIM)[:seq].copy_(q) slot[nbytes:nbytes + L * 4].view(torch.float32)[:seq].copy_(scale) @@ -104,9 +127,10 @@ + self._pair_streams[key] = st + return st + -+ def _kv_targets(self, use: list, n_extra: int): -+ # (rank, cache rows, received rows) for every MLA cache the engine -+ # reads: rank 0 holds the main layers, ranks 1..7 add the MTP layer. ++ def _kv_targets(self, use: list, n_extra: int) -> list: ++ # (rank, layer, kind, cache rows, received rows) for every MLA cache the ++ # engine reads: rank 0 holds the main layers, ranks 1..7 add the MTP layer. ++ out = [] + for rank in range(self.npes): + if rank == 0 and (not self._pure_tp8): + continue @@ -114,52 +138,81 @@ + n_pairs = self.n_layers + (n_extra if rank != 0 else 0) + for lid in range(min(len(use), n_pairs)): + _ki, kv, pe = use[lid] -+ yield rank, caches[2 * lid][0], kv -+ yield rank, caches[2 * lid + 1][0], pe ++ out.append((rank, lid, 'kv', caches[2 * lid][0], kv)) ++ out.append((rank, lid, 'pe', caches[2 * lid + 1][0], pe)) ++ return out + + def _reuse_start(self, req, seq: int) -> int: + if not self._prefix_reuse or self._resident_ids is None: + return 0 -+ ids = getattr(req, 'prompt_token_ids', None) or [] -+ if len(ids) != seq: -+ logger.warning('glm5_rocm inject: %d prompt ids for seq %d; prefix reuse skipped', len(ids), seq) ++ ids = _ids_tensor(getattr(req, 'prompt_ids_i32', b'')) ++ if ids is None or ids.numel() != seq: ++ logger.warning('glm5_rocm inject: %s prompt ids for seq %d; prefix reuse skipped', ++ 'no' if ids is None else ids.numel(), seq) + return 0 + prev = self._resident_ids -+ keep = min(_common_prefix(prev, ids), len(prev)) - 1 ++ keep = min(_common_prefix(prev, ids), prev.numel()) - 1 + return max(0, keep // _REUSE_PAGE * _REUSE_PAGE) + -+ def _prefix_matches(self, use: list, n_extra: int, start: int) -> bool: -+ # The prefill resends the whole prompt, so the resident rows can be ++ def _check_rows(self, targets: list, start: int) -> tuple[list, list]: ++ # The prefill resends the whole prompt, so each kept cache can be + # checked byte for byte against it: a spread of rows plus the last -+ # page before start, on every rank and every layer. ++ # page before start. Returns per-target (differs, all-zero) flags. + spread = torch.linspace(0, start - 1, steps=min(_VERIFY_SPREAD, start)).round().long() + tail = torch.arange(max(0, start - _VERIFY_TAIL), start) + pos = torch.unique(torch.cat([spread, tail])) -+ flags: dict = {} + pos_on: dict = {} -+ for rank, dst, src in self._kv_targets(use, n_extra): ++ per_dev: dict = {} ++ for i, (_rank, _lid, _kind, dst, src) in enumerate(targets): + dd, sd = dst.device, src.device + if dd not in pos_on: + pos_on[dd] = pos.to(dd) + if sd not in pos_on: + pos_on[sd] = pos.to(sd) -+ got = dst.index_select(0, pos_on[dd]) -+ want = torch.empty_like(got) ++ got = dst.index_select(0, pos_on[dd]).view(torch.uint8) ++ want = torch.empty((pos.numel(),) + tuple(dst.shape[1:]), dtype=dst.dtype, device=dd) + want.copy_(src.index_select(0, pos_on[sd])) -+ bad = (got.view(torch.uint8) != want.view(torch.uint8)).any() -+ flags[dd] = bad if dd not in flags else (flags[dd] | bad) -+ return not any((bool(f.item()) for f in flags.values())) ++ per_dev.setdefault(dd, []).append((i, (got != want.view(torch.uint8)).any(), (got == 0).all())) ++ bad = [False] * len(targets) ++ zero = [False] * len(targets) ++ for items in per_dev.values(): ++ b = torch.stack([x[1] for x in items]).cpu().tolist() ++ z = torch.stack([x[2] for x in items]).cpu().tolist() ++ for (i, _b, _z), bb, zz in zip(items, b, z): ++ bad[i], zero[i] = bool(bb), bool(zz) ++ return bad, zero ++ ++ @staticmethod ++ def _summary(targets: list, bad: list, zero: list) -> str: ++ hit = [t for t, b in zip(targets, bad) if b] ++ if not hit: ++ return '0 differ' ++ nz = sum(1 for b, z in zip(bad, zero) if b and z) ++ kinds = {} ++ for rank, lid, kind, _d, _s in hit: ++ kinds.setdefault(kind, (set(), set())) ++ kinds[kind][0].add(lid) ++ kinds[kind][1].add(rank) ++ parts = [f'{k}: layers {_span(list(v[0]))} ranks {_span(list(v[1]))}' for k, v in sorted(kinds.items())] ++ return f'{len(hit)}/{len(targets)} differ ({nz} all-zero); ' + '; '.join(parts) + def inject(self, req) -> None: dl = self.dl layers = req.layers -@@ -86,16 +171,36 @@ +@@ -86,16 +208,47 @@ if n_extra and len(layers) == self.n_layers: raise RuntimeError('glm5_rocm inject: engine has an MTP block but the prefill sent no MTP-layer KV (prefill must run with --speculative-config mtp)') use = layers[:self.n_layers + n_extra] ++ targets = self._kv_targets(use, n_extra) + t0 = time.time() + start = self._reuse_start(req, seq) + self._resident_ids = None ++ pre = None ++ if start > 0 and self._pre_checks_left > 0: ++ # Diagnostic: the same check before the reset shows whether the ++ # reset or the previous decode changed the kept rows. ++ self._pre_checks_left -= 1 ++ pre = self._check_rows(targets, start) dl.reset_sequence() - for rank in range(self.npes): - if rank == 0 and (not self._pure_tp8): @@ -174,32 +227,36 @@ + # from other streams. + for d in range(self.npes): + torch.cuda.synchronize(d) ++ starts = [0] * len(targets) + verified = 'none' ++ n_ok = 0 + if start > 0: -+ if self._prefix_matches(use, n_extra, start): -+ verified = 'ok' -+ else: -+ logger.warning('glm5_rocm inject: resident prefix of %d rows differs from the transfer; copying all %d', start, seq) -+ verified = 'mismatch' -+ start = 0 ++ bad, zero = self._check_rows(targets, start) ++ starts = [0 if b else start for b in bad] ++ n_ok = len(targets) - sum(bad) ++ verified = 'ok' if n_ok == len(targets) else ('partial' if n_ok else 'mismatch') ++ if n_ok < len(targets) or pre is not None: ++ logger.warning('glm5_rocm inject: seq=%d keep=%d before reset: %s | after reset: %s', ++ seq, start, self._summary(targets, *pre) if pre is not None else 'not checked', ++ self._summary(targets, bad, zero)) + t1 = time.time() + # One stream per (destination, source) device pair: torch fences a + # cross-device copy against the current stream on both devices, so a + # shared stream serialises all eight ranks behind one link. + by_pair: dict = {} -+ for rank, dst, src in self._kv_targets(use, n_extra): -+ by_pair.setdefault((dst.device.index, src.device.index), []).append((dst, src)) ++ for (_rank, _lid, _kind, dst, src), s0 in zip(targets, starts): ++ by_pair.setdefault((dst.device.index, src.device.index), []).append((dst, src, s0)) + used = [] -+ for (dd, sd), pairs in by_pair.items(): ++ for (dd, sd), items in by_pair.items(): + ds, ss = self._pair_stream(dd, sd), self._pair_stream(sd, dd) + used.extend((ds, ss)) + with torch.cuda.stream(ds), torch.cuda.stream(ss): -+ for dst, src in pairs: -+ dst[start:seq].copy_(src[start:seq], non_blocking=True) ++ for dst, src, s0 in items: ++ dst[s0:seq].copy_(src[s0:seq], non_blocking=True) caches0 = dl._caches[0] ki_base = 2 * self.n_layers if self._pure_tp8 else 0 ki_layers = self._full_layers + ([self.n_layers] if n_extra else []) -@@ -103,9 +208,20 @@ +@@ -103,9 +256,21 @@ if lid >= len(use): break self._write_ki(caches0[ki_base + ki_slot], use[lid][0], seq) @@ -212,16 +269,17 @@ dl.set_cur_pos(seq) self._seq_len = seq + if self._prefix_reuse: -+ ids = getattr(req, 'prompt_token_ids', None) or [] -+ self._resident_ids = list(ids) if len(ids) == seq else None -+ self._inject_stats = {'reuse_rows': start, 'copy_rows': seq - start, 'reuse_verify': verified, -+ 'verify_ms': round(1000 * (t1 - t0), 1), 'copy_ms': round(1000 * (t2 - t1), 1)} -+ logger.info('glm5_rocm inject: seq=%d reused=%d copied=%d verify=%s (%.1f ms) copy %.1f ms', -+ seq, start, seq - start, verified, 1000 * (t1 - t0), 1000 * (t2 - t1)) ++ ids = _ids_tensor(getattr(req, 'prompt_ids_i32', b'')) ++ self._resident_ids = ids if ids is not None and ids.numel() == seq else None ++ copied = sum(seq - s0 for s0 in starts) // max(1, len(targets)) ++ self._inject_stats = {'reuse_rows': start, 'reuse_tensors': f'{n_ok}/{len(targets)}', 'reuse_verify': verified, ++ 'mean_copy_rows': copied, 'verify_ms': round(1000 * (t1 - t0), 1), 'copy_ms': round(1000 * (t2 - t1), 1)} ++ logger.info('glm5_rocm inject: seq=%d keep=%d reused %d/%d caches, mean rows copied %d, verify=%s (%.1f ms) copy %.1f ms', ++ seq, start, n_ok, len(targets), copied, verified, 1000 * (t1 - t0), 1000 * (t2 - t1)) def decode(self, first_token_id, max_tokens, sampling, on_token=None, cancel_event=None, grammar_session=None, top_logprobs=None): if grammar_session is not None: -@@ -140,7 +256,7 @@ +@@ -140,7 +305,7 @@ finish = self._decode_mtp(first, budget, tokens, stop_ids, on_token, cancel_event) else: finish = self._decode_plain(first, budget, tokens, stop_ids, on_token, cancel_event) @@ -232,7 +290,20 @@ return tokens --- a/receive_server.py +++ b/receive_server.py -@@ -41,6 +41,9 @@ +@@ -1,3 +1,4 @@ ++import base64 + import logging + import os + import queue +@@ -22,6 +23,7 @@ + first_token_id: int | None + sampling: dict | None + prompt_token_ids: list = field(default_factory=list) ++ prompt_ids_i32: bytes = b'' + done_ranks: set = field(default_factory=set) + t_first_conn: float = 0.0 + t_complete: float = 0.0 +@@ -41,6 +43,9 @@ self.ctrl_port = ctrl_port self.device = device self.request_timeout = request_timeout @@ -242,7 +313,16 @@ total = profile.buffer_bytes(max_seq_len) nshards = int((os.environ.get('TILERT_PD_SHARDS') or '8').strip() or 8) nshards = max(1, min(nshards, torch.cuda.device_count() or 1)) -@@ -212,7 +215,10 @@ +@@ -188,6 +193,8 @@ + else: + self._current = cur = ReceivedRequest(rid=rid, seq_len=int(req['seq_len']), last_prompt_token=int(req.get('last_prompt_token', 0)), first_token_id=req.get('first_token_id'), sampling=req.get('sampling'), prompt_token_ids=list(req.get('prompt_token_ids') or []), t_first_conn=time.time(), generation=self._next_generation(), state=RESERVED) + logger.info('request %s: seq_len=%d (generation %d)', rid, cur.seq_len, cur.generation) ++ if not cur.prompt_ids_i32 and req.get('prompt_ids_i32'): ++ cur.prompt_ids_i32 = base64.b64decode(req['prompt_ids_i32']) + if not cur.prompt_token_ids and req.get('prompt_token_ids'): + cur.prompt_token_ids = list(req['prompt_token_ids']) + logger.info('request %s: prompt bitmap seeded from rank %d (%d ids)', rid, rank, len(cur.prompt_token_ids)) +@@ -212,7 +219,10 @@ conn.settimeout(self.request_timeout) with self._lock: advisory_busy = self._current is not None and (not self._reusable(self._current)) diff --git a/docs/waiver/3376.md b/docs/waiver/3376.md index 200c4d3495..84d42134a5 100644 --- a/docs/waiver/3376.md +++ b/docs/waiver/3376.md @@ -25,10 +25,10 @@ vLLM is **not** patched; it runs exactly as shipped in `ghcr.io/tile-ai/tilert-r | file | change | |---|---| -| `pd_vllm/profiles/glm5_rocm_engine.py` | `RocmGlm52EngineAdapter.inject` keeps the last prompt's token ids; for the next request it copies only the KV rows from `floor((min(common prefix, previous length) - 1) / 64) * 64` onward, after checking a sample of the kept rows byte for byte against the fresh transfer on every rank and layer (a mismatch copies everything). Index-key caches are always rewritten in full. The KV copies run on one stream per (destination, source) device pair instead of the default streams. Per-request reuse statistics are logged and added to `REQSTAT`. | +| `pd_vllm/profiles/glm5_rocm_engine.py` | `RocmGlm52EngineAdapter.inject` keeps the last prompt's token ids; for the next request it copies only the KV rows from `floor((min(common prefix, previous length) - 1) / 64) * 64` onward, after checking a sample of the kept rows byte for byte against the fresh transfer, separately for each rank's kv and pe cache of every layer; a cache whose sample differs is copied in full. Mismatches are logged by layer, rank and kind, with an all-zero check and, for the first three reuse attempts, the same check before `reset_sequence()`. Index-key caches are always rewritten in full. The KV copies run on one stream per (destination, source) device pair instead of the default streams. Per-request reuse statistics are logged and added to `REQSTAT`. | | `pd_vllm/receive_server.py` | with `TILERT_PD_PREFIX_REUSE` set, the control-plane hello carries `want_prompt_ids: true` | -| `pd_vllm/prefill_connector.py` | TP rank 0 sends the prompt ids when the decode hello asks for them (previously only for repetition penalties) | -| `pd_vllm/decode_server.py` | passes the received prompt ids to `inject` | +| `pd_vllm/prefill_connector.py` | TP rank 0 sends the prompt ids as packed int32 when the decode hello asks for them; the existing list form for repetition penalties is unchanged | +| `pd_vllm/decode_server.py` | passes the received prompt ids to `inject` (`receive_server.py` also decodes the packed ids) | The wire protocol version, the buffer layout, the bytes transferred, the router and vLLM are unchanged. With `TILERT_PD_PREFIX_REUSE=0` the patch is not applied and the wheel runs as shipped. diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 24bbb602c5..d38b6a2d65 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8764,6 +8764,6 @@ scenario-type: - agentic-coding description: - - "Keep AgentX KV prefixes resident on the TileRT decode node (experiment on top of #3366). setup_deps.sh applies patches/tilert-0.1.6.post1-pd-prefix-reuse.patch to the pip-installed tilert 0.1.6.post1 on both ranks when the recipe sets TILERT_PD_PREFIX_REUSE=1 (engine-patch waiver docs/waiver/3376.md; vLLM and the router are unpatched). On #3366's AgentX run the decode inject step, which resets the sequence and copies the whole prompt's KV into all eight rank caches serially on the default streams, took 2426 ms at p50 and 4309 ms at p90 of a 5.76 s / 8.09 s TTFT, while 200 of 239 requests continued the previous request's conversation with a median 334 new tokens. The patched inject keeps the last prompt's ids and KV and copies only the rows from the 64-token page before min(common prefix, previous length) - 1 onward, after a byte-for-byte check of sampled kept rows against the fresh transfer on every rank and layer (any mismatch copies everything); index-key caches are always rewritten. The KV copies run on one stream per (destination, source) device pair. The decode hello asks for the prompt ids and the prefill connector sends them; the wire layout and bytes transferred are unchanged." - - "在 TileRT decode 节点上保留 AgentX 的 KV 前缀（基于 #3366 的实验）。配方设置 TILERT_PD_PREFIX_REUSE=1 时，setup_deps.sh 在两侧对 pip 安装的 tilert 0.1.6.post1 应用 patches/tilert-0.1.6.post1-pd-prefix-reuse.patch（引擎补丁豁免 docs/waiver/3376.md；vLLM 与 router 未改动）。#3366 的 AgentX 运行中，decode 的 inject 步骤每次复位序列并在默认流上把整段提示的 KV 串行拷入 8 个 rank 的缓存，在 5.76 秒 / 8.09 秒的 TTFT 中占 p50 2426 ms、p90 4309 ms，而 239 个请求中有 200 个延续上一请求的对话，中位新增 334 个 token。补丁后的 inject 保留上一轮提示的 id 与 KV，只拷贝从 min(共同前缀, 上一轮长度) - 1 所在 64-token 页起的行；拷贝前在每个 rank、每层上将抽样保留行与新传输按字节比对，任何不一致都回退为全量拷贝；索引键缓存始终重写。KV 拷贝按（目标卡，源卡）成对使用独立流。decode 的 hello 请求提示 id，prefill connector 据此发送；线上布局与传输字节数不变。" + - "Keep AgentX KV prefixes resident on the TileRT decode node (experiment on top of #3366). setup_deps.sh applies patches/tilert-0.1.6.post1-pd-prefix-reuse.patch to the pip-installed tilert 0.1.6.post1 on both ranks when the recipe sets TILERT_PD_PREFIX_REUSE=1 (engine-patch waiver docs/waiver/3376.md; vLLM and the router are unpatched). On #3366's AgentX run the decode inject step, which resets the sequence and copies the whole prompt's KV into all eight rank caches serially on the default streams, took 2426 ms at p50 and 4309 ms at p90 of a 5.76 s / 8.09 s TTFT, while 200 of 239 requests continued the previous request's conversation with a median 334 new tokens. The patched inject keeps the last prompt's ids and KV and copies only the rows from the 64-token page before min(common prefix, previous length) - 1 onward, after a byte-for-byte check of sampled kept rows against the fresh transfer, per rank and per kv/pe cache of every layer (a cache whose sample differs is copied in full); index-key caches are always rewritten. The KV copies run on one stream per (destination, source) device pair. The decode hello asks for the prompt ids and the prefill connector sends them as packed int32; the KV layout and bytes transferred are unchanged." + - "在 TileRT decode 节点上保留 AgentX 的 KV 前缀（基于 #3366 的实验）。配方设置 TILERT_PD_PREFIX_REUSE=1 时，setup_deps.sh 在两侧对 pip 安装的 tilert 0.1.6.post1 应用 patches/tilert-0.1.6.post1-pd-prefix-reuse.patch（引擎补丁豁免 docs/waiver/3376.md；vLLM 与 router 未改动）。#3366 的 AgentX 运行中，decode 的 inject 步骤每次复位序列并在默认流上把整段提示的 KV 串行拷入 8 个 rank 的缓存，在 5.76 秒 / 8.09 秒的 TTFT 中占 p50 2426 ms、p90 4309 ms，而 239 个请求中有 200 个延续上一请求的对话，中位新增 334 个 token。补丁后的 inject 保留上一轮提示的 id 与 KV，只拷贝从 min(共同前缀, 上一轮长度) - 1 所在 64-token 页起的行；拷贝前按每个 rank、每层的 kv/pe 缓存分别将抽样保留行与新传输按字节比对，抽样不一致的缓存全量拷贝；索引键缓存始终重写。KV 拷贝按（目标卡，源卡）成对使用独立流。decode 的 hello 请求提示 id，prefill connector 以打包 int32 发送；KV 布局与传输字节数不变。" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3376 From 865925a1ddc5977bba8254ac0fc07485e64e1f80 Mon Sep 17 00:00:00 2001 From: Oseltamivir <58582368+Oseltamivir@users.noreply.github.com> Date: Wed, 23 Sep 2026 14:35:33 +0800 Subject: [PATCH 23/25] fix(tilert): reuse the KV prefix on matching tokens, always copy the MTP layer, delta-encode prompt ids The v2 diagnostics (sweep 35817675288, Slurm job 45577) showed why the byte-exact check rejected reuse: on 221 of 246 continuing AgentX turns layer 0 matched on every rank while layers 1-78 differed, before and after reset_sequence(), and no kept row was zeroed except the MTP layer, which the reset clears. vLLM recomputes part of the prefix with different chunking, so the kept KV is the same tokens' KV with different rounding. On the other 25 turns every main layer matched exactly. - Reuse now requires the token ids to match (as before), layer 0 to match the fresh transfer byte for byte on every rank, and five sampled deeper layers on ranks 0 and 1 to be within a relative L2 error of 0.25 (TILERT_PD_REUSE_REL_TOL); otherwise every row is copied. The worst relative error is logged and added to REQSTAT. - The MTP layer is always copied in full. - Verification checks 36 caches instead of 1262 (it cost ~170 ms). - Prompt ids are sent as a suffix over the previous request's ids when the decode hello's hash shows the decode node still holds them. --- .../tilert-0.1.6.post1-pd-prefix-reuse.patch | 232 +++++++++++------- docs/waiver/3376.md | 6 +- perf-changelog.yaml | 4 +- 3 files changed, 145 insertions(+), 97 deletions(-) diff --git a/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6.post1-pd-prefix-reuse.patch b/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6.post1-pd-prefix-reuse.patch index a621d74484..813ed077a2 100644 --- a/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6.post1-pd-prefix-reuse.patch +++ b/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6.post1-pd-prefix-reuse.patch @@ -9,14 +9,15 @@ engine.inject(conv) t_inj = time.time() --- a/prefill_connector.py 2026-09-23 10:51:39 -+++ b/prefill_connector.py 2026-09-23 12:15:52 -@@ -1,3 +1,5 @@ ++++ b/prefill_connector.py 2026-09-23 14:34:58 +@@ -1,3 +1,6 @@ +import array +import base64 ++import hashlib import logging import queue import os -@@ -26,6 +28,7 @@ +@@ -26,6 +29,7 @@ tilert_ctrl_port: int sampling: dict | None = None prompt_token_ids: list = field(default_factory=list) @@ -24,7 +25,32 @@ @dataclass class TileRTMetadata(KVConnectorMetadata): -@@ -120,7 +123,7 @@ +@@ -39,6 +43,16 @@ + block_ids_per_group: list + params: dict + ++def _common_bytes(a: bytes, b: bytes) -> int: ++ lo, hi = 0, min(len(a), len(b)) ++ while lo < hi: ++ mid = (lo + hi + 1) // 2 ++ if a[:mid] == b[:mid]: ++ lo = mid ++ else: ++ hi = mid - 1 ++ return lo ++ + class TileRTConnector(KVConnectorBase_V1, SupportsHMA): + + def __init__(self, vllm_config, role, kv_cache_config=None): +@@ -58,6 +72,7 @@ + self._staging = None + self._send_q: queue.Queue = queue.Queue() + self._sender_thread: threading.Thread | None = None ++ self._last_ids: dict = {} + logger.info('TileRTConnector: role=%s profile=%s target=%s:%s', role, self._profile.name, self._default_host, self._default_port) + + @staticmethod +@@ -120,7 +135,7 @@ return meta def _emit(self, req_id, token_ids, groups, params) -> _ReqMeta: @@ -33,7 +59,7 @@ logger.info('claimed %s (rid=%s, %d tokens) -> %s:%d', req_id, m.rid, m.num_tokens, m.tilert_host, m.tilert_ctrl_port) return m -@@ -222,8 +225,12 @@ +@@ -222,8 +237,24 @@ remote_max_seq = int(hello['max_seq_len']) assert seq <= remote_max_seq, f'seq {seq} exceeds decode max_seq_len {remote_max_seq}' msg = {'rid': m.rid, 'rank': self._tp_rank, 'seq_len': seq, 'last_prompt_token': m.last_prompt_token, 'sampling': m.sampling, 'admission_window_s': self._admission_window()} @@ -41,15 +67,36 @@ + if self._tp_rank == 0 and m.prompt_token_ids and m.wants_ids: msg['prompt_token_ids'] = m.prompt_token_ids + # InferenceX patch: a decode node that keeps KV prefixes asks for -+ # the prompt ids in its hello; they travel as packed int32. ++ # the prompt ids in its hello; they travel as packed int32, as a ++ # suffix over the last ids this rank sent when the hello shows the ++ # decode node still holds them. ++ ids_blob = None + if self._tp_rank == 0 and m.prompt_token_ids and hello.get('want_prompt_ids'): -+ msg['prompt_ids_i32'] = base64.b64encode(array.array('i', m.prompt_token_ids).tobytes()).decode('ascii') ++ ids_blob = array.array('i', m.prompt_token_ids).tobytes() ++ dest = (m.tilert_host, m.tilert_ctrl_port) ++ prev = self._last_ids.get(dest) ++ if prev is not None and hello.get('prompt_ids_sha1') == prev[1]: ++ keep = _common_bytes(prev[0], ids_blob) // 4 ++ msg['prompt_ids_base'] = prev[1] ++ msg['prompt_ids_keep'] = keep ++ msg['prompt_ids_i32'] = base64.b64encode(ids_blob[4 * keep:]).decode('ascii') ++ else: ++ msg['prompt_ids_i32'] = base64.b64encode(ids_blob).decode('ascii') wire.send_msg(conn, msg) ack = wire.recv_msg(conn) if not ack.get('accepted'): +@@ -234,6 +265,8 @@ + logger.error('discarding %s rank=%d: admission does not match the request (%s)', m.rid, self._tp_rank, ack) + return _REJECTED_PERMANENT + generation = ack['generation'] ++ if ids_blob is not None: ++ self._last_ids[(m.tilert_host, m.tilert_ctrl_port)] = (ids_blob, hashlib.sha1(ids_blob).hexdigest()) + base = ([t.data_ptr() for t in self._staging] + if isinstance(self._staging, list) else self._staging.data_ptr()) + srcs, dsts, lens = self._profile.rdma_plan(hello, job['sections'], self._tp_rank, seq, base) --- a/profiles/glm5_rocm_engine.py +++ b/profiles/glm5_rocm_engine.py -@@ -1,13 +1,51 @@ +@@ -1,13 +1,50 @@ from __future__ import annotations import logging import os @@ -61,16 +108,22 @@ _FP8_MAX = 448.0 _INDEX_HEAD_DIM = 128 +# Prefix reuse (InferenceX patch): the engine keeps the KV of the last injected -+# prompt. When the next prompt extends it, only the rows from the first -+# differing 64-token page onward are copied in again. The prefill's vLLM -+# recomputes a partial last page, and the MTP layer's KV at position i depends -+# on token i + 1, so reuse stops at a page boundary at or before -+# min(common prefix, previous length) - 1. ++# prompt. When the next prompt's token ids extend it, the main layers keep the ++# rows below a 64-token page boundary at or before ++# min(common prefix, previous length) - 1 (vLLM recomputes a partial last ++# page) and only the rows above it are copied in again. The kept rows are the ++# prefill's KV for the same tokens from an earlier turn: vLLM recomputes part ++# of the prefix with different chunking, so layers past 0 differ from the ++# fresh transfer in rounding but not in meaning. Layer 0, which depends only on ++# the token embeddings, must match byte for byte on every rank; a few deeper ++# layers must match within a relative error. The MTP layer, which ++# reset_sequence() clears, and the index keys are always copied in full. +_REUSE_ENV = 'TILERT_PD_PREFIX_REUSE' +_REUSE_PAGE = 64 +_VERIFY_SPREAD = 192 +_VERIFY_TAIL = 64 -+_PRE_RESET_CHECKS = 3 ++_REUSE_REL_TOL = float(os.environ.get('TILERT_PD_REUSE_REL_TOL') or '0.25') ++_TOL_RANKS = (0, 1) + +def _env_on(name: str) -> bool: @@ -90,24 +143,18 @@ + diff = (a[:n] != b[:n]).nonzero() + return int(diff[0, 0]) if diff.numel() else n + -+ -+def _span(xs: list) -> str: -+ xs = sorted(set(xs)) -+ if len(xs) <= 12: -+ return ','.join(str(x) for x in xs) -+ return f'{len(xs)} in [{xs[0]}..{xs[-1]}]' -+ + def is_rocm_torch() -> bool: return getattr(torch.version, 'hip', None) is not None -@@ -45,6 +83,13 @@ +@@ -45,6 +82,14 @@ self._full_layers = list(full_layer_ordinals(self.n_layers)) self._pure_tp8 = True if pure_tp8 is None else pure_tp8 self._fp8_ki = fp8_ki_enabled() if fp8_ki is None else fp8_ki + self._prefix_reuse = _env_on(_REUSE_ENV) + self._resident_ids: torch.Tensor | None = None -+ self._pre_checks_left = _PRE_RESET_CHECKS ++ n = self.n_layers ++ self._tol_layers = sorted({1, n // 4, n // 2, (3 * n) // 4, n - 1} - {0}) if n > 1 else [] + self._pair_streams: dict = {} + self._inject_stats: dict = {} + logger.info('glm5_rocm inject: parallel per-pair copy streams; prefix reuse %s (%s)', @@ -115,7 +162,7 @@ def supports_logprobs(self) -> bool: return False -@@ -74,6 +119,83 @@ +@@ -74,6 +119,75 @@ slot[:nbytes].view(torch.float8_e4m3fn).view(L, _INDEX_HEAD_DIM)[:seq].copy_(q) slot[nbytes:nbytes + L * 4].view(torch.float32)[:seq].copy_(scale) @@ -154,52 +201,44 @@ + keep = min(_common_prefix(prev, ids), prev.numel()) - 1 + return max(0, keep // _REUSE_PAGE * _REUSE_PAGE) + -+ def _check_rows(self, targets: list, start: int) -> tuple[list, list]: -+ # The prefill resends the whole prompt, so each kept cache can be -+ # checked byte for byte against it: a spread of rows plus the last -+ # page before start. Returns per-target (differs, all-zero) flags. ++ @staticmethod ++ def _sample_rows(start: int) -> torch.Tensor: + spread = torch.linspace(0, start - 1, steps=min(_VERIFY_SPREAD, start)).round().long() + tail = torch.arange(max(0, start - _VERIFY_TAIL), start) -+ pos = torch.unique(torch.cat([spread, tail])) ++ return torch.unique(torch.cat([spread, tail])) ++ ++ def _check_prefix(self, targets: list, start: int) -> tuple[bool, float, str]: ++ # Returns (keep, worst relative error over the tolerance layers, reason). ++ pos = self._sample_rows(start) + pos_on: dict = {} -+ per_dev: dict = {} -+ for i, (_rank, _lid, _kind, dst, src) in enumerate(targets): -+ dd, sd = dst.device, src.device -+ if dd not in pos_on: -+ pos_on[dd] = pos.to(dd) -+ if sd not in pos_on: -+ pos_on[sd] = pos.to(sd) -+ got = dst.index_select(0, pos_on[dd]).view(torch.uint8) -+ want = torch.empty((pos.numel(),) + tuple(dst.shape[1:]), dtype=dst.dtype, device=dd) -+ want.copy_(src.index_select(0, pos_on[sd])) -+ per_dev.setdefault(dd, []).append((i, (got != want.view(torch.uint8)).any(), (got == 0).all())) -+ bad = [False] * len(targets) -+ zero = [False] * len(targets) -+ for items in per_dev.values(): -+ b = torch.stack([x[1] for x in items]).cpu().tolist() -+ z = torch.stack([x[2] for x in items]).cpu().tolist() -+ for (i, _b, _z), bb, zz in zip(items, b, z): -+ bad[i], zero[i] = bool(bb), bool(zz) -+ return bad, zero + -+ @staticmethod -+ def _summary(targets: list, bad: list, zero: list) -> str: -+ hit = [t for t, b in zip(targets, bad) if b] -+ if not hit: -+ return '0 differ' -+ nz = sum(1 for b, z in zip(bad, zero) if b and z) -+ kinds = {} -+ for rank, lid, kind, _d, _s in hit: -+ kinds.setdefault(kind, (set(), set())) -+ kinds[kind][0].add(lid) -+ kinds[kind][1].add(rank) -+ parts = [f'{k}: layers {_span(list(v[0]))} ranks {_span(list(v[1]))}' for k, v in sorted(kinds.items())] -+ return f'{len(hit)}/{len(targets)} differ ({nz} all-zero); ' + '; '.join(parts) ++ def rows(t): ++ d = t.device ++ if d not in pos_on: ++ pos_on[d] = pos.to(d) ++ return t.index_select(0, pos_on[d]) ++ exact, rel = [], [] ++ for rank, lid, _kind, dst, src in targets: ++ if lid == 0: ++ got = rows(dst) ++ want = torch.empty_like(got) ++ want.copy_(rows(src)) ++ exact.append((got.view(torch.uint8) != want.view(torch.uint8)).any().cpu()) ++ elif lid in self._tol_layers and rank in _TOL_RANKS: ++ got = rows(dst).float() ++ want = rows(src).to(got.device).float() ++ rel.append(((got - want).norm() / want.norm().clamp_min(1e-6)).cpu()) ++ if any(bool(x) for x in exact): ++ return False, float('nan'), 'layer 0 differs' ++ worst = max((float(x) for x in rel), default=0.0) ++ if not worst <= _REUSE_REL_TOL: ++ return False, worst, f'relative error {worst:.3g} > {_REUSE_REL_TOL:g}' ++ return True, worst, 'ok' + def inject(self, req) -> None: dl = self.dl layers = req.layers -@@ -86,16 +208,47 @@ +@@ -86,16 +200,38 @@ if n_extra and len(layers) == self.n_layers: raise RuntimeError('glm5_rocm inject: engine has an MTP block but the prefill sent no MTP-layer KV (prefill must run with --speculative-config mtp)') use = layers[:self.n_layers + n_extra] @@ -207,12 +246,6 @@ + t0 = time.time() + start = self._reuse_start(req, seq) + self._resident_ids = None -+ pre = None -+ if start > 0 and self._pre_checks_left > 0: -+ # Diagnostic: the same check before the reset shows whether the -+ # reset or the previous decode changed the kept rows. -+ self._pre_checks_left -= 1 -+ pre = self._check_rows(targets, start) dl.reset_sequence() - for rank in range(self.npes): - if rank == 0 and (not self._pure_tp8): @@ -227,18 +260,15 @@ + # from other streams. + for d in range(self.npes): + torch.cuda.synchronize(d) -+ starts = [0] * len(targets) -+ verified = 'none' -+ n_ok = 0 ++ verified, rel = 'none', float('nan') + if start > 0: -+ bad, zero = self._check_rows(targets, start) -+ starts = [0 if b else start for b in bad] -+ n_ok = len(targets) - sum(bad) -+ verified = 'ok' if n_ok == len(targets) else ('partial' if n_ok else 'mismatch') -+ if n_ok < len(targets) or pre is not None: -+ logger.warning('glm5_rocm inject: seq=%d keep=%d before reset: %s | after reset: %s', -+ seq, start, self._summary(targets, *pre) if pre is not None else 'not checked', -+ self._summary(targets, bad, zero)) ++ keep, rel, why = self._check_prefix(targets, start) ++ verified = 'ok' if keep else 'rejected' ++ if not keep: ++ logger.warning('glm5_rocm inject: seq=%d keep=%d rejected (%s); copying all rows', seq, start, why) ++ start = 0 ++ # Main layers keep [0, start); the MTP layer is always copied in full. ++ starts = [start if lid < self.n_layers else 0 for _r, lid, _k, _d, _s in targets] + t1 = time.time() + # One stream per (destination, source) device pair: torch fences a + # cross-device copy against the current stream on both devices, so a @@ -256,7 +286,7 @@ caches0 = dl._caches[0] ki_base = 2 * self.n_layers if self._pure_tp8 else 0 ki_layers = self._full_layers + ([self.n_layers] if n_extra else []) -@@ -103,9 +256,21 @@ +@@ -103,9 +239,20 @@ if lid >= len(use): break self._write_ki(caches0[ki_base + ki_slot], use[lid][0], seq) @@ -271,15 +301,14 @@ + if self._prefix_reuse: + ids = _ids_tensor(getattr(req, 'prompt_ids_i32', b'')) + self._resident_ids = ids if ids is not None and ids.numel() == seq else None -+ copied = sum(seq - s0 for s0 in starts) // max(1, len(targets)) -+ self._inject_stats = {'reuse_rows': start, 'reuse_tensors': f'{n_ok}/{len(targets)}', 'reuse_verify': verified, -+ 'mean_copy_rows': copied, 'verify_ms': round(1000 * (t1 - t0), 1), 'copy_ms': round(1000 * (t2 - t1), 1)} -+ logger.info('glm5_rocm inject: seq=%d keep=%d reused %d/%d caches, mean rows copied %d, verify=%s (%.1f ms) copy %.1f ms', -+ seq, start, n_ok, len(targets), copied, verified, 1000 * (t1 - t0), 1000 * (t2 - t1)) ++ self._inject_stats = {'reuse_rows': start, 'reuse_verify': verified, 'reuse_rel_err': round(rel, 5) if rel == rel else 'nan', ++ 'verify_ms': round(1000 * (t1 - t0), 1), 'copy_ms': round(1000 * (t2 - t1), 1)} ++ logger.info('glm5_rocm inject: seq=%d kept=%d copied=%d verify=%s rel_err=%.3g (%.1f ms) copy %.1f ms', ++ seq, start, seq - start, verified, rel, 1000 * (t1 - t0), 1000 * (t2 - t1)) def decode(self, first_token_id, max_tokens, sampling, on_token=None, cancel_event=None, grammar_session=None, top_logprobs=None): if grammar_session is not None: -@@ -140,7 +305,7 @@ +@@ -140,7 +287,7 @@ finish = self._decode_mtp(first, budget, tokens, stop_ids, on_token, cancel_event) else: finish = self._decode_plain(first, budget, tokens, stop_ids, on_token, cancel_event) @@ -290,12 +319,13 @@ return tokens --- a/receive_server.py +++ b/receive_server.py -@@ -1,3 +1,4 @@ +@@ -1,3 +1,5 @@ +import base64 ++import hashlib import logging import os import queue -@@ -22,6 +23,7 @@ +@@ -22,6 +24,7 @@ first_token_id: int | None sampling: dict | None prompt_token_ids: list = field(default_factory=list) @@ -303,26 +333,41 @@ done_ranks: set = field(default_factory=set) t_first_conn: float = 0.0 t_complete: float = 0.0 -@@ -41,6 +43,9 @@ +@@ -41,6 +44,11 @@ self.ctrl_port = ctrl_port self.device = device self.request_timeout = request_timeout + # InferenceX patch: ask the prefill for the prompt ids on every request + # so the engine can keep the KV prefix the next turn shares. + self._want_prompt_ids = (os.environ.get('TILERT_PD_PREFIX_REUSE') or '').strip().lower() not in ('', '0', 'false', 'no', 'off') ++ self._last_ids = b'' ++ self._last_ids_sha1 = None total = profile.buffer_bytes(max_seq_len) nshards = int((os.environ.get('TILERT_PD_SHARDS') or '8').strip() or 8) nshards = max(1, min(nshards, torch.cuda.device_count() or 1)) -@@ -188,6 +193,8 @@ +@@ -188,6 +196,21 @@ else: self._current = cur = ReceivedRequest(rid=rid, seq_len=int(req['seq_len']), last_prompt_token=int(req.get('last_prompt_token', 0)), first_token_id=req.get('first_token_id'), sampling=req.get('sampling'), prompt_token_ids=list(req.get('prompt_token_ids') or []), t_first_conn=time.time(), generation=self._next_generation(), state=RESERVED) logger.info('request %s: seq_len=%d (generation %d)', rid, cur.seq_len, cur.generation) + if not cur.prompt_ids_i32 and req.get('prompt_ids_i32'): -+ cur.prompt_ids_i32 = base64.b64decode(req['prompt_ids_i32']) ++ blob = base64.b64decode(req['prompt_ids_i32']) ++ base = req.get('prompt_ids_base') ++ if base is not None: ++ keep = int(req.get('prompt_ids_keep', 0)) ++ if base == self._last_ids_sha1 and 0 <= 4 * keep <= len(self._last_ids): ++ blob = self._last_ids[:4 * keep] + blob ++ else: ++ logger.warning('request %s: prompt id delta against an unknown base; prefix reuse skipped', rid) ++ blob = b'' ++ cur.prompt_ids_i32 = blob ++ if len(blob) == 4 * cur.seq_len: ++ self._last_ids, self._last_ids_sha1 = blob, hashlib.sha1(blob).hexdigest() ++ else: ++ self._last_ids, self._last_ids_sha1 = b'', None if not cur.prompt_token_ids and req.get('prompt_token_ids'): cur.prompt_token_ids = list(req['prompt_token_ids']) logger.info('request %s: prompt bitmap seeded from rank %d (%d ids)', rid, rank, len(cur.prompt_token_ids)) -@@ -212,7 +219,10 @@ +@@ -212,7 +235,13 @@ conn.settimeout(self.request_timeout) with self._lock: advisory_busy = self._current is not None and (not self._reusable(self._current)) @@ -330,6 +375,9 @@ + hello = wire.hello_msg(self._transport.name, self._transport_meta, self.max_seq_len, self.profile.layout_version, self._hello_layout, busy=advisory_busy) + if self._want_prompt_ids: + hello['want_prompt_ids'] = True ++ with self._lock: ++ if self._last_ids_sha1: ++ hello['prompt_ids_sha1'] = self._last_ids_sha1 + wire.send_msg(conn, hello) req = wire.recv_msg(conn) rid, rank = (req['rid'], int(req['rank'])) diff --git a/docs/waiver/3376.md b/docs/waiver/3376.md index 84d42134a5..92615290e4 100644 --- a/docs/waiver/3376.md +++ b/docs/waiver/3376.md @@ -25,9 +25,9 @@ vLLM is **not** patched; it runs exactly as shipped in `ghcr.io/tile-ai/tilert-r | file | change | |---|---| -| `pd_vllm/profiles/glm5_rocm_engine.py` | `RocmGlm52EngineAdapter.inject` keeps the last prompt's token ids; for the next request it copies only the KV rows from `floor((min(common prefix, previous length) - 1) / 64) * 64` onward, after checking a sample of the kept rows byte for byte against the fresh transfer, separately for each rank's kv and pe cache of every layer; a cache whose sample differs is copied in full. Mismatches are logged by layer, rank and kind, with an all-zero check and, for the first three reuse attempts, the same check before `reset_sequence()`. Index-key caches are always rewritten in full. The KV copies run on one stream per (destination, source) device pair instead of the default streams. Per-request reuse statistics are logged and added to `REQSTAT`. | -| `pd_vllm/receive_server.py` | with `TILERT_PD_PREFIX_REUSE` set, the control-plane hello carries `want_prompt_ids: true` | -| `pd_vllm/prefill_connector.py` | TP rank 0 sends the prompt ids as packed int32 when the decode hello asks for them; the existing list form for repetition penalties is unchanged | +| `pd_vllm/profiles/glm5_rocm_engine.py` | `RocmGlm52EngineAdapter.inject` keeps the last prompt's token ids; for the next request it copies only the KV rows from `floor((min(common prefix, previous length) - 1) / 64) * 64` onward, The kept rows are the prefill's KV for the same tokens from an earlier turn; vLLM recomputes part of the prefix with different chunking, so layers past 0 differ from the fresh transfer in rounding only (measured on this PR: layers 1-78 differed on 221 of 246 continuing turns, layer 0 never). Before reuse, layer 0, which depends only on the token embeddings, must match the fresh transfer byte for byte on every rank, and a sample of five deeper layers on ranks 0 and 1 must match within a relative L2 error of 0.25 (`TILERT_PD_REUSE_REL_TOL`); otherwise every row is copied. The MTP layer, which `reset_sequence()` clears, and the index-key caches are always copied in full. Index-key caches are always rewritten in full. The KV copies run on one stream per (destination, source) device pair instead of the default streams. Per-request reuse statistics are logged and added to `REQSTAT`. | +| `pd_vllm/receive_server.py` | with `TILERT_PD_PREFIX_REUSE` set, the control-plane hello carries `want_prompt_ids: true` and the hash of the last prompt ids; it rebuilds delta-encoded ids | +| `pd_vllm/prefill_connector.py` | TP rank 0 sends the prompt ids as packed int32 when the decode hello asks for them, as a suffix over the previous request's ids when the hello's hash shows the decode node holds those; the existing list form for repetition penalties is unchanged | | `pd_vllm/decode_server.py` | passes the received prompt ids to `inject` (`receive_server.py` also decodes the packed ids) | The wire protocol version, the buffer layout, the bytes transferred, the router and vLLM are unchanged. diff --git a/perf-changelog.yaml b/perf-changelog.yaml index d38b6a2d65..19baa78e70 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8764,6 +8764,6 @@ scenario-type: - agentic-coding description: - - "Keep AgentX KV prefixes resident on the TileRT decode node (experiment on top of #3366). setup_deps.sh applies patches/tilert-0.1.6.post1-pd-prefix-reuse.patch to the pip-installed tilert 0.1.6.post1 on both ranks when the recipe sets TILERT_PD_PREFIX_REUSE=1 (engine-patch waiver docs/waiver/3376.md; vLLM and the router are unpatched). On #3366's AgentX run the decode inject step, which resets the sequence and copies the whole prompt's KV into all eight rank caches serially on the default streams, took 2426 ms at p50 and 4309 ms at p90 of a 5.76 s / 8.09 s TTFT, while 200 of 239 requests continued the previous request's conversation with a median 334 new tokens. The patched inject keeps the last prompt's ids and KV and copies only the rows from the 64-token page before min(common prefix, previous length) - 1 onward, after a byte-for-byte check of sampled kept rows against the fresh transfer, per rank and per kv/pe cache of every layer (a cache whose sample differs is copied in full); index-key caches are always rewritten. The KV copies run on one stream per (destination, source) device pair. The decode hello asks for the prompt ids and the prefill connector sends them as packed int32; the KV layout and bytes transferred are unchanged." - - "在 TileRT decode 节点上保留 AgentX 的 KV 前缀（基于 #3366 的实验）。配方设置 TILERT_PD_PREFIX_REUSE=1 时，setup_deps.sh 在两侧对 pip 安装的 tilert 0.1.6.post1 应用 patches/tilert-0.1.6.post1-pd-prefix-reuse.patch（引擎补丁豁免 docs/waiver/3376.md；vLLM 与 router 未改动）。#3366 的 AgentX 运行中，decode 的 inject 步骤每次复位序列并在默认流上把整段提示的 KV 串行拷入 8 个 rank 的缓存，在 5.76 秒 / 8.09 秒的 TTFT 中占 p50 2426 ms、p90 4309 ms，而 239 个请求中有 200 个延续上一请求的对话，中位新增 334 个 token。补丁后的 inject 保留上一轮提示的 id 与 KV，只拷贝从 min(共同前缀, 上一轮长度) - 1 所在 64-token 页起的行；拷贝前按每个 rank、每层的 kv/pe 缓存分别将抽样保留行与新传输按字节比对，抽样不一致的缓存全量拷贝；索引键缓存始终重写。KV 拷贝按（目标卡，源卡）成对使用独立流。decode 的 hello 请求提示 id，prefill connector 以打包 int32 发送；KV 布局与传输字节数不变。" + - "Keep AgentX KV prefixes resident on the TileRT decode node (experiment on top of #3366). setup_deps.sh applies patches/tilert-0.1.6.post1-pd-prefix-reuse.patch to the pip-installed tilert 0.1.6.post1 on both ranks when the recipe sets TILERT_PD_PREFIX_REUSE=1 (engine-patch waiver docs/waiver/3376.md; vLLM and the router are unpatched). On #3366's AgentX run the decode inject step, which resets the sequence and copies the whole prompt's KV into all eight rank caches serially on the default streams, took 2426 ms at p50 and 4309 ms at p90 of a 5.76 s / 8.09 s TTFT, while 200 of 239 requests continued the previous request's conversation with a median 334 new tokens. The patched inject keeps the last prompt's ids and KV and copies only the rows from the 64-token page before min(common prefix, previous length) - 1 onward, The kept rows are the prefill's KV for the same tokens from an earlier turn, which vLLM partly recomputes with different chunking, so layers past 0 differ from the fresh transfer in rounding only; reuse requires layer 0 to match byte for byte on every rank and five sampled deeper layers on ranks 0 and 1 to match within a relative L2 error of 0.25. The MTP layer, which reset_sequence() clears, and the index-key caches are always copied in full. The KV copies run on one stream per (destination, source) device pair. The decode hello asks for the prompt ids and the prefill connector sends them as packed int32, delta-encoded against the previous request when the decode still holds it; the KV layout and bytes transferred are unchanged." + - "在 TileRT decode 节点上保留 AgentX 的 KV 前缀（基于 #3366 的实验）。配方设置 TILERT_PD_PREFIX_REUSE=1 时，setup_deps.sh 在两侧对 pip 安装的 tilert 0.1.6.post1 应用 patches/tilert-0.1.6.post1-pd-prefix-reuse.patch（引擎补丁豁免 docs/waiver/3376.md；vLLM 与 router 未改动）。#3366 的 AgentX 运行中，decode 的 inject 步骤每次复位序列并在默认流上把整段提示的 KV 串行拷入 8 个 rank 的缓存，在 5.76 秒 / 8.09 秒的 TTFT 中占 p50 2426 ms、p90 4309 ms，而 239 个请求中有 200 个延续上一请求的对话，中位新增 334 个 token。补丁后的 inject 保留上一轮提示的 id 与 KV，只拷贝从 min(共同前缀, 上一轮长度) - 1 所在 64-token 页起的行；保留行是 prefill 在更早一轮为相同 token 计算的 KV；vLLM 以不同分块重算部分前缀，因此第 0 层之后的层与新传输只有舍入差异；复用要求第 0 层在每个 rank 上逐字节一致，且 rank 0 与 1 上抽样的 5 个更深层相对 L2 误差不超过 0.25。reset_sequence() 会清空的 MTP 层与索引键缓存始终全量拷贝。KV 拷贝按（目标卡，源卡）成对使用独立流。decode 的 hello 请求提示 id，prefill connector 以打包 int32 发送，若 decode 仍持有上一请求则只发增量；KV 布局与传输字节数不变。" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3376 From bf9d952da139109fb6f3e2aa5bfecedc9747a660 Mon Sep 17 00:00:00 2001 From: Oseltamivir <58582368+Oseltamivir@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:23:54 +0800 Subject: [PATCH 24/25] perf(tilert): run the GLM-5.3 AgentX vLLM prefill without --enforce-eager server_tilert.sh hardcoded --enforce-eager on the vLLM prefill. It is now the PREFILL_ENFORCE_EAGER recipe knob (0 or 1, validated, forwarded by job.slurm); the AgentX recipe sets 0 so vLLM uses torch.compile and CUDA graphs. On this PR's AgentX runs the prefill-side part of TTFT stayed at 1.8 s p50 / 4.4 s p90 after the decode inject dropped to 0.13 s p50. The TileRT prefix-reuse patch is unchanged from 865925a1d, so this run compares against sweep 35827481885. --- .../multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh | 7 +++++++ benchmarks/multi_node/amd_utils/job.slurm | 3 ++- benchmarks/multi_node/amd_utils/server_tilert.sh | 10 ++++++++-- perf-changelog.yaml | 9 +++++++++ 4 files changed, 26 insertions(+), 3 deletions(-) diff --git a/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh b/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh index bff28ca342..5bc0726659 100644 --- a/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh +++ b/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh @@ -114,6 +114,13 @@ export PREFILL_KV_DTYPE=bfloat16 # The ROCm backend supports block sizes [1, 64] and vLLM picks 1, which makes # the connector's KI plane copy fail and MLA address the wrong rows. export PREFILL_BLOCK_SIZE=64 +# vLLM prefill with torch.compile and CUDA graphs (no --enforce-eager). On +# #3376's AgentX runs the prefill-side part of TTFT (vLLM prefill plus router, +# after the decode-side steps and the KV send) stayed at 1.8 s p50 / 4.4 s p90 +# for a few thousand uncached tokens per turn. Graph pools come out of the +# ~24 GiB left outside vLLM's budget after the staging shard and the non-torch +# baseline; 1 restores eager mode. +export PREFILL_ENFORCE_EAGER=0 export DECODE_KV_DTYPE=bf16 # The PD staging shard sits outside vLLM's budget, so vLLM needs 90.45 (weights) # + 40.3 (profiling) + 91.71 GiB (KV for one 1048576-token request) = 222.5 GiB diff --git a/benchmarks/multi_node/amd_utils/job.slurm b/benchmarks/multi_node/amd_utils/job.slurm index de8541a4b2..7cfd20b7a9 100755 --- a/benchmarks/multi_node/amd_utils/job.slurm +++ b/benchmarks/multi_node/amd_utils/job.slurm @@ -62,7 +62,7 @@ if [[ "$ENGINE" == "tilert" ]]; then # server_tilert.sh takes the container-creation barrier timeout from the # recipe (no 300s default as on the SGLang path); fail here, before sbatch # work is done, rather than inside the container. - check_env_vars CONTAINER_BARRIER_TIMEOUT TILERT_PD_PREFIX_REUSE + check_env_vars CONTAINER_BARRIER_TIMEOUT TILERT_PD_PREFIX_REUSE PREFILL_ENFORCE_EAGER fi # Resolve the models.yaml entry the same way server_sglang.sh does: agentic runs @@ -555,6 +555,7 @@ elif [[ "$ENGINE" == "tilert" ]]; then -e \"TILERT_EXTRA_ENV=${TILERT_EXTRA_ENV:-}\" -e SERVED_MODEL_NAME=${SERVED_MODEL_NAME} -e PREFILL_KV_DTYPE=${PREFILL_KV_DTYPE} + -e PREFILL_ENFORCE_EAGER=${PREFILL_ENFORCE_EAGER} -e PREFILL_BLOCK_SIZE=${PREFILL_BLOCK_SIZE} -e PREFILL_SPEC_TOKENS=${PREFILL_SPEC_TOKENS} -e DECODE_KV_DTYPE=${DECODE_KV_DTYPE} diff --git a/benchmarks/multi_node/amd_utils/server_tilert.sh b/benchmarks/multi_node/amd_utils/server_tilert.sh index a22cd5c433..9e99d8878a 100644 --- a/benchmarks/multi_node/amd_utils/server_tilert.sh +++ b/benchmarks/multi_node/amd_utils/server_tilert.sh @@ -15,7 +15,7 @@ check_env_vars \ TILERT_PROFILE TILERT_MODEL_TYPE TILERT_MODEL_PKG TILERT_MAX_MODEL_LEN \ TILERT_TRANSPORT TILERT_PARSER TILERT_QUEUE_TIMEOUT TILERT_WEIGHTS_DIR \ TILERT_RDMA_STRICT TILERT_CONVERT_LOCK_WAIT TILERT_SIMULATE_ACC_METHOD \ - PREFILL_KV_DTYPE PREFILL_BLOCK_SIZE PREFILL_SPEC_TOKENS DECODE_KV_DTYPE \ + PREFILL_KV_DTYPE PREFILL_BLOCK_SIZE PREFILL_SPEC_TOKENS PREFILL_ENFORCE_EAGER DECODE_KV_DTYPE \ DECODE_MTP_SIZE GPU_MEM_UTIL SERVED_MODEL_NAME \ DECODE_CTRL_PORT DECODE_HTTP_PORT PREFILL_PORT ROUTER_PORT \ DECODE_WAIT PREFILL_WAIT ROUTER_WAIT SKIP_CONTAINER_BARRIER \ @@ -291,13 +291,19 @@ start_prefill() { [[ -n "$MODEL_NAME" && "$MODEL_NAME" != "$SERVED_MODEL_NAME" ]] && served+=("$MODEL_NAME") # shellcheck disable=SC2206 local extra=( ${TILERT_PREFILL_EXTRA_FLAGS} ) + local eager=() + case "$PREFILL_ENFORCE_EAGER" in + 1) eager=(--enforce-eager) ;; + 0) ;; + *) echo "ERROR: PREFILL_ENFORCE_EAGER must be 0 or 1 (got '$PREFILL_ENFORCE_EAGER')" >&2; exit 1 ;; + esac local kv_cfg kv_cfg=$(printf '{"kv_connector":"TileRTConnector","kv_connector_module_path":"tilert.pd_vllm.prefill_connector","kv_role":"kv_producer","kv_connector_extra_config":{"tilert_host":"%s","tilert_ctrl_port":%s,"tilert_model":"%s","tilert_max_seq_len":%s,"tilert_transport":"%s"}}' \ "$DECODE_HOST" "$DECODE_CTRL_PORT" "$TILERT_PROFILE" "$TILERT_MAX_MODEL_LEN" "$TILERT_TRANSPORT") local cmd=(vllm serve "$MODEL_PATH" --served-model-name "${served[@]}" --port "$PREFILL_PORT" --tensor-parallel-size "$PREFILL_TP_SIZE" --max-model-len "$TILERT_MAX_MODEL_LEN" - --enforce-eager --trust-remote-code --return-tokens-as-token-ids + "${eager[@]}" --trust-remote-code --return-tokens-as-token-ids --gpu-memory-utilization "$GPU_MEM_UTIL" --kv-cache-dtype "$PREFILL_KV_DTYPE" --block-size "$PREFILL_BLOCK_SIZE" "${PREFILL_SPEC[@]}" diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 19baa78e70..80f3888cfe 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8767,3 +8767,12 @@ - "Keep AgentX KV prefixes resident on the TileRT decode node (experiment on top of #3366). setup_deps.sh applies patches/tilert-0.1.6.post1-pd-prefix-reuse.patch to the pip-installed tilert 0.1.6.post1 on both ranks when the recipe sets TILERT_PD_PREFIX_REUSE=1 (engine-patch waiver docs/waiver/3376.md; vLLM and the router are unpatched). On #3366's AgentX run the decode inject step, which resets the sequence and copies the whole prompt's KV into all eight rank caches serially on the default streams, took 2426 ms at p50 and 4309 ms at p90 of a 5.76 s / 8.09 s TTFT, while 200 of 239 requests continued the previous request's conversation with a median 334 new tokens. The patched inject keeps the last prompt's ids and KV and copies only the rows from the 64-token page before min(common prefix, previous length) - 1 onward, The kept rows are the prefill's KV for the same tokens from an earlier turn, which vLLM partly recomputes with different chunking, so layers past 0 differ from the fresh transfer in rounding only; reuse requires layer 0 to match byte for byte on every rank and five sampled deeper layers on ranks 0 and 1 to match within a relative L2 error of 0.25. The MTP layer, which reset_sequence() clears, and the index-key caches are always copied in full. The KV copies run on one stream per (destination, source) device pair. The decode hello asks for the prompt ids and the prefill connector sends them as packed int32, delta-encoded against the previous request when the decode still holds it; the KV layout and bytes transferred are unchanged." - "在 TileRT decode 节点上保留 AgentX 的 KV 前缀（基于 #3366 的实验）。配方设置 TILERT_PD_PREFIX_REUSE=1 时，setup_deps.sh 在两侧对 pip 安装的 tilert 0.1.6.post1 应用 patches/tilert-0.1.6.post1-pd-prefix-reuse.patch（引擎补丁豁免 docs/waiver/3376.md；vLLM 与 router 未改动）。#3366 的 AgentX 运行中，decode 的 inject 步骤每次复位序列并在默认流上把整段提示的 KV 串行拷入 8 个 rank 的缓存，在 5.76 秒 / 8.09 秒的 TTFT 中占 p50 2426 ms、p90 4309 ms，而 239 个请求中有 200 个延续上一请求的对话，中位新增 334 个 token。补丁后的 inject 保留上一轮提示的 id 与 KV，只拷贝从 min(共同前缀, 上一轮长度) - 1 所在 64-token 页起的行；保留行是 prefill 在更早一轮为相同 token 计算的 KV；vLLM 以不同分块重算部分前缀，因此第 0 层之后的层与新传输只有舍入差异；复用要求第 0 层在每个 rank 上逐字节一致，且 rank 0 与 1 上抽样的 5 个更深层相对 L2 误差不超过 0.25。reset_sequence() 会清空的 MTP 层与索引键缓存始终全量拷贝。KV 拷贝按（目标卡，源卡）成对使用独立流。decode 的 hello 请求提示 id，prefill connector 以打包 int32 发送，若 decode 仍持有上一请求则只发增量；KV 布局与传输字节数不变。" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3376 + +- config-keys: + - glm5.3-fp8-mi355x-tilert-agentic + scenario-type: + - agentic-coding + description: + - "Run the vLLM prefill of the TileRT AgentX recipe without --enforce-eager (torch.compile and CUDA graphs), behind a new PREFILL_ENFORCE_EAGER recipe knob (0 here; 1 restores eager mode) that job.slurm forwards and server_tilert.sh validates. On #3376's AgentX runs the prefill-side part of TTFT, after the decode-side steps and the KV send, stayed at 1.8 s p50 / 4.4 s p90 for a few thousand uncached tokens per turn." + - "TileRT AgentX 配方的 vLLM prefill 去掉 --enforce-eager（启用 torch.compile 与 CUDA graph），由新的配方开关 PREFILL_ENFORCE_EAGER 控制（此处为 0，设为 1 恢复 eager 模式），job.slurm 负责传递、server_tilert.sh 负责校验。在 #3376 的 AgentX 运行中，扣除 decode 侧步骤与 KV 发送后，TTFT 中 prefill 侧部分在每轮只有几千个未缓存 token 的情况下仍为 p50 1.8 秒、p90 4.4 秒。" + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3376 From 18d1d87b6a8e34ae2b249d15ec55db7c77f9ea8b Mon Sep 17 00:00:00 2001 From: Oseltamivir <58582368+Oseltamivir@users.noreply.github.com> Date: Wed, 23 Sep 2026 17:47:34 +0800 Subject: [PATCH 25/25] revert(tilert): return to the unpatched GLM-5.3 AgentX recipe [skip-sweep] Restore the tree of a0caa02b9 (#3366's head, the unpatched tilert==0.1.6.post1 recipe whose AgentX path produced sweep 35741321941): drop the pd_vllm patch, docs/waiver/3376.md, TILERT_PD_PREFIX_REUSE and PREFILL_ENFORCE_EAGER, and the two changelog entries this PR appended. The parallel KV inject that cut AgentX TTFT from 5.76 s to 3.20 s at p50 goes upstream to TileRT instead; it is re-tested here unpatched once a TileRT release carries it. Prefix reuse (25 of 246 turns bit-identical; the rest differed by 0.25-0.36 relative error) and dropping --enforce-eager (no TTFT change over the first 184 requests) are not pursued. --- .../agentic/glm5.3_fp8_mi355x_tilert.sh | 13 - benchmarks/multi_node/amd_utils/job.slurm | 4 +- .../tilert-0.1.6.post1-pd-prefix-reuse.patch | 384 ------------------ .../multi_node/amd_utils/server_tilert.sh | 10 +- benchmarks/multi_node/amd_utils/setup_deps.sh | 47 +-- docs/waiver/3376.md | 61 --- perf-changelog.yaml | 18 - 7 files changed, 10 insertions(+), 527 deletions(-) delete mode 100644 benchmarks/multi_node/amd_utils/patches/tilert-0.1.6.post1-pd-prefix-reuse.patch delete mode 100644 docs/waiver/3376.md diff --git a/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh b/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh index 5bc0726659..a8f6c3ac9f 100644 --- a/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh +++ b/benchmarks/multi_node/agentic/glm5.3_fp8_mi355x_tilert.sh @@ -95,12 +95,6 @@ export TILERT_PARSER=none export TILERT_RDMA_STRICT=0 export TILERT_CONVERT_LOCK_WAIT=21600 export TILERT_SIMULATE_ACC_METHOD=match-expected -# Keep the KV of the previous turn on the decode node and copy in only what the -# new prompt adds (patches/tilert-0.1.6.post1-pd-prefix-reuse.patch, applied by -# setup_deps.sh). At concurrency 1, 200 of 239 AgentX turns continue the -# previous request's conversation, adding 334 tokens at the median; the -# unpatched wheel re-copies the whole context into all eight ranks each turn. -export TILERT_PD_PREFIX_REUSE=1 export TILERT_WEIGHTS_DIR="/models/${MODEL_NAME}-tilert-tp${DECODE_TP}" # bf16 MLA KV on both roles. This is the only layout TileRT 0.1.6 can consume # from vLLM on ROCm: MlaNsaProfile.classify_layers infers the layout from the @@ -114,13 +108,6 @@ export PREFILL_KV_DTYPE=bfloat16 # The ROCm backend supports block sizes [1, 64] and vLLM picks 1, which makes # the connector's KI plane copy fail and MLA address the wrong rows. export PREFILL_BLOCK_SIZE=64 -# vLLM prefill with torch.compile and CUDA graphs (no --enforce-eager). On -# #3376's AgentX runs the prefill-side part of TTFT (vLLM prefill plus router, -# after the decode-side steps and the KV send) stayed at 1.8 s p50 / 4.4 s p90 -# for a few thousand uncached tokens per turn. Graph pools come out of the -# ~24 GiB left outside vLLM's budget after the staging shard and the non-torch -# baseline; 1 restores eager mode. -export PREFILL_ENFORCE_EAGER=0 export DECODE_KV_DTYPE=bf16 # The PD staging shard sits outside vLLM's budget, so vLLM needs 90.45 (weights) # + 40.3 (profiling) + 91.71 GiB (KV for one 1048576-token request) = 222.5 GiB diff --git a/benchmarks/multi_node/amd_utils/job.slurm b/benchmarks/multi_node/amd_utils/job.slurm index 7cfd20b7a9..0c779476ea 100755 --- a/benchmarks/multi_node/amd_utils/job.slurm +++ b/benchmarks/multi_node/amd_utils/job.slurm @@ -62,7 +62,7 @@ if [[ "$ENGINE" == "tilert" ]]; then # server_tilert.sh takes the container-creation barrier timeout from the # recipe (no 300s default as on the SGLang path); fail here, before sbatch # work is done, rather than inside the container. - check_env_vars CONTAINER_BARRIER_TIMEOUT TILERT_PD_PREFIX_REUSE PREFILL_ENFORCE_EAGER + check_env_vars CONTAINER_BARRIER_TIMEOUT fi # Resolve the models.yaml entry the same way server_sglang.sh does: agentic runs @@ -551,11 +551,9 @@ elif [[ "$ENGINE" == "tilert" ]]; then -e TILERT_RDMA_STRICT=${TILERT_RDMA_STRICT} -e TILERT_CONVERT_LOCK_WAIT=${TILERT_CONVERT_LOCK_WAIT} -e TILERT_SIMULATE_ACC_METHOD=${TILERT_SIMULATE_ACC_METHOD} - -e TILERT_PD_PREFIX_REUSE=${TILERT_PD_PREFIX_REUSE} -e \"TILERT_EXTRA_ENV=${TILERT_EXTRA_ENV:-}\" -e SERVED_MODEL_NAME=${SERVED_MODEL_NAME} -e PREFILL_KV_DTYPE=${PREFILL_KV_DTYPE} - -e PREFILL_ENFORCE_EAGER=${PREFILL_ENFORCE_EAGER} -e PREFILL_BLOCK_SIZE=${PREFILL_BLOCK_SIZE} -e PREFILL_SPEC_TOKENS=${PREFILL_SPEC_TOKENS} -e DECODE_KV_DTYPE=${DECODE_KV_DTYPE} diff --git a/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6.post1-pd-prefix-reuse.patch b/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6.post1-pd-prefix-reuse.patch deleted file mode 100644 index 813ed077a2..0000000000 --- a/benchmarks/multi_node/amd_utils/patches/tilert-0.1.6.post1-pd-prefix-reuse.patch +++ /dev/null @@ -1,384 +0,0 @@ ---- a/decode_server.py -+++ b/decode_server.py -@@ -145,6 +145,7 @@ - return JSONResponse({'error': 'kv_transfer_timeout', 'rid': body.rid}, status_code=504) - t_recv = time.time() - conv = server.profile.convert(server.buffer, server.base_ptr, server.max_seq_len, req, server.profile.num_ranks) -+ conv.prompt_ids_i32 = req.prompt_ids_i32 - t_conv = time.time() - engine.inject(conv) - t_inj = time.time() ---- a/prefill_connector.py 2026-09-23 10:51:39 -+++ b/prefill_connector.py 2026-09-23 14:34:58 -@@ -1,3 +1,6 @@ -+import array -+import base64 -+import hashlib - import logging - import queue - import os -@@ -26,6 +29,7 @@ - tilert_ctrl_port: int - sampling: dict | None = None - prompt_token_ids: list = field(default_factory=list) -+ wants_ids: bool = False - - @dataclass - class TileRTMetadata(KVConnectorMetadata): -@@ -39,6 +43,16 @@ - block_ids_per_group: list - params: dict - -+def _common_bytes(a: bytes, b: bytes) -> int: -+ lo, hi = 0, min(len(a), len(b)) -+ while lo < hi: -+ mid = (lo + hi + 1) // 2 -+ if a[:mid] == b[:mid]: -+ lo = mid -+ else: -+ hi = mid - 1 -+ return lo -+ - class TileRTConnector(KVConnectorBase_V1, SupportsHMA): - - def __init__(self, vllm_config, role, kv_cache_config=None): -@@ -58,6 +72,7 @@ - self._staging = None - self._send_q: queue.Queue = queue.Queue() - self._sender_thread: threading.Thread | None = None -+ self._last_ids: dict = {} - logger.info('TileRTConnector: role=%s profile=%s target=%s:%s', role, self._profile.name, self._default_host, self._default_port) - - @staticmethod -@@ -120,7 +135,7 @@ - return meta - - def _emit(self, req_id, token_ids, groups, params) -> _ReqMeta: -- m = _ReqMeta(req_id=req_id, rid=derive_rid(req_id), num_tokens=len(token_ids), last_prompt_token=int(token_ids[-1]), prompt_token_ids=list(token_ids) if params.get('_wants_prompt_ids') else [], block_ids_per_group=groups, tilert_host=params.get('tilert_host') or self._default_host, tilert_ctrl_port=int(params.get('tilert_ctrl_port', self._default_port)), sampling=params.get('sampling')) -+ m = _ReqMeta(req_id=req_id, rid=derive_rid(req_id), num_tokens=len(token_ids), last_prompt_token=int(token_ids[-1]), prompt_token_ids=list(token_ids), wants_ids=bool(params.get('_wants_prompt_ids')), block_ids_per_group=groups, tilert_host=params.get('tilert_host') or self._default_host, tilert_ctrl_port=int(params.get('tilert_ctrl_port', self._default_port)), sampling=params.get('sampling')) - logger.info('claimed %s (rid=%s, %d tokens) -> %s:%d', req_id, m.rid, m.num_tokens, m.tilert_host, m.tilert_ctrl_port) - return m - -@@ -222,8 +237,24 @@ - remote_max_seq = int(hello['max_seq_len']) - assert seq <= remote_max_seq, f'seq {seq} exceeds decode max_seq_len {remote_max_seq}' - msg = {'rid': m.rid, 'rank': self._tp_rank, 'seq_len': seq, 'last_prompt_token': m.last_prompt_token, 'sampling': m.sampling, 'admission_window_s': self._admission_window()} -- if self._tp_rank == 0 and m.prompt_token_ids: -+ if self._tp_rank == 0 and m.prompt_token_ids and m.wants_ids: - msg['prompt_token_ids'] = m.prompt_token_ids -+ # InferenceX patch: a decode node that keeps KV prefixes asks for -+ # the prompt ids in its hello; they travel as packed int32, as a -+ # suffix over the last ids this rank sent when the hello shows the -+ # decode node still holds them. -+ ids_blob = None -+ if self._tp_rank == 0 and m.prompt_token_ids and hello.get('want_prompt_ids'): -+ ids_blob = array.array('i', m.prompt_token_ids).tobytes() -+ dest = (m.tilert_host, m.tilert_ctrl_port) -+ prev = self._last_ids.get(dest) -+ if prev is not None and hello.get('prompt_ids_sha1') == prev[1]: -+ keep = _common_bytes(prev[0], ids_blob) // 4 -+ msg['prompt_ids_base'] = prev[1] -+ msg['prompt_ids_keep'] = keep -+ msg['prompt_ids_i32'] = base64.b64encode(ids_blob[4 * keep:]).decode('ascii') -+ else: -+ msg['prompt_ids_i32'] = base64.b64encode(ids_blob).decode('ascii') - wire.send_msg(conn, msg) - ack = wire.recv_msg(conn) - if not ack.get('accepted'): -@@ -234,6 +265,8 @@ - logger.error('discarding %s rank=%d: admission does not match the request (%s)', m.rid, self._tp_rank, ack) - return _REJECTED_PERMANENT - generation = ack['generation'] -+ if ids_blob is not None: -+ self._last_ids[(m.tilert_host, m.tilert_ctrl_port)] = (ids_blob, hashlib.sha1(ids_blob).hexdigest()) - base = ([t.data_ptr() for t in self._staging] - if isinstance(self._staging, list) else self._staging.data_ptr()) - srcs, dsts, lens = self._profile.rdma_plan(hello, job['sections'], self._tp_rank, seq, base) ---- a/profiles/glm5_rocm_engine.py -+++ b/profiles/glm5_rocm_engine.py -@@ -1,13 +1,50 @@ - from __future__ import annotations - import logging - import os -+import time - import torch - from tilert.pd_vllm.grammar_spec import GrammarUnsupported - from tilert.pd_vllm.sampling import resolve_top_p - logger = logging.getLogger('pd_vllm.profile.glm5_rocm') - _FP8_MAX = 448.0 - _INDEX_HEAD_DIM = 128 -+# Prefix reuse (InferenceX patch): the engine keeps the KV of the last injected -+# prompt. When the next prompt's token ids extend it, the main layers keep the -+# rows below a 64-token page boundary at or before -+# min(common prefix, previous length) - 1 (vLLM recomputes a partial last -+# page) and only the rows above it are copied in again. The kept rows are the -+# prefill's KV for the same tokens from an earlier turn: vLLM recomputes part -+# of the prefix with different chunking, so layers past 0 differ from the -+# fresh transfer in rounding but not in meaning. Layer 0, which depends only on -+# the token embeddings, must match byte for byte on every rank; a few deeper -+# layers must match within a relative error. The MTP layer, which -+# reset_sequence() clears, and the index keys are always copied in full. -+_REUSE_ENV = 'TILERT_PD_PREFIX_REUSE' -+_REUSE_PAGE = 64 -+_VERIFY_SPREAD = 192 -+_VERIFY_TAIL = 64 -+_REUSE_REL_TOL = float(os.environ.get('TILERT_PD_REUSE_REL_TOL') or '0.25') -+_TOL_RANKS = (0, 1) - -+ -+def _env_on(name: str) -> bool: -+ return (os.environ.get(name) or '').strip().lower() not in ('', '0', 'false', 'no', 'off') -+ -+ -+def _ids_tensor(blob: bytes) -> torch.Tensor | None: -+ if not blob or len(blob) % 4: -+ return None -+ return torch.frombuffer(bytearray(blob), dtype=torch.int32) -+ -+ -+def _common_prefix(a: torch.Tensor, b: torch.Tensor) -> int: -+ n = min(a.numel(), b.numel()) -+ if n <= 0: -+ return 0 -+ diff = (a[:n] != b[:n]).nonzero() -+ return int(diff[0, 0]) if diff.numel() else n -+ -+ - def is_rocm_torch() -> bool: - return getattr(torch.version, 'hip', None) is not None - -@@ -45,6 +82,14 @@ - self._full_layers = list(full_layer_ordinals(self.n_layers)) - self._pure_tp8 = True if pure_tp8 is None else pure_tp8 - self._fp8_ki = fp8_ki_enabled() if fp8_ki is None else fp8_ki -+ self._prefix_reuse = _env_on(_REUSE_ENV) -+ self._resident_ids: torch.Tensor | None = None -+ n = self.n_layers -+ self._tol_layers = sorted({1, n // 4, n // 2, (3 * n) // 4, n - 1} - {0}) if n > 1 else [] -+ self._pair_streams: dict = {} -+ self._inject_stats: dict = {} -+ logger.info('glm5_rocm inject: parallel per-pair copy streams; prefix reuse %s (%s)', -+ 'on' if self._prefix_reuse else 'off', _REUSE_ENV) - - def supports_logprobs(self) -> bool: - return False -@@ -74,6 +119,75 @@ - slot[:nbytes].view(torch.float8_e4m3fn).view(L, _INDEX_HEAD_DIM)[:seq].copy_(q) - slot[nbytes:nbytes + L * 4].view(torch.float32)[:seq].copy_(scale) - -+ def _pair_stream(self, dev: int, peer: int) -> torch.cuda.Stream: -+ key = (dev, peer) -+ st = self._pair_streams.get(key) -+ if st is None: -+ st = torch.cuda.Stream(device=dev) -+ self._pair_streams[key] = st -+ return st -+ -+ def _kv_targets(self, use: list, n_extra: int) -> list: -+ # (rank, layer, kind, cache rows, received rows) for every MLA cache the -+ # engine reads: rank 0 holds the main layers, ranks 1..7 add the MTP layer. -+ out = [] -+ for rank in range(self.npes): -+ if rank == 0 and (not self._pure_tp8): -+ continue -+ caches = self.dl._caches[rank] -+ n_pairs = self.n_layers + (n_extra if rank != 0 else 0) -+ for lid in range(min(len(use), n_pairs)): -+ _ki, kv, pe = use[lid] -+ out.append((rank, lid, 'kv', caches[2 * lid][0], kv)) -+ out.append((rank, lid, 'pe', caches[2 * lid + 1][0], pe)) -+ return out -+ -+ def _reuse_start(self, req, seq: int) -> int: -+ if not self._prefix_reuse or self._resident_ids is None: -+ return 0 -+ ids = _ids_tensor(getattr(req, 'prompt_ids_i32', b'')) -+ if ids is None or ids.numel() != seq: -+ logger.warning('glm5_rocm inject: %s prompt ids for seq %d; prefix reuse skipped', -+ 'no' if ids is None else ids.numel(), seq) -+ return 0 -+ prev = self._resident_ids -+ keep = min(_common_prefix(prev, ids), prev.numel()) - 1 -+ return max(0, keep // _REUSE_PAGE * _REUSE_PAGE) -+ -+ @staticmethod -+ def _sample_rows(start: int) -> torch.Tensor: -+ spread = torch.linspace(0, start - 1, steps=min(_VERIFY_SPREAD, start)).round().long() -+ tail = torch.arange(max(0, start - _VERIFY_TAIL), start) -+ return torch.unique(torch.cat([spread, tail])) -+ -+ def _check_prefix(self, targets: list, start: int) -> tuple[bool, float, str]: -+ # Returns (keep, worst relative error over the tolerance layers, reason). -+ pos = self._sample_rows(start) -+ pos_on: dict = {} -+ -+ def rows(t): -+ d = t.device -+ if d not in pos_on: -+ pos_on[d] = pos.to(d) -+ return t.index_select(0, pos_on[d]) -+ exact, rel = [], [] -+ for rank, lid, _kind, dst, src in targets: -+ if lid == 0: -+ got = rows(dst) -+ want = torch.empty_like(got) -+ want.copy_(rows(src)) -+ exact.append((got.view(torch.uint8) != want.view(torch.uint8)).any().cpu()) -+ elif lid in self._tol_layers and rank in _TOL_RANKS: -+ got = rows(dst).float() -+ want = rows(src).to(got.device).float() -+ rel.append(((got - want).norm() / want.norm().clamp_min(1e-6)).cpu()) -+ if any(bool(x) for x in exact): -+ return False, float('nan'), 'layer 0 differs' -+ worst = max((float(x) for x in rel), default=0.0) -+ if not worst <= _REUSE_REL_TOL: -+ return False, worst, f'relative error {worst:.3g} > {_REUSE_REL_TOL:g}' -+ return True, worst, 'ok' -+ - def inject(self, req) -> None: - dl = self.dl - layers = req.layers -@@ -86,16 +200,38 @@ - if n_extra and len(layers) == self.n_layers: - raise RuntimeError('glm5_rocm inject: engine has an MTP block but the prefill sent no MTP-layer KV (prefill must run with --speculative-config mtp)') - use = layers[:self.n_layers + n_extra] -+ targets = self._kv_targets(use, n_extra) -+ t0 = time.time() -+ start = self._reuse_start(req, seq) -+ self._resident_ids = None - dl.reset_sequence() -- for rank in range(self.npes): -- if rank == 0 and (not self._pure_tp8): -- continue -- caches = dl._caches[rank] -- n_pairs = self.n_layers + (n_extra if rank != 0 else 0) -- for lid in range(min(len(use), n_pairs)): -- _ki, kv, pe = use[lid] -- caches[2 * lid][0, :seq].copy_(kv, non_blocking=True) -- caches[2 * lid + 1][0, :seq].copy_(pe, non_blocking=True) -+ # Fence the reset on every device before reading or writing the caches -+ # from other streams. -+ for d in range(self.npes): -+ torch.cuda.synchronize(d) -+ verified, rel = 'none', float('nan') -+ if start > 0: -+ keep, rel, why = self._check_prefix(targets, start) -+ verified = 'ok' if keep else 'rejected' -+ if not keep: -+ logger.warning('glm5_rocm inject: seq=%d keep=%d rejected (%s); copying all rows', seq, start, why) -+ start = 0 -+ # Main layers keep [0, start); the MTP layer is always copied in full. -+ starts = [start if lid < self.n_layers else 0 for _r, lid, _k, _d, _s in targets] -+ t1 = time.time() -+ # One stream per (destination, source) device pair: torch fences a -+ # cross-device copy against the current stream on both devices, so a -+ # shared stream serialises all eight ranks behind one link. -+ by_pair: dict = {} -+ for (_rank, _lid, _kind, dst, src), s0 in zip(targets, starts): -+ by_pair.setdefault((dst.device.index, src.device.index), []).append((dst, src, s0)) -+ used = [] -+ for (dd, sd), items in by_pair.items(): -+ ds, ss = self._pair_stream(dd, sd), self._pair_stream(sd, dd) -+ used.extend((ds, ss)) -+ with torch.cuda.stream(ds), torch.cuda.stream(ss): -+ for dst, src, s0 in items: -+ dst[s0:seq].copy_(src[s0:seq], non_blocking=True) - caches0 = dl._caches[0] - ki_base = 2 * self.n_layers if self._pure_tp8 else 0 - ki_layers = self._full_layers + ([self.n_layers] if n_extra else []) -@@ -103,9 +239,20 @@ - if lid >= len(use): - break - self._write_ki(caches0[ki_base + ki_slot], use[lid][0], seq) -- torch.cuda.synchronize() -+ for st in used: -+ st.synchronize() -+ for d in range(self.npes): -+ torch.cuda.synchronize(d) -+ t2 = time.time() - dl.set_cur_pos(seq) - self._seq_len = seq -+ if self._prefix_reuse: -+ ids = _ids_tensor(getattr(req, 'prompt_ids_i32', b'')) -+ self._resident_ids = ids if ids is not None and ids.numel() == seq else None -+ self._inject_stats = {'reuse_rows': start, 'reuse_verify': verified, 'reuse_rel_err': round(rel, 5) if rel == rel else 'nan', -+ 'verify_ms': round(1000 * (t1 - t0), 1), 'copy_ms': round(1000 * (t2 - t1), 1)} -+ logger.info('glm5_rocm inject: seq=%d kept=%d copied=%d verify=%s rel_err=%.3g (%.1f ms) copy %.1f ms', -+ seq, start, seq - start, verified, rel, 1000 * (t1 - t0), 1000 * (t2 - t1)) - - def decode(self, first_token_id, max_tokens, sampling, on_token=None, cancel_event=None, grammar_session=None, top_logprobs=None): - if grammar_session is not None: -@@ -140,7 +287,7 @@ - finish = self._decode_mtp(first, budget, tokens, stop_ids, on_token, cancel_event) - else: - finish = self._decode_plain(first, budget, tokens, stop_ids, on_token, cancel_event) -- self.last_stats = {'finish_reason': finish} -+ self.last_stats = {'finish_reason': finish, **self._inject_stats} - if self.with_mtp: - self.last_stats.update(self._mtp_stats(base_step)) - return tokens ---- a/receive_server.py -+++ b/receive_server.py -@@ -1,3 +1,5 @@ -+import base64 -+import hashlib - import logging - import os - import queue -@@ -22,6 +24,7 @@ - first_token_id: int | None - sampling: dict | None - prompt_token_ids: list = field(default_factory=list) -+ prompt_ids_i32: bytes = b'' - done_ranks: set = field(default_factory=set) - t_first_conn: float = 0.0 - t_complete: float = 0.0 -@@ -41,6 +44,11 @@ - self.ctrl_port = ctrl_port - self.device = device - self.request_timeout = request_timeout -+ # InferenceX patch: ask the prefill for the prompt ids on every request -+ # so the engine can keep the KV prefix the next turn shares. -+ self._want_prompt_ids = (os.environ.get('TILERT_PD_PREFIX_REUSE') or '').strip().lower() not in ('', '0', 'false', 'no', 'off') -+ self._last_ids = b'' -+ self._last_ids_sha1 = None - total = profile.buffer_bytes(max_seq_len) - nshards = int((os.environ.get('TILERT_PD_SHARDS') or '8').strip() or 8) - nshards = max(1, min(nshards, torch.cuda.device_count() or 1)) -@@ -188,6 +196,21 @@ - else: - self._current = cur = ReceivedRequest(rid=rid, seq_len=int(req['seq_len']), last_prompt_token=int(req.get('last_prompt_token', 0)), first_token_id=req.get('first_token_id'), sampling=req.get('sampling'), prompt_token_ids=list(req.get('prompt_token_ids') or []), t_first_conn=time.time(), generation=self._next_generation(), state=RESERVED) - logger.info('request %s: seq_len=%d (generation %d)', rid, cur.seq_len, cur.generation) -+ if not cur.prompt_ids_i32 and req.get('prompt_ids_i32'): -+ blob = base64.b64decode(req['prompt_ids_i32']) -+ base = req.get('prompt_ids_base') -+ if base is not None: -+ keep = int(req.get('prompt_ids_keep', 0)) -+ if base == self._last_ids_sha1 and 0 <= 4 * keep <= len(self._last_ids): -+ blob = self._last_ids[:4 * keep] + blob -+ else: -+ logger.warning('request %s: prompt id delta against an unknown base; prefix reuse skipped', rid) -+ blob = b'' -+ cur.prompt_ids_i32 = blob -+ if len(blob) == 4 * cur.seq_len: -+ self._last_ids, self._last_ids_sha1 = blob, hashlib.sha1(blob).hexdigest() -+ else: -+ self._last_ids, self._last_ids_sha1 = b'', None - if not cur.prompt_token_ids and req.get('prompt_token_ids'): - cur.prompt_token_ids = list(req['prompt_token_ids']) - logger.info('request %s: prompt bitmap seeded from rank %d (%d ids)', rid, rank, len(cur.prompt_token_ids)) -@@ -212,7 +235,13 @@ - conn.settimeout(self.request_timeout) - with self._lock: - advisory_busy = self._current is not None and (not self._reusable(self._current)) -- wire.send_msg(conn, wire.hello_msg(self._transport.name, self._transport_meta, self.max_seq_len, self.profile.layout_version, self._hello_layout, busy=advisory_busy)) -+ hello = wire.hello_msg(self._transport.name, self._transport_meta, self.max_seq_len, self.profile.layout_version, self._hello_layout, busy=advisory_busy) -+ if self._want_prompt_ids: -+ hello['want_prompt_ids'] = True -+ with self._lock: -+ if self._last_ids_sha1: -+ hello['prompt_ids_sha1'] = self._last_ids_sha1 -+ wire.send_msg(conn, hello) - req = wire.recv_msg(conn) - rid, rank = (req['rid'], int(req['rank'])) - if req.get('seq_len', 0) > self.max_seq_len: diff --git a/benchmarks/multi_node/amd_utils/server_tilert.sh b/benchmarks/multi_node/amd_utils/server_tilert.sh index 9e99d8878a..a22cd5c433 100644 --- a/benchmarks/multi_node/amd_utils/server_tilert.sh +++ b/benchmarks/multi_node/amd_utils/server_tilert.sh @@ -15,7 +15,7 @@ check_env_vars \ TILERT_PROFILE TILERT_MODEL_TYPE TILERT_MODEL_PKG TILERT_MAX_MODEL_LEN \ TILERT_TRANSPORT TILERT_PARSER TILERT_QUEUE_TIMEOUT TILERT_WEIGHTS_DIR \ TILERT_RDMA_STRICT TILERT_CONVERT_LOCK_WAIT TILERT_SIMULATE_ACC_METHOD \ - PREFILL_KV_DTYPE PREFILL_BLOCK_SIZE PREFILL_SPEC_TOKENS PREFILL_ENFORCE_EAGER DECODE_KV_DTYPE \ + PREFILL_KV_DTYPE PREFILL_BLOCK_SIZE PREFILL_SPEC_TOKENS DECODE_KV_DTYPE \ DECODE_MTP_SIZE GPU_MEM_UTIL SERVED_MODEL_NAME \ DECODE_CTRL_PORT DECODE_HTTP_PORT PREFILL_PORT ROUTER_PORT \ DECODE_WAIT PREFILL_WAIT ROUTER_WAIT SKIP_CONTAINER_BARRIER \ @@ -291,19 +291,13 @@ start_prefill() { [[ -n "$MODEL_NAME" && "$MODEL_NAME" != "$SERVED_MODEL_NAME" ]] && served+=("$MODEL_NAME") # shellcheck disable=SC2206 local extra=( ${TILERT_PREFILL_EXTRA_FLAGS} ) - local eager=() - case "$PREFILL_ENFORCE_EAGER" in - 1) eager=(--enforce-eager) ;; - 0) ;; - *) echo "ERROR: PREFILL_ENFORCE_EAGER must be 0 or 1 (got '$PREFILL_ENFORCE_EAGER')" >&2; exit 1 ;; - esac local kv_cfg kv_cfg=$(printf '{"kv_connector":"TileRTConnector","kv_connector_module_path":"tilert.pd_vllm.prefill_connector","kv_role":"kv_producer","kv_connector_extra_config":{"tilert_host":"%s","tilert_ctrl_port":%s,"tilert_model":"%s","tilert_max_seq_len":%s,"tilert_transport":"%s"}}' \ "$DECODE_HOST" "$DECODE_CTRL_PORT" "$TILERT_PROFILE" "$TILERT_MAX_MODEL_LEN" "$TILERT_TRANSPORT") local cmd=(vllm serve "$MODEL_PATH" --served-model-name "${served[@]}" --port "$PREFILL_PORT" --tensor-parallel-size "$PREFILL_TP_SIZE" --max-model-len "$TILERT_MAX_MODEL_LEN" - "${eager[@]}" --trust-remote-code --return-tokens-as-token-ids + --enforce-eager --trust-remote-code --return-tokens-as-token-ids --gpu-memory-utilization "$GPU_MEM_UTIL" --kv-cache-dtype "$PREFILL_KV_DTYPE" --block-size "$PREFILL_BLOCK_SIZE" "${PREFILL_SPEC[@]}" diff --git a/benchmarks/multi_node/amd_utils/setup_deps.sh b/benchmarks/multi_node/amd_utils/setup_deps.sh index d5d786af72..8adb438681 100644 --- a/benchmarks/multi_node/amd_utils/setup_deps.sh +++ b/benchmarks/multi_node/amd_utils/setup_deps.sh @@ -92,18 +92,18 @@ _tilert_install_missing() { install_tilert_container_tools() { if command -v ip >/dev/null 2>&1 && command -v curl >/dev/null 2>&1 \ - && command -v ibv_devices >/dev/null 2>&1 && command -v patch >/dev/null 2>&1; then + && command -v ibv_devices >/dev/null 2>&1; then echo "[SETUP] Container RDMA/net tools already present" return 0 fi - echo "[SETUP] Installing iproute2 + curl + patch + ibverbs userspace in container..." + echo "[SETUP] Installing iproute2 + curl + ibverbs userspace in container..." apt-get update -q -y && apt-get install -q -y --no-install-recommends \ - iproute2 curl patch ibverbs-utils libibverbs1 librdmacm1 ibverbs-providers \ + iproute2 curl ibverbs-utils libibverbs1 librdmacm1 ibverbs-providers \ && rm -rf /var/lib/apt/lists/* - if ! command -v ip >/dev/null 2>&1 || ! command -v curl >/dev/null 2>&1 || ! command -v patch >/dev/null 2>&1; then - echo "[SETUP] ERROR: failed to install iproute2/curl/patch"; exit 1 + if ! command -v ip >/dev/null 2>&1 || ! command -v curl >/dev/null 2>&1; then + echo "[SETUP] ERROR: failed to install iproute2/curl"; exit 1 fi - _SETUP_INSTALLED+=("iproute2+curl+patch+ibverbs") + _SETUP_INSTALLED+=("iproute2+curl+ibverbs") } _tilert_install_wheel() { @@ -127,37 +127,9 @@ _tilert_install_wheel() { _SETUP_INSTALLED+=("$TILERT_PACKAGE==$TILERT_VERSION($mode)") } -# KV prefix reuse across AgentX turns. tilert 0.1.6.post1 resets the decode -# sequence and copies the whole prompt's KV into all eight rank caches on every -# request, one layer after another on the default streams: 2.4 s at the p50 -# AgentX context, although most turns only extend the previous prompt. The -# patch keeps the last prompt's KV resident, copies only the rows from the -# first differing page onward after checking a sample of the kept rows against -# the transfer, and fans the copies out on one stream per device pair. The -# prefill connector sends the prompt ids when the decode hello asks for them. -# Engine-patch waiver: docs/waiver/3376.md. Applied when the recipe sets -# TILERT_PD_PREFIX_REUSE=1; with 0 the wheel runs as shipped. -_TILERT_PD_REUSE_PATCH="$(dirname "${BASH_SOURCE[0]}")/patches/tilert-0.1.6.post1-pd-prefix-reuse.patch" - -_tilert_apply_pd_reuse_patch() { - [[ "$TILERT_PD_PREFIX_REUSE" == "1" ]] || { echo "[SETUP] KV prefix reuse off (TILERT_PD_PREFIX_REUSE=$TILERT_PD_PREFIX_REUSE); tilert unpatched"; return 0; } - local dir - dir="$("$PY" -c 'import os, tilert.pd_vllm as m; print(os.path.dirname(m.__file__))')" || { echo "[SETUP] ERROR: cannot locate tilert.pd_vllm"; exit 1; } - if grep -q "_REUSE_ENV = 'TILERT_PD_PREFIX_REUSE'" "$dir/profiles/glm5_rocm_engine.py"; then - echo "[SETUP] tilert KV prefix reuse patch already applied in $dir" - return 0 - fi - [[ -f "$_TILERT_PD_REUSE_PATCH" ]] || { echo "[SETUP] ERROR: missing $_TILERT_PD_REUSE_PATCH"; exit 1; } - echo "[SETUP] applying $(basename "$_TILERT_PD_REUSE_PATCH") to $dir" - patch -p1 -s -N -d "$dir" < "$_TILERT_PD_REUSE_PATCH" || { echo "[SETUP] ERROR: patch failed to apply"; exit 1; } - "$PY" -m py_compile "$dir"/*.py "$dir"/profiles/*.py || { echo "[SETUP] ERROR: patched tilert.pd_vllm does not compile"; exit 1; } - _SETUP_INSTALLED+=("tilert-pd-prefix-reuse.patch") -} - install_tilert_decode() { install_tilert_container_tools _tilert_install_wheel full - _tilert_apply_pd_reuse_patch _tilert_install_missing uvicorn $TILERT_HTTP_DEPS _tilert_install_missing mooncake.engine "$TILERT_TRANSPORT_DEPS" _tilert_install_missing transformers "$TILERT_TRANSFORMERS_SPEC" @@ -177,7 +149,6 @@ install_tilert_prefill() { echo "[SETUP] prefill-side vLLM $vllm_v" install_tilert_container_tools _tilert_install_wheel no-deps - _tilert_apply_pd_reuse_patch _tilert_install_missing mooncake.engine "$TILERT_TRANSPORT_DEPS" "$PY" -c "import tilert.pd_vllm.prefill_connector" 2>/dev/null || { echo "[SETUP] WARN: import tilert.pd_vllm.prefill_connector failed (vLLM will report again when loading the connector plugin):" @@ -194,11 +165,7 @@ if [[ "$ENGINE" == "vllm-disagg" ]]; then export PATH="${UCX_HOME}/bin:/usr/local/bin/etcd:/root/.cargo/bin:${PATH}" export LD_LIBRARY_PATH="${UCX_HOME}/lib:${RIXL_HOME}/lib:${RIXL_HOME}/lib/x86_64-linux-gnu:${LD_LIBRARY_PATH:-}" elif [[ "$ENGINE" == "tilert" ]]; then - check_env_vars TILERT_VERSION TILERT_PD_PREFIX_REUSE - if [[ "$TILERT_PD_PREFIX_REUSE" == "1" && "$TILERT_VERSION" != "0.1.6.post1" ]]; then - echo "[SETUP] ERROR: $_TILERT_PD_REUSE_PATCH targets tilert 0.1.6.post1, got $TILERT_VERSION; rebase the patch or set TILERT_PD_PREFIX_REUSE=0" - exit 1 - fi + check_env_vars TILERT_VERSION TILERT_PIP_SPEC="$TILERT_PACKAGE==$TILERT_VERSION" _tilert_resolve_python case "${TILERT_ROLE:-}" in diff --git a/docs/waiver/3376.md b/docs/waiver/3376.md deleted file mode 100644 index 92615290e4..0000000000 --- a/docs/waiver/3376.md +++ /dev/null @@ -1,61 +0,0 @@ -# Inference-engine patch waiver — PR #3376 - -Filed per [`docs/PR_REVIEW_CHECKLIST.md`](../PR_REVIEW_CHECKLIST.md) (image-provenance / patch item): the -TileRT launcher in this PR patches the TileRT PD (prefill/decode) data plane, the pip-installed -`tilert==0.1.6.post1` wheel's `pd_vllm` package, inside both containers at start, before serving. -vLLM is **not** patched; it runs exactly as shipped in `ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6`. - -## Config covered - -- **Master config entry:** `glm5.3-fp8-mi355x-tilert-agentic` in - [`configs/amd-master.yaml`](../../configs/amd-master.yaml). -- **Pinned images:** `ghcr.io/tile-ai/tilert-rocm-decode:0.1.6` and `ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6`; - neither carries the TileRT wheel, which - [`setup_deps.sh`](../../benchmarks/multi_node/amd_utils/setup_deps.sh) pip-installs as - `tilert==0.1.6.post1` from PyPI at container start. -- **Patch entrypoint:** `_tilert_apply_pd_reuse_patch` in `setup_deps.sh`, called from - `install_tilert_decode` and `install_tilert_prefill`, only when the recipe sets - `TILERT_PD_PREFIX_REUSE=1`. Marker-gated, compiled after application, and pinned to - `TILERT_VERSION=0.1.6.post1`; any other version fails loudly. -- **Patch file:** - [`benchmarks/multi_node/amd_utils/patches/tilert-0.1.6.post1-pd-prefix-reuse.patch`](../../benchmarks/multi_node/amd_utils/patches/tilert-0.1.6.post1-pd-prefix-reuse.patch) - (4 pure-Python files under `tilert/pd_vllm/`; no compiled component is touched). - -## What is patched - -| file | change | -|---|---| -| `pd_vllm/profiles/glm5_rocm_engine.py` | `RocmGlm52EngineAdapter.inject` keeps the last prompt's token ids; for the next request it copies only the KV rows from `floor((min(common prefix, previous length) - 1) / 64) * 64` onward, The kept rows are the prefill's KV for the same tokens from an earlier turn; vLLM recomputes part of the prefix with different chunking, so layers past 0 differ from the fresh transfer in rounding only (measured on this PR: layers 1-78 differed on 221 of 246 continuing turns, layer 0 never). Before reuse, layer 0, which depends only on the token embeddings, must match the fresh transfer byte for byte on every rank, and a sample of five deeper layers on ranks 0 and 1 must match within a relative L2 error of 0.25 (`TILERT_PD_REUSE_REL_TOL`); otherwise every row is copied. The MTP layer, which `reset_sequence()` clears, and the index-key caches are always copied in full. Index-key caches are always rewritten in full. The KV copies run on one stream per (destination, source) device pair instead of the default streams. Per-request reuse statistics are logged and added to `REQSTAT`. | -| `pd_vllm/receive_server.py` | with `TILERT_PD_PREFIX_REUSE` set, the control-plane hello carries `want_prompt_ids: true` and the hash of the last prompt ids; it rebuilds delta-encoded ids | -| `pd_vllm/prefill_connector.py` | TP rank 0 sends the prompt ids as packed int32 when the decode hello asks for them, as a suffix over the previous request's ids when the hello's hash shows the decode node holds those; the existing list form for repetition penalties is unchanged | -| `pd_vllm/decode_server.py` | passes the received prompt ids to `inject` (`receive_server.py` also decodes the packed ids) | - -The wire protocol version, the buffer layout, the bytes transferred, the router and vLLM are unchanged. -With `TILERT_PD_PREFIX_REUSE=0` the patch is not applied and the wheel runs as shipped. - -## Why the unmodified upstream stack cannot run this benchmark - -**It can.** [#3366](https://github.com/SemiAnalysisAI/InferenceX/pull/3366) runs this AgentX benchmark -on the unpatched `tilert==0.1.6.post1` wheel. This patch is a performance change, so it does not meet the -checklist's condition for a waiver. It is filed so the patch is documented and reviewable, and this PR -is an experiment measuring what upstream TileRT would gain. On #3366's AgentX run (sweep 35741321941, -239 requests, concurrency 1) the decode `inject` step took 2426 ms at p50 and 4309 ms at p90 of a -5.76 s / 8.09 s TTFT, while 200 of 239 requests continued the previous request's conversation. -Maintainers decide whether results from this PR may be published. - -## Upstream PR / issue links - -- TileRT upstream: https://github.com/tile-ai/TileRT; **upstream issue pending**, to be linked here when - opened. The measurements behind the patch are on - https://github.com/SemiAnalysisAI/InferenceX/pull/3376. -- vLLM: no change requested. - -## Removal plan - -When a TileRT release keeps decode KV prefixes across requests (or merges an equivalent of this patch), -in one PR: bump `TILERT_VERSION` in the AgentX recipe; delete -`patches/tilert-0.1.6.post1-pd-prefix-reuse.patch`, `_TILERT_PD_REUSE_PATCH`, -`_tilert_apply_pd_reuse_patch`, its two call sites and the version pin check in `setup_deps.sh`, and the -`patch` package in `install_tilert_container_tools`; drop `TILERT_PD_PREFIX_REUSE` from the recipe and -its forwarding and validation in `job.slurm` (or map it to the release's own knob); and delete this -waiver. If the PR is closed unmerged, nothing needs removing on `main`. diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 80f3888cfe..bce3819e20 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8758,21 +8758,3 @@ - "Replace the PD-buffers-in-DRAM route with tilert==0.1.6.post1 on both TileRT ranks (PyPI, 2026-09-22): the shipped wheel shards the mooncake receive buffer (decode, receive_server.py) and the connector staging buffer (prefill TP rank 0, prefill_connector.py) by layer across the eight devices (TILERT_PD_SHARDS, default 8, capped at device_count; layer lid on device lid % 8; the glm5_2 profile has 79 layers, so the heaviest shard holds 10 of them: 99.06 GiB x 10/79 = 12.54 GiB per card) instead of one dense 99.06 GiB buffer per role, and both fit on the GPU at the full 1048576-token context without host memory. Each shard is registered with mooncake on its own device, hello_layout carries per-shard bases and rdma_plan writes layer by layer into the matching remote shard, so the KV path is device-to-device again (81 GB in 751 ms = 108 GB/s on 2x8 MI350X per CrimsonDump/InferenceX@56a1e5a, against the 20.9 GiB/s measured for the host-to-host hop) and mla_nsa.convert() dequantises each layer on the shard that received it rather than on cuda:0. patches/tilert-0.1.6-pd-buffers-in-dram.patch, docs/waiver/3330.md and TILERT_PD_BUFFER_DEVICE (the AgentX recipe, server_tilert.sh, job.slurm, setup_deps.sh, including the 0.1.6 pin guard and the patch apt package) are removed; neither image is modified. Router metadata 0.1.6 -> 0.1.6.post1: pd_router.py is one of the six files post1 changes, its fixed 600 s request timeouts to vLLM and decode become TILERT_PD_HTTP_TIMEOUT_S (default 3600). GPU_MEM_UTIL stays 0.85; the 12.54 GiB staging shard sits outside vLLM's budget next to the ~6.3 GiB non-torch baseline. server_tilert.sh now honours EVAL_ONLY/RUN_EVAL on the AgentX path (GSM8K on the same router instead of an unconditional trace replay) and exports PORT for benchmark_lib's run_lm_eval; evals run with real MTP verification since simulated acceptance is already disabled under those flags. Measured by CrimsonDump on 2x8 MI350X at this context with bf16 KV, 3600 s AgentX at concurrency 1: submission_valid true, 239 successful requests, 0 errors, TTFT p50 5711.8 ms, ITL p50 2.18 ms, peak device memory 202.1 GiB per decode card and 269.2 GiB per prefill card of 287.69 GiB." - "在两侧 TileRT rank 上以 tilert==0.1.6.post1（PyPI，2026-09-22）取代 PD 缓冲进 DRAM 的方案：正式 wheel 把 mooncake 接收缓冲（decode，receive_server.py）与 connector 暂存缓冲（prefill TP rank 0，prefill_connector.py）按层分片到 8 张卡（TILERT_PD_SHARDS，默认 8，以 device_count 为上限；第 lid 层落在第 lid % 8 张卡；glm5_2 profile 共 79 层，最重分片含 10 层：99.06 GiB x 10/79 = 每卡 12.54 GiB），不再是每个角色一块致密的 99.06 GiB 缓冲，于是在完整 1048576 上下文下两者都能放进 GPU，无需主机内存。各分片在所在卡上向 mooncake 注册，hello_layout 携带各分片基址，rdma_plan 逐层写入对应的远端分片，KV 通路重回设备到设备（2x8 MI350X 上 81 GB / 751 ms = 108 GB/s，见 CrimsonDump/InferenceX@56a1e5a；对比主机到主机实测 20.9 GiB/s），mla_nsa.convert() 在收到该层的分片上反量化而非集中于 cuda:0。移除 patches/tilert-0.1.6-pd-buffers-in-dram.patch、docs/waiver/3330.md 与 TILERT_PD_BUFFER_DEVICE（AgentX 配方、server_tilert.sh、job.slurm、setup_deps.sh，含 0.1.6 版本锁定检查与 patch apt 包）；两侧镜像均无改动。router 元数据 0.1.6 -> 0.1.6.post1：pd_router.py 是 post1 改动的六个文件之一，其对 vLLM 与 decode 固定的 600 秒请求超时改为 TILERT_PD_HTTP_TIMEOUT_S（默认 3600）。GPU_MEM_UTIL 保持 0.85；12.54 GiB 暂存分片与约 6.3 GiB 非 torch 基线同在 vLLM 预算之外。server_tilert.sh 现在在 AgentX 路径也遵循 EVAL_ONLY/RUN_EVAL（在同一 router 上跑 GSM8K，而非无条件回放轨迹），并为 benchmark_lib 的 run_lm_eval 导出 PORT；这些标志下模拟接受已关闭，评测使用真实 MTP 验证。CrimsonDump 在 2x8 MI350X、该上下文、bf16 KV 下实测 3600 秒 AgentX 并发 1：submission_valid true、239 条成功、0 错误、TTFT p50 5711.8 ms、ITL p50 2.18 ms，decode 每卡峰值 202.1 GiB、prefill 每卡 269.2 GiB（卡容量 287.69 GiB）。" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3366 - -- config-keys: - - glm5.3-fp8-mi355x-tilert-agentic - scenario-type: - - agentic-coding - description: - - "Keep AgentX KV prefixes resident on the TileRT decode node (experiment on top of #3366). setup_deps.sh applies patches/tilert-0.1.6.post1-pd-prefix-reuse.patch to the pip-installed tilert 0.1.6.post1 on both ranks when the recipe sets TILERT_PD_PREFIX_REUSE=1 (engine-patch waiver docs/waiver/3376.md; vLLM and the router are unpatched). On #3366's AgentX run the decode inject step, which resets the sequence and copies the whole prompt's KV into all eight rank caches serially on the default streams, took 2426 ms at p50 and 4309 ms at p90 of a 5.76 s / 8.09 s TTFT, while 200 of 239 requests continued the previous request's conversation with a median 334 new tokens. The patched inject keeps the last prompt's ids and KV and copies only the rows from the 64-token page before min(common prefix, previous length) - 1 onward, The kept rows are the prefill's KV for the same tokens from an earlier turn, which vLLM partly recomputes with different chunking, so layers past 0 differ from the fresh transfer in rounding only; reuse requires layer 0 to match byte for byte on every rank and five sampled deeper layers on ranks 0 and 1 to match within a relative L2 error of 0.25. The MTP layer, which reset_sequence() clears, and the index-key caches are always copied in full. The KV copies run on one stream per (destination, source) device pair. The decode hello asks for the prompt ids and the prefill connector sends them as packed int32, delta-encoded against the previous request when the decode still holds it; the KV layout and bytes transferred are unchanged." - - "在 TileRT decode 节点上保留 AgentX 的 KV 前缀（基于 #3366 的实验）。配方设置 TILERT_PD_PREFIX_REUSE=1 时，setup_deps.sh 在两侧对 pip 安装的 tilert 0.1.6.post1 应用 patches/tilert-0.1.6.post1-pd-prefix-reuse.patch（引擎补丁豁免 docs/waiver/3376.md；vLLM 与 router 未改动）。#3366 的 AgentX 运行中，decode 的 inject 步骤每次复位序列并在默认流上把整段提示的 KV 串行拷入 8 个 rank 的缓存，在 5.76 秒 / 8.09 秒的 TTFT 中占 p50 2426 ms、p90 4309 ms，而 239 个请求中有 200 个延续上一请求的对话，中位新增 334 个 token。补丁后的 inject 保留上一轮提示的 id 与 KV，只拷贝从 min(共同前缀, 上一轮长度) - 1 所在 64-token 页起的行；保留行是 prefill 在更早一轮为相同 token 计算的 KV；vLLM 以不同分块重算部分前缀，因此第 0 层之后的层与新传输只有舍入差异；复用要求第 0 层在每个 rank 上逐字节一致，且 rank 0 与 1 上抽样的 5 个更深层相对 L2 误差不超过 0.25。reset_sequence() 会清空的 MTP 层与索引键缓存始终全量拷贝。KV 拷贝按（目标卡，源卡）成对使用独立流。decode 的 hello 请求提示 id，prefill connector 以打包 int32 发送，若 decode 仍持有上一请求则只发增量；KV 布局与传输字节数不变。" - pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3376 - -- config-keys: - - glm5.3-fp8-mi355x-tilert-agentic - scenario-type: - - agentic-coding - description: - - "Run the vLLM prefill of the TileRT AgentX recipe without --enforce-eager (torch.compile and CUDA graphs), behind a new PREFILL_ENFORCE_EAGER recipe knob (0 here; 1 restores eager mode) that job.slurm forwards and server_tilert.sh validates. On #3376's AgentX runs the prefill-side part of TTFT, after the decode-side steps and the KV send, stayed at 1.8 s p50 / 4.4 s p90 for a few thousand uncached tokens per turn." - - "TileRT AgentX 配方的 vLLM prefill 去掉 --enforce-eager（启用 torch.compile 与 CUDA graph），由新的配方开关 PREFILL_ENFORCE_EAGER 控制（此处为 0，设为 1 恢复 eager 模式），job.slurm 负责传递、server_tilert.sh 负责校验。在 #3376 的 AgentX 运行中，扣除 decode 侧步骤与 KV 发送后，TTFT 中 prefill 侧部分在每轮只有几千个未缓存 token 的情况下仍为 p50 1.8 秒、p90 4.4 秒。" - pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3376