[DinhKhangHoang]

Description

Serving YAMLs can run dynamo.vllm engines with replicas: N behind the Dynamo frontend (KV router). This PR adds the LiveKit 111868 arm on that path, plus a script that reads LiveKit results.

Engine/launcher (9bb5712)

• render/schema: python3 -m dynamo.vllm is a Dynamo engine (framework vllm, router profile dynamo).
  • replicas > 1 with the Dynamo frontend is allowed for dynamo.vllm only, and only without a --kv-events-config of its own.
  • DYN_SYSTEM_PORTS / DYN_KV_EVENT_PORTS are reserved.
• launch_hardware-hcm.sh reserves one Dynamo system port and one KV-event port per replica.
• serve_generic.sh starts one engine per GPU slice, each with its own --kv-events-config.
  • It waits for each replica to become healthy.
  • aiperf scrapes every replica.
• benchmarks/dynamo-vllm/Dockerfile: vllm/vllm-openai:v0.31.0 + ai-dynamo 1.6.0.dev20261018. This is the first ai-dynamo release that routes Gemma-4's hybrid KV groups.

LiveKit arm (bdc706b)

• serving/gemma4-livekit-2tp4-dyn-kv-sa.yaml: Gemma-4-31B FP8, 2× TP4 dynamo.vllm (8×H200).
  • KV router with --router-prefill-load-scale=16 and hard session affinity.
  • Scenario custom-agentic-livekit111868-v1, c96.
• Image vcr.vngcloud.vn/108942-public69/dynamo-stack/vllm-runtime:0.31.0-dyn1.6.0.dev20261018-g4strip. It is public; Dockerfile.g4strip documents the build.
  • It carries an ai-dynamo-runtime patch for the Gemma 4 empty <|channel>thought\n<channel|> that leaks into content after a tool result ([BUG]: Gemma 4 with thinking off leaks an empty <|channel>thought\n<channel|> into content after a tool result ai-dynamo/dynamo#16781).
  • The patch does not affect performance.

Results tooling (73b483c, 21fa1e6, b48f97a)

• utils/serving/livekit_ttft.py: TTFT per conc, split by a room's first turn vs later turns. First turns carry the whole system prompt uncached and set the p95 tail.
  • It reads dataset.txt to pick the plan CSV, then fetches it from the lock.
  • New lock entries livekit-111868-{v1,v2}-plan: same HF repo and revision as the traces, public.
• The serving-bench skill §4 and serving-yaml-procedures (en/zh) now describe the artifacts a run really uploads:
  • no results/ subdir;
  • server.<i>.log per replica;
  • agg_bmk.json and the fields to read in it;
  • tp in artifact names = gpus;
  • readiness 503s before warmup are not failures.
• A fresh agent read run 39270751257 through the skill without workarounds.

Evidence: run 39270751257, c96, on hardware-hcm-8x_02.

• 0 / 16,822 errors.
• TTFT p95 0.64 s: first turns 0.79, later turns 0.60.
• GPU prefix-cache hit 86.7%; the two workers split prompt tokens 52/48.
• This matches the same config on Slurm (0.66 / 0.64 / 0.65 s). The run used the pre-patch image; the -g4strip image passed a tool-call smoke test on Slurm.

Related Issue

ai-dynamo/dynamo#16781 (upstream bug the -g4strip image patches)

Type of Change

• Bug fix
• New feature
• Configuration change
• Documentation update
• Other (please describe)

Checklist

• I have tested my changes locally
  • utils/test_serving_{livekit_ttft,fetch_dataset,to_matrix,router,render,launcher_hcm}.py pass.
  • test_serving_schema.py passes except test_router_needs_hardware_hcm_runner / test_storage_needs_hardware_hcm_runner, which already fail on vng-benchmark (stale message after the rtx6000pro runner was added).
• I have updated documentation if necessary
• perf-changelog: not added. Serving YAMLs are dispatched via utils/serving/dispatch.sh, not the changelog sweep, and earlier serving/*.yaml commits carry no entry either.
• /reuse-sweep-run: N/A (no sweep).

中文

Serving YAML 现可在 Dynamo frontend（KV router）后以 replicas: N 运行 dynamo.vllm 引擎。本 PR 在此路径上新增 LiveKit 111868 配置，以及读取 LiveKit 结果的脚本。

• 引擎与启动：
  • render/schema 支持 dynamo.vllm。
  • launch_hardware-hcm.sh 与 serve_generic.sh 为每个副本分配独立的 system port 与 KV-event port。
  • 新增 benchmarks/dynamo-vllm/Dockerfile。
• LiveKit 配置： gemma4-livekit-2tp4-dyn-kv-sa.yaml，2× TP4，KV router + hard session affinity，c96。
  • 镜像 -g4strip 修补 [BUG]: Gemma 4 with thinking off leaks an empty <|channel>thought\n<channel|> into content after a tool result ai-dynamo/dynamo#16781，不影响性能。
• 结果工具：
  • utils/serving/livekit_ttft.py 按房间首轮与后续轮次拆分 TTFT，plan CSV 从锁文件获取。
  • serving-bench skill 与 procedures 文档（中英）已按实际产物更新。
• 验证： run 39270751257，c96，0 错误。
  • TTFT p95 0.64 s：首轮 0.79，后续 0.60。
  • 缓存命中 86.7%，与 Slurm 结果一致。
• 测试： 新增与相关测试通过。test_serving_schema.py 的两个失败在 vng-benchmark 上已存在。

🤖 Generated with Claude Code

[github-actions]

Thanks for the contribution!

• Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
• PR verification: Sweeps only run on labeled PRs. Add full-sweep-fail-fast (strongly recommended); use full-sweep-enabled only when matrix jobs should continue after a failure.
• After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

• **审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
• **PR 验证：**扫描仅在带有标签的 PR 上运行。强烈建议添加 full-sweep-fail-fast；仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。
• **合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。