Requesting team / customer: AMD FDE / Cohere Inference v-team (customer: Cohere — enterprise, on-prem/no-API; this benchmarking work supports Cohere's internal training pipeline evaluation and the longer-term goal of Cohere serving on AMD Instinct).
Jira Epic: pending — AIMODELS project key/access still being confirmed by requester; will link once created per Phase 1 of the intake process.
Model
HF repo: zai-org/GLM-5.1-FP8 (confirmed 2026-08-20 via HF Hub API as a distinct checkpoint from the bf16 zai-org/GLM-5.1)
Size / class: large MoE (MLA+DSA architecture)
Framework: vLLM disaggregated (prefill/decode split), MoRI EP backend
Scope: this is the WideEP disaggregation workstream specifically — separate from the vLLM co-located command-r7b/command-a-plus entries filed as companion issues. GLM-5.2 was considered earlier but is out of scope — GLM-5.1-FP8 is the only model in scope here. Targets disagg_models.yaml + run_xPyD_models.slurm / vllm_disagg_mori_io_ep.sh allowlists.
Config basis: MoRI EP flags and cuda-graph-bs sweeps copied from the existing DeepSeek-V3/DeepSeek-R1 entries as a starting point; UNTUNED for GLM-5.1-FP8 and flagged for revisit once real WideEP runs land.
kv-cache-dtype fp8 included in base_flags: originally justified by GLM-5.2's 1M-token context window OOMing KV cache in bf16 at meaningful concurrency. With GLM-5.2 now out of scope, this flag is being kept as a reasonable default for an FP8 checkpoint, but is unvalidated for GLM-5.1-FP8's own (smaller) context — revisit once real GLM-5.1-FP8 WideEP runs land.
Hardware target
AMD Instinct MI300X (gfx942). This scope is explicitly multi-node (xPyD disaggregated prefill/decode).
Workload / win-criteria
WideEP disaggregated serving benchmark — validates large-MoE expert-parallel + prefill/decode disaggregation on MI300X as a step toward Cohere production serving on Instinct. Success = clean run through the standard xPyD sweep at the tiers already agreed for Milestone 1.
Proposed changes
scripts/vllm_disagg/models.yaml — 1 new entry (GLM-5.1-FP8)
run_xPyD_models.slurm — VALID_MODELS addition (mandatory gate) + optional MORI_DP_MODE1_ALLOWED_MODELS / MORI_EP_VALID_MODELS additions
vllm_disagg_mori_io_ep.sh — MORI_DP_MODE1_ALLOWED_MODELS addition (only if DP_MODE=1 support pursued)
Draft config content available on request; no PR opened yet pending sign-off from requester and confirmation of Jira Epic link.
Open items before Phase 3 (Scoping) can close
Jira AIMODELS https://amd-hub.atlassian.net/browse/AIMODELS-1340
Confirm whether DP_MODE=1 (dp-attention MoRI EP) support is actually needed for GLM-5.1-FP8, or if DP_MODE=0 (tp-only) is sufficient for initial onboarding — affects whether the two DP_MODE=1 allowlist edits are needed
Filed per AMD's AI Models Intake Process (Confluence: MLSE space, "AI Models Intake Process"), Phase 2 "Technical Intake." This is 1 of 3 companion issues (command-r7b / command-a-plus / GLM-5.1-FP8) covering the same customer engagement.
Requesting team / customer: AMD FDE / Cohere Inference v-team (customer: Cohere — enterprise, on-prem/no-API; this benchmarking work supports Cohere's internal training pipeline evaluation and the longer-term goal of Cohere serving on AMD Instinct).
Jira Epic: pending — AIMODELS project key/access still being confirmed by requester; will link once created per Phase 1 of the intake process.
Model
HF repo: zai-org/GLM-5.1-FP8 (confirmed 2026-08-20 via HF Hub API as a distinct checkpoint from the bf16 zai-org/GLM-5.1)
Size / class: large MoE (MLA+DSA architecture)
Framework: vLLM disaggregated (prefill/decode split), MoRI EP backend
Scope: this is the WideEP disaggregation workstream specifically — separate from the vLLM co-located command-r7b/command-a-plus entries filed as companion issues. GLM-5.2 was considered earlier but is out of scope — GLM-5.1-FP8 is the only model in scope here. Targets disagg_models.yaml + run_xPyD_models.slurm / vllm_disagg_mori_io_ep.sh allowlists.
Config basis: MoRI EP flags and cuda-graph-bs sweeps copied from the existing DeepSeek-V3/DeepSeek-R1 entries as a starting point; UNTUNED for GLM-5.1-FP8 and flagged for revisit once real WideEP runs land.
kv-cache-dtype fp8 included in base_flags: originally justified by GLM-5.2's 1M-token context window OOMing KV cache in bf16 at meaningful concurrency. With GLM-5.2 now out of scope, this flag is being kept as a reasonable default for an FP8 checkpoint, but is unvalidated for GLM-5.1-FP8's own (smaller) context — revisit once real GLM-5.1-FP8 WideEP runs land.
Hardware target
AMD Instinct MI300X (gfx942). This scope is explicitly multi-node (xPyD disaggregated prefill/decode).
Workload / win-criteria
WideEP disaggregated serving benchmark — validates large-MoE expert-parallel + prefill/decode disaggregation on MI300X as a step toward Cohere production serving on Instinct. Success = clean run through the standard xPyD sweep at the tiers already agreed for Milestone 1.
Proposed changes
scripts/vllm_disagg/models.yaml — 1 new entry (GLM-5.1-FP8)
run_xPyD_models.slurm — VALID_MODELS addition (mandatory gate) + optional MORI_DP_MODE1_ALLOWED_MODELS / MORI_EP_VALID_MODELS additions
vllm_disagg_mori_io_ep.sh — MORI_DP_MODE1_ALLOWED_MODELS addition (only if DP_MODE=1 support pursued)
Draft config content available on request; no PR opened yet pending sign-off from requester and confirmation of Jira Epic link.
Open items before Phase 3 (Scoping) can close
Jira AIMODELS https://amd-hub.atlassian.net/browse/AIMODELS-1340
Confirm whether DP_MODE=1 (dp-attention MoRI EP) support is actually needed for GLM-5.1-FP8, or if DP_MODE=0 (tp-only) is sufficient for initial onboarding — affects whether the two DP_MODE=1 allowlist edits are needed
Filed per AMD's AI Models Intake Process (Confluence: MLSE space, "AI Models Intake Process"), Phase 2 "Technical Intake." This is 1 of 3 companion issues (command-r7b / command-a-plus / GLM-5.1-FP8) covering the same customer engagement.