Skip to content

fix: classify GLM SM90 KV rejection in FPE audits - #575

Closed
tedzhouhk wants to merge 2 commits into
mainfrom
hzhou-codex/fix-nightly-fpe-qualification
Closed

tedzhouhk wants to merge 2 commits into
mainfrom
hzhou-codex/fix-nightly-fpe-qualification

Conversation

@tedzhouhk

@tedzhouhk tedzhouhk commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Classify the exact native GLM-5.3-Flash SM9x FP8 sparse-KV rejection as FRAMEWORK_INCOMPATIBLE, only for a build-stage ValueError with the complete recognized diagnostic.
  • Preserve the requested identity, original error, all probe phases and absent latency. Do not change KV precision, native admission, performance math, measurement tables, required probes or the qualification gate.
  • Cover the real native producer through probe_plan, plus wrapped errors, wrong types, changed messages, query failures and required-sentinel rejection.

Root cause

Main nightly 39922445918 built/staged the artifacts but stopped during FPE qualification. All 328 unexpected rows were the intentional native rejection of FP8 sparse-MLA KV on H100/H200 with the currently modeled older vLLM/SGLang runtimes. The classifier did not recognize that diagnostic and emitted BUILD_FAILED.

This is a runtime-compatibility category, not hardware FP8 incapability. Unknown build failures and query-time failures remain blocking. A FRAMEWORK_INCOMPATIBLE row remains nonpassing and cannot satisfy a required known-good probe.

Validation

  • Before the production fix, all six new classifier/native regression cases failed with BUILD_FAILED instead of the expected compatibility category.
  • PYTHONPATH=python/aisimulate python -m pytest -c python/aisimulate/pytest.ini python/aisimulate/tests/unit/support_matrix tests/test_release_fpe.py -q: 292 passed. The synthetic Git fixture used a process-local commit.gpgsign=false override; actual commits remain signed.
  • Python/native runtime came from the unchanged, SHA256-verified staged nightly 0.13.0.dev202610230000000080, source aeb05094a243dc0a054731ac11e63d1af49b6e89; only the checked-out support-matrix tooling was changed.
  • Replayed 96 distinct real EngineHandle.compile plans, regenerating all 328 originally blocking phase rows. Every field except status matched the original report exactly, including the error and reproducer. No status-only synthetic replacement was used.
  • With those actual new results overlaid in memory onto the original 28 shards / 922,991 rows, the unchanged qualifier passes all identity, completeness and four required-sentinel checks. All 281,672 PASS rows and all original on-disk reports remain unchanged.
  • Independent review: no actionable findings; 16 additional adversarial diagnostic probes stayed blocking, and four real BF16 controls remained PERF_DATA_MISSING, not framework-incompatible.
  • Fast-CI policy/qualification/release tests, using the same Python 3.12 and locked CI dependencies: 449 passed. Removed an unnecessary build marker from the unit regression to comply with CI shard policy; all six new cases remain collected and pass under -m unit.
  • Changed-file pre-commit hooks, Ruff lint/format and git diff --check passed.
  • Independent review reconfirmed exact head 21618d24686a9963b966ad30e03cc80c785c8a7a; no actionable findings. GitHub CI is running on this head; no remote CI success claim yet.

This is local diagnostic validation, not a completed new release qualification or GPU accuracy claim. No nightly rerun/publication was triggered. The fix must land and a fresh nightly must complete before downstream dependency pins advance.

Coordination

Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
@tedzhouhk
tedzhouhk requested review from a team as code owners October 23, 2026 16:29
@coderabbitai

coderabbitai Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration
📥 Commits

Reviewing files that changed from the base of the PR and between 246d51b and 21618d2.

📒 Files selected for processing: 1
  • python/aisimulate/tests/unit/support_matrix/test_fpe_support_matrix.py

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
📥 Commits

Reviewing files that changed from the base of the PR and between aeb0509 and 246d51b.

📒 Files selected for processing: 3
  • python/aisimulate/tests/unit/support_matrix/test_fpe_qualification.py
  • python/aisimulate/tests/unit/support_matrix/test_fpe_support_matrix.py
  • python/aisimulate/tools/support_matrix/fpe_support_matrix.py
🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

📜 Recent review details
⚠️ CI failures not shown inline: 18

GitHub Actions: Full CI / 1_Full CI Success.txt: fix: classify GLM SM90 KV rejection in FPE audits

Conclusion: failure

View job details

##[group]Run python - <<'PY'
 �[36;1mpython - <<'PY'�[0m
 �[36;1mimport json�[0m
 �[36;1mimport os�[0m
 �[36;1m�[0m
 �[36;1mneeds = json.loads(os.environ["NEEDS_JSON"])�[0m
 �[36;1mplan = json.loads(os.environ["PLAN_JSON"])�[0m
 �[36;1mcomponent_outputs = {�[0m
 �[36;1m    "readme-commands": "readme_commands",�[0m
 �[36;1m    "platform-wheels": "platform_wheels",�[0m
 �[36;1m    "collector-data": "collector_data",�[0m
 �[36;1m    "prediction-regression": "prediction_regression",�[0m
 �[36;1m    "cargo-deny": "cargo_deny",�[0m
 �[36;1m    "rust": "rust",�[0m
 �[36;1m    "rust-feature-modes": "rust_feature_modes",�[0m
 �[36;1m    "public-api-rust": "public_api_rust",�[0m
 �[36;1m    "application-wheel": "application_wheel",�[0m
 �[36;1m    "python-compliance": "application_wheel",�[0m
 �[36;1m    "application-test-wheel": "application_tests",�[0m
 �[36;1m    "application-tests": "application_tests",�[0m
 �[36;1m    "python-compatibility": "python_compatibility",�[0m
 �[36;1m    "engine-golden-regression": "engine_golden_regression",�[0m
 �[36;1m    "release-artifact-contract": "release_artifact_contract",�[0m
 �[36;1m}�[0m
 �[36;1mexpected = {�[0m
 �[36;1m    "verify-target": "success",�[0m
 �[36;1m    "select-full-ci": "success",�[0m
 �[36;1m    "fast-ci": "success",�[0m
 �[36;1m}�[0m
 �[36;1minvalid_outputs = []�[0m
 �[36;1mfor job, output in component_outputs.items():�[0m
 �[36;1m    selected = plan.get(output)�[0m
 �[36;1m    if selected not in {"true", "false"}:�[0m
 �[36;1m        invalid_outputs.append(f"{output}={selected!r}")�[0m
 �[36;1m        continue�[0m
 �[36;1m    expected[job] = "success" if selected == "true" else "skipped"�[0m
 �[36;1mfailures = []�[0m
 �[36;1msummary = [�[0m
 �[36;1m    f"### Full CI evidence for `{os.environ.get('GITHUB_SHA', 'local')}`",�[0m
 �[36;1m    "",�[0m
 �[36;1m    "| Required job | Result |",�[0m
 �[36;1m    "| --- | --- |",�[0m
 �[36;1m]�[0m
 �[36;1mfor name, payload in sorted(needs.items()):�[0m
 �[36;1m    result = ...

GitHub Actions: Full CI / Full CI Success: fix: classify GLM SM90 KV rejection in FPE audits

Conclusion: failure

View job details

##[group]Run python - <<'PY'
 �[36;1mpython - <<'PY'�[0m
 �[36;1mimport json�[0m
 �[36;1mimport os�[0m
 �[36;1m�[0m
 �[36;1mneeds = json.loads(os.environ["NEEDS_JSON"])�[0m
 �[36;1mplan = json.loads(os.environ["PLAN_JSON"])�[0m
 �[36;1mcomponent_outputs = {�[0m
 �[36;1m    "readme-commands": "readme_commands",�[0m
 �[36;1m    "platform-wheels": "platform_wheels",�[0m
 �[36;1m    "collector-data": "collector_data",�[0m
 �[36;1m    "prediction-regression": "prediction_regression",�[0m
 �[36;1m    "cargo-deny": "cargo_deny",�[0m
 �[36;1m    "rust": "rust",�[0m
 �[36;1m    "rust-feature-modes": "rust_feature_modes",�[0m
 �[36;1m    "public-api-rust": "public_api_rust",�[0m
 �[36;1m    "application-wheel": "application_wheel",�[0m
 �[36;1m    "python-compliance": "application_wheel",�[0m
 �[36;1m    "application-test-wheel": "application_tests",�[0m
 �[36;1m    "application-tests": "application_tests",�[0m
 �[36;1m    "python-compatibility": "python_compatibility",�[0m
 �[36;1m    "engine-golden-regression": "engine_golden_regression",�[0m
 �[36;1m    "release-artifact-contract": "release_artifact_contract",�[0m
 �[36;1m}�[0m
 �[36;1mexpected = {�[0m
 �[36;1m    "verify-target": "success",�[0m
 �[36;1m    "select-full-ci": "success",�[0m
 �[36;1m    "fast-ci": "success",�[0m
 �[36;1m}�[0m
 �[36;1minvalid_outputs = []�[0m
 �[36;1mfor job, output in component_outputs.items():�[0m
 �[36;1m    selected = plan.get(output)�[0m
 �[36;1m    if selected not in {"true", "false"}:�[0m
 �[36;1m        invalid_outputs.append(f"{output}={selected!r}")�[0m
 �[36;1m        continue�[0m
 �[36;1m    expected[job] = "success" if selected == "true" else "skipped"�[0m
 �[36;1mfailures = []�[0m
 �[36;1msummary = [�[0m
 �[36;1m    f"### Full CI evidence for `{os.environ.get('GITHUB_SHA', 'local')}`",�[0m
 �[36;1m    "",�[0m
 �[36;1m    "| Required job | Result |",�[0m
 �[36;1m    "| --- | --- |",�[0m
 �[36;1m]�[0m
 �[36;1mfor name, payload in sorted(needs.items()):�[0m
 �[36;1m    result = ...

GitHub Actions: Full CI / 17_Select Full CI Scope.txt: fix: classify GLM SM90 KV rejection in FPE audits

Conclusion: failure

View job details

##[group]Run set -euo pipefail
 �[36;1mset -euo pipefail�[0m
 �[36;1mpaths="${RUNNER_TEMP}/full-ci-paths.b64"�[0m
 �[36;1mforce_all=false�[0m
 �[36;1m�[0m
 �[36;1mcase "${GITHUB_EVENT_NAME}:${GITHUB_REF}" in�[0m
 �[36;1m  workflow_dispatch:*|push:refs/heads/main|push:refs/heads/release/*)�[0m
 �[36;1m    force_all=true�[0m
 �[36;1m    : > "${paths}"�[0m
 �[36;1m    ;;�[0m
 �[36;1m  push:refs/heads/pull-request/*)�[0m
 �[36;1m    pr_number="${GITHUB_REF#refs/heads/pull-request/}"�[0m
 �[36;1m    if [[ ! "${pr_number}" =~ ^[0-9]+$ ]]; then�[0m
 �[36;1m      echo "::error::invalid trusted PR copy ref: ${GITHUB_REF}"�[0m
...
 �[36;1m    fi�[0m
 �[36;1m    pr_head="$(�[0m
 �[36;1m      gh api -X GET "repos/${REPOSITORY}/pulls/${pr_number}" --jq '.head.sha'�[0m
 �[36;1m    )"�[0m
 �[36;1m    if [[ "${pr_head}" != "${GITHUB_SHA}" ]]; then�[0m
 �[36;1m      echo "::error::PR head changed while resolving its changed-file set"�[0m
 �[36;1m      exit 1�[0m
 �[36;1m    fi�[0m
 �[36;1m    ;;�[0m
 �[36;1m  *)�[0m
 �[36;1m    echo "::error::unsupported Full CI event/ref: ${GITHUB_EVENT_NAME}:${GITHUB_REF}"�[0m

GitHub Actions: Fast CI / 0_Fast CI Success.txt: fix: classify GLM SM90 KV rejection in FPE audits

Conclusion: failure

View job details

##[group]Run set -euo pipefail
 �[36;1mset -euo pipefail�[0m
 �[36;1m�[0m
 �[36;1mfailures=0�[0m
 �[36;1m{�[0m
 �[36;1m  echo "### Fast CI evidence"�[0m
 �[36;1m  echo�[0m
 �[36;1m  echo "| Required job | Result |"�[0m
 �[36;1m  echo "| --- | --- |"�[0m
 �[36;1m} >> "${GITHUB_STEP_SUMMARY}"�[0m
 �[36;1m�[0m
 �[36;1mrecord_required() {�[0m
 �[36;1m  local job_name="$1"�[0m
 �[36;1m  local job_result="$2"�[0m
 �[36;1m  local outcome="PASS"�[0m
 �[36;1m  if [[ "${job_result}" != "success" ]]; then�[0m
 �[36;1m    outcome="FAIL"�[0m
 �[36;1m    failures=$((failures + 1))�[0m
 �[36;1m    echo "::error::${job_name} finished with ${job_result:-missing}"�[0m
...
 env:
   POLICY_RESULT: failure
   PYTHON_STATIC_RESULT: success
   RUST_FORMAT_RESULT: success
 ##[endgroup]
 ##[error]Repository Policy finished with failure
 ##[error]Process completed with exit code 1.

GitHub Actions: Fast CI / Fast CI Success: fix: classify GLM SM90 KV rejection in FPE audits

Conclusion: failure

View job details

##[group]Run set -euo pipefail
 �[36;1mset -euo pipefail�[0m
 �[36;1m�[0m
 �[36;1mfailures=0�[0m
 �[36;1m{�[0m
 �[36;1m  echo "### Fast CI evidence"�[0m
 �[36;1m  echo�[0m
 �[36;1m  echo "| Required job | Result |"�[0m
 �[36;1m  echo "| --- | --- |"�[0m
 �[36;1m} >> "${GITHUB_STEP_SUMMARY}"�[0m
 �[36;1m�[0m
 �[36;1mrecord_required() {�[0m
 �[36;1m  local job_name="$1"�[0m
 �[36;1m  local job_result="$2"�[0m
 �[36;1m  local outcome="PASS"�[0m
 �[36;1m  if [[ "${job_result}" != "success" ]]; then�[0m
 �[36;1m    outcome="FAIL"�[0m
 �[36;1m    failures=$((failures + 1))�[0m
 �[36;1m    echo "::error::${job_name} finished with ${job_result:-missing}"�[0m
...
 env:
   POLICY_RESULT: failure
   PYTHON_STATIC_RESULT: success
   RUST_FORMAT_RESULT: success
 ##[endgroup]
 ##[error]Repository Policy finished with failure
 ##[error]Process completed with exit code 1.

GitHub Actions: Full CI / Select Full CI Scope: fix: classify GLM SM90 KV rejection in FPE audits

Conclusion: failure

View job details

##[group]Run set -euo pipefail
 �[36;1mset -euo pipefail�[0m
 �[36;1mpaths="${RUNNER_TEMP}/full-ci-paths.b64"�[0m
 �[36;1mforce_all=false�[0m
 �[36;1m�[0m
 �[36;1mcase "${GITHUB_EVENT_NAME}:${GITHUB_REF}" in�[0m
 �[36;1m  workflow_dispatch:*|push:refs/heads/main|push:refs/heads/release/*)�[0m
 �[36;1m    force_all=true�[0m
 �[36;1m    : > "${paths}"�[0m
 �[36;1m    ;;�[0m
 �[36;1m  push:refs/heads/pull-request/*)�[0m
 �[36;1m    pr_number="${GITHUB_REF#refs/heads/pull-request/}"�[0m
 �[36;1m    if [[ ! "${pr_number}" =~ ^[0-9]+$ ]]; then�[0m
 �[36;1m      echo "::error::invalid trusted PR copy ref: ${GITHUB_REF}"�[0m
...
 �[36;1m    fi�[0m
 �[36;1m    pr_head="$(�[0m
 �[36;1m      gh api -X GET "repos/${REPOSITORY}/pulls/${pr_number}" --jq '.head.sha'�[0m
 �[36;1m    )"�[0m
 �[36;1m    if [[ "${pr_head}" != "${GITHUB_SHA}" ]]; then�[0m
 �[36;1m      echo "::error::PR head changed while resolving its changed-file set"�[0m
 �[36;1m      exit 1�[0m
 �[36;1m    fi�[0m
 �[36;1m    ;;�[0m
 �[36;1m  *)�[0m
 �[36;1m    echo "::error::unsupported Full CI event/ref: ${GITHUB_EVENT_NAME}:${GITHUB_REF}"�[0m

GitHub Actions: Fast CI / 1_Python Static Checks.txt: fix: classify GLM SM90 KV rejection in FPE audits

Conclusion: failure

View job details

##[group]Run # Copy branches can be rebased or stacked. Their previous push head
 �[36;1m# Copy branches can be rebased or stacked. Their previous push head�[0m
 �[36;1m# is not the PR base; resolve the originating PR before diffing.�[0m
 �[36;1mif [[ "${GITHUB_REF}" == refs/heads/pull-request/* ]]; then�[0m
 �[36;1m  pr_number="${GITHUB_REF#refs/heads/pull-request/}"�[0m
 �[36;1m  if [[ ! "${pr_number}" =~ ^[0-9]+$ ]]; then�[0m
 �[36;1m    echo "::error::invalid trusted PR copy ref"�[0m
 �[36;1m    exit 1�[0m
 �[36;1m  fi�[0m
 �[36;1m  pr_metadata="$(gh api -X GET "repos/${REPOSITORY}/pulls/${pr_number}" --jq '[.head.sha, .base.sha] | @tsv')"�[0m
 �[36;1m  IFS=$'\t' read -r pr_head BASE_SHA <<< "${pr_metadata}"�[0m
 �[36;1m  if [[ "${pr_head}" != "${TARGET_SHA}" || ! "${BASE_SHA}" =~ ^[0-9a-f]{40}$ ]]; then�[0m
 �[36;1m    echo "::error::trusted copy or PR comparison base changed"�[0m

GitHub Actions: Full CI / 18_Require Fast CI.txt: fix: classify GLM SM90 KV rejection in FPE audits

Conclusion: failure

View job details

##[group]Run python scripts/ci/require_fast_ci.py
 �[36;1mpython scripts/ci/require_fast_ci.py�[0m
 shell: /usr/bin/bash -e {0}
 env:
   pythonLocation: /opt/hostedtoolcache/Python/3.12.15/x64
   PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.12.15/x64/lib/pkgconfig
   Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.15/x64
   Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.15/x64
   Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.15/x64
   LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.12.15/x64/lib
   GH_***REDACTED_SECRET_ASSIGNMENT***
 ##[endgroup]
 Waiting for standalone Fast CI on pull-request/575@246d51bb69f41ce9a751df6ed262da78bb118830: (39987194856, 'in_progress', None)
 Waiting for standalone Fast CI on pull-request/575@246d51bb69f41ce9a751df6ed262da78bb118830: (39987194856, 'completed', 'failure')
 ##[error]Latest Fast CI run 39987194856 finished with failure
 ##[error]Process completed with exit code 1.

GitHub Actions: Full CI / Require Fast CI: fix: classify GLM SM90 KV rejection in FPE audits

Conclusion: failure

View job details

##[group]Run python scripts/ci/require_fast_ci.py
 �[36;1mpython scripts/ci/require_fast_ci.py�[0m
 shell: /usr/bin/bash -e {0}
 env:
   pythonLocation: /opt/hostedtoolcache/Python/3.12.15/x64
   PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.12.15/x64/lib/pkgconfig
   Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.15/x64
   Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.15/x64
   Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.15/x64
   LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.12.15/x64/lib
   GH_***REDACTED_SECRET_ASSIGNMENT***
 ##[endgroup]
 Waiting for standalone Fast CI on pull-request/575@246d51bb69f41ce9a751df6ed262da78bb118830: (39987194856, 'in_progress', None)
 Waiting for standalone Fast CI on pull-request/575@246d51bb69f41ce9a751df6ed262da78bb118830: (39987194856, 'completed', 'failure')
 ##[error]Latest Fast CI run 39987194856 finished with failure
 ##[error]Process completed with exit code 1.

GitHub Actions: Fast CI / Python Static Checks: fix: classify GLM SM90 KV rejection in FPE audits

Conclusion: failure

View job details

##[group]Run # Copy branches can be rebased or stacked. Their previous push head
 �[36;1m# Copy branches can be rebased or stacked. Their previous push head�[0m
 �[36;1m# is not the PR base; resolve the originating PR before diffing.�[0m
 �[36;1mif [[ "${GITHUB_REF}" == refs/heads/pull-request/* ]]; then�[0m
 �[36;1m  pr_number="${GITHUB_REF#refs/heads/pull-request/}"�[0m
 �[36;1m  if [[ ! "${pr_number}" =~ ^[0-9]+$ ]]; then�[0m
 �[36;1m    echo "::error::invalid trusted PR copy ref"�[0m
 �[36;1m    exit 1�[0m
 �[36;1m  fi�[0m
 �[36;1m  pr_metadata="$(gh api -X GET "repos/${REPOSITORY}/pulls/${pr_number}" --jq '[.head.sha, .base.sha] | @tsv')"�[0m
 �[36;1m  IFS=$'\t' read -r pr_head BASE_SHA <<< "${pr_metadata}"�[0m
 �[36;1m  if [[ "${pr_head}" != "${TARGET_SHA}" || ! "${BASE_SHA}" =~ ^[0-9a-f]{40}$ ]]; then�[0m
 �[36;1m    echo "::error::trusted copy or PR comparison base changed"�[0m

GitHub Actions: Full CI / 19_Verify Exact Target.txt: fix: classify GLM SM90 KV rejection in FPE audits

Conclusion: failure

View job details

##[group]Run if [[ "${GITHUB_EVENT_NAME}" == "workflow_dispatch" && -z "${EXPECTED_SHA}" ]]; then
 �[36;1mif [[ "${GITHUB_EVENT_NAME}" == "workflow_dispatch" && -z "${EXPECTED_SHA}" ]]; then�[0m
 �[36;1m  echo "::error::manual Full CI requires a nonempty expected_sha"�[0m
 �[36;1m  exit 1�[0m
 �[36;1mfi�[0m
 �[36;1mif [[ -n "${EXPECTED_SHA}" && "${EXPECTED_SHA}" != "${RUN_SHA}" ]]; then�[0m
 �[36;1m  echo "::error::expected ${EXPECTED_SHA}, but workflow is pinned to ${RUN_SHA}"�[0m
...
 ##[group]Run set -euo pipefail
 �[36;1mset -euo pipefail�[0m
 �[36;1mpr_number="${GITHUB_REF#refs/heads/pull-request/}"�[0m
 �[36;1mif [[ ! "${pr_number}" =~ ^[0-9]+$ ]]; then�[0m
 �[36;1m  echo "::error::invalid trusted PR copy ref: ${GITHUB_REF}"�[0m
...
 �[36;1m  gh api -X GET "repos/${REPOSITORY}/pulls/${pr_number}" \�[0m
 �[36;1m    --jq '[.head.sha, .base.sha] | @tsv'�[0m
 �[36;1m)"�[0m
 �[36;1mIFS=$'\t' read -r pr_head pr_base <<< "${pr_metadata}"�[0m
 �[36;1mif [[ "${pr_head}" != "${RUN_SHA}" ]]; then�[0m
 �[36;1m  echo "::error::trusted copy ${RUN_SHA} does not match PR #${pr_number} head ${pr_head}"�[0m
 �[36;1m  exit 1�[0m
 �[36;1mfi�[0m
 �[36;1mif [[ ! "${pr_base}" =~ ^[0-9a-f]{40}$ ]]; then�[0m
 �[36;1m  echo "::error::PR comparison base must resolve to a full commit SHA"�[0m
...
 ##[group]Run set -euo pipefail
...
 �[36;1m  # Other manual runs and newly created release branches compare with�[0m
 �[36;1m  # one snapshot of main, shared by the collector and prediction gates.�[0m
 �[36;1m  comparison_sha="$(gh api -X GET "repos/${REPOSITORY}/commits/main" --jq '.sha')"�[0m
 �[36;1mfi�[0m
 �[36;1mif [[ ! "${comparison_sha}" =~ ^[0-9a-f]{40}$ ]]; then�[0m
 �[36;1m  echo "::error::comparison base must resolve to a full commit SHA"�[0m

GitHub Actions: Full CI / Verify Exact Target: fix: classify GLM SM90 KV rejection in FPE audits

Conclusion: failure

View job details

##[group]Run if [[ "${GITHUB_EVENT_NAME}" == "workflow_dispatch" && -z "${EXPECTED_SHA}" ]]; then
 �[36;1mif [[ "${GITHUB_EVENT_NAME}" == "workflow_dispatch" && -z "${EXPECTED_SHA}" ]]; then�[0m
 �[36;1m  echo "::error::manual Full CI requires a nonempty expected_sha"�[0m
 �[36;1m  exit 1�[0m
 �[36;1mfi�[0m
 �[36;1mif [[ -n "${EXPECTED_SHA}" && "${EXPECTED_SHA}" != "${RUN_SHA}" ]]; then�[0m
 �[36;1m  echo "::error::expected ${EXPECTED_SHA}, but workflow is pinned to ${RUN_SHA}"�[0m

GitHub Actions: Fast CI / 2_Repository Policy.txt: fix: classify GLM SM90 KV rejection in FPE audits

Conclusion: failure

View job details

##[group]Run groups=$(python -c 'import pathlib, tomllib; print(" ".join(tomllib.loads(pathlib.Path("scripts/pyproject.toml").read_text())["dependency-groups"]))')
 �[36;1mgroups=$(python -c 'import pathlib, tomllib; print(" ".join(tomllib.loads(pathlib.Path("scripts/pyproject.toml").read_text())["dependency-groups"]))')�[0m
 �[36;1mfor group in ${groups}; do�[0m
 �[36;1m  directory=${group//-/_}�[0m
 �[36;1m  uv pip compile --group "scripts/pyproject.toml:${group}" --generate-hashes \�[0m
 �[36;1m    --python-version 3.12 --universal -o "scripts/${directory}/requirements.txt" > /dev/null�[0m
 �[36;1mdone�[0m
 �[36;1mif [[ -n "$(git status --porcelain --untracked-files=all -- scripts)" ]]; then�[0m
 �[36;1m  git diff -- scripts�[0m
 �[36;1m  git status --short --untracked-files=all -- scripts�[0m
 �[36;1m  echo "::error::Script dependency locks are out of date or missing."�[0m
...
 ##[group]Run python -m pytest -c /dev/null .github/codeowners/test_*.py -q \
...
 �[36;1m  --contributors-out CONTRIBUTORS.md�[0m
 �[36;1mif [[ -n "$(git status --porcelain --untracked-files=all -- \�[0m
 �[36;1m    CODEOWNERS CONTRIBUTORS.md)" ]]; then�[0m
 �[36;1m  git diff -- CODEOWNERS CONTRIBUTORS.md || true�[0m
 �[36;1m  git status --short --untracked-files=all -- CODEOWNERS CONTRIBUTORS.md�[0m
 �[36;1m  echo "::error::Generated CODEOWNERS artifacts are out of date."�[0m
...
 ##[group]Run python scripts/prediction_regression/check_prediction_numerics.py --fetch-baseline-only
...
 tests/test_ci_workflow_contracts.py:749: AssertionError
 =========================== short test summary info ============================
 FAILED ../../../../../dev::test_full_ci_owns_migrated_expensive_suites - assert False
  +  where False = all(<generator object test_full_ci_owns_migrated_expensive_suites.<locals>.<genexpr> at 0x7fb897904d40>)
 1 failed, 448 passed in 11.53s
 ##[error]Process completed with exit code 1.

GitHub Actions: Full CI / Verify Exact Target: fix: classify GLM SM90 KV rejection in FPE audits

Conclusion: failure

View job details

##[group]Run set -euo pipefail
 �[36;1mset -euo pipefail�[0m
 �[36;1mpr_number="${GITHUB_REF#refs/heads/pull-request/}"�[0m
 �[36;1mif [[ ! "${pr_number}" =~ ^[0-9]+$ ]]; then�[0m
 �[36;1m  echo "::error::invalid trusted PR copy ref: ${GITHUB_REF}"�[0m
...
 �[36;1m  gh api -X GET "repos/${REPOSITORY}/pulls/${pr_number}" \�[0m
 �[36;1m    --jq '[.head.sha, .base.sha] | @tsv'�[0m
 �[36;1m)"�[0m
 �[36;1mIFS=$'\t' read -r pr_head pr_base <<< "${pr_metadata}"�[0m
 �[36;1mif [[ "${pr_head}" != "${RUN_SHA}" ]]; then�[0m
 �[36;1m  echo "::error::trusted copy ${RUN_SHA} does not match PR #${pr_number} head ${pr_head}"�[0m
 �[36;1m  exit 1�[0m
 �[36;1mfi�[0m
 �[36;1mif [[ ! "${pr_base}" =~ ^[0-9a-f]{40}$ ]]; then�[0m
 �[36;1m  echo "::error::PR comparison base must resolve to a full commit SHA"�[0m

GitHub Actions: Full CI / Verify Exact Target: fix: classify GLM SM90 KV rejection in FPE audits

Conclusion: failure

View job details

##[group]Run set -euo pipefail
 �[36;1mset -euo pipefail�[0m
 �[36;1mcomparison_sha=""�[0m
 �[36;1mif [[ "${GITHUB_REF}" == refs/heads/pull-request/* ]]; then�[0m
 �[36;1m  # Compare a stacked PR with its validated target, not later main changes.�[0m
 �[36;1m  comparison_sha="${PR_BASE_SHA}"�[0m
 �[36;1melif [[ "${GITHUB_EVENT_NAME}" == "push" &&�[0m
 �[36;1m      ( "${GITHUB_REF}" == "refs/heads/main" || "${GITHUB_REF}" == refs/heads/release/* ) &&�[0m
 �[36;1m      -n "${BEFORE_SHA}" && "${BEFORE_SHA}" != "0000000000000000000000000000000000000000" ]]; then�[0m
 �[36;1m  comparison_sha="${BEFORE_SHA}"�[0m
 �[36;1melse�[0m
 �[36;1m  # Other manual runs and newly created release branches compare with�[0m
 �[36;1m  # one snapshot of main, shared by the collector and prediction gates.�[0m
 �[36;1m  comparison_sha="$(gh api -X GET "repos/${REPOSITORY}/commits/main" --jq '.sha')"�[0m
 �[36;1mfi�[0m
 �[36;1mif [[ ! "${comparison_sha}" =~ ^[0-9a-f]{40}$ ]]; then�[0m
 �[36;1m  echo "::error::comparison base must resolve to a full commit SHA"�[0m

GitHub Actions: Fast CI / Repository Policy: fix: classify GLM SM90 KV rejection in FPE audits

Conclusion: failure

View job details

##[group]Run groups=$(python -c 'import pathlib, tomllib; print(" ".join(tomllib.loads(pathlib.Path("scripts/pyproject.toml").read_text())["dependency-groups"]))')
 �[36;1mgroups=$(python -c 'import pathlib, tomllib; print(" ".join(tomllib.loads(pathlib.Path("scripts/pyproject.toml").read_text())["dependency-groups"]))')�[0m
 �[36;1mfor group in ${groups}; do�[0m
 �[36;1m  directory=${group//-/_}�[0m
 �[36;1m  uv pip compile --group "scripts/pyproject.toml:${group}" --generate-hashes \�[0m
 �[36;1m    --python-version 3.12 --universal -o "scripts/${directory}/requirements.txt" > /dev/null�[0m
 �[36;1mdone�[0m
 �[36;1mif [[ -n "$(git status --porcelain --untracked-files=all -- scripts)" ]]; then�[0m
 �[36;1m  git diff -- scripts�[0m
 �[36;1m  git status --short --untracked-files=all -- scripts�[0m
 �[36;1m  echo "::error::Script dependency locks are out of date or missing."�[0m

GitHub Actions: Fast CI / Repository Policy: fix: classify GLM SM90 KV rejection in FPE audits

Conclusion: failure

View job details

##[group]Run python -m pytest -c /dev/null .github/codeowners/test_*.py -q \
 �[36;1mpython -m pytest -c /dev/null .github/codeowners/test_*.py -q \�[0m
 �[36;1m  -p no:cacheprovider \�[0m
 �[36;1m  --override-ini="addopts=" \�[0m
 �[36;1m  --override-ini="filterwarnings="�[0m
 �[36;1mpython .github/codeowners/build_codeowners.py \�[0m
 �[36;1m  --areas .github/codeowners/areas.yaml \�[0m
 �[36;1m  --repo . \�[0m
 �[36;1m  --strict�[0m
 �[36;1mpython .github/codeowners/emit_codeowners.py \�[0m
 �[36;1m  --areas .github/codeowners/areas.yaml \�[0m
 �[36;1m  --out CODEOWNERS \�[0m
 �[36;1m  --external .github/codeowners/external_contributors.yaml \�[0m
 �[36;1m  --contributors-out CONTRIBUTORS.md�[0m
 �[36;1mif [[ -n "$(git status --porcelain --untracked-files=all -- \�[0m
 �[36;1m    CODEOWNERS CONTRIBUTORS.md)" ]]; then�[0m
 �[36;1m  git diff -- CODEOWNERS CONTRIBUTORS.md || true�[0m
 �[36;1m  git status --short --untracked-files=all -- CODEOWNERS CONTRIBUTORS.md�[0m
 �[36;1m  echo "::error::Generated CODEOWNERS artifacts are out of date."�[0m

GitHub Actions: Fast CI / Repository Policy: fix: classify GLM SM90 KV rejection in FPE audits

Conclusion: failure

View job details

##[group]Run python scripts/prediction_regression/check_prediction_numerics.py --fetch-baseline-only
 �[36;1mpython scripts/prediction_regression/check_prediction_numerics.py --fetch-baseline-only�[0m
 �[36;1mpython -m pytest -c /dev/null tests/test_ci_workflow_contracts.py tests/test_ci_qualification.py tests/test_release_fpe.py -q \�[0m
 �[36;1m  -p no:cacheprovider \�[0m
 �[36;1m  --override-ini="addopts=" \�[0m
 �[36;1m  --override-ini="filterwarnings=error"�[0m
 shell: /usr/bin/bash -e {0}
 env:
   EXPECTED_SHA:
   TARGET_SHA: 246d51bb69f41ce9a751df6ed262da78bb118830
   pythonLocation: /opt/hostedtoolcache/Python/3.12.15/x64
   PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.12.15/x64/lib/pkgconfig
   Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.15/x64
   Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.15/x64
   Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.15/x64
   LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.12.15/x64/lib
 ##[endgroup]
 Numerical baseline available: d066e918705b98e2d55eed55743ce8d225f129ea
 ........................................................................ [ 16%]
 .........F.............................................................. [ 32%]
 ........................................................................ [ 48%]
 ........................................................................ [ 64%]
 ........................................................................ [ 80%]
 ........................................................................ [ 96%]
 .................                                                        [100%]
 =================================== FAILURES ===================================
 _________________ test_full_ci_owns_migrated_expensive_suites __________________
     def test_full_ci_owns_migrated_expensive_suites() -> None:
         jobs = _workflow("ci.yml")["jobs"]
         assert {
             "rust-feature-modes",
             "python-compatibility",
             "engine-gol...
🧰 Additional context used
📚 Code guidelines: 3
python/aisimulate/.claude/rules/perfmodel-api.md — auto-discovered
AGENTS.md — configured
REVIEW.md — configured
📓 Path-based instructions: 4
Read REVIEW.md before commenting.

⚙️ CodeRabbit configuration file

Files:

  • python/aisimulate/tools/support_matrix/fpe_support_matrix.py
  • python/aisimulate/tests/unit/support_matrix/test_fpe_qualification.py
  • python/aisimulate/tests/unit/support_matrix/test_fpe_support_matrix.py
Applicability reference: AGENTS.md: # AGENTS Before changing a performance model, its configuration, or a caller in Rust, Python, CLI, Sweeper, Replay, or Planner, MUST read and follow [`perfmodel-api.md`](python/aisimulat...

📄 CodeRabbit inference engine (python/aisimulate/.claude/rules/perfmodel-api.md)

Files:

  • python/aisimulate/tests/unit/support_matrix/test_fpe_qualification.py
  • python/aisimulate/tests/unit/support_matrix/test_fpe_support_matrix.py
Source excerpt: Keep run logs, investigation notes, and benchmark results out of `docs/`; measured evidence belongs under `benchmarks/evidence/`.

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • python/aisimulate/tools/support_matrix/fpe_support_matrix.py
  • python/aisimulate/tests/unit/support_matrix/test_fpe_qualification.py
  • python/aisimulate/tests/unit/support_matrix/test_fpe_support_matrix.py
Source excerpt: Binary parquet changes require a machine-readable summary of keys, shapes, units, row counts, coverage, and anomalies plus proof that a real consumer reaches the new data.

📄 CodeRabbit inference engine (REVIEW.md)

Files:

  • python/aisimulate/tools/support_matrix/fpe_support_matrix.py
  • python/aisimulate/tests/unit/support_matrix/test_fpe_qualification.py
  • python/aisimulate/tests/unit/support_matrix/test_fpe_support_matrix.py
🔀 Multi-repo context ai-dynamo/dynamo, ai-dynamo/aiconfigurator

Linked repositories findings

ai-dynamo/dynamo

Inspected ref f021648, the branch for open PR #16945.

  • Dynamo pins AISimulate exactly to 0.13.0.dev2 in both project dependencies and the staged wheel requirements, so this classifier change must land in a compatible published artifact before Dynamo advances its pin. [::ai-dynamo/dynamo::pyproject.toml:17] [::ai-dynamo/dynamo::container/deps/requirements.aisimulate.txt:4-6]
  • Dynamo’s replay boundary delegates agentic snapshot handling to AISimulate and has AgentX tests comparing snapshot outputs, but no changed public Dynamo API was identified. [::ai-dynamo/dynamo::components/src/dynamo/replay/api.py:285] [::ai-dynamo/dynamo::components/src/dynamo/replay/tests/test_agentx_g2.py:357-360]

ai-dynamo/aiconfigurator

Inspected ref f254959.

  • The frozen support-matrix implementation already recognizes FRAMEWORK_INCOMPATIBLE as a valid status and preserves it when a known framework gap is detected; only successful hybrid retries become HYBRID_PASS. [::ai-dynamo/aiconfigurator::tools/support_matrix/support_matrix.py:38-45] [::ai-dynamo/aiconfigurator::tools/support_matrix/support_matrix.py:686-745]
  • Matrix validation requires every framework-incompatible row to retain a nonempty diagnostic reason, matching this PR’s preservation of the original build error. [::ai-dynamo/aiconfigurator::tools/support_matrix/compare_support_matrix.py:158-168]
  • Comparison tooling explicitly blocks transitions from FRAMEWORK_INCOMPATIBLE to ordinary FAIL, reinforcing that reclassifying the targeted build rejection must not lose its compatibility category. [::ai-dynamo/aiconfigurator::tools/support_matrix/compare_support_matrix.py:341-360]
  • AIC’s migration policy treats development AISimulate releases as unsuitable stable baselines, relevant to the downstream 0.13.0.dev2 pin and the requirement for fresh qualification before release advancement. [::ai-dynamo/aiconfigurator::docs/aisimulate_migration.md:13-19]
🔇 Additional comments: 1
python/aisimulate/tests/unit/support_matrix/test_fpe_qualification.py: 1

100-100: LGTM!

Also applies to: 164-169

📝 Summary

Risk and human attention

Risk: Medium. Review these three areas:

  1. Diagnostic matching: The new exception applies only during build and only to a complete ValueError diagnostic. Check that the diagnostic matches the intended runtime rejection and does not match a different incompatibility.
  2. Qualification behavior: FRAMEWORK_INCOMPATIBLE remains disqualifying for required known-good probes. Confirm that only exploratory compatibility rows receive the new classification.
  3. Validation and release readiness: The supplied evidence is local diagnostic validation, not a fresh nightly qualification or GPU accuracy result.

Changed behavior and contract

classify_failure now returns FRAMEWORK_INCOMPATIBLE for a matching GLM-5.3-Flash SM9x FP8 sparse-MLA KV ValueError during build. It also recognizes the rejection when it appears in an exception chain. Query-stage errors, wrong exception types, and altered or incomplete diagnostics retain blocking classifications.

The classifier does not change KV precision or native admission. Tests assert that build results preserve probe phases, requested plan identity, FP8 KV mode, the original error type and message, and absent latency.

Qualification tests confirm that this status cannot satisfy a required known-good probe. They also allow exploratory rows with this status to remain visible.

Evidence and readiness

The supplied results report 292 passing tests, replay of 96 real compile plans producing 328 corrected phase rows, and in-memory checks against the existing shards and qualification gate. They also report unchanged PASS rows and on-disk reports, plus adversarial diagnostic and BF16 control checks. These are reported results; this inspection did not rerun them.

No current review findings were supplied, so review severity counts are unavailable. No fresh nightly qualification or downstream publication is evidenced. Merge readiness therefore still depends on a completed fresh nightly before downstream dependency pins advance.

The support matrix now classifies a matching GLM-5.3-Flash SM90 FP8 sparse-MLA KV build error as framework-incompatible. Tests cover build and query classification, H100 and H200 probe results, and qualification handling of that status.

Changes: FP8 KV Failure Classification

Layer / File(s) Summary
Classify SM90 FP8 KV build errors
python/aisimulate/tools/support_matrix/fpe_support_matrix.py, python/aisimulate/tests/unit/support_matrix/test_fpe_support_matrix.py
classify_failure classifies the matching GLM-5.3-Flash SM90 FP8 sparse-MLA KV build error as FRAMEWORK_INCOMPATIBLE. Tests check direct and wrapped errors, query-stage behavior, nonmatching errors, and H100/H200 probe results.
Test qualification status handling
python/aisimulate/tests/unit/support_matrix/test_fpe_qualification.py
Tests include FRAMEWORK_INCOMPATIBLE among required-case failures and exploratory unsupported-case statuses.

Priority: ⚪ Pending latest changes

Estimated code review effort: 2 (Simple) | ~12 minutes

Merge Risk: ⚪ Minimal · up to 246d5

No actionable issue is established for this change; it is mergeable after normal checks.

Pre-merge checks: Passed 7 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Description check Warning The description clearly states the behavior change, root cause, risk, validation results, compatibility constraints, and related work. It does not follow the required template structure and omits the … Add the missing template sections: Review map with riskiest files and contract impact; Evidence with explicit Fast CI, Full CI, CodeRabbit reviewed commit, and required Codex reviewed commit; Modeling or data provenance with N/A if not appl…
Passed Passed checks: 7 passed
Check name Status Explanation
Title check Passed The title accurately identifies the GLM SM90 KV rejection classification change. It is specific and directly related to the main behavioral change, although it does not name FRAMEWORK_INCOMPATIBLE or …
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Cross-Layer Contract Passed PASS. The PR changes only failure classification; it adds no input, schema field, default, or serialization field. The new FRAMEWORK_INCOMPATIBLE path is called only for build-stage matching errors …
Modeling And Data Evidence Passed The PR changes failure-selection logic, not formulas, model data, or performance outputs. The new rule uses an exact typed, build-stage diagnostic match (fpe_support_matrix.py:550-563). Regression tes…
Compatibility Boundaries Passed No compatibility boundary changed. The PR modifies only two support-matrix tests and the Python classifier. The classifier adds a build-stage mapping to the existing FRAMEWORK_INCOMPATIBLE status; it …
Review Evidence Passed The PR description names the pytest command and result, pre-commit/Ruff/format checks, and the real native replay. It includes boundary validation for wrapped errors, wrong types, altered or incomplet…
Full details: Title check

Explanation

The title accurately identifies the GLM SM90 KV rejection classification change. It is specific and directly related to the main behavioral change, although it does not name FRAMEWORK_INCOMPATIBLE or FP8 explicitly.

Full details: Cross-Layer Contract

Explanation

PASS. The PR changes only failure classification; it adds no input, schema field, default, or serialization field. The new FRAMEWORK_INCOMPATIBLE path is called only for build-stage matching errors (fpe_support_matrix.py:550-577, 707-752). Existing JSON/CSV serialization carries the unchanged status field (fpe_support_matrix.py:193-224, 897-905). The qualifier already accepts the status and still rejects it for required probes (qualify_fpe_support_matrix.py:33-42, 98-129). The web rollup and Pages publisher already preserve and render this status (build_fpe_support_matrix.py:139-145, prepare_fpe_pages.py:26-35, 130-144). Tests cover direct and wrapped errors, query-stage failures, altered diagnostics, native H100/H200 and vLLM/SGLang cases, all probe phases, identity, and qualification behavior (test_fpe_support_matrix.py:477-529; test_fpe_qualification.py:88-169). The Rust producer emits the exact diagnostic matched by the classifier (crates/core/src/perfmodel/engine/readiness.rs:101-116).

Full details: Modeling And Data Evidence

Explanation

The PR changes failure-selection logic, not formulas, model data, or performance outputs. The new rule uses an exact typed, build-stage diagnostic match (fpe_support_matrix.py:550-563). Regression tests cover wrapped errors, query-stage errors, wrong types, altered messages, altered SM, all four probe phases, H100/H200, and both backend/version pairs (test_fpe_support_matrix.py:477-529). The supplied validation provides provenance and reproducible before/after evidence: nightly run 39922445918, staged source SHA, replay of 96 real plans producing 328 rows, and a field-level comparison showing only status changed. It also reports anomaly checks for unchanged PASS rows and qualification sentinels.

Full details: Compatibility Boundaries

Explanation

No compatibility boundary changed. The PR modifies only two support-matrix tests and the Python classifier. The classifier adds a build-stage mapping to the existing FRAMEWORK_INCOMPATIBLE status; it does not change result fields, schema version, function signatures, or defaults. The status and schema version were already present in the base revision and remain accepted by the qualifier. Cargo.toml and python/aisimulate/pyproject.toml are unchanged, including version 0.13.0, maturin binding settings, and the package layout. The changed module adds no Dynamo dependency or binding change, so the one-wheel/one-crate contract and standalone boundary remain intact.

Full details: Review Evidence

Explanation

The PR description names the pytest command and result, pre-commit/Ruff/format checks, and the real native replay. It includes boundary validation for wrapped errors, wrong types, altered or incomplete messages, query-stage failures, adversarial diagnostics, and BF16 controls. It distinguishes local and synthetic-fixture checks from real-plan parity results and states that no new nightly qualification or publication occurred. The changed files contain no workflow changes, and the description does not cite inactive nested workflows as hosted CI evidence.

Full details: Description check

Explanation

The description clearly states the behavior change, root cause, risk, validation results, compatibility constraints, and related work. It does not follow the required template structure and omits the required Codex reviewed commit for a medium-risk change, plus explicit Review map, Modeling or data provenance, and Tracking fields.

Resolution

Add the missing template sections: Review map with riskiest files and contract impact; Evidence with explicit Fast CI, Full CI, CodeRabbit reviewed commit, and required Codex reviewed commit; Modeling or data provenance with N/A if not applicable; and Tracking with Closes and Related PRs. Preserve the existing validation details and state the exact review commit values.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
@tedzhouhk
tedzhouhk enabled auto-merge (squash) October 23, 2026 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants