[tedzhouhk]

Overview

This is a follow-up to #17474, in response to its review. With ai-dynamo/aisimulate#626, AISimulate's generator bridge lowers a trace workload without request lengths and rejects placeholder ones. The AIC DGD renderer therefore goes back to passing the workload, and the manifest AIC returns, through unchanged.

Note

This PR now pins AISimulate nightly 87 (0.13.0.dev202610290000000087, Rust 0.13.0-dev.202610290000000087). Nightly 87 is built from AISimulate 03eadda3 and contains ai-dynamo/aisimulate#626.

The revert and the pin bump have to land together, in this PR:

• main pins nightly 86 (feat(planner): load local AISimulate execution profiles #17228), which was built before fix: lower trace workloads in the Sweeper generator bridge aisimulate#626. The revert on that pin breaks trace rendering again.
• Nightly 87 and later reject the placeholder lengths that main sends. Any other PR that moves the pin past 86 fails test_installed_aic_renders_trace_workloads_like_synthetic_ones[trtllm|vllm|sglang] on main's renderer, and needs this change.

Key Changes

• components/src/dynamo/aisimulate/output/dgd/renderers/aic/renderer.py: restore the renderer from before fix(aisimulate): render trace workloads through the AIC generator #17474, which passes the workload straight through. This drops the placeholder lengths and the scrub of --trtllm.cache_transceiver_config.max_tokens_in_buffer, and keeps the check that AIC returned k8s_deploy.yaml.
• components/src/dynamo/aisimulate/tests/dgd/test_renderers.py:
  • The placeholder and scrub unit tests are removed.
  • The real-bridge test_installed_aic_renders_trace_workloads_like_synthetic_ones now expects the trace DGD to equal the synthetic one without the TensorRT-LLM transfer-buffer size. The test computes that expected DGD itself; the renderer no longer edits anything.
• AISimulate pin, 86 → 87:
  • pyproject.toml
  • benchmarks/pyproject.toml
  • container/deps/requirements.aisimulate.txt
  • Cargo.toml
  • lib/bindings/python/Cargo.toml
  • Cargo.lock, lib/bindings/python/Cargo.lock, lib/bindings/kvbm/Cargo.lock: only the aisimulate-core version and checksum change, through cargo update -p aisimulate-core --precise. The checksum matches the crates.io index.
  • The pinned versions quoted in docs/fern/pages/reference/components/planner-configuration.mdx.
• docs/fern/pages/cli/operations/simulation-with-dynosim/simulation-model.md: nightly 87 also contains feat: share P/D KV transfer links and offload bandwidth max-min fairly aisimulate#616. Offline P/D transfers now share each prefill rank's sending link and each decode rank's receiving link max-min fairly. The page said concurrent requests do not contend for a shared link; it now describes link sharing in offline prediction and notes that live Mocker workers still wait a fixed delay per transfer.
• The branch merges main at ebad592bf0 to pick up the nightly 86 pin before bumping it.

Validation

These runs used the published nightly 87. A fresh virtual environment held the PyPI wheel and Dynamo bindings rebuilt against the crates.io crate (maturin develop --release --features kv-indexer,slot-tracker,select-service,mm-routing,ais-forward-pass).

• Python: 3094 passed, 21 skipped and 2 failed, run with -n 16 over:
  • components/src/dynamo/{aisimulate,planner,replay,profiler,router,mocker}/tests
  • lib/bindings/python/tests/test_ais_capacity.py
  • lib/bindings/python/tests/replay
  • tests/dependencies
  • The two failures are test_profiler_protocol.py::test_materialize_dgd_inaccessible_model_path[True|False]. They assert on caplog and do not call AISimulate. Both pass alone, and the file passes 103 of 103 when run by itself.
  • The run includes the three real-bridge trace tests and test_aisimulate_consistency.py.
  • It ran before the review fixes in 88bca4d. After them, components/src/dynamo/aisimulate/tests passes 102 of 102.
• Rust:
  • cargo test --locked -p dynamo-mocker --all-targets: 294 passed.
  • The canonical replay bench from pre-merge.yml (offline_replay_bench on mooncake_trace_tiny.jsonl) completed.
  • cargo clippy --locked --no-deps -p dynamo-mocker --all-targets -- -D warnings passed.
  • In lib/bindings/python, cargo test --locked --features ais-forward-pass: 85 passed.
  • cargo test --locked -p dynamo-llm --lib mocker: 48 passed, including live_and_offline_handoff_surfaces_share_one_conformance_report and no_bootstrap_prefill_delays_terminal_finish_once.
• Nightly 86 → 87 review: I compared the two published wheels, the two crates, and the AISimulate source between 17625dc8 and 03eadda3 against Dynamo's callers.
  • No API or dependency change. No public Python or Rust API, PyO3 surface, feature flag, entry point, or dependency changed. from_sweeper_candidate has no Dynamo caller other than this renderer, and the profiler's generator paths always pass request lengths. perf: allocation-free grid-table queries aisimulate#627, fix: fix max_local_prefill_length not being printed out in disagg router log #628, fix: add DYNAMO_HOME env var to vLLM docker image #629 and build: Define UCX env var to use NVLink when available #631 are internal and bit-for-bit equivalent. fix: the SGLang MLA module collector recognises the ModelOpt mixed recipe aisimulate#614 changes only the collector, which is not shipped in the wheel.
  • Behavior change: feat: share P/D KV transfer links and offload bandwidth max-min fairly aisimulate#616. It changes offline replay timing only when kv_transfer_bandwidth is set and transfers overlap on one rank, or when a ReplaySpec gives decode a lower bandwidth than prefill. Shared G2/G3 offload (cluster_shared) now redistributes unused pool bandwidth max-min fairly. The default (no bandwidth) is unchanged, and no Dynamo test asserts the affected timings. Results from planner-gym matches or offline_replay_bench runs that set a transfer bandwidth can shift.
• Earlier parity check: the trace DGDs from this renderer equal those from main's placeholder-and-scrub renderer in 32 of 32 cases: TensorRT-LLM rc20/rc23, vLLM 0.20.1 and SGLang 0.5.9, each × agg/disagg × block size 32/64 × context_length None/10000. That check used nightly 82 with the fix: lower trace workloads in the Sweeper generator bridge aisimulate#626 generator overlaid; nightly 87's generator/ matches that source.
• pre-commit and docs_lint.py: passed on the changed files.
• Not verified:
  • A GPU deployment.
  • Full CI, which has not been triggered.

Issues and Ongoing Work

• There is no tracked issue; this is a follow-up to fix(aisimulate): render trace workloads through the AIC generator #17474.
• This PR covers the first review thread on fix(aisimulate): render trace workloads through the AIC generator #17474. It does not change behavior for TensorRT-LLM 1.3.0rc3 to rc22 runtimes (second thread), or which CI lanes collect these tests (third thread).
• With feat: share P/D KV transfer links and offload bandwidth max-min fairly aisimulate#616, the offline replay shares transfer links but the live mocker still sleeps a fixed delay per transfer. The two can differ under concurrent handoffs. This PR does not change the live mocker.

Pull Request Checklist

• I used Dynamo's /pr-checklist skill.

🤖 Generated with Claude Code

[devin-ai-integration]

✅ Dynamo PR CI passed — run 40917773565 (attempt 1) on 88bca4d08f

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

_{Posted automatically by Devin for run 40917773565. Updated on every full-CI run of this PR.}

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a188a7-81cc-7479-85bf-f82bb750e45b.docs.buildwithfern.com/dynamo/dev

[tedzhouhk]

/ok to test 99429c6

[coderabbitai]

No actionable comments were generated in the recent review.

Recent review info

Run configuration

• Source mode: Auto
• Repository: ai-dynamo/dynamo
• Branch: hzhou/aic-trace-passthrough
• File: .coderabbit.yaml
• Commit: 99429c6
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: e586be67-fd0d-4fb3-8b0f-8a1882da597b

Commits

Reviewing files that changed from the base of the PR and between ebad592 and 99429c6.

Files ignored due to path filters: 3

• Cargo.lock is excluded by !**/*.lock
• lib/bindings/kvbm/Cargo.lock is excluded by !**/*.lock
• lib/bindings/python/Cargo.lock is excluded by !**/*.lock

Files selected for processing: 9

• Cargo.toml
• benchmarks/pyproject.toml
• components/src/dynamo/aisimulate/output/dgd/renderers/aic/renderer.py
• components/src/dynamo/aisimulate/tests/dgd/test_renderers.py
• container/deps/requirements.aisimulate.txt
• docs/fern/pages/cli/operations/simulation-with-dynosim/simulation-model.md
• docs/fern/pages/reference/components/planner-configuration.mdx
• lib/bindings/python/Cargo.toml
• pyproject.toml

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

The AIC renderer now passes trace workloads through without placeholder lengths or manifest buffer-argument filtering. AISimulate pins and planner documentation use the October 29 build. Simulation documentation now describes shared-link contention with max-min fairness.

Changes

AIC trace-workload rendering

Layer / File(s)	Summary
Trace workload rendering and validation components/src/dynamo/aisimulate/output/dgd/renderers/aic/renderer.py, components/src/dynamo/aisimulate/tests/dgd/test_renderers.py	The renderer passes the original workload to the generator and no longer removes the transfer-buffer-size argument from generated manifests. Tests check workload pass-through and compare manifests with a test-local argument filter.

AISimulate version pins

Layer / File(s)	Summary
Dependency and documented version updates Cargo.toml, lib/bindings/python/Cargo.toml, pyproject.toml, benchmarks/pyproject.toml, container/deps/requirements.aisimulate.txt, docs/fern/pages/reference/components/planner-configuration.mdx	AISimulate dependency pins and planner documentation now reference the October 29 build. Existing Python version constraints and optional feature settings remain unchanged.

Offline transfer contention documentation

Layer / File(s)	Summary
Shared-link transfer model docs/fern/pages/cli/operations/simulation-with-dynosim/simulation-model.md	The documentation describes line-rate transfer pricing and max-min sharing when concurrent transfers use the same prefill sending link or decode receiving link.

Priority: Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: Minimal · up to 99429

Trace workloads now pass directly to the generator; no verified functional issue blocks merging.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title is concise, specific, and accurately describes the primary change: passing trace workloads directly to the AIC generator.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description check		The description follows the repository template closely. It has Overview, Key Changes, Validation, and Issues and Ongoing Work sections, plus the checklist. It explains the revert and the pin bump fro…

Full details: Description check

Explanation

The description follows the repository template closely. It has Overview, Key Changes, Validation, and Issues and Ongoing Work sections, plus the checklist. It explains the revert and the pin bump from nightly 86 to 87, names the affected files, and gives detailed validation results. It also lists known gaps: GPU deployment and full CI are unverified. The 'Closes #XXXX' line is absent, but the template marks that item as advisory today, and the description states there is no tracked issue. The pull request checklist item is unchecked, and the template's required item is '/pr-checklist' skill use. Those are minor gaps and do not affect the overall completeness.

Full details: Docstring Coverage

Explanation

Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 4 functions across 2 files. (7 skipped: 7 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[jthomson04]

LGTM. The renderer matches the pre-#17474 version exactly, and every AISimulate pin is consistent at nightly 87. A few optional nits inline.

The description still says "It stays a draft for now", but the PR is marked ready for review.

[tedzhouhk]

/ok to test 88bca4d

[dmitry-tokarev-nv]

No P0, P1, P2 or P3 findings at 88bca4d08f5b6a3b4186783b9a40c6c6458ce073.

This review read all 12 changed files at that head, and the commit 88bca4d08f in full. The changes that the three review threads of 2026-10-29 asked for are in this head.

Open items: none.

What this review measured, with controls.

1. Pins. All 9 pin locations name nightly 87, and no file names nightly 86. The same search finds the 9 nightly 86 pins on main. The sha256 of the aisimulate-core 0.13.0-dev.202610290000000087 crate is eb823782e806dd74cba719906a6b828b86f415aad54c6d5016c29ff278e89319. It matches the crates.io index and all three Cargo.lock files. In each lock file, only the aisimulate-core version and checksum change. cargo tree --locked passes on the three manifests at this head, at main, and at a merge of this head with main at 261a106a3d. As a control, a pin set back to nightly 86 fails that check.

2. Nightly 86 against nightly 87. The two wheels have the same Requires-Dist lines, entry points and file list. The 26 changed files are all under aisimulate/: the generator and _runtime.abi3.so. The two crates have the same dependencies and features, and every changed Rust item is pub(crate). So the statement "No API or dependency change" in the description holds.

3. Trace rendering, components/src/dynamo/aisimulate/tests/dgd:

   Tree	Nightly	Result
   this head	87	66 passed
   this head	86	3 failed with SweeperCandidateError: the matching workload must provide isl and osl (or set SlaConfig.isl/osl overrides)
   main	86	69 passed
   main	87	3 failed with SweeperCandidateError: a trace workload has no fixed request length; remove workload.isl, workload.osl

   The 3 failures are the three cases of test_installed_aic_renders_trace_workloads_like_synthetic_ones. So each half fails alone, as the description says.

4. Test strength. Two deliberate breaks of the renderer at this head ran with nightly 87. The renderer from main fails the same 3 cases. An edit of the manifest for trace workloads only fails the TensorRT-LLM case. So the real-bridge test still covers what the removed fake-bridge test covered.

5. TensorRT-LLM transfer buffer. The disaggregated TensorRT-LLM trace DGD is byte-identical at main with nightly 86 and at this head with nightly 87. Both keep --trtllm.cache_transceiver_config.backend DEFAULT on prefill and decode. Neither has --trtllm.cache_transceiver_config.max_tokens_in_buffer. The synthetic DGD sets that size to 6528. The aggregated and disaggregated trace DGDs for TensorRT-LLM, vLLM and SGLang are also identical between the two setups.

6. simulation-model.md. A probe on the two crates replayed two concurrent 128 ms transfers at 1 GB/s. With nightly 86, each transfer took 128 ms in every layout. With nightly 87, two transfers into one decode rank took 256 ms each. Two transfers from one prefill rank also took 256 ms each. Transfers on separate ranks took 128 ms. The live Mocker sleeps delay_ms for each StartTransfer in lib/llm/src/mocker/handoff.rs. The new text matches both.

7. Rust on macOS. cargo test --locked -p dynamo-mocker --all-targets passes 290 of 290 at this head with nightly 87, and at main with nightly 86.

8. CI at this head. dynamo-runtime / test / parallel cuda13.0, amd64 ran the three trace cases and passed them. test_installed_aisimulate_matches_the_declared_release also passed there, so that image has nightly 87. rust-tests (.) ran on a merge of this head with 9f88a47f25 and compiled nightly 87. It passed 3946 dynamo-llm and 294 dynamo-mocker unit tests.

[dmitry-tokarev-nv]

Approving at 88bca4d08f5b6a3b4186783b9a40c6c6458ce073.

The revert of the AIC renderer and the nightly 87 pin must land together, and together they work. At this head with nightly 87, the AISimulate DGD tests pass. Each half fails alone: this head with nightly 86 and main with nightly 87 both fail the three trace cases. The TensorRT-LLM trace DGD is byte-identical to the one from main with nightly 86, and it has no transfer buffer size. All 9 pins name nightly 87, and the crate checksum matches all three Cargo.lock files. The new link-sharing text in simulation-model.md matches the nightly 87 replay.

In CI at this head, the three trace cases pass, and rust-tests (.) passes the dynamo-llm and dynamo-mocker tests with nightly 87.

Open items: none.