Repository navigation
fix(mocker): adopt AISimulate nightly 82 - #16945
Conversation
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
✅ Dynamo PR CI passed — run 40380859564 (attempt 2) on
|
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
dreamtalen
left a comment
There was a problem hiding this comment.
LGTM, approving once CI is all green. I'll add KVCR router hint support in a follow-up PR.
dmitry-tokarev-nv
left a comment
There was a problem hiding this comment.
No P0 or P1 found. One P2, inline. This review read head d38d790c89.
- [P2] A merge with current
main(dea508e1c1, #17095) does not compile thedynamo-mockertests.composition.rs:635needsrouter_hint: false. Inline oncomposition.rs:576. The newestmain,8222ff098c, has the same literal and merges with this head without a conflict.
Open items:
- In the
PRrun 40380859564,planner / Build multi-arch cpuwas cancelled at the 60-minute job limit, andbackend-status-checkfailed because of that. The image build with the nightly 82 pins passed for both architectures. TheCompliance extractstep then printed nothing for 21 minutes after the amd64 export. The same job was also cancelled on run 40382487453 of #17229. This review does not relate it to this change. - The description says that the branch merged
mainat2d17836993b0. The head also merged4b487d890dlater. Both merge commits match a freshgit merge-treeand add nothing to the diff of this PR.
What this review measured, on the GPU box and on macOS, with base `4b487d890d` as the control.
Pins and locks:
- All five pins name
0.13.0.dev202610240000000082or0.13.0-dev.202610240000000082. No other file names nightly 73. - Each of the three
Cargo.lockfiles changes 4 lines, foraisimulate-coreonly. cargo tree --lockedpasses in.,lib/bindings/pythonandlib/bindings/kvbm, at base and at head. The only dependency change isaisimulate-core73 to 82.
Published artifacts:
- The crate sha256 equals the lock checksum
fa3e351f.... The x86_64 wheel sha256 equals the PyPI digest7f985fff.... - The crate names
f3a3e3156530574f5e7a10446f19e571b82db871in.cargo_vcs_info.json. 266 of its 270 files are byte-equal tocrates/coreat that commit. The other 4 are the packaging files and the version stamp. - Every Python and data file in the wheel is byte-equal to
python/aisimulateat that commit. Only the compiled.sois not in git. - AISimulate #463, #465, #476, #477, #492 and #574 are in the history of that commit. The nightly run 40076769417 built that commit.
Rust (cargo test --locked -p dynamo-mocker --all-targets):
| tree | result |
|---|---|
| base | 272 passed |
| head | 277 passed |
A deliberate break of each new check fails exactly one new test:
| break | failing test |
|---|---|
remove the KV-hint check in native_request |
unhandled_request_fields_do_not_mutate_compatibility_state |
remove the image check in native_request |
unhandled_request_fields_do_not_mutate_compatibility_state |
remove the sglang.host_loop check |
host_loop_is_rejected_before_live_tasks_start |
| change the TTFT milestone message | host_aware_ttft_milestones_are_rejected_explicitly |
| remove the KV-hint check in the offline KV router | unhandled_request_fields_are_rejected_instead_of_dropped |
| remove the image check in the offline KV router | unhandled_request_fields_are_rejected_instead_of_dropped |
| remove the KV-hint check in online replay | online_replay_rejects_unhandled_request_fields_before_placement |
| remove the image check in online replay | online_replay_rejects_unhandled_request_fields_before_placement |
Where each new error goes: online replay returns it from the whole replay (joined?? in live_runtime.rs). The offline KV router returns it from on_request_arrival. The grouped live engine sends it on the command reply. Outside the tests, the DirectRequest and synthetic trace builders in this tree set no images and no KV hints. So these checks do not stop a text-only request.
Python, with a _core built from each tree (maturin build --features ais-forward-pass --profile dev):
| suite | base, nightly 73 | head, nightly 82 |
|---|---|---|
AISimulate tests in tests/dependencies |
9 passed | 9 passed |
| the 4 binding modules that the description names | 58 passed | 58 passed |
the 30 test files that use aisimulate, with -n 8 |
594 passed, 7 failed, 6 skipped | 595 passed, 6 failed, 6 skipped |
Each head failure also fails at base. Three router end-to-end tests fail on both trees, also in a run of their own. In each, a server process of the test exits with code 1 before it opens its port. The other failures are smoke tests in profiler/tests/test_replay_optimize.py under -n 8. In a run of their own, the three that failed at head pass at head and at base.
tests/dependencies/test_aisimulate_consistency.py: 5 passed at head. A changed pin in lib/bindings/python/Cargo.toml or in container/deps/requirements.aisimulate.txt makes it fail. At base, the same change in lib/bindings/python/Cargo.toml still passes (5 passed), so the new check in this PR catches it.
dmitry-tokarev-nv
left a comment
There was a problem hiding this comment.
Reviewed head d38d790c89. No P0 or P1 found.
All five pins and the three Cargo.lock files move to AISimulate nightly 82, and cargo tree --locked passes in all three workspaces. The published crate and wheel match source commit f3a3e315. A removal of each new request check fails a test, and dynamo-mocker passes 277 tests. The Python suites that use aisimulate show no new failures against base.
Open item: one P2. A merge with current main (dea508e1c1, #17095) needs router_hint: false in single_worker_topology() in composition.rs, or the dynamo-mocker tests do not compile. Please merge main with that line before this lands.
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
Summary
100cce2e48c4, including the earlier2d17836993b0and4b487d890dmerges.images: None, and adapt thesingle_worker_topology()fixture added by main feat(planner): engine-owned retune with replay router control #17095 withrouter_hint: false.Published artifacts
aisimulate0.13.0.dev202610240000000082aisimulate-core0.13.0-dev.202610240000000082f3a3e3156530574f5e7a10446f19e571b82db871Published x86_64 wheel and crate bytes were downloaded and their SHA-256 digests verified against nightly run 82 provenance. This source includes AISimulate #463, #465, #476, #477, #492, and #574. Validation uses the published artifacts, not an unpublished source override.
Validation
Review fix and latest main synchronization
100cce2e48c4without conflicts and addedrouter_hint: falseto the newsingle_worker_topology()fixture. Independent review confirmed the merge differs from the clean merge tree only by that compatibility fix.cargo test --locked -p dynamo-mocker --libon the updated merge tree: 283 passed, including the new replay-router control tests from main.cargo tree --lockedsucceeds in the root, Python-bindings, and KVBM workspaces, all using nightly 82. Dependency-consistency tests: 5 passed. Changed-file pre-commit,cargo fmt --all --check, andgit diff --checkpassed.d38d790c89. New-head CI must run again; those earlier results do not qualify the newer main merge.Initial nightly 82 integration qualification
cargo test --locked -p dynamo-mocker --lib --no-default-featureson Rust 1.96.1: 277 passed, including AgentX replay, grouped scheduling, and unsupported-field/host-loop regression tests.maturin build --manifest-path lib/bindings/python/Cargo.toml --features ais-forward-pass --profile dev, then installed that runtime wheel with the editable Dynamo Python package and the actual published AISimulate 82 wheel.uv pip checkpassed.test_ais_capacity.py(34),test_replay_ais_integration.py(9),test_replay_ais_capacity.py(7), andtest_replay_agentic_timing.py(8). These exercise real AIS callbacks, aggregated/disaggregated replay, capacity, and AgentX timing; two live tests use the actual in-memory runtime.tests/dependencies/test_aisimulate_consistency.py.best_availablefactory. An additional real AIS session/native static-oracle matrix passed 16 cases (nextn 0/2, attention DP 1/2, prefill/decode); vLLM and SGLang capacity probes passed.git diff --checkpassed. Independent review completed with its host-loop early-rejection finding fixed and retested.Python suites were run offline (
HF_HUB_OFFLINE=1,TRANSFORMERS_OFFLINE=1), one module per process, with--noconftestand the required pytest plugins explicitly enabled. The selected network-free suites do not require the parent conftest's blanket NATS/etcd autouse fixture. Separate module processes avoid apytest-forkedparent fixture-state teardown error seen in a combined invocation; no test assertions or repository warning filters were relaxed.Validation boundaries
d38d790c89head; qualification of the latest main synchronization is limited to the explicitly listed local checks until its new-head CI completes.