feat(planner): cut live retune windows and reconcile the trace claim by tedzhouhk · Pull Request #17735 · ai-dynamo/dynamo

tedzhouhk · 2026-10-30T23:42:24Z

Note

Stacked PR, live retune (split from #17562). Merge order: #17483 → #17733 → #17734 → #17735 (this PR) → #17736 → #17562

This PR targets hzhou/live-retune-2-config, the branch of #17734 (the PR below it), so its diff shows only this layer. #17562 was split into these layers for review. Together they carry #17562's original diff exactly, plus a 40-line replay check in #17734 that rejects executor: dgdr in offline replay.

Overview

This is the third layer of live retune (#17562). It adds the two pieces that work on the shared ReadWriteMany trace claim:

The window cutter turns the frontends' request-trace segments into the one window file a DGDR Run replays.
The reconciler garbage-collects what live retune leaves on the claim and in the namespace.

Both are libraries that only the live retune supervisor in #17562 calls. Nothing in this layer wires them into a Planner, and a live Planner still rejects retune at startup.

Key Changes

About 2.3k lines of implementation and 2.3k lines of tests, all in new files.

planner/retune/live/window_cut.py: merges each frontend pod's request_end jsonl.gz segments into <windows_dir>/<dgd>/<window_end_ms>.jsonl.gz. Row selection, censoring, de-duplication and row_counts follow retune.window.read_trace_window, and the rows are the same bytes.
- The cut runs in a spawned subprocess bounded by cut_budget_seconds, which is killed at the deadline or on cancel.
- It validates each gzip member (CRC32 and ISIZE) and skips corrupt or truncated ones. A byte-ratio and member-ratio gate fails a mostly corrupt window (trace_corrupt). An in-flight flush at the end of a writer's newest segment is counted but not held against either ratio.
- Two passes keep about 45 bytes per selected row, so max_window_rows bounds memory (window_too_large).
- The output is atomic: a temp file is fsynced, set to mode 0600 and hard-linked into place. An existing window file is reused.
planner/retune/live/reconciler.py: garbage-collects finished Requests and window files once no live pod still reads them, based on pod phase.
- Requests are kept for the newest keep_requests windows. Non-terminal Requests past their window deadline, queued deletions and Requests of a recreated DGD are deleted.
- A window file goes once every Request that references it has a completed Run with no Job pod left.
- Cut temp files and old frontend segments are deleted too, but never a live pod's newest segment.
- The caller is asked before every delete, so a round stops as soon as leadership is lost.
- It is also the SweepRegistry the DGDR sweep uses. In-flight windows are protected by ref-counted entries, so two sweeps of one window cannot drop each other's protection.
Tests: unit/test_live_retune_window_cut.py (corrupt and truncated gzip members, zero-filled holes, ratio gates, golden output against read_trace_window, the child process and its interruption) and unit/test_live_retune_reconciler.py (segment, window, temp file and Request rules, the deletion queue, should_continue, DGD recreation, and reference counting during GC).

The only imports are contracts, planner_config, retune.window and offline.trace_data, so no module of this layer imports a later layer.

Validation

Ran on this layer (aacfacb02b) and compared with the same suites at the stack base 6e8087d359 (#17483's head):

Check	Result
`test_live_retune_window_cut.py` and `test_live_retune_reconciler.py`	157 passed
Planner and replay suites (`components/src/dynamo/planner/tests`, `components/src/dynamo/replay/tests`, no xdist)	59 failed, 2594 passed, 2 skipped. The failing tests are the same 59 as at the base (59 failed, 2131 passed).
`pre-commit run --files` on the 4 changed files	All hooks pass, no files changed
CI mypy (the `dynamo-pipeline.yml` command, `MYPYPATH=components/src:lib/bindings/python/src mypy --explicit-package-bases`) on `dynamo/planner` and `dynamo/replay`	The same 9 errors as at the base (missing `yaml`/`pandas`/`nvtx` stubs and existing `arg-type` errors); none is new, and none is in `retune/live`

The 59 failures come from this host. Its prebuilt bindings lack conversation_affinity (#16854) and the ais-forward-pass feature, and it has no etcd binary. CI builds the bindings from source.

This layer adds library code that nothing reachable calls until #17562. It needs no guard of its own. A live Planner still fails in NativePlannerBase.__init__, and replay still rejects executor: dgdr through #17734's replay check.

Issues and Ongoing Work

Part of live retune (feat(planner): enable live retune through DGDR v1beta2 #17562), the retune work in feat(planner): engine-owned retune with replay router control #17095 and the runtime reconfiguration DEP DEP (light): Planner runtime reconfiguration with predictor warmup #16835.
The next layer is feat(planner): run a live retune sweep through DGDR v1beta2 Requests #17736.

Pull Request Checklist

I used Dynamo's /pr-checklist skill.

🤖 Generated with Claude Code

Add the two trace-claim pieces of live retune: - retune/live/window_cut.py cuts one retune window from the frontends' gzip request-trace segments on the shared RWX claim into the window file a DGDR Run replays, with the selection, censoring and de-duplication of retune.window.read_trace_window, bounded memory (window_too_large), corrupt-member handling (trace_corrupt) and an atomic output, in a spawn child killed at the deadline or on cancel. - retune/live/reconciler.py garbage-collects what live retune leaves on the claim and in the namespace (old frontend segments, window and temp files, and DGDR Requests beyond retune.dgdr.keep_requests), and is the SweepRegistry a sweep uses to protect its window and Requests. Both are libraries that only the live retune supervisor calls; nothing in this layer wires them into a Planner, which still rejects retune at startup. Third layer of the live retune stack split from #17562. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: hongkuanz <hongkuanz@nvidia.com>

Stacks this PR on #17736. The library layers of live retune now come from #17733 (CRN replay), #17734 (configuration and premises), #17735 (window cut and reconciler) and #17736 (DGDR executor), so this PR's diff against #17736 is the supervisor, state, actuator, metrics, the Planner wiring, Helm RBAC, docs and the end-to-end tests. #17734 also adds a replay check that rejects executor 'dgdr'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: hongkuanz <hongkuanz@nvidia.com>

devin-ai-integration · 2026-10-31T00:57:33Z

✅ Dynamo PR CI passed — run 41146051842 (attempt 1) on `73c0e5889c`

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

_{Posted automatically by Devin for run 41146051842. Updated on every full-CI run of this PR.}

jthomson04

Reviewed at 73c0e58. Two should-fix items inline (re-cut ordering and the scope of pass 1), plus nits. Nothing here blocks the event loop, and the atomic publish, partial-member handling and retention all check out.

jthomson04 · 2026-10-31T17:39:30Z

+
+        if selection.num_rows == 0:
+            return result(None)
+        if os.path.lexists(out_path):


Check out_path before pass 1. Today re-cutting an already published window scans every segment first, so a later leader adopting the window can fail it as trace_corrupt/window_too_large, or return counts that #17736 (dgdr.py:1399) writes into new Requests while they replay the unchanged file. Take the counts from the existing file or the adopted Requests' rows annotation.

jthomson04 · 2026-10-31T17:39:30Z

+# ---------------------------------------------------------------- pass 1
+
+
+def _segment_files(root: str) -> list[tuple[str, bool]]:


Pass 1 decodes every retained segment, about 2T + 11 min with the defaults, so roughly half the parse is outside the window and counts against cut_budget_seconds. Corruption or a malformed row there can also fail windows it holds no rows for. Skip segments whose mtime is before start_ms minus a skew margin.

jthomson04 · 2026-10-31T17:39:30Z

+                elif view.dgd_uid != dgd_uid:
+                    # The DGD was deleted and recreated under the same name.
+                    reasons.setdefault(view.name, "other_dgd_uid")
+            states = {view.name: self._is_terminal(state, view) for view in ours}


should_continue is only asked before a delete; this comprehension still does a get_run (10 s timeout) per uncached Request after the budget has expired or leadership was lost, and the supervisor won't start the next round until it returns. Check it before these reads too.

jthomson04 · 2026-10-31T17:39:30Z

+
+
+@dataclass(frozen=True)
+class _PlainBlock:


Production never feeds this a plain JSONL file: the guide requires jsonl_gz, and the reconciler only collects .NNNNNN.jsonl.gz segments, so a plain .jsonl sink would be re-read in full on every cut and never collected. Drop _PlainBlock/_plain_blocks, or make the reconciler handle it.

jthomson04 · 2026-10-31T17:39:30Z

+        self._sink_reported = current
+        if missing:
+            logger.warning(
+                "retune reconciler: Ready frontends without a trace directory "


This warns every 15 s round while the condition holds, and the gauge already tracks it. Log on state change. Same for the missing-label warning at :682.

jthomson04 · 2026-10-31T17:39:30Z

+        pass
+    # The writer thread takes the GIL back after every zlib block; at the
+    # default 5 ms interval it would wait behind parsing (pass 2 ~30% slower).
+    sys.setswitchinterval(_CHILD_SWITCH_INTERVAL_S)


Is the ~30% pass-2 gain worth a writer thread, setswitchinterval and os.nice in the child? A synchronous GzipFile.write drops about 70 lines and a thread error path.

tedzhouhk requested a review from a team as a code owner October 30, 2026 23:42

pull-request-size Bot added the size/XXL label Oct 30, 2026

github-actions Bot added the feat label Oct 30, 2026

This was referenced Oct 30, 2026

feat(replay): replay DGDR trace Runs with common random numbers #17733

Open

feat(planner): add the live retune configuration and deployment premises #17734

Open

github-actions Bot added the planner label Oct 30, 2026

tedzhouhk mentioned this pull request Oct 30, 2026

feat(planner): run a live retune sweep through DGDR v1beta2 Requests #17736

Open

1 task

tedzhouhk mentioned this pull request Oct 30, 2026

feat(planner): enable live retune through DGDR v1beta2 #17562

Open

1 task

jthomson04 reviewed Oct 31, 2026

View reviewed changes

Navigation Menu

Repository navigation

feat(planner): cut live retune windows and reconcile the trace claim - #17735

feat(planner): cut live retune windows and reconcile the trace claim#17735
tedzhouhk wants to merge 1 commit into
hzhou/live-retune-2-configfrom
hzhou/live-retune-3-trace

tedzhouhk commented Oct 30, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot commented Oct 31, 2026

Uh oh!

jthomson04 left a comment

Uh oh!

jthomson04 Oct 31, 2026

Uh oh!

jthomson04 Oct 31, 2026

Uh oh!

jthomson04 Oct 31, 2026

Uh oh!

jthomson04 Oct 31, 2026

Uh oh!

jthomson04 Oct 31, 2026

Uh oh!

jthomson04 Oct 31, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		# ---------------------------------------------------------------- pass 1


		def _segment_files(root: str) -> list[tuple[str, bool]]:



		@dataclass(frozen=True)
		class _PlainBlock:

Conversation

tedzhouhk commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Key Changes

Validation

Issues and Ongoing Work

Uh oh!

devin-ai-integration Bot commented Oct 31, 2026

✅ Dynamo PR CI passed — run 41146051842 (attempt 1) on 73c0e5889c

Uh oh!

jthomson04 left a comment

Choose a reason for hiding this comment

Uh oh!

jthomson04 Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

jthomson04 Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

jthomson04 Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

jthomson04 Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

jthomson04 Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

jthomson04 Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tedzhouhk commented Oct 30, 2026 •

edited

Loading

✅ Dynamo PR CI passed — run 41146051842 (attempt 1) on `73c0e5889c`