[atchernych]

Summary

An alternative to #17276. Both make MCS pools routable by giving each pool a model and live frontend addresses. #17276 reads the model from a hand-written --routing-config file. This PR gets it from the PoolRelay itself, through POST /pools, as the design says.

How it works

• In MCS mode, POST /pools now works (it returned 501). Each call is the pool's heartbeat, as in memory mode.
• A pool is listed only when both sides agree: MCS shows a Ready PoolRelay, and the pool registered within the lease (15 s by default).
• MCS gives the addresses: the Relay's, and the frontends' from the <dgd>-frontend Service, the same way as feat(global-router): resolve MCS pool models and live frontends #17276. The registration gives the model, capabilities, capacity and location.
• Discovery instances share registrations. Each one asks the others for theirs (GET /registrations) twice per heartbeat. It finds them through MCS, the way PoolRelays find the routers (--peer-service). So every region lists every pool, and a copy lapses with its original. A new instance is ready only after it has asked the others once, so it never serves a partial list.

Problems with the file in #17276

1. The model is typed twice: in the pool's deployment and in the file. If they differ, requests go to a pool that doesn't serve that model.
2. The same file goes to every cluster. Adding a pool or changing a model means editing it, applying it everywhere, and restarting every discovery service.
3. MCS pools get no capacity or capabilities. So every MCS pool in feat(global-router): resolve MCS pool models and live frontends #17276 shows "capabilities": [] and "capacity": []. In feat(global-router): take MCS pool models from registrations, not a file #17282, both come from the relay's registration. Today's round-robin router doesn't use either field yet. They matter for later work, such as load-aware routing (how big is each pool) or sending requests only to pools that can handle them.
4. In MCS mode, POST /pools still returns 501, but the design has PoolRelays register in every mode. A PoolRelay pointed at an MCS-mode service would get an error on every heartbeat.
5. A pool is live only by MCS readiness. If a cluster loses contact with the MCS implementation, its pools can stay Ready after they are gone (see the comment above ready_addresses in hub.rs).

What this PR adds

• No routing file. New pools and model changes need no config change or restart.
• Capacity and capabilities in MCS mode.
• One PoolRelay contract for every store.
• Missed heartbeats drop a pool even when MCS still shows it Ready.
• More than one discovery replica per cluster, since replicas share registrations too.
• A warning when MCS and the registrations disagree for two lease durations, for example when site_id isn't the pool's MCS cluster name. Today such a pool is silently missing.

Trade-offs

• A pool appears only once its PoolRelay calls POST /pools, and no PoolRelay does yet. Until then, register a pool with curl, as feat(global-router): route round robin over pools from discovery #17273 did.
• Discovery instances talk to each other: one small request per instance, twice per heartbeat.
• With --peer-service, the lease must be more than twice the heartbeat interval, so copies stay alive between refreshes. The defaults (15 s and 5 s) are fine.
• In MCS mode, GET /pools now says complete: false, because registrations are learned, not read from a shared store. The Global Router only logs this.
• If every discovery instance in a cluster is down, or the other clusters can't reach them, that cluster's pools drop out of every region within a lease (15 s by default), even when their PoolRelays and frontends are healthy. feat(global-router): resolve MCS pool models and live frontends #17276 doesn't have this, because MCS alone keeps those pools listed. Running two or more instances per cluster makes an outage unlikely. A registration store that every region reads, such as the planned one on DynamoDB global tables, would remove the need to reach the other clusters' instances.

To deploy, export the discovery service's own Service through MCS, like the router's, and pass --peer-service <name>. Each pool also exports <dgd>-frontend, as in #17276.

Validation

• cargo test -p dynamo-global-discovery-service: 50 passed, 7 of them new. They cover:
  • the join: only pools both sides list, which side each field comes from, watch events from both sides, a lapsed registration dropping a pool that MCS still lists, and one-sided pools (one warning each, forgotten after a relist);
  • frontends from MCS: frontend-only pools stay hidden, IPv6, and the named port;
  • copies: the latest registration wins, only local ones are shared, a copy never outlasts the lease, and one with no time left is ignored;
  • sharing between two instances over real HTTP: a registration and its changes reach the other instance, the copy lapses with the original, an unreachable instance is skipped, and an instance is ready only after its first round. An instance also skips its own reply and does not count time spent waiting for a slower one.
• Breaking each of 11 guarded behaviors on purpose (for example the self-skip, the lease cap, and the readiness gate) makes a test fail.
• cargo test -p dynamo-global-ingress -p dynamo-global-router -p dynamo-global-discovery-client: all pass, including the round-robin test against the real service.
• cargo clippy --all-targets -- -D warnings and cargo fmt on the crate: clean.
• Not yet run on a cluster.

🤖 Generated with Claude Code

[devin-ai-integration]

❌ Dynamo PR CI failed — run 40246321389 (attempt 1) on c1a4c16956

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ❌ dynamo-status-check

Other	Jobs
dynamo-runtime	❌ 3 ✅ 5

Failure details

3 jobs failed: on both amd64 and arm64, test_examples_default_to_v1beta1 fails because it parses the Helm template examples/global-router/cdk/helm/dynamo-mocker/templates/dgd.yaml as plain YAML. rust-gpu was killed with exit code 137 (SIGKILL, likely OOM or a lost runner) while compiling, so it looks like an infrastructure failure.

❌ dynamo-runtime / test / sequential cuda13.0, amd64: yaml ScannerError in test_examples_default_to_v1beta1

Job: dynamo-runtime / test / sequential cuda13.0, amd64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 121332266048 -R ai-dynamo/dynamo --log-failed

manifest_path = PosixPath('/workspace/examples/global-router/cdk/helm/dynamo-mocker/templates/dgd.yaml')
E   yaml.scanner.ScannerError: while scanning a simple key
E     in "<unicode string>", line 28, column 5:
E           {{- if .Values.runtimeVersionOve ...
E           ^
E   could not find expected ':'
E     in "<unicode string>", line 29, column 5:
E           runtimeVersionOverride: {{ .Valu ...
FAILED deploy/utils/tests/test_convert_api_version.py::test_examples_default_to_v1beta1 - yaml.scanner.ScannerError: while scanning a simple key
== 1 failed, 5519 passed, 153 skipped, 2593 deselected in 1012.63s (0:16:52) ===

deploy/utils/tests/test_convert_api_version.py::test_examples_default_to_v1beta1 scans example manifests under examples/. One of them is the Helm template examples/global-router/cdk/helm/dynamo-mocker/templates/dgd.yaml, and its {{- if .Values.runtimeVersionOverride }} directive (line 28) is not valid YAML, so the parse fails. The test failed on all 3 auto-retries too.

❌ dynamo-runtime / test / sequential cuda13.0, arm64: same yaml ScannerError in test_examples_default_to_v1beta1

Job: dynamo-runtime / test / sequential cuda13.0, arm64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 121332266106 -R ai-dynamo/dynamo --log-failed

FAILED deploy/utils/tests/test_convert_api_version.py::test_examples_default_to_v1beta1 - yaml.scanner.ScannerError: while scanning a simple key
  in "<unicode string>", line 28, column 5:
        {{- if .Values.runtimeVersionOve ...
        ^
== 1 failed, 5519 passed, 153 skipped, 2593 deselected in 1202.87s (0:20:02) ===

Same failure as on amd64: the same test fails on the same Helm template file.

❌ dynamo-runtime / rust-gpu: exit code 137 (killed) during cargo build

Job: dynamo-runtime / rust-gpu · Failed step: Run Rust checks (block-manager + media-ffmpeg + MM routing + integration tests) · Logs: gh run view --job 121332265945 -R ai-dynamo/dynamo --log-failed

   Compiling dynamo-bench v1.6.0 (/__w/dynamo/dynamo/lib/bench)
   Compiling dynamo-mocker v1.6.0 (/__w/dynamo/dynamo/lib/mocker)
##[error]Error: failed to run script step: Error: command terminated with non-zero exit code: command terminated with exit code 137
##[error]Process completed with exit code 1.
##[error]Executing the custom container implementation failed. Please contact your self hosted runner administrator.
##[error]Error: Internal error occurred: ... failed to create exec "...": task 0af513c5... not found

The build was SIGKILLed (exit code 137) while compiling crates. There is no compiler error, and afterwards the runner container was gone (task ... not found). This looks like an infrastructure failure (OOM kill or lost runner pod), so rerunning the failed jobs is likely enough.

For agents

{"pr": 17282, "run_id": 40246321389, "run_attempt": 1, "head_sha": "c1a4c16956accc08d56eea1b4c23dd1ba65cc45d",
 "failures": [
  {"job": "dynamo-runtime / test / sequential cuda13.0, amd64", "job_id": 121332266048, "failed_step": "Run CPU-only tests (parallelized)", "signature": "yaml.scanner.ScannerError parsing Helm template examples/global-router/cdk/helm/dynamo-mocker/templates/dgd.yaml", "tests": ["deploy/utils/tests/test_convert_api_version.py::test_examples_default_to_v1beta1"], "log_cmd": "gh run view --job 121332266048 -R ai-dynamo/dynamo --log-failed"},
  {"job": "dynamo-runtime / test / sequential cuda13.0, arm64", "job_id": 121332266106, "failed_step": "Run CPU-only tests (parallelized)", "signature": "yaml.scanner.ScannerError parsing Helm template examples/global-router/cdk/helm/dynamo-mocker/templates/dgd.yaml", "tests": ["deploy/utils/tests/test_convert_api_version.py::test_examples_default_to_v1beta1"], "log_cmd": "gh run view --job 121332266106 -R ai-dynamo/dynamo --log-failed"},
  {"job": "dynamo-runtime / rust-gpu", "job_id": 121332265945, "failed_step": "Run Rust checks (block-manager + media-ffmpeg + MM routing + integration tests)", "signature": "exit code 137 (killed) during cargo build; runner container lost", "tests": [], "log_cmd": "gh run view --job 121332265945 -R ai-dynamo/dynamo --log-failed"}
 ]}

_{Posted automatically by Devin for run 40246321389. Updated on every full-CI run of this PR.}

[atchernych]

Replaced by #17332, the same change rebuilt for main on top of #17331. Closing this one, since it targets the global-router-v2 test branch.