[atchernych]

Summary

Adds --registration-storage dynamodb: registrations kept in one DynamoDB table that every discovery instance reads, such as a Global Table with a replica in each region. Instances in every cluster then share registrations without asking each other, and a restarted instance lists every live pool at once.

Stacked on #17332. It is the alternative to #17637, which shares registrations between instances over HTTP: use this where DynamoDB is available, and #17637 (or a single instance) where it is not. If both land, --peer-service with dynamodb should be refused.

Part of #17600, the pool discovery DEP.

How it works

• POST /pools writes the registration with the end of its lease, from the writing instance's clock. The pool is listed from the next read of the table, within a second.
• Each instance reads the whole table every second, with strongly consistent reads of its region's replica, and lists the registrations whose leases have not ended. A registration made in another region arrives with replication, usually within a second. The lease's slack over the heartbeat interval (10 s with the defaults) covers that delay and the clock skew between instances.
• GET /pools says complete: true as soon as one read succeeds, so there is no one-lease wait after a restart. A failed read keeps the last list, with complete: false, until reads work again. A registration needs the table, so POST /pools answers 503 while the table cannot be reached.
• It works with every --pool-discovery value. With karmada or mcs, the join from feat(global-router): read pool models from registrations, not a file #17332 reads it in place of the memory store.

The table
The same key layout as the MCS registry's table in #17615: pk and sk strings, the expires_at time to live, and % and # escaped in keys.

• One item per pool: pk is POOL, sk is <cluster_id>#<namespace>#<dgd_name>, registration holds the JSON the PoolRelay sent, lease_ends_ms the end of its lease, and expires_at a time an hour later, so that DynamoDB deletes pools that stopped registering.
• The service does not create the table. The module docs have the aws dynamodb create-table and update-time-to-live commands. It needs dynamodb:PutItem and dynamodb:Query on the table, and AWS credentials from the usual chain, such as IRSA on EKS.
• Every instance reads every item once a second: 100 pools of about 1 KB take about 25 read units a second per instance.

Flags

• --registration-storage dynamodb, with --dynamodb-table (required), --dynamodb-region (otherwise from the AWS environment) and --dynamodb-endpoint-url (for DynamoDB Local). The three fail at startup with another storage, as the other mode-specific flags do.

Build

• The AWS SDK is behind a dynamodb Cargo feature, off by default so that workspace builds skip it. Its dependencies are set up as in lib/kvbm-engine's s3 feature. aws-sdk-dynamodb is 1.105.0, the newest that fits the AWS crates already in Cargo.lock, which only gains that one package.
• The image builds with --features dynamodb. A binary built without it refuses --registration-storage dynamodb at startup, with a message that says how to build it.
• Two steps in pre-merge.yml run the feature's tests and clippy, as for other feature-gated crates.

Validation

• cargo test -p dynamo-global-discovery-service --features dynamodb: 83 passed (70 library, 13 binary), with the DynamoDB Local test ignored; without the feature, 74 (61 and 13). The new tests use a fake table. They cover:
  • listing from the table, with ended leases left out;
  • the watch: it starts at the first read, then follows registrations and changes, and removes a pool whose lease ended;
  • a heartbeat with its lists in another order is not a change;
  • a failed read keeps the last list as Estimated and recovers to Exact, and an unreadable table before the first read is unavailable;
  • the item codec and the sort-key escaping;
  • the new flags.
• Against DynamoDB Local (amazon/dynamodb-local:2.6.0, as feat(global-router): add the Multi-Cluster Services registry and controller #17615 uses): an ignored test, run with --ignored and DYNAMODB_LOCAL_URL, creates a table, skips an item that does not parse, registers, lists, watches, and sees the pool removed when its 2 s lease ends.
• Ran two binaries on one DynamoDB Local table: a pool registered with one instance was listed by the other with complete: true, and a restarted instance listed it at once, also with complete: true. A binary built without the feature refuses --registration-storage dynamodb with exit code 1.
• cargo clippy --all-targets -- -D warnings with and without the feature, cargo fmt, and cargo doc with -D warnings: clean.
• An adversarial review found no runtime bug. From it: the refresh starts after a registration's write, the DynamoDB Local test is ignored instead of passing in CI without running, a flag test that passed the flag twice now checks each error kind, and some docs are fixed. Left as they are: a write that can never succeed, such as a pool key longer than DynamoDB's 1 KB sort key, answers 503 like an outage, and listing and sorted stay in memory.rs, which this backend imports them from.
• Not yet run against AWS.

🤖 Generated with Claude Code

[devin-ai-integration]

❌ Dynamo PR CI failed — run 41146616385 (attempt 1) on 5b8c68d092

Gate checks: ❌ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Other	Jobs
Operator Integration	❌ 1

Failure details

1 job failed: the operator cluster tests fail in two packages — TestClusterComponentHistoryBlocksWorkloads is rejected by the DynamoComponentDeployment admission webhook, and every TestCluster* test in internal/controller fails at clusterenv startup with a CRD update conflict.

❌ Operator Integration: DCD webhook "spec.podTemplate.spec.containers: Required value" + CRD conversion patch conflict

Job: Operator Integration · Failed step: Run operator integration tests · Logs: gh run view --job 124376032497 -R ai-dynamo/dynamo --log-failed

history_clustertest_test.go:194:
    Error: Received unexpected error:
        create DynamoComponentDeployment worker: admission webhook "vdynamocomponentdeployment.kb.io" denied the request: DynamoComponentDeployment.nvidia.com "worker" is invalid: spec.podTemplate.spec.containers: Required value: is required
--- FAIL: TestClusterComponentHistoryBlocksWorkloads (1.78s)
FAIL	github.com/ai-dynamo/dynamo/deploy/operator/internal/testing/clusterenv	12.053s
...
env.go:108: start clusterenv: point conversion CRD "dynamocomponentdeployments.nvidia.com" at cluster-test webhook: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "dynamocomponentdeployments.nvidia.com": the object has been modified; please apply your changes to the latest version and try again
--- FAIL: TestClusterDynamoComponentDeploymentManifests (10.69s)
--- FAIL: TestClusterDynamoGraphDeploymentManifests (0.00s)
--- FAIL: TestClusterDynamoGraphDeploymentRequestProfilesAndCreatesWorkloadManifests (0.00s)
--- FAIL: TestClusterDynamoGraphDeploymentRequestV1beta2ProducesCandidates (0.00s)
--- FAIL: TestClusterFrontendDependencyStartup (0.00s)
--- FAIL: TestClusterGroveScalerUsesScaleSubresource (0.00s)
FAIL	github.com/ai-dynamo/dynamo/deploy/operator/internal/controller	10.744s
make[1]: *** [Makefile:375: clustertest] Error 1

In internal/testing/clusterenv, TestClusterComponentHistoryBlocksWorkloads (history_clustertest_test.go:194) cannot create the worker DynamoComponentDeployment because the validating webhook requires spec.podTemplate.spec.containers. In internal/controller, all 6 TestCluster* tests (13 subtests) fail in clusterenv setup (env.go:108) with a resourceVersion conflict while patching the dynamocomponentdeployments.nvidia.com CRD's conversion webhook; the first subtest fails after 10.7s and the remaining ones fail immediately with the same error.

For agents

{"pr": 17644, "run_id": 41146616385, "run_attempt": 1, "head_sha": "5b8c68d092e10a9a36602242867c717628fb612b",
 "failures": [{"job": "Operator Integration", "job_id": 124376032497, "failed_step": "Run operator integration tests", "signature": "DCD admission webhook: spec.podTemplate.spec.containers Required; clusterenv CRD conversion patch conflict (object has been modified)", "tests": ["deploy/operator/internal/testing/clusterenv::TestClusterComponentHistoryBlocksWorkloads", "deploy/operator/internal/controller::TestClusterDynamoComponentDeploymentManifests", "deploy/operator/internal/controller::TestClusterDynamoGraphDeploymentManifests", "deploy/operator/internal/controller::TestClusterDynamoGraphDeploymentRequestProfilesAndCreatesWorkloadManifests", "deploy/operator/internal/controller::TestClusterDynamoGraphDeploymentRequestV1beta2ProducesCandidates", "deploy/operator/internal/controller::TestClusterFrontendDependencyStartup", "deploy/operator/internal/controller::TestClusterGroveScalerUsesScaleSubresource"], "log_cmd": "gh run view --job 124376032497 -R ai-dynamo/dynamo --log-failed"}]}

_{Posted automatically by Devin for run 41146616385. Updated on every full-CI run of this PR.}