fix(manager): acquire only as many deployments as can be lease-guarded - #244
Merged
Conversation
Greptile SummaryThe PR aligns deployment acquisition with processing capacity and tightens lease-renewal safety margins.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| crates/alien-manager/src/loops/deployment.rs | Aligns the deployment acquisition batch size with the maximum processing concurrency. |
| crates/alien-manager/src/loops/heartbeat.rs | Reworks heartbeat processing into bounded acquisition and processing batches. |
| crates/alien-deployment/src/runner.rs | Tightens lease-renewal timeouts and adds compile-time timing invariants. |
| client-sdks/platform/openapi.json | Refreshes the Platform API schema for the updated lease protocol. |
| client-sdks/platform/rust/openapi-3.0.json | Refreshes the Rust client’s OpenAPI 3.0 schema copy. |
| client-sdks/platform/rust/openapi.json | Refreshes the Rust client’s source OpenAPI schema copy. |
Reviews (2): Last reviewed commit: "chore(sdk): refresh the Platform API spe..." | Re-trigger Greptile
Both loops acquired far more deployments than they process concurrently: the deployment loop took 10 and ran 4 at a time, and the heartbeat loop took 100 and ran 4 at a time. A deployment's lease is only renewed once it is actually being processed, so everything waiting behind the concurrency limit held a lease that nothing was extending. With a long lease this merely wasted ownership; with a short one the surplus can lapse before its turn arrives. Acquire in batches sized to the processing concurrency and loop, so every acquired lease is immediately guarded. Per-tick throughput is unchanged: the deployment loop already batched, and the heartbeat loop now batches too. Also bound lease renewal so the runner gives up ownership well before the lease could be reacquired elsewhere, and assert that relationship at compile time so the constants cannot drift into an unsafe combination. Refs ALIEN-378
Copied by `pnpm run generate:api-copy-openapi` from the platform repo. Adds the provisional lease expiry on the acquire response and the batch form of the renewal request. The generated TypeScript SDK is intentionally not regenerated here. The checked in output is already stale against the pinned Speakeasy generator — a run from an unmodified main rewrites ~600 files — so regenerating it inside this change would bury two schema fields in unrelated churn. That drift needs its own pass. Refs ALIEN-378
alongubkin
force-pushed
the
alon/alien-378-bound-lease-renewal
branch
from
July 27, 2026 19:27
1b81264 to
018607f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A deployment's lease is only renewed once the deployment is actually being
processed. Both manager loops acquired far more than they process at once:
MAX_CONCURRENT_DEPLOYMENTS(4)Everything waiting behind the concurrency limit therefore held a lease that
nothing was extending. With a long lease that only wasted ownership. Platform
is moving to a short provisional lease that the manager confirms before
starting work (ALIEN-378), and under that model the surplus can lapse before
its turn arrives — so this needs fixing first.
What
Acquire in batches sized to the processing concurrency, and loop. Every
acquired lease is immediately guarded. Per-tick throughput is unchanged: the
deployment loop already batched, and the heartbeat loop now batches too.
Tighten lease-renewal timing so the runner abandons a deployment well before
the lease could be reacquired elsewhere, and assert that relationship at
compile time so the constants cannot drift into an unsafe combination:
Worst case moves from 30s of margin to 80s.
Refresh the Platform API spec (
pnpm run generate:api-copy-openapi) for thenew acquire/renew shape.
Note on the generated TypeScript SDK
Deliberately not regenerated here. The checked-in output is already stale
against the pinned Speakeasy generator — a run from an unmodified
mainrewrites ~600 files / ~73k lines — so regenerating it in this PR would bury two
schema fields in unrelated churn. I verified this by regenerating from a clean
main: 603 files changed, slightly more than with this branch's spec. Thatdrift needs its own pass.
Testing
cargo test -p alien-manager -p alien-deployment— all green.Note:
test_platformis flaky under parallelism onmain(a temp-dir race invault secrets; different tests fail per run). It passes deterministically with
--test-threads=1, on this branch and onmain. Not introduced here.Refs ALIEN-378