Skip to content

fix(manager): acquire only as many deployments as can be lease-guarded - #244

Merged
alongubkin merged 2 commits into
mainfrom
alon/alien-378-bound-lease-renewal
Jul 27, 2026
Merged

fix(manager): acquire only as many deployments as can be lease-guarded#244
alongubkin merged 2 commits into
mainfrom
alon/alien-378-bound-lease-renewal

Conversation

@alongubkin

Copy link
Copy Markdown
Member

Why

A deployment's lease is only renewed once the deployment is actually being
processed. Both manager loops acquired far more than they process at once:

  • the deployment loop took 10 and ran MAX_CONCURRENT_DEPLOYMENTS (4)
  • the heartbeat loop took 100 and ran 4

Everything waiting behind the concurrency limit therefore held a lease that
nothing was extending. With a long lease that only wasted ownership. Platform
is moving to a short provisional lease that the manager confirms before
starting work (ALIEN-378), and under that model the surplus can lapse before
its turn arrives — so this needs fixing first.

What

  • Acquire in batches sized to the processing concurrency, and loop. Every
    acquired lease is immediately guarded. Per-tick throughput is unchanged: the
    deployment loop already batched, and the heartbeat loop now batches too.

  • Tighten lease-renewal timing so the runner abandons a deployment well before
    the lease could be reacquired elsewhere, and assert that relationship at
    compile time so the constants cannot drift into an unsafe combination:

    const _: () = assert!(
        LEASE_RENEW_FAILURE_DEADLINE.as_secs() + LEASE_RENEW_REQUEST_TIMEOUT.as_secs()
            < MINIMUM_CONFIRMED_LEASE.as_secs(),
    );

    Worst case moves from 30s of margin to 80s.

  • Refresh the Platform API spec (pnpm run generate:api-copy-openapi) for the
    new acquire/renew shape.

Note on the generated TypeScript SDK

Deliberately not regenerated here. The checked-in output is already stale
against the pinned Speakeasy generator — a run from an unmodified main
rewrites ~600 files / ~73k lines — so regenerating it in this PR would bury two
schema fields in unrelated churn. I verified this by regenerating from a clean
main: 603 files changed, slightly more than with this branch's spec. That
drift needs its own pass.

Testing

cargo test -p alien-manager -p alien-deployment — all green.

Note: test_platform is flaky under parallelism on main (a temp-dir race in
vault secrets; different tests fail per run). It passes deterministically with
--test-threads=1, on this branch and on main. Not introduced here.

Refs ALIEN-378

@greptile-apps

greptile-apps Bot commented Jul 27, 2026

Copy link
Copy Markdown

Greptile Summary

The PR aligns deployment acquisition with processing capacity and tightens lease-renewal safety margins.

  • Limits deployment and heartbeat acquisition batches to the configured concurrency.
  • Processes heartbeat deployments in sequential bounded batches while preserving the per-tick cap.
  • Adds compile-time checks relating renewal timing to the minimum confirmed lease.
  • Refreshes the checked-in Platform OpenAPI specifications for the lease protocol.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
crates/alien-manager/src/loops/deployment.rs Aligns the deployment acquisition batch size with the maximum processing concurrency.
crates/alien-manager/src/loops/heartbeat.rs Reworks heartbeat processing into bounded acquisition and processing batches.
crates/alien-deployment/src/runner.rs Tightens lease-renewal timeouts and adds compile-time timing invariants.
client-sdks/platform/openapi.json Refreshes the Platform API schema for the updated lease protocol.
client-sdks/platform/rust/openapi-3.0.json Refreshes the Rust client’s OpenAPI 3.0 schema copy.
client-sdks/platform/rust/openapi.json Refreshes the Rust client’s source OpenAPI schema copy.

Reviews (2): Last reviewed commit: "chore(sdk): refresh the Platform API spe..." | Re-trigger Greptile

Both loops acquired far more deployments than they process concurrently:
the deployment loop took 10 and ran 4 at a time, and the heartbeat loop took
100 and ran 4 at a time. A deployment's lease is only renewed once it is
actually being processed, so everything waiting behind the concurrency limit
held a lease that nothing was extending. With a long lease this merely wasted
ownership; with a short one the surplus can lapse before its turn arrives.

Acquire in batches sized to the processing concurrency and loop, so every
acquired lease is immediately guarded. Per-tick throughput is unchanged: the
deployment loop already batched, and the heartbeat loop now batches too.

Also bound lease renewal so the runner gives up ownership well before the
lease could be reacquired elsewhere, and assert that relationship at compile
time so the constants cannot drift into an unsafe combination.

Refs ALIEN-378
Copied by `pnpm run generate:api-copy-openapi` from the platform repo. Adds the
provisional lease expiry on the acquire response and the batch form of the
renewal request.

The generated TypeScript SDK is intentionally not regenerated here. The checked
in output is already stale against the pinned Speakeasy generator — a run from
an unmodified main rewrites ~600 files — so regenerating it inside this change
would bury two schema fields in unrelated churn. That drift needs its own pass.

Refs ALIEN-378
@alongubkin
alongubkin force-pushed the alon/alien-378-bound-lease-renewal branch from 1b81264 to 018607f Compare July 27, 2026 19:27
@alongubkin
alongubkin merged commit 47eb8f6 into main Jul 27, 2026
21 checks passed
@alongubkin
alongubkin deleted the alon/alien-378-bound-lease-renewal branch July 27, 2026 20:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant