Repository navigation
feat: suspend and idempotent resume for sandboxes - #1
Merged
Merged
Conversation
added 19 commits
October 9, 2026 19:27
Suspend releases a sandbox's capacity slot and keeps its workspace;
resume re-acquires a slot, reusing the directory on the original worker
when it is healthy and otherwise reassigning and restoring from a shared
workspace or an object-store snapshot. Both are idempotent.
- Route states SUSPENDING/SUSPENDED, compare-and-set transitions; exec
admission and suspend decide on the same row (also fixes the SQLite
path where FOR UPDATE is a no-op).
- Worker suspend takes the exclusive lifecycle lock without waiting, so
an in-flight command in any scope refuses it (SANDBOX_SUSPEND_BUSY).
- Optional snapshot of /workspace, /home, /envs (SANDBOX_SUSPEND_SNAPSHOT).
- Retention (SANDBOX_SUSPENDED_RETENTION_SECONDS, default 7 days) and
disk-pressure eviction of snapshotted dormant copies by the reaper.
- REST /api/v1/sandboxes/{id}/suspend|resume and MCP sandbox_suspend /
sandbox_resume; optional DormantRouteStore / DormantLifecycle plugin
protocols (501 when unsupported).
- Docs: docs/SUSPEND_RESUME.md, configuration, architecture, READMEs.
…ifier verify-deployment.py pins the published MCP toolset and halted on a real deployment with 'unexpected [sandbox_resume, sandbox_suspend]', which would fail CI's deployment job. Add both tools, and assert in the MCP server test that the verifier's pin matches what the server publishes.
Both observed against a running deployment: resume after retention expiry answers 409 STALE_SANDBOX_ROUTE, and a cross-worker restore leaves the old directory on the original worker.
A cross-worker resume (restore from snapshot) left the old directory on the original worker forever. Each worker's maintenance cycle now lists its local workspace directories, asks the metadata store which are orphaned (route released or pointing at another worker) and deletes those that stayed orphaned for SANDBOX_ORPHAN_DORMANT_DIR_TTL_SECONDS (default 86400, 0 keeps them). The clock is an on-disk marker under the template cache root, so a restart does not reset it; a directory whose route names this worker, or with no route, or on shared storage, is never deleted, and the deletion re-checks under the sandbox lifecycle lock.
A SUSPEND_EXPIRED release that could not reach the worker stayed RELEASING; the reaper's RELEASE_RETRY then finished it but skipped the snapshot, which was only deleted for routes still SUSPENDED, leaking it in the object store. Delete the dormant snapshot for RELEASING routes too, count routes already gone in the dormant sweep like the idle sweep does, and document that retention expiry is an ordinary release (same cleanup, audit and counters).
…rmant copy Found in Docker: a suspended sandbox whose expiry was finished while its worker was paused (RELEASE_RETRY with the worker away) kept an in-memory dormant entry on that worker, which hid its directory from the orphan cleanup forever. Only live sandboxes and releases in flight are protected by memory now; a dormant entry reported orphaned by its route is dropped together with its directory.
SANDBOX_SUSPENDED_RETENTION_SECONDS defaults to 2592000 (30 days) instead of 604800 (7 days). Docs, .env.example and the tests pinning the reported period follow, and the docs now say that an object-store lifecycle rule used as a backstop on checkpoints/ must expire later than the retention.
…and S3, and the 501 for adapters without them
…s under disk pressure again
…so any S3-compatible store can run the suite
…ted worker relearns
…e file-replacement race test independent of the timer
…nd give the failure-diagnostic steps the token they need
… profile on the Ubuntu runners
…an probe can resolve the image
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lets a waiting agent give its capacity slot back without losing its workspace, then continue later.
409 SANDBOX_SUSPEND_BUSY); keeps the directory, releases the slot. Idempotent.SANDBOX_SUSPEND_SNAPSHOT). Idempotent; failures roll back to SUSPENDED.resolvewakes a suspended sandbox.SANDBOX_SUSPENDED_RETENTION_SECONDS(default 30 days, 0 = keep; a backstop bucket lifecycle rule oncheckpoints/must expire later than this); the reaper releases expired ones (SUSPEND_EXPIRED) and evicts snapshotted dormant copies under disk pressure./api/v1/sandboxes/{id}/suspend|resume, MCPsandbox_suspend/sandbox_resume, optionalDormantRouteStore/DormantLifecycleplugin protocols (501 when unsupported). No schema change.docs/SUSPEND_RESUME.md, configuration, architecture, operations, adapters, READMEs.scan_public_tree.pyclean. Bubblewrap integration tests were not runnable in the dev environment.test_stat_open_races_never_read_replacement[file-replacement]is flaky independent of this change.Known limitations: a template mounted before a cross-worker resume must be re-mounted; worker-side slot check is not atomic; with local-only storage and no snapshot, losing the original worker loses the dormant workspace until retention reclaims the route.