Snapshots you take on purpose, mirrored to S3, and the way back from it - #150
Merged
Conversation
`studio.sh up --bind 0.0.0.0` printed "starting Studio" and exited. No daemon, no error, an empty api.log — the script died working out which names to allow rather than starting anything. reachable_hosts asks every interface for its address so the guard can allow the names this machine actually answers to. Every Mac holds interfaces that have none — en4, awdl0, llw0 — and `ipconfig getifaddr` exits 1 on each. Under `set -e` that aborts the loop and the function and the launch, which is why the `return 0` at the end never got the chance to make it harmless: it is the last command that decides a function's status, and control never reached it. So the `|| true` goes on the failing command rather than at the end, and the `hostname` call above it gets the same treatment for the same reason. Loopback binds never call this, which is why it only ever hit the one flag it exists to serve. Claude-Session: https://claude.ai/code/session_011ba7PUJ23wb1V1aBhVbnE8
A snapshot was something the crash net recorded on a timer and `recover` found
for you. This makes it something you can ask for — before a risky migration,
around an agent you are not sure about — through the SDK and Studio, gives it
somewhere to live off this machine, and gives you a command to bring it back.
The mechanism is unchanged, which is the point: a commit of the working tree
under refs/sandbox/snapshots/, written through a private GIT_INDEX_FILE and
through internal/githard, so your index, HEAD, branches and working tree are
never touched and a hostile repository cannot make the capture run a command on
the host. What is new is who can ask, where the bytes may go, and how they come
home.
rescue.Capture one snapshot now; Available() failing is an error here
where Begin is silent — a caller who asked explicitly is
owed the answer.
rescue.Mirror upload as a git bundle, record the result on the manifest.
rescue.Fetch bring one back into refs/sandbox/.
POST /v1/snapshots and /restore, /retention, /upload, /verify, /settings,
s3/check, plus /v1/runs/{id}/snapshot.
ws.snapshot() and restore/snapshots/uploadSnapshot/verifySnapshot.
Snapshots screen lists them, restores the ones that came from a run.
The format is the decision. A **git bundle**, not an archive, so the object in
the bucket is a packfile git alone can read on a machine that has never seen the
repository — `git init && git fetch <bundle>` — rather than something that needs
this tool. `git clone` of it does not work, and that follows from the rule this
package keeps rather than from an oversight: a snapshot ref is not a branch, so
the bundle carries no HEAD, and giving it one would mean writing under
refs/heads to bundle it from. The accepted cost is that a bundle is
self-contained — sized like a clone, not like a diff — which is why the default
is `upload: manual` and why max_object_mb refuses one up front rather than at the
far end of a transfer S3 was always going to reject at 5 GiB.
Mirroring is a **backup, never an offload**: the local ref stays and is what
Restore reads, so Fetch puts the objects back where they always were and none of
the three restore modes learns that a network exists.
internal/s3 is the smallest client that can hold a snapshot — put, get, stat,
delete, list — hand-rolled SigV4 against a published AWS test vector rather than
the AWS SDK, for the reason this module depends only on cobra and yaml.v3: 15MB
of transitive code and a release cadence to track, in exchange for five requests
whose signing has been stable since 2012. Vendors are configuration (Endpoint,
PathStyle), not a list. The costs are named rather than hidden: no IMDS, no SSO,
no config-file profiles, no multipart.
The credential is **named, never held**. access_key_env is the name of a variable
read on this machine, the same shape gateway: uses, so there is nowhere in the
config file, the settings file, the API response or the browser for a secret to
sit. And snapshot.s3 is refused from a project .sandbox.yaml along with the rest
of the snapshot key: it names a network destination *and* which of your
credentials is read to reach it, which is an exfiltration target and the means to
fill it, handed to a file the agent can write.
Three refusals are load-bearing. An unchanged tree is refused rather than
recorded — a snapshot id pointing at no commit is worse than none, because you
find out at the moment you try to roll back. Fetch validates the manifest's ref
before spending a download, since a manifest is a file on disk and is never
allowed to name a branch. And it compares the fetched sha against the one
recorded locally: `git bundle verify` proves a bundle is well formed, not that it
is *yours*, and a well-formed bundle of somebody else's commit served under this
key restores as silent success.
`sandbox-cli recover fetch` is the way back on a machine that still has this
tool. With no argument it reads the manifests stored beside the bundles and says
what is in the bucket, which works on a machine that has never seen these
snapshots — that self-describing half is what those few hundred bytes per
snapshot were always for. With an id it fetches, adopts the manifest locally, and
hands the snapshot to `show` and `restore` as one that never left.
Two things it refuses to guess at, both consequences of decisions made elsewhere.
A repository is addressed in the bucket by worktree.RepoID, a hash of its
absolute path, so a clone in a new location looks in an empty namespace of its
own; an empty listing therefore names the other ids that are in the bucket and
--repo-id reads one of them. And a session read out of the bucket is checked
against a sha that came from the same bucket, which proves the bundle and its
manifest agree rather than that either is yours — the command says which of the
two checks ran, because a successful fetch looks identical either way.
rescue.DeleteRemote is removed. It had no caller and a doc comment reading "for
pruning", which contradicts the decision recorded beside it: retention prunes the
local copy only, since deleting an off-machine backup on a timer that runs while
the laptop is open loses the copy meant to survive the laptop. Left in place it
is a trap, not a loose end — the next reader wires it into Prune. s3.Delete stays;
that package's surface is small on purpose and generic by design.
Also fixed, both stale rather than new: docs/GUIDE.md showed `snapshot:` inside
its .sandbox.yaml sample and left it out of the refused-key list, so the guide
advertised a project setting the loader has been refusing; and the site's config
mirror described snapshot as project-settable for the same reason. Both now say
user-config-only, and carry manual_retention and s3.
POST /v1/runs/{id}/recover restored the state a Studio run *started* from: the
daemon records a baseline before every launch, and it matched on branch, agent
and recency. Not an error — a restore that looked like it worked. Baselines are
skipped when looking for something to recover, and hidden from the listing.
Verified: make fmt, go build ./..., go test ./..., the two contract mirror sync
tests, npm test in sdk/typescript (51/51), tsc --noEmit in studio and web. The
fetch path was also run end to end against a stand-in S3 — seeded a bucket,
destroyed the ref, the objects and the rescue directory, then listed, fetched and
restored, including across two repository namespaces.
Claude-Session: https://claude.ai/code/session_011ba7PUJ23wb1V1aBhVbnE8
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Eight findings from the review of #150, two of them the same defect wearing different clothes. **rescue.Find returns its error *and* the snapshot** when the manifest is here and the objects are not — deleted ref, collected commit. Both callers with somewhere to look were written against `err != nil`, so: - `POST /v1/snapshots/{id}/restore` answered 404 for a snapshot with a perfectly good bundle in the bucket. The fetch-back branch below it, and the sentence in the changelog about the local ref being what restore reads, were unreachable code. - `recover fetch <id>` never took its local-record path either, so every fetch fell through to the bucket manifest and printed "this machine had no record of it" about a session this machine had a record of — losing the one check that proves provenance rather than internal consistency, which is the difference the command exists to report. - the upload handler's "no objects left to upload" refusal was dead for the same reason, replaced by a 404 that reads as a wrong id. So the recoverable refusal gets a name — `ErrSnapshotGone` — and the callers match on it. A refusal that carries the means to undo it has to be matched on, not merely reported. `TestRestoreFetchesASnapshotThatIsOnlyInTheBucket` is the guard, against a fake bucket that now stores what it is given: reverting the fix reproduces the 404, which was checked before the fix went in. **A settings write clobbered what config.yaml pins.** `SnapshotSettings` was both the read and the write shape, and "" cannot mean "leave this alone" and "clear it" in one string field. The screen sends "" for a field it has just told the user it cannot change, so editing the manual window blanked the run window in the running daemon — reverting crash pruning to the built-in default for the life of the process — and the next read, finding nothing left to attribute to config.yaml, un-pinned the field it had just discarded. Fixed only by a restart. `SnapshotSettingsUpdate` is the write shape: optional fields, absent means leave it alone, "" means clear this daemon's override. That is the rule the bucket already had, said in a way a string field can carry. `configPins` is now one function shared by the read and the write, because a screen that says "config.yaml sets this" while the write path overwrites it anyway is worse than either behaviour alone. **Studio's storage card wrote retention it had never read as its own.** `save.mutate({ ...data!, s3: form })` spread the *resolved* windows into the write, copying config.yaml's values — or the built-in defaults — into Studio's override file, where they outlive the line they came from. It is the exact mistake the retention card next door carries a comment about avoiding. It now sends only the bucket, and the retention card sends `undefined` rather than "" for a pinned field. **Row actions ignored the row's repository.** The listing defaults to "All repositories" and spans every registered project, but restore/upload/verify/ retention sent no `repo`, so the daemon resolved them against its *default* project: every row from a second repository 404'd. Each row already carries its repoId; the four mutations take it per call now. **A namespace was read by searching for a marker inside the key.** `strings.Index(key, "snapshots/")` finds it inside a configured prefix, so `prefix: my-snapshots` made `RemoteRepoIDs` report "snapshots" as a repository — a namespace that does not exist, printed to the user as the `--repo-id` to try. Read against the client's own base instead; a key outside it belongs to something else sharing the bucket and is skipped. **RemoteSessions rewrote Repo and not Workspace.** studioapi's restore passes `sess.Workspace` to Restore, so a snapshot fetched and adopted from the bucket would have been restored against the *uploading* machine's path — absent here, or on a shared layout, present and something else. **`upload: all` could hold a Ctrl-C for twenty minutes.** Stop's final snapshot mirrors synchronously, and remoteTimeout is generous because a bundle is megabytes over a connection nobody chose — the right trade on the interval and the wrong one on the way out. The final mirror gets its own budget. It loses at most the last upload, and loses it visibly: the manifest records the failure and `POST /v1/snapshots/{id}/upload` exists for exactly that copy. Not changed: `Mirror` mutating the session outside `s.mu`. That is how `take` has always written the manifest, and narrowing it is a separate change to the Snapshotter's locking rather than part of this one. Verified: make fmt, go build ./..., go test ./..., make contract with both mirror sync tests, npm test in sdk/typescript (51/51), tsc --noEmit in studio and web. The CLI fetch was re-run end to end against a stand-in S3 and now takes the local-record path — no provenance caveat, and the restore returns the work. Claude-Session: https://claude.ai/code/session_011ba7PUJ23wb1V1aBhVbnE8
The Launch form had an "Extra host directories" field. The preview warned that
`--share` widens the boundary on purpose. `toRunCreate` never copied it into the
request body, and `RunCreateRequest` had nowhere to put it if it had — so two
agents told to exchange a file through /shared were running in containers with
three mounts and no /shared, and nothing anywhere said so.
Found by inspecting the containers rather than the code: two live Studio runs in
different repositories, workspace + .git/hooks + persisted HOME, and that is all.
It is now a toggle, and it does what the CLI's flag does through the same code.
`sandbox.ShareMount` is `cli.shareMount`'s resolution moved into internal/sandbox
— the daemon cannot import internal/cli (and should not: that would drag cobra
and every wrapper into the control plane binary), and a second copy of the code
that decides what a container can reach is the thing this repository refuses
everywhere else. What stays in internal/cli is the sentence it prints: a terminal
has somewhere to announce the widening, an HTTP handler does not. The path-safety
tests moved with the code; the flag-parsing and dry-run ones stayed with the
flags.
A boolean, not the list of host paths the field looked like. An arbitrary
directory named in a request is a browser choosing what a container reaches;
this is one well-known directory the daemon creates, seeds, checks with
RefuseUnsafeHostPath and opens to the container's group. The wider thing is
`--mount` and it is deliberately still not offered over HTTP. `shareName`
narrows to a namespace and is refused without `share` — implying would let one
field switch the cross-project channel on, and `share:false, shareName:"work"`
would be a contradiction settled by guessing, with the wrong guess widening the
boundary for somebody who wrote false.
A second defect sat behind the first and would have outlived fixing it: the
worktree case **assigned** its linked-.git mounts over `extraMounts` rather than
appending, so a worktree run — the runs most likely to want a handoff — would
have silently lost the share mount it asked for. internal/fleet has had a test
for that exact rule since sharing reached the fleet ("it must be *added* to the
worktree's .git mount, not put in its place"); the daemon now has the mirror of
it, along with tests that the mount is the shared dir, that a namespace mounts
the leaf and not the root, that an unsafe name is refused, and that a run which
does not ask gets nothing.
The read side needed no change: resolve.go's mountOrigin already returns "share"
for a /shared destination, and the run-detail view already explains it. Only the
write side was missing.
Verified: make fmt, go build ./..., go test ./..., make contract with both mirror
sync tests, npm test in sdk/typescript (51/51), tsc --noEmit and eslint in studio.
Claude-Session: https://claude.ai/code/session_011ba7PUJ23wb1V1aBhVbnE8
The site's Studio walkthrough took a run from launch to answering an agent and never mentioned that two runs can hand a file to each other, because until the previous commit they could not — the Launch form collected the option and the request dropped it. The new step sits beside "Launch a run" rather than at the end, since the case that wants it is two tabs open on two repositories, and that is a thing you set up while launching the second one. It carries the two facts that are not guessable: what to say in *both* prompts, since a mounted directory is invisible to an agent nobody told about it, and that a namespace prevents collisions rather than access. The warn line is the boundary decision, in the place somebody meets it: the toggle is a boolean and not a path you name, because an arbitrary host directory chosen in a browser is a different act from one typed in your own terminal. The wider --mount stays a CLI flag, and the page now says so rather than leaving the absence to be noticed. The features card said --share works "in every sandbox that asks for it", which was true and read as CLI-only next to a flag name. It now names both front ends. Not reformatted: web/src is not prettier-clean on main and CI does not check it, so these edits match the file rather than the tool. Claude-Session: https://claude.ai/code/session_011ba7PUJ23wb1V1aBhVbnE8
Amitgb14
added a commit
that referenced
this pull request
Sep 9, 2026
Reported as lost work: an agent wrote a file, was killed before committing, and the restored branch did not contain it. The restore was correct. The snapshot never held the file, because it was a *baseline* — the workspace as the run started, captured before the agent ran — and nothing said so. A run started from Studio records only that: studioapi takes one before-image at launch and closes the session immediately. So killing an agent before it commits leaves exactly one session, holding the state from before it did anything. On the reporter's machine every one of seven OneNutri sessions was a baseline, and `recover list` reported all of them as `clean` — the same word a finished run's snapshot gets. The daemon already guards against this: #150 taught it to skip baselines when looking for something to recover and to hide them from the snapshot listing. That fix never reached internal/cli, which did no filtering on Outcome at all. Same defect, other front end. `OutcomeBaseline` moves to internal/rescue beside the other outcomes, and studioapi's private literal now refers to it. Its comment there was already right — "a literal that has to match in three places is one typo away from offering a run's starting state as its work" — and it had matched in two, which is how the CLI came to be blind to a value the daemon has been writing for weeks. `Status()` names it, ahead of the closed-session case that was swallowing it, and `IsBaseline` is the question callers actually ask. The listing counts them and says what they are; the restore says it *before* the git commands it prints, because it changes what those commands are for — somebody who reads "created branch X" and goes looking will find the state they had before the run and conclude the snapshot lost their work rather than that it never held it. Marked rather than hidden, and that is a deliberate difference from the daemon. Studio hides them because its screen offers a Restore button beside every row. This listing is what somebody reads while hunting for lost work, where "no snapshots recorded" is the worse answer: there *was* a run, and this is all it left. Claude-Session: https://claude.ai/code/session_011ba7PUJ23wb1V1aBhVbnE8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Takes the crash net's snapshot mechanism and makes it something you can ask for, store off this machine, and bring back — through the SDK, Studio and one new CLI command. Two commits: a
studio.shfix that is independent of the rest, and the feature.What lands
internal/rescueCapture(snapshot now),Mirror/Fetch/Verify/CheckRemote(object storage),RemoteSessions/RemoteRepoIDs(read the bucket), per-snapshot retentioninternal/s3Put/Get/Stat/Delete/List/Checkinternal/configsnapshot.manual_retention,snapshot.s3, both refused from a project.sandbox.yaml/v1/snapshots(+restore,retention,upload,verify,settings,s3/check) and/v1/runs/{id}/snapshotws.snapshot(),restore,snapshots,uploadSnapshot,verifySnapshot, typedNothingToSnapshotErrorsandbox-cli recover fetchThe decisions worth reviewing
A git bundle, not an archive. The object in the bucket is a packfile git alone can read on a machine that has never seen the repository.
git cloneof it does not work — a snapshot ref is not a branch, so the bundle carries no HEAD, and giving it one would mean writing underrefs/headsto bundle it from. The accepted cost is that a bundle is self-contained, sized like a clone rather than like a diff, which is why the default isupload: manualand whymax_object_mbrefuses one up front.A backup, never an offload. The local ref stays and is what
Restorereads.Fetchputs the objects back where they always were, so none of the three restore modes learns that a network exists.The credential is named, never held.
access_key_envis the name of a variable read on this machine — there is nowhere in the config file, the settings file, the API response or the browser for a secret to sit.snapshot.s3is refused from a project.sandbox.yaml: it names a network destination and which of your credentials is read to reach it.Three refusals. An unchanged tree is refused rather than recorded.
Fetchvalidates the manifest's ref before spending a download. And it compares the fetched sha against the one recorded locally —git bundle verifyproves a bundle is well formed, not that it is yours.recover fetch, and the two things it will not guessWith no argument it reads the manifests stored beside the bundles, so it works on a machine that has never seen these snapshots. With an id it fetches, adopts the manifest, and hands the snapshot to
show/restoreas one that never left.worktree.RepoID— a hash of its absolute path — so a clone in a new location looks in an empty namespace of its own. An empty listing names the other ids in the bucket;--repo-idreads one.Removed
rescue.DeleteRemote— no caller, and a doc comment reading "for pruning" that contradicts the decision recorded beside it (retention prunes the local copy only, because deleting an off-machine backup on a timer that runs while the laptop is open loses the copy meant to survive it). Left in place it is a trap rather than a loose end.s3.Deletestays; that package's surface is small on purpose and generic by design.Stale docs fixed on the way past
docs/GUIDE.mdshowedsnapshot:inside its.sandbox.yamlsample and omitted it from the refused-key list, so the guide advertised a project setting the loader has been refusing since before this branch. The site's config mirror said the same. Both now say user-config-only and carrymanual_retentionands3.Verification
make fmt,go build ./...,go test ./..., the two contract mirror sync tests,npm testinsdk/typescript(51/51),tsc --noEmitinstudioandweb. No Docker needed for any of it.The fetch path was also run end to end against a stand-in S3: seed a bucket, destroy the ref, the objects and the rescue directory, then list, fetch and restore — including across two repository namespaces, which is the case
--repo-idexists for.🤖 Generated with Claude Code
https://claude.ai/code/session_011ba7PUJ23wb1V1aBhVbnE8