A failed object-storage HEAD on a cache miss renders a populated repo as an empty 200 on the read endpoints #300
Copy link
Copy link
Open
Labels
crate:nodegitlawb-node — the serving node and REST APIgitlawb-node — the serving node and REST APIkind:bugDefect fix — wrong or unsafe behaviorDefect fix — wrong or unsafe behaviorsev:mediumDegraded but workaround existsDegraded but workaround existssubsystem:apiNode REST API request/response surfaceNode REST API request/response surfacesubsystem:storageBlob/object store, Arweave, IPFS, archivesBlob/object store, Arweave, IPFS, archives
Description
Activity
- addedsev:mediumDegraded but workaround existsDegraded but workaround existscrate:nodegitlawb-node — the serving node and REST APIgitlawb-node — the serving node and REST APIkind:bugDefect fix — wrong or unsafe behaviorDefect fix — wrong or unsafe behaviorsubsystem:apiNode REST API request/response surfaceNode REST API request/response surfacesubsystem:storageBlob/object store, Arweave, IPFS, archivesBlob/object store, Arweave, IPFS, archives
on Aug 3, 2026
Metadata
Metadata
Assignees
Labels
crate:nodegitlawb-node — the serving node and REST APIgitlawb-node — the serving node and REST APIkind:bugDefect fix — wrong or unsafe behaviorDefect fix — wrong or unsafe behaviorsev:mediumDegraded but workaround existsDegraded but workaround existssubsystem:apiNode REST API request/response surfaceNode REST API request/response surfacesubsystem:storageBlob/object store, Arweave, IPFS, archivesBlob/object store, Arweave, IPFS, archives
RepoStore::acquirecollapses a failed object-storage HEAD into "no archive exists":crates/gitlawb-node/src/git/repo_store.rs:141Err(_)andOk(false)fall through to the same terminalOk(local_path), with no log line on the error arm. On the fast path this never evaluates, because a local copy short-circuits above it. It only matters on a cache miss, where there is no local copy to return, so the function hands back a path that does not exist.The part worth leading with: a blip renders a populated repo as empty, with a 200
Several read endpoints swallow the downstream failure with
unwrap_or_default():list_commits(crates/gitlawb-node/src/api/repos.rs:361) returns{"commits": []}list_tree/get_tree(repos.rs:443,repos.rs:481) return{"entries": []}crates/gitlawb-node/src/api/changelog.rs:46) returns no commit eventsAll with HTTP 200. A client cannot tell a transient storage failure from a genuinely empty repository. The remaining callers are less subtle but still wrong-classed: clone advertisement, blob reads and fork return a non-retryable 500
git_errorwhere the condition is transient and retryable.Why this is not the same call as the one just made in #285
#285 fixed the sibling
acquire_fresh, which now refuses with a retryable 503 when the HEAD cannot be read, and its comment names the construct still present here. That fix deliberately left the read path alone, on the grounds that serving a possibly-stale local copy is reasonable for a read. That argument is sound for the fast path and does not apply here: on a cache miss there is nothing to serve, so the availability-over-consistency trade has nothing to trade with.Worth recording, since it affected the earlier scoping: the prior-art learning that was cited to defer this site actually names
acquire_fresh, which is now fixed. Nothing on record coversacquire, and there is no comment at line 141 marking the behavior intentional.Reachability, honestly
Narrow. There is no cache eviction (no reaper over
repos_dir, and the Fly volumes are persistent withauto_stop_machines = false), so a node keeps a repo once it has it. The window is first-read-of-a-repo-on-this-node intersected with a concurrent object-storage failure. That is not exotic in a multi-node deployment, where a repo pushed to one node is genuinely absent on the others until someone reads it there, and fresh deploys start on empty volumes.Also worth stating: if the HEAD fails, the download would very likely fail too. Fixing this changes the status code and the retryability signal, not whether the request succeeds.
Suggested shape
Classify the
Errarm the wayacquire_freshnow does, so a storage failure surfaces as the retryable 503 rather than a 500 or a false-success 200. Theunwrap_or_default()swallows at the three read sites are the part that was never a deliberate decision, and they are worth addressing whether or not the classification changes.