Repository navigation
Conversation
The S3 SDK retries failed calls three times by default and the GCS client retries uploads that carry a precondition, the manifest CAS included. When a CAS lands and its reply is lost, the retry gets a 412 from our own write. process_batch, publish_compact_classified and publish_settings_impl all read any 412 as a lost race and delete their log segment, which the committed manifest now lists: the writer reports a failure for a durable change and every reader then fails on the missing segment. Run the existing cas_landed check on a 412 too before dropping the segment, and keep an unknown outcome unknown when that check fails. The fault store gains the missing fault kind (applied, then 412), sharing the existing CAS roll so seeded simulations keep their sequences, and a test drives a push, a compaction and a settings change through it. One extra manifest GET on the 412 path only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A 412 on the manifest CAS can answer a transport retry of our own landed
write. The previous commit looks before dropping the segment, but treats
"not listed" as "never committed", and the pack seal still drops on any
412:
- seal_pack_replacement answered a landed 412 by deleting the segment the
committed manifest lists: the same corruption, on the maintainer's
retirement path.
- A checkpoint that folds the log at or past our seq drops our segment
descriptor. If one lands between our CAS and the read-back, "not listed"
proves nothing: the writer deleted its committed (folded) segment and
re-verified its own push into a conflict against its own commit.
slot_evidence now classifies the fresh manifest as Committed (our exact
segment is listed), Unlisted (our key is not listed and nothing is folded
at or past our seq, so no committed manifest ever listed it; with a 412
no copy of our request can land later) or Unknown (folded, our key listed
with other contents, or no manifest). Every 412 path - push, compact,
settings and seal - drops the segment only on Unlisted; Unknown keeps it
and reports CommitUnknown. The non-412 paths are unchanged (cas_landed).
Round trips: unchanged. The 412 path still costs one manifest GET (plus
one log GET when our descriptor is listed); successful publishes do not
change.
Tests: a unit test for every evidence case; a_landed_seal_answered_412_
keeps_the_commit (fails before: "manifest lists log/...2.pb but it is
gone"); a_landed_cas_folded_before_the_412_is_resolved_stays_unknown
(fails before: the push is answered Conflict{expected: (new), actual:
<its own commit>}); and a sim liveness scenario,
liveness_after_a_landed_cas_answered_412.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This was referenced Oct 11, 2026
nibzard
added a commit
to nibzard/walgit
that referenced
this pull request
Oct 14, 2026
…s it uncommitted Co-Authored-By: Claude Code <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This builds on #103 and contains its commit unchanged (9fb0223). Once #103 merges, this PR is the second commit only. If you'd rather take both together, merging this PR merges #103 too.
Problem
A 412 on the manifest CAS doesn't always mean another writer won. The S3 SDK retries in standard mode and GCS retries uploads that carry a precondition. So a CAS that lands but loses its reply gets 412 from its own write. #103 re-reads the manifest before dropping the segment on the push, compaction and settings paths. Two cases are still open:
seal_pack_replacement(classification.rs) still answers a 412 withdrop_own_slot. The conserving pack lifecycle commits through it, so a landed seal answered 412 deletes the log segment the committed manifest lists. That is the same corruption Don't delete a committed log segment when the manifest CAS answers 412 #103 fixes: every later sync fails on the missing segment.log_segments.retain(|s| s.last_seq > seq)) and drops our segment descriptor. If another instance commits and checkpoints between our landed CAS and our read-back, the manifest no longer lists us. Don't delete a committed log segment when the manifest CAS answers 412 #103 then deletes the committed (folded) segment and retries. The retry re-verifies the push against refs that already hold it, so the pusher is toldConflict { expected: "(new)", actual: <its own commit> }for a push that is durable.Fix
slot_evidencereads the manifest and returns one of:Committed: our exact segment is listed (key, first seq, size, bytes).Unlisted: our key is not listed and no checkpoint is at or past our first seq, so no committed manifest ever listed it. With a 412 the request's precondition is gone for good, so the CAS provably lost.Unknown: the log was folded, our key is listed with other contents, or there is no manifest.All four 412 paths (push, compaction, settings, seal) drop the segment only on
Unlisted.Unknownkeeps it and answersCommitUnknown, as D47 asks for missing evidence. The non-412 paths are unchanged (cas_landednow wrapsslot_evidence).Round trips are unchanged: still one manifest GET on the 412 path only, plus one log GET when our descriptor is listed. The ROUNDTRIPS row from #103 now names the seal and the folded rule.
Tests (each fails without this change)
a_landed_seal_answered_412_keeps_the_commit(walgit-wal/tests/coverage.rs). On main and on Don't delete a committed log segment when the manifest CAS answers 412 #103 alone:manifest lists log/0000000000000002.pb but it is gone.a_landed_cas_folded_before_the_412_is_resolved_stays_unknown(walgit-wal/tests/wal.rs). AFaultStoregate holds the writer's read-back while a second instance pushes and checkpoints. On Don't delete a committed log segment when the manifest CAS answers 412 #103 alone the push is answeredOk([("refs/heads/main", Err(Conflict { expected: "(new)", actual: "84069d0…" }))]), which is its own commit, and its segment is deleted.only_an_unlisted_unfolded_segment_is_proven_uncommitted(unit,publish.rs) covers every evidence case.liveness_after_a_landed_cas_answered_412(sim) isliveness_after_a_lost_cas_responsewith Don't delete a committed log segment when the manifest CAS answers 412 #103'sp_cas_fail_after. On main:truth inconsistent after a landed CAS answered 412: object not found: repos/sim/r15/log/0000000000000002.pb.Checked on Linux (Rust 1.97.1, git 2.47.3, 2-CPU container, non-root). This branch alone passes fmt, clippy (
walgit-wal,walgit-store,walgit-server, all targets,-D warnings), thewalgit-walsuites and the sim lost-response scenarios.just ci(warnings, clippy, test, e2e, sim, smoke) passes on a merge of this branch with my two other open fixes (#105, #106).Same pattern, left alone here
These are safe, but worth knowing:
Createthat lands and is answered 412 by a retry looks like another writer's orphan. The writer burns its own seq after the 300 ms grace and sweeps that segment after it commits.Registry::createanswered the same way reportsAlreadyExistsfor the repository it just created.🤖 Generated with Claude Code