Skip to content

Keep a 412'd log segment unless the manifest proves it uncommitted - #104

Open
jilio wants to merge 2 commits into
tobi:mainfrom
jilio:fix/landed-cas-412
Open

jilio wants to merge 2 commits into
tobi:mainfrom
jilio:fix/landed-cas-412

Conversation

@jilio

@jilio jilio commented Oct 11, 2026 •

Copy link
Copy Markdown

This builds on #103 and contains its commit unchanged (9fb0223). Once #103 merges, this PR is the second commit only. If you'd rather take both together, merging this PR merges #103 too.

Problem

A 412 on the manifest CAS doesn't always mean another writer won. The S3 SDK retries in standard mode and GCS retries uploads that carry a precondition. So a CAS that lands but loses its reply gets 412 from its own write. #103 re-reads the manifest before dropping the segment on the push, compaction and settings paths. Two cases are still open:

  1. The seal still deletes on any 412. seal_pack_replacement (classification.rs) still answers a 412 with drop_own_slot. The conserving pack lifecycle commits through it, so a landed seal answered 412 deletes the log segment the committed manifest lists. That is the same corruption Don't delete a committed log segment when the manifest CAS answers 412 #103 fixes: every later sync fails on the missing segment.
  2. "Not listed" is not proof. A checkpoint folds the log at or past our seq (log_segments.retain(|s| s.last_seq > seq)) and drops our segment descriptor. If another instance commits and checkpoints between our landed CAS and our read-back, the manifest no longer lists us. Don't delete a committed log segment when the manifest CAS answers 412 #103 then deletes the committed (folded) segment and retries. The retry re-verifies the push against refs that already hold it, so the pusher is told Conflict { expected: "(new)", actual: <its own commit> } for a push that is durable.

Fix

slot_evidence reads the manifest and returns one of:

  • Committed: our exact segment is listed (key, first seq, size, bytes).
  • Unlisted: our key is not listed and no checkpoint is at or past our first seq, so no committed manifest ever listed it. With a 412 the request's precondition is gone for good, so the CAS provably lost.
  • Unknown: the log was folded, our key is listed with other contents, or there is no manifest.

All four 412 paths (push, compaction, settings, seal) drop the segment only on Unlisted. Unknown keeps it and answers CommitUnknown, as D47 asks for missing evidence. The non-412 paths are unchanged (cas_landed now wraps slot_evidence).

Round trips are unchanged: still one manifest GET on the 412 path only, plus one log GET when our descriptor is listed. The ROUNDTRIPS row from #103 now names the seal and the folded rule.

Tests (each fails without this change)

Checked on Linux (Rust 1.97.1, git 2.47.3, 2-CPU container, non-root). This branch alone passes fmt, clippy (walgit-wal, walgit-store, walgit-server, all targets, -D warnings), the walgit-wal suites and the sim lost-response scenarios. just ci (warnings, clippy, test, e2e, sim, smoke) passes on a merge of this branch with my two other open fixes (#105, #106).

Same pattern, left alone here

These are safe, but worth knowing:

  • A log-slot Create that lands and is answered 412 by a retry looks like another writer's orphan. The writer burns its own seq after the 300 ms grace and sweeps that segment after it commits.
  • A lease acquire answered the same way waits out the TTL.
  • Registry::create answered the same way reports AlreadyExists for the repository it just created.

🤖 Generated with Claude Code

0bserver07 and others added 2 commits October 1, 2026 13:45
The S3 SDK retries failed calls three times by default and the GCS
client retries uploads that carry a precondition, the manifest CAS
included. When a CAS lands and its reply is lost, the retry gets a 412
from our own write. process_batch, publish_compact_classified and
publish_settings_impl all read any 412 as a lost race and delete their
log segment, which the committed manifest now lists: the writer reports
a failure for a durable change and every reader then fails on the
missing segment.

Run the existing cas_landed check on a 412 too before dropping the
segment, and keep an unknown outcome unknown when that check fails.
The fault store gains the missing fault kind (applied, then 412),
sharing the existing CAS roll so seeded simulations keep their
sequences, and a test drives a push, a compaction and a settings change
through it. One extra manifest GET on the 412 path only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A 412 on the manifest CAS can answer a transport retry of our own landed
write. The previous commit looks before dropping the segment, but treats
"not listed" as "never committed", and the pack seal still drops on any
412:

- seal_pack_replacement answered a landed 412 by deleting the segment the
  committed manifest lists: the same corruption, on the maintainer's
  retirement path.
- A checkpoint that folds the log at or past our seq drops our segment
  descriptor. If one lands between our CAS and the read-back, "not listed"
  proves nothing: the writer deleted its committed (folded) segment and
  re-verified its own push into a conflict against its own commit.

slot_evidence now classifies the fresh manifest as Committed (our exact
segment is listed), Unlisted (our key is not listed and nothing is folded
at or past our seq, so no committed manifest ever listed it; with a 412
no copy of our request can land later) or Unknown (folded, our key listed
with other contents, or no manifest). Every 412 path - push, compact,
settings and seal - drops the segment only on Unlisted; Unknown keeps it
and reports CommitUnknown. The non-412 paths are unchanged (cas_landed).

Round trips: unchanged. The 412 path still costs one manifest GET (plus
one log GET when our descriptor is listed); successful publishes do not
change.

Tests: a unit test for every evidence case; a_landed_seal_answered_412_
keeps_the_commit (fails before: "manifest lists log/...2.pb but it is
gone"); a_landed_cas_folded_before_the_412_is_resolved_stays_unknown
(fails before: the push is answered Conflict{expected: (new), actual:
<its own commit>}); and a sim liveness scenario,
liveness_after_a_landed_cas_answered_412.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
nibzard added a commit to nibzard/walgit that referenced this pull request Oct 14, 2026
…s it uncommitted

Co-Authored-By: Claude Code <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants