Conversation
The artifact plane was the only one with a retention rule and a reaper. An agent run's log streams, their segments and the routed CAS objects behind them had neither: once a run ended, its archive sat at its destination for good, and nothing in the schema could even answer whether something still needed it. Two things were missing, and the first is why the second is safe. A sealed paired-qualification result is a statement ABOUT a set of runs, expressed nowhere as a foreign key — so "is this stream still cited" was unanswerable, and an unanswerable question has to mean keep. The pin table answers it, and is written in the seal's own transaction so a committed seal always has its pins. Migration 0235 backfills the same four closures for results sealed before it existed, because a pin table that only fills forward would make every older result read as citing nothing. The reaper then reclaims only what both waits and a fail-closed citation probe have cleared, bytes first and the tombstone last. The head row deliberately outlives its bytes: a reader that finds nothing cannot tell "reclaimed by policy" from "capture lost it", so the Room reads Purged instead of meeting a stream that will not load.
`.gitignore:19` ignores every directory named `Logs/`, so the cursor — the only file in this change that actually reclaims anything — was silently skipped by `git add` and never appeared in `git status`. The build and the tests were green locally because the file was on disk; CI would have failed to compile, because the integration test references the class. `Cursors/` is also the name rule 18.3 asks for: the sub-folder is named for the variant axis, which here is which plane a cursor sweeps, not which plane the first one happens to be.
A review of the first cut found four ways the plane could look like it was working while doing nothing, or say something it had not earned. A keep wrote nothing at all, so a stream nothing can ever reclaim — one a sealed result pins, or one whose bytes another stream also names — was re-claimed by the oldest-first query on every tick. A few hundred of those deployment-wide would have filled the batch for ever while the sweep reported a healthy claim count. Every decision is now settled, including the keeps, and the claim query leaves a record alone for its recheck interval. The read path answered a policy reclamation in the vocabulary of data loss: the segment walk found no available location and reported ArtifactMissing, byte-identical to an object that vanished. Worse, the metadata kept projecting the surviving manifest receipt as an integrity proof over bytes that were gone. A purged stream now answers with its own code before the walk, and withholds the proof. The citer list was neither complete nor pinned. It is enumerated, covered by a drift detector over the EF model, and it deliberately does NOT include a transfer intent: ck_artifact_transfer_intent_outcome makes that column null on every non-terminal row, so the only rows naming an object are finished transfers — and a transfer is what wrote each segment, so probing it would have kept every log stream in the deployment for ever. Three smaller reversals of meaning: a stream whose bytes were lost by some other path is no longer tombstoned as purged; a drain that is still making progress no longer backs off like a refusal; and the guard now requires the quarantine to have been recorded by an EARLIER statement, so one sweep cannot both propose and execute a collection. The rule table lists one class, because one cursor exists. The five classes it advertised without one are gone until their cursors arrive.
ppXD
force-pushed
the
feat/a-sealed-result-pins-what-it-cites-and-a-reaper-collects-the-rest
branch
from
September 19, 2026 05:19
5e6c189 to
a69c4d5
Compare
… team Three things the first rework got wrong, each of which quietly kept bytes nobody wanted. Deduplication is per WRITE, not per stream: two byte-identical segments are one content-addressed object with a placement each, and because the object key is chosen per write both can sit at the same destination. The drain refused any object with more than one placement, so a capture that repeated itself was unreclaimable for good — and the suite could not see it, because every test wrote distinct bytes per segment. The drain now walks placements and names each one in its purge request, which is what the coordinator asks for when an object has more than one. The refusal it replaced was also misreported: "holds bytes at N destinations" was one destination, N keys. The per-tenant fair claim returns one record per team, so claiming once capped the sweep at one stream per team per tick — about a dozen a day against the two a single run writes. The loop asks again until the batch is full, skipping ids already settled this tick so a drain that is still making progress cannot spin. And the log reader's own frontend rejected the new Purged verdict as a malformed response, which is worse than the wrong-but-well-formed answer it replaced: a 410 an operator could read became "invalid log response".
This was referenced Sep 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
paired_qualification_result_pin(migration 0235) holds one row per agent run, log stream, cleanup receipt and offloaded event payload behind the observations it was computed from. 0235 backfills the same four closures for results sealed earlier, so the table is complete rather than complete-going-forward.pinned_artifact_idjoinsArtifactReferenceOracle.ReferenceSites, so the existing artifact reaper can never collect an artifact a sealed result pins.DurableRetentionPolicy), one pure decision (DurableRetentionDecision.Decide— age floor, then an independent quarantine, and every answer but a definite "nothing cites this" keeps), and a generic claim/decide/act loop (DurableRetentionReaper) over per-plane cursors. Hourly at:45viaReapExpiredDurableRecordsCommand(INonTransactionalCommand). One class is registered, because one cursor exists.Retention/Cursors/LogStreamRetentionCursor: for a terminal stream past 30 days that no pin cites and whose objects no other stream orworkflow_artifactrow also names, it purges the routed CAS bytes first and then stampsagent_run_log_stream.purged_at. The head row deliberately outlives its bytes — the read path answersPurgedwith its own code and the Room says purged, rather than either of them reporting a policy reclamation as data loss.agent_run_log_stream_guard()refuses every update to a terminal row, so a retention statement is admitted explicitly — those two columns only, revision advancing, and a purge that has no previously recordedretain_untilor that tries to be undone is refused. A statement that touches anything else reads the refusal it always did, word for word.Test plan
*artifact_object_idcolumn;Pinned_row_is_never_collected,First_unreferenced_observation_only_quarantines,Young_row_keeps_whatever_its_verdict,A_citation_clears_the_quarantine_it_contradicts. Room fold: a purged stream is not folded as integrity-verified.Retention|AgentRunLog|Qualification|Room|Reconcil|Sweep|ArtifactLocationVerifierin one invocation (23 of them new).NEW.retain_untilinstead ofOLD; a purge by object id rather than placement; the claim not looped; a listed citation site with no probe beside it.pnpm buildandpnpm lintclean (0 errors),vitest193 passed."Purged"is in theRoomAgentLogStatusunion AND in the log reader'sAgentRunLogReadAvailabilityunion + its runtimeSet(without both, the backend's 410 decoded asInvalidResponse),purgedAtrides on the stream summary, and the reader renders its own words for it.dotnet build backend/CodeSpace.sln— 0 errors, after rebase ontomain.Deploy notes
agent_run_log_stream.retain_until/purged_atare nullableADD COLUMNs with no default and no index — metadata-only, no table rewrite — and NULL is exactly what an older binary writes and reads.paired_qualification_result_pinis a new table nothing older touches.CREATE OR REPLACEofagent_run_log_stream_guard()at its own number (the function has been redefined at 0129, 0132, 0133, 0201, 0230, 0232 — one number each, whichMigrationDiscoveryTests.No_two_migrations_at_one_number_redefine_the_same_functionenforces). It only widens: every update the previous body admitted is still admitted, with its message unchanged.Corruptone, which is claimable but never deletable — is a keep, logged with the refusal's own code every sweep.Follow-ups
artifact_transfer_intentneeds a schema change first, not just a cursor: a terminal intent still holdingtemporary_object_keyis permanently unreachable, becauseArtifactCasRuntimeCoordinator.Resume.cs:~187-190gates the clearing write on a live lease that a terminal row never has. Relaxingartifact_cas_transfer_guard()(defined at 0127, 0128, 0131, 0226) is its own migration and its own negative test.agent_run_log_segment_guard()refuses every non-INSERT andagent_run_log_verificationrefuses DELETE, both pinned by a test in this PR. The bytes — which is what actually accumulates — are reclaimed without them.agent_run_log_stream (completed_at, id) WHERE purged_at IS NULL AND state <> 'Open'wantsCREATE INDEX CONCURRENTLY, which cannot appear in a transactional DbUp script).