[tobydigz]

Second layer of the DW-992 stack, on top of #2113. This is the change that replaces the retention design; the first layer made the API honest about expires_at and validates the file expiry settings everywhere.

Why

Three loops ran in sequence to delete batch content, and each had a failure mode of its own:

• An archive week waited 28 days after the last batch in it froze. A single late retry could hold every other batch's completions in the week far beyond the retention period.
• Retiring an archive week stamped retention_expired_at on every batch in the week, in one statement inside the drop transaction. A large week can exceed the statement timeout; the transaction rolls back, drop included, and the engine retries on every tick, so that week never completes and every later week queues behind it.
• That stamp was the only thing that released input files, which an hourly sweep then re-examined in bulk to expire almost none; only expired files released a template week. The sweep was also the files writer that stalled uploads behind weekly partition attach (DW-950).

Retention was also configured separately from the expiry the API shows, so the two could disagree and content could go before the date a file reported.

What

One clock, one period

Both weekly families (batch archive and request templates) drop a partition for one reason: the week is past the retention period R, measured from the week itself, and no batch created in it (archive) or reading input uploaded in it (templates) is still running.

R is derived from batches.files.max_expiry_seconds, rounded up to whole days. Every file's expires_at is at most created_at + max_expiry_seconds, which is at most the week's end plus R, so the date the API shows can never outrun the content. There is no separate retention setting and nothing left to warn about.

• batch_archive_retirement.rs: candidate anchored on the week. completion_sql now stamps retention_expired_at only on the week's explicitly deleted batches, read through idx_batches_explicitly_deleted, so they leave the erasure purge's tombstone set once their archived rows are dropped. Finished batches are not written, so the statement stays small however large the week.
• template_retirement.rs: candidate anchored on the week with the unfinished-batch guard; the two per-file scans removed. A dedicated (file-less) template row still fails the gate closed. The guard also bounds batches.created_at from the week start (a batch is always created after its input), so it range-scans the new index.
• The guard reads batches (created_at) WHERE counts_frozen_at IS NULL AND deleted_at IS NULL, a new partial index that holds only the batches still running. Three-file concurrent migration (create / reindex / validate) with down files and checksum entries.
• The file-content sweep (expire_file_content, FileExpiryCursor/FileExpiryStep, the daemon's hourly phase and constants) is deleted across fusillade-core, fusillade-arsenal and fusillade. Its metric, fusillade_file_content_expired_total, is removed; any dashboard or alert on it will flat-line and should be dropped.
• Batch creation and retry additionally require the input's template week to be active, so a request is never populated from content that is gone.
• The retirement enable flags are unchanged.

Deprecated settings

batch_daemon.batch_archive_retention_days and batch_daemon.template_retention_days are deprecated and ignored. They still parse so existing values files keep loading, and dwctl logs a warning at startup when either is set. Operators should remove them. For a deployment that set them, the effective retention period may change to the one derived from max_expiry_seconds.

In the fusillade crate, RetentionMaintenanceConfig takes one with_content_retention_days in place of the two per-family setters.

Nothing reads batches.retention_expired_at or files.retention_expired_at any more except the orphan purge, which uses them to tell explicit deletes apart. Archive retirement still sets the batch column on deleted batches (see above), and the columns stay.

Retry age cut-off

A retry makes a batch unfinished again, and an unfinished batch holds its weeks open. To keep that hold short and predictable, a batch cannot be retried after a certain age. While content retirement is enabled, retry (whole batch, or selected requests) is refused with a 400 once now > created_at + R - 1 day, counted from the batch's creation and from its input file's upload, whichever is earlier (a retry re-reads the input templates).

A week drops no earlier than week_start + 7 days + R, and everything in it was created at or after week_start, so the one-day margin always puts the cut-off before the partition can drop, never after.

Races between writers and retirement

• Batch creation and both retry paths read the input's request_template_buckets row FOR SHARE (alongside the existing FOR UPDATE on the file).
• Retrying an archived or split batch reads its own batch_archive_buckets row FOR SHARE (as archive_batch does) and is refused unless the week is active. Without this, the move-back would read a dropped partition, find nothing, and reset the batch over completions that are gone.
• The shared engine's claim() now re-runs the family's unfinished-batch guard (FamilySpec::recheck_sql) in a fresh statement after locking the bucket row FOR UPDATE and before fencing it. Under READ COMMITTED that statement sees a batch committed while the claim waited for the lock; if one is found the claim backs off. A writer that arrives after the fence waits for it and then reads the week as no longer active.
• The archive guard keys on the week a batch was created in, while archived rows live in archive_bucket. These differ only for a batch archived into a later week because its creation week was already fenced; the retry archive-week check and the age cut-off bound that case (documented in batch_archive_retirement.rs).

Rollout notes

• Once live, every week past R becomes eligible at once. The engine retires one week per tick and re-ticks after 1 s; with no per-row work each drop is sub-second, so the backlog clears one week per tick shortly after deploy. No paced backfill is needed.
• A week that previously failed to complete because its completion statement timed out completes on the next attempt (there is no completion statement any more).
• Remove batch_archive_retention_days and template_retention_days from values files; until then each logs a deprecation warning at startup. Check that max_expiry_seconds is the retention you want, since it now sets it.
• Retries of batches older than the cut-off now return 400 instead of being accepted.

Tests

• Archive family: retires without touching batch metadata; a late finish no longer holds the week; a young week is not a candidate; only an unfinished batch created in the week blocks (live/split/other-week do not); a batch committed while the claim waits on the bucket lock stops the fence (fails without the re-check).
• Template family: retires on age unless an unfinished batch reads it, and the file row is untouched; a young week is not a candidate; an unfinished batch on input from another week does not block; the unowned-row gate is unchanged.
• Refusals: batch creation, batch retry and request retry each refuse with a validation error when the input week is retiring or retired, and succeed when it is active; an archived batch's retry is refused when its archive week is retiring or retired, without resetting the batch.
• Retry age: just inside the cut-off is accepted and just past it is refused, from the batch's creation and from the input file's upload.
• Config: retention rounds max_expiry_seconds up to whole days, and the deprecated keys still parse.
• Daemon: the expiry-walk test is removed with the phase. Full just test rust passes.

[cloudflare-workers-and-pages]

Deploying control-layer with    Cloudflare Pages

Latest commit:	c32232d
Status:	✅  Deploy successful!
Preview URL:	https://85a612c4.control-layer.pages.dev
Branch Preview URL:	https://feat-dw-992-single-clock-ret.control-layer.pages.dev

View logs

[cubic-dev-ai]

Running ultrareview automatically — this PR appears to change concurrency-sensitive code such as locks, queues, or retries, where a missed race may only surface under production load, so a deeper multi-pass review is worth running. I'll post findings when complete.

[cubic-dev-ai]

Ultrareview ran automatically — this PR appears to change concurrency-sensitive code such as locks, queues, or retries, where a missed race may only surface under production load, so a deeper multi-pass review is worth running.

All reported issues were addressed across 26 files

<sub>Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 4 reviewed files. 1 file intentionally excluded from review.

<sub>Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>

[cubic-dev-ai]

0 issues found across 2 files (changes from recent commits).

<sub>Requires human review: Retention redesign: one creation-anchored clock from max_expiry_seconds, partition retirement, retry cut-off, new index; drops content and changes retry API. Requires human sign-off for data deletion and request-path behavior change.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>