Skip to content

feat(files): make expires_at the truth for file content (DW-992) - #2113

Open
tobydigz wants to merge 3 commits into
mainfrom
feat/dw-992/file-expiry-api
Open

tobydigz wants to merge 3 commits into
mainfrom
feat/dw-992/file-expiry-api

Conversation

@tobydigz

@tobydigz tobydigz commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Bottom layer of the DW-992 stack (single-clock content retention). Stands on its own; the retirement-rule layer follows.

Why

The API shows an expires_at on every input file, but nothing enforced it:

  • content stayed downloadable past the date, and the row was only marked expired as a side effect of being read;
  • output and error files had no expires_at at all, and kept reporting processed after their content had been dropped with a retired archive week;
  • expires_after[seconds] on upload was uncapped (and silently ignored when it didn't parse), so a file could record a date content retention would never reach;
  • a batch created from an expired input was refused by the repository and surfaced as a 500.

Content retention is moving to one creation-anchored clock for prompts and completions. That only works if the API cannot show a date the content will not reach.

What

One setting governs all three files of a batch. batches.files.default_expiry_seconds (previously an unused key defaulting to 24 h; now 30 days, which is what uploads already received) is the lifetime given to an upload without expires_after, and to a batch's output and error files measured from the batch's creation. max_expiry_seconds caps what an upload may request. Both reach the repository through a new FileExpiry setting on PostgresRequestManager.

  • Upload: expires_after[seconds] outside [min, max] (1 h to 30 d by default) or unparseable → 400, and nothing is stored (counted in dwctl_upload_rejections_total as invalid_expiry). Sent before the file part, it is refused before the file is read; sent after, the file is read first and the upload is then aborted, as with purpose. Same range as OpenAI.
  • Output and error files are created with expires_at. Ones created before this read as created_at + default, derived on read, with no backfill of existing rows.
  • GET /files/{id}/content past expires_at (or status expired) → 410, matching how deleted results files already answer.
  • GET /batches/{id}/results → 410 once the output or error file is past expires_at, alongside the existing 410 for a deleted results file. It reads both files through the repository, so older results files get the same derived date.
  • POST /batches: an input that is not processed → 400. The repository's own ValidationError on a deleted, expired or unprocessed source now maps to 400 instead of 500. There is no separate check that the input outlives the completion window; OpenAI does not make one either.
  • Expiry config (min/default/max positive and ordered) is validated whether or not batches.enabled is set. The request manager is built from these keys on every instance, so a misordered config is now a config error instead of a panic at startup.
  • config.yaml and docs/src/reference/configuration.md document the three keys: the 30-day default, the upload range, the 400 and 410 responses, and when results files expire.

Output and error files expire default_expiry_seconds after the batch is created. OpenAI counts from when the batch completes, so ours can expire up to one completion window earlier. The docs say so.

Not changed: files.status is still only written on read (check_and_mark_expired); nothing deletes content here. That is the next layer.

Tests

Ten new tests, all against Postgres:

  • Repository: results files carry the batch's clock; legacy results files derive the default; an upload beyond max is refused and leaves no row.
  • Uploads: the default lifetime, using a non-default default_expiry_seconds so the test shows the setting reaches fusillade; range rejection, including the bounds.
  • Downloads and results: 410 on download past expires_at. 410 from /batches/{id}/results when either results file has a past date, and when the derived date has passed. Both paginated and streaming paths are covered.
  • Batch creation: results files show created_at + default under a non-default config. An expired input gets 400. That test runs the repository's ValidationError → 400 mapping.
  • Config: a misordered or non-positive expiry is rejected with batches and the daemon disabled.

Full just test rust: 4,296 passed.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Deploying control-layer with  Cloudflare Pages  Cloudflare Pages

Latest commit: 97cfae9
Status: ✅  Deploy successful!
Preview URL: https://10c8590e.control-layer.pages.dev
Branch Preview URL: https://feat-dw-992-file-expiry-api.control-layer.pages.dev

View logs

@tobydigz tobydigz changed the title feat/dw 992/file expiry api feat(files): make expires_at the truth for file content (DW-992) Oct 23, 2026
@tobydigz
tobydigz added this pull request to stack #2116 October 23, 2026 12:33
@tobydigz
tobydigz marked this pull request as ready for review October 26, 2026 12:47
@cubic-dev-ai

cubic-dev-ai Bot commented Oct 26, 2026

Copy link
Copy Markdown

Running ultrareview automatically — this PR appears to change a public API contract, where a missed bug can break the clients and integrations that depend on it, so a deeper multi-pass review is worth running. I'll post findings when complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ultrareview ran automatically — this PR appears to change a public API contract, where a missed bug can break the clients and integrations that depend on it, so a deeper multi-pass review is worth running.

All reported issues were addressed across 10 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread dwctl/src/api/handlers/files.rs Resolved
Comment thread docs/src/reference/configuration.md Resolved
Comment thread docs/src/reference/configuration.md Resolved
Comment thread dwctl/src/config.rs Resolved

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 2 files (changes from recent commits).

Requires human review: Auto-approval blocked by 1 unresolved issue from previous reviews.

View guided diff | Turn on auto-fix | Re-trigger cubic

The API shows an expires_at on every input file, but nothing enforced it:
content stayed downloadable past the date, output and error files never
had one, and an upload could ask for any expires_after at all, recording
a date that content retention would not honour. Batch creation then
turned an expired input into a 500 from the repository's check.

One setting now governs the three: batches.files.default_expiry_seconds
(30 days, what uploads already received) is the lifetime given to an
upload without expires_after and to a batch's results files from the
batch's creation; max_expiry_seconds caps expires_after. Results files
created before this read as created_at + default rather than being
backfilled. Downloads past expires_at answer 410, batch creation refuses
an input that would expire inside the completion window with 400, and
an out-of-range expires_after is a 400 before any bytes are read.

Groundwork for retiring content weeks on a single creation-anchored
clock: once the API cannot show a date content will not reach, the
retirement rules no longer need per-file state.
…e (DW-992)

/batches/{id}/results kept streaming bodies after the output and error
files had passed expires_at, so expiry was only enforced on
/files/{id}/content. It now reads both results files through the
repository, which supplies the date older results files are given, and
answers 410 once either has expired.

The request manager is built from the expiry keys on every instance and
refuses a misordered or non-positive expiry, but config validation of
those keys only ran with batches enabled. With batches disabled a bad
value panicked at startup; it is now a config error.

Drop the create_batch check that refused an input expiring inside the
completion window. OpenAI does not make it, and the repository already
refuses deleted, expired and unprocessed sources, now mapped to 400.

Tests build the app with a non-default default_expiry_seconds so they
show the setting reaches fusillade, rather than matching its fallback.
The configuration reference describes the 30-day default, the upload
range, the 400 and 410 responses, and that results files expire from
batch creation rather than completion.
File metadata has no status field, so it cannot report 'expired';
invalid_expiry is a metric label, not an API error code; and the
default expiry is not the longest content lifetime, since uploads may
request up to max_expiry_seconds.
@tobydigz
tobydigz force-pushed the feat/dw-992/file-expiry-api branch from 0df3bcb to 97cfae9 Compare October 26, 2026 14:01

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant