[Crown421]

Closes DW-994. Follow-up to DW-985 (#2082, #2083).

Compressed request bodies and compressed batch uploads shipped with good coverage for realtime refusals: unsupported_content_encoding, body_decode_failed and request_too_large go into onwards_rejections_total, and the gateway rejection alerts pick them up. Two failure modes from that work had no signal. This PR adds one for each.

Gap 1: file-upload refusals weren't counted by reason

Batch uploads don't go through the inference body layer. upload_file refuses them through dwctl's own Error: a corrupt .jsonl.gz gets a 400, an xz/bzip2/zip/lz4 file gets a 415, and so does a body that isn't multipart/form-data. None of these reach onwards_rejections_total, and dwctl_http_requests_total only gives the status, so they were mixed in with every other 400 and 415 on the files route.

New counter: dwctl_file_upload_rejections_total{code}. It covers every FileUploadError variant, plus the refusals upload_file makes before it reads the file:

code	status	cause
not_multipart	415	body isn't multipart/form-data
upload_capacity	429	the upload queue is full, or no upload slot came free in time
batch_disabled	403	the batch API is switched off for the account
invalid_multipart	400	multipart content type, but the form can't be read (e.g. no boundary)
stream_interrupted	400	multipart stream broke mid-upload
file_too_large	413	over limits.files.max_file_size, from either Content-Length or the stream
too_many_lines	400	over max_requests_per_file
invalid_json / invalid_utf8	400	a bad line
invalid_compression	400	starts as gzip/zstd but is corrupt or truncated
unsupported_compression	415	xz, bzip2, zip, lz4
no_file / empty_file / invalid_purpose	400	form contents
model_access_denied	403	a line names a model the caller can't use
validation_error / unprocessable_line	400 / 422	per-line validation
image_unfetchable	422	an image origin refused access
image_transient / image_store_failed / image_access_unavailable / image_fetch_failed	503	image normalisation failed on our side or upstream

Every code is registered at 0 when the Prometheus recorder is installed, so the first refusal of a kind is a 0 → 1 step that increase() can see.

Why a dedicated counter rather than rejections::refuse. I looked at recording these through the existing rejection machinery so they'd land in onwards_rejections_total. It doesn't fit well:

• The files route is outside the onwards router, so the Recorder extension isn't installed there. refuse also works on a Response, but the handler returns dwctl's Error.
• The labels don't apply. model would always be empty, because one upload spans many models. traffic only distinguishes realtime from dispatched inference, so an upload would be labelled realtime inference traffic.
• onwards_rejections_total counts client errors. Several upload failures are 503s caused by our image store or an image origin. Mixing them in would make a server problem look like a client one.
• GatewayRejectionNew and GatewayRejectionSpike are tuned for inference traffic. A separate series lets upload alerts get their own thresholds.

The cost is that the existing dashboard and alerts don't pick the new series up on their own, so they need adding there.

Gap 2: decode-slot waits weren't measured

Decoding a compressed realtime body takes one of DECODE_SLOTS, one per CPU. Requests over that bound wait rather than fail, so if the slots saturate (a burst of compressed traffic, or abuse, since decoding runs before authentication) compressed requests just get slower, and nothing measured that.

in_slot now records:

• dwctl_inference_decode_slot_wait_seconds (histogram): time from asking for a slot to getting one. Buckets are 1 ms to 5 s with an edge at 0.25 s, so a 250 ms threshold reads an exact bucket. A free slot is taken at once, so almost every observation lands in the first bucket.
• dwctl_inference_decodes_waiting (gauge): decodes waiting for a slot right now. It's set to 0 at startup. A drop guard decrements it, so a request dropped while waiting leaves the gauge and records no wait.

Behaviour is unchanged: requests still wait and are never refused, and the permit still moves into the blocking task.

Tests

• test_every_upload_refusal_is_counted_by_its_code: builds every FileUploadError variant, refuses it under a local recorder, and checks it's counted once under its own code, and that the code is in the zero-registered list.
• test_upload_rejection_codes_start_at_zero: every code renders at 0 after initialisation.
• test_upload_capacity_refusals_are_counted: holds the only upload slot and checks the refused upload's 429 is counted as upload_capacity.
• The existing end-to-end tests for a truncated gzip, an xz file, non-multipart bodies, a broken multipart form and an org with batch disabled now also check that each refusal moves its counter.
• records_how_long_a_decode_waited_for_its_slot: uses a paused tokio clock, so it's deterministic with no sleeps. It holds the only slot, checks the gauge reads 1, advances the clock 300 ms, releases the slot, and checks the wait lands between the 0.25 and 0.5 buckets. A decode that found a free slot lands in the first bucket.
• a_request_dropped_while_waiting_leaves_the_queue: dropping a waiting decode takes the gauge back to 0 and records no duration.

just lint rust and just test rust pass.

[cubic-dev-ai]

All reported issues were addressed across 3 files

<sub>Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

[cloudflare-workers-and-pages]

Deploying control-layer with    Cloudflare Pages

Latest commit:	64924fc
Status:	✅  Deploy successful!
Preview URL:	https://a07e7907.control-layer.pages.dev
Branch Preview URL:	https://steffen-dw-994-make-compress.control-layer.pages.dev

View logs

[cubic-dev-ai]

0 issues found across 3 files (changes from recent commits).

<sub>Auto-approved: Adds per-code upload-refusal counters and decode-slot wait histogram/gauge; observability-only, no request-handling behavior change, codes zero-registered and tests cover the counting.
Tip: Get a Slack DM when cubic finishes reviewing your PRs. Connect Slack

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>