fix(otel): alert on Cloud delivery failures only while they are happening

[dviejokfs]

Problem

The project Telemetry storage page showed a red alert whenever any dead-lettered span existed:

N spans were never delivered to Temps Cloud … Credential rejected by the backend — re-enroll this instance

Dead letters are a permanent record and never change state. So once an instance recovered (credential re-issued, Cloud outage over), the alert stayed red and kept telling a healthy instance to re-enroll, with no way to clear it. Operators learn to ignore red banners, and "re-enroll" is the wrong instruction once the link works.

What changes

The alert only shows while delivery is failing now. "Failing now" comes from spans that failed at least one attempt and are still pending (state = 'pending' AND last_error IS NOT NULL). That set appears on the first refused attempt, hours before anything is dead-lettered, and it empties on its own once Cloud accepts again. Nothing needs dismissing.

The alert names the fix only when there is one, and links to Temps Cloud settings:

Link state while spans are failing	Alert says
Credential rejected	Re-enroll the instance; retrying spans are delivered once it is accepted
Instance unlinked	Spans are held here, not retried; link again to deliver them
Telemetry export off	Spans are held here, not retried; turn export on
Linked and healthy (e.g. Cloud 5xx)	The failure and its reason; no invented fix

Past losses become history, not an alert. Dead letters are grouped by span time into separate gaps: a new gap starts after DELIVERY_GAP_SEPARATION (1 h) without one. They are listed under Storage history → Spans never delivered to Temps Cloud with the time range, the count, when retries ran out, and why. Two outages days apart are two rows, not one range claiming the days in between were lost.

API (GET /otel/cloud-telemetry/projects/{project_id}), additive only

• delivery_failing, retrying_spans, delivery_failing_since, delivery_failure_error, delivery_failure_action, delivery_failure_setup_path
• delivery_gaps: CloudDeliveryGapResponse[] (first_span_at, last_span_at, undelivered_spans, gave_up_at, last_error)
• The existing dead_lettered_spans / last_dead_letter_* fields are unchanged, for older clients.

CLI parity

bunx @temps-sdk/cli cloud telemetry write-mode get prints the same split: a warning only while failing, and a "Spans never delivered to Temps Cloud" history section. It uses ?? [] so it doesn't break against an older server.

Load / efficiency

Control-plane only: one settings-page read, nothing on the ingest path.

• delivery_failure_for_project is a filtered count over this project's pending rows (the pending set is bounded by the outbox byte cap).
• delivery_gaps_for_project walks idx_cloud_telemetry_outbox_entity_project in enqueued_at order, so both window functions run without a sort. Gaps are hash-aggregated (a handful of rows), and each gap's reason is one index probe (at most 100). Measured at 520,300 dead letters for one project: ~280 ms end-to-end, 32 kB aggregate memory, no full sort. An earlier version with an ordered ARRAY_AGG sorted all 520k rows (69 MB, ~600 ms); that was replaced before this PR.

Evidence

Tests

• crates/temps-otel/tests/cloud_primary_write_mode_test.rs::past_delivery_failures_are_history_and_only_retrying_spans_mean_failing_now runs against real Postgres (not skipped). It covers:
  • Two outages three days apart come back as 2 gaps, newest first, each with its own reason.
  • A queued span that was never refused is not "failing".
  • Another project's dead letters don't leak.
  • A retrying span makes delivery failing; once it's delivered, failing clears on its own.

  test result: ok. 27 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 42.35s
  

• cloud_telemetry_handler unit tests cover the four link states in the table above:

  temps-otel lib:            test result: ok. 801 passed; 0 failed
  temps-cloud-client lib:    test result: ok. 147 passed; 0 failed
  

• apps/temps-cli describeDelivery tests (recovered → no alert + history; failing → alert + fix; no fix → none invented): 45 pass, 0 fail across src/commands/cloud/.
• Playwright web/e2e/authenticated/cloud-delivery-failure.spec.ts (recovered / failing / never-failed), run against a local server built from this branch:

  4 passed (6.4s)
  

• cargo clippy -p temps-cloud-client -p temps-otel --all-targets -- -D warnings: clean. cargo fmt --check: clean. Web tsc --noEmit, CLI tsc --noEmit, spec:check, source_attribution.py check: clean.

Live, against a local server built from this branch

Seeded a project's outbox with 520k dead letters from three outages, and later 1,200 retrying spans.

Recovered (nothing retrying):

delivery_failing = False
retrying_spans = 0
 gap {first_span_at: 2026-09-24T05:47:31Z, last_span_at: 2026-09-24T11:20:50Z, undelivered_spans: 20000, gave_up_at: 2026-09-24T13:47:30Z, last_error: 'Credential rejected by the backend — re-enroll this instance'}
 gap {first_span_at: 2026-09-17T13:47:31Z, last_span_at: 2026-09-17T13:52:30Z, undelivered_spans: 300,   gave_up_at: 2026-09-17T17:47:30Z, last_error: 'Backend returned 503'}

With spans retrying:

delivery_failing = True
retrying_spans = 1200
delivery_failing_since = 2026-09-26T13:32:35Z
delivery_failure_error = Credential rejected by the backend — re-enroll this instance

CLI against the same server (recovered):

Spans never delivered to Temps Cloud (newest first)
ℹ   9/24/2026, 7:47:31 AM → 9/24/2026, 1:20:50 PM: 20,000 span(s) never delivered (gave up 9/24/2026, 3:47:30 PM)
ℹ     Credential rejected by the backend — re-enroll this instance
ℹ   9/23/2026, 3:47:38 PM → 9/23/2026, 9:20:58 PM: 500,000 span(s) never delivered (gave up 9/23/2026, 7:47:38 PM)
ℹ   9/17/2026, 3:47:31 PM → 9/17/2026, 3:52:30 PM: 300 span(s) never delivered (gave up 9/17/2026, 7:47:30 PM)
ℹ     Backend returned 503

I checked the Playwright run's screenshots by eye (they're written to /tmp/cloud-delivery-{failing,recovered}.png; not attached here). Failing: red alert with count, reason, and an Open Temps Cloud settings link. Recovered: no alert; the gap is a neutral row under Storage history.

Not verified live

• The unlinked / export-off wording is unit-tested only. My hand-made outbox rows used placeholder payloads, and the worker correctly dead-lettered them as undecodable before they could be observed as "held".

Notes

• apps/temps-cli/src/api/sdk.gen.ts gains a 3-line doc comment on updateSettings. That's pre-existing generator drift against the committed spec on main, not part of this change's API.

Review follow-up (bcb4a40)

• Web polls every 15 s while spans are failing or queued (60 s otherwise), so the alert clears by itself. The draft-reset effect now depends only on the saved values; otherwise every poll would wipe unsaved edits. The new e2e test fails if the old dependency is restored.
• CLI against older servers falls back to their dead-letter total and reason.
• Gap gave_up_at and last_error come from the same row.
• The current failure reason comes from the front of the FIFO queue (the most recently attempted rows), not the highest id.
• delivery_gaps_truncated is set past 100 gaps; web and CLI then state how many older spans are not listed.

Evidence after the follow-up: integration tests 2 passed (delivery + dead-letter), temps-otel lib 802 passed, temps-cloud-client lib 147 passed, CLI 47 pass, Playwright 6 passed, clippy --all-targets -D warnings clean.

[github-actions]

📓 Changelog preview

This is what your commits will add to the generated CHANGELOG.md at release time (via git-cliff). Do not edit CHANGELOG.md by hand — it is generated from your Conventional Commit messages.

## [Unreleased]

### Fixed

- **otel:** Alert on Cloud delivery failures only while they are happening
- **otel:** Keep delivery alert and history accurate after review

[greptile-apps]

Confidence Score: 5/5

[Medium risk] Adds telemetry delivery failure tracking to the cloud settings UI.

The PR appears safe to merge based on the reviewed changes and resolved previous findings.

Summary

The PR separates active Cloud delivery failures from permanent loss history in the API, web console, and CLI. The follow-up adds polling, legacy-server fallback, consistent gap metadata, and an explicit truncation indicator.

_{Reviews (2) · Last reviewed commit: "fix(otel): keep delivery alert and histo..."}

[dviejokfs]

@greptileai review