Add explicit timeout to firehose collector fetch #SUPERLOG

[superlog-app]

Summary

The proxy's forwardFirehose function POSTs AWS Data Firehose batches to the co-located OTel collector's receiver (FIREHOSE_LOGS_COLLECTOR_URL / FIREHOSE_METRICS_COLLECTOR_URL) with no explicit timeout. When the collector is briefly unresponsive, the proxy waits up to undici's internal headersTimeout (300 s default) before propagating the error and returning 500 to Firehose. On 2026-08-09, two Firehose batches (signal=logs, both on ip-10-0-12-4) hit this path and received HeadersTimeoutError at 19:55 and 19:57 UTC before the collector recovered.

Root Cause

At apps/proxy/src/index.ts:1077, the fetch(collectorUrl, { method: 'POST', … }) call in forwardFirehose carries no signal. During a brief collector stall undici's built-in 300 s timer fires the HeadersTimeoutError, the outer catch at line 1117 returns 500 to Firehose (which is correct and retriable), but the proxy held the bufferSemaphore slot for the full stall duration—potentially blocking concurrent firehose requests on that instance.

Remediation

Adds signal: AbortSignal.timeout(FIREHOSE_COLLECTOR_TIMEOUT_MS) (default 30 000 ms, env-overridable) to the collector POST. 30 s is well inside Firehose's 60 s endpoint-response window, so the proxy will return a 500 promptly and release the semaphore, giving Firehose time to retry before it declares a delivery failure. All 132 proxy tests pass.

Incident: rusty-hedgehog (f1bff3df-30f5-4d74-8c84-0493b65b7ad2)

---

Was this PR helpful? Leave feedback — goes straight to the Superlog team.

---

Summary by cubic

Add an explicit timeout to the Firehose collector POST in the proxy. Stalled collector calls now fail within 30s instead of ~5m, freeing the semaphore and letting Firehose retry.

• Bug Fixes
  • Add AbortSignal.timeout(FIREHOSE_COLLECTOR_TIMEOUT_MS) to the collector fetch in forwardFirehose.
  • Default timeout is 30s (FIREHOSE_COLLECTOR_TIMEOUT_MS, env-overridable), staying within Firehose’s 60s response window.

^{Written for commit 96a0e65. Summary will update on new commits.}

[superlog-app]

Observability review

• 1 blocking

[superlog-app]

logs · blocking — Log the timeout signal type when the collector POST is aborted

Add a structured timedOut: true (or abortReason) field in the outer catch when err.name === 'TimeoutError' so operators can distinguish a self-imposed 30 s abort from a collector-side TCP reset or TLS error; without it, both appear as identical 500s with no way to confirm the new timeout path is firing.

Suggested change

	signal: AbortSignal.timeout(FIREHOSE_COLLECTOR_TIMEOUT_MS),
	signal: AbortSignal.timeout(FIREHOSE_COLLECTOR_TIMEOUT_MS),
	});
	// Note: if the signal fires, the catch block should check
	// err.name === 'TimeoutError' and log { timedOut: true, timeoutMs: FIREHOSE_COLLECTOR_TIMEOUT_MS }

Useful? React with 👍 / 👎.