HookRelay is a multi-tenant webhook delivery platform built to make failure behavior explicit. An API transaction stores an event, its delivery, and a Kafka outbox record together; workers then deliver signed requests with durable retries, leases, distributed rate limits, dead letters, and controlled replay.
The guarantee is at least once, not exactly once. The system prevents silent loss across its own PostgreSQL/Kafka boundary, but a receiver can process a request immediately before a worker crashes. Receivers therefore must deduplicate on X-HookRelay-Delivery-ID.
flowchart LR
Client -->|API key + event| API
API -->|one transaction| PG[(PostgreSQL)]
PG --> Outbox[Outbox publishers]
Outbox -->|endpoint ID key| Kafka[(Kafka)]
Kafka --> Workers[Delivery workers]
Workers --> Redis[(Redis token buckets)]
Workers -->|signed HTTP| Receiver
Workers -->|attempt + state transaction| PG
Scheduler[Retry scheduler / lease reclaimer] --> PG
PG -->|retry outbox| Outbox
| Component | Responsibility |
|---|---|
| API | Authentication, endpoint management, idempotent ingestion, operations API, health and metrics |
| Outbox publisher | FOR UPDATE SKIP LOCKED, synchronous Kafka acknowledgement, then published_at |
| Delivery worker | Atomic lease claim, Redis limits, HMAC request, bounded HTTP, durable attempt, then offset commit |
| Retry scheduler | Poll durable retry times, reclaim expired leases, dead-letter expired deliveries, create retry outbox rows |
| Mock receiver | Verify signatures; emulate 500, 429, delay, disconnect and fail-first-N; report duplicate delivery IDs |
Requirements: Docker with Compose, Go 1.26+, curl, and openssl.
make setup
make up
make demomake setup generates a local 256-bit master key in ignored .env. make up builds and starts PostgreSQL, Kafka, Redis, the API, two publishers, two workers, the scheduler, and receiver. make demo creates a tenant, registers the receiver through the API, demonstrates a successful retry, creates a dead letter, and replays it after recovery.
Stop and remove local volumes with make down.
Create a tenant key (the plaintext is printed once and never stored):
API_KEY=$(make seed | sed -n 's/^ API key //p')Register an endpoint. HTTP/private destinations are accepted only because the Compose environment explicitly enables both development switches.
curl -X POST http://localhost:8080/v1/endpoints \
-H "Authorization: Bearer $API_KEY" \
-H 'Content-Type: application/json' \
-d '{"url":"http://mock-receiver:9000/webhook","max_attempts":5,"max_retry_seconds":3600}'The signing secret is returned on creation (or rotation) only. Submit an event:
curl -X POST http://localhost:8080/v1/events \
-H "Authorization: Bearer $API_KEY" \
-H 'Content-Type: application/json' \
-H 'Idempotency-Key: order-123-paid' \
-d '{"endpoint_id":"<uuid>","event_type":"order.paid","payload":{"order_id":"123"}}'Repeating identical endpoint/type/payload bytes with that key returns HTTP 200 and the original event. Reusing the key with changed content returns HTTP 409. New events return HTTP 202 only after the event, delivery and outbox rows commit.
Operations routes are JSON-only and tenant-scoped:
GET /v1/events GET /v1/events/{id}
GET /v1/deliveries GET /v1/deliveries/{id}
GET /v1/deliveries/{id}/attempts GET /v1/dead-letters
POST /v1/dead-letters/{id}/replay
Lists accept limit/offset; deliveries accept state and event_id; events accept endpoint_id.
HookRelay sends these headers:
X-HookRelay-Event-ID: <UUID>
X-HookRelay-Delivery-ID: <UUID>
X-HookRelay-Timestamp: <base-10 Unix seconds>
X-HookRelay-Signature: v1=<lowercase hex HMAC-SHA256>
X-HookRelay-Attempt: <base-10 integer>
The exact signed byte sequence is the UTF-8 decimal timestamp, one ASCII period (0x2e), then the raw request-body bytes:
timestamp + "." + raw_request_body
Do not parse and re-encode JSON before verification. Compute HMAC-SHA256 using the endpoint secret, decode the supplied hex, compare in constant time, reject timestamps outside a short tolerance, and deduplicate the delivery ID. Go verification code is in internal/signing/signing.go; the receiver uses that same public contract.
Delivery states are queued, delivering, retry_scheduled, succeeded, dead_letter, and cancelled.
Each Kafka message is keyed by endpoint ID. A worker atomically changes an eligible delivery to delivering, increments its attempt, and installs lease_owner/lease_expires_at. Duplicate Kafka records cannot acquire that claim and make no HTTP request. HTTP 408, 429, 5xx, timeouts and network errors retry with exponential full jitter; Retry-After is honored. Retry times live in PostgreSQL, not sleeping goroutines.
The scheduler claims due rows with FOR UPDATE SKIP LOCKED, changes them to queued, and inserts the uniquely keyed retry outbox row in the same transaction. Redis failure also reschedules: workers never silently bypass the limit.
Duplicates can still occur in three important windows:
- Kafka acknowledges an outbox publish, but the publisher crashes before committing
published_at. The row is republished; the delivery claim normally prevents a second HTTP request. - A receiver processes the webhook, but the worker crashes before recording success. The lease expires and the same delivery ID is sent again.
- A live but slow worker exceeds its lease. Reclamation can let a replacement send while the original request remains in flight. The old worker is fenced from updating state, but the receiver may see both requests.
The platform therefore makes an at-least-once guarantee. Receivers should store delivery IDs under a uniqueness constraint before applying side effects and return success for an already-seen ID.
Tenant and endpoint limits are token buckets implemented by one atomic Redis Lua script. Two worker processes share the same keys and therefore the same budget. The tenant bucket is consumed first; if the endpoint bucket rejects, that tenant token is intentionally not refunded. This conservative over-count avoids a non-atomic compensation round trip.
- API keys contain 256 random secret bits. Only SHA-256 hashes are stored; authentication errors do not reveal whether a key exists.
- Endpoint signing secrets are encrypted with AES-256-GCM under
HOOKRELAY_MASTER_KEY, returned only on creation/rotation, and decrypted only for a send. - Production accepts HTTPS only, rejects embedded credentials/fragments, bounds redirects and revalidates each redirect target.
- Production DNS results are blocked for loopback, RFC1918/private, link-local, carrier-grade NAT, multicast, reserved and cloud-metadata ranges. Development requires explicit switches for HTTP/private Docker destinations.
- Connect and whole-request timeouts are separate. Request and response bodies are bounded; response control characters and invalid UTF-8 are sanitized before storage.
- HMAC timestamp tolerance limits replay usefulness, but receiver-side delivery-ID deduplication remains required if a signing secret leaks.
DNS rebinding protection is out of scope. See Known limitations.
The migration in migrations/0001_init.up.sql creates all foreign keys and constraints. Important hot-path indexes are partial: unpublished outbox rows, due retries, expired leases, and dead letters. This keeps terminal/published history out of scheduler and publisher indexes. Authentication and (tenant_id, idempotency_key) are unique index lookups; (delivery_id, attempt_number) makes attempt writes idempotent.
Services write JSON logs. The API accepts or creates X-Correlation-ID, stores it, adds it to Kafka headers, and workers include it in their logs.
curl http://localhost:8080/health/live
curl http://localhost:8080/health/ready
curl http://localhost:8080/metrics
make metricsMetrics include ingestion latency, successes/failures, retries, dead letters, delivery latency, outbox/retry backlog, duplicate claims, reclaimed leases, rate-limit outcomes and in-flight work.
make test # unit + race detector
make lint # formatting + vet (+ staticcheck when installed)
make test-integration # real PostgreSQL and Redis; concurrency/transactions/rate limit/replay
make test-e2e # API -> outbox -> Kafka -> worker -> signed receiver
make test-fault # leases/crash windows and Kafka/Redis/PostgreSQL outagesStart the infrastructure with make up before container-backed suites. Fault hooks also exist at after-claim, after-send, and after-record boundaries in internal/worker for deeper process-level injection.
make benchmark performs one excluded warm-up, then three measured runs of each profile: ingestion only, successful end-to-end, destination failure, and worker recovery. It stores every raw JSON run plus machine/Docker details under benchmarks/results/<timestamp>/, and writes a median report. There is no synthetic target.
The latest checked-in measured report is benchmarks/results/latest/report.md. The 2026-08-03 run on Docker Desktop (Apple M5 Pro, 15 Docker CPUs, 7.75 GiB Docker memory) produced these medians:
| Profile | Ingest/s | Delivery/s | p50 ingest | p95 ingest | p99 ingest |
|---|---|---|---|---|---|
| Ingestion only | 1,888.2 | n/a | 6.81 ms | 41.62 ms | 44.09 ms |
| Successful end-to-end | 1,173.7 | 163.9 | 11.46 ms | 46.38 ms | 47.25 ms |
| Destination returning 500 | 476.8 | 8.8 terminal/s | 45.07 ms | 47.91 ms | 47.97 ms |
| Worker restart probe | 50.6 | 0.2 | 13.95 ms | 13.95 ms | 13.95 ms |
The 500 profile made three attempts per delivery, so 66.67% of receiver requests repeated a delivery ID; these are configured retries, not an unplanned internal duplicate. The worker-restart probe observed 8.063 seconds from restart command to a queued probe delivery reaching success. That profile is intentionally reported as a recovery probe, not a full backlog-recovery throughput result. Interpret local Docker numbers as comparative rather than production capacity: the stack uses one API, two publishers, two workers (32 handlers each), one Kafka broker with 12 partitions, and 1 KiB payloads.
docs/adr/0001-at-least-once.mddocs/adr/0002-transactional-outbox.mddocs/adr/0003-kafka-partitioning.mddocs/adr/0004-redis-token-bucket.mddocs/interview-prep.md
- At-least-once can expose duplicate requests; receiver deduplication is mandatory.
- DNS rebinding protection is not implemented. IPs are checked at resolution/dial and redirects, but no DNS pinning layer is provided.
- Kafka Compose uses one broker/replica for local resource cost; production durability requires multiple brokers and an appropriate ISR.
- Offset commits are batch-wide, so one process crash can redeliver already completed records in that fetched batch. Claims make these database-idempotent.
- One endpoint maps to one Kafka partition, giving predictable ordering at the transport level, but retries can interleave with later events and delivery concurrency means receiver completion order is not strict.
- Secrets have no online re-wrapping workflow; changing the master key without migration dead-letters affected work.
- Offset pagination can drift under concurrent writes and is intended for support tooling, not bulk export.
Deepen process-level fault injection around real container kills, add DNS pinning/rebinding defense, run multi-broker Kafka durability tests, add master-key versioning/re-wrapping, and benchmark on a dedicated Linux host with fixed CPU/memory limits.
MIT licensed. See LICENSE.