coderconnoisseur · Oct 15, 2026 · Oct 15, 2026 · Oct 15, 2026 · Oct 15, 2026 · Oct 15, 2026
diff --git a/README.md b/README.md
@@ -39,8 +39,8 @@ RidePulse follows an event-driven microservices design where each component has
 | **Rider Simulator** | Generates ride requests at configurable RPS against the HTTP gateway |
 | **API Gateway** | Accepts `POST /rides`, publishes `ride.requested` to Kafka |
 | **Pricing Service** | Consumes `ride.requested`, computes fare, publishes `ride.priced`; failures retried, then dead-lettered to `ride.requested.dlq` |
-| **Matching Service** | Consumes `ride.priced`, runs geospatial search + atomic driver locking, assigns the driver (idempotent per ride), publishes `ride.matched`; infrastructure failures retried, then dead-lettered to `ride.priced.dlq` |
-| **Assignment Service** | Consumes `ride.matched`, records the assignment, exposes end-to-end latency metrics; dead-letters to `ride.matched.dlq` |
+| **Matching Service** | Consumes `ride.priced`, runs geospatial search + atomic driver locking, assigns the driver (idempotent per ride), publishes `ride.matched`; business failures (`no_drivers`, `match_deadline`) publish `ride.unmatched`; infrastructure failures retried, then dead-lettered to `ride.priced.dlq` |
+| **Assignment Service** | Consumes `ride.matched` and `ride.unmatched`, records each ride's outcome once (duplicates ignored), exposes end-to-end latency metrics and `GET /rides/<id>`; dead-letters to `ride.matched.dlq` / `ride.unmatched.dlq` |
 | **Driver Simulator** | Continuously updates live driver locations in Redis GEO; keeps assigned drivers out of the GEO set and releases them after `DRIVER_TRIP_DURATION` |
 | **Kafka** | Decouples every stage of the pipeline |
 | **Redis** | GEO indexing, distributed driver locking (Lua safe-unlock) and driver assignment state |
@@ -71,14 +71,15 @@ RiderSimulator ──HTTP POST /rides──▶ API Gateway
                                         │   (batch losers explicitly released)
                                         ├── atomic assign: driver:assigned, ride:driver,
                                         │   remove from GEO (replays return same driver)
-                                        ▼
-                                 Kafka: ride.matched
+                                        ├─ matched ──▶ Kafka: ride.matched
+                                        └─ no driver / deadline ──▶ Kafka: ride.unmatched
                                         │
                                         ▼
                                 Assignment Service
                                         │
                                         ▼
-                        records assignment; end_to_end_latency
+              records the ride's outcome once; end_to_end_latency
+              (GET /rides/<id> returns matched or unmatched)
 ```
 
 ---
@@ -109,13 +110,13 @@ Full write-up: [shuffle-nearest-first-fix.md](docs/shuffle-nearest-first-fix.md)
 
 ### 4. Progressive Parallel Locking with hierarchical deadlines
 
-Each match has its own 800 ms budget; each batch of 5 parallel `SETNX` attempts has a 200 ms budget nested inside it; Kafka publish gets its own independent 300 ms budget so a slow match can't starve the publish RTT.
+Each match has its own budget (`MATCH_DEADLINE`, default 500 ms; the write-up below was done at 800 ms); each batch of 5 parallel `SETNX` attempts has a 200 ms budget nested inside it; Kafka publish gets its own independent 300 ms budget so a slow match can't starve the publish RTT.
 
 Full write-up: [deadline-fix.md](docs/deadline-fix.md).
 
 ### 5. At-least-once consumers with ordered commits and DLQs
 
-Pricing, matching and assignment share one runner (`pkg/kafkaconsumer`). An offset is committed only after its message, and every earlier message of the partition, was handled or dead-lettered, so a crash redelivers instead of losing work. Failed handlers are retried with backoff, then written to `<topic>.dlq` with `x-error` / `x-original-*` headers; shutdown drains in-flight messages for up to 10 s.
+Pricing, matching and assignment share one runner (`pkg/kafkaconsumer`). An offset is committed only after its message, and every earlier message of the partition, was handled or dead-lettered, so a crash redelivers instead of losing work. Failed handlers are retried with backoff, then written to `<topic>.dlq` with `x-error` / `x-original-*` headers; a handler panic is dead-lettered without retries instead of crashing the process; a consumer-group rewind starts a new tracker epoch so stale completions cannot corrupt commits; shutdown drains in-flight messages for up to 10 s.
 
 Full write-up: [at-least-once-consumers.md](docs/at-least-once-consumers.md).
 
@@ -136,9 +137,10 @@ Every stage exposes Prometheus metrics on its own port. `RequestedAt` is stamped
 Key exports:
 - `gateway_request_duration_seconds`, `gateway_requests_total`, `gateway_publish_errors_total`
 - `pricing_processing_duration_seconds`, `pricing_events_published_total`
-- `matching_latency_seconds`, `matching_success_total`, `matching_failures_total`
+- `matching_latency_seconds`, `matching_success_total`, `matching_failures_total`, `match_unmatched_published_total{reason}`
 - `driver_lock_conflict_total`, `driver_lock_released_phantom_total`
-- `end_to_end_latency_seconds`, `assignments_total`
+- `end_to_end_latency_seconds`, `assignments_total` (each ride counted once), `assignment_duplicates_total`
+- `unmatched_total{reason}`, `unmatched_duplicates_total`, `unmatched_end_to_end_latency_seconds`
 - `pricing_retries_total`, `matching_retries_total`, `pricing_dlq_messages_total`, `matching_dlq_messages_total`, `assignment_dlq_messages_total`, `match_idempotent_replays_total`
 
 ---
@@ -162,19 +164,20 @@ Headline from that run (Windows dev box, single-partition Kafka, single Redis, ~
 | Peak "healthy" throughput (>95 % success) | ~50 RPS |
 | Successful `matching_latency_seconds` p95 | stayed 33–84 ms across every step |
 
-The primary bottleneck is matcher pool starvation from slow-fail matches: under contention, ~30 % of matches consume workers for the full 800 ms `matchDeadline`, blowing past `NumCPU × 5` worker capacity. Fixing this (fail-fast on batch 1 saturation, multi-partition Kafka topics, larger pool) is the next set of unlocks.
+The primary bottleneck in that run was matcher pool starvation from slow-fail matches: under contention, ~30 % of matches consumed workers for the full 800 ms match deadline, blowing past the `NumCPU × 5` worker capacity of the time. The matcher defaults have since changed (`MATCH_WORKERS` = `NumCPU × 25`, `MATCH_DEADLINE` = 500 ms) but the new values have **not been measured**; the ladder must be re-run. Fail-fast on batch 1 saturation and multi-partition Kafka topics are still open.
 
 See [rps-benchmarking-setup.md](docs/rps-benchmarking-setup.md) for the runbook, ladder shape, PromQL queries, and interpretation notes.
 
 ---
 
 ## Known limitations and next steps
 
-- **Matcher pool starvation.** The pool is `runtime.NumCPU() * 5` workers and each match may hold one for up to the 800 ms `matchDeadline`. Under contention, slow-fail matches occupy workers until the deadline and the queue backs up. Next: fail fast when the first batch is fully contended instead of waiting out the deadline, then re-measure pool size.
-- **Single-partition Kafka topics.** One partition per topic caps consumer parallelism per service. Next: partition by ride ID and run multiple consumers per stage.
-- **Hardcoded endpoints.** Services connect to `localhost:9092` and `localhost:6379` directly; only the rider simulator and driver simulator (`REDIS_ADDR`) read their config from environment variables. Next: env-based config so services can run inside Compose.
-- **In-memory assignment store.** The assignment service keeps results in a map for benchmark verification; nothing is persisted.
-- **Test coverage is unit-level.** Unit tests cover the tier-preserving shuffle, the at-least-once consumer runner (ordered commits, retries, DLQ, drain) and the Redis Lua scripts (locking, assignment, release) against an in-process Redis (miniredis). Nothing yet exercises real Kafka/Redis end to end. Next: integration tests against containerized Kafka and Redis.
+- **Matcher pool starvation: new defaults, still unmeasured.** The pool now defaults to `runtime.NumCPU() * 25` workers (`MATCH_WORKERS`) and the match deadline to 500 ms (`MATCH_DEADLINE`), down from `NumCPU * 5` and 800 ms. Slow-fail matches still hold a worker until the deadline, and no ladder has been run with these values, so they are a starting point, not a result. Next: re-run the ladder, and fail fast when the first batch is fully contended instead of waiting out the deadline.
+- **Single-partition Kafka topics.** One partition per topic caps consumer parallelism per service. Next: create topics with more partitions, partition by ride ID and run multiple consumers per stage. Until then the rewind handling below is not exercised.
+- **Rewind detection is partial.** The offset tracker detects a rewind only when a re-fetched offset is at or below the last offset it tracked. A partition re-assigned starting *above* that offset is not detected, so the committed offset can move backwards and messages are redelivered. That stays within at-least-once, and handlers absorb duplicates.
+- **Two readers, two consumer groups.** assignment-service reads `ride.matched` in group `assignment-service` and `ride.unmatched` in group `assignment-service-unmatched`. A shared group would make the leader's metadata request for both topics fail as a whole while `ride.unmatched` does not exist yet (it is created lazily on the first business failure), stalling the success path too; with separate groups a missing `ride.unmatched` topic only delays the unmatched reader, and rebalances are independent.
+- **In-memory assignment store.** The assignment service keeps ride outcomes (matched and unmatched) in maps for benchmark verification; nothing is persisted, so after a restart a redelivered event is counted as new.
+- **Test coverage is unit-level.** Unit tests cover the tier-preserving shuffle, the at-least-once consumer runner (ordered commits, retries, DLQ, drain, rewind epochs, panic recovery) and the Redis Lua scripts (locking, assignment, release) against an in-process Redis (miniredis). Nothing yet exercises real Kafka/Redis end to end: the unmatched path, the epoch-stamped commits and the new matcher defaults have not been run against a live cluster. Next: integration tests against containerized Kafka and Redis.
 
 ---
 
@@ -243,6 +246,25 @@ Prometheus scrape targets:
 
 Confirm all four are `up` at http://localhost:9090/targets.
 
+**Configuration.** Every service reads its endpoints from environment variables; unset or blank values fall back to the defaults below, so the commands above work unchanged on one host. `KAFKA_BROKERS` is a comma-separated list. Invalid values make the services exit at startup, except rider-simulator, which falls back to the default for a non-numeric or non-positive `RIDER_RPS` / `RIDER_CONCURRENCY`; double-check these when running the ladder.
+
+| Variable | Default | Used by | Meaning |
+|---|---|---|---|
+| `KAFKA_BROKERS` | `localhost:9092` | api-gateway, pricing, matching, assignment | Comma-separated broker list. |
+| `REDIS_ADDR` | `localhost:6379` | matching, driver-simulator | Redis address. |
+| `HTTP_ADDR` | `:8080` (api-gateway), `:2114` (assignment) | api-gateway, assignment | HTTP listen address (also serves `/metrics`). |
+| `METRICS_ADDR` | `:2115` (pricing), `:2112` (matching) | pricing, matching | Metrics listen address. |
+| `MATCH_WORKERS` | `runtime.NumCPU() * 25` | matching | Matcher worker-pool size; positive integer. |
+| `REDIS_POOL_SIZE` | `MATCH_WORKERS * 5` | matching | Redis connection pool size; positive integer. Each worker runs up to 5 parallel `SETNX` attempts, while the go-redis default is only 10 x GOMAXPROCS. |
+| `MATCH_DEADLINE` | `500ms` | matching | Total per-match budget (Go duration); must exceed the 200 ms batch deadline. |
+| `DRIVER_COUNT` | `1000` | driver-simulator | Number of simulated drivers. |
+| `DRIVER_TRIP_DURATION` | `5s` | driver-simulator | How long an assigned driver stays busy (Go duration). |
+| `RIDER_URL` | `http://localhost:8080/rides` | rider-simulator | Gateway endpoint to POST rides to. |
+| `RIDER_RPS` | `500` | rider-simulator | Target requests per second. |
+| `RIDER_CONCURRENCY` | `200` | rider-simulator | Max in-flight requests; sends beyond it are counted as dropped. |
+
+The `MATCH_WORKERS` and `MATCH_DEADLINE` defaults have not been benchmarked; treat them as untested until the ladder is re-run.
+
 ### 3. Send load
 
 **Automated ladder (recommended):**
@@ -277,15 +299,21 @@ histogram_quantile(0.95, sum by (le) (rate(end_to_end_latency_seconds_bucket[1m]
 # Where matches are failing (contention vs deadline)
 rate(driver_lock_conflict_total[1m])
 rate(driver_lock_released_phantom_total[1m])
+
+# Rides that found no driver, by reason (no_drivers | match_deadline)
+sum by (reason) (rate(unmatched_total[1m]))
 ```
 
 Verify pipeline closure by querying assignment-service directly:
 
 ```bash
-curl http://localhost:2114/count            # total assignments observed
-curl http://localhost:2114/assignments/<id> # look up a specific ride
+curl http://localhost:2114/count            # total matched rides observed
+curl http://localhost:2114/assignments/<id> # look up a matched ride (404 if unmatched)
+curl http://localhost:2114/rides/<id>       # outcome of any ride: matched or unmatched
 ```
 
+`GET /rides/<id>` returns `{"ride_id","status":"matched","driver_id","matched_at","latency_ms"}` or `{"ride_id","status":"unmatched","reason","failed_at"}` (`reason` is `no_drivers` or `match_deadline`), and 404 for a ride with no outcome yet.
+
 ### 5. Run the checks CI runs
 
 Each service is its own Go module (tied together by `services/go.work` for local development). CI runs `go vet`, `go build`, and `go test` in every module; the unit tests need no Kafka or Redis: