[coderconnoisseur]

Summary

Phase 1 correctness work: stop losing Kafka messages, stop double-booking drivers, and shut services down cleanly.

At-least-once consumers (pkg/kafkaconsumer)

• New shared module used by pricing, matching and assignment (their ad-hoc worker pools are removed).
• An offset is committed only after that message and every earlier offset in its partition has finished, so a crash replays work instead of dropping it.
• Failed handlers are retried with exponential backoff (3 retries, 50 ms → 1 s). Messages that still fail, or can't be decoded, go to <topic>.dlq with x-error / x-original-* headers.
• Readers commit asynchronously every 1 s, and commit errors are logged. DLQ topics are auto-created.
• New metrics: *_retries_total, *_dlq_messages_total.

Driver state model (matching-service + driver-simulator)

• A matched driver is marked driver:assigned:<id> and removed from the GEO index in one atomic Lua script. Locking refuses assigned drivers.
• ride:driver:<rideID> makes matching idempotent: retries and redeliveries return the same driver (match_idempotent_replays_total).
• Infrastructure errors in matching are retried. "No drivers" and the overall match deadline are logged and committed (a ride.unmatched event is the next roadmap item).
• The driver simulator keeps assigned drivers out of the GEO set and frees them after DRIVER_TRIP_DURATION (default 5s). New env vars: DRIVER_COUNT, REDIS_ADDR.

Graceful shutdown

• SIGINT/SIGTERM drains in-flight work (up to 10 s), shuts down the HTTP servers, then closes Kafka readers and writers and Redis. This applies to pricing, matching, assignment and the driver simulator.

Docs and CI

• New: docs/at-least-once-consumers.md, docs/driver-state-model.md.
• Updated: README, docs/rps-benchmarking-setup.md.
• CI now also checks the pkg module.

Behaviour changes to note

• Benchmarks are not comparable with the committed CSV. Drivers now stay busy for the trip duration. With the defaults (1000 drivers, 5 s trips), throughput tops out at roughly 180 rides/s. The ladder needs re-running.
• After a consumer-group generation change, Kafka rewinds to the last committed offset. The commit watermark can then stall until a restart (nothing is lost). This needs a proper fix before going multi-partition.

Testing

• go vet, go build, go test -race pass in all 8 modules, in both workspace and GOWORK=off mode.
• New unit tests cover the runner (ordered commits, retries, DLQ, drain), the tracker, and the Redis Lua scripts against miniredis (lock, assign, replay, release). They also cover matching's failure classification.
• Not yet run end to end against real Kafka and Redis.