diff --git a/README.md b/README.md index 2a497e4..505d409 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,28 @@ # Etch -Etch is a small notification hub built on Kafka. You place an order, the order service answers right away, and the emails and texts go out in the background through their own services. +Distributed Notification Hub Etch GIF +## What's going on here + +Etch is a small notification hub built on Kafka. When someone places an order, the order service answers right away. The emails and texts go out later, in the background, through their own services. So a slow or broken email provider never makes an order fail. + +- Place an order, and the answer comes back fast +- Notifications get sent separately, on their own time +- If a send fails, Etch retries it +- If it can never work, Etch parks it safely instead of losing it + ## Why I built it -I wanted to understand what happens when the slow, unreliable part of a system sits behind a message queue. Sending a notification is a good example. Email and SMS providers time out, go down, and occasionally reject a message outright, and none of that should make an order fail or make a customer wait. +I wanted to understand what happens when the slow, unreliable part of a system sits behind a message queue. Sending a notification is a good example. Email and SMS providers time out, go down, and sometimes reject a message outright. None of that should make an order fail or make a customer wait. -So I built the whole path to see how it holds up: an order comes in, an event goes onto Kafka, and a separate service picks it up and handles delivery. The parts I cared about most were the unglamorous ones. What happens when a send fails? How many times do you retry? Where does a message go when it can never succeed? And what stops a customer from getting the same email twice when Kafka redelivers an event? Etch is my answer to those questions, kept small enough to read in an afternoon. +So I built the whole path to see how it holds up. I cared most about the unglamorous parts. Etch is my answer to these questions, kept small enough to read in an afternoon. + +- What happens when a send fails? +- How many times should you retry? +- Where does a message go when it can never succeed? +- What stops a customer from getting the same email twice when Kafka redelivers an event? ## How it fits together @@ -43,36 +57,51 @@ flowchart LR NOTIFY -->|sent / failed / dead-letter| KAFKA ``` -1. The client creates an order through the gateway. -2. The order service saves it to MySQL, publishes an `order-created` event, and responds immediately. -3. The notification service consumes the event and creates one notification per requested channel (email, SMS, or both). -4. Each notification is sent through the email or SMS service. If a send fails temporarily, it's retried with exponential backoff. -5. If it fails for good, or runs out of retries, it goes to a dead-letter topic instead of being dropped. +Here's the journey of one order. + +- The client creates an order through the gateway +- The order service saves it to MySQL and publishes an `order-created` event +- It responds right away, without waiting for any notifications +- The notification service picks up the event +- It creates one notification for each channel requested (email, SMS, or both) +- Each notification goes out through the email or SMS service +- If a send fails temporarily, it's retried with exponential backoff +- If it fails for good, or runs out of retries, it goes to a dead-letter topic instead of being dropped ## What it does -- **Async by design.** Creating an order never waits on delivery. -- **Retries and a dead-letter queue.** Temporary failures are retried with backoff. Permanent failures and exhausted retries land in the `notification-dlt` topic and get logged. -- **No duplicates.** Redis tracks which events and notifications have already been handled, so a redelivered Kafka message doesn't send twice. -- **Five services, each in its own container.** Gateway, order, notification, email, and SMS. The email and SMS services are mocks that fail on purpose, so you can watch the failure handling work. -- **Tested.** Unit tests for the logic, plus Testcontainers integration tests that run against real MySQL, Kafka, and Redis. -- **Ships to AWS.** A GitHub Actions pipeline tests the code, builds the Docker images, and deploys to ECS Fargate. +- **Async by design.** Creating an order never waits on delivery +- **Retries and a dead-letter queue.** Temporary failures are retried with backoff, and permanent ones land in the `notification-dlt` topic and get logged +- **No duplicates.** Redis tracks which events and notifications were already handled, so a redelivered Kafka message doesn't send twice +- **Five services, each in its own container.** Gateway, order, notification, email, and SMS +- **Mock providers that fail on purpose.** The email and SMS services are stand-ins, so you can watch the failure handling work +- **Easy to follow in the logs.** Requests carry an `X-Correlation-Id` header +- **Tested.** Unit tests for the logic, plus Testcontainers integration tests against real MySQL, Kafka, and Redis +- **Set up for AWS.** A GitHub Actions pipeline tests the code, builds the Docker images, and deploys to ECS Fargate ## Tech stack -Java 25, Spring Boot, Spring Cloud Gateway, Apache Kafka, MySQL, Redis, JUnit and Testcontainers, Docker and Docker Compose, GitHub Actions, and AWS (ECS/Fargate, ECR, RDS, ElastiCache). +- Java 25 +- Spring Boot and Spring Cloud Gateway +- Apache Kafka +- MySQL +- Redis +- JUnit and Testcontainers +- Docker and Docker Compose +- GitHub Actions +- AWS (ECS/Fargate, ECR, RDS, ElastiCache) ## Running it -You need Docker with Compose v2. To build or test outside Docker you also need JDK 25 and Maven 3.9 or newer. +You'll need Docker with Compose v2. To build or test outside Docker, you'll also need JDK 25 and Maven 3.9 or newer. -From the repo root: +From the repo root, start everything. ```bash docker compose up -d --build ``` -That starts Kafka, MySQL, Redis, and all five services. The first build takes a few minutes. +That starts Kafka, MySQL, Redis, and all five services. The first build takes a few minutes. Then you can reach each service here. | Service | URL | |---|---| @@ -84,7 +113,7 @@ That starts Kafka, MySQL, Redis, and all five services. The first build takes a ### Try it -Create an order through the gateway. Three demo users are created on startup. +Three demo users are created on startup. Start by creating an order through the gateway. ```bash curl -s -X POST http://localhost:8080/orders \ @@ -92,13 +121,13 @@ curl -s -X POST http://localhost:8080/orders \ -d '{"userId":1,"orderNumber":"ORD-1001","total":49.99,"channels":["EMAIL","SMS"]}' ``` -Then look at what happened to its notifications: +Then look at what happened to its notifications. ```bash curl -s http://localhost:8080/notifications/order/1 ``` -The mock services reject some requests on purpose, so create a few orders and watch the notification service work through retries and dead-lettering: +The mock services reject some requests on purpose. So create a few more orders, then watch the notification service work through retries and dead-lettering. ```bash docker compose logs -f notification-service @@ -110,11 +139,18 @@ docker compose logs -f notification-service docker compose down ``` -Add `-v` to also wipe the MySQL volume and start clean next time. +Add `-v` to also wipe the MySQL volume, so you start clean next time. + +## Endpoints + +- `POST /orders` creates an order +- `GET /orders` and `GET /orders/{id}` read orders +- `GET /notifications/order/{orderId}` shows the notifications for an order +- The mock providers listen at `POST /email/send` and `POST /sms/send` ## Configuration -Everything has a sensible local default, and `docker-compose.yml` overrides what needs to change inside containers. These are the ones you're most likely to touch: +Everything has a sensible local default. The `docker-compose.yml` file overrides what needs to change inside containers. These are the ones you're most likely to touch. | Variable | Used by | Default | What it does | |---|---|---|---| @@ -128,6 +164,8 @@ Everything has a sensible local default, and `docker-compose.yml` overrides what ## Testing +Run these from the repo root. + ```bash mvn test # unit tests mvn verify # unit tests plus the Testcontainers integration tests (needs Docker) @@ -163,4 +201,10 @@ infrastructure/ ## Deployment -The repo includes ECS task definitions for each service, targeting ECS Fargate with images in ECR, MySQL on RDS, and Redis on ElastiCache. The GitHub Actions pipeline runs the tests, builds the images, and pushes and deploys them when AWS credentials and repository secrets are configured. See [infrastructure/aws/README.md](infrastructure/aws/README.md) for the setup notes. +The repo is set up to deploy to AWS. The infrastructure itself isn't created by code in this repo, so you set that up once by hand. + +- There are ECS task definitions for each service, targeting ECS Fargate +- Images live in ECR, MySQL runs on RDS, and Redis runs on ElastiCache +- The GitHub Actions pipeline runs the tests, builds the images, then pushes and deploys them +- The deploy steps only work once AWS credentials and repository secrets are configured +- The setup notes are in [infrastructure/aws/README.md](infrastructure/aws/README.md) diff --git a/services/notification-service/src/main/java/com/etch/notificationservice/redis/IdempotencyService.java b/services/notification-service/src/main/java/com/etch/notificationservice/redis/IdempotencyService.java index 3d7fdc3..0f6c61f 100644 --- a/services/notification-service/src/main/java/com/etch/notificationservice/redis/IdempotencyService.java +++ b/services/notification-service/src/main/java/com/etch/notificationservice/redis/IdempotencyService.java @@ -33,6 +33,9 @@ public boolean claim(String key) { } public boolean claim(String key, Duration ttl) { + if (ttl.isZero() || ttl.isNegative()) { + throw new IllegalArgumentException("Idempotency claim ttl must be positive (was " + ttl + ")"); + } Boolean firstClaim = redisTemplate.opsForValue().setIfAbsent(key, "1", ttl); return Boolean.TRUE.equals(firstClaim); } diff --git a/services/notification-service/src/main/java/com/etch/notificationservice/retry/ExponentialBackOff.java b/services/notification-service/src/main/java/com/etch/notificationservice/retry/ExponentialBackOff.java index c4fbc4f..c26d171 100644 --- a/services/notification-service/src/main/java/com/etch/notificationservice/retry/ExponentialBackOff.java +++ b/services/notification-service/src/main/java/com/etch/notificationservice/retry/ExponentialBackOff.java @@ -11,6 +11,9 @@ public class ExponentialBackOff implements BackOff { private final long baseDelayMs; public ExponentialBackOff(@Value("${etch.retry.base-delay-ms:500}") long baseDelayMs) { + if (baseDelayMs <= 0) { + throw new IllegalArgumentException("etch.retry.base-delay-ms must be positive (was " + baseDelayMs + ")"); + } this.baseDelayMs = baseDelayMs; } diff --git a/services/notification-service/src/main/java/com/etch/notificationservice/service/NotificationDispatchService.java b/services/notification-service/src/main/java/com/etch/notificationservice/service/NotificationDispatchService.java index 89e50ed..ad26ab9 100644 --- a/services/notification-service/src/main/java/com/etch/notificationservice/service/NotificationDispatchService.java +++ b/services/notification-service/src/main/java/com/etch/notificationservice/service/NotificationDispatchService.java @@ -49,6 +49,9 @@ public NotificationDispatchService(List channelClients, NotificationMessageRenderer messageRenderer, BackOff backOff, @Value("${etch.retry.max-attempts:3}") int maxAttempts) { + if (maxAttempts < 1) { + throw new IllegalArgumentException("etch.retry.max-attempts must be at least 1 (was " + maxAttempts + ")"); + } this.clientsByChannel = channelClients.stream() .collect(Collectors.toMap(ChannelClient::channel, Function.identity())); this.notificationRepository = notificationRepository; diff --git a/services/notification-service/src/test/java/com/etch/notificationservice/redis/IdempotencyServiceTest.java b/services/notification-service/src/test/java/com/etch/notificationservice/redis/IdempotencyServiceTest.java index 7a8c174..b5db7ef 100644 --- a/services/notification-service/src/test/java/com/etch/notificationservice/redis/IdempotencyServiceTest.java +++ b/services/notification-service/src/test/java/com/etch/notificationservice/redis/IdempotencyServiceTest.java @@ -11,6 +11,7 @@ import java.time.Duration; import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; import static org.mockito.ArgumentMatchers.any; import static org.mockito.ArgumentMatchers.eq; import static org.mockito.Mockito.lenient; @@ -58,6 +59,14 @@ void claim_usesTheSuppliedTtlWhenGiven() { verify(valueOperations).setIfAbsent("k2", "1", customTtl); } + @Test + void claim_rejectsNonPositiveTtl() { + assertThatThrownBy(() -> idempotencyService.claim("k3", Duration.ZERO)) + .isInstanceOf(IllegalArgumentException.class); + assertThatThrownBy(() -> idempotencyService.claim("k3", Duration.ofSeconds(-1))) + .isInstanceOf(IllegalArgumentException.class); + } + @Test void release_deletesTheKey() { idempotencyService.release("k1"); diff --git a/services/notification-service/src/test/java/com/etch/notificationservice/retry/ExponentialBackOffTest.java b/services/notification-service/src/test/java/com/etch/notificationservice/retry/ExponentialBackOffTest.java index e2817db..1f1f3b3 100644 --- a/services/notification-service/src/test/java/com/etch/notificationservice/retry/ExponentialBackOffTest.java +++ b/services/notification-service/src/test/java/com/etch/notificationservice/retry/ExponentialBackOffTest.java @@ -3,6 +3,7 @@ import org.junit.jupiter.api.Test; import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; class ExponentialBackOffTest { @@ -15,6 +16,12 @@ void delayDoublesWithEachAttempt() { assertThat(backOff.delayMsFor(3)).isEqualTo(400L); } + @Test + void rejectsNonPositiveBaseDelay() { + assertThatThrownBy(() -> new ExponentialBackOff(0L)).isInstanceOf(IllegalArgumentException.class); + assertThatThrownBy(() -> new ExponentialBackOff(-1L)).isInstanceOf(IllegalArgumentException.class); + } + @Test void neverProducesANegativeExponentForAttemptZeroOrBelow() { ExponentialBackOff backOff = new ExponentialBackOff(100L); diff --git a/services/notification-service/src/test/java/com/etch/notificationservice/service/NotificationDispatchServiceTest.java b/services/notification-service/src/test/java/com/etch/notificationservice/service/NotificationDispatchServiceTest.java index 301c3cc..6df56f6 100644 --- a/services/notification-service/src/test/java/com/etch/notificationservice/service/NotificationDispatchServiceTest.java +++ b/services/notification-service/src/test/java/com/etch/notificationservice/service/NotificationDispatchServiceTest.java @@ -20,6 +20,7 @@ import java.util.List; import java.util.Optional; +import static org.assertj.core.api.Assertions.assertThatThrownBy; import static org.mockito.ArgumentMatchers.any; import static org.mockito.ArgumentMatchers.anyInt; import static org.mockito.ArgumentMatchers.anyString; @@ -58,6 +59,14 @@ void setUp() { messageRenderer, backOff, MAX_ATTEMPTS); } + @Test + void constructor_rejectsMaxAttemptsBelowOne() { + assertThatThrownBy(() -> new NotificationDispatchService( + List.of(emailClient), notificationRepository, deadLetterService, outcomeRecorder, + messageRenderer, backOff, 0)) + .isInstanceOf(IllegalArgumentException.class); + } + private Notification pendingNotification() { Notification notification = new Notification(1L, NotificationChannel.EMAIL, "a@example.com", "corr-1"); setId(notification, 10L);