Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 14 additions & 10 deletions .github/workflows/ci-cd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,30 +9,28 @@ on:
env:
AWS_REGION: us-east-1

permissions:
contents: read

jobs:
build-and-test:
name: Build and test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Set up JDK 21
- name: Set up JDK 25
uses: actions/setup-java@v4
with:
distribution: temurin
java-version: '21'
java-version: '25'
cache: maven

- name: Build all services
run: mvn -B -DskipTests package

- name: Run unit tests
run: mvn -B test

- name: Run integration tests
- name: Build and test
# Docker is preinstalled on ubuntu-latest runners, so the
# Testcontainers-backed tests (order-service, notification-service)
# run against real MySQL/Kafka/Redis containers here.
# run against real MySQL/Kafka containers here. verify covers
# compile, unit tests (surefire) and integration tests (failsafe).
run: mvn -B verify

- name: Upload test reports
Expand All @@ -50,6 +48,9 @@ jobs:
needs: build-and-test
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write
strategy:
matrix:
service: [order-service, notification-service, email-service, sms-service, api-gateway]
Expand Down Expand Up @@ -84,6 +85,9 @@ jobs:
needs: build-and-push-images
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write
strategy:
matrix:
service: [order-service, notification-service, email-service, sms-service, api-gateway]
Expand Down
128 changes: 50 additions & 78 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,16 @@
# Etch

Etch is a distributed notification hub using Apache Kafka to decouple high-volume business orders from email and SMS delivery.
Etch is a small notification hub built on Kafka. You place an order, the order service answers right away, and the emails and texts go out in the background through their own services.

<img src="/etch.gif" alt="Etch GIF" width="300">

## System architecture
## Why I built it

I wanted to understand what happens when the slow, unreliable part of a system sits behind a message queue. Sending a notification is a good example. Email and SMS providers time out, go down, and occasionally reject a message outright, and none of that should make an order fail or make a customer wait.

So I built the whole path to see how it holds up: an order comes in, an event goes onto Kafka, and a separate service picks it up and handles delivery. The parts I cared about most were the unglamorous ones. What happens when a send fails? How many times do you retry? Where does a message go when it can never succeed? And what stops a customer from getting the same email twice when Kafka redelivers an event? Etch is my answer to those questions, kept small enough to read in an afternoon.

## How it fits together

```mermaid
flowchart LR
Expand Down Expand Up @@ -37,39 +43,36 @@ flowchart LR
NOTIFY -->|sent / failed / dead-letter| KAFKA
```

## Tech stack
1. The client creates an order through the gateway.
2. The order service saves it to MySQL, publishes an `order-created` event, and responds immediately.
3. The notification service consumes the event and creates one notification per requested channel (email, SMS, or both).
4. Each notification is sent through the email or SMS service. If a send fails temporarily, it's retried with exponential backoff.
5. If it fails for good, or runs out of retries, it goes to a dead-letter topic instead of being dropped.

- **Backend:** Java 21, Spring Boot, Spring Cloud Gateway
- **Messaging:** Apache Kafka
- **Database:** MySQL, Flyway
- **Caching and idempotency:** Redis
- **Testing:** JUnit, Spring Boot Test, Testcontainers
- **DevOps:** Docker, Docker Compose, GitHub Actions
- **Deployment:** AWS ECS/Fargate, Amazon ECR, Amazon RDS, ElastiCache
## What it does

## Key features
- **Async by design.** Creating an order never waits on delivery.
- **Retries and a dead-letter queue.** Temporary failures are retried with backoff. Permanent failures and exhausted retries land in the `notification-dlt` topic and get logged.
- **No duplicates.** Redis tracks which events and notifications have already been handled, so a redelivered Kafka message doesn't send twice.
- **Five services, each in its own container.** Gateway, order, notification, email, and SMS. The email and SMS services are mocks that fail on purpose, so you can watch the failure handling work.
- **Tested.** Unit tests for the logic, plus Testcontainers integration tests that run against real MySQL, Kafka, and Redis.
- **Ships to AWS.** A GitHub Actions pipeline tests the code, builds the Docker images, and deploys to ECS Fargate.

- five Spring Boot services that process orders and deliver email and SMS notifications asynchronously over Kafka
- retry handling with exponential backoff, and a dead-letter topic for notifications that fail permanently
- Redis-backed deduplication, so a redelivered Kafka event is never processed or sent twice
- persisted notification status per order and channel
- unit and Testcontainers-based integration tests
- Dockerfiles for every service, a Docker Compose stack, and a GitHub Actions pipeline that tests, builds, and deploys to AWS ECS
## Tech stack

## Prerequisites
Java 25, Spring Boot, Spring Cloud Gateway, Apache Kafka, MySQL, Redis, JUnit and Testcontainers, Docker and Docker Compose, GitHub Actions, and AWS (ECS/Fargate, ECR, RDS, ElastiCache).

- Docker with Docker Compose v2, to run the full stack
- JDK 21 and Maven 3.9 or newer, to build and test outside Docker
## Running it

## Running locally
You need Docker with Compose v2. To build or test outside Docker you also need JDK 25 and Maven 3.9 or newer.

From the repo root:

```bash
docker compose up -d --build
```

This starts Kafka (KRaft, single node), MySQL, Redis, and all five services. The first build takes a few minutes; subsequent starts are faster.
That starts Kafka, MySQL, Redis, and all five services. The first build takes a few minutes.

| Service | URL |
|---|---|
Expand All @@ -79,73 +82,64 @@ This starts Kafka (KRaft, single node), MySQL, Redis, and all five services. The
| Email Service (mock) | http://localhost:8083 |
| SMS Service (mock) | http://localhost:8084 |

### Try it end to end
### Try it

Create an order through the gateway. Demo users are seeded by Flyway.
Create an order through the gateway. Three demo users are created on startup.

```bash
curl -s -X POST http://localhost:8080/orders \
-H 'Content-Type: application/json' \
-d '{"userId":1,"orderNumber":"ORD-1001","total":49.99,"channels":["EMAIL","SMS"]}'
```

Check the notification status for the order:
Then look at what happened to its notifications:

```bash
curl -s http://localhost:8080/notifications/order/1
```

The mock email and SMS services intentionally fail a configurable percentage of requests, which makes retries and dead-letter handling easy to observe. Watch them in the logs:
The mock services reject some requests on purpose, so create a few orders and watch the notification service work through retries and dead-lettering:

```bash
docker compose logs -f notification-service
```

### Stopping the stack
### Stopping it

```bash
docker compose down
```

Add `-v` to also delete the MySQL volume and start from a clean database next time.
Add `-v` to also wipe the MySQL volume and start clean next time.

## Configuration

Every setting has a local default, and `docker-compose.yml` overrides the ones that differ inside containers. The variables you are most likely to change:
Everything has a sensible local default, and `docker-compose.yml` overrides what needs to change inside containers. These are the ones you're most likely to touch:

| Variable | Used by | Default | Purpose |
| Variable | Used by | Default | What it does |
|---|---|---|---|
| `ORDER_SERVICE_URL`, `NOTIFICATION_SERVICE_URL` | api-gateway | `http://localhost:8081`, `http://localhost:8082` | Downstream routes |
| `ORDER_SERVICE_URL`, `NOTIFICATION_SERVICE_URL` | api-gateway | `http://localhost:8081`, `http://localhost:8082` | Where the gateway forwards requests |
| `KAFKA_BOOTSTRAP_SERVERS` | order-service, notification-service | `localhost:9092` | Kafka brokers |
| `DB_HOST`, `DB_PORT`, `DB_NAME`, `DB_USERNAME`, `DB_PASSWORD` | order-service, notification-service | `localhost`, `3306`, per-service schema, `etch`, `etch` | MySQL connection |
| `REDIS_HOST`, `REDIS_PORT` | notification-service | `localhost`, `6379` | Idempotency |
| `EMAIL_SERVICE_URL`, `SMS_SERVICE_URL` | notification-service | `http://localhost:8083`, `http://localhost:8084` | Channel service endpoints |
| `EMAIL_FAILURE_RATE`, `SMS_FAILURE_RATE` | email-service, sms-service | `0.05` | Fraction of requests the mock rejects as a permanent failure |
| `EMAIL_UNAVAILABLE_RATE`, `SMS_UNAVAILABLE_RATE` | email-service, sms-service | `0.0` | Fraction of requests the mock answers with a 503, which notification-service retries |
| `DB_HOST`, `DB_PORT`, `DB_NAME`, `DB_USERNAME`, `DB_PASSWORD` | order-service, notification-service | `localhost`, `3306`, one schema per service, `etch`, `etch` | MySQL connection |
| `REDIS_HOST`, `REDIS_PORT` | notification-service | `localhost`, `6379` | Redis, used for deduplication |
| `EMAIL_SERVICE_URL`, `SMS_SERVICE_URL` | notification-service | `http://localhost:8083`, `http://localhost:8084` | Where deliveries are sent |
| `EMAIL_FAILURE_RATE`, `SMS_FAILURE_RATE` | email-service, sms-service | `0.05` | Share of requests the mock rejects permanently |
| `EMAIL_UNAVAILABLE_RATE`, `SMS_UNAVAILABLE_RATE` | email-service, sms-service | `0.0` | Share of requests the mock answers with a 503, which gets retried |

## Testing

Run unit tests:

```bash
mvn test
```

Run unit and Testcontainers integration tests:

```bash
mvn verify
mvn test # unit tests
mvn verify # unit tests plus the Testcontainers integration tests (needs Docker)
```

The integration suite requires Docker.

Run the tests for one service together with the shared modules it depends on:
To test one service along with the shared modules it depends on:

```bash
mvn -pl services/order-service -am test
```

## Project structure
## Project layout

```text
services/
Expand All @@ -156,39 +150,17 @@ services/
└── sms-service/

shared/
├── common-events/
├── common-dto/
└── common-utils/
├── common-events/ event classes shared between services
├── common-dto/ request and response types
└── common-utils/ exceptions and logging helpers

infrastructure/
├── docker/
└── aws/
├── docker/ MySQL init script
└── aws/ ECS task definitions and setup notes

.github/
└── workflows/
.github/workflows/ CI/CD pipeline
```

## Deployment

The repository includes AWS ECS deployment configuration for:

- ECS Fargate
- Amazon ECR
- Amazon RDS for MySQL
- ElastiCache for Redis

The GitHub Actions pipeline builds and tests the services, builds Docker images, and contains the configuration required to publish and deploy them when the appropriate AWS credentials and repository secrets are provided.

## Design highlights

### Asynchronous processing

Creating an order does not wait for notification delivery. The Order Service persists the order and publishes an event to Kafka, allowing notification processing to happen independently.

### Failure handling

Transient delivery failures are retried with exponential backoff. Permanent failures, and messages that exhaust their retry attempts, are published to a dead-letter topic (`notification-dlt`) and logged by the notification service.

### Idempotency

Redis is used to prevent the same notification from being processed or delivered multiple times.
The repo includes ECS task definitions for each service, targeting ECS Fargate with images in ECR, MySQL on RDS, and Redis on ElastiCache. The GitHub Actions pipeline runs the tests, builds the images, and pushes and deploys them when AWS credentials and repository secrets are configured. See [infrastructure/aws/README.md](infrastructure/aws/README.md) for the setup notes.
16 changes: 5 additions & 11 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -15,16 +15,16 @@
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>3.3.4</version>
<version>3.5.16</version>
<relativePath/>
</parent>

<properties>
<java.version>21</java.version>
<maven.compiler.source>21</maven.compiler.source>
<maven.compiler.target>21</maven.compiler.target>
<java.version>25</java.version>
<maven.compiler.source>25</maven.compiler.source>
<maven.compiler.target>25</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<spring-cloud.version>2023.0.3</spring-cloud.version>
<spring-cloud.version>2025.0.3</spring-cloud.version>
</properties>

<modules>
Expand Down Expand Up @@ -90,11 +90,6 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<!-- Mockito's bundled Byte Buddy doesn't yet officially
recognize bytecode from JDKs newer than it was built
against; this opts into Byte Buddy's experimental mode
so class mocking still works on newer JDKs. -->
<argLine>-Dnet.bytebuddy.experimental=true</argLine>
<!-- Testcontainers-based integration tests run separately
via failsafe (mvn verify), not on every `mvn test`. -->
<excludes>
Expand All @@ -106,7 +101,6 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<argLine>-Dnet.bytebuddy.experimental=true</argLine>
<includes>
<include>**/*IntegrationTest.java</include>
</includes>
Expand Down
6 changes: 3 additions & 3 deletions services/api-gateway/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Build from the repository root: docker build -f services/api-gateway/Dockerfile .

FROM maven:3.9-eclipse-temurin-21 AS build
FROM maven:3.9-eclipse-temurin-25 AS build
WORKDIR /workspace

COPY pom.xml .
Expand All @@ -9,11 +9,11 @@ COPY services services

RUN mvn -q -pl services/api-gateway -am -DskipTests package

FROM eclipse-temurin:21-jre-alpine
FROM eclipse-temurin:25-jre-alpine
RUN addgroup -S etch && adduser -S etch -G etch
WORKDIR /app
COPY --from=build /workspace/services/api-gateway/target/api-gateway.jar app.jar
USER etch

EXPOSE 8080
ENTRYPOINT ["java", "-jar", "/app/app.jar"]
ENTRYPOINT ["java", "-XX:MaxRAMPercentage=75", "-jar", "/app/app.jar"]
1 change: 1 addition & 0 deletions services/api-gateway/src/main/resources/application.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ spring:

server:
port: 8080
shutdown: graceful

management:
endpoints:
Expand Down
6 changes: 3 additions & 3 deletions services/email-service/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Build from the repository root: docker build -f services/email-service/Dockerfile .

FROM maven:3.9-eclipse-temurin-21 AS build
FROM maven:3.9-eclipse-temurin-25 AS build
WORKDIR /workspace

COPY pom.xml .
Expand All @@ -9,11 +9,11 @@ COPY services services

RUN mvn -q -pl services/email-service -am -DskipTests package

FROM eclipse-temurin:21-jre-alpine
FROM eclipse-temurin:25-jre-alpine
RUN addgroup -S etch && adduser -S etch -G etch
WORKDIR /app
COPY --from=build /workspace/services/email-service/target/email-service.jar app.jar
USER etch

EXPOSE 8083
ENTRYPOINT ["java", "-jar", "/app/app.jar"]
ENTRYPOINT ["java", "-XX:MaxRAMPercentage=75", "-jar", "/app/app.jar"]
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,11 @@ public class EmailSendService {

public EmailSendService(@Value("${etch.simulation.unavailable-rate:0.0}") double unavailableRate,
@Value("${etch.simulation.failure-rate:0.05}") double failureRate) {
if (unavailableRate < 0 || failureRate < 0 || unavailableRate + failureRate > 1.0) {
throw new IllegalArgumentException(
"etch.simulation rates must be non-negative and sum to at most 1.0 (unavailable="
+ unavailableRate + ", failure=" + failureRate + ")");
}
this.unavailableRate = unavailableRate;
this.failureRate = failureRate;
}
Expand Down
1 change: 1 addition & 0 deletions services/email-service/src/main/resources/application.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ spring:

server:
port: 8083
shutdown: graceful

etch:
simulation:
Expand Down
Loading
Loading