For interviewers and evaluators, Quick Project Walkthrough. View Deck
A connected-vehicle data pipeline built to the shape of the problem: many car manufacturers send the same physical facts (speed, fuel, location) in different field names and different units, and all of it has to be pulled in, converted into one consistent model, watched for meaningful events in real time, and served to a UI.
Everything runs locally in Docker. The stack is TypeScript end to end.
- Ingest raw OEM feeds from Ford, Toyota, and BMW. Each sends speed, fuel, temperature, and location in its own shape.
- Normalize every feed into one canonical model so nothing downstream knows which brand a car came from.
- Load telemetry into Postgres with batched, idempotent writes. Replaying the same message changes nothing.
- Detect events on the canonical stream: hard braking, extended idling, geofence exits, and low fuel.
- Send notifications through Hasura event triggers, also idempotent, so retries never spam.
- Serve a live dashboard over GraphQL subscriptions: a map, an event feed, and a fleet table.
- Durability. Kafka is a distributed append-only log, not a queue. Messages are kept until retention expires and each consumer group tracks its own offset, so a buggy consumer can be fixed and rerun over history.
- Ordering. One vehicle's readings must stay in time order, otherwise a detector cannot tell a hard brake from two unrelated readings. Kafka guarantees order only inside a partition, so every message is keyed by
vehicleId. - Fault tolerance. The local cluster runs three Kafka brokers in KRaft mode with replication factor 3 and
min.insync.replicas = 2. Losing one broker does not stop reads or writes. - Idempotency by natural key. The loader, detector, and notifier all ignore duplicate deliveries rather than trying to prevent them.
- Separation of concerns. Normalization, loading, detection, and notification are separate services that communicate through Kafka or Postgres, not direct calls.
A small React app shows the fleet live over GraphQL subscriptions.
The dots are vehicles, coloured by how fresh their last reading is. The event feed on the right fills in real time as the detector raises hard brakes, extended idles, and geofence exits.
The fleet table makes the point of the two timestamps visible. The rows from the slow feed read "delayed" while the others read "live", because that manufacturer publishes about eight seconds behind.
flowchart LR
subgraph sim["OEM simulators"]
F[Ford<br/>mph, camelCase]
T[Toyota<br/>kmh, snake_case]
B[BMW<br/>kmh, German keys]
end
subgraph kafka["Kafka — distributed log"]
OF[oem.ford]
OT[oem.toyota]
OB[oem.bmw]
CT[canonical.telemetry]
DLQ[telemetry.dlq]
end
subgraph pipeline["Stream processing"]
N[Normalizer]
L[Loader]
D[Detector]
end
subgraph store["Storage & serving"]
PG[(Postgres)]
H[Hasura / GraphQL]
NT[Notifier]
end
F -- key=vehicleId --> OF
T -- key=vehicleId --> OT
B -- key=vehicleId --> OB
OF --> N
OT --> N
OB --> N
N --> CT
N -.rejected.-> DLQ
CT --> L
CT --> D
L --> PG
D --> PG
PG --> H
PG -.event trigger.-> NT
H --> UI[React dashboard]
Data flows left to right. Everything after the normalizer works with one clean format and never sees brand-specific code again.
flowchart TD
F["Ford: speedMph 62"] --> N[Normalizer]
T["Toyota: velocity_kmh 100"] --> N
B["BMW: geschwindigkeit_kmh 100"] --> N
N --> C["one shape: speed_kph 100"]
Ford uses miles, the others use kilometres, and BMW writes in German. The normalizer has one small adapter per brand. Adding a brand later means writing one adapter and nothing else.
The local stack runs three brokers in KRaft mode. Each broker is also a controller, and the three controllers form a Raft quorum that elects partition leaders without ZooKeeper.
Every topic is created with replication factor 3 and min.insync.replicas = 2, so a write is acknowledged once the leader and one follower have it. The cluster survives the loss of any single broker.
Kafka UI is at http://localhost:8080 once the brokers are healthy. It shows the cluster overview, topics, partitions, replicas, consumer groups, and live messages.
Every reading carries two timestamps: when the car measured it, and when we received it.
flowchart LR
A["car measured it<br/>10:00:00"] -->|8 second gap| B["we received it<br/>10:00:08"]
That gap is staleness. One feed runs about eight seconds behind, so on the dashboard its cars show as "delayed" while the others show as "live". Without the second timestamp you would serve old data as if it were current and never notice.
You cannot tell from one reading; "speed is 40" means nothing on its own. So the detector keeps the last few readings and looks for a pattern.
flowchart LR
R1["80"] --> R2["78"] --> R3["40"]
R3 --> Q{"dropped 38<br/>in 2 seconds?"}
Q -->|yes| E["hard brake"]
This "remember the last few" is a sliding window. The same idea finds idling, leaving an area, and low fuel.
Kafka splits a topic into lanes and only promises order within a lane. Tagging each message with the car's id makes Kafka always put the same car in the same lane.
flowchart LR
L0["lane 0: FORD-002 readings, in order"]
L1["lane 1: FORD-003 readings, in order"]
L2["lane 2: FORD-001 readings, in order"]
So one car's readings stay in order even with several workers reading at once. That is the whole reason "80 then 40" can be trusted to mean a brake, and it is measured in the scale experiment: 1602 readings, none out of order.
Kafka and Hasura can deliver the same message twice after a hiccup. Rather than prevent that, the system makes a repeat do nothing: each item has a natural key, and saving a key that already exists is ignored.
flowchart TD
A[message arrives] --> B{seen this<br/>exact one before?}
B -->|no| C[save it and act]
B -->|yes| D[ignore it]
This is why replaying the whole stream changes nothing, and it appears in the same shape four times: the loader, the detector, the notifier, and the end-to-end suite.
flowchart TD
E[(an event happens)]
E -->|order matters| D["detector reads Kafka in order"]
E -->|order does not matter| A["alerts fired by a Hasura trigger"]
The detector must see readings in order, so it reads Kafka. The alert sender does not care which alert goes out first, so a Hasura trigger's parallel delivery is fine for it. Using the right tool for each is the point.
- Apache Kafka sits between every producer and every consumer. It is a durable, replayable log, not a queue: messages are not deleted when read, so a consumer with a bug can be fixed and re-run over the same history. Keying messages by vehicle guarantees that one vehicle's readings stay in order, which is what makes time-based event detection possible.
- Three brokers in KRaft mode make the durability claim real. With replication factor 3 and
min.insync.replicas = 2, the cluster keeps working if any single broker is lost. - A normalizer converts each manufacturer's raw format into one canonical model. This is the core problem and it is deliberately kept in one place.
- Postgres plus Hasura serve the normalized data. Hasura generates the GraphQL API and its live subscriptions directly from the database schema, so there is no hand-written API layer.
Requirements: Docker with Compose, Node 20 or newer, pnpm, and the Nhost CLI.
pnpm install # install workspace dependencies
pnpm kafka:up # start the 3-node Kafka cluster and Kafka UI
pnpm topics:setup # create the declared topics
pnpm smoke # produce and consume test messages end to end
pnpm test # run unit tests
pnpm nhost:up # start Postgres and Hasura
pnpm loader # write the canonical stream into the database
pnpm detector # raise events from the stream
pnpm notifier # send alerts via Hasura event trigger
pnpm dashboard # http://localhost:3000Kafka UI is at http://localhost:8080. The cluster has three brokers; connect to any one to reach all of them. From the host the brokers are at localhost:9092, localhost:9094, and localhost:9096, and from other containers at kafka-1:29092, kafka-2:29092, and kafka-3:29092. A client only needs one of these as its bootstrap: the broker it names hands back the full list.
To stop the broker:
pnpm kafka:downThe default setup uses one partition per topic. That keeps ordering trivial to reason about while the pipeline is being built, but it also caps parallelism: each consumer group can have only one active reader per topic.
The unit tests cover the logic that can be checked without infrastructure: unit conversions, the window arithmetic, the ordering rule, staleness buckets, payload parsing. They run in about a second.
The real ceiling of the current layout is not a hard number from Kafka — it depends on the single broker's disk, the loader's Postgres round-trips, and the detector's memory — but the architecture makes the next bottlenecks predictable:
The end-to-end suite covers the properties that only appear once the pieces are wired together and need Kafka, Postgres and Hasura running. It brings the whole stack up, drives five real scenarios against the services, prints a table, and tears the stack down.
flowchart LR
subgraph before ["Before: 1 partition"]
T1[canonical.telemetry]
D1[Detector]
T1 --> D1
end
subgraph after ["After: 3 partitions"]
P0[partition 0]
P1[partition 1]
P2[partition 2]
D0[Detector 0]
D1b[Detector 1]
P0 --> D0
P1 --> D0
P2 --> D1b
end
before -->|grow partitions| after
Set E2E_KEEP_INFRA=1 to run against a stack that is already up and leave it running, which is faster while iterating.
Set E2E_KEEP_INFRA=1 to run against a stack that is already up and leave it running, which is faster while iterating.
docker-compose.yml 3-node Kafka (KRaft, replication factor 3) and Kafka UI
packages/kafka shared Kafka client, config, topic registry, and admin
packages/canonical the one reading shape everything downstream agrees on
services/simulator the manufacturer simulators, one profile per OEM
services/normalizer per-manufacturer adapters onto the canonical shape
services/loader batched, idempotent writes into Postgres
services/detector sliding-window detection of events on the stream
services/notifier idempotent notifier behind a Hasura event trigger
apps/dashboard live React dashboard, coloured by staleness
nhost/migrations the schema, as versioned migrations
nhost/metadata which tables Hasura exposes, and to whom
scripts/ topic setup, repartition, ordering check, smoke check
scripts/e2e the end-to-end suite: bring up, drive, tear down
docs/architecture.md full design, decisions, and roadmap
docs/scale-experiment.md partitioning, parallelism, and the ordering proof
docs/HLD.md high-level design for a quick overview
One simulator service runs three times with different configuration rather than existing as three near-identical copies. The physics is shared; only the output format differs, so adding a manufacturer means adding one profile under services/simulator/src/profiles and registering it. Nothing else changes.