diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..2a45ab7 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,115 @@ +name: CI + +on: + push: + pull_request: + +jobs: + firmware-build: + name: Build firmware (ESP32-S3) + runs-on: ubuntu-latest + container: + image: espressif/idf:release-v5.4 + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Write placeholder Wi-Fi credentials + run: | + cat > main/wifi_credentials.h <<'EOF' + #pragma once + constexpr const char* WIFI_SSID = "ci-placeholder-ssid"; + constexpr const char* WIFI_PASSWORD = "ci-placeholder-password"; + EOF + + - name: Write placeholder admin credentials + run: | + cat > main/admin_credentials.h <<'EOF' + #pragma once + constexpr const char* ADMIN_USER = "ci-placeholder-admin"; + constexpr const char* ADMIN_PASS = "ci-placeholder-password"; + EOF + + - name: Build firmware + run: | + . "$IDF_PATH/export.sh" + idf.py set-target esp32s3 + idf.py build + + host-tests: + name: Host-side unit tests (dns_wire) + runs-on: ubuntu-latest + container: + image: espressif/idf:release-v5.4 + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Build host_test (linux target) + working-directory: host_test + run: | + . "$IDF_PATH/export.sh" + idf.py --preview set-target linux build + + - name: Run host tests + run: ./host_test/build/host_test.elf + + release: + name: Publish release artifacts + needs: [firmware-build, host-tests] + if: startsWith(github.ref, 'refs/tags/v') + permissions: + contents: write + runs-on: ubuntu-latest + container: + image: espressif/idf:release-v5.4 + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Write placeholder Wi-Fi credentials + run: | + cat > main/wifi_credentials.h <<'EOF' + #pragma once + constexpr const char* WIFI_SSID = "ci-placeholder-ssid"; + constexpr const char* WIFI_PASSWORD = "ci-placeholder-password"; + EOF + + - name: Write placeholder admin credentials + run: | + cat > main/admin_credentials.h <<'EOF' + #pragma once + constexpr const char* ADMIN_USER = "ci-placeholder-admin"; + constexpr const char* ADMIN_PASS = "ci-placeholder-password"; + EOF + + - name: Build firmware + run: | + . "$IDF_PATH/export.sh" + idf.py set-target esp32s3 + idf.py build + + - name: Create GitHub Release + uses: softprops/action-gh-release@v2 + with: + files: | + build/bootloader/bootloader.bin + build/partition_table/partition-table.bin + build/mini_dns.bin + body: | + Prebuilt binaries for `${{ github.ref_name }}`. + + **These binaries use placeholder Wi-Fi and admin credentials and will + not connect to any real network.** To flash a working device: + + 1. Clone the repo at this tag: `git checkout ${{ github.ref_name }}` + 2. Create `main/wifi_credentials.h` and `main/admin_credentials.h` with + your real values (see README.md's Setup section) + 3. `idf.py set-target esp32s3 && idf.py build` + 4. `idf.py -p flash monitor` + + Flash command for these prebuilt binaries (placeholder credentials — + device will boot but will not join your Wi-Fi): + ``` + python -m esptool --chip esp32s3 -b 460800 --before default_reset --after hard_reset write_flash --flash_mode dio --flash_size 16MB --flash_freq 80m 0x0 bootloader.bin 0x8000 partition-table.bin 0x20000 mini_dns.bin + ``` diff --git a/.gitignore b/.gitignore index ed29c10..1b2e220 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,8 @@ main/wifi_credentials.h +main/admin_credentials.h build/ +managed_components/ cmake-build-debug/ cmake-build-debug-esp-idf/ .idea/ diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 6e54fb0..86bdac4 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -2,9 +2,9 @@ ## What this is -`mini_dns` is ESP32-S3 firmware moving from proof-of-concept toward a marketable "edge DNS" appliance: a single device that connects to Wi-Fi, serves a small hardcoded set of hostnames authoritatively, forwards everything else to an upstream recursive resolver with TTL caching, and serves an HTTP page + JSON API showing the static record table. It was built incrementally — Wi-Fi → raw UDP → DNS parsing → single-record response → multi-record + NXDOMAIN → HTTP server → JSON API → page wired to the API → **forwarding resolver + cache (Phase 1 of the edge-DNS roadmap)** — each step flashed and confirmed on real hardware before moving on. +`mini_dns` is ESP32-S3 firmware moving from proof-of-concept toward a marketable "edge DNS" appliance: a single device that connects to Wi-Fi, serves a runtime-managed set of hostnames authoritatively (A and/or AAAA), forwards everything else to an upstream recursive resolver with TTL caching and secondary-upstream failover, sinkholes ad/tracker domains, exposes Prometheus metrics, advertises itself via mDNS, and serves an HTTP page + a JSON CRUD API for managing that record table. It was built incrementally — Wi-Fi → raw UDP → DNS parsing → single-record response → multi-record + NXDOMAIN → HTTP server → JSON API → page wired to the API → **forwarding resolver + cache (Phase 1)** → **NVS-backed ad-block (Phase 2)** → **Prometheus `/metrics` (Phase 3)** → **mDNS responder (Phase 4)** → **record management: persistence + CRUD API + auth (Phase 5)** → **hardening: dual-stack records + secondary upstream + host tests (Phase 6)** — each step flashed and confirmed on real hardware before moving on. -**Records are still compile-time constants, read-only after boot** — there is no persistence, no provisioning UI, no OTA, no runtime record editing, no auth. Ad-block, metrics/OTEL export, and mDNS are planned as later phases (see Future scoping), not built yet. If you're extending this, read the Non-Goals section before adding anything that smells like a "real" feature. +**As of Phase 5, records are runtime-managed** — persisted in NVS, editable via `POST`/`PUT`/`DELETE /api/records` (Basic-auth-gated), no longer a reflash-only compile-time table. **As of Phase 6, a record can hold an A, an AAAA, or both**, and a forwarded query that times out against the primary upstream gets one retry against a secondary before SERVFAIL. There is still no provisioning UI, no OTA, no TLS. If you're extending this, read the Non-Goals section before adding anything that smells like a "real" feature. ## Target hardware / toolchain @@ -25,31 +25,42 @@ sequenceDiagram wifi_connect-->>app_main: returns once IP obtained app_main->>dns_task: dns_server_start() [xTaskCreate, returns immediately] app_main->>httpd: http_server_start() [httpd_start, returns immediately] + app_main->>mdns: mdns_responder_start() [mdns_init, returns immediately] app_main--)app_main: app_main() returns, its task exits - Note over dns_task,httpd: Both keep running independently.
No shared mutable state between them. + Note over dns_task,httpd: All three keep running independently.
No shared mutable state between them. ``` -Three independent runtime pieces exist after boot, with **no synchronization between them**: +Four independent runtime pieces exist after boot, with **no synchronization between them**: -1. **Wi-Fi event handling** (`wifi_connect.cpp`) — registered event handlers keep running for the life of the device (auto-reconnect on disconnect), but `wifi_connect()` itself only blocks `app_main` during initial bring-up. +1. **Wi-Fi event handling** (`wifi_connect.cpp`) — registered event handlers keep running for the life of the device (auto-reconnect on disconnect, plus IPv6 link-local bring-up as of Phase 4), but `wifi_connect()` itself only blocks `app_main` during initial IPv4 bring-up. 2. **DNS task** (`dns_server.cpp`) — a task we create explicitly via `xTaskCreate`, because raw BSD sockets have no framework driving an accept/receive loop for us. 3. **HTTP server task(s)** (`http_server.cpp`) — `esp_http_server` owns and drives its own task(s) internally; we only configure and start it. +4. **mDNS responder task** (`mdns_responder.cpp`, Phase 4) — the `mdns` component owns and drives its own internal task, the same shape as `esp_http_server`; we only configure and start it. It reads whatever A/AAAA addresses the STA netif currently holds and shares no mutable state with the other three pieces. -`DNS_RECORDS` is still `constexpr`, shared read-only across tasks with no locking needed. **As of the forwarding resolver (Phase 1), the DNS task also owns a cache and an in-flight query table** (`DnsCache`, `DnsForwarder`) — these *are* mutable, but they're only ever touched from within the single DNS task's `select()` loop, so the "no shared mutable state across tasks" property still holds; it's just no longer literally "no mutable state," only "no mutable state shared *between* tasks." +**As of the forwarding resolver (Phase 1), the DNS task also owns a cache and an in-flight query table** (`DnsCache`, `DnsForwarder`) — these *are* mutable, but they're only ever touched from within the single DNS task's `select()` loop, so the "no shared mutable state across tasks" property still holds; it's just no longer literally "no mutable state," only "no mutable state shared *between* tasks." + +Three pieces of state cross the DNS task / HTTP task boundary. Two do it without a mutex: `DnsBlocklist`'s domain set (Phase 2) is populated once at boot and never mutated again, so concurrent reads need no synchronization; and `DnsMetrics` (Phase 3) is a set of `std::atomic` counters/gauges — the DNS task increments/sets them inline, the HTTP task's `/metrics` handler reads a `snapshot()`. Both were narrow, deliberate exceptions to "no shared mutable state," justified the same way: either the shared value is immutable after boot, or it's a single atomic word. + +**`DnsRecordStore` (Phase 5) breaks that pattern on purpose — it's the first genuinely shared *mutable* state, guarded by a real `std::mutex`.** The record table can be neither immutable-after-boot (records are edited at runtime by design) nor a bag of atomics (a record is a hostname+IP pair that must be read as one consistent unit, not field-by-field). The DNS task's `record_store().find()` locks briefly per query; the HTTP task's `create()`/`update()`/`remove()` lock to mutate and persist. This is a deliberate, documented departure from the "lock-free DNS path" property every earlier phase preserved — see the Phase 5 design doc for why a plain mutex was chosen over a lock-free (atomic-snapshot) alternative, and for why `find()` returns a *copy* of the IP rather than a pointer/reference into the table. ## Component map | File | Responsibility | |---|---| -| `main/main.cpp` | Boot sequence: logs detected PSRAM size → `wifi_connect()` → `dns_server_start()` → `http_server_start()`. | -| `main/wifi_connect.h/.cpp` | Blocking Wi-Fi station bring-up with hardcoded credentials. Retries forever on disconnect (2s backoff), no give-up state. | +| `main/main.cpp` | Boot sequence: logs detected PSRAM size → `wifi_connect()` → `dns_server_start()` → `http_server_start()` → `mdns_responder_start()`. | +| `main/wifi_connect.h/.cpp` | Blocking Wi-Fi station bring-up with hardcoded credentials. Retries forever on disconnect (2s backoff), no give-up state. As of Phase 4, also triggers IPv6 link-local address creation on `WIFI_EVENT_STA_CONNECTED` (non-blocking, doesn't gate boot readiness) so mDNS has an address to advertise as AAAA. | | `main/wifi_credentials.h` | `WIFI_SSID`/`WIFI_PASSWORD` constants. **Gitignored** — does not exist on a fresh checkout, must be recreated. | -| `main/dns_records.h` | The hardcoded hostname→IPv4 table (`DNS_RECORDS`, a `constexpr std::array`). Pure data, no logic, no dependencies beyond ``/`` — intentionally kept dependency-free. | -| `main/dns_wire.h/.cpp` | Pure DNS wire-format functions: header/question parsing, name skipping, answer-section TTL scanning, and response building (A-record, NXDOMAIN, and a generic "relay a captured answer section" builder used by both cache hits and forwarded replies). No I/O, no FreeRTOS/lwIP dependency — the natural home for host-side unit tests. | +| `main/admin_credentials.h` | `ADMIN_USER`/`ADMIN_PASS` constants for Basic auth on the mutating `/api/records` routes (Phase 5). **Gitignored**, same pattern as `wifi_credentials.h` — does not exist on a fresh checkout, must be recreated. | +| `main/dns_records.h` | The seed hostname→IPv4 table (`DNS_RECORDS_DEFAULTS`, a `constexpr std::array`), consulted only once by `DnsRecordStore::load_from_nvs()` on first boot (Phase 5) — the live table lives in NVS afterward. Pure data, no logic, no dependencies beyond ``/``. IPv4-only by design — AAAA for a seeded name is added later through the CRUD API (Phase 6). | +| `main/dns_record_store.h/.cpp` | Runtime-managed DNS record table (`DnsRecordStore`, Phase 5) — NVS-persisted, guarded by a `std::mutex` since it's the first cross-task state that's genuinely shared *and* mutable. Each entry holds an optional IPv4 *and* optional IPv6 address (Phase 6 dual-stack — never neither). DNS task calls `find()` (returns a copy of the whole entry, not a pointer); HTTP task calls `snapshot()`/`create()`/`update()`/`remove()`. | +| `main/dns_wire.h/.cpp` | Pure DNS wire-format functions: header/question parsing, name skipping, answer-section TTL scanning, and response building (A-record, AAAA-record, NXDOMAIN, NODATA, and a generic "relay a captured answer section" builder used by both cache hits and forwarded replies). No I/O, no FreeRTOS/lwIP dependency — covered by the `host_test/` Unity suite (Phase 6). | | `main/dns_cache.h/.cpp` | TTL cache (`DnsCache`) keyed by (lowercased qname, qtype), storing captured answer-section bytes + a clamped TTL. Single-owner (the DNS task); no mutex. | -| `main/dns_forwarder.h/.cpp` | Upstream UDP client socket (`DnsForwarder`) plus the in-flight query table that correlates upstream replies back to the client that asked, using a slot-index + generation-counter transaction ID scheme. Single-owner; no mutex. | -| `main/dns_server.h/.cpp` | UDP/53 listener in its own FreeRTOS task, running a `select()` loop over the listen socket and the forwarder's upstream socket. Resolves each query via local table → cache → forward, in that order, and reaps timed-out forwarded queries into SERVFAIL. | -| `main/http_server.h/.cpp` | `esp_http_server` bring-up with two routes: `GET /` (static HTML page with an inline `fetch()` script) and `GET /api/records` (JSON array via cJSON). Unchanged by Phase 1 — still only reflects the static table, not cache/forwarding activity. | +| `main/dns_forwarder.h/.cpp` | Upstream UDP client socket (`DnsForwarder`) plus the in-flight query table that correlates upstream replies back to the client that asked, using a slot-index + generation-counter transaction ID scheme. As of Phase 6, a slot whose primary attempt times out is retried once against a secondary upstream before giving up, reusing the same transaction ID and socket. Single-owner; no mutex. | +| `main/dns_server.h/.cpp` | UDP/53 listener in its own FreeRTOS task, running a `select()` loop over the listen socket and the forwarder's upstream socket. Resolves each query via record store → blocklist → cache → forward, in that order, and reaps timed-out forwarded queries into SERVFAIL. Also the sole writer of `DnsMetrics` (Phase 3). | +| `main/dns_blocklist.h/.cpp` | NVS-backed ad-block domain set (`DnsBlocklist`), suffix-matched on label boundaries. Populated once at boot, never mutated afterward — safely read from both the DNS and HTTP tasks with no lock; only its atomic block counter changes at runtime. | +| `main/dns_metrics.h/.cpp` | Runtime counters/gauges/histogram for `/metrics` (`DnsMetrics`, Phase 3) — query/cache/forward/SERVFAIL counts, an upstream-latency histogram, and cache/in-flight occupancy gauges. Written only by the DNS task, read only by the HTTP task's `/metrics` handler, all via `std::atomic` — the same cross-task shape as `DnsBlocklist`'s counter, generalized into its own module. | +| `main/http_server.h/.cpp` | `esp_http_server` bring-up with routes: `GET /` (static HTML page with an inline `fetch()` script), `GET`/`POST`/`PUT`/`DELETE`/`OPTIONS /api/records` (JSON CRUD via cJSON, mutating routes Basic-auth-gated and CORS-enabled — Phase 5), `GET /api/blocklist` (Phase 2), and `GET /metrics` (Prometheus plaintext, Phase 3). | +| `main/mdns_responder.h/.cpp` | mDNS responder bring-up (Phase 4, `espressif/mdns` managed component) — advertises the device as `edge-dns.local` (A + AAAA) plus an `_http._tcp` service pointing at the dashboard. Owns its own internal task, no shared mutable state with the DNS/HTTP tasks. | ## Data flow @@ -58,9 +69,11 @@ Three independent runtime pieces exist after boot, with **no synchronization bet ``` select() wakes on listen socket readable → recvfrom() → parse_dns_header() → parse_question_name() → read qtype/qclass - → find_dns_record() (case-insensitive linear scan of DNS_RECORDS) - match, qtype==A → build_a_record_response() → sendto() - match, other type→ build_nxdomain_response() (existing simplification, see Gotchas) + → record_store().find() (locked, case-insensitive scan of the record table) + match, qtype==A, entry has ipv4 → build_a_record_response() → sendto() + match, qtype==AAAA, entry has ipv6 → build_aaaa_record_response() → sendto() + match, family absent for this qtype→ build_nodata_response() (Phase 6: name + exists, NOERROR/no answers — not NXDOMAIN) no match → DnsCache::lookup(lowercased qname, qtype) hit → build_relayed_response(cached answer) → sendto() miss → DnsForwarder::forward() @@ -71,14 +84,17 @@ select() wakes on listen socket readable select() wakes on forwarder socket readable → DnsForwarder::handle_upstream_readable() (matches reply to its in-flight slot via slot-index + generation in the - transaction ID, scans the answer section for min TTL) + transaction ID — accepting a reply from either upstream, Phase 6 — scans + the answer section for min TTL) → DnsCache::insert() (caches the answer, or a capped-TTL negative entry if ancount == 0) → build_relayed_response() → sendto() to the original client select() times out, or every loop iteration regardless of wake reason → DnsForwarder::reap_expired() (in-flight queries past their deadline) - → build_relayed_response(SERVFAIL) → sendto() to each one's client + attempt==0 (primary only) → resend to secondary upstream, same slot, + fresh deadline (Phase 6) — stays in-flight + attempt==1 (secondary too) → build_relayed_response(SERVFAIL) → sendto() → (on a bare timeout) DnsCache::sweep_expired() ``` @@ -87,8 +103,13 @@ Local-table responses still echo the request's question section verbatim and reu **Upstream transaction ID scheme** (`dns_forwarder.cpp`): the in-flight table has a fixed power-of-two size (32). Each outstanding query is assigned an upstream-facing transaction ID whose low bits are the table slot index and whose high bits are a per-slot generation counter, incremented every time the slot is reused. This gives O(1) matching of an upstream reply to its slot (no linear scan), and the generation counter is what safely rejects a stale reply for an already-expired-and-reused slot instead of misrouting it to the wrong client. **HTTP request** (`http_server.cpp`): -- `GET /` → static HTML shell with an inline `