CanSystems

Firmware monorepo for a distributed home-IoT system: ESP8266/ESP32 nodes talk to a Mosquitto broker over MQTT/TLS (with optional Home Assistant auto-discovery), and an ESP32 gateway bridges a 500 kbit/s CAN bus of ATmega328P devices onto MQTT. Everything — firmware, configuration files, even the CAN devices' firmware — is updatable over the air through MQTT.

About this repository. This is a personal project built for my own home setup, shared as a reference to draw ideas from rather than a turnkey product to deploy as-is — the hardware, CAN IDs and device list are specific to my installation. If something here is useful to you, take the pattern and adapt it. Things it may be worth a look for: a CAN↔MQTT bridge, an MQTT-based OTA scheme that also updates the AVR CAN nodes through the ESP32 gateway, a native (host) test setup with hardware shims for embedded code, and a one-command release gate (build + tests + static analysis + lint/type/format) mirrored in CI.

Architecture

                 MQTT over TLS (8883)
  Server side  ─────────────────────────  Device side
┌──────────────────┐                  ┌─────────────────────────┐
│ Mosquitto broker │◄────────────────►│ ESP8266 nodes           │
│ Home Assistant   │                  │  • rad    (ENC28J60 LAN)│
│ ota/otaUpdate.py │                  │  • thermo (Wi-Fi)       │
└──────────────────┘                  ├─────────────────────────┤
                                      │ ESP32 CAN gateway       │
                                      │  (LAN8720 Ethernet)     │
                                      └───────────┬─────────────┘
                                          CAN bus │ 500 kbit/s, ext. ID
                                      ┌───────────┴─────────────┐
                                      │ ATmega328P devices      │
                                      │  • alert (LED+MP3+sens.)│
                                      │  • irrigation (pumps)   │
                                      └─────────────────────────┘

Firmware targets (PlatformIO environments)

Environment	MCU	Role
nanoatmega328_alert	ATmega328P	CAN alert node: WS2812 strip, DFPlayer MP3, Si7021 + LDR, pushbutton
nanoatmega328_irrigation	ATmega328P	CAN irrigation node: 4 pump channels, flow/current safety checks, moisture sensors (CAN-only — see below)
project_esp8266_rad	ESP8266	Geiger counter (CPM → µSv/h) + 433 MHz RF transceiver, ENC28J60 Ethernet
project_esp8266_thermo	ESP8266	DS18B20 multi-probe thermometer (works with zero probes — handy as a test board)
project_esp32_can	ESP32	CAN↔MQTT gateway for the alert nodes, LAN8720 Ethernet
native_test	host	Native unit-test suite (custom runner + shims)

The irrigation node is CAN-only for now: it publishes MOISTURE_DATA and IRRIGATION_ERROR on the bus, but the gateway has no driver registered for its CAN ID, so those frames are dropped and nothing reaches MQTT or Home Assistant. The node is dormant and a CanIrrigationDriver is future work; CanMqttGateway already carries the shared parts (availability, info, OTA, discovery), so only processCanFrameArrived() would need writing.

The nanoatmega328_bootloader_* environments only burn the urboot bootloader and fuses (see bootloader/README.md for the variants, fuses and rebuild steps).

MQTT scheme

• Device → server: iot/dtos/<mac>/<subtopic>; server → device: iot/stod/<mac>/<subtopic>.

• Every node publishes a retained availability topic (LWT) and a retained info topic (fw version = git commit count, git hash, dirty flag, reset reason, and boot: how far the previous run's startup got as a BootStage ordinal — 0 when nothing was recorded, 12 once the main loop had the device).

• The info topic's rr is the SDK's own enum on the ESP nodes. On the CAN nodes it is the bitmask ResetHandler::getResetReason() builds — MCUSR in bits 0-3, a deliberate-restart flag in bit 4, and why in bits 5-7. Read it back as three fields rather than looking a value up:

  Bits	What they carry
  0-3	MCUSR as the hardware left it. 0x05 is a power-on (brown-out comes up with it), 0x04 a brown-out while running — the urboot envs set BOD to 4.3 V — and 0x08 a reset pin or a hang the watchdog caught: urboot consumes EXTRF, so the two look alike. A whole rr of 0x00 means the node is not running urboot, the only bootloader that hands MCUSR on.
  4	Set when the reset came from restartMCU(). Those always arrive through the watchdog, so bit 3 is set alongside.
  5-7	The ResetHandler::RestartCause ordinal, meaningful only while bit 4 is set. The enum is the list; a new cause needs no change here.

• After every reconnect the node publishes a retained diag topic: the cause of the last disconnect (MQTT status or NETWORK_LOST), its UTC timestamp, the offline duration in seconds (measured from client-side detection, i.e. up to ~2x keepalive after the actual drop), a since-boot reconnect counter, and pingRetry, the keep-alive pings the TCP client refused to take since boot — those are retried rather than dropped, so the count is what tells a keep-alive loss caused here apart from one the network caused. Kept in RAM only: an outage that ends in the offline-watchdog MCU reset is reported by the info topic's reset reason instead.

• CAN sub-devices get their own sub-tree: iot/dtos/<mac>/alert1/{availability,info,ota,button}. An alert node takes {"Colors":[r,g,b]} on its iot/stod topic to set the LEDs, and {"Sound":n,"Volume":v} to play a track - with Colors if it should light up while playing.

• The gateway hands out CAN addresses on its own can subtopic. A node with nothing in EEPROM answers on an address it derives from its unique id (0x300 | (crc16(uid) & 0xFF)) and announces itself there every 5 seconds; the gateway keeps what it hears for 30 seconds. Both answers go to the subtopic the question arrived on:

  {"list":true}                            -> {"waiting":[{"uid":"a1b2c3d4e5f60708","at":789}]}
  {"assign":{"uid":"a1b2c3d4e5f60708","id":26}}   -> {"type":1,"cmd":9,"err":0}   (ACK)
  

  assign refuses an id of 0, the master's, one a driver already answers on, or one from the provisional block; the node itself refuses a request that did not come from the master or that names an address it no longer holds. On success it stores the pair and restarts.

• Home Assistant MQTT discovery is opt-in via "haDiscovery": true in server.json; when disabled, the nodes actively retract their previously published entities.

On-device configuration (LittleFS)

File	Purpose
/config/server.json	Wi-Fi + MQTT credentials, server URL/port, haDiscovery toggle
/config/mosq-ca.crt	CA certificate for TLS server validation (NTP sync runs first for X.509)
/config/tube.json	Geiger tube type (rad node only)

server.json is rendered per device by ota/otaUpdate.py from ota/secrets.yaml (git-ignored secrets) + ota/devices.yaml (non-secret fields); the first LittleFS image of a fresh device is flashed over USB by the tool's Initial provisioning action, which stages the device's /config/* files into the transient, git-ignored data/ directory and runs uploadfs. See ota/README.md.

OTA and file transfer

ESP firmware / files (MQTT): ota/otaUpdate.py (interactive curses menu; configured by ota/secrets.yaml + ota/devices.yaml) sends files as base64 pieces with per-piece ACK. See ota/README.md for setup and a copy-paste secrets.yaml example. A firmware upload must carry a binId, which the running firmware checks against its own PIO env before accepting the transfer; from there it streams into the Updater and reboots. Other files go to a temp file, are MD5-verified and renamed into place (file names are allow-listed). The 100-byte piece size is deliberate — larger pieces measured slower in practice.

CAN device firmware (two-stage): the ATmega firmware is first uploaded to the gateway's LittleFS as /canAlertFw.bin (this also auto-triggers the CAN OTA), then streamed over the CAN bus one OtaCanFrame::dataPieceSize chunk per frame — the eighth data byte carries the piece's place in the stream — checked against a CRC16 over the whole image. The ATmega stages it to its SPI flash (W25Q64); on reset the urboot dual-boot bootloader programs the MCU from SPI flash. Result: {"OTA":"[OK]"} / {"OTA":"[ERR]"} on the device's ota subtopic.

Every node expecting that file is updated from the one upload, one transfer at a time: they share a CAN bus and a controller that holds a single frame, so running them together finishes no sooner and keeps each node in transfer twice as long. A node that is not answering its ping is skipped rather than queued, and the image's checksum is computed once for the whole batch.

Repository layout

Path	Contents
src/main_*.cpp	One entry point per environment (selected via build_src_filter)
lib/	Feature libraries (task scheduler, MQTT/CAN stacks, drivers, OTA, …)
test/	Native test suites + test/_shims/ (Arduino/LittleFS/Update/PubSubClient fakes)
web/	Browser firmware and configuration installer for the ESP boards, published with each release — see web/README.md
ota/	Server-side OTA/file-transfer tool + its own README (runtime deps in ota/requirements.txt: paho-mqtt, pyyaml, tqdm)
scripts/	Build helpers (git version injection, ELF→BIN, library patching, size compare) and the release gate (release_check.py)
bootloader/	Prebuilt urboot images for the ATmega nodes
data/	LittleFS image source — transient and git-ignored; ota/otaUpdate.py provisioning generates and clears it
audio/	MP3 set for the alert node's DFPlayer SD card (see audio/README.md)

Building, testing, flashing

pio run                                  # build all environments
pio run -e project_esp8266_thermo -t upload      # serial flash one target
pio test -e native_test                  # native test suite
pio check                                # cppcheck on all environments
python scripts/analysis_check.py         # clang-tidy on the check_* environments

A fresh device is set up entirely from ota/otaUpdate.py: the Initial firmware flash action serial-flashes the firmware, and Initial provisioning flashes the first LittleFS config image (it generates data/ and runs uploadfs itself — see ota/README.md).

A release is a v<commit count> tag on a pushed master, created by python scripts/release_tag.py. CI then publishes the ESP firmware as a release on both forges, keeping only the newest one, and puts the browser installer around it on GitHub Pages, from where a board can be updated, or set up from scratch, over USB without a local toolchain.

Watching a board on the bench goes through scripts/board_console.py, which restarts it over the USB-serial adapter's control lines and prints what it says next:

python scripts/board_console.py --board esp               # ESP8266 / ESP32
python scripts/board_console.py --board avr --listen 20   # ATmega328P

The wiring has to be named because nothing in the USB descriptors reveals it — the same CH340 sits under a D1 mini and under a Nano — and the two differ in a way that matters: an ESP board restarts from RTS and is left alone by merely attaching, while an ATmega restarts from DTR, which opening the port already asserts, so there is no way to watch one without restarting it first. The script's own docstring carries the rest, including what each board then reports about the restart.

The whole build is warning-clean under -Wall -Wextra -Werror; keep it that way. Firmware version comes from the git commit count, so commit before flashing release builds (the dirty flag is published in the info topic).

Development setup

C/C++ builds and tests run through PlatformIO. Setting up a fresh clone:

git clone <repo> && cd CanSystems
git config core.autocrlf input         # Windows only: keep LF endings (the gate checks them)

python -m venv .venv                    # release-gate Python tooling + OTA runtime deps
.venv/bin/pip install -r requirements-dev.txt

• PlatformIO provides the build (pio). Install it via the VS Code pioarduino IDE extension (recommended in .vscode/extensions.json) or pip install pioarduino; it runs from its own install (~/.platformio/penv/bin/pio), not from .venv, and the root .venv does not interfere with it. Toolchains download on the first build (needs internet).
• The Python tooling for the release gate (clang-format, ruff, pyright, pytest, gcovr) plus the OTA tool's runtime deps are pinned in requirements-dev.txt; installed into the project-root .venv, every gate guard discovers it automatically. CI's image installs the same file plus pioarduino and intelhex on top.
• Remotes: the canonical repo is the self-hosted Gitea, which push-mirrors to GitHub automatically — a single origin (the Gitea URL) is all a working clone needs. A clone from GitHub works too; it just cannot push.
• Per-deployment files are not in the repo (git-ignored): ota/secrets.yaml holds all broker and device credentials in one file — copy it from your other machine or recreate it from the template in ota/README.md. The CA bundle (ota/mosq-ca.crt) regenerates automatically from the system trust store. Both are only needed to run OTA or to provision a device; building and testing the firmware needs neither.

Release gate

scripts/release_check.py chains its steps fail-fast: check that the pinned tooling is the tooling installed (deps_check.py), then the quick guards - clang-format + final-newline (format_check.py), no key, address or private host in a tracked file (secret_check.py), ruff lint (lint_check.py), actionlint + zizmor over the CI workflows (workflow_check.py), pyright strict (typecheck_check.py) and pytest (pytest_check.py) - then the builds, the native test suite, and static analysis in two steps: cppcheck over every environment, then clang-tidy over the check_* ones, which are the same builds with the framework include paths clang-tidy needs to parse anything. cppcheck runs with --fail-on-defect low/medium/high, so a single defect of any severity fails the gate, unlike a bare pio check, which reports SUCCESS even with findings. Every guard is required: a missing tool fails the gate rather than skipping, so set up the dev venv (above) before running it. The guards run their tools out of the environment deps_check.py checked - the project .venv, or the interpreter the gate runs under when there is none - and an unrelated copy earlier on PATH cannot stand in for one. Step 0 checks the git tree with the same rule the firmware uses for its GIT_DIRTY flag.

python3 scripts/release_check.py                  # whole gate; a dirty tree is a warning
python3 scripts/release_check.py --strict         # release mode: a dirty tree fails immediately
python3 scripts/release_check.py --sync           # install the pinned tooling into .venv first
python3 scripts/release_check.py --group host     # the quick guards alone, without PlatformIO
python3 scripts/release_check.py --group esp32    # one CI group of it (scripts/gate_groups.py)

Every group checks the dependencies first, so --sync works with any of them. Each step streams its output unchanged (through a pseudo-terminal, colors included) and keeps a full log under .pio/gate-logs/. The run ends machine-friendly: a summary table, the last lines of the failed step, and a final RELEASE CHECK: PASS|FAIL (step: <name>) marker line - tail is enough to know everything. Exit code is 0 only on a fully clean run.

scripts/analysis_check.py does not trust pio check to tell whether clang-tidy ran: that passes a crashed tool and drops clang's own errors. So scripts/pio_tidy.py keeps PlatformIO's compiler configuration but replaces its clang-tidy driver. Each input runs as its own process, with a timeout and a log under .pio/analysis/, and passes only on an exit status and dependency output that show it parsed its input; every library header gets a pass of its own. A clang error in project code fails the gate, as clang-tidy skips whatever depends on it. A file written for another target is left out of an environment by PLATFORM_EXCLUSIONS, with the reason; the files left out are still parsed, an entry none of whose files fails to parse any more fails as stale, and so does a file left out of every environment. The number of tidy processes follows the CPUs and the memory the gate may use (build_jobs.analysis_jobs()).

CI runs the gate split into the groups of scripts/gate_groups.py, one job each, and generates its matrix from them; the gate fails when the groups leave a default or check_* environment out or cover one twice. A push that changes nothing but Markdown runs the host group alone, which checks the format of the Markdown and scans it for secrets: no firmware can differ. When the push's base cannot be fetched and compared, everything runs. The jobs run in the image ci/toolchain.Dockerfile builds: the pinned Python tooling, and PlatformIO with every toolchain in a venv of its own. CI builds and pushes it whenever what decides its contents changes (scripts/toolchain_key.py). A registry that answers the lookup with an error other than a missing or refused image fails the job instead of starting a rebuild.

A firmware group's job hands the sizes its build measured to the size comparison (--sizes FILE), which then builds only the baseline. On a release tag, the ESP groups also pass the firmware they checked to the release job through the cache both forges provide, with a manifest naming the commit, the group and each binary's SHA-256; the release job refuses a bundle that does not match, and builds through the gate when there is none.

CI also runs scripts/coverage_check.py: the native tests, instrumented, and a gcovr report over lib/ under .pio/coverage/, leaving out the upstream DFPlayerMiniFast driver. It fails when the line coverage falls below the floor in ci/coverage-baseline.json, which only a reviewed commit changes; --record-baseline writes a floor from a local measurement for that review.

Gotchas

• Cross-project reflash: a running firmware only accepts an OTA image that names its own environment, so converting a board to another project needs a one-time serial flash.
• CAN IDs are stored in EEPROM (CRC-protected). A node with none announces itself and is given one over the gateway's can subtopic (see "MQTT scheme"). Building once with NEW_CAN_ADDRESS defined in platformio.ini (master ID is MASTER_CAN_ADDRESS=10) and then removing it again writes the pair directly — that is the route for a node that already has an address, since only a node still waiting for one is on the commissioning list.