Detect repeated whole-hour meter registers, and announce state over MQTT

[gskjold]

Fixes #1119. Implements #1128, which covers the notification asked for in #1120.

Reject repeated whole-hour registers (#1119)

Some meters publish the previous whole hour's accumulated registers again at the next whole hour — identical registers and an identical meter clock, while the instantaneous values in the same payload are current. Confirmed on three Aidon meters; the four captured payloads are already in the corpus as gh1119-1..4.hex:

Payload logged at	Meter clock inside	1.8.0 A+ (kWh)	Delta	1.7.0
16:00:11	16:00:00	109552.248	—	4027 W
17:00:11	17:00:00	109554.016	1.768	319 W
18:00:11	17:00:00	109554.016	0.000	12633 W
19:00:11	19:00:00	109562.072	8.056	763 W

Stored history got a zero hour followed by a double hour, which destroys the month's tariff peak until the month rolls over.

The 2.3 workaround never took effect on history. AmsData::apply() substituted the integrated estimate into meterState, but handleDataSuccess() passes the raw packet to ds.update() and meterState is never used to build history. That is why #1119 reports the workaround "does not work".

Detection

AmsData::isStaleCounter(), meter-agnostic. A List ≥ 3 packet is a repeat when its meter timestamp equals the last accepted one, the registers are unchanged, and the meter's own observed register cadence is at least 30 minutes. After two consecutive rejections the reading is accepted anyway, so a frozen clock cannot suppress history indefinitely.

The cadence is learned from the meter rather than compared against the system clock: adjustForKnownIssues() applies a flat -3600 for Aidon that is only correct outside DST, so an absolute comparison would misfire for half the year. Learning the step also keeps meters that publish registers every few seconds out of scope — their clock advances between updates, so they never match.

Value used for the affected hour

• Uptime over an hour: the integrated estimate in meterState covers the whole hour, so the storage update is fed the estimate. The next hour's delta is measured against the estimate, so the two-hour total closes exactly on the meter's own register and the estimation error is absorbed by the second hour.
• Uptime under an hour: no reliable baseline, so nothing is stored. At the next whole hour the gap is ~7200 s, which clears AmsDataStorage's < 4000 test and lands in its existing "average" branch, distributing the delta evenly across both hours.

AmsDataStorage is unchanged. If the estimate ever overshoots by more than the whole of the next hour's consumption, its existing day.activeImport > importCounter guard stores that hour as 0 and snaps the counter back — bounded, and it heals on the following hour.

The Aidon-specific gate, the exact-double counter compare and the diff < 1.0 case are gone. That last one accepted the stale value whenever the hour's estimate was below 1 kWh, i.e. most nights.

MQTT

All four handlers withhold the repeated accumulated registers. The instantaneous values in the same payload are current and still go out — RawMqttHandler skips publishList3(), JsonMqttHandler publishes as List 2, HA and Domoticz extend their existing estimated-counter gate.

The flag describes the packet most recently applied and is cleared by the next sub-hour packet, so the state snapshots the handlers compose in stateUpdate mode do not inherit it for the rest of the hour.

Announce service state and events over MQTT (#1128)

<topic>/services — retained state snapshot, built from the same AmsJsonGenerator::generateServicesJson() the web UI is served from, republished on change or once a minute. Byte for byte what the web UI receives:

{"services":[{"k":"han","s":3,"e":90,"d":"AIDON_V0001"},
             {"k":"mqtt","s":1,"e":0,"d":"192.168.1.10"}, ...]}

The aggregate problem flag rides in the system payload (<topic>/state for HA, <topic>/system for JSON) rather than on the services topic, so that topic is nothing but the array.

<topic>/event — one message per transition, not retained:

Event	Trigger
hourly_data_substituted	the repeat above, with the affected local hour and fill: estimate or split
han_no_data	nothing arriving on the HAN port (#1120, #1122)
han_undecodable	frames arrive but cannot be decoded (#1241, #1245)
han_error	any other HAN error
han_restored	back to no error
price_error / price_ok	PriceService transitions

The raw error code travels with the event rather than being mapped to a category in firmware — the codes are already documented and translated for the UI. HAN events are rate limited to one per 30 s on top of the debounce already in AmsData::getLastError().

Home Assistant

• avty_t on every discovered entity, so all entities go unavailable when the reader drops off the broker.
• A device_class: problem, entity_category: diagnostic binary sensor reading problem from <topic>/state — the one entity a notification automation can trigger on, which is what Feature request: Notification in HA payload when no data from the meter #1120 is really asking for.
• An event entity on <topic>/event declaring all seven event types, so the transitions are visible in HA rather than only to an mqtt trigger. The MQTT event platform reads event_type natively and turns the remaining keys into attributes, which is why the payload names the field that way — no value_template needed.

Two things that differ from the plan in #1128

• <topic>/status was already taken by the retained online/offline availability payload backed by the last will, so the snapshot uses /services. The upside: avty_t was a single line, since the availability topic already existed.
• The generator moved out of AmsWebServer. buildServicesJson() stopped being web-only once the announcements rendered from it, which left the MQTT path reaching through the web server for state it does not own. It is now AmsJsonGenerator::generateServicesJson() alongside hanState() and mqttHandlerState(), with a ServiceStatusContext carrying the services to report on so neither caller depends on the other. AmsWebServer keeps computeServicesAggregate(), which is web-only, but builds it from the moved helpers.

One correction made along the way

An earlier revision trimmed the d detail fields from the MQTT services payload, on the belief that the 256 byte MQTTClient buffer on ESP8266 caps outbound payloads. It does not. lwmqtt_encode_publish() writes only the fixed header and topic into the write buffer, and lwmqtt_write_to_network() then streams the payload straight to the socket — the 256 bytes cap inbound messages, which is what dropOverflow(true) is for. It is also why the ~400 byte HA discovery configs have always published fine from ESP8266. The trimming and its withDetail flag are gone.

Verification

Six new native tests over the gh1119 fixtures in test/test_decoder/test_stale_counter.cpp:

• the fixtures still carry the defect (third payload repeats the second's clock and registers while its instantaneous power differs; fourth is two hours on with an 8.056 kWh delta)
• the detector flags the third payload and only the third
• a repeat leaves the integrated estimate and the estimated flag untouched
• it gives up after two consecutive rejections
• it stays out of the way until an hourly cadence has been observed
• the flag clears on the next sub-hour packet, so composed snapshots keep their counters

Assertions are relative between fixtures — decodeCosemDateTime() uses mktime(), so absolute values depend on the host timezone.

pio test -e native: 23 cases, all pass, golden sweep unchanged.
Builds: esp8266, esp32, esp32s2, esp32s2psram, esp32c3, esp32s3, esp32solo. The CUSTOM_MQTT_HOST and ENERGY_SPEEDOMETER_PASS branches were compile-checked with forced defines.
ESP8266 flash: 98.0% → 98.4% of the 1 MB budget.

Not covered

EnergyAccounting offers only now - 3600 to updateMax() when it rolls the hour. When the affected hour straddles local midnight the earlier hour belongs to the previous day and its corrected value is never offered as a peak candidate for that day. Left as a known gap in #1119 — updateMax() only ever raises a peak, so re-offering would be idempotent if we want to close it.

AmsDataStorage and EnergyAccounting are Arduino-only, so the storage behaviour needs a device check: after a stale hour the day plot should show a plausible value for both hours instead of a zero followed by a double, and the month peak should not jump.

🤖 Generated with Claude Code

[github-actions]

🔧 PR Build Artifacts

Version: 60ecfa3

All environments built successfully. Download the zip files:

• esp32s2: Download esp32s2.zip
• esp32s3: Download esp32s3.zip
• esp32c3: Download esp32c3.zip
• esp32: Download esp32.zip
• esp32solo: Download esp32solo.zip
• esp8266: Download esp8266.zip

Artifacts expire after 7 days. View workflow run