From c53a600fb174d5f7de52cfeb1d7b312f30154fdc Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Fri, 18 Sep 2026 19:18:20 +0000 Subject: [PATCH 1/3] docs(runbook): the A1437 is fitted, the design figures moved, and the clock stepped (#454) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The cell went into `prometheus` on 2026-09-18. This page still said it was in hand and nothing was fitted, so the status banner, the Observed column and steps 6 through 9 are now the record of what happened rather than a plan. TWO THINGS THE PAGE HAD WRONG, both found by doing it. The step 1 table marked `charge_full_design` and `voltage_min_design` "must not move". Both moved: 6.6 -> 6.8 Ah and 11.21 -> 11.4 V. They are figures the pack's own gas gauge reports, not properties of the model, and an aftermarket A1437 reports its own. The expectation is inverted here rather than restated, because on a machine that exports no serial number that movement is the only proof the part changed at all — `manufacturer` and `model_name` came back byte-identical, `SMP` and `bq20z451`, exactly as step 7 warned. Step 7's three proof rows become four, and the new one is the strongest of them. Disconnecting the cell clears the RTC. The post-swap boot opens `2026-07-28 15:04:45`; systemd-timesyncd restored the clock to 14:53:37, the second the machine went down, and the stack then ran about fourteen minutes writing samples at backdated timestamps before NTP stepped it forward to the true 18:12. So step 6's "measure the hole" query returns a confident wrong answer after a battery swap: the 184-minute gap it shows is not downtime, and fourteen minutes of real post-swap operation sits inside it. The step now says to take the window from `journalctl --list-boots`. Whether anything should notice a backward boot is #519 — `HostClockSkew` reads the kernel's current offset, which stayed at 0.004 s throughout, while `node_timex_sync_status` was 0 for exactly the backdated window and is read by no rule. STEP 2 WAS NEVER PERFORMED, and it can never be performed for this swap: the old cell is gone. `HostOnBattery` had never fired for `prometheus` across the whole 30-day retention. What fired on the day was accidental — the machine came back unplugged at 41 % — so the discrimination step 2 was written to buy, a step 8 failure being the cell or the adapter and nothing else, is unavailable. That is recorded in *What is still open* rather than glossed. STEP 8 IS STILL OWED and is the whole of what keeps #454 open. The pack was still charging when the fit was recorded. Also: the old cell went for recycling the same day (step 9); the window ran to about three hours and eighteen minutes against a stated bound of two; the firing set came back identical to the step 1 snapshot, all four alerts pre-existing; and the first *What is still open* bullet is deleted because #506 already fixed what it described, with its closure recorded at the end of the section instead. Verified: `npx markdownlint-cli2` clean; `python3 scripts/check_docs.py` prints docs OK — 83 Prometheus + 18 Loki rules, 7 dashboards, 141 panels, 10 assertions. Every reading quoted here was read from this host's own Prometheus on 2026-09-18, not remembered. Refs #454 Refs #519 Co-Authored-By: Claude Opus 5 --- docs/runbooks/replace-the-laptop-cell.md | 212 +++++++++++++++++------ 1 file changed, 155 insertions(+), 57 deletions(-) diff --git a/docs/runbooks/replace-the-laptop-cell.md b/docs/runbooks/replace-the-laptop-cell.md index 6be51468..746868e6 100644 --- a/docs/runbooks/replace-the-laptop-cell.md +++ b/docs/runbooks/replace-the-laptop-cell.md @@ -3,15 +3,30 @@ **One glued-in cell, one power-down of the host that watches everything else, and one test that only works with the machine running.** -> **Status — 2026-09-18: the cell is in hand and nothing is fitted. The -> baseline below is measured, not remembered.** +> **Status — 2026-09-18: the cell is fitted. Step 8, the mains pull on the new +> cell, has not been run, so +> [#454](https://github.com/Gerrrt/HomeLab/issues/454) stays open.** > -> Every reading in step 1 was taken from this host's own Prometheus on -> 2026-09-17 and re-read unchanged on 2026-09-18, the day the cell arrived: -> `charge_full` 6.196 Ah of 6.6 Ah design (94 %), unmoved across the retained -> window, 108 cycles, `status` `Full`, on mains via `ADP1`. Nothing in -> `host.rules.yaml` is firing for `prometheus`. The baseline table is therefore -> current as it stands — it does not need retaking before the fit. +> The host was down from 14:53:37 to about 18:12 UTC — roughly three hours and +> eighteen minutes, over the two-hour bound this page sets below, and *not* the +> 184 minutes the series appear to show. Step 6 has why. +> +> The new pack reads `charge_full` 6.889 Ah against a `charge_full_design` of +> 6.8 Ah (101 %), `cyclecount` 1, `present` 1, `status` `Charging`. The cell it +> replaced, measured 2026-09-17, read 6.196 Ah of 6.6 Ah (94 %) after 108 +> cycles, and went for recycling the same day it came out. **Both design +> figures moved** — 6.6 → 6.8 Ah, and `voltage_min_design` 11.21 → 11.4 V — +> which step 1's table said they could not. The correction is beneath that +> table. +> +> **Step 2 was never performed, and now cannot be.** The alert path was not +> proved on the old cell before it came out: `HostOnBattery` had never fired +> for `prometheus` across the whole 30-day retention. What fired on 2026-09-18 +> was accidental — the machine came back from the swap unplugged at 41 %, so +> `HostOnBattery` went pending at 14:56 and firing at 14:57 *on the skewed +> clock*, for `prometheus` and nothing else, with `UpsOnBattery` quiet +> throughout. That is the discriminator step 2 wanted, arrived at by accident. +> It is not the bounded test, and it measured no runtime. > > `HostBatteryHealthLow` fired for **`oracle`** from 2026-09-14 and was > silenced on 2026-09-17 until 2026-10-08 @@ -40,9 +55,9 @@ network is enough for both mains-pull tests. ## Why this is not urgent, and why it is not nothing -The cell reads **94 % of design capacity after 108 cycles**, which is *above* -`HostBatteryHealthLow`'s 80 % line. **No alert asked for this.** It is bought -on age and on the failure mode, which is the exception +The cell that came out read **94 % of design capacity after 108 cycles**, +which was *above* `HostBatteryHealthLow`'s 80 % line. **No alert asked for +this.** It was bought on age and on the failure mode, which is the exception [`../roadmap.md`](../roadmap.md) names in the same breath as its *Never* line: a consumable whose failure is a safety or availability event is not an upgrade. @@ -106,6 +121,11 @@ The two battery runbooks beside this one did not need this section, because tightest staleness threshold that matters here is 90 minutes, so a window under two hours costs at most one late job on the way back. + **2026-09-18 ran to about three hours and eighteen minutes, not two** — + 14:53:37 to roughly 18:12. Solvent soak and the clock confusion in step 6 are + where it went. Budget three and a half hours and treat two as the target + rather than the observed. + ## 1. Record the baseline — before touching anything This is the step that makes the fix provable rather than assumed. The UPS pack @@ -152,27 +172,43 @@ curl -sS http://localhost:9090/api/v1/alerts | python3 -c 'import json,sys; [print(a["labels"].get("alertname"), a["labels"].get("instance"), a["state"]) for a in json.load(sys.stdin)["data"]["alerts"]]' ``` -The **Reads** column was measured on 2026-09-17 and is the baseline; fill -**Observed** in step 7. +The **Reads** column was measured on 2026-09-17, on the cell that came out. +**Observed** was filled on 2026-09-18, minutes after the fit and part-way +through the first charge — `status` was `Charging` and `capacity` still +climbing — so every row there is a first reading rather than a settled one. +`charge_full` in particular is the gauge's figure before it has learned this +pack across a cycle; step 8 is what settles it. -| Metric | Reads 2026-09-17, thirteen-year-old cell | After a good new cell | Observed | +| Metric | Reads 2026-09-17, thirteen-year-old cell | After a good new cell | Observed 2026-09-18 | | --- | --- | --- | --- | -| `node_power_supply_charge_full` | `6.196` | at or near `6.6` | | -| `node_power_supply_charge_full_design` | `6.6` | `6.6` — must not move | | -| `charge_full / charge_full_design` | `0.9388` | `0.98`–`1.0` | | -| `node_power_supply_cyclecount` | `108` | low — `0` to a handful | | -| `node_power_supply_charge_ampere` | `6.12`, drifting in the last digit | any value that **moves** | | -| `node_power_supply_capacity` | `93` | rises to `100` on charge | | -| `node_power_supply_current_ampere` | `0` | non-zero while discharging | | -| `node_power_supply_voltage_volt` | `12.43` on mains (24h span `12.425`–`12.438`) | `10.9`–`12.6`, and varying under load | | -| `node_power_supply_voltage_min_design` | `11.21` | `11.21` — must not move | | -| `node_power_supply_temp_celsius` | `32.7`–`39.2` over 24h, averaging `33.3` | a similar band, never far above it | | -| `node_power_supply_present` | `1` | `1` | | -| `node_power_supply_online{power_supply="ADP1"}` | `1` | `1` | | -| info `manufacturer` | `SMP` | may or may not change | | -| info `model_name` | `bq20z451` | may or may not change | | -| info `status` | `Full` | `Charging`, then `Full` | | - +| `node_power_supply_charge_full` | `6.196` | at or near its own design figure | `6.889` | +| `node_power_supply_charge_full_design` | `6.6` | **may move — see below** | `6.8` | +| `charge_full / charge_full_design` | `0.9388` | `0.98`–`1.0`, or above it | `1.013` | +| `node_power_supply_cyclecount` | `108` | low — `0` to a handful | `1` | +| `node_power_supply_charge_ampere` | `6.12`, drifting in the last digit | any value that **moves** | `3.48`, climbing | +| `node_power_supply_capacity` | `93` | rises to `100` on charge | `51` and climbing | +| `node_power_supply_current_ampere` | `0` | non-zero while discharging | `1.677`, charging | +| `node_power_supply_voltage_volt` | `12.43` on mains (24h span `12.425`–`12.438`) | `10.9`–`12.6`, and varying under load | `12.607` | +| `node_power_supply_voltage_min_design` | `11.21` | **may move — see below** | `11.4` | +| `node_power_supply_temp_celsius` | `32.7`–`39.2` over 24h, averaging `33.3` | a similar band, never far above it | `25.5`, one sample | +| `node_power_supply_present` | `1` | `1` | `1` | +| `node_power_supply_online{power_supply="ADP1"}` | `1` | `1` | `1` | +| info `manufacturer` | `SMP` | may or may not change | `SMP` — unchanged | +| info `model_name` | `bq20z451` | may or may not change | `bq20z451` — unchanged | +| info `status` | `Full` | `Charging`, then `Full` | `Charging` | + +> **The two "must not move" rows were wrong, and their moving is the strongest +> proof available that the pack is genuinely different.** This page asserted +> that `charge_full_design` (`6.6`) and `voltage_min_design` (`11.21`) are +> properties of the model and therefore fixed. They are not. They are figures +> the pack's own gas gauge reports, and an aftermarket A1437 reports its own: +> on 2026-09-18 they read `6.8` Ah and `11.4` V. **Read the expectation the +> other way round from here — a design figure that moves is evidence of a +> different pack, and one that does not is evidence of nothing.** That matters +> more on this machine than it would elsewhere, because it exports no serial +> number (step 7) and `manufacturer` and `model_name` came back byte-identical, +> `SMP` and `bq20z451`, exactly as step 7 warned they would. +> > **Three names in this family are not what you would guess, and a wrong one > returns an empty result — which on a command line is indistinguishable from > "the cell is gone".** The cycle count is `node_power_supply_cyclecount`, one @@ -190,6 +226,10 @@ what is being compared against; the kernel agreeing with itself proves less. ## 2. Prove the alert path on the cell you are about to throw away +> **Not done on 2026-09-18, and it can never be done for this swap — the old +> cell is gone.** Read this step as written before reusing the page on +> `oracle`; what it costs to skip is in *What is still open*. + A lithium cell should come out discharged rather than full, and the only way to discharge this one is to run the machine on it. That is the mains-pull test, so do it **now, on the old cell**, and get a proven alert path out of the same @@ -352,11 +392,35 @@ Then, in order: python3 -c 'import json,sys; [print(a["labels"].get("alertname"), a["labels"].get("instance"), a["state"]) for a in json.load(sys.stdin)["data"]["alerts"]]' ``` + On 2026-09-18 it came back identical to step 1's snapshot: + `ScheduledJobFailed` (`backup-volumes`, exited 2), `SecretsKeyBackupUnproven`, + `RebootRequired`, and `GatewayMonitorUnreliable` (`morpheus`, `WAN_DHCP6`). + All four had been firing continuously for the four days before the swap, so + none of them is this work. `HostBatteryNotReported` stayed quiet and every + target came back up. The reboot in step 4 was not taken, so `RebootRequired` + is still firing rather than cleared. + - The off-host healthcheck is green again within about ten minutes of Alertmanager starting. If it is still red after that, it is [`verify-the-alert-path.md`](verify-the-alert-path.md) and not this runbook. - **Measure the hole, and measure which half of it filled in.** One query, - twice, against two jobs that behave differently: + twice, against two jobs that behave differently. + + > **After a battery swap, do not read this query as the window.** + > Disconnecting the cell clears the RTC. On 2026-09-18 the post-swap journal + > boot opens `2026-07-28 15:04:45`; `systemd-timesyncd` restored the clock to + > `14:53:37` — the second the machine went down — and the stack then ran about + > fourteen minutes writing samples at **backdated** timestamps, 14:54:30 to + > 15:08, before NTP stepped the clock forward to the true 18:12. So the TSDB + > holds a 184-minute hole from 15:08 to 18:12 that is *not* downtime, and + > fourteen minutes of real post-swap operation filed inside the outage. The + > true window was 14:53:37 to about 18:12. **Take it from + > `journalctl --list-boots` and the recorded shutdown, not from + > `query_range`.** Nothing noticed the backdated boot — `HostClockSkew` reads + > `node_timex_offset_seconds`, which is small once timesyncd has restored a + > wrong but stable clock — and whether anything should is + > [#519](https://github.com/Gerrrt/HomeLab/issues/519). A further reboot at + > 18:28 is in the series too and is not part of the swap. ```bash for j in snmp oracle-metrics; do @@ -370,8 +434,9 @@ Then, in order: done ``` - `snmp` is scraped by Prometheus and its hole is the true length of the - window. `oracle-metrics` arrives by remote write and may be shorter, because + With the window taken from the journal, this still separates the two jobs' + behaviour. `snmp` is scraped by Prometheus, so its hole is the shape of the + outage. `oracle-metrics` arrives by remote write and may be shorter, because that agent's WAL retried. A shorter `oracle-metrics` hole is the backfill working; an equal one means it did not. Either way the number is now known rather than assumed. @@ -382,7 +447,8 @@ Then, in order: ## 7. Confirm the metrics actually moved -Re-run step 1's loop and fill the **Observed** column. +Re-run step 1's loop and fill the **Observed** column. It was filled on +2026-09-18 and the readings are there; what follows is how they were judged. > **There is no serial number to appeal to.** `shiva`'s pack was proven by > `cpqHeSysBatterySerialNumber` changing — the one row that could not be the old @@ -392,19 +458,26 @@ Re-run step 1's loop and fill the **Observed** column. > Dell does carry one, which is worth knowing when this runbook is reused > there.) Worse, an aftermarket A1437 commonly reuses the same gas gauge, so > `SMP` / `bq20z451` reading identically afterwards proves nothing either way. -> The honest proof is three rows together, not one: **`cyclecount` falling from -> `108` to something low, `charge_full` rising toward `6.6`, and -> `charge_ampere` moving across a charge cycle.** A reading that is -> byte-for-byte identical to the baseline means the machine is reporting the old -> pack's stored values — treat that as "the new cell is not seen", not as "the -> numbers happen to match", and go back to the connector. +> The honest proof is several rows together, not one. On 2026-09-18 it was +> **four**: `cyclecount` falling from `108` to `1`, `charge_full` rising from +> `6.196` to `6.889`, `charge_ampere` moving across a charge cycle, and — the +> row this page did not know it had — **both design figures changing**, +> `charge_full_design` `6.6` → `6.8` and `voltage_min_design` `11.21` → `11.4`. +> That last one turned out to be the strongest of them, and the table above now +> says so. A reading that is byte-for-byte identical to the baseline means the +> machine is reporting the old pack's stored values — treat that as "the new +> cell is not seen", not as "the numbers happen to match", and go back to the +> connector. Two more failure shapes worth naming: -- **`charge_full` reads exactly `6.6` and never moves.** That is the gauge - reporting its design figure because it has not learned a capacity yet, not a - cell at 100 % health. It becomes a measurement after one full charge and one - substantial discharge, and step 8 provides the discharge. +- **`charge_full` reads exactly its own `charge_full_design` and never moves.** + That is the gauge reporting the design figure because it has not learned a + capacity yet, not a cell at 100 % health. It becomes a measurement after one + full charge and one substantial discharge, and step 8 provides the discharge. + The 2026-09-18 reading is not this case: `6.889` is *above* the `6.8` design + figure, so the gauge is reporting something it measured rather than something + it was told. - **`HostBatteryHealthLow` fires for `prometheus`.** The new cell is measurably worse than the thirteen-year-old one it replaced. That is a return to the seller, not a finding to write up. @@ -415,6 +488,13 @@ Two more failure shapes worth naming: > [#454](https://github.com/Gerrrt/HomeLab/issues/454). Everything before it > makes a claim about capacity. Only this one makes the claim the cell was > bought for. +> +> **Not done as of 2026-09-18.** The pack was still charging when the fit was +> recorded — `capacity` 51 % and rising — and this step needs it full. Until it +> runs, the property the cell was bought for is untested and that issue stays +> open. The accidental discharge on the day is not a substitute: it began at +> 41 %, was not bounded, measured no runtime, and its timestamps are the +> backdated ones step 6 describes. Same procedure as step 2, now on the new cell and with the machine fully charged. Because the numbers are finally meaningful, also **measure the runtime @@ -453,7 +533,8 @@ note that it is one measurement at one load on one day. the removed one there keeps the hazard and gives up the capacity. - Keep it in a non-flammable container, away from the rack and away from anything that burns, and take it to a council household-waste battery point or - a retailer take-back **within days, not months**. Note the date. + a retailer take-back **within days, not months**. Note the date. **The cell + removed on 2026-09-18 went for recycling the same day.** - **Never post it.** A damaged lithium cell is not a mailable item. - If it came out swollen or vented: outside, metal container, same day, and do not apply solvent to it. @@ -485,15 +566,23 @@ host being down. [`verify-the-alert-path.md`](verify-the-alert-path.md). ## What is still open -- **`HostBatteryHealthLow`'s description names - [`fit-the-ups-battery.md`](fit-the-ups-battery.md), not this file.** That was - the only runbook there was when the rule was written. Correcting it changes - `stacks/observability/prometheus/rules/host.rules.yaml` *and* the - `exp_annotations` block in - `stacks/observability/prometheus/tests/host.test.yaml`, which asserts the - string exactly — a rules-and-tests change rather than a documentation one, so - it is not made here. A reader following the annotation lands on that - runbook's laptop-cell paragraph, which links here. +- **Step 8 has not been run, and it is the whole of what keeps + [#454](https://github.com/Gerrrt/HomeLab/issues/454) open.** Everything else + the issue asked for is done and measured. The cell needs a full charge first. +- **Step 2 can never be run for this swap: the old cell is gone.** The + discrimination it was written to buy — a step-8 failure being the cell or the + adapter and nothing else, because the path was already proven — is + unavailable. A failure in step 8 will be ambiguous between the pack, the brick + and the rule. The only evidence the path works is the accidental firing on + 2026-09-18, which did at least exercise the real rule against the real + adapter. Run step 2 properly when this page is reused on `oracle`. +- **A battery disconnect resets the RTC, and the TSDB records the result as a + hole that is not one** — [#519](https://github.com/Gerrrt/HomeLab/issues/519). + Nothing notices a backward boot: `HostClockSkew` reads the kernel's current + offset, which stays small once the clock is wrong but stable, and + `node_timex_sync_status` — which was `0` for exactly the backdated window — is + read by no rule. Step 6 says what to do instead; whether anything should alert + is that issue's question. - **`oracle`'s cell reads 72 %, is unbought, and its alert is silenced until 2026-10-08** — `01cb81d7-5e19-4e6d-b386-f5c8c843032b`, matching `alertname="HostBatteryHealthLow"`, `instance="oracle"`, @@ -507,9 +596,18 @@ host being down. [`verify-the-alert-path.md`](verify-the-alert-path.md). This runbook is written to be reused for that cell. The differences: its mains supply is `AC` and not `ADP1`; its firmware reports `cyclecount` as `0` - and always has, so the strongest of the three proof rows in step 7 is - unavailable there; it exports no `temp_celsius`; and its info series *does* - carry a `serial_number`, which is a proof row this machine lacks. + and always has, so one of step 7's proof rows is unavailable there; it exports + no `temp_celsius`; and its info series *does* carry a `serial_number`, which + is a proof row this machine lacks and the cleanest of them all. Whether that + Dell's clock survives the disconnect the way this MacBook's did not is + unknown and worth watching for — see [#519](https://github.com/Gerrrt/HomeLab/issues/519). - **Nothing watches runtime-on-battery continuously.** The figure step 8 produces is one measurement, at one load, on one day. There is no rule and no series that would notice it halving. + +**Closed since this page was written:** `HostBatteryHealthLow`'s description +named [`fit-the-ups-battery.md`](fit-the-ups-battery.md) — the rack pack in +`mjolnir`, not this cell — because that was the only runbook there was when the +rule was written. [#506](https://github.com/Gerrrt/HomeLab/pull/506) retargeted +it at this file on 2026-09-17, moving the `exp_annotations` blocks in +`stacks/observability/prometheus/tests/host.test.yaml` with it. From 324877fa1cda3fc09d070f56331c61b54c4db68e Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Fri, 18 Sep 2026 19:18:47 +0000 Subject: [PATCH 2/3] docs: the 94 % figure is the cell that came out (#454) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four documents restated `prometheus`'s cell as 94 % of design after 108 cycles in the present tense. That pack was removed on 2026-09-18 and recycled the same day. Each now says what the estate has: `oracle` at 72 % and unbought, `prometheus` replaced and reading 101 % of its design capacity at one cycle. `hardware.md` carries the durable record. The Accessories bullet goes from "delivered and not yet fitted" to fitted the same day, and the compatible-cell wording — written to be settled once someone could look at the pack — is settled sideways: the gas gauge reports identically, `SMP` and `bq20z451`, because an aftermarket A1437 reuses it, but both design figures moved, which is the only proof of a different part on a machine that exports no serial. The forward-looking checklist becomes what was checked at the fit and what was not, with the mains pull named as the reason #454 is still open. The mains-cut dependency paragraph gains the sentence that is actually load-bearing: how long either laptop runs on its cell has never been measured. `security.md`'s mains-power row said the same thing and now ends on the same limit — the row is answered as far as the cells being healthy and no further. `fit-the-ups-battery.md`'s laptop-cell paragraph delegates to the other runbook; it now records that the capacity half was proved at the fit and the mains pull was not. `roadmap.md` gets four corrections. The 2026-09-13 purchase paragraph pointed at `fit-the-ups-battery.md` for the test that would settle the issue — the rack pack in `mjolnir`, not this cell — which is the same wrong pointer #506 fixed in the alert's own description; it now names `replace-the-laptop-cell.md` and says it was wrong. A dated paragraph records the fit. The cell leaves "already paid for and still moving" into neither list, because those two paragraphs track money that has not yet become a working part and this has. The *Never* carve-out keeps its 94 % measurement, scoped to the cell that was there then, and records the exception being exercised rather than restated. THE BUY TABLE DOES NOT MOVE. The cell left it on 2026-09-13 and fitting is not buying, so README's "two items now, one later" is unaffected. There is also NO `## Done` ENTRY: #454 is open until step 8 runs, and the precedent for work landed against an open issue is #418, which is recorded as dated progress paragraphs rather than a checkbox. Verified: `python3 scripts/check_docs.py` prints docs OK — 83 Prometheus + 18 Loki rules, 7 dashboards, 141 panels, 10 assertions; `npx markdownlint-cli2` clean across all 92 files. Refs #454 Co-Authored-By: Claude Opus 5 --- docs/hardware.md | 43 +++++++++++++++--------- docs/roadmap.md | 49 +++++++++++++++++++++------- docs/runbooks/fit-the-ups-battery.md | 14 +++++--- docs/security.md | 2 +- 4 files changed, 76 insertions(+), 32 deletions(-) diff --git a/docs/hardware.md b/docs/hardware.md index b9740078..667ae263 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -24,7 +24,10 @@ dependency of the mains-cut path and is watched as one: Alloy's node collector exports `node_power_supply_*` from both, and `host.rules.yaml` alerts when the shelf is off mains, when a cell falls below 80 % of its design capacity, and when a laptop reports no cell at all -([#454](https://github.com/Gerrrt/HomeLab/issues/454)). +([#454](https://github.com/Gerrrt/HomeLab/issues/454)). `prometheus`'s cell was +replaced on 2026-09-18 and reads 101 % of its design capacity at one cycle; +`oracle`'s is the original, reads 72 %, and is unbought. How long either laptop +actually runs on its cell has never been measured. The patch panel and the PDU were listed the other way round here until 2026-08-29. U8 is the panel and U7 is the PDU, confirmed against the rack. @@ -403,7 +406,7 @@ revisions of this repository treated `shiva` as the hypervisor itself. ([#76](https://github.com/Gerrrt/HomeLab/issues/76)) - A1437 battery cell for `prometheus`[^A1437] — the pack that fits the `A1425`, the late-2012 Retina 13" in the Compute table — bought new - 2026-09-13, **delivered 2026-09-18 and not yet fitted.** A consumable and + 2026-09-13, delivered 2026-09-18 and **fitted the same day.** A consumable and not an upgrade: it is the one exception the roadmap's *Never* line names, bought because the estate's mains-cut path rests on this cell and the failure mode of a @@ -411,19 +414,29 @@ revisions of this repository treated `shiva` as the hypervisor itself. ([#454](https://github.com/Gerrrt/HomeLab/issues/454)). The listing calls it genuine and its brand field says unbranded, so it is recorded as a compatible cell, not an Apple part. That wording was written to be settled - once the part could be looked at, which it now can: the cell arrived - 2026-09-18 and this line stays as it is until someone reads the pack's own - markings and says otherwise. The cell it - replaces read 94 % of design capacity after 108 cycles on 2026-09-12, which - is above `HostBatteryHealthLow`'s 80 % — it is bought on age, not on the - alert. Nothing in the Compute table changes; a cell is not a spec. - **Checked at the fit, not assumed:** `charge_full` at or near - `charge_full_design` and a cycle count reading low, then mains pulled and - the host staying up with `HostOnBattery` firing — the property the cell is - there for, and untested since the machine was commissioned - ([`replace-the-laptop-cell.md`](runbooks/replace-the-laptop-cell.md), which - also carries the stack-down window and the old cell's disposal). `oracle`'s - cell reads 72 % and is second in line, unbought. + once the part could be looked at, and the fit settled it sideways. + `manufacturer` and `model_name` came back unchanged — `SMP`, `bq20z451` — + because an aftermarket A1437 reuses the same gas gauge; but **both design + figures moved**, `charge_full_design` 6.6 → 6.8 Ah and `voltage_min_design` + 11.21 → 11.4 V, which is a pack reporting its own numbers and is the only + proof of a different part available on a machine that exports no serial. It + stays recorded as a compatible cell. The new pack reads `charge_full` + 6.889 Ah of 6.8 Ah design (101 %) at `cyclecount` 1. The cell it replaced + read 94 % of design capacity after 108 cycles, on 2026-09-12 and again on + 2026-09-17 — above `HostBatteryHealthLow`'s 80 %, so it was bought on age and + not on the alert — and went for recycling on 2026-09-18. Nothing in the + Compute table changes; a cell is not a spec. + **Checked at the fit:** `charge_full` above `charge_full_design`, + `cyclecount` 1, `charge_ampere` moving, and both design figures changed — all + read from this host's own Prometheus on 2026-09-18. **Not checked, and the + reason [#454](https://github.com/Gerrrt/HomeLab/issues/454) is still open:** + the mains pull on the fully charged pack, which is the property the cell was + bought for and a runtime the estate has never had. The pack was still + charging when the fit was recorded. + ([`replace-the-laptop-cell.md`](runbooks/replace-the-laptop-cell.md) carries + the baseline, the stack-down window — 14:53 to about 18:12 on the day, over + its own two-hour bound — and the disposal.) `oracle`'s cell reads 72 % and is + second in line, unbought. - ViewSonic N1700W LCD, used as a rack console via the KVM - RJ45 Cat6 in-line couplers[^Couplers] - Cat6 patch cables[^Patchcables] diff --git a/docs/roadmap.md b/docs/roadmap.md index e412d9bf..d74a4414 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -62,9 +62,24 @@ three rows to two, and neither of them is a part: two licences and a drive kept at another address. What closes #454 is not the cell landing but the fit and the test after it — `charge_full` at or near design, a low cycle count, and mains pulled with the host staying up — which -[`fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) names. `oracle`'s -cell, the one that measures worse, stays unbought and second in line, as that -issue ranked it. +[`replace-the-laptop-cell.md`](runbooks/replace-the-laptop-cell.md) names. +This paragraph pointed at +[`fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) until 2026-09-18 — +the rack pack in `mjolnir`, not this cell — which is the same wrong pointer +[#506](https://github.com/Gerrrt/HomeLab/pull/506) corrected in the alert's own +description. `oracle`'s cell, the one that measures worse, stays unbought and +second in line, as that issue ranked it. + +**2026-09-18: the cell is fitted, and the issue stays open.** The A1437 went +into `prometheus` the day it was delivered; the host was down from 14:53 to +about 18:12 UTC. The new pack reads `charge_full` 6.889 Ah of 6.8 Ah design at +one cycle, and both of its design figures differ from the old pack's — an +aftermarket cell reports its own, which is the only proof of a changed part on +a machine that exports no serial number. **The buy table does not move**: the +cell left it on 2026-09-13, and fitting is not buying. What is still owed is +the mains pull on the charged pack and the runtime it measures, so +[#454](https://github.com/Gerrrt/HomeLab/issues/454) stays open and this gets +no *Done* entry. **Buy these, and the estate as decided is fully bought:** @@ -104,11 +119,12 @@ restore onto `trinity` needs installer stick — between them, the whole of what the rehearsal now waits on; the two Exos X20 18 TB drives for `smaug`'s mirror ([#413](https://github.com/Gerrrt/HomeLab/issues/413)); the CRS326 that -replaces `neo` ([#444](https://github.com/Gerrrt/HomeLab/issues/444)); the -A1437 cell that replaces `prometheus`'s thirteen-year-old one -([#454](https://github.com/Gerrrt/HomeLab/issues/454)); and two HP SFF drive -trays for `Saruman`, the carriers its SM863a pair needs to go in the bays -([#418](https://github.com/Gerrrt/HomeLab/issues/418)). +replaces `neo` ([#444](https://github.com/Gerrrt/HomeLab/issues/444)); and two +HP SFF drive trays for `Saruman`, the carriers its SM863a pair needs to go in +the bays ([#418](https://github.com/Gerrrt/HomeLab/issues/418)). The A1437 cell +for `prometheus` was on this list until 2026-09-18 and is on neither now: it is +fitted ([#454](https://github.com/Gerrrt/HomeLab/issues/454)). These two +paragraphs track money that has not yet become a working part, and that has. **2026-09-17: the trays were bought on 2026-09-11 and are only now written down.** They were paid for in the same sitting as the I226 card, the Exos pair @@ -201,9 +217,12 @@ is the point, not a problem to spend money on. **consumable whose failure is a safety or availability event** is not an upgrade. `prometheus`'s battery is the worked example ([#454](https://github.com/Gerrrt/HomeLab/issues/454)) — the estate's -mains-cut path depends on it, `host.rules.yaml` measures it since 2026-09-12 -(94 % of design after 108 cycles), and the failure mode of a thirteen-year-old -cell is a fire on a shelf. `oracle` has the same kind of cell and it measures +mains-cut path depends on it, `host.rules.yaml` has measured it since +2026-09-12 — 94 % of design after 108 cycles, on the cell that was there then — +and the failure mode of a thirteen-year-old cell is a fire on a shelf. **That +cell was bought 2026-09-13 and fitted 2026-09-18**, so this is the exception +being exercised rather than restated; the new pack reads 101 % of its design +capacity at one cycle, and the runtime it was bought for is still unmeasured. `oracle` has the same kind of cell and it measures worse, 72 %, so `HostBatteryHealthLow` fires for it first; it stays second in line only because `prometheus` is the host whose death is the estate going blind. The cell for `prometheus` was bought on 2026-09-13; nothing else about @@ -475,6 +494,14 @@ new since this file was last honest: catches a silently dead notification path, and it cannot report a mains cut, because the switch between the two laptops has no battery (#110). + **That last clause has been stale since 2026-09-08 and is corrected rather + than swapped.** The switch was racked in U4 on UPS power by + [#110](https://github.com/Gerrrt/HomeLab/issues/110), and each laptop rides a + cut on its own cell — measured since 2026-09-12 and, on `prometheus`, replaced + on 2026-09-18 ([#454](https://github.com/Gerrrt/HomeLab/issues/454)). So the + off-host check *can* report a mains cut, for exactly as long as `prometheus` + stays up on its cell, which is the one thing still unmeasured. + Two collection faults of the same kind were fixed in [#62](https://github.com/Gerrrt/HomeLab/pull/62): the agent was answering to the name of the server, and cAdvisor could only see its own cgroup. Both ran healthy diff --git a/docs/runbooks/fit-the-ups-battery.md b/docs/runbooks/fit-the-ups-battery.md index d57628e2..7fce3ba8 100644 --- a/docs/runbooks/fit-the-ups-battery.md +++ b/docs/runbooks/fit-the-ups-battery.md @@ -86,11 +86,13 @@ is measured rather than assumed ([#454](https://github.com/Gerrrt/HomeLab/issues/454)). No collector was needed: Alloy's node collector already exports `node_power_supply_*` from both hosts, and `host.rules.yaml` reads it — `HostBatteryHealthLow` when a cell's -`charge_full` falls below 80 % of `charge_full_design` (`prometheus` read 94 % -after 108 cycles, `oracle` 72 %), `HostOnBattery` when the mains adapter loses +`charge_full` falls below 80 % of `charge_full_design` (`oracle` reads 72 %; +`prometheus` read 94 % after 108 cycles until its cell was replaced on +2026-09-18, and now reads 101 % at one cycle), `HostOnBattery` when the mains +adapter loses input, `HostBatteryNotReported` when a laptop stops reporting a cell. The cell -for `prometheus`, an A1437, was bought 2026-09-13 and is recorded in -[`hardware.md`](../hardware.md#accessories). The swap and the tests that prove +for `prometheus`, an A1437, was bought 2026-09-13, **fitted 2026-09-18**, and +is recorded in [`hardware.md`](../hardware.md#accessories). The swap and the tests that prove it are [`replace-the-laptop-cell.md`](replace-the-laptop-cell.md), which is where the baseline, the disposal and the stack-down window are written down — that cell is glued into the machine running this stack, so fitting it blinds @@ -99,7 +101,9 @@ pack did not carry. What it proves is the same shape as step 5 here: `charge_full` at or near `charge_full_design` and the cycle count reading low, then that laptop's mains pulled with the host staying up and `HostOnBattery` firing — the property the cell is there for, and untested since the machine was -commissioned. +commissioned. The first half was proved at the fit on 2026-09-18: `charge_full` +above design, `cyclecount` 1. **The mains pull on the charged pack has still +not been run**, so that property remains untested and #454 is still open. ## Before you start diff --git a/docs/security.md b/docs/security.md index b92e8beb..29807667 100644 --- a/docs/security.md +++ b/docs/security.md @@ -19,7 +19,7 @@ What this network is actually built to survive: | Someone with the trusted Wi-Fi key quietly joining | Kea's lease log reaches Loki; `UnknownDeviceOnTrustedSegment` fires the first time a MAC appears on VLAN 50 in seven days ([ADR-0019](adr/0019-read-device-joins-from-the-dhcp-server.md)) | | Losing visibility of a failure | 101 alert rules, 30 days of metrics and logs | | Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana | -| Mains power loss | **The rack, yes; the monitoring path, yes — on two laptop cells that were measured for the first time on 2026-09-12.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the TP-Link carrying `prometheus` and `oracle` has been on UPS power since 2026-09-08 ([#110](https://github.com/Gerrrt/HomeLab/issues/110)); the laptops ride a cut out on their own batteries, which `HostBatteryHealthLow` in `host.rules.yaml` now reads — 94 % of design on `prometheus`, 72 % on `oracle` ([#454](https://github.com/Gerrrt/HomeLab/issues/454)) — see below | +| Mains power loss | **The rack, yes; the monitoring path, yes — on two laptop cells that were measured for the first time on 2026-09-12.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the TP-Link carrying `prometheus` and `oracle` has been on UPS power since 2026-09-08 ([#110](https://github.com/Gerrrt/HomeLab/issues/110)); the laptops ride a cut out on their own batteries, which `HostBatteryHealthLow` in `host.rules.yaml` now reads — `prometheus`'s cell was replaced on 2026-09-18 and reads 101 % of design, `oracle`'s is the original at 72 % and unbought ([#454](https://github.com/Gerrrt/HomeLab/issues/454)) — and **how long either laptop runs on its cell has never been measured**, so this row is answered as far as the cells being healthy and no further — see below | | The estate being down while the person who runs it is unavailable | **Documentation, yes; data, not yet.** ADR-0011 puts the emergency tier on paper; [ADR-0023](adr/0023-keep-the-household-recovery-path-outside-the-estate.md) extends the same reasoning to the sensitive tier's data before that tier exists — see below | What it explicitly does **not** defend against: a determined attacker with From 483e8dbe5efd557f6ad54e0acb0f18d1b3779ccf Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Fri, 18 Sep 2026 19:18:47 +0000 Subject: [PATCH 3/3] test(prometheus): the quiet case is the cell that was replaced, and a ratio above 1 is new (#454) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `prometheus`'s cell was replaced on 2026-09-18. Nothing in these three expressions changes, and that is the point worth recording rather than assuming: `HostBatteryHealthLow` reads the RATIO of `charge_full` to `charge_full_design`, so a pack that reports its own design figure — 6.6 -> 6.8 Ah here, with `voltage_min_design` 11.21 -> 11.4 V beside it — is absorbed rather than breaking the rule. The section comment in `host.rules.yaml` says so, keeps the 2026-09-12 readings because they are what these rules were written against, and records that the pack they describe is gone. THE QUIET CASE KEEPS ITS NUMBERS DELIBERATELY. 6.196 of 6.6 Ah is 0.9388, which sits 0.14 above the 0.8 line; the pack now fitted sits 0.21 above it. Refreshing the case to the new reading would weaken it — a threshold quietly raised to 0.95 fails the 94 % case and would pass one built on 101 %. Only the comment changes, because it called these "prometheus's real reading" and that is no longer true. A SECOND QUIET CASE IS ADDED at 6.889 / 6.8. A ratio above 1 is an input shape this rule had never been fed: an aftermarket pack reports its own design figure and can measure above it, and the rule has to stay quiet rather than fall through to the `energy_*` branch. Adding a case for an alert already in the tested set moves no counted claim, which is why `docs/observability.md`'s coverage sentences are untouched. Verified: `./scripts/validate.sh` — `promtool test rules` over 11 observability files passes through the pinned `prom/prometheus` image. The new case was proved to assert something rather than pass vacuously by flipping its `exp_alerts: []` to expect a firing alert, confirming the run FAILS, and flipping it back. Refs #454 Co-Authored-By: Claude Opus 5 --- .../prometheus/rules/host.rules.yaml | 17 +++++++++++ .../prometheus/tests/host.test.yaml | 29 +++++++++++++++++-- 2 files changed, 43 insertions(+), 3 deletions(-) diff --git a/stacks/observability/prometheus/rules/host.rules.yaml b/stacks/observability/prometheus/rules/host.rules.yaml index 64dc1553..5ce7e609 100644 --- a/stacks/observability/prometheus/rules/host.rules.yaml +++ b/stacks/observability/prometheus/rules/host.rules.yaml @@ -501,6 +501,23 @@ groups: # cell still worth relying on; is there a cell at all. Only the last # needs to know which hosts are laptops, and that list lives in its # expression and nowhere else. + # + # PROMETHEUS'S CELL WAS REPLACED ON 2026-09-18 (#454). The figures above + # are the pack that came out, and they stay: they are what these rules + # were written against. The A1437 fitted in its place reads charge_full + # 6.889 Ah of 6.8 Ah design (101 %) at cyclecount 1. + # + # BOTH DESIGN FIGURES MOVED: charge_full_design 6.6 -> 6.8 Ah and + # voltage_min_design 11.21 -> 11.4 V. An aftermarket pack reports its own + # design numbers, and on this machine that is the only proof the part + # changed at all — it exports no serial, and manufacturer and model_name + # came back identical, SMP and bq20z451. Nothing in these expressions + # depends on either figure: HostBatteryHealthLow reads the RATIO, so a + # design figure that moves is absorbed rather than breaking the rule. + # + # oracle's cell is unchanged, unbought and still firing. + # docs/runbooks/replace-the-laptop-cell.md has the fit, and what it still + # owes: the mains pull on the charged pack, which is why #454 is open. - alert: HostOnBattery # The mains adapter reports online=0, joined to the info series so that diff --git a/stacks/observability/prometheus/tests/host.test.yaml b/stacks/observability/prometheus/tests/host.test.yaml index 317a567b..d00e7891 100644 --- a/stacks/observability/prometheus/tests/host.test.yaml +++ b/stacks/observability/prometheus/tests/host.test.yaml @@ -1146,9 +1146,14 @@ tests: docs/runbooks/replace-the-laptop-cell.md says what to check before and after the swap. - # --- HostBatteryHealthLow: QUIET for prometheus's real reading ------------- - # 6.196 Ah of 6.6 Ah is 94 %, read on 2026-09-12 after 108 cycles. Without - # this the threshold could be deleted and the firing tests would still pass. + # --- HostBatteryHealthLow: QUIET just above the line ---------------------- + # 6.196 Ah of 6.6 Ah is 94 %, read on 2026-09-12 after 108 cycles. THAT CELL + # WAS REPLACED ON 2026-09-18 (#454) and these values are kept deliberately + # rather than refreshed: they are the tighter guard on the 0.8 threshold. + # 0.9388 sits 0.14 above the line where the pack now fitted sits 0.21 above + # it, so a threshold quietly raised to 0.95 fails this case and would pass + # one built on the new numbers. Without a quiet case the threshold could be + # deleted outright and the firing tests would still pass. - interval: 1m input_series: - series: 'node_power_supply_charge_full{instance="prometheus",job="prometheus-metrics",power_supply="BAT0"}' @@ -1160,6 +1165,24 @@ tests: alertname: HostBatteryHealthLow exp_alerts: [] + # --- HostBatteryHealthLow: QUIET above 100 % of design -------------------- + # The A1437 fitted to prometheus on 2026-09-18: charge_full 6.889 Ah against + # a charge_full_design of 6.8 Ah, a ratio of 1.013. A ratio ABOVE 1 is an + # input shape this rule had never been fed — an aftermarket pack reports its + # own design figure and can measure above it — and it has to stay quiet + # rather than falling through to the `or` branch. + # docs/runbooks/replace-the-laptop-cell.md records the fit. + - interval: 1m + input_series: + - series: 'node_power_supply_charge_full{instance="prometheus",job="prometheus-metrics",power_supply="BAT0"}' + values: "6.889x100" + - series: 'node_power_supply_charge_full_design{instance="prometheus",job="prometheus-metrics",power_supply="BAT0"}' + values: "6.8x100" + alert_rule_test: + - eval_time: 65m + alertname: HostBatteryHealthLow + exp_alerts: [] + # --- HostBatteryNotReported: fires when a laptop reports no battery ------- # up is 1 and no BAT series exists at all — the cell is gone, or the kernel # stopped seeing it.