diff --git a/README.md b/README.md index 43880117..ca0b8865 100644 --- a/README.md +++ b/README.md @@ -93,7 +93,7 @@ documents for different readers. from `compose.yaml` too, so an image that is not pinned there cannot be run at all. - **Documented decisions and runbooks.** 42 ADRs covering what was chosen - and what was rejected — including the costs accepted knowingly; 27 + and what was rejected — including the costs accepted knowingly; 28 runbooks for the operations that are easy to get wrong at 1am, one of which is the handover page a successor reads first. diff --git a/docs/hardware.md b/docs/hardware.md index d0384549..610ef51f 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -392,13 +392,17 @@ revisions of this repository treated `shiva` as the hypervisor itself. ([#76](https://github.com/Gerrrt/HomeLab/issues/76)) - A1437 battery cell for `prometheus`[^A1437] — the pack that fits the `A1425`, the late-2012 Retina 13" in the Compute table — bought new - 2026-09-13, in transit. A consumable and not an upgrade: it is the one + 2026-09-13, **delivered 2026-09-18 and not yet fitted.** A consumable and + not an upgrade: it is the one exception the roadmap's *Never* line names, bought because the estate's mains-cut path rests on this cell and the failure mode of a thirteen-year-old lithium cell on a shelf is swelling ([#454](https://github.com/Gerrrt/HomeLab/issues/454)). The listing calls it genuine and its brand field says unbranded, so it is recorded as a - compatible cell, not an Apple part, until it is in hand. The cell it + compatible cell, not an Apple part. That wording was written to be settled + once the part could be looked at, which it now can: the cell arrived + 2026-09-18 and this line stays as it is until someone reads the pack's own + markings and says otherwise. The cell it replaces read 94 % of design capacity after 108 cycles on 2026-09-12, which is above `HostBatteryHealthLow`'s 80 % — it is bought on age, not on the alert. Nothing in the Compute table changes; a cell is not a spec. @@ -406,7 +410,8 @@ revisions of this repository treated `shiva` as the hypervisor itself. `charge_full_design` and a cycle count reading low, then mains pulled and the host staying up with `HostOnBattery` firing — the property the cell is there for, and untested since the machine was commissioned - ([`fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md)). `oracle`'s + ([`replace-the-laptop-cell.md`](runbooks/replace-the-laptop-cell.md), which + also carries the stack-down window and the old cell's disposal). `oracle`'s cell reads 72 % and is second in line, unbought. - ViewSonic N1700W LCD, used as a rack console via the KVM - RJ45 Cat6 in-line couplers[^Couplers] diff --git a/docs/observability.md b/docs/observability.md index 630277bb..c74eb95e 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -437,7 +437,7 @@ argument and for what to do when it exits 1. | File | Covers | | --- | --- | -| `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew, reboots — and, for the two laptops, whether the shelf is on mains and whether the cell that carries them through a cut is still worth relying on ([#454](https://github.com/Gerrrt/HomeLab/issues/454)); and whether the wiki's drift check on `oracle` is still running ([#470](https://github.com/Gerrrt/HomeLab/issues/470)) | +| `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew, reboots — and, for the two laptops, whether the shelf is on mains and whether the cell that carries them through a cut is still worth relying on ([#454](https://github.com/Gerrrt/HomeLab/issues/454), and [`runbooks/replace-the-laptop-cell.md`](runbooks/replace-the-laptop-cell.md) for the swap); and whether the wiki's drift check on `oracle` is still running ([#470](https://github.com/Gerrrt/HomeLab/issues/470)) | | `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware and Smart Array cache. `shiva`'s Smart Storage Battery read failed from 2026-08-18 until it was replaced on 2026-09-02, with the array in write-through as a result, so stored metrics before that date show the failed pack — `IloBatteryCondition` names the spare part to order, and the controller rollups are deliberately read at *failed* rather than *degraded* ([#76](https://github.com/Gerrrt/HomeLab/issues/76)) | | `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. A pack was fitted on 2026-08-28 and passed its self-test, so these read real hardware; stored metrics older than that date are the card's fabricated values — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) | | `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling | diff --git a/docs/runbooks/fit-the-ups-battery.md b/docs/runbooks/fit-the-ups-battery.md index 729d3a1d..d57628e2 100644 --- a/docs/runbooks/fit-the-ups-battery.md +++ b/docs/runbooks/fit-the-ups-battery.md @@ -90,11 +90,16 @@ hosts, and `host.rules.yaml` reads it — `HostBatteryHealthLow` when a cell's after 108 cycles, `oracle` 72 %), `HostOnBattery` when the mains adapter loses input, `HostBatteryNotReported` when a laptop stops reporting a cell. The cell for `prometheus`, an A1437, was bought 2026-09-13 and is recorded in -[`hardware.md`](../hardware.md#accessories). After a cell is swapped, -`charge_full` should sit at or near `charge_full_design` and the cycle count -read low. Then pull that laptop's mains and confirm the host -stays up and `HostOnBattery` fires: that is the property the cell is there for, -and it had not been tested since the machine was commissioned. +[`hardware.md`](../hardware.md#accessories). The swap and the tests that prove +it are [`replace-the-laptop-cell.md`](replace-the-laptop-cell.md), which is +where the baseline, the disposal and the stack-down window are written down — +that cell is glued into the machine running this stack, so fitting it blinds +the estate for the length of the job, which is the one cost this runbook's own +pack did not carry. What it proves is the same shape as step 5 here: +`charge_full` at or near `charge_full_design` and the cycle count reading low, +then that laptop's mains pulled with the host staying up and `HostOnBattery` +firing — the property the cell is there for, and untested since the machine was +commissioned. ## Before you start diff --git a/docs/runbooks/replace-the-laptop-cell.md b/docs/runbooks/replace-the-laptop-cell.md new file mode 100644 index 00000000..6be51468 --- /dev/null +++ b/docs/runbooks/replace-the-laptop-cell.md @@ -0,0 +1,515 @@ +# Runbook: Replace the laptop cell in `prometheus` + +**One glued-in cell, one power-down of the host that watches everything else, +and one test that only works with the machine running.** + +> **Status — 2026-09-18: the cell is in hand and nothing is fitted. The +> baseline below is measured, not remembered.** +> +> Every reading in step 1 was taken from this host's own Prometheus on +> 2026-09-17 and re-read unchanged on 2026-09-18, the day the cell arrived: +> `charge_full` 6.196 Ah of 6.6 Ah design (94 %), unmoved across the retained +> window, 108 cycles, `status` `Full`, on mains via `ADP1`. Nothing in +> `host.rules.yaml` is firing for `prometheus`. The baseline table is therefore +> current as it stands — it does not need retaking before the fit. +> +> `HostBatteryHealthLow` fired for **`oracle`** from 2026-09-14 and was +> silenced on 2026-09-17 until 2026-10-08 +> (`01cb81d7-5e19-4e6d-b386-f5c8c843032b`). That is the other laptop, it is the +> finding [#454](https://github.com/Gerrrt/HomeLab/issues/454) produced rather +> than a fault in this procedure, and that cell is second in line and unbought. +> The silence suppresses 72 % **and anything lower**, so that cell's further +> decay is not visible until it expires — see *What is still open*. +> +> **No silence is created anywhere in this runbook, and that is deliberate.** +> Both of the battery runbooks beside this one are largely about a silence that +> had to be deleted at the right moment, and both record deleting it late as +> their one regret. A silence never created cannot be forgotten. + +The host is `prometheus` at `10.0.99.20` — the Apple MacBook Pro (2012, Retina +13") in the Compute table of [`../hardware.md`](../hardware.md), and the machine +the whole observability stack runs on. The cell is an A1437, the pack that fits +the `A1425`, bought new 2026-09-13 and recorded under Accessories. It is +**glued to the case**, which is why the purchase was the kit with tools and +adhesive solvent rather than the bare cell. + +Budget an evening, most of it solvent soak and charge time, plus a bounded +window in which the estate has no monitoring at all. You will need the kit, and +something to read Prometheus from that is not this laptop — a phone on the same +network is enough for both mains-pull tests. + +## Why this is not urgent, and why it is not nothing + +The cell reads **94 % of design capacity after 108 cycles**, which is *above* +`HostBatteryHealthLow`'s 80 % line. **No alert asked for this.** It is bought +on age and on the failure mode, which is the exception +[`../roadmap.md`](../roadmap.md) names in the same breath as its *Never* line: +a consumable whose failure is a safety or availability event is not an upgrade. + +Two stakes, and they are different from each other. + +**Safety.** A thirteen-year-old lithium pouch cell, on a shelf, beside the +rack, in an occupied room. The failure mode of a cell this old is swelling and +then fire. Nothing in `host.rules.yaml` can see that coming: +`HostBatteryHealthLow` compares capacity against design capacity, and a cell +can sit at 94 % and still be bulging. Capacity and mechanical integrity are +different questions and only one of them is measured. + +**Availability.** `prometheus` is the observability stack. Its cell is the last +link of the mains-cut path that [#93](https://github.com/Gerrrt/HomeLab/issues/93) +and [#110](https://github.com/Gerrrt/HomeLab/issues/110) built — the TP-Link in +U4 is on UPS power so the laptops stay *reachable* through a cut, and the cell +is what keeps this one *running* through it. That half has never been tested +since the machine was commissioned. Step 8 is where it stops being an +assumption. + +What this is **not** is a fix for `oracle`, whose cell measures worse at 72 % +and is already firing. That cell is unbought and second in line, and this +runbook is written to be reused for it — see *What is still open* for the +differences between the two machines. + +## What goes blind while the lid is off + +The two battery runbooks beside this one did not need this section, because +`shiva` and `mjolnir` are not the monitoring host. This one is. + +- **Everything stops.** Prometheus, Alertmanager, Loki, Grafana, the SNMP and + blackbox exporters and this host's Alloy are one compose stack on this + laptop. While it is down no rule is evaluated, so **no alert can fire and + none can be recorded**, and Alertmanager could not notify even if something + else knew. Loki is not ingesting; lines arriving in the window are lost, not + queued. +- **Any `for:` in flight resets.** An alert that starts and ends inside the + window leaves no trace anywhere at all. `HostBatteryHealthLow` on `oracle` + needs a fresh hour after the stack returns before it fires again, and its + absence for that hour is the rule restarting rather than the cell recovering. +- **Two kinds of hole, and only one of them fills in.** The SNMP and blackbox + jobs are *scraped by* Prometheus — `morpheus`, `neo`, `mjolnir`, `shiva` and + every probe — so those minutes are a true hole that nothing can backfill. + Host metrics from the agent hosts arrive by **remote write** from their own + Alloy, which buffers to a WAL and retries, so a short outage may backfill. + Do not rely on it; step 6 measures which half actually did. +- **What keeps working.** `morpheus` keeps filtering — nothing about + enforcement stops, only observation. The lab's own Prometheus on `alexander` + keeps watching its guest and is unaffected. `oracle` keeps running the wiki + and its own jobs. +- **One thing does notice, and it is supposed to.** The off-host healthcheck + from [`verify-the-alert-path.md`](verify-the-alert-path.md) goes down roughly + twenty minutes after the host powers off, and emails through a path that is + not this stack's. **Do not pause it.** That email is the dead man's switch + working, and a check you have only ever seen green is a check you have not + tested. Note when it arrives; step 6 wants it back green. +- **Bound the window.** Keep it under two hours. Start after the half hour so + the hourly converge has just run, and **not on a Sunday near 03:30**, which + is when `backup-volumes` quiesces the stack — + [`schedule-maintenance.md`](schedule-maintenance.md) has the timetable. The + tightest staleness threshold that matters here is 90 minutes, so a window + under two hours costs at most one late job on the way back. + +## 1. Record the baseline — before touching anything + +This is the step that makes the fix provable rather than assumed. The UPS pack +was only ever proven because its before-values were written down first. + +```bash +for m in node_power_supply_charge_full node_power_supply_charge_full_design \ + node_power_supply_charge_ampere node_power_supply_capacity \ + node_power_supply_cyclecount node_power_supply_present \ + node_power_supply_voltage_volt node_power_supply_voltage_min_design \ + node_power_supply_temp_celsius node_power_supply_current_ampere; do + printf '%-44s ' "$m" + curl -sG http://localhost:9090/api/v1/query \ + --data-urlencode "query=${m}{instance=\"prometheus\",power_supply=\"BAT0\"}" | + python3 -c 'import json,sys; r=json.load(sys.stdin)["data"]["result"]; print(r[0]["value"][1] if r else "no data")' +done +``` + +Two things the loop cannot carry, because one answer is a label set and the +other is a different supply: + +```bash +curl -sS -G http://localhost:9090/api/v1/query \ + --data-urlencode 'query=node_power_supply_info{instance="prometheus"}' | + python3 -m json.tool + +curl -sG http://localhost:9090/api/v1/query \ + --data-urlencode 'query=node_power_supply_online{instance="prometheus",power_supply="ADP1"}' +``` + +The ratio the rule itself reads, so the comparison is against the rule rather +than against arithmetic done by hand: + +```bash +curl -sG http://localhost:9090/api/v1/query --data-urlencode \ + 'query=node_power_supply_charge_full{instance="prometheus"} + / node_power_supply_charge_full_design{instance="prometheus"}' +``` + +And the state of the world, so step 6 has something to compare against: + +```bash +curl -sS http://localhost:9090/api/v1/alerts | + python3 -c 'import json,sys; [print(a["labels"].get("alertname"), a["labels"].get("instance"), a["state"]) for a in json.load(sys.stdin)["data"]["alerts"]]' +``` + +The **Reads** column was measured on 2026-09-17 and is the baseline; fill +**Observed** in step 7. + +| Metric | Reads 2026-09-17, thirteen-year-old cell | After a good new cell | Observed | +| --- | --- | --- | --- | +| `node_power_supply_charge_full` | `6.196` | at or near `6.6` | | +| `node_power_supply_charge_full_design` | `6.6` | `6.6` — must not move | | +| `charge_full / charge_full_design` | `0.9388` | `0.98`–`1.0` | | +| `node_power_supply_cyclecount` | `108` | low — `0` to a handful | | +| `node_power_supply_charge_ampere` | `6.12`, drifting in the last digit | any value that **moves** | | +| `node_power_supply_capacity` | `93` | rises to `100` on charge | | +| `node_power_supply_current_ampere` | `0` | non-zero while discharging | | +| `node_power_supply_voltage_volt` | `12.43` on mains (24h span `12.425`–`12.438`) | `10.9`–`12.6`, and varying under load | | +| `node_power_supply_voltage_min_design` | `11.21` | `11.21` — must not move | | +| `node_power_supply_temp_celsius` | `32.7`–`39.2` over 24h, averaging `33.3` | a similar band, never far above it | | +| `node_power_supply_present` | `1` | `1` | | +| `node_power_supply_online{power_supply="ADP1"}` | `1` | `1` | | +| info `manufacturer` | `SMP` | may or may not change | | +| info `model_name` | `bq20z451` | may or may not change | | +| info `status` | `Full` | `Charging`, then `Full` | | + +> **Three names in this family are not what you would guess, and a wrong one +> returns an empty result — which on a command line is indistinguishable from +> "the cell is gone".** The cycle count is `node_power_supply_cyclecount`, one +> word, not `cycle_count`. The capacity family is `charge_*`, not `energy_*`: +> [#454](https://github.com/Gerrrt/HomeLab/issues/454)'s body asked for +> `energy_full` and `energy_full_design`, and neither exists on either laptop. +> And what would be `charge_now` is exported as +> `node_power_supply_charge_ampere`. The `or` branch in `HostBatteryHealthLow` +> keeps the `energy_*` spelling for a future machine whose driver reports +> watt-hours; nothing here reports them. + +Query Prometheus rather than reaching for the host's own +`/sys/class/power_supply/`. The stack is what the alerts read and therefore +what is being compared against; the kernel agreeing with itself proves less. + +## 2. Prove the alert path on the cell you are about to throw away + +A lithium cell should come out discharged rather than full, and the only way to +discharge this one is to run the machine on it. That is the mains-pull test, so +do it **now, on the old cell**, and get a proven alert path out of the same +twenty minutes. + +1. Have the phone in hand and note the time. +2. Pull `prometheus`'s power brick — **only that brick.** Not the shelf, not the + TP-Link, not the UPS. This is the discriminator the rule was written for: + `HostOnBattery` should fire for `prometheus` and for nothing else, and + `UpsOnBattery` should stay quiet, because the rack did not lose power. +3. Watch it fire: + + ```bash + curl -sG http://localhost:9090/api/v1/query --data-urlencode \ + 'query=node_power_supply_online == 0 and on (instance, power_supply) + node_power_supply_info{type="Mains"}' + + curl -sG http://localhost:9090/api/v1/query \ + --data-urlencode 'query=ALERTS{alertname="HostOnBattery"}' + + docker exec alertmanager amtool alert query \ + --alertmanager.url=http://localhost:9093 alertname=HostOnBattery + ``` + +4. **Budget under four minutes to the notification.** Alloy scrapes the node + exporter every 60s, the rule carries `for: 2m`, Prometheus evaluates every + 30s, and the power route has `group_wait: 0s`. Nothing arriving inside five + minutes is a finding about the alert path rather than about the cell, and + [`verify-the-alert-path.md`](verify-the-alert-path.md) is the runbook for it. +5. Let it run down to roughly **30 %** on `node_power_supply_capacity`, then + plug back in and confirm the alert resolves and the notification clears. Do + not run it flat: a deep discharge of a cell about to be removed buys nothing, + and a hard shutdown of the monitoring host is the thing this whole page + exists to keep controlled. + +Record the times — brick pulled, first `online == 0` sample, alert pending, +alert firing, notification on the phone, brick back, resolved. + +**Why this belongs here rather than only at the end:** when step 8 repeats it on +the new cell, the path is already known to work. A failure in step 8 after a +pass here is the cell or the adapter, and nothing about Prometheus, +Alertmanager or the route. That is a discrimination you cannot buy afterwards. + +### Do not silence `HostOnBattery` for either test + +It is `severity: critical` and `category: power`, which the routing sends to the +urgent receiver with `group_wait: 0s` and a 30-minute repeat. It will really +page. Let it, both times, deliberately — tell anyone else on that receiver +first, and stay out of the Sunday backup window. + +The case against silencing, which is the one that wins here: + +- The blast radius is one notification to the person standing at the machine + holding the plug. A bounded twenty-minute test produces at most two. +- [`verify-the-alert-path.md`](verify-the-alert-path.md) proves the *transport* + using `Watchdog`, which is `vector(1)` and carries no labels. It cannot prove + this rule's `for: 2m`, its join against + `node_power_supply_info{type="Mains"}`, or that `category: power` reaches the + urgent receiver instead of sitting in the default 12h bucket. Nothing else in + the estate exercises that specific path end to end, and the mains-cut path is + the reason the cell was bought. +- **A silence created is a silence to remember to delete.** + [`fit-the-ups-battery.md`](fit-the-ups-battery.md) records deleting one 29 + minutes late; [`replace-the-smart-storage-battery.md`](replace-the-smart-storage-battery.md) + records the same inversion nine minutes late and says "delete first next time + too". Both regrets are about a silence standing over a freshly fitted part — + exactly what a silence created here would be. The way not to repeat it a + third time is not to create one. + +## 3. A thirteen-year-old lithium cell is the hazard, not the laptop + +- **Inspect before you plan.** A swollen MacBook cell pushes the trackpad up + from underneath. A trackpad that has stopped clicking properly, a lid that no + longer closes flat, or a bottom case that rocks are the tells, and all three + appear before anything electrical does. If any is present, treat the machine + as a hazard from that moment: do not charge it further, do not leave it + unattended, and bring step 4 forward rather than scheduling it. +- **The temperature baseline is the one number that speaks to this, and it + is a band rather than a figure.** `node_power_supply_temp_celsius` ranged + `32.7`–`39.2` over the 24 hours to 2026-09-17, averaging `33.3`, idling on + mains. Nothing alerts on it and nothing watches it; it is written down here + so a reading far above that band has something to be far above. One + instantaneous sample proves nothing on its own — a single reading near `39` + is ordinary for this cell. +- **Order of operations, and it is not negotiable.** Machine off (step 4), + bottom case off, then **disconnect the battery connector from the logic board + before touching anything else.** A metal tool near a live cell's terminals is + the failure this sequence prevents. +- **Never pry, fold or puncture a pouch.** The cells are glued flat across the + case. Use the kit's adhesive solvent, give it the time the instructions say, + and pull on the tabs. No metal spudger goes under a cell. If one tears or + vents, stop, ventilate, and do not continue on that machine that evening. +- **This is not a repair guide, on purpose.** The mechanical procedure is the + kit's own instructions and iFixit's A1425 battery guide. What is written down + here is what this repository can be authoritative about: the readings, the + alerts, the window and the disposal. + +## 4. Take the stack down, and the machine with it + +1. **Back up first.** The TSDB, Loki's chunks and Grafana's database are all on + the disk inside the machine you are about to open. + + ```bash + make backup + ``` + + It quiesces the stack, archives the volumes and verifies each archive + readable. [`restore-the-stack.md`](restore-the-stack.md) is the other end of + that, and it is the fallback if the machine does not come back. +2. Note the time. The window starts here, and the healthcheck email is keyed to + it. +3. Stop the stack cleanly, then the host: + + ```bash + make down + sudo systemctl poweroff + ``` + + `make down` preserves volumes. Holding the power button on a running + Prometheus is not the same thing and is not what this step says. +4. **Take any pending reboot for free.** `RebootRequired` waits three days + precisely because rebooting this host blinds the estate for the duration — + and this window is that duration, already being spent. Apply outstanding + updates before the power-down if any are queued; step 6 checks the alert has + cleared. + +## 5. Fit the cell + +Bottom case screws out, **battery connector off the logic board first**, +solvent under each cell in turn, lift by the tabs, clean the residue, seat the +new pack, connector back on, screws back in. + +**Do not judge any number until it has had a full charge.** A new cell arriving +part-charged reports a `charge_full` that is not yet its true learned capacity; +the gas gauge learns across a charge cycle. This is the same reason the write +cache alert on `shiva` waits an hour after a pack is fitted. Charge to +`status="Full"` before believing the table in step 7. + +## 6. Bring the stack back and confirm nothing else broke + +```bash +make up +make ps +``` + +Then, in order: + +- Every target healthy. No output is the pass: + + ```bash + curl -sS http://localhost:9090/api/v1/targets | + python3 -c 'import json,sys; [print(t["labels"]["job"], t["scrapeUrl"], t["health"]) for t in json.load(sys.stdin)["data"]["activeTargets"] if t["health"] != "up"]' + ``` + +- The firing set matches the step 1 snapshot, minus `RebootRequired` if you + took the reboot: + + ```bash + curl -sS http://localhost:9090/api/v1/alerts | + python3 -c 'import json,sys; [print(a["labels"].get("alertname"), a["labels"].get("instance"), a["state"]) for a in json.load(sys.stdin)["data"]["alerts"]]' + ``` + +- The off-host healthcheck is green again within about ten minutes of + Alertmanager starting. If it is still red after that, it is + [`verify-the-alert-path.md`](verify-the-alert-path.md) and not this runbook. +- **Measure the hole, and measure which half of it filled in.** One query, + twice, against two jobs that behave differently: + + ```bash + for j in snmp oracle-metrics; do + printf '%-16s ' "$j" + curl -sG http://localhost:9090/api/v1/query_range \ + --data-urlencode "query=count(up{job=\"${j}\"})" \ + --data-urlencode "start=$(date -u -d '-6 hours' +%s)" \ + --data-urlencode "end=$(date -u +%s)" \ + --data-urlencode 'step=60' | + python3 -c 'import json,sys; r=json.load(sys.stdin)["data"]["result"]; print(len(r[0]["values"]) if r else 0, "minutes with data, of 360")' + done + ``` + + `snmp` is scraped by Prometheus and its hole is the true length of the + window. `oracle-metrics` arrives by remote write and may be shorter, because + that agent's WAL retried. A shorter `oracle-metrics` hole is the backfill + working; an equal one means it did not. Either way the number is now known + rather than assumed. +- **`HostBatteryNotReported` is the automatic verdict on "is there a cell at + all", and it arrives fifteen minutes after the stack is back.** Silence at + t+15m is the pass. Do not skip ahead to step 7's table before it has had that + quarter of an hour. + +## 7. Confirm the metrics actually moved + +Re-run step 1's loop and fill the **Observed** column. + +> **There is no serial number to appeal to.** `shiva`'s pack was proven by +> `cpqHeSysBatterySerialNumber` changing — the one row that could not be the old +> part reporting differently. This machine does not export one: +> `node_power_supply_info` on `prometheus` carries `manufacturer`, +> `model_name`, `technology`, `status` and `type`, and no serial. (`oracle`'s +> Dell does carry one, which is worth knowing when this runbook is reused +> there.) Worse, an aftermarket A1437 commonly reuses the same gas gauge, so +> `SMP` / `bq20z451` reading identically afterwards proves nothing either way. +> The honest proof is three rows together, not one: **`cyclecount` falling from +> `108` to something low, `charge_full` rising toward `6.6`, and +> `charge_ampere` moving across a charge cycle.** A reading that is +> byte-for-byte identical to the baseline means the machine is reporting the old +> pack's stored values — treat that as "the new cell is not seen", not as "the +> numbers happen to match", and go back to the connector. + +Two more failure shapes worth naming: + +- **`charge_full` reads exactly `6.6` and never moves.** That is the gauge + reporting its design figure because it has not learned a capacity yet, not a + cell at 100 % health. It becomes a measurement after one full charge and one + substantial discharge, and step 8 provides the discharge. +- **`HostBatteryHealthLow` fires for `prometheus`.** The new cell is measurably + worse than the thirteen-year-old one it replaced. That is a return to the + seller, not a finding to write up. + +## 8. Pull the mains again — the test the cell is there for + +> This is the step that closes +> [#454](https://github.com/Gerrrt/HomeLab/issues/454). Everything before it +> makes a claim about capacity. Only this one makes the claim the cell was +> bought for. + +Same procedure as step 2, now on the new cell and with the machine fully +charged. Because the numbers are finally meaningful, also **measure the runtime +rather than estimating it** — a figure the estate has never had: + +```bash +# Ampere-hours per hour being drawn, at the load the stack actually presents +curl -sG http://localhost:9090/api/v1/query --data-urlencode \ + 'query=-deriv(node_power_supply_charge_ampere{instance="prometheus",power_supply="BAT0"}[10m]) * 3600' + +# Hours left at that draw +curl -sG http://localhost:9090/api/v1/query --data-urlencode \ + 'query=node_power_supply_charge_ampere{instance="prometheus",power_supply="BAT0"} + / (-deriv(node_power_supply_charge_ampere{instance="prometheus",power_supply="BAT0"}[10m]) * 3600)' +``` + +Bound the test: **stop at twenty minutes or 50 % capacity, whichever comes +first.** The point is that the property holds and is measurable, not that the +cell can be flattened. + +Confirm on the way back that the alert resolves, the notification clears, +`online` returns to `1` and `status` goes to `Charging`. + +And check the same discriminator as step 2: `HostOnBattery` fires for +`prometheus` and not for `oracle`, and `UpsOnBattery` stays quiet throughout. +**If `UpsOnBattery` fired, you pulled the wrong plug.** + +Record the measured runtime in the status banner at the top of this page, and +note that it is one measurement at one load on one day. + +## 9. Retire the old cell + +- **Tape the connector and any exposed terminals** the moment it is out. +- **Not the bin, and not back on the shelf.** A thirteen-year-old cell sitting + beside the rack is the hazard this whole exercise was opened about; leaving + the removed one there keeps the hazard and gives up the capacity. +- Keep it in a non-flammable container, away from the rack and away from + anything that burns, and take it to a council household-waste battery point or + a retailer take-back **within days, not months**. Note the date. +- **Never post it.** A damaged lithium cell is not a mailable item. +- If it came out swollen or vented: outside, metal container, same day, and do + not apply solvent to it. + +## If something goes wrong + +**The machine does not come back.** Nothing on the disk changed — the stack is +volumes plus a checkout. Re-seat the battery connector, and try mains with the +cell disconnected, because a MacBook runs on the adapter alone. `make up` once +it boots. The estate stays blind until it does, and +[`restore-the-stack.md`](restore-the-stack.md) is the path if the disk itself +turns out to be the problem. + +**`HostBatteryNotReported` fires fifteen minutes after the lid closed.** The +connector, almost always. Then `/sys/class/power_supply/` on the host, to see +whether the kernel enumerates `BAT0` at all. + +**Everything reads exactly as it did before.** Step 7's warning. That is not a +coincidence, and it is not the numbers happening to match. + +**Step 8's pull produced nothing, but step 2's worked.** The path is proven, so +the difference is the new cell or the adapter. Check that +`node_power_supply_online{power_supply="ADP1"}` actually went to `0` — if it did +not, the brick is still feeding the machine and nothing was tested. + +**The healthcheck email never arrived during the window.** That is more serious +than anything else on this page, because it is the one signal that survives this +host being down. [`verify-the-alert-path.md`](verify-the-alert-path.md). + +## What is still open + +- **`HostBatteryHealthLow`'s description names + [`fit-the-ups-battery.md`](fit-the-ups-battery.md), not this file.** That was + the only runbook there was when the rule was written. Correcting it changes + `stacks/observability/prometheus/rules/host.rules.yaml` *and* the + `exp_annotations` block in + `stacks/observability/prometheus/tests/host.test.yaml`, which asserts the + string exactly — a rules-and-tests change rather than a documentation one, so + it is not made here. A reader following the annotation lands on that + runbook's laptop-cell paragraph, which links here. +- **`oracle`'s cell reads 72 %, is unbought, and its alert is silenced until + 2026-10-08** — `01cb81d7-5e19-4e6d-b386-f5c8c843032b`, matching + `alertname="HostBatteryHealthLow"`, `instance="oracle"`, + `power_supply="BAT0"`. It shares an expiry with the `#351` disk silence on the + same host so one look covers both. Because the rule is `< 0.8` and no label + carries the ratio, that silence hides any *further* decay of the cell as well + as the 72 % it was created for; re-read + `node_power_supply_charge_full{instance="oracle"} / + node_power_supply_charge_full_design{instance="oracle"}` rather than trusting + the absence of an alert. + + This runbook is written to be reused for that cell. The differences: its + mains supply is `AC` and not `ADP1`; its firmware reports `cyclecount` as `0` + and always has, so the strongest of the three proof rows in step 7 is + unavailable there; it exports no `temp_celsius`; and its info series *does* + carry a `serial_number`, which is a proof row this machine lacks. +- **Nothing watches runtime-on-battery continuously.** The figure step 8 + produces is one measurement, at one load, on one day. There is no rule and no + series that would notice it halving.