Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ documents for different readers.
from `compose.yaml` too, so an image that is not pinned there cannot be run
at all.
- **Documented decisions and runbooks.** 42 ADRs covering what was chosen
and what was rejected — including the costs accepted knowingly; 27
and what was rejected — including the costs accepted knowingly; 28
runbooks for the operations that are easy to get wrong at 1am, one of which
is the handover page a successor reads first.

Expand Down
11 changes: 8 additions & 3 deletions docs/hardware.md
Original file line number Diff line number Diff line change
Expand Up @@ -392,21 +392,26 @@ revisions of this repository treated `shiva` as the hypervisor itself.
([#76](https://github.com/Gerrrt/HomeLab/issues/76))
- A1437 battery cell for `prometheus`[^A1437] — the pack that fits the
`A1425`, the late-2012 Retina 13" in the Compute table — bought new
2026-09-13, in transit. A consumable and not an upgrade: it is the one
2026-09-13, **delivered 2026-09-18 and not yet fitted.** A consumable and
not an upgrade: it is the one
exception the roadmap's *Never* line names, bought because the estate's
mains-cut path rests on this cell and the failure mode of a
thirteen-year-old lithium cell on a shelf is swelling
([#454](https://github.com/Gerrrt/HomeLab/issues/454)). The listing calls
it genuine and its brand field says unbranded, so it is recorded as a
compatible cell, not an Apple part, until it is in hand. The cell it
compatible cell, not an Apple part. That wording was written to be settled
once the part could be looked at, which it now can: the cell arrived
2026-09-18 and this line stays as it is until someone reads the pack's own
markings and says otherwise. The cell it
replaces read 94 % of design capacity after 108 cycles on 2026-09-12, which
is above `HostBatteryHealthLow`'s 80 % — it is bought on age, not on the
alert. Nothing in the Compute table changes; a cell is not a spec.
**Checked at the fit, not assumed:** `charge_full` at or near
`charge_full_design` and a cycle count reading low, then mains pulled and
the host staying up with `HostOnBattery` firing — the property the cell is
there for, and untested since the machine was commissioned
([`fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md)). `oracle`'s
([`replace-the-laptop-cell.md`](runbooks/replace-the-laptop-cell.md), which
also carries the stack-down window and the old cell's disposal). `oracle`'s
cell reads 72 % and is second in line, unbought.
- ViewSonic N1700W LCD, used as a rack console via the KVM
- RJ45 Cat6 in-line couplers[^Couplers]
Expand Down
2 changes: 1 addition & 1 deletion docs/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -437,7 +437,7 @@ argument and for what to do when it exits 1.

| File | Covers |
| --- | --- |
| `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew, reboots — and, for the two laptops, whether the shelf is on mains and whether the cell that carries them through a cut is still worth relying on ([#454](https://github.com/Gerrrt/HomeLab/issues/454)); and whether the wiki's drift check on `oracle` is still running ([#470](https://github.com/Gerrrt/HomeLab/issues/470)) |
| `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew, reboots — and, for the two laptops, whether the shelf is on mains and whether the cell that carries them through a cut is still worth relying on ([#454](https://github.com/Gerrrt/HomeLab/issues/454), and [`runbooks/replace-the-laptop-cell.md`](runbooks/replace-the-laptop-cell.md) for the swap); and whether the wiki's drift check on `oracle` is still running ([#470](https://github.com/Gerrrt/HomeLab/issues/470)) |
| `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware and Smart Array cache. `shiva`'s Smart Storage Battery read failed from 2026-08-18 until it was replaced on 2026-09-02, with the array in write-through as a result, so stored metrics before that date show the failed pack — `IloBatteryCondition` names the spare part to order, and the controller rollups are deliberately read at *failed* rather than *degraded* ([#76](https://github.com/Gerrrt/HomeLab/issues/76)) |
| `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. A pack was fitted on 2026-08-28 and passed its self-test, so these read real hardware; stored metrics older than that date are the card's fabricated values — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) |
| `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling |
Expand Down
15 changes: 10 additions & 5 deletions docs/runbooks/fit-the-ups-battery.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,11 +90,16 @@ hosts, and `host.rules.yaml` reads it — `HostBatteryHealthLow` when a cell's
after 108 cycles, `oracle` 72 %), `HostOnBattery` when the mains adapter loses
input, `HostBatteryNotReported` when a laptop stops reporting a cell. The cell
for `prometheus`, an A1437, was bought 2026-09-13 and is recorded in
[`hardware.md`](../hardware.md#accessories). After a cell is swapped,
`charge_full` should sit at or near `charge_full_design` and the cycle count
read low. Then pull that laptop's mains and confirm the host
stays up and `HostOnBattery` fires: that is the property the cell is there for,
and it had not been tested since the machine was commissioned.
[`hardware.md`](../hardware.md#accessories). The swap and the tests that prove
it are [`replace-the-laptop-cell.md`](replace-the-laptop-cell.md), which is
where the baseline, the disposal and the stack-down window are written down —
that cell is glued into the machine running this stack, so fitting it blinds
the estate for the length of the job, which is the one cost this runbook's own
pack did not carry. What it proves is the same shape as step 5 here:
`charge_full` at or near `charge_full_design` and the cycle count reading low,
then that laptop's mains pulled with the host staying up and `HostOnBattery`
firing — the property the cell is there for, and untested since the machine was
commissioned.

## Before you start

Expand Down
Loading
Loading