Skip to content

docs(prometheus): the A1437 cell is fitted, and the swap corrected two documents (#454) - #520

Merged
Gerrrt merged 3 commits into
mainfrom
gerrrt/prometheus-battery-replacement-941acd
Sep 18, 2026
Merged

Gerrrt merged 3 commits into
mainfrom
gerrrt/prometheus-battery-replacement-941acd

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Sep 18, 2026

Copy link
Copy Markdown
Owner

What changed

The cell went into prometheus on 2026-09-18. Every document still said it was
in hand and unfitted; they now describe the machine that exists. Nothing here
changes a rule expression, adds a metric or needs a deploy.

Metric Old cell, 2026-09-17 New cell, 2026-09-18
charge_full 6.196 Ah 6.889 Ah
charge_full_design 6.6 Ah 6.8 Ah
ratio 0.9388 (94 %) 1.013 (101 %)
cyclecount 108 1
voltage_min_design 11.21 V 11.4 V
info manufacturer / model_name SMP / bq20z451 unchanged

HostBatteryNotReported is quiet, every scrape target is up, and the firing set
came back identical to the pre-swap snapshot — ScheduledJobFailed
(backup-volumes), SecretsKeyBackupUnproven, RebootRequired and
GatewayMonitorUnreliable had all been firing continuously for the four days
before, so none of them is this work.

#454 stays open

The runbook names step 8 — mains pulled on a fully charged cell with the
runtime measured — as the step that settles the issue, and it has not been run.
The pack was still charging when these readings were taken. Every commit here
says Refs, and there is deliberately no ## Done entry on the roadmap; the
precedent for work landed against an open issue is #418.

What fired on the day was accidental: the machine came back from the swap
unplugged at 41 %, so HostOnBattery went pending and then firing for
prometheus and nothing else, with UpsOnBattery quiet throughout. That is the
discriminator the runbook wanted, arrived at by accident, on a clock that was
three hours wrong. It is not a bounded test and it measured no runtime.

Step 2 was never performed and now cannot be. HostOnBattery had never
fired for prometheus across the whole 30-day retention, and the old cell is
gone, so the discrimination step 2 was written to buy — a step 8 failure being
the cell or the adapter and nothing else — is unavailable. Recorded rather than
glossed, and worth running properly when the page is reused on oracle.

Two things the swap showed the documents had wrong

The design figures moved, and the runbook said they could not. Step 1's
table marked charge_full_design and voltage_min_design "must not move". Both
did. They are figures the pack's own gas gauge reports, not properties of the
model, and an aftermarket A1437 reports its own. The expectation is inverted
rather than restated, because on a machine that exports no serial number
that movement is the only proof the part changed at all — the gauge identifiers
came back byte-identical, exactly as step 7 warned. Step 7's three proof rows
become four. Nothing in the rules depends on either figure:
HostBatteryHealthLow reads the ratio, so the movement is absorbed.

Disconnecting the cell reset the RTC, and nothing noticed — filed as
#519. The post-swap boot opens
2026-07-28 15:04:45; systemd-timesyncd restored the clock to 14:53:37, the
second the machine went down, and the stack then ran ~14 minutes writing samples
at backdated timestamps before NTP stepped it forward to the true 18:12.

So the 184-minute "hole" at 15:08–18:12 is not downtime, real post-swap
operation is filed inside the outage, and the true window — 14:53:37 to about
18:12, roughly 3 h 18 m — is over this page's own two-hour bound and not
recoverable from the series at all. Step 6's measure the hole query therefore
gives a confident wrong answer after a battery swap, and now says to take the
window from journalctl --list-boots. HostClockSkew could not have caught it:
it reads abs(node_timex_offset_seconds), which peaked at 0.004 s, because
the kernel's correction says nothing about a clock that is wrong but stable.
node_timex_sync_status was 0 for exactly the backdated window and is read by
no rule — that is #519's question, and no rule lands here.

Blast radius

  • No rule expression changed — one YAML comment block and one added test case
  • No change to network segmentation or firewall rules
  • No new port, no new image, no new credential
  • Nothing to deploy; the running rules are byte-identical

Verification

  • ./scripts/validate.sh passes — promtool test rules over 11
    observability files through the pinned image, documents agree with the configs, 3 expected host-only skips
  • The new test case was proved non-vacuous: flipping its exp_alerts: [] to
    expect a firing alert makes the run FAIL with got:[], confirming the
    rule is genuinely quiet at a ratio of 1.013. Flipped back and re-run green.
  • python3 scripts/check_docs.py — docs OK, 83 Prometheus + 18 Loki rules,
    7 dashboards, 141 panels, 10 assertions. No counted claim moves: the buy
    table keeps its row count, no runbook file is added, and adding a case for
    an already-tested alert leaves the coverage sentences alone.
  • npx markdownlint-cli2 clean across all 92 files (MD028 caught and fixed)
  • Commit messages and this description scanned multiline for GitHub's
    close keywords — none present, so merging leaves the issue open. Worth
    knowing: the parser does not read negation, so a sentence saying the
    change does not close an issue still closes it. An earlier draft of
    this description did exactly that, twice.

One fact here is not a measurement: the old cell's recycling date,
2026-09-18, comes from the operator rather than from an instrument.

What stays with the operator

Step 8, once status reads Full: pull prometheus's brick only, expect
HostOnBattery within four minutes and UpsOnBattery silent, measure the
runtime, stop at 20 minutes or 50 %. That is what settles #454.

🤖 Generated with Claude Code

Gerrrt and others added 3 commits September 18, 2026 19:18
… clock stepped (#454)

The cell went into `prometheus` on 2026-09-18. This page still said it was in
hand and nothing was fitted, so the status banner, the Observed column and
steps 6 through 9 are now the record of what happened rather than a plan.

TWO THINGS THE PAGE HAD WRONG, both found by doing it.

The step 1 table marked `charge_full_design` and `voltage_min_design` "must not
move". Both moved: 6.6 -> 6.8 Ah and 11.21 -> 11.4 V. They are figures the
pack's own gas gauge reports, not properties of the model, and an aftermarket
A1437 reports its own. The expectation is inverted here rather than restated,
because on a machine that exports no serial number that movement is the only
proof the part changed at all — `manufacturer` and `model_name` came back
byte-identical, `SMP` and `bq20z451`, exactly as step 7 warned. Step 7's three
proof rows become four, and the new one is the strongest of them.

Disconnecting the cell clears the RTC. The post-swap boot opens
`2026-07-28 15:04:45`; systemd-timesyncd restored the clock to 14:53:37, the
second the machine went down, and the stack then ran about fourteen minutes
writing samples at backdated timestamps before NTP stepped it forward to the
true 18:12. So step 6's "measure the hole" query returns a confident wrong
answer after a battery swap: the 184-minute gap it shows is not downtime, and
fourteen minutes of real post-swap operation sits inside it. The step now says
to take the window from `journalctl --list-boots`. Whether anything should
notice a backward boot is #519 — `HostClockSkew` reads the kernel's current
offset, which stayed at 0.004 s throughout, while `node_timex_sync_status` was
0 for exactly the backdated window and is read by no rule.

STEP 2 WAS NEVER PERFORMED, and it can never be performed for this swap: the
old cell is gone. `HostOnBattery` had never fired for `prometheus` across the
whole 30-day retention. What fired on the day was accidental — the machine came
back unplugged at 41 % — so the discrimination step 2 was written to buy, a
step 8 failure being the cell or the adapter and nothing else, is unavailable.
That is recorded in *What is still open* rather than glossed.

STEP 8 IS STILL OWED and is the whole of what keeps #454 open. The pack was
still charging when the fit was recorded.

Also: the old cell went for recycling the same day (step 9); the window ran to
about three hours and eighteen minutes against a stated bound of two; the
firing set came back identical to the step 1 snapshot, all four alerts
pre-existing; and the first *What is still open* bullet is deleted because #506
already fixed what it described, with its closure recorded at the end of the
section instead.

Verified: `npx markdownlint-cli2` clean; `python3 scripts/check_docs.py` prints
docs OK — 83 Prometheus + 18 Loki rules, 7 dashboards, 141 panels, 10
assertions. Every reading quoted here was read from this host's own Prometheus
on 2026-09-18, not remembered.

Refs #454
Refs #519

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four documents restated `prometheus`'s cell as 94 % of design after 108 cycles
in the present tense. That pack was removed on 2026-09-18 and recycled the same
day. Each now says what the estate has: `oracle` at 72 % and unbought,
`prometheus` replaced and reading 101 % of its design capacity at one cycle.

`hardware.md` carries the durable record. The Accessories bullet goes from
"delivered and not yet fitted" to fitted the same day, and the compatible-cell
wording — written to be settled once someone could look at the pack — is
settled sideways: the gas gauge reports identically, `SMP` and `bq20z451`,
because an aftermarket A1437 reuses it, but both design figures moved, which is
the only proof of a different part on a machine that exports no serial. The
forward-looking checklist becomes what was checked at the fit and what was not,
with the mains pull named as the reason #454 is still open. The mains-cut
dependency paragraph gains the sentence that is actually load-bearing: how long
either laptop runs on its cell has never been measured.

`security.md`'s mains-power row said the same thing and now ends on the same
limit — the row is answered as far as the cells being healthy and no further.

`fit-the-ups-battery.md`'s laptop-cell paragraph delegates to the other
runbook; it now records that the capacity half was proved at the fit and the
mains pull was not.

`roadmap.md` gets four corrections. The 2026-09-13 purchase paragraph pointed
at `fit-the-ups-battery.md` for the test that would settle the issue — the rack
pack in `mjolnir`, not this cell — which is the same wrong pointer #506 fixed
in the alert's own description; it now names `replace-the-laptop-cell.md` and
says it was wrong.
A dated paragraph records the fit. The cell leaves "already paid for and still
moving" into neither list, because those two paragraphs track money that has
not yet become a working part and this has. The *Never* carve-out keeps its
94 % measurement, scoped to the cell that was there then, and records the
exception being exercised rather than restated.

THE BUY TABLE DOES NOT MOVE. The cell left it on 2026-09-13 and fitting is not
buying, so README's "two items now, one later" is unaffected. There is also NO
`## Done` ENTRY: #454 is open until step 8 runs, and the precedent for work
landed against an open issue is #418, which is recorded as dated progress
paragraphs rather than a checkbox.

Verified: `python3 scripts/check_docs.py` prints docs OK — 83 Prometheus + 18
Loki rules, 7 dashboards, 141 panels, 10 assertions; `npx markdownlint-cli2`
clean across all 92 files.

Refs #454

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… ratio above 1 is new (#454)

`prometheus`'s cell was replaced on 2026-09-18. Nothing in these three
expressions changes, and that is the point worth recording rather than
assuming: `HostBatteryHealthLow` reads the RATIO of `charge_full` to
`charge_full_design`, so a pack that reports its own design figure — 6.6 -> 6.8
Ah here, with `voltage_min_design` 11.21 -> 11.4 V beside it — is absorbed
rather than breaking the rule. The section comment in `host.rules.yaml` says
so, keeps the 2026-09-12 readings because they are what these rules were
written against, and records that the pack they describe is gone.

THE QUIET CASE KEEPS ITS NUMBERS DELIBERATELY. 6.196 of 6.6 Ah is 0.9388, which
sits 0.14 above the 0.8 line; the pack now fitted sits 0.21 above it. Refreshing
the case to the new reading would weaken it — a threshold quietly raised to 0.95
fails the 94 % case and would pass one built on 101 %. Only the comment changes,
because it called these "prometheus's real reading" and that is no longer true.

A SECOND QUIET CASE IS ADDED at 6.889 / 6.8. A ratio above 1 is an input shape
this rule had never been fed: an aftermarket pack reports its own design figure
and can measure above it, and the rule has to stay quiet rather than fall
through to the `energy_*` branch. Adding a case for an alert already in the
tested set moves no counted claim, which is why `docs/observability.md`'s
coverage sentences are untouched.

Verified: `./scripts/validate.sh` — `promtool test rules` over 11 observability
files passes through the pinned `prom/prometheus` image. The new case was proved
to assert something rather than pass vacuously by flipping its `exp_alerts: []`
to expect a firing alert, confirming the run FAILS, and flipping it back.

Refs #454

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Gerrrt
Gerrrt merged commit 16e92d3 into main Sep 18, 2026
3 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/prometheus-battery-replacement-941acd branch September 18, 2026 19:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant