docs(prometheus): the A1437 cell is fitted, and the swap corrected two documents (#454) - #520
Merged
Merged
Conversation
… clock stepped (#454) The cell went into `prometheus` on 2026-09-18. This page still said it was in hand and nothing was fitted, so the status banner, the Observed column and steps 6 through 9 are now the record of what happened rather than a plan. TWO THINGS THE PAGE HAD WRONG, both found by doing it. The step 1 table marked `charge_full_design` and `voltage_min_design` "must not move". Both moved: 6.6 -> 6.8 Ah and 11.21 -> 11.4 V. They are figures the pack's own gas gauge reports, not properties of the model, and an aftermarket A1437 reports its own. The expectation is inverted here rather than restated, because on a machine that exports no serial number that movement is the only proof the part changed at all — `manufacturer` and `model_name` came back byte-identical, `SMP` and `bq20z451`, exactly as step 7 warned. Step 7's three proof rows become four, and the new one is the strongest of them. Disconnecting the cell clears the RTC. The post-swap boot opens `2026-07-28 15:04:45`; systemd-timesyncd restored the clock to 14:53:37, the second the machine went down, and the stack then ran about fourteen minutes writing samples at backdated timestamps before NTP stepped it forward to the true 18:12. So step 6's "measure the hole" query returns a confident wrong answer after a battery swap: the 184-minute gap it shows is not downtime, and fourteen minutes of real post-swap operation sits inside it. The step now says to take the window from `journalctl --list-boots`. Whether anything should notice a backward boot is #519 — `HostClockSkew` reads the kernel's current offset, which stayed at 0.004 s throughout, while `node_timex_sync_status` was 0 for exactly the backdated window and is read by no rule. STEP 2 WAS NEVER PERFORMED, and it can never be performed for this swap: the old cell is gone. `HostOnBattery` had never fired for `prometheus` across the whole 30-day retention. What fired on the day was accidental — the machine came back unplugged at 41 % — so the discrimination step 2 was written to buy, a step 8 failure being the cell or the adapter and nothing else, is unavailable. That is recorded in *What is still open* rather than glossed. STEP 8 IS STILL OWED and is the whole of what keeps #454 open. The pack was still charging when the fit was recorded. Also: the old cell went for recycling the same day (step 9); the window ran to about three hours and eighteen minutes against a stated bound of two; the firing set came back identical to the step 1 snapshot, all four alerts pre-existing; and the first *What is still open* bullet is deleted because #506 already fixed what it described, with its closure recorded at the end of the section instead. Verified: `npx markdownlint-cli2` clean; `python3 scripts/check_docs.py` prints docs OK — 83 Prometheus + 18 Loki rules, 7 dashboards, 141 panels, 10 assertions. Every reading quoted here was read from this host's own Prometheus on 2026-09-18, not remembered. Refs #454 Refs #519 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four documents restated `prometheus`'s cell as 94 % of design after 108 cycles in the present tense. That pack was removed on 2026-09-18 and recycled the same day. Each now says what the estate has: `oracle` at 72 % and unbought, `prometheus` replaced and reading 101 % of its design capacity at one cycle. `hardware.md` carries the durable record. The Accessories bullet goes from "delivered and not yet fitted" to fitted the same day, and the compatible-cell wording — written to be settled once someone could look at the pack — is settled sideways: the gas gauge reports identically, `SMP` and `bq20z451`, because an aftermarket A1437 reuses it, but both design figures moved, which is the only proof of a different part on a machine that exports no serial. The forward-looking checklist becomes what was checked at the fit and what was not, with the mains pull named as the reason #454 is still open. The mains-cut dependency paragraph gains the sentence that is actually load-bearing: how long either laptop runs on its cell has never been measured. `security.md`'s mains-power row said the same thing and now ends on the same limit — the row is answered as far as the cells being healthy and no further. `fit-the-ups-battery.md`'s laptop-cell paragraph delegates to the other runbook; it now records that the capacity half was proved at the fit and the mains pull was not. `roadmap.md` gets four corrections. The 2026-09-13 purchase paragraph pointed at `fit-the-ups-battery.md` for the test that would settle the issue — the rack pack in `mjolnir`, not this cell — which is the same wrong pointer #506 fixed in the alert's own description; it now names `replace-the-laptop-cell.md` and says it was wrong. A dated paragraph records the fit. The cell leaves "already paid for and still moving" into neither list, because those two paragraphs track money that has not yet become a working part and this has. The *Never* carve-out keeps its 94 % measurement, scoped to the cell that was there then, and records the exception being exercised rather than restated. THE BUY TABLE DOES NOT MOVE. The cell left it on 2026-09-13 and fitting is not buying, so README's "two items now, one later" is unaffected. There is also NO `## Done` ENTRY: #454 is open until step 8 runs, and the precedent for work landed against an open issue is #418, which is recorded as dated progress paragraphs rather than a checkbox. Verified: `python3 scripts/check_docs.py` prints docs OK — 83 Prometheus + 18 Loki rules, 7 dashboards, 141 panels, 10 assertions; `npx markdownlint-cli2` clean across all 92 files. Refs #454 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… ratio above 1 is new (#454) `prometheus`'s cell was replaced on 2026-09-18. Nothing in these three expressions changes, and that is the point worth recording rather than assuming: `HostBatteryHealthLow` reads the RATIO of `charge_full` to `charge_full_design`, so a pack that reports its own design figure — 6.6 -> 6.8 Ah here, with `voltage_min_design` 11.21 -> 11.4 V beside it — is absorbed rather than breaking the rule. The section comment in `host.rules.yaml` says so, keeps the 2026-09-12 readings because they are what these rules were written against, and records that the pack they describe is gone. THE QUIET CASE KEEPS ITS NUMBERS DELIBERATELY. 6.196 of 6.6 Ah is 0.9388, which sits 0.14 above the 0.8 line; the pack now fitted sits 0.21 above it. Refreshing the case to the new reading would weaken it — a threshold quietly raised to 0.95 fails the 94 % case and would pass one built on 101 %. Only the comment changes, because it called these "prometheus's real reading" and that is no longer true. A SECOND QUIET CASE IS ADDED at 6.889 / 6.8. A ratio above 1 is an input shape this rule had never been fed: an aftermarket pack reports its own design figure and can measure above it, and the rule has to stay quiet rather than fall through to the `energy_*` branch. Adding a case for an alert already in the tested set moves no counted claim, which is why `docs/observability.md`'s coverage sentences are untouched. Verified: `./scripts/validate.sh` — `promtool test rules` over 11 observability files passes through the pinned `prom/prometheus` image. The new case was proved to assert something rather than pass vacuously by flipping its `exp_alerts: []` to expect a firing alert, confirming the run FAILS, and flipping it back. Refs #454 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
The cell went into
prometheuson 2026-09-18. Every document still said it wasin hand and unfitted; they now describe the machine that exists. Nothing here
changes a rule expression, adds a metric or needs a deploy.
charge_fullcharge_full_designcyclecountvoltage_min_designmanufacturer/model_nameSMP/bq20z451HostBatteryNotReportedis quiet, every scrape target is up, and the firing setcame back identical to the pre-swap snapshot —
ScheduledJobFailed(
backup-volumes),SecretsKeyBackupUnproven,RebootRequiredandGatewayMonitorUnreliablehad all been firing continuously for the four daysbefore, so none of them is this work.
#454 stays open
The runbook names step 8 — mains pulled on a fully charged cell with the
runtime measured — as the step that settles the issue, and it has not been run.
The pack was still charging when these readings were taken. Every commit here
says
Refs, and there is deliberately no## Doneentry on the roadmap; theprecedent for work landed against an open issue is #418.
What fired on the day was accidental: the machine came back from the swap
unplugged at 41 %, so
HostOnBatterywent pending and then firing forprometheusand nothing else, withUpsOnBatteryquiet throughout. That is thediscriminator the runbook wanted, arrived at by accident, on a clock that was
three hours wrong. It is not a bounded test and it measured no runtime.
Step 2 was never performed and now cannot be.
HostOnBatteryhad neverfired for
prometheusacross the whole 30-day retention, and the old cell isgone, so the discrimination step 2 was written to buy — a step 8 failure being
the cell or the adapter and nothing else — is unavailable. Recorded rather than
glossed, and worth running properly when the page is reused on
oracle.Two things the swap showed the documents had wrong
The design figures moved, and the runbook said they could not. Step 1's
table marked
charge_full_designandvoltage_min_design"must not move". Bothdid. They are figures the pack's own gas gauge reports, not properties of the
model, and an aftermarket A1437 reports its own. The expectation is inverted
rather than restated, because on a machine that exports no serial number
that movement is the only proof the part changed at all — the gauge identifiers
came back byte-identical, exactly as step 7 warned. Step 7's three proof rows
become four. Nothing in the rules depends on either figure:
HostBatteryHealthLowreads the ratio, so the movement is absorbed.Disconnecting the cell reset the RTC, and nothing noticed — filed as
#519. The post-swap boot opens
2026-07-28 15:04:45;systemd-timesyncdrestored the clock to 14:53:37, thesecond the machine went down, and the stack then ran ~14 minutes writing samples
at backdated timestamps before NTP stepped it forward to the true 18:12.
So the 184-minute "hole" at 15:08–18:12 is not downtime, real post-swap
operation is filed inside the outage, and the true window — 14:53:37 to about
18:12, roughly 3 h 18 m — is over this page's own two-hour bound and not
recoverable from the series at all. Step 6's measure the hole query therefore
gives a confident wrong answer after a battery swap, and now says to take the
window from
journalctl --list-boots.HostClockSkewcould not have caught it:it reads
abs(node_timex_offset_seconds), which peaked at 0.004 s, becausethe kernel's correction says nothing about a clock that is wrong but stable.
node_timex_sync_statuswas0for exactly the backdated window and is read byno rule — that is #519's question, and no rule lands here.
Blast radius
Verification
./scripts/validate.shpasses —promtool test rulesover 11observability files through the pinned image,
documents agree with the configs, 3 expected host-only skipsexp_alerts: []toexpect a firing alert makes the run FAIL with
got:[], confirming therule is genuinely quiet at a ratio of 1.013. Flipped back and re-run green.
python3 scripts/check_docs.py— docs OK, 83 Prometheus + 18 Loki rules,7 dashboards, 141 panels, 10 assertions. No counted claim moves: the buy
table keeps its row count, no runbook file is added, and adding a case for
an already-tested alert leaves the coverage sentences alone.
npx markdownlint-cli2clean across all 92 files (MD028 caught and fixed)close keywords — none present, so merging leaves the issue open. Worth
knowing: the parser does not read negation, so a sentence saying the
change does not close an issue still closes it. An earlier draft of
this description did exactly that, twice.
One fact here is not a measurement: the old cell's recycling date,
2026-09-18, comes from the operator rather than from an instrument.
What stays with the operator
Step 8, once
statusreadsFull: pullprometheus's brick only, expectHostOnBatterywithin four minutes andUpsOnBatterysilent, measure theruntime, stop at 20 minutes or 50 %. That is what settles #454.
🤖 Generated with Claude Code