docs: the SSDs are in Saruman's bays, unassigned, and three predictions were wrong (#418)

[Gerrrt]

Refs #418, Refs #148, Refs #76 — no close keywords. The drives are fitted and nothing after the bays has happened; the measurement is what closes #418.

What happened

Both SM863a drives went into Saruman's bays 3 and 4 on 2026-09-18. The iLO reported them in the 20:11 UTC scrape, absent at 20:10 — both in the same minute:

Reading	Bay 3 (idx 2)	Bay 4 (idx 3)
cpqDaPhyDrvSerialNum	S3F3NX0K601487	S3F3NX0K806107
cpqDaPhyDrvModel / FWRev	SAMSUNG / GXM5304Q	same
cpqDaPhyDrvSize	915715 MB	same
MediaType / RotationalSpeed / Type / NegotiatedLinkRate	3 solidState / 5 rpmSsd / 3 sata / 4	same
Condition / Status / SmartStatus	2 / 2 / 2 ok	same
ConfigurationStatus	3 notConfigured	same
SmartCarrierAppFWRev / BootldrFWRev	11 / 6 — the same as bays 1–2	same

cpqDaLogDrv* still reads index 1 only; cpqDaAccel* and cpqDaCntlr* are unchanged from the 2026-09-17 baseline; scrape_duration_seconds{device="shiva"} is 11.4 s either side of the fit. No alert on device="shiva" fired for it, and no silence existed — step 1's silences are for step 5's sync, and inserting a drive into an empty bay moves no condition column.

This is step 4 of the runbook and no other step. No ssacli, no logical drive, no thin pool, alexander still on the HDD mirror, no fio, no cache reading for #76.

What this PR does

• fit-the-saruman-ssds.md — status block rewritten for 2026-09-18; step 4's tray precondition becomes past tense with the carrier firmware as the proof; step 11's Observed column is filled for the physical-drive rows; items 2, 3 and 5 of "what this runbook does not know" are struck through with the reading that settled them; step 1 says to create its silences immediately before step 5, not before step 4.
• hardware.md — the Compute table's Storage column moves, as the accessories entry said it would on the day Fit the two SSDs in Saruman, and re-derive what the spindles decided #418 fitted them: 2× 1 TB SAS HDD, RAID 1; 2× 960 GB SATA SSD, unassigned. The SM863a entry carries both serials, the firmware and the link rate in the S3520 entry's shape, and says whose reading they are. The tray entry is fitted. The duplicated "reading them back afterwards" sentence in the SSD entry is gone.
• roadmap.md — the trays leave still moving, the SSDs leave on hand, and a dated note records the fit; the Fit the two SSDs in Saruman, and re-derive what the spindles decided #418 entry says fitted-and-stopped-at-the-bays; the Build the lab Windows domain on Saruman (ADR-0029) #414 paragraph no longer says the drives are on a shelf.

Three things the runbook predicted wrongly

• The model string. Step 11 expected MZ7KM960...; the iLO reports SAMSUNG. It names a third-party SATA drive by vendor where it gives the HPE disks their part number. The part is proved by size, firmware and serial prefix instead, and the row now says so.
• The walk cost. Step 10's table predicted 13–17 s; two more drives cost the walk nothing measurable.
• Labels first. Step 4 said to read the serials off the labels before insertion, because reading them back afterwards is reading them through the tool the fit is meant to verify. That was not done. The runbook now records that while both drives are Unassigned a label check is still safe — a drive that is a member of nothing can be pulled and reseated — and that the window closes at step 5.

Two readings worth having

• Wear is blank on the SSDs: cpqDaPhyDrvSSDWearStatus 1 other, SSDPercntEndrnceUsed / PowerOnHours / SSDEstTimeRemainingHours all 4294967295, as on the HDDs. cpqDaPhyDrvHasMonInfo also reads false on all four, and the MIB allows an unconfigured drive to go unpolled, so step 11 re-reads these after step 5. If they are still blank once the drives are in a logical drive, the closing paragraph's smartctl -d cciss issue gets opened — not before.
• cpqDaPhyDrvMaximumTemperature is a lifetime figure: 62 °C on Bay 3 against a 60 threshold, 51 on Bay 4, both at 30–32 °C now. Second-hand drives carrying their previous life's number — the kind of reading direct SMART would have shown at purchase.

What is deliberately not here

No ADR changes. ADR-0029's ninety is stale the day step 8's measurement lands, not the day the drives did. No network.rules.yaml silence paragraph and no test-fixture changes — no silence exists yet. No wear issue — gated on the re-read after step 5. No thin-pool blind-spot issue — the pool does not exist.

Also seen and not chased: the iLO's SNMP was unreachable 14:54–15:03 UTC the same day (InstanceDown fired on 10.0.30.10); Saruman and alexander stayed up throughout.

Blast radius

Documentation only.

• No change to network segmentation or firewall rules
• No new port published to a VLAN that could not already reach the service
• No credential added outside secrets/*.sops.yaml

Verification

python3 scripts/check_docs.py → docs OK — 83 Prometheus + 18 Loki rules, 7 dashboards, 141 panels, 10 assertions
./scripts/lint.sh --require-all → lint passed
Serials and bays in the diff re-read from Prometheus at commit time and match.

• make validate passes (check-docs and lint stages; rules unchanged)
• Deployed to the lab and confirmed working — n/a, documentation only
• Docs updated

What is next, and who runs it

From the Mac, on Saruman: step 2 (ssacli, record which path worked), step 3 (capture the Cache Ratio and Interface Type lines verbatim — the #76 and item-12 readings), the optional label check while both drives are unassigned, then step 1's silences from prometheus immediately before step 5, steps 5–9 on the Mac, steps 10–11 from prometheus. The runbook's "Flipping the documents" list is the completion PR.

Refs #418

🤖 Generated with Claude Code