From b612b97cdedc236ed2189c07960afca8fbfd39a8 Mon Sep 17 00:00:00 2001 From: Garrett Allen Date: Thu, 17 Sep 2026 23:37:17 +0000 Subject: [PATCH] docs(runbooks): the Saruman SSD fit becomes a procedure, and the trays gate it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit nothing the documents say. This adds the document that makes the fit provable when it happens, and records what has actually been decided. The layout question the issue left open is answered: the SSDs become a second RAID 1 logical drive on the P440ar, Smart Array managed, with the 7.2K mirror keeping Proxmox, the ISOs and the backups. That is the smaller change, it keeps a spindle for the things that do not need IOPS, and it leaves IloDrivePredictiveFailure, IloDriveSmartUnreadable and the replaceDriveSSDWearOut(4) state reading cpqida.mib exactly as they do today — so #351's decision to skip smart-state on this host stays right rather than needing re-checking. The runbook's "before" column is read, not assumed: every cpqDa* value in it came out of Prometheus on 2026-09-17. That reading also sharpens #76. cpqDaAccelWriteCachePercent has been the one column anyone looked at; the baseline shows cpqDaAccelMemory, cpqDaAccelReadMemory and both cache percents all at 0 against a 2 GiB module, with cpqDaLogDrvHasAccel{1} reading other rather than enabled. Four columns agreeing is much weaker support for "the iLO does not populate the ratio" than one was. The runbook takes the ssacli reading that settles it and hands the finding to #76 without acting on it: modify cacheratio= is controller-wide and would land on the array holding every guest, and it would move two variables at once underneath the measurement. What is deliberately not here: ADR-0029's ninety random write IOPS is derived, not measured — seek plus half a rotation at 7200 rpm — so replacing it with a differently-shaped measured number would be worse than leaving it alone. The runbook measures both arrays at the parameters that derivation implies, 4 KiB at queue depth 1, three times. Until that reading exists ADR-0029, ADR-0007, ADR-0017 and the roadmap's #414 paragraph stand as written, and then each gets a dated note rather than a silent edit. The fit is blocked, and the runbook says so at the top. #498 recorded two SFF trays bought 2026-09-11 and still in transit; a Gen9 bay holds a drive only in a carrier, so step 4 is a stop condition rather than something discovered at the rack. Steps 0 to 3 can all be done first, and step 2 — getting ssacli onto Debian 13, which nothing in this repository is evidence of — is the one most worth failing early. README's runbook count goes to 28 for the runbook this adds. The ADR counter and the duplicate ADR-0041 that had check-docs failing on main are both fixed there now, by #503, so nothing about them is carried here. Refs #418, #76, #148 Co-Authored-By: Claude Opus 5 --- README.md | 2 +- docs/hardware.md | 15 +- docs/roadmap.md | 53 +- docs/runbooks/fit-the-saruman-ssds.md | 781 ++++++++++++++++++++++++++ 4 files changed, 837 insertions(+), 14 deletions(-) create mode 100644 docs/runbooks/fit-the-saruman-ssds.md diff --git a/README.md b/README.md index 43880117..ca0b8865 100644 --- a/README.md +++ b/README.md @@ -93,7 +93,7 @@ documents for different readers. from `compose.yaml` too, so an image that is not pinned there cannot be run at all. - **Documented decisions and runbooks.** 42 ADRs covering what was chosen - and what was rejected — including the costs accepted knowingly; 27 + and what was rejected — including the costs accepted knowingly; 28 runbooks for the operations that are easy to get wrong at 1am, one of which is the handover page a successor reads first. diff --git a/docs/hardware.md b/docs/hardware.md index d0384549..12c4a16a 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -325,10 +325,18 @@ revisions of this repository treated `shiva` as the hypervisor itself. decision on `Saruman` starts from: a 7.2K mirror serving about ninety random write IOPS ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)). - Serials go here when they land. The Compute table's Storage column changes + Serials go here when they land — read off the labels before the drives go + into the bays, which is step 4 of + [`fit-the-saruman-ssds.md`](runbooks/fit-the-saruman-ssds.md), because + reading them back afterwards means reading them through the tool the fit is + trying to verify. The Compute table's Storage column changes when [#418](https://github.com/Gerrrt/HomeLab/issues/418) fits them, and not before — that issue also names the ADRs whose arithmetic the fit makes - stale + stale. The fit itself is + [`fit-the-saruman-ssds.md`](runbooks/fit-the-saruman-ssds.md), whose step 4 + reads both serials off the labels before the drives go into the bays — + reading them back afterwards means reading them through the tool the fit is + trying to verify - 2× HP 2.5" SFF drive tray, `651687-001`[^Caddy] — bought 2026-09-11, in transit, quoted to arrive by 2026-09-17 and not here on the morning of it. The carriers the SM863a pair above needs to sit in `Saruman`'s SFF bays @@ -350,6 +358,9 @@ revisions of this repository treated `shiva` as the hypervisor itself. this machine; that both arrived; and that each has its screws, since a carrier without them holds a drive no better than no carrier. `651687-001` is the listing's part number and not yet the tray's. + **Until they land the SSDs cannot be fitted at all**, which is the sharpest + form of the point #418 is built on: arriving is not fitting, and neither the + Compute table nor ADR-0029's arithmetic moves for a drive on a shelf. - MikroTik CRS326-24G-2S+RM[^CRS326] — 24 × 1 GbE, 2 × SFP+, 1U, dual-boot RouterOS / SwOS — bought used 2026-09-13; in transit, delivery estimated 2026-09-23, moved out from the 09-16 to 09-21 window quoted at purchase. The diff --git a/docs/roadmap.md b/docs/roadmap.md index 90c8307f..30eaaab7 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -851,7 +851,9 @@ what left this one unfireable for months. the endpoints per session, because a 7.2K mirror serves about ninety random write IOPS and six idle Windows guests would be most of them — the number #418's SSDs were bought against, and the one their *fit* re-derives; a drive - in transit changes no duty cycle. Three things + on the shelf changes no duty cycle either, and since 2026-09-11 that is + where they are — waiting on the trays that let them into the bays at all. + Three things that ADR left explicit because they fail quietly: the DC takes its clock from the gateway, not `time.windows.com` — ADR-0014 named that failure and did not fix it, and the alert reads the *sync source* rather than the offset, because @@ -943,16 +945,45 @@ what left this one unfireable for months. → [runbook](runbooks/build-the-playground.md) - **[#418](https://github.com/Gerrrt/HomeLab/issues/418) Fit the two SSDs in `Saruman`.** Two Samsung SM863a 960 GB SATA enterprise drives, bought - 2026-09-09, in transit. ADR-0007's constraint — "the fleet - is sized against spindles, not RAM" — became a number in ADR-0029, about - ninety random write IOPS for the whole machine, and that number sized the - lab domain's duty cycle and #266's indexer. The SSDs raise the ceiling; what - the issue leaves to the fit is whether they replace the mirror or sit beside - it, whether the Smart Array manages them — the drive rules read `cpqida.mib` - and a layout the controller does not own falls out of all of them — and - what #76's write cache does for the new logical drive. The fit makes - ADR-0029's derivation stale on the day, and the issue says so rather than - leaving the ADR to be silently edited. + 2026-09-09, **delivered 2026-09-11, not fitted and not yet fittable**. + ADR-0007's constraint — "the + fleet is sized against spindles, not RAM" — became a number in ADR-0029, + about ninety random write IOPS for the whole machine, and that number sized + the lab domain's duty cycle and #266's indexer. The SSDs raise the ceiling. + + **Two of the three questions the issue left to the fit are now answered, and + the one that matters is not.** The layout is decided: the SSDs become a + *second* logical drive on the P440ar, RAID 1, Smart Array managed, with the + 7.2K mirror keeping Proxmox, the ISOs and the backups. That is the smaller + change, it keeps a spindle for the things that do not need IOPS, and it + leaves `IloDrivePredictiveFailure`, `IloDriveSmartUnreadable` and the + `replaceDriveSSDWearOut(4)` state reading `cpqida.mib` exactly as they do + today — so #351's decision to skip `smart-state` on this host stays right + rather than needing re-checking. #76's cache is **read and handed over, not + changed**: `modify cacheratio=` is controller-wide and would land on the + array holding every guest, so the runbook takes the `ssacli` reading nobody + has ever taken on this machine and #76 owns what to do about it. + + **What blocks it is neither of those: the trays.** A Gen9 bay holds a drive + only in a SmartDrive carrier, two were bought `651687-001` on 2026-09-11, + and on the morning of 2026-09-17 they had not arrived. Until they do, the + drives cannot enter the machine, which is why the runbook makes the carriers + a stop condition at step 4 rather than something discovered at the rack. The + same step confirms both drives are physically present — the delivery notice + covers one order, not two units, which the paragraph above leaves to this + issue to prove at the bays. + + What is still open after that is the only thing the purchase was for. + **ADR-0029's + ninety is derived, not measured** — seek plus half a rotation at 7200 rpm — + so the runbook measures both arrays at the parameters that derivation + implies, 4 KiB at queue depth 1, and three times: the HDD mirror loaded, the + SSD array idle, and the HDD mirror idle once the guest has moved off it. + Until that reading exists, ADR-0029, ADR-0007, ADR-0017 and the #414 + paragraph above all stand as written. The fit makes them stale on the day, + and it gets a dated note on each rather than a silent edit — ADR-0001 makes + them immutable. + → [runbook](runbooks/fit-the-saruman-ssds.md) ## Automation diff --git a/docs/runbooks/fit-the-saruman-ssds.md b/docs/runbooks/fit-the-saruman-ssds.md new file mode 100644 index 00000000..cec69617 --- /dev/null +++ b/docs/runbooks/fit-the-saruman-ssds.md @@ -0,0 +1,781 @@ +# Runbook: Fit the two SSDs in `Saruman` + +**Two drives, no maintenance window, and a number that has to be measured on +both arrays — not assumed from either.** + +> **Status — 2026-09-17: the drives are here, the trays are not. Nothing is +> fitted, and nothing can be.** +> +> Two Samsung SM863a 960 GB (`MZ-7KM960N`), bought 2026-09-09 and delivered +> 2026-09-11. The baseline in step 0 below was read from Prometheus on +> 2026-09-17 and is the "Before" column of every table here. +> +> **This runbook is blocked at step 4 until the drive trays land.** A Gen9 bay +> holds a drive only in an HPE SmartDrive carrier. Two were bought +> (`651687-001`, 2026-09-11) and had not arrived by the morning of 2026-09-17 +> — see [`../hardware.md`](../hardware.md). Steps 0 through 3 can all be done +> before they turn up, and doing them early is worth it: step 2 in particular +> is the one most likely to fail, and finding that out with the drives already +> in your hand is the wrong order. +> +> **The layout is decided and the measurement is not.** The SSDs become a +> *second* logical drive on the P440ar, RAID 1, Smart Array managed; the two +> 7.2K disks keep Proxmox, the ISOs and the backups. That answers the first two +> questions [#418](https://github.com/Gerrrt/HomeLab/issues/418) left to the +> fit. What it does not answer is the one the purchase exists for: the write +> IOPS this machine actually has. Step 8 measures it, on both arrays, with the +> parameters ADR-0029 derived its number from — and until that reading exists, +> no ADR here changes. + +`shiva` is the iLO, not the hypervisor. The host behind it is `Saruman` at +`10.0.30.110`; see [`../hardware.md`](../hardware.md). Both are on VLAN 30. + +## Where each step runs, and why it matters + +VLAN 30 is not reachable from the monitoring host. That is not an inconvenience +to work around in this runbook — it is the reason the runbook is split the way +it is. + +| Steps | Host | Who runs them | +| --- | --- | --- | +| 0, 1, 10, 11 | `prometheus`, `10.0.99.20`, VLAN 99 | the monitoring host — Prometheus and Alertmanager are here | +| 2–9 | `Saruman`, `10.0.30.110`, VLAN 30 | **you, from the Mac.** Nothing on VLAN 99 can reach this host | + +Run the `Saruman` half inside `tmux`. A dropped SSH session during step 9's +disk move leaves a half-mirrored disk, and that is a worse afternoon than a +reconnect. + +## Why this is not a maintenance window + +The SFF bays are hot-plug and `qm move-disk` on a running guest is an online +operation, so **no guest stops and no alert needs to be scheduled around**. +[`schedule-maintenance.md`](schedule-maintenance.md) is not the shape of this +job. Two alerts still get silenced in step 1, not because anything is going +down but because a freshly created logical drive may legitimately report +`degraded` while the controller syncs it, and `IloHardwareDegraded` is +`severity: critical` with a ten-second `group_wait` on the `urgent` receiver. + +## What this deliberately does not do + +- **It does not change the controller-wide cache ratio.** + `ssacli ctrl slot=0 modify cacheratio=` flushes and re-partitions the whole + 2 GiB module, and that pause lands on the logical drive holding Proxmox and + every guest. Fitting two drives into empty bays is not the occasion to + re-partition the cache the rest of the machine runs on. Step 6 *reads* the + ratio — which is the reading + [#76](https://github.com/Gerrrt/HomeLab/issues/76) has been waiting for since + 2026-09-02 — and hands the finding over. #76 owns the change. +- **It does not edit ADR-0029 or ADR-0007.** They are stale the day the + measurement lands, not the day the drives land. See "Flipping the documents". +- **It does not touch the existing mirror.** No `delete`, no `modify raid=`, + no `erase`, and nothing is pulled from bay 1 or bay 2. + +--- + +## 0. Record the baseline — on `prometheus`, before anything + +This is the step that makes the fit provable rather than assumed, and it is the +same argument [`replace-the-smart-storage-battery.md`](replace-the-smart-storage-battery.md) +makes: every other row in the after-table could in principle be the old +hardware reporting differently. + +```bash +curl -sS -G http://127.0.0.1:9090/api/v1/query --data-urlencode 'query={__name__=~"cpqDa(Accel|Cntlr|LogDrv|PhyDrv).*",device="shiva"}' | python3 -m json.tool +``` + +Two more, because they are the only readings that come from the *host* rather +than the iLO, and they are how you prove in step 11 that the guest's writes +actually moved: + +```bash +curl -sS -G http://127.0.0.1:9090/api/v1/query --data-urlencode 'query=scrape_duration_seconds{device="shiva"}' +curl -sS -G http://127.0.0.1:9090/api/v1/query --data-urlencode 'query=rate(node_disk_writes_completed_total{instance="Saruman"}[30m])' +``` + +**What that read on 2026-09-17**, and what each value is doing in this runbook: + +| Reading | Value | Why it is here | +| --- | --- | --- | +| `cpqDaCntlrHwLocation` | `Slot 0` | every `slot=0` below depends on it | +| `cpqDaCntlrOperatingMode` | `2` smartArrayMode | not HBA mode — the chosen layout is the one the controller is already in | +| `cpqDaCntlrModel` / `FWRev` | `75` / `7.00` | the controller this runbook was written against | +| `cpqDaLogDrv*` index set | **`1` only** | the new one should become `2`; step 5 verifies rather than assumes | +| `cpqDaLogDrvFaultTol{1}` | `3` mirroring | the existing RAID 1 | +| `cpqDaLogDrvSize{1}` | `953837` MB | | +| `cpqDaLogDrvHasAccel{1}` | **`1` other** | not `3` enabled — see step 6 | +| `cpqDaPhyDrv*` index set | **`0` and `1`** | the new drives should become `2` and `3` | +| `cpqDaPhyDrvLocationString{0,1}` | `Port 1I Box 1 Bay 1` / `Bay 2` | bays 3 and 4 are the target | +| `cpqDaPhyDrvModel{0,1}` | `MM1000GBKAL` | | +| `cpqDaPhyDrvSerialNum{0,1}` | `9XG62LSC` / `9XG72NXP` | the rows that cannot be faked | +| `cpqDaPhyDrvMediaType{0,1}` | `2` rotatingPlatters | the SSDs must read `3` solidState | +| `cpqDaPhyDrvRotationalSpeed{0,1}` | `2` rpm7200 | the SSDs must read `5` rpmSsd | +| `cpqDaPhyDrvType{0,1}` | **`3` sata** | see "What this runbook does not know", item 12 | +| `cpqDaPhyDrvSmartStatus{0,1}` | `2` ok | | +| `cpqDaPhyDrvSmartCarrierAppFWRev{0,1}` | `11` | the existing drives are in HPE SmartDrive carriers, which have their own firmware — step 4's carrier precondition is not hypothetical | +| `cpqDaAccelStatus` | `3` enabled | must not move | +| `cpqDaAccelTotalMemory` | `2097152` KB | 2 GiB fitted | +| `cpqDaAccelMemory` (write) | **`0`** | 0 KB allocated, against 2 GiB fitted | +| `cpqDaAccelReadMemory` | **`0`** | | +| `cpqDaAccelWriteCachePercent` | **`0`** | #76 | +| `cpqDaAccelReadCachePercent` | **`0`** | | +| `cpqDaAccelBadData` | `2` none | must not move; `3` is a different conversation | +| `cpqDaAccelBattery` / `BackupPowerSource` | `2` ok / `4` smartbattery | the pack fitted 2026-09-02 | +| `cpqDaAccelFailedBatteries` | `1` | a known curiosity from #76, not a target here | +| `cpqDaCntlrDriveWriteCacheState` | `1` other | step 6 leaves this alone deliberately | +| `scrape_duration_seconds{device="shiva"}` | `11.8` s | against `SnmpScrapeSlow`'s 30 s | +| guests | one — `alexander`, vmid `140`, qemu | step 9 is one command, not a campaign | + +**Four columns, not one.** #76 has been open on `cpqDaAccelWriteCachePercent` +reading `0`. The baseline above shows `cpqDaAccelMemory`, +`cpqDaAccelReadMemory`, `cpqDaAccelWriteCachePercent` and +`cpqDaAccelReadCachePercent` *all* reading `0` while +`cpqDaAccelTotalMemory` reports 2 GiB fitted, and `cpqDaLogDrvHasAccel{1}` +reading `other` rather than `enabled`. Four independent columns agreeing that +nothing is allocated is much weaker support for "the iLO simply does not +populate the ratio" than one column was. Step 6 is where that gets settled. + +## 1. Silence the two alerts a new array legitimately trips — on `prometheus` + +The repo documents *deleting* a silence +([`fit-the-ups-battery.md`](fit-the-ups-battery.md) §3) and has never written +down creating one. This is that call. + +```bash +START=$(date -u +%Y-%m-%dT%H:%M:%SZ) +END=$(date -u -d '+5 hours' +%Y-%m-%dT%H:%M:%SZ) +``` + +```bash +curl -sS -X POST http://localhost:9093/api/v2/silences -H 'Content-Type: application/json' --data "$(printf '{"matchers":[{"name":"alertname","value":"IloHardwareDegraded","isRegex":false,"isEqual":true},{"name":"device","value":"shiva","isRegex":false,"isEqual":true},{"name":"cpqDaLogDrvIndex","value":"2","isRegex":false,"isEqual":true}],"startsAt":"%s","endsAt":"%s","createdBy":"#418 fit the SSDs in Saruman","comment":"New RAID 1 logical drive on the P440ar may report degraded while the controller syncs it. Scoped to the NEW logical drive index so temperature, supplies, the controller and LD 1 stay live. Delete BEFORE the proving reading."}' "$START" "$END")" | python3 -m json.tool +``` + +```bash +curl -sS -X POST http://localhost:9093/api/v2/silences -H 'Content-Type: application/json' --data "$(printf '{"matchers":[{"name":"alertname","value":"IloHardwareDegraded","isRegex":false,"isEqual":true},{"name":"device","value":"shiva","isRegex":false,"isEqual":true},{"name":"cpqDaPhyDrvIndex","value":"2|3","isRegex":true,"isEqual":true}],"startsAt":"%s","endsAt":"%s","createdBy":"#418 fit the SSDs in Saruman","comment":"Two newly inserted physical drives on the P440ar. Scoped to the NEW drive indexes so drives 0 and 1 stay live. Delete BEFORE the proving reading."}' "$START" "$END")" | python3 -m json.tool +``` + +Record both UUIDs — they go in the roadmap and in the rules-file comment +afterwards: + +```bash +curl -sS http://localhost:9093/api/v2/silences | python3 -c 'import json,sys; [print(s["id"], s["status"]["state"], s["endsAt"], [(m["name"],m["value"]) for m in s["matchers"]]) for s in json.load(sys.stdin)]' +``` + +**Three properties, each of which this repository already argues for somewhere.** + +**Narrower than the precedent.** The battery silence on 2026-08-31 could only +manage `alertname` + `device`, and for five weeks a power-supply fault on the +same box would have been suppressed with the battery. +`IloHardwareDegraded`'s expression is an `or` of six series, so the firing +alert inherits the labels of whichever series matched — which means the +matcher can name `cpqDaLogDrvIndex` or `cpqDaPhyDrvIndex` and leave +temperature, supplies, the controller and the *existing* mirror completely +live. Use it. + +**Five hours, not a month.** The battery silence was written to +2026-10-01 and deleted on 2026-09-02. A silence should expire embarrassingly +soon rather than outlive the job. + +**`cpqDaLogDrvIndex` `2` is a prediction.** It is the most likely index and it +is not read yet. Step 5 verifies it within a minute of creating the drive; if +it comes back different, create the correct silence and delete this one +immediately. A silence keyed on the wrong index is a silence that does nothing +while you believe it is doing something. + +**Three alerts are deliberately *not* silenced.** See the verdict table before +step 10 for why each one is better left live. + +## 2. Get `ssacli` onto the hypervisor — the step most likely to stop you + +> **This is the largest unknown in this runbook and it is written as one.** +> `ssacli` is not in Debian and not in the Proxmox repositories. It comes from +> HPE's MCP SDR, which is not known to publish a Debian 13 `trixie` suite, and +> **nothing in this repository is evidence that it has ever been installed on +> this host.** Proxmox VE 9 is Debian 13. None of the three paths below has +> been watched working on this box. + +Path 1 — pin the `bullseye` suite and accept an out-of-suite package: + +```bash +curl -fsSL https://downloads.linux.hpe.com/SDR/hpPublicKey2048_key1.pub | gpg --dearmor > /usr/share/keyrings/hpe.gpg +echo "deb [signed-by=/usr/share/keyrings/hpe.gpg] https://downloads.linux.hpe.com/SDR/repo/mcp bullseye/current non-free" > /etc/apt/sources.list.d/hpe-mcp.list +apt-get update && apt-get install -y ssacli +``` + +`ssacli` is a largely self-contained vendor binary; it may or may not satisfy +trixie's `libc6` and `libstdc++6`. + +Path 2 — take the `.deb` directly from +`https://downloads.linux.hpe.com/SDR/repo/mcp/pool/non-free/` and `dpkg -i`, +resolving whatever it complains about. + +Path 3 — **no host package at all.** Reboot into Intelligent Provisioning and +use the offline Smart Storage Administrator through the iLO remote console. +This works with certainty and costs a reboot. The iLO 4 web UI's Storage page +is read-only and **cannot** create a logical drive, so it is not a fourth +option. + +A reboot changes the alert picture a little and not much: `HostRebooted` is +`severity: info` and routes to the `null` receiver; `RemoteWriteJobStale` needs +roughly twenty minutes of absence before it fires. A reboot shorter than that +is quiet. If path 3 is taken, steps 7 and 9 still need a shell, so you are +installing nothing and rebooting twice — prefer paths 1 and 2, and record +which one worked. + +## 3. Read the controller before touching a bay + +```bash +ssacli ctrl all show status +ssacli ctrl all show config +ssacli ctrl slot=0 show detail +ssacli ctrl slot=0 pd all show detail +ssacli ctrl slot=0 ld all show detail +``` + +`ctrl all show` must print `Smart Array P440ar in Slot 0 (Embedded)`. If it +prints a different slot, every `slot=0` below is wrong. That is the one thing +to check rather than copy. + +Capture verbatim and keep it — several of these lines are readings nothing in +this estate has ever taken: + +- **Cache Ratio**, **Total Cache Size**, **Total Cache Memory Available** +- **No-Battery Write Cache**, **Drive Write Cache** +- **Battery/Capacitor Status** and **Count** +- from `ld 1 show detail`: the **Caching** line +- from `pd all show detail`: the **Interface Type** of drives 1I:1:1 and + 1I:1:2 — see item 12 at the end + +## 4. Fit the drives + +The SFF bays are genuinely hot-plug, and inserting drives into **empty** bays +while logical drive 1 serves the running OS is a supported operation. Two +preconditions, stated here rather than discovered at the rack: + +- **HPE Gen8/Gen9 SFF SmartDrive carriers — bought, and not here yet.** Two + HP 2.5" SFF drive trays, `651687-001`, bought 2026-09-11 and still in + transit on 2026-09-17; [`../hardware.md`](../hardware.md) carries them. A + bare 2.5" drive does not seat in a ProLiant bay, so **this is where the job + stops until they land**. The baseline in step 0 shows + `cpqDaPhyDrvSmartCarrierAppFWRev` reading `11` on the existing drives, so + the carriers this machine uses are the firmware-carrying kind and not + improvised — a Gen10 part or a 3.5" LFF tray will not do. + + Check on arrival, per `hardware.md`: that both trays came, that each has its + screws, and that they are the Gen8/Gen9 SmartDrive carrier. One tray short + is one SSD fitted and one on a shelf, which is a worse outcome than waiting. + +- **Both drives, physically.** The delivery notice covers one order for the + pair, not two units confirmed. `docs/roadmap.md` leaves proving that to this + runbook, and the bays are where it gets proved: two drives in, two new + `cpqDaPhyDrv` indexes out. +- **Bays 3 and 4 must be cabled.** Both existing drives read `Box 1 Bay 1` and + `Bay 2`. The backplane variant on this chassis is recorded nowhere in this + repository. + +**Read both serial numbers off the drive labels before they go in.** They are +the one row in step 11's table that cannot be produced by the old hardware +reporting differently, and they are what +[`../hardware.md`](../hardware.md)'s *"Serials go here when they land"* is +waiting for. Writing them down after the drives are in a chassis means reading +them back through the tool you are trying to verify. + +Insert into bays 3 and 4, then: + +```bash +ssacli ctrl slot=0 pd all show +ssacli ctrl slot=0 pd 1I:1:3 show detail +ssacli ctrl slot=0 pd 1I:1:4 show detail +``` + +Both must appear as `1I:1:3` and `1I:1:4`, `Status: OK`, `Drive Type: +Unassigned Drive`. Confirm the model string and that **the serials match the +labels** before creating anything. + +Give the iLO time before reading Prometheus. The scrape is every 60 s, but the +iLO's own agentless refresh is not, and a drive absent from Prometheus five +minutes after insertion is not necessarily a fault. + +## 5. Create the logical drive + +```bash +ssacli ctrl slot=0 create type=ld drives=1I:1:3,1I:1:4 raid=1 +``` + +**Type the bay list. Never `drives=all`, never `drives=allunassigned`.** The two +HDDs are already assigned and would not be selected by either keyword — and the +cost of being wrong about that is the array holding every guest. A keyword one +letter from a different meaning is not worth the saving. + +```bash +ssacli ctrl slot=0 ld all show +ssacli ctrl slot=0 ld 2 show detail +``` + +**Verify the new index now**, against the silence created in step 1: + +```bash +# from prometheus +curl -sS -G http://127.0.0.1:9090/api/v1/query --data-urlencode 'query=cpqDaLogDrvCondition{device="shiva"}' +``` + +### Safe while logical drive 1 is in use, and not + +| Safe | Not safe, ever, in this runbook | +| --- | --- | +| Pulling a bay blank; inserting a drive into an empty bay | Pulling bay 1 or bay 2 | +| Any `ssacli ... show` | `ssacli ctrl slot=0 delete ...`, `modify raid=`, any `erase` | +| `create type=ld` from an explicit bay list | `create` with `drives=all` or `drives=allunassigned` | +| `ld 2 modify aa=enable`, `array B modify ssdsmartpath=` | `ctrl slot=0 modify cacheratio=` — see step 6 | +| `pvcreate` / `pvesh ... lvmthin` on the new device | Controller firmware update | +| `fio` against a purpose-made LV | `fio --filename=/dev/sda` or `/dev/sdb` | +| `qm move-disk` on the running guest | Anything at all while a `move-disk` is mirroring | + +## 6. The cache: read it, hand it to #76 + +Two cache commands are in scope here, and both are scoped to the new array. +Neither touches logical drive 1: + +```bash +ssacli ctrl slot=0 ld 2 modify aa=enable +ssacli ctrl slot=0 array B modify ssdsmartpath=enable +ssacli ctrl slot=0 ld 2 show detail +``` + +Check the spellings against the installed version before trusting them — +`ssacli ctrl slot=0 help create` and `help modify` are the authority, and these +options have moved between versions. Read **both** the `Caching` and the +`SSD Smart Path` lines out of `ld 2 show detail` afterwards rather than +predicting how they interact. + +Then take the reading that settles #76: + +```bash +ssacli ctrl slot=0 show detail | grep -iE "cache ratio|cache size|cache memory|drive write cache|no-battery" +ssacli ctrl slot=0 ld 1 show detail | grep -i caching +``` + +**State which answer you expect before you look**, so the reading can +contradict you: + +- **(a) `ssacli` reports a real ratio** — say `10% Read / 90% Write` — while + SNMP reports `0`/`0`. Then the iLO's agentless view does not populate those + columns on iLO 4 2.82, **#76 resolves as "this column is not readable on this + hardware"**, no rule may ever be written on it, and `IloWriteCacheDisabled` + reading `cpqDaAccelStatus` instead was right all along. +- **(b) `ssacli` agrees — `0% / 0%`, or `ld 1 ... Caching: Disabled`.** Then the + array has been running write-through since the pack went in *despite + reporting the accelerator enabled*, ADR-0029's *"That figure assumes no write + cache, which is the honest assumption here"* was literally true rather than + conservative, and the fix is `modify cacheratio=`. + +**The four columns in step 0 lean towards (b).** Either answer resolves #76. + +**The fix, if it is (b), is not taken here.** `modify cacheratio=` is +controller-wide: it flushes and re-partitions the 2 GiB module, and the pause +lands on logical drive 1 — Proxmox and `alexander`. It would also move two +variables at once underneath step 8's measurement, which is the one thing this +runbook exists to get right. Hand #76 the reading; #76 takes the change. + +For when it does: the ratio that suits an SSD logical drive is also the ratio +that suits the HDD mirror, so the mixed-media controller forces no compromise. +Read-ahead buys an SSD essentially nothing — which is why HPE built SSD Smart +Path to bypass the cache entirely for reads on RAID 0/1/1+0 SSD arrays — and +posted writes buy it little, because the drive services a 4 KiB write in well +under 100 µs and the controller's cache path is not an order of magnitude +faster than that. On the HDD mirror, write cache is the single largest lever +available on this machine. Low read, high write: `10/90`. + +### Drive-level cache: leave it alone + +```bash +ssacli ctrl slot=0 modify dwc=enable # do NOT run this +``` + +`cpqDaCntlrDriveWriteCacheState` reads `1` (other) today. The SM863a's own DRAM +buffer is capacitor-backed — that is the property the part was bought for, and +it means enabling the drives' caches would be safe *for these two drives*. But +`dwc` is **controller-wide** on this generation, so it would also enable the +caches on the two 7.2K spinners, whose buffers have no capacitor and whose only +protection is `mjolnir` — a UPS whose runtime reading has sat on the fabricated +`63` for the whole retained window +([`fit-the-ups-battery.md`](fit-the-ups-battery.md)). The SSDs lose almost +nothing; the HDDs would lose a guarantee resting on a pack whose runtime is not +yet proven. + +If `ssacli ctrl slot=0 help modify` shows a per-array form on the installed +version, that changes the answer and this runbook should take it. Check; do not +assume either way. + +## 7. Give Proxmox the new logical drive + +```bash +lsblk -o NAME,SIZE,MODEL,SERIAL +ls -l /dev/disk/by-id/ +``` + +If the new device does not appear, rescan rather than reboot: + +```bash +echo "- - -" > /sys/class/scsi_host/host0/scan +lsblk -o NAME,SIZE,MODEL,SERIAL +``` + +Then create the thin pool the way the GUI does, substituting the device that +actually appeared: + +```bash +pvesh create /nodes/Saruman/disks/lvmthin --name ssd --device /dev/sdb --add_storage 1 +pvesm status +``` + +**Why LVM-thin, as a decision rather than a default.** Plain LVM is disqualified +outright: PVE cannot snapshot it, and ADR-0027 leaves the lab with *revert and +not backup*, so snapshots are the only rollback this estate has for +`alexander`. A directory with qcow2 would give snapshots too, but stacks ext4 +plus qcow2 metadata on top of a logical drive that is already an abstraction, +and adds write amplification on flash for a capability this host does not use. +LVM-thin gives snapshots, thin provisioning and discard pass-through, and is +what [`build-the-lab-guest.md`](build-the-lab-guest.md) already assumes. + +> **One new blind spot, recorded rather than fixed.** A thin pool is not a +> filesystem, so `node_filesystem_*` never sees it — `Saruman` reports only +> `/`, `/boot/efi` and `/etc/pve`. `HostDiskCritical` and +> `HostDiskWillFillIn24h` are therefore **structurally blind** to the new pool, +> and a thin pool that fills makes every guest on it read-only. Either do not +> overprovision, or give the pool a textfile collector beside the existing +> `collect-guest-state.sh`. This is the same shape of gap as +> [#351](https://github.com/Gerrrt/HomeLab/issues/351) and deserves its own +> issue, not a step here. + +## 8. Measure what the array actually does — both of them + +This is the step the purchase exists for. **ADR-0029's number is derived, not +measured**, and replacing a derived number with a differently-shaped measured +one would be worse than leaving it alone. + +**What ADR-0029's figure implies.** *"A 7.2K SAS drive is roughly 83 random +IOPS at queue depth 1"* is a mechanical derivation — average seek plus half a +rotation at 7200 rpm is about 12 ms, which is about 83 operations a second. It +therefore carries **queue depth 1, one stream, no concurrency**, and no block +size at all, because seek-plus-rotate does not depend on block size at small +sizes. The comparable reconstruction is **4 KiB, `iodepth=1`, `numjobs=1`, +`direct=1`, random write**, and the comparison is only honest if those exact +parameters are used on both logical drives. A queue-depth-32 run is a +*different benchmark* — worth having as the new ceiling, not as the +re-derivation. + +```bash +apt-get install -y fio +vgs # confirm free extents in both VGs first +lvcreate -n fiotest -L 8G ssd +lvcreate -n fiotest -L 8G pve +``` + +Precondition both targets identically, so the SSD is measured in steady state +rather than on fresh flash: + +```bash +fio --name=fill --filename=/dev/ssd/fiotest --rw=write --bs=1M --iodepth=8 --ioengine=libaio --direct=1 --size=8G +fio --name=fill --filename=/dev/pve/fiotest --rw=write --bs=1M --iodepth=8 --ioengine=libaio --direct=1 --size=8G +``` + +The ADR-0029 comparison, run identically on both: + +```bash +fio --name=adr0029 --filename=/dev/pve/fiotest --rw=randwrite --bs=4k --iodepth=1 --numjobs=1 --ioengine=libaio --direct=1 --size=8G --time_based --runtime=60 --ramp_time=10 --randrepeat=0 --norandommap --group_reporting +``` + +The new ceiling, on the SSD array only: + +```bash +fio --name=ceiling --filename=/dev/ssd/fiotest --rw=randwrite --bs=4k --iodepth=32 --numjobs=4 --ioengine=libaio --direct=1 --size=8G --time_based --runtime=60 --ramp_time=10 --group_reporting +``` + +Record IOPS, `clat` mean and `clat` p99 from every run. + +**Three runs, and the ordering is the design:** + +| | Target | State | What it is for | +| --- | --- | --- | --- | +| **M1** | HDD `pve` | guest running | what the array delivers *today, under its real load*. Deliberately a loaded number | +| **M2** | SSD `ssd` | idle, before any guest data | the number that replaces ADR-0029's | +| **M3** | HDD `pve` | idle, **after step 9's move** | the clean re-derivation of the 83. Only obtainable once the array is quiet | + +M3 is why the measurement is not finished when step 8 ends. Until `alexander` +has moved off, this array is never idle, and a loaded HDD number compared +against an idle SSD number would flatter the SSDs by an unknown amount. Run the +queue-depth-32 variant against the HDD here too, and nowhere else. + +**The risk, plainly.** A queue-depth-1 4 KiB random write stream consumes +essentially the whole of the machine's ~90 IOPS budget for its sixty seconds, +so M1 **will** visibly stall `alexander`. Do it deliberately and briefly. A +queue-depth-32 run against the HDD mirror with a guest live would queue 32 deep +on a 90-IOPS array and hand the guest multi-second IO latency — which is why +that variant is confined to M3. + +Clean up. This is part of the step, not a footnote: + +```bash +lvremove /dev/ssd/fiotest +lvremove /dev/pve/fiotest +``` + +## 9. Move the guest + +There is exactly one guest, so this is one command rather than a campaign. + +```bash +qm config 140 +qm listsnapshot 140 +``` + +**`qm listsnapshot` must come back empty.** LVM-thin's export format carries no +snapshots and an online mirror cannot carry them either, so a snapshot present +here is a refusal halfway through rather than an error up front. Delete them +first if there are any. + +```bash +qm move-disk 140 scsi0 ssd +``` + +**It is genuinely online.** `qm move-disk` on a running VM drives a QEMU +`drive-mirror`: the guest keeps running throughout, with a brief pivot at the +end. `HypervisorGuestStopped` needs a full hour of `homelab_guest_running == 0` +and never sees anything. The honest cost is that copying the disk *off* a +90-IOPS mirror saturates it for the duration — the disk is 64 GiB as built +([`build-the-lab-guest.md`](build-the-lab-guest.md) §1; confirm with +`qm config 140`, it may have grown), and the copy is sequential, so likely ten +to twenty minutes — and the guest is slow while it runs. `--bwlimit` is there +if that matters. + +**`--delete` is omitted deliberately.** The source becomes `unused0`. Verify +the guest is healthy on the new storage first, then remove it: + +```bash +qm config 140 +qm set 140 --delete unused0 +``` + +Prove, then destroy — the same order as step 5's refusal to use `drives=all`. + +Then tell the guest it is on flash, using the exact volid `qm config` now +reports and preserving the flags +[`build-the-lab-guest.md`](build-the-lab-guest.md) §1 already sets: + +```bash +qm set 140 --scsi0 ssd:vm-140-disk-0,discard=on,iothread=1,ssd=1 +``` + +`ssd=1` sets the emulated rotation rate so the guest's own scheduler and TRIM +behave; it takes effect at the guest's next start. + +**Backups.** `vzdump` jobs are per-VM, not per-storage, so nothing here changes +them — confirm with `cat /etc/pve/jobs.cfg`. Existing backups stay restorable, +but a restore defaults back to the storage recorded in the archive, so pass +`--storage ssd` explicitly when the time comes. + +**Now run M3** from step 8: the HDD mirror is finally idle, and that is the +reading ADR-0029's `83` gets compared against. + +## 10. Delete the silences — immediately, not on expiry + +Both rack runbooks in this repository record getting this order inverted — +23:14 against a 22:45 self-test, 23:11 against a 23:02 proving reading — and +both cost nothing only by luck. **Delete first, then read.** A silence standing +over freshly fitted hardware suppresses exactly the thing you most want to hear +about. + +```bash +curl -sS -X DELETE http://localhost:9093/api/v2/silence/ +``` + +```bash +curl -sS http://localhost:9093/api/v2/silences | python3 -c 'import json,sys; [print(s["id"], s["status"]["state"], s["endsAt"]) for s in json.load(sys.stdin)]' +``` + +Both must read `expired` with an `endsAt` at the moment you deleted them rather +than the five-hour mark — that is the difference between deleted and lapsed, +and it is what the roadmap entry should record. + +### Which alerts were silenced, and which were deliberately left live + +| Alert | What it does during this job | Silenced? | +| --- | --- | --- | +| `IloHardwareDegraded` (critical, 5m) | **May fire.** Reads `cpqDaLogDrvCondition > 2`. RAID 1 has no parity to initialise, but `cpqDaLogDrvStatus` carries `recovering(5)`, `rebuilding(7)` and `rapidParityInit*(18/19)` and the condition column tracks them. Critical, and it pages | **Yes** — two, both scoped to the new indexes | +| `IloDrivePredictiveFailure` (warning, 15m) | Should not fire. `cpqDaPhyDrvSmartStatus` `4` is `replaceDriveSSDWearOut` — on *used* enterprise SSDs that is a **true finding**, not noise | **No.** Silencing it would suppress precisely what you have just introduced. If it fires, read `cpqDaPhyDrvSSDPercntEndrnceUsed` and `cpqDaPhyDrvSSDWearStatus` at that index | +| `IloDriveSmartUnreadable` (info, 1h) | May fire — whether a new non-HPE SATA SSD reports `SmartStatus` `1` before the controller configures it is unknown. `severity: info` routes to the `null` receiver, so it pages nobody | **No.** Let it fire; the fit is the experiment that settles it. Still firing an hour after the drive reads `ok` is a finding about SMART visibility, not an alerting problem | +| `IloWriteCacheDisabled` (warning, 1h) | Will not fire. Reads `cpqDaAccelStatus > 3`; it reads `3`, and with the ratio change deferred to #76 there is no flush to blip it to `tmpDisabled(4)` | **No** — it is the most informative rule for this job | +| `HypervisorGuestStopped` (warning, 1h) | Will not fire — the move is online and needs a full hour of `== 0` | **No** | +| `GuestStateStopped` (warning, 30m) | Will not fire — nothing here touches `homelab-guest-state.timer` | **No** | +| `SnmpScrapeSlow` | Will not fire. The walk gains two drives and one logical drive: expect roughly 13–17 s against a 30 s threshold | **No**, but put before and after in the table | +| `SmartDrive*` (`host.rules.yaml`) | Will not fire — `collect-smart-state.sh` excludes `Saruman` by design, because the iLO already walks its array | **No** | +| `HostDiskCritical` / `HostDiskWillFillIn24h` | Cannot fire for the new pool — see step 7's blind spot | **No** | + +## 11. Confirm the metrics actually moved — on `prometheus` + +Re-run step 0's queries and fill the Observed column. Every row must reach its +right-hand value. + +| Metric | Before, 2026-09-17 | After a successful fit | Observed | +| --- | --- | --- | --- | +| `cpqDaPhyDrv*` index set | `0`, `1` | `0`, `1`, `2`, `3` | | +| `cpqDaPhyDrvLocationString{2,3}` | absent | `Port 1I Box 1 Bay 3` / `Bay 4` | | +| `cpqDaPhyDrvSerialNum{2,3}` | absent | **the two label serials — the row that cannot be faked** | | +| `cpqDaPhyDrvModel{2,3}` | absent | an `MZ7KM960...` string | | +| `cpqDaPhyDrvMediaType{2,3}` | absent (`{0,1}` = `2`) | `3` solidState | | +| `cpqDaPhyDrvRotationalSpeed{2,3}` | absent (`{0,1}` = `2`) | `5` rpmSsd | | +| `cpqDaPhyDrvType{2,3}` | absent | `3` sata | | +| `cpqDaPhyDrvNegotiatedLinkRate{2,3}` | absent (`{0,1}` = `4`) | `4`. A `3` means the drive negotiated down to 3 Gb/s | | +| `cpqDaPhyDrvConfigurationStatus{2,3}` | absent | `3` notConfigured after step 4 → `2` configured after step 5 | | +| `cpqDaPhyDrvCondition{2,3}` / `Status{2,3}` | absent | `2` ok / `2` ok | | +| `cpqDaPhyDrvSmartStatus{2,3}` | absent | `2` ok. `1` trips `IloDriveSmartUnreadable` after 1 h, unrouted. **`4` is SSD wear-out and is a true finding** | | +| `cpqDaPhyDrvSSDWearStatus{2,3}` | absent (`{0,1}` = `1` other) | `2` ok, or `1` if the controller will not read a third-party SSD | | +| `cpqDaPhyDrvSSDPercntEndrnceUsed{2,3}` | absent (`{0,1}` = `4294967295`) | a percentage, or `4294967295` — unknown is this box's norm | | +| `cpqDaLogDrv*` index set | `1` | `1`, `2` | | +| `cpqDaLogDrvCondition{2}` / `Status{2}` | absent | `2` ok / `2` ok (`5`, `7`, `18`, `19` while syncing) | | +| `cpqDaLogDrvFaultTol{2}` | absent | `3` mirroring | | +| `cpqDaLogDrvSize{2}` | absent | roughly `915700` MB | | +| `cpqDaLogDrvHasAccel{2}` | absent (`{1}` = `1` other) | `3` enabled — **or `1` other, if the existing drive's reading is the iLO's habit rather than a fault.** Which one it is, is itself the #76 answer arriving from a second direction | | +| `cpqDaLogDrvSSDSmartPathStatus{2}` | absent | `4` ssdSmartPathEnabled | | +| `cpqDaLogDrvCondition{1}` | `2` ok | `2` ok — **must not move** | | +| `cpqDaAccelStatus` | `3` enabled | `3` enabled — **must not move**; no `modify cacheratio=` means no flush | | +| `cpqDaAccelBadData` | `2` none | `2` none. `3` means dirty cache was lost and is a different conversation | | +| `cpqDaAccelWriteCachePercent` | `0` | **`0`, expected unchanged** — the ratio is not set here. If `ld 2 modify aa=enable` alone moves it, that is itself a finding for #76 | | +| `cpqDaAccelMemory` / `ReadMemory` | `0` / `0` | `0` / `0`, same reasoning | | +| `cpqDaAccelTotalMemory` | `2097152` | unchanged | | +| `cpqDaAccelFailedBatteries` | `1` | unchanged — a #76 curiosity, not a target | | +| `cpqDaCntlrCondition` / `BoardCondition` | `2` / `2` | `2` / `2` | | +| `cpqDaCntlrDriveWriteCacheState` | `1` other | `1` other — **unchanged is the pass condition** | | +| `ssacli ctrl slot=0 show detail` → Cache Ratio | never read | a real ratio, or `0/0` — **this is the #76 reading, and either answer resolves it** | | +| `ssacli ctrl slot=0 ld 1 show detail` → Caching | never read | `Enabled`, or the (b) branch of step 6 | | +| `scrape_duration_seconds{device="shiva"}` | `11.8` s | roughly 13–17 s, and well under 30 | | +| `rate(node_disk_writes_completed_total{device="sda"}[1h])` | current | falls to the host's own writes | | +| the same for `sdb` | absent | present, carrying `alexander`'s writes | | +| **fio 4k QD1 randwrite, HDD** | ADR-0029's **derived** `83` | **measured** — M1 loaded, M3 idle | | +| **fio 4k QD1 randwrite, SSD** | n/a | **measured** — M2. The number that replaces ADR-0029's | | +| **fio 4k QD32 randwrite** | n/a | measured on both — the new ceiling, a different benchmark | | + +The runbook is done when every row has moved, `cpqDaAccelBadData` still reads +`2`, `cpqDaLogDrvCondition{1}` still reads `2`, and the walk is still +comfortably under 30 s. + +## If something goes wrong + +| Symptom | Cause | Fix | +| --- | --- | --- | +| `ssacli` will not install | No trixie suite from HPE's MCP SDR | Paths 2 and 3 in step 2. Record which one worked | +| `ctrl all show` prints a slot other than 0 | This chassis is not wired as assumed | Stop. Every `slot=0` in this runbook is wrong; re-derive them all | +| Drives do not seat in the bays | No HPE SmartDrive carriers | Stop at step 4. This is a purchase, and a purchase edits the roadmap's buy table | +| Drives seat but never appear to `ssacli` | Bays 3–4 not cabled on this backplane | Try bays that are known-good; if none, this layout is not available on this chassis | +| New drives appear, `create` refuses | Drives not `Unassigned`, or a leftover configuration on a used drive | `pd 1I:1:3 show detail`. Do **not** reach for `erase` without re-reading step 5's table | +| `IloHardwareDegraded` pages anyway | The silence named the wrong `cpqDaLogDrvIndex` | Read the real index, silence that, delete the wrong one. Then check whether the alert is *true* | +| `cpqDaAccelStatus` moves to `4` tmpDisabled | The controller parked the cache | Wait — the rule's 1 h `for:` exists for this. If it persists, it is real and #76 wants to know | +| `qm move-disk` refuses | Snapshots on vmid 140 | `qm listsnapshot 140`, delete them, retry | +| Move completes, guest will not boot | Storage moved, guest config did not | `qm config 140` — `scsi0` must name the `ssd` storage. `unused0` is still the intact original | +| The new pool fills | Thin overprovisioning, and nothing alerts on it | Step 7's blind spot, arriving. `pvesm status`, `lvs` | +| fio numbers are wildly high | Caching somewhere in the path | `--direct=1` on every run, and a raw LV target — never a file, never `/dev/sda` | + +## Flipping the documents + +The fit does not finish when the drives are in. These are the documents it makes +stale, in the order they should be touched: + +- **[`../hardware.md`](../hardware.md)** — the Compute table's Storage column + for `Saruman`, which reads `2× 1 TB SAS HDD, RAID 1` and is the reason the + accessories entry says *"The Compute table's Storage column changes when #418 + fits them, and not before."* Model, capacity and **both serials** go into the + accessories entry, which answers + [#148](https://github.com/Gerrrt/HomeLab/issues/148)'s question about where + serials live. +- **ADR-0029** — *"By spindle"* and *"The duty cycle is a spindle decision"* + both rest on ~90 write IOPS. Once step 8 has measured otherwise, the endpoints + running per session are a **choice** rather than a constraint. A dated + `[!NOTE]` on the ADR, status left `Accepted`, in the shape of the one already + at the top of ADR-0029 and the three on ADR-0007. **Never a silent edit** — + [ADR-0001](../adr/0001-record-architecture-decisions.md) makes ADRs immutable. +- **ADR-0007**'s constraint sentence, quoted in #266 and #414, and **ADR-0017**, + which quotes it verbatim to justify NVMe for the range. ADR-0017's argument + was a separate fault domain and not only IOPS, so it survives — but the + sentence it leans on has moved and the note should say so. +- **[`../roadmap.md`](../roadmap.md)** — the #414 paragraph that repeats the + ninety-IOPS reasoning, the #418 entry, and #76 with whichever branch of step 6 + turned out to be true. +- **`stacks/observability/prometheus/rules/network.rules.yaml`** — the iLO + section-header comment records silences by UUID and expiry. Both silences from + step 1 get a paragraph there, in the shape of the 2026-08-31 one already + present. +- **`stacks/observability/prometheus/tests/network.test.yaml`** — the + `IloDrivePredictiveFailure` and `IloDriveSmartUnreadable` fixtures are + single-drive, `cpqDaPhyDrvIndex="0"`. The rule expressions are index-agnostic + so nothing is broken, but the tests stop describing the machine the day this + runbook succeeds. +- **[`build-the-lab-guest.md`](build-the-lab-guest.md)** and + **[`build-the-lab-domain.md`](build-the-lab-domain.md)** — both tell the + reader to leave the Proxmox disk cache at the default *because #76 is open and + nobody has confirmed the controller absorbs writes*. Whichever branch step 6 + lands on changes that argument. +- **`README.md`** — the runbook count, if this file was the one that moved it. + +## What this runbook does not know + +Written down rather than asserted, in the shape of +[`replace-the-smart-storage-battery.md`](replace-the-smart-storage-battery.md)'s +*"What is still open"*. Every item here is something the fit can settle, and +settling it is most of the value of doing the fit carefully. + +1. **Whether `ssacli` can be installed on Proxmox VE 9 at all.** Debian 13 + trixie; HPE's MCP SDR is not known to publish a trixie suite; nothing in this + repository is evidence it has ever run here. Three paths in step 2, none + observed working on this box. +2. **When the drive trays arrive, and whether both do.** No longer *whether* + — `651687-001` × 2 were bought 2026-09-11 and are recorded in + `hardware.md`. But they were quoted for 2026-09-17 and were not here that + morning, and a carrier without its screws holds a drive no better than no + carrier. A hard stop, not a detail. +3. **Whether bays 3 and 4 are cabled.** Both existing drives are Bay 1 and + Bay 2; the backplane variant is recorded nowhere. +4. **Whether a newly created RAID 1 transits `cpqDaLogDrvCondition = 3`.** This + is the whole reason step 1 creates a silence rather than skipping one, and + the fit should record the answer so the next one need not guess. +5. **Whether a brand-new non-HPE SATA SSD reports `cpqDaPhyDrvSmartStatus = 1`** + before the controller configures it. The info severity makes finding out + free. +6. **Whether `cpqDaAccelWriteCachePercent = 0` is a reporting gap or a real + 0 % allocation.** Four columns lean towards real. Only step 6's `ssacli` + reading decides, and it has never been run on this machine. +7. **Whether SSD Smart Path and the array accelerator conflict** on a P440ar — + specifically whether enabling Smart Path disables caching for that logical + drive. Read both lines out of `ld 2 show detail`; do not predict. +8. **Whether `dwc` has a per-array form** on the installed `ssacli`. The + recommendation to leave drive write cache alone depends on it being + controller-wide. +9. **Whether `qm move-disk` refuses with snapshots present**, and whether + `alexander` has any. Check; do not learn it from the error message. +10. **Whether `hpsa` surfaces the new logical drive without a SCSI rescan.** +11. **Whether discard reaches the SSDs** through LVM-thin → hpsa → P440ar, or + stops at the thin pool. It affects long-term steady-state write performance + and nothing in the first day's readings will show it. +12. **Whether the existing drives are SAS or SATA.** `cpqDaPhyDrvType` reads + `3` (sata) for both, on a metric whose enumeration carries a distinct + `4: sas`, and the model string is `MM1000GBKAL` — while + [`../hardware.md`](../hardware.md) and ADR-0007 both say **SAS**. Step 3's + `pd all show detail` prints the interface type authoritatively, and settling + it is a free by-product of a visit that is happening anyway. **Do not + correct either document from an SNMP enum alone**, and note that ADR-0007 + is immutable — if it is wrong, it gets a dated note like everything else. +13. **That the new logical drive's index will be `2`.** Predicted, load-bearing + for step 1's silence, and verifiable within a minute of creating it. +14. **How long the iLO takes to reflect a configuration change.** The 60 s + scrape is not the bound; the iLO's own agentless refresh may be minutes. +15. **The exact option spellings** on the installed `ssacli` — `aa=`, + `ssdsmartpath=`, `dwc=` have all moved between versions. + `ssacli ctrl slot=0 help create` and `help modify` are the authority. + +One thing that belongs beyond this runbook rather than in it: +`collect-smart-state.sh` skips `Saruman` because the iLO covers its array — but +the iLO already reports `4294967295` for the HDDs' endurance columns and +`cpqDaPhyDrvSSDWearStatus` `1` (other), and it will very likely do the same for +two third-party SSDs. That would leave wear monitoring blind on the newest and +most wear-sensitive parts in the estate, on drives bought second-hand. +`smartctl -d cciss,N /dev/sda` reads the drives directly through the `hpsa` +path and would close it. That is an issue, not a step here.