diff --git a/README.md b/README.md index 43880117..ca0b8865 100644 --- a/README.md +++ b/README.md @@ -93,7 +93,7 @@ documents for different readers. from `compose.yaml` too, so an image that is not pinned there cannot be run at all. - **Documented decisions and runbooks.** 42 ADRs covering what was chosen - and what was rejected — including the costs accepted knowingly; 27 + and what was rejected — including the costs accepted knowingly; 28 runbooks for the operations that are easy to get wrong at 1am, one of which is the handover page a successor reads first. diff --git a/docs/hardware.md b/docs/hardware.md index d0384549..12c4a16a 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -325,10 +325,18 @@ revisions of this repository treated `shiva` as the hypervisor itself. decision on `Saruman` starts from: a 7.2K mirror serving about ninety random write IOPS ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)). - Serials go here when they land. The Compute table's Storage column changes + Serials go here when they land — read off the labels before the drives go + into the bays, which is step 4 of + [`fit-the-saruman-ssds.md`](runbooks/fit-the-saruman-ssds.md), because + reading them back afterwards means reading them through the tool the fit is + trying to verify. The Compute table's Storage column changes when [#418](https://github.com/Gerrrt/HomeLab/issues/418) fits them, and not before — that issue also names the ADRs whose arithmetic the fit makes - stale + stale. The fit itself is + [`fit-the-saruman-ssds.md`](runbooks/fit-the-saruman-ssds.md), whose step 4 + reads both serials off the labels before the drives go into the bays — + reading them back afterwards means reading them through the tool the fit is + trying to verify - 2× HP 2.5" SFF drive tray, `651687-001`[^Caddy] — bought 2026-09-11, in transit, quoted to arrive by 2026-09-17 and not here on the morning of it. The carriers the SM863a pair above needs to sit in `Saruman`'s SFF bays @@ -350,6 +358,9 @@ revisions of this repository treated `shiva` as the hypervisor itself. this machine; that both arrived; and that each has its screws, since a carrier without them holds a drive no better than no carrier. `651687-001` is the listing's part number and not yet the tray's. + **Until they land the SSDs cannot be fitted at all**, which is the sharpest + form of the point #418 is built on: arriving is not fitting, and neither the + Compute table nor ADR-0029's arithmetic moves for a drive on a shelf. - MikroTik CRS326-24G-2S+RM[^CRS326] — 24 × 1 GbE, 2 × SFP+, 1U, dual-boot RouterOS / SwOS — bought used 2026-09-13; in transit, delivery estimated 2026-09-23, moved out from the 09-16 to 09-21 window quoted at purchase. The diff --git a/docs/roadmap.md b/docs/roadmap.md index 90c8307f..30eaaab7 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -851,7 +851,9 @@ what left this one unfireable for months. the endpoints per session, because a 7.2K mirror serves about ninety random write IOPS and six idle Windows guests would be most of them — the number #418's SSDs were bought against, and the one their *fit* re-derives; a drive - in transit changes no duty cycle. Three things + on the shelf changes no duty cycle either, and since 2026-09-11 that is + where they are — waiting on the trays that let them into the bays at all. + Three things that ADR left explicit because they fail quietly: the DC takes its clock from the gateway, not `time.windows.com` — ADR-0014 named that failure and did not fix it, and the alert reads the *sync source* rather than the offset, because @@ -943,16 +945,45 @@ what left this one unfireable for months. → [runbook](runbooks/build-the-playground.md) - **[#418](https://github.com/Gerrrt/HomeLab/issues/418) Fit the two SSDs in `Saruman`.** Two Samsung SM863a 960 GB SATA enterprise drives, bought - 2026-09-09, in transit. ADR-0007's constraint — "the fleet - is sized against spindles, not RAM" — became a number in ADR-0029, about - ninety random write IOPS for the whole machine, and that number sized the - lab domain's duty cycle and #266's indexer. The SSDs raise the ceiling; what - the issue leaves to the fit is whether they replace the mirror or sit beside - it, whether the Smart Array manages them — the drive rules read `cpqida.mib` - and a layout the controller does not own falls out of all of them — and - what #76's write cache does for the new logical drive. The fit makes - ADR-0029's derivation stale on the day, and the issue says so rather than - leaving the ADR to be silently edited. + 2026-09-09, **delivered 2026-09-11, not fitted and not yet fittable**. + ADR-0007's constraint — "the + fleet is sized against spindles, not RAM" — became a number in ADR-0029, + about ninety random write IOPS for the whole machine, and that number sized + the lab domain's duty cycle and #266's indexer. The SSDs raise the ceiling. + + **Two of the three questions the issue left to the fit are now answered, and + the one that matters is not.** The layout is decided: the SSDs become a + *second* logical drive on the P440ar, RAID 1, Smart Array managed, with the + 7.2K mirror keeping Proxmox, the ISOs and the backups. That is the smaller + change, it keeps a spindle for the things that do not need IOPS, and it + leaves `IloDrivePredictiveFailure`, `IloDriveSmartUnreadable` and the + `replaceDriveSSDWearOut(4)` state reading `cpqida.mib` exactly as they do + today — so #351's decision to skip `smart-state` on this host stays right + rather than needing re-checking. #76's cache is **read and handed over, not + changed**: `modify cacheratio=` is controller-wide and would land on the + array holding every guest, so the runbook takes the `ssacli` reading nobody + has ever taken on this machine and #76 owns what to do about it. + + **What blocks it is neither of those: the trays.** A Gen9 bay holds a drive + only in a SmartDrive carrier, two were bought `651687-001` on 2026-09-11, + and on the morning of 2026-09-17 they had not arrived. Until they do, the + drives cannot enter the machine, which is why the runbook makes the carriers + a stop condition at step 4 rather than something discovered at the rack. The + same step confirms both drives are physically present — the delivery notice + covers one order, not two units, which the paragraph above leaves to this + issue to prove at the bays. + + What is still open after that is the only thing the purchase was for. + **ADR-0029's + ninety is derived, not measured** — seek plus half a rotation at 7200 rpm — + so the runbook measures both arrays at the parameters that derivation + implies, 4 KiB at queue depth 1, and three times: the HDD mirror loaded, the + SSD array idle, and the HDD mirror idle once the guest has moved off it. + Until that reading exists, ADR-0029, ADR-0007, ADR-0017 and the #414 + paragraph above all stand as written. The fit makes them stale on the day, + and it gets a dated note on each rather than a silent edit — ADR-0001 makes + them immutable. + → [runbook](runbooks/fit-the-saruman-ssds.md) ## Automation diff --git a/docs/runbooks/fit-the-saruman-ssds.md b/docs/runbooks/fit-the-saruman-ssds.md new file mode 100644 index 00000000..cec69617 --- /dev/null +++ b/docs/runbooks/fit-the-saruman-ssds.md @@ -0,0 +1,781 @@ +# Runbook: Fit the two SSDs in `Saruman` + +**Two drives, no maintenance window, and a number that has to be measured on +both arrays — not assumed from either.** + +> **Status — 2026-09-17: the drives are here, the trays are not. Nothing is +> fitted, and nothing can be.** +> +> Two Samsung SM863a 960 GB (`MZ-7KM960N`), bought 2026-09-09 and delivered +> 2026-09-11. The baseline in step 0 below was read from Prometheus on +> 2026-09-17 and is the "Before" column of every table here. +> +> **This runbook is blocked at step 4 until the drive trays land.** A Gen9 bay +> holds a drive only in an HPE SmartDrive carrier. Two were bought +> (`651687-001`, 2026-09-11) and had not arrived by the morning of 2026-09-17 +> — see [`../hardware.md`](../hardware.md). Steps 0 through 3 can all be done +> before they turn up, and doing them early is worth it: step 2 in particular +> is the one most likely to fail, and finding that out with the drives already +> in your hand is the wrong order. +> +> **The layout is decided and the measurement is not.** The SSDs become a +> *second* logical drive on the P440ar, RAID 1, Smart Array managed; the two +> 7.2K disks keep Proxmox, the ISOs and the backups. That answers the first two +> questions [#418](https://github.com/Gerrrt/HomeLab/issues/418) left to the +> fit. What it does not answer is the one the purchase exists for: the write +> IOPS this machine actually has. Step 8 measures it, on both arrays, with the +> parameters ADR-0029 derived its number from — and until that reading exists, +> no ADR here changes. + +`shiva` is the iLO, not the hypervisor. The host behind it is `Saruman` at +`10.0.30.110`; see [`../hardware.md`](../hardware.md). Both are on VLAN 30. + +## Where each step runs, and why it matters + +VLAN 30 is not reachable from the monitoring host. That is not an inconvenience +to work around in this runbook — it is the reason the runbook is split the way +it is. + +| Steps | Host | Who runs them | +| --- | --- | --- | +| 0, 1, 10, 11 | `prometheus`, `10.0.99.20`, VLAN 99 | the monitoring host — Prometheus and Alertmanager are here | +| 2–9 | `Saruman`, `10.0.30.110`, VLAN 30 | **you, from the Mac.** Nothing on VLAN 99 can reach this host | + +Run the `Saruman` half inside `tmux`. A dropped SSH session during step 9's +disk move leaves a half-mirrored disk, and that is a worse afternoon than a +reconnect. + +## Why this is not a maintenance window + +The SFF bays are hot-plug and `qm move-disk` on a running guest is an online +operation, so **no guest stops and no alert needs to be scheduled around**. +[`schedule-maintenance.md`](schedule-maintenance.md) is not the shape of this +job. Two alerts still get silenced in step 1, not because anything is going +down but because a freshly created logical drive may legitimately report +`degraded` while the controller syncs it, and `IloHardwareDegraded` is +`severity: critical` with a ten-second `group_wait` on the `urgent` receiver. + +## What this deliberately does not do + +- **It does not change the controller-wide cache ratio.** + `ssacli ctrl slot=0 modify cacheratio=` flushes and re-partitions the whole + 2 GiB module, and that pause lands on the logical drive holding Proxmox and + every guest. Fitting two drives into empty bays is not the occasion to + re-partition the cache the rest of the machine runs on. Step 6 *reads* the + ratio — which is the reading + [#76](https://github.com/Gerrrt/HomeLab/issues/76) has been waiting for since + 2026-09-02 — and hands the finding over. #76 owns the change. +- **It does not edit ADR-0029 or ADR-0007.** They are stale the day the + measurement lands, not the day the drives land. See "Flipping the documents". +- **It does not touch the existing mirror.** No `delete`, no `modify raid=`, + no `erase`, and nothing is pulled from bay 1 or bay 2. + +--- + +## 0. Record the baseline — on `prometheus`, before anything + +This is the step that makes the fit provable rather than assumed, and it is the +same argument [`replace-the-smart-storage-battery.md`](replace-the-smart-storage-battery.md) +makes: every other row in the after-table could in principle be the old +hardware reporting differently. + +```bash +curl -sS -G http://127.0.0.1:9090/api/v1/query --data-urlencode 'query={__name__=~"cpqDa(Accel|Cntlr|LogDrv|PhyDrv).*",device="shiva"}' | python3 -m json.tool +``` + +Two more, because they are the only readings that come from the *host* rather +than the iLO, and they are how you prove in step 11 that the guest's writes +actually moved: + +```bash +curl -sS -G http://127.0.0.1:9090/api/v1/query --data-urlencode 'query=scrape_duration_seconds{device="shiva"}' +curl -sS -G http://127.0.0.1:9090/api/v1/query --data-urlencode 'query=rate(node_disk_writes_completed_total{instance="Saruman"}[30m])' +``` + +**What that read on 2026-09-17**, and what each value is doing in this runbook: + +| Reading | Value | Why it is here | +| --- | --- | --- | +| `cpqDaCntlrHwLocation` | `Slot 0` | every `slot=0` below depends on it | +| `cpqDaCntlrOperatingMode` | `2` smartArrayMode | not HBA mode — the chosen layout is the one the controller is already in | +| `cpqDaCntlrModel` / `FWRev` | `75` / `7.00` | the controller this runbook was written against | +| `cpqDaLogDrv*` index set | **`1` only** | the new one should become `2`; step 5 verifies rather than assumes | +| `cpqDaLogDrvFaultTol{1}` | `3` mirroring | the existing RAID 1 | +| `cpqDaLogDrvSize{1}` | `953837` MB | | +| `cpqDaLogDrvHasAccel{1}` | **`1` other** | not `3` enabled — see step 6 | +| `cpqDaPhyDrv*` index set | **`0` and `1`** | the new drives should become `2` and `3` | +| `cpqDaPhyDrvLocationString{0,1}` | `Port 1I Box 1 Bay 1` / `Bay 2` | bays 3 and 4 are the target | +| `cpqDaPhyDrvModel{0,1}` | `MM1000GBKAL` | | +| `cpqDaPhyDrvSerialNum{0,1}` | `9XG62LSC` / `9XG72NXP` | the rows that cannot be faked | +| `cpqDaPhyDrvMediaType{0,1}` | `2` rotatingPlatters | the SSDs must read `3` solidState | +| `cpqDaPhyDrvRotationalSpeed{0,1}` | `2` rpm7200 | the SSDs must read `5` rpmSsd | +| `cpqDaPhyDrvType{0,1}` | **`3` sata** | see "What this runbook does not know", item 12 | +| `cpqDaPhyDrvSmartStatus{0,1}` | `2` ok | | +| `cpqDaPhyDrvSmartCarrierAppFWRev{0,1}` | `11` | the existing drives are in HPE SmartDrive carriers, which have their own firmware — step 4's carrier precondition is not hypothetical | +| `cpqDaAccelStatus` | `3` enabled | must not move | +| `cpqDaAccelTotalMemory` | `2097152` KB | 2 GiB fitted | +| `cpqDaAccelMemory` (write) | **`0`** | 0 KB allocated, against 2 GiB fitted | +| `cpqDaAccelReadMemory` | **`0`** | | +| `cpqDaAccelWriteCachePercent` | **`0`** | #76 | +| `cpqDaAccelReadCachePercent` | **`0`** | | +| `cpqDaAccelBadData` | `2` none | must not move; `3` is a different conversation | +| `cpqDaAccelBattery` / `BackupPowerSource` | `2` ok / `4` smartbattery | the pack fitted 2026-09-02 | +| `cpqDaAccelFailedBatteries` | `1` | a known curiosity from #76, not a target here | +| `cpqDaCntlrDriveWriteCacheState` | `1` other | step 6 leaves this alone deliberately | +| `scrape_duration_seconds{device="shiva"}` | `11.8` s | against `SnmpScrapeSlow`'s 30 s | +| guests | one — `alexander`, vmid `140`, qemu | step 9 is one command, not a campaign | + +**Four columns, not one.** #76 has been open on `cpqDaAccelWriteCachePercent` +reading `0`. The baseline above shows `cpqDaAccelMemory`, +`cpqDaAccelReadMemory`, `cpqDaAccelWriteCachePercent` and +`cpqDaAccelReadCachePercent` *all* reading `0` while +`cpqDaAccelTotalMemory` reports 2 GiB fitted, and `cpqDaLogDrvHasAccel{1}` +reading `other` rather than `enabled`. Four independent columns agreeing that +nothing is allocated is much weaker support for "the iLO simply does not +populate the ratio" than one column was. Step 6 is where that gets settled. + +## 1. Silence the two alerts a new array legitimately trips — on `prometheus` + +The repo documents *deleting* a silence +([`fit-the-ups-battery.md`](fit-the-ups-battery.md) §3) and has never written +down creating one. This is that call. + +```bash +START=$(date -u +%Y-%m-%dT%H:%M:%SZ) +END=$(date -u -d '+5 hours' +%Y-%m-%dT%H:%M:%SZ) +``` + +```bash +curl -sS -X POST http://localhost:9093/api/v2/silences -H 'Content-Type: application/json' --data "$(printf '{"matchers":[{"name":"alertname","value":"IloHardwareDegraded","isRegex":false,"isEqual":true},{"name":"device","value":"shiva","isRegex":false,"isEqual":true},{"name":"cpqDaLogDrvIndex","value":"2","isRegex":false,"isEqual":true}],"startsAt":"%s","endsAt":"%s","createdBy":"#418 fit the SSDs in Saruman","comment":"New RAID 1 logical drive on the P440ar may report degraded while the controller syncs it. Scoped to the NEW logical drive index so temperature, supplies, the controller and LD 1 stay live. Delete BEFORE the proving reading."}' "$START" "$END")" | python3 -m json.tool +``` + +```bash +curl -sS -X POST http://localhost:9093/api/v2/silences -H 'Content-Type: application/json' --data "$(printf '{"matchers":[{"name":"alertname","value":"IloHardwareDegraded","isRegex":false,"isEqual":true},{"name":"device","value":"shiva","isRegex":false,"isEqual":true},{"name":"cpqDaPhyDrvIndex","value":"2|3","isRegex":true,"isEqual":true}],"startsAt":"%s","endsAt":"%s","createdBy":"#418 fit the SSDs in Saruman","comment":"Two newly inserted physical drives on the P440ar. Scoped to the NEW drive indexes so drives 0 and 1 stay live. Delete BEFORE the proving reading."}' "$START" "$END")" | python3 -m json.tool +``` + +Record both UUIDs — they go in the roadmap and in the rules-file comment +afterwards: + +```bash +curl -sS http://localhost:9093/api/v2/silences | python3 -c 'import json,sys; [print(s["id"], s["status"]["state"], s["endsAt"], [(m["name"],m["value"]) for m in s["matchers"]]) for s in json.load(sys.stdin)]' +``` + +**Three properties, each of which this repository already argues for somewhere.** + +**Narrower than the precedent.** The battery silence on 2026-08-31 could only +manage `alertname` + `device`, and for five weeks a power-supply fault on the +same box would have been suppressed with the battery. +`IloHardwareDegraded`'s expression is an `or` of six series, so the firing +alert inherits the labels of whichever series matched — which means the +matcher can name `cpqDaLogDrvIndex` or `cpqDaPhyDrvIndex` and leave +temperature, supplies, the controller and the *existing* mirror completely +live. Use it. + +**Five hours, not a month.** The battery silence was written to +2026-10-01 and deleted on 2026-09-02. A silence should expire embarrassingly +soon rather than outlive the job. + +**`cpqDaLogDrvIndex` `2` is a prediction.** It is the most likely index and it +is not read yet. Step 5 verifies it within a minute of creating the drive; if +it comes back different, create the correct silence and delete this one +immediately. A silence keyed on the wrong index is a silence that does nothing +while you believe it is doing something. + +**Three alerts are deliberately *not* silenced.** See the verdict table before +step 10 for why each one is better left live. + +## 2. Get `ssacli` onto the hypervisor — the step most likely to stop you + +> **This is the largest unknown in this runbook and it is written as one.** +> `ssacli` is not in Debian and not in the Proxmox repositories. It comes from +> HPE's MCP SDR, which is not known to publish a Debian 13 `trixie` suite, and +> **nothing in this repository is evidence that it has ever been installed on +> this host.** Proxmox VE 9 is Debian 13. None of the three paths below has +> been watched working on this box. + +Path 1 — pin the `bullseye` suite and accept an out-of-suite package: + +```bash +curl -fsSL https://downloads.linux.hpe.com/SDR/hpPublicKey2048_key1.pub | gpg --dearmor > /usr/share/keyrings/hpe.gpg +echo "deb [signed-by=/usr/share/keyrings/hpe.gpg] https://downloads.linux.hpe.com/SDR/repo/mcp bullseye/current non-free" > /etc/apt/sources.list.d/hpe-mcp.list +apt-get update && apt-get install -y ssacli +``` + +`ssacli` is a largely self-contained vendor binary; it may or may not satisfy +trixie's `libc6` and `libstdc++6`. + +Path 2 — take the `.deb` directly from +`https://downloads.linux.hpe.com/SDR/repo/mcp/pool/non-free/` and `dpkg -i`, +resolving whatever it complains about. + +Path 3 — **no host package at all.** Reboot into Intelligent Provisioning and +use the offline Smart Storage Administrator through the iLO remote console. +This works with certainty and costs a reboot. The iLO 4 web UI's Storage page +is read-only and **cannot** create a logical drive, so it is not a fourth +option. + +A reboot changes the alert picture a little and not much: `HostRebooted` is +`severity: info` and routes to the `null` receiver; `RemoteWriteJobStale` needs +roughly twenty minutes of absence before it fires. A reboot shorter than that +is quiet. If path 3 is taken, steps 7 and 9 still need a shell, so you are +installing nothing and rebooting twice — prefer paths 1 and 2, and record +which one worked. + +## 3. Read the controller before touching a bay + +```bash +ssacli ctrl all show status +ssacli ctrl all show config +ssacli ctrl slot=0 show detail +ssacli ctrl slot=0 pd all show detail +ssacli ctrl slot=0 ld all show detail +``` + +`ctrl all show` must print `Smart Array P440ar in Slot 0 (Embedded)`. If it +prints a different slot, every `slot=0` below is wrong. That is the one thing +to check rather than copy. + +Capture verbatim and keep it — several of these lines are readings nothing in +this estate has ever taken: + +- **Cache Ratio**, **Total Cache Size**, **Total Cache Memory Available** +- **No-Battery Write Cache**, **Drive Write Cache** +- **Battery/Capacitor Status** and **Count** +- from `ld 1 show detail`: the **Caching** line +- from `pd all show detail`: the **Interface Type** of drives 1I:1:1 and + 1I:1:2 — see item 12 at the end + +## 4. Fit the drives + +The SFF bays are genuinely hot-plug, and inserting drives into **empty** bays +while logical drive 1 serves the running OS is a supported operation. Two +preconditions, stated here rather than discovered at the rack: + +- **HPE Gen8/Gen9 SFF SmartDrive carriers — bought, and not here yet.** Two + HP 2.5" SFF drive trays, `651687-001`, bought 2026-09-11 and still in + transit on 2026-09-17; [`../hardware.md`](../hardware.md) carries them. A + bare 2.5" drive does not seat in a ProLiant bay, so **this is where the job + stops until they land**. The baseline in step 0 shows + `cpqDaPhyDrvSmartCarrierAppFWRev` reading `11` on the existing drives, so + the carriers this machine uses are the firmware-carrying kind and not + improvised — a Gen10 part or a 3.5" LFF tray will not do. + + Check on arrival, per `hardware.md`: that both trays came, that each has its + screws, and that they are the Gen8/Gen9 SmartDrive carrier. One tray short + is one SSD fitted and one on a shelf, which is a worse outcome than waiting. + +- **Both drives, physically.** The delivery notice covers one order for the + pair, not two units confirmed. `docs/roadmap.md` leaves proving that to this + runbook, and the bays are where it gets proved: two drives in, two new + `cpqDaPhyDrv` indexes out. +- **Bays 3 and 4 must be cabled.** Both existing drives read `Box 1 Bay 1` and + `Bay 2`. The backplane variant on this chassis is recorded nowhere in this + repository. + +**Read both serial numbers off the drive labels before they go in.** They are +the one row in step 11's table that cannot be produced by the old hardware +reporting differently, and they are what +[`../hardware.md`](../hardware.md)'s *"Serials go here when they land"* is +waiting for. Writing them down after the drives are in a chassis means reading +them back through the tool you are trying to verify. + +Insert into bays 3 and 4, then: + +```bash +ssacli ctrl slot=0 pd all show +ssacli ctrl slot=0 pd 1I:1:3 show detail +ssacli ctrl slot=0 pd 1I:1:4 show detail +``` + +Both must appear as `1I:1:3` and `1I:1:4`, `Status: OK`, `Drive Type: +Unassigned Drive`. Confirm the model string and that **the serials match the +labels** before creating anything. + +Give the iLO time before reading Prometheus. The scrape is every 60 s, but the +iLO's own agentless refresh is not, and a drive absent from Prometheus five +minutes after insertion is not necessarily a fault. + +## 5. Create the logical drive + +```bash +ssacli ctrl slot=0 create type=ld drives=1I:1:3,1I:1:4 raid=1 +``` + +**Type the bay list. Never `drives=all`, never `drives=allunassigned`.** The two +HDDs are already assigned and would not be selected by either keyword — and the +cost of being wrong about that is the array holding every guest. A keyword one +letter from a different meaning is not worth the saving. + +```bash +ssacli ctrl slot=0 ld all show +ssacli ctrl slot=0 ld 2 show detail +``` + +**Verify the new index now**, against the silence created in step 1: + +```bash +# from prometheus +curl -sS -G http://127.0.0.1:9090/api/v1/query --data-urlencode 'query=cpqDaLogDrvCondition{device="shiva"}' +``` + +### Safe while logical drive 1 is in use, and not + +| Safe | Not safe, ever, in this runbook | +| --- | --- | +| Pulling a bay blank; inserting a drive into an empty bay | Pulling bay 1 or bay 2 | +| Any `ssacli ... show` | `ssacli ctrl slot=0 delete ...`, `modify raid=`, any `erase` | +| `create type=ld` from an explicit bay list | `create` with `drives=all` or `drives=allunassigned` | +| `ld 2 modify aa=enable`, `array B modify ssdsmartpath=` | `ctrl slot=0 modify cacheratio=` — see step 6 | +| `pvcreate` / `pvesh ... lvmthin` on the new device | Controller firmware update | +| `fio` against a purpose-made LV | `fio --filename=/dev/sda` or `/dev/sdb` | +| `qm move-disk` on the running guest | Anything at all while a `move-disk` is mirroring | + +## 6. The cache: read it, hand it to #76 + +Two cache commands are in scope here, and both are scoped to the new array. +Neither touches logical drive 1: + +```bash +ssacli ctrl slot=0 ld 2 modify aa=enable +ssacli ctrl slot=0 array B modify ssdsmartpath=enable +ssacli ctrl slot=0 ld 2 show detail +``` + +Check the spellings against the installed version before trusting them — +`ssacli ctrl slot=0 help create` and `help modify` are the authority, and these +options have moved between versions. Read **both** the `Caching` and the +`SSD Smart Path` lines out of `ld 2 show detail` afterwards rather than +predicting how they interact. + +Then take the reading that settles #76: + +```bash +ssacli ctrl slot=0 show detail | grep -iE "cache ratio|cache size|cache memory|drive write cache|no-battery" +ssacli ctrl slot=0 ld 1 show detail | grep -i caching +``` + +**State which answer you expect before you look**, so the reading can +contradict you: + +- **(a) `ssacli` reports a real ratio** — say `10% Read / 90% Write` — while + SNMP reports `0`/`0`. Then the iLO's agentless view does not populate those + columns on iLO 4 2.82, **#76 resolves as "this column is not readable on this + hardware"**, no rule may ever be written on it, and `IloWriteCacheDisabled` + reading `cpqDaAccelStatus` instead was right all along. +- **(b) `ssacli` agrees — `0% / 0%`, or `ld 1 ... Caching: Disabled`.** Then the + array has been running write-through since the pack went in *despite + reporting the accelerator enabled*, ADR-0029's *"That figure assumes no write + cache, which is the honest assumption here"* was literally true rather than + conservative, and the fix is `modify cacheratio=`. + +**The four columns in step 0 lean towards (b).** Either answer resolves #76. + +**The fix, if it is (b), is not taken here.** `modify cacheratio=` is +controller-wide: it flushes and re-partitions the 2 GiB module, and the pause +lands on logical drive 1 — Proxmox and `alexander`. It would also move two +variables at once underneath step 8's measurement, which is the one thing this +runbook exists to get right. Hand #76 the reading; #76 takes the change. + +For when it does: the ratio that suits an SSD logical drive is also the ratio +that suits the HDD mirror, so the mixed-media controller forces no compromise. +Read-ahead buys an SSD essentially nothing — which is why HPE built SSD Smart +Path to bypass the cache entirely for reads on RAID 0/1/1+0 SSD arrays — and +posted writes buy it little, because the drive services a 4 KiB write in well +under 100 µs and the controller's cache path is not an order of magnitude +faster than that. On the HDD mirror, write cache is the single largest lever +available on this machine. Low read, high write: `10/90`. + +### Drive-level cache: leave it alone + +```bash +ssacli ctrl slot=0 modify dwc=enable # do NOT run this +``` + +`cpqDaCntlrDriveWriteCacheState` reads `1` (other) today. The SM863a's own DRAM +buffer is capacitor-backed — that is the property the part was bought for, and +it means enabling the drives' caches would be safe *for these two drives*. But +`dwc` is **controller-wide** on this generation, so it would also enable the +caches on the two 7.2K spinners, whose buffers have no capacitor and whose only +protection is `mjolnir` — a UPS whose runtime reading has sat on the fabricated +`63` for the whole retained window +([`fit-the-ups-battery.md`](fit-the-ups-battery.md)). The SSDs lose almost +nothing; the HDDs would lose a guarantee resting on a pack whose runtime is not +yet proven. + +If `ssacli ctrl slot=0 help modify` shows a per-array form on the installed +version, that changes the answer and this runbook should take it. Check; do not +assume either way. + +## 7. Give Proxmox the new logical drive + +```bash +lsblk -o NAME,SIZE,MODEL,SERIAL +ls -l /dev/disk/by-id/ +``` + +If the new device does not appear, rescan rather than reboot: + +```bash +echo "- - -" > /sys/class/scsi_host/host0/scan +lsblk -o NAME,SIZE,MODEL,SERIAL +``` + +Then create the thin pool the way the GUI does, substituting the device that +actually appeared: + +```bash +pvesh create /nodes/Saruman/disks/lvmthin --name ssd --device /dev/sdb --add_storage 1 +pvesm status +``` + +**Why LVM-thin, as a decision rather than a default.** Plain LVM is disqualified +outright: PVE cannot snapshot it, and ADR-0027 leaves the lab with *revert and +not backup*, so snapshots are the only rollback this estate has for +`alexander`. A directory with qcow2 would give snapshots too, but stacks ext4 +plus qcow2 metadata on top of a logical drive that is already an abstraction, +and adds write amplification on flash for a capability this host does not use. +LVM-thin gives snapshots, thin provisioning and discard pass-through, and is +what [`build-the-lab-guest.md`](build-the-lab-guest.md) already assumes. + +> **One new blind spot, recorded rather than fixed.** A thin pool is not a +> filesystem, so `node_filesystem_*` never sees it — `Saruman` reports only +> `/`, `/boot/efi` and `/etc/pve`. `HostDiskCritical` and +> `HostDiskWillFillIn24h` are therefore **structurally blind** to the new pool, +> and a thin pool that fills makes every guest on it read-only. Either do not +> overprovision, or give the pool a textfile collector beside the existing +> `collect-guest-state.sh`. This is the same shape of gap as +> [#351](https://github.com/Gerrrt/HomeLab/issues/351) and deserves its own +> issue, not a step here. + +## 8. Measure what the array actually does — both of them + +This is the step the purchase exists for. **ADR-0029's number is derived, not +measured**, and replacing a derived number with a differently-shaped measured +one would be worse than leaving it alone. + +**What ADR-0029's figure implies.** *"A 7.2K SAS drive is roughly 83 random +IOPS at queue depth 1"* is a mechanical derivation — average seek plus half a +rotation at 7200 rpm is about 12 ms, which is about 83 operations a second. It +therefore carries **queue depth 1, one stream, no concurrency**, and no block +size at all, because seek-plus-rotate does not depend on block size at small +sizes. The comparable reconstruction is **4 KiB, `iodepth=1`, `numjobs=1`, +`direct=1`, random write**, and the comparison is only honest if those exact +parameters are used on both logical drives. A queue-depth-32 run is a +*different benchmark* — worth having as the new ceiling, not as the +re-derivation. + +```bash +apt-get install -y fio +vgs # confirm free extents in both VGs first +lvcreate -n fiotest -L 8G ssd +lvcreate -n fiotest -L 8G pve +``` + +Precondition both targets identically, so the SSD is measured in steady state +rather than on fresh flash: + +```bash +fio --name=fill --filename=/dev/ssd/fiotest --rw=write --bs=1M --iodepth=8 --ioengine=libaio --direct=1 --size=8G +fio --name=fill --filename=/dev/pve/fiotest --rw=write --bs=1M --iodepth=8 --ioengine=libaio --direct=1 --size=8G +``` + +The ADR-0029 comparison, run identically on both: + +```bash +fio --name=adr0029 --filename=/dev/pve/fiotest --rw=randwrite --bs=4k --iodepth=1 --numjobs=1 --ioengine=libaio --direct=1 --size=8G --time_based --runtime=60 --ramp_time=10 --randrepeat=0 --norandommap --group_reporting +``` + +The new ceiling, on the SSD array only: + +```bash +fio --name=ceiling --filename=/dev/ssd/fiotest --rw=randwrite --bs=4k --iodepth=32 --numjobs=4 --ioengine=libaio --direct=1 --size=8G --time_based --runtime=60 --ramp_time=10 --group_reporting +``` + +Record IOPS, `clat` mean and `clat` p99 from every run. + +**Three runs, and the ordering is the design:** + +| | Target | State | What it is for | +| --- | --- | --- | --- | +| **M1** | HDD `pve` | guest running | what the array delivers *today, under its real load*. Deliberately a loaded number | +| **M2** | SSD `ssd` | idle, before any guest data | the number that replaces ADR-0029's | +| **M3** | HDD `pve` | idle, **after step 9's move** | the clean re-derivation of the 83. Only obtainable once the array is quiet | + +M3 is why the measurement is not finished when step 8 ends. Until `alexander` +has moved off, this array is never idle, and a loaded HDD number compared +against an idle SSD number would flatter the SSDs by an unknown amount. Run the +queue-depth-32 variant against the HDD here too, and nowhere else. + +**The risk, plainly.** A queue-depth-1 4 KiB random write stream consumes +essentially the whole of the machine's ~90 IOPS budget for its sixty seconds, +so M1 **will** visibly stall `alexander`. Do it deliberately and briefly. A +queue-depth-32 run against the HDD mirror with a guest live would queue 32 deep +on a 90-IOPS array and hand the guest multi-second IO latency — which is why +that variant is confined to M3. + +Clean up. This is part of the step, not a footnote: + +```bash +lvremove /dev/ssd/fiotest +lvremove /dev/pve/fiotest +``` + +## 9. Move the guest + +There is exactly one guest, so this is one command rather than a campaign. + +```bash +qm config 140 +qm listsnapshot 140 +``` + +**`qm listsnapshot` must come back empty.** LVM-thin's export format carries no +snapshots and an online mirror cannot carry them either, so a snapshot present +here is a refusal halfway through rather than an error up front. Delete them +first if there are any. + +```bash +qm move-disk 140 scsi0 ssd +``` + +**It is genuinely online.** `qm move-disk` on a running VM drives a QEMU +`drive-mirror`: the guest keeps running throughout, with a brief pivot at the +end. `HypervisorGuestStopped` needs a full hour of `homelab_guest_running == 0` +and never sees anything. The honest cost is that copying the disk *off* a +90-IOPS mirror saturates it for the duration — the disk is 64 GiB as built +([`build-the-lab-guest.md`](build-the-lab-guest.md) §1; confirm with +`qm config 140`, it may have grown), and the copy is sequential, so likely ten +to twenty minutes — and the guest is slow while it runs. `--bwlimit` is there +if that matters. + +**`--delete` is omitted deliberately.** The source becomes `unused0`. Verify +the guest is healthy on the new storage first, then remove it: + +```bash +qm config 140 +qm set 140 --delete unused0 +``` + +Prove, then destroy — the same order as step 5's refusal to use `drives=all`. + +Then tell the guest it is on flash, using the exact volid `qm config` now +reports and preserving the flags +[`build-the-lab-guest.md`](build-the-lab-guest.md) §1 already sets: + +```bash +qm set 140 --scsi0 ssd:vm-140-disk-0,discard=on,iothread=1,ssd=1 +``` + +`ssd=1` sets the emulated rotation rate so the guest's own scheduler and TRIM +behave; it takes effect at the guest's next start. + +**Backups.** `vzdump` jobs are per-VM, not per-storage, so nothing here changes +them — confirm with `cat /etc/pve/jobs.cfg`. Existing backups stay restorable, +but a restore defaults back to the storage recorded in the archive, so pass +`--storage ssd` explicitly when the time comes. + +**Now run M3** from step 8: the HDD mirror is finally idle, and that is the +reading ADR-0029's `83` gets compared against. + +## 10. Delete the silences — immediately, not on expiry + +Both rack runbooks in this repository record getting this order inverted — +23:14 against a 22:45 self-test, 23:11 against a 23:02 proving reading — and +both cost nothing only by luck. **Delete first, then read.** A silence standing +over freshly fitted hardware suppresses exactly the thing you most want to hear +about. + +```bash +curl -sS -X DELETE http://localhost:9093/api/v2/silence/ +``` + +```bash +curl -sS http://localhost:9093/api/v2/silences | python3 -c 'import json,sys; [print(s["id"], s["status"]["state"], s["endsAt"]) for s in json.load(sys.stdin)]' +``` + +Both must read `expired` with an `endsAt` at the moment you deleted them rather +than the five-hour mark — that is the difference between deleted and lapsed, +and it is what the roadmap entry should record. + +### Which alerts were silenced, and which were deliberately left live + +| Alert | What it does during this job | Silenced? | +| --- | --- | --- | +| `IloHardwareDegraded` (critical, 5m) | **May fire.** Reads `cpqDaLogDrvCondition > 2`. RAID 1 has no parity to initialise, but `cpqDaLogDrvStatus` carries `recovering(5)`, `rebuilding(7)` and `rapidParityInit*(18/19)` and the condition column tracks them. Critical, and it pages | **Yes** — two, both scoped to the new indexes | +| `IloDrivePredictiveFailure` (warning, 15m) | Should not fire. `cpqDaPhyDrvSmartStatus` `4` is `replaceDriveSSDWearOut` — on *used* enterprise SSDs that is a **true finding**, not noise | **No.** Silencing it would suppress precisely what you have just introduced. If it fires, read `cpqDaPhyDrvSSDPercntEndrnceUsed` and `cpqDaPhyDrvSSDWearStatus` at that index | +| `IloDriveSmartUnreadable` (info, 1h) | May fire — whether a new non-HPE SATA SSD reports `SmartStatus` `1` before the controller configures it is unknown. `severity: info` routes to the `null` receiver, so it pages nobody | **No.** Let it fire; the fit is the experiment that settles it. Still firing an hour after the drive reads `ok` is a finding about SMART visibility, not an alerting problem | +| `IloWriteCacheDisabled` (warning, 1h) | Will not fire. Reads `cpqDaAccelStatus > 3`; it reads `3`, and with the ratio change deferred to #76 there is no flush to blip it to `tmpDisabled(4)` | **No** — it is the most informative rule for this job | +| `HypervisorGuestStopped` (warning, 1h) | Will not fire — the move is online and needs a full hour of `== 0` | **No** | +| `GuestStateStopped` (warning, 30m) | Will not fire — nothing here touches `homelab-guest-state.timer` | **No** | +| `SnmpScrapeSlow` | Will not fire. The walk gains two drives and one logical drive: expect roughly 13–17 s against a 30 s threshold | **No**, but put before and after in the table | +| `SmartDrive*` (`host.rules.yaml`) | Will not fire — `collect-smart-state.sh` excludes `Saruman` by design, because the iLO already walks its array | **No** | +| `HostDiskCritical` / `HostDiskWillFillIn24h` | Cannot fire for the new pool — see step 7's blind spot | **No** | + +## 11. Confirm the metrics actually moved — on `prometheus` + +Re-run step 0's queries and fill the Observed column. Every row must reach its +right-hand value. + +| Metric | Before, 2026-09-17 | After a successful fit | Observed | +| --- | --- | --- | --- | +| `cpqDaPhyDrv*` index set | `0`, `1` | `0`, `1`, `2`, `3` | | +| `cpqDaPhyDrvLocationString{2,3}` | absent | `Port 1I Box 1 Bay 3` / `Bay 4` | | +| `cpqDaPhyDrvSerialNum{2,3}` | absent | **the two label serials — the row that cannot be faked** | | +| `cpqDaPhyDrvModel{2,3}` | absent | an `MZ7KM960...` string | | +| `cpqDaPhyDrvMediaType{2,3}` | absent (`{0,1}` = `2`) | `3` solidState | | +| `cpqDaPhyDrvRotationalSpeed{2,3}` | absent (`{0,1}` = `2`) | `5` rpmSsd | | +| `cpqDaPhyDrvType{2,3}` | absent | `3` sata | | +| `cpqDaPhyDrvNegotiatedLinkRate{2,3}` | absent (`{0,1}` = `4`) | `4`. A `3` means the drive negotiated down to 3 Gb/s | | +| `cpqDaPhyDrvConfigurationStatus{2,3}` | absent | `3` notConfigured after step 4 → `2` configured after step 5 | | +| `cpqDaPhyDrvCondition{2,3}` / `Status{2,3}` | absent | `2` ok / `2` ok | | +| `cpqDaPhyDrvSmartStatus{2,3}` | absent | `2` ok. `1` trips `IloDriveSmartUnreadable` after 1 h, unrouted. **`4` is SSD wear-out and is a true finding** | | +| `cpqDaPhyDrvSSDWearStatus{2,3}` | absent (`{0,1}` = `1` other) | `2` ok, or `1` if the controller will not read a third-party SSD | | +| `cpqDaPhyDrvSSDPercntEndrnceUsed{2,3}` | absent (`{0,1}` = `4294967295`) | a percentage, or `4294967295` — unknown is this box's norm | | +| `cpqDaLogDrv*` index set | `1` | `1`, `2` | | +| `cpqDaLogDrvCondition{2}` / `Status{2}` | absent | `2` ok / `2` ok (`5`, `7`, `18`, `19` while syncing) | | +| `cpqDaLogDrvFaultTol{2}` | absent | `3` mirroring | | +| `cpqDaLogDrvSize{2}` | absent | roughly `915700` MB | | +| `cpqDaLogDrvHasAccel{2}` | absent (`{1}` = `1` other) | `3` enabled — **or `1` other, if the existing drive's reading is the iLO's habit rather than a fault.** Which one it is, is itself the #76 answer arriving from a second direction | | +| `cpqDaLogDrvSSDSmartPathStatus{2}` | absent | `4` ssdSmartPathEnabled | | +| `cpqDaLogDrvCondition{1}` | `2` ok | `2` ok — **must not move** | | +| `cpqDaAccelStatus` | `3` enabled | `3` enabled — **must not move**; no `modify cacheratio=` means no flush | | +| `cpqDaAccelBadData` | `2` none | `2` none. `3` means dirty cache was lost and is a different conversation | | +| `cpqDaAccelWriteCachePercent` | `0` | **`0`, expected unchanged** — the ratio is not set here. If `ld 2 modify aa=enable` alone moves it, that is itself a finding for #76 | | +| `cpqDaAccelMemory` / `ReadMemory` | `0` / `0` | `0` / `0`, same reasoning | | +| `cpqDaAccelTotalMemory` | `2097152` | unchanged | | +| `cpqDaAccelFailedBatteries` | `1` | unchanged — a #76 curiosity, not a target | | +| `cpqDaCntlrCondition` / `BoardCondition` | `2` / `2` | `2` / `2` | | +| `cpqDaCntlrDriveWriteCacheState` | `1` other | `1` other — **unchanged is the pass condition** | | +| `ssacli ctrl slot=0 show detail` → Cache Ratio | never read | a real ratio, or `0/0` — **this is the #76 reading, and either answer resolves it** | | +| `ssacli ctrl slot=0 ld 1 show detail` → Caching | never read | `Enabled`, or the (b) branch of step 6 | | +| `scrape_duration_seconds{device="shiva"}` | `11.8` s | roughly 13–17 s, and well under 30 | | +| `rate(node_disk_writes_completed_total{device="sda"}[1h])` | current | falls to the host's own writes | | +| the same for `sdb` | absent | present, carrying `alexander`'s writes | | +| **fio 4k QD1 randwrite, HDD** | ADR-0029's **derived** `83` | **measured** — M1 loaded, M3 idle | | +| **fio 4k QD1 randwrite, SSD** | n/a | **measured** — M2. The number that replaces ADR-0029's | | +| **fio 4k QD32 randwrite** | n/a | measured on both — the new ceiling, a different benchmark | | + +The runbook is done when every row has moved, `cpqDaAccelBadData` still reads +`2`, `cpqDaLogDrvCondition{1}` still reads `2`, and the walk is still +comfortably under 30 s. + +## If something goes wrong + +| Symptom | Cause | Fix | +| --- | --- | --- | +| `ssacli` will not install | No trixie suite from HPE's MCP SDR | Paths 2 and 3 in step 2. Record which one worked | +| `ctrl all show` prints a slot other than 0 | This chassis is not wired as assumed | Stop. Every `slot=0` in this runbook is wrong; re-derive them all | +| Drives do not seat in the bays | No HPE SmartDrive carriers | Stop at step 4. This is a purchase, and a purchase edits the roadmap's buy table | +| Drives seat but never appear to `ssacli` | Bays 3–4 not cabled on this backplane | Try bays that are known-good; if none, this layout is not available on this chassis | +| New drives appear, `create` refuses | Drives not `Unassigned`, or a leftover configuration on a used drive | `pd 1I:1:3 show detail`. Do **not** reach for `erase` without re-reading step 5's table | +| `IloHardwareDegraded` pages anyway | The silence named the wrong `cpqDaLogDrvIndex` | Read the real index, silence that, delete the wrong one. Then check whether the alert is *true* | +| `cpqDaAccelStatus` moves to `4` tmpDisabled | The controller parked the cache | Wait — the rule's 1 h `for:` exists for this. If it persists, it is real and #76 wants to know | +| `qm move-disk` refuses | Snapshots on vmid 140 | `qm listsnapshot 140`, delete them, retry | +| Move completes, guest will not boot | Storage moved, guest config did not | `qm config 140` — `scsi0` must name the `ssd` storage. `unused0` is still the intact original | +| The new pool fills | Thin overprovisioning, and nothing alerts on it | Step 7's blind spot, arriving. `pvesm status`, `lvs` | +| fio numbers are wildly high | Caching somewhere in the path | `--direct=1` on every run, and a raw LV target — never a file, never `/dev/sda` | + +## Flipping the documents + +The fit does not finish when the drives are in. These are the documents it makes +stale, in the order they should be touched: + +- **[`../hardware.md`](../hardware.md)** — the Compute table's Storage column + for `Saruman`, which reads `2× 1 TB SAS HDD, RAID 1` and is the reason the + accessories entry says *"The Compute table's Storage column changes when #418 + fits them, and not before."* Model, capacity and **both serials** go into the + accessories entry, which answers + [#148](https://github.com/Gerrrt/HomeLab/issues/148)'s question about where + serials live. +- **ADR-0029** — *"By spindle"* and *"The duty cycle is a spindle decision"* + both rest on ~90 write IOPS. Once step 8 has measured otherwise, the endpoints + running per session are a **choice** rather than a constraint. A dated + `[!NOTE]` on the ADR, status left `Accepted`, in the shape of the one already + at the top of ADR-0029 and the three on ADR-0007. **Never a silent edit** — + [ADR-0001](../adr/0001-record-architecture-decisions.md) makes ADRs immutable. +- **ADR-0007**'s constraint sentence, quoted in #266 and #414, and **ADR-0017**, + which quotes it verbatim to justify NVMe for the range. ADR-0017's argument + was a separate fault domain and not only IOPS, so it survives — but the + sentence it leans on has moved and the note should say so. +- **[`../roadmap.md`](../roadmap.md)** — the #414 paragraph that repeats the + ninety-IOPS reasoning, the #418 entry, and #76 with whichever branch of step 6 + turned out to be true. +- **`stacks/observability/prometheus/rules/network.rules.yaml`** — the iLO + section-header comment records silences by UUID and expiry. Both silences from + step 1 get a paragraph there, in the shape of the 2026-08-31 one already + present. +- **`stacks/observability/prometheus/tests/network.test.yaml`** — the + `IloDrivePredictiveFailure` and `IloDriveSmartUnreadable` fixtures are + single-drive, `cpqDaPhyDrvIndex="0"`. The rule expressions are index-agnostic + so nothing is broken, but the tests stop describing the machine the day this + runbook succeeds. +- **[`build-the-lab-guest.md`](build-the-lab-guest.md)** and + **[`build-the-lab-domain.md`](build-the-lab-domain.md)** — both tell the + reader to leave the Proxmox disk cache at the default *because #76 is open and + nobody has confirmed the controller absorbs writes*. Whichever branch step 6 + lands on changes that argument. +- **`README.md`** — the runbook count, if this file was the one that moved it. + +## What this runbook does not know + +Written down rather than asserted, in the shape of +[`replace-the-smart-storage-battery.md`](replace-the-smart-storage-battery.md)'s +*"What is still open"*. Every item here is something the fit can settle, and +settling it is most of the value of doing the fit carefully. + +1. **Whether `ssacli` can be installed on Proxmox VE 9 at all.** Debian 13 + trixie; HPE's MCP SDR is not known to publish a trixie suite; nothing in this + repository is evidence it has ever run here. Three paths in step 2, none + observed working on this box. +2. **When the drive trays arrive, and whether both do.** No longer *whether* + — `651687-001` × 2 were bought 2026-09-11 and are recorded in + `hardware.md`. But they were quoted for 2026-09-17 and were not here that + morning, and a carrier without its screws holds a drive no better than no + carrier. A hard stop, not a detail. +3. **Whether bays 3 and 4 are cabled.** Both existing drives are Bay 1 and + Bay 2; the backplane variant is recorded nowhere. +4. **Whether a newly created RAID 1 transits `cpqDaLogDrvCondition = 3`.** This + is the whole reason step 1 creates a silence rather than skipping one, and + the fit should record the answer so the next one need not guess. +5. **Whether a brand-new non-HPE SATA SSD reports `cpqDaPhyDrvSmartStatus = 1`** + before the controller configures it. The info severity makes finding out + free. +6. **Whether `cpqDaAccelWriteCachePercent = 0` is a reporting gap or a real + 0 % allocation.** Four columns lean towards real. Only step 6's `ssacli` + reading decides, and it has never been run on this machine. +7. **Whether SSD Smart Path and the array accelerator conflict** on a P440ar — + specifically whether enabling Smart Path disables caching for that logical + drive. Read both lines out of `ld 2 show detail`; do not predict. +8. **Whether `dwc` has a per-array form** on the installed `ssacli`. The + recommendation to leave drive write cache alone depends on it being + controller-wide. +9. **Whether `qm move-disk` refuses with snapshots present**, and whether + `alexander` has any. Check; do not learn it from the error message. +10. **Whether `hpsa` surfaces the new logical drive without a SCSI rescan.** +11. **Whether discard reaches the SSDs** through LVM-thin → hpsa → P440ar, or + stops at the thin pool. It affects long-term steady-state write performance + and nothing in the first day's readings will show it. +12. **Whether the existing drives are SAS or SATA.** `cpqDaPhyDrvType` reads + `3` (sata) for both, on a metric whose enumeration carries a distinct + `4: sas`, and the model string is `MM1000GBKAL` — while + [`../hardware.md`](../hardware.md) and ADR-0007 both say **SAS**. Step 3's + `pd all show detail` prints the interface type authoritatively, and settling + it is a free by-product of a visit that is happening anyway. **Do not + correct either document from an SNMP enum alone**, and note that ADR-0007 + is immutable — if it is wrong, it gets a dated note like everything else. +13. **That the new logical drive's index will be `2`.** Predicted, load-bearing + for step 1's silence, and verifiable within a minute of creating it. +14. **How long the iLO takes to reflect a configuration change.** The 60 s + scrape is not the bound; the iLO's own agentless refresh may be minutes. +15. **The exact option spellings** on the installed `ssacli` — `aa=`, + `ssdsmartpath=`, `dwc=` have all moved between versions. + `ssacli ctrl slot=0 help create` and `help modify` are the authority. + +One thing that belongs beyond this runbook rather than in it: +`collect-smart-state.sh` skips `Saruman` because the iLO covers its array — but +the iLO already reports `4294967295` for the HDDs' endurance columns and +`cpqDaPhyDrvSSDWearStatus` `1` (other), and it will very likely do the same for +two third-party SSDs. That would leave wear monitoring blind on the newest and +most wear-sensitive parts in the estate, on drives bought second-hand. +`smartctl -d cciss,N /dev/sda` reads the drives directly through the `hpsa` +path and would close it. That is an issue, not a step here.