Saruman's SSDs are fitted and in no logical drive: steps 5–11 of the fit runbook, and the IOPS number ADR-0029 is sized on

[Gerrrt]

What is true

Both SM863a drives went into Saruman's bays on 2026-09-18 and #418 closed on that. docs/runbooks/fit-the-saruman-ssds.md says what that means in its status line: step 4 is done and every other step is not. No ssacli on the host, no logical drive (cpqDaLogDrv still reads index 1 only), no thin pool, alexander still on the HDD mirror, no fio, and no cache reading for #76.

Nothing tracks steps 5–11. The milestone is titled assign the disks, then the domain, and until this issue the first half had no issue.

Why it gates the rest

ADR-0029 sizes the domain — six guests, servers continuous, endpoints per session — on ~90 derived write IOPS that nobody has measured. docs/roadmap.md says so: "an unassigned drive changes no duty cycle either — the sentence above holds until the runbook's step 8 measures both arrays." #414, #266 and #451 all rest on that number.

Order of work

The runbook's own, from where it stopped:

• §2 — ssacli on PVE 9 (item 1 of "What this runbook does not know": never observed working there)
• §5 — create logical drive 2 on the SSD pair
• §6 — the cache reading owed to IloBatteryCondition has been firing on shiva with no silence and no owner #76
• §7 — the thin pool
• §8 — fio on both arrays at 4 KiB / QD1; write the numbers down
• §9 — move alexander
• §10 — delete the silences the fit needed
• §11 — confirm the metrics
• "Flipping the documents": hardware.md, dated notes on ADR-0029/0007/0017, roadmap.md, the network.test.yaml single-drive fixtures, both lab runbooks' disk-cache paragraph

Done when

Step 8's two numbers are in hardware.md, alexander boots from the SSD pool, and the roadmap's "holds until step 8 measures both arrays" sentence has been replaced by what it measured. If the wear columns are still blank after §5, that opens its own issue (SSD wear through hpsa).

Refs #418, #414, #76.

[Gerrrt]

Steps 5 and 7 done on 2026-09-19; 2, 3, 6, 8, 9, 10, 11 not. Recorded in #539.

• §5 — logical drive 2 exists: RAID 1, 915683 MB, PhyDrvIDs 0x0203, index 2 as predicted. The iLO showed it in the 13:40 UTC scrape, absent at 13:39, Condition/Status 2 ok from that first scrape with PercentRebuild never populated; both SSDs went notConfigured → configured in the same walk. Made through the offline Smart Storage Administrator (the runbook's path 3), not ssacli.
• §7 — the thin pool is Large_data (VG, pool and storage id), on sdb; its device-mapper volumes appeared 15:46–15:47 UTC, one ~2.6 GB metadata burst, nothing since. The runbook's ssd is renamed throughout steps 8 and 9.
• §2 — resolved by path 3, which means ssacli is still not on the host and neither install path was tried. §3 and §6 both need it or a second SSA session, so the cache reading owed to IloBatteryCondition has been firing on shiva with no silence and no owner #76 is still owed. The runbook now says: try path 1 once first, and if it is path 3 again, silence by name before powering off (below).
• §6, §8, §9, §11 — not started. sda reads flat at the host's own ~50 KB/s; sdb has carried nothing since the pool was made. alexander is still on pve.
• §1 / §10 — no silence was created and none was needed: IloHardwareDegraded never went pending. Item 4 of "what this runbook does not know" is settled by that (one observation, 60 s granularity).

What did fire. Path 3 was not the twenty-minute reboot §2 priced. The iLO power meter reads 0 W from about 23:00 UTC on 09-18 until the host came back at 13:43 UTC on 09-19 — a 14.5 h power-off — and in that window RemoteWriteJobStale (both Saruman jobs), GuestStateStopped and PatchStateStopped fired at warning, unsilenced, until the boot resolved them; HostRebooted (info) for eight minutes after. cpqDaAccelBadData stayed 2 through it. The runbook's §2 paragraph is struck and replaced.

Label check: skipped, and the window closed with the array. The serials stay as the iLO's.

Wear, re-read with the drives configured: still blank — SSDWearStatus 1, endurance / power-on hours / estimated remaining 4294967295, HasMonInfo false — so #529's condition holds and it is no longer gated.

New issue: #538, the thin-pool blind spot §7 described (HostDiskCritical reads filesystems; a pool is not one), opened while Large_data is still empty.

Settled from the "does not know" list: 4 (no degraded transit seen), 13 (index 2), 14 (a create shows within one scrape, even with the host still in SSA). Still open: 1 (ssacli on PVE 9 — untried), 10 (fresh boot says nothing about a live rescan), and everything §6 and §8 exist to answer.

One residual outside this issue. Saruman's remote write also went silent 15:11–19:10 UTC on 09-18 with the host drawing its usual 92 W, the same afternoon the iLO was unreachable 14:54–15:03 — before the fit, unexplained by it, and not investigated here. Say if it wants an issue.

Next, in order: §2 path 1 from the Mac (apt-get and nothing else if it fails) → §3 and §6 readings (verbatim: Cache Ratio, Caching, Interface Type of 1I:1:1) → §8 M1 and M2 → §9 → §8 M3 → §11 → the document flips.

[Gerrrt]

Operator sheet: the Saruman half of #527, in one sitting

Corrected 2026-09-19 23:00 UTC: the pool is large_data, lowercase, everywhere below. An earlier revision of this sheet wrote Large_data; LVM and PVE storage ids are case-sensitive, and block 5 run against the capitalised name fails on the SSD side while the pve side runs. Copy the name out of pvesm status.

Everything below runs on Saruman as root, from the Mac, inside tmux. It is docs/runbooks/fit-the-saruman-ssds.md steps 2, 3, 6, 8 and 9 in order, with the two corrections in #547 (the trixie suite exists; the fio targets are thin volumes) and the reboot the done condition implies. Nothing on the monitoring host needs silencing for any of it. When the log is posted back here, Claude does step 11 from Prometheus and flips the documents.

Rough time: about an hour, most of it the disk copy in block 6.

Start — one log for everything:

tmux new -s fit527
script -f /root/fit-527-$(date -u +%Y%m%dT%H%MZ).log
date -u

Block 1 — ssacli (step 2, path 1)

for k in key1 key2; do curl -fsSL "https://downloads.linux.hpe.com/SDR/hpePublicKey2048_$k.pub" | gpg --dearmor; done > /usr/share/keyrings/hpe-mcp.gpg
echo "deb [signed-by=/usr/share/keyrings/hpe-mcp.gpg] https://downloads.linux.hpe.com/SDR/repo/mcp trixie/current non-free" > /etc/apt/sources.list.d/hpe-mcp.list
apt-get update && apt-get install -y ssacli
ssacli version
ssacli ctrl all show status

• Must print Smart Array P440ar in Slot 0 (Embedded). Any other slot: STOP — every slot=0 below is wrong.
• apt refuses → path 2: curl -O https://downloads.linux.hpe.com/SDR/repo/mcp/pool/non-free/ssacli-6.60-8.0_amd64.deb && dpkg -i ssacli-6.60-8.0_amd64.deb.
• Installs but finds no controller → STOP and post the log. That is path 3 (offline SSA, a power-off), and the three silences go in from prometheus before the host goes down.

Block 2 — read the controller (step 3)

date -u
ssacli ctrl all show status
ssacli ctrl all show config
ssacli ctrl slot=0 show detail
ssacli ctrl slot=0 pd all show detail
ssacli ctrl slot=0 ld all show detail
ssacli ctrl slot=0 help modify
ssacli ctrl slot=0 help create

Everything read-only. The lines that matter are captured by the log; no need to copy them out.

Block 3 — the cache: enable it on LD 2 only, read the ratio (step 6)

STOP: read help modify first for the exact spellings of aa= and ssdsmartpath= on this version. Then:

date -u
ssacli ctrl slot=0 ld 2 show detail
ssacli ctrl slot=0 ld 2 modify aa=enable
ssacli ctrl slot=0 array B modify ssdsmartpath=enable
ssacli ctrl slot=0 ld 2 show detail
ssacli ctrl slot=0 show detail | grep -iE "cache ratio|cache size|cache memory|drive write cache|no-battery"
ssacli ctrl slot=0 ld 1 show detail | grep -i caching

• Never modify cacheratio= (controller-wide flush, lands on the array holding Proxmox; IloBatteryCondition has been firing on shiva with no silence and no owner #76's to take) and never dwc= (controller-wide; enables the HDDs' unprotected caches too). If help modify shows a per-array dwc form, just note it.
• If either modify is refused, leave it refused and carry on. The reading is the point.

Block 4 — the pools before anything is written

date -u
pvesm status
vgs
lvs -a -o +data_percent,metadata_percent

Block 5 — measure (step 8: M1 loaded HDD, M2 idle SSD)

apt-get install -y fio
lvcreate -V 8G -T large_data/large_data -n fiotest
lvcreate -V 8G -T pve/data -n fiotest
fio --name=fill --filename=/dev/large_data/fiotest --rw=write --bs=1M --iodepth=8 --ioengine=libaio --direct=1 --size=8G
fio --name=fill --filename=/dev/pve/fiotest --rw=write --bs=1M --iodepth=8 --ioengine=libaio --direct=1 --size=8G

STOP: the next run stalls alexander for about 70 seconds. Deliberate, once.

date -u   # M1 start — note it
fio --name=adr0029 --filename=/dev/pve/fiotest --rw=randwrite --bs=4k --iodepth=1 --numjobs=1 --ioengine=libaio --direct=1 --size=8G --time_based --runtime=60 --ramp_time=10 --randrepeat=0 --norandommap --group_reporting
date -u   # M2 start
fio --name=adr0029 --filename=/dev/large_data/fiotest --rw=randwrite --bs=4k --iodepth=1 --numjobs=1 --ioengine=libaio --direct=1 --size=8G --time_based --runtime=60 --ramp_time=10 --randrepeat=0 --norandommap --group_reporting
date -u   # SSD ceiling
fio --name=ceiling --filename=/dev/large_data/fiotest --rw=randwrite --bs=4k --iodepth=32 --numjobs=4 --ioengine=libaio --direct=1 --size=8G --time_based --runtime=60 --ramp_time=10 --group_reporting

Not yet on the HDD at QD32 — that is block 7, once the guest is off it.

Block 6 — move alexander (step 9)

date -u
qm config 140
qm listsnapshot 140

STOP unless listsnapshot is empty. Then, without --delete:

qm move-disk 140 scsi0 large_data
qm config 140

Ten to twenty-five minutes; the guest stays up and is slow. Then STOP: confirm the guest answers (lab Grafana loads, or qm guest exec 140 -- uptime). Only then:

qm set 140 --delete unused0
qm config 140
qm set 140 --scsi0 large_data:vm-140-disk-0,discard=on,iothread=1,ssd=1   # use the exact volid qm config printed
qm reboot 140
qm status 140
qm guest exec 140 -- lsblk -d -o NAME,ROTA,SIZE

ROTA 0 is the guest seeing flash. The reboot is the done condition, not a courtesy.

Block 7 — M3 and the HDD ceiling, now the mirror is idle

date -u   # M3 start
fio --name=adr0029 --filename=/dev/pve/fiotest --rw=randwrite --bs=4k --iodepth=1 --numjobs=1 --ioengine=libaio --direct=1 --size=8G --time_based --runtime=60 --ramp_time=10 --randrepeat=0 --norandommap --group_reporting
date -u   # HDD ceiling
fio --name=ceiling --filename=/dev/pve/fiotest --rw=randwrite --bs=4k --iodepth=32 --numjobs=4 --ioengine=libaio --direct=1 --size=8G --time_based --runtime=60 --ramp_time=10 --group_reporting

Block 8 — clean up and close the log

lvremove -y /dev/large_data/fiotest /dev/pve/fiotest
lvs -a -o +data_percent,metadata_percent
pvesm status
qm config 140
cat /etc/pve/jobs.cfg
date -u
exit

exit ends script. Post the log file here (attach it, or paste it), with the date -u lines intact — they are how the fio runs get found on the graphs.

What Claude needs from the log, if it has to be trimmed

1. ssacli version, the ctrl all show status slot line, and which path worked.
2. ctrl slot=0 show detail in full: Cache Ratio, Total Cache Size, Total Cache Memory Available, No-Battery Write Cache, Drive Write Cache, Battery/Capacitor Status and Count.
3. ld 1 show detail → Caching; ld 2 show detail → Caching and SSD Smart Path, before and after block 3; whether either modify was refused.
4. help modify: the aa=, ssdsmartpath=, dwc= spellings, and whether dwc has an array-scoped form.
5. pd all show detail → Interface Type for 1I:1:1, 1I:1:2, 1I:1:3, 1I:1:4.
6. Every fio run: write: IOPS=, clat … avg=, clat percentiles … 99.00th=, and its date -u.
7. vgs / lvs before lvcreate, and whether the -V … -T form worked.
8. qm listsnapshot 140, qm config 140 after the move and after --delete unused0, qm status 140 after the reboot, the guest's lsblk, pvesm status, lvs -a, /etc/pve/jobs.cfg.

Never, in this sitting

drives=all or drives=allunassigned; delete, modify raid=, erase; fio against /dev/sda or /dev/sdb; modify cacheratio=; dwc=; --delete on move-disk; QD32 on the HDD before the guest has moved.