Compaction has no I/O budget: a pass rewrites 300 MB per shard and starves the seal's barriers

[PaulSnow]

Invariant: 1.6, "the protocol path never waits for maintenance", and 1.2. With #92 closed, no maintenance runs on the protocol path; but the seal's fsync barriers still wait behind it in the device queue, and that is the store's cost to bound (2.7: bounded per pass).

Measured. Soak 20260916T185711Z, 60 minutes, bvn2-val4 bvnn store 5.9 GB: each shard's dynamic history is a 220 MB segment plus a 90 MB index behind a few small ones, so a fold rewrites ~300 MB per shard, ~2.4 GB per Compress per node. All four BVN2 nodes were streaming ~200 MB/s of reads and 60-80 MB/s of writes at once; I/O pressure (/proc/pressure/io full) 24-34%; a 4 KB fsync on the volume 16 ms p50 / 48 ms max; the host's dirty budget (vm.dirty_bytes 256 MB) full, so seal goroutines blocked inside write() of the manifest JSON. Twenty goroutine snapshots: seal shard goroutines in fsync 16 times and in throttled write() 11 times. BVN2's execution lag sat at 15-29, refusing all load; BVN1's nodes, not compacting at that moment, sat at lag 6.

CompactPassRecords bounds a pass by records (4 Mi; 32 Mi deep). It says nothing about bytes moved, and nothing about the rate at which they are moved relative to the commit path sharing the device.

Ask. A pass bounded by bytes as well as records, and paced: a rate limit on the merge's reads and writes (or a pass that yields between segments) so a fold costs the seal a bounded wait. A seal-duration histogram (accumulate #4257 asks for it) so the wait is displayed, not inferred from goroutine dumps.

[PaulSnow]

Reproduced on the platform alone (PR #95, cmd/bdbench -stores 9, defaults: 8 shards per store, the soak's per-block volumes, 1 s blocks, same NVMe, no Accumulate):

minute	seal p50 / p90 / max	block p90 / max	blocks of 540	maintenance work per minute	write
1	65 / 84 / 107 ms	426 / 651 ms	540	20 s	117 MB/s
2	95 / 207 / 496 ms	444 / 1,745 ms	535	67 s	175 MB/s
3	116 / 731 / 1,244 ms	1,218 / 2,889 ms	494	229 s	240 MB/s

One store alone ran 30 minutes flat (seal p50 51 ms, block p50 205 ms, compaction 0.7 s per pass). Nine stores on the disk reach the soak's knee by minute 3.

Twenty goroutine snapshots at minute 4: 518 of 519 seal shard goroutine samples in fsync; the store drivers not sleeping were in fsync (29) or waiting on the seal's WaitGroup (29); the maintenance goroutines were in fsync too (55 of 56, writeMergedRun). I/O pressure full 20%, dirty pages 117 MB against the host's 256 MB budget.

Two costs stack: each seal is several barriers per shard (2.3: "four barriers, three deep"), so nine stores of eight shards ask the device for a few hundred fsyncs a second before any maintenance; and compaction's own fsyncs and 240 MB/s of writes fill the queue those barriers wait in. A pass bounded by bytes and rate addresses the second; the first is #33 (residual fsyncs) and is what a heap-and-holes dynamic layer with the index as the sealed object would cut to one barrier per block per store.