Dyna-history lookups walk every segment with one-byte preads: blooms freed by handoffBelowWindow; add fsync duration

[PaulSnow]

Seen from accumulate's soak 20260906T134054Z (BlockchainDB as the node store, 8 shards, 1 s blocks, ~250 user tx per block; accumulate review docs/reviews/throughput-latency-2026-09-06.md, finding F1).

Measured on one node (bvnn stats.json after 5600 commits): perm LookupTotal 95.7M (FilterAbsent 94.6M), dyna LookupTotal 78.9M (LiveHit 11.7M, FilterAbsent 62.5M, FilterWalked 4.7M) — ~31k lookups per commit; 11,152 filter-absent lookups per commit fall through to a full dyna-history walk. Process-wide 77.5k read syscalls/s; the block-producing goroutine sampled 36 times was in pread 24 times (segment.lookup bloom/index probes, SegmentStore.lookupHistory, readValue). In the 30 s CPU profile SegmentStore.lookupHistory is 77% of getAt; segment.bloomTest 0.92 s, segment.lookup 1.28 s, pread 2.54 s.

Cause (code): KV2.Get (kv_2.go:281-289) asks the dynamic layer first; the mutable store never short-circuits a miss — segstore.go:1791-1795 walks lookupHistory for anything outside the live window; handoffBelowWindow (segstore.go:1085) frees the bloom of every history segment, so each probe is K=3 one-byte ReadAts per history segment (segstore.go:289-313) plus a ~14-ReadAt binary search of the index on a maybe (:347-357); loadBloom runs only for the active segment (seal.go:447). #64 freed blooms for perm history, which grows without bound; dyna history does not, so its blooms (~19 KB per 12.5k-record segment) can stay resident, or a single resident key filter over dyna history can answer absence in memory.

Also: per commit the seal issues ~3-5 fsync barriers per shard that took writes (perm tail seal.go:394, finish :559, index indexmerge.go:237, manifest segstore.go:1680, dyna tail) — ~30-40 per commit — and their duration is not measured (fsyncs are counted). Please add an fsync/seal duration histogram and a pread counter per Get so the next review measures instead of sampling goroutines.

Accumulate-side counterpart (routing write-once shapes to perm first, negative cache, dead probes): accumulatenetwork/accumulate#4249.

[PaulSnow]

Follow-up from the same soak: at 4 h 26 the run stalled — on every node the block commit was in KVShard.SealBlock waiting on the shards' KV2.Seal goroutines, all in syscall.Fsync (pendingSync.finish, pendingSeal.promote -> writeIndexTmp): 12-27 fsync goroutines per process, ~100 across sixteen stores on one LUKS NVMe, none returning for minutes; no kernel error. accumulatenetwork/accumulate#4259. A seal-duration histogram would have said whether the device degraded or one seal stalled the rest; and the seal needs to cost one barrier rather than ~5 waves per shard.

[PaulSnow]

Decision on the accumulate side (accumulatenetwork/accumulate#4259): the consensus log becomes the durability point and the store's seal runs behind the commit, at the merge watermark (20 blocks) or later under disk pressure. The store question this raises: the adapter will call SealBlock(h) for h = last sealed + k (k ≥ 1) while writes for blocks above h have already landed in the tails. From KV2.Seal's comments the cut happens at the current tail position, so the sealed segment would carry k blocks' writes tagged with height h, and the dyna window ages k at once. Please confirm that is sound (height tags, dyna aging, replay of the unsealed tail on open), or provide a seal-range/seal-through call that says which heights a seal covers. The seal-duration histogram asked for above becomes the sealer's main gauge.