Accumulate soak 20260904T012004Z (8 nodes, 2 BVNs, bcdb adapter over KVShard with 8 shards, N=20, 1 s blocks, 500 tps), 30 s CPU profile of acc-bvn1-val1 at 18 minutes.
(*segment).lookup is 17.8% of CPU. 94% of it is reached from SegmentStore.lookupHistory, and 82% of the lookup's own time is (*segment).bloomTest; File.ReadAt is 16%. So the cost is not reading values, it is testing the bloom filter of one history segment after another until one admits the key.
Per commit on one shard set (stats.json at commit 950): perm lookups 28,756, filter walked 296, misled 131; dyna lookups 31,984, walked 965, live hits 1,921. The shard directories hold ~40 perm .dat segments and 6–7 dyna each at block ~950 (seg-00000719-0001 through seg-00000842-0000 on Shard0000/perm).
Fleet CPU rose from 6.6 to 17.6 cores between minutes 5 and 15 with execution work flat; roughly half of the rise is GC on the adapter's side (accumulate S4), the rest is this walk lengthening as segments accumulate.
The reads that walk history are legitimate: at 500 tps the load generator touches ~10,000 accounts, most last written more than N blocks ago, so their state is outside the active window by design.
Asks
- A shard-level filter over the history segments (a filter per merged set already exists for packs — a filter over the unpacked history segments too), so a history lookup tests one filter, not forty.
- Or keep the last-touched segment per key in the live index when a key is rewritten, so the common "account touched recently but outside N" case is one seek.
- Report the segment count walked per lookup in
Stats, so the adapter's soak can watch it.
Accumulate soak
20260904T012004Z(8 nodes, 2 BVNs, bcdb adapter over KVShard with 8 shards, N=20, 1 s blocks, 500 tps), 30 s CPU profile ofacc-bvn1-val1at 18 minutes.(*segment).lookupis 17.8% of CPU. 94% of it is reached fromSegmentStore.lookupHistory, and 82% of the lookup's own time is(*segment).bloomTest;File.ReadAtis 16%. So the cost is not reading values, it is testing the bloom filter of one history segment after another until one admits the key.Per commit on one shard set (
stats.jsonat commit 950): perm lookups 28,756, filter walked 296, misled 131; dyna lookups 31,984, walked 965, live hits 1,921. The shard directories hold ~40 perm.datsegments and 6–7 dyna each at block ~950 (seg-00000719-0001throughseg-00000842-0000on Shard0000/perm).Fleet CPU rose from 6.6 to 17.6 cores between minutes 5 and 15 with execution work flat; roughly half of the rise is GC on the adapter's side (accumulate S4), the rest is this walk lengthening as segments accumulate.
The reads that walk history are legitimate: at 500 tps the load generator touches ~10,000 accounts, most last written more than N blocks ago, so their state is outside the active window by design.
Asks
Stats, so the adapter's soak can watch it.