Repository navigation
Conversation
Exercise the existing raw byte-vector binding through mp::BuildField and mp::ReadField. Check that received bytes survive mutation and destruction of the Cap'n Proto message, and that an empty field clears an existing payload. Share the checks so other owned payload types can reuse them.
Allow clients to receive a Data field as ipc::Serialized<T>, retaining an owned byte copy without constructing T on the IPC event loop. Provide build and read hooks using the existing libmultiprocess customization points. Run the existing ownership and empty-payload checks for the new wrapper, including arbitrary bytes that cannot be decoded as a block.
Add a manual hot-cache mp::ReadField benchmark comparing CBlock with ipc::Serialized<CBlock>. Use an empty control and a synthetic large-witness block, select production SHA256 acceleration, and alternate conversion order across 101 samples after five warmups. GCC 15.2, RelWithDebInfo -O2, x86 SHA-NI, median milliseconds: payload CBlock Serialized<CBlock> 142 bytes, 1 transaction 0.00020 0.00002 3,713,460 bytes, 1,021 txs 1.829 0.053 The interval includes field conversion and destruction, excluding encoding, transport, and disk I/O. Run with: IPC_SERIALIZED_BENCH=1 build/bin/test_sv2 -t ipc_serialized_tests See doc/benchmark-ipc-serialized.md for methodology.
Check that accepted and rejected submissions both save the complete block returned by getBlock, including transaction witness data. Allow the mock to return a known block and compare the saved file byte for byte. This test passes with the existing typed getBlock implementation.
Keep getBlock returning CBlock and add getSerializedBlock for owned IPC bytes. Map the serialized accessor to RPC ordinal 2 and provide the typed getBlock through the client adapter, decoding on the calling thread. Use the serialized accessor for diagnostic saves. Parse only the header for the filename and write the exact returned bytes, avoiding transaction construction and hashing on the IPC loop. This can save several milliseconds for large blocks. Extend the diagnostic-save test with malformed transaction data after accepted and rejected submissions. Check both block accessors agree and that typed decoding failures remain local. All 64 test_sv2 cases pass.
|
@ryanofsky do you think |
cb34bff to
9d71796
Compare
xyephy
left a comment
There was a problem hiding this comment.
Concept ACK. Tested 9d71796: each commit builds and passes the unit tests on macOS (clang), and the head also on Linux (GCC). The two new template provider tests are clean under repeated TSan runs.
End-to-end against Bitcoin Core v31.1 and v32.1 on regtest, with segwit transactions in the mempool:
- pool + mining_device: every saved
.datfile is byte-identical togetblock <hash> 0, including blocks with witness transactions. - pool with embedded JDS + JDC on sv2-tp: every
RequestTransactionDatasucceeded and the JDS accepted every declared job.
I ran the same JDS + JDC setup against v32.1 on signet and on mainnet for a few minutes each. There were no RequestTransactionData errors and every declaration was accepted, including mainnet templates with ~7,600-7,900 transactions. On signet the CPU miner reached the proof-of-work target and Core rejected the blocks for the missing signet signature (bad-signet-blksig). sv2-tp still saved them, and all 31 unique files parse fully, with header hash and merkle root consistent.
I also timed the CBlock decode on six recent mainnet blocks (479 to 7,580 transactions). Full blocks took about as long as the synthetic 3.7 MB benchmark block, up to roughly twice as long, on both an arm64/clang and an x86/GCC machine. The synthetic block is mostly a few large witness items, so "several milliseconds" holds for real blocks.
Test coverage at 9d71796:
- No mock template transaction carries a witness (neither the coinbase nor
MakeDummyTx()). Switching the client decode inmining.cpptoTX_NO_WITNESSstill passes every test. In 7a67a91,save_getblocksent a witness through the typedgetBlock(), but 9d71796 switched that override to raw bytes, so the coverage was lost. Real Core's coinbase carries the 32-byte witness reserved value, whichRequestTransactionDatasends asexcess_data. I tried adding it to the mock coinbase:client_teststhen needs itsRequestTransactionData.Successsize updated, and with an assertion onexcess_datathe mutation is caught byget_block_accessors. Happy to share the diff. MockBlockTemplate::getBlock(), including its override branch, is never reached over IPC, since@2now dispatches togetSerializedBlock()(anabort()there still passes).
With Apple clang the benchmark reports Serialized_ms=0. The timed loop only uses value.bytes.size(), so the copy is elided, while GCC still performs it. Reading one byte of the result makes it measurable.
An optional alternative that may suit sv2-tp, as a pure IPC client: instead of renaming the capnp method and adding the ProxyClientCustom override, interfaces::BlockTemplate::getBlock() could return ipc::Serialized<CBlock> directly, with RequestTransactionData decoding via a SpanReader. That keeps the BlockTemplate capnp interface identical to Core's and drops mining-client.h and the $Proxy.include. The difference moves to the return type in interfaces/mining.h, which already diverges from Core. I tried it: it builds, the remaining tests pass (get_block_accessors no longer applies), and the end-to-end results above are the same on v31.1 and v32.1. The cost is that callers decode by hand. In Core, local callers such as miner_tests want a CBlock, so if ipc::Serialized<T> goes upstream first, the current shape is probably the better fit and this can be ignored.
// interfaces/mining.h
virtual ipc::Serialized<CBlock> getBlock() = 0;
// RequestTransactionData
const auto serialized{(*cached_block->second.second).getBlock()};
SpanReader reader{MakeByteSpan(serialized.bytes)};
reader >> TX_WITH_WITNESS(block);Nits:
- "getBlock RPC" (the
mining.capnpcomment and theget_block_accessorscomment) and "RPC ordinal 2" (the 9d71796 message) could read as JSON-RPC; "IPC method" would avoid that. CustomReadFieldforSerialized<T>leaves existing bytes in place when the field is absent. That's unreachable fromclientInvoke, which default-constructs the result, but clearing first would match the Data hook.
I prefer to keep the regular |
|
re: #154 (comment)
I think these changes are reasonable workarounds for libmultiprocess performance problems, but I'd rather not add
|
|
Thanks. I'm marking this as draft pending the direction in bitcoin/bitcoin#36525. |
Thanks. One correction to my test-coverage note: with #141 / sv2-spec#241 moving |
While working on an alternative approach to bitcoin/bitcoin#36525, I noticed that
CBlockdeserialization takes several milliseconds, and that this blocks the main IPC thread.Since we want to write the block straight to disk (for diagnostics), there's no need to deserialize it.
Introduce
ipc::Serialized<T>as a general way to bypass this and then havegetBlock()wrap aroundgetSerializedBlock().Add a manual hot-cache
mp::ReadFieldbenchmark comparingCBlockwithipc::Serialized<CBlock>. Use an empty control and a synthetic large-witnessblock, select production SHA256 acceleration, and alternate conversion
order across 101 samples after five warmups.
GCC 15.2, RelWithDebInfo -O2, x86 SHA-NI, median milliseconds:
payload CBlock Serialized
142 bytes, 1 transaction 0.00020 0.00002
3,713,460 bytes, 1,021 txs 1.829 0.053
The interval includes field conversion and destruction, excluding encoding,
transport, and disk I/O.