Repository navigation
Description
Found by ThreadSanitizer in a Bitcoin Core bitcoin-node build (libmultiprocess e8de5c7b68, capnproto 1.0.1) running under Antithesis (see here for how use it in Bitcoin Core).
LLM analysis:
Summary
Server method bodies run on a worker thread and call
call_context.getResults()directly from that thread. When the peer disconnects abruptly, capnp'sRpcConnectionState::disconnect()runs on the event loop thread and clobbers the connection state under the worker's feet. Nothing synchronizes the two.Both TSAN reports hit the same
RpcConnectionStateheap block, on the samekj::OneOf<Own<VatNetworkBase::Connection>, Exception> connectionmember:
offset writer — T2 b-capnp-loopreader — T27 (IPC worker) +0x38(ptr)Own::Own(Own&&)Own::operator->()+0x28(tag)destroyVariant/init<Exception>OneOf::is<Own<Connection>>()capnproto-c++-1.0.1
src/capnp/rpc.c++:// writer: RpcConnectionState::disconnect(), :426-427 auto dyingConnection = kj::mv(connection.get<Connected>()); // write +0x38 connection.init<Disconnected>(kj::cp(networkException)); // write +0x28// reader: RpcConnectionState::RpcCallContext::getResults(), :2581-2584 if (redirectResults || !connectionState->connection.is<Connected>()) { // read +0x28 ... } else { auto message = connectionState->connection.get<Connected>()->newOutgoingMessage( // read +0x38A TOCTOU on the connection state: the worker passes
is<Connected>(), the event loop then disconnects, and the worker dereferences a moved-from (null)Own— or takes the raw pointer just beforedyingConnectiondrops the last reference and writes the return message into a freed transport.Abridged stacks:
Write of size 8 at 0x72780002f438 by thread T2: #0 kj::Own<capnp::_::VatNetworkBase::Connection>::Own(Own&&) #1 capnp::_::RpcConnectionState::disconnect(kj::Exception&&) rpc.c++ #2 capnp::_::RpcConnectionState::taskFailed(kj::Exception&&) rpc.c++ #3 kj::TaskSet::Task::fire() #5 kj::EventLoop::turn() #9 kj::Promise<unsigned long>::wait(kj::WaitScope&, kj::SourceLocation) #10 mp::EventLoop::loop() src/mp/proxy.cpp:247 Previous read of size 8 at 0x72780002f438 by thread T27 (mutexes: write M0): #0 kj::Own<capnp::_::VatNetworkBase::Connection>::operator->() #1 capnp::_::RpcConnectionState::RpcCallContext::getResults(kj::Maybe<capnp::MessageSize>) rpc.c++ #2 capnp::CallContext<Mining::CreateNewBlockParams, Mining::CreateNewBlockResults>::getResults(...) #3 mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>::invoke<...> ... #16 mp::Unlock<mp::Lock, kj::Function<void ()>&>(mp::Lock&, kj::Function<void ()>&) include/mp/util.h:216 #17 mp::Waiter::wait<mp::ProxyServer<mp::ThreadMap>::makeThread(...)::$_0::...> #20 mp::ProxyServer<mp::ThreadMap>::makeThread(...)::$_0::operator()() constWhy it happens
PassFieldformp.Context(include/mp/type-context.h:74-196) posts the method body to a dedicated worker thread viaProxyServer<Thread>::post. The generatedServerRet::invokethen callscall_context.getResults()on that thread — but capnpRpcCallContextis event-loop-thread-only.The event loop holds no lock while capnp runs.
EventLoop::loop()(src/mp/proxy.cpp:246-247) acquiresm_mutexonly afterwait_stream->read(...).wait(waitScope)returns; the entire kj loop — includingtaskFailed→disconnect()— turns inside thatwait()with the mutex released.The
M0the worker holds is the per-requestcancel_mutex(include/mp/type-context.h:100-101), which is documented (:113-126) to block the event loop thread from freeing the request's params/results structs mid-execution. But it only guards the promise-cancellation path:Connection::~Connection→m_canceler.cancel()→CancelProbedtor →CancelMonitor::m_on_cancel→Lock cancel_lock{cancel_mutex}.On a remote disconnect, capnp's
RpcConnectionState::disconnect()runs first, directly fromtaskFailed, before theonDisconnecthandler destroys theConnection. That path never touchescancel_mutexand never waits for in-flight workers. That's the unprotected window.(TSAN's "created at" stack for
M0points at a different request'spost; expected, sincecancel_mutexis a stack local and the address is reused.)Reproduction
Sequence from the run:
22.618 node: "IPC server: socket connected" (client connects) 23.511 test harness SIGKILLs the client (exit 137) 23.617 node: "CreateNewBlock(): block weight: 804" (worker T27 executing the call) 23.675 TSAN: data raceMinimal recipe: TSAN-instrumented server, client issues a
Mining.createNewBlock(any method whose body is slow enough to still be executing), SIGKILL the client mid-call. The abrupt EOF drivestaskFailed→disconnect()concurrently with the worker'sgetResults().Antithesis' coverage instrumentation (
__sanitizer_cov_trace_pc_guard_internal→sleep_for) lands inside the window, which is why TSAN addsAs if synchronized via sleep— that's what widened the race, not a false positive.Possible fixes
- Route
getParams()/getResults()throughloop.sync()so all capnp access stays on the event loop thread. Correct, but a round-trip per field access.- Copy params into a worker-owned
MessageBuilderbefore dispatch and write results back in a singleloop.sync()after the body returns. One hop per call, and it removes the need for thecancel_mutex/request_canceledhandshake entirely, since the worker would no longer alias anything the event loop can free.- Make the
cancel_mutexhandshake cover capnp's internal disconnect — would need a hook insideRpcConnectionState::disconnect(), i.e. patching capnp.
Activity
dergoegge commentedon Aug 19, 2026
The LLM made up the capnp version. This happens with the version we have in depends (1.5.0), and the capnp rpc code remains the same.
ryanofsky commentedon Aug 19, 2026
Thanks! The thing I'm confused by here is that case is supposed to be already handled by cancelling the request in the onDisconnect handler
libmultiprocess/src/mp/proxy.cpp
Lines 117 to 118 in 275c8ee
But the bug report seems to suggest that capnproto is actually discarding all connection state before triggering onDisconnect, which is surprising. If connection state can literally just disappear at any time with no notification than that really leaves us with few options other than the unappealing possible fixes suggested by the LLM above of synchronizing every access to parameters and results (1) or making complete copies of them (2). I thought we were already doing (3)...
ryanofsky commentedon Aug 19, 2026
To answer my questions above:
onDisconnecthandler doesn't help with this case because it runs too late, only after theVatNetworkBase::Connectionobject is destroyed, not before.cancel_mutexto handle disconnects already prevents the results message from being deleted while the worker thread might access it. It just doesn't prevent the connection state which is read by theRpcCallContext::getResultsfrom being updated by the capnproto event loop in the background.Turns out there is a nice one-line fix for this implemented in #349:
proxy: fix data race between server request threads and disconnect ha…
Merge #349: type-context: fix async disconnect race condition found b…
proxy: fix data race between server request threads and disconnect ha…
ryanofsky commentedon Sep 28, 2026
Closing this, since #349 fixed it. The fix is something that could be easily backported to 32.0, but it looks like the worst form of the bug is very unlikely to happen in practice so not planning backporting it currently.
Technically the bug can happen when there's a disconnect during any method call taking an
mp.Contextparameter if theres unlucky scheduling between the event loop thread inRpcConnectionState::disconnect():and the worker thread in
RpcCallContext::getResults(), which tests the tag and then dereferences the pointer:If the worker reads between the two writes in the event loop thread, there could be a null pointer dereference on the last line because get() returns null. But coupled with the fact that the disconnect has to happen at the moment the worker thread is calling
getResultsmeans that this crash is very unlikely to happen while the node is running.This bug is more likely to happen during shutdown, but even then requires the bad thread interleaving, and we have only seen symptoms in practice with antithesis which uses artificial thread scheduling and in the #349 unit test which relies on threadsanitizer. Crashes during shutdown are a real problem but less serious than crashes during regular node operation and there is bigger shutdown crash tracked in bitcoin/bitcoin#35845 and bitcoin/bitcoin#33387 and fixed in bitcoin/bitcoin#35932 and #335 which is easier to trigger than this.