Repository navigation
event-loop: replace pipe-based sync call scheduling with kj::Executor::executeSync - #376
Conversation
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ReviewsSee the guideline and AI policy for information on the review process.
If your review is incorrectly listed, please copy-paste ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
967be7a to
df6a91f
Compare
kj::Executor::executeAsynckj::Executor::executeAsync
36143b6 to
c16c467
Compare
kj::Executor::executeAsynckj::Executor::executeSync
|
I’ve been investigating the Bitcoin Core TSan data race failure. Here’s what’s happening, for anyone wondering: Bear in mind that this PR enables cross-thread communication between KJ components for the first time. It exposed a conflict between atomic writes and non-atomic reads of the futex state (Linux only) owned by KJ's event-loop executor object. These non-atomic reads occur when capnproto is built in debug mode and have already been fixed upstream, but only in the At the libmultiprocess level, this issue occurs when:
This is harmless in practice because, despite both targeting the same futex word, they access different bits: the read end only reads the The local sanitize job didn't catch this because capnproto is compiled in release mode, so the assertions are compiled out. In contrast, the Core job compiles capnp in debug mode. In the sanitize job, the nix compiler wrapper adds I'm not sure if this is intentional, but we should decide whether to compile capnp in debug or release mode at both sites (in which case we must keep it consistent) or in one. Nevertheless, I'll be working on a backport PR to fix this upstream for v1 ( |
|
Big concept ACK c16c467! This is a really nice simplification, but it will require careful review to make sure edge case conditions around previous |
ryanofsky
left a comment
There was a problem hiding this comment.
Code review c16c467. Thanks for picking this up, and for the benchmark. It's nice that removing the pipe makes this both simpler and faster.
My main comments are about shutdown: loop() now exits the first time done() is true instead of re-checking it, and sync() no longer holds an EventLoopRef while its call is pending. Details and suggested fixes are in the threads below.
c16c467 to
e04cf68
Compare
|
Thanks for the review, @ryanofsky! I’ve just pushed some changes based on your feedback (compare), including squashing the two commits into one. Regarding the TSan failure, I think the depends patch in Core is the right approach. I also believe libmultiprocess's sanitize job should build in debug mode to keep the libmultiprocess and Core jobs consistent, as mentioned in #376 (comment). |
KJ's debug assertions read the mutex futex word and `AtomicRefcounted` refcounts non-atomically. The TSan job builds capnp without optimization, which automatically enables these assertions, and TSan reports these reads as data races the moment KJ objects start communicating across threads. libmultiprocess enables cross-thread communication for the first time in bitcoin-core/libmultiprocess#376, which replaces the pipe used in `EventLoop::sync()` with `kj::Executor::executeSync()`. Patches are the upstream commits capnproto/capnproto@b91a6ba7c7 and capnproto/capnproto@cc13c8c45f, which are backports of the v2 fixes in capnproto/capnproto#2781 and capnproto/capnproto#2843 (capnproto/capnproto#2855).
ryanofsky
left a comment
There was a problem hiding this comment.
Code review ACK e04cf68. Thanks for the updates and for the new comments, which make the shutdown rules clear.
The TSan failure is the KJ debug-assertion race that bitcoin/bitcoin#36556 fixes, and the NetBSD jobs fail checking out pkgsrc, which looks fixed on master by the NetBSD version bump, so a rebase should help there. I think this should wait for bitcoin/bitcoin#36556 so the TSan job here stays green, but this is just waiting on a guix build.
e04cf68 to
74d5d8b
Compare
6e8e85d depends: fix non-atomic reads in capnp debug assertions (xyzconstant) Pull request description: KJ's debug assertions read the mutex futex word and `AtomicRefcounted` refcounts non-atomically. The TSan job builds capnp without optimization, which automatically enables these assertions, and TSan reports these reads as data races the moment KJ objects start communicating across threads. libmultiprocess enables cross-thread communication for the first time in bitcoin-core/libmultiprocess#376, which replaces the pipe used in `EventLoop::sync()` with `kj::Executor::executeSync()`. This is what triggered the [failure](https://github.com/bitcoin-core/libmultiprocess/actions/runs/38094008438/job/114336046070?pr=376). Patches are the upstream commits capnproto/capnproto@b91a6ba7c7 and capnproto/capnproto@cc13c8c45f, which are backports of the v2 fixes in capnproto/capnproto#2781 and capnproto/capnproto#2843 (backport PR: capnproto/capnproto#2855). ACKs for top commit: Sjors: ACK 6e8e85d ryanofsky: Code review ACK 6e8e85d. Thanks for tracking down the race and getting the fix merged upstream first, which makes the patches easy to check. Tree-SHA512: 07bc877c39b9e598b29eba73d066561dfd981ddf2d750d1cab383697084ecea497b6c7e7d1925fb4eed59bd7ccadfc05eef734b46e83b654a2e31033d34189ab
…uteSync Before this commit, sync calls originated from external threads were coordinated over a pipe: - External threads called `EventLoop::sync()`, which stored a function in `m_sync_fn` and wrote a byte to the pipe using `m_post_writer`. - `EventLoop::loop()` waited on the read end of the pipe. When a byte arrived, it woke up and executed the function stored in `m_sync_fn`, if present. - If `m_sync_fn` was not set, it then checked the `done()` predicate to determine whether to stop the loop. `EventLoopRef::reset()` could also write to the pipe if the last reference was destroyed. KJ already provides a [cross-thread communication mechanism](https://github.com/capnproto/capnproto/blob/v0.9.2/kjdoc/tour.md#threads), which this commit takes advantage of. Additionally, get rid of the pipe-related members and replace them with a promise-fulfiller pair. The pair wakes `EventLoop::loop()` so it can exit and is now triggered only by `EventLoopRef::reset()`. With `m_post_writer` gone, the comments and conditional locking in `EventLoopRef::reset()` no longer hold. Drop them and simplify the function body. `reset()` now calls `fulfill()` with `m_mutex` held, so the `relock` parameter is removed since it is not used anymore. `sync()` no longer holds its own `EventLoopRef` while the call is pending, and `loop()` now returns the first time `done()` is true. Document what callers must do for both to be safe in the `loop()` and `sync()` declarations. Also document that `ListenConnections()` callers need to hold an `EventLoopRef` for as long as connections should be accepted. This is a purely internal improvement and does not change the mp::EventLoop API.
74d5d8b to
174419a
Compare
|
Rebased again to re-run the TSan job now that bitcoin/bitcoin#36556 has just been merged: 74d5d8b -> 174419a (compare). |
|
The macOS job being canceled is unrelated, the good thing is that the TSAn job now passes. |
KJ's debug assertions read the mutex futex word and `AtomicRefcounted` refcounts non-atomically. The TSan job builds capnp without optimization, which automatically enables these assertions, and TSan reports these reads as data races the moment KJ objects start communicating across threads. libmultiprocess enables cross-thread communication for the first time in bitcoin-core/libmultiprocess#376, which replaces the pipe used in `EventLoop::sync()` with `kj::Executor::executeSync()`. Patches are the upstream commits capnproto/capnproto@b91a6ba7c7 and capnproto/capnproto@cc13c8c45f, which are backports of the v2 fixes in capnproto/capnproto#2781 and capnproto/capnproto#2843 (capnproto/capnproto#2855). Backport of bitcoin/bitcoin#36556.
There was a problem hiding this comment.
Code review ACK 174419a. Really nice simplification and performance improvement!
Only change since last review is the rebase onto master, and TSan job is passing now that bitcoin/bitcoin#36556 is merged.

Suggested by ryanofsky in #274 (comment) and #347 (comment)
Currently, synchronous calls from threads without an event loop are coordinated via a pipe.
EventLoop::loop()blocks on the read end until work is scheduled viaEventLoop::sync(). That method stores the function inm_sync_fnand writes a byte to the pipe, waking the loop to run that function. This is what allows other threads to perform I/O without an event loop.But KJ already provides an API for cross-thread communication.: a
const kj::Executor&obtained throughkj::getCurrentThreadExecutor()and shared with other threads so they can schedule calls on the event loop.This PR replaces the existing pipe-based sync-call scheduling with
kj::Executor::executeSync.It also uses
kj::PromiseCrossThreadFulfillerPair<void>to wakeEventLoop::loop()and let it exit. This wake-up is now only triggered fromEventLoopRef::reset(), which before this PR was sharingm_post_writerwithEventLoop::sync().