Skip to content

event-loop: replace pipe-based sync call scheduling with kj::Executor::executeSync - #376

Merged
ryanofsky merged 1 commit into
bitcoin-core:masterfrom
xyzconstant:2026-10-09-use-kj-executor
Nov 12, 2026
Merged

ryanofsky merged 1 commit into
bitcoin-core:masterfrom
xyzconstant:2026-10-09-use-kj-executor

Conversation

@xyzconstant

@xyzconstant xyzconstant commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Suggested by ryanofsky in #274 (comment) and #347 (comment)


Currently, synchronous calls from threads without an event loop are coordinated via a pipe. EventLoop::loop() blocks on the read end until work is scheduled via EventLoop::sync(). That method stores the function in m_sync_fn and writes a byte to the pipe, waking the loop to run that function. This is what allows other threads to perform I/O without an event loop.

But KJ already provides an API for cross-thread communication.: a const kj::Executor& obtained through kj::getCurrentThreadExecutor() and shared with other threads so they can schedule calls on the event loop.

This PR replaces the existing pipe-based sync-call scheduling with kj::Executor::executeSync.

It also uses kj::PromiseCrossThreadFulfillerPair<void> to wake EventLoop::loop() and let it exit. This wake-up is now only triggered from EventLoopRef::reset(), which before this PR was sharing m_post_writer with EventLoop::sync().

@DrahtBot

DrahtBot commented Oct 10, 2026 •

Copy link
Copy Markdown

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews

See the guideline and AI policy for information on the review process.

Type Reviewers
ACK ryanofsky

If your review is incorrectly listed, please copy-paste <!--meta-tag:bot-skip--> into the comment that the bot should ignore.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #390 (proxy: Rethrow EventLoop::sync() callback exceptions in the calling thread by ryanofsky)
  • #368 (proxy: defer thread-local client cleanup and worker joins by Sjors)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@xyzconstant

xyzconstant commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor Author

Ran some numbers, comparing against master in a program that calls EventLoop::sync() from worker threads 50k times each and measures how long each batch of workers takes to finish.

Ran this on my M2 Pro, each variant in a dedicated worktree, built with -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS='-O3'. I ran it many times, and the results have been consistent with what is shown in this plot:

image

I find this interesting. The kj::Executor variant seems almost 2x faster than the pipe-based one.

NOTE: I'm not an expert at this benchmarking thing, and the setup is a bit awkward, so I'm just pasting the generated plot for now and will edit this comment with a Gist link later if/when I structure it better.

@xyzconstant
xyzconstant force-pushed the 2026-10-09-use-kj-executor branch 2 times, most recently from 967be7a to df6a91f Compare October 10, 2026 05:08
@xyzconstant xyzconstant changed the title loop: replace pipe-based sync call scheduling with kj::Executor::executeAsync event-loop: replace pipe-based sync call scheduling with kj::Executor::executeAsync Oct 10, 2026
@xyzconstant
xyzconstant force-pushed the 2026-10-09-use-kj-executor branch 2 times, most recently from 36143b6 to c16c467 Compare October 10, 2026 23:08
@xyzconstant xyzconstant changed the title event-loop: replace pipe-based sync call scheduling with kj::Executor::executeAsync event-loop: replace pipe-based sync call scheduling with kj::Executor::executeSync Oct 11, 2026
@xyzconstant

Copy link
Copy Markdown
Contributor Author

I’ve been investigating the Bitcoin Core TSan data race failure. Here’s what’s happening, for anyone wondering:

Bear in mind that this PR enables cross-thread communication between KJ components for the first time. It exposed a conflict between atomic writes and non-atomic reads of the futex state (Linux only) owned by KJ's event-loop executor object. These non-atomic reads occur when capnproto is built in debug mode and have already been fixed upstream, but only in the v2 branch: capnproto/capnproto#2843.

At the libmultiprocess level, this issue occurs when:

  1. A worker thread calling EventLoop::sync() goes through m_executor.executeSync() into the mutex wait, where a KJ_ASSERT in Mutex::assertLockedByCaller() performs a non-atomic read.

  2. The event loop thread then wakes up inside the promise.wait() call in EventLoop::loop(), and Executor::poll() takes the same mutex with Mutex::lock(), which writes the futex word atomically.

This is harmless in practice because, despite both targeting the same futex word, they access different bits: the read end only reads the EXCLUSIVE_HELD bit, while the write end sets the EXCLUSIVE_REQUESTED bit.

The local sanitize job didn't catch this because capnproto is compiled in release mode, so the assertions are compiled out. In contrast, the Core job compiles capnp in debug mode. In the sanitize job, the nix compiler wrapper adds -O2. In the Core job, CXXFLAGS is passed to depends on the command line, which makes depends drop its default -O2. If no optimization flags are enabled, KJ then assumes it should build in debug mode.

I'm not sure if this is intentional, but we should decide whether to compile capnp in debug or release mode at both sites (in which case we must keep it consistent) or in one. Nevertheless, I'll be working on a backport PR to fix this upstream for v1 (master).

@ryanofsky

Copy link
Copy Markdown
Collaborator

Big concept ACK c16c467! This is a really nice simplification, but it will require careful review to make sure edge case conditions around previous done() and relock code are handled correctly. Also curious how you think we should work around the issue fixed in capnproto/capnproto#2855 so CI passes.

@ryanofsky ryanofsky left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review c16c467. Thanks for picking this up, and for the benchmark. It's nice that removing the pipe makes this both simpler and faster.

My main comments are about shutdown: loop() now exits the first time done() is true instead of re-checking it, and sync() no longer holds an EventLoopRef while its call is pending. Details and suggested fixes are in the threads below.

Comment thread src/mp/proxy.cpp Resolved
Comment thread src/mp/proxy.cpp Resolved
Comment thread src/mp/proxy.cpp Resolved
Comment thread src/mp/proxy.cpp Resolved
Comment thread src/mp/proxy.cpp Resolved
Comment thread include/mp/proxy-io.h Resolved
@xyzconstant
xyzconstant force-pushed the 2026-10-09-use-kj-executor branch from c16c467 to e04cf68 Compare October 16, 2026 20:25
@xyzconstant

Copy link
Copy Markdown
Contributor Author

Thanks for the review, @ryanofsky! I’ve just pushed some changes based on your feedback (compare), including squashing the two commits into one.

Regarding the TSan failure, I think the depends patch in Core is the right approach. I also believe libmultiprocess's sanitize job should build in debug mode to keep the libmultiprocess and Core jobs consistent, as mentioned in #376 (comment).

xyzconstant added a commit to xyzconstant/bitcoin that referenced this pull request Oct 17, 2026
KJ's debug assertions read the mutex futex word and `AtomicRefcounted`
refcounts non-atomically. The TSan job builds capnp without optimization,
which automatically enables these assertions, and TSan reports these reads
as data races the moment KJ objects start communicating across threads.

libmultiprocess enables cross-thread communication for the first time
in bitcoin-core/libmultiprocess#376, which replaces
the pipe used in `EventLoop::sync()` with `kj::Executor::executeSync()`.

Patches are the upstream commits
capnproto/capnproto@b91a6ba7c7 and
capnproto/capnproto@cc13c8c45f, which are backports
of the v2 fixes in capnproto/capnproto#2781 and capnproto/capnproto#2843
(capnproto/capnproto#2855).
@DrahtBot DrahtBot mentioned this pull request Oct 21, 2026

@ryanofsky ryanofsky left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review ACK e04cf68. Thanks for the updates and for the new comments, which make the shutdown rules clear.

The TSan failure is the KJ debug-assertion race that bitcoin/bitcoin#36556 fixes, and the NetBSD jobs fail checking out pkgsrc, which looks fixed on master by the NetBSD version bump, so a rebase should help there. I think this should wait for bitcoin/bitcoin#36556 so the TSan job here stays green, but this is just waiting on a guix build.

Comment thread src/mp/proxy.cpp Resolved
Comment thread include/mp/proxy-io.h Resolved
Comment thread src/mp/proxy.cpp Resolved
@xyzconstant
xyzconstant force-pushed the 2026-10-09-use-kj-executor branch from e04cf68 to 74d5d8b Compare November 10, 2026 17:56
@xyzconstant

xyzconstant commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

Just rebased onto master e04cf68 -> 74d5d8b (compare).

fanquake added a commit to bitcoin/bitcoin that referenced this pull request Nov 10, 2026
6e8e85d depends: fix non-atomic reads in capnp debug assertions (xyzconstant)

Pull request description:

  KJ's debug assertions read the mutex futex word and `AtomicRefcounted` refcounts non-atomically. The TSan job builds capnp without optimization, which automatically enables these assertions, and TSan reports these reads as data races the moment KJ objects start communicating across threads.

  libmultiprocess enables cross-thread communication for the first time in bitcoin-core/libmultiprocess#376, which replaces the pipe used in `EventLoop::sync()` with `kj::Executor::executeSync()`. This is what triggered the [failure](https://github.com/bitcoin-core/libmultiprocess/actions/runs/38094008438/job/114336046070?pr=376).

  Patches are the upstream commits capnproto/capnproto@b91a6ba7c7 and capnproto/capnproto@cc13c8c45f, which are backports of the v2 fixes in capnproto/capnproto#2781 and capnproto/capnproto#2843 (backport PR: capnproto/capnproto#2855).

ACKs for top commit:
  Sjors:
    ACK 6e8e85d
  ryanofsky:
    Code review ACK 6e8e85d. Thanks for tracking down the race and getting the fix merged upstream first, which makes the patches easy to check.

Tree-SHA512: 07bc877c39b9e598b29eba73d066561dfd981ddf2d750d1cab383697084ecea497b6c7e7d1925fb4eed59bd7ccadfc05eef734b46e83b654a2e31033d34189ab
…uteSync

Before this commit, sync calls originated from external threads were coordinated
over a pipe:
- External threads called `EventLoop::sync()`, which stored a function in `m_sync_fn` and wrote a byte to the pipe using `m_post_writer`.
- `EventLoop::loop()` waited on the read end of the pipe. When a byte arrived, it woke up and executed the function stored in `m_sync_fn`, if present.
- If `m_sync_fn` was not set, it then checked the `done()` predicate to determine whether to stop the loop. `EventLoopRef::reset()` could also write to the pipe if the last reference was destroyed.

KJ already provides a [cross-thread communication mechanism](https://github.com/capnproto/capnproto/blob/v0.9.2/kjdoc/tour.md#threads), which this commit takes advantage of.

Additionally, get rid of the pipe-related members and replace them with a
promise-fulfiller pair. The pair wakes `EventLoop::loop()` so it can exit and is
now triggered only by `EventLoopRef::reset()`.

With `m_post_writer` gone, the comments and conditional locking in
`EventLoopRef::reset()` no longer hold. Drop them and simplify the function
body. `reset()` now calls `fulfill()` with `m_mutex` held, so the `relock`
parameter is removed since it is not used anymore.

`sync()` no longer holds its own `EventLoopRef` while the call is pending, and
`loop()` now returns the first time `done()` is true. Document what callers
must do for both to be safe in the `loop()` and `sync()` declarations. Also
document that `ListenConnections()` callers need to hold an `EventLoopRef` for
as long as connections should be accepted.

This is a purely internal improvement and does not change the mp::EventLoop API.
@xyzconstant
xyzconstant force-pushed the 2026-10-09-use-kj-executor branch from 74d5d8b to 174419a Compare November 10, 2026 19:42
@xyzconstant

Copy link
Copy Markdown
Contributor Author

Rebased again to re-run the TSan job now that bitcoin/bitcoin#36556 has just been merged: 74d5d8b -> 174419a (compare).

@xyzconstant

Copy link
Copy Markdown
Contributor Author

The macOS job being canceled is unrelated, the good thing is that the TSAn job now passes.

Sjors pushed a commit to Sjors/sv2-tp that referenced this pull request Nov 11, 2026
KJ's debug assertions read the mutex futex word and `AtomicRefcounted`
refcounts non-atomically. The TSan job builds capnp without
optimization, which automatically enables these assertions, and TSan
reports these reads as data races the moment KJ objects start
communicating across threads.

libmultiprocess enables cross-thread communication for the first time in
bitcoin-core/libmultiprocess#376, which replaces
the pipe used in `EventLoop::sync()` with `kj::Executor::executeSync()`.

Patches are the upstream commits
capnproto/capnproto@b91a6ba7c7 and
capnproto/capnproto@cc13c8c45f, which are
backports of the v2 fixes in capnproto/capnproto#2781 and
capnproto/capnproto#2843
(capnproto/capnproto#2855).

Backport of bitcoin/bitcoin#36556.

@ryanofsky ryanofsky left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review ACK 174419a. Really nice simplification and performance improvement!

Only change since last review is the rebase onto master, and TSan job is passing now that bitcoin/bitcoin#36556 is merged.

@ryanofsky
ryanofsky merged commit 3c74d87 into bitcoin-core:master Nov 12, 2026
16 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants