Skip to content

Add windows support - #231

Open
ryanofsky wants to merge 11 commits into
bitcoin-core:masterfrom
ryanofsky:pr/win
Open

ryanofsky wants to merge 11 commits into
bitcoin-core:masterfrom
ryanofsky:pr/win

Conversation

@ryanofsky

@ryanofsky ryanofsky commented Oct 22, 2025 •

Copy link
Copy Markdown
Collaborator

Add support for running on windows. These changes make the libmultiprocess API more generic, using stream types instead of file descriptors. All features are supported, including spawning processes with socket connections to the parent process. These changes were originally made in bitcoin/bitcoin#32387

@DrahtBot

DrahtBot commented Oct 22, 2025 •

Copy link
Copy Markdown

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews

See the guideline and AI policy for information on the review process.

Type Reviewers
Concept ACK enirox001
Approach ACK hebasto, ViniciusCestarii

If your review is incorrectly listed, please copy-paste <!--meta-tag:bot-skip--> into the comment that the bot should ignore.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #383 (ci: Compile capnproto with GCC in gnu32 job by xyzconstant)
  • #376 (event-loop: replace pipe-based sync call scheduling with kj::Executor::executeSync by xyzconstant)
  • #368 (proxy: defer thread-local client cleanup and worker joins by Sjors)
  • #333 (nix: Make clang tooling and IWYU find the right standard library headers by ryanofsky)
  • #175 (Set cmake_minimum_required(VERSION 3.22) by maflcko)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

LLM Linter (✨ experimental)

Possible typos and grammar issues:

  • include/mp/util.h: It it possible → It is possible [typo]

Possible places where named args for integral literals may be used (e.g. func(x, /*named_arg=*/0) in C++, and func(x, named_arg=0) in Python):

  • src/mp/util.cpp: WSASocket(FROM_PROTOCOL_INFO, FROM_PROTOCOL_INFO, FROM_PROTOCOL_INFO, &info, 0, WSA_FLAG_OVERLAPPED | WSA_FLAG_NO_HANDLE_INHERIT)

2026-10-20 07:26:50

@ryanofsky

Copy link
Copy Markdown
Collaborator Author

Rebased 7d2f3c4 -> 2975fac (pr/win.1 -> pr/win.2, compare) due to conflicts with #237

@pavlenex

pavlenex commented Apr 8, 2026

Copy link
Copy Markdown

We have miners using Stratum V2 that are on Windows and would like to see Windows IPC support landing, is there any estimate when they can expect this to materialize? We have a dozen of users who can help with testing if that's a blocker.

@Sjors

Sjors commented Apr 8, 2026 •

Copy link
Copy Markdown
Member

@pavlenex testing would definitely be useful. I guess for that to work we'd need a stack of pull requests: this one here in libmultiprocess, one in Bitcoin Core that enables IPC support in the Windows Guix build, and then an SRI pull request that uses it. Testers would then have download (or build themselves) the custom bitcoin core and SRI binaries.

I've added Windows support to my v32 wish list: bitcoin/bitcoin#33777

No guarantees obviously.

@ryanofsky

Copy link
Copy Markdown
Collaborator Author

I'll try to get this PR ready for review this week, split up into smaller commits and with ci passing. From there as Sjors mentioned there is a lot more work to do: more code changes in bitcoin/bitcoin#32387 that need to be made in bitcoin core, enabling IPC into windows builds in bitcoin core, enabling it in windows CI jobs with pycapnp, adding client support, probably adding a windows CI job to this repository.

v32 sounds like a good target though and it is very useful to know there is demand for this feature, because it hasn't been a priority so far

@ryanofsky

ryanofsky commented Apr 15, 2026 •

Copy link
Copy Markdown
Collaborator Author

PR is split up into commits now and should be reviewable. CI is not passing but failures look like IWYU errors. I also opened bitcoin/bitcoin#35084 with corresponding bitcoin core changes. Windows support for bitcoin core can be tested with bitcoin/bitcoin#32387 which combines both PRs and enables IPC by default in windows builds.

Rebased 2975fac -> cb16d2e (pr/win.2 -> pr/win.3, compare) splitting changes up into more reviewable commits

Updated cb16d2e -> e563c96 (pr/win.3 -> pr/win.4, compare) to fix various ci failures #231 (comment): macos/freebsd shutdownwrite fail, bitcoin core ci jobs api incompatibility, iwyu and olddeps fixes

Updated e563c96 -> d9fcac6 (pr/win.4 -> pr/win.5, compare) to fix IWYU errors https://github.com/bitcoin-core/libmultiprocess/actions/runs/24539430990/job/71741856575?pr=231

Added 1 commits d9fcac6 -> a1748e2 (pr/win.5 -> pr/win.6, compare) to fix bitcoin core macos exception type error https://github.com/bitcoin-core/libmultiprocess/actions/runs/24541950788/job/71749481354?pr=231

Updated a1748e2 -> 18fc188 (pr/win.6 -> pr/win.7, compare) with updates from bitcoin/bitcoin#32387 pr/ipc-win.23

Updated 18fc188 -> 7fd5ec4 (pr/win.7 -> pr/win.8, compare) adding workaround for ubuntu packaging bug exposed by cmake change https://github.com/bitcoin-core/libmultiprocess/actions/runs/24746487506/job/72399345861, also fixing olddeps include and iwyu errors https://github.com/bitcoin-core/libmultiprocess/actions/runs/24746487510, also fixing more MSVC errors, and rearranging commits

@Sjors

Sjors commented Apr 20, 2026 •

Copy link
Copy Markdown
Member

How realistic is it to add a Windows CI job here? (can be cross-compiled)

This was referenced Apr 20, 2026
Sjors added a commit to Sjors/sv2-tp that referenced this pull request Apr 20, 2026
The lifecycle test repeatedly constructs and destroys TPTester to
exercise IPC EventLoop teardown. On Windows it hangs intermittently
in std::thread::join during teardown of libmultiprocess thread-local
state. This is a known libmultiprocess Windows issue, not specific
to sv2-tp:

  bitcoin-core/libmultiprocess#231
  bitcoin/bitcoin#32387

The fix lives upstream and rewrites the EventLoop wakeup primitive
(raw fd -> KJ stream) and adds shutdownWrite() in ~Connection. Until
that lands and is backported into our libmultiprocess subtree, gate
this particular test off on _WIN32. The other sv2 unit tests run
normally on Windows; only the explicit teardown loop is affected.

Assisted-by: GitHub Copilot
Assisted-by: Anthropic Claude Opus 4
Sjors added a commit to Sjors/sv2-tp that referenced this pull request Apr 20, 2026
The template_provider_tests `client_tests`, `fee_timer_blocking_test`
and `new_tip_bypasses_fee_timer_test` cases each construct and tear
down a TPTester. Like the lifecycle test skipped in the previous
commit, that teardown deadlocks intermittently on Windows in
std::thread::join during libmultiprocess thread-local state cleanup.
This is a known libmultiprocess Windows issue, not specific to
sv2-tp:

  bitcoin-core/libmultiprocess#231
  bitcoin/bitcoin#32387

Gate the three TPTester-using cases off on _WIN32 with a TODO. The
non-IPC test `block_reserved_weight_floor` continues to run on
Windows.

Assisted-by: GitHub Copilot
Assisted-by: Anthropic Claude Opus 4
Sjors added a commit to Sjors/sv2-tp that referenced this pull request Sep 28, 2026
Rebase bitcoin-core/libmultiprocess#231 onto upstream master
2dba130478 and include the MinGW ThreadContext workaround from
bitcoin/bitcoin#32387. Preserve current upstream race fixes and CI.

The rebased upstream history is retained locally on
codex/libmultiprocess-windows.
Sjors added a commit to Sjors/sv2-tp that referenced this pull request Sep 28, 2026
Rebase bitcoin-core/libmultiprocess#231 onto upstream master
2dba130478 and include the MinGW ThreadContext workaround from
bitcoin/bitcoin#32387. Preserve current upstream race fixes and CI.

The rebased upstream history is retained locally on
codex/libmultiprocess-windows.
ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Oct 1, 2026
…g to named pipe

SpawnProcess on Windows creates a named pipe with PIPE_WAIT and then calls
ConnectNamedPipe synchronously. If the child exits or crashes before
connecting, ConnectNamedPipe blocks forever with no recovery path.

Fix by opening the pipe with FILE_FLAG_OVERLAPPED and using
WaitForMultipleObjects on both the connect event and the child process
handle. If the process handle signals first, the child died without
connecting and SpawnProcess throws instead of hanging. Since the pipe is
now in overlapped mode, WriteFile also requires an OVERLAPPED structure;
use GetOverlappedResult with bWait=TRUE to handle both synchronous and
asynchronous completion.

Add a Windows-only test that spawns a child which exits immediately
without opening the named pipe and asserts SpawnProcess does not block.

(bitcoin-core/libmultiprocess#231 (comment))

Co-Authored-By: ViniciusCestarii <viniciuscestari01@gmail.com>
This change was written with Claude Sonnet 4.6.
ryanofsky added a commit to ryanofsky/libmultiprocess that referenced this pull request Oct 2, 2026
…g to named pipe

SpawnProcess on Windows creates a named pipe with PIPE_WAIT and then calls
ConnectNamedPipe synchronously. If the child exits or crashes before
connecting, ConnectNamedPipe blocks forever with no recovery path.

Fix by opening the pipe with FILE_FLAG_OVERLAPPED and using
WaitForMultipleObjects on both the connect event and the child process
handle. If the process handle signals first, the child died without
connecting and SpawnProcess throws instead of hanging. Since the pipe is
now in overlapped mode, WriteFile also requires an OVERLAPPED structure;
use GetOverlappedResult with bWait=TRUE to handle both synchronous and
asynchronous completion.

Add a Windows-only test that spawns a child which exits immediately
without opening the named pipe and asserts SpawnProcess does not block.

(bitcoin-core#231 (comment))

Co-Authored-By: ViniciusCestarii <viniciuscestari01@gmail.com>
This change was written with Claude Sonnet 4.6.
ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Oct 2, 2026
…g to named pipe

SpawnProcess on Windows creates a named pipe with PIPE_WAIT and then calls
ConnectNamedPipe synchronously. If the child exits or crashes before
connecting, ConnectNamedPipe blocks forever with no recovery path.

Fix by opening the pipe with FILE_FLAG_OVERLAPPED and using
WaitForMultipleObjects on both the connect event and the child process
handle. If the process handle signals first, the child died without
connecting and SpawnProcess throws instead of hanging. Since the pipe is
now in overlapped mode, WriteFile also requires an OVERLAPPED structure;
use GetOverlappedResult with bWait=TRUE to handle both synchronous and
asynchronous completion.

Add a Windows-only test that spawns a child which exits immediately
without opening the named pipe and asserts SpawnProcess does not block.

(bitcoin-core/libmultiprocess#231 (comment))

Co-Authored-By: ViniciusCestarii <viniciuscestari01@gmail.com>
This change was written with Claude Sonnet 4.6.
ryanofsky added a commit to ryanofsky/libmultiprocess that referenced this pull request Oct 2, 2026
…g to named pipe

SpawnProcess on Windows creates a named pipe with PIPE_WAIT and then calls
ConnectNamedPipe synchronously. If the child exits or crashes before
connecting, ConnectNamedPipe blocks forever with no recovery path.

Fix by opening the pipe with FILE_FLAG_OVERLAPPED and using
WaitForMultipleObjects on both the connect event and the child process
handle. If the process handle signals first, the child died without
connecting and SpawnProcess throws instead of hanging. Since the pipe is
now in overlapped mode, WriteFile also requires an OVERLAPPED structure;
use GetOverlappedResult with bWait=TRUE to handle both synchronous and
asynchronous completion.

Add a Windows-only test that spawns a child which exits immediately
without opening the named pipe and asserts SpawnProcess does not block.

(bitcoin-core#231 (comment))

Co-Authored-By: ViniciusCestarii <viniciuscestari01@gmail.com>
This change was written with Claude Sonnet 4.6.
ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Oct 2, 2026
…g to named pipe

SpawnProcess on Windows creates a named pipe with PIPE_WAIT and then calls
ConnectNamedPipe synchronously. If the child exits or crashes before
connecting, ConnectNamedPipe blocks forever with no recovery path.

Fix by opening the pipe with FILE_FLAG_OVERLAPPED and using
WaitForMultipleObjects on both the connect event and the child process
handle. If the process handle signals first, the child died without
connecting and SpawnProcess throws instead of hanging. Since the pipe is
now in overlapped mode, WriteFile also requires an OVERLAPPED structure;
use GetOverlappedResult with bWait=TRUE to handle both synchronous and
asynchronous completion.

Add a Windows-only test that spawns a child which exits immediately
without opening the named pipe and asserts SpawnProcess does not block.

(bitcoin-core/libmultiprocess#231 (comment))

Co-Authored-By: ViniciusCestarii <viniciuscestari01@gmail.com>
This change was written with Claude Sonnet 4.6.
ryanofsky and others added 11 commits October 13, 2026 02:14
Remove recently introduced SpawnConnectInfo and SpawnConnectInfoToArgsFn type
aliases since they are the same on all platforms and might obscure the fact
that connect info should be treated as an opaque string.

Co-authored-by: Sjors Provoost <sjors@sprovoost.nl>
This change was written with Claude Sonnet 4.6.
Replace the Unix-only UnixListener test helper with a cross-platform
SocketListener backed by a std::variant<sockaddr_un> (TCP arm will be
added later for Wine compatibility).

- test/socketlistener.h: New header with SocketListener class extracted from
  the pattern of UnixListener but using SocketId/SocketError types,
  std::filesystem for temp dir cleanup, and mp::CloseSocket for teardown.
  Uses std::variant<sockaddr_un> so the TCP arm can be appended later without
  restructuring the class.
- test/unixlistener.h: Deleted.
- test/listen_tests.cpp: Switch to socketlistener.h and socketlistener.
- test/connect_tests.cpp: Switch to socketlistener.h and socketlistener.

This change was written with Claude Sonnet 4.6.
Move POSIX-only headers in util.cpp into a #ifndef WIN32 block so the file
compiles on Windows without modification. Guard the Unix-only helpers (MakeArgv,
MaxFd, ChildFail, SpawnError*, ReadSpawnResult, WriteSpawnError, KillAndReapChild)
with a single #ifndef WIN32 block. Guard the extern "C" environ declaration
likewise. Guard Unix-only includes in spawn_tests.cpp and connect_tests.cpp and
wrap the Unix-only spawn tests with #ifndef WIN32.

No Windows implementations are added here; those come in a later commit.

This change was written with Claude Sonnet 4.6.
Co-authored-by: Sjors Provoost <sjors@sprovoost.nl>
Add Windows-specific code to support building and running on Windows:

- util.h: Guard ProcessId/SocketId/SocketError type aliases with WIN32
  ifdefs so they use SOCKET/uintptr_t on Windows and int on Unix.
  Add winsock2.h include on Windows.
- util.cpp: Guard Unix-specific system headers with WIN32 ifdefs. Add
  Windows-specific includes (windows.h, winsock2.h). Guard MaxFd() with
  #ifndef WIN32. Add GetCurrentThreadId() branch in ThreadName(). Add
  win32Socketpair() forward-declare. Add Windows branch in SocketPair()
  using win32Socketpair(). Add CommandLineFromArgv() helper needed to
  construct CreateProcess command lines. Add Windows branch in
  SpawnProcess() using named pipes and WSADuplicateSocket to pass socket
  to child. Add Windows branch in StartSpawned() reading socket from
  named pipe. Add Windows branch in WaitProcess() using
  WaitForSingleObject/GetExitCodeProcess.
- proxy.cpp: Add SocketOutputStream class on Windows (analogous to
  FdOutputStream but using SOCKET/send()). Add Windows branch in
  EventLoop constructor to create m_post_writer using SocketOutputStream.

Co-Authored-By: ViniciusCestarii <viniciuscestari01@gmail.com>
…g to named pipe

SpawnProcess on Windows creates a named pipe with PIPE_WAIT and then calls
ConnectNamedPipe synchronously. If the child exits or crashes before
connecting, ConnectNamedPipe blocks forever with no recovery path.

Fix by opening the pipe with FILE_FLAG_OVERLAPPED and using
WaitForMultipleObjects on both the connect event and the child process
handle. If the process handle signals first, the child died without
connecting and SpawnProcess throws instead of hanging. Since the pipe is
now in overlapped mode, WriteFile also requires an OVERLAPPED structure;
use GetOverlappedResult with bWait=TRUE to handle both synchronous and
asynchronous completion.

Add a Windows-only test that spawns a child which exits immediately
without opening the named pipe and asserts SpawnProcess does not block.

(bitcoin-core#231 (comment))

Co-Authored-By: ViniciusCestarii <viniciuscestari01@gmail.com>
This change was written with Claude Sonnet 4.6.
Allow code to compile without pthreads available, as required for MSVC
compatibility. Avoid unconditional POSIX calls (fork, posix_spawn,
pthread_getname_np) by moving them into #ifndef WIN32 or HAVE_PTHREAD_*
guards. When pthreads is available on Windows (detected via cmake
HAVE_PTHREAD_* checks), still use it for thread name reporting since it
provides useful information at low cost.

Also add Threads::Threads as an explicit dependency of the multiprocess
library. proxy.cpp directly uses thread_local, std::this_thread, and
std::thread, and the dependency was previously satisfied only through
transitive linkage from CapnProto::kj-async.

This change was written with Claude Sonnet 4.6.
Add TCP (sockaddr_in) support to SocketListener and platform-conditional
header guards so the class compiles and works on Windows:

- Wrap POSIX headers (arpa/inet.h, netinet/in.h, sys/socket.h, sys/un.h)
  in #else of #ifdef WIN32, with Windows equivalents (afunix.h,
  ws2tcpip.h) in the #ifdef WIN32 branch.
- Add a WsaInit static initializer on Windows that calls WSAStartup
  before any socket operations. Tests create sockets directly (not via
  StartSpawned), so Winsock must be initialized here.
- Add Init(sockaddr_in&) and Connect(const sockaddr_in&) overloads for
  TCP loopback connections.
- Use TCP on Windows to work around Wine's lack of the AcceptEx extension
  required by KJ's AF_UNIX listener; Unix platforms continue using
  AF_UNIX.
- Expand std::variant<sockaddr_un> to std::variant<sockaddr_in, sockaddr_un>.

This change was written with Claude Sonnet 4.6.
- shell.nix: add `windows` parameter that selects pkgs.pkgsCross.mingwW64
  as the cross target; also change crossPkgs default from import<nixpkgs>{}
  to null (cleaner API). When windows=true, add native pkgs.capnproto to
  nativeBuildInputs so capnp/capnpc-c++ are in PATH for cmake code generation,
  and add wine64Packages.staging so ctest can run mptest.exe via wine.
  Change llvmBase to always use pkgs (native) instead of crossPkgs.

- ci/configs/windows.bash: new config that cross-compiles with mingw,
  sets CMAKE_SYSTEM_NAME=Windows, CMAKE_FIND_ROOT_PATH_MODE_PROGRAM=NEVER
  (so cmake finds native capnp from PATH), CMAKE_CROSSCOMPILING_EMULATOR=wine
  (so ctest runs mptest.exe via wine), and sets MPGEN_PRE_BUILD=1.

- ci/scripts/ci.sh: add MPGEN_PRE_BUILD support: when set, build native mpgen
  in $CI_DIR-native before the main cross build, then inject
  -DMPGEN_EXECUTABLE into CMAKE_ARGS. This is needed because cmake's
  add_custom_command does not use CMAKE_CROSSCOMPILING_EMULATOR, so the
  cross-compiled mpgen.exe cannot be used as a code generator directly.

This change was written with Claude Sonnet 4.6.
The proxy-server code generated for test/mp/test/foo.capnp now has more
sections than MSVC's default object file format allows, failing the
Windows native VS CI job with "error C1128: number of sections exceeded
object file format limit: compile with /bigobj"
(https://github.com/bitcoin/bitcoin/actions/runs/36945085020/job/110645241551).
Bitcoin Core already passes /bigobj for its own debug builds.

Not reproduced locally; based on the error message.

This change was written with Claude Opus 5.5 (1M context).
@Sjors

Sjors commented Oct 20, 2026

Copy link
Copy Markdown
Member

Can you give this a rebase? I'll push an update to #272 after that.

@ryanofsky

Copy link
Copy Markdown
Collaborator Author

Rebased de40396 -> a8e8c55 (pr/win.22 -> pr/win.23, compare) due to conflicts with #370, which merged the SocketError inline change, so the separate commit for it is dropped. Also adds a commit compiling mptest with /bigobj to fix the MSVC build (error C1128: number of sections exceeded object file format limit) (Windows native VS job on bitcoin/bitcoin#32387)

@DrahtBot

DrahtBot commented Nov 3, 2026

Copy link
Copy Markdown

🐙 This pull request conflicts with the target branch and needs rebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants