Skip to content

sv2-tp: reconnect to bitcoin core IPC after disconnect - #101

Closed
enochaz wants to merge 11 commits into
stratum-mining:masterfrom
enochaz:reconnect-subtree-clean-series
Closed

enochaz wants to merge 11 commits into
stratum-mining:masterfrom
enochaz:reconnect-subtree-clean-series

Conversation

@enochaz

@enochaz enochaz commented May 4, 2026 •

Copy link
Copy Markdown
Contributor

This PR makes sv2-tp recover from a lost IPC connection without exiting.

On top of that, sv2-tp keeps the template provider and pool-facing connection manager alive across backend loss, reconnects to Bitcoin Core with backoff, installs a fresh IPC backend, and resumes serving templates on the existing pool connection.

IPC objects are tied to backend generations so stale objects can be cleaned up safely without invalidating a replacement backend. IPC exceptions trigger reconnection instead of terminating client handlers

@enochaz enochaz changed the title sv2-tp: reconnect to Bitcoin Core IPC after disconnect sv2-tp: reconnect to bitcoin core IPC after disconnect May 4, 2026
@enochaz
enochaz force-pushed the reconnect-subtree-clean-series branch 5 times, most recently from 79e0653 to 8f0ba30 Compare May 6, 2026 09:57
// capnp interface does not define a destroy method, this will just call
// an empty stub defined in the ProxyClientBase class and do nothing.
Sub::destroy(*this);
if (m_context.connection && !m_context.connection->m_disconnected.load()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In commit "ipc: skip proxy destroy calls after disconnect starts" (fa010b9)

FWIW, there's already a PR on upstream that fixes this issue: bitcoin-core/libmultiprocess#273

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing that out @xyzconstant. I was not aware that there was already an upstream libmultiprocess PR addressing this.

I have reworked the branch to remove the local libmultiprocess change and replace it with a proper subtree update based on bitcoin-core/libmultiprocess#273. The sv2-tp reconnect commits are now rebased on top of that.

I reran the regtest stack on the updated branch and the reconnect path is still working as expected.

@enochaz
enochaz force-pushed the reconnect-subtree-clean-series branch from 8f0ba30 to 451b144 Compare May 8, 2026 14:31
@Sjors

Sjors commented May 11, 2026 •

Copy link
Copy Markdown
Collaborator

I'm marking this draft until bitcoin-core/libmultiprocess#273 lands.

I also plan to review this.

@Sjors
Sjors marked this pull request as draft May 11, 2026 09:14
@xyzconstant

Copy link
Copy Markdown
Contributor

FYI: bitcoin-core/libmultiprocess#273 just went through

@enochaz
enochaz force-pushed the reconnect-subtree-clean-series branch 3 times, most recently from 62f92cd to 41cbc55 Compare June 3, 2026 09:07
@enochaz

enochaz commented Jun 3, 2026

Copy link
Copy Markdown
Contributor Author

FYI: bitcoin-core/libmultiprocess#273 just went through

Thanks for the heads up @xyzconstant.

@Sjors, I have updated the subtree to use the latest changes. I think this should be ready for review now.

@Sjors
Sjors marked this pull request as ready for review June 6, 2026 07:24
@Sjors

Sjors commented Jun 6, 2026

Copy link
Copy Markdown
Collaborator

The test each commit CI job is taking unexpectedly long. My agent thinks a9f45af is hanging in sv2_tester_lifecycle_tests and sv2_template_provider_tests.

@enochaz
enochaz force-pushed the reconnect-subtree-clean-series branch 2 times, most recently from decadbd to 99a1bed Compare June 8, 2026 13:30
@enochaz

enochaz commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

The test each commit CI job is taking unexpectedly long. My agent thinks a9f45af is hanging in sv2_tester_lifecycle_tests and sv2_template_provider_tests.

Thanks for giving this a look @Sjors. It seems, in a9f45af, resetting m_backend before calling Interrupt() made InterruptBackend() a no-op, leaving client threads hung in waitNext() during teardown.

@Sjors

Sjors commented Jun 17, 2026 •

Copy link
Copy Markdown
Collaborator

I did the subtree update in #109 to reduce the scope here. Still reviewing the changes, but you'll need a trivial rebase.

@Sjors Sjors left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I shut the node down gracefully (ctrl + c) I get a flood of waitNext() calls. Probably because the node cancels them, but then sv2-tp tries another one immediately. We need to back-off mechanism if waitNext returns null.

I left inline suggestions for some other issues I ran into while testing (on mainnet, since it has more mempool churn, and using the sv2-apps pool role).

Comment thread src/sv2-tp.cpp Resolved
Comment thread src/sv2/template_provider.h Resolved
Comment thread src/sv2/template_provider.cpp Resolved
Comment thread src/sv2/template_provider.cpp Resolved
@enochaz
enochaz force-pushed the reconnect-subtree-clean-series branch from 99a1bed to 2b7a8dd Compare June 22, 2026 13:17
@enochaz

enochaz commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review @Sjors

When I shut the node down gracefully (ctrl + c) I get a flood of waitNext() calls. Probably because the node cancels them, but then sv2-tp tries another one immediately. We need to back-off mechanism if waitNext returns null.

waitNext() returning nullptr during backend shutdown was causing the client loop to immediately call back into the same dying backend. To prevent this i added an interruptible exponential backoff in that path.

I avoid plain sleep here because that would delay shutdown.

@enochaz

enochaz commented Jun 22, 2026 •

Copy link
Copy Markdown
Contributor Author

Restored some comments that were mistakenly deleted (LLM mistakes).

I also noticed that the commits were a bit messy. So I restructured them to be easier to review

@enochaz
enochaz force-pushed the reconnect-subtree-clean-series branch 2 times, most recently from 3ee7cfe to f26dbaf Compare June 22, 2026 14:50
@enochaz
enochaz force-pushed the reconnect-subtree-clean-series branch from e0471d7 to 37db9a8 Compare July 20, 2026 15:52
@enochaz

enochaz commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

Rebased this PR onto the current master to include the test coverage from #112, and updated the resume_after_backend_reconnect test to exercise two clients waiting concurrently.

Also split the original first commit into three commits. It was not atomic and combined several changes, so the new commit structure should make the PR easier to review. Thanks

@Sjors

Sjors commented Jul 21, 2026 •

Copy link
Copy Markdown
Collaborator

Thanks, looks like one CI job failed. I haven't checked if it's related or a pre-existing issue.

Update: I added a commit to #113 that hopefully fixes it.

Sjors added a commit to Sjors/sv2-tp that referenced this pull request Jul 21, 2026
WITH_LOCK uses decltype(auto), so prefix increment returns a reference
that is read after the mutex is released. Copy the new template id
before leaving the critical section to avoid the data race.

The code reported at template_provider.cpp:465 was:

    uint64_t template_id{
        WITH_LOCK(m_tp_mutex, return ++m_template_id;)
    };

The TSan job in stratum-mining#101 reported:

    WARNING: ThreadSanitizer: data race
      Write of size 8 ... by thread T39
      (mutexes: write M0):
        #0 ... template_provider.cpp:465:38
      Previous read of size 8 ... by thread T41:
        #0 ... template_provider.cpp:465:38

Co-authored-by: OpenAI Codex 5.6 Sol <noreply@openai.com>
Sjors added a commit to Sjors/sv2-tp that referenced this pull request Jul 21, 2026
WITH_LOCK uses decltype(auto), so prefix increment returns a reference
that is read after the mutex is released. Copy the new template id
before leaving the critical section to avoid the data race.

The code reported at template_provider.cpp:465 was:

    uint64_t template_id{
        WITH_LOCK(m_tp_mutex, return ++m_template_id;)
    };

The TSan job in stratum-mining#101 reported:

    WARNING: ThreadSanitizer: data race
      Write of size 8 ... by thread T39
      (mutexes: write M0):
        #0 ... template_provider.cpp:465:38
      Previous read of size 8 ... by thread T41:
        #0 ... template_provider.cpp:465:38

Co-authored-by: OpenAI Codex 5.6 Sol <noreply@openai.com>
@enochaz
enochaz force-pushed the reconnect-subtree-clean-series branch from 37db9a8 to f457c6d Compare July 21, 2026 09:54
@Sjors

Sjors commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

#113 landed, so you'll need a small rebase.

@enochaz
enochaz force-pushed the reconnect-subtree-clean-series branch from f457c6d to 133e5a6 Compare July 21, 2026 11:26
@enochaz

enochaz commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

#113 landed, so you'll need a small rebase.

Done. Rebased this PR onto the current master. Thanks

@Sjors

Sjors commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Can you split 52360f7 or better document how it can be reviewed?

enochaz added 11 commits July 25, 2026 07:39
The Mining proxy depends on the Init connection that created it. Move both objects into a
BackendSession owned by the template provider, and let client handlers retain that session
while using its proxies.

This establishes the lifetime boundary needed for later backend replacement without changing
disconnect or reconnection behavior.
Construct the template provider independently from its Bitcoin Core backend and publish BackendSession
replacements under a dedicated mutex. Take shared session references before using IPC proxies, and
interrupt or destroy those proxies outside the template-cache lock.

This commit only establishes synchronized replacement and shutdown. Runtime IPC failure recovery
is added separately.
Mark a backend generation disconnected when one of its IPC calls fails. Clear templates and proxies
belonging to that generation only when it is still active, so a late failure cannot invalidate a
newer backend.

Copy cached template proxies before interrupting them to avoid making IPC calls while holding the
template mutex. Propagate test IPC errors as ipc::Exception so this path is exercised like production.
Wake client handlers when a backend is installed and make them wait whenever their current generation
is disconnected. Re-run the IBD check for each replacement generation before creating templates.

Shutdown wakes the same waiters, allowing all handler threads to exit without requiring a backend.
ForEachClient only guarantees the Sv2Client reference during its callback. Copy the id into
the new thread instead of retaining that reference after the callback returns.
Catch ipc failures in template operations and discard state tied to the disconnected backend.
Existing client handlers can then wait for the replacement backend and regenerate work without
dropping their sv2 connections.
Losing Bitcoin Core does not invalidate the pool-facing SV2 connection.
Restarting the whole template provider would unnecessarily force pools
to reconnect and repeat the handshake.

Keep the listener and client handlers alive while reconnecting the IPC
backend, then resume template creation on the existing sessions.
Keep the template provider main loop polling the Bitcoin Core IPC backend even when no Stratum v2 client handler is making template IPC calls.

Use the existing IBD check before startup and a periodic getTip liveness check afterwards so backend shutdown is detected and the reconnect loop can run.
An IPC backend shutting down can make waitNext() return empty immediately. Retry with bounded exponential backoff to avoid a hot loop while preserving prompt interrupt handling.

Extend the mock backend to exercise empty waits and expose shorter retry delays in tests.
Track waitNext() state in the template cache instead of Sv2Client. Connman now
only signals the template provider after disconnects or coinbase constraint updates,
and the provider interrupts cached templates that actually have a wait in progress.
Do not send a new same-tip template unless fee updates are active and the total
fee delta reaches the configured threshold. New tips still propagate immediately.
@enochaz
enochaz force-pushed the reconnect-subtree-clean-series branch from 133e5a6 to 4e0e99a Compare July 25, 2026 06:42
@enochaz

enochaz commented Jul 25, 2026

Copy link
Copy Markdown
Contributor Author

Can you split 52360f7 or better document how it can be reviewed?

I agree that 52360f7 was doing too much, so I split it into the following review order:

  • own the Bitcoin Core ipc backend
  • synchronize ipc backend replacement
  • invalidate failed ipc backends
  • wait for ipc backend replacement
  • capture the client id safely in the handler thread

This leaves the PR with 11 commits, but each commit now has a focused purpose, which should make the changes easier to review.

@Sjors

Sjors commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

I think this is getting too complicated, and 700 lines of code is a lot.

Maybe instead of trying to recover from a disconnect, we can simply shut down?

A well configured (systemd) backend should stop the SRI client first, then stop sv2-tp and then the node. If the system isn't configured that way, it seems fine to just bail out if the node is stopped from under us.

@enochaz

enochaz commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

I think this is getting too complicated, and 700 lines of code is a lot.

Maybe instead of trying to recover from a disconnect, we can simply shut down?

A well configured (systemd) backend should stop the SRI client first, then stop sv2-tp and then the node. If the system isn't configured that way, it seems fine to just bail out if the node is stopped from under us.

I agree that this is getting a bit complicated, and shutting does would be a simpler approach to solving this problem.

Will be opening a PR to shut down cleanly when the node is stopped. Thanks for all the reviews so far

@enochaz

enochaz commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Opened 115 for a simpler approach that simply stops sv2-tp when the node is disconnected.

@Sjors

Sjors commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Thanks, closing this PR in favor of #115.

@Sjors Sjors closed this Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants