Repository navigation
Conversation
79e0653 to
8f0ba30
Compare
| // capnp interface does not define a destroy method, this will just call | ||
| // an empty stub defined in the ProxyClientBase class and do nothing. | ||
| Sub::destroy(*this); | ||
| if (m_context.connection && !m_context.connection->m_disconnected.load()) { |
There was a problem hiding this comment.
In commit "ipc: skip proxy destroy calls after disconnect starts" (fa010b9)
FWIW, there's already a PR on upstream that fixes this issue: bitcoin-core/libmultiprocess#273
There was a problem hiding this comment.
Thanks for pointing that out @xyzconstant. I was not aware that there was already an upstream libmultiprocess PR addressing this.
I have reworked the branch to remove the local libmultiprocess change and replace it with a proper subtree update based on bitcoin-core/libmultiprocess#273. The sv2-tp reconnect commits are now rebased on top of that.
I reran the regtest stack on the updated branch and the reconnect path is still working as expected.
8f0ba30 to
451b144
Compare
|
I'm marking this draft until bitcoin-core/libmultiprocess#273 lands. I also plan to review this. |
|
FYI: bitcoin-core/libmultiprocess#273 just went through |
62f92cd to
41cbc55
Compare
Thanks for the heads up @xyzconstant. @Sjors, I have updated the subtree to use the latest changes. I think this should be ready for review now. |
|
The test each commit CI job is taking unexpectedly long. My agent thinks a9f45af is hanging in |
decadbd to
99a1bed
Compare
Thanks for giving this a look @Sjors. It seems, in a9f45af, resetting |
|
I did the subtree update in #109 to reduce the scope here. Still reviewing the changes, but you'll need a trivial rebase. |
There was a problem hiding this comment.
When I shut the node down gracefully (ctrl + c) I get a flood of waitNext() calls. Probably because the node cancels them, but then sv2-tp tries another one immediately. We need to back-off mechanism if waitNext returns null.
I left inline suggestions for some other issues I ran into while testing (on mainnet, since it has more mempool churn, and using the sv2-apps pool role).
99a1bed to
2b7a8dd
Compare
|
Thanks for the review @Sjors
I avoid plain sleep here because that would delay shutdown. |
|
Restored some comments that were mistakenly deleted (LLM mistakes). I also noticed that the commits were a bit messy. So I restructured them to be easier to review |
3ee7cfe to
f26dbaf
Compare
e0471d7 to
37db9a8
Compare
|
Rebased this PR onto the current master to include the test coverage from #112, and updated the Also split the original first commit into three commits. It was not atomic and combined several changes, so the new commit structure should make the PR easier to review. Thanks |
|
Thanks, looks like one CI job failed. I haven't checked if it's related or a pre-existing issue. Update: I added a commit to #113 that hopefully fixes it. |
WITH_LOCK uses decltype(auto), so prefix increment returns a reference
that is read after the mutex is released. Copy the new template id
before leaving the critical section to avoid the data race.
The code reported at template_provider.cpp:465 was:
uint64_t template_id{
WITH_LOCK(m_tp_mutex, return ++m_template_id;)
};
The TSan job in stratum-mining#101 reported:
WARNING: ThreadSanitizer: data race
Write of size 8 ... by thread T39
(mutexes: write M0):
#0 ... template_provider.cpp:465:38
Previous read of size 8 ... by thread T41:
#0 ... template_provider.cpp:465:38
Co-authored-by: OpenAI Codex 5.6 Sol <noreply@openai.com>
WITH_LOCK uses decltype(auto), so prefix increment returns a reference
that is read after the mutex is released. Copy the new template id
before leaving the critical section to avoid the data race.
The code reported at template_provider.cpp:465 was:
uint64_t template_id{
WITH_LOCK(m_tp_mutex, return ++m_template_id;)
};
The TSan job in stratum-mining#101 reported:
WARNING: ThreadSanitizer: data race
Write of size 8 ... by thread T39
(mutexes: write M0):
#0 ... template_provider.cpp:465:38
Previous read of size 8 ... by thread T41:
#0 ... template_provider.cpp:465:38
Co-authored-by: OpenAI Codex 5.6 Sol <noreply@openai.com>
37db9a8 to
f457c6d
Compare
|
#113 landed, so you'll need a small rebase. |
f457c6d to
133e5a6
Compare
Done. Rebased this PR onto the current master. Thanks |
|
Can you split 52360f7 or better document how it can be reviewed? |
The Mining proxy depends on the Init connection that created it. Move both objects into a BackendSession owned by the template provider, and let client handlers retain that session while using its proxies. This establishes the lifetime boundary needed for later backend replacement without changing disconnect or reconnection behavior.
Construct the template provider independently from its Bitcoin Core backend and publish BackendSession replacements under a dedicated mutex. Take shared session references before using IPC proxies, and interrupt or destroy those proxies outside the template-cache lock. This commit only establishes synchronized replacement and shutdown. Runtime IPC failure recovery is added separately.
Mark a backend generation disconnected when one of its IPC calls fails. Clear templates and proxies belonging to that generation only when it is still active, so a late failure cannot invalidate a newer backend. Copy cached template proxies before interrupting them to avoid making IPC calls while holding the template mutex. Propagate test IPC errors as ipc::Exception so this path is exercised like production.
Wake client handlers when a backend is installed and make them wait whenever their current generation is disconnected. Re-run the IBD check for each replacement generation before creating templates. Shutdown wakes the same waiters, allowing all handler threads to exit without requiring a backend.
ForEachClient only guarantees the Sv2Client reference during its callback. Copy the id into the new thread instead of retaining that reference after the callback returns.
Catch ipc failures in template operations and discard state tied to the disconnected backend. Existing client handlers can then wait for the replacement backend and regenerate work without dropping their sv2 connections.
Losing Bitcoin Core does not invalidate the pool-facing SV2 connection. Restarting the whole template provider would unnecessarily force pools to reconnect and repeat the handshake. Keep the listener and client handlers alive while reconnecting the IPC backend, then resume template creation on the existing sessions.
Keep the template provider main loop polling the Bitcoin Core IPC backend even when no Stratum v2 client handler is making template IPC calls. Use the existing IBD check before startup and a periodic getTip liveness check afterwards so backend shutdown is detected and the reconnect loop can run.
An IPC backend shutting down can make waitNext() return empty immediately. Retry with bounded exponential backoff to avoid a hot loop while preserving prompt interrupt handling. Extend the mock backend to exercise empty waits and expose shorter retry delays in tests.
Track waitNext() state in the template cache instead of Sv2Client. Connman now only signals the template provider after disconnects or coinbase constraint updates, and the provider interrupts cached templates that actually have a wait in progress.
Do not send a new same-tip template unless fee updates are active and the total fee delta reaches the configured threshold. New tips still propagate immediately.
133e5a6 to
4e0e99a
Compare
I agree that 52360f7 was doing too much, so I split it into the following review order:
This leaves the PR with 11 commits, but each commit now has a focused purpose, which should make the changes easier to review. |
|
I think this is getting too complicated, and 700 lines of code is a lot. Maybe instead of trying to recover from a disconnect, we can simply shut down? A well configured (systemd) backend should stop the SRI client first, then stop sv2-tp and then the node. If the system isn't configured that way, it seems fine to just bail out if the node is stopped from under us. |
I agree that this is getting a bit complicated, and shutting does would be a simpler approach to solving this problem. Will be opening a PR to shut down cleanly when the node is stopped. Thanks for all the reviews so far |
|
Opened 115 for a simpler approach that simply stops sv2-tp when the node is disconnected. |
|
Thanks, closing this PR in favor of #115. |
This PR makes sv2-tp recover from a lost IPC connection without exiting.
On top of that, sv2-tp keeps the template provider and pool-facing connection manager alive across backend loss, reconnects to Bitcoin Core with backoff, installs a fresh IPC backend, and resumes serving templates on the existing pool connection.
IPC objects are tied to backend generations so stale objects can be cleaned up safely without invalidating a replacement backend. IPC exceptions trigger reconnection instead of terminating client handlers