Skip to content

fix: races in watch-connection seed, partial sig-share batches, and ChainLock shutdown - #8117

Merged
PastaPastaPasta merged 3 commits into
dashpay:developfrom
PastaPastaPasta:fix/llmq-watch-batch-shutdown
Nov 13, 2026
Merged

PastaPastaPasta merged 3 commits into
dashpay:developfrom
PastaPastaPasta:fix/llmq-watch-batch-shutdown

Conversation

@PastaPastaPasta

@PastaPastaPasta PastaPastaPasta commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Issue being fixed or feature implemented

Concurrent first calls can regenerate the watch-connection seed while another caller reads it. A missing quorum during signature-share collection leaves a partial batch that the caller dispatches to code using quorums.at(). ChainLock shutdown also returns before its scheduler thread exits. The thread was only joined when the node context was destroyed, so a task already running could continue after mn_sync and the mempool were destroyed, or block shutdown in ActivateBestChain waiting on the validation queue of the already stopped main scheduler.

What was done?

  • Initialize the immutable watch-connection seed with thread-safe function-local static initialization.
  • Clear both batch outputs when quorum lookup fails, so the caller cannot dispatch an incomplete batch.
  • Join the ChainLock scheduler thread in Stop() and reuse that method from the destructor, guarding repeated joins with joinable().

Each fix is its own commit.

Reviewed every handler stop/destruction path: production shutdown joins before stopping the main scheduler, with no chain or handler lock held; the full unit fixture calls Stop() before later destruction; reindex resets the handler without those locks while the main scheduler runs. The simpler base fixture leaves the handler dormant. No stop/destructor runs on the handler's own scheduler thread.

How Has This Been Tested?

  • Debug build with warnings treated as errors: make -C src -j4 dashd test/test_dash.
  • ./src/test/test_dash --run_test='llmq*' --log_level=message: all 84 cases passed.
  • ./src/test/test_dash --run_test='validation_chainstatemanager_tests/chainstatemanager,chainstatemanager_snapshot_discarded_on_reindex' --log_level=message: both cases passed, covering base-fixture teardown and reindex destruction/recreation.
  • RAM-disk functional runner with an isolated port range and -j1 --timeout-factor=3: feature_llmq_chainlocks.py, feature_llmq_signing.py, and its --spork21 variant all passed.
  • Whitespace lint, git diff --check, and self-review passed.

The seed and partial-batch failure paths were verified by code inspection. No TSAN run or deterministic forced-race reproducer is claimed.

Breaking Changes

None.

Checklist:

  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation (not needed for these internal fixes)
  • I have assigned this pull request to a milestone

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The changes update chainlock shutdown so the destructor calls Stop(), which joins the scheduler thread when joinable. The signing-share collection function clears its output maps when a required quorum is missing. Watch-connection seed initialization now uses a function-local static initialized by GetRandHash().

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: Minimal · up to 97076

This change fixes three race and shutdown issues without introducing a new concrete regression that was found in review. It looks ready to merge after normal checks.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names three fixes: a race in the watch-connection seed, partial signature-share batches, and ChainLock shutdown. The changeset matches all three: the seed initialization in utils.cpp, the ou…
Description check Passed The description states the three problems and the matching fixes. It names the affected paths (seed initialization, quorum-lookup failure in signature-share collection, and the ChainLock scheduler joi…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names three fixes: a race in the watch-connection seed, partial signature-share batches, and ChainLock shutdown. The changeset matches all three: the seed initialization in utils.cpp, the output clearing in signing_shares.cpp, and the Stop()/destructor change in handler.cpp. The title is a long list of three items, so it is broad, but it describes real changes and is not off-topic.

Full details: Description check

Explanation

The description states the three problems and the matching fixes. It names the affected paths (seed initialization, quorum-lookup failure in signature-share collection, and the ChainLock scheduler join in Stop()). It also lists the testing performed. This relates directly to the changeset.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 9e45c12d-628a-47ff-b64a-bbffd0ba8941
Commits

Reviewing files that changed from the base of the PR and between 5c34dcf and 9707616.

Files selected for processing: 3

  • src/chainlock/handler.cpp
  • src/llmq/signing_shares.cpp
  • src/llmq/utils.cpp

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

@PastaPastaPasta PastaPastaPasta added this to the 24.1 milestone Nov 11, 2026
@PastaPastaPasta PastaPastaPasta changed the title fix(llmq): races in watch-connection seed, partial sig-share batches, and ChainLock shutdown fix: races in watch-connection seed, partial sig-share batches, and ChainLock shutdown Nov 11, 2026
@thepastaclaw

thepastaclaw commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

♻️ No change to this PR's diff since commit a3933fa (an empty commit, a merge of the base or a rebase onto it): that review stands — approved; @thepastaclaw review asks for a review of this commit (commit 9707616)

@PastaPastaPasta
PastaPastaPasta force-pushed the fix/llmq-watch-batch-shutdown branch from e6e52d1 to a3933fa Compare November 11, 2026 18:18

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final validation — Phase 2 only (queue backlog)

The reviewed changes correctly address the three stated concurrency and shutdown defects, and no blocking correctness issue is evident. Add deterministic regression coverage for the missing-quorum batch contract; the relevant build, lint, and sanitizer checks were still queued in the supplied CI snapshot.

🟡 1 suggestion(s)

Approved: nothing here blocks the merge. The non-blocking findings are optional: fix them forward, in this PR or a follow-up, without waiting for another review round.

Review provenance

Source: reviewer 1: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: dash-core-commit-history); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 4: gpt-6.1-sol (agent: phase2-reviewer, role: dash-core-commit-history); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: normal by gpt-6.1-sol (effort low) — Although small, the diff changes concurrent initialization, batch failure handling, and shutdown thread joining, requiring reasoning about caller invariants and lifecycle ordering rather than straightforward local verification.
  • Phase 1 reviewers: not run (skipped for throughput: 12 PRs queued, above the 10 limit)
  • Second Phase-2 pass: on this first full review every Phase-2 reviewer also ran a second, independent time, and the verifier weighed both sets
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — dash-core-commit-history (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — general (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — dash-core-commit-history (completed, effort high); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review (PastaClaw). How to work through them:
- Verify each finding against the code; fix only what is real, as narrowly as the finding.
- If a finding is wrong, decline it on its thread in one or two sentences; the bot answers once and stops.
- To defer one, link a tracking issue on its thread (or reply asking the bot to file one).
- Suggestions don't block the merge, but most are worth fixing in this PR; judge each on its merits.
- Keep the PR's scope; don't grow it to satisfy hardening suggestions.
- Carried items were raised in an earlier round: read the thread first, and don't re-fix what you already fixed, declined or deferred there. Disputed and deferred items are already handled and are not listed.

In `src/llmq/signing_shares.cpp`:
- [SUGGESTION] src/llmq/signing_shares.cpp:599-600: Add deterministic regression coverage for incomplete batch suppression
  The fix correctly clears both output maps when quorum lookup fails, which prevents NetSigning::WorkThreadDispatcher from dispatching an incomplete batch. However, no test exercises this contract. Add a focused regression that collects a batch containing a quorum unavailable to GetQuorum, verifies that both output maps are empty, and confirms that a later valid batch can still be collected. This is important because the caller uses the boolean return only as a more-work indicator and dispatches whenever the shares map is nonempty; returning false alone would not prevent the original quorums.at() failure.

Comment thread src/llmq/signing_shares.cpp Resolved
@thepastaclaw thepastaclaw added the pastaclaw:approved thepastaclaw's latest review approved this PR label Nov 11, 2026
PastaPastaPasta and others added 3 commits November 12, 2026 23:48
… static

Two first callers of CalcDeterministicWatchConnections could both see the
generated flag unset, and one could regenerate the seed while the other
was already reading it.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
CollectPendingSigSharesToVerify returned with the shares collected so far
but without their quorums. The caller dispatches whenever the share map is
non-empty, so verification then hit quorums.at() for a missing key.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Stop() only asked the scheduler to stop, and the thread was joined in the
destructor at NodeContext destruction. A task already running could keep
going through the rest of shutdown, after mn_sync and the mempool were
destroyed, or block in ActivateBestChain on the validation queue of the
already stopped main scheduler.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@PastaPastaPasta
PastaPastaPasta force-pushed the fix/llmq-watch-batch-shutdown branch from a3933fa to 9707616 Compare November 13, 2026 05:49
@thepastaclaw thepastaclaw added pastaclaw:approved thepastaclaw's latest review approved this PR and removed pastaclaw:approved thepastaclaw's latest review approved this PR labels Nov 13, 2026

@PastaPastaPasta PastaPastaPasta left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approevd

@PastaPastaPasta
PastaPastaPasta merged commit 92ed025 into dashpay:develop Nov 13, 2026
55 checks passed
@thepastaclaw thepastaclaw removed the pastaclaw:approved thepastaclaw's latest review approved this PR label Nov 13, 2026
PastaPastaPasta added a commit that referenced this pull request Nov 14, 2026
…atches, and ChainLock shutdown

9707616 fix(chainlock): join the scheduler thread in ChainlockHandler::Stop() (pasta)
547392c fix(llmq): drop the whole sig-share batch when a quorum lookup fails (pasta)
5e524df fix(llmq): initialize the watch-connection seed with a function-local static (pasta)

Pull request description:

  ## Issue being fixed or feature implemented

  Concurrent first calls can regenerate the watch-connection seed while another caller reads it. A missing quorum during signature-share collection leaves a partial batch that the caller dispatches to code using `quorums.at()`. ChainLock shutdown also returns before its scheduler thread exits. The thread was only joined when the node context was destroyed, so a task already running could continue after `mn_sync` and the mempool were destroyed, or block shutdown in `ActivateBestChain` waiting on the validation queue of the already stopped main scheduler.

  ## What was done?

  - Initialize the immutable watch-connection seed with thread-safe function-local static initialization.
  - Clear both batch outputs when quorum lookup fails, so the caller cannot dispatch an incomplete batch.
  - Join the ChainLock scheduler thread in `Stop()` and reuse that method from the destructor, guarding repeated joins with `joinable()`.

  Each fix is its own commit.

  Reviewed every handler stop/destruction path: production shutdown joins before stopping the main scheduler, with no chain or handler lock held; the full unit fixture calls `Stop()` before later destruction; reindex resets the handler without those locks while the main scheduler runs. The simpler base fixture leaves the handler dormant. No stop/destructor runs on the handler's own scheduler thread.

  ## How Has This Been Tested?

  - Debug build with warnings treated as errors: `make -C src -j4 dashd test/test_dash`.
  - `./src/test/test_dash --run_test='llmq*' --log_level=message`: all 84 cases passed.
  - `./src/test/test_dash --run_test='validation_chainstatemanager_tests/chainstatemanager,chainstatemanager_snapshot_discarded_on_reindex' --log_level=message`: both cases passed, covering base-fixture teardown and reindex destruction/recreation.
  - RAM-disk functional runner with an isolated port range and `-j1 --timeout-factor=3`: `feature_llmq_chainlocks.py`, `feature_llmq_signing.py`, and its `--spork21` variant all passed.
  - Whitespace lint, `git diff --check`, and self-review passed.

  The seed and partial-batch failure paths were verified by code inspection. No TSAN run or deterministic forced-race reproducer is claimed.

  ## Breaking Changes

  None.

  ## Checklist:
  - [x] I have performed a self-review of my own code
  - [ ] I have made corresponding changes to the documentation (not needed for these internal fixes)
  - [x] I have assigned this pull request to a milestone

  🤖 Generated with [Claude Code](https://claude.com/claude-code)

Top commit has no ACKs.

Tree-SHA512: 0d518624c79f2cdee35b2daa341a5905de3e0a27b63105ca06941baa6e620f53ba1f26a0eebe0c843a5118c4615f680b0da7e4f5798eb22c1aefadd717c02baa
(cherry picked from commit 92ed025)
PastaPastaPasta added a commit that referenced this pull request Nov 14, 2026
b534385 chore: prepare v24.0.2 release (pasta)
70e06ae Merge #8117: fix: races in watch-connection seed, partial sig-share batches, and ChainLock shutdown (merge-script)
d50ea34 Merge #8115: fix: free pooled secure allocations with their allocated size (merge-script)
824e58a Merge #8113: fix(wallet): schedule CoinJoin auto-denomination per wallet and guard an empty round list (merge-script)
068529f Merge #8112: fix: answer govsync only to the requesting peer and index replayed orphan votes (merge-script)
484ec63 Merge #8119: fix(stats): wrong inputValue gauge, counter data race, and null-tip crash (merge-script)
28d5711 Merge #8120: fix(utils): accept -printcrashinfo in dash-cli and dash-tx (merge-script)
578b914 Merge bitcoin-core/gui#770: Revert "gui: provide wallet controller context to wallet actions" (Konstantin Akimov)
d6902fa Merge bitcoin-core/gui#765: Fix wallet list hover crash on shutdown (Konstantin Akimov)

Pull request description:

  ## Issue being fixed or feature implemented

  Backports and release prep for v24.0.2, on top of `v24.0.x` at 137d999 (v24.0.1). One PR carries both, so the release branch gets a single merge.

  Two fixes are the reason for the release:

  - **Memory leak (#8115).** Since v24.0.0 a node's locked-memory use grows for as long as it runs. On `v24.0.x` BLS still uses relic, where a G2 element is 296 bytes; the pooled secure allocator frees such an allocation with one chunk fewer than it allocated. The batch-verification weighting added in v24.0.0 makes one such allocation per verified message, so the pool keeps growing until restart. On develop (blst, 288-byte elements) the same code path does not leak, which is why #8115 describes the bug as latent.
  - **macOS crash on close (#8073's bitcoin-core/gui#765 and gui#770).** On macOS the menu bar stays usable after the main window is hidden for shutdown. `File > Open Wallet`, `Close Wallet` and the other wallet items then call the wallet controller after it has been deleted.

  The rest are small fixes that were already merged on develop and are low risk to carry.

  ## What was done?

  ### Picked

  In develop merge order. The Dash PRs are picked from their develop merge commits with `git cherry-pick -m1 -x`; the two GUI backports are picked with `-x` from the commits #8073 merged, like earlier partial picks on this branch.

  - bitcoin-core/gui#765 Fix wallet list hover crash on shutdown, and bitcoin-core/gui#770, which reverts its first commit (from #8073). Together they leave one change: `appMenuBar->clear()` when shutdown starts.
  - #8120 fix(utils): accept -printcrashinfo in dash-cli and dash-tx
  - #8119 fix(stats): wrong inputValue gauge, counter data race, and null-tip crash
  - #8112 fix: answer govsync only to the requesting peer and index replayed orphan votes
  - #8113 fix(wallet): schedule CoinJoin auto-denomination per wallet and guard an empty round list
  - #8115 fix: free pooled secure allocations with their allocated size
  - #8117 fix: races in watch-connection seed, partial sig-share batches, and ChainLock shutdown

  Then one commit, `chore: prepare v24.0.2 release`:

  - `configure.ac`: `_CLIENT_VERSION_BUILD` 1 → 2.
  - `doc/release-notes.md`: v24.0.2 notes. The v24.0.1 notes are archived verbatim to `doc/release-notes/dash/release-notes-24.0.1.md` and added to "Older releases".
  - `contrib/flatpak/org.dash.dash-core.metainfo.xml`: `24.0.2` entry dated 2026-11-13. The date needs changing if the tag is cut on another day.
  - `doc/man/*.1`: version string only. `-printcrashinfo` (#8120) is a hidden option, so no help text changed.

  ### Adaptations

  Every pick changes the same lines as its develop counterpart except:

  - **#8073:** only the two GUI commits are taken. The other two backports in that PR (bitcoin#28629, partial bitcoin#28639) are test and refactor changes for code `v24.0.x` doesn't need.
  - **#8112:** `test/functional/feature_governance.py` differs on `v24.0.x` (no pruned node from #7860). The `v24.0.x` file is kept and only #8112's additions are applied: the imports it needs, `GovernanceSyncPeer`, `check_govsync_is_peer_local()` and its call. The source change is identical.
  - **#8113:** one line of the new unit test. `v24.0.x` has the five-argument `CWallet` constructor and `CreateMockWalletDatabase()`, so the second wallet is built the way the fixture in the same file builds the first. The source change is identical.
  - **#8117:** in `src/llmq/utils.cpp` the neighbouring includes differ; the pick removes `<atomic>` and adds nothing. The source change is otherwise identical.

  ### Not picked

  - #8111 (check every BIP39 checksum bit): it can refuse a mnemonic that v24.0.1 accepts, so it should come with a major or minor release and a release note, not a patch release.
  - #8118 (RPC argument and locking fixes): changes RPC error behaviour and needs a hand-port to `rpc/masternode.cpp`.
  - #8122 (CacheMultiMap): the use-after-free can't be reached on `v24.0.x`, and the capacity part is hardening with no behaviour change.
  - #8047 (index startup after an unclean stop): on `v24.0.x` a synced node never records the uncommitted tip it handles, and the governance index it mainly covers doesn't exist here.
  - #8092 (wording): changes two translatable strings.
  - #8101, #8102, #8098, #8095 with #8121, #8116 with #7815, #8085: unchanged from the v24.0.1 decision in #8128.
  - Features, refactors, Bitcoin Core backports, CI and test-only changes merged since the branch point.

  #8103 (EvoDB left ahead of the coins DB after a reindex) is a candidate that is still open on develop. If it merges before this PR does, its pick will be added here.

  ## How Has This Been Tested?

  Linux x86_64, depends build with `--enable-debug --enable-werror`, GUI enabled. Head is b534385.

  - `dashd`, `dash-qt`, `test_dash` and `test_dash-qt` build cleanly; `dashd -version` reports `v24.0.2`.
  - `test_dash` (1106 cases), `test_dash-qt` and `test/util/test_runner.py` pass. That includes the picked unit tests: `allocator_tests/mt_pooled_secure_allocator_reuses_all_chunks` (#8115), `coinjoin_tests/coinjoin_maintenance_runs_for_every_wallet` (#8113) and the orphan-vote case in `governance_vote_processing_tests` (#8112).
  - The full functional suite (`test_runner.py -j14`): 312 passed, 12 skipped, 1 failed. The skipped ones are the usual environment-dependent tests (`feature_bind_port_*`, `rpc_bind.py --ipv6`, `interface_usdt_*`, previous-release compatibility and `feature_unsupported_utxo_db.py`). The run includes the scenarios the picks add or extend: `feature_governance.py` (#8112), `feature_stats.py` (#8119) and `interface_bitcoin_cli.py` (#8120).
  - The failure was `feature_llmq_is_cl_conflicts.py`, in `mine_cycle_quorum`: the node logs show the block for the final commitments being built about 40 ms before the second quorum's final commitment reached the mining node, so that quorum was not mined. It is a timing race in the test helper under load, in code none of the picks touch (develop reworked those waits in #7892). Run again on its own four times in parallel, it passed all four.
  - **Leak, end to end.** A regtest script (not part of the tree) mines a single-member InstantSend quorum, sends InstantSend-locked transactions and reads `getmemoryinfo` `locked.used` on each node every 50 locks.
    - The published v24.0.1 Linux binary: the four non-masternode nodes go from 39,568 bytes to 467,664 bytes over 600 locks, and are still growing at the end.
    - This branch: the same nodes stay at 20,096 bytes for all 600 locks.
  - The man pages were regenerated with `contrib/devtools/gen-manpages.py` from this build. They are byte-identical to the committed ones except for the `lock` debug category in `dashd.1` and `dash-qt.1`, which only an `--enable-debug` build lists.
  - The archived v24.0.1 notes are byte-identical to the previous `doc/release-notes.md`.

  Not tested: the macOS shutdown crash itself. The GUI change is the upstream one-liner and was built and run through `test_dash-qt` on Linux only.

  ## Breaking Changes

  None. No consensus change, and no change to which P2P messages a node accepts or how it scores peers. A node that syncs governance from a v24.0.2 node receives the same inventories as before; other peers of that node no longer receive them.

  ## Checklist:
  - [x] I have performed a self-review of my own code
  - [x] I have made corresponding changes to the documentation
  - [x] I have assigned this pull request to a milestone _(if you have permission to set one)_

  🤖 Generated with [Claude Code](https://claude.com/claude-code)

Top commit has no ACKs.

Tree-SHA512: 6d9861f4a8c2341889bb0ce5dd500a7083ea59395777a21c5e5d01e70ec8e8b489cf6425bc26ca79f15b1919610ceab10baada361043322dc689b7dfcae7cb5d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants