Skip to content

fix(dashmate): stop masternodes safely around DKGs on Core 24 - #5841

Merged
shumkov merged 11 commits into
v5.1-devfrom
shumkov/dashmate-dkg-safe-stop
Nov 12, 2026
Merged

shumkov merged 11 commits into
v5.1-devfrom
shumkov/dashmate-dkg-safe-stop

Conversation

@shumkov

@shumkov shumkov commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Issue being fixed or feature implemented

Bottom of GitHub Stack #5847: this PR, then #5842 (Core 24 default image), then #5843 (--safe progress).

dashmate stop, restart and their --safe wait decide when stopping a masternode cannot cost it a PoSe penalty for missing DKG messages.

On Core 24.0 the v5.1 rule (#5231) almost never lets a masternode stop. Once Core reports upcoming_dkgs, the rule treats any non-zero quorum dkginfo active_dkgs as a running session. That rule was written for dashpay/dash#7812, which derives active_dkgs from membership. dash#7812 landed in develop for 24.1 and is not in v24.0.0. Core 24.0 still counts the DKG worker's session records, which linger until the next round of the same quorum type and index.

On mainnet (360 evonodes, ~2195 masternodes) a masternode holds such a record about 94% of the time; llmq_60_75 records alone last ~48 h. So plain stop refused, and --safe waited hours to days. This blocks moving dashmate's default image to Core 24 (follow-up PR).

Operators also wait longer than DKGs require. A mainnet operator reported a 1 h --safe wait on an evonode under 5.0.x. Simulated over a random mainnet membership history, 5.0.x's rule makes 13.6% of evonode stops wait an hour or more (p90 ~85 min, worst ~3.7 h). Most of that comes from rotated llmq_60_75 sessions: they block for 42 blocks counted from each quorum index's own start. 42 is the cycle's shared mining window, so it was counted from the wrong base.

A stop at a DKG start could pass. Core older than 24.1 records a session only after the start block connects. A check right at that block saw the previous round's stale record, or none, and said "safe".

group restart --safe never waited. Its flag was declared as a plain object, which oclif parses as false.

What was done?

getDkgStopBlockers is now the single safety rule. It returns height ranges in which stopping is blocked; stopping is safe when none covers the chain tip. It works on Core 23, 24.0 and 24.1:

  • Upcoming DKGs. A DKG this node is a member of, or whose membership Core cannot determine, blocks from 6 blocks before its start until its Commit phase ends. Core 24 reports membership in upcoming_dkgs. Without that data (Core 23, or no proTxHash), any DKG starting within 6 blocks blocks, as before.
  • Session records (quorum dkgstatus).
    • A record blocks from the session's start to the end of its Commit phase (5 × dkgPhaseBlocks), the same window Core 24.1 counts. Each rotated quorum index is measured from its own start, so llmq_60_75 blocks for 10 blocks instead of 42.
    • A record that has reached its next round but that Core has not replaced yet blocks as that new round.
    • Records are read on every Core version. On 24.1 a mid-session ProUpServTx clears Core's proTxHash, so active_dkgs drops to 0 while the session still runs.
  • Core 24.1+ (getnetworkinfo version ≥ 240100): a non-zero active_dkgs blocks. Core's version is read over RPC, not taken from the image tag, because dashpay/dashd:24 will float to 24.1 on pull.
  • A session Core has not recorded (Core older than 24.1): quorum dkgstatus lists every enabled quorum with the start of its current session. Core records a session only after computing its members, with no bound on how long that takes, and until then nothing shows whether this node is a member. So a current session without a record blocks from its start until its Commit phase ends.
    • Sessions Core announced without this node don't block. The wait and the confirming check remember those announcements, on the chain they were made on. Each poll checks that the previous poll's tip is still on the chain, and drops remembered non-memberships after a reorg.
  • Consistent reads and confirmation.
    • Each check reads the chain tip before and after the DKG RPCs, and is discarded if the tip moved.
    • On Core older than 24.1 a safe result is confirmed 15 s after the first check finished. The chain may have grown by at most 6 blocks, and the confirming check must still find the first check's tip at that height (getblockhash, read inside its snapshot), which catches a reorg that ended higher.
    • A record count that differs between dkginfo and dkgstatus (Core changing its records mid-read) is unsettled too.
    • A Core version that is not an integer is an error, not a guess at older Core.
  • --safe remembers every DKG of this node it has seen, announced or running, until its Commit phase ends. This covers the gap at a session start where Core has erased last round's record but not yet created the new one, and a Core restarted mid-session that no longer reports the session.
  • Announced starts come from Core's absolute quorumHeight, not the tip plus blocksUntilStart.
  • Core in initial block download runs no DKG session until synced, so session records don't apply to it, even while blocks connect between the reads. DKGs it announces still block, because it may finish syncing in time for them.
  • Core's version is read on every --safe poll, since Core may be replaced between polls.
  • Local networks: group commands stop a local network's miner before its masternodes. With --safe those masternodes then waited for blocks that could never come. Once the miner is stopped, later local-network masternodes in its group skip the DKG check: every masternode goes down together, so their DKG produces no commitment to penalize them in. Configs of other networks in the same run are still checked.
  • Plain stop retries an unsettled check up to 3 times. It reports unreadable Core data with its own error, because --safe would wait for that forever.
  • group restart --safe now parses as a boolean flag.

With these rules on Core 24.0, the same mainnet simulation gives a mean evonode wait of ~6 min (from ~22), p90 ~28 min, and 0.9% of waits ≥ 1 h. A regular masternode outside a DKG is almost never blocked.

Cost: on Core older than 24.1, a plain stop or restart of a masternode that is safe to stop takes about 15 s longer. A plain stop also refuses during the first blocks (10, or 20 for 400-member quorums) of every DKG session Core has not recorded, members or not, because it hasn't seen the announcements. --safe doesn't wait for sessions it saw announced without this node. Local group stops/restarts stop masternodes one at a time, so that adds up per node.

The progress/ETA display for --safe comes in a follow-up PR stacked on this one.

How Has This Been Tested?

  • Core RPC semantics were checked against Dash Core sources: v24.0.0 for quorum dkginfo/dkgstatus, record reset and initialisation, the DKG worker and IBD; develop/24.1 for dash#7812. The LLMQ parameters come from src/llmq/params.h and are pinned in tests by literal values.
  • New unit tests drive the wait loop, the check and the stop task against a scripted Core RPC (src/test/mock/getCoreRpcMock.js).
  • Run against the previous code, these fail. ✖ before, ✔ after:
    • should stop waiting on Core 24.0 when active_dkgs counts only finished sessions
    • should end a rotated DKG session once its own Commit phase is over
    • should keep waiting at a DKG start block while Core still shows the previous round
    • should not stop at a DKG start that Core has not recorded yet
    • should not trust DKG reads that saw a newer tip than the height read
    • should keep waiting on Core 24.1 while a session runs after the proTxHash was cleared
    • should parse --safe so the restart waits for a safe DKG window
    • should not stop at a DKG start reached in Core's record gap, however long it lasts
    • should refuse a plain stop at a DKG start that Core has not recorded yet
    • should report unsettled when a reorg replaced the first check's tip and the chain grew
    • should not wait for DKGs once the local network's miner has stopped
    • should treat Core in initial block download as safe even while blocks keep connecting
    • should refuse to guess when Core reports no integer version
    • should keep blocking after the start block while Core still has no record
    • should not stop in a session Core has not recorded when waiting starts after its start block
    • should forget DKGs announced without this node when a reorg replaces their chain
    • should report a replaced chain when the previous check's tip is gone
    • should still check a masternode of another network after a local miner stopped
    • should not hold a session it saw announced without this node after its start block
    • should keep blocking a DKG it saw announced when Core restarts into initial block download
    • should not read a DKG of a type dashmate does not know that may include this node
    • should still block a DKG this node is in while Core syncs
    • should keep the first check's DKGs when Core restarts into initial block download during confirmation
    • should judge each poll by the Core version answering it
    • should place an announced DKG at the start Core gives
    • should count next_dkg from the earliest tip dkginfo may have seen
    • should not read Core's announcements as covering a tip that moved too far past them
    • should keep blocking a running session after Core restarts and forgets it
  • --safe's memory of announced DKGs: its test fails with that memory removed.
  • yarn workspace dashmate test:unit: 1092 passing, run locally.
  • eslint on the changed files: no errors. One warning, an unused no-console directive in stopNodeTaskFactory.js, already existed.
  • Not run: a live local network or testnet masternode.

Breaking Changes

None. Plain stop/restart on Core older than 24.1 is ~15 s slower when it is safe to stop (see above).

Checklist:

  • I have performed a self-review of my own code
  • My Rust, FFI, WASM, Swift or Kotlin changes follow the coding conventions
  • I have commented my code, particularly in hard-to-understand areas
  • I have added or updated relevant unit/integration/functional/e2e tests
  • I have added "!" to the title and described breaking changes in the corresponding section if my code contains any
  • I have made corresponding changes to the documentation if needed
  • If I added or changed GroveDB structure, I described it in the area's structure.rs, regenerated grovedb-structure.json, and checked the structure viewer link posted on this pull request
  • If I added or changed a system contract's document schemas, I checked the contract viewer link posted on this pull request

For repository code-owners and collaborators only

  • I have assigned this pull request to a milestone

🤖 Generated with Claude Code

PR Hygiene · 140b310

  • Bots — thepastaclaw requested changes — dismiss the review or push a fix
  • Self-review — post /self-reviewed
  • Build failed
  • Approvals — you own every area touched; none needed

When every merge requirement is met, the PR Hygiene check passes. Reviewer limits do not block merging; other required GitHub checks and protections still apply.

Summary by CodeRabbit

  • Bug Fixes
    • Improved masternode safe-stop checks to account for DKG activity, chain changes, and Core version differences.
    • Safe-stop checks retry when chain or DKG information is unsettled and report when required DKG data is unavailable.
    • DKG checks are skipped for a group after its local miner has stopped, while other networks in that group remain checked.
  • Improvements
    • The group restart --safe option and its -s alias are parsed consistently.

@shumkov shumkov added this to the v5.1.0 milestone Nov 10, 2026
@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 2 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between d1a2eb5 and 140b310.

Files selected for processing: 3

  • packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js
  • packages/dashmate/test/unit/core/quorum/checkMasternodeSafeToStop.spec.js
  • packages/dashmate/test/unit/core/quorum/waitForDKGWindowPass.spec.js
Walkthrough

The change adds Core-version-aware DKG stop blockers and safe-stop checks, tracks DKG state while waiting, and integrates those checks into node stopping. It also updates the group restart command’s safe flag and adds tests for the updated behavior.

Changes: DKG-safe stop flow

Layer / File(s) Summary
Calculate DKG stop blockers
packages/dashmate/src/core/quorum/getDkgStopBlockers.js, packages/dashmate/src/core/quorum/isMasternodeSafeToStopDuringDkg.js, packages/dashmate/test/unit/core/quorum/getDkgStopBlockers.spec.js, packages/dashmate/test/unit/core/quorum/isMasternodeSafeToStopDuringDkg.spec.js
Adds blocker intervals for upcoming DKGs, session records, and unrecorded sessions. Core version selects membership and active-DKG handling. The prior per-session check and its tests are removed.
Evaluate safe-stop checks
packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js, packages/dashmate/src/test/mock/getCoreRpcMock.js, packages/dashmate/test/unit/core/quorum/checkMasternodeSafeToStop.spec.js
Returns structured safe, blocked, or unsettled results. Reads chain and DKG state around RPC calls, detects inconsistent snapshots, and confirms safe results on older Core versions.
Track blockers during DKG waits
packages/dashmate/src/core/quorum/waitForDKGWindowPass.js, packages/dashmate/test/unit/core/quorum/waitForDKGWindowPass.spec.js
Tracks announced DKGs and known non-members across polling checks. Handles reorgs and removes tracked DKG keys when they are no longer current.
Apply safe-stop checks during node stops
packages/dashmate/src/listr/tasks/stopNodeTaskFactory.js, packages/dashmate/src/commands/group/restart.js, packages/dashmate/test/unit/listr/tasks/stopNodeTaskFactory.spec.js, packages/dashmate/test/unit/commands/group/restart.spec.js
Retries unsettled stop checks up to three times and records stopped local-network miners. Updates the restart command’s safe flag declaration and tests its parser behavior.

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant StopNodeTaskFactory
  participant SafeStopCheck
  participant CoreRPC
  participant DkgStopBlockers
  participant DkgWait
  StopNodeTaskFactory->>SafeStopCheck: Request version-aware safety check
  SafeStopCheck->>CoreRPC: Read chain and DKG state
  SafeStopCheck->>DkgStopBlockers: Evaluate DKG stop blockers
  DkgStopBlockers-->>SafeStopCheck: Return blockers
  SafeStopCheck-->>StopNodeTaskFactory: Return safe, blocked, or unsettled result
  StopNodeTaskFactory->>DkgWait: Wait in safe mode
  DkgWait->>SafeStopCheck: Poll with tracked DKG state
Loading

Possibly related issues

Possibly related PRs

  • dashpay/platform#5843: Extends this DKG stop-safety flow with blocker reporting, wait-time estimates, and failed-check handling.

Suggested reviewers: thepastaclaw

Merge Risk: Moderate · previous review d1a2e

Stopping a masternode that is syncing quickly can fail with a "chain kept advancing" error, and a safe stop can wait indefinitely. This should be fixed or explicitly accepted before merge. The DKG-safety logic otherwise looks sound, and most stops are unaffected.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 73.91% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed

Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: making Dashmate stop operations safe around DKGs on Core 24.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Full details: Docstring Coverage

Explanation

Docstring coverage is 73.91% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 23 functions across 11 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches

Generate docstrings

  • Commit to this branch
  • Create a new PR
Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions github-actions Bot added the waiting-bots Waiting for the review bots to report on this head label Nov 10, 2026
@thepastaclaw

thepastaclaw commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

⛔ Final review complete — 1 blocking finding(s) (commit 140b310) · triage: normal

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final validation — Phase 1 + Phase 2

Verified the findings against head aba4078 and the relevant Dash Core sources. Three blocking issues remain: delayed session initialization can permit an unsafe stop, confirmation does not detect higher-tip reorganizations, and local group restart --safe can deadlock after stopping the miner. Current code checks pass, including Dashmate tests, linting, and all three Dashmate E2E jobs; this verification was static and ran no builds or tests.

🔴 3 blocking | 🟡 2 suggestion(s)

Review provenance

Source: reviewer 1: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: architecture-layering); reviewer 4: gpt-6.1-sol (agent: phase2-reviewer, role: security-auditor); reviewer 5: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 6: gpt-6.1-sol (agent: phase2-reviewer, role: architecture-layering); reviewer 7: gpt-6.1-sol (agent: phase2-reviewer, role: security-auditor); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: normal by gpt-6.1-sol (effort low) — This is a large, intricate dashmate control-flow change affecting masternode shutdown safety, DKG timing, RPC consistency, retries, and confirmation behavior, but it does not modify consensus, funds, cryptography, signatures, peer-facing deserialization, or storage migrations.
  • Phase 1 reviewers: muse-spark-1.3-contributor — general (completed, effort xhigh); agent phase1-reviewer
  • Phase 1 model: muse-spark-1.3-contributor — not quota-gated; passed over gemini-3.8-flash-high (antigravity quota lookup failed), glm-5.3-flash (not used above high effort; tier asks max)
  • Second Phase-2 pass: on this first full review every Phase-2 reviewer also ran a second, independent time, and the verifier weighed both sets
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — architecture-layering (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — security-auditor (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — general (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — architecture-layering (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — security-auditor (completed, effort high); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review (PastaClaw). How to work through them:
- Verify each finding against the code; fix only what is real, as narrowly as the finding.
- If a finding is wrong, decline it on its thread in one or two sentences; the bot answers once and stops.
- To defer one, link a tracking issue on its thread (or reply asking the bot to file one).
- Suggestions are optional: fix them forward in a later PR rather than pushing another round just for them.
- Address every current finding in one push: self-review the full diff first, and don't push piecemeal.
- Keep the PR's scope; don't grow it to satisfy hardening suggestions.
- Carried items were raised in an earlier round: read the thread first, and don't re-fix what you already fixed, declined or deferred there. Disputed and deferred items are already handled and are not listed.

In `packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js`:
- [BLOCKING] packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js:143-145: Do not treat the confirmation delay as proof of session initialization
  A plain stop or fresh --safe invocation beginning in Core's session-record gap has no remembered announcement. On Core older than 24.1, both evaluations can see active_dkgs=0, next_dkg=24, and an empty session list at a non-rotated start height, and therefore permit stopping after 15 seconds. Core resets the debug record before initializing the new session and publishing its membership record; that work has no 15-second completion guarantee. The new wait-loop test explicitly models this gap lasting longer than the confirmation delay, but avoids the failure only because polling began before the start block. Keep an unobserved current-session window conservative unless participation or initialization completion can be established, and add a regression that starts directly in the empty-record gap and keeps it empty beyond the delay.
- [BLOCKING] packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js:147-149: Verify ancestry when the confirming tip is higher
  The hash comparison detects replacement of the first tip only when both snapshots have the same height. A reorganization that replaces that tip and finishes one to six blocks higher passes this condition. For example, the first snapshot can predict non-membership before a DKG starts, while the replacement branch changes its work block and reaches the start block shortly before confirmation. On Core 24.0, the new member session may not yet have a debug record, so the second evaluation can report safe without the promised initialization interval on the replacement branch. Read the active-chain hash at first.height within the confirming snapshot's consistency checks and return unsettled if it differs from first.hash. Add coverage for a replacement branch with a higher final tip.
- [SUGGESTION] packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js:40-47: Allow the IBD exemption when the tip advances between reads
  The tip-hash rejection runs before evaluate can apply the initial-block-download exemption. When an IBD node advances between the bracketing reads, the snapshot is discarded even if both reads report initialblockdownload=true. Plain stop can exhaust all three attempts and reject a node that is safe to stop, while --safe continues polling until it gets a stable read or synchronization finishes. Core's DKG manager returns without processing tip updates during IBD, so DKG snapshot consistency is unnecessary when both bracketing reads confirm that state. Preserve that exemption across tip movement and add coverage with IBD true on both reads and different tip hashes.

In `packages/dashmate/src/commands/group/restart.js`:
- [BLOCKING] packages/dashmate/src/commands/group/restart.js:11-15: Coordinate safe group restart with local miner shutdown
  Making --safe effective exposes a deadlock on the standard local network. The stop tasks run sequentially over configGroup.reverse(), stopping local_seed first. That node hosts the detached mining loop started by startGroupNodesTaskFactory. If a remaining masternode is blocked by an imminent or active DKG at that height, waitForDKGWindowPass cannot finish because block production has stopped, and the subsequent task that restarts the miner is unreachable. The parser test does not exercise execution, and the local-network E2E restart forces its stops. Coordinate safety waiting and miner shutdown at group level so a height-dependent wait cannot begin after mining has stopped, and cover a restart at an imminent-DKG height.

In `packages/dashmate/src/core/quorum/getDkgStopBlockers.js`:
- [SUGGESTION] packages/dashmate/src/core/quorum/getDkgStopBlockers.js:284-287: Track the remaining local-session reporting gap in Dash Core
  Link an upstream issue or PR for the specific lifecycle gap behind this fallback. Current Core clears the active manager's proTxHash after a service-address change when reinitialization cannot accept the new address, while an existing DKG session retains its own myProTxHash. quorum dkginfo computes active_dkgs only when the current proTxHash is non-null, so it can report zero despite an outstanding local session. The linked dashpay/dash#7812 fixes membership-based counting but explicitly retains zero for calls without a proTxHash; it does not resolve this case. Keeping compatibility handling here is reasonable, but the missing authoritative local-activity capability belongs in Dash Core. Track an owner-side repair that accounts for unexpired locally owned sessions, with a functional test for a service update during DKG.

@github-actions

Copy link
Copy Markdown
Contributor

Your move: thepastaclaw requested changes on this head; dismiss the review or push a fix; thepastaclaw left review threads unresolved; resolve them.
Full checklist in the description.

@github-actions github-actions Bot added waiting-self-review Waiting for the author to post /self-reviewed and removed waiting-bots Waiting for the review bots to report on this head labels Nov 10, 2026
@shumkov
shumkov changed the base branch from v5.1-dev to v5.0-dev November 10, 2026 15:29
@shumkov
shumkov added this pull request to stack #5846 November 10, 2026 15:29
@github-actions github-actions Bot modified the milestones: v5.1.0, v5.0.0 Nov 10, 2026
@shumkov
shumkov removed this pull request from stack #5846 November 10, 2026 15:30
@shumkov
shumkov changed the base branch from v5.0-dev to v5.1-dev November 10, 2026 15:30
@shumkov
shumkov added this pull request to stack #5847 November 10, 2026 15:30
@github-actions github-actions Bot modified the milestones: v5.0.0, v5.1.0 Nov 10, 2026
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Your move: thepastaclaw left review threads unresolved; resolve them.
Full checklist in the description.

@github-actions

Copy link
Copy Markdown
Contributor

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions github-actions Bot added waiting-bots Waiting for the review bots to report on this head and removed waiting-self-review Waiting for the author to post /self-reviewed labels Nov 10, 2026

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review — Final validation — Phase 1 + Phase 2

This head still contains an unsafe-stop path for older Core versions: advancing one block beyond an unrecorded DKG start is treated as evidence that the session is irrelevant. It also introduces stale cross-poll membership evidence across reorgs and applies the local-miner exemption to unrelated networks. Dashmate tests, lint, and builds passed in the supplied CI snapshot; local-network E2E and broader suites were still pending.

🔴 3 blocking

Review provenance

Source: reviewer 1: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: architecture-layering); reviewer 4: gpt-6.1-sol (agent: phase2-reviewer, role: security-auditor); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: normal by gpt-6.1-sol (effort low) — This is a large and intricate dashmate safety-logic change involving DKG timing, Core-version behavior, reorg handling, confirmation races, and command integration, but it does not modify consensus, funds movement, cryptography, or another critical surface listed for the critical tier.
  • Phase 1 reviewers: muse-spark-1.3-contributor — general (completed, effort xhigh); agent phase1-reviewer
  • Phase 1 model: muse-spark-1.3-contributor — not quota-gated; passed over gemini-3.8-flash-high (antigravity quota lookup failed), glm-5.3-flash (not used above high effort; tier asks max)
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — architecture-layering (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — security-auditor (completed, effort high); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review (PastaClaw). How to work through them:
- Verify each finding against the code; fix only what is real, as narrowly as the finding.
- If a finding is wrong, decline it on its thread in one or two sentences; the bot answers once and stops.
- To defer one, link a tracking issue on its thread (or reply asking the bot to file one).
- Suggestions are optional: fix them forward in a later PR rather than pushing another round just for them.
- Address every current finding in one push: self-review the full diff first, and don't push piecemeal.
- Keep the PR's scope; don't grow it to satisfy hardening suggestions.
- Carried items were raised in an earlier round: read the thread first, and don't re-fix what you already fixed, declined or deferred there. Disputed and deferred items are already handled and are not listed.

In `packages/dashmate/src/core/quorum/getDkgStopBlockers.js`:
- [BLOCKING] packages/dashmate/src/core/quorum/getDkgStopBlockers.js:299-300: Do not treat the confirmation delay as proof of session initialization
  The start-block guard only covers the exact height at which the session starts. If a fresh check begins at start height S with no announcement or session record, then the chain advances to S+1 while Core is still initializing membership, `pQuorumBaseBlockIndex === height` no longer matches, and the blocker disappears. With `active_dkgs === 0`, no session record, and no upcoming entry, two stable reads 15 seconds apart can authorize stopping even though Core has not established that this node is a non-member. The existing regression models this unsafe assumption by declaring that an absent record at S+1 proves non-membership. Keep an unobserved session conservative throughout the initialization window unless authoritative membership or initialization-completion evidence exists, and add a regression starting directly at S+1 with the record still absent.

In `packages/dashmate/src/core/quorum/waitForDKGWindowPass.js`:
- [BLOCKING] packages/dashmate/src/core/quorum/waitForDKGWindowPass.js:26-40: Invalidate remembered non-membership when its chain changes
  `nonMemberDkgs` is retained across polls using only the LLMQ type, quorum index, and start height. `getAnnouncedNonMemberDkgs` receives Core's `workBlockHash`, but discards it when creating the key. If a reorg replaces the work block and the replacement branch selects this node as a member, the stale negative prediction still suppresses `getStartingDkgBlockers` at the replacement branch's start height. The confirming check only verifies ancestry within the replacement branch, so it cannot detect that the cached prediction originated on another branch. Preserve Core's work-block identity with the cached prediction and validate or discard it before allowing it to override the conservative start-block guard.

In `packages/dashmate/src/listr/tasks/stopNodeTaskFactory.js`:
- [BLOCKING] packages/dashmate/src/listr/tasks/stopNodeTaskFactory.js:55-57: Restrict the stopped-miner exemption to the affected local network
  `ctx.isLocalMinerStopped` is shared by the entire group task, but the enabled predicates consume it for every config regardless of that config's network. Groups are selected by the configurable group name and the schema does not require all members to use the same network. Therefore, after a local miner is stopped, a later mainnet or testnet masternode in the same group skips both the DKG check and the `--safe` wait, even though stopping an unrelated local miner does not stop block production or remove DKG obligations on that public network. Scope the exemption to the coordinated local network, or have the group orchestration pass a network-specific authorization rather than using a run-wide boolean.

@github-actions

Copy link
Copy Markdown
Contributor

Your move: thepastaclaw requested changes on this head; dismiss the review or push a fix; thepastaclaw left review threads unresolved; resolve them.
Full checklist in the description.

@github-actions github-actions Bot added waiting-self-review Waiting for the author to post /self-reviewed and removed waiting-bots Waiting for the review bots to report on this head labels Nov 10, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Your move: thepastaclaw left review threads unresolved; resolve them.
Full checklist in the description.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions github-actions Bot added waiting-bots Waiting for the review bots to report on this head and removed waiting-self-review Waiting for the author to post /self-reviewed labels Nov 11, 2026

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review — Final validation — Phase 2 only (queue backlog)

At ff137eb, work-block validation and the post-DKG ancestor lookup address the reported IBD branch-switch orderings, while moving IBD snapshots are accepted again; two prior safety findings remain blocking. Current CI shows passing Dashmate tests, JavaScript linting, functional tests, Docker builds, and Dashmate E2E jobs, with review and PR Hygiene pending. This was a static review; no builds or tests were run locally.

🔴 2 blocking

Still open from earlier rounds (2)

  • 🔴 Blocking: Keep DKG announcements anchored when accepting moving IBD snapshots — packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js:87-95 (thread)
  • 🔴 Blocking: Preserve member DKGs learned before confirmation enters IBD — packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js:251-254 (thread)
Review provenance

Source: reviewer 1: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: architecture-layering); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: security-auditor); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: normal by gpt-6.1-sol (effort low) — The change is intricate but modifies dashmate shutdown orchestration and DKG safety checks rather than consensus rules, funds movement, cryptography, or another qualifying critical surface.
  • Phase 1 reviewers: not run (skipped for throughput: 18 PRs queued, above the 10 limit)
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — architecture-layering (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — security-auditor (completed, effort high); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review (PastaClaw). How to work through them:
- Verify each finding against the code; fix only what is real, as narrowly as the finding.
- If a finding is wrong, decline it on its thread in one or two sentences; the bot answers once and stops.
- To defer one, link a tracking issue on its thread (or reply asking the bot to file one).
- Suggestions don't block the merge, but most are worth fixing in this PR; judge each on its merits.
- Keep the PR's scope; don't grow it to satisfy hardening suggestions.
- Carried items were raised in an earlier round: read the thread first, and don't re-fix what you already fixed, declined or deferred there. Disputed and deferred items are already handled and are not listed.

In `packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js`:
- [BLOCKING] packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js:87-95: Keep DKG announcements anchored when accepting moving IBD snapshots
  (carried; thread: https://github.com/dashpay/platform/pull/5841#discussion_r4530984956)
- [BLOCKING] packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js:251-254: Preserve member DKGs learned before confirmation enters IBD
  (carried; thread: https://github.com/dashpay/platform/pull/5841#discussion_r4529658849)

@github-actions

Copy link
Copy Markdown
Contributor

Your move: thepastaclaw requested changes on this head; dismiss the review or push a fix.
Full checklist in the description.

@github-actions github-actions Bot added waiting-self-review Waiting for the author to post /self-reviewed and removed waiting-bots Waiting for the review bots to report on this head labels Nov 11, 2026
When the read confirming a safe result was discarded, as when a block
connects during it, the check came back with no height and no blockers.
`--safe` remembers DKGs only from a check with a height, so it forgot
the DKG of this node the first read announced, and could stop the node
once Core no longer showed it. Such a check now reports the first read,
unsettled, so its announced DKGs are remembered.

Test would have caught this: ✖ before, ✔ after ("should keep what a
check announced when its confirming read is discarded").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shumkov added a commit that referenced this pull request Nov 11, 2026
`--safe` retries failing Core calls. A session it saw running must still
block once Core comes back with no record, count or announcement of it,
as Core 24.1 does when restarted before it knows its proTxHash. The loop
remembers every blocker with a known range since #5841's change, carried
into this branch by the restack.

✖ with the previous loop, ✔ now ("should keep blocking a running session
through failed calls and a Core restart").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions github-actions Bot added waiting-bots Waiting for the review bots to report on this head and removed waiting-self-review Waiting for the author to post /self-reviewed labels Nov 11, 2026

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review — Final validation — Phase 1 + Phase 2

The head fixes most previously identified safe-stop lifecycle issues, and the CI snapshot contains no failed code checks; the JavaScript and Docker builds are still queued. Two blocking safety gaps remain: record-count mismatches can discard a previously observed member DKG, and moving IBD snapshots can be accepted without proving that their announcement list covers the accepted tip.

🔴 2 blocking

Still open from earlier rounds (2)

  • 🔴 Blocking: Preserve member DKGs learned before confirmation enters IBD — packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js:191-192 (thread)
  • 🔴 Blocking: Keep DKG announcements anchored when accepting moving IBD snapshots — packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js:87-95 (thread)
Review provenance

Source: reviewer 1: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: architecture-layering); reviewer 4: gpt-6.1-sol (agent: phase2-reviewer, role: security-auditor); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: normal by gpt-6.1-sol (effort low) — This is a large and intricate dashmate behavioral change involving quorum RPC interpretation, chain reorg consistency, version-dependent logic, and stop-safety timing, but it does not modify consensus, funds, cryptography, or another critical surface defined by the rubric.
  • Phase 1 reviewers: muse-spark-1.3-contributor — general (completed, effort xhigh); agent phase1-reviewer
  • Phase 1 model: muse-spark-1.3-contributor — not quota-gated; passed over gemini-3.8-flash-high (antigravity quota lookup failed), glm-5.3-flash (not used above high effort; tier asks max)
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — architecture-layering (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — security-auditor (completed, effort high); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review (PastaClaw). How to work through them:
- Verify each finding against the code; fix only what is real, as narrowly as the finding.
- If a finding is wrong, decline it on its thread in one or two sentences; the bot answers once and stops.
- To defer one, link a tracking issue on its thread (or reply asking the bot to file one).
- Suggestions don't block the merge, but most are worth fixing in this PR; judge each on its merits.
- Keep the PR's scope; don't grow it to satisfy hardening suggestions.
- Carried items were raised in an earlier round: read the thread first, and don't re-fix what you already fixed, declined or deferred there. Disputed and deferred items are already handled and are not listed.

In `packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js`:
- [BLOCKING] packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js:191-192: Preserve member DKGs learned before confirmation enters IBD
  (carried; thread: https://github.com/dashpay/platform/pull/5841#discussion_r4529658849)
- [BLOCKING] packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js:87-95: Keep DKG announcements anchored when accepting moving IBD snapshots
  (carried; thread: https://github.com/dashpay/platform/pull/5841#discussion_r4530984956)

@github-actions

Copy link
Copy Markdown
Contributor

Your move: thepastaclaw requested changes on this head; dismiss the review or push a fix.
Full checklist in the description.

@github-actions github-actions Bot added waiting-self-review Waiting for the author to post /self-reviewed and removed waiting-bots Waiting for the review bots to report on this head labels Nov 11, 2026
…e confirming

When Core older than 24.1 changed its session records during the read
confirming a safe result, the check came back unsettled with no
blockers. `--safe` then forgot the DKG of this node the first read
announced, and could stop the node once Core no longer showed it. Such
a check now carries the first read's blockers.

Test would have caught this: ✖ before, ✔ after ("should keep what a
check announced when Core changes its session records during the
confirming read").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shumkov added a commit that referenced this pull request Nov 11, 2026
`--safe` retries failing Core calls. A session it saw running must still
block once Core comes back with no record, count or announcement of it,
as Core 24.1 does when restarted before it knows its proTxHash. The loop
remembers every blocker with a known range since #5841's change, carried
into this branch by the restack.

✖ with the previous loop, ✔ now ("should keep blocking a running session
through failed calls and a Core restart").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions github-actions Bot added waiting-bots Waiting for the review bots to report on this head and removed waiting-self-review Waiting for the author to post /self-reviewed labels Nov 11, 2026

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review — Final validation — Phase 1 + Phase 2

The new head preserves first-read DKG blockers through discarded and record-changing confirmations and validates announcement work blocks during moving IBD reads. One disputed announcement-coverage gap remains blocking, alongside a narrow version-refresh suggestion for plain-stop retries. This was a static review; current CI shows dashmate tests, JS lint, builds, and dashmate E2E tests passing, with additional test-suite jobs still pending.

🔴 1 blocking

Still open from earlier rounds (1)

  • 🔴 Blocking: Keep DKG announcements anchored when accepting moving IBD snapshots — packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js:87-95 (thread)
Follow-up candidates (optional) (1)

New non-blocking observations on code unchanged since the last full review. No inline threads are opened for them and later reviews do not repeat them: fix forward or track separately if they are worth it.

🟡 Suggestion: Re-read Core version on each plain-stop retry
packages/dashmate/src/listr/tasks/stopNodeTaskFactory.js:83-92

The plain-stop retry loop reuses the version read before its first attempt. If Core is replaced between an unsettled attempt and the next attempt, the subsequent snapshot is interpreted using the previous process's rules. In particular, a replacement from 24.1 to 24.0 leaves the cached version selecting membership-based counting and disabling both the unrecorded-session fallback and confirmation, allowing an empty record gap to pass. The safe-wait loop already refreshes the version on each poll to handle replacement. Refresh it inside this retry loop as well, and cover a version change between attempts.

          let check;
          for (let attempt = 0; attempt < SAFE_STOP_CHECK_ATTEMPTS; attempt += 1) {
            const coreVersion = await readCoreVersion(rpcClient);
            check = await checkMasternodeSafeToStop(rpcClient, coreVersion);

            if (check.status !== 'unsettled') {
              break;
            }
          }

source: muse-spark-1.3-contributor (phase1-reviewer: general)

Review provenance

Source: reviewer 1: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: architecture-layering); reviewer 4: gpt-6.1-sol (agent: phase2-reviewer, role: security-auditor); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: normal by gpt-6.1-sol (effort low) — This is a large and intricate dashmate safety-control rewrite involving DKG state, chain reorg handling, RPC consistency, version-dependent behavior, and confirmation timing, but it does not modify consensus, funds, cryptography, or another critical surface.
  • Phase 1 reviewers: muse-spark-1.3-contributor — general (completed, effort xhigh); agent phase1-reviewer
  • Phase 1 model: muse-spark-1.3-contributor — not quota-gated; passed over gemini-3.8-flash-high (antigravity quota lookup failed), glm-5.3-flash (not used above high effort; tier asks max)
  • Code unchanged since the last final review (148c936b): 1 new suggestion(s) listed as follow-up candidates, not posted inline
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — architecture-layering (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — security-auditor (completed, effort high); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review (PastaClaw). How to work through them:
- Verify each finding against the code; fix only what is real, as narrowly as the finding.
- If a finding is wrong, decline it on its thread in one or two sentences; the bot answers once and stops.
- To defer one, link a tracking issue on its thread (or reply asking the bot to file one).
- Suggestions don't block the merge, but most are worth fixing in this PR; judge each on its merits.
- Keep the PR's scope; don't grow it to satisfy hardening suggestions.
- Carried items were raised in an earlier round: read the thread first, and don't re-fix what you already fixed, declined or deferred there. Disputed and deferred items are already handled and are not listed.

In `packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js`:
- [BLOCKING] packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js:87-95: Keep DKG announcements anchored when accepting moving IBD snapshots
  (carried; thread: https://github.com/dashpay/platform/pull/5841#discussion_r4530984956)

@github-actions

Copy link
Copy Markdown
Contributor

Your move: thepastaclaw requested changes on this head; dismiss the review or push a fix.
Full checklist in the description.

@github-actions github-actions Bot added waiting-self-review Waiting for the author to post /self-reviewed and removed waiting-bots Waiting for the review bots to report on this head labels Nov 11, 2026
@shumkov
shumkov removed this pull request from stack #5847 November 12, 2026 01:42
@shumkov
shumkov merged commit 82d9a9a into v5.1-dev Nov 12, 2026
28 of 30 checks passed
@shumkov
shumkov deleted the shumkov/dashmate-dkg-safe-stop branch November 12, 2026 01:42
shumkov added a commit that referenced this pull request Nov 12, 2026
`--safe` retries failing Core calls. A session it saw running must still
block once Core comes back with no record, count or announcement of it,
as Core 24.1 does when restarted before it knows its proTxHash. The loop
remembers every blocker with a known range since #5841's change, carried
into this branch by the restack.

✖ with the previous loop, ✔ now ("should keep blocking a running session
through failed calls and a Core restart").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-self-review Waiting for the author to post /self-reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants