From 65fa535a6648302c38a078ff80abd86567f4a1c4 Mon Sep 17 00:00:00 2001 From: Ivan Shumkov Date: Tue, 10 Nov 2026 20:12:22 +0700 Subject: [PATCH 1/4] feat(dashmate): show what a DKG-safe stop waits for and how long `dashmate stop|restart --safe` could wait for an hour showing only "Wait for DKG window to pass", and a refused plain stop printed one long line cut at the terminal width. Under `restart`, `group stop` and `group restart` neither showed at all: listr leaves out a subtree whose tasks all lack a title, which hid the stop steps there, refusals included. While `--safe` waits it now shows the tip height, each DKG it waits for and the earliest block at which stopping is safe, with a time estimate from the network's target block spacing (mainnet, testnet, devnets; blocks only on local networks). The estimate joins every DKG this node is announced for, so it is a lower bound: Core announces membership only 8 blocks before a cycle starts. Output changes only when there is something new to say, which keeps non-TTY logs to about one line a block. A refused plain stop explains the risk in short lines with the same estimate, and suggests only the flags the running command has: `group restart` has no --force, `core reindex` no --safe. `--safe` now rides out failing Core RPC calls for about a minute (six polls in a row), reading Core's version again after each failure, and gives up after three polls in a row of DKG data dashmate cannot read instead of waiting forever. Either way it leaves the node running and says why. It no longer confirms a safe result while a DKG it saw announced still blocks, and drops a remembered DKG once Core has recorded its session. The stop and start steps of `restart`, and each node of `group stop` and `group restart`, now have titles. The command docs describe what a safe stop checks, what `--safe` shows, typical waits, and when it gives up. No bug-fix test applies to the new display; the titled steps are pinned by a test of listr's rule, and the rendering was checked in a terminal. Co-Authored-By: Claude Opus 5.5 --- .../dashmate/docs/commands/group/restart.md | 3 + packages/dashmate/docs/commands/group/stop.md | 2 + packages/dashmate/docs/commands/restart.md | 1 + packages/dashmate/docs/commands/stop.md | 15 ++ .../dashmate/src/commands/group/restart.js | 1 + packages/dashmate/src/commands/group/stop.js | 1 + .../core/quorum/checkMasternodeSafeToStop.js | 14 +- .../src/core/quorum/formatDkgStopStatus.js | 195 +++++++++++++++ .../src/core/quorum/getDkgStopBlockers.js | 59 +++++ .../src/core/quorum/waitForDKGWindowPass.js | 156 +++++++++--- .../src/listr/tasks/restartNodeTaskFactory.js | 4 + .../src/listr/tasks/stopNodeTaskFactory.js | 105 +++++--- .../quorum/checkMasternodeSafeToStop.spec.js | 46 ++++ .../core/quorum/formatDkgStopStatus.spec.js | 225 +++++++++++++++++ .../core/quorum/getDkgStopBlockers.spec.js | 50 ++++ .../core/quorum/waitForDKGWindowPass.spec.js | 229 +++++++++++++++++- .../tasks/restartNodeTaskFactory.spec.js | 17 ++ .../listr/tasks/stopNodeTaskFactory.spec.js | 149 +++++++++++- 18 files changed, 1197 insertions(+), 75 deletions(-) create mode 100644 packages/dashmate/src/core/quorum/formatDkgStopStatus.js create mode 100644 packages/dashmate/test/unit/core/quorum/formatDkgStopStatus.spec.js create mode 100644 packages/dashmate/test/unit/listr/tasks/restartNodeTaskFactory.spec.js diff --git a/packages/dashmate/docs/commands/group/restart.md b/packages/dashmate/docs/commands/group/restart.md index 95eb172db0f..78f19fd8879 100644 --- a/packages/dashmate/docs/commands/group/restart.md +++ b/packages/dashmate/docs/commands/group/restart.md @@ -27,6 +27,9 @@ The restart process happens in two phases: If the nodes are masternodes participating in quorums, restarting them during a DKG (Distributed Key Generation) session can cause issues. Use the `--safe` flag to wait for any current DKG round to complete before stopping the nodes. +See [Stopping a masternode safely](../stop.md#stopping-a-masternode-safely) for what dashmate checks and what `--safe` shows while it waits. +Nodes are stopped one at a time, and with `--safe` each waits for its own safe moment, so nodes already stopped stay down while the next one waits. +If stopping a node fails, the command ends there and the nodes it already stopped stay down until you start the group again. This command is useful when: - You've changed configuration options that require a service restart diff --git a/packages/dashmate/docs/commands/group/stop.md b/packages/dashmate/docs/commands/group/stop.md index 48ac5763943..efcc7705e68 100644 --- a/packages/dashmate/docs/commands/group/stop.md +++ b/packages/dashmate/docs/commands/group/stop.md @@ -26,6 +26,8 @@ The command handles the stop sequence appropriately to ensure that interdependen If the nodes are masternodes participating in DKG (Distributed Key Generation) session, stopping them can cause issues. Use the `--safe` flag to wait for any current DKG round to complete before stopping the nodes, or use `--force` to ignore this protection (which may result in masternodes being temporarily banned from quorums). +See [Stopping a masternode safely](../stop.md#stopping-a-masternode-safely) for what dashmate checks and what `--safe` shows while it waits. +Nodes are stopped one at a time, and with `--safe` each waits for its own safe moment, so nodes already stopped stay down while the next one waits. ## Examples diff --git a/packages/dashmate/docs/commands/restart.md b/packages/dashmate/docs/commands/restart.md index a4cf4372ed2..1b4ce85db8c 100644 --- a/packages/dashmate/docs/commands/restart.md +++ b/packages/dashmate/docs/commands/restart.md @@ -32,6 +32,7 @@ Use the `--platform` flag to restart only the platform services while keeping Co If the node is a masternode participating in a quorum, restarting during a DKG (Distributed Key Generation) session can cause issues. Use the `--safe` flag to wait for any current DKG round to complete before restarting, or use `--force` to ignore this protection (which may result in the masternode being banned and lost rewards). +See [Stopping a masternode safely](./stop.md#stopping-a-masternode-safely) for what dashmate checks and what `--safe` shows while it waits. ## Examples diff --git a/packages/dashmate/docs/commands/stop.md b/packages/dashmate/docs/commands/stop.md index 19ddb1a9f0e..0ebbfc43ec1 100644 --- a/packages/dashmate/docs/commands/stop.md +++ b/packages/dashmate/docs/commands/stop.md @@ -29,6 +29,21 @@ Using the `--safe` flag ensures the command waits for any current DKG round to c The `--force` flag can be used to forcibly stop services even if there's a DKG in progress or other conditions that would normally prevent shutdown. +## Stopping a masternode safely + +A masternode that misses its part in a DKG, the session that creates a new quorum's keys, can get a PoSe penalty. Before stopping a masternode, dashmate asks Core whether stopping now could cause that: + +- a DKG this node is a member of is still running: from its start block to the end of its Commit phase, 10 blocks for most quorum types and 20 for the 400-member ones; +- or one starts within the next 6 blocks, too soon for a restart to finish. Core 24 tells dashmate whether this node is a member; with older Core, any DKG counts. + +Without `--safe` or `--force`, the command refuses in that case and prints what blocks the stop and the earliest block at which stopping is safe. + +With `--safe`, it checks every 10 seconds and stops the node as soon as that is safe. While it waits it shows the DKGs it is waiting for and the earliest safe block, with a time estimate on mainnet, testnet and devnets. The estimate can grow: Core announces a node's place in a DKG only about 8 blocks before the DKG starts. With Core 24, an evonode usually waits under half an hour, longer around the twice-daily start of the InstantSend quorum cycle. Older Core does not report membership, so every start of that cycle blocks for about an hour and a half. Pressing Ctrl+C ends the wait and leaves the node running. + +`--safe` gives up and leaves the node running when Core stops answering for about a minute, or when it keeps reporting DKG data dashmate cannot read. + +On Core older than 24.1, dashmate checks a safe result again 15 seconds later, so stopping a masternode takes about that much longer. + ## Examples ```bash diff --git a/packages/dashmate/src/commands/group/restart.js b/packages/dashmate/src/commands/group/restart.js index 79d7ab097d7..a24c8403b55 100644 --- a/packages/dashmate/src/commands/group/restart.js +++ b/packages/dashmate/src/commands/group/restart.js @@ -48,6 +48,7 @@ export default class GroupRestartCommand extends GroupBaseCommand { // So we stop the miner first, as there's a chance that MNs will get banned // if the miner is still running when stopping them new Listr(configGroup.reverse().map((config) => ({ + title: `Stop ${config.getName()} node`, task: () => stopNodeTask(config), }))) ), diff --git a/packages/dashmate/src/commands/group/stop.js b/packages/dashmate/src/commands/group/stop.js index 50c5fc6f5b5..f214a4384fc 100644 --- a/packages/dashmate/src/commands/group/stop.js +++ b/packages/dashmate/src/commands/group/stop.js @@ -49,6 +49,7 @@ export default class GroupStopCommand extends GroupBaseCommand { // So we stop the miner first, as there's a chance that MNs will get banned // if the miner is still running when stopping them new Listr(configGroup.reverse().map((config) => ({ + title: `Stop ${config.getName()} node`, task: () => stopNodeTask(config), }))) ), diff --git a/packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js b/packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js index aeb626958f2..8569899e016 100644 --- a/packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js +++ b/packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js @@ -225,12 +225,20 @@ async function evaluate(rpcClient, coreVersion, knownNonMemberDkgs, ancestorHeig * without this node at earlier checks * @param {{ height: number, hash: string }} [options.ancestor] tip of the * latest earlier check + * @param {function(SafeStopCheck): boolean} [options.beforeConfirmation] + * called with the first, safe check before waiting to confirm it; returning + * `false` skips the confirmation and returns that check, for a caller that + * will not stop the node at its height anyway * @return {Promise} */ export default async function checkMasternodeSafeToStop( rpcClient, coreVersion, - { knownNonMemberDkgs = new Set(), ancestor } = {}, + { + knownNonMemberDkgs = new Set(), + ancestor, + beforeConfirmation = () => true, + } = {}, ) { const first = await evaluate(rpcClient, coreVersion, knownNonMemberDkgs, ancestor && ancestor.height); @@ -243,6 +251,10 @@ export default async function checkMasternodeSafeToStop( return first; } + if (!beforeConfirmation(first)) { + return first; + } + await wait(SAFE_STOP_CONFIRMATION_DELAY_MS); const nonMemberDkgs = new Set([...knownNonMemberDkgs, ...first.nonMemberDkgs]); diff --git a/packages/dashmate/src/core/quorum/formatDkgStopStatus.js b/packages/dashmate/src/core/quorum/formatDkgStopStatus.js new file mode 100644 index 00000000000..19a75862acb --- /dev/null +++ b/packages/dashmate/src/core/quorum/formatDkgStopStatus.js @@ -0,0 +1,195 @@ +import prettyMs from 'pretty-ms'; +import { MIN_BLOCKS_BEFORE_DKG } from '../../constants.js'; +import { SAFE_STOP_CONFIRMATION_DELAY_MS } from './checkMasternodeSafeToStop.js'; +import { + getBlockersAt, + getEarliestSafeHeight, + withoutRecordedSessionCount, +} from './getDkgStopBlockers.js'; + +// Lines for mainnet quorum types stay within 64 characters up to 8-digit +// heights: the default listr renderer cuts every line at the terminal width, +// less the indentation of the task it belongs to. + +function describeQuorum({ llmqType, quorumIndex }) { + const name = typeof llmqType === 'number' ? `LLMQ type ${llmqType}` : llmqType; + + return Number.isInteger(quorumIndex) && quorumIndex > 0 ? `${name} #${quorumIndex}` : name; +} + +function getStart(blocker) { + return blocker.reason === 'upcoming' ? blocker.from + MIN_BLOCKS_BEFORE_DKG : blocker.from; +} + +function describeRunningDkg(blocker) { + const end = blocker.until === null ? 'is running' : `runs until block ${blocker.until}`; + + return `${describeQuorum(blocker)} DKG at block ${getStart(blocker)} ${end}`; +} + +/** + * @param {DkgStopBlocker} blocker + * @param {number} height + * @return {string[]} + */ +function describeBlocker(blocker, height) { + switch (blocker.reason) { + case 'session': + case 'new-round': + return [describeRunningDkg(blocker)]; + case 'upcoming': + if (getStart(blocker) <= height) { + return [describeRunningDkg(blocker)]; + } + + return [`${describeQuorum(blocker)} DKG starts at block ${getStart(blocker)} ` + + `(${blocker.membershipKnown ? 'member' : 'membership unknown'})`]; + case 'imminent': + return [`A DKG starts at block ${blocker.from + MIN_BLOCKS_BEFORE_DKG} (membership not reported)`]; + case 'active-count': + return [`Core reports ${blocker.count} DKG session${blocker.count === 1 ? '' : 's'} ` + + 'in progress for this node']; + default: + return ['dashmate cannot read Core\'s DKG data:', blocker.source]; + } +} + +/** + * A DKG the wait remembered from its announcement, once Core shows a record + * of the same session. + */ +function isShownAsRecord(blocker, blockers) { + return blocker.reason === 'upcoming' + && blockers.some((other) => (other.reason === 'session' || other.reason === 'new-round') + && other.llmqType === blocker.llmqType + && other.quorumIndex === blocker.quorumIndex + && other.from === getStart(blocker)); +} + +/** + * @param {number} seconds + * @return {string} + */ +function formatDuration(seconds) { + const minutes = Math.max(1, Math.round(seconds / 60)); + + return prettyMs(minutes * 60 * 1000, { unitCount: 2 }); +} + +/** + * @param {number|null} safeHeight + * @param {number} height + * @param {number|null} blockTimeSeconds + * @return {string} + */ +function describeEarliestSafeStop(safeHeight, height, blockTimeSeconds) { + if (safeHeight === null) { + return 'Earliest safe stop: unknown'; + } + + const blocks = safeHeight - height; + const time = blockTimeSeconds ? ` (~${formatDuration(blocks * blockTimeSeconds)})` : ''; + + return `Earliest safe stop: block ${safeHeight}, in ${blocks}+ blocks${time}`; +} + +/** + * What blocks stopping until the earliest safe height, or, when that is + * unknown, what blocks it now; then the earliest safe height. + * + * @param {number} height + * @param {DkgStopBlocker[]} blockers + * @param {number|null} blockTimeSeconds + * @return {string[]} + */ +function describeBlockers(height, blockers, blockTimeSeconds) { + const safeHeight = getEarliestSafeHeight(blockers, height); + const relevant = withoutRecordedSessionCount(blockers, height); + + const listed = safeHeight === null + ? getBlockersAt(relevant, height) + : relevant.filter(({ from, until }) => (from === null || from < safeHeight) + && (until === null || until > height)); + + return [ + ...listed + .filter((blocker) => !isShownAsRecord(blocker, listed)) + .flatMap((blocker) => describeBlocker(blocker, height)), + describeEarliestSafeStop(safeHeight, height, blockTimeSeconds), + ]; +} + +/** + * Progress of `waitForDKGWindowPass` for the operator. + * + * @param {Object} status as passed to its `progressCallback` + * @param {Object} options + * @param {number|null} options.blockTimeSeconds target block spacing, or + * `null` to estimate in blocks only + * @return {string} + */ +export default function formatDkgStopStatus(status, { blockTimeSeconds }) { + switch (status.type) { + case 'blocked': + return [ + `At block ${status.height}, waiting for:`, + ...describeBlockers(status.height, status.blockers, blockTimeSeconds), + ].join('\n'); + case 'confirming': + return `Block ${status.height} looks safe; checking again ` + + `in ${SAFE_STOP_CONFIRMATION_DELAY_MS / 1000}s`; + case 'unsettled': + return 'Core\'s DKG state changed while reading it; checking again'; + default: + return `Core RPC failed (${status.failedPolls} of ${status.maxFailedPolls}), retrying:\n` + + `${status.error.message}`; + } +} + +/** + * Why a plain stop refuses to stop the masternode, and what the operator can + * do. Only the flags the running command has are suggested. + * + * @param {SafeStopCheck} check a check that is not `safe` + * @param {Object} options + * @param {number|null} options.blockTimeSeconds + * @param {boolean} options.canWait the command has `--safe` + * @param {boolean} options.canForce the command has `--force` + * @return {string} + */ +export function formatDkgStopRefusal(check, { blockTimeSeconds, canWait, canForce }) { + const unreadable = check.status === 'blocked' + && getBlockersAt(check.blockers, check.height).find(({ reason }) => reason === 'unreadable'); + + // Unreadable data does not clear on its own, so --safe is not offered. + if (unreadable) { + return [ + ...describeBlocker(unreadable, check.height), + canForce ? 'Update dashmate, or use --force to stop now' : 'Update dashmate', + ].join('\n'); + } + + const lines = check.status === 'blocked' + ? [ + 'Stopping now risks a PoSe penalty for this masternode:', + ...describeBlockers(check.height, check.blockers, blockTimeSeconds), + ] + : ['Core\'s DKG state kept changing while reading it', '(is Core still syncing?)']; + + const retry = check.status === 'blocked' && getEarliestSafeHeight(check.blockers, check.height) !== null + ? 'Retry then' + : 'Retry later'; + + if (canWait && canForce) { + lines.push(`${retry}, use --safe to wait and stop automatically,`); + lines.push('or --force to stop now and accept the PoSe risk'); + } else if (canWait) { + lines.push(`${retry}, or use --safe to wait and stop automatically`); + } else if (canForce) { + lines.push(`${retry}, or use --force to stop now and accept the PoSe risk`); + } else { + lines.push(retry); + } + + return lines.join('\n'); +} diff --git a/packages/dashmate/src/core/quorum/getDkgStopBlockers.js b/packages/dashmate/src/core/quorum/getDkgStopBlockers.js index 140942265e0..fc65da983d2 100644 --- a/packages/dashmate/src/core/quorum/getDkgStopBlockers.js +++ b/packages/dashmate/src/core/quorum/getDkgStopBlockers.js @@ -118,6 +118,65 @@ export function getBlockersAt(blockers, height) { return blockers.filter((blocker) => isBlockedAt(blocker, height)); } +/** + * Core 24.1+ reports how many sessions this node is in, not which. When the + * running sessions dashmate knows of at `height`, recorded or remembered + * from their announcement, account for that count, the count tells nothing + * more about what to wait for or how long. Only for display and estimates: + * the count still blocks. + * + * @param {DkgStopBlocker[]} blockers + * @param {number} height + * @return {DkgStopBlocker[]} + */ +export function withoutRecordedSessionCount(blockers, height) { + const runningSessions = new Set(getBlockersAt(blockers, height) + .map((blocker) => { + const start = blocker.reason === 'upcoming' + ? blocker.from + MIN_BLOCKS_BEFORE_DKG + : blocker.from; + + return ['session', 'new-round', 'upcoming'].includes(blocker.reason) && start <= height + ? `${blocker.llmqType}:${blocker.quorumIndex}:${start}` + : null; + }) + .filter((key) => key !== null)); + const recordedSessions = runningSessions.size; + + return blockers.filter((blocker) => blocker.reason !== 'active-count' + || blocker.count > recordedSessions); +} + +/** + * The first height at or above `height` that no blocker covers, or `null` + * when a blocker with no known end stands in the way. It is a lower bound: + * Core announces membership of a DKG only 8 blocks before its cycle starts, + * so a DKG not announced yet can push it further out. + * + * @param {DkgStopBlocker[]} blockers + * @param {number} height + * @return {number|null} + */ +export function getEarliestSafeHeight(blockers, height) { + const relevant = withoutRecordedSessionCount(blockers, height); + + let candidate = height; + + for (;;) { + const covering = getBlockersAt(relevant, candidate); + + if (covering.length === 0) { + return candidate; + } + + if (covering.some(({ until }) => until === null)) { + return null; + } + + candidate = Math.max(...covering.map(({ until }) => until)); + } +} + function isCount(value) { return Number.isInteger(value) && value >= 0; } diff --git a/packages/dashmate/src/core/quorum/waitForDKGWindowPass.js b/packages/dashmate/src/core/quorum/waitForDKGWindowPass.js index 5e006f8f4aa..a15956a12d7 100644 --- a/packages/dashmate/src/core/quorum/waitForDKGWindowPass.js +++ b/packages/dashmate/src/core/quorum/waitForDKGWindowPass.js @@ -4,40 +4,58 @@ import { getBlockersAt, isDkgKeyCurrent } from './getDkgStopBlockers.js'; export const CHECK_INTERVAL_MS = 10000; +// Unreadable DKG data does not clear while dashmate and Core stay as they +// are; a few polls rule out a read that caught Core mid-update. +export const MAX_UNREADABLE_POLLS = 3; + +// About a minute of failing RPC calls, enough to ride out Core being busy +// or briefly unreachable without ending a long wait. +export const MAX_FAILED_POLLS = 6; + /** * Poll Core until the masternode is safe to stop without disrupting a * DKG session. See {@link checkMasternodeSafeToStop} for the - * safety rule. The only acceptable exit is reaching a safe state, so - * that `--safe` cannot silently fall back to an unsafe restart. + * safety rule. It resolves only once that is safe, so that `--safe` cannot + * silently fall back to an unsafe restart, and throws, leaving the node + * running, when Core's DKG data stays unreadable or Core keeps failing to + * answer. * * DKGs this node takes part in, announced or running, are remembered until * their Commit phase ends. Once a session starts Core no longer announces * it, Core older than 24.1 reports it only after its DKG worker has computed * the session's members and recorded it, which can take a while at the * start of a rotation cycle, and a restarted Core keeps no record of it at - * all. DKGs announced without this node are remembered too, - * so their sessions need not wait for a record that never comes, as long as - * the chain they were announced on stays: a reorg replacing it may make this + * all. DKGs announced without this node are remembered too, so their + * sessions need not wait for a record that never comes, as long as the + * chain they were announced on stays: a reorg replacing it may make this * node a member after all. * + * `progressCallback` receives one of: + * - `{ type: 'blocked', height, blockers }`: blockers at and above `height`, + * remembered DKGs included + * - `{ type: 'confirming', height }`: a safe result waits to be confirmed + * - `{ type: 'unsettled', height }`: Core's state changed while reading it + * - `{ type: 'rpc-error', error, failedPolls, maxFailedPolls }` + * + * The error for unreadable data carries the `blocker` that could not be + * read. + * * @param {RpcClient} rpcClient + * @param {Object} [options] + * @param {function(Object)} [options.progressCallback] * @return {Promise} */ -export default async function waitForDKGWindowPass(rpcClient) { +export default async function waitForDKGWindowPass(rpcClient, { progressCallback = () => {} } = {}) { const knownDkgs = new Map(); let nonMemberDkgs = new Set(); let lastTip; - for (;;) { - // Which rule applies depends on the version, and Core may have been - // replaced since the last poll. - const coreVersion = await readCoreVersion(rpcClient); - - const check = await checkMasternodeSafeToStop(rpcClient, coreVersion, { - knownNonMemberDkgs: nonMemberDkgs, - ancestor: lastTip, - }); + let failedPolls = 0; + let unreadablePolls = 0; + // Take in what a check learned: the DKGs it saw announced, with or without + // this node, and the tip it read them on. + const remember = (check) => { nonMemberDkgs = check.isReorg ? check.nonMemberDkgs : new Set([...nonMemberDkgs, ...check.nonMemberDkgs]); @@ -46,28 +64,102 @@ export default async function waitForDKGWindowPass(rpcClient) { lastTip = check.tip; } - if (check.height !== null) { - nonMemberDkgs = new Set([...nonMemberDkgs] - .filter((key) => isDkgKeyCurrent(key, check.height))); + if (check.height === null) { + return; + } - check.blockers - .filter(({ from, until }) => from !== null && until !== null) - .forEach((dkg) => { - knownDkgs.set(`${dkg.reason}:${dkg.llmqType}:${dkg.quorumIndex}:${dkg.from}`, dkg); - }); + nonMemberDkgs = new Set([...nonMemberDkgs] + .filter((key) => isDkgKeyCurrent(key, check.height))); - knownDkgs.forEach((dkg, key) => { - if (dkg.until <= check.height) { - knownDkgs.delete(key); - } + check.blockers + .filter(({ from, until }) => from !== null && until !== null) + .forEach((dkg) => { + knownDkgs.set(`${dkg.reason}:${dkg.llmqType}:${dkg.quorumIndex}:${dkg.from}`, dkg); + }); + + knownDkgs.forEach((dkg, key) => { + if (dkg.until <= check.height) { + knownDkgs.delete(key); + } + }); + }; + + for (;;) { + let check; + + try { + // Which rule applies depends on the version, and Core may have been + // replaced since the last poll. + const coreVersion = await readCoreVersion(rpcClient); + + check = await checkMasternodeSafeToStop(rpcClient, coreVersion, { + knownNonMemberDkgs: nonMemberDkgs, + ancestor: lastTip, + beforeConfirmation: (first) => { + // Keep what the first check learned even if its confirmation is cut + // short, as by Core restarting and forgetting its announcements. + remember(first); + + // A remembered DKG blocks this height whatever the confirmation finds. + if (getBlockersAt([...knownDkgs.values()], first.height).length > 0) { + return false; + } + + progressCallback({ type: 'confirming', height: first.height }); + + return true; + }, + }); + + failedPolls = 0; + } catch (e) { + failedPolls += 1; + + if (failedPolls >= MAX_FAILED_POLLS) { + throw new Error(`Core did not answer ${MAX_FAILED_POLLS} checks in a row, ` + + `so the node was not stopped:\n${e.message}`, { cause: e }); + } + + progressCallback({ + type: 'rpc-error', error: e, failedPolls, maxFailedPolls: MAX_FAILED_POLLS, }); } - // A remembered DKG blocks even while Core syncs: Core restarted into initial - // block download may finish in time for it. - if (check.status === 'safe' - && getBlockersAt([...knownDkgs.values()], check.height).length === 0) { - return; + if (check) { + remember(check); + } + + // Unreadable data counts only over polls that read it in a row. + if (!check) { + unreadablePolls = 0; + } else if (check.status === 'unsettled') { + unreadablePolls = 0; + progressCallback({ type: 'unsettled', height: check.height }); + } else { + const blockers = [ + // Bounded blockers are among the remembered ones. + ...check.blockers.filter(({ from, until }) => from === null || until === null), + ...knownDkgs.values(), + ]; + const current = getBlockersAt(blockers, check.height); + + // A remembered DKG blocks even while Core syncs: Core restarted into + // initial block download may finish in time for it. + if (check.status === 'safe' && current.length === 0) { + return; + } + + const unreadable = current.find(({ reason }) => reason === 'unreadable'); + unreadablePolls = unreadable ? unreadablePolls + 1 : 0; + + if (unreadablePolls >= MAX_UNREADABLE_POLLS) { + const error = new Error(`dashmate cannot read Core's DKG data: ${unreadable.source}`); + error.blocker = unreadable; + + throw error; + } + + progressCallback({ type: 'blocked', height: check.height, blockers }); } await wait(CHECK_INTERVAL_MS); diff --git a/packages/dashmate/src/listr/tasks/restartNodeTaskFactory.js b/packages/dashmate/src/listr/tasks/restartNodeTaskFactory.js index 8751233e9da..571d5194c41 100644 --- a/packages/dashmate/src/listr/tasks/restartNodeTaskFactory.js +++ b/packages/dashmate/src/listr/tasks/restartNodeTaskFactory.js @@ -26,10 +26,14 @@ export default function restartNodeTaskFactory(startNodeTask, stopNodeTask, buil return buildServicesTask(config); }, }, + // Titled, because listr leaves out a subtree whose tasks all lack one, + // and with it what stopping prints while it waits or why it failed. { + title: 'Stop node', task: () => stopNodeTask(config), }, { + title: 'Start node', task: () => startNodeTask(config), }, ]); diff --git a/packages/dashmate/src/listr/tasks/stopNodeTaskFactory.js b/packages/dashmate/src/listr/tasks/stopNodeTaskFactory.js index 686417d09db..12090435c9a 100644 --- a/packages/dashmate/src/listr/tasks/stopNodeTaskFactory.js +++ b/packages/dashmate/src/listr/tasks/stopNodeTaskFactory.js @@ -4,8 +4,15 @@ import waitForDKGWindowPass from '../../core/quorum/waitForDKGWindowPass.js'; import checkMasternodeSafeToStop, { readCoreVersion, } from '../../core/quorum/checkMasternodeSafeToStop.js'; -import { getBlockersAt } from '../../core/quorum/getDkgStopBlockers.js'; -import { NETWORK_LOCAL } from '../../constants.js'; +import formatDkgStopStatus, { + formatDkgStopRefusal, +} from '../../core/quorum/formatDkgStopStatus.js'; +import { + NETWORK_DEVNET, + NETWORK_LOCAL, + NETWORK_MAINNET, + NETWORK_TESTNET, +} from '../../constants.js'; /** * Group commands stop a local network's miner before its masternodes. Without @@ -28,6 +35,29 @@ function isLocalNetworkHalted(config, ctx) { // consistent DKG state, such as while Core catches up after a restart. const SAFE_STOP_CHECK_ATTEMPTS = 3; +// Dash's target block spacing, which difficulty adjustment holds mainnet and +// testnet to on average. +const TARGET_BLOCK_SECONDS = 150; + +/** + * Block spacing to estimate waits with, or `null` where blocks come on their + * own schedule, as on local networks. + * + * @param {Config} config + * @return {number|null} + */ +function getTargetBlockSeconds(config) { + switch (config.get('network')) { + case NETWORK_MAINNET: + case NETWORK_TESTNET: + return TARGET_BLOCK_SECONDS; + case NETWORK_DEVNET: + return config.get('core.devnet.powTargetSpacing'); + default: + return null; + } +} + /** * @param {DockerCompose} dockerCompose * @param {createRpcClient} createRpcClient @@ -72,7 +102,7 @@ export default function stopNodeTaskFactory( title: 'Check node is participating in DKG', enabled: (ctx) => config.get('core.masternode.enable') && !ctx.isForce && !ctx.isSafe && !ctx.platformOnly && !isLocalNetworkHalted(config, ctx), - task: async () => { + task: async (ctx) => { const rpcClient = createRpcClient({ port: config.get('core.rpc.port'), user: 'dashmate', @@ -91,25 +121,14 @@ export default function stopNodeTaskFactory( } } - const unreadable = check.status === 'blocked' - && getBlockersAt(check.blockers, check.height) - .find(({ reason }) => reason === 'unreadable'); - - if (unreadable) { - throw new Error(`dashmate cannot read Core's DKG state (${unreadable.source}). ` - + 'Update dashmate, or continue with --force'); - } - - if (check.status === 'blocked') { - throw new Error('Your node is currently participating in a DKG exchange session ' - + '(or one is about to start) and stopping it right now may result in a PoSe ban. ' - + 'Try again later, or continue with --force or --safe flags'); - } - - if (check.status === 'unsettled') { - throw new Error('The chain kept advancing while checking for DKG sessions ' - + '(is Core still syncing?). ' - + 'Try again later, or continue with --force or --safe flags'); + if (check.status !== 'safe') { + // Commands pass the flags they have, so `undefined` means the + // running command offers no such flag. + throw new Error(formatDkgStopRefusal(check, { + blockTimeSeconds: getTargetBlockSeconds(config), + canWait: ctx.isSafe !== undefined, + canForce: ctx.isForce !== undefined, + })); } }, }, @@ -117,12 +136,42 @@ export default function stopNodeTaskFactory( title: 'Wait for DKG window to pass', enabled: (ctx) => config.get('core.masternode.enable') && !ctx.isForce && ctx.isSafe && !ctx.platformOnly && !isLocalNetworkHalted(config, ctx), - task: async () => waitForDKGWindowPass(createRpcClient({ - port: config.get('core.rpc.port'), - user: 'dashmate', - pass: config.get('core.rpc.users.dashmate.password'), - host: await getConnectionHost(config, 'core', 'core.rpc.host'), - })), + task: async (ctx, task) => { + const rpcClient = createRpcClient({ + port: config.get('core.rpc.port'), + user: 'dashmate', + pass: config.get('core.rpc.users.dashmate.password'), + host: await getConnectionHost(config, 'core', 'core.rpc.host'), + }); + + const blockTimeSeconds = getTargetBlockSeconds(config); + let output; + + try { + await waitForDKGWindowPass(rpcClient, { + // Without a terminal, every output change is printed as a new line, + // so the output changes only when there is something new to say. + progressCallback: (status) => { + const text = formatDkgStopStatus(status, { blockTimeSeconds }); + + if (text !== output) { + output = text; + // eslint-disable-next-line no-param-reassign + task.output = text; + } + }, + }); + } catch (e) { + if (!e.blocker) { + throw e; + } + + throw new Error(formatDkgStopRefusal( + { status: 'blocked', height: null, blockers: [e.blocker] }, + { blockTimeSeconds, canWait: false, canForce: ctx.isForce !== undefined }, + ), { cause: e }); + } + }, }, { title: `Stopping ${config.getName()} node`, diff --git a/packages/dashmate/test/unit/core/quorum/checkMasternodeSafeToStop.spec.js b/packages/dashmate/test/unit/core/quorum/checkMasternodeSafeToStop.spec.js index 8376867488e..2277be0d6e0 100644 --- a/packages/dashmate/test/unit/core/quorum/checkMasternodeSafeToStop.spec.js +++ b/packages/dashmate/test/unit/core/quorum/checkMasternodeSafeToStop.spec.js @@ -209,6 +209,52 @@ describe('checkMasternodeSafeToStop', () => { expect(rpcClient.quorum.withArgs('dkginfo')).to.have.been.calledOnce(); }); + describe('beforeConfirmation', () => { + function startCheckWith(beforeConfirmation) { + const state = {}; + checkMasternodeSafeToStop(rpcClient, core.version, { beforeConfirmation }) + .then((result) => { state.result = result; }); + + return state; + } + + it('should be asked before the confirmation delay', async function it() { + const beforeConfirmation = this.sinon.stub().returns(true); + + const state = startCheckWith(beforeConfirmation); + await clock.tickAsync(0); + + expect(beforeConfirmation).to.have.been.calledOnceWith(this.sinon.match({ status: 'safe', height: 1012 })); + expect(state.result).to.be.undefined(); + }); + + it('should skip the confirmation when the caller will not stop at that height anyway', async () => { + const state = startCheckWith(() => false); + await clock.tickAsync(0); + + expect(state.result).to.include({ status: 'safe', height: 1012 }); + expect(rpcClient.quorum.withArgs('dkginfo')).to.have.been.calledOnce(); + }); + + it('should not be asked about a confirmation that does not happen', async function it() { + const beforeConfirmation = this.sinon.stub().returns(true); + + core.version = CORE_24_1; + startCheckWith(beforeConfirmation); + await clock.tickAsync(0); + + core.version = CORE_23; + core.dkgInfo = { active_dkgs: 0, next_dkg: 3 }; + startCheckWith(beforeConfirmation); + await clock.tickAsync(0); + + core.isSyncing = true; + startCheckWith(beforeConfirmation); + await clock.tickAsync(0); + + expect(beforeConfirmation).to.not.have.been.called(); + }); + }); it('should report unsettled when a reorg replaced the first check\'s tip and the chain grew', async () => { // The replacing branch can put this node in a DKG starting on it that // Core has not recorded yet. diff --git a/packages/dashmate/test/unit/core/quorum/formatDkgStopStatus.spec.js b/packages/dashmate/test/unit/core/quorum/formatDkgStopStatus.spec.js new file mode 100644 index 00000000000..3de8c932d12 --- /dev/null +++ b/packages/dashmate/test/unit/core/quorum/formatDkgStopStatus.spec.js @@ -0,0 +1,225 @@ +import formatDkgStopStatus, { + formatDkgStopRefusal, +} from '../../../../src/core/quorum/formatDkgStopStatus.js'; + +// Eight-digit heights, the longest these lines are made to fit. +const HEIGHT = 12345678; +const MAX_LINE_LENGTH = 64; + +const MAINNET_BLOCK_SECONDS = 150; + +function session(from, extra = {}) { + return { + reason: 'session', llmqType: 'llmq_100_67', quorumIndex: 0, from, until: from + 10, ...extra, + }; +} + +function upcoming(start, extra = {}) { + return { + reason: 'upcoming', + llmqType: 'llmq_100_67', + quorumIndex: 0, + membershipKnown: true, + from: start - 6, + until: start + 10, + ...extra, + }; +} + +function expectLinesFit(text) { + text.split('\n').forEach((line) => { + expect(line.length, `"${line}" is too long`).to.be.at.most(MAX_LINE_LENGTH); + }); +} + +function blocked(blockers, blockTimeSeconds = MAINNET_BLOCK_SECONDS) { + const text = formatDkgStopStatus({ type: 'blocked', height: HEIGHT, blockers }, { blockTimeSeconds }); + expectLinesFit(text); + + return text.split('\n'); +} + +describe('formatDkgStopStatus', () => { + it('should say what blocks and when stopping becomes safe', () => { + expect(blocked([session(HEIGHT - 3)])).to.deep.equal([ + `At block ${HEIGHT}, waiting for:`, + `llmq_100_67 DKG at block ${HEIGHT - 3} runs until block ${HEIGHT + 7}`, + `Earliest safe stop: block ${HEIGHT + 7}, in 7+ blocks (~18m)`, + ]); + }); + + it('should describe every kind of blocker on a line that fits the renderer', () => { + expect(blocked([ + session(HEIGHT - 3, { llmqType: 'llmq_60_75', quorumIndex: 31 }), + { + reason: 'new-round', llmqType: 'llmq_400_85', quorumIndex: 0, from: HEIGHT, until: HEIGHT + 20, + }, + upcoming(HEIGHT + 4, { llmqType: 'llmq_60_75', quorumIndex: 31, membershipKnown: false }), + upcoming(HEIGHT + 6, { llmqType: 250, until: null }), + { reason: 'imminent', from: HEIGHT - 1, until: null }, + { + reason: 'active-count', count: 3, from: null, until: null, + }, + { + reason: 'unreadable', source: 'quorum dkgstatus session llmq_future_unknown', from: null, until: null, + }, + ])).to.deep.equal([ + `At block ${HEIGHT}, waiting for:`, + `llmq_60_75 #31 DKG at block ${HEIGHT - 3} runs until block ${HEIGHT + 7}`, + `llmq_400_85 DKG at block ${HEIGHT} runs until block ${HEIGHT + 20}`, + `llmq_60_75 #31 DKG starts at block ${HEIGHT + 4} (membership unknown)`, + `LLMQ type 250 DKG starts at block ${HEIGHT + 6} (member)`, + `A DKG starts at block ${HEIGHT + 5} (membership not reported)`, + 'Core reports 3 DKG sessions in progress for this node', + 'dashmate cannot read Core\'s DKG data:', + 'quorum dkgstatus session llmq_future_unknown', + 'Earliest safe stop: unknown', + ]); + }); + + it('should list the later DKGs the estimate waits for', () => { + expect(blocked([ + session(HEIGHT - 3), + upcoming(HEIGHT + 12, { llmqType: 'llmq_60_75', quorumIndex: 5 }), + upcoming(HEIGHT + 100), + ])).to.deep.equal([ + `At block ${HEIGHT}, waiting for:`, + `llmq_100_67 DKG at block ${HEIGHT - 3} runs until block ${HEIGHT + 7}`, + `llmq_60_75 #5 DKG starts at block ${HEIGHT + 12} (member)`, + `Earliest safe stop: block ${HEIGHT + 22}, in 22+ blocks (~55m)`, + ]); + }); + + it('should describe a DKG it saw announced as running once it has started', () => { + expect(blocked([upcoming(HEIGHT - 2)])).to.deep.equal([ + `At block ${HEIGHT}, waiting for:`, + `llmq_100_67 DKG at block ${HEIGHT - 2} runs until block ${HEIGHT + 8}`, + `Earliest safe stop: block ${HEIGHT + 8}, in 8+ blocks (~20m)`, + ]); + }); + + it('should list a DKG once when Core shows a record of the session it saw announced', () => { + expect(blocked([ + { ...session(HEIGHT - 2), reason: 'new-round' }, + upcoming(HEIGHT - 2), + ])).to.deep.equal([ + `At block ${HEIGHT}, waiting for:`, + `llmq_100_67 DKG at block ${HEIGHT - 2} runs until block ${HEIGHT + 8}`, + `Earliest safe stop: block ${HEIGHT + 8}, in 8+ blocks (~20m)`, + ]); + }); + + it('should leave out Core\'s session count when the sessions it lists account for it', () => { + const count = { + reason: 'active-count', count: 1, from: null, until: null, + }; + + expect(blocked([session(HEIGHT - 3), count])).to.deep.equal([ + `At block ${HEIGHT}, waiting for:`, + `llmq_100_67 DKG at block ${HEIGHT - 3} runs until block ${HEIGHT + 7}`, + `Earliest safe stop: block ${HEIGHT + 7}, in 7+ blocks (~18m)`, + ]); + expect(blocked([count]).slice(1)).to.deep.equal([ + 'Core reports 1 DKG session in progress for this node', + 'Earliest safe stop: unknown', + ]); + }); + + it('should give long waits in hours and short ones as at least a minute', () => { + const wait = (blocks, blockTimeSeconds) => blocked([session(HEIGHT - 10 + blocks)], blockTimeSeconds).at(-1); + + expect(wait(10, MAINNET_BLOCK_SECONDS)).to.match(/in 10\+ blocks \(~25m\)$/); + expect(wait(4, 20)).to.match(/in 4\+ blocks \(~1m\)$/); + expect(wait(1, 20)).to.match(/in 1\+ blocks \(~1m\)$/); + expect(wait(10, 600)).to.match(/in 10\+ blocks \(~1h 40m\)$/); + expect(wait(10, null)).to.match(/in 10\+ blocks$/); + }); + + it('should describe a safe result being confirmed and a state that changed', () => { + expect(formatDkgStopStatus({ type: 'confirming', height: HEIGHT }, { blockTimeSeconds: null })) + .to.equal(`Block ${HEIGHT} looks safe; checking again in 15s`); + expect(formatDkgStopStatus({ type: 'unsettled', height: null }, { blockTimeSeconds: null })) + .to.equal('Core\'s DKG state changed while reading it; checking again'); + }); + + it('should describe a failed RPC call it retries, with the cause on a line of its own', () => { + const text = formatDkgStopStatus({ + type: 'rpc-error', error: new Error('Work queue depth exceeded'), failedPolls: 2, maxFailedPolls: 6, + }, { blockTimeSeconds: null }); + + expect(text.split('\n')).to.deep.equal([ + 'Core RPC failed (2 of 6), retrying:', + 'Work queue depth exceeded', + ]); + }); +}); + +describe('formatDkgStopRefusal', () => { + function refuse(check, flags) { + const text = formatDkgStopRefusal(check, { blockTimeSeconds: MAINNET_BLOCK_SECONDS, ...flags }); + expectLinesFit(text); + + return text.split('\n'); + } + + const inSession = { status: 'blocked', height: HEIGHT, blockers: [session(HEIGHT - 3)] }; + + it('should explain the PoSe risk and offer both flags when the command has them', () => { + expect(refuse(inSession, { canWait: true, canForce: true })).to.deep.equal([ + 'Stopping now risks a PoSe penalty for this masternode:', + `llmq_100_67 DKG at block ${HEIGHT - 3} runs until block ${HEIGHT + 7}`, + `Earliest safe stop: block ${HEIGHT + 7}, in 7+ blocks (~18m)`, + 'Retry then, use --safe to wait and stop automatically,', + 'or --force to stop now and accept the PoSe risk', + ]); + }); + + it('should offer only the flags the command has', () => { + const hints = (flags) => refuse(inSession, flags).at(-1); + + expect(hints({ canWait: true, canForce: false })) + .to.equal('Retry then, or use --safe to wait and stop automatically'); + expect(hints({ canWait: false, canForce: true })) + .to.equal('Retry then, or use --force to stop now and accept the PoSe risk'); + expect(hints({ canWait: false, canForce: false })).to.equal('Retry then'); + }); + + it('should say to retry later when it cannot tell when stopping becomes safe', () => { + const lines = refuse({ + status: 'blocked', height: HEIGHT, blockers: [{ reason: 'imminent', from: HEIGHT - 4, until: null }], + }, { canWait: false, canForce: true }); + + expect(lines.slice(-2)).to.deep.equal([ + 'Earliest safe stop: unknown', + 'Retry later, or use --force to stop now and accept the PoSe risk', + ]); + }); + + it('should not offer --safe for data dashmate cannot read, which does not clear on its own', () => { + const unreadable = { + status: 'blocked', + height: HEIGHT, + blockers: [{ + reason: 'unreadable', source: 'quorum dkginfo', from: null, until: null, + }], + }; + + expect(refuse(unreadable, { canWait: true, canForce: true })).to.deep.equal([ + 'dashmate cannot read Core\'s DKG data:', + 'quorum dkginfo', + 'Update dashmate, or use --force to stop now', + ]); + expect(refuse(unreadable, { canWait: true, canForce: false }).at(-1)).to.equal('Update dashmate'); + }); + + it('should explain a state that kept changing', () => { + expect(refuse({ status: 'unsettled', height: null, blockers: [] }, { + canWait: true, canForce: true, + })).to.deep.equal([ + 'Core\'s DKG state kept changing while reading it', + '(is Core still syncing?)', + 'Retry later, use --safe to wait and stop automatically,', + 'or --force to stop now and accept the PoSe risk', + ]); + }); +}); diff --git a/packages/dashmate/test/unit/core/quorum/getDkgStopBlockers.spec.js b/packages/dashmate/test/unit/core/quorum/getDkgStopBlockers.spec.js index 1886d11713a..43e9031cc77 100644 --- a/packages/dashmate/test/unit/core/quorum/getDkgStopBlockers.spec.js +++ b/packages/dashmate/test/unit/core/quorum/getDkgStopBlockers.spec.js @@ -1,6 +1,7 @@ import getDkgStopBlockers, { getAnnouncedNonMemberDkgs, getBlockersAt, + getEarliestSafeHeight, hasMembershipActiveDkgs, LLMQ_TYPES, } from '../../../../src/core/quorum/getDkgStopBlockers.js'; @@ -459,6 +460,55 @@ describe('getDkgStopBlockers', () => { }); }); + describe('getEarliestSafeHeight', () => { + function blocker(from, until, extra = {}) { + return { + reason: 'session', from, until, ...extra, + }; + } + + it('should be the current height when nothing blocks', () => { + expect(getEarliestSafeHeight([blocker(HEIGHT + 3, HEIGHT + 10)], HEIGHT)).to.equal(HEIGHT); + }); + + it('should be where the last overlapping blocker ends', () => { + // An llmq_400_60 session ends at +16, but the next hourly DKG of this + // node starts blocking at +14 and runs until +30. + const blockers = [ + blocker(HEIGHT - 4, HEIGHT + 16), + blocker(HEIGHT + 14, HEIGHT + 30, { reason: 'upcoming' }), + blocker(HEIGHT + 40, HEIGHT + 50, { reason: 'upcoming' }), + ]; + + expect(getEarliestSafeHeight(blockers, HEIGHT)).to.equal(HEIGHT + 30); + }); + + it('should be unknown while a blocker with no known end stands in the way', () => { + expect(getEarliestSafeHeight([ + blocker(HEIGHT - 1, HEIGHT + 5), + blocker(HEIGHT + 3, null, { reason: 'imminent' }), + ], HEIGHT)).to.be.null(); + }); + + it('should ignore a blocker with no known end that starts after it', () => { + expect(getEarliestSafeHeight([ + blocker(HEIGHT - 1, HEIGHT + 5), + blocker(HEIGHT + 8, null, { reason: 'imminent' }), + ], HEIGHT)).to.equal(HEIGHT + 5); + }); + + it('should not wait for Core\'s session count when recorded sessions account for it', () => { + const count = { + reason: 'active-count', count: 1, from: null, until: null, + }; + + expect(getEarliestSafeHeight([blocker(HEIGHT - 1, HEIGHT + 5), count], HEIGHT)) + .to.equal(HEIGHT + 5); + expect(getEarliestSafeHeight([blocker(HEIGHT - 1, HEIGHT + 5), { ...count, count: 2 }], HEIGHT)) + .to.be.null(); + }); + }); + describe('LLMQ table vs Dash Core src/llmq/params.h', () => { it('should match every LLMQ type Core defines', () => { expect(LLMQ_TYPES.map(({ diff --git a/packages/dashmate/test/unit/core/quorum/waitForDKGWindowPass.spec.js b/packages/dashmate/test/unit/core/quorum/waitForDKGWindowPass.spec.js index d82c7dcd451..c725fcbc477 100644 --- a/packages/dashmate/test/unit/core/quorum/waitForDKGWindowPass.spec.js +++ b/packages/dashmate/test/unit/core/quorum/waitForDKGWindowPass.spec.js @@ -1,5 +1,7 @@ import waitForDKGWindowPass, { CHECK_INTERVAL_MS, + MAX_FAILED_POLLS, + MAX_UNREADABLE_POLLS, } from '../../../../src/core/quorum/waitForDKGWindowPass.js'; import { SAFE_STOP_CONFIRMATION_DELAY_MS } from '../../../../src/core/quorum/checkMasternodeSafeToStop.js'; import getCoreRpcMock from '../../../../src/test/mock/getCoreRpcMock.js'; @@ -13,6 +15,27 @@ const CORE_24_1 = 240100; // read by the next poll. const NEXT_POLL_MS = CHECK_INTERVAL_MS + SAFE_STOP_CONFIRMATION_DELAY_MS + 1; +/** + * A Core RPC client that, while `core.down` is set, rejects every call the + * way a restarting Core does. + */ +function getRestartableCoreRpcMock(sinon, core) { + const running = getCoreRpcMock(sinon, core); + const guard = (method) => sinon.stub().callsFake(async (...args) => { + if (core.down) { + throw new Error('connect ECONNREFUSED'); + } + + return running[method](...args); + }); + + return { + getNetworkInfo: guard('getNetworkInfo'), + getBlockchainInfo: guard('getBlockchainInfo'), + quorum: guard('quorum'), + }; +} + function track(promise) { const state = { resolved: false, error: undefined }; promise.then(() => { state.resolved = true; }, (e) => { state.error = e; }); @@ -170,7 +193,9 @@ describe('waitForDKGWindowPass', () => { expect(waiting.resolved).to.be.true(); }); - it('should keep waiting while Core reports a session dashmate cannot read', async function it() { + it('should give up when Core keeps reporting DKG data dashmate cannot read', async function it() { + // An unknown quorum type does not go away while dashmate and Core stay + // as they are, so waiting for it would never end. const rpcClient = getCoreRpcMock(this.sinon, { version: CORE_23, height: 1000, @@ -183,7 +208,207 @@ describe('waitForDKGWindowPass', () => { const waiting = track(waitForDKGWindowPass(rpcClient)); await clock.tickAsync(CHECK_INTERVAL_MS * 10); - await expectStillPolling(waiting, rpcClient); + expect(waiting.error.message).to.include('dashmate cannot read Core\'s DKG data') + .and.include('llmq_future_unknown'); + expect(waiting.error.blocker).to.include({ reason: 'unreadable' }); + expect(rpcClient.quorum.withArgs('dkginfo')).to.have.callCount(MAX_UNREADABLE_POLLS); + }); + + it('should count only unreadable polls in a row, starting again after a readable one', async function it() { + const unreadable = { active_dkgs: 0, next_dkg: 20, upcoming_dkgs: {} }; + const blocked = { active_dkgs: 1, next_dkg: 20, upcoming_dkgs: [] }; + const core = { + version: CORE_24_1, height: 1000, dkgInfo: unreadable, dkgStatus: { session: [] }, + }; + const rpcClient = getCoreRpcMock(this.sinon, core); + + const waiting = track(waitForDKGWindowPass(rpcClient)); + await clock.tickAsync(CHECK_INTERVAL_MS * (MAX_UNREADABLE_POLLS - 1) - 1); + core.dkgInfo = blocked; + await clock.tickAsync(CHECK_INTERVAL_MS); + core.dkgInfo = unreadable; + await clock.tickAsync(CHECK_INTERVAL_MS * (MAX_UNREADABLE_POLLS - 1)); + + expect(waiting.error).to.be.undefined(); + expect(waiting.resolved).to.be.false(); + expect(rpcClient.quorum.withArgs('dkginfo')).to.have.callCount(MAX_UNREADABLE_POLLS * 2 - 1); + }); + + it('should keep waiting through a single read dashmate cannot make sense of', async function it() { + const core = { + version: CORE_24_1, + height: 1000, + dkgInfo: { active_dkgs: 0, next_dkg: 20, upcoming_dkgs: {} }, + dkgStatus: { session: [] }, + }; + const rpcClient = getCoreRpcMock(this.sinon, core); + + const waiting = track(waitForDKGWindowPass(rpcClient)); + await clock.tickAsync(1); + + expect(waiting.resolved).to.be.false(); + + core.dkgInfo = { active_dkgs: 0, next_dkg: 20, upcoming_dkgs: [] }; + await clock.tickAsync(NEXT_POLL_MS); + + expect(waiting.resolved).to.be.true(); + }); + + it('should report what it waits for, a DKG it saw announced included', async function it() { + const progressCallback = this.sinon.stub(); + const start = 24 * 5000; + const core = { + version: CORE_24_0, + height: start - 1, + dkgInfo: { + active_dkgs: 0, + next_dkg: 1, + upcoming_dkgs: [{ + llmqType: 4, quorumIndex: 0, quorumHeight: start, blocksUntilStart: 1, known: true, isMember: true, + }], + }, + dkgStatus: { session: [] }, + }; + const rpcClient = getCoreRpcMock(this.sinon, core); + + waitForDKGWindowPass(rpcClient, { progressCallback }); + await clock.tickAsync(1); + + core.height = start; + core.dkgInfo = { active_dkgs: 0, next_dkg: 24, upcoming_dkgs: [] }; + await clock.tickAsync(NEXT_POLL_MS); + + const announced = { + reason: 'upcoming', llmqType: 'llmq_100_67', from: start - 6, until: start + 10, + }; + expect(progressCallback.firstCall.args[0]).to.deep.include({ type: 'blocked', height: start - 1 }); + expect(progressCallback.firstCall.args[0].blockers).to.have.lengthOf(1); + expect(progressCallback.firstCall.args[0].blockers[0]).to.include(announced); + expect(progressCallback.lastCall.args[0]).to.deep.include({ type: 'blocked', height: start }); + expect(progressCallback.lastCall.args[0].blockers).to.have.lengthOf(1); + expect(progressCallback.lastCall.args[0].blockers[0]).to.include(announced); + // The remembered DKG blocks the start block whatever a confirmation finds. + expect(progressCallback).to.not.have.been.calledWithMatch({ type: 'confirming' }); + }); + + it('should rely on Core\'s record once it has recorded a DKG it saw announced', async function it() { + const progressCallback = this.sinon.stub(); + const start = 24 * 5000; + const core = { + version: CORE_24_0, + height: start - 1, + dkgInfo: { + active_dkgs: 0, + next_dkg: 1, + upcoming_dkgs: [{ + llmqType: 4, quorumIndex: 0, quorumHeight: start, blocksUntilStart: 1, known: true, isMember: true, + }], + }, + dkgStatus: { session: [] }, + }; + const rpcClient = getCoreRpcMock(this.sinon, core); + + waitForDKGWindowPass(rpcClient, { progressCallback }); + await clock.tickAsync(1); + + core.height = start + 2; + core.dkgInfo = { active_dkgs: 1, next_dkg: 22, upcoming_dkgs: [] }; + core.dkgStatus = { + session: [{ llmqType: 'llmq_100_67', quorumIndex: 0, status: { quorumHeight: start } }], + }; + await clock.tickAsync(NEXT_POLL_MS); + + expect(progressCallback.lastCall.args[0].blockers.map(({ reason }) => reason)) + .to.deep.equal(['session']); + }); + + it('should report when Core\'s state changed while reading it', async function it() { + const progressCallback = this.sinon.stub(); + const core = { + version: CORE_24_1, + height: 1000, + dkgInfo: { active_dkgs: 1, next_dkg: 20, upcoming_dkgs: [] }, + dkgStatus: { session: [] }, + }; + core.beforeDkgInfo = () => { core.height += 1; }; + const rpcClient = getCoreRpcMock(this.sinon, core); + + waitForDKGWindowPass(rpcClient, { progressCallback }); + await clock.tickAsync(1); + + expect(progressCallback).to.have.been.calledOnceWith({ type: 'unsettled', height: null }); + }); + + it('should ride out Core restarting and judge by the version it comes back as', async function it() { + // Core 24.0 needs 15 s to confirm a safe result; Core 24.1 does not. + const progressCallback = this.sinon.stub(); + const core = { + version: CORE_24_0, + height: 1005, + dkgInfo: { active_dkgs: 1, next_dkg: 19, upcoming_dkgs: [] }, + dkgStatus: { + session: [{ llmqType: 'llmq_test_platform', quorumIndex: 0, status: { quorumHeight: 1000 } }], + }, + }; + const rpcClient = getRestartableCoreRpcMock(this.sinon, core); + + const waiting = track(waitForDKGWindowPass(rpcClient, { progressCallback })); + await clock.tickAsync(1); + + core.down = true; + await clock.tickAsync(CHECK_INTERVAL_MS * 2); + + expect(waiting.error).to.be.undefined(); + expect(progressCallback).to.have.been.calledWithMatch({ + type: 'rpc-error', failedPolls: 2, maxFailedPolls: MAX_FAILED_POLLS, + }); + + core.down = false; + core.version = CORE_24_1; + core.height = 1010; + core.dkgInfo = { active_dkgs: 0, next_dkg: 14, upcoming_dkgs: [] }; + await clock.tickAsync(CHECK_INTERVAL_MS + 1); + + expect(waiting.resolved).to.be.true(); + }); + + it('should count only failed RPC calls in a row, starting again after one that works', async function it() { + const core = { + version: CORE_24_1, + height: 1000, + dkgInfo: { active_dkgs: 1, next_dkg: 20, upcoming_dkgs: [] }, + dkgStatus: { session: [] }, + down: true, + }; + const rpcClient = getRestartableCoreRpcMock(this.sinon, core); + + const waiting = track(waitForDKGWindowPass(rpcClient)); + await clock.tickAsync(CHECK_INTERVAL_MS * (MAX_FAILED_POLLS - 1) - 1); + core.down = false; + await clock.tickAsync(CHECK_INTERVAL_MS); + core.down = true; + await clock.tickAsync(CHECK_INTERVAL_MS * (MAX_FAILED_POLLS - 1)); + + expect(waiting.error).to.be.undefined(); + expect(waiting.resolved).to.be.false(); + }); + + it('should give up, leaving the node running, after too many failed RPC calls in a row', async function it() { + const core = { + version: CORE_24_1, + height: 1000, + dkgInfo: { active_dkgs: 0, next_dkg: 20, upcoming_dkgs: [] }, + dkgStatus: { session: [] }, + down: true, + }; + const rpcClient = getRestartableCoreRpcMock(this.sinon, core); + + const waiting = track(waitForDKGWindowPass(rpcClient)); + await clock.tickAsync(CHECK_INTERVAL_MS * (MAX_FAILED_POLLS + 1)); + + expect(waiting.error.message).to.equal(`Core did not answer ${MAX_FAILED_POLLS} checks in a row, ` + + 'so the node was not stopped:\nconnect ECONNREFUSED'); + expect(rpcClient.getNetworkInfo).to.have.callCount(MAX_FAILED_POLLS); }); it('should stop waiting on Core 24.0 when active_dkgs counts only finished sessions', async function it() { diff --git a/packages/dashmate/test/unit/listr/tasks/restartNodeTaskFactory.spec.js b/packages/dashmate/test/unit/listr/tasks/restartNodeTaskFactory.spec.js new file mode 100644 index 00000000000..f6a26e1ae46 --- /dev/null +++ b/packages/dashmate/test/unit/listr/tasks/restartNodeTaskFactory.spec.js @@ -0,0 +1,17 @@ +import restartNodeTaskFactory from '../../../../src/listr/tasks/restartNodeTaskFactory.js'; + +describe('restartNodeTaskFactory', () => { + it('should title a step so listr renders what stopping prints and why it failed', function it() { + // listr's default renderer leaves out a subtree whose tasks all lack a + // title, and with it a DKG wait's progress or a refused stop's reason. + const restartNodeTask = restartNodeTaskFactory( + this.sinon.stub(), + this.sinon.stub(), + this.sinon.stub(), + ); + + const config = { get: this.sinon.stub().returns(null), getName: () => 'mainnet' }; + + expect(restartNodeTask(config).tasks.some((task) => task.hasTitle())).to.be.true(); + }); +}); diff --git a/packages/dashmate/test/unit/listr/tasks/stopNodeTaskFactory.spec.js b/packages/dashmate/test/unit/listr/tasks/stopNodeTaskFactory.spec.js index a8c2fe02924..0a9d9fbd308 100644 --- a/packages/dashmate/test/unit/listr/tasks/stopNodeTaskFactory.spec.js +++ b/packages/dashmate/test/unit/listr/tasks/stopNodeTaskFactory.spec.js @@ -1,6 +1,7 @@ -import { Listr } from 'listr2'; +import { Listr, ListrEventType } from 'listr2'; import stopNodeTaskFactory from '../../../../src/listr/tasks/stopNodeTaskFactory.js'; import { SAFE_STOP_CONFIRMATION_DELAY_MS } from '../../../../src/core/quorum/checkMasternodeSafeToStop.js'; +import { CHECK_INTERVAL_MS } from '../../../../src/core/quorum/waitForDKGWindowPass.js'; import getConfigMock from '../../../../src/test/mock/getConfigMock.js'; import getCoreRpcMock from '../../../../src/test/mock/getCoreRpcMock.js'; @@ -27,6 +28,7 @@ describe('stopNodeTaskFactory', () => { config = getConfigMock(this.sinon); config.get.withArgs('core.masternode.enable').returns(true); + config.get.withArgs('network').returns('mainnet'); config.getName.returns('mainnet'); dockerCompose = { @@ -46,14 +48,38 @@ describe('stopNodeTaskFactory', () => { }); /** - * Run a plain stop, advancing the fake clock past any confirmation delays. + * Run the stop task as a command would, collecting what the DKG wait + * prints. `ctx` holds the flags of the running command. */ - async function stop() { - const state = {}; - new Listr([{ task: () => stopNodeTask(config) }], { renderer: 'silent' }) - .run({ isForce: false, isSafe: false, platformOnly: false }) + function run(ctx) { + const state = { outputs: [] }; + + new Listr([{ + task: () => { + const listr = stopNodeTask(config); + listr.tasks + .find(({ title }) => title === 'Wait for DKG window to pass') + .subscribe(({ type, data }) => { + if (type === ListrEventType.DATA) { + state.outputs.push(data); + } + }); + + return listr; + }, + }], { renderer: 'silent' }) + .run({ platformOnly: false, ...ctx }) .then(() => { state.stopped = true; }, (e) => { state.error = e; }); + return state; + } + + /** + * Run a plain stop, advancing the fake clock past any confirmation delays. + */ + async function stop(ctx = { isForce: false, isSafe: false }) { + const state = run(ctx); + await clock.tickAsync(SAFE_STOP_CONFIRMATION_DELAY_MS * 4); return state; @@ -66,15 +92,37 @@ describe('stopNodeTaskFactory', () => { expect(dockerCompose.stop).to.have.been.calledOnce(); }); - it('should refuse to stop a masternode during its DKG session', async () => { + it('should refuse to stop a masternode during its DKG session and say until when', async () => { core.dkgInfo = { active_dkgs: 1, next_dkg: 12, upcoming_dkgs: [] }; + core.dkgStatus = { + session: [{ llmqType: 'llmq_100_67', quorumIndex: 0, status: { quorumHeight: 1008 } }], + }; const { error } = await stop(); - expect(error.message).to.include('may result in a PoSe ban'); + expect(error.message.split('\n')).to.deep.equal([ + 'Stopping now risks a PoSe penalty for this masternode:', + 'llmq_100_67 DKG at block 1008 runs until block 1018', + 'Earliest safe stop: block 1018, in 6+ blocks (~15m)', + 'Retry then, use --safe to wait and stop automatically,', + 'or --force to stop now and accept the PoSe risk', + ]); expect(dockerCompose.stop).to.not.have.been.called(); }); + it('should only suggest the flags the running command has', async () => { + // group restart has --safe but no --force; core reindex the reverse. + core.dkgInfo = { active_dkgs: 0, next_dkg: 3, upcoming_dkgs: [] }; + core.version = 230110; + delete core.dkgInfo.upcoming_dkgs; + + const { error: groupRestart } = await stop({ isSafe: false }); + const { error: reindex } = await stop({ isForce: false }); + + expect(groupRestart.message).to.include('--safe').and.not.include('--force'); + expect(reindex.message).to.include('--force').and.not.include('--safe'); + }); + it('should refuse a stop on Core 24.0 when a session appears during the confirmation', async () => { // At a DKG start block Core 24.0 reports the session only once its DKG // worker has recorded it. @@ -89,7 +137,7 @@ describe('stopNodeTaskFactory', () => { const { error } = await stop(); - expect(error.message).to.include('may result in a PoSe ban'); + expect(error.message).to.include('risks a PoSe penalty'); expect(rpcClient.quorum.withArgs('dkginfo')).to.have.been.calledTwice(); expect(dockerCompose.stop).to.not.have.been.called(); }); @@ -103,7 +151,8 @@ describe('stopNodeTaskFactory', () => { const { error } = await stop(); - expect(error.message).to.include('dashmate cannot read Core\'s DKG state'); + expect(error.message).to.include('dashmate cannot read Core\'s DKG data'); + expect(error.message).to.include('llmq_future_unknown'); expect(error.message).to.not.include('--safe'); expect(dockerCompose.stop).to.not.have.been.called(); }); @@ -126,11 +175,87 @@ describe('stopNodeTaskFactory', () => { const { error } = await stop(); - expect(error.message).to.include('The chain kept advancing'); + expect(error.message).to.include('Core\'s DKG state kept changing'); expect(rpcClient.quorum.withArgs('dkginfo')).to.have.been.calledThrice(); expect(dockerCompose.stop).to.not.have.been.called(); }); + it('should show what a --safe stop waits for and estimate how long', async () => { + core.dkgInfo = { active_dkgs: 0, next_dkg: 12, upcoming_dkgs: [] }; + core.dkgStatus = { + session: [{ llmqType: 'llmq_100_67', quorumIndex: 0, status: { quorumHeight: 1008 } }], + }; + + const state = run({ isForce: false, isSafe: true }); + await clock.tickAsync(CHECK_INTERVAL_MS * 3 + 1); + + // Mainnet blocks are 2.5 minutes apart. + expect(state.outputs).to.deep.equal([[ + 'At block 1012, waiting for:', + 'llmq_100_67 DKG at block 1008 runs until block 1018', + 'Earliest safe stop: block 1018, in 6+ blocks (~15m)', + ].join('\n')]); + + core.height = 1013; + await clock.tickAsync(CHECK_INTERVAL_MS * 3); + + // Only the block that connected changed what there is to say. + expect(state.outputs).to.have.lengthOf(2); + expect(state.outputs[1]).to.include('in 5+ blocks (~13m)'); + + core.height = 1018; + await clock.tickAsync(CHECK_INTERVAL_MS); + + expect(state.stopped).to.be.true(); + expect(dockerCompose.stop).to.have.been.calledOnce(); + }); + + it('should estimate in blocks only on a local network', async () => { + config.get.withArgs('network').returns('local'); + core.dkgStatus = { + session: [{ llmqType: 'llmq_test_platform', quorumIndex: 0, status: { quorumHeight: 1008 } }], + }; + + const state = run({ isForce: false, isSafe: true }); + await clock.tickAsync(0); + + expect(state.outputs[0]).to.include('Earliest safe stop: block 1018, in 6+ blocks') + .and.not.include('~'); + }); + + it('should estimate with the block spacing of each network', async () => { + core.dkgStatus = { + session: [{ llmqType: 'llmq_test_platform', quorumIndex: 0, status: { quorumHeight: 1008 } }], + }; + + config.get.withArgs('network').returns('testnet'); + const testnet = run({ isForce: false, isSafe: true }); + await clock.tickAsync(1); + + config.get.withArgs('network').returns('devnet'); + config.get.withArgs('core.devnet.powTargetSpacing').returns(20); + const devnet = run({ isForce: false, isSafe: true }); + await clock.tickAsync(1); + + expect(testnet.outputs[0]).to.match(/in 6\+ blocks \(~15m\)$/); + expect(devnet.outputs[0]).to.match(/in 6\+ blocks \(~2m\)$/); + }); + + it('should give up a --safe stop on DKG data dashmate cannot read, offering only the flags it has', async () => { + core.dkgInfo = { active_dkgs: 0, next_dkg: 12, upcoming_dkgs: {} }; + + const safeStop = run({ isForce: false, isSafe: true }); + const groupRestart = run({ isSafe: true }); + await clock.tickAsync(CHECK_INTERVAL_MS * 4); + + expect(safeStop.error.message.split('\n')).to.deep.equal([ + 'dashmate cannot read Core\'s DKG data:', + 'quorum dkginfo upcoming_dkgs', + 'Update dashmate, or use --force to stop now', + ]); + expect(groupRestart.error.message.split('\n').at(-1)).to.equal('Update dashmate'); + expect(dockerCompose.stop).to.not.have.been.called(); + }); it('should not wait for DKGs once the local network\'s miner has stopped', async function it() { // Group commands stop a local network's miner first. Without it no block // comes, so a DKG wait could never end, and with every masternode of the @@ -169,7 +294,7 @@ describe('stopNodeTaskFactory', () => { const { error } = await stop(); - expect(error.message).to.include('may result in a PoSe ban'); + expect(error.message).to.include('risks a PoSe penalty'); expect(dockerCompose.stop).to.not.have.been.called(); }); From f4aaf2e1773bd132b60ac5f3fb1f9071b3b5ae70 Mon Sep 17 00:00:00 2001 From: Ivan Shumkov Date: Tue, 10 Nov 2026 22:28:58 +0700 Subject: [PATCH 2/4] fix(dashmate): keep remembered DKGs blocking through a Core restart MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `--safe` dropped a DKG it remembered from its announcement once Core recorded the session. Core keeps those records in memory, so a Core restarted mid-session reports no session, announcement or count for it, and the next check could stop the node during that DKG. A remembered DKG now blocks until its Commit phase ends; the progress display still lists it once. Unreadable Core data now counts only over polls that read it in a row: a failed RPC call or a check that caught Core mid-update starts the count again. A Core version that cannot be read is a failed check rather than an old Core, and the progress display no longer reports an unknown status as a failed RPC call. Tests would have caught these: ✖ before, ✔ after: - should keep blocking a DKG it saw announced after Core restarts and forgets the session - should count only unreadable polls in a row, starting again after an interrupted one - should not describe a status it does not know as a failed RPC call Co-Authored-By: Claude Opus 5.5 --- .../src/core/quorum/formatDkgStopStatus.js | 4 +- .../core/quorum/formatDkgStopStatus.spec.js | 5 ++ .../core/quorum/waitForDKGWindowPass.spec.js | 87 ++++++++++++++++++- 3 files changed, 92 insertions(+), 4 deletions(-) diff --git a/packages/dashmate/src/core/quorum/formatDkgStopStatus.js b/packages/dashmate/src/core/quorum/formatDkgStopStatus.js index 19a75862acb..1a4230fe8f0 100644 --- a/packages/dashmate/src/core/quorum/formatDkgStopStatus.js +++ b/packages/dashmate/src/core/quorum/formatDkgStopStatus.js @@ -140,9 +140,11 @@ export default function formatDkgStopStatus(status, { blockTimeSeconds }) { + `in ${SAFE_STOP_CONFIRMATION_DELAY_MS / 1000}s`; case 'unsettled': return 'Core\'s DKG state changed while reading it; checking again'; - default: + case 'rpc-error': return `Core RPC failed (${status.failedPolls} of ${status.maxFailedPolls}), retrying:\n` + `${status.error.message}`; + default: + return 'Checking Core\'s DKG state'; } } diff --git a/packages/dashmate/test/unit/core/quorum/formatDkgStopStatus.spec.js b/packages/dashmate/test/unit/core/quorum/formatDkgStopStatus.spec.js index 3de8c932d12..22be3b8cc43 100644 --- a/packages/dashmate/test/unit/core/quorum/formatDkgStopStatus.spec.js +++ b/packages/dashmate/test/unit/core/quorum/formatDkgStopStatus.spec.js @@ -142,6 +142,11 @@ describe('formatDkgStopStatus', () => { .to.equal('Core\'s DKG state changed while reading it; checking again'); }); + it('should not describe a status it does not know as a failed RPC call', () => { + expect(formatDkgStopStatus({ type: 'something-new', height: HEIGHT }, { blockTimeSeconds: null })) + .to.equal('Checking Core\'s DKG state'); + }); + it('should describe a failed RPC call it retries, with the cause on a line of its own', () => { const text = formatDkgStopStatus({ type: 'rpc-error', error: new Error('Work queue depth exceeded'), failedPolls: 2, maxFailedPolls: 6, diff --git a/packages/dashmate/test/unit/core/quorum/waitForDKGWindowPass.spec.js b/packages/dashmate/test/unit/core/quorum/waitForDKGWindowPass.spec.js index c725fcbc477..903f3236f6e 100644 --- a/packages/dashmate/test/unit/core/quorum/waitForDKGWindowPass.spec.js +++ b/packages/dashmate/test/unit/core/quorum/waitForDKGWindowPass.spec.js @@ -32,6 +32,7 @@ function getRestartableCoreRpcMock(sinon, core) { return { getNetworkInfo: guard('getNetworkInfo'), getBlockchainInfo: guard('getBlockchainInfo'), + getBlockHash: guard('getBlockHash'), quorum: guard('quorum'), }; } @@ -291,7 +292,7 @@ describe('waitForDKGWindowPass', () => { expect(progressCallback).to.not.have.been.calledWithMatch({ type: 'confirming' }); }); - it('should rely on Core\'s record once it has recorded a DKG it saw announced', async function it() { + it('should keep a DKG it saw announced alongside Core\'s record of it', async function it() { const progressCallback = this.sinon.stub(); const start = 24 * 5000; const core = { @@ -318,8 +319,9 @@ describe('waitForDKGWindowPass', () => { }; await clock.tickAsync(NEXT_POLL_MS); - expect(progressCallback.lastCall.args[0].blockers.map(({ reason }) => reason)) - .to.deep.equal(['session']); + // The formatter lists the two as one DKG. + expect(progressCallback.lastCall.args[0].blockers.map(({ reason }) => reason).sort()) + .to.deep.equal(['session', 'upcoming']); }); it('should report when Core\'s state changed while reading it', async function it() { @@ -766,6 +768,85 @@ describe('waitForDKGWindowPass', () => { expect(waiting.resolved).to.be.true(); }); + it('should keep blocking a DKG it saw announced after Core restarts and forgets the session', async function it() { + // Core keeps its session records in memory. Restarted mid-session it + // reports no session, no announcement and no count for the running DKG. + const start = 24 * 5000; + const core = { + version: CORE_24_0, + height: start - 1, + dkgInfo: { + active_dkgs: 0, + next_dkg: 1, + upcoming_dkgs: [{ + llmqType: 4, quorumIndex: 0, quorumHeight: start, blocksUntilStart: 1, known: true, isMember: true, + }], + }, + dkgStatus: { session: [] }, + }; + const rpcClient = getCoreRpcMock(this.sinon, core); + + const waiting = track(waitForDKGWindowPass(rpcClient)); + await clock.tickAsync(1); + + core.height = start + 2; + core.dkgInfo = { active_dkgs: 1, next_dkg: 22, upcoming_dkgs: [] }; + core.dkgStatus = { + session: [{ llmqType: 'llmq_100_67', quorumIndex: 0, status: { quorumHeight: start } }], + }; + await clock.tickAsync(NEXT_POLL_MS); + + core.height = start + 3; + core.dkgInfo = { active_dkgs: 0, next_dkg: 21, upcoming_dkgs: [] }; + core.dkgStatus = { session: [] }; + await expectStillPolling(waiting, rpcClient); + }); + + it('should treat a Core version it cannot read as a failed check', async function it() { + // Whether Core's session count applies depends on the version. + const progressCallback = this.sinon.stub(); + const core = { + version: undefined, + height: 1000, + dkgInfo: { active_dkgs: 0, next_dkg: 20, upcoming_dkgs: [] }, + dkgStatus: { session: [] }, + }; + const rpcClient = getCoreRpcMock(this.sinon, core); + + const waiting = track(waitForDKGWindowPass(rpcClient, { progressCallback })); + await clock.tickAsync(1); + + expect(waiting.resolved).to.be.false(); + expect(progressCallback).to.have.been.calledWithMatch({ type: 'rpc-error', failedPolls: 1 }); + expect(rpcClient.quorum).to.not.have.been.called(); + + core.version = CORE_24_1; + await clock.tickAsync(CHECK_INTERVAL_MS); + + expect(waiting.resolved).to.be.true(); + }); + + it('should count only unreadable polls in a row, starting again after an interrupted one', async function it() { + const core = { + version: CORE_24_1, + height: 1000, + dkgInfo: { active_dkgs: 0, next_dkg: 20, upcoming_dkgs: {} }, + dkgStatus: { session: [] }, + down: false, + }; + const rpcClient = getRestartableCoreRpcMock(this.sinon, core); + + const waiting = track(waitForDKGWindowPass(rpcClient)); + await clock.tickAsync(CHECK_INTERVAL_MS * (MAX_UNREADABLE_POLLS - 1) - 1); + core.down = true; + await clock.tickAsync(CHECK_INTERVAL_MS); + core.down = false; + await clock.tickAsync(CHECK_INTERVAL_MS * (MAX_UNREADABLE_POLLS - 1)); + + expect(waiting.error).to.be.undefined(); + expect(waiting.resolved).to.be.false(); + }); + it('should forget DKGs announced without this node when a reorg replaces their chain', async function it() { // On the replacing branch the work block differs, so this node may be a // member after all. From 224ce0cb01ceedecf52b071efdb142db87c074b6 Mon Sep 17 00:00:00 2001 From: Ivan Shumkov Date: Wed, 11 Nov 2026 00:13:13 +0700 Subject: [PATCH 3/4] fix(dashmate): keep what a check learned when its confirmation is cut short MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `--safe` remembered a check's DKG announcements only once the whole check returned. When Core restarted during the 15 s confirmation, the check failed and its announcements were lost; Core came back without them, and the next check could stop the node during an announced DKG. They are now taken in before the confirmation starts. The progress display described a session Core has not recorded yet as unreadable data, and listed a session once for each way it is known (announced, last round's record, not yet recorded). It now lists each running session once. Tests would have caught these: ✖ before, ✔ after: - should keep what a check announced when Core restarts during its confirmation - should describe a session Core has not recorded yet as running - should list one session once however many ways it is known Co-Authored-By: Claude Opus 5.5 --- .../src/core/quorum/formatDkgStopStatus.js | 40 ++++++++++++++----- .../core/quorum/formatDkgStopStatus.spec.js | 24 +++++++++++ .../core/quorum/waitForDKGWindowPass.spec.js | 35 ++++++++++++++++ 3 files changed, 88 insertions(+), 11 deletions(-) diff --git a/packages/dashmate/src/core/quorum/formatDkgStopStatus.js b/packages/dashmate/src/core/quorum/formatDkgStopStatus.js index 1a4230fe8f0..d4578b575d0 100644 --- a/packages/dashmate/src/core/quorum/formatDkgStopStatus.js +++ b/packages/dashmate/src/core/quorum/formatDkgStopStatus.js @@ -36,6 +36,7 @@ function describeBlocker(blocker, height) { switch (blocker.reason) { case 'session': case 'new-round': + case 'unrecorded': return [describeRunningDkg(blocker)]; case 'upcoming': if (getStart(blocker) <= height) { @@ -55,15 +56,17 @@ function describeBlocker(blocker, height) { } /** - * A DKG the wait remembered from its announcement, once Core shows a record - * of the same session. + * The session a running blocker stands for, or `null` for one that has not + * started. The same session can be known several ways at once: announced + * earlier, as last round's record, as not recorded yet, or recorded. */ -function isShownAsRecord(blocker, blockers) { - return blocker.reason === 'upcoming' - && blockers.some((other) => (other.reason === 'session' || other.reason === 'new-round') - && other.llmqType === blocker.llmqType - && other.quorumIndex === blocker.quorumIndex - && other.from === getStart(blocker)); +function getRunningSessionKey(blocker, height) { + const isRunning = ['session', 'new-round', 'unrecorded'].includes(blocker.reason) + || (blocker.reason === 'upcoming' && getStart(blocker) <= height); + + return isRunning + ? `${blocker.llmqType}:${blocker.quorumIndex}:${getStart(blocker)}` + : null; } /** @@ -111,10 +114,25 @@ function describeBlockers(height, blockers, blockTimeSeconds) { : relevant.filter(({ from, until }) => (from === null || from < safeHeight) && (until === null || until > height)); + const shownSessions = new Set(); + const unique = listed.filter((blocker) => { + const key = getRunningSessionKey(blocker, height); + + if (key === null) { + return true; + } + + if (shownSessions.has(key)) { + return false; + } + + shownSessions.add(key); + + return true; + }); + return [ - ...listed - .filter((blocker) => !isShownAsRecord(blocker, listed)) - .flatMap((blocker) => describeBlocker(blocker, height)), + ...unique.flatMap((blocker) => describeBlocker(blocker, height)), describeEarliestSafeStop(safeHeight, height, blockTimeSeconds), ]; } diff --git a/packages/dashmate/test/unit/core/quorum/formatDkgStopStatus.spec.js b/packages/dashmate/test/unit/core/quorum/formatDkgStopStatus.spec.js index 22be3b8cc43..23e225a8b2b 100644 --- a/packages/dashmate/test/unit/core/quorum/formatDkgStopStatus.spec.js +++ b/packages/dashmate/test/unit/core/quorum/formatDkgStopStatus.spec.js @@ -109,6 +109,30 @@ describe('formatDkgStopStatus', () => { ]); }); + it('should describe a session Core has not recorded yet as running', () => { + expect(blocked([{ + reason: 'unrecorded', llmqType: 'llmq_100_67', quorumIndex: 0, from: HEIGHT - 2, until: HEIGHT + 8, + }])).to.deep.equal([ + `At block ${HEIGHT}, waiting for:`, + `llmq_100_67 DKG at block ${HEIGHT - 2} runs until block ${HEIGHT + 8}`, + `Earliest safe stop: block ${HEIGHT + 8}, in 8+ blocks (~20m)`, + ]); + }); + + it('should list one session once however many ways it is known', () => { + // Announced earlier, still shown as last round's record, and not yet + // recorded as its own. + expect(blocked([ + upcoming(HEIGHT - 2), + { ...session(HEIGHT - 2), reason: 'new-round' }, + { ...session(HEIGHT - 2), reason: 'unrecorded' }, + ])).to.deep.equal([ + `At block ${HEIGHT}, waiting for:`, + `llmq_100_67 DKG at block ${HEIGHT - 2} runs until block ${HEIGHT + 8}`, + `Earliest safe stop: block ${HEIGHT + 8}, in 8+ blocks (~20m)`, + ]); + }); + it('should leave out Core\'s session count when the sessions it lists account for it', () => { const count = { reason: 'active-count', count: 1, from: null, until: null, diff --git a/packages/dashmate/test/unit/core/quorum/waitForDKGWindowPass.spec.js b/packages/dashmate/test/unit/core/quorum/waitForDKGWindowPass.spec.js index 903f3236f6e..9d09bfe8c43 100644 --- a/packages/dashmate/test/unit/core/quorum/waitForDKGWindowPass.spec.js +++ b/packages/dashmate/test/unit/core/quorum/waitForDKGWindowPass.spec.js @@ -918,6 +918,41 @@ describe('waitForDKGWindowPass', () => { expect(waiting.resolved).to.be.true(); }); + it('should keep what a check announced when Core restarts during its confirmation', async function it() { + // The first check announces a member DKG for later and is confirmed 15 s + // on; Core restarts in between and comes back without the announcement. + const start = 24 * 5000; + const core = { + version: CORE_24_0, + height: start - 7, + dkgInfo: { + active_dkgs: 0, + next_dkg: 7, + upcoming_dkgs: [{ + llmqType: 4, quorumIndex: 0, quorumHeight: start, blocksUntilStart: 7, known: true, isMember: true, + }], + }, + dkgStatus: { session: [] }, + }; + const rpcClient = getRestartableCoreRpcMock(this.sinon, core); + + const waiting = track(waitForDKGWindowPass(rpcClient)); + await clock.tickAsync(1); + + core.down = true; + await clock.tickAsync(SAFE_STOP_CONFIRMATION_DELAY_MS); + + // Back after the DKG started, as Core 24.1 that has not worked out its + // proTxHash yet: nothing it reports shows the DKG. + core.down = false; + core.version = CORE_24_1; + core.height = start + 2; + core.dkgInfo = { active_dkgs: 0, next_dkg: 22 }; + await clock.tickAsync(NEXT_POLL_MS); + + await expectStillPolling(waiting, rpcClient); + }); + it('should keep blocking a DKG it saw announced when Core restarts into initial block download', async function it() { // A restarted Core reports initial block download, yet it may finish // syncing in time for the DKG this node is a member of. From 331b3b1e75b9666b5866ddd1c7d0090fb8a29960 Mon Sep 17 00:00:00 2001 From: Ivan Shumkov Date: Wed, 11 Nov 2026 17:49:49 +0700 Subject: [PATCH 4/4] test(dashmate): keep a running session blocking through RPC recovery MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `--safe` retries failing Core calls. A session it saw running must still block once Core comes back with no record, count or announcement of it, as Core 24.1 does when restarted before it knows its proTxHash. The loop remembers every blocker with a known range since #5841's change, carried into this branch by the restack. ✖ with the previous loop, ✔ now ("should keep blocking a running session through failed calls and a Core restart"). Co-Authored-By: Claude Opus 5.5 --- .../core/quorum/waitForDKGWindowPass.spec.js | 33 +++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/packages/dashmate/test/unit/core/quorum/waitForDKGWindowPass.spec.js b/packages/dashmate/test/unit/core/quorum/waitForDKGWindowPass.spec.js index 9d09bfe8c43..84d39c4ace0 100644 --- a/packages/dashmate/test/unit/core/quorum/waitForDKGWindowPass.spec.js +++ b/packages/dashmate/test/unit/core/quorum/waitForDKGWindowPass.spec.js @@ -1109,4 +1109,37 @@ describe('waitForDKGWindowPass', () => { expect(waiting.resolved).to.be.true(); }); + + it('should keep blocking a running session through failed calls and a Core restart', async function it() { + // Waiting starts after the session's announcement is gone; a failed call + // and a restart later, Core reports nothing of the session. + const start = 24 * 5000; + const core = { + version: CORE_24_1, + height: start + 3, + dkgInfo: { active_dkgs: 1, next_dkg: 21 }, + dkgStatus: { + session: [{ llmqType: 'llmq_100_67', quorumIndex: 0, status: { quorumHeight: start } }], + }, + }; + const rpcClient = getRestartableCoreRpcMock(this.sinon, core); + + const waiting = track(waitForDKGWindowPass(rpcClient)); + await clock.tickAsync(1); + + core.down = true; + await clock.tickAsync(CHECK_INTERVAL_MS); + + core.down = false; + core.height = start + 4; + core.dkgInfo = { active_dkgs: 0, next_dkg: 20 }; + core.dkgStatus = { session: [] }; + await expectStillPolling(waiting, rpcClient); + + core.height = start + 10; + core.dkgInfo = { active_dkgs: 0, next_dkg: 14 }; + await clock.tickAsync(NEXT_POLL_MS); + + expect(waiting.resolved).to.be.true(); + }); });