Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions packages/dashmate/docs/commands/group/restart.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ The restart process happens in two phases:

If the nodes are masternodes participating in quorums, restarting them during a DKG (Distributed Key Generation) session can cause issues.
Use the `--safe` flag to wait for any current DKG round to complete before stopping the nodes.
See [Stopping a masternode safely](../stop.md#stopping-a-masternode-safely) for what dashmate checks and what `--safe` shows while it waits.
Nodes are stopped one at a time, and with `--safe` each waits for its own safe moment, so nodes already stopped stay down while the next one waits.
If stopping a node fails, the command ends there and the nodes it already stopped stay down until you start the group again.

This command is useful when:
- You've changed configuration options that require a service restart
Expand Down
2 changes: 2 additions & 0 deletions packages/dashmate/docs/commands/group/stop.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ The command handles the stop sequence appropriately to ensure that interdependen

If the nodes are masternodes participating in DKG (Distributed Key Generation) session, stopping them can cause issues.
Use the `--safe` flag to wait for any current DKG round to complete before stopping the nodes, or use `--force` to ignore this protection (which may result in masternodes being temporarily banned from quorums).
See [Stopping a masternode safely](../stop.md#stopping-a-masternode-safely) for what dashmate checks and what `--safe` shows while it waits.
Nodes are stopped one at a time, and with `--safe` each waits for its own safe moment, so nodes already stopped stay down while the next one waits.

## Examples

Expand Down
1 change: 1 addition & 0 deletions packages/dashmate/docs/commands/restart.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ Use the `--platform` flag to restart only the platform services while keeping Co

If the node is a masternode participating in a quorum, restarting during a DKG (Distributed Key Generation) session can cause issues.
Use the `--safe` flag to wait for any current DKG round to complete before restarting, or use `--force` to ignore this protection (which may result in the masternode being banned and lost rewards).
See [Stopping a masternode safely](./stop.md#stopping-a-masternode-safely) for what dashmate checks and what `--safe` shows while it waits.

## Examples

Expand Down
15 changes: 15 additions & 0 deletions packages/dashmate/docs/commands/stop.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,21 @@ Using the `--safe` flag ensures the command waits for any current DKG round to c

The `--force` flag can be used to forcibly stop services even if there's a DKG in progress or other conditions that would normally prevent shutdown.

## Stopping a masternode safely

A masternode that misses its part in a DKG, the session that creates a new quorum's keys, can get a PoSe penalty. Before stopping a masternode, dashmate asks Core whether stopping now could cause that:

- a DKG this node is a member of is still running: from its start block to the end of its Commit phase, 10 blocks for most quorum types and 20 for the 400-member ones;
- or one starts within the next 6 blocks, too soon for a restart to finish. Core 24 tells dashmate whether this node is a member; with older Core, any DKG counts.

Without `--safe` or `--force`, the command refuses in that case and prints what blocks the stop and the earliest block at which stopping is safe.

With `--safe`, it checks every 10 seconds and stops the node as soon as that is safe. While it waits it shows the DKGs it is waiting for and the earliest safe block, with a time estimate on mainnet, testnet and devnets. The estimate can grow: Core announces a node's place in a DKG only about 8 blocks before the DKG starts. With Core 24, an evonode usually waits under half an hour, longer around the twice-daily start of the InstantSend quorum cycle. Older Core does not report membership, so every start of that cycle blocks for about an hour and a half. Pressing Ctrl+C ends the wait and leaves the node running.

`--safe` gives up and leaves the node running when Core stops answering for about a minute, or when it keeps reporting DKG data dashmate cannot read.

On Core older than 24.1, dashmate checks a safe result again 15 seconds later, so stopping a masternode takes about that much longer.

## Examples

```bash
Expand Down
1 change: 1 addition & 0 deletions packages/dashmate/src/commands/group/restart.js
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ export default class GroupRestartCommand extends GroupBaseCommand {
// So we stop the miner first, as there's a chance that MNs will get banned
// if the miner is still running when stopping them
new Listr(configGroup.reverse().map((config) => ({
title: `Stop ${config.getName()} node`,
task: () => stopNodeTask(config),
})))
),
Expand Down
1 change: 1 addition & 0 deletions packages/dashmate/src/commands/group/stop.js
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ export default class GroupStopCommand extends GroupBaseCommand {
// So we stop the miner first, as there's a chance that MNs will get banned
// if the miner is still running when stopping them
new Listr(configGroup.reverse().map((config) => ({
title: `Stop ${config.getName()} node`,
task: () => stopNodeTask(config),
})))
),
Expand Down
14 changes: 13 additions & 1 deletion packages/dashmate/src/core/quorum/checkMasternodeSafeToStop.js
Original file line number Diff line number Diff line change
Expand Up @@ -225,12 +225,20 @@ async function evaluate(rpcClient, coreVersion, knownNonMemberDkgs, ancestorHeig
* without this node at earlier checks
* @param {{ height: number, hash: string }} [options.ancestor] tip of the
* latest earlier check
* @param {function(SafeStopCheck): boolean} [options.beforeConfirmation]
* called with the first, safe check before waiting to confirm it; returning
* `false` skips the confirmation and returns that check, for a caller that
* will not stop the node at its height anyway
* @return {Promise<SafeStopCheck>}
*/
export default async function checkMasternodeSafeToStop(
rpcClient,
coreVersion,
{ knownNonMemberDkgs = new Set(), ancestor } = {},
{
knownNonMemberDkgs = new Set(),
ancestor,
beforeConfirmation = () => true,
} = {},
) {
const first = await evaluate(rpcClient, coreVersion, knownNonMemberDkgs, ancestor && ancestor.height);

Expand All @@ -243,6 +251,10 @@ export default async function checkMasternodeSafeToStop(
return first;
}

if (!beforeConfirmation(first)) {
return first;
}

await wait(SAFE_STOP_CONFIRMATION_DELAY_MS);

const nonMemberDkgs = new Set([...knownNonMemberDkgs, ...first.nonMemberDkgs]);
Expand Down
215 changes: 215 additions & 0 deletions packages/dashmate/src/core/quorum/formatDkgStopStatus.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,215 @@
import prettyMs from 'pretty-ms';
import { MIN_BLOCKS_BEFORE_DKG } from '../../constants.js';
import { SAFE_STOP_CONFIRMATION_DELAY_MS } from './checkMasternodeSafeToStop.js';
import {
getBlockersAt,
getEarliestSafeHeight,
withoutRecordedSessionCount,
} from './getDkgStopBlockers.js';

// Lines for mainnet quorum types stay within 64 characters up to 8-digit
// heights: the default listr renderer cuts every line at the terminal width,
// less the indentation of the task it belongs to.

function describeQuorum({ llmqType, quorumIndex }) {
const name = typeof llmqType === 'number' ? `LLMQ type ${llmqType}` : llmqType;

return Number.isInteger(quorumIndex) && quorumIndex > 0 ? `${name} #${quorumIndex}` : name;
}

function getStart(blocker) {
return blocker.reason === 'upcoming' ? blocker.from + MIN_BLOCKS_BEFORE_DKG : blocker.from;
}

function describeRunningDkg(blocker) {
const end = blocker.until === null ? 'is running' : `runs until block ${blocker.until}`;

return `${describeQuorum(blocker)} DKG at block ${getStart(blocker)} ${end}`;
}

/**
* @param {DkgStopBlocker} blocker
* @param {number} height
* @return {string[]}
*/
function describeBlocker(blocker, height) {
switch (blocker.reason) {
case 'session':
case 'new-round':
case 'unrecorded':
return [describeRunningDkg(blocker)];
case 'upcoming':
if (getStart(blocker) <= height) {
return [describeRunningDkg(blocker)];
}

return [`${describeQuorum(blocker)} DKG starts at block ${getStart(blocker)} `
+ `(${blocker.membershipKnown ? 'member' : 'membership unknown'})`];
case 'imminent':
return [`A DKG starts at block ${blocker.from + MIN_BLOCKS_BEFORE_DKG} (membership not reported)`];
case 'active-count':
return [`Core reports ${blocker.count} DKG session${blocker.count === 1 ? '' : 's'} `
+ 'in progress for this node'];
default:
return ['dashmate cannot read Core\'s DKG data:', blocker.source];
}
Comment thread
shumkov marked this conversation as resolved.
Resolved
}

/**
* The session a running blocker stands for, or `null` for one that has not
* started. The same session can be known several ways at once: announced
* earlier, as last round's record, as not recorded yet, or recorded.
*/
function getRunningSessionKey(blocker, height) {
const isRunning = ['session', 'new-round', 'unrecorded'].includes(blocker.reason)
|| (blocker.reason === 'upcoming' && getStart(blocker) <= height);

return isRunning
? `${blocker.llmqType}:${blocker.quorumIndex}:${getStart(blocker)}`
: null;
}

/**
* @param {number} seconds
* @return {string}
*/
function formatDuration(seconds) {
const minutes = Math.max(1, Math.round(seconds / 60));

return prettyMs(minutes * 60 * 1000, { unitCount: 2 });
}

/**
* @param {number|null} safeHeight
* @param {number} height
* @param {number|null} blockTimeSeconds
* @return {string}
*/
function describeEarliestSafeStop(safeHeight, height, blockTimeSeconds) {
if (safeHeight === null) {
return 'Earliest safe stop: unknown';
}

const blocks = safeHeight - height;
const time = blockTimeSeconds ? ` (~${formatDuration(blocks * blockTimeSeconds)})` : '';

return `Earliest safe stop: block ${safeHeight}, in ${blocks}+ blocks${time}`;
}

/**
* What blocks stopping until the earliest safe height, or, when that is
* unknown, what blocks it now; then the earliest safe height.
*
* @param {number} height
* @param {DkgStopBlocker[]} blockers
* @param {number|null} blockTimeSeconds
* @return {string[]}
*/
function describeBlockers(height, blockers, blockTimeSeconds) {
const safeHeight = getEarliestSafeHeight(blockers, height);
const relevant = withoutRecordedSessionCount(blockers, height);

const listed = safeHeight === null
? getBlockersAt(relevant, height)
: relevant.filter(({ from, until }) => (from === null || from < safeHeight)
&& (until === null || until > height));

const shownSessions = new Set();
const unique = listed.filter((blocker) => {
const key = getRunningSessionKey(blocker, height);

if (key === null) {
return true;
}

if (shownSessions.has(key)) {
return false;
}

shownSessions.add(key);

return true;
});

return [
...unique.flatMap((blocker) => describeBlocker(blocker, height)),
describeEarliestSafeStop(safeHeight, height, blockTimeSeconds),
];
}

/**
* Progress of `waitForDKGWindowPass` for the operator.
*
* @param {Object} status as passed to its `progressCallback`
* @param {Object} options
* @param {number|null} options.blockTimeSeconds target block spacing, or
* `null` to estimate in blocks only
* @return {string}
*/
export default function formatDkgStopStatus(status, { blockTimeSeconds }) {
switch (status.type) {
case 'blocked':
return [
`At block ${status.height}, waiting for:`,
...describeBlockers(status.height, status.blockers, blockTimeSeconds),
].join('\n');
case 'confirming':
return `Block ${status.height} looks safe; checking again `
+ `in ${SAFE_STOP_CONFIRMATION_DELAY_MS / 1000}s`;
case 'unsettled':
return 'Core\'s DKG state changed while reading it; checking again';
case 'rpc-error':
return `Core RPC failed (${status.failedPolls} of ${status.maxFailedPolls}), retrying:\n`
+ `${status.error.message}`;
default:
return 'Checking Core\'s DKG state';
}
}

/**
* Why a plain stop refuses to stop the masternode, and what the operator can
* do. Only the flags the running command has are suggested.
*
* @param {SafeStopCheck} check a check that is not `safe`
* @param {Object} options
* @param {number|null} options.blockTimeSeconds
* @param {boolean} options.canWait the command has `--safe`
* @param {boolean} options.canForce the command has `--force`
* @return {string}
*/
export function formatDkgStopRefusal(check, { blockTimeSeconds, canWait, canForce }) {
const unreadable = check.status === 'blocked'
&& getBlockersAt(check.blockers, check.height).find(({ reason }) => reason === 'unreadable');

// Unreadable data does not clear on its own, so --safe is not offered.
if (unreadable) {
return [
...describeBlocker(unreadable, check.height),
canForce ? 'Update dashmate, or use --force to stop now' : 'Update dashmate',
].join('\n');
}

const lines = check.status === 'blocked'
? [
'Stopping now risks a PoSe penalty for this masternode:',
...describeBlockers(check.height, check.blockers, blockTimeSeconds),
]
: ['Core\'s DKG state kept changing while reading it', '(is Core still syncing?)'];

const retry = check.status === 'blocked' && getEarliestSafeHeight(check.blockers, check.height) !== null
? 'Retry then'
: 'Retry later';

if (canWait && canForce) {
lines.push(`${retry}, use --safe to wait and stop automatically,`);
lines.push('or --force to stop now and accept the PoSe risk');
} else if (canWait) {
lines.push(`${retry}, or use --safe to wait and stop automatically`);
} else if (canForce) {
lines.push(`${retry}, or use --force to stop now and accept the PoSe risk`);
} else {
lines.push(retry);
}

return lines.join('\n');
}
59 changes: 59 additions & 0 deletions packages/dashmate/src/core/quorum/getDkgStopBlockers.js
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,65 @@ export function getBlockersAt(blockers, height) {
return blockers.filter((blocker) => isBlockedAt(blocker, height));
}

/**
* Core 24.1+ reports how many sessions this node is in, not which. When the
* running sessions dashmate knows of at `height`, recorded or remembered
* from their announcement, account for that count, the count tells nothing
* more about what to wait for or how long. Only for display and estimates:
* the count still blocks.
*
* @param {DkgStopBlocker[]} blockers
* @param {number} height
* @return {DkgStopBlocker[]}
*/
export function withoutRecordedSessionCount(blockers, height) {
const runningSessions = new Set(getBlockersAt(blockers, height)
.map((blocker) => {
const start = blocker.reason === 'upcoming'
? blocker.from + MIN_BLOCKS_BEFORE_DKG
: blocker.from;

return ['session', 'new-round', 'upcoming'].includes(blocker.reason) && start <= height
? `${blocker.llmqType}:${blocker.quorumIndex}:${start}`
: null;
})
.filter((key) => key !== null));
const recordedSessions = runningSessions.size;

return blockers.filter((blocker) => blocker.reason !== 'active-count'
|| blocker.count > recordedSessions);
}

/**
* The first height at or above `height` that no blocker covers, or `null`
* when a blocker with no known end stands in the way. It is a lower bound:
* Core announces membership of a DKG only 8 blocks before its cycle starts,
* so a DKG not announced yet can push it further out.
*
* @param {DkgStopBlocker[]} blockers
* @param {number} height
* @return {number|null}
*/
export function getEarliestSafeHeight(blockers, height) {
const relevant = withoutRecordedSessionCount(blockers, height);

let candidate = height;

for (;;) {
const covering = getBlockersAt(relevant, candidate);

if (covering.length === 0) {
return candidate;
}

if (covering.some(({ until }) => until === null)) {
return null;
}

candidate = Math.max(...covering.map(({ until }) => until));
}
}

function isCount(value) {
return Number.isInteger(value) && value >= 0;
}
Expand Down
Loading
Loading