Skip to content

test: wait for a ChainLock only on a block the healed quorum signs - #8171

Open
PastaPastaPasta wants to merge 1 commit into
dashpay:developfrom
PastaPastaPasta:test/data-recovery-chainlock-quorum
Open

PastaPastaPasta wants to merge 1 commit into
dashpay:developfrom
PastaPastaPasta:test/data-recovery-chainlock-quorum

Conversation

@PastaPastaPasta

Copy link
Copy Markdown
Member

Issue being fixed or feature implemented

feature_llmq_data_recovery.py fails intermittently: the final wait_for_chainlocked_block_all_nodes after the recovery scenario times out. It reproduced 5 times in 144 parallel runs on current develop, and once in 96 runs on the version before #8156, so it predates that PR.

The test heals one quorum (quorum_hash_recover) and then expects the next block to be chainlocked. But two llmq_test quorums are active, and the ChainLock for a height can fall to either one. The other quorum's members lost their DKG data to the same reindex, and the test never waits for them to recover. When that quorum is picked for the block, the ChainLock either never forms or forms only by luck. The failing runs showed both ways it breaks:

  • its members have not recovered their key share yet (CreateSigShare -- we don't have our skShare for quorum ...), or
  • enough members sign, but the recovery member chosen for the session still lacks the quorum verification vector and drops every share (PreVerifySigShareQuorum -- we don't have the quorum vvec ... no verification possible).

In both cases nothing re-signs the tip, because mocktime is frozen and no new block arrives, so the wait times out.

What was done?

Before waiting for the ChainLock, mine blocks until quorum selectquorum names the healed quorum for the tip's clsig request id. That call resolves the quorum the same way the signers do at that tip. The ChainLock check then covers what the test claims: the healed quorum can sign again. The clsig request-id calculation already in test_no_chainlock becomes a small helper so both places share it.

How Has This Been Tested?

The same release dashd ran feature_llmq_data_recovery.py 16 at a time:

  • Before the fix: 5 of 144 runs failed, all at this ChainLock wait (1 of 96 with the version before test: shorten the default functional suite #8156).
  • After the fix: 192 of 192 passed, and so did one run through test_runner.py on an --enable-debug build. With instrumentation, 2 of 16 runs had to mine extra blocks to reach the healed quorum, so both paths run.

Breaking Changes

None. The change only touches a functional test.

Checklist:

  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation
  • I have assigned this pull request to a milestone (if you have permission to set one)

🤖 Generated with Claude Code

@PastaPastaPasta PastaPastaPasta added this to the 24.1 milestone Nov 16, 2026
@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The functional test adds a helper to derive ChainLock signing request IDs. After recovery, it mines and validation-syncs blocks until the selected signing quorum matches the recovered quorum, then checks the ChainLock on that block.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: Minimal · up to 3a4a5

This only changes a functional test so it waits for a ChainLock on a block signed by the recovered quorum. It does not affect production behavior, and no merge-blocking risk remains.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: the test now waits for a ChainLock only on a block whose signing request selects the healed quorum. It matches the raw summary (mining until quorum selectquorum names …
Description check Passed The description explains the intermittent failure in feature_llmq_data_recovery.py, the root cause (a second active quorum whose members have not recovered DKG data), the fix (mine blocks until quorum…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: the test now waits for a ChainLock only on a block whose signing request selects the healed quorum. It matches the raw summary (mining until quorum selectquorum names the healed quorum before the ChainLock wait) and the PR objectives. It is specific and not vague.

Full details: Description check

Explanation

The description explains the intermittent failure in feature_llmq_data_recovery.py, the root cause (a second active quorum whose members have not recovered DKG data), the fix (mine blocks until quorum selectquorum names the healed quorum, and a shared clsig request-id helper), and the test results. It relates directly to the changeset.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 2cddf1bc-87c4-4976-bdac-e36268552bf3
Commits

Reviewing files that changed from the base of the PR and between b136890 and 3a4a591.

Files selected for processing: 1

  • test/functional/feature_llmq_data_recovery.py

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 1 remain after this review.

feature_llmq_data_recovery.py heals one quorum and then expects the
next block to be chainlocked. Two llmq_test quorums are active, and the
ChainLock for a height can fall to either. The other quorum lost its key
shares to the same reindex, and the test never waits for it to recover.
When it is picked, its members either have no key share yet or the
recovery member chosen for the session has no verification vector and
drops every share, and nothing re-signs the tip, so the wait times out.

Mine until selectquorum names the healed quorum for the tip, then wait
for its ChainLock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@PastaPastaPasta
PastaPastaPasta force-pushed the test/data-recovery-chainlock-quorum branch from b136890 to 3a4a591 Compare November 16, 2026 05:03
@thepastaclaw

thepastaclaw commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Final review complete — no blockers (commit 3a4a591) · triage: low

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final validation — Phase 1 + Phase 2

The change is a focused functional-test fix that selects the healed quorum for the newly mined block before asserting its ChainLock. The request-ID helper preserves the existing serialization, and the quorum-selection check matches the production ChainLock signing path. Reviewed CI checks are successful where completed, with remaining jobs queued or in progress and no reported failures caused by this change.

Review provenance

Source: reviewer 1: glm-5.3-flash (agent: phase1-reviewer, role: general); reviewer 2: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: dash-core-commit-history); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: low by gpt-6.1-sol (effort low) — This is a small, well-contained functional-test change that adds deterministic quorum selection before an existing ChainLock assertion without changing production behavior.
  • Phase 1 reviewers: glm-5.3-flash — general (completed, effort high); agent phase1-reviewer
  • Phase 1 model: glm-5.3-flash — zai quota: 5h 99% left, weekly 50% left; passed over gemini-3.8-flash-high (antigravity quota lookup failed)
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort medium); agent phase2-reviewer, gpt-6.1-sol — dash-core-commit-history (completed, effort medium); agent phase2-reviewer

@thepastaclaw thepastaclaw added the pastaclaw:approved thepastaclaw's latest review approved this PR label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pastaclaw:approved thepastaclaw's latest review approved this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants