Skip to content

test: interface_zmq_dash.py InstantLock timeout on branches missing wait_for_instantlock sync fixes #7270

Open

Description

@thepastaclaw

Summary

interface_zmq_dash.py can time out waiting for an InstantLock on older/stale branches that do not contain the later wait_for_instantlock() sync fixes now present on develop.

Failing CI examples

Failure mode

test/functional/interface_zmq_dash.py times out at:

  • DashZMQTest.test_instantsend_publishers()
  • line 300: self.wait_for_instantlock(rpc_raw_tx_1_hash, self.nodes[0])

CI traceback excerpt:

AssertionError: Predicate ''''
    def check_instantlock():
        try:
            return node.getrawtransaction(txid, True)["instantlock"]
        except Exception:
            return False
''' not true after 240.0 seconds

Root cause

This is not caused by PR #7237's own changes.

I verified in the PR worktree:

git log upstream/develop..HEAD -- test/functional/interface_zmq_dash.py test/functional/test_framework/test_framework.py

which returns no commits, so the PR branch itself does not modify the failing test/helper path.

develop already contains the fixes for this exact issue:

  • c79c2d981d — refactor: improve is-waiting helper by including mempool sync bump of mocktime
  • d8ac7815e3 — test: fix additional IS lock sync sites that passed nodes=[self.nodes[0]]

Those commits restore the wait_for_instantlock() behavior to bump mocktime and sync mempools across the relevant nodes, which is required so the MN signer nodes actually see the transaction before InstantLock signing.

The failing branch is behind develop and does not contain those fixes, so it can still hit the old timeout.

Reproduction notes

  1. Use a branch that predates c79c2d981d / d8ac7815e3
  2. Run python3 test/functional/interface_zmq_dash.py in a CI-capable environment with python3-zmq installed
  3. Observe intermittent timeout waiting for getrawtransaction(...)["instantlock"]

Expected behavior

Branches with the current develop test framework fixes should not rely on the old unsynchronized behavior and should avoid this timeout.

Suggested resolution

For stale backport branches, rebase or cherry-pick the helper/test fixes from:

  • c79c2d981d
  • d8ac7815e3

Activity

  1. thepastaclaw commented on Apr 18, 2026

    @thepastaclaw
    CollaboratorAuthor

    Another occurrence from PR #7242 (wallet deprecation + RPC type checking backport), current head 24daadcf6d.

    The failure is the same InstantLock timeout in interface_zmq_dash.py:

    AssertionError: Predicate "" not true after 240.0 seconds
        def check_instantlock():
            try:
                return all(
                    node.getrawtransaction(txid, True)["instantlock"]
                    for txid in txids
                    for node in nodes
                )
            except Exception:
                return False
    

    I rechecked the current PR head and both of these are empty / unchanged relative to the branch diff:

    • git log upstream/develop..HEAD -- test/functional/interface_zmq_dash.py
    • the wait_for_instantlock() helper hunk in test/functional/test_framework/test_framework.py

    The PR does touch test_framework.py, but only for unrelated RPC bool/type-check adaptations (1 -> True, None handling, stringifying operator_reward) and util.py config plumbing. So this still matches the stale-branch / missing-sync-fix failure mode tracked here, not an in-scope regression from #7242.

  2. thepastaclaw commented on Apr 21, 2026

    @thepastaclaw
    CollaboratorAuthor

    Another occurrence on #7242 (head 24daadcf6d, run https://github.com/dashpay/dash/actions/runs/24580460650):

    Fresh scope check on this PR still says the failure is not caused by the branch changes:

    • git diff upstream/develop..HEAD --name-only does not touch test/functional/interface_zmq_dash.py
    • this PR does touch test/functional/test_framework/test_framework.py, but only in unrelated RPC-bool / masternode helper hunks; the wait_for_instantlock() helper path is unchanged
    • the branch is still behind the develop-side fixes already referenced here (c79c2d981d, d8ac7815e3, merged via 3957e09d59)

    So this is another stale-branch / pre-existing InstantLock wait timeout occurrence, not a regression introduced by PR #7242.

  3. thepastaclaw commented on Apr 24, 2026

    @thepastaclaw
    CollaboratorAuthor

    Another occurrence on PR #7242 (head 24daadcf6dd495c76b5ec2ab6c1e6c9dfec979b8):

    Same failure shape as before:

    • interface_zmq_dash.py --legacy-wallet
    • timeout while waiting for self.wait_for_instantlock(rpc_raw_tx_1_hash)
    • traceback still ends in AssertionError: Predicate "" not true after ... seconds

    Scope check for PR #7242 still shows the branch does not touch test/functional/interface_zmq_dash.py, and the diff against upstream/develop does not modify the wait_for_instantlock() helper hunk in test/functional/test_framework/test_framework.py either. This is still stale-branch / unrelated test debt rather than a regression from the PR changes.

  4. thepastaclaw commented on Apr 26, 2026

    @thepastaclaw
    CollaboratorAuthor

    Another occurrence on PR #7290 (head 317c3768959a6b606e1583d5a1b3f1b89846f408):

    Same top-level symptom as before:

    • interface_zmq_dash.py
    • timeout in DashZMQTest.test_instantsend_publishers() while waiting in self.wait_for_instantlock(rpc_raw_tx_1_hash)

    Fresh scope check for this PR still says the branch did not introduce the failing test path:

    • git diff upstream/develop..HEAD -- test/functional/interface_zmq_dash.py is empty
    • the branch diff in test/functional/test_framework/test_framework.py does not touch the wait_for_instantlock() helper hunk

    New detail from the archived combined log: this run hit repeated lower-level LLMQ/DKG failures before the InstantLock timeout, for example:

    • CSigningManager::AsyncSignIfMember -- failed to select quorum
    • CQuorumManager::BuildQuorumFromCommitment -- ... ReadContributions and BuildQuorumContributions ... failed
    • ActiveDKGSessionHandler::InitNewQuorum -- ... quorum initialization failed for llmq_test_instantsend

    Also unlike the earlier stale-branch reports, this PR head already contains the prior wait/sync fixes referenced here (3957e09d59, c79c2d981d, d8ac7815e3 are all ancestors of 317c376895).

    So this looks broader than “branch missing wait_for_instantlock sync fixes”: the same interface_zmq_dash timeout is now also showing up when the wait/sync patches are already present, with the proximate failure being intermittent LLMQ quorum/DKG setup under UBSan rather than a change from PR #7290 itself.

  5. thepastaclaw commented on Apr 28, 2026

    @thepastaclaw
    CollaboratorAuthor

    Another scope-check confirmation for PR #7290 / tracker issue #1106.

    Fresh verification in the PR worktree (backport-0.26-batch-2-staging, head 317c3768959a6b606e1583d5a1b3f1b89846f408):

    git log upstream/develop..HEAD -- test/functional/interface_zmq_dash.py test/functional/test_framework/test_framework.py test/functional/feature_multikeysporks.py src/instantsend/instantsend.cpp src/spork.h

    returns no commits. The batch-2 backport merges themselves only touch wallet/rpc/compactblocks/config-args paths; they do not introduce changes in the failing InstantSend/ZMQ path.

    The relevant differences in test_framework.py / spork.h / instantsend.cpp are from the branch being behind develop, not from PR #7290's own commits. In particular, HEAD..upstream/develop contains cfad4141d6 / afcf3d148e (SPORK_3 removal follow-ups) plus other post-branch fixes.

    So this is still not an in-scope regression from PR #7290 itself. No branch change is appropriate here; tracker issue #1106 should be closed as unrelated CI/test debt tracked here.

  6. thepastaclaw commented on Apr 28, 2026

    @thepastaclaw
    CollaboratorAuthor

    Fresh tracker-1107 scope recheck on PR #7242 head 24daadcf6d:\n\n- failing workflow run: https://github.com/dashpay/dash/actions/runs/24580460650\n- failing job: https://github.com/dashpay/dash/actions/runs/24580460650/job/71880218247\n- failure: interface_zmq_dash.py --legacy-wallet timed out again in self.wait_for_instantlock(rpc_raw_tx_1_hash)\n\nI rechecked the exact branch diff with git diff upstream/develop..HEAD --name-only on the PR head. It still does not touch test/functional/interface_zmq_dash.py, and the PR’s test_framework.py hunks are unrelated to the wait_for_instantlock() helper path. I also verified this PR head already contains c79c2d981d, d8ac7815e3, and 3957e09d59 in its ancestry, so this occurrence is not explained by a branch-local omission of those earlier wait/sync fixes.\n\nSo this is still unrelated CI/test debt rather than an in-scope regression from PR #7242, and no PR-branch mutation is appropriate for this failure alone.

  7. thepastaclaw commented on Apr 28, 2026

    @thepastaclaw
    CollaboratorAuthor

    Another scope check for PR #7290 / tracker #1121:

    Rechecked in the PR worktree at 317c3768959a6b606e1583d5a1b3f1b89846f408:

    git log upstream/develop..HEAD -- test/functional/interface_zmq_dash.py test/functional/test_framework/test_framework.py src/instantsend/instantsend.cpp src/spork.h

    returns no PR commits touching the failing InstantSend/ZMQ path, so the failure is still not coming from this backport batch's own changes.

    One new data point: the latest completed develop CI run I checked (25059702916, head 7e524b10eb2b2cc7c8289931c70bc99c65622fcf) had linux64_ubsan-test / Test source green. So this does not look like a deterministic branch-local regression that needs a PR-branch fixup; it still looks intermittent, with this occurrence lining up with the same lower-level LLMQ/DKG instability already noted earlier in this issue.

  8. thepastaclaw commented on May 4, 2026

    @thepastaclaw
    CollaboratorAuthor

    Fresh current-develop recheck (tracker #1176):

    • Worktree: ~/Projects/dash/worktrees/tracker-1176
    • Ref: upstream/develop at 4a0fd854750caf2a8945ac567447ddd3a341c246 (confirmed HEAD...upstream/develop = 0 0 after fetch)
    • Confirmed ancestors present:
      • c79c2d981d — refactor: improve is-waiting helper by including mempool sync bump of mocktime
      • d8ac7815e3 — test: fix additional IS lock sync sites that passed nodes=[self.nodes[0]]
      • merge 3957e09d59 — Merge #7241: test: fix intermittent IS lock timeouts by syncing mempools before waiting

    Local validation on this current develop head:

    PYTHONPATH=/Users/claw/Library/Python/3.9/lib/python/site-packages \
      .venv/bin/python test/functional/interface_zmq_dash.py --legacy-wallet --tmpdir=/tmp/dash-tracker-1176-legacy-1
    # passed, PRNG seed 385690915384688966
    
    # repeated legacy runs:
    # passed, PRNG seed 6775434089131662380
    # passed, PRNG seed 1972982303692681424
    
    PYTHONPATH=/Users/claw/Library/Python/3.9/lib/python/site-packages \
      .venv/bin/python test/functional/interface_zmq_dash.py --tmpdir=/tmp/dash-tracker-1176-default-1
    # passed, PRNG seed 4132865026894307226
    

    So I still do not have a current-develop reproduction from the local check: interface_zmq_dash.py passed 3/3 in --legacy-wallet mode and 1/1 in default wallet mode at current develop.

    Recommended handling remains:

    1. For branches that are actually missing the wait/sync fixes, rebase onto current develop or cherry-pick exactly:
      • c79c2d981d
      • d8ac7815e3
        (or take the containing merge 3957e09d59 if the branch policy allows merge commits).
    2. For unrelated PR/backport branches that already contain those commits, do not mutate the PR branch just because interface_zmq_dash.py timed out. First scope-check whether the branch touches test/functional/interface_zmq_dash.py, wait_for_instantlock() in test_framework.py, or the LLMQ/InstantSend code path. If not, treat the failure as intermittent test/CI debt and rerun/track it here rather than force-pushing unrelated changes.

    No Dash Core PR opened from this check because I could not reproduce the flake on current develop.

  9. thepastaclaw commented on May 18, 2026

    @thepastaclaw
    CollaboratorAuthor

    New CI data point from #7326 (v23.1.x PR head 4d269f73):

    Failure mode matches this issue: test_instantsend_publishers() timed out waiting for an InstantLock. The failing predicate was wait_for_instantlock() on node.getrawtransaction(txid, True)["instantlock"], timing out after 240s at interface_zmq_dash.py:300 / test_framework.py:1959.

    Scope note: #7326 only changes activate_by_name() spork synchronization waits on the v23.1.x branch (wait_for_sporks_same(timeout=60) around temporary spork17 disable/restore). It does not change interface_zmq_dash.py or wait_for_instantlock(), and the failing test does not use that activation helper. This looks like the same pre-existing release-branch InstantLock/ZMQ flake tracked here, not a regression introduced by #7326.

  10. thepastaclaw commented on May 18, 2026

    @thepastaclaw
    CollaboratorAuthor

    New CI data point from #7327 (v23.1.x PR head fb832662):

    Failure mode matches this issue: test_instantsend_publishers() timed out waiting for an InstantLock. The failing predicate was wait_for_instantlock() on node.getrawtransaction(txid, True)["instantlock"], timing out after 240s at interface_zmq_dash.py:300 / test_framework.py:1959.

    Scope note: #7327 only extends the feature_dip3_v19.py spork-sync wait on v23.1.x by threading an optional spork_sync_timeout argument through activate_by_name(). It does not change interface_zmq_dash.py, wait_for_instantlock(), or InstantSend/ZMQ logic. This looks like the same pre-existing release-branch InstantLock/ZMQ flake tracked here, not a regression introduced by #7327.

  11. thepastaclaw commented on May 18, 2026

    @thepastaclaw
    CollaboratorAuthor

    Confirmed during tracker #1305 scope check for #7327:

    • Failing job: https://github.com/dashpay/dash/actions/runs/26015843317/job/76472838342
    • Failing test: interface_zmq_dash.py --legacy-wallet
    • Failure: test_instantsend_publishers() timed out after 240s in wait_for_instantlock(rpc_raw_tx_1_hash, self.nodes[0]), waiting for getrawtransaction(txid, True)["instantlock"].
    • PR head: fb832662467a241a70c97826d2d3863bd44858f1

    Scope check:

    git diff upstream/v23.1.x..HEAD --name-only
     test/functional/feature_dip3_v19.py
     test/functional/test_framework/test_framework.py
    

    The PR's test_framework.py hunk only threads spork_sync_timeout through activate_by_name() / wait_for_sporks_same(). It does not touch interface_zmq_dash.py, wait_for_instantlock(), InstantSend, or LLMQ signing code. The modified feature_dip3_v19.py test passed in this same job.

    This is another occurrence of the existing release-branch InstantLock/ZMQ flake, not a regression introduced by #7327. No PR branch change was made.

  12. thepastaclaw commented on May 20, 2026

    @thepastaclaw
    CollaboratorAuthor

    Additional occurrence seen on PR #7333 (ci: tolerate conflict comment permission failures):

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions