You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
interface_zmq_dash.py can time out waiting for an InstantLock on older/stale branches that do not contain the later wait_for_instantlock() sync fixes now present on develop.
Failing CI examples
PR branch failure (dashpay/dash#7237, head c425124c7e57a3f5e851d9f5ceb8934d850e84ff):
which returns no commits, so the PR branch itself does not modify the failing test/helper path.
develop already contains the fixes for this exact issue:
c79c2d981d — refactor: improve is-waiting helper by including mempool sync bump of mocktime
d8ac7815e3 — test: fix additional IS lock sync sites that passed nodes=[self.nodes[0]]
Those commits restore the wait_for_instantlock() behavior to bump mocktime and sync mempools across the relevant nodes, which is required so the MN signer nodes actually see the transaction before InstantLock signing.
The failing branch is behind develop and does not contain those fixes, so it can still hit the old timeout.
Reproduction notes
Use a branch that predates c79c2d981d / d8ac7815e3
Run python3 test/functional/interface_zmq_dash.py in a CI-capable environment with python3-zmq installed
Observe intermittent timeout waiting for getrawtransaction(...)["instantlock"]
Expected behavior
Branches with the current develop test framework fixes should not rely on the old unsynchronized behavior and should avoid this timeout.
Suggested resolution
For stale backport branches, rebase or cherry-pick the helper/test fixes from:
The failure is the same InstantLock timeout in interface_zmq_dash.py:
AssertionError: Predicate "" not true after 240.0 seconds
def check_instantlock():
try:
return all(
node.getrawtransaction(txid, True)["instantlock"]
for txid in txids
for node in nodes
)
except Exception:
return False
I rechecked the current PR head and both of these are empty / unchanged relative to the branch diff:
the wait_for_instantlock() helper hunk in test/functional/test_framework/test_framework.py
The PR does touch test_framework.py, but only for unrelated RPC bool/type-check adaptations (1 -> True, None handling, stringifying operator_reward) and util.py config plumbing. So this still matches the stale-branch / missing-sync-fix failure mode tracked here, not an in-scope regression from #7242.
failure: interface_zmq_dash.py --legacy-wallet timed out waiting for getrawtransaction(txid, True)["instantlock"] in DashZMQTest.test_instantsend_publishers()
Fresh scope check on this PR still says the failure is not caused by the branch changes:
git diff upstream/develop..HEAD --name-only does not touch test/functional/interface_zmq_dash.py
this PR does touch test/functional/test_framework/test_framework.py, but only in unrelated RPC-bool / masternode helper hunks; the wait_for_instantlock() helper path is unchanged
the branch is still behind the develop-side fixes already referenced here (c79c2d981d, d8ac7815e3, merged via 3957e09d59)
So this is another stale-branch / pre-existing InstantLock wait timeout occurrence, not a regression introduced by PR #7242.
timeout while waiting for self.wait_for_instantlock(rpc_raw_tx_1_hash)
traceback still ends in AssertionError: Predicate "" not true after ... seconds
Scope check for PR #7242 still shows the branch does not touch test/functional/interface_zmq_dash.py, and the diff against upstream/develop does not modify the wait_for_instantlock() helper hunk in test/functional/test_framework/test_framework.py either. This is still stale-branch / unrelated test debt rather than a regression from the PR changes.
timeout in DashZMQTest.test_instantsend_publishers() while waiting in self.wait_for_instantlock(rpc_raw_tx_1_hash)
Fresh scope check for this PR still says the branch did not introduce the failing test path:
git diff upstream/develop..HEAD -- test/functional/interface_zmq_dash.py is empty
the branch diff in test/functional/test_framework/test_framework.py does not touch the wait_for_instantlock() helper hunk
New detail from the archived combined log: this run hit repeated lower-level LLMQ/DKG failures before the InstantLock timeout, for example:
CSigningManager::AsyncSignIfMember -- failed to select quorum
CQuorumManager::BuildQuorumFromCommitment -- ... ReadContributions and BuildQuorumContributions ... failed
ActiveDKGSessionHandler::InitNewQuorum -- ... quorum initialization failed for llmq_test_instantsend
Also unlike the earlier stale-branch reports, this PR head already contains the prior wait/sync fixes referenced here (3957e09d59, c79c2d981d, d8ac7815e3 are all ancestors of 317c376895).
So this looks broader than “branch missing wait_for_instantlock sync fixes”: the same interface_zmq_dash timeout is now also showing up when the wait/sync patches are already present, with the proximate failure being intermittent LLMQ quorum/DKG setup under UBSan rather than a change from PR #7290 itself.
returns no commits. The batch-2 backport merges themselves only touch wallet/rpc/compactblocks/config-args paths; they do not introduce changes in the failing InstantSend/ZMQ path.
The relevant differences in test_framework.py / spork.h / instantsend.cpp are from the branch being behind develop, not from PR #7290's own commits. In particular, HEAD..upstream/develop contains cfad4141d6 / afcf3d148e (SPORK_3 removal follow-ups) plus other post-branch fixes.
So this is still not an in-scope regression from PR #7290 itself. No branch change is appropriate here; tracker issue #1106 should be closed as unrelated CI/test debt tracked here.
Fresh tracker-1107 scope recheck on PR #7242 head 24daadcf6d:\n\n- failing workflow run: https://github.com/dashpay/dash/actions/runs/24580460650\n- failing job: https://github.com/dashpay/dash/actions/runs/24580460650/job/71880218247\n- failure: interface_zmq_dash.py --legacy-wallet timed out again in self.wait_for_instantlock(rpc_raw_tx_1_hash)\n\nI rechecked the exact branch diff with git diff upstream/develop..HEAD --name-only on the PR head. It still does not touch test/functional/interface_zmq_dash.py, and the PR’s test_framework.py hunks are unrelated to the wait_for_instantlock() helper path. I also verified this PR head already contains c79c2d981d, d8ac7815e3, and 3957e09d59 in its ancestry, so this occurrence is not explained by a branch-local omission of those earlier wait/sync fixes.\n\nSo this is still unrelated CI/test debt rather than an in-scope regression from PR #7242, and no PR-branch mutation is appropriate for this failure alone.
returns no PR commits touching the failing InstantSend/ZMQ path, so the failure is still not coming from this backport batch's own changes.
One new data point: the latest completed develop CI run I checked (25059702916, head 7e524b10eb2b2cc7c8289931c70bc99c65622fcf) had linux64_ubsan-test / Test sourcegreen. So this does not look like a deterministic branch-local regression that needs a PR-branch fixup; it still looks intermittent, with this occurrence lining up with the same lower-level LLMQ/DKG instability already noted earlier in this issue.
So I still do not have a current-develop reproduction from the local check: interface_zmq_dash.py passed 3/3 in --legacy-wallet mode and 1/1 in default wallet mode at current develop.
Recommended handling remains:
For branches that are actually missing the wait/sync fixes, rebase onto current develop or cherry-pick exactly:
c79c2d981d
d8ac7815e3
(or take the containing merge 3957e09d59 if the branch policy allows merge commits).
For unrelated PR/backport branches that already contain those commits, do not mutate the PR branch just because interface_zmq_dash.py timed out. First scope-check whether the branch touches test/functional/interface_zmq_dash.py, wait_for_instantlock() in test_framework.py, or the LLMQ/InstantSend code path. If not, treat the failure as intermittent test/CI debt and rerun/track it here rather than force-pushing unrelated changes.
No Dash Core PR opened from this check because I could not reproduce the flake on current develop.
Failure mode matches this issue: test_instantsend_publishers() timed out waiting for an InstantLock. The failing predicate was wait_for_instantlock() on node.getrawtransaction(txid, True)["instantlock"], timing out after 240s at interface_zmq_dash.py:300 / test_framework.py:1959.
Scope note: #7326 only changes activate_by_name() spork synchronization waits on the v23.1.x branch (wait_for_sporks_same(timeout=60) around temporary spork17 disable/restore). It does not change interface_zmq_dash.py or wait_for_instantlock(), and the failing test does not use that activation helper. This looks like the same pre-existing release-branch InstantLock/ZMQ flake tracked here, not a regression introduced by #7326.
Failure mode matches this issue: test_instantsend_publishers() timed out waiting for an InstantLock. The failing predicate was wait_for_instantlock() on node.getrawtransaction(txid, True)["instantlock"], timing out after 240s at interface_zmq_dash.py:300 / test_framework.py:1959.
Scope note: #7327 only extends the feature_dip3_v19.py spork-sync wait on v23.1.x by threading an optional spork_sync_timeout argument through activate_by_name(). It does not change interface_zmq_dash.py, wait_for_instantlock(), or InstantSend/ZMQ logic. This looks like the same pre-existing release-branch InstantLock/ZMQ flake tracked here, not a regression introduced by #7327.
Failure: test_instantsend_publishers() timed out after 240s in wait_for_instantlock(rpc_raw_tx_1_hash, self.nodes[0]), waiting for getrawtransaction(txid, True)["instantlock"].
The PR's test_framework.py hunk only threads spork_sync_timeout through activate_by_name() / wait_for_sporks_same(). It does not touch interface_zmq_dash.py, wait_for_instantlock(), InstantSend, or LLMQ signing code. The modified feature_dip3_v19.py test passed in this same job.
This is another occurrence of the existing release-branch InstantLock/ZMQ flake, not a regression introduced by #7327. No PR branch change was made.
Failure: interface_zmq_dash.py --legacy-wallet timed out after 240s in wait_for_instantlock() while waiting for node.getrawtransaction(txid, True)["instantlock"] to become true.
Activity
thepastaclaw commentedon Apr 18, 2026
Another occurrence from PR #7242 (wallet deprecation + RPC type checking backport), current head
24daadcf6d.linux64_ubsan-test / Test sourceThe failure is the same InstantLock timeout in
interface_zmq_dash.py:I rechecked the current PR head and both of these are empty / unchanged relative to the branch diff:
git log upstream/develop..HEAD -- test/functional/interface_zmq_dash.pywait_for_instantlock()helper hunk intest/functional/test_framework/test_framework.pyThe PR does touch
test_framework.py, but only for unrelated RPC bool/type-check adaptations (1->True,Nonehandling, stringifyingoperator_reward) andutil.pyconfig plumbing. So this still matches the stale-branch / missing-sync-fix failure mode tracked here, not an in-scope regression from #7242.thepastaclaw commentedon Apr 21, 2026
Another occurrence on #7242 (head
24daadcf6d, run https://github.com/dashpay/dash/actions/runs/24580460650):interface_zmq_dash.py --legacy-wallettimed out waiting forgetrawtransaction(txid, True)["instantlock"]inDashZMQTest.test_instantsend_publishers()Fresh scope check on this PR still says the failure is not caused by the branch changes:
git diff upstream/develop..HEAD --name-onlydoes not touchtest/functional/interface_zmq_dash.pytest/functional/test_framework/test_framework.py, but only in unrelated RPC-bool / masternode helper hunks; thewait_for_instantlock()helper path is unchangedc79c2d981d,d8ac7815e3, merged via3957e09d59)So this is another stale-branch / pre-existing InstantLock wait timeout occurrence, not a regression introduced by PR #7242.
thepastaclaw commentedon Apr 24, 2026
Another occurrence on PR #7242 (head
24daadcf6dd495c76b5ec2ab6c1e6c9dfec979b8):Same failure shape as before:
interface_zmq_dash.py --legacy-walletself.wait_for_instantlock(rpc_raw_tx_1_hash)AssertionError: Predicate "" not true after ... secondsScope check for PR #7242 still shows the branch does not touch
test/functional/interface_zmq_dash.py, and the diff againstupstream/developdoes not modify thewait_for_instantlock()helper hunk intest/functional/test_framework/test_framework.pyeither. This is still stale-branch / unrelated test debt rather than a regression from the PR changes.thepastaclaw commentedon Apr 26, 2026
Another occurrence on PR #7290 (head
317c3768959a6b606e1583d5a1b3f1b89846f408):linux64_ubsan-test / Test sourceSame top-level symptom as before:
interface_zmq_dash.pyDashZMQTest.test_instantsend_publishers()while waiting inself.wait_for_instantlock(rpc_raw_tx_1_hash)Fresh scope check for this PR still says the branch did not introduce the failing test path:
git diff upstream/develop..HEAD -- test/functional/interface_zmq_dash.pyis emptytest/functional/test_framework/test_framework.pydoes not touch thewait_for_instantlock()helper hunkNew detail from the archived combined log: this run hit repeated lower-level LLMQ/DKG failures before the InstantLock timeout, for example:
CSigningManager::AsyncSignIfMember -- failed to select quorumCQuorumManager::BuildQuorumFromCommitment -- ... ReadContributions and BuildQuorumContributions ... failedActiveDKGSessionHandler::InitNewQuorum -- ... quorum initialization failed for llmq_test_instantsendAlso unlike the earlier stale-branch reports, this PR head already contains the prior wait/sync fixes referenced here (
3957e09d59,c79c2d981d,d8ac7815e3are all ancestors of317c376895).So this looks broader than “branch missing wait_for_instantlock sync fixes”: the same interface_zmq_dash timeout is now also showing up when the wait/sync patches are already present, with the proximate failure being intermittent LLMQ quorum/DKG setup under UBSan rather than a change from PR #7290 itself.
thepastaclaw commentedon Apr 28, 2026
Another scope-check confirmation for PR #7290 / tracker issue #1106.
interface_zmq_dash.py --legacy-walletinlinux64_ubsan-test / Test sourceFresh verification in the PR worktree (
backport-0.26-batch-2-staging, head317c3768959a6b606e1583d5a1b3f1b89846f408):returns no commits. The batch-2 backport merges themselves only touch wallet/rpc/compactblocks/config-args paths; they do not introduce changes in the failing InstantSend/ZMQ path.
The relevant differences in
test_framework.py/spork.h/instantsend.cppare from the branch being behinddevelop, not from PR #7290's own commits. In particular,HEAD..upstream/developcontainscfad4141d6/afcf3d148e(SPORK_3 removal follow-ups) plus other post-branch fixes.So this is still not an in-scope regression from PR #7290 itself. No branch change is appropriate here; tracker issue #1106 should be closed as unrelated CI/test debt tracked here.
thepastaclaw commentedon Apr 28, 2026
Fresh tracker-1107 scope recheck on PR #7242 head
24daadcf6d:\n\n- failing workflow run: https://github.com/dashpay/dash/actions/runs/24580460650\n- failing job: https://github.com/dashpay/dash/actions/runs/24580460650/job/71880218247\n- failure:interface_zmq_dash.py --legacy-wallettimed out again inself.wait_for_instantlock(rpc_raw_tx_1_hash)\n\nI rechecked the exact branch diff withgit diff upstream/develop..HEAD --name-onlyon the PR head. It still does not touchtest/functional/interface_zmq_dash.py, and the PR’stest_framework.pyhunks are unrelated to thewait_for_instantlock()helper path. I also verified this PR head already containsc79c2d981d,d8ac7815e3, and3957e09d59in its ancestry, so this occurrence is not explained by a branch-local omission of those earlier wait/sync fixes.\n\nSo this is still unrelated CI/test debt rather than an in-scope regression from PR #7242, and no PR-branch mutation is appropriate for this failure alone.thepastaclaw commentedon Apr 28, 2026
Another scope check for PR #7290 / tracker #1121:
interface_zmq_dash.py --legacy-walletinlinux64_ubsan-test / Test sourceRechecked in the PR worktree at
317c3768959a6b606e1583d5a1b3f1b89846f408:returns no PR commits touching the failing InstantSend/ZMQ path, so the failure is still not coming from this backport batch's own changes.
One new data point: the latest completed
developCI run I checked (25059702916, head7e524b10eb2b2cc7c8289931c70bc99c65622fcf) hadlinux64_ubsan-test / Test sourcegreen. So this does not look like a deterministic branch-local regression that needs a PR-branch fixup; it still looks intermittent, with this occurrence lining up with the same lower-level LLMQ/DKG instability already noted earlier in this issue.thepastaclaw commentedon May 4, 2026
Fresh current-
developrecheck (tracker #1176):~/Projects/dash/worktrees/tracker-1176upstream/developat4a0fd854750caf2a8945ac567447ddd3a341c246(confirmedHEAD...upstream/develop=0 0after fetch)c79c2d981d—refactor: improve is-waiting helper by including mempool sync bump of mocktimed8ac7815e3—test: fix additional IS lock sync sites that passed nodes=[self.nodes[0]]3957e09d59—Merge #7241: test: fix intermittent IS lock timeouts by syncing mempools before waitingLocal validation on this current develop head:
So I still do not have a current-
developreproduction from the local check:interface_zmq_dash.pypassed 3/3 in--legacy-walletmode and 1/1 in default wallet mode at current develop.Recommended handling remains:
developor cherry-pick exactly:c79c2d981dd8ac7815e3(or take the containing merge
3957e09d59if the branch policy allows merge commits).interface_zmq_dash.pytimed out. First scope-check whether the branch touchestest/functional/interface_zmq_dash.py,wait_for_instantlock()intest_framework.py, or the LLMQ/InstantSend code path. If not, treat the failure as intermittent test/CI debt and rerun/track it here rather than force-pushing unrelated changes.No Dash Core PR opened from this check because I could not reproduce the flake on current
develop.thepastaclaw commentedon May 18, 2026
New CI data point from #7326 (
v23.1.xPR head4d269f73):linux64_multiprocess-test / Test sourcetest_logs-linux64_multiprocess-4d269f73)interface_zmq_dash.py --legacy-walletFailure mode matches this issue:
test_instantsend_publishers()timed out waiting for an InstantLock. The failing predicate waswait_for_instantlock()onnode.getrawtransaction(txid, True)["instantlock"], timing out after 240s atinterface_zmq_dash.py:300/test_framework.py:1959.Scope note: #7326 only changes
activate_by_name()spork synchronization waits on thev23.1.xbranch (wait_for_sporks_same(timeout=60)around temporary spork17 disable/restore). It does not changeinterface_zmq_dash.pyorwait_for_instantlock(), and the failing test does not use that activation helper. This looks like the same pre-existing release-branch InstantLock/ZMQ flake tracked here, not a regression introduced by #7326.thepastaclaw commentedon May 18, 2026
New CI data point from #7327 (
v23.1.xPR headfb832662):linux64_multiprocess-test / Test sourcetest_logs-linux64_multiprocess-fb832662)interface_zmq_dash.py --legacy-walletFailure mode matches this issue:
test_instantsend_publishers()timed out waiting for an InstantLock. The failing predicate waswait_for_instantlock()onnode.getrawtransaction(txid, True)["instantlock"], timing out after 240s atinterface_zmq_dash.py:300/test_framework.py:1959.Scope note: #7327 only extends the
feature_dip3_v19.pyspork-sync wait onv23.1.xby threading an optionalspork_sync_timeoutargument throughactivate_by_name(). It does not changeinterface_zmq_dash.py,wait_for_instantlock(), or InstantSend/ZMQ logic. This looks like the same pre-existing release-branch InstantLock/ZMQ flake tracked here, not a regression introduced by #7327.thepastaclaw commentedon May 18, 2026
Confirmed during tracker #1305 scope check for #7327:
interface_zmq_dash.py --legacy-wallettest_instantsend_publishers()timed out after 240s inwait_for_instantlock(rpc_raw_tx_1_hash, self.nodes[0]), waiting forgetrawtransaction(txid, True)["instantlock"].fb832662467a241a70c97826d2d3863bd44858f1Scope check:
The PR's
test_framework.pyhunk only threadsspork_sync_timeoutthroughactivate_by_name()/wait_for_sporks_same(). It does not touchinterface_zmq_dash.py,wait_for_instantlock(), InstantSend, or LLMQ signing code. The modifiedfeature_dip3_v19.pytest passed in this same job.This is another occurrence of the existing release-branch InstantLock/ZMQ flake, not a regression introduced by #7327. No PR branch change was made.
thepastaclaw commentedon May 20, 2026
Additional occurrence seen on PR #7333 (
ci: tolerate conflict comment permission failures):linux64_tsan-test / Test sourcetest_logs-linux64_tsan-2992445binterface_zmq_dash.py --legacy-wallettimed out after 240s inwait_for_instantlock()while waiting fornode.getrawtransaction(txid, True)["instantlock"]to become true..github/workflows/predict-conflicts.yml; it does not touchinterface_zmq_dash.py, InstantSend/LLMQ code, or test framework code. This is therefore another instance of the pre-existing flaky InstantLock timeout tracked here, not caused by PR ci: tolerate conflict comment permission failures #7333.