fix: stop transient RPC errors from wedging the tx verifier

[jorgecuesta]

fix: stop transient RPC errors from wedging the tx verifier

What happened on mainnet

On 2026-08-10 at 13:34:30 UTC the scheduled VerifyPendingTransactions run stopped making progress and stayed Running for three days. With ScheduleOverlapPolicy: Skip, every subsequent fire (one every 30s) was skipped — SkippedOverlap climbed to 9533 — so no pending transaction was verified for three days. Five stake transactions sat at status = pending, and the suppliers behind them never got their nodes row, so users saw their supplier as "pending" in the UI while it was staked and healthy on chain.

The chain of causation, taken from the workflow history:

5   ACTIVITY_TASK_SCHEDULED  listPendingWithHash  args=[]
7   ACTIVITY_TASK_COMPLETED  result=[{"id":387,"executionHeight":872892}]
11  ACTIVITY_TASK_SCHEDULED  verifyTxHash  args=['387']
13  ACTIVITY_TASK_FAILED     msg={"code":-32603,"message":"Internal error",
                                  "data":"could not find results for height #872893"}
16+ WORKFLOW_TASK_FAILED     "VerifyPendingTransactions: all transactions failed"  (forever)

1. Transaction 387 landed on chain in block 872893 (index 1772, code 0 — successful).
2. The block scan matched the hash inside block 872893, then called blockResults(872893) on the same height. That call hit a node which had the block but had not yet persisted its ABCI responses, which CometBFT reports as could not find results for height #N. Per-node cometbft_consensus_height from Prometheus confirms all three backends were sitting exactly at 872893 at that moment — the failure landed on the block's commit boundary.
3. blockResults sat outside verifyTransaction's try/catch (only comet.block(h) was wrapped), so the error escaped instead of degrading to the unavailable state the tri-state contract already defines for "the tx is on chain but we cannot read its outcome yet".
4. The activity's retry policy — 3 attempts at the default 1s base — exhausted in about 4 seconds, well inside a single ~60s block, so a condition tied to the chain's own cadence was reported as a permanent failure.
5. It was the only pending transaction, so results.every(rejected) held and the workflow threw WorkflowError.
6. WorkflowError extends plain Error, not TemporalFailure. The SDK fails the workflow only for TemporalFailure subclasses; anything else fails the workflow task, which then retries forever. The run stayed Running, and Skip did the rest.

Note that WorkflowError is exported from @temporalio/workflow and does construct — verified at runtime, instanceof Error true, instanceof TemporalFailure false. Two comments in provider-workflows claimed it was unexported and threw a TypeError; the conclusion they drew was right but the stated mechanism was wrong, and they are corrected here.

What this PR changes

packages/pocket — blockResults is wrapped so a failed read of a matched tx's result row degrades to result-missing → unavailable, which makes the verifier retry rather than fail. Covered by a new regression test that reproduces the exact production error string.

middleman-workflows — all five throw new WorkflowError(...) sites become non-retryable ApplicationFailure, matching the pattern provider-workflows already used. A failed execution now ends the run, so the schedule's next fire proceeds normally.

Retry policy for the verification sweep (both apps) — from 3 attempts in ~4s to 5 attempts with a 5s base, 2x backoff, capped at 30s. That spans more than one block, so a commit-boundary hiccup resolves itself without any human involvement.

noWorkflowError.test.ts — a structural guard over both apps' workflow directories that fails if new WorkflowError( reappears. It greps source rather than exercising the runtime, because @temporalio/testing is not a dependency of this repo and adding it was out of scope here. Verified to go red by injecting the pattern.

Verification

• packages/pocket: 7/7 (new commit-boundary test red before the fix, green after)
• middleman-workflows: 91/91 · provider-workflows: 33/33
• turbo lint, turbo build, no-console-guard.sh: pass

Review follow-ups already folded in

• blockResults failing is now result-unreadable, kept apart from result-missing, because the old shared path logged "matched tx has no block result entry" for a read that never happened. Both still map to unavailable.
• The ApplicationFailure details payload is capped at 20 reasons in both apps. An oversized payload would fail the workflow task that reports the failure — the same wedge this PR removes.
• The new test's blockResults rejection is one-shot. clearAllMocks() resets calls but not implementations, and this jest config sets neither resetMocks nor restoreMocks, so a persistent rejection would have silently poisoned any confirmed-path test appended after it.

Known gaps, deliberately not addressed here

• The watchdog cannot see this failure mode. evaluateLiveness returns healthy whenever runningActions.length > 0 (packages/temporal/src/scheduleWatchdog.ts:166), so a schedule blocked by a wedged run reads green in the Workflows UI. This PR removes one cause, not the blind spot: a nondeterminism error after a workflow-code deploy, an unregistered activity name, or an oversized payload all reproduce the same silent stall, and the new grep-based guard catches none of them. The concrete proposal for a follow-up is to have evaluateLiveness treat a running action older than K × the schedule interval as stale instead of trusting its mere presence.
• The sweep's worst-case wall clock roughly doubles and nothing bounds it. Per activity it goes from ~363s (3 × 120s startToCloseTimeout + ~3s backoff) to ~665s (5 × 120s + 65s backoff), and createOptions sets no workflowRunTimeout. This is accepted here because the run now ends — under the old behaviour it never did — so starvation is bounded by the failure rather than unbounded. Two follow-ups worth doing: a run timeout on the sweep, and nonRetryableErrorTypes for the deterministic activity throws (tx missing hash, tx not found), which today burn all 5 attempts plus 65s of backoff on permanent conditions.
• ExecuteTransaction has a false-failure path on the broadcast leg. The if (!result) guard is unreachable — sendTransaction always returns an object, returning { transactionHash: '', success: false } from its own catch — so a socket timeout on broadcastTxSync after the node accepted the tx into mempool takes the !result.transactionHash branch, marks the tx Failure and emails the user. Pre-existing and untouched here, but it is the same false-negative class this PR removes from the verification leg, and it is worth its own fix.
• provider-workflows/src/activities/index.ts has three pre-existing rawLog type errors (TS2339), present on staging before this branch and confirmed by re-running the check with this branch's provider changes stashed. check-types is continue-on-error: true in CI.
• apps/middleman-workflows/src/lib/blockchain/index.ts is dead code — nothing imports it — and carries its own copy of the same blockResults call. Left untouched; it should probably be deleted.