Skip to content

fix(sdk): ban a node that fails a block query, but not one that is busy - #5839

Merged
QuantumExplorer merged 2 commits into
v5.1-devfrom
fix/block-query-follow-ups
Nov 10, 2026
Merged

QuantumExplorer merged 2 commits into
v5.1-devfrom
fix/block-query-follow-ups

Conversation

@QuantumExplorer

@QuantumExplorer QuantumExplorer commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Basic explanation

What this does: When a DAPI node fails a request, the client normally stops using that node for a while (a "ban"). The two block queries, getBlocks and getOwnerTransactionsInBlock, had bans switched off completely, because a node often refuses them only for being busy or rate limited. This change makes the rule finer: a node that says it is busy, or that the newest block is not ready yet, is still not banned, but a node that fails in any other way (down, or not answering in time) is banned again.

Value: A client with a short node list no longer keeps going back to a dead node for block queries. Before, such a node was only moved out of rotation and could come straight back, so each block query could wait out the long response timeout on it again. Busy nodes stay available to the client's other requests, as before.

Risks: Low. No consensus, state or wire change; only the client's choice of nodes changes, and only for the two block queries. One case where a healthy node can now get the first, 60 second ban: a getBlocks that takes longer than the node's 15 second handler limit, whose time on a live node has not been measured. The retry-after header that keeps a node unbanned while its newest block is not ready has not been checked through a live gateway; the existing ratelimit-reset header travels the same way. Rust code that builds RequestSettings or AppliedRequestSettings by naming every field needs one more field.

Issue being fixed or feature implemented

Follow-up to #5815, from two of its review threads (one, two). Both block queries set ban_failed_address: Some(false). The reason was overload answers (ResourceExhausted from rs-dapi's concurrency limits and from the gateway's 30 requests per hour on getBlocks), but the setting turned off bans for every retryable failure.

What was done?

  • RequestSettings::ban_busy_address (and the same field on AppliedRequestSettings), default true. When false, a node that answers ResourceExhausted is moved out of rotation without a ban; every other retryable failure is still banned as ban_failed_address says.
  • CanRetry::is_node_busy(), default false, true for a gRPC ResourceExhausted status (and for the not-ready answer below) and forwarded by TransportError, DapiClientError and ExecutionError.
  • update_address_ban_status bans when ban_failed_address && (ban_busy_address || !error.is_node_busy()).
  • Tenderdash stores a block before its results, so rs-dapi answers a block query for the newest block with Unavailable for a moment. results_not_stored_yet (rs-dapi, get_blocks.rs) now adds a retry-after: 1 header to that answer, and is_node_busy() is also true for Unavailable with a retry-after header. The node is then retried without a ban; Unavailable without the header is still a failure.
  • GetBlocksRequest and GetOwnerTransactionsInBlockRequest now set ban_busy_address: Some(false) in place of ban_failed_address: Some(false).
  • The four places outside rs-dapi-client that build RequestSettings field by field (rs-sdk, rs-platform-wallet, wasm-sdk) and the example in the book pass ban_busy_address: None.

Examples (a block query, client with two nodes A and B)

Node A is down:

# Before
A fails with Unavailable  ->  A leaves rotation, not banned; with two nodes it comes straight back
next block queries        ->  can try A again and wait on it

# After
A fails with Unavailable  ->  A is banned (60 seconds first, then longer)
next block queries        ->  go to B only

Node A is at its limit:

# Before and after
A answers ResourceExhausted  ->  the request moves to B; A is not banned and keeps serving other requests

Any other request, for example getIdentity:

# Before and after
A answers ResourceExhausted  ->  A is banned for the advertised window (or 60 seconds without one)

Public interface changes

rs-dapi-client (re-exported by dash-sdk)

pub struct RequestSettings {
    // ...
    pub ban_failed_address: Option<bool>,
    pub ban_busy_address: Option<bool>,   // new; None means true
    // ...
}

pub struct AppliedRequestSettings {
    // ...
    pub ban_failed_address: bool,
    pub ban_busy_address: bool,           // new
    // ...
}

pub trait CanRetry {
    // ...
    fn is_node_busy(&self) -> bool { false }   // new, with a default
}
  • ban_busy_address: Some(false) means: do not ban a node for a ResourceExhausted answer to this request, or for an Unavailable answer that carries a retry-after header. ban_failed_address: Some(false) still turns off every ban, busy or not.
  • Code that builds either settings struct with ..RequestSettings::default() or through finalize() compiles unchanged. Code that names every field has to add the new one.
  • CanRetry implementors outside this repository keep compiling; their errors count as "not busy" until they override the method.

Behaviour of the two block queries

Failure of a node Before After
ResourceExhausted (rs-dapi limit or gateway rate limit) not banned not banned
Unavailable with retry-after (the newest block's results are not stored yet) not banned not banned
Unavailable without the header, DeadlineExceeded, connection error, other retryable codes not banned banned

rs-dapi: the Unavailable answer for a block whose results are not stored yet now carries the response header retry-after: 1. Code and message are unchanged.

wasm-sdk: no new JavaScript setting. banFailedAddress keeps its meaning.

What stays the same: every other request, the ban durations, the wire format, and the SDK's own handling of an answer whose proof does not verify (it never read these per-request settings).

How Has This Been Tested?

  • cargo test -p rs-dapi-client --all-features --test rate_limit_ban: 14 passed. New cases: both block queries ban on Unavailable and DeadlineExceeded but not on ResourceExhausted, with and without a rate limit header; other requests still ban a busy node; ban_failed_address: false still turns every ban off; is_node_busy is forwarded through each error wrapper; and with a list of only two nodes under the default settings, a failed node is asked once in ten requests and banned, while a busy node is asked again and not banned. A client with a single node that first answers "results not ready" gets its answer on the retry with the node unbanned, and the same node answering plain Unavailable is banned.
  • cargo test -p rs-dapi --lib -- platform_service clients::tenderdash: 179 passed, including that the not-ready answer is Unavailable with retry-after: 1.
  • cargo test -p rs-dapi-client --all-features --lib --test request_inventory --test request_errors --test unimplemented_failover: 176 passed, 1 ignored.
  • cargo test -p dash-sdk --lib block_query: 12 passed.
  • cargo test -p platform-wallet --lib locate: 23 passed.
  • cargo check -p dash-sdk -p platform-wallet -p wasm-sdk --tests: clean.
  • cargo clippy -p rs-dapi -p rs-dapi-client --all-features --all-targets --no-deps -- -D warnings: clean.

Not tested: against a live network.

Breaking Changes

None on the wire or in behaviour of existing requests. At the source level, Rust code that builds RequestSettings or AppliedRequestSettings by naming every field must add ban_busy_address, as with earlier fields added to these structs.

Related

Checklist:

  • I have performed a self-review of my own code
  • My Rust, FFI, WASM, Swift or Kotlin changes follow the coding conventions
  • I have commented my code, particularly in hard-to-understand areas
  • I have added or updated relevant unit/integration/functional/e2e tests
  • I have added "!" to the title and described breaking changes in the corresponding section if my code contains any
  • I have made corresponding changes to the documentation if needed
  • If I added or changed GroveDB structure, I described it in the area's structure.rs, regenerated grovedb-structure.json, and checked the structure viewer link posted on this pull request
  • If I added or changed a system contract's document schemas, I checked the contract viewer link posted on this pull request

For repository code-owners and collaborators only

  • I have assigned this pull request to a milestone

🤖 Generated with Claude Code

PR Hygiene · 2602a9e

  • Bots — thepastaclaw not yet — /skip-bots proceeds without the ones not yet reported
  • Self-review — post /self-reviewed
  • Reviewer requests paused — your 5 review slots are occupied; this PR is excluded from reviewers' queues. Required approvals still count without a slot.
  • Build running
  • Approvals
    • files with no dedicated owner — you own it
    • rust-dapi (packages/rs-dapi/src/services/platform_service/get_blocks.rs) — lklimek or shumkov
    • rs-platform-wallet (packages/rs-platform-wallet/src/wallet/asset_lock/sync/locate.rs) — HashEngineering or ZocoLini or llbartekll or romchornyi
    • rust-sdk (packages/rs-sdk/src/sdk.rs) — lklimek or shumkov
    • js-wasm-sdk (packages/wasm-sdk/src/settings.rs) — shumkov

When every merge requirement is met, the PR Hygiene check passes. Reviewer limits do not block merging; other required GitHub checks and protections still apply.

The two block queries turned bans off for every retryable failure, though the
reason was only a node answering that it is busy or rate limited. A node that
was down was then only moved out of rotation, and with a short node list it
came straight back.

`RequestSettings::ban_busy_address` lets a request keep a busy node unbanned
while every other failure is banned as before. The block queries now use it
in place of `ban_failed_address: Some(false)`.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 8 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between f970f0e and 2602a9e.

Files selected for processing: 13

  • book/src/sdk/builder-pattern.md
  • packages/rs-dapi-client/src/address_list.rs
  • packages/rs-dapi-client/src/dapi_client.rs
  • packages/rs-dapi-client/src/executor.rs
  • packages/rs-dapi-client/src/lib.rs
  • packages/rs-dapi-client/src/request_settings.rs
  • packages/rs-dapi-client/src/transport.rs
  • packages/rs-dapi-client/src/transport/grpc.rs
  • packages/rs-dapi-client/tests/rate_limit_ban.rs
  • packages/rs-dapi/src/services/platform_service/get_blocks.rs
  • packages/rs-platform-wallet/src/wallet/asset_lock/sync/locate.rs
  • packages/rs-sdk/src/sdk.rs
  • packages/wasm-sdk/src/settings.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions github-actions Bot added this to the v5.1.0 milestone Nov 10, 2026
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

📖 Book Preview built successfully.

Download the preview from the workflow artifacts.
To view locally: download the artifact, unzip, and open index.html.

Updated at 2026-11-10T12:08:32.034Z

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions github-actions Bot added the waiting-bots Waiting for the review bots to report on this head label Nov 10, 2026
@thepastaclaw

thepastaclaw commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

🕓 Review not started yet because the new head is waiting for the 30-minute push debounce.

  • Request normal review — click when the PR is ready for review.
  • Request priority review — click to move this review to the front of the queue.

Commit 2602a9e. Normal review starts when eligible; priority review starts as soon as a slot is available.

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final validation — Phase 1 + Phase 2

The busy-node exemption is correctly propagated through request settings and transport error wrappers, and the relevant CI checks are successful. However, both block-query overrides now health-ban a node for the documented transient state where Tenderdash has stored a newest block but not its results; this can make a healthy single-node client unavailable for 60 seconds and is an introduced availability regression. The Dashmate helper image and review-hygiene checks were still pending.

🔴 1 blocking

Review provenance

Source: reviewer 1: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 2: muse-spark-1.3-contributor (agent: phase1-reviewer, role: rust-quality); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 4: gpt-6.1-sol (agent: phase2-reviewer, role: architecture-layering); reviewer 5: gpt-6.1-sol (agent: phase2-reviewer, role: rust-quality); reviewer 6: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 7: gpt-6.1-sol (agent: phase2-reviewer, role: architecture-layering); reviewer 8: gpt-6.1-sol (agent: phase2-reviewer, role: rust-quality); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)

  • Triage: normal by gpt-6.1-sol (effort low) — This is a cross-cutting Rust client behavior change affecting request retry, timeout, error classification, node rotation, and ban state across multiple crates, but it does not alter consensus, funds, cryptography, signatures, peer-facing deserialization, or storage.
  • Phase 1 reviewers: muse-spark-1.3-contributor — general (completed, effort xhigh); agent phase1-reviewer, muse-spark-1.3-contributor — rust-quality (completed, effort xhigh); agent phase1-reviewer
  • Phase 1 model: muse-spark-1.3-contributor — not quota-gated; passed over gemini-3.8-flash-high (antigravity quota lookup failed), glm-5.3-flash (not used above high effort; tier asks max)
  • Second Phase-2 pass: on this first full review every Phase-2 reviewer also ran a second, independent time, and the verifier weighed both sets
  • Fresh verifier: gpt-6.1-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.1-sol — general (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — architecture-layering (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — rust-quality (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — general (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — architecture-layering (completed, effort high); agent phase2-reviewer, gpt-6.1-sol — rust-quality (completed, effort high); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review (PastaClaw). How to work through them:
- Verify each finding against the code; fix only what is real, as narrowly as the finding.
- If a finding is wrong, decline it on its thread in one or two sentences; the bot answers once and stops.
- To defer one, link a tracking issue on its thread (or reply asking the bot to file one).
- Suggestions are optional: fix them forward in a later PR rather than pushing another round just for them.
- Address every current finding in one push: self-review the full diff first, and don't push piecemeal.
- Keep the PR's scope; don't grow it to satisfy hardening suggestions.
- Carried items were raised in an earlier round: read the thread first, and don't re-fix what you already fixed, declined or deferred there. Disputed and deferred items are already handled and are not listed.

In `packages/rs-dapi-client/src/transport/grpc.rs`:
- [BLOCKING] packages/rs-dapi-client/src/transport/grpc.rs:825-829: Do not health-ban nodes while newest-block results are still pending
  The new `ban_busy_address: Some(false)` setting only exempts `ResourceExhausted`; `Unavailable` responses still enter the normal health-ban path. This is a problem for a documented, normal DAPI state: `held_block_results` in `packages/rs-dapi/src/services/platform_service/get_blocks.rs` returns `Unavailable` when Tenderdash has stored a block but has not stored its execution results yet, and `get_owner_transactions_in_block` uses the same helper. A request for a transaction-containing newest block can therefore ban an otherwise healthy node for the initial 60-second period. With one configured node, the retry immediately reaches `NoAvailableAddressesToRetry`; unrelated requests also cannot use that node during the ban. Distinguish this results-not-ready response from genuine transport unavailability at the rs-dapi/rs-dapi-client boundary, and preserve non-banning retry behavior for it. Add a regression that requests through a single node, observes results-not-ready, then succeeds once results become available.

@github-actions

Copy link
Copy Markdown
Contributor

Your move: thepastaclaw requested changes on this head; dismiss the review or push a fix; thepastaclaw left review threads unresolved; resolve them.
Full checklist in the description.

@github-actions github-actions Bot added waiting-self-review Waiting for the author to post /self-reviewed and removed waiting-bots Waiting for the review bots to report on this head labels Nov 10, 2026
Tenderdash stores a block before its results, so rs-dapi can answer a block
query for the newest block with `Unavailable` for a moment. That answer now
carries a `retry-after` header, and the client counts an `Unavailable` answer
with that header as a busy node: a block query retries it without a ban. An
`Unavailable` answer without the header is still a failure.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post /skip-bots to proceed without them; blocking findings still need addressing.
Full checklist in the description.

@github-actions github-actions Bot added waiting-bots Waiting for the review bots to report on this head and removed waiting-self-review Waiting for the author to post /self-reviewed labels Nov 10, 2026
@QuantumExplorer
QuantumExplorer merged commit f603f98 into v5.1-dev Nov 10, 2026
26 of 28 checks passed
@QuantumExplorer
QuantumExplorer deleted the fix/block-query-follow-ups branch November 10, 2026 12:25
QuantumExplorer added a commit that referenced this pull request Nov 10, 2026
…uests

Brings in #5556 (shared broadcast keeps uncertain submission outcomes),
#5823 (proven state transition outcome, getRecentStateTransitionStatus),
#5815, #5834, #5825, #5835, #5838 and #5839.

One textual conflict, in packages/rs-sdk/src/platform/transition/broadcast.rs
at the end of broadcast_with_retries. This branch (#5837) had the shared
broadcast fail over on a nonce-ahead answer and on a missing DPNS owner
(retry_with_additional_error, from #5562), and made is_address_nonce_ahead
public for the wallet's single-dispatch broadcast. #5556 replaced both
failovers with the plain shared retry, so that a concrete refusal ends
sending, with tests saying so (should_stop_on_nonce_rejection_without_
rebuilding_or_resending, should_stop_on_concrete_rejection_regardless_of_
node_failover_settings), and removed is_address_nonce_ahead and
is_missing_transition_owner. The file is taken as v5.1-dev has it.

The wallet's single-dispatch broadcast (broadcast_outcome::track_delivery)
still goes on to the next dispatch after a nonce-ahead answer, as #5837
meant: the classifier moves into broadcast_outcome.rs, unchanged, and its
doc no longer says the SDK's own broadcast fails over too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-bots Waiting for the review bots to report on this head

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants