Repository navigation
fix(sdk): ban a node that fails a block query, but not one that is busy - #5839
Conversation
The two block queries turned bans off for every retryable failure, though the reason was only a node answering that it is busy or rate limited. A node that was down was then only moved out of rotation, and with a short node list it came straight back. `RequestSettings::ban_busy_address` lets a request keep a busy node unbanned while every other failure is banned as before. The block queries now use it in place of `ban_failed_address: Some(false)`. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 8 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
|
📖 Book Preview built successfully. Download the preview from the workflow artifacts. Updated at 2026-11-10T12:08:32.034Z |
|
Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post |
|
🕓 Review not started yet because the new head is waiting for the 30-minute push debounce.
Commit 2602a9e. Normal review starts when eligible; priority review starts as soon as a slot is available. |
thepastaclaw
left a comment
There was a problem hiding this comment.
Final validation — Phase 1 + Phase 2
The busy-node exemption is correctly propagated through request settings and transport error wrappers, and the relevant CI checks are successful. However, both block-query overrides now health-ban a node for the documented transient state where Tenderdash has stored a newest block but not its results; this can make a healthy single-node client unavailable for 60 seconds and is an introduced availability regression. The Dashmate helper image and review-hygiene checks were still pending.
🔴 1 blocking
Review provenance
Source: reviewer 1: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 2: muse-spark-1.3-contributor (agent: phase1-reviewer, role: rust-quality); reviewer 3: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 4: gpt-6.1-sol (agent: phase2-reviewer, role: architecture-layering); reviewer 5: gpt-6.1-sol (agent: phase2-reviewer, role: rust-quality); reviewer 6: gpt-6.1-sol (agent: phase2-reviewer, role: general); reviewer 7: gpt-6.1-sol (agent: phase2-reviewer, role: architecture-layering); reviewer 8: gpt-6.1-sol (agent: phase2-reviewer, role: rust-quality); final verifier: gpt-6.1-sol (agent: sol-verifier, role: final-verifier)
- Triage:
normalbygpt-6.1-sol(effort low) — This is a cross-cutting Rust client behavior change affecting request retry, timeout, error classification, node rotation, and ban state across multiple crates, but it does not alter consensus, funds, cryptography, signatures, peer-facing deserialization, or storage. - Phase 1 reviewers:
muse-spark-1.3-contributor— general (completed, effort xhigh); agentphase1-reviewer,muse-spark-1.3-contributor— rust-quality (completed, effort xhigh); agentphase1-reviewer - Phase 1 model:
muse-spark-1.3-contributor— not quota-gated; passed overgemini-3.8-flash-high(antigravity quota lookup failed),glm-5.3-flash(not used above high effort; tier asks max) - Second Phase-2 pass: on this first full review every Phase-2 reviewer also ran a second, independent time, and the verifier weighed both sets
- Fresh verifier:
gpt-6.1-sol— final-verifier; agentsol-verifier - Phase 2 reviewers:
gpt-6.1-sol— general (completed, effort high); agentphase2-reviewer,gpt-6.1-sol— architecture-layering (completed, effort high); agentphase2-reviewer,gpt-6.1-sol— rust-quality (completed, effort high); agentphase2-reviewer,gpt-6.1-sol— general (completed, effort high); agentphase2-reviewer,gpt-6.1-sol— architecture-layering (completed, effort high); agentphase2-reviewer,gpt-6.1-sol— rust-quality (completed, effort high); agentphase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review (PastaClaw). How to work through them:
- Verify each finding against the code; fix only what is real, as narrowly as the finding.
- If a finding is wrong, decline it on its thread in one or two sentences; the bot answers once and stops.
- To defer one, link a tracking issue on its thread (or reply asking the bot to file one).
- Suggestions are optional: fix them forward in a later PR rather than pushing another round just for them.
- Address every current finding in one push: self-review the full diff first, and don't push piecemeal.
- Keep the PR's scope; don't grow it to satisfy hardening suggestions.
- Carried items were raised in an earlier round: read the thread first, and don't re-fix what you already fixed, declined or deferred there. Disputed and deferred items are already handled and are not listed.
In `packages/rs-dapi-client/src/transport/grpc.rs`:
- [BLOCKING] packages/rs-dapi-client/src/transport/grpc.rs:825-829: Do not health-ban nodes while newest-block results are still pending
The new `ban_busy_address: Some(false)` setting only exempts `ResourceExhausted`; `Unavailable` responses still enter the normal health-ban path. This is a problem for a documented, normal DAPI state: `held_block_results` in `packages/rs-dapi/src/services/platform_service/get_blocks.rs` returns `Unavailable` when Tenderdash has stored a block but has not stored its execution results yet, and `get_owner_transactions_in_block` uses the same helper. A request for a transaction-containing newest block can therefore ban an otherwise healthy node for the initial 60-second period. With one configured node, the retry immediately reaches `NoAvailableAddressesToRetry`; unrelated requests also cannot use that node during the ban. Distinguish this results-not-ready response from genuine transport unavailability at the rs-dapi/rs-dapi-client boundary, and preserve non-banning retry behavior for it. Add a regression that requests through a single node, observes results-not-ready, then succeeds once results become available.
|
Your move: thepastaclaw requested changes on this head; dismiss the review or push a fix; thepastaclaw left review threads unresolved; resolve them. |
Tenderdash stores a block before its results, so rs-dapi can answer a block query for the newest block with `Unavailable` for a moment. That answer now carries a `retry-after` header, and the client counts an `Unavailable` answer with that header as a busy node: a block query retries it without a ban. An `Unavailable` answer without the header is still a failure. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Waiting for bot review — thepastaclaw not yet. Wait for the missing reviews, or a writer can post |
…uests Brings in #5556 (shared broadcast keeps uncertain submission outcomes), #5823 (proven state transition outcome, getRecentStateTransitionStatus), #5815, #5834, #5825, #5835, #5838 and #5839. One textual conflict, in packages/rs-sdk/src/platform/transition/broadcast.rs at the end of broadcast_with_retries. This branch (#5837) had the shared broadcast fail over on a nonce-ahead answer and on a missing DPNS owner (retry_with_additional_error, from #5562), and made is_address_nonce_ahead public for the wallet's single-dispatch broadcast. #5556 replaced both failovers with the plain shared retry, so that a concrete refusal ends sending, with tests saying so (should_stop_on_nonce_rejection_without_ rebuilding_or_resending, should_stop_on_concrete_rejection_regardless_of_ node_failover_settings), and removed is_address_nonce_ahead and is_missing_transition_owner. The file is taken as v5.1-dev has it. The wallet's single-dispatch broadcast (broadcast_outcome::track_delivery) still goes on to the next dispatch after a nonce-ahead answer, as #5837 meant: the classifier moves into broadcast_outcome.rs, unchanged, and its doc no longer says the SDK's own broadcast fails over too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Basic explanation
What this does: When a DAPI node fails a request, the client normally stops using that node for a while (a "ban"). The two block queries,
getBlocksandgetOwnerTransactionsInBlock, had bans switched off completely, because a node often refuses them only for being busy or rate limited. This change makes the rule finer: a node that says it is busy, or that the newest block is not ready yet, is still not banned, but a node that fails in any other way (down, or not answering in time) is banned again.Value: A client with a short node list no longer keeps going back to a dead node for block queries. Before, such a node was only moved out of rotation and could come straight back, so each block query could wait out the long response timeout on it again. Busy nodes stay available to the client's other requests, as before.
Risks: Low. No consensus, state or wire change; only the client's choice of nodes changes, and only for the two block queries. One case where a healthy node can now get the first, 60 second ban: a
getBlocksthat takes longer than the node's 15 second handler limit, whose time on a live node has not been measured. Theretry-afterheader that keeps a node unbanned while its newest block is not ready has not been checked through a live gateway; the existingratelimit-resetheader travels the same way. Rust code that buildsRequestSettingsorAppliedRequestSettingsby naming every field needs one more field.Issue being fixed or feature implemented
Follow-up to #5815, from two of its review threads (one, two). Both block queries set
ban_failed_address: Some(false). The reason was overload answers (ResourceExhaustedfrom rs-dapi's concurrency limits and from the gateway's 30 requests per hour ongetBlocks), but the setting turned off bans for every retryable failure.What was done?
RequestSettings::ban_busy_address(and the same field onAppliedRequestSettings), defaulttrue. Whenfalse, a node that answersResourceExhaustedis moved out of rotation without a ban; every other retryable failure is still banned asban_failed_addresssays.CanRetry::is_node_busy(), defaultfalse, true for a gRPCResourceExhaustedstatus (and for the not-ready answer below) and forwarded byTransportError,DapiClientErrorandExecutionError.update_address_ban_statusbans whenban_failed_address && (ban_busy_address || !error.is_node_busy()).Unavailablefor a moment.results_not_stored_yet(rs-dapi,get_blocks.rs) now adds aretry-after: 1header to that answer, andis_node_busy()is also true forUnavailablewith aretry-afterheader. The node is then retried without a ban;Unavailablewithout the header is still a failure.GetBlocksRequestandGetOwnerTransactionsInBlockRequestnow setban_busy_address: Some(false)in place ofban_failed_address: Some(false).rs-dapi-clientthat buildRequestSettingsfield by field (rs-sdk,rs-platform-wallet,wasm-sdk) and the example in the book passban_busy_address: None.Examples (a block query, client with two nodes A and B)
Node A is down:
Node A is at its limit:
Any other request, for example
getIdentity:Public interface changes
rs-dapi-client(re-exported bydash-sdk)ban_busy_address: Some(false)means: do not ban a node for aResourceExhaustedanswer to this request, or for anUnavailableanswer that carries aretry-afterheader.ban_failed_address: Some(false)still turns off every ban, busy or not...RequestSettings::default()or throughfinalize()compiles unchanged. Code that names every field has to add the new one.CanRetryimplementors outside this repository keep compiling; their errors count as "not busy" until they override the method.Behaviour of the two block queries
ResourceExhausted(rs-dapi limit or gateway rate limit)Unavailablewithretry-after(the newest block's results are not stored yet)Unavailablewithout the header,DeadlineExceeded, connection error, other retryable codesrs-dapi: theUnavailableanswer for a block whose results are not stored yet now carries the response headerretry-after: 1. Code and message are unchanged.wasm-sdk: no new JavaScript setting.banFailedAddresskeeps its meaning.What stays the same: every other request, the ban durations, the wire format, and the SDK's own handling of an answer whose proof does not verify (it never read these per-request settings).
How Has This Been Tested?
cargo test -p rs-dapi-client --all-features --test rate_limit_ban: 14 passed. New cases: both block queries ban onUnavailableandDeadlineExceededbut not onResourceExhausted, with and without a rate limit header; other requests still ban a busy node;ban_failed_address: falsestill turns every ban off;is_node_busyis forwarded through each error wrapper; and with a list of only two nodes under the default settings, a failed node is asked once in ten requests and banned, while a busy node is asked again and not banned. A client with a single node that first answers "results not ready" gets its answer on the retry with the node unbanned, and the same node answering plainUnavailableis banned.cargo test -p rs-dapi --lib -- platform_service clients::tenderdash: 179 passed, including that the not-ready answer isUnavailablewithretry-after: 1.cargo test -p rs-dapi-client --all-features --lib --test request_inventory --test request_errors --test unimplemented_failover: 176 passed, 1 ignored.cargo test -p dash-sdk --lib block_query: 12 passed.cargo test -p platform-wallet --lib locate: 23 passed.cargo check -p dash-sdk -p platform-wallet -p wasm-sdk --tests: clean.cargo clippy -p rs-dapi -p rs-dapi-client --all-features --all-targets --no-deps -- -D warnings: clean.Not tested: against a live network.
Breaking Changes
None on the wire or in behaviour of existing requests. At the source level, Rust code that builds
RequestSettingsorAppliedRequestSettingsby naming every field must addban_busy_address, as with earlier fields added to these structs.Related
Checklist:
structure.rs, regeneratedgrovedb-structure.json, and checked the structure viewer link posted on this pull requestFor repository code-owners and collaborators only
🤖 Generated with Claude Code
PR Hygiene ·
2602a9e/skip-botsproceeds without the ones not yet reported/self-reviewedrust-dapi(packages/rs-dapi/src/services/platform_service/get_blocks.rs) — lklimek or shumkovrs-platform-wallet(packages/rs-platform-wallet/src/wallet/asset_lock/sync/locate.rs) — HashEngineering or ZocoLini or llbartekll or romchornyirust-sdk(packages/rs-sdk/src/sdk.rs) — lklimek or shumkovjs-wasm-sdk(packages/wasm-sdk/src/settings.rs) — shumkovWhen every merge requirement is met, the
PR Hygienecheck passes. Reviewer limits do not block merging; other required GitHub checks and protections still apply.