Repository navigation
blindbit-lib: tell a stalled block transfer from a missing block (SNB-688) - #62
Closed
setavenger wants to merge 10 commits into
Closed
setavenger wants to merge 10 commits into
setavenger wants to merge 10 commits into
Conversation
Remove public items nothing calls, in this workspace or in the other repos that use blindbit-lib (blindbit-node-apps, testkit, the talk benchmark): - broadcast_tx: its only caller, friglet's Electrum broadcast, uses friglet's own relay since #26. - Scanner::from_config, get_last_scanned_block_height_rescan, staged_mut, take_staged and the commented-out get_outputs. - watch_chain: friglet calls watch_chain_until; its doc moves there. - BlockIdentifierDisplay (only .0.block_height was read), OwnedOutput (never built), ProbableMatch::new, _match_short_pubkey_bytes and the commented-out ScannerConfig Default. electrum_index becomes a private module; its items stay re-exported. - The spent-outpoint and probabilistic-match channels and the code that only fed them. They had no subscribers and never sent anything: the sends were guarded by Sender::is_empty(), which means "no queued value", so they were always skipped. The malformed-hash branch that guarded them cannot run: check_block_identifier rejects such a block before scan_short_block_data sees it. The "possible spend of owned output" log line stays. - save_to_file's back-fill of keys and max_label_num: stage holds them from construction on and only take_staged could clear them. A test run with assertions in place of the back-fill passed every workspace test. - P2P: TimeoutParams setters the handshake never reads (it only uses the ping interval; the socket timeouts are set directly), history notes about deleted code, and a retry doc that predates the witness-stripped and bad-block retries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scanner::new and from_changeset each wrote out the whole struct; both now hand their staged changeset to one private constructor, from_stage, which derives the rest (checkpoints with genesis, owned outputs, remembered hashes, Electrum index) from it. - "New indexer, labels 0..=max, then apply_changeset" was written in from_changeset and in the reorg rollback; it is restored_indexer now, with the reason the labels must come first. - The genesis checkpoint hash was built in three places; genesis_hash(). - new() merged add_label(0..=max) into the stage a second time after initial_changeset() already carried those labels (label_lookup is a set, so the merge changed nothing); dropped. - The remembered-hash lookup in reorg.rs is one helper. - Scanner.last_scanned_block_height_rescan was only restored, lowered on rollback and copied into every save. The stage already holds the same value, so the field goes; the ChangeSet field and the state format stay as they are. A probe (not committed) dumped stage, saved file and full scanner state for fresh scanners, a test-produced wallet, the old-format fixture, restores and rollbacks of each, before and after this change: all 42 outputs identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The scan loop's two arms (no match / matched block) each advanced the scanned height, recorded the block hash and notified Electrum; that now happens once after the optional matched-block step, in the same order as before. The periodic checkpoint still runs only for blocks without a match (a matched block is saved earlier). - The oracle's display-order block hash is decoded once: check_block_identifier returns it (via health.rs's indexed_block_hash, the same length and all-zero rules), and the P2P fetch uses it instead of decoding the bytes again. block_hash_from_oracle goes. - The balance over the sparse checkpoints is one helper, balance_at, used for the per-block "balance" log, the "balance after scan" log and the owned-output tests (which had a third copy). - The matched block's scan_progress write is dropped; the progress notification right after it writes the same value. - The debug loop that logged every graph transaction after each matched block is dropped. - apply_matched_block computed every transaction's txid once per matched txid; it now builds a txid map and hashes each transaction once. - OracleStreams only delegated to the oracle client: the client now implements BlockStreamSource itself (it already implemented OracleProbe). An identity match in probabilistic_match goes. - Stale TODOs and comments (notifications that no longer exist, a config-constructor TODO the talk benchmark rules out, Electrum index docs that missed the restore and broadcast writers). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…SNB-666) The test wallet's keys, the Tokio runner, the never-connecting oracle client, state-file paths, TempState and the scanner constructors were defined again in five to six test modules. They now live in scanner/test_support.rs: - oracle_client() uses the caller's runtime when there is one and a process-wide one otherwise, so it serves both the async tests and the synchronous owned-output and BIP-352 tests. All tests now point at 127.0.0.1:1, where nothing listens (two modules used [::1]:50051 and 127.0.0.1:8333). - state_file(tag) returns a path no other call returns, so modules can no longer collide on a tag (reorg and owned-output tests both used "restart"). - scanner(tag), scanner_at(path, max_label_num) and restore_from(path) replace stream_safety's scanner, reorg's wallet/restart, owned_outputs' scanner/restore, witness' restart and state_file's new_scanner/client. bip352_vectors keeps its own constructor (vector keys, mainnet) on the shared client and paths. Also: - a_fetched_block_keeps_its_witnesses is dropped: the next test fetches the same kind of block whole on its second connection and compares the whole block, witnesses included. - p2p_tests' Node has three constructors instead of four, and the witness-stripping loop is one helper (stripped) used by the fake node and the witness tests. - The stale allow(dead_code) on the vectors' `sending` field goes (the field is read). - The pre-owned-outputs restore test restores the file it just saved instead of a re-serialised copy, so it leaves no file behind. Test count: 254 -> 253 passed (the one duplicate removed), 1 ignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-688) bitcoin-p2p reads each message with read_exact, so a read timeout part way through the block throws away the bytes read so far. A node that sent half a block and went quiet therefore looked like one that sent nothing: at the deadline the fetch reported NoAnswer, with advice that the node is probably pruned, and NoAnswer is not tried again in that round. The fetch now sends a ping right after getdata. A node handles a peer's messages in order, so it answers the ping once it has sent the block or decided not to. At the deadline, a pong means the node does not serve the block (NoAnswer, as before). No pong means the node went quiet, so the attempt fails as a broken connection, which is tried again on a fresh one. The wait itself is unchanged. Tests: a_transfer_that_stalls_mid_block_is_retried_on_a_fresh_connection and a_transfer_that_keeps_stalling_is_not_reported_as_a_missing_block use the fake node's StallMidBlock. Before the fix both ended after one attempt with NoAnswer and "the node sent nothing for 303 ms ... It is probably pruned below this height". The fake node's Ignore now answers pings, as Bitcoin Core does while ignoring a request for a block it does not have. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A block fetched only because an owned output may be spent carries no matched txids. The txid filter still hashed every transaction of it (about 2 ms per block); the code before #52 did nothing there. Skip the filter when there are no matched txids. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…NB-666) Brings #52's spend-only fix (skip the txid filter when no txid matched) into this stacked branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- owned_outputs_labels_and_spends_survive_restart builds its scanner on the state file it saves, restores and removes. state_file() returns a new path on every call, so the second call gave the scanner a different file than the one the test checks. - default_scanner in the BIP-352 vectors points its P2P peer at 127.0.0.1:1 like every other test (it is never contacted). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A no-pong deadline is still exposed as a broken connection, but keep its private attempt classification long enough to allow one fresh connection and then end the round. This reports a persistent stall after about 181 s instead of consuming all five 90 s attempts, while other transient failures retain their configured retry budget. Correct the NoAnswer message now that a matching pong proves the peer did answer, and strengthen the fake-peer tests for both recovery and the quiet-stall retry cap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 3, 2026
Owner
Author
|
Merged into |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #57; review only the commits after it.
Bug fix for SNB-688. It changes how a stalled block download is classified and limits the cost of retrying it.
What was wrong
bitcoin-p2preads each message withread_exact. A read timeout part way through a block discards that partial message. A peer that sent half a block and then went quiet therefore looked like a peer that sent nothing. At the deadline the fetch reportedNoAnswer, with advice that the node was probably pruned or still syncing.The initial fix distinguished that stall, but exposed it as the existing transient
Brokenfailure. A persistently quiet peer then consumed all five default 90 s attempts: about 7.5 minutes before the daemon could publish the failure and begin its next scan round.Fix
The fetch sends one
pingimmediately aftergetdata. Bitcoin Core handles these peer messages in order, so it answers the ping only after it has sent the requested block or decided not to send one. If no block has arrived by the deadline:NoAnswer, because the peer processed the preceding block request without sending the block.Brokenfailure.A quiet deadline gets one fresh-connection retry when the retry policy permits it. A second quiet deadline ends the round. Other transient failures retain the policy's full retry budget. The existing 1 s delay before attempt two remains, so a persistent hard stall surfaces after about 181 s with the default 90 s deadline, rather than about 465 s across five attempts.
The wait itself keeps the same per-read timeout and deadline. Cancellation remains prompt because it still shuts down the socket and wins the async
select!. There is no public API change and no error-string matching; the quiet classification exists only between one connection attempt and the fetch loop.The
NoAnswertext now says the node “did not send the block” instead of saying it “sent nothing”, since a matching pong is an answer.Wire cost: one 32-byte ping and one 32-byte pong per block fetch, 64 bytes total.
Regression coverage
The real-TCP fake-peer tests cover both paths:
a_transfer_that_stalls_mid_block_is_retried_on_a_fresh_connection:[StallMidBlock, Serve]recovers on connection two.a_transfer_that_keeps_stalling_gets_one_fresh_retry_then_ends_the_round: uses a five-attempt policy but asserts that two persistent stalls open exactly two connections and returnBroken, without missing/pruned advice.a_silent_node_behind_the_block_is_reported_as_syncing: a peer that answers the ping but omits the block remainsNoAnswerand gets the syncing advice.a_stop_during_a_stalled_block_download_ends_the_scan_at_once_and_it_resumes_there: cancellation still ends a stalled transfer promptly.The strengthened persistent-stall test fails on the previous implementation: it consumes all five attempts and eventually reports connection refused after the two scripted peers are exhausted. It passes with this change after exactly two connections.
Verification (debug build, this branch)
NoAnswer, and cancellation tests: pass.cargo check --workspace --all-targets --all-features: 0 warnings.cargo check -p blindbit-lib --no-default-features --all-targets: 5 warnings (baseline 5).cargo clippy --no-deps -p friglet -p friglet-ipc -p friglet-tray --all-targets -- -D warnings: clean.cargo fmt -p friglet -p friglet-ipc -p friglet-tray -- --check: clean.cargo test --workspace: 255 passed, 1 ignored, 0 failed.Bitcoin Core's message ordering was verified from its peer-processing source. A live Bitcoin Core mid-transfer fault injection was not performed; the runtime regression coverage uses the repository's TCP fake peer.
Left out
Ending the wait early once the pong arrives without a block would shorten the missing-block case. This PR retains the existing deadline before returning
NoAnswer.