Skip to content

fix(friglet): real tip headers, honest broadcast, honest fee answers - #26

Merged
setavenger merged 1 commit into
integrationfrom
fix/electrum-tip-broadcast-fees
Oct 1, 2026
Merged

setavenger merged 1 commit into
integrationfrom
fix/electrum-tip-broadcast-fees

Conversation

@setavenger

Copy link
Copy Markdown
Owner

This makes friglet's Electrum server tell Sparrow the truth about three things: new blocks, broadcasts, and fees.

Linear: SNB-626 (tip headers) · SNB-627 (broadcast) · SNB-628 (fees) · follow-up SNB-629 (verification)

Tip headers (SNB-626)

Problem

  • New-block notifications sent whatever header hex sat in WalletElectrumIndex.tip.
  • For blocks without wallet matches, that is the previous matched block's header. Before the first match, after a restart, and after a fix(scanner): detect chain reorganisations and roll back to the fork point #21 rollback, it is empty.
  • Sparrow 2.5.5 ignores a tip whose header does not parse (ElectrumServer.java:2927-2938), so confirmations froze. A stale header was recorded at the wrong height.

Fix

  • tip_header no longer reads the tip's stored hex.
    • It uses the header stored for exactly that height.
    • Otherwise it asks the oracle for the block hash at that height, on one reused gRPC channel.
    • It then fetches that block's 80-byte header from the P2P peer with getheaders and an empty locator, on one reused connection, and checks it against the hash.
  • Each tip is announced once. A tip replaced at the same height has a new hash, so it is announced again.
  • If the header can't be fetched, the tip is not announced and friglet retries every 15 s. Before, it sent an empty or stale header.
  • blockchain.block.header backfill uses the same path. It no longer downloads a whole block to read 80 bytes.

Honest broadcast (SNB-627)

Problem

  • The old path sent an unsolicited tx and always answered success.
  • Bitcoin Core has sent no reject since 0.20, so the old code could not tell a rejection from success.
  • Pending txs lived only in memory.

New P2P client: electrum/p2p.rs

  • bitcoin_p2p hard-codes relay = false in its version message. With that, Core answers every mempool getdata with notfound, and we need those answers.
  • Framing is buffered, so a read timeout never loses half a message.

How a broadcast works: electrum/relay.rs

  1. Ask whether the peer already has the tx (getdata → tx/notfound).
  2. Announce it with inv, and hand it over on the peer's getdata.
  3. Success only once the peer serves the tx from its mempool. That is the same criterion as sendrawtransaction, and it usually takes 5–10 s.

What ends the wait early with a real error

  • The peer asks for a parent → bad-txns-inputs-missingorspent: …. Sparrow shows its "UTXOs missing or already spent" dialog.
  • The peer drops the connection after the hand-over → consensus-invalid.
  • The tx is not accepted and its fee is below the peer's feefilter → the transaction was rejected by network rules. … mempool min fee not met, X < Y. Sparrow parses this and tells the user the fee to pay.
  • Otherwise: an explicit "did not accept within 25 s, the P2P protocol does not say why" error.
  • Core's context-free checks (CheckTransaction reasons) run locally first.

Fitting Sparrow's timeouts

  • Sparrow's first read timeout is 3 s, and it resends on a timeout (TcpTransport.java:37-41, RetryLogic).
  • A resend waits for the first attempt and gets its answer (cached 120 s).
  • After an inconclusive answer, a resend checks the mempool once more.

Persistence: electrum/pending.rs, <state>.pending.json

  • Accepted txs are shown as unconfirmed again after a restart.
  • Every 10 minutes, once the scan has caught up, pending txs are:
    • forgotten when confirmed,
    • dropped when a confirmed wallet tx spends one of their inputs,
    • re-announced when the peer's mempool lost them,
    • dropped after 14 days out of the mempool (Core's expiry).
  • A newer accepted broadcast spending the same input (an RBF bump) replaces the old one in the index.

Decided: no fan-out to other peers. Broadcasting from your own IP to random nodes marks you as the origin. The configured peer stays the only relay.

Fees (SNB-628)

What Sparrow 2.5.5 does (from its source):

  • Its default fee source is mempool.space, and it is used whatever server you connect to.
  • -1 becomes 1 sat/vB.
  • An error from relayfee or mempool.get_fee_histogram drops the connection.

What friglet now answers:

  • blockchain.estimatefee → -1, the protocol's "no estimate". Before, it returned a made-up 10 sat/vB. friglet has no mempool or fee-market view, and P2P cannot supply fees because input values are unknown.
  • blockchain.relayfee → the peer's feefilter, i.e. its current mempool minimum. The signet peer reports 0.1 sat/vB, as Bitcoin Core 30 does. If the peer can't be read, the answer is 1 sat/vB.
  • mempool.get_fee_histogram → [], unchanged.
  • The README tells users to keep Sparrow's fee source external, or to set fees by hand.

Verification task 4: not in this PR

blockchain.block.headers would be cheap with the new header path. transaction.get_merkle needs a full block per wallet tx. Sparrow also turns any reply other than -32601 into "verification refused" and marks the tx unconfirmed. Both stay -32601 (verification cleanly off). The design and costs are in SNB-629.

Verified

  • cargo test --workspace: all pass. That includes friglet's 50 tests, 25 of them new.
    • The new tests drive a scripted P2P peer: accepted, already in the mempool, silently dropped, disconnect, orphan, an announcement the peer does not request, recheck.
    • They also cover the handlers: stale tip header, unavailable header, error codes, pending store and restart, replacement, conflict, fees.
  • cargo clippy --no-deps -p friglet -p friglet-ipc -p friglet-tray --all-targets -- -D warnings: clean. cargo fmt -p friglet --check: clean.
  • Live, on signet: friglet at tip−3 with a throwaway key (scan secret …01), the hosted signet oracle, and peer 152.53.151.148:38333 (Bitcoin Core 30). The client was a small python script.
headers.subscribe -> {'height': 324498, 'hex': '00000020a2326a4cf2fb234d...'}
   height 324498 hash 00000007ee4ebf6aa26f171e19aa8594156cace96d75c6f9db901387bcf6d5e7 vs explorer: OK
blockchain.relayfee[] -> 1e-06
blockchain.estimatefee[2] -> -1          blockchain.estimatefee[25] -> -1
mempool.get_fee_histogram[] -> []
broadcast(garbage) -> {'code': 1, 'message': "TX decode failed: invalid hex: Invalid character 'z' at position 0"}
broadcast(invalid: made-up input) -> {'code': 1, 'message': 'bad-txns-inputs-missingorspent: the P2P peer asked for a parent of this transaction, ...'}  (6.41 s)
broadcast(tx already in signet mempool 4497f865…) -> '4497f865f96dfe30c2e0ae19ad1bdbf020303b49fcda3d6605d576198017a6cf'  (0.17 s)
  • A new-block notification checked live is posted below as a comment.

Not covered live: a fresh, valid tx going through the full inv→getdata→tx→accepted path. That needs signet coins. The scripted-peer tests cover this path.

Notes for other tracks

Tip headers (SNB-626). New-block notifications carried whatever header
the scanner had stored in the index tip. That is the previous matched
block's header, or none at all before the first match, after a restart
and after a reorg rollback. Sparrow ignores a tip whose header does not
parse, so confirmations froze; a stale header was recorded at the wrong
height. The Electrum server now ignores the stored tip header. It uses
the header stored for exactly that height, or asks the oracle for the
block hash at that height and fetches the 80-byte header from the P2P
peer (getheaders with an empty locator), checked against that hash.
Each tip is announced once. A tip whose header cannot be fetched is not
announced; friglet retries every 15 s. blockchain.block.header backfill
uses the same path and no longer downloads whole blocks.

Broadcast (SNB-627). Broadcasting sent an unsolicited tx to the peer and
always reported success. friglet now has its own small P2P client:
bitcoin_p2p hard-codes relay=false, and with that Bitcoin Core never
serves mempool transactions. A broadcast:
- asks with getdata whether the peer already has the tx,
- announces it with inv and hands it over on the peer's getdata,
- succeeds only once the peer serves it from its mempool.
These end the wait early with a real error:
- the peer asks for a parent: bad-txns-inputs-missingorspent;
- the peer drops the connection after the hand-over: consensus-invalid;
- the fee is below the peer's feefilter: "mempool min fee not met",
  the wording Sparrow parses.
Bitcoin Core's context-free checks run locally first. A resend of the
same tx (Sparrow resends after its 3 s read timeout) gets the first
attempt's answer. Accepted txs are kept in <state>.pending.json and
shown as unconfirmed again after a restart. Every 10 minutes, once the
scan has caught up, pending txs are forgotten when confirmed, dropped
when a conflicting tx confirms, and re-announced when the peer's
mempool lost them. A newer broadcast spending the same input replaces
the older one.

Fees (SNB-628). estimatefee returns -1, the Electrum protocol's "no
estimate"; it used to return a made-up 10 sat/vB. relayfee returns the
peer's feefilter (0.1 sat/vB on the signet peer, which runs Bitcoin
Core 30) instead of a fixed 1 sat/vB. The fee histogram stays empty,
because an error there makes Sparrow reconnect.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@setavenger

Copy link
Copy Markdown
Owner Author

Live check on signet: a new block reached the Electrum client with its real header.

The throwaway wallet has no matches, so no header was stored for this height. The notification therefore took the oracle-hash plus P2P getheaders path, which is exactly the case that used to send an empty header.

subscribed: height 324498 hash 00000007ee4ebf6aa26f171e19aa8594156cace96d75c6f9db901387bcf6d5e7 vs explorer: OK
13:47:16 notification: height 324499 hash 000000022c54c6824a2019c10af1ab9e7ee793a9bbb3c84084ff75b6e622b91d vs explorer: OK prev-link OK

The smoke client hashes the announced 80-byte header and compares it with mempool.space/signet's hash for that height. It also checks that the header's prev-hash equals the previous tip's hash.

CI (test) passes on ea2296b.

setavenger added a commit that referenced this pull request Oct 1, 2026
…nto dogfood/preview

No textual conflicts. Resolved by hand so all behaviours stay:
- control: restore the `scanner` module import (track A's scan_health
  mapping uses scanner::ScanHealth); scan_health stays in GetStatus
  alongside the restart-on-apply semantics.
- config: moving a legacy state file to its per-wallet name also moves
  #26's pending-broadcast file (`*.pending.json`), so unconfirmed
  broadcasts survive the migration; the pending file is per wallet
  because it follows the keyed state path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setavenger added a commit that referenced this pull request Oct 1, 2026
…ood/preview

Brings in the #17/#20/#19/#21/#31 stack. One conflict, in
friglet/src/electrum/mod.rs, resolved keeping both sides:
- #26's ChainSource (p2p/network/oracle) and pending-broadcast store,
- #21's reorg handling: checkpoints behind a mutex, reorg_rx feeding
  forget_blocks_above; its test now builds the server state with #26's
  test helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@setavenger
setavenger merged commit 69e07ce into integration Oct 1, 2026
1 check passed
@setavenger
setavenger deleted the fix/electrum-tip-broadcast-fees branch October 1, 2026 20:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant