Skip to content

DO NOT MERGE: dogfood preview for first-user testing - #28

Merged
setavenger merged 35 commits into
integrationfrom
dogfood/preview
Oct 1, 2026
Merged

setavenger merged 35 commits into
integrationfrom
dogfood/preview

Conversation

@setavenger

@setavenger setavenger commented Oct 1, 2026 •

Copy link
Copy Markdown
Owner

Do not merge. Throwaway branch for first-user testing. Each PR still lands on its own, in the planned order.

This is integration plus these PRs, merged with merge commits:

PR What it adds
#20 stops at unindexed oracle blocks instead of silently skipping them; no panics on stream errors
#19 detects spends; labels survive restarts; old state files are rewound so missed spends are found (#17 comes in through #19)
#21 reorg detection and rollback to the fork point; hash-only chain checks instead of re-streaming blocks
#31 atomic state-file writes; a state file that cannot be restored is moved aside and reported, never silently replaced (stacked on #21)
#24 Sparrow descriptor paste in the tray, start at tip, network → hosted oracle, host:port peers
#29 daemon restarts itself when settings are saved; tray restarts a crashed daemon with backoff and shows the crash; start at login
#26 real tip headers, broadcasts that report real failures, honest fee answers
#25 optimised libsecp256k1 in dev/test builds
#23 release bundle workflow, which builds the installers for this PR as workflow artifacts

Not in yet: nothing currently planned is missing. The branch is refreshed as PRs change.

Test on signet first. In Sparrow: SP wallet → Settings → right-click Descriptor → Copy Output Descriptor → paste into the tray setup. Then in Sparrow: Preferences → Server → Private Electrum 127.0.0.1:50001, no SSL.

setavenger and others added 30 commits September 24, 2026 10:26
…SNB-541)

The official vectors only reached Scanner::scan_transaction_full, which has
no production caller. The daemon receives through scan_short_block_data ->
scan_transaction_short (k = 0, labels enumerated by hand, 8-byte x-only
prefixes) and, on a probable match, apply_block_relevant on the external
indexer. A false negative in the short step means the block is never
fetched: silent loss.

- Serve every receiving vector the way the oracle does (taproot outputs'
  8-byte x-only prefixes, the 33-byte tweak, the display-order txid) and
  assert a probable match wherever the vector expects >= 1 output. Pins
  which k = 0 branch did the work: unlabelled P_0, labels of both output
  parities, and the change label m = 0. False positives are counted and
  reported (zero on these vectors).
- Drive the real post-match step for the same vectors: the vector tx in a
  full block through apply_block_relevant, asserting the stored outputs and
  priv_key_tweaks equal expected.outputs.
- Extract the post-match step (txid mapping + apply_block_relevant + stage
  merge) from scan_block_range into Scanner::apply_matched_block, unchanged,
  so the test exercises the production code rather than a copy.
- K_max through the live path is SNB-540: an ignored test records that it
  stores 2324 outputs where BIP-352 allows 2323.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The scanner's `owned_outputs` set, which the per-block spent-output check
matches the oracle's 8-byte `spent_outputs` prefixes against, had a writer
(`add_owned_output`) that no call site ever invoked. The set stayed empty,
the spent check never fired, and a block whose only wallet-relevant
transaction was a spend (no Silent Payments output back to the wallet) was
never fetched. The spending transaction never reached the wallet graph, so
blindbit-cli and friglet reported a balance that still included spent coins
and Electrum clients never saw the spend. Spends were only noticed by
accident, when the spending transaction also paid the wallet (e.g. change).

- `Scanner::sync_owned_outputs` rebuilds the owned-output records from the
  indexer after every applied block and on restore: outpoint, full x-only
  key, amount, height, and the BIP-352 label `m` the output matched.
- A record is marked spent only when a confirmed transaction in the wallet
  graph spends exactly that outpoint. The 8-byte prefixes, matched through
  a prefix -> unspent-outpoint map, only decide which blocks to fetch; a
  foreign key sharing the prefix fetches the block but marks nothing.
- The state file's `owned_outputs` array now holds these records (label
  and spend included). Legacy bare-hex entries are accepted and dropped;
  the records are re-derived from the indexer on load.
- Restore registered labels only after re-indexing the stored
  transactions, and `load_from_file` blanks the persisted label table, so
  every labelled output (change included, m = 0) silently dropped out of
  the index, the balance and spend detection after each restart. Labels
  are now registered before the changeset is applied.
- `apply_matched_block` is extracted from `scan_block_range` (same shape
  as in #17) so the tests drive the real per-block path without P2P.

Tests (in-process, `owned_outputs_tests.rs`): receive then spend in a later
block marks the output spent and the balance drops; a colliding foreign
prefix marks nothing; records, labels and spends survive save/load/restore
and spend detection keeps working after it; a labelled receive records the
right m.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… messages

Adds an in-process seam (BlockScanDataStream) between scan_block_range and
the scan loop, with no behaviour change, and tests that drive a real Scanner
through it. They fail on this commit: an empty-hash block mid-range is
treated as an empty block (payments in it are lost and the scan reports
success), height gaps and short streams are accepted, and any error status
panics the process via unwrap().

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ipping it

An oracle answers a height it has not indexed with an OK message carrying an
empty block hash and no data. scan_block_range took that as a block with no
payments: it warned, advanced the scan height past it and reported success,
so any payment in that block was never found (proven by the testkit mutation
check unindexed-ok-empty: 80000 sats reported instead of 120600).
Separately, every stream message was unwrap()ed, so an error status
(NOT_FOUND, UNAVAILABLE) panicked the process, as did failing to open the
stream.

Now each message must be the next expected height and carry a valid
32-byte, non-zero block hash; otherwise, and on any error status or a stream
that ends before the requested end, the scan returns an error naming the
height to rescan from. The scan height stays at the last fully processed
block, so watch_chain retries from there on its next poll (every 10 s, no
spin) and blindbit-cli exits non-zero after saving that height. The example
is updated the same way.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…point

The live scanner had no reorg handling: after a reorganisation it kept
every output and spend learned from disconnected blocks. An evicted
payment stayed in the balance as a phantom output, a payment that
re-confirmed at another height kept its old height, and an output whose
spend was disconnected stayed marked spent.

The scanner now remembers the block hash of each scanned height for the
last REORG_LOOKBACK (144) heights, persisted in the state file as
`scanned_block_hashes`. A range scan that continues the scanned chain
streams one block early and compares it; watch_chain also runs that
check when the oracle has no new block, which catches a same-height tip
replacement. On a mismatch the scan walks up from the oldest remembered
hash to find the fork point. It rolls back everything above it: graph
transactions and anchors from disconnected blocks with their tweaks, the
owned-output records and spent marks derived from them, checkpoints,
remembered hashes, and Electrum history/SP history/headers. It persists
that state and scans the new branch from the fork. A fork at or below
the oldest remembered hash is a ReorgTooDeep error. That error changes
nothing and repeats on every attempt, and it tells the user to rescan
from the wallet birthday.

friglet drops its snapshotted checkpoint hashes and cached headers above
the fork (and rewrites the header sidecar) so it never serves a header
of a disconnected block. blindbit-cli goes through scan_block_range and
gets the same check.

Tests drive the stream seam with an in-process oracle, mirroring the
testkit T2 reorg scenario (fork at 104, 2 disconnected, 3 mined):
eviction, re-confirmation at a new height, a disconnected spend, a
same-height tip replacement, persistence plus restart mid-rescan, and a
reorg deeper than the lookback.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…d-labels

Resolve scanning.rs: keep sync_owned_outputs() after apply_matched_block,
keep both test-module lines, and drop the dead
`if probable_match.spent {}` block that #17's extraction carried over.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a packaging-only workflow that builds the tray app with the friglet
daemon sidecar via Tauri's bundler: Linux deb+AppImage (ubuntu-22.04),
a universal macOS dmg (macos-15), and best-effort Windows msi+nsis.
Each job checks the daemon is inside the bundle. Tags and manual runs
attach the bundles plus SHA256SUMS to a draft release; pull requests
only run when packaging files change and only upload artifacts.

prepare-sidecar.sh learns universal-apple-darwin (lipo of both slices)
and honours CARGO_TARGET_DIR. macOS bundles are ad-hoc signed so
Gatekeeper offers "Open Anyway" instead of reporting a damaged app.
README documents the workflow and how to open unsigned builds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The universal tauri build compiles the tray once per arch, and
tauri-build checks that binaries/friglet-<arch> exists each time, so
the universal staging also keeps both slices next to the lipo output.
The Linux deb check now prints the package listing and accepts entries
with or without a leading "./".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Onboarding no longer needs hand-decoded hex keys:

- friglet-ipc: BIP-392 SP descriptor parser (spscan/spspend, two-key
  export form, BIP-393 bh= birth height, BIP-380 checksum), tested
  against Sparrow 2.5.5/drongo fixtures; per-network hosted oracles,
  default P2P ports and host[:port] peer parsing/DNS resolution.
- friglet: `descriptor` config key / FRIGLET_DESCRIPTOR supplies the spend
  key, the scan key (moved into the 0600 key file) and the birthday. It is
  read beside DaemonConfig so GetConfig never returns it. A spspend
  descriptor on disk is refused (friglet is watch-only).
- New wallets: `start_at_tip` uses the oracle tip as the birthday and
  records it as start_height in the config file on first start.
- oracle_url follows the network unless set; a hosted oracle of another
  network is rejected. p2p_node_addr accepts hostnames and bare hosts.
- friglet-tray: one "Sparrow wallet descriptor" field fills network and
  keys (scan secret never reaches the webview), shows the SP address to
  compare with Sparrow, drops a pasted spend private key; birthday choice
  (new wallet at tip / existing wallet from a height); network switch sets
  the hosted oracle; Save checks the node's DNS name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s below it; report stalls

A wallet whose start height lies below the oracle's first indexed block
(the hosted oracle has nothing below 800000 on mainnet and 100000 on
signet) stopped at its first height on every poll, forever, with only an
error-level log line. Since #20 refuses to scan past an unindexed height,
that is the only thing such a wallet ever did.

- When `watch_chain`'s scan stops at the wallet's start height because
  the oracle has not indexed it, the scanner locates the oracle's lowest
  indexed height with GetBlockHashByHeight lookups and starts there,
  logs a warning, persists the adjustment (`oracle_floor_start`, optional
  in the state file) and publishes it in the scan health.
- Only heights below the oracle's floor are skipped. A height above the
  first scanned block, or one with an indexed height below it (a gap in
  the oracle's index), keeps #20's stop-and-retry. The search is a binary
  search verified by exponential back-off probes below the boundary, so a
  gap above the floor does not move the floor up. NOT_FOUND and an empty
  or zero hash both mean "not indexed".
- A scan that cannot get past a height (or cannot reach the oracle) is
  published as a stall: height, reason, and the time of the first failed
  attempt at that height. It clears when the scan makes progress.
- Stream stops are a typed `ScanStopped` error (same message text) so the
  loop can tell "not indexed" from other failures.
- Health lives in the wallet's Electrum index, which status readers lock
  without waiting on the scan task. friglet's GetStatus reports it as
  `StatusInfo::scan_health` (serde default, so old trays and daemons
  interoperate), and the tray's status window shows a stall line and a
  note for an adjusted start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd-labels

#20 lands before this PR, so this branch carries it to merge cleanly
after it. changeset.rs: keep both imports (#20's OracleFloorStart and
this branch's OwnedOutputRecord); the serde_hex module #20 still had is
the one this branch replaced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
secp256k1-sys builds the C libsecp256k1 with the profile's opt-level, so
debug and test builds run every EC operation at -O0. The BIP-352 vector
tests feel this most: the K_max vector scans 2324 outputs quadratically,
and the blindbit-lib test binary spent ~3 minutes of every CI run on it
(~4 minutes once SNB-540 un-ignores the live-path K_max test).

Set `opt-level = 3` for secp256k1-sys in the dev profile (the test profile
inherits it). Only that one C library changes; the rest of the dev build is
untouched. Locally the live K_max test drops from 219s to 41s of CPU time;
also optimising the secp256k1 and bdk_sp Rust crates gained <5% more, so
they stay at the default. Debug builds of the daemon scan faster too.

SNB-540

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tauri deletes the intermediate .app after building the dmg, so the
macOS check now mounts the dmg read-only and verifies the shipped
app: both binaries are arm64+x86_64, the daemon runs, and the bundle
carries a valid ad-hoc signature. AGENTS.md now records that the MSVC
build and msi/nsis bundling work on windows-2025.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tip headers (SNB-626). New-block notifications carried whatever header
the scanner had stored in the index tip. That is the previous matched
block's header, or none at all before the first match, after a restart
and after a reorg rollback. Sparrow ignores a tip whose header does not
parse, so confirmations froze; a stale header was recorded at the wrong
height. The Electrum server now ignores the stored tip header. It uses
the header stored for exactly that height, or asks the oracle for the
block hash at that height and fetches the 80-byte header from the P2P
peer (getheaders with an empty locator), checked against that hash.
Each tip is announced once. A tip whose header cannot be fetched is not
announced; friglet retries every 15 s. blockchain.block.header backfill
uses the same path and no longer downloads whole blocks.

Broadcast (SNB-627). Broadcasting sent an unsolicited tx to the peer and
always reported success. friglet now has its own small P2P client:
bitcoin_p2p hard-codes relay=false, and with that Bitcoin Core never
serves mempool transactions. A broadcast:
- asks with getdata whether the peer already has the tx,
- announces it with inv and hands it over on the peer's getdata,
- succeeds only once the peer serves it from its mempool.
These end the wait early with a real error:
- the peer asks for a parent: bad-txns-inputs-missingorspent;
- the peer drops the connection after the hand-over: consensus-invalid;
- the fee is below the peer's feefilter: "mempool min fee not met",
  the wording Sparrow parses.
Bitcoin Core's context-free checks run locally first. A resend of the
same tx (Sparrow resends after its 3 s read timeout) gets the first
attempt's answer. Accepted txs are kept in <state>.pending.json and
shown as unconfirmed again after a restart. Every 10 minutes, once the
scan has caught up, pending txs are forgotten when confirmed, dropped
when a conflicting tx confirms, and re-announced when the peer's
mempool lost them. A newer broadcast spending the same input replaces
the older one.

Fees (SNB-628). estimatefee returns -1, the Electrum protocol's "no
estimate"; it used to return a made-up 10 sat/vB. relayfee returns the
peer's feefilter (0.1 sat/vB on the signet peer, which runs Bitcoin
Core 30) instead of a fixed 1 sat/vB. The fee histogram stays empty,
because an error there makes Sparrow reconnect.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ends it missed

A state file written before owned outputs were recorded still loads, but
its wallet never fetched blocks whose only wallet-relevant transaction
was a spend, so those spends stay missing and nothing said so.

- The state file now carries `format_version` (STATE_FORMAT_VERSION = 1;
  absent = 0, written before the owned-output records). Every save writes
  the current version.
- Restoring a format-0 state rewinds the scan to just below its oldest
  unspent output, the lowest height a missed spend can be at. The next
  scan revisits those blocks and, with the outputs now recorded, fetches
  the ones that spend them. Re-applying blocks already in the graph is
  idempotent. With no unspent output there is nothing to re-check and
  only the version is bumped.
- The rewind is logged and published in the scan health
  (`StateRescan`), shown in friglet's status and the tray, and cleared
  once the scan is back where the state had been. While it runs, the
  Electrum tip stays at that height, so clients never see wallet history
  above the tip.
- Test fixture `testdata/state_before_owned_outputs.json` was written by
  the pre-fix code (integration 93f6a92): payment + change at 100,
  scanned to 300, spend at 200 missing. The test loads it as written and
  with legacy bare-hex `owned_outputs` entries, checks the restored
  records and labels, the rewind to 99, the held Electrum tip, that the
  rescan finds the spend, and that the upgraded file restores without
  another rewind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fix/reorg-rollback

Keeps this branch a superset of the stack below it.

- changeset.rs / scanner.rs: keep both sides' new fields
  (scanned_block_hashes; format_version, oracle_floor_start).
- scanning.rs: keep this branch's scan_range_from / BlockStreamSource;
  OracleStreams::open now reports a refused stream as #20's typed
  ScanStopped. watch_chain combines both: the no-new-block hash check
  from here, and #20's floor start (only for a real range) and stall
  reporting, with the ReorgTooDeep log kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ing blocks; catch mid-stream oracle reorgs

The reorg check streamed the tip block again on every idle poll (~45 KB
of compute index every 10 s on mainnet, ~385 MB a day per friglet and the
same egress on the oracle), streamed the last scanned block again in
front of every new block, and located a fork by streaming the whole
144-block lookback window.

- The block below a range is checked with one GetBlockHashByHeight
  lookup against the remembered hash. An idle poll streams nothing; a new
  block is streamed once. An empty hash or NOT_FOUND means "cannot check
  yet", not a reorganisation.
- A disagreement is located with a binary search over the remembered
  heights (lookups only), and the rollback stream starts at the fork
  point. The oracle disagreeing at the oldest remembered height is
  ReorgTooDeep without streaming anything, so a halted wallet costs two
  lookups per poll.
- The short stream carries no previous-block hash, so an oracle that
  switches branches while a stream runs could hand over blocks of both
  and nothing noticed. When a stream ends, its newest blocks
  (POST_STREAM_CHECK = 6) and the block below them are checked again; a
  disagreement is rolled back like any other reorganisation (bounded to 3
  restarts per scan).
- The per-poll body of watch_chain is `watch_step`, generic over the
  oracle, so the tests drive the real loop: a ReorgTooDeep is published
  as a stall (reason says to rescan from the wallet birthday), like any
  other scan that cannot proceed.
- record_scanned_block_hash no longer copies the 144-hash window into the
  staged changeset on every block (save_to_file writes the current
  window), and trims the window in place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…SNB-623)

The GTK-themed native select was light-on-light in the dark form; the
`hidden` attribute lost to .card/.col display:flex, so the empty
"Last error" row always showed; "From block height" reads right both
in first-run setup and for an already configured wallet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n attribute)

The status window's error, stall and scan-notes blocks use class="col",
whose display: flex beats the user-agent [hidden] rule, so an empty
"Last error" label was always shown and the new stall/notes blocks
would be too. Seen in a screenshot against a live daemon.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd-labels

Picks up the tray CSS fix for hidden status blocks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Picks up #20's tray CSS fix for hidden status blocks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rthday (SNB-623)

Found in the Xvfb run: pasting the descriptor of the wallet already
configured flipped the birthday to "New wallet", so Save would have
moved it to the tip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… login (SNB-624, SNB-52)

Daemon:
- Settings saved over the control socket (SetConfig, SetScanKey, new
  ApplySettings = both in one step) are validated, persisted and then
  applied by an in-process restart: each run gets its own tokio runtime,
  so dropping it ends every task of the old run (Electrum client
  connections, startup-time subscriptions, listeners) and Sparrow
  reconnects to the new wallet view. Same PID; unchanged input does not
  restart. Replaces the live scanner rebuild that left the Electrum
  server on the old wallet.
- The default state file is keyed by wallet (scanner_state-<net>-<id>),
  so new keys start fresh; a legacy state of the same wallet is moved in,
  an explicit state file of another wallet is refused.
- Moving the birthday back below what the state covers rescans (state
  kept as .json.bak).
- A dead HTTP/Electrum/control server exits non-zero; a control socket
  owned by another daemon is no longer deleted on the way out.

Tray:
- Supervision: a tray-owned daemon that dies (crash, kill, panic=abort)
  is restarted with exponential backoff (1 s .. 60 s, reset after a
  healthy minute); a wedged or handle-less owned daemon is detected too.
  The Status tab shows exit status/signal, time, last output lines,
  restart count and countdown; the tray label says it is restarting.
- Settings save sends one ApplySettings and waits for the restarted
  daemon.
- Start at login via tauri-plugin-autostart, toggled in Settings and
  ticked by default in first-run setup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setavenger and others added 5 commits October 1, 2026 14:04
…shed (SNB-624)

Found in the Xvfb run: when the first spawn (startup or first-run Save)
failed, nothing scheduled another attempt — supervision only reacted to
exits. attach_and_record now records every failed attempt and schedules
the next one with backoff.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… one that cannot be restored

save_to_file wrote the state in place, so a crash or power loss during a
save could leave a truncated file. load_scanner then treated any file it
could not parse or restore as absent: it logged a warning, created a
fresh scanner, and that scanner's first save overwrote the old file.

- save_to_file writes `<state>.tmp` (created 0600: the file holds the
  scan secret), fsyncs it, renames it over the state file and fsyncs the
  directory. A failed write leaves the previous file untouched and
  removes the temporary file; a temporary file left by a crash is
  replaced. A new state file is no longer created world-readable until
  friglet's next start tightens it.
- load_scanner (via restore_or_create):
  - no state file: new scanner, as before;
  - a state file that cannot be read (permissions, I/O): error, nothing
    is moved or created;
  - a state file that cannot be restored (invalid JSON, missing keys, or a
    format_version newer than this build knows): moved to
    `<state>.unreadable-<unix time>`, never overwritten; a new scan starts
    from the configured start height; the error is logged and published as
    ScanHealth::state_file_reset, shown in friglet's GetStatus
    (`scan_health.state_reset`) and as a note in the tray.
- The oracle connection is opened once for both paths (previously each
  path connected on its own; failure behaviour is unchanged).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nto dogfood/preview

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nto dogfood/preview

No textual conflicts. Resolved by hand so all behaviours stay:
- control: restore the `scanner` module import (track A's scan_health
  mapping uses scanner::ScanHealth); scan_health stays in GetStatus
  alongside the restart-on-apply semantics.
- config: moving a legacy state file to its per-wallet name also moves
  #26's pending-broadcast file (`*.pending.json`), so unconfirmed
  broadcasts survive the migration; the pending file is per wallet
  because it follows the keyed state path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ood/preview

Brings in the #17/#20/#19/#21/#31 stack. One conflict, in
friglet/src/electrum/mod.rs, resolved keeping both sides:
- #26's ChainSource (p2p/network/oracle) and pending-broadcast store,
- #21's reorg handling: checkpoints behind a mutex, reorg_rx feeding
  forget_blocks_above; its test now builds the server state with #26's
  test helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@setavenger
setavenger merged commit 6d4930a into integration Oct 1, 2026
5 checks passed
@setavenger
setavenger deleted the dogfood/preview branch October 1, 2026 20:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant