Skip to content

fix(index): remove a reorged-out block's data when its height is replaced - #67

Open
setavenger wants to merge 2 commits into
integrationfrom
fix/reorg-stale-index
Open

setavenger wants to merge 2 commits into
integrationfrom
fix/reorg-stale-index

Conversation

@setavenger

@setavenger setavenger commented Sep 24, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #66. This branch contains #66's commit (the start-height fix and the fake-chain test helper it adds). Merge #66 first; after that, this PR's diff is only the reorg commit.

Problem

Nothing was deleted on a reorg. attachBlockToBatch only calls Set, so a block displaced by a reorg kept every row it had written:

  • Compute index. The compute index is keyed by height+txid and served by height, so rows of txs that were only in the displaced block were served mixed with the new block's rows. A client acting on compute-index data could see outputs that do not exist.
  • Best-chain membership. The displaced block kept its KCIBlock (blockhash→height) row, so heightIfOnBestChain and BlockhashInDB still treated it as best chain. A spend that existed only in the orphan kept marking its output spent under cut-through and spent checks. The continuous-sync walk-back could also stop early at an orphaned parent.
  • Evicted txs. Tweaks and outputs of evicted txs stayed.

Fix

Store: ApplyBlock → replaceBlock (new internal/database/dbpebble/reorg.go). When a height already holds a different block, one pebble batch:

  1. deletes, for the displaced block:
    • KCIBlock(old)
    • KBlockTx(old,*)
    • KTxOccur(*,old)
    • KSpend(*,*,old), enumerated through KTxidOutpoints(old,*)
    • KTxidOutpoints(old,*)
    • KSpentOutputsShort(old)
  2. deletes every KComputeIndex(height,*) row.
  3. deletes KTx/KOut of the displaced block's txs that no other best-chain block contains, checked via KTxOccur plus a height↔hash round trip. A tx re-confirmed elsewhere keeps its rows. A tx re-confirmed in the replacing block has its rows deleted and then rewritten in the same batch.
  4. writes the new block. KCIHeight is overwritten as before.

Ordering, and why it is safe:

  • replaceBlock first calls FlushBatch(true). This PR also makes that wait for in-flight background commits. Every block handed to the store earlier is therefore visible when the deletes are computed from committed state.
  • The deletes precede the new block's Sets in the batch. Pebble applies a batch in order, so rewritten keys end up set.
  • The batch is committed synchronously, so no later batch can overtake it.
  • Initial sync pulls concurrently and applies blocks out of height order, but only for heights that are not indexed yet. It never takes this path and keeps its asynchronous batching.
  • The continuous-sync writer already flushes synchronously after every block.

Walk-back (internal/indexer/pull.go).

  • SingleBlockPullAndHandle now finds the true fork point by comparing the index's hash with the node's hash at each height, going down from min(node, index tip). It no longer asks "is the parent hash in KCIBlock".
  • It then pulls and applies the new branch in ascending order, one block at a time, instead of holding the whole descent in recursion.
  • If the node switches branches mid-pull, the step stops and the next check starts over.
  • ContinuousSync now also triggers when the node is at the index's height on a different best block (nodeAheadOrForked). Before, a reorg that did not lengthen the chain went unnoticed until the next block.

Existing databases. InitialSyncToTip first runs PurgeOrphanedBlocks. It finds KCIBlock rows whose height maps to another block, which is the trace every past reorg left. It deletes those blocks' rows the same way and rebuilds the compute index of their heights from the best block. Mainnet and signet databases therefore do not need a reindex.

  • On a clean database the purge is one scan of the KCIBlock rows plus one point read each, and it writes nothing.
  • It does not run under server-only, or under sync --start-height/--end-height, which skips InitialSyncToTip.

Tests

In-memory pebble:

  • dbpebble.TestReorgRemovesDisconnectedBlockData, with the new branch applied both ascending and descending. The fixture reorgs A2(R,E) A3(S spends X:0) into B2(N) B3(R). After the reorg:
    • the whole database is byte-for-byte equal to a fresh index of the new chain
    • the compute index at heights 2 and 3 holds exactly the new blocks' rows
    • E's tweak and outputs are gone
    • X:0 is no longer spent, for both cut-through outputs and tweaks
    • BlockhashInDB(A2/A3) returns false
    • R is served at height 3
    • no row references an orphan hash
  • TestReorgSameTxSameHeight and TestReapplySameBlockIsNoop cover the remaining cases.
  • TestPurgeOrphanedBlocks reproduces a pre-fix database after the same reorg. The purge must leave it equal to a fresh index of the new chain, and a second run must be a no-op.
  • indexer.TestContinuousSyncReorg uses the fake Bitcoin Core REST server from fix(indexer): index the configured start height on a fresh database #66. It reorgs the node at the same height, to a longer chain, and at the tip only. One continuous-sync step must leave every height on the node's hash and no orphan KCIBlock row.

Red on integration, with the store and pull changes reverted and the tests kept (excerpt):

reorg_test.go:162: compute index at 2: got [R… E…]            (E evicted, still served)
reorg_test.go:171: evicted tx E still has a tweak
reorg_test.go:179: X:0 still reported spent by the orphaned block A3
reorg_test.go:193: orphaned block a2a2 still reported in DB
reorg_test.go:103: orphaned block 73f69e1b… still reported in DB     (TestContinuousSyncReorg, all 3 cases)

go build ./... && go vet ./... && go test ./... passes, both with the default build and with CGO_ENABLED=1 -tags libsecp256k1. go test -race ./internal/... passes.

Performance

  • Read path unchanged. heightIfOnBestChain is on the hot path of every spent check and still does exactly one point read. I measured a variant that re-verified height→hash on every call at +~20% on a 2,000-spent-output cut-through fetch (8.6 ms vs 7.2 ms), so correctness relies on the delete invariant plus the startup purge instead. The round trip is only used where it is cheap: BlockhashInDB, and the reorg-time "tx elsewhere" check.
  • Write path. ApplyBlock adds one point read per block. The delete work only happens on an actual reorg.

Not covered

  • Node chain shorter than the index tip. If the node reports a lower height than the index tip, heights above the node's tip are not disconnected, as before. Doing so automatically would also delete data whenever an operator points the oracle at a node that is still syncing.
  • Initial sync racing a node reorg (not fixed). Initial sync still pulls by height without linking parents. A node reorg during initial sync is corrected by the next continuous-sync check only if the divergence reaches the index tip. Divergence below the tip is not detected; a restart runs PurgeOrphanedBlocks but does not compare heights against the node.

Overlap

🤖 Generated with Claude Code

setavenger and others added 2 commits September 24, 2026 12:37
InitialSyncToTip started at max(dbTip, SyncStartHeight)+1. On an empty
database GetChainTip reports height 0, so the first pulled height was
SyncStartHeight+1 and the configured start height itself was never
indexed; a regtest oracle logged "syncTip=1" and answered height 1 with an
empty hash. The same +1 skipped the start height whenever the database tip
was below it.

The start height is now the configured height on an empty database and
max(tip+1, configured) otherwise. Tests run the initial sync against a
fake Bitcoin Core REST server on an in-memory pebble store and require
every height from the start height to the tip to carry the node's hash.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…aced

Nothing was deleted on reorg: attachBlockToBatch only Sets, so the block
that a reorg displaced kept every row it had written.

- The compute index is keyed by height+txid and served by height, so rows
  of txs that were only in the displaced block were served mixed with the
  new block's rows.
- Its blockhash->height row made heightIfOnBestChain treat it as best
  chain: a spend that existed only in the orphan kept marking the output
  spent under cut-through, and BlockhashInDB let the continuous-sync
  walk-back stop at an orphaned parent.
- Tweaks and outputs of evicted txs stayed reachable.

ApplyBlock now detects that a height already holds a different block and
replaces it: in one batch it deletes the displaced block's KCIBlock,
block->tx, tx-occurrence, spend, spent-outputs-short and txid-outpoints
rows, every compute-index row at the height, and the tweak and outputs of
its txs that no other best-chain block contains, then writes the new block.
It flushes and waits for background commits first, so deletes are computed
from complete state, and commits the batch synchronously, so no later batch
can overtake it. Initial sync only writes unindexed heights and keeps its
asynchronous batching.

The continuous-sync walk-back now finds the fork point by comparing the
index's hash at each height with the node's, then applies the new branch in
ascending order, pulling one block at a time. It also triggers when the node
is at the index's height on a different best block, a reorg that does not
lengthen the chain.

Databases indexed before this fix are repaired at the start of the initial
sync: PurgeOrphanedBlocks finds blockhash->height rows whose height maps to
another block, deletes those blocks' rows the same way, and rebuilds the
compute index of their heights. No reindex is needed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant