Skip to content

fix(indexer): stop the initial sync from deadlocking on a failed request - #70

Open
jrsd86 wants to merge 3 commits into
setavenger:integrationfrom
jrsd86:fix/indexer-pull-deadlock
Open

jrsd86 wants to merge 3 commits into
setavenger:integrationfrom
jrsd86:fix/indexer-pull-deadlock

Conversation

@jrsd86

@jrsd86 jrsd86 commented Oct 10, 2026

Copy link
Copy Markdown

Base: integration (1b0e113) · Depends on: nothing · 3 commits (they can be squashed)

Problem

On a mainnet node, the initial sync stopped for good three times (13 h, 23 h and until stopped), while the process kept running at ~0 % CPU. Each stop came right after a single failed REST request to Bitcoin Core, a 100 s client timeout:

ERR internal/indexer/pull.go:29 > failed to pull spentutxos error="... context deadline exceeded ..."

After that line, no further progress or error was logged. In particular, the ended with err message never appears.

Root cause

  • PullBlockData: the two fetch goroutines send their error on an unbuffered channel before defer wg.Done() runs, while the caller sits in wg.Wait() and only reads the channel afterwards. Any single failed fetch blocks its goroutine forever, so wg.Wait() never returns.
  • SyncBlocks: the same pattern repeats.
    • A failed pull returns without wg.Done(), so newBlockChan is never closed.
    • Further errors block on the unbuffered errChan once the first one has been received.
    • The dispatch loop keeps starting pulls.
  • Sends without a cancellation exit: handleBlock (to writerChan) and pullBlockToChan (to newBlockChan) send without watching ctx, so they can block forever once the consumer has stopped.
  • ContinuousSync: the writer can fail while the main loop is blocked handing it a block, and each side waits on the other.

Goroutine states, reproduced by the new tests against integration (raw dumps available on request):

[chan send]            indexer.PullBlockData.func1 / func2
[sync.WaitGroup.Wait]  indexer.PullBlockData
[sync.WaitGroup.Wait]  indexer.(*Builder).SyncBlocks.func1

Changes

Commit 1: the fix

  • PullBlockData records each fetch's error and joins them after wg.Wait(); there is no channel.
  • SyncBlocks derives a cancellable context.
    • Errors are recorded without blocking (sync.Once), and the first one recorded cancels all stages.
    • Every channel send also selects on ctx.
    • SyncBlocks returns only after every goroutine it started has exited, so nothing outlives the call or touches the builder's channels afterwards.
    • Precedence: cancel() is called only after an error has been recorded, so an error that merely results from that internal cancellation cannot be recorded first. A context.Canceled is returned when the caller cancelled. If the caller cancels at the same moment as a real failure, either error may be returned.
  • ContinuousSync cancels on writer failure, uses a buffered error channel and waits for the writer before returning. It returns the writer's error rather than the cancellation that error caused.
  • The REST fetch helpers take a context.Context (http.NewRequestWithContext), so cancellation also aborts in-flight requests. Their errors are wrapped with %w, so callers can match context.Canceled and context.DeadlineExceeded.

Commit 2: cancel the sibling request

  • When one of PullBlockData's two fetches fails, the other one is cancelled at once, instead of being left to run until the client timeout. The cancellation PullBlockData causes itself is dropped from the returned error.
  • TestContinuousSyncWriterError now pins config.SyncStartHeight instead of relying on the package default.

Commit 3: cancellable chain-state requests

  • GetChainInfo now takes a context. Its callers are SyncBlocks' progress report, ContinuousSync's polling and InitialSyncToTip. Before this commit, a cancelled sync stayed blocked in that request for up to the client timeout when the node did not answer.
  • The progress-report interval becomes a package variable, so tests can shorten it.

Behaviour is unchanged on the happy path. On a failed request the sync now returns the error (main exits non-zero and a supervisor restarts it; the existing DBIntegrityCheck refills missing heights at the next start) instead of hanging. No retry is added here.

Tests

  • Harness: a self-contained fake node (httptest) that injects faults per endpoint and height (dropped connection, request that never answers, also on chaininfo), plus an in-memory store with optional write failures. It does not depend on the fake chain in fix(indexer): index the configured start height on a fresh database #66; the names are distinct.
  • Cases:
    • a dropped request on each endpoint;
    • concurrent failures (4 heights × 2 endpoints);
    • a writer failure with a chain longer than both channel buffers;
    • cancellation while a block request never answers;
    • the writer error in ContinuousSync;
    • PullBlockData with one or both fetches failing, success and cancellation;
    • sibling cancellation in both directions;
    • cancelling SyncBlocks and ContinuousSync while chaininfo never answers.
  • Leak check: every regression test also asserts that no indexer goroutine is left running after the call returns.
  • Regressions demonstrated:
    • against integration, 5 of the 6 original SyncBlocks/ContinuousSync tests fail (deadlock or leaked goroutines);
    • the sibling test fails without commit 2;
    • the two chaininfo tests fail without commit 3 (they only return after the 100 s client timeout).

Results (author's runs, Go 1.24.1 as in CI, Linux/amd64):

go test ./...                                  ok
go test -race -count=5 ./internal/indexer/     ok, no data races
go vet ./...                                   ok
staticcheck ./internal/... ./cmd/...           no new findings vs integration

End to end against a real mainnet node (commit 1): a local proxy dropped one spenttxouts request. The old binary hung indefinitely; the patched binary exited with status 1 after 1.8 s, with the error chain logged. This covers one failure variant only; the other variants are covered by the unit tests.

Notes for merging

Disclosure

This change was developed with the assistance of Claude Code (Anthropic). The analysis, code and tests were reviewed before submission, including an independent static review of the patches.

A single failed REST request (timeout, connection reset) left the indexer
blocked forever while the process kept running:

- PullBlockData sent fetch errors on an unbuffered channel that was only
  read after wg.Wait(). The failing goroutine blocked before wg.Done(), so
  wg.Wait() never returned.
- In SyncBlocks a failed pull returned without wg.Done(), so newBlockChan
  was never closed. Later errors blocked on the unbuffered errChan once the
  first one had been received, and the pull loop kept dispatching.
- handleBlock sent to writerChan without watching ctx, and
  pullBlockToChan sent to newBlockChan without watching ctx, so both could
  block forever once the consumer had stopped.
- In ContinuousSync the writer could fail while the main loop was blocked
  handing it a block: each side waited on the other.

Changes:

- PullBlockData records each fetch's error and joins them after wg.Wait().
- SyncBlocks derives a cancellable context; the first error from any stage
  is recorded without blocking and cancels the others. Every channel send
  also selects on ctx. SyncBlocks returns only after all goroutines it
  started have exited.
- ContinuousSync cancels on writer failure, uses a buffered error channel
  and waits for the writer before returning; it returns the writer's error
  rather than the cancellation it caused.
- The REST fetch helpers take a context (http.NewRequestWithContext) so
  cancellation also aborts in-flight requests, and wrap errors with %w so
  callers can match context.Canceled / DeadlineExceeded.

Tests add a fault-injecting fake node (httptest) and an in-memory store.
They cover a dropped request on each endpoint, concurrent failures, a
writer failure with full channels, cancellation with a request that never
answers, and the writer error in ContinuousSync, and assert that no
indexer goroutine is left running. Against the previous code the five
regression tests fail (deadlock or leaked goroutines).
…ata fails

PullBlockData fetches a block and its spent outputs concurrently. When one
fetch failed, the other was left running until it completed or hit the
client timeout (100 s), so a failed block pull could still take up to that
long to return.

Each fetch now runs under a context derived from the caller's. The first
failure cancels it, which aborts the sibling request at once. The
cancellation that PullBlockData causes itself is dropped from the returned
error, so callers see the real failure; if the caller's own context is done,
its error is kept.

TestPullBlockDataCancelsSiblingRequest drops the spenttxouts request while
the block request never answers. Without this change PullBlockData only
returns after the client timeout, and the test fails at its 10 s limit.

TestContinuousSyncWriterError now pins config.SyncStartHeight instead of
relying on the package default, so it keeps working once reorg handling
that walks back to the start height is merged.
GetChainInfo took no context. SyncBlocks' progress report and
ContinuousSync's polling call it while running, so with an unresponsive
node a cancelled sync stayed blocked in that request until the HTTP client
timeout (100 s) before it could return.

GetChainInfo now takes a context and builds its request with
http.NewRequestWithContext; its three callers pass the context they run
under. The progress report interval becomes a package variable so tests can
shorten it.

Tests:
- SyncBlocks and ContinuousSync are cancelled while the node never answers
  chaininfo. Without this change both only return after the client timeout
  and fail at the tests' 10 s limit; with it they return promptly.
- The sibling-cancellation test from the previous commit now covers both
  directions: block failing while spenttxouts hangs, and the reverse.

Also documents in SyncBlocks why an error that merely results from the
internal cancellation can never be recorded before the error that caused it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant