Repository navigation
Conversation
A single failed REST request (timeout, connection reset) left the indexer blocked forever while the process kept running: - PullBlockData sent fetch errors on an unbuffered channel that was only read after wg.Wait(). The failing goroutine blocked before wg.Done(), so wg.Wait() never returned. - In SyncBlocks a failed pull returned without wg.Done(), so newBlockChan was never closed. Later errors blocked on the unbuffered errChan once the first one had been received, and the pull loop kept dispatching. - handleBlock sent to writerChan without watching ctx, and pullBlockToChan sent to newBlockChan without watching ctx, so both could block forever once the consumer had stopped. - In ContinuousSync the writer could fail while the main loop was blocked handing it a block: each side waited on the other. Changes: - PullBlockData records each fetch's error and joins them after wg.Wait(). - SyncBlocks derives a cancellable context; the first error from any stage is recorded without blocking and cancels the others. Every channel send also selects on ctx. SyncBlocks returns only after all goroutines it started have exited. - ContinuousSync cancels on writer failure, uses a buffered error channel and waits for the writer before returning; it returns the writer's error rather than the cancellation it caused. - The REST fetch helpers take a context (http.NewRequestWithContext) so cancellation also aborts in-flight requests, and wrap errors with %w so callers can match context.Canceled / DeadlineExceeded. Tests add a fault-injecting fake node (httptest) and an in-memory store. They cover a dropped request on each endpoint, concurrent failures, a writer failure with full channels, cancellation with a request that never answers, and the writer error in ContinuousSync, and assert that no indexer goroutine is left running. Against the previous code the five regression tests fail (deadlock or leaked goroutines).
…ata fails PullBlockData fetches a block and its spent outputs concurrently. When one fetch failed, the other was left running until it completed or hit the client timeout (100 s), so a failed block pull could still take up to that long to return. Each fetch now runs under a context derived from the caller's. The first failure cancels it, which aborts the sibling request at once. The cancellation that PullBlockData causes itself is dropped from the returned error, so callers see the real failure; if the caller's own context is done, its error is kept. TestPullBlockDataCancelsSiblingRequest drops the spenttxouts request while the block request never answers. Without this change PullBlockData only returns after the client timeout, and the test fails at its 10 s limit. TestContinuousSyncWriterError now pins config.SyncStartHeight instead of relying on the package default, so it keeps working once reorg handling that walks back to the start height is merged.
GetChainInfo took no context. SyncBlocks' progress report and ContinuousSync's polling call it while running, so with an unresponsive node a cancelled sync stayed blocked in that request until the HTTP client timeout (100 s) before it could return. GetChainInfo now takes a context and builds its request with http.NewRequestWithContext; its three callers pass the context they run under. The progress report interval becomes a package variable so tests can shorten it. Tests: - SyncBlocks and ContinuousSync are cancelled while the node never answers chaininfo. Without this change both only return after the client timeout and fail at the tests' 10 s limit; with it they return promptly. - The sibling-cancellation test from the previous commit now covers both directions: block failing while spenttxouts hangs, and the reverse. Also documents in SyncBlocks why an error that merely results from the internal cancellation can never be recorded before the error that caused it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Base:
integration(1b0e113) · Depends on: nothing · 3 commits (they can be squashed)Problem
On a mainnet node, the initial sync stopped for good three times (13 h, 23 h and until stopped), while the process kept running at ~0 % CPU. Each stop came right after a single failed REST request to Bitcoin Core, a 100 s client timeout:
After that line, no further progress or error was logged. In particular, the
ended with errmessage never appears.Root cause
PullBlockData: the two fetch goroutines send their error on an unbuffered channel beforedefer wg.Done()runs, while the caller sits inwg.Wait()and only reads the channel afterwards. Any single failed fetch blocks its goroutine forever, sowg.Wait()never returns.SyncBlocks: the same pattern repeats.wg.Done(), sonewBlockChanis never closed.errChanonce the first one has been received.handleBlock(towriterChan) andpullBlockToChan(tonewBlockChan) send without watchingctx, so they can block forever once the consumer has stopped.ContinuousSync: the writer can fail while the main loop is blocked handing it a block, and each side waits on the other.Goroutine states, reproduced by the new tests against
integration(raw dumps available on request):Changes
Commit 1: the fix
PullBlockDatarecords each fetch's error and joins them afterwg.Wait(); there is no channel.SyncBlocksderives a cancellable context.sync.Once), and the first one recorded cancels all stages.ctx.SyncBlocksreturns only after every goroutine it started has exited, so nothing outlives the call or touches the builder's channels afterwards.cancel()is called only after an error has been recorded, so an error that merely results from that internal cancellation cannot be recorded first. Acontext.Canceledis returned when the caller cancelled. If the caller cancels at the same moment as a real failure, either error may be returned.ContinuousSynccancels on writer failure, uses a buffered error channel and waits for the writer before returning. It returns the writer's error rather than the cancellation that error caused.context.Context(http.NewRequestWithContext), so cancellation also aborts in-flight requests. Their errors are wrapped with%w, so callers can matchcontext.Canceledandcontext.DeadlineExceeded.Commit 2: cancel the sibling request
PullBlockData's two fetches fails, the other one is cancelled at once, instead of being left to run until the client timeout. The cancellationPullBlockDatacauses itself is dropped from the returned error.TestContinuousSyncWriterErrornow pinsconfig.SyncStartHeightinstead of relying on the package default.Commit 3: cancellable chain-state requests
GetChainInfonow takes a context. Its callers areSyncBlocks' progress report,ContinuousSync's polling andInitialSyncToTip. Before this commit, a cancelled sync stayed blocked in that request for up to the client timeout when the node did not answer.Behaviour is unchanged on the happy path. On a failed request the sync now returns the error (
mainexits non-zero and a supervisor restarts it; the existingDBIntegrityCheckrefills missing heights at the next start) instead of hanging. No retry is added here.Tests
httptest) that injects faults per endpoint and height (dropped connection, request that never answers, also onchaininfo), plus an in-memory store with optional write failures. It does not depend on the fake chain in fix(indexer): index the configured start height on a fresh database #66; the names are distinct.ContinuousSync;PullBlockDatawith one or both fetches failing, success and cancellation;SyncBlocksandContinuousSyncwhilechaininfonever answers.integration, 5 of the 6 originalSyncBlocks/ContinuousSynctests fail (deadlock or leaked goroutines);chaininfotests fail without commit 3 (they only return after the 100 s client timeout).Results (author's runs, Go 1.24.1 as in CI, Linux/amd64):
End to end against a real mainnet node (commit 1): a local proxy dropped one
spenttxoutsrequest. The old binary hung indefinitely; the patched binary exited with status 1 after 1.8 s, with the error chain logged. This covers one failure variant only; the other variants are covered by the unit tests.Notes for merging
builder.go(a helper inserted aboveSyncBlocks; theContinuousSynccondition) and, for fix(index): remove a reorged-out block's data when its height is replaced #67, inpull.go(findForkPoint). Resolution keeps both sides, and fix(index): remove a reorged-out block's data when its height is replaced #67's new calls must passctxtopullBlockandgetBlockHashByHeight; the compiler enforces it. Functional compatibility with fix(index): remove a reorged-out block's data when its height is replaced #67's reorg handling is expected from reading the code, but has not been tested in a combined build. Happy to rebase and run both test suites together once either lands.Store.Close()setsclosedbefore its finalFlushBatch(true), andcommitBatchreturns early whenclosedis set, so the final flush inClose()may be a no-op. This is independent of this PR; it could be a separate issue.Disclosure
This change was developed with the assistance of Claude Code (Anthropic). The analysis, code and tests were reviewed before submission, including an independent static review of the patches.