Repository navigation
fix(indexer): wait for Bitcoin Core instead of exiting while it is unavailable - #71
Open
setavenger wants to merge 6 commits into
Open
setavenger wants to merge 6 commits into
setavenger wants to merge 6 commits into
Conversation
A single failed REST request (timeout, connection reset) left the indexer blocked forever while the process kept running: - PullBlockData sent fetch errors on an unbuffered channel that was only read after wg.Wait(). The failing goroutine blocked before wg.Done(), so wg.Wait() never returned. - In SyncBlocks a failed pull returned without wg.Done(), so newBlockChan was never closed. Later errors blocked on the unbuffered errChan once the first one had been received, and the pull loop kept dispatching. - handleBlock sent to writerChan without watching ctx, and pullBlockToChan sent to newBlockChan without watching ctx, so both could block forever once the consumer had stopped. - In ContinuousSync the writer could fail while the main loop was blocked handing it a block: each side waited on the other. Changes: - PullBlockData records each fetch's error and joins them after wg.Wait(). - SyncBlocks derives a cancellable context; the first error from any stage is recorded without blocking and cancels the others. Every channel send also selects on ctx. SyncBlocks returns only after all goroutines it started have exited. - ContinuousSync cancels on writer failure, uses a buffered error channel and waits for the writer before returning; it returns the writer's error rather than the cancellation it caused. - The REST fetch helpers take a context (http.NewRequestWithContext) so cancellation also aborts in-flight requests, and wrap errors with %w so callers can match context.Canceled / DeadlineExceeded. Tests add a fault-injecting fake node (httptest) and an in-memory store. They cover a dropped request on each endpoint, concurrent failures, a writer failure with full channels, cancellation with a request that never answers, and the writer error in ContinuousSync, and assert that no indexer goroutine is left running. Against the previous code the five regression tests fail (deadlock or leaked goroutines).
…ata fails PullBlockData fetches a block and its spent outputs concurrently. When one fetch failed, the other was left running until it completed or hit the client timeout (100 s), so a failed block pull could still take up to that long to return. Each fetch now runs under a context derived from the caller's. The first failure cancels it, which aborts the sibling request at once. The cancellation that PullBlockData causes itself is dropped from the returned error, so callers see the real failure; if the caller's own context is done, its error is kept. TestPullBlockDataCancelsSiblingRequest drops the spenttxouts request while the block request never answers. Without this change PullBlockData only returns after the client timeout, and the test fails at its 10 s limit. TestContinuousSyncWriterError now pins config.SyncStartHeight instead of relying on the package default, so it keeps working once reorg handling that walks back to the start height is merged.
GetChainInfo took no context. SyncBlocks' progress report and ContinuousSync's polling call it while running, so with an unresponsive node a cancelled sync stayed blocked in that request until the HTTP client timeout (100 s) before it could return. GetChainInfo now takes a context and builds its request with http.NewRequestWithContext; its three callers pass the context they run under. The progress report interval becomes a package variable so tests can shorten it. Tests: - SyncBlocks and ContinuousSync are cancelled while the node never answers chaininfo. Without this change both only return after the client timeout and fail at the tests' 10 s limit; with it they return promptly. - The sibling-cancellation test from the previous commit now covers both directions: block failing while spenttxouts hangs, and the reverse. Also documents in SyncBlocks why an error that merely results from the internal cancellation can never be recorded before the error that caused it.
…available After a VM reboot on 2026-10-17 both oracles on oracle.setor.dev exited within a second of starting, because Bitcoin Core was not ready yet: - mainnet: REST answered 503 while Core loaded its block index, and rest.go called log.Fatal on any non-200 status; - signet: the connection was refused, InitialSyncToTip returned the error and the process exited. The same happens at runtime: when Core restarts while the oracle syncs or waits at the tip, the indexer stops and the process exits, taking the HTTP and gRPC servers with it. Every REST request now goes through restGet. It retries failures that waiting for Core can fix, with a backoff from 0.5 s capped at 15 s plus jitter, until Core answers or the context is cancelled: - transport errors: refused, reset or dropped connections, timeouts, name lookups, a connection lost mid-body; - 503 (Core starting, shutting down or with a full work queue) and 502, 504, 429 from a proxy in front of it. Answers that waiting cannot change are returned at once, so a misconfiguration still stops the oracle, now with an error that says what to check instead of log.Fatal: - 4xx and other 5xx statuses; - a complete body that does not decode; - a core_rest_endpoint that is not an http(s) URL; - an HTTP/HTTPS or certificate mismatch. Core's reason from the response body is included in the error. While Core is away the indexer logs one warning when it goes, a reminder every minute and one line when it is back, not one line per request. Because `run` starts the servers before the indexer, already-indexed data keeps being served while the indexer waits. Builds on #70: its context plumbing lets a waiting request stop when the sync is cancelled or a sibling fetch fails for good, and its error propagation makes a permanent failure stop the sync instead of deadlocking it. #70's tests injected dropped connections to exercise that propagation; those are retried now, so the tests inject a 404 instead and new tests cover the retries. The SyncBlocks progress report asks for the chain state once rather than waiting, so it cannot stall the writer. blockCheckInterval becomes a variable so the tests can poll quickly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BenchmarkSyncBlocksRealBlocks syncs 16 heights per op from a fake REST node that serves the four mainnet blocks in test_data (with their spent outputs, rebuilt from the getblock verbosity-3 fixtures and checked against the fixture hashes). BenchmarkSyncBlocksTinyBlocks syncs 200 coinbase-only blocks, so its time is dominated by the request path. The file only uses APIs that predate the retry change, so it can be copied onto older trees for A/B comparisons. Run it with the C backend: go test -tags libsecp256k1 -run '^$' -bench SyncBlocks -benchtime 20x ./internal/indexer/ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: OpenAI Codex <noreply@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Base:
integration. Stacked on #70: this branch still contains its three deadlock/cancellation commits. Please merge #70 first; merge and deployment remain owner decisions.Problem and behavior
After a reboot, the oracle exited while Bitcoin Core was still starting: REST returned 503 or refused connections. The same failure stopped the oracle during a Core restart.
Core REST requests now wait with capped, jittered backoff for transient transport failures and HTTP 503/502/504/429. Context cancellation ends active requests and retry waits. Permanent HTTP/decode/endpoint errors return promptly, including hints for disabled REST and authentication mistakes.
PR #71 adds the resilience implementation (
a889318), sync benchmarks (053cc24), and the review repairs with their regression tests. It does not change database schemas or deploy anything.Validation
Current local checks passed on
b954ff15c4bc2d77f09ac084fcf07026729a5216(the committed tree is byte-identical to the fully checked tree), with both the default Go backend and-tags libsecp256k1:go test -v ./...,go vet ./..., and fullgo test -race -count=1 ./....go test -race -count=100 -run 'TestCoreAvailability(Logging|IgnoresStaleFailures)$' ./internal/indexerpassed.Flake evidence
The forced interleaving holds eight failed attempts until a newer request succeeds. On
053cc24, both healthy/recovering cases fail (2/2): stale failures emit an extra unavailable/available pair. With the repair, both cases pass in all 100 race-detector repetitions (0/200 failures), and the original parallel-outage logging test passes 100/100. A later genuinely new failure must still open an outage. No timeout, assertion or retry gate was loosened.The availability helper and these tests do not exist on the integration base or #70, so the same probe cannot compile there; the offending helper was introduced by
a889318. The previous CI run on053cc24passed, showing why a deterministic interleaving is needed. Routing: fix in this PR.Earlier real Bitcoin Core regtest restart and benchmark measurements were performed on
053cc24; they were not rerun for these repairs. The final healthy-path locking change has no new performance claim. The test audit also identified two nonblocking coverage gaps: the known-outage request-skip branch and exact cancellation-log suppression have no direct diagnostic assertions.Known limits