Skip to content

fix(indexer): wait for Bitcoin Core instead of exiting while it is unavailable - #71

Open
setavenger wants to merge 6 commits into
integrationfrom
fix/core-unavailable-retry
Open

setavenger wants to merge 6 commits into
integrationfrom
fix/core-unavailable-retry

Conversation

@setavenger

@setavenger setavenger commented Oct 30, 2026 •

Copy link
Copy Markdown
Owner

Base: integration. Stacked on #70: this branch still contains its three deadlock/cancellation commits. Please merge #70 first; merge and deployment remain owner decisions.

Problem and behavior

After a reboot, the oracle exited while Bitcoin Core was still starting: REST returned 503 or refused connections. The same failure stopped the oracle during a Core restart.

Core REST requests now wait with capped, jittered backoff for transient transport failures and HTTP 503/502/504/429. Context cancellation ends active requests and retry waits. Permanent HTTP/decode/endpoint errors return promptly, including hints for disabled REST and authentication mistakes.

  • Parallel failed requests share availability logging. Every successful attempt invalidates older in-flight failures, so a delayed failure cannot reopen an outage after recovery.
  • Continuous sync returns the original storage error when its writer cancels an overlapping chain-info poll.
  • Optional progress reporting skips known outages, uses a two-second request deadline, and never reports an unknown Core tip as “up to date.” Failed progress checks log at debug level.
  • Cancellation does not produce redundant block-pull error logs. The polling cancellation regression waits for an actual request before cancelling.

PR #71 adds the resilience implementation (a889318), sync benchmarks (053cc24), and the review repairs with their regression tests. It does not change database schemas or deploy anything.

Validation

Current local checks passed on b954ff15c4bc2d77f09ac084fcf07026729a5216 (the committed tree is byte-identical to the fully checked tree), with both the default Go backend and -tags libsecp256k1:

  • CLI build, full go test -v ./..., go vet ./..., and full go test -race -count=1 ./....
  • Focused recovery/error/progress/cancellation tests passed three times under the race detector.
  • go test -race -count=100 -run 'TestCoreAvailability(Logging|IgnoresStaleFailures)$' ./internal/indexer passed.
  • Functional gstack-qa-only probes verified all 60 expected block heights/hashes were applied once, forced eight stale failures across a newer success, preserved an injected writer error, and completed sync with 503/hung progress requests. These use synthetic HTTP fixtures and an in-memory store.
  • Skills run: gstack-investigate, gstack-review, gstack-qa-only, and gstack-ship. No UI surface changed. A fresh Codex source adversarial review and test coverage audit cover the repairs. A refreshed Claude review remains unavailable because its session is usage-blocked; this is not a Claude pass.

Flake evidence

The forced interleaving holds eight failed attempts until a newer request succeeds. On 053cc24, both healthy/recovering cases fail (2/2): stale failures emit an extra unavailable/available pair. With the repair, both cases pass in all 100 race-detector repetitions (0/200 failures), and the original parallel-outage logging test passes 100/100. A later genuinely new failure must still open an outage. No timeout, assertion or retry gate was loosened.

The availability helper and these tests do not exist on the integration base or #70, so the same probe cannot compile there; the offending helper was introduced by a889318. The previous CI run on 053cc24 passed, showing why a deterministic interleaving is needed. Routing: fix in this PR.

Earlier real Bitcoin Core regtest restart and benchmark measurements were performed on 053cc24; they were not rerun for these repairs. The final healthy-path locking change has no new performance claim. The test audit also identified two nonblocking coverage gaps: the known-outage request-skip branch and exact cancellation-log suppression have no direct diagnostic assertions.

Known limits

  • Unknown transport errors, including some permanent endpoint/proxy mistakes, retry until cancellation, with periodic warnings. This preserves the intended wait-through-restart behavior.
  • A Core restart with a shorter tip (for example reindexing) can still return a permanent 404 for an in-flight requested height and stop sync. Recovery from that case is not implemented here.
  • Mixed successes and transient failures can still produce multiple availability transitions; the generation fix addresses stale completion ordering, not partial-service health policy.
  • Continuous sync already writes a new tip before backfilling lower heights. An outage can prolong that existing gap; requests for missing heights return errors until backfill finishes.
  • Existing SIGTERM handling and the fresh-database start-height issue (fix(indexer): index the configured start height on a fresh database #66) are unchanged. Real users/data remain on their existing deployment.
  • No public-repo merge or production deployment was performed. The owner will choose the released master version to run.

jrsd86 and others added 6 commits October 10, 2026 18:49
A single failed REST request (timeout, connection reset) left the indexer
blocked forever while the process kept running:

- PullBlockData sent fetch errors on an unbuffered channel that was only
  read after wg.Wait(). The failing goroutine blocked before wg.Done(), so
  wg.Wait() never returned.
- In SyncBlocks a failed pull returned without wg.Done(), so newBlockChan
  was never closed. Later errors blocked on the unbuffered errChan once the
  first one had been received, and the pull loop kept dispatching.
- handleBlock sent to writerChan without watching ctx, and
  pullBlockToChan sent to newBlockChan without watching ctx, so both could
  block forever once the consumer had stopped.
- In ContinuousSync the writer could fail while the main loop was blocked
  handing it a block: each side waited on the other.

Changes:

- PullBlockData records each fetch's error and joins them after wg.Wait().
- SyncBlocks derives a cancellable context; the first error from any stage
  is recorded without blocking and cancels the others. Every channel send
  also selects on ctx. SyncBlocks returns only after all goroutines it
  started have exited.
- ContinuousSync cancels on writer failure, uses a buffered error channel
  and waits for the writer before returning; it returns the writer's error
  rather than the cancellation it caused.
- The REST fetch helpers take a context (http.NewRequestWithContext) so
  cancellation also aborts in-flight requests, and wrap errors with %w so
  callers can match context.Canceled / DeadlineExceeded.

Tests add a fault-injecting fake node (httptest) and an in-memory store.
They cover a dropped request on each endpoint, concurrent failures, a
writer failure with full channels, cancellation with a request that never
answers, and the writer error in ContinuousSync, and assert that no
indexer goroutine is left running. Against the previous code the five
regression tests fail (deadlock or leaked goroutines).
…ata fails

PullBlockData fetches a block and its spent outputs concurrently. When one
fetch failed, the other was left running until it completed or hit the
client timeout (100 s), so a failed block pull could still take up to that
long to return.

Each fetch now runs under a context derived from the caller's. The first
failure cancels it, which aborts the sibling request at once. The
cancellation that PullBlockData causes itself is dropped from the returned
error, so callers see the real failure; if the caller's own context is done,
its error is kept.

TestPullBlockDataCancelsSiblingRequest drops the spenttxouts request while
the block request never answers. Without this change PullBlockData only
returns after the client timeout, and the test fails at its 10 s limit.

TestContinuousSyncWriterError now pins config.SyncStartHeight instead of
relying on the package default, so it keeps working once reorg handling
that walks back to the start height is merged.
GetChainInfo took no context. SyncBlocks' progress report and
ContinuousSync's polling call it while running, so with an unresponsive
node a cancelled sync stayed blocked in that request until the HTTP client
timeout (100 s) before it could return.

GetChainInfo now takes a context and builds its request with
http.NewRequestWithContext; its three callers pass the context they run
under. The progress report interval becomes a package variable so tests can
shorten it.

Tests:
- SyncBlocks and ContinuousSync are cancelled while the node never answers
  chaininfo. Without this change both only return after the client timeout
  and fail at the tests' 10 s limit; with it they return promptly.
- The sibling-cancellation test from the previous commit now covers both
  directions: block failing while spenttxouts hangs, and the reverse.

Also documents in SyncBlocks why an error that merely results from the
internal cancellation can never be recorded before the error that caused it.
…available

After a VM reboot on 2026-10-17 both oracles on oracle.setor.dev exited
within a second of starting, because Bitcoin Core was not ready yet:

- mainnet: REST answered 503 while Core loaded its block index, and
  rest.go called log.Fatal on any non-200 status;
- signet: the connection was refused, InitialSyncToTip returned the
  error and the process exited.

The same happens at runtime: when Core restarts while the oracle syncs or
waits at the tip, the indexer stops and the process exits, taking the
HTTP and gRPC servers with it.

Every REST request now goes through restGet. It retries failures that
waiting for Core can fix, with a backoff from 0.5 s capped at 15 s plus
jitter, until Core answers or the context is cancelled:

- transport errors: refused, reset or dropped connections, timeouts,
  name lookups, a connection lost mid-body;
- 503 (Core starting, shutting down or with a full work queue) and 502,
  504, 429 from a proxy in front of it.

Answers that waiting cannot change are returned at once, so a
misconfiguration still stops the oracle, now with an error that says
what to check instead of log.Fatal:

- 4xx and other 5xx statuses;
- a complete body that does not decode;
- a core_rest_endpoint that is not an http(s) URL;
- an HTTP/HTTPS or certificate mismatch.

Core's reason from the response body is included in the error.

While Core is away the indexer logs one warning when it goes, a reminder
every minute and one line when it is back, not one line per request.
Because `run` starts the servers before the indexer, already-indexed
data keeps being served while the indexer waits.

Builds on #70: its context plumbing lets a waiting request stop when the
sync is cancelled or a sibling fetch fails for good, and its error
propagation makes a permanent failure stop the sync instead of
deadlocking it. #70's tests injected dropped connections to exercise
that propagation; those are retried now, so the tests inject a 404
instead and new tests cover the retries. The SyncBlocks progress report
asks for the chain state once rather than waiting, so it cannot stall
the writer. blockCheckInterval becomes a variable so the tests can poll
quickly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BenchmarkSyncBlocksRealBlocks syncs 16 heights per op from a fake REST
node that serves the four mainnet blocks in test_data (with their spent
outputs, rebuilt from the getblock verbosity-3 fixtures and checked
against the fixture hashes). BenchmarkSyncBlocksTinyBlocks syncs 200
coinbase-only blocks, so its time is dominated by the request path.

The file only uses APIs that predate the retry change, so it can be
copied onto older trees for A/B comparisons. Run it with the C backend:

  go test -tags libsecp256k1 -run '^$' -bench SyncBlocks -benchtime 20x ./internal/indexer/

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: OpenAI Codex <noreply@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants