Repository navigation
indexer: fetch retry/backoff, loud placeholder warning, streams + stream_events folds (#566–#569) - #738
Merged
Conversation
…eam_events folds (conduit-protocol#566, conduit-protocol#567, conduit-protocol#568, conduit-protocol#569) conduit-protocol#569 — retry-with-backoff around the RPC fetch step: - new src/retry.ts: retryWithBackoff (exponential, default 3 attempts, 250ms base → 5s cap, abort-signal aware, one structured warn line per retry) plus the shared sleep helper. - pollOnce and Poller.tick now run getEvents/fetchEvents through it; the cursor is only advanced after a successful fetch, and startPollLoop keeps catching/backing off so an exhausted retry delays progress instead of stalling the worker. - tests: retry unit tests, pollOnce retries-and-advances, exhausted retries leave the cursor untouched, and both pollers provably keep advancing after a transient failure. conduit-protocol#568 — placeholder event source is loud, not silent: - StubSorobanEventSource logs a one-time warning on construction, and worker.ts warns once at boot/first fetch (the path npm run start:worker actually runs), so "running the stub" and "RPC returns nothing" no longer look identical in logs. conduit-protocol#567 — streams table + creation fold: - indexer/db/schema.sql gains `streams` (id, sender, recipient, token, rate_per_second, start_time, end_time, created_at) and applyEvent folds the creation event into it with an idempotent ON CONFLICT upsert. conduit-protocol#566 — stream_events table + DripStream fold handlers: - indexer/db/schema.sql gains `stream_events` (PK ledger, tx_hash, event_type, stream_id) and applyEvent now handles withdrawn, cancelled, force_cxl, paused, resumed, topped_up, clawback and xfer_rec — both the README names and the raw on-chain topic aliases — instead of only the DAO-voting events the deployed contracts never emit. Unrecognized types still log-and-drop (conduit-protocol#578). Also fixes the pre-existing prefer-const lint error in worker.ts so the indexer CI job (lint/typecheck/test) is green, and documents the failure handling and derived tables in indexer/README.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes four indexer scaffold gaps that together meant the worker could either
stall invisibly or never index the protocol it ships alongside.
closes #569 — no retry/backoff around the fetch step
indexer/src/retry.ts—retryWithBackoff(): exponential (default 3attempts, 250ms base doubling to a 5s cap), abort-signal aware, emits one
structured warn line per retry so a retrying worker never looks "healthy
but idle".
pollOnce()andPoller.tick()now runs through it.The cursor is only checkpointed after a successful fetch; when retries are
exhausted the error is logged, the cursor stays put, and the next tick /
loop iteration resumes from the same position.
rollback on failure) and are retried by the loop-level backoff in
startPollLoop— now documented in code rather than incidental.closes #568 — stub event source silently indexed nothing
StubSorobanEventSourcenow logs a one-time warning on construction:"using placeholder SorobanEventSource — no events will be indexed".
src/worker.tswarns once at boot and on first fetch, covering thenpm run start:workerpath where the placeholder lives infetchEventsFromRPC. "Intentionally running the scaffold" and "the realRPC implementation is broken and returning nothing" are now distinguishable
from the logs.
closes #567 — no table tracks stream creation
streamstable:id, sender, recipient, token, rate_per_second, start_time, end_time, created_at.handleStreamCreatedfold inapplyEvent(handlescreatedandstream_createdtags) with an idempotentON CONFLICT (id) DO UPDATE—created_atis never rewritten, so "streams created in the last 24h"survives page replays.
closes #566 — applyEvent only handled DAO-voting events
stream_eventstable (PKledger, tx_hash, event_type, stream_id).applyEventnow foldsstream_withdrawnandstream_cancelled(plusforce_cxl,stream_paused,stream_resumed,stream_topped_up,stream_clawback,xfer_rec) using the existing upsert pattern, acceptingboth the README event names and the raw on-chain topic aliases.
applyEvent'sdefault: returnsilently drops unrecognized event types #578).Acceptance criteria
poller.tshas no retry/backoff aroundfetchPage/ingestPagefailures #569 explicit retry-with-backoff around the RPC fetch steppoller.tshas no retry/backoff aroundfetchPage/ingestPagefailures #569 test asserting the poller doesn't silently stop advancing after asingle transient failure (both pollers)
chainEventSource.tsstub silently returns an empty page forever #568 one-time warning on stub constructionDripFactory::create_stream(stream creation) #567streamstable with the required columns + stream-creation foldapplyEventhas no handler for any DripStream event — only DAO-voting events #566stream_eventstable + handlers for at leaststream_withdrawnand
stream_cancelled, following the existing upsert patternTesting
retry,poller,indexerPoller,applyEvent,placeholderSource,schemanpm run lint/npm run typecheck/npm test(indexer CI steps): 72passed, 1 skipped, 0 failed
prefer-constlint error inworker.tsthat was keeping the indexer CI job red