Skip to content

indexer: fetch retry/backoff, loud placeholder warning, streams + stream_events folds (#566–#569) - #738

Merged
Jaydbrown merged 1 commit into
conduit-protocol:mainfrom
markdavid000:fix/indexer-566-569
Oct 6, 2026
Merged

Jaydbrown merged 1 commit into
conduit-protocol:mainfrom
markdavid000:fix/indexer-566-569

Conversation

@markdavid000

Copy link
Copy Markdown
Contributor

Summary

Fixes four indexer scaffold gaps that together meant the worker could either
stall invisibly or never index the protocol it ships alongside.

closes #569 — no retry/backoff around the fetch step

  • Adds indexer/src/retry.ts — retryWithBackoff(): exponential (default 3
    attempts, 250ms base doubling to a 5s cap), abort-signal aware, emits one
    structured warn line per retry so a retrying worker never looks "healthy
    but idle".
  • The RPC fetch in pollOnce() and Poller.tick() now runs through it.
    The cursor is only checkpointed after a successful fetch; when retries are
    exhausted the error is logged, the cursor stays put, and the next tick /
    loop iteration resumes from the same position.
  • Ingest failures stay transactional (BEGIN → ingest → saveCursor → COMMIT,
    rollback on failure) and are retried by the loop-level backoff in
    startPollLoop — now documented in code rather than incidental.

closes #568 — stub event source silently indexed nothing

  • StubSorobanEventSource now logs a one-time warning on construction:
    "using placeholder SorobanEventSource — no events will be indexed".
  • src/worker.ts warns once at boot and on first fetch, covering the
    npm run start:worker path where the placeholder lives in
    fetchEventsFromRPC. "Intentionally running the scaffold" and "the real
    RPC implementation is broken and returning nothing" are now distinguishable
    from the logs.

closes #567 — no table tracks stream creation

  • New streams table: id, sender, recipient, token, rate_per_second, start_time, end_time, created_at.
  • New handleStreamCreated fold in applyEvent (handles created and
    stream_created tags) with an idempotent ON CONFLICT (id) DO UPDATE —
    created_at is never rewritten, so "streams created in the last 24h"
    survives page replays.

closes #566 — applyEvent only handled DAO-voting events

  • New stream_events table (PK ledger, tx_hash, event_type, stream_id).
  • applyEvent now folds stream_withdrawn and stream_cancelled (plus
    force_cxl, stream_paused, stream_resumed, stream_topped_up,
    stream_clawback, xfer_rec) using the existing upsert pattern, accepting
    both the README event names and the raw on-chain topic aliases.
  • Unrecognized types still log-and-drop (indexer: applyEvent's default: return silently drops unrecognized event types #578).

Acceptance criteria

Testing

  • 33 new tests across retry, poller, indexerPoller, applyEvent,
    placeholderSource, schema
  • npm run lint / npm run typecheck / npm test (indexer CI steps): 72
    passed, 1 skipped, 0 failed
  • Drive-by: fixed the pre-existing prefer-const lint error in worker.ts
    that was keeping the indexer CI job red

…eam_events folds (conduit-protocol#566, conduit-protocol#567, conduit-protocol#568, conduit-protocol#569)

conduit-protocol#569 — retry-with-backoff around the RPC fetch step:
- new src/retry.ts: retryWithBackoff (exponential, default 3 attempts,
  250ms base → 5s cap, abort-signal aware, one structured warn line per
  retry) plus the shared sleep helper.
- pollOnce and Poller.tick now run getEvents/fetchEvents through it; the
  cursor is only advanced after a successful fetch, and startPollLoop
  keeps catching/backing off so an exhausted retry delays progress
  instead of stalling the worker.
- tests: retry unit tests, pollOnce retries-and-advances, exhausted
  retries leave the cursor untouched, and both pollers provably keep
  advancing after a transient failure.

conduit-protocol#568 — placeholder event source is loud, not silent:
- StubSorobanEventSource logs a one-time warning on construction, and
  worker.ts warns once at boot/first fetch (the path npm run start:worker
  actually runs), so "running the stub" and "RPC returns nothing" no
  longer look identical in logs.

conduit-protocol#567 — streams table + creation fold:
- indexer/db/schema.sql gains `streams` (id, sender, recipient, token,
  rate_per_second, start_time, end_time, created_at) and applyEvent folds
  the creation event into it with an idempotent ON CONFLICT upsert.

conduit-protocol#566 — stream_events table + DripStream fold handlers:
- indexer/db/schema.sql gains `stream_events` (PK ledger, tx_hash,
  event_type, stream_id) and applyEvent now handles withdrawn, cancelled,
  force_cxl, paused, resumed, topped_up, clawback and xfer_rec — both the
  README names and the raw on-chain topic aliases — instead of only the
  DAO-voting events the deployed contracts never emit. Unrecognized types
  still log-and-drop (conduit-protocol#578).

Also fixes the pre-existing prefer-const lint error in worker.ts so the
indexer CI job (lint/typecheck/test) is green, and documents the failure
handling and derived tables in indexer/README.md.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants