Skip to content

perf(stellar): pipeline streaming scan to overlap RPC fetch with decrypt#154

Open
collinsezedike wants to merge 1 commit into
wraith-protocol:developfrom
collinsezedike:perf/stellar-streaming-scan-pipeline
Open

perf(stellar): pipeline streaming scan to overlap RPC fetch with decrypt#154
collinsezedike wants to merge 1 commit into
wraith-protocol:developfrom
collinsezedike:perf/stellar-streaming-scan-pipeline

Conversation

@collinsezedike

Copy link
Copy Markdown

Summary

  • Add a generic bounded-queue pipeline(source, capacity) helper in src/chains/stellar/scanner/pipeline.ts. A background pump keeps pulling from source up to capacity items ahead of the consumer, so the next RPC fetch's I/O is already in flight while the consumer does CPU-bound work on already-buffered items, instead of alternating "await a full window, then process it."
  • scanAnnouncementsStream now wraps its source in pipeline(source, window) instead of prefetching a strict window before scanning any of it. Peak memory stays O(window); cancellation and error propagation are unchanged.
  • The old windowed algorithm is kept as scanAnnouncementsStreamSequential for benchmark comparisons, matching how scanAnnouncementsLegacySharedSecretTag is retained for the view-tag-batching benchmark.
  • Fixed test/chains/stellar/bench/scan.bench.ts, which had corrupted/duplicated content on develop (left over from an earlier merge) making it fail to even parse — unrelated to this change but blocking it since I needed to extend that file.
  • Docs in docs/chains/stellar-streaming-scan-pipeline.md, changelog entry under the upcoming release.

Closes #126.

Semver

Additive-only: new internal module (src/chains/stellar/scanner/, not part of the public package exports) plus a new benchmark-only export scanAnnouncementsStreamSequential. fetchAnnouncementsStream and scanAnnouncementsStream's public signatures and behavior are unchanged.

Test plan

  • pnpm build
  • pnpm test — full suite passes (793 passed in the Stellar module, no regressions)
  • New test/chains/stellar/scanner/pipeline.test.ts (runs under pnpm test, not excluded like bench/): order preservation, overlap timing (pipelined run is reliably ≥25% faster than a measured sequential baseline with matched I/O/CPU delays), backpressure bound with a fast producer + slow consumer, cancellation propagation, error propagation
  • test/chains/stellar/bench/scan.bench.ts — new "Stellar streaming scan pipelining" section: correctness test plus before/after benchmarks against a mock paginated RPC source; measured 36% wall-clock reduction on the 10k-announcement canned dataset (537ms → 343ms), clearing the 30% target
  • Existing scanAnnouncementsStream tests (window sizing, cancellation, memory-bounded) pass unmodified
  • pnpm format:check

@drips-wave

drips-wave Bot commented Jul 24, 2026

Copy link
Copy Markdown

@collinsezedike Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Streaming decrypt: overlap RPC with CPU work

1 participant