Skip to content

feat(logpoller): implement opt-in batched unfinalized walk - #590

Draft
j-nowak wants to merge 1 commit into
l1-logpoller-batch-configfrom
l1-logpoller-batch-walk
Draft

j-nowak wants to merge 1 commit into
l1-logpoller-batch-configfrom
l1-logpoller-batch-walk

Conversation

@j-nowak

@j-nowak j-nowak commented Nov 10, 2026 •

Copy link
Copy Markdown

Context

Stacked on #589. This is PR 2 of the L1 stack: implement the opt-in batched unfinalized-block walk. The feature remains disabled by default; the legacy path is untouched and is still used unless LogPollerBatchUnfinalizedBlocks = true.

What changed

  • Add getUnfinalizedLogsBatched, a new walk that:
    • preserves the legacy anchor log query;
    • fetches headers in one JSON-RPC batch per chunk;
    • verifies the parent-hash chain locally before issuing any log query;
    • preserves the replay SelectBlockByNumber check;
    • fetches logs for the chunk with a bounded worker pool (4 concurrent single-block eth_getLogs calls);
    • preserves skipEmptyBlocks, the finalized checkpoint, and the deferred last-block checkpoint;
    • caps the number of walked headers per poll at 2,000;
    • saves only the contiguous successful prefix on partial batches / log failures, so the next poll resumes without a gap;
    • discards a chunk and returns reorgError when the parent chain or replay DB state disagrees.
  • Dispatch in pollAndSaveLogs through getUnfinalizedLogsForMode.
  • Log an info line at startup when the feature is enabled, including chunk size and log concurrency.
  • Add focused unit tests for parity with legacy, anchor inclusion, mid-chunk reorgs, partial header batches, null headers at the tip, log failures, replay DB checks, walked cap, concurrency bound, per-block timestamps, and mode selection. The selector tests assert that the flag-off path uses HeadByNumber and never BatchCallContext, while the flag-on path uses BatchCallContext and never HeadByNumber. Additional tests cover default-off, chunk-size defaulting/clamping, and log concurrency.

Why logs are not JSON-RPC-batched

A single eth_getLogs filter accepts one blockHash, so batching logs would be N independent requests in one envelope. The bounded worker pool keeps per-call metrics and the latency monitor intact, avoids provider batch-size limits and large responses, and works on chains where BatchCallContext rejects eth_getLogs (for example Astar). This follows the POC's recommended implementation shape in #588.

Safety

  • Default-off; the legacy path is unchanged.
  • No DB schema changes; the batch path writes the same rows through the same InsertLogsWithBlocks call.
  • A partial header batch or log failure never skips a block.
  • A reorg discards the current chunk before any orphaned logs are saved.
  • Chunk size is bounded (default 16, max 50); log concurrency is fixed at 4 in this PR.

Testing

  • go test -short ./pkg/logpoller/... passes.
  • Focused batch tests cover the cases listed above.
  • DB-backed property/integration tests are added in the next PR (Add zksync compilation scripts #3) and require CL_DATABASE_URL.
  • Also ran locally with a real Postgres: CL_DATABASE_URL=... go test -race -run 'TestGetUnfinalizedLogsBatched|TestGetUnfinalizedLogsForMode|TestLogPoller_SynchronizedWithGeth' ./pkg/logpoller.

CI is green: check-tidy, golangci-lint, Build Chainlink, and the full go test -race ./... matrix (including DB-backed tests).

Stack

  1. feat(logpoller): add opt-in config for batched unfinalized walk #589 — config only, no behavior.
  2. This PR — batched walk + focused unit tests.
  3. l1-logpoller-batch-tests-metrics — property/integration coverage + metrics.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

📊 API Diff Results

No changes detected for module github.com/smartcontractkit/chainlink-evm

View full report

@j-nowak
j-nowak force-pushed the l1-logpoller-batch-walk branch from d668f60 to e846df2 Compare November 10, 2026 09:34
@j-nowak
j-nowak force-pushed the l1-logpoller-batch-walk branch from e846df2 to 276ff0b Compare November 10, 2026 09:38
@j-nowak
j-nowak force-pushed the l1-logpoller-batch-walk branch from 276ff0b to 3caefb6 Compare November 10, 2026 09:42
@j-nowak
j-nowak added this pull request to stack #592 November 10, 2026 10:15
@j-nowak
j-nowak force-pushed the l1-logpoller-batch-walk branch 2 times, most recently from b74c4fe to 7591d0a Compare November 10, 2026 10:24
@j-nowak
j-nowak requested a balanced review from Copilot November 10, 2026 10:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Replay handling can silently leave log ranges unqueried when the walked-block cap ends a replay prematurely or reorg recovery skips a discarded chunk’s still-canonical prefix.

3 open issues
What changed in this PR

Implements the opt-in batched unfinalized-block walk using the configuration introduced in #589. Batching remains disabled by default, preserving the legacy polling path.

Changes:

  • Adds chunked header fetching, chain validation, bounded concurrent log queries, and a 2,000-block walk cap.
  • Routes polling through the feature flag and logs batching settings when enabled.
  • Adds focused tests for legacy parity, partial failures, reorgs, replay checks, timestamps, and concurrency limits.
File Description
pkg/​logpoller/​unfinalized_batch.go Implements batched header walking and concurrent log retrieval.
pkg/​logpoller/​unfinalized_batch_internal_test.go Tests batching behavior, failure handling, and legacy compatibility.
pkg/​logpoller/​log_poller.go Adds feature-flag dispatch and startup logging.

Review effort: Balanced

Comment on lines +198 to +201
remaining := int64(maxUnfinalizedBlocksWalked) - walked
if remaining <= 0 {
lp.lggr.Warnw("Too many unfinalized blocks, stopping log retrieval to avoid OOM", "currentBlockNumber", currentBlock.Number, "latestBlockNumber", latest)
return blocks, logs, nil
Comment on lines +229 to +230
if dbBlock != nil && header.Hash != dbBlock.BlockHash {
return blocks, logs, newReorgError(header)

blocks, logs, err := m.LP.getUnfinalizedLogsBatched(t.Context(), headers[start], latest, latest, start+1, false)
require.Error(t, err)
require.NotErrorIs(t, err, new(reorgError))
@j-nowak
j-nowak force-pushed the l1-logpoller-batch-walk branch from 7591d0a to 310af40 Compare November 10, 2026 11:08

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants