Repository navigation
Conversation
Contributor
📊 API Diff Results
|
j-nowak
force-pushed
the
l1-logpoller-batch-walk
branch
from
November 10, 2026 09:34
d668f60 to
e846df2
Compare
j-nowak
force-pushed
the
l1-logpoller-batch-walk
branch
from
November 10, 2026 09:38
e846df2 to
276ff0b
Compare
j-nowak
force-pushed
the
l1-logpoller-batch-walk
branch
from
November 10, 2026 09:42
276ff0b to
3caefb6
Compare
j-nowak
force-pushed
the
l1-logpoller-batch-walk
branch
2 times, most recently
from
November 10, 2026 10:24
b74c4fe to
7591d0a
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Replay handling can silently leave log ranges unqueried when the walked-block cap ends a replay prematurely or reorg recovery skips a discarded chunk’s still-canonical prefix.
3 open issues
What changed in this PR
Implements the opt-in batched unfinalized-block walk using the configuration introduced in #589. Batching remains disabled by default, preserving the legacy polling path.
Changes:
- Adds chunked header fetching, chain validation, bounded concurrent log queries, and a 2,000-block walk cap.
- Routes polling through the feature flag and logs batching settings when enabled.
- Adds focused tests for legacy parity, partial failures, reorgs, replay checks, timestamps, and concurrency limits.
| File | Description |
|---|---|
pkg/logpoller/unfinalized_batch.go |
Implements batched header walking and concurrent log retrieval. |
pkg/logpoller/unfinalized_batch_internal_test.go |
Tests batching behavior, failure handling, and legacy compatibility. |
pkg/logpoller/log_poller.go |
Adds feature-flag dispatch and startup logging. |
Review effort: Balanced
Comment on lines
+198
to
+201
| remaining := int64(maxUnfinalizedBlocksWalked) - walked | ||
| if remaining <= 0 { | ||
| lp.lggr.Warnw("Too many unfinalized blocks, stopping log retrieval to avoid OOM", "currentBlockNumber", currentBlock.Number, "latestBlockNumber", latest) | ||
| return blocks, logs, nil |
Comment on lines
+229
to
+230
| if dbBlock != nil && header.Hash != dbBlock.BlockHash { | ||
| return blocks, logs, newReorgError(header) |
|
|
||
| blocks, logs, err := m.LP.getUnfinalizedLogsBatched(t.Context(), headers[start], latest, latest, start+1, false) | ||
| require.Error(t, err) | ||
| require.NotErrorIs(t, err, new(reorgError)) |
j-nowak
force-pushed
the
l1-logpoller-batch-walk
branch
from
November 10, 2026 11:08
7591d0a to
310af40
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Context
Stacked on #589. This is PR 2 of the L1 stack: implement the opt-in batched unfinalized-block walk. The feature remains disabled by default; the legacy path is untouched and is still used unless
LogPollerBatchUnfinalizedBlocks = true.What changed
getUnfinalizedLogsBatched, a new walk that:SelectBlockByNumbercheck;eth_getLogscalls);skipEmptyBlocks, the finalized checkpoint, and the deferred last-block checkpoint;reorgErrorwhen the parent chain or replay DB state disagrees.pollAndSaveLogsthroughgetUnfinalizedLogsForMode.HeadByNumberand neverBatchCallContext, while the flag-on path usesBatchCallContextand neverHeadByNumber. Additional tests cover default-off, chunk-size defaulting/clamping, and log concurrency.Why logs are not JSON-RPC-batched
A single
eth_getLogsfilter accepts oneblockHash, so batching logs would be N independent requests in one envelope. The bounded worker pool keeps per-call metrics and the latency monitor intact, avoids provider batch-size limits and large responses, and works on chains whereBatchCallContextrejectseth_getLogs(for example Astar). This follows the POC's recommended implementation shape in #588.Safety
InsertLogsWithBlockscall.Testing
go test -short ./pkg/logpoller/...passes.CL_DATABASE_URL.CL_DATABASE_URL=... go test -race -run 'TestGetUnfinalizedLogsBatched|TestGetUnfinalizedLogsForMode|TestLogPoller_SynchronizedWithGeth' ./pkg/logpoller.CI is green: check-tidy, golangci-lint, Build Chainlink, and the full
go test -race ./...matrix (including DB-backed tests).Stack
l1-logpoller-batch-tests-metrics— property/integration coverage + metrics.