Repository navigation
Enable OTel metric export batching - #23491
Merged
Merged
Conversation
pkcll
force-pushed
the
otel-metric-export-batching
branch
from
October 6, 2026 03:28
5f4c6ed to
9aa41a9
Compare
Contributor
|
✅ No conflicts with other open PRs targeting |
1 task
pkcll
added a commit
that referenced
this pull request
Oct 6, 2026
Enables node operators to turn on gRPC-level retries for the node's chip ingress client (and LOOP plugins' own clients) via [Telemetry] TOML, wiring the pass-through added by chainlink-common#2441 on top of the retry support from chainlink-common#2373. - [Telemetry] ChipIngressRetry* keys: Enabled (off by default) plus optional MaxAttempts / InitialBackoff / MaxBackoff / BackoffMultiplier / RetryableStatusCodes overrides; unset knobs fall back to chipingress.DefaultRetryPolicy values. - core/cmd/shell.go newBeholderClient resolves them into beholder.Config.ChipIngressRetryPolicy for the node's shared chip ingress client (custom events + telemetry/heartbeat path). - plugins/loop_registry.go forwards the resolved values to LOOP plugin subprocesses over the existing CL_CHIP_INGRESS_* env channel. - Validation: MaxAttempts >= 2 (gRPC silently installs no retry policy below 2), positive backoffs/multiplier; unknown status-code names fail node startup via chipingress.ParseStatusCodes. - docs: core.toml keys, regenerated CONFIG.md, new docs/CHIP_INGRESS_RETRY.md operator page. Follows the wiring pattern of #23491 (OTel metric export batching). Retries remain opt-in: replayed publishes can duplicate events unless producers set an idempotency key; the durable emitter path is intentionally excluded (its retransmit loop already guarantees delivery). Jira: INFOPLAT-19422
|
|
pkcll
added a commit
that referenced
this pull request
Oct 6, 2026
Enables node operators to turn on gRPC-level retries for the node's chip ingress client (and LOOP plugins' own clients) via [Telemetry] TOML, wiring the pass-through added by chainlink-common#2441 on top of the retry support from chainlink-common#2373. - [Telemetry] ChipIngressRetry* keys: Enabled (off by default) plus optional MaxAttempts / InitialBackoff / MaxBackoff / BackoffMultiplier / RetryableStatusCodes overrides; unset knobs fall back to chipingress.DefaultRetryPolicy values. - core/cmd/shell.go newBeholderClient resolves them into beholder.Config.ChipIngressRetryPolicy for the node's shared chip ingress client (custom events + telemetry/heartbeat path). - plugins/loop_registry.go forwards the resolved values to LOOP plugin subprocesses over the existing CL_CHIP_INGRESS_* env channel. - Validation: MaxAttempts >= 2 (gRPC silently installs no retry policy below 2), positive backoffs/multiplier; unknown status-code names fail node startup via chipingress.ParseStatusCodes. - docs: core.toml keys, regenerated CONFIG.md, new docs/CHIP_INGRESS_RETRY.md operator page. Follows the wiring pattern of #23491 (OTel metric export batching). Retries remain opt-in: replayed publishes can duplicate events unless producers set an idempotency key; the durable emitter path is intentionally excluded (its retransmit loop already guarantees delivery). Jira: INFOPLAT-19422
pkcll
added a commit
that referenced
this pull request
Oct 6, 2026
Enables node operators to turn on gRPC-level retries for the node's chip ingress client (and LOOP plugins' own clients) via [Telemetry] TOML, wiring the pass-through added by chainlink-common#2441 on top of the retry support from chainlink-common#2373. - [Telemetry] ChipIngressRetry* keys: Enabled (off by default) plus optional MaxAttempts / InitialBackoff / MaxBackoff / BackoffMultiplier / RetryableStatusCodes overrides; unset knobs fall back to chipingress.DefaultRetryPolicy values. - core/cmd/shell.go newBeholderClient resolves them into beholder.Config.ChipIngressRetryPolicy for the node's shared chip ingress client (custom events + telemetry/heartbeat path). - plugins/loop_registry.go forwards the resolved values to LOOP plugin subprocesses over the existing CL_CHIP_INGRESS_* env channel. - Validation: MaxAttempts >= 2 (gRPC silently installs no retry policy below 2), positive backoffs/multiplier; unknown status-code names fail node startup via chipingress.ParseStatusCodes. - docs: core.toml keys, regenerated CONFIG.md, new docs/CHIP_INGRESS_RETRY.md operator page. Follows the wiring pattern of #23491 (OTel metric export batching). Retries remain opt-in: replayed publishes can duplicate events unless producers set an idempotency key; the durable emitter path is intentionally excluded (its retransmit loop already guarantees delivery). Jira: INFOPLAT-19422
pkcll
force-pushed
the
otel-metric-export-batching
branch
from
November 17, 2026 16:52
9aa41a9 to
5e4d0a0
Compare
pkcll
marked this pull request as ready for review
November 17, 2026 19:38
product-security-plaid-production
Bot
requested review from
chudilka1 and
kalverra
November 17, 2026 19:38
jmank88
previously approved these changes
Nov 17, 2026
kirqz23
previously approved these changes
Nov 18, 2026
jmank88
approved these changes
Nov 18, 2026
pkcll
enabled auto-merge
November 18, 2026 15:32
ilija42
approved these changes
Nov 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.




Summary
chainlink-commontov0.11.2-0.20261117163148-84b62b920030, which includes the batching support from Add OTel metric export batching support chainlink-common#2337 and the OTel v1.47 /WithMaxExportBatchSizemigration from dependabump chainlink-common#2550.Telemetry.MetricExportBatchSizecore TOML config (default0= batching disabled). A positive value limits metric data points per OTLP export request; the count is not a byte bound and does not replace the collector receive limit.newBeholderClient(beholder.Config.MetricExportBatchSize) and LOOP plugin registry env config (CL_TELEMETRY_METRIC_EXPORT_BATCH_SIZE), matching theTelemetry.MetricCardinalityLimitwiring from Wire Telemetry.MetricCardinalityLimit to Beholder #23084.core/config/docs/core.toml,docs/CONFIG.md) and golden fixtures (core/services/chainlink/testdata,core/web/resolver/testdata,testdata/scriptstxtar).Validation
github.com/smartcontractkit/chainlink-common v0.11.2-0.20261117163148-84b62b920030,go.opentelemetry.io/otel/sdk/metric v1.47.0go build ./core/config/... ./core/web/resolver/... ./plugins/...— passgo test ./plugins/ -run TestLoopRegistry— passgo test ./core/web/resolver/...— passgo test ./core/services/chainlink/ -run 'TestTelemetryConfig_MetricExportBatchSize|TestConfig_Marshal'— passgo test . -run 'TestScripts/(node/validate|config)'andTestConfigDocs— passgolangci-lintreports no new issues on touched filesVerification:
MetricExportBatchSize = 0disables batchingChecked by reading the code at each layer (chainlink-common
84b62b9, OTelsdk/metricv1.47.0):MetricExportBatchSize()returns 0 for nil, 0 and negative values.newPeriodicReaderWithMaxExportBatchSizeonly when the size is > 0, so at 0 the reader is built as before.CL_TELEMETRY_METRIC_EXPORT_BATCH_SIZE=0, parsed intoTelemetryMetricExportBatchSize, passed on asMetricExportBatchSize: 0, which takes the same path.WithMaxExportBatchSizeignores sizes <= 0, and batching only runs whenbatcher.size > 0. SDK tests assert 0 and -1 give default behavior.Tests: chainlink-common #2337 covers unset, zero and negative as unbatched;
TestTelemetryConfig_MetricExportBatchSizecovers nil and 0. Negative values behave like 0 and are not rejected.Follow-up / limitations
WithMaxExportBatchSizereader option in chainlink-commonnewPeriodicReader; see dependabump chainlink-common#2550) and the collector receive limit remains the independent guardrail.