Skip to content

Enable OTel metric export batching - #23491

Merged
pkcll merged 6 commits into
developfrom
otel-metric-export-batching
Nov 18, 2026
Merged

pkcll merged 6 commits into
developfrom
otel-metric-export-batching

Conversation

@pkcll

@pkcll pkcll commented Aug 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Bump chainlink-common to v0.11.2-0.20261117163148-84b62b920030, which includes the batching support from Add OTel metric export batching support chainlink-common#2337 and the OTel v1.47 / WithMaxExportBatchSize migration from dependabump chainlink-common#2550.
  • Add Telemetry.MetricExportBatchSize core TOML config (default 0 = batching disabled). A positive value limits metric data points per OTLP export request; the count is not a byte bound and does not replace the collector receive limit.
  • Wire the value through newBeholderClient (beholder.Config.MetricExportBatchSize) and LOOP plugin registry env config (CL_TELEMETRY_METRIC_EXPORT_BATCH_SIZE), matching the Telemetry.MetricCardinalityLimit wiring from Wire Telemetry.MetricCardinalityLimit to Beholder #23084.
  • Update generated docs (core/config/docs/core.toml, docs/CONFIG.md) and golden fixtures (core/services/chainlink/testdata, core/web/resolver/testdata, testdata/scripts txtar).

Validation

  • Resolved modules: github.com/smartcontractkit/chainlink-common v0.11.2-0.20261117163148-84b62b920030, go.opentelemetry.io/otel/sdk/metric v1.47.0
  • go build ./core/config/... ./core/web/resolver/... ./plugins/... — pass
  • go test ./plugins/ -run TestLoopRegistry — pass
  • go test ./core/web/resolver/... — pass
  • go test ./core/services/chainlink/ -run 'TestTelemetryConfig_MetricExportBatchSize|TestConfig_Marshal' — pass
  • go test . -run 'TestScripts/(node/validate|config)' and TestConfigDocs — pass
  • golangci-lint reports no new issues on touched files

Verification: MetricExportBatchSize = 0 disables batching

Checked by reading the code at each layer (chainlink-common 84b62b9, OTel sdk/metric v1.47.0):

Layer Behavior at 0
Core TOML getter MetricExportBatchSize() returns 0 for nil, 0 and negative values.
Beholder newPeriodicReader Adds WithMaxExportBatchSize only when the size is > 0, so at 0 the reader is built as before.
LOOP plugins Sent as CL_TELEMETRY_METRIC_EXPORT_BATCH_SIZE=0, parsed into TelemetryMetricExportBatchSize, passed on as MetricExportBatchSize: 0, which takes the same path.
OTel SDK v1.47.0 WithMaxExportBatchSize ignores sizes <= 0, and batching only runs when batcher.size > 0. SDK tests assert 0 and -1 give default behavior.

Tests: chainlink-common #2337 covers unset, zero and negative as unbatched; TestTelemetryConfig_MetricExportBatchSize covers nil and 0. Negative values behave like 0 and are not rejected.

Follow-up / limitations

  • Add the constrained local OTLP receiver integration coverage before marking this PR ready for review.
  • Do not set a production batch size without canary measurements; SDK batching is count-based (applied via the OTel SDK WithMaxExportBatchSize reader option in chainlink-common newPeriodicReader; see dependabump chainlink-common#2550) and the collector receive limit remains the independent guardrail.

@pkcll
pkcll force-pushed the otel-metric-export-batching branch from 5f4c6ed to 9aa41a9 Compare October 6, 2026 03:28
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

✅ No conflicts with other open PRs targeting develop

pkcll added a commit that referenced this pull request Oct 6, 2026
Enables node operators to turn on gRPC-level retries for the node's chip
ingress client (and LOOP plugins' own clients) via [Telemetry] TOML, wiring
the pass-through added by chainlink-common#2441 on top of the retry support
from chainlink-common#2373.

- [Telemetry] ChipIngressRetry* keys: Enabled (off by default) plus optional
  MaxAttempts / InitialBackoff / MaxBackoff / BackoffMultiplier /
  RetryableStatusCodes overrides; unset knobs fall back to
  chipingress.DefaultRetryPolicy values.
- core/cmd/shell.go newBeholderClient resolves them into
  beholder.Config.ChipIngressRetryPolicy for the node's shared chip ingress
  client (custom events + telemetry/heartbeat path).
- plugins/loop_registry.go forwards the resolved values to LOOP plugin
  subprocesses over the existing CL_CHIP_INGRESS_* env channel.
- Validation: MaxAttempts >= 2 (gRPC silently installs no retry policy below
  2), positive backoffs/multiplier; unknown status-code names fail node
  startup via chipingress.ParseStatusCodes.
- docs: core.toml keys, regenerated CONFIG.md, new docs/CHIP_INGRESS_RETRY.md
  operator page.

Follows the wiring pattern of #23491 (OTel metric export batching). Retries
remain opt-in: replayed publishes can duplicate events unless producers set
an idempotency key; the durable emitter path is intentionally excluded (its
retransmit loop already guarantees delivery).

Jira: INFOPLAT-19422
@cl-sonarqube-production

Copy link
Copy Markdown

@trunk-io

trunk-io Bot commented Oct 6, 2026

Copy link
Copy Markdown

Static Badge   Static Badge   Static Badge

Failed Test Failure Summary Logs
Test_workflowRegisteredHandler_confidentialRouting/confidential_workflow_module_is_hooked_correctly The test failed without a specific error message, likely due to an issue during workflow module creation or execution. Logs ↗︎
Test_workflowRegisteredHandler_confidentialRouting The test named 'Test_workflowRegisteredHandler_confidentialRouting' failed during execution. Logs ↗︎

View Full Report ↗︎ ⋅ Docs

pkcll added a commit that referenced this pull request Oct 6, 2026
Enables node operators to turn on gRPC-level retries for the node's chip
ingress client (and LOOP plugins' own clients) via [Telemetry] TOML, wiring
the pass-through added by chainlink-common#2441 on top of the retry support
from chainlink-common#2373.

- [Telemetry] ChipIngressRetry* keys: Enabled (off by default) plus optional
  MaxAttempts / InitialBackoff / MaxBackoff / BackoffMultiplier /
  RetryableStatusCodes overrides; unset knobs fall back to
  chipingress.DefaultRetryPolicy values.
- core/cmd/shell.go newBeholderClient resolves them into
  beholder.Config.ChipIngressRetryPolicy for the node's shared chip ingress
  client (custom events + telemetry/heartbeat path).
- plugins/loop_registry.go forwards the resolved values to LOOP plugin
  subprocesses over the existing CL_CHIP_INGRESS_* env channel.
- Validation: MaxAttempts >= 2 (gRPC silently installs no retry policy below
  2), positive backoffs/multiplier; unknown status-code names fail node
  startup via chipingress.ParseStatusCodes.
- docs: core.toml keys, regenerated CONFIG.md, new docs/CHIP_INGRESS_RETRY.md
  operator page.

Follows the wiring pattern of #23491 (OTel metric export batching). Retries
remain opt-in: replayed publishes can duplicate events unless producers set
an idempotency key; the durable emitter path is intentionally excluded (its
retransmit loop already guarantees delivery).

Jira: INFOPLAT-19422
pkcll added a commit that referenced this pull request Oct 6, 2026
Enables node operators to turn on gRPC-level retries for the node's chip
ingress client (and LOOP plugins' own clients) via [Telemetry] TOML, wiring
the pass-through added by chainlink-common#2441 on top of the retry support
from chainlink-common#2373.

- [Telemetry] ChipIngressRetry* keys: Enabled (off by default) plus optional
  MaxAttempts / InitialBackoff / MaxBackoff / BackoffMultiplier /
  RetryableStatusCodes overrides; unset knobs fall back to
  chipingress.DefaultRetryPolicy values.
- core/cmd/shell.go newBeholderClient resolves them into
  beholder.Config.ChipIngressRetryPolicy for the node's shared chip ingress
  client (custom events + telemetry/heartbeat path).
- plugins/loop_registry.go forwards the resolved values to LOOP plugin
  subprocesses over the existing CL_CHIP_INGRESS_* env channel.
- Validation: MaxAttempts >= 2 (gRPC silently installs no retry policy below
  2), positive backoffs/multiplier; unknown status-code names fail node
  startup via chipingress.ParseStatusCodes.
- docs: core.toml keys, regenerated CONFIG.md, new docs/CHIP_INGRESS_RETRY.md
  operator page.

Follows the wiring pattern of #23491 (OTel metric export batching). Retries
remain opt-in: replayed publishes can duplicate events unless producers set
an idempotency key; the durable emitter path is intentionally excluded (its
retransmit loop already guarantees delivery).

Jira: INFOPLAT-19422
@pkcll
pkcll force-pushed the otel-metric-export-batching branch from 9aa41a9 to 5e4d0a0 Compare November 17, 2026 16:52
@pkcll
pkcll marked this pull request as ready for review November 17, 2026 19:38
@pkcll
pkcll requested review from a team as code owners November 17, 2026 19:38
jmank88
jmank88 previously approved these changes Nov 17, 2026
kirqz23
kirqz23 previously approved these changes Nov 18, 2026
@kirqz23
kirqz23 dismissed stale reviews from jmank88 and themself via d56bdcf November 18, 2026 10:17
@pkcll
pkcll enabled auto-merge November 18, 2026 15:32
@pkcll
pkcll removed the request for review from pavel-raykov November 18, 2026 15:34
@pkcll
pkcll disabled auto-merge November 18, 2026 15:49
@pkcll
pkcll added this pull request to the merge queue Nov 18, 2026
Merged via the queue into develop with commit 07f40e0 Nov 18, 2026
239 of 240 checks passed
@pkcll
pkcll deleted the otel-metric-export-batching branch November 18, 2026 16:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants