Skip to content

Enable chip ingress gRPC retries via [Telemetry] TOML config - #23912

Draft
pkcll wants to merge 1 commit into
developfrom
incident-2673-pmai-chipingress-retry-config
Draft

pkcll wants to merge 1 commit into
developfrom
incident-2673-pmai-chipingress-retry-config

Conversation

@pkcll

@pkcll pkcll commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Wires the chip ingress gRPC retry policy (chainlink-common #2373, pass-through via chainlink-common #2441) into the node's [Telemetry] TOML config, following the pattern of #23491 (OTel metric export batching).

  • [Telemetry] ChipIngressRetry* keys: ChipIngressRetryEnabled (off by default) plus optional MaxAttempts, InitialBackoff, MaxBackoff, BackoffMultiplier, RetryableStatusCodes overrides. Unset knobs fall back to chipingress.DefaultRetryPolicy values (3 attempts, 100ms–1s backoff ×2, Unavailable + ResourceExhausted).
  • core/cmd/shell.go newBeholderClient resolves them into beholder.Config.ChipIngressRetryPolicy for the node's shared chip ingress client — covers both the custom-event emitter and the telemetry/heartbeat path (telemetry/manager.go reuses beholder.GetClient().Chip).
  • plugins/loop_registry.go forwards the resolved values to LOOP plugin subprocesses over the existing CL_CHIP_INGRESS_* env channel (CL_CHIP_INGRESS_RETRY_*).
  • Validation at config load: MaxAttempts ≥ 2 (gRPC silently installs no retry policy below 2), positive backoffs/multiplier; unknown status-code names fail startup via chipingress.ParseStatusCodes.
  • Docs: core.toml keys, regenerated CONFIG.md, new docs/CHIP_INGRESS_RETRY.md operator page (duplicate-Publish caveat, throttling pairing, env overrides, LOOP passthrough).
  • Bumps chainlink-common + nested pkg/chipingress pins to the Bump github.com/gin-contrib/cors from 1.3.0 to 1.3.1 #2441 branch head (91c6604e5); re-pins to its merge commit before ready-for-review. Note: subsumes the common bump in deps: bump chainlink-common to latest main #23910 once Bump github.com/gin-contrib/cors from 1.3.0 to 1.3.1 #2441 merges.

Retries stay opt-in: replayed publishes can duplicate events unless producers set an idempotency key; the durable emitter path is intentionally excluded (its retransmit loop already guarantees delivery). Incident context: INCIDENT-2673 (Beholder heartbeats not ingested) — without retries, every transient UNAVAILABLE/RESOURCE_EXHAUSTED chip publish is a permanent drop.

Validation

  • go test ./core/services/chainlink/ config suites (marshal + effective-config goldens + new getter tests, incl. defaults-mirror-DefaultRetryPolicy check) — green
  • go test ./plugins/ (LOOP env assertions extended to the retry fields) — green
  • go test ./core/config/docs/ (TOML↔CONFIG.md sync) — green
  • go build of all touched packages, gofmt clean on touched files
  • Dep diff is exactly the two chainlink-common pins; no transitive bumps required
  • CI

Jira: INFOPLAT-19422

Known CI note (stacked on chainlink-common#2441)

Validate go.mod dependencies is expected to fail while #2441 is unmerged: the go.mod pins point at the #2441 branch head (84edf0848) so this PR compiles against the retry pass-through, and that commit is by definition not on chainlink-common main yet. As soon as #2441 merges, the pins are re-pointed at its merge commit and the check goes green. All other checks are unaffected.

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

✅ No conflicts with other open PRs targeting develop

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

I see you updated files related to core. Please run make gocs in the root directory to add a changeset as well as in the text include at least one of the following tags:

  • #added For any new functionality added.
  • #breaking_change For any functionality that requires manual action for the node to boot.
  • #bugfix For bug fixes.
  • #changed For any change to the existing functionality.
  • #db_update For any feature that introduces updates to database schema.
  • #deprecation_notice For any upcoming deprecation functionality.
  • #internal For changesets that need to be excluded from the final changelog.
  • #nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
  • #removed For any functionality/config that is removed.
  • #updated For any functionality that is updated.
  • #wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

@pkcll
pkcll force-pushed the incident-2673-pmai-chipingress-retry-config branch from 8823d06 to da18bd8 Compare October 6, 2026 03:33
@trunk-io

trunk-io Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Static Badge   Static Badge   Static Badge

Failed Test Failure Summary Logs
Test_CRE_V2_ConfidentialWorkflows_Relay/Confidential_Workflows_Relay_-_workflow-gateway-capabilities-confidential-workflows The test failed during the setup of confidential workflows enclaves, likely due to an error or timeout in starting or responding from the enclave a... Logs ↗︎
Test_CRE_V2_ConfidentialWorkflows_Relay The test 'Test_CRE_V2_ConfidentialWorkflows_Relay' failed to run successfully. Logs ↗︎
Test_workflowRegisteredHandler_confidentialRouting/confidential_workflow_module_is_hooked_correctly The test failed without a specific error message, likely due to an issue during workflow module creation or initialization. Logs ↗︎
TestResolver_ConfigV2/empty The test failed without a specific error message, indicating an unspecified failure during test execution. Logs ↗︎

... and 17 more

View Full Report ↗︎ ⋅ Docs

@pkcll
pkcll force-pushed the incident-2673-pmai-chipingress-retry-config branch from da18bd8 to 3c21acb Compare October 6, 2026 03:54
@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

CORA - Pending Reviewers

Codeowners Entry Overall Num Files Owners
* ⏳ 14 @smartcontractkit/foundations, @smartcontractkit/core
/core/services/workflows/ ⏳ 1 @smartcontractkit/keystone
go.mod ⏳ 6 @smartcontractkit/core, @smartcontractkit/foundations
go.sum ⏳ 6 @smartcontractkit/core, @smartcontractkit/foundations
integration-tests/go.mod ⏳ 1 @smartcontractkit/core, @smartcontractkit/devex-tooling, @smartcontractkit/foundations
integration-tests/go.sum ⏳ 1 @smartcontractkit/core, @smartcontractkit/devex-tooling, @smartcontractkit/foundations
/docs/CONFIG.md ⏳ 1 @smartcontractkit/foundations, @smartcontractkit/core, @smartcontractkit/devrel

Legend: ✅ Approved | ❌ Changes Requested | 💬 Commented | 🚫 Dismissed | ⏳ Pending | ❓ Unknown

For more details, see the full review summary.

Enables node operators to turn on gRPC-level retries for the node's chip
ingress client (and LOOP plugins' own clients) via [Telemetry] TOML, wiring
the pass-through added by chainlink-common#2441 on top of the retry support
from chainlink-common#2373.

- [Telemetry] ChipIngressRetry* keys: Enabled (off by default) plus optional
  MaxAttempts / InitialBackoff / MaxBackoff / BackoffMultiplier /
  RetryableStatusCodes overrides; unset knobs fall back to
  chipingress.DefaultRetryPolicy values.
- core/cmd/shell.go newBeholderClient resolves them into
  beholder.Config.ChipIngressRetryPolicy for the node's shared chip ingress
  client (custom events + telemetry/heartbeat path).
- plugins/loop_registry.go forwards the resolved values to LOOP plugin
  subprocesses over the existing CL_CHIP_INGRESS_* env channel.
- Validation: MaxAttempts >= 2 (gRPC silently installs no retry policy below
  2), positive backoffs/multiplier; unknown status-code names fail node
  startup via chipingress.ParseStatusCodes.
- docs: core.toml keys, regenerated CONFIG.md, new docs/CHIP_INGRESS_RETRY.md
  operator page.

Follows the wiring pattern of #23491 (OTel metric export batching). Retries
remain opt-in: replayed publishes can duplicate events unless producers set
an idempotency key; the durable emitter path is intentionally excluded (its
retransmit loop already guarantees delivery).

Jira: INFOPLAT-19422
@pkcll
pkcll force-pushed the incident-2673-pmai-chipingress-retry-config branch from 3c21acb to 02f4dfe Compare October 6, 2026 03:59
@cl-sonarqube-production

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant