Skip to content

[INFOPLAT-19706] Send node telemetry through Beholder domain emitters - #24098

Merged
4of9 merged 28 commits into
developfrom
INFOPLAT-19706-telemetry-domain-manager
Nov 18, 2026
Merged

4of9 merged 28 commits into
developfrom
INFOPLAT-19706-telemetry-domain-manager

Conversation

@4of9

@4of9 4of9 commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

INFOPLAT-19706

What

  • New telemetry.DomainManager: implements MonitoringEndpointGenerator (job code unchanged) and sends telemetry through Beholder's per-domain emitters (ForDomain → EmitEvent)
    • type → domain routing (data-feeds, data-streams, ccip) for the telemetry types ingested in production today; other types (functions, llo-report, llo-attributed-observation, S4, threshold, rebalancer, ...) are dropped and counted
    • no per-network config: any chain resolvable by chain-selectors sends (EVM and non-EVM); TelemetryIngress.Endpoints are ignored; an unresolvable chain is logged at Error once and counted
    • CloudEvents carry the OTI-parity extensions legacy-mode chip ingress requires
    • metrics: OTel (Beholder) sent / send errors / dropped (buffer full, shutdown) / unsupported, by domain + telemetry type; the existing Prometheus telemetry_client_messages_* counters are incremented too (endpoint="beholder/<domain>")
  • New [TelemetryIngress] BeholderEnabled (default false): selects DomainManager and turns on Beholder's domain emitters (beholder.Config.ChipDomainEmittersEnabled); requires Telemetry.Enabled + Telemetry.ChipIngressEndpoint, not combinable with ChipIngressEnabled
  • Beholder client passed to NewApplication via ApplicationOpts
  • telemetry.Manager is now an interface (services.Service + MonitoringEndpointGenerator) implemented by DomainManager and the existing WSRPC manager, renamed telemetry.WSRPCManager
  • chainlink-common (incl. chainlink-common#2504 per-domain emitters) and chipingress (no-ack) come from develop (chainlink-common 51f4bebad41d via chainlink#24213); this PR pins neither. Its only go.mod change makes otlploggrpc and otel/sdk/log direct requirements, because the new tests use them for a no-op Beholder log exporter (test-only, same versions as develop)

Why

  • Moves node telemetry onto Beholder with per-domain routing (INFOPLAT-19704 direction doc), replacing a dedicated chip-ingress client
  • Opt-in: with the flag off the node is unchanged, and the WSRPC manager stays the fallback

Notes

  • Before enabling BeholderEnabled anywhere: every domain's chip ingress must serve PublishBatchNoAck (INFOPLAT-19548; there is no fallback), plus the per-domain deployments (INFOPLAT-19697) and ALB chainlink-domain header rules.
  • With the flag on, TelemetryIngress sending settings are ignored: Telemetry.ChipIngress* sizes one buffer per domain, shared by that domain's jobs.
  • Routing table, event builder and chain-selector lookup are ported from [INFOPLAT-19425] Send legacy telemetry via a dedicated chip-ingress batch client #23949 (thanks @skippaDaBitFlippa)
  • head-report → ccip pending INFOPLAT-19709

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

CORA - Analysis Skipped

Reason: The number of code owners (4) is less than the minimum required (5) and/or the number of CODEOWNERS entries with changed files (6) is less than the minimum required (2).

@github-actions

Copy link
Copy Markdown
Contributor

✅ No conflicts with other open PRs targeting develop

@trunk-io

trunk-io Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Static Badge   Static Badge   Static Badge

View Full Report ↗︎ ⋅ Docs

4of9 and others added 11 commits October 28, 2026 15:37
Picks up the domain emitter review fixes: connections closed once,
rate-limited per-event failure logs, outage tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Count buffer-full and shutdown drops as dropped, not send errors
- Also increment the Prometheus telemetry_client_messages_* counters
- Meter provider option; precomputed per-route metric options
- Log unsupported types (Warn) and unresolvable chains (Error) once each
- Lower-case networkname; recordedtime from the manager clock
- Supported types are the ones ingested in production; source/entity
  come from synchronization.TelemetryTypeToDomainAndEntity
- Drop llo-attributed-observation; deterministic missing-domain error
- Tests: real Beholder client end to end (no-ack, header, clean Close),
  failure injection, -race concurrency, chain families, multitype

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- newTelemetryManager picks DomainManager or the WSRPC Manager, and
  rejects a nil Beholder client before it becomes an interface value
- New tests run in parallel; the cmd test starts and closes its client

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s no-ack

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ace)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The chipingress bump added PublishBatchNoAck to chipingress.Client; the new
tests import otel log packages directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ames

- AppFactory.NewApplication takes the Shell's Beholder client explicitly
  instead of reading the global
- Log the first refused/unbuildable event per route and error code at Error
- Warn once for unsupported types on multitype endpoints; cache the option
- Precompute per-endpoint partitionkey and chain selector strings
- Rename metrics to core convention (telemetry_messages_*_total); note the
  Prometheus counter semantics next to the code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fter Close

- Real-client tests point OTel at the local server (10s -> <1s) and close via
  t.Cleanup; per-event header lookup cannot panic
- Buffer-full through a real Beholder client: dropped, never send errors;
  sent + dropped == total and sent == received after Close
- Helper test covers the no-op client (Beholder off) and closes its client

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…etry types

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follows the upstream rename of beholder Config.ChipIngressDomainEmittersEnabled
to ChipDomainEmittersEnabled (chip ingress is an implementation detail). Test
Beholder configs set ChipIngressLogger, now required with domain emitters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tters, orgresolver registry field removal, OCR3 config digests); adapt newOrgResolver to the removed orgresolver fields
@thomaska thomaska added the build-publish Build and Publish image to SDLC label Nov 13, 2026
# Conflicts:
#	core/scripts/go.mod
#	core/scripts/go.sum
#	core/services/chainlink/application.go
#	core/services/cre/cre.go
#	deployment/go.mod
#	deployment/go.sum
#	go.mod
#	go.sum
#	integration-tests/go.mod
#	integration-tests/go.sum
#	integration-tests/load/go.mod
#	integration-tests/load/go.sum
#	system-tests/lib/go.mod
#	system-tests/lib/go.sum
#	system-tests/tests/go.mod
#	system-tests/tests/go.sum

@pkcll pkcll left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review: chainlink#24098 — [INFOPLAT-19706] Send node telemetry through Beholder domain emitters

PR: #24098 (head cac894ae, base develop, author 4of9)
Scope: +1542/−68, 50 files (about 20 non-test, non-go.mod files). CI: 258 success, 8 skipped.
Method: read the full diff plus shell.go, shell_local.go, synchronization/common.go at the PR head. No local checkout, no tests run.

Summary

Adds [TelemetryIngress] BeholderEnabled (default false). When on, telemetry.DomainManager replaces the WSRPC Manager. It routes each telemetry type to a Beholder per-domain emitter (ForDomain → EmitEvent) as a CloudEvent with the legacy-mode OTI extensions. Config validation, wiring and metrics look sound. With the flag off, behaviour is unchanged.

Verified

  • Event sources from TelemetryTypeToDomainAndEntity (data-feeds.*, data-streams.*, ccip.*) match each route's domain, so the emitter's source guard in chainlink-common#2504 accepts them.
  • The Beholder client is started in BeforeNode (shell_local.go:1245) before the app sends. Validation requires Telemetry.Enabled, so the client is never the noop client when the flag is on. newTelemetryManager also guards a nil client.
  • Validation rejects BeholderEnabled together with ChipIngressEnabled, and rejects it without Telemetry.Enabled and ChipIngressEndpoint.

Findings

Critical / Major

None.

Minor

  1. OCR3_1Mercury is not routed. TelemetryTypeToDomainAndEntity maps it to data-streams.telemetry.ocr3_1-mercury, but supportedTelemetryDomains() omits it, so it is dropped and counted as unsupported. If OCR3.1 Mercury jobs run in production, flipping the flag silently loses their telemetry. Confirm it is intentionally excluded (as with LLOAttributedObservation), or add it to DomainDataStreams.
  2. Merge gating on chainlink-common. go.mod pins chainlink-common v0.11.2-0.20261112155645-a3881feda2f8, which is the head of #2504. That PR is open and has a review thread that will change Client lifecycle internals (closeOnce). The API used here (ForDomain, EmitEvent, Config.ChipDomainEmittersEnabled) should not change, but re-pin and re-run CI after #2504 merges. The PR notes already say the bump lands separately. This also brings the wasmtime-go v47→v50 major bump, so make sure that bump lands first.
  3. Prometheus semantic mix under the same names. telemetry_client_messages_* counts messages here but batches on the WSRPC path, with a different endpoint label. This is documented, but any alert or dashboard built on those counters changes meaning when the flag is on. Flag it in the rollout notes.
  4. Rollout has no fallback. PublishBatchNoAck must be served by every domain's chip ingress (INFOPLAT-19548), plus the per-domain deployments and ALB header rules. Nothing in the node can detect that. A misconfigured env fails as send_errors and drops, not at startup. The PR notes already say this. Add a runbook line or alert on telemetry_messages_send_errors_total before enabling.

Nitpicks

  • newEvent sets time as now.UTC() and recordedtime as now (local). Use the same form for both.
  • partitionKey is contractID + "-" + csaPubKeyHex. For endpoints with an empty contractID, all of the node's events share one key and one partition. This matches OTI per the comment, so it is fine if intended.

Already raised by thomaska (agree; not repeated)

  • Unbounded unsupportedOpts and reported growth, and unbounded label cardinality, from arbitrary unknown telemetry-type strings. This is the most substantive of the three; normalize unknown types to a bounded value.
  • Chain-unresolvable drops lose the domain label.
  • Pass WithMeterProvider(beholderClient.MeterProvider).

Questions

  • Is OCR3_1Mercury out of scope (finding 1)?
  • Is HeadReport → ccip settled? The TODO references INFOPLAT-19709, and head reports are emitted for every chain on every node.

Positive Notes

  • Flag is default off, with strict validation and a clear error for a missing client.
  • Routes and metric options are precomputed per endpoint, so the hot path does no lookups.
  • Refusals are logged once per route and error code, and every fate is counted in both OTel and Prometheus.
  • The event builder documents the OTI header each extension replaces.

Recommendation

Approve with minor changes. Resolve finding 1 and thomaska's cardinality comment before merge, and re-pin chainlink-common after #2504 lands.

4of9 added 2 commits November 16, 2026 15:29
telemetry_messages_{sent,send_errors,dropped}_total now carry the target
chain's chain_selector, so volume and drops can be attributed to a chain
on multi-chain DONs. Metric options are cached per (selector, telemetry
type, error code); unsupported-type/chain counts keep their (type, reason)
attributes since no selector resolves for them.
@pkcll

pkcll commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Correction to my review above: Minor finding 2 ("Merge gating on chainlink-common") is wrong. chainlink-common#2504 merged on 2026-11-11, and the go.mod pin (a3881feda2f8) is a main commit from after that merge, so no re-pin is needed. I had taken "pinned to its head" from the PR description without checking the PR's state.

The wasmtime-go v47 to v50 major bump from the dependency update still applies and should land separately first, as the PR notes say. My other findings are unchanged.

4of9 and others added 4 commits November 16, 2026 15:54
The (selector, telemetry type, error code) cache key was a NUL-joined
string; a small struct is self-documenting and collision-proof. No
behaviour change.
@4of9

4of9 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks @pkcll, follow-up on the review:

  • OCR3_1Mercury: routed to data-streams in 445f7b7. It's a new type from llo: route OCR3.1 protocol telemetry as ocr3_1-mercury #24197 (LLO v31 oracles now send ocr3_1-mercury), and OTI ingestion for it is in progress.
  • chainlink-common pin / wasmtime: moot now. This PR takes chainlink-common and chipingress from develop (Update cron to the new PSB #24213) and pins neither; develop was already on wasmtime v50.
  • Prometheus counters: these are node-local only (our view is the OTel counters through Beholder). With the flag on, the existing telemetry_client_messages_* series carry endpoint="beholder/<domain>" and count messages, so they stay separate from the WSRPC series. That's documented with the metric definitions in domain_manager.go.
  • No fallback: handled by ordering. The chip ingress side (INFOPLAT-19548 no-ack, INFOPLAT-19697 per-domain deployments) lands before this release reaches NOPs, and BeholderEnabled is off by default.
  • recordedtime: now UTC like time (8f734c8).
  • partitionKey with an empty contractID: intended, it matches OTI.
  • HeadReport → ccip: still open in INFOPLAT-19709. Nothing changes while the flag is off.

thomaska's three threads are addressed and resolved.

@4of9
4of9 marked this pull request as ready for review November 17, 2026 21:43
@4of9
4of9 requested review from a team as code owners November 17, 2026 21:43
pkcll
pkcll previously approved these changes Nov 18, 2026

@pkcll pkcll left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 2 — 2026-11-18 (head 7035964c, since a2d24a13)

CI: 254 success, 8 skipped. Merge state BLOCKED (review required).

Addressed (verified in code)

  • thomaska unknown-type cardinality: unsupportedTypeLabel clamps to registry types or "unknown"; cache key is the bounded struct; reportOnce capped at 25 keys. Resolved.
  • thomaska domain label: countUnsupported always sets domain (route's domain, else "unknown"), looked up per message, so multitype endpoints are covered. Resolved.
  • thomaska meter provider: newTelemetryManager passes WithMeterProvider only when non-nil; test added. Resolved.
  • jmank88 lifecycle context: send and countUnsupported now use m.eng.NewCtx(). Verified EmitEvent only checks ctx.Err() synchronously and the async callbacks only use it for OTel adds, so cancelling on return is safe. Resolved.
  • Mine: OCR3_1Mercury routed to data-streams (445f7b7), docs list updated, test case added. recordedtime now UTC (8f734c8). chainlink-common/chipingress now come from develop, no pin. Prometheus, rollout, partitionKey and HeadReport answered by the author.

Still open

  • jmank88: "Do we need another interface?" on telemetry_manager.go:19. thomaska showed it is used as both Service and MonitoringEndpointGenerator; jmank88 asked to promote it to telemetry.Manager. No author response yet.

New findings in the new commits

  1. Minor — the 25-key reportOnce cap is shared by refusals. refused:<type>/<code>, type:<t> and chain:<net>/<id> all draw from one budget. Refusal keys are bounded already (routes × 2 codes), but once plugin-supplied type/chain keys fill the cap, the first "Telemetry refused" Error log (e.g. emitter not started, chip ingress broken) is suppressed. Cap only the sender-influenced categories (types, chains), or give each category its own budget.
  2. Minor — NewCtx per message spawns a goroutine. StopRChan.CtxCancel (non-race build) starts a goroutine per call; send and countUnsupported call it per telemetry message, on the hot path (LLO/OCR). Short-lived, but avoidable: create one service-scoped ctx in a Start hook and cancel it in Close, which still ties sends to the lifecycle as jmank88 asked.
  3. Nit — unknown types allocate per message. unsupportedTypeLabel calls TelemetryTypeToDomainAndEntity, which builds a fmt.Errorf for every unknown type. A static set of registry types would avoid it (the map comment says this path "must not allocate").
  4. Nit — past the cap, every new-key call does LoadOrStore + Delete. For an unknown type on the multitype path that is per message. A Load + cap check before storing avoids the churn.

Move the Service + MonitoringEndpointGenerator interface from
core/services/chainlink into the telemetry package as Manager, and rename
the WSRPC struct it displaced to WSRPCManager (NewManager ->
NewWSRPCManager). The service name stays "TelemetryManager", so logs and
health checks are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
send and countUnsupported called m.eng.NewCtx() per message, which starts a
goroutine each time on the telemetry hot path. Create one context in
NewDomainManager and cancel it in the service's Close hook instead; sends
are still tied to the manager's lifecycle and are refused after Close.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@4of9

4of9 commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @pkcll. Item 2 is fixed in 2a9bd9f: the manager now creates one context and cancels it on Close, so there's no goroutine per message, and sends after Close are still refused. Items 1 and 4 need more distinct types/chains than a node ever sees, so we're leaving them. Item 3 does fire for llo-report on LLO nodes, but it's one small allocation per already-dropped message, so we're leaving it too.

libocr's MonitoringEndpoint.SendLog takes no context, so the manager keeps
one cancelled on Close; same exception as ocr3_1/beholderwrapper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@4of9
4of9 added this pull request to the merge queue Nov 18, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 18, 2026
@4of9
4of9 added this pull request to the merge queue Nov 18, 2026
Merged via the queue into develop with commit 3c4a720 Nov 18, 2026
262 checks passed
@4of9
4of9 deleted the INFOPLAT-19706-telemetry-domain-manager branch November 18, 2026 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build-publish Build and Publish image to SDLC

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants