Repository navigation
[INFOPLAT-19706] Send node telemetry through Beholder domain emitters - #24098
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CORA - Analysis SkippedReason: The number of code owners (4) is less than the minimum required (5) and/or the number of CODEOWNERS entries with changed files (6) is less than the minimum required (2). |
|
✅ No conflicts with other open PRs targeting |
Picks up the domain emitter review fixes: connections closed once, rate-limited per-event failure logs, outage tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Count buffer-full and shutdown drops as dropped, not send errors - Also increment the Prometheus telemetry_client_messages_* counters - Meter provider option; precomputed per-route metric options - Log unsupported types (Warn) and unresolvable chains (Error) once each - Lower-case networkname; recordedtime from the manager clock - Supported types are the ones ingested in production; source/entity come from synchronization.TelemetryTypeToDomainAndEntity - Drop llo-attributed-observation; deterministic missing-domain error - Tests: real Beholder client end to end (no-ack, header, clean Close), failure injection, -race concurrency, chain families, multitype Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- newTelemetryManager picks DomainManager or the WSRPC Manager, and rejects a nil Beholder client before it becomes an interface value - New tests run in parallel; the cmd test starts and closes its client Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s no-ack Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ace) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The chipingress bump added PublishBatchNoAck to chipingress.Client; the new tests import otel log packages directly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ames - AppFactory.NewApplication takes the Shell's Beholder client explicitly instead of reading the global - Log the first refused/unbuildable event per route and error code at Error - Warn once for unsupported types on multitype endpoints; cache the option - Precompute per-endpoint partitionkey and chain selector strings - Rename metrics to core convention (telemetry_messages_*_total); note the Prometheus counter semantics next to the code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fter Close - Real-client tests point OTel at the local server (10s -> <1s) and close via t.Cleanup; per-event header lookup cannot panic - Buffer-full through a real Beholder client: dropped, never send errors; sent + dropped == total and sent == received after Close - Helper test covers the no-op client (Beholder off) and closes its client Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…etry types Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follows the upstream rename of beholder Config.ChipIngressDomainEmittersEnabled to ChipDomainEmittersEnabled (chip ingress is an implementation detail). Test Beholder configs set ChipIngressLogger, now required with domain emitters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tters, orgresolver registry field removal, OCR3 config digests); adapt newOrgResolver to the removed orgresolver fields
# Conflicts: # core/scripts/go.mod # core/scripts/go.sum # core/services/chainlink/application.go # core/services/cre/cre.go # deployment/go.mod # deployment/go.sum # go.mod # go.sum # integration-tests/go.mod # integration-tests/go.sum # integration-tests/load/go.mod # integration-tests/load/go.sum # system-tests/lib/go.mod # system-tests/lib/go.sum # system-tests/tests/go.mod # system-tests/tests/go.sum
pkcll
left a comment
There was a problem hiding this comment.
PR Review: chainlink#24098 — [INFOPLAT-19706] Send node telemetry through Beholder domain emitters
PR: #24098 (head cac894ae, base develop, author 4of9)
Scope: +1542/−68, 50 files (about 20 non-test, non-go.mod files). CI: 258 success, 8 skipped.
Method: read the full diff plus shell.go, shell_local.go, synchronization/common.go at the PR head. No local checkout, no tests run.
Summary
Adds [TelemetryIngress] BeholderEnabled (default false). When on, telemetry.DomainManager replaces the WSRPC Manager. It routes each telemetry type to a Beholder per-domain emitter (ForDomain → EmitEvent) as a CloudEvent with the legacy-mode OTI extensions. Config validation, wiring and metrics look sound. With the flag off, behaviour is unchanged.
Verified
- Event sources from
TelemetryTypeToDomainAndEntity(data-feeds.*,data-streams.*,ccip.*) match each route's domain, so the emitter's source guard in chainlink-common#2504 accepts them. - The Beholder client is started in
BeforeNode(shell_local.go:1245) before the app sends. Validation requiresTelemetry.Enabled, so the client is never the noop client when the flag is on.newTelemetryManageralso guards a nil client. - Validation rejects
BeholderEnabledtogether withChipIngressEnabled, and rejects it withoutTelemetry.EnabledandChipIngressEndpoint.
Findings
Critical / Major
None.
Minor
OCR3_1Mercuryis not routed.TelemetryTypeToDomainAndEntitymaps it todata-streams.telemetry.ocr3_1-mercury, butsupportedTelemetryDomains()omits it, so it is dropped and counted as unsupported. If OCR3.1 Mercury jobs run in production, flipping the flag silently loses their telemetry. Confirm it is intentionally excluded (as withLLOAttributedObservation), or add it toDomainDataStreams.- Merge gating on chainlink-common.
go.modpinschainlink-common v0.11.2-0.20261112155645-a3881feda2f8, which is the head of #2504. That PR is open and has a review thread that will changeClientlifecycle internals (closeOnce). The API used here (ForDomain,EmitEvent,Config.ChipDomainEmittersEnabled) should not change, but re-pin and re-run CI after #2504 merges. The PR notes already say the bump lands separately. This also brings thewasmtime-gov47→v50 major bump, so make sure that bump lands first. - Prometheus semantic mix under the same names.
telemetry_client_messages_*counts messages here but batches on the WSRPC path, with a differentendpointlabel. This is documented, but any alert or dashboard built on those counters changes meaning when the flag is on. Flag it in the rollout notes. - Rollout has no fallback.
PublishBatchNoAckmust be served by every domain's chip ingress (INFOPLAT-19548), plus the per-domain deployments and ALB header rules. Nothing in the node can detect that. A misconfigured env fails assend_errorsand drops, not at startup. The PR notes already say this. Add a runbook line or alert ontelemetry_messages_send_errors_totalbefore enabling.
Nitpicks
newEventsetstimeasnow.UTC()andrecordedtimeasnow(local). Use the same form for both.partitionKeyiscontractID + "-" + csaPubKeyHex. For endpoints with an emptycontractID, all of the node's events share one key and one partition. This matches OTI per the comment, so it is fine if intended.
Already raised by thomaska (agree; not repeated)
- Unbounded
unsupportedOptsandreportedgrowth, and unbounded label cardinality, from arbitrary unknown telemetry-type strings. This is the most substantive of the three; normalize unknown types to a bounded value. - Chain-unresolvable drops lose the
domainlabel. - Pass
WithMeterProvider(beholderClient.MeterProvider).
Questions
- Is
OCR3_1Mercuryout of scope (finding 1)? - Is
HeadReport → ccipsettled? The TODO references INFOPLAT-19709, and head reports are emitted for every chain on every node.
Positive Notes
- Flag is default off, with strict validation and a clear error for a missing client.
- Routes and metric options are precomputed per endpoint, so the hot path does no lookups.
- Refusals are logged once per route and error code, and every fate is counted in both OTel and Prometheus.
- The event builder documents the OTI header each extension replaces.
Recommendation
Approve with minor changes. Resolve finding 1 and thomaska's cardinality comment before merge, and re-pin chainlink-common after #2504 lands.
…emetry-domain-manager
telemetry_messages_{sent,send_errors,dropped}_total now carry the target
chain's chain_selector, so volume and drops can be attributed to a chain
on multi-chain DONs. Metric options are cached per (selector, telemetry
type, error code); unsupported-type/chain counts keep their (type, reason)
attributes since no selector resolves for them.
|
Correction to my review above: Minor finding 2 ("Merge gating on chainlink-common") is wrong. chainlink-common#2504 merged on 2026-11-11, and the The |
The (selector, telemetry type, error code) cache key was a NUL-joined string; a small struct is self-documenting and collision-proof. No behaviour change.
…omain on unsupported drops
# Conflicts: # go.mod
|
Thanks @pkcll, follow-up on the review:
thomaska's three threads are addressed and resolved. |
pkcll
left a comment
There was a problem hiding this comment.
Round 2 — 2026-11-18 (head 7035964c, since a2d24a13)
CI: 254 success, 8 skipped. Merge state BLOCKED (review required).
Addressed (verified in code)
- thomaska unknown-type cardinality:
unsupportedTypeLabelclamps to registry types or"unknown"; cache key is the bounded struct;reportOncecapped at 25 keys. Resolved. - thomaska domain label:
countUnsupportedalways setsdomain(route's domain, else"unknown"), looked up per message, so multitype endpoints are covered. Resolved. - thomaska meter provider:
newTelemetryManagerpassesWithMeterProvideronly when non-nil; test added. Resolved. - jmank88 lifecycle context:
sendandcountUnsupportednow usem.eng.NewCtx(). VerifiedEmitEventonly checksctx.Err()synchronously and the async callbacks only use it for OTel adds, so cancelling on return is safe. Resolved. - Mine:
OCR3_1Mercuryrouted to data-streams (445f7b7), docs list updated, test case added.recordedtimenow UTC (8f734c8). chainlink-common/chipingress now come from develop, no pin. Prometheus, rollout, partitionKey and HeadReport answered by the author.
Still open
- jmank88: "Do we need another interface?" on
telemetry_manager.go:19. thomaska showed it is used as bothServiceandMonitoringEndpointGenerator; jmank88 asked to promote it totelemetry.Manager. No author response yet.
New findings in the new commits
- Minor — the 25-key
reportOncecap is shared by refusals.refused:<type>/<code>,type:<t>andchain:<net>/<id>all draw from one budget. Refusal keys are bounded already (routes × 2 codes), but once plugin-supplied type/chain keys fill the cap, the first "Telemetry refused" Error log (e.g. emitter not started, chip ingress broken) is suppressed. Cap only the sender-influenced categories (types, chains), or give each category its own budget. - Minor —
NewCtxper message spawns a goroutine.StopRChan.CtxCancel(non-race build) starts a goroutine per call;sendandcountUnsupportedcall it per telemetry message, on the hot path (LLO/OCR). Short-lived, but avoidable: create one service-scoped ctx in aStarthook and cancel it inClose, which still ties sends to the lifecycle as jmank88 asked. - Nit — unknown types allocate per message.
unsupportedTypeLabelcallsTelemetryTypeToDomainAndEntity, which builds afmt.Errorffor every unknown type. A static set of registry types would avoid it (the map comment says this path "must not allocate"). - Nit — past the cap, every new-key call does LoadOrStore + Delete. For an unknown type on the multitype path that is per message. A
Load+ cap check before storing avoids the churn.
Move the Service + MonitoringEndpointGenerator interface from core/services/chainlink into the telemetry package as Manager, and rename the WSRPC struct it displaced to WSRPCManager (NewManager -> NewWSRPCManager). The service name stays "TelemetryManager", so logs and health checks are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
send and countUnsupported called m.eng.NewCtx() per message, which starts a goroutine each time on the telemetry hot path. Create one context in NewDomainManager and cancel it in the service's Close hook instead; sends are still tied to the manager's lifecycle and are refused after Close. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks @pkcll. Item 2 is fixed in 2a9bd9f: the manager now creates one context and cancels it on Close, so there's no goroutine per message, and sends after Close are still refused. Items 1 and 4 need more distinct types/chains than a node ever sees, so we're leaving them. Item 3 does fire for |
libocr's MonitoringEndpoint.SendLog takes no context, so the manager keeps one cancelled on Close; same exception as ocr3_1/beholderwrapper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
INFOPLAT-19706
What
telemetry.DomainManager: implementsMonitoringEndpointGenerator(job code unchanged) and sends telemetry through Beholder's per-domain emitters (ForDomain→EmitEvent)TelemetryIngress.Endpointsare ignored; an unresolvable chain is logged at Error once and countedtelemetry_client_messages_*counters are incremented too (endpoint="beholder/<domain>")[TelemetryIngress] BeholderEnabled(default false): selectsDomainManagerand turns on Beholder's domain emitters (beholder.Config.ChipDomainEmittersEnabled); requiresTelemetry.Enabled+Telemetry.ChipIngressEndpoint, not combinable withChipIngressEnabledNewApplicationviaApplicationOptstelemetry.Manageris now an interface (services.Service+MonitoringEndpointGenerator) implemented byDomainManagerand the existing WSRPC manager, renamedtelemetry.WSRPCManagerdevelop(chainlink-common51f4bebad41dvia chainlink#24213); this PR pins neither. Its only go.mod change makesotlploggrpcandotel/sdk/logdirect requirements, because the new tests use them for a no-op Beholder log exporter (test-only, same versions as develop)Why
Notes
BeholderEnabledanywhere: every domain's chip ingress must servePublishBatchNoAck(INFOPLAT-19548; there is no fallback), plus the per-domain deployments (INFOPLAT-19697) and ALBchainlink-domainheader rules.TelemetryIngresssending settings are ignored:Telemetry.ChipIngress*sizes one buffer per domain, shared by that domain's jobs.🤖 Generated with Claude Code