Repository navigation
feat(llo): add per-stream observed value gauge (INFOPLAT-19823) - #24222
skippaDaBitFlippa wants to merge 3 commits into
Conversation
|
👋 skippaDaBitFlippa, thanks for creating this pull request! To help reviewers, please consider creating future PRs as drafts first. This allows you to self-review and make any final changes before notifying the team. Once you're ready, you can mark it as "Ready for review" to request feedback. Thanks! |
|
✅ No conflicts with other open PRs targeting |
pkcll
left a comment
There was a problem hiding this comment.
PR Review: #24222 — feat(llo): add per-stream observed value gauge (INFOPLAT-19823)
Summary
Adds llo_datasource_stream_observed_value{stream_id, value_type, config_digest}, recorded in dataSource.observe() after cache.UpdateStreamValues, production lifecycle stage only, with explicit series deletion (stale/nil value, stream removed, kind change, 10s digest TTL swept on record + ticker, Close). The implementation is small, well-scoped and the series lifecycle is correct as far as I could trace it. No blocking issues; a naming decision and some validation/operator-doc gaps below.
Jira Context
INFOPLAT-19823 (gauge), supports INFOPLAT-19824 (NOP bridge allowlist) and INFOPLAT-19827 (dashboard v2).
Files Reviewed
core/services/llo/observation/observed_value_gauge.go(new)core/services/llo/observation/data_source.gocore/services/llo/observation/observed_value_gauge_test.go(new),data_source_test.go.changeset/infoplat-19823-llo-observed-value.md- Context:
core/services/llo/delegate.go,observation/cache.go,chainlink-data-streams@v1.1.3llo/v30/plugin_observation.go,llo/dev/v31/blobpump.go
Verification
| Check | Result |
|---|---|
go build ./core/services/llo/... |
pass |
go vet ./core/services/llo/observation/ |
pass |
go test ./core/services/llo/observation/ -race -count=1 |
pass (skipped TestObservationContext_Observe_integrationRealPipeline, needs CL_DATABASE_URL, untouched by this PR) |
New tests, -race -count=20 |
pass |
BenchmarkObservedValueGaugesRecord (Apple Silicon, 3 runs) |
unchanged ~44.5 µs / 0 allocs; all_changed ~475 µs / 552 KB / 29,020 allocs |
| CI | all green |
Traced and confirmed:
- Blue/Green share one
dataSource(delegate.gopasses the samedsto both oracle factories) → per-digest series sets, production gating prevents duplicates from staging. delegate.Close()closes the oracles beforeds.Close(), so noObservecan re-create series afterdeleteAll().Cache.UpdateStreamValueshands back the stored pointer until the stream is re-written, so theentry.last != origpointer compare is valid;entry.lastholds a reference, so no address-reuse false positive.- The round-based sweep (
seen < len(dg.streams)), kind change (getOrCreateStream) and exact-label deletes are correct; noDeletePartialMatch.
Findings
Critical
None.
Major
None.
Minor
-
Staging validation doesn't cover the large-DON shape. The soak ran on
cl-mercury-hf-testnet(~190 streams). From prod metrics, HF DONs make ~94–188Observecalls/s with p50 cache-entry age ~25 ms, so most pointers are unchanged between calls andrecord()stays on the cheap path. The ~3.1k-stream v03 DONs (the shape the benchmark targets) make ~4–8 calls/s with p50 cache-entry age ~230 ms, so a large share of streams change between calls and cost trends towardall_changed. All prod DONs are v30, whereDataSource.Observeis inline inObservation(critical path). Meanocr3_reporting_plugin_duration{plugin="llo",function="observation"}on v03 DONs is ~7.8 ms (arb-mainnet) / ~23.8 ms (arb-sepolia), so I'd expect roughly +0.2–0.5 ms per Observe (~1–6%). That's acceptable, but the "+0.5%" result shouldn't be read as covering those DONs. Suggest either a short soak on a v03 node or stating the expected overhead in the PR. Optional: since the gauge is sampled every 15–60s, a per-digest minimum record interval (e.g. 1s) would make the hot-path cost negligible everywhere. -
value_type="benchmark"for every Decimal stream. Not every Decimal stream is a benchmark price (market status, NAV/reserves, timestamps, etc.). Since the name/labels are being locked in for the allowlist and dashboards, worth an explicit decision: keepbenchmark(and document it as "the stream's scalar value"), or use a distinct value such asvaluefor Decimal streams and reservebenchmarkfor the quote component. -
Operator instructions are incomplete.
[Telemetry.PrometheusBridge].Enableddefaults tofalse(marked ADVANCED), so adding the prefix alone does nothing. Also, settingPrefixesreplaces the default["go_"]rather than appending, so the INFOPLAT-19824 bundle should ship e.g.Prefixes = ["go_", "llo_datasource_stream_observed_value"](plus whatever is already forwarded). Worth stating in the PR description / changeset.
Nitpicks
- Label casing.
stream_idhere vsstreamIDon the otherllo_datasource_*metrics (stream_observation_error_count,cache_hit_entry_age_ms,stream_missing_count). snake_case is the right convention and needed forconfig_digest, but dashboards correlating these will needlabel_replace; worth a note for INFOPLAT-19827. requireinsiderequire.Eventually. testify runs the condition in a separate goroutine, sorequire.NoError(t, ds.Observe(...))/observedSeries(t, ...)callFailNowoff the test goroutine; a failure surfaces only as a 5s "Condition never satisfied".assert.EventuallyWithTwith*assert.CollectTavoids that. Same forobservedSeries(t, vec)insidewg.Goin the concurrent test.- Precision.
decimal.Float64()loses precision above 2^53 (e.g. nanosecond timestamps, large integer streams). Fine for monitoring; maybe mention in the Help text. - "Staging does not record" test could assert
Len(observedSeries(t, vec), 3)after the stagingObserveto make the "no extra digest" check explicit rather than relying on the loop over keys.
Questions
- Has the Beholder metrics backend budgeted this once NOPs enable it? ~3.3k series/node on v03 DONs × nodes per DON, and since
config_digestis part of the series identity, each Blue/Green rotation replaces the full set. Up to ~10s of overlap between the two digests at handover means dashboards should aggregate withmax by (stream_id)rather thansum/count. - When v31 rolls out,
DataSource.Observemoves into the blob pump (off the OCR critical path), so the observation-duration metric will no longer reflect this cost. Is there a pump cycle-duration metric to watch instead? (Not blocking — v30 everywhere today.)
Positive Notes
- Exact-label deletes and per-
dataSourceownership mean one job can never delete another's series. - The stall case (no
Observeat all) is handled by the ticker, not just byrecord(). - Pointer-compare skip gives a zero-alloc unchanged path; the benchmark backs the claim.
- Good test coverage of the lifecycle edges (nil, removal, kind change, TTL, concurrent Blue/Green,
Close). - Thorough staging validation write-up with reproducible queries.
Recommendation
Approve with minor changes. Settle the value_type for Decimal streams (Minor 2) since it gets locked in, and fix the operator instructions (Minor 3). The rest is optional.
- convert decimals without big.Rat (decimalToFloat64): all_changed record 900us -> 112us, 38k -> 3.2k allocs on realistic values - benchmark uses parsed-float and even-count-median coefficient shapes - Help text: benchmark is a decimal stream's value; float64 precision - tests: EventuallyWithT, no require off the test goroutine, staging test asserts 3 series - changeset: bridge Enabled and Prefixes replace semantics
|
@pkcll, I pushed 82f5c73 for your review. Below is the status of each item. Minor 1 (cost on v03 DONs): fixed.
Minor 2 (
Minor 3 (operator instructions): fixed.
Nitpicks: all done.
Questions
|
…T-19823) observedSeries uses require again (CollectT supports FailNow in EventuallyWithT conditions); the concurrent reader goroutine uses testutil.CollectAndCount, which makes no assertions.
Data Streams needs the value that each node observes for each stream. The node has this value, but it does not export it (INFOPLAT-19823).
What this PR adds
One gauge in
core/services/llo/observation:value_typeisbenchmarkfor a decimal stream. For a quote stream, it isbid,benchmarkorask.productionlifecycle stage records the gauge.config_digesthas the same value as the libocr label, soon(config_digest)joins the gauge with libocr metrics.When a series is deleted
Observefor 10 s. This also covers a DON stall.Thus a series never shows an old value as current.
Cost
Observeis on the OCR observation path. The benchmark uses the DON 4 shape (3,186 series) on an Apple M5 Max:ObserveThe code converts each decimal with
decimalToFloat64, not withdecimal.Float64(). In the same case,decimal.Float64()takes ~900 µs and makes ~38,000 allocs. The result is within 2 ulps ofdecimal.Float64().Operators
The gauge leaves the node only through the Prometheus bridge. Set both values:
Prefixesreplaces the default["go_"]. It does not add to it.Dashboards
max by (stream_id), notsumorcount.llo_datasource_*metrics use the labelstreamID. To join them with this gauge, uselabel_replace.For reviewers
Please approve the metric name and the label names. NOPs will put this name in their bridge config (INFOPLAT-19824), so a later change is expensive.
Requires
None.
Supports
INFOPLAT-19824 (NOP bridge config), INFOPLAT-19827 (dashboard v2).
Deployment Validation Steps
We tested this on staging on 2026-11-17. A backport of this PR on
v2.62.3-rc.3ran oncl-mercury-hf-testnetnode 0 for 80 minutes. Nodes 1–3 were the controls. Node 0 is now back on2.62.3-rc.3.benchmark, 2bid, 2askconfig_digeston the gauge, 2 onocr3_epochon(config_digest)withocr3_epochmatched all 196 seriesThe staging run used
decimal.Float64().Test_decimalToFloat64and the benchmark cover the new conversion. No config digest rotation occurred during the run, so unit tests are the proof for rotation and stall cleanup. Queries and full results are in INFOPLAT-19823.