Skip to content

feat(llo): add per-stream observed value gauge (INFOPLAT-19823) - #24222

Open
skippaDaBitFlippa wants to merge 3 commits into
developfrom
infoplat-19823-llo-observed-value-gauge
Open

skippaDaBitFlippa wants to merge 3 commits into
developfrom
infoplat-19823-llo-observed-value-gauge

Conversation

@skippaDaBitFlippa

@skippaDaBitFlippa skippaDaBitFlippa commented Nov 17, 2026 •

Copy link
Copy Markdown

Data Streams needs the value that each node observes for each stream. The node has this value, but it does not export it (INFOPLAT-19823).

What this PR adds

One gauge in core/services/llo/observation:

llo_datasource_stream_observed_value{stream_id, value_type, config_digest}
  • The gauge value is the latest value that the node gives to OCR for the stream. The price is the sample value, not a label.
  • value_type is benchmark for a decimal stream. For a quote stream, it is bid, benchmark or ask.
  • Only the OCR instance in the production lifecycle stage records the gauge.
  • config_digest has the same value as the libocr label, so on(config_digest) joins the gauge with libocr metrics.
  • Each node has one series per stream, plus two more series per quote stream. On DON 4, this is about 3,200 series.

When a series is deleted

  • The stream has no fresh value in the cache (about 1 s).
  • The stream leaves the channel definitions, or it changes between decimal and quote.
  • Its config digest does not call Observe for 10 s. This also covers a DON stall.
  • The job is deleted.

Thus a series never shows an old value as current.

Cost

Observe is on the OCR observation path. The benchmark uses the DON 4 shape (3,186 series) on an Apple M5 Max:

Case Time per Observe Allocs
No stream changed ~45 µs 0
All streams changed ~112 µs 3,186

The code converts each decimal with decimalToFloat64, not with decimal.Float64(). In the same case, decimal.Float64() takes ~900 µs and makes ~38,000 allocs. The result is within 2 ulps of decimal.Float64().

Operators

The gauge leaves the node only through the Prometheus bridge. Set both values:

[Telemetry.PrometheusBridge]
Enabled = true # default false
Prefixes = ["go_", "llo_datasource_stream_observed_value"] # also keep the prefixes that you send now

Prefixes replaces the default ["go_"]. It does not add to it.

Dashboards

  • At a Blue/Green handover, two digests can have series for up to ~15 s. Use max by (stream_id), not sum or count.
  • The other llo_datasource_* metrics use the label streamID. To join them with this gauge, use label_replace.

For reviewers

Please approve the metric name and the label names. NOPs will put this name in their bridge config (INFOPLAT-19824), so a later change is expensive.

Requires

None.

Supports

INFOPLAT-19824 (NOP bridge config), INFOPLAT-19827 (dashboard v2).

Deployment Validation Steps

We tested this on staging on 2026-11-17. A backport of this PR on v2.62.3-rc.3 ran on cl-mercury-hf-testnet node 0 for 80 minutes. Nodes 1–3 were the controls. Node 0 is now back on 2.62.3-rc.3.

Check Result
Gauge is emitted 196 series: 192 benchmark, 2 bid, 2 ask
Series count is stable 196 for the full hour, while 187 series changed value
Production instance only 1 config_digest on the gauge, 2 on ocr3_epoch
Join with libocr on(config_digest) with ocr3_epoch matched all 196 series
Observe latency Node 0 +0.5%, controls −1.5% to +2.9%
Memory and CPU No change against the controls

The staging run used decimal.Float64(). Test_decimalToFloat64 and the benchmark cover the new conversion. No config digest rotation occurred during the run, so unit tests are the proof for rotation and stall cleanup. Queries and full results are in INFOPLAT-19823.

@skippaDaBitFlippa
skippaDaBitFlippa requested review from a team as code owners November 17, 2026 19:17
@github-actions

Copy link
Copy Markdown
Contributor

👋 skippaDaBitFlippa, thanks for creating this pull request!

To help reviewers, please consider creating future PRs as drafts first. This allows you to self-review and make any final changes before notifying the team.

Once you're ready, you can mark it as "Ready for review" to request feedback. Thanks!

@github-actions

Copy link
Copy Markdown
Contributor

✅ No conflicts with other open PRs targeting develop

@pkcll pkcll left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review: #24222 — feat(llo): add per-stream observed value gauge (INFOPLAT-19823)

Summary

Adds llo_datasource_stream_observed_value{stream_id, value_type, config_digest}, recorded in dataSource.observe() after cache.UpdateStreamValues, production lifecycle stage only, with explicit series deletion (stale/nil value, stream removed, kind change, 10s digest TTL swept on record + ticker, Close). The implementation is small, well-scoped and the series lifecycle is correct as far as I could trace it. No blocking issues; a naming decision and some validation/operator-doc gaps below.

Jira Context

INFOPLAT-19823 (gauge), supports INFOPLAT-19824 (NOP bridge allowlist) and INFOPLAT-19827 (dashboard v2).

Files Reviewed

  • core/services/llo/observation/observed_value_gauge.go (new)
  • core/services/llo/observation/data_source.go
  • core/services/llo/observation/observed_value_gauge_test.go (new), data_source_test.go
  • .changeset/infoplat-19823-llo-observed-value.md
  • Context: core/services/llo/delegate.go, observation/cache.go, chainlink-data-streams@v1.1.3 llo/v30/plugin_observation.go, llo/dev/v31/blobpump.go

Verification

Check Result
go build ./core/services/llo/... pass
go vet ./core/services/llo/observation/ pass
go test ./core/services/llo/observation/ -race -count=1 pass (skipped TestObservationContext_Observe_integrationRealPipeline, needs CL_DATABASE_URL, untouched by this PR)
New tests, -race -count=20 pass
BenchmarkObservedValueGaugesRecord (Apple Silicon, 3 runs) unchanged ~44.5 µs / 0 allocs; all_changed ~475 µs / 552 KB / 29,020 allocs
CI all green

Traced and confirmed:

  • Blue/Green share one dataSource (delegate.go passes the same ds to both oracle factories) → per-digest series sets, production gating prevents duplicates from staging.
  • delegate.Close() closes the oracles before ds.Close(), so no Observe can re-create series after deleteAll().
  • Cache.UpdateStreamValues hands back the stored pointer until the stream is re-written, so the entry.last != orig pointer compare is valid; entry.last holds a reference, so no address-reuse false positive.
  • The round-based sweep (seen < len(dg.streams)), kind change (getOrCreateStream) and exact-label deletes are correct; no DeletePartialMatch.

Findings

Critical

None.

Major

None.

Minor

  1. Staging validation doesn't cover the large-DON shape. The soak ran on cl-mercury-hf-testnet (~190 streams). From prod metrics, HF DONs make ~94–188 Observe calls/s with p50 cache-entry age ~25 ms, so most pointers are unchanged between calls and record() stays on the cheap path. The ~3.1k-stream v03 DONs (the shape the benchmark targets) make ~4–8 calls/s with p50 cache-entry age ~230 ms, so a large share of streams change between calls and cost trends toward all_changed. All prod DONs are v30, where DataSource.Observe is inline in Observation (critical path). Mean ocr3_reporting_plugin_duration{plugin="llo",function="observation"} on v03 DONs is ~7.8 ms (arb-mainnet) / ~23.8 ms (arb-sepolia), so I'd expect roughly +0.2–0.5 ms per Observe (~1–6%). That's acceptable, but the "+0.5%" result shouldn't be read as covering those DONs. Suggest either a short soak on a v03 node or stating the expected overhead in the PR. Optional: since the gauge is sampled every 15–60s, a per-digest minimum record interval (e.g. 1s) would make the hot-path cost negligible everywhere.

  2. value_type="benchmark" for every Decimal stream. Not every Decimal stream is a benchmark price (market status, NAV/reserves, timestamps, etc.). Since the name/labels are being locked in for the allowlist and dashboards, worth an explicit decision: keep benchmark (and document it as "the stream's scalar value"), or use a distinct value such as value for Decimal streams and reserve benchmark for the quote component.

  3. Operator instructions are incomplete. [Telemetry.PrometheusBridge].Enabled defaults to false (marked ADVANCED), so adding the prefix alone does nothing. Also, setting Prefixes replaces the default ["go_"] rather than appending, so the INFOPLAT-19824 bundle should ship e.g. Prefixes = ["go_", "llo_datasource_stream_observed_value"] (plus whatever is already forwarded). Worth stating in the PR description / changeset.

Nitpicks

  1. Label casing. stream_id here vs streamID on the other llo_datasource_* metrics (stream_observation_error_count, cache_hit_entry_age_ms, stream_missing_count). snake_case is the right convention and needed for config_digest, but dashboards correlating these will need label_replace; worth a note for INFOPLAT-19827.
  2. require inside require.Eventually. testify runs the condition in a separate goroutine, so require.NoError(t, ds.Observe(...)) / observedSeries(t, ...) call FailNow off the test goroutine; a failure surfaces only as a 5s "Condition never satisfied". assert.EventuallyWithT with *assert.CollectT avoids that. Same for observedSeries(t, vec) inside wg.Go in the concurrent test.
  3. Precision. decimal.Float64() loses precision above 2^53 (e.g. nanosecond timestamps, large integer streams). Fine for monitoring; maybe mention in the Help text.
  4. "Staging does not record" test could assert Len(observedSeries(t, vec), 3) after the staging Observe to make the "no extra digest" check explicit rather than relying on the loop over keys.

Questions

  1. Has the Beholder metrics backend budgeted this once NOPs enable it? ~3.3k series/node on v03 DONs × nodes per DON, and since config_digest is part of the series identity, each Blue/Green rotation replaces the full set. Up to ~10s of overlap between the two digests at handover means dashboards should aggregate with max by (stream_id) rather than sum/count.
  2. When v31 rolls out, DataSource.Observe moves into the blob pump (off the OCR critical path), so the observation-duration metric will no longer reflect this cost. Is there a pump cycle-duration metric to watch instead? (Not blocking — v30 everywhere today.)

Positive Notes

  • Exact-label deletes and per-dataSource ownership mean one job can never delete another's series.
  • The stall case (no Observe at all) is handled by the ticker, not just by record().
  • Pointer-compare skip gives a zero-alloc unchanged path; the benchmark backs the claim.
  • Good test coverage of the lifecycle edges (nil, removal, kind change, TTL, concurrent Blue/Green, Close).
  • Thorough staging validation write-up with reproducible queries.

Recommendation

Approve with minor changes. Settle the value_type for Decimal streams (Minor 2) since it gets locked in, and fix the operator instructions (Minor 3). The rest is optional.

- convert decimals without big.Rat (decimalToFloat64): all_changed record 900us -> 112us, 38k -> 3.2k allocs on realistic values
- benchmark uses parsed-float and even-count-median coefficient shapes
- Help text: benchmark is a decimal stream's value; float64 precision
- tests: EventuallyWithT, no require off the test goroutine, staging test asserts 3 series
- changeset: bridge Enabled and Prefixes replace semantics
@skippaDaBitFlippa

skippaDaBitFlippa commented Nov 18, 2026 •

Copy link
Copy Markdown
Author

@pkcll, I pushed 82f5c73 for your review. Below is the status of each item.

Minor 1 (cost on v03 DONs): fixed.

  • I checked your prod numbers, and they are correct. v03 DONs make 4–8 Observe calls/s, and the mean observation is 7.8–24 ms.
  • The old benchmark used small integer values. With real pipeline values, the old cost was higher than your estimate: about 900 µs per record() when all streams change.
  • The new function decimalToFloat64 replaces decimal.Float64(). It converts the top 64 bits of the coefficient, then divides by an exact power of ten. The result is within 2 ulps of decimal.Float64().
  • When all streams change, record() now takes about 112 µs and makes 3,186 allocs. On v03 DONs, this is about 0.5–1.4% of the mean observation.
  • Test_decimalToFloat64 compares the function with decimal.Float64() on edge cases and on 20,000 random values.
  • I did not add the 1 s throttle. With the faster conversion, each call has a low cost, and the gauge value stays current.

Minor 2 (value_type="benchmark" for decimal streams): no change.

  • The INFOPLAT-19775 direction doc defines this label, and Data Streams approved the doc on 2026-11-16. The doc says that plain streams emit only benchmark.
  • The spread alert in that doc (DS_OCR_V03_020) queries value_type="benchmark" for all streams. A new value such as value would make every query use value_type=~"benchmark|value".
  • The Help text now says that for a decimal stream, benchmark is the value of the stream, whatever the stream measures.

Minor 3 (operator instructions): fixed.

  • The changeset and the PR body now say that operators must set Enabled = true.
  • They also say that Prefixes replaces the default ["go_"], so operators must keep "go_" and the prefixes that they already send. The PR body has a TOML example.
  • I will give the same information to INFOPLAT-19824.

Nitpicks: all done.

  1. Label case: the PR body tells dashboard authors to use label_replace for streamID.
  2. Both integration subtests now use require.EventuallyWithT, and their conditions pass the *assert.CollectT to the helpers. The reader goroutine of the concurrent test now calls testutil.CollectAndCount, which makes no assertions. Thus no test calls FailNow on a *testing.T outside the test goroutine.
  3. The Help text now states the float64 precision limit.
  4. The staging subtest now asserts exactly 3 series.

Questions

  1. Volume: the direction doc measured about 3,200 series per node on DON 4. That is about 3% of the series that a node of that DON already exports. I did not find a separate budget for the Beholder bridge path. The PR body now tells dashboard authors to use max by (stream_id), because two digests can have series for up to about 15 s at a handover.
  2. v31: I do not know a metric for the duration of the pump cycle yet. I will find one before nodes use v31.

…T-19823)

observedSeries uses require again (CollectT supports FailNow in EventuallyWithT conditions); the concurrent reader goroutine uses testutil.CollectAndCount, which makes no assertions.
@pkcll
pkcll requested a review from dav009 November 19, 2026 20:15
@pkcll
pkcll enabled auto-merge November 19, 2026 20:16
@pkcll
pkcll disabled auto-merge November 19, 2026 20:16

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants