Skip to content

fix(metrics): rpc_call_latency success label is inverted - #125

Draft
pkcll wants to merge 1 commit into
mainfrom
rpc-call-latency-fix-success-label
Draft

pkcll wants to merge 1 commit into
mainfrom
rpc-call-latency-fix-success-label

Conversation

@pkcll

@pkcll pkcll commented Nov 3, 2026

Copy link
Copy Markdown

Why

RecordRequest sets success = strconv.FormatBool(err != nil), so success="true" means the call failed. The doc comment says failures use success="false", and chainlink-evm's deprecated promEVMPoolRPCCallTiming uses err == nil.

Confirmed in prod VictoriaMetrics (10m rates): rpc_call_latency_milliseconds_count{success="true"} = 627/s vs evm_pool_rpc_node_calls_failed_total = 605/s; success="false" = 73.4k/s vs evm_pool_rpc_node_calls_success_total = 63.4k/s.

What

  • success = err == nil (Prometheus histogram and the Beholder OTel histogram share the value).
  • Regression test on the Prometheus histogram (fails without the fix).

Rollout notes

  • Mixed-fleet window: upgraded nodes report the corrected meaning, older nodes the inverted one.
  • Consumers hardcoding success="true" (e.g. Grafana DON-MultiNode-RPC-Health panel 503 "RPC p99 call latency") currently show failed-call latency and need updating alongside the node rollout.

Jira: INFOPLAT-19678 (series-reduction follow-up)

RecordRequest set success=FormatBool(err != nil), so success="true" meant the call
failed. Docs and the deprecated chainlink-evm timing metric use success=true for
successful calls. Add a regression test on the Prometheus histogram.
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

📊 API Diff Results

No changes detected for module github.com/smartcontractkit/chainlink-framework/metrics

View full report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant