Add Prometheus metrics to readiness condition reporter by rawadhossain · Pull Request #379 · kubernetes-sigs/node-readiness-controller

rawadhossain · 2026-08-09T10:59:27Z

Description

Implements the Reporter metrics specified in the observability design

node_readiness_reporter_build_info
node_readiness_reporter_check_duration_seconds
node_readiness_reporter_checks_total
node_readiness_reporter_condition_writes_total

Also adds the Prometheus /metrics and /healthz endpoints to the readiness condition reporter.

Implementation choices

Single HTTP server for both /metrics and /healthz to keep the Reporter to one listener and port.
Dedicated Prometheus registry to keep the Reporter's metrics isolated and avoid registration conflicts in tests, following the same general pattern used by the controller metrics.
Port :9445 as the default metrics port because the Reporter runs with hostNetwork: true. The address is configurable through METRICS_BIND_ADDRESS.

Build fix

Updated Dockerfile.reporter to build the Reporter package instead of single file main.go, since the new metrics.go also needs to be included.

Fixes #380

Type of Change

/kind feature

Testing

Verified metrics scraping with a live kind cluster and Prometheus.
Verified graceful SIGTERM shutdown.
Verified the Reporter continues its normal check/condition-update loop when the metrics port is unavailable.

Checklist

make test passes
make lint passes

netlify · 2026-08-09T10:59:33Z

✅ Deploy Preview for node-readiness-controller ready!

Name	Link
🔨 Latest commit	`2de0068`
🔍 Latest deploy log	https://app.netlify.com/projects/node-readiness-controller/deploys/6a84326f1f7e2800082aff28
😎 Deploy Preview	https://deploy-preview-379--node-readiness-controller.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

kubernetes-prow · 2026-08-09T10:59:38Z

Hi @rawadhossain. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ajaysundark · 2026-08-09T16:15:26Z

/cc @AvineshTripathi @ajaysundark

ajaysundark · 2026-08-10T02:38:27Z

/ok-to-test

AvineshTripathi

Thank you for starting this PR! I have left some comments

AvineshTripathi · 2026-08-10T19:58:11Z

+	// reporterChecksTotal tracks total probe check results over time.
+	reporterChecksTotal = prometheus.NewCounterVec(
+		prometheus.CounterOpts{
+			Name: "node_readiness_reporter_checks_total",


node_readiness_reporter_checks instead

but I see in #344 it says node_readiness_reporter_checks_total

Yeah, was confused about this too, so kept it as in the design doc.

I followed https://prometheus.io/docs/practices/naming/#metric-names. The metrics convention suggests that an accumulating count has total as a suffix. Also on a similar note, https://www.robustperception.io/on-the-naming-of-things/ was a good read on why our node_readiness_rules_total (a gauge) metric could drop the suffix as can cause to confusions like this.

AvineshTripathi · 2026-08-10T20:04:45Z

+	mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) {
+		w.WriteHeader(http.StatusOK)
+	})
+	metricsServer := &http.Server{


why not using "sigs.k8s.io/controller-runtime/pkg/metrics/server"? We already have it as part of go.mod

Yeah, I looked into this one. It can run standalone without a full Manager setup, but I kept the current approach for a few reasons:

We use a dedicated registry for the Reporter, so the /metrics endpoint only exposes the metrics we explicitly register.

The current implementation gives us explicit control over the metrics server shutdown and keeps the existing klog setup.

Using metrics/server doesn't really make the implementation simpler once the registry and logging setup are included.

I also tested the current setup with a real SIGTERM against a live pod, and it shuts down cleanly in ~0.3s. So I kept the current implementation since it gives us the behavior we wanted and is already validated. Im okay to switch to metrics/server if you think that's preferable here.

Signed-off-by: Rawad Hossain <rawad.hossain00@gmail.com>

ajaysundark · 2026-08-14T02:37:45Z

+	// reporterBuildInfo tracks the reporter binary version to track fleet version skew.
+	reporterBuildInfo = prometheus.NewGaugeVec(
+		prometheus.GaugeOpts{
+			Name: "node_readiness_reporter_build_info",


@rawadhossain just reviewed #406. Don't we need the ldFlags on this one?

Yeah, we need it here too. Was planning to get #406 merged first and then follow up here. I’ll push the changes once #406 lands.

@ajaysundark done with the idFlags changes now that #406 is merged. PTAL.

Signed-off-by: Rawad Hossain <rawad.hossain00@gmail.com>

AvineshTripathi · 2026-08-17T19:31:02Z

/lgtm

I think this is all good now. @ajaysundark can you take a final look

ajaysundark · 2026-08-18T06:45:19Z

+		prometheus.HistogramOpts{
+			Name:    "node_readiness_reporter_check_duration_seconds",
+			Help:    "Duration of health probe checks.",
+			Buckets: prometheus.DefBuckets,


An AI review caught this as excessive, so I looked it up - https://pkg.go.dev/github.com/prometheus/client_golang/prometheus#pkg-variables. The default bucket seems to be 11 buckets from 0.005 granularity to 10s.

I wonder whether these many milliseconds accuracy is needed. May be we could keep it fewer to something like {0.1, 0.5, 1, 5, 10 }?

Looked into this and I agree that the default buckets are more than we need here. Tested it by running the reporter against real traffic + some deliberate slow responses to see which buckets actually mattered.

So I ended up with {0.005, 0.1, 0.25, 0.5, 1, 2.5, 5, 10}. Kept a few more buckets since they were catching useful delays in testing.

Committed the changes. PTAL.

ajaysundark · 2026-08-18T06:50:47Z

/lgtm

please address the comments when you get time before we could merge this.

AvineshTripathi · 2026-08-18T15:32:09Z

/lgtm

Adding it again! Changes looks good

ajaysundark · 2026-08-18T22:31:56Z

/approve

kubernetes-prow · 2026-08-18T22:32:07Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ajaysundark, rawadhossain

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [ajaysundark]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kubernetes-prow Bot added the kind/feature Categorizes issue or PR as related to a new feature. label Aug 9, 2026

kubernetes-prow Bot requested review from dchen1107 and mrunalp August 9, 2026 10:59

kubernetes-prow Bot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Aug 9, 2026

kubernetes-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Aug 9, 2026

kubernetes-prow Bot requested review from AvineshTripathi and ajaysundark August 9, 2026 16:15

Ayush4958 mentioned this pull request Aug 9, 2026

Clear FailedNodes on successful node reconciliation #381

Merged

2 tasks

kubernetes-prow Bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 10, 2026

AvineshTripathi reviewed Aug 10, 2026

View reviewed changes

add Prometheus metrics

b5cce66

Signed-off-by: Rawad Hossain <rawad.hossain00@gmail.com>

rawadhossain force-pushed the reporter branch from 95e009f to b5cce66 Compare August 11, 2026 03:40

tejassinghbhati mentioned this pull request Aug 13, 2026

fix: stop the chart suggesting an empty nodeSelector #404

Merged

2 tasks

ajaysundark reviewed Aug 14, 2026

View reviewed changes

rawadhossain added 2 commits August 14, 2026 10:02

add node_readiness_build_info metric

10fa1a0

Signed-off-by: Rawad Hossain <rawad.hossain00@gmail.com>

Merge branch 'main' into reporter

2e35f85

rawadhossain force-pushed the reporter branch 2 times, most recently from 3613030 to 2e35f85 Compare August 14, 2026 23:10

ajaysundark requested review from AvineshTripathi and removed request for dchen1107 and mrunalp August 17, 2026 05:00

kubernetes-prow Bot assigned AvineshTripathi Aug 17, 2026

kubernetes-prow Bot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 17, 2026

rawadhossain mentioned this pull request Aug 17, 2026

Add reporter metrics to examples #419

Open

2 tasks

ajaysundark reviewed Aug 18, 2026

View reviewed changes

Comment thread cmd/readiness-condition-reporter/metrics.go Outdated

ajaysundark reviewed Aug 18, 2026

View reviewed changes

Comment thread cmd/readiness-condition-reporter/metrics.go Outdated

kubernetes-prow Bot assigned ajaysundark Aug 18, 2026

reporter metric buckets and descriptions

2de0068

kubernetes-prow Bot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 18, 2026

kubernetes-prow Bot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 18, 2026

kubernetes-prow Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 18, 2026

kubernetes-prow Bot merged commit 33bdbc9 into kubernetes-sigs:main Aug 18, 2026
10 checks passed

Uh oh!

Conversation

rawadhossain commented Aug 9, 2026 • edited by ajaysundark Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Implementation choices

Build fix

Type of Change

Testing

Checklist

Uh oh!

netlify Bot commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for node-readiness-controller ready!

Uh oh!

kubernetes-prow Bot commented Aug 9, 2026

Uh oh!

ajaysundark commented Aug 9, 2026

Uh oh!

ajaysundark commented Aug 10, 2026

Uh oh!

AvineshTripathi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ajaysundark Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AvineshTripathi commented Aug 17, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rawadhossain Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ajaysundark commented Aug 18, 2026

Uh oh!

AvineshTripathi commented Aug 18, 2026

Uh oh!

ajaysundark commented Aug 18, 2026

Uh oh!

kubernetes-prow Bot commented Aug 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rawadhossain commented Aug 9, 2026 •

edited by ajaysundark

Loading

netlify Bot commented Aug 9, 2026 •

edited

Loading

ajaysundark Aug 11, 2026 •

edited

Loading

rawadhossain Aug 18, 2026 •

edited

Loading