feat: Add selector and bootstrap observability metrics

[rawadhossain]

Description

Adds node_readiness_rule_matched_nodes metric to the scrape-time collector pattern introduced in #398.

The metric is computed on each scrape from the cached Node list, sharing the same snapshot as node_readiness_rule_nodes. Dry-run and deleting rules are also included.

(exisitng) node_readiness_failures_total (new reason: AnnotationPatchFailed)

Also adds AnnotationPatchFailed as a new reason for the existing node_readiness_failures_total metric. Tracks failures when writing the bootstrap-completion annotation, keeping all failure reporting under the existing failures_total counter.

Changes

• Moved node_readiness_rule_matched_nodes to the scrape-time ReadinessCollector.
• Shares the Node list with node_readiness_rule_nodes.
• Includes dry-run and deleting rules.
• Added AnnotationPatchFailed to node_readiness_failures_total.
• Failure reasons (EvaluationError, AddTaintError, RemoveTaintError, AnnotationPatchFailed) are now defined as a type with named constants.

Related to #397

Type of Change

/kind feature

Testing

• make test, make lint, go test ./... -race all pass
• Verified the metric against a live cluster and confirmed the counts match the current Node selectors.

Checklist

• make test passes
• make lint passes

[netlify]

✅ Deploy Preview for node-readiness-controller ready!

Name	Link
🔨 Latest commit	4ea706d
🔍 Latest deploy log	https://app.netlify.com/projects/node-readiness-controller/deploys/6a860e375a20db0008caf841
😎 Deploy Preview	https://deploy-preview-286--node-readiness-controller.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[kubernetes-prow]

Hi @rawadhossain. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[AvineshTripathi]

Thanks for the PR! Left a few comments.

[rawadhossain]

@AvineshTripathi PTAL. I'll commit the changes once you approve.
Thanks.

[AvineshTripathi]

@rawadhossain I left one comment PHAL and maybe remove the Bootstrap duration metrics till we merge #224

[rawadhossain]

@AvineshTripathi I have made the changes. PTAL.
Thanks.

[AvineshTripathi]

/ok-to-test

[AvineshTripathi]

/lgtm

cc @ajaysundark for final look

[rawadhossain]

Could node_readiness_bootstrap_completion_errors_total be folded within failures_total with a dedicated reason?

Yeah, agreed. Folded it into failures_total with a dedicated reason, so bootstrap_completion_errors_total has been removed.

[ajaysundark]

Hi @rawadhossain can you update the description to reflect the changes?

[rawadhossain]

updated the description. Thanks.

[AvineshTripathi]

/lgtm

/assign @ajaysundark
for review and approve

[ajaysundark]

We discussed this in design about having this behind the scrape-collector. I see you have a PR (#398) open for that. Do you want to merge this and migrate later or you want to handle this after handling #398 first?

[rawadhossain]

Yes, that was the plan to get this merged first. Since this PR was raised before we introduced the scrape-time collector, I didn’t want to discard it and block the current work.

I’ll do a follow-up migration PR once this and the collector PRs are merged.

[ajaysundark]

Now that scrape-time collector is already in, would it make sense to make these changes right now than to plan a followup?

[rawadhossain]

Yeah, I was thinking the same. I think it'd be better to do it now, and it'll be easier for you to review as well. I'll try to get the changes done today. Thanks.

[kubernetes-prow]

New changes are detected. LGTM label has been removed.

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rawadhossain
Once this PR has been reviewed and has the lgtm label, please ask for approval from ajaysundark. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[AvineshTripathi]

@rawadhossain can you please resolve the conflicts?

[rawadhossain]

@AvineshTripathi resolved the conflicts. GitHub was down for quite a while 😥
PTAL.

[rawadhossain]

@ajaysundark if everything looks right here, can we get it merged?
Will do the collector migration for rule_matched_nodes as a follow-up PR once this land.

[rawadhossain]

@ajaysundark made the changes based on scrape time collector and updated the description.

Could we get #431 (blocked_nodes) reviewed and merged first? This PR has a small adjustment that depends on the changes in #431. Once it's merged, I'll make that change here and push the final commit.