[BUG] Orphaned readiness taint remains when a node's labels no longer match a rule's selector

[bhuvan-somisetty]

What happened?

Split out of #338 per @ajaysundark's review comment: when a node's labels change so it no longer matches a NodeReadinessRule's spec.nodeSelector, NodeReconciler simply stops seeing the rule as applicable via getApplicableRulesForNode and moves on. Any taint that rule previously applied to the node is never removed, and the node's stale entries in rule.Status.NodeEvaluations/AppliedNodes/FailedNodes are never cleaned up.

Steps to Reproduce

1. Create a rule with nodeSelector: {tier: frontend} and taint readiness.k8s.io/frontend-ready:NoSchedule.
2. Label a node tier=frontend; once a required condition is False, the controller applies the taint.
3. Relabel the node: kubectl label node worker-1 tier=backend --overwrite.
4. Inspect worker-1.spec.taints — readiness.k8s.io/frontend-ready is still present even though the node no longer matches tier: frontend.

Expected Behavior

Ideally the taint gets removed and the node drops out of the rule's status once it stops matching. In practice this needs a reliable way for NodeReconciler to detect "used to match, no longer does" without depending on rule.Status as the signal for what to act on, since the controller's own previously-written status isn't a trustworthy basis for deciding what action to take next (a first attempt at this in #339 did exactly that, and @ajaysundark flagged it as fragile in review).

Controller Version / Image Tag

main (commit 43bff46)

Kubernetes Version

Not version-specific.

Additional Environment Details

Carried over from review discussion on #338/#339:

• The straightforward alternative — have NodeReconciler evaluate all rules (not just currently-matching ones) on every node update, so it can catch "still has a managed taint but no longer matches" — works, but is expensive, especially with the extra status writes it implies for every node event.
• If a manual relabel (e.g. via script) causes this, removing the taint could reasonably be handled by the same script/automation rather than the controller. dry-run mode also exists to catch this kind of misconfiguration before it happens.
• Given that, this is a real gap but arguably not high-impact, and needs a proper design discussion on how to detect the "used to match" transition reliably before implementing anything — rather than being bundled into the node-deletion cleanup fix in [BUG] Handle node deletion, label selector unmatching, and status drift in NodeReconciler #338/fix(controller): handle node deletion, label selector unmatching, and status drift in NodeReconciler #339.

[ajaysundark]

I just realized we already have a bug tracking this. So this can be marked duplicate. Sorry I missed!