[#376] fix: clear stale FailedNodes entry in node reconciler after successful evaluation

[tejassinghbhati]

Summary

processNodeAgainstAllRules permanently re-asserts a stale Status.FailedNodes entry for a node on every reconcile, even after evaluateRuleForNode succeeds, until the rule is manually edited.

This is the unfixed mirror of the fix that landed for processAllNodesForRule — that fix only touched nodereadinessrule_controller.go. The node reconciler path in node_controller.go was never updated with the equivalent clearing logic.

Root cause

The status patch loop rebuilds FailedNodes in two passes. Pass 1 correctly drops this node's current entry from the API server's state. Pass 2 unconditionally re-appends whatever the cached rule.Status.FailedNodes holds for the node — regardless of whether the evaluation just succeeded. Because a successful evaluation never clears the cached copy's FailedNodes field, pass 2 actively re-asserts the stale entry on every node reconcile.

Fix

Gate pass 2 on evalErr != nil, matching the else branch already present in processAllNodesForRule:

// Re-add this node's failure entry only when evaluation just
// failed. On success we leave it absent so that a stale error
// recorded by a previous transient failure is cleared.
if evalErr != nil {
    for _, failure := range rule.Status.FailedNodes {
        if failure.NodeName == node.Name {
            updatedFailedNodes = append(updatedFailedNodes, failure)
            break
        }
    }
}

Testing

Added a regression test in node_controller_test.go using a fake client with WithStatusSubresource:

1. Seed a FailedNodes entry for the node to simulate a prior transient error.
2. Run processNodeAgainstAllRules with the node satisfying all conditions (evaluation succeeds).
3. Assert the stale FailedNodes entry is absent from the patched rule status.

go build ./... passes cleanly. The controller test suite requires envtest binaries (etcd / kube-apiserver) which are unavailable locally on Windows; CI runs on ubuntu-latest where they are present.

Related Issue

Fixes #376

Type of Change

/kind bug

Checklist

• make test passes (CI; envtest not available locally on Windows)
• go build ./... passes

Does this PR introduce a user-facing change?

NONE

[netlify]

✅ Deploy Preview for node-readiness-controller canceled.

Name	Link
🔨 Latest commit	87ec116
🔍 Latest deploy log	https://app.netlify.com/projects/node-readiness-controller/deploys/6a85a3d7c88f7d00081e4b44

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tejassinghbhati
Once this PR has been reviewed and has the lgtm label, please assign mrunalp for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kubernetes-prow]

Hi @tejassinghbhati. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[kubernetes-prow]

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ajaysundark]

Duplicate Of #381

/close

[kubernetes-prow]

@ajaysundark: Closed this PR.

Details

In response to this:

Duplicate Of #381

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.