fix: stop reporting adoption for taints the controller applied itself - #384
Conversation
NodeReconciler evaluates rules from the in-memory cache, and the cached copy carries whatever status the rule had the last time RuleReconciler ran. Node evaluations written since then are not in it. Because the rule spec is immutable, GenerationChangedPredicate means that reconcile rarely fires again, so the snapshot can stay stale for the life of the process. evaluateRuleForNode derives isFirstEvaluation from that status, so for a node the snapshot never saw it is always true. Every reconcile of a node that is still unready and already tainted logs "Adopting pre-existing taint" and emits a TaintAdopted event, for a taint the controller applied itself moments earlier. TaintAdopted is meant to tell an operator that a taint existed before the rule took ownership. Emitting it repeatedly for the controller's own taints makes it useless as a signal and adds writes to the events API. Refresh NodeEvaluations from the live object before evaluating. This reads through the controller-runtime cache, so it is not an extra API call, and it leaves the FailedNodes handling alone. Signed-off-by: tejassinghbhati <tejassinghbhati077@gmail.com>
✅ Deploy Preview for node-readiness-controller canceled.
|
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: tejassinghbhati The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Hi @tejassinghbhati. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Thanks for the PR @tejassinghbhati. K8s events are by design best-effort, I dont think this is required to refresh rule cache (which can be a large object) when a node adoption need to be verified. |
|
Fair enough. The Get deep copies the whole rule status, so on a big rule that cost per node is not worth it just to quiet down an event. Thanks for looking. |
Description
NodeReconcilerevaluates rules from the in-memory cache, and the cached copy carries whatever status the rule had the last timeRuleReconcilerran. Node evaluations persisted since then are not in it, and because the rule spec is immutableGenerationChangedPredicatemeans that reconcile rarely fires again, so the snapshot can stay stale for the life of the process.evaluateRuleForNodedecides adoption from that status, so for a node the snapshot never containedisFirstEvaluationis always true and the adoption branch fires on every pass. The result is aTaintAdoptedevent and anAdopting pre-existing taintlog line every time an unready, already tainted node reconciles, for a taint the controller applied itself.The fix refreshes
NodeEvaluationsfrom the live object before evaluating. That read goes through the controller-runtime cache, so it is not an extra API call, and the persist block a few lines below already does the sameGet. It deliberately touches onlyNodeEvaluationsand leavesFailedNodesalone, so it does not overlap with #381 or #209.Related Issue
Fixes #383
Type of Change
/kind bug
Testing
Two specs added under
when a node is re-evaluated while still unready:The first applies the taint, confirms the evaluation is persisted, then reconciles three more times and asserts no further
TaintAdoptedevents. It fails onmain:The second puts the taint on the node before the rule ever evaluates it and asserts exactly one
TaintAdoptedevent, so the feature still works. That one passes both with and without the fix, which is what shows the change is narrow rather than just switching adoption off.Full controller suite passes, 64 of 64 specs, up from 62 before these two.
internal/webhookandcmd/readiness-condition-reporterpass too. Run against envtest with Kubernetes 1.36.2 binaries.golangci-lintv2.12.1 reports 0 issues.Checklist
make testpassesmake lintpassesDoes this PR introduce a user-facing change?