Implement rollout halting and degraded node surfacing#96
Open
jlebon wants to merge 4 commits into
Open
Conversation
When a daemon reports Degraded=True on a BootcNode, the controller now sets the pool's Degraded condition with reason NodeDegraded and a message listing the affected node names. Assisted-by: Pi (Claude Opus 4.6) Signed-off-by: Jonathan Lebon <jonathan@jlebon.com>
Add a new Degraded condition reason for when the controller stops assigning reboot slots because 2 or more nodes in reboot slots are unhealthy (Degraded or not Ready after rebooting into the target image). No behavioral changes yet; the constant is wired up in a follow-up commit. Assisted-by: Pi (Claude Opus 4.6) Signed-off-by: Jonathan Lebon <jonathan@jlebon.com>
When 2 or more nodes occupying reboot slots are unhealthy (BootcNode Degraded after booting the target digest, or K8s Node not Ready after update), the controller stops assigning new reboot slots and sets the pool Degraded with reason RolloutHalted. A single unhealthy node might be a hardware issue, but two suggest a bad image. The rollout resumes automatically once the unhealthy count drops below the threshold (e.g. nodes recover or are manually fixed). Assisted-by: Pi (Claude Opus 4.6) Signed-off-by: Jonathan Lebon <jonathan@jlebon.com>
The controller concurrently patches spec.unschedulable on the same Node, which bumps the resourceVersion. A Status().Update fails in that case because it checks the top-level resourceVersion. Switch to Status().Patch which avoids the conflict entirely. Assisted-by: Pi (Claude Opus 4.6) Signed-off-by: Jonathan Lebon <jonathan@jlebon.com>
alicefr
reviewed
Jul 10, 2026
Comment on lines
+178
to
+187
| g.Eventually(func(g Gomega) { | ||
| var p bootcv1alpha1.BootcNodePool | ||
| g.Expect(k8sClient.Get(ctx, client.ObjectKey{Name: poolName}, &p)).To(Succeed()) | ||
| g.Expect(p.Status.Conditions).To(ContainElement(And( | ||
| HaveField("Type", bootcv1alpha1.PoolDegraded), | ||
| HaveField("Status", metav1.ConditionTrue), | ||
| HaveField("Reason", bootcv1alpha1.PoolNodeDegraded), | ||
| HaveField("Message", ContainSubstring("degraded-w1")), | ||
| ))) | ||
| }).Should(Succeed()) |
Collaborator
There was a problem hiding this comment.
nit: This can be re written as:
Suggested change
| g.Eventually(func(g Gomega) { | |
| var p bootcv1alpha1.BootcNodePool | |
| g.Expect(k8sClient.Get(ctx, client.ObjectKey{Name: poolName}, &p)).To(Succeed()) | |
| g.Expect(p.Status.Conditions).To(ContainElement(And( | |
| HaveField("Type", bootcv1alpha1.PoolDegraded), | |
| HaveField("Status", metav1.ConditionTrue), | |
| HaveField("Reason", bootcv1alpha1.PoolNodeDegraded), | |
| HaveField("Message", ContainSubstring("degraded-w1")), | |
| ))) | |
| }).Should(Succeed()) | |
| g.Eventually(func() ([]metav1.Condition, error) { | |
| var p bootcv1alpha1.BootcNodePool | |
| err := k8sClient.Get(ctx, client.ObjectKey{Name: poolName}, &p) | |
| return p.Status.Conditions, err | |
| }).Should(ContainElement(And( | |
| HaveField("Type", bootcv1alpha1.PoolDegraded), | |
| HaveField("Status", metav1.ConditionTrue), | |
| HaveField("Reason", bootcv1alpha1.PoolNodeDegraded), | |
| HaveField("Message", ContainSubstring("degraded-w1")), | |
| ))) |
Comment on lines
+191
to
+196
| g.Eventually(func(g Gomega) { | ||
| var bn bootcv1alpha1.BootcNode | ||
| g.Expect(k8sClient.Get(ctx, client.ObjectKey{Name: "degraded-w2"}, &bn)).To(Succeed()) | ||
| g.Expect(bn.Annotations).To(HaveKey(bootcv1alpha1.AnnotationInRebootSlot), | ||
| "non-degraded node should get a reboot slot") | ||
| }).Should(Succeed()) |
Collaborator
There was a problem hiding this comment.
nit:
Suggested change
| g.Eventually(func(g Gomega) { | |
| var bn bootcv1alpha1.BootcNode | |
| g.Expect(k8sClient.Get(ctx, client.ObjectKey{Name: "degraded-w2"}, &bn)).To(Succeed()) | |
| g.Expect(bn.Annotations).To(HaveKey(bootcv1alpha1.AnnotationInRebootSlot), | |
| "non-degraded node should get a reboot slot") | |
| }).Should(Succeed()) | |
| g.Eventually(func() (map[string]string, error) { | |
| var bn bootcv1alpha1.BootcNode | |
| err := k8sClient.Get(ctx, client.ObjectKey{Name: "degraded-w2"}, &bn) | |
| return bn.Annotations, err | |
| }).Should(HaveKey(bootcv1alpha1.AnnotationInRebootSlot), "non-degraded node should get a reboot slot") |
| Image: "quay.io/example/myos@" + bootedDigest, | ||
| ImageDigest: bootedDigest, | ||
| } | ||
| bn.Status.Conditions = []metav1.Condition{ |
Collaborator
There was a problem hiding this comment.
nit: this is probably fine but if the bootc node has other conditions, you are wipe them out.
Comment on lines
+236
to
+250
| bn.Status.Conditions = []metav1.Condition{ | ||
| { | ||
| Type: bootcv1alpha1.NodeIdle, | ||
| Status: metav1.ConditionFalse, | ||
| Reason: bootcv1alpha1.NodeReasonStaging, | ||
| LastTransitionTime: metav1.Now(), | ||
| }, | ||
| { | ||
| Type: bootcv1alpha1.NodeDegraded, | ||
| Status: metav1.ConditionTrue, | ||
| Reason: bootcv1alpha1.NodeReasonError, | ||
| Message: "simulated staging failure", | ||
| LastTransitionTime: metav1.Now(), | ||
| }, | ||
| } |
Collaborator
There was a problem hiding this comment.
nit:
Suggested change
| bn.Status.Conditions = []metav1.Condition{ | |
| { | |
| Type: bootcv1alpha1.NodeIdle, | |
| Status: metav1.ConditionFalse, | |
| Reason: bootcv1alpha1.NodeReasonStaging, | |
| LastTransitionTime: metav1.Now(), | |
| }, | |
| { | |
| Type: bootcv1alpha1.NodeDegraded, | |
| Status: metav1.ConditionTrue, | |
| Reason: bootcv1alpha1.NodeReasonError, | |
| Message: "simulated staging failure", | |
| LastTransitionTime: metav1.Now(), | |
| }, | |
| } | |
| meta.SetStatusCondition(&bn.Status.Conditions, metav1.Condition{ | |
| Type: bootcv1alpha1.NodeIdle, | |
| Status: metav1.ConditionFalse, | |
| Reason: bootcv1alpha1.NodeReasonStaging, | |
| }) | |
| meta.SetStatusCondition(&bn.Status.Conditions, metav1.Condition{ | |
| Type: bootcv1alpha1.NodeDegraded, | |
| Status: metav1.ConditionTrue, | |
| Reason: bootcv1alpha1.NodeReasonError, | |
| Message: "simulated staging failure", | |
| }) |
| // indicate a bad image. Now, this _probably_ should scale with number | ||
| // of nodes somehow. E.g. a 500 node cluster could tolerate more | ||
| // unhealthy nodes. Just starting conservative here and we can adjust. | ||
| rolloutHalted := len(unhealthy) >= 2 |
Collaborator
There was a problem hiding this comment.
Should we create an issue to eventually extend the BootcPoolNode API?
Comment on lines
+278
to
+288
| g.Eventually(func(g Gomega) { | ||
| var p bootcv1alpha1.BootcNodePool | ||
| g.Expect(k8sClient.Get(ctx, client.ObjectKey{Name: poolName}, &p)).To(Succeed()) | ||
| g.Expect(p.Status.Conditions).To(ContainElement(And( | ||
| HaveField("Type", bootcv1alpha1.PoolDegraded), | ||
| HaveField("Status", metav1.ConditionTrue), | ||
| HaveField("Reason", bootcv1alpha1.PoolRolloutHalted), | ||
| HaveField("Message", ContainSubstring("halt-w1")), | ||
| HaveField("Message", ContainSubstring("halt-w2")), | ||
| ))) | ||
| }).Should(Succeed()) |
Collaborator
There was a problem hiding this comment.
nit:
Suggested change
| g.Eventually(func(g Gomega) { | |
| var p bootcv1alpha1.BootcNodePool | |
| g.Expect(k8sClient.Get(ctx, client.ObjectKey{Name: poolName}, &p)).To(Succeed()) | |
| g.Expect(p.Status.Conditions).To(ContainElement(And( | |
| HaveField("Type", bootcv1alpha1.PoolDegraded), | |
| HaveField("Status", metav1.ConditionTrue), | |
| HaveField("Reason", bootcv1alpha1.PoolRolloutHalted), | |
| HaveField("Message", ContainSubstring("halt-w1")), | |
| HaveField("Message", ContainSubstring("halt-w2")), | |
| ))) | |
| }).Should(Succeed()) | |
| g.Eventually(func() ([]metav1.Condition, error) { | |
| var p bootcv1alpha1.BootcNodePool | |
| err := k8sClient.Get(ctx, client.ObjectKey{Name: poolName}, &p) | |
| return p.Status.Conditions, err | |
| }).Should(ContainElement(And( | |
| HaveField("Type", bootcv1alpha1.PoolDegraded), | |
| HaveField("Status", metav1.ConditionTrue), | |
| HaveField("Reason", bootcv1alpha1.PoolRolloutHalted), | |
| HaveField("Message", ContainSubstring("halt-w1")), | |
| HaveField("Message", ContainSubstring("halt-w2")), | |
| ))) |
| g.Expect(k8sClient.Get(ctx, client.ObjectKey{Name: "halt-w3"}, &bn3)).To(Succeed()) | ||
| g.Expect(bn3.Annotations).NotTo(HaveKey(bootcv1alpha1.AnnotationInRebootSlot), | ||
| "w3 slot should be freed") | ||
| }).Should(Succeed()) |
Collaborator
There was a problem hiding this comment.
nit:
Suggested change
| }).Should(Succeed()) | |
| g.Eventually(func() (map[string]string, error) { | |
| var bn3 bootcv1alpha1.BootcNode | |
| err := k8sClient.Get(ctx, client.ObjectKey{Name: "halt-w3"}, &bn3) | |
| return bn3.Annotations, err | |
| }).ShouldNot(HaveKey(bootcv1alpha1.AnnotationInRebootSlot), "w3 slot should be freed") |
Comment on lines
+300
to
+304
| var bn bootcv1alpha1.BootcNode | ||
| g.Expect(k8sClient.Get(ctx, client.ObjectKey{Name: "halt-w4"}, &bn)).To(Succeed()) | ||
| g.Expect(bn.Annotations).NotTo(HaveKey(bootcv1alpha1.AnnotationInRebootSlot), | ||
| "w4 should not get a slot while rollout is halted") | ||
| }, "2s", pollInterval).Should(Succeed()) |
Collaborator
There was a problem hiding this comment.
Same nit as previous comment
Comment on lines
+314
to
+319
| g.Eventually(func(g Gomega) { | ||
| var bn bootcv1alpha1.BootcNode | ||
| g.Expect(k8sClient.Get(ctx, client.ObjectKey{Name: "halt-w4"}, &bn)).To(Succeed()) | ||
| g.Expect(bn.Annotations).To(HaveKey(bootcv1alpha1.AnnotationInRebootSlot), | ||
| "w4 should get a slot after rollout resumes") | ||
| }).Should(Succeed()) |
Collaborator
There was a problem hiding this comment.
nit: same remove the succeeded with the actual condition
Comment on lines
+322
to
+331
| g.Eventually(func(g Gomega) { | ||
| var p bootcv1alpha1.BootcNodePool | ||
| g.Expect(k8sClient.Get(ctx, client.ObjectKey{Name: poolName}, &p)).To(Succeed()) | ||
| g.Expect(p.Status.Conditions).To(ContainElement(And( | ||
| HaveField("Type", bootcv1alpha1.PoolDegraded), | ||
| HaveField("Status", metav1.ConditionFalse), | ||
| HaveField("Reason", bootcv1alpha1.PoolHealthy), | ||
| ))) | ||
| }).Should(Succeed()) | ||
| } |
Collaborator
|
Overall looks good, I only had a question and small nits. Thanks for the work! Love the test cases! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Various patches that add more error-handling as described in the 3b implementation plan.