Skip to content

fix: break infinite status reconciliation loop in notebook controller - #1341

Open
Raakshass wants to merge 2 commits into
kubeflow:notebooks-v1from
Raakshass:fix/notebook-controller-status-loop
Open

fix: break infinite status reconciliation loop in notebook controller#1341
Raakshass wants to merge 2 commits into
kubeflow:notebooks-v1from
Raakshass:fix/notebook-controller-status-loop

Conversation

@Raakshass

Copy link
Copy Markdown

Summary

Fixes the infinite reconciliation loop in the notebook controller where updateNotebookStatus issues a Status().Update() PUT on every reconcile even when nothing has changed.

Root Cause

Three-link causal chain:

  1. PodCondToNotebookCond (line 330) — kubelet never sets LastProbeTime on pod conditions, so the else branch injects metav1.Now() on every reconcile, producing a different timestamp each time.

  2. updateNotebookStatus (line 238-240) — unconditional r.Status().Update(ctx, nb) with no comparison guard.

  3. SetupWithManager (line 713-714)For(&v1beta1.Notebook{}) with no GenerationChangedPredicate, so the status write bumps resourceVersion, emits a watch event, and re-enqueues.

Changes

File Change
controllers/notebook_controller.go Reorder PodCondToNotebookCond to resolve LastTransitionTime first, fall back LastProbeTime to condition.LastTransitionTime instead of metav1.Now(), add reflect.DeepEqual guard in updateNotebookStatus

Design Decisions

  • LastProbeTime falls back to LastTransitionTime, not zero — the CRD schema declares lastProbeTime as type: string, format: date-time without nullable: true, and omitempty on metav1.Time (a struct) does not suppress the zero value.
  • Block reorderLastTransitionTime resolved first so LastProbeTime can reference it without a second metav1.Now() call.
  • Both changes neededDeepEqual guard alone does not break the loop because metav1.Now() makes it return false every time. The timestamp fix makes the status deterministic; the guard eliminates the residual no-op PUT.
  • No new importsreflect already used at line 257.

Test Results

All 5 TestCreateNotebookStatus subtests pass. go vet and go build clean.

Follow-up

  • Add test case with LastProbeTime omitted asserting fallback equals LastTransitionTime
  • CopyStatefulSetFields secondary loop is a separate concern

Contributor Checklist

  • DCO sign-off
  • All existing tests pass
  • go vet clean
  • go build clean
  • Single file, minimal diff

PodCondToNotebookCond defaulted LastProbeTime to metav1.Now() when
kubelet left it unset, producing a different timestamp on every
reconcile. updateNotebookStatus wrote unconditionally with no
DeepEqual guard. Combined with For(&Notebook{}) having no
GenerationChangedPredicate, this created an infinite loop.

- Reorder PodCondToNotebookCond to resolve LastTransitionTime first
- Fall back LastProbeTime to LastTransitionTime instead of metav1.Now()
- Add reflect.DeepEqual guard in updateNotebookStatus to skip no-op writes

Signed-off-by: Siddhant Jain <siddhantjainofficial26@gmail.com>
@github-project-automation github-project-automation Bot moved this to Needs Triage in Kubeflow Notebooks Aug 21, 2026
@google-oss-prow google-oss-prow Bot added the area/controller area - related to controller components label Aug 21, 2026
@google-oss-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign thesuperzapper for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow Bot added area/v1 area - version - kubeflow notebooks v1 size/S labels Aug 21, 2026
@Raakshass Raakshass changed the title fix(notebook-controller): break infinite status reconciliation loop fix: break infinite status reconciliation loop in notebook controller Aug 21, 2026
Adds a test case where LastProbeTime is intentionally omitted (zero
value) to simulate kubelet behavior. Asserts that PodCondToNotebookCond
falls back to LastTransitionTime instead of metav1.Now().

This test fails if the LastProbeTime fix is reverted, proving the
guard is load-bearing.

Signed-off-by: Siddhant Jain <siddhantjainofficial26@gmail.com>
@google-oss-prow google-oss-prow Bot added size/M and removed size/S labels Aug 21, 2026
@Raakshass

Copy link
Copy Markdown
Author

@juliusvonkohout

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/controller area - related to controller components area/v1 area - version - kubeflow notebooks v1 size/M

Projects

Status: Needs Triage

Development

Successfully merging this pull request may close these issues.

1 participant