Repository navigation
[Docs] A model's status.observedGeneration and lastReconcileTime advance only when its status changes - #213
Open
github-actions[bot] wants to merge 2 commits into
Open
github-actions[bot] wants to merge 2 commits into
github-actions[bot] wants to merge 2 commits into
Conversation
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
github-actions
Bot
requested review from
XinyueZhang369,
pallasathena92 and
slin1237
as code owners
November 1, 2026 17:02
|
Important Review skippedBot user detected. To trigger a single review, invoke the
|
Contributor
Author
|
Documentation maintenance: ready. Unsuccessful content rounds: 0/3. Operational attempts: 0/3. Head: The diff adds one subsection ("Status freshness") to base-models.md's Model lifecycle section, documenting the freshness contract of status.observedGeneration and status.lastReconcileTime — a single concern on the page that owns model status. Every claim verifies against the pinned OME code: the fields exist on ModelStatusSpec (basemodel_types.go:491-503) with the documented JSON names used in the kubectl example; StampObservedReconcile (shared/shared.go:187-196) sets ObservedGeneration to the current generation and LastReconcileTime to now, and its only two call sites are inside status writes; the per-node writer (pernode/status.go:148-152) early-returns before stamping when nodesReady, nodesFailed and the computed state are unchanged, and the agent's update handler (scout.go:436-455) diffs only labels, annotations and storage/download inputs, so a displayName edit (spec-inline, bumps generation) triggers no status write and observedGeneration lags indefinitely; the PVC writer (pvc.go:574-589) stamps unconditionally on every status write, which recurs via the Job TTL (default 3600) deletion firing the Owns(Job) watch and Job recreation — the cycle the linked serve-models-from-pvc.md#troubleshooting section documents. Both link anchors resolve, the style follows writing-docs.md (sentence-case heading, paired bash/output blocks with the guide's custom-columns pattern, relative .md links), no since badges or release comparisons are added, and no generated, nav or redirect files are touched. The context contains no review threads, so none are marked addressed.
Human review threads and CODEOWNER approval remain under repository policy. |
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2 tasks done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
I edited my BaseModel's displayName and status.observedGeneration still lags metadata.generation, and a healthy Ready model's lastReconcileTime is hours old — is the model controller stuck?
Why we need it
Source change: ome-projects/ome@e8cc64f
From commit 50 (#653, '[API] Add conditions + PVC reasons to model status'), whose patch adds ObservedGeneration and LastReconcileTime to ModelStatusSpec (patch lines 441-453, read today) with the descriptions that now render into the generated API reference ('If this lags behind .metadata.generation, the controller has not yet observed the latest spec edit'; 'Liveness signal — if it falls behind by more than the reconcile period, the controller is wedged or not scheduled'). Current code verified today: pkg/apis/ome/v1beta1/basemodel_types.go:491-503 carries those comments (rendered at reference/api/ome.v1beta1.md:8263-8282, generated, must not be edited); the ONE stamping helper shared.StampObservedReconcile (pkg/controller/v1beta1/basemodel/shared/shared.go:187-196) has exactly two call sites, both inside status writes: backends/pernode/status.go:140-170, whose updateModelStatusWithRetry returns at :148-152 BEFORE stamping whenever nodesReady, nodesFailed and the computed state are all unchanged, and backends/pvc/pvc.go:574-589, which stamps on every PVC-backend status write; basemodel/controller.go itself writes no status. So for an hf://, oci://, modelpack://, local:// or vendor:// model the two fields advance only when the node lists or the state change: an edit base-models.md:166 classifies as 'displayName, modelCapabilities or other descriptive metadata' ('Nodes keep their copy as it is') leaves observedGeneration behind metadata.generation indefinitely with nothing wrong, and a quiet healthy Ready model's lastReconcileTime goes stale — so the generated reference's 'controller is wedged' liveness reading misleads exactly the operator who checks it. A pvc:// model behaves differently: its backend rewrites status as it re-checks the claim and Job (including the TTL Job-recreation cycle documented at serve-models-from-pvc.md:299), so its stamps advance regularly. Docs gap verified today: base-models.md's Model lifecycle section (:211-223) documents only status.state, nodesReady and nodesFailed; a repo-wide grep for observedGeneration and lastReconcileTime under src/lib/content finds no authored mention for BaseModel/ClusterBaseModel (matches are ISVC, quota, WorkloadCluster, TrafficMap pages and the generated reference; benchmarks.md:197 is BenchmarkJob's own field). One independent concern: the freshness contract of the two status fields one commit added, as a short addition to the Model lifecycle section that owns model status. Distinct from merged #186 (the InferenceService-side ome.io/template-hash / observedTemplateHash analog on a different resource and page) and from the PVC condition reasons already documented at serve-models-from-pvc.md:276-298.
Scope: model-storage / model-status-freshness-fields. Other concerns are deferred.
How to test
git diff --checkand website content/link tests, type checks, lint and production build.Checklist
git commit -s)pnpm lint && pnpm check && pnpm test && pnpm buildpasses (run by the publisher on an isolated copy)