[neuyilan]

A worker group stops scaling (both up and down) indefinitely when a single worker Pod gets stuck in a terminal/terminating state (e.g. a Failed Pod whose node becomes NotReady, so the kubelet never confirms termination and the Pod keeps a deletionTimestamp that never clears).

Root cause - two gates, both tripped by the same stuck Pod:

1. Scale-expectation gate (expectations/scale_expectations.go): isPodScaled() has a timeout fallback for Create but none for Delete. A Pod requested for deletion that never disappears keeps its Delete expectation unsatisfied forever; IsSatisfied() short-circuits on it, freezing all scaling.
2. Unhealthy-deletion early-return (raycluster_controller.go): even past gate 1, a Failed Pod is unconditionally judged shouldDelete=true, counted into numDeletedUnhealthyWorkerPods, triggering the early return before workersToDelete / replica-diff logic runs. The stuck Pod re-triggers this every reconcile.

Fix:

• scale_expectations.go: give the Delete branch a timeout fallback, but only after Kubernetes has set deletionTimestamp, so an expectation is never satisfied prematurely before deletion actually starts.
• raycluster_controller.go: split out worker Pods that already have a deletionTimestamp (already terminating) so they no longer re-trigger unhealthy deletion or the early-return, and are skipped when draining workersToDelete. Terminating Pods still reserve their replica index so new Pods do not reuse it.
• Observability: when Pods are stuck terminating, log their names (terminatingPodNames), not just a count, and log which Pod/action is blocking a group's reconcile.

Tests:

• TestIsPodScaledDeleteTimeout / TestIsPodScaledDeleteDoesNotTimeoutBeforeDeletion.
• TestReconcilePodsStuckDeleteContinuesScaling / TestReconcilePodsStuckDeleteDoesNotReuseReplicaIndex.

Why are these changes needed?

Related issue number

Labels

• If this PR has user-facing changes that require documentation updates at release time, I have added the doc-updates-required label.
• If this PR contains breaking changes, I have added the breaking-change label.

AI assistance

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

Manual test instructions

[Copilot]

🟢 Approval recommended

No active review issues block approval of the changes addressing scaling deadlocks caused by stuck terminating Pods.

No open issues 🎉

What changed in this PR

This PR addresses worker-group scaling deadlocks in the Ray operator when a Pod remains stuck terminating.

Changes:

• Allow delete expectations to expire after a timeout, but only once the Pod has a deletion timestamp.
• Exclude terminating Pods from active-worker reconciliation while preserving their replica indices, and add diagnostic logging.
• Add regression tests for timeout safeguards, continued scaling, and replica-index preservation; remove sleep-based timing from the existing timeout test.

File	Description
ray-operator/​controllers/​ray/​raycluster_controller.go	Separates active and terminating workers, reserves replica indices, and logs terminating Pod names.
ray-operator/​controllers/​ray/​raycluster_controller_unit_test.go	Tests continued scaling and index preservation with stuck terminating Pods.
ray-operator/​controllers/​ray/​expectations/​scale_expectations.go	Adds guarded delete-expectation expiry and logging for blocked reconciliation.
ray-operator/​controllers/​ray/​expectations/​scale_expectations_test.go	Tests delete-timeout behavior and replaces sleep-based timeout testing.

_{Review effort: Balanced}

[neuyilan]

@rueian Could you please review it? thanks.

[rueian]

Hi @neuyilan, could you fix the lint error as well as the DCO check?

[neuyilan]

Hi @rueian, I have opened one new PR #5489 and fix the lint error as well as the DCO check. please review it again.