Skip to content

fix(keda): keep the autoscaled replica count on redeploy - #4089

Open
Elvand-Lie wants to merge 1 commit into
knative:mainfrom
Elvand-Lie:fix/keda-preserve-replicas
Open

Elvand-Lie wants to merge 1 commit into
knative:mainfrom
Elvand-Lie:fix/keda-preserve-replicas

Conversation

@Elvand-Lie

Copy link
Copy Markdown
Contributor

Changes

  • 🐛 Redeploying a function with the keda deployer no longer resets its replica count. The raw deployer, which keda embeds, replaces the Deployment on every update with spec.replicas set to max(1, scale.min). KEDA's HPA owns that field, so a function the HPA had scaled to 6 replicas dropped to 1 on redeploy, and pods serving traffic were terminated until the HPA scaled it back up.
  • On update, the keda deployer now keeps the live replica count, raised to scale.min (at least 1) and lowered to an explicit scale.max. This follows the existing preserveDeploymentSelector pattern for cluster-owned Deployment state. HTTP and Kafka triggers both go through it.
  • The raw deployer is unchanged. Its documented contract is a fixed replica count, so scale.min stays authoritative there. A keda function scaled to zero is still started at one replica on redeploy, as before.

/kind bug

There is no issue for this. The Kubernetes HPA docs describe the failure: while an HPA is active, applying an object that sets spec.replicas "will instruct Kubernetes to scale the current number of Pods to the value of the spec.replicas key".

Testing

  • TestDeploy_ReplicasOnRedeploy runs the full keda Deploy (HTTP and Kafka triggers) against a fake Kubernetes API server and checks the replica count sent in the Deployment update. On main, the live count, Kafka and scale.max cases fail with replicas = 1. The scale from zero, raised scale.min and raw cases pass on both, as controls.
  • Test_preserveAutoscaledReplicas covers the bounds directly.
  • go test -race ./pkg/keda/ ./pkg/k8s/ and go vet pass locally, apart from two existing pkg/k8s tests that need a reachable cluster. Not run: an end to end test on a cluster with KEDA installed.

Release Note

Redeploying a function with the keda deployer no longer resets its replica count. The count set by the autoscaler is kept within scale.min and scale.max.

Docs


@knative-prow knative-prow Bot added the kind/bug Bugs label Oct 18, 2026
@knative-prow

knative-prow Bot commented Oct 18, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Elvand-Lie
Once this PR has been reviewed and has the lgtm label, please assign matzew for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@knative-prow knative-prow Bot added size/L 🤖 PR changes 100-499 lines, ignoring generated files. needs-ok-to-test 🤖 Needs an org member to approve testing labels Oct 18, 2026
@knative-prow

knative-prow Bot commented Oct 18, 2026

Copy link
Copy Markdown

Hi @Elvand-Lie. Thanks for your PR.

I'm waiting for a knative member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@lkingland

Copy link
Copy Markdown
Member

/ok-to-test

@knative-prow knative-prow Bot added ok-to-test 🤖 Non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test 🤖 Needs an org member to approve testing labels Oct 30, 2026
@knative-prow-robot knative-prow-robot added the needs-rebase Cannot be merged due to conflicts with HEAD. label Oct 30, 2026
@lkingland

Copy link
Copy Markdown
Member

Keeping the HPA's count on redeploy seems obviously right to me.

Needs a rebase, but I ran this against main and tests all pass, and correctly fail without the fix.

Two things from an adversarial AI review came up (both nonblocking and FYI):

  1. Because the update carries the live ResourceVersion, an HPA scale between the GET and the PUT returns a 409 and fails the deploy. That's safe (we never overwrite the HPA's newer count) and it predates this PR, but it's now the remaining way a redeploy under active autoscaling can fail. Would wrapping the GET/preserve/PUT in retry.RetryOnConflict make sense, here or as a follow-up?

  2. A keda function at zero is restarted at one replica on redeploy, and KEDA can re-idle it while func deploy is still waiting. That's the race k8s deployer: WaitForDeploymentAvailable errors on a 0-replica Deployment (keda scale-to-zero redeploy) #4068 / fix(k8s): tolerate scale-to-zero during deployment readiness #4075 address. Maybe worth a line in the description that this pairs with fix(k8s): tolerate scale-to-zero during deployment readiness #4075?

Rebase and we can merge.

KEDA's HPA owns the live replica count, but redeploying replaces it
with scale.min and can remove capacity under load.
Preserve the count within new bounds while raw keeps its fixed count.

Assisted-by: Codex (gpt-6.1-sol)
Signed-off-by: Elvand Lie Nababan <elvandlie@gmail.com>
@Elvand-Lie
Elvand-Lie force-pushed the fix/keda-preserve-replicas branch from 66c9290 to 525c0aa Compare October 30, 2026 14:25
@knative-prow-robot knative-prow-robot removed the needs-rebase Cannot be merged due to conflicts with HEAD. label Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/bug Bugs ok-to-test 🤖 Non-member PR verified by an org member that is safe to test. size/L 🤖 PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants