SUPPORT-17719 Drain long requests on apps-proxy shutdown

[pepamartinec]

Release Notes

https://keboola.atlassian.net/browse/SUPPORT-17719
Depends on https://github.com/keboola/kbc-stacks/pull/25705

On shutdown, apps-proxy waited only 20 s for in-flight requests, while a proxied request may run for up to upstream.httpTimeout (4m30s). Every apps-proxy rollout or node drain therefore cut long requests (MCP tool calls, SSE streams) in the middle. The HTTP server drain is now bounded by upstream.httpTimeout instead of the fixed 20 s.

The drain was sized at 20 s because the pod only had the Kubernetes default of 30 s before SIGKILL, and the session-event flush runs after the drain. The linked kbc-stacks PR raises terminationGracePeriodSeconds to 300 s, which covers the 4m30s drain, the 5 s session drain and the remaining shutdown callbacks. The drain setup moved into registerGracefulShutdown, so it can be tested with a real server.

Not covered here:

• WebSocket connections are hijacked, so http.Server.Shutdown does not wait for them.
• Shutdown closes the listener at once, and the ingress does not retry (proxy-next-upstream-tries: 1), so requests routed to a stopping pod before its endpoint is removed are refused. A preStop delay would cover that.

Plans for customer communication

None.

Impact analysis

A stopping apps-proxy pod now lingers for up to 4m30s while in-flight requests finish. Rollouts are not slowed (new pods start first); a node drain waits up to about 5 minutes per apps-proxy pod.

Change type

Improvement

Justification

Long requests are cut on every apps-proxy restart (SUPPORT-17719).

Deployment

Merge https://github.com/keboola/kbc-stacks/pull/25705 and let it sync on all stacks first. Then merge this and release a production-apps-proxy-v* tag. Releasing this first would get the pod SIGKILLed at 30 s before the session flush runs.

Rollback plan

Revert of this PR.

Post release support plan

None.

[keboola-pr-reviewer-bot]

Verdict: needs_human (risk 4/5) · profile psgo

Head commit has failing checks (Lint / lint); no required-check rules discovered on the base branch, falling back to block-on-any-failure.

[Copilot]

Copilot review overview

🟢 Approval recommended

The implementation correctly drains active HTTP requests within the configured bound and includes focused coverage.

Review effort: Balanced
Findings: None

What changed in this PR

Updates apps-proxy shutdown behavior to preserve long-running proxied requests during deployments and node drains.

Changes:

• Uses the upstream HTTP timeout for graceful server draining.
• Adds real-server tests for completion and timeout scenarios.
• Updates session-drain timing documentation.

File	Description
internal/​pkg/​service/​appsproxy/​proxy/​server.go	Configures graceful shutdown using the upstream timeout.
internal/​pkg/​service/​appsproxy/​proxy/​shutdown_test.go	Tests graceful draining and timeout bounds.
internal/​pkg/​service/​appsproxy/​dataapps/​sessions/​writer.go	Documents the updated shutdown budget.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.