Version
spark 3.5+ (k8s)
Description
I observed a behavior difference between Spark on Kubernetes and YARN for executor heartbeat-timeout scenarios.
In heartbeat timeout handling, HeartbeatReceiver triggers both:
- sc.killAndReplaceExecutor(executorId) (which uses countFailures=true)
- RemoveExecutor(executorId, ExecutorProcessLost(timeout))
However, in Kubernetes backend, doKillExecutors() historically removes executors eagerly with ExecutorKilled.
This can win the race against the later timeout-based RemoveExecutor(...ExecutorProcessLost...), so the final loss reason becomes ExecutorKilled.
Since ExecutorKilled is treated as non-app-caused failure (exitCausedByApp=false), task failures may not be counted towards spark.task.maxFailures, and fast-fail is less likely to happen.
On k8s, heartbeat timeouts can lead to repeated task retries without triggering fast-fail.
On yarn，the same task reaches the retry limit (4 attempts) and the application fails fast.
Impact
- Heartbeat-timeout task failures may bypass failure counting.
- Jobs can retry for a long time instead of fast-failing.
- Behavior diverges from YARN under similar failure conditions.
Suggested fix direction
In K8s doKillExecutors, preserve countFailures semantics from killExecutors:
Keep ExecutorKilled for explicit driver-initiated non-counting kills.
For replacement kills (countFailures=true), avoid eagerly converting to ExecutorKilled; keep a non-ExecutorKilled reason (or allow timeout reason to land) so task-failure counting semantics are preserved.
If this direction is acceptable to maintainers, I can contribute a PR with a minimal fix and corresponding unit tests.
Additional question:
Could maintainers advise how to obtain an Apache JIRA account for new contributors?
Version
spark 3.5+ (k8s)
Description
I observed a behavior difference between Spark on Kubernetes and YARN for executor heartbeat-timeout scenarios.
In heartbeat timeout handling, HeartbeatReceiver triggers both:
However, in Kubernetes backend, doKillExecutors() historically removes executors eagerly with ExecutorKilled.
This can win the race against the later timeout-based RemoveExecutor(...ExecutorProcessLost...), so the final loss reason becomes ExecutorKilled.
Since ExecutorKilled is treated as non-app-caused failure (exitCausedByApp=false), task failures may not be counted towards spark.task.maxFailures, and fast-fail is less likely to happen.
On k8s, heartbeat timeouts can lead to repeated task retries without triggering fast-fail.
On yarn，the same task reaches the retry limit (4 attempts) and the application fails fast.
Impact
Suggested fix direction
In K8s doKillExecutors, preserve countFailures semantics from killExecutors:
Keep ExecutorKilled for explicit driver-initiated non-counting kills.
For replacement kills (countFailures=true), avoid eagerly converting to ExecutorKilled; keep a non-ExecutorKilled reason (or allow timeout reason to land) so task-failure counting semantics are preserved.
If this direction is acceptable to maintainers, I can contribute a PR with a minimal fix and corresponding unit tests.
Additional question:
Could maintainers advise how to obtain an Apache JIRA account for new contributors?