Repository navigation
User Request: Support for pod state disruptions #352
Description
Is your feature request related to a problem? Please describe.
Pod state failures (e.g. graceful/non-graceful deletion) are common disruptions in the Chaos Engineering community.
The reasoning behind pod failures is that Kubernetes pods are ephemeral resources; they get destroyed, restarted, recreated.
This happens in many cases:
- When deploying a new version of an application
- In case the liveness probe of any container running inside the pod fails
- As a consequence of draining a node
- When the autoscaler updates the number of replicas of a deployment
Pod state disruptions can expose a number of reliability concerns including:
- Long-living pods and all the issues that may arise from them
- Cold start issues
- Scalability issues (e.g. autoscaling misconfigurations)
- Inconsistent/unknown startup times
- Uneven traffic distribution across pods
- Non-graceful shutdown
- Issues related to Java's DNS cache TTL leading to terminated pods still receiving requests
- Cascading failures
- We also wrote a blogpost on issues we found when using Kube Monkey
Describe the solution you'd like
Pod deletions can be executed in many different ways. The easiest is through the Kubernetes client which supports graceful and non-graceful deletions through its gracePeriodSeconds parameter. This is how tools like Kube Monkey and our internal controller execute that disruption.
The other option would be to do this at container level which provides more granularity. This is how Pumba executes these disruptions.
A few more implementation details/ideas:
- The level will always be
podfor these disruptions. - In the CRD this might get a bit confusing but one option is to introduce a
podFailure, similar tonodeFailure, with options (e.g. graceful/non-graceful deletion). - There is already a
containersfield which would allow targeting containers.
Activity
nikos912000 commentedon Jun 24, 2021
I'm happy to take a stab on this soon once we decide on these implementation details.
Devatoria commentedon Jul 1, 2021
Hi, sorry for the very late reply but we couldn't find time to answer to this yet.
We never implemented this disruption in the controller because at Datadog we consider a pod being killed (either softly or violently) should not impact us at all as long as the pod can be rescheduled elsewhere to keep the scale. But we definitely agree that it could be an interesting feature outside of the Datadog usecase.
If you want to tackle this one, we can work together on a design document in this issue.
In the CRD, I'd go with a new field
podFailureas you described (in comparison to the existingnodeFailurefield). The level would always be pod (we should catch this in the admission webhook when someone tries to apply such a disruption with thelevel: nodeas it does not make sense).In the injector, as we always try to be container engine agnostic, I'd go with signals which are often an issue (with apps in containers not catching termination signals so they end up with a
SIGKILLwhen the graceful period is reached). The implementation of the primary use case of apodFailuredisruption could be aSIGKILLsent to the target container(s) main PID which is a signal applications can't trap and could be sent for different reasons (OOMkill, end of the graceful period, etc.). An application should not cause any issue when terminated violently because it is unpredictable and uncatchable.What do you think?
nikos912000 commentedon Jul 6, 2021
Sure thing.
In that case this will be a process/container termination that would lead to a pod termination achieving the same goal.
I'll go through the code in the following days and I will post an update for the design.