Skip to content

etcd-dev runs on emptyDir with no resource limits, and restarts under load #4

Open

Description

@tdakkota

manifests/etcd-dev.yaml backs etcd with an emptyDir volume and sets no resources at all. During a long interactive session on kind, that etcd restarted 3 times.

Those restarts are what surfaced oteldb/storage#375 (a lease-loss bug where nodes never re-register and the ring dissolves permanently) — so in one sense the fixture did its job. But it also means the suite's blast radius includes "etcd went away", which is neither what most specs intend to test nor something they can distinguish from a real failure.

Two separable problems:

No resource limits. Every other workload in the stack is bounded — the storage StatefulSet has memory: 1Gi, the role pools 512Mi. etcd has nothing, so under memory pressure the kubelet's eviction ranking treats it as best-effort and it is among the first things reclaimed. For the component whose availability the entire ring depends on, that is backwards. Requests and limits sized for a dev cluster (etcd is tiny here — the keyspace is a handful of member and owner keys) would make it the last thing evicted rather than the first.

emptyDir. Container restarts keep the volume, so this is less severe than it first appears — but a pod reschedule loses the whole keyspace, and with #375 unfixed that is unrecoverable without deleting every oteldb pod. For a test fixture, ephemeral storage is defensible; the question is whether the suite wants etcd's durability to be part of what it exercises. If not, a small PVC removes a variable.

Worth deciding alongside: should a spec fail loudly if etcd restarts mid-run? Right now an etcd blip degrades the cluster in a way that looks like an oteldb bug, and the suite has no way to say "the fixture broke, not the system under test". Asserting etcd's restart count at spec boundaries would keep the two apart.

Context: found while manually exercising the cluster deployed by #2's role pools. The oteldb-side bug it exposed is real and now filed; this issue is only about the fixture making it easy to hit and hard to attribute.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions