🐛 fix(controller): snapshot CronJob drops scheduling constraints and wedges permanently

[thilak009]

Problem

The snapshot CronJob can never run on a cluster whose nodes carry taints, and once it fails the first time it never recovers. The CronJob object looks perfectly healthy in kubectl get cronjob the whole time — .status.lastSuccessfulTime simply stays empty forever.

Two compounding defects:

1. Scheduling fields are dropped. buildSnapshotCronJob built its pod without NodeSelector, Tolerations or Affinity, even though the CRD accepts all three and buildStatefulSet honours them (statefulset.go:147-149). On tainted node pools the snapshot pod is unschedulable and pends forever:

FailedScheduling: 0/N nodes are available: N node(s) had untolerated taint(s)

Nothing in the CR could work around this — spec.tolerations reached the StatefulSet, but never the CronJob. So the memgraph pods schedule fine while the snapshot pod cannot.

2. Forbid made it permanent. ConcurrencyPolicy was hardcoded to Forbid with no activeDeadlineSeconds or startingDeadlineSeconds anywhere. A permanently-Pending job stays Active forever, so every later tick is skipped. lastScheduleTime also stops advancing under Forbid, which eventually trips the CronJob controller's too many missed start times (> 100) lockout — a second, independent wedge.

Changes

1. snapshot.go — propagate NodeSelector/Tolerations/Affinity to the snapshot pod template, matching the StatefulSet.

2. memgraphcluster_types.go — three new optional SnapshotSpec fields: concurrencyPolicy (default Forbid), activeDeadlineSeconds (default 600), startingDeadlineSeconds (default 300).

   Forbid is kept as the default rather than switching to Replace: it prevents overlapping CREATE SNAPSHOT runs, whereas Replace would kill a legitimately slow snapshot on every tick and never let one finish — trading "never runs" for "never finishes". The wedge is broken by activeDeadlineSeconds instead: a stuck job is marked Failed and its pods terminated, freeing the slot so the next tick runs, without disturbing a healthy job that simply needs more time. concurrencyPolicy is exposed for anyone who does want Replace.

3. snapshot.go — replace the update check. Without this the rest of the PR has no effect on any existing deployment. needsUpdate compared only the schedule string and the container count; adding tolerations changes neither, so any cluster that already has a CronJob would silently keep its broken pod template after upgrade.

   The new snapshotCronJobNeedsUpdate compares only operator-managed fields. Comparing whole specs would report drift on every reconcile, because the API server defaults many PodSpec fields the operator never sets; TestSnapshotCronJobNeedsUpdateIgnoresServerDefaults guards against that update loop.

Compatibility

No CronJobs are deleted by this PR — it only repairs existing ones.

The CRD change is purely additive (three new optional fields with defaults), so the updated CRD must be installed before the operator writes them.

Upgrade note

activeDeadlineSeconds is baked into a Job at creation, so it does not apply retroactively to a Job that is already stuck Active. Under Forbid such a Job keeps blocking even after this ships, and needs a one-time manual delete:

kubectl -n <namespace> delete job <cluster>-snapshot-<id>

If the CronJob still refuses to schedule afterwards, check its events for too many missed start times (> 100) — startingDeadlineSeconds prevents this recurring, but a CronJob already in that state may need recreating.

When auditing whether snapshots have ever worked, check lastSuccessfulTime rather than whether the object exists:

kubectl -n <namespace> get cronjob <cluster>-snapshot \
  -o jsonpath='{.status.lastSuccessfulTime}'

An empty value means it has never run once.

Testing

make ci passes — lint 0 issues, all packages green, build succeeds. New coverage: scheduling-field propagation, deadline defaults and overrides, update-detection drift cases plus the server-defaults loop guard.

Follow-up

A second PR handles the other half of this: snapshot.enabled: false is currently unsettable (non-pointer bool + omitempty + default=true) and the CronJob cleanup path is unreachable. Split out because it changes behaviour on existing clusters.

Also not addressed here: spec.snapshot.retentionCount and spec.snapshot.s3.retentionDays are accepted by the CRD and read by no controller code.

🤖 Generated with Claude Code