feat(operator): implement ReplicationPolicy, PVCAnnotationWatcher, ReplicationPair, and ReplicationOps controllers

[geoffrey1330]

Summary

Fixes: issue-442

Adds policy-driven snapshot replication to the operator. The old SnapshotReplication CR is gone (removed in the previous commit) — this replaces it with a three-tier CRD hierarchy: ReplicationPair, ReplicationPolicy, ReplicationSlot, and ReplicationOps.

New CRDs

CRD	Short name	Purpose
ReplicationPair	relpair	Reusable site-pair config: source + target cluster. Creates and owns the backend ReplicationTarget. Multiple policies may share one pair.
ReplicationPolicy	repl	Replication schedule and retention, referencing a pair via spec.pairRef. Creates and owns the backend ReplicationPolicy.
ReplicationSlot	relslot	One per PVC; auto-created by the PVCAnnotationWatcher. Drives all backend calls and tracks the live source↔target state machine.
ReplicationOps	replops	One-shot user-driven CR for failover and failback (same pattern as StorageClusterOps).

Annotation mechanism

Opt PVCs into replication via storage.simplyblock.io/replication-policy on a StorageClass (applies to all PVCs) or directly on a PVC (overrides the StorageClass). Post-bind annotation changes are fully handled: a policy change detaches the old slot and attaches a new one; removing the annotation detaches and cleans up snapshots on both sides.

Controllers

• ReplicationPair reconciler — check-or-create the backend ReplicationTarget for the source/target cluster pair; stores the UUID in status.backendTargetID; guards deletion until no ReplicationPolicy CRs reference it.
• ReplicationPolicy reconciler — waits for the referenced ReplicationPair to be ready, then check-or-create the backend ReplicationPolicy using the pair's backendTargetID; counts owned slots in status.slotCount; guards deletion until all slots are gone; on deletion removes only the backend policy (the backend target is owned by the pair).
• PVCAnnotationWatcher — watches PVCs and StorageClasses for annotation changes; creates, replaces, or deletes ReplicationSlot CRs; waits for the PVC to be Bound before creating a slot.
• ReplicationSlot reconciler — drives the per-volume state machine (attaching → poll_attach → replicating → cutover_pending / cutover_done / failed_over → detaching → deleted); syncs lastReplicatedAt from the backend; finalizer ensures detach completes before the CR is GC'd.
• ReplicationOps reconciler — one-active-per-policy guard via ReplicationPolicy.status.activeOpsRef; failover and failback for scope=policy and scope=volume.

Test plan

• Apply all four CRD YAMLs to a dev cluster; confirm kubectl get relpair,repl,relslot,replops work
• Create a ReplicationPair pointing at source and target clusters; verify status.ready=true and status.backendTargetID populated
• Create a ReplicationPolicy referencing the pair; verify status.ready=true and status.backendPolicyID populated
• Annotate a StorageClass with storage.simplyblock.io/replication-policy; provision a PVC; verify a ReplicationSlot is created and reaches status.state=replicating
• Add annotation directly to an existing bound PVC; verify slot is created without reprovisioning
• Change annotation on a bound PVC to a different policy; verify old slot reaches detaching and new slot is created once the old one is GC'd
• Remove the annotation; verify slot reaches detaching, backend DELETE is called, and slot CR is removed
• Create ReplicationOps{action: failover, scope: policy}; verify all slots move to failed_over and status.phase=Succeeded
• Create a second ReplicationOps for the same policy while the first is Running; verify it stays Pending (activeOpsRef guard)
• Create ReplicationOps{action: failback}; verify replication_failback + replication_commit called per volume and slots return to direction=source
• Delete a PVC with an active slot; verify finalizer blocks GC until backend detach completes
• Delete a ReplicationPolicy; verify blocked while slots exist, then GC'd cleanly without deleting the backend target
• Delete a ReplicationPair; verify blocked while policies reference it, then GC'd and backend target deleted

Test Result

ReplicationPair

user@users-MacBook-Pro-2 ~ % kubectl -n simplyblock get ReplicationPair -o yaml
apiVersion: v1
items:
- apiVersion: storage.simplyblock.io/v1alpha1
  kind: ReplicationPair
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"storage.simplyblock.io/v1alpha1","kind":"ReplicationPair","metadata":{"annotations":{},"name":"test-repl-pair","namespace":"simplyblock"},"spec":{"sourceCluster":"cluster1","targetCluster":"cluster2"}}
    creationTimestamp: "2026-08-20T14:43:38Z"
    finalizers:
    - storage.simplyblock.io/replicationpair-finalizer
    generation: 1
    name: test-repl-pair
    namespace: simplyblock
    resourceVersion: "44998"
    uid: c226d075-0820-47cd-bde2-45d713ddad52
  spec:
    sourceCluster: cluster1
    targetCluster: cluster2
  status:
    backendTargetID: 61c0ac8e-64c7-43b7-8308-da1fad0ac691
    message: ReplicationTarget ready
    ready: true
kind: List
metadata:
  resourceVersion: ""

ReplicationPolicy

user@users-MacBook-Pro-2 ~ % kubectl -n simplyblock get ReplicationPolicy -o yaml
apiVersion: v1
items:
- apiVersion: storage.simplyblock.io/v1alpha1
  kind: ReplicationPolicy
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"storage.simplyblock.io/v1alpha1","kind":"ReplicationPolicy","metadata":{"annotations":{},"name":"test-repl-policy","namespace":"simplyblock"},"spec":{"interval":"2m","mode":"failover","pairRef":"test-repl-pair","snapshotRetention":2}}
    creationTimestamp: "2026-08-20T14:43:41Z"
    finalizers:
    - storage.simplyblock.io/replicationpolicy-finalizer
    generation: 1
    name: test-repl-policy
    namespace: simplyblock
    resourceVersion: "45009"
    uid: 0065fd95-830e-40cb-b2f9-678ed194f31d
  spec:
    interval: 2m
    mode: failover
    pairRef: test-repl-pair
    snapshotRetention: 2
  status:
    backendPolicyID: 78914e12-e86c-47ce-aeb6-2d42431c32b0
    ready: true
kind: List
metadata:
  resourceVersion: ""

ReplicationSlot

user@users-MacBook-Pro-2 ~ % kubectl -n simplyblock get ReplicationSlot -o yaml
apiVersion: v1
items:
- apiVersion: storage.simplyblock.io/v1alpha1
  kind: ReplicationSlot
  metadata:
    creationTimestamp: "2026-08-20T14:49:41Z"
    finalizers:
    - storage.simplyblock.io/replicationslot-finalizer
    generation: 1
    name: test-repl-policy-test-pvc-repl
    namespace: simplyblock
    ownerReferences:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      name: test-pvc-repl
      uid: 2c872ca5-e7ef-41f8-bd0d-70227c7bf84c
    resourceVersion: "46001"
    uid: 81d3cdb8-354e-4db1-b00d-d854be424aa8
  spec:
    policyRef: test-repl-policy
    pvcRef: test-pvc-repl
    volumeID: 167588b0-c152-45cf-ba5f-a4653b168cdc:0a0d2526-1b44-4c5c-a903-8a7624e2b51d:e6a48c8b-3b27-491d-a25a-6d513aa2d18e
  status:
    direction: source
    lastReplicatedAt: "2026-08-20T14:49:41Z"
    message: Replicating
    state: replicating
kind: List
metadata:
  resourceVersion: ""