Add EtcdBackup CRD enhancement for OADP integration

[jparrill]

Summary

• Introduces a new EtcdBackup CRD in the hypershift.openshift.io/v1beta1 API group
• Serves as the contract between the OADP HyperShift plugin and the Hypershift Operator for triggering etcd backups
• Controller in the HO orchestrates snapshot and upload Jobs while keeping management credentials isolated from HCP namespaces

Details

• Tracking: https://issues.redhat.com/browse/CNTRLPLANE-2676
• Status: Implementation OnGoing
• CRD: EtcdBackup (namespaced, in HCP namespace)
• Key design decision: Backup Jobs run in the HO namespace to keep AWS/S3 credentials isolated from customer-scoped HCP namespaces

Test plan

• Unit tests: Controller reconcile logic, Job manifest construction, Secret copy/cleanup, status condition updates
• Integration tests: Create EtcdBackup CR, verify Job creation, simulate completion, verify status
• E2E tests: Full backup flow with real HostedCluster

🤖 Generated with Claude Code

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign sjenning for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• enhancements/hypershift/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jparrill]

If you have time @sjenning @enxebre @csrwng @muraee @bryan-cox, please review 🙏 .

[bryan-cox]

The ticket says the solution should work for both Azure and AWS. This should mention the Azure bits as well.

[bryan-cox]

This applies to the rest of the proposal where AWS only language is used.

[jparrill]

Updated in the latest revision. The enhancement now covers both AWS S3 and Azure Blob Storage:

• EtcdBackupStorageType enum includes S3 and AzureBlob
• New EtcdBackupAzureBlob struct with container, storageAccount, keyPrefix, and credentialsSecretRef
• All AWS-only language has been generalized throughout the proposal (workflow, diagram, credential isolation, alternatives, test plan)
• The Non-Goals section clarifies that S3 is implemented first, with Azure Blob following within the same epic via an uploader interface

[bryan-cox]

The enhancement doesn't address the interaction with KMS encryption at rest. When a hosted cluster has KMS encryption configured, the etcd snapshot will contain DEKs wrapped by the KMS key. The snapshot and upload process should work fine since etcdctl snapshot save is encryption-agnostic, but the resulting backup is only restorable if the KMS key remains available and accessible.

A few questions:

1. Should the EtcdBackupStatus capture metadata about the encryption state (e.g., whether KMS was active, which key ID was used)? This would let the restore path validate key availability before attempting a restore, rather than failing opaquely.
2. Should there be a note in the Risks and Mitigations table about the dependency between KMS key lifecycle and backup usability?
3. Even though restore is out of scope here, does the existing RestoreSnapshotURL mechanism already account for KMS key availability, or is that a gap that needs to be tracked separately?

[jparrill]

Addressed in the latest revision:

• Added encryptionMetadata to EtcdBackupStatus with kmsEnabled (bool) and kmsKeyID (string). The controller reads the KMS config from the HostedCluster's SecretEncryptionSpec at backup time and records it in the status. The kmsKeyID stores the AWS KMS ARN or Azure Key Vault key URL — both formats are self-contained and sufficient to validate key availability with the appropriate credentials.
• Added a new entry in the Risks and Mitigations table covering the KMS key lifecycle dependency: if the key is deleted after backup, the snapshot becomes unrestorable, and the encryptionMetadata allows the restore path to validate this upfront.
• Regarding RestoreSnapshotURL: it does not currently validate KMS key availability — that should be tracked separately.

[Ajpantuso]

From a ROSA perspective it would be ideal to check the KMS key availability prior to attempting restoration. We would want an early indication that the restoration would fail and encoding the encryptionMetadata with the backup would definitely help to expose that earlier.

[jparrill]

Agreed. The encryptionMetadata field in the status (kmsEnabled + kmsKeyID) is designed exactly for this — allowing the restore path to validate KMS key availability upfront before attempting the restore. This is already in the enhancement. Note that key rotation is safe (both AWS KMS and Azure Key Vault retain previous key material), but key deletion after the waiting/retention period makes the backup permanently unrestorable — this is documented in the Drawbacks section.

[lance5890]

Although the this won't be confused with the etcd-operator's backup crd ceo backup crd, which is also named EtcdBackup. since they are installed in different management clusters, in rare (or future) cases where both CRDs coexist in the same cluster, using a plain oc get etcdbackups (without specifying the group) could lead to ambiguous behavior or errors.

[jparrill]

Yeah, the name is not written in stone, maybe hcpEtcdBackup or etcdBackupHCPcould be good ones.

[enxebre]

This doesn't need to be an open question, let's just double check the implementation and update the proposal accordingly.
FWIW If we were not blocking this traffic, that would be a bug

[jparrill]

Addressed. Moved out of Open Questions into a new "NetworkPolicy for Cross-Namespace Etcd Access" section under Implementation Details. The same-namespace NetworkPolicy in HCP namespaces blocks cross-namespace ingress by default. The controller will create a temporary NetworkPolicy in the HCP namespace allowing ingress from the HO namespace to etcd pods on port 2379, scoped to app: etcd. It is cleaned up after the Job completes (and also covered by owner references for GC in failure scenarios). Workflow steps 2 and 4 updated accordingly.

[enxebre]

Can you point to this interface? https://github.com/search?q=repo%3Aopenshift%2Fhypershift%20BackupUploader&type=code

[enxebre]

Also what happens with the existing EtcdBackupCronJob

[jparrill]

Good catch — the BackupUploader interface does not exist in the codebase. Updated the enhancement: all references now state that the uploader interface will be implemented as part of this work. The existing etcd-backup binary currently supports S3 upload directly; the interface will be added to make it extensible to Azure Blob Storage (within this epic) and other backends in the future.

[jparrill]

Addressed. The existing EtcdBackupCronJob and EtcdBackupServiceAccount in manifests/etcd.go are unused scaffolding — no controller references them. The enhancement now documents this in a new "Existing Binary and Manifest Reuse" section: the EtcdBackupReconciler will create on-demand batch/v1.Job resources reusing the same control-plane-operator image and etcd-backup subcommand. The existing CronJob scaffolding will be removed or superseded by this CRD-driven approach.

[jparrill]

Also what happens with the existing EtcdBackupCronJob

The existing EtcdBackupCronJob and EtcdBackupServiceAccount in manifests/etcd.go are unused scaffolding — no controller references them. The EtcdBackupReconciler introduced by this enhancement will create on-demand batch/v1.Job resources following the same two-container pattern established in PR #3034:

• InitContainer (CPO image): copies the control-plane-operator binary to a shared emptyDir volume and fetches etcd TLS certs from the HCP namespace.
• Main container (etcd image): provides etcdctl natively, executes the copied CPO binary (control-plane-operator etcd-backup) which calls etcdctl snapshot save then uploads to cloud storage.

The existing CronJob scaffolding will be removed or superseded by this CRD-driven approach. See the new "Container Image Strategy" and "Existing Binary and Manifest Reuse" sections.

[enxebre]

does this workflow leave old CRs indefinitely in etcd?

[jparrill]

Good point — addressed. The workflow now includes a step 6: the OADP plugin deletes the EtcdBackup CR as a post-hook cleanup step after reading the status. As a fallback, the controller will garbage-collect CRs that have been in a terminal state (BackupCompleted=True or BackupCompleted=False) for longer than a configurable retention period (default: 24h).

[lance5890]

Besides the retention period, do we need a retention count limit?

[jparrill]

Good suggestion. Updated the GC strategy to include both:

• Retention period: terminal CRs older than 24h (configurable) are deleted
• Retention count: max 5 completed CRs per HostedControlPlane (configurable), oldest deleted first

[jparrill]

Update: We've revised the GC approach based on feedback from @Ajpantuso. Instead of custom retention logic (time-based + count-based), the enhancement now uses a finalizer-based lifecycle management. The EtcdBackup CR has a finalizer that cleans up the corresponding cloud storage object on deletion. This ties the CR lifecycle to the external artifact and eliminates the need for custom GC — retention is managed externally via Velero TTL or direct deletion. See the updated "Finalizer-Based Lifecycle Management" section.

[enxebre]

can you elaborate why we choose putting etcd-client-tls and etcd-ca for all HCP in the HO namespace over temporary put the creds secret in targeted HCP namespace?

[muraee]

agreed, the job should run in the HCP namespace with temporary S3 credentials passed.

[jparrill]

Elaborated in the latest revision under "Credential Isolation". Two reasons:

1. Cloud storage credentials are management-scoped, not per-cluster. The backup bucket is shared infrastructure owned by the service provider. All credentials in HCP namespaces are customer-scoped (STS Web Identity roles for EC2, Route53, EBS, etc.) — the ControlPlaneOperatorARN has zero S3/storage permissions. Placing a management-scoped credential into a customer-scoped namespace breaks this trust model.

2. Privilege escalation risk. In a multi-tenant management cluster, any workload in an HCP namespace with RBAC to read Secrets could access a management-level credential granting access to shared infrastructure across all hosted clusters. Etcd TLS secrets are inherently single-cluster-scoped — copying them into the more-privileged HO namespace does not expand their blast radius.

See also @muraee's comment below — this is an open design discussion point.

[jparrill]

Thanks for the input @muraee. The current design runs the Job in the HO namespace to avoid placing management-scoped storage credentials (Red Hat/CSP-owned, shared across all hosted clusters) into customer-scoped HCP namespaces. The ControlPlaneOperatorARN has no S3/storage permissions, so a new management-level credential would need to be temporarily injected.

The concern is that even temporarily, a management credential in the HCP namespace could be read by any workload with get secrets RBAC, potentially exposing shared infrastructure. Copying etcd TLS secrets (single-cluster-scoped) to the HO namespace is the safer direction since it doesn't escalate privilege.

That said, this is a trade-off worth discussing — if we can sufficiently scope the temporary credential and ensure cleanup, the HCP namespace approach would avoid the NetworkPolicy complexity. Happy to discuss further.

[jparrill]

Update: This was discussed in a team meeting and the decision is to keep the Job in the HO namespace. The key reasons remain the credential isolation model — management-scoped storage credentials must not enter customer-scoped HCP namespaces.

To address the etcd TLS certificate access without copying Secrets as persistent objects, the Job will use an InitContainer that fetches the etcd TLS certificates (etcd-client-tls, etcd-ca) from the HCP namespace and makes them available to the main container via a shared emptyDir volume.

The ServiceAccount used by the Job is pending a decision from Managed Services: either extend the existing HO SA with storage permissions, or create a dedicated etcd-backup-sa with minimal permissions.

The trade-offs of running in the HO namespace (service network overhead, node scheduling, NetworkPolicy management) are documented and mitigated by serializing backup Jobs (one at a time). See the updated "Consequences of Running the Job in the HO Namespace" section.

[enxebre]

please include SRE/stakeholders for ROSA and ARO cc @mmazur @typeid @joshbranham

[Ajpantuso]

I'll be the ROSA SRE contact for this, thanks!

[jparrill]

Done — added @mmazur, @typeid, @joshbranham, and @Ajpantuso as reviewers in the enhancement header.

[enxebre]

does this UX satisfy use cases for both rosa and aro?

[Ajpantuso]

The more ergonomic usage for ROSA would be EtcdBackup resources being bound to the external S3 object state such that removal of an EtcdBackup would be blocked by a finalizer which allows the owning controller to cleanup the corresponding s3 object.

That would:

1. Eliminate this gc logic
2. Allow cascading deletion of EtcdBackups when the corresponding Velero Backup is deleted.
3. Leverage the Velero TTL mechanism for pruning s3 backups instead of having to apply external lifecycle policies
4. Make it easier to correlate Velero resource backups to the corresponding EtcdBackup

Obviously there would be raciness with cleanup if the upload credentials are removed before resources are cleaned up, but this is a similar issue in OADP/Velero and is unavoidable without some extended APIs for credential handling which is probably not worth the effort.

[jparrill]

We've received initial feedback from ROSA SRE (@Ajpantuso) with concrete suggestions that we've incorporated into the latest revision (finalizer-based lifecycle, etcd health pre-checks). ARO input is still welcome — cc @mmazur @joshbranham.

[jparrill]

Great proposal — adopted in the latest revision. The enhancement now uses a finalizer-based lifecycle management approach:

• The controller adds a finalizer (hypershift.openshift.io/etcd-backup-cleanup) when it first processes the CR.
• On CR deletion, the finalizer triggers cleanup of the corresponding cloud storage object before the CR is removed.
• This eliminates custom GC logic, enables cascading deletion from Velero Backup, and leverages Velero TTL for retention.
• The credential availability race condition at deletion time is documented as an accepted trade-off (consistent with Velero's approach).

See the new "Finalizer-Based Lifecycle Management" section in the enhancement.

[enxebre]

can we specify a recovery point goal (how fresh a backup can be) / SLO for the worst case scenario (management cluster a max HC capacity) and articulate how this proposal meets it

[typeid]

Adding @Ajpantuso who is running point for ROSA DR

[typeid]

Is my understanding correct here that we are no longer using EBS volume snapshots but instead directly using etcd snapshots (using underlying etcdctl) - then uploading to a target S3? This solves some of our problems we've had, such as moving snapshots from one account to another for cross-mc recoveries.

If I'm reading correctly, we don't encrypt the S3 etcd snapshots - therefore they would only be encrypted via the existing KMS if etcd itself was already encrypted - correct?

[Ajpantuso]

If we are aligning with ROSA requirements for velero resource backups then we would need per-HCP KMS encryption for the stored artifacts. Obviously S3 encrypts at rest by default, but we want further isolation for tenants.

etcd encryption does not apply to the whole snapshot anyway, only values so etcd encryption doesn't satisfy privacy requirements and is inconsistenly applied as you mentioned Claudio..

TL;DR Explicitly adding a requirement that KMS encryption is configurable per EtcdBackup makes sense to me.

[jparrill]

Correct on both points:

1. etcdctl snapshots, not EBS snapshots. The Job runs etcdctl snapshot save and uploads the resulting file to S3/Azure Blob. This is indeed portable across accounts and solves the cross-MC recovery use case.

2. No additional encryption on the S3 artifact itself. The snapshot is uploaded as-is. If etcd had KMS encryption at rest enabled, the snapshot contains DEKs wrapped by the KMS key, but only the values are encrypted — not the full snapshot. That's a requirement we're evaluating.

[jparrill]

Valid point — etcd encryption at rest only covers values, not the full snapshot, so it doesn't satisfy privacy requirements for the stored artifact. We're evaluating adding per-HCP KMS encryption for the S3/Blob objects as a configurable option in the EtcdBackup spec. Will update the enhancement once we've worked through the details.

[jparrill]

Agreed — addressed in the latest revision. The EtcdBackup spec now supports optional per-tenant encryption of the backup artifact:

• AWS S3: spec.s3.kmsKeyARN — enables SSE-KMS with a customer-managed KMS key. Bucket keys are recommended to reduce KMS API costs.
• Azure Blob: spec.azureBlob.encryptionKeyURL — enables encryption with a customer-managed Key Vault key.

Both fields are optional. If not set, the bucket/storage account's default encryption is used.

Importantly, this is independent of etcd encryption at rest — the enhancement now documents both encryption layers clearly:

• Etcd encryption at rest (KMS provider): encrypts values inside etcd. Only values are encrypted, not the full snapshot. Tracked in status.encryptionMetadata.kmsKeyID.
• Artifact encryption (SSE-KMS / Azure CMK): encrypts the entire backup artifact in cloud storage. Provides per-tenant isolation. Configured in the spec.

The SA used by the backup Job must have permissions on both the storage backend and the tenant's KMS key when artifact encryption is configured. See the new "Artifact Encryption (SSE-KMS / Azure CMK)" section.

[tony-schndr]

@jparrill can you elaborate on the encryption mechanism for Azure? Will it be client side or server side encryption? If encryption is done server side I assume it will be via encryption scopes?

[jparrill]

@tony-schndr Good question. For Azure, the mechanism that best maps to SSE-KMS on S3 would be Encryption Scopes with a customer-managed key (CMK) from Azure Key Vault. This is server-side encryption — the blob is encrypted transparently at upload time by Azure Storage using a Key Vault key scoped to a specific encryption scope. The etcd-upload subcommand would pass the x-ms-encryption-scope header on the blob upload request.

However, there are some trade-offs worth discussing before committing to this approach:

1. Cost per encryption scope — Each encryption scope is billed with a minimum of 30 days. If we create one scope per tenant (per HCP), that's a fixed cost per hosted cluster with backups configured. At scale (e.g., 64 HCPs per MC), this adds up.

2. Encryption scopes cannot be deleted — They can only be disabled. When a HostedCluster is deprovisioned, the encryption scope remains orphaned in the storage account (disabled but present). Over time, residual scopes accumulate.

3. 10,000 scope limit with auto-rotation — With automatic key version updates, there's a limit of 10,000 encryption scopes per storage account. Given that scopes can't be deleted, long-lived management clusters with high HC churn could eventually exhaust this limit.

4. Access tier restrictions — Blobs uploaded with an encryption scope cannot change access tier (e.g., move to Archive). This could limit cost-saving retention strategies via tiering for older backups.

5. Pre-provisioning required — The encryption scope must be created on the storage account before uploading the blob. This adds an operational step — either the controller or the managed service team (ROSA SRE / ARO SRE) needs to create the scope beforehand.

The simpler alternative would be storage account-level CMK (a single Key Vault key for the entire account), but that doesn't provide per-tenant isolation which @Ajpantuso has flagged as a requirement for ROSA.

Given these trade-offs, I think Encryption Scopes are still the right choice for per-tenant isolation on Azure, but I'd like your input on whether the downsides (especially cost, non-deletable scopes, and access tier restrictions) are acceptable for ARO HCP at scale. Alternatively, if there's a different server-side mechanism you had in mind, happy to discuss.

cc @Ajpantuso

[tony-schndr]

@jparrill thanks for the breakdown. Per tenant CMK isolation has not been established as a hard requirement for ARO-HCP. The current backup encryption is done at the storage account level with platform managed keys + etcd encryption at rest with CMK. I am still trying to dig into what AKS does for etcd backups and I'm not ready to commit to encryption scopes with CMK.

[jparrill]

Understood — no commitment to encryption scopes with CMK for ARO-HCP at this point. The API field (encryptionKeyURL) exists but the implementation is pending your team's decision on the mechanism.

The architecture is ready to support whichever approach ARO chooses (encryption scopes, account-level CMK, or client-side encryption) with localized changes to the Azure uploader. We'll coordinate once ARO has a defined approach.

[jparrill]

The credentialsSecretRef references a Secret in the HO namespace that is shared — it is a copy of the Velero BackupStorageLocation (BSL) credential, extracted from the OADP/Velero namespace by the plugin during backup. The same BSL credential is used for all HCPs that back up to the same storage location.

The flow is: the OADP plugin reads the BSL credential (either from bsl.spec.credential or falling back to the cloud-credentials Secret in the Velero namespace), copies it to the HO namespace, and sets the credentialsSecretRef on the HCPEtcdBackup CR pointing to that copy. This means the credential is not per-HCP — it's per-BSL, which in practice maps to a single shared credential for all hosted clusters on the management cluster.

The controller then auto-detects the credential type (static, STS/IRSA, Workload Identity) from the Secret content and configures the backup Job accordingly. See the Credential Auto-Detection section.

[Ajpantuso]

The more ergonomic usage for ROSA would be EtcdBackup resources being bound to the external S3 object state such that removal of an EtcdBackup would be blocked by a finalizer which allows the owning controller to cleanup the corresponding s3 object.

That would:

1. Eliminate this gc logic
2. Allow cascading deletion of EtcdBackups when the corresponding Velero Backup is deleted.
3. Leverage the Velero TTL mechanism for pruning s3 backups instead of having to apply external lifecycle policies
4. Make it easier to correlate Velero resource backups to the corresponding EtcdBackup

Obviously there would be raciness with cleanup if the upload credentials are removed before resources are cleaned up, but this is a similar issue in OADP/Velero and is unavoidable without some extended APIs for credential handling which is probably not worth the effort.

[Ajpantuso]

It would be nice to expand on this as some pre-checks should be made to check etcd cluster health prior to saving snapshots.

[jparrill]

Agreed — added in the latest revision. The controller now performs etcd health pre-checks (endpoint availability and quorum verification) before creating the backup Job. If the cluster is unhealthy, the CR is set to BackupCompleted=False with reason EtcdUnhealthy and the Job is not created. See the new "Etcd Health Pre-Checks" section.

[Ajpantuso]

From a ROSA perspective it would be ideal to check the KMS key availability prior to attempting restoration. We would want an early indication that the restoration would fail and encoding the encryptionMetadata with the backup would definitely help to expose that earlier.

[Ajpantuso]

@slopezz Please review for any observability like metrics/logging that you think should be called out.

[jparrill]

can we specify a recovery point goal (how fresh a backup can be) / SLO for the worst case scenario (management cluster a max HC capacity) and articulate how this proposal meets it

@enxebre ^^
The RPO (Recovery Point Objective) is not directly governed by this enhancement — this covers on-demand backups only, and scheduling is an OADP/Velero concern (Non-Goal #1). The freshness of a backup depends on when the OADP plugin triggers the EtcdBackup CR.

It's worth noting that the current backup method uses CSI snapshots of all etcd PVCs, which is significantly slower. This proposal replaces that with a single etcdctl snapshot save + upload, which should be considerably faster per backup.

That said, the serialization constraint (one backup Job at a time) does impact throughput on a management cluster at max HC capacity. This is a known trade-off of running the Job in the HO namespace (service network overhead). If this proves to be a problem during testing, we will evaluate solutions (e.g., controlled parallelism with a configurable concurrency limit), but that is initially out of scope for this enhancement.

[jparrill]

The more ergonomic usage for ROSA would be EtcdBackup resources being bound to the external S3 object state such that removal of an EtcdBackup would be blocked by a finalizer which allows the owning controller to cleanup the corresponding s3 object.

That would:...
...
...

Added to the design @Ajpantuso

[jparrill]

@Ajpantuso We have an open question that would benefit from ROSA SRE input:

Service Account for backup Jobs: The backup Job needs a ServiceAccount in the HO namespace with permissions to read Secrets/ConfigMaps from HCP namespaces (for fetching etcd TLS certs) and read/write access to cloud storage. Two options:

1. Extend the existing HO ServiceAccount — cloud storage credentials are already provisioned, only the IAM role needs to be extended with storage permissions.
2. Create a dedicated ServiceAccount (etcd-backup-sa) — minimal permissions, provisioned by the EtcdBackupReconciler.

From a ROSA operational perspective, do you have a preference? Option 1 is simpler but broadens the HO SA's scope. Option 2 follows least-privilege but adds provisioning complexity.

[slopezz]

Thanks @jparrill for this detailed enhancement. As requested by @Ajpantuso, I'm providing feedback on observability requirements, along with some architectural concerns based on my experience troubleshooting problems while working with OADP on ROSA-86.

Apologies if some of this is already addressed in the enhancement - I wanted to share current operational challenges since this new architecture is being built from scratch and we have the opportunity to address known issues from the beginning.

These are just my thoughts and suggestions - my 2 cents (maybe more like 20 cents at this point 😅). The final design decisions are up to the architects.

---

Observability Requirements

@Ajpantuso asked me to review observability aspects. Current CSI-based etcd backups expose minimal metrics:

• velero_csi_snapshot_attempt_total{schedule}
• velero_csi_snapshot_success_total{schedule}
• velero_csi_snapshot_failure_total{schedule}

CSI is essentially a black box - no visibility into snapshot duration, size, or internal phases. This new controller is an opportunity to instrument properly from the start.

Metrics

From an SRE perspective, these are the metrics I'd find useful for operating and troubleshooting the system. The final implementation is up to the architects, but this is what would help us:

Counters (low cardinality, essential for SLOs):

Metric	Labels	Why
hypershift_etcd_backup_attempt_total	schedule, hcp_namespace	Track total backup attempts. Matches existing Velero pattern.
hypershift_etcd_backup_success_total	schedule, hcp_namespace	Success count for SLO calculation.
hypershift_etcd_backup_failure_total	schedule, hcp_namespace	Failure count for alerting and SLO calculation.

Gauges (preferred over histograms to avoid cardinality explosion):

Metric	Labels	Why
hypershift_etcd_backup_last_duration_seconds	schedule, hcp_namespace, phase	Last backup duration per phase (snapshot, upload, total). In current OADP, we end up using "last duration" despite having histograms - gauges give us what we actually need without the cardinality cost.
hypershift_etcd_backup_last_size_bytes	schedule, hcp_namespace	Last backup size for capacity planning and anomaly detection.
hypershift_etcd_backup_last_success_timestamp	schedule, hcp_namespace	Unix timestamp of last successful backup. Essential for "backup too old" alerts.
hypershift_etcd_backup_in_progress	schedule, hcp_namespace	Know which specific backup is running. Detect stuck backups.
hypershift_etcd_backup_queue_depth		Early warning if serial processing becomes a bottleneck.
hypershift_etcd_backup_credentials_valid	hcp_namespace	Early detection of credential issues before backup attempts fail. 1=valid, 0=invalid.

Why gauges over histograms: Histograms with schedule × hcp_namespace × phase × bucket labels explode cardinality (64 HCPs × phases × histogram buckets = thousands of series per MC). The schedule label is critical - it enables correlation with existing Velero metrics and per-schedule SLO tracking.

Metric Lifecycle

There's an important bug in current Velero metrics: when an HC is deleted (along with its schedule), metrics persist indefinitely, showing stale data. The controller should implement metric garbage collection - when HC is deleted, stop reporting metrics for that hcp_namespace. This prevents dashboard pollution, misleading SLO calculations, and false positive alerts.

Velero Integration

This is critical for our backup SLO: will EtcdBackup CR status propagate to Velero's backup status and metrics?

• EtcdBackup failure must → Velero Backup status = Failed (not PartiallyFailed - etcd is critical, partial is not acceptable)
• Does EtcdBackup failure → velero_backup_failure_total increment?
• Does EtcdBackup success → velero_backup_success_total increment?

If EtcdBackup status doesn't propagate, existing dashboards and alerts won't reflect etcd backup health, requiring separate monitoring infrastructure.

Structured Logging

Key events to log with structured fields for debugging and audit:

Event	Fields	Why
Backup start	hcp_namespace, schedule, timestamp	Audit trail, duration calculation
Backup end	hcp_namespace, schedule, timestamp, result, duration_ms	Audit trail, debugging
Etcd health pre-check	hcp_namespace, healthy, endpoints_available, quorum_status	Debug pre-check failures
NetworkPolicy lifecycle	hcp_namespace, action (create/delete), result	Troubleshoot cross-namespace connectivity
Snapshot complete	hcp_namespace, size_bytes, duration_ms	Debug slow snapshots
Upload complete	hcp_namespace, size_bytes, duration_ms, storage_url	Debug slow uploads
KMS metadata captured	hcp_namespace, kms_enabled, kms_key_id	Restore validation
Errors	hcp_namespace, phase, error_message, retry_count	Actionable debugging

---

Architectural Concerns from Current OADP Experience

Based on my experience troubleshooting OADP issues, I want to share some context and raise a few concerns that might be relevant for this design.

Context: Evolution of Etcd Backup Approaches

We've been through multiple etcd backup approaches:

Approach	Speed	Compute Load	Storage Model	Cost	Portable
Kopia (past)	Slow	High CPU/mem/network	Full backups (compressed)	expensive full_etcd_size × HCs × freq × TTL	Yes
CSI snapshots (current)	Instant (AWS API call)	None	Incremental	cheap (full_size × HCs) + (delta_size × HCs × freq × TTL)	No
etcdctl (proposed)	Fast	CPU/mem/network	Full backups (compressed)	expensive full_etcd_size × HCs × freq × TTL	Yes

We moved from Kopia to CSI because Kopia was slow, resource-intensive, and caused many operational problems. Kopia does filesystem-level backups which should be incremental, but etcd stores everything in a single bbolt file - so every backup was effectively a full backup.

CSI is instant (just an AWS API call), has no compute overhead, and uses true incremental storage at block level. The problem is portability - CSI snapshots are tied to the AWS account.

This proposal trades CSI's speed and storage efficiency for portability. etcdctl snapshots are fast (native etcd operation) but still:

• Slower than CSI (requires actual data processing vs instant API call)
• Requires compute resources (CPU/mem/network during backup and upload)
• Produces full backups to S3 (compressed, but still full)
• Adds sequential processing risk if controller is not parallelized
• Requires complex coordination (cross-namespace access, temporary NetworkPolicy, credential management, HO controller orchestration)

1. Serial Processing Bottleneck

The enhancement states "only one backup at a time is allowed across the controller." This concerns me given our RPO targets.

Desired goal: 1h RPO
Worst case scenario: 64 HCPs per MC

Math:

• 64 backups/hour needed
• At 2 min/backup: 128 min/hour of work required
• Impossible with serial execution

Current CSI snapshots use instant AWS API calls + multiple controller replicas. This design loses both advantages.

OADP 1.6 context: Current OADP processes backups sequentially, which is problematic - even 6h RPO is difficult due to long backup execution times. OADP 1.6 introduces parallelism with configurable concurrent backups. With OADP 1.6, each Velero controller (multiple in parallel) will:

1. Process backup items (Velero pod)
2. Create EtcdBackup CR and wait for completion (HO controller)
3. Both must complete before proceeding to next backup

If the HO controller remains serial, it becomes the bottleneck. Multiple Velero controllers will queue EtcdBackup CRs waiting for the single HO controller. We need parallelism in the HO controller from the beginning to ensure Velero doesn't wait excessively and EtcdBackup CRs don't accumulate.

What's the parallelization path for the HO controller?

2. HC Deletion and Backup Lifecycle

In current OADP architecture, when an HC is deleted, the HCP namespace deletion takes BSL/credentials with it. Velero then fails to honor TTL for existing backups (no S3 access), causing constant failed deletion attempts and orphan AWS resources. This is being addressed in OCM-22239.

For this design, I'd like clarification on credential relationships:

• Cloud storage credentials (HO namespace) - for uploading etcd snapshots
• Velero BSL credentials - for Velero S3 access
• etcd TLS credentials (HCP namespace) - for accessing etcd
• Are cloud storage creds and BSL creds the same or different?
• Which credentials are needed for restore after HC deletion?

When HC is deleted before backup TTL expires, backups must remain accessible and credentials must remain available to honor TTL cleanup. Who owns S3 object deletion - EtcdBackup finalizer or Velero TTL mechanism which would need valid credentials?

Recommendation: Add a "Hosted Cluster Deprovisioning" section documenting credential inventory, what remains after HC deletion, and S3 cleanup ownership.

3. Restore: How Does Velero Know Which Etcd Snapshot to Use?

As far as I know, with current CSI snapshots, VolumeSnapshot metadata is stored inside the backup tarball in S3. During restore, Velero reads this metadata to know which snapshots to restore.

With the new EtcdBackup CR, the snapshotURL lives in the CR status (HCP namespace), and the CR is ephemeral (may be deleted via TTL, HC deletion, etc.).

How does Velero associate the snapshotURL with the backup for restore purposes?

• Is the EtcdBackup CR (or its status) persisted in the Velero backup tarball?
• Is there a separate mechanism to store the snapshotURL in Velero's backup metadata?
• If the EtcdBackup CR is deleted before restore, how does Velero find the etcd snapshot?

This connects to the HC deletion topic above: I guess the main reason why currently we need to preserve backups after HC deletion is for restore scenarios. If an HC is deleted (along with its namespace and all resources), and we later need to restore it:

• The HCP namespace is gone
• The EtcdBackup CR is gone
• The BSL/credentials may be gone (theoretically addressed by OCM-22239)
• How does restore succeed in this scenario?

4. Cross-MC Restore Consideration

While cross-MC restore is out of scope in this enhancement, it's being actively worked on.

In current OADP architecture, each MC has its own S3 bucket and credentials. If an HC needs to be restored on a different MC than where it was backed up, the target MC needs access to the source MC's backup storage.

Worth considering whether the design accommodates this scenario, to avoid rework later.

[Ajpantuso]

@Ajpantuso We have an open question that would benefit from ROSA SRE input:

Service Account for backup Jobs: The backup Job needs a ServiceAccount in the HO namespace with permissions to read Secrets/ConfigMaps from HCP namespaces (for fetching etcd TLS certs) and read/write access to cloud storage. Two options:

1. **Extend the existing HO ServiceAccount** — cloud storage credentials are already provisioned, only the IAM role needs to be extended with storage permissions.

2. **Create a dedicated ServiceAccount** (`etcd-backup-sa`) — minimal permissions, provisioned by the `EtcdBackupReconciler`.

From a ROSA operational perspective, do you have a preference? Option 1 is simpler but broadens the HO SA's scope. Option 2 follows least-privilege but adds provisioning complexity.

@jparrill The best experience for ROSA would be the ability for us to provide a ServiceAccount which the reconciler will then apply to backup jobs. That would allow us full control of the ServiceAccount and any related kube RBAC reducing the need for further code changes should security concerns be raised.

Other customers might benefit from a simpler experience (like in non-prod environments) so Option 1 or 2 could still be relevant if no ServiceAccount is configured by the customer.

[wgordon17]

OCPSTRAT-2802 references "Minimize Pause Duration", however I don't see that addressed in this enhancement? Would that be handled/addressed separately in CNTRLPLANE-2676 as part of the plugin implementation?

[wgordon17]

@jparrill This is intended to reference a single secret in HO namespace for *all HCP's on the management cluster? Or this is a named reference to a secret, that we (as the management service) can configure to be per-HCP cluster?

[wgordon17]

Would it be the OADP plugin that's responsible for orchestrating the usage of the RestoreSnapshotURL spec field during restoration?

[wgordon17]

Given that this enhancement is focused solely on the snapshot and upload mechanism, is it safe to assume that backup credentials can be write-only? And that read credentials would be provided via OADP for usage by the plugin?

[jparrill]

Yes, the OADP plugin orchestrates the RestoreSnapshotURL usage during restoration. The flow is:

1. During backup, the plugin stores the snapshotURL from the HCPEtcdBackup CR status as an annotation on the HostedControlPlane/HostedCluster objects in the Velero tarball.
2. During restore, the plugin's RestoreItemAction reads the annotation, generates a presigned URL (S3 presigned GET or Azure Blob SAS), and injects it into spec.etcd.managed.storage.restoreSnapshotURL.
3. The existing HyperShift restore machinery picks up the URL and restores etcd from the snapshot.

This is implemented in openshift/hypershift-oadp-plugin#247.

[wgordon17]

While this enhancement lists the plugin implementation as a Non-Goal, can we assume that CNTRLPLANE-2676 will still encompass the necessary work for the plugin to utilize this new enhancement?

[jparrill]

Yes — CNTRLPLANE-2676 encompasses all the plugin work. The OADP HyperShift plugin implementation is in openshift/hypershift-oadp-plugin#247, which integrates the HCPEtcdBackup CR lifecycle (create, poll, cleanup) and handles the restore flow (presigned URLs for RestoreSnapshotURL injection).

Regarding "Minimize Pause Duration" from OCPSTRAT-2802: this enhancement is the solution. The previous backup approach required pausing HostedCluster/NodePool reconciliation during CSI volume snapshots (20-30 min per cycle). With HCPEtcdBackup, the backup Job runs independently without pausing reconciliation. Future considerations around pausing during restore (CAPI node re-adoption) are out of scope and will be addressed separately.

[enxebre]

probably good opportunity to start using omitzero instead

[jparrill]

Done — updated S3 and AzureBlob fields to use omitzero instead of omitempty.

[enxebre]

thanks, lgtm overall

[enxebre]

api: this don't need to be a pointer unless we want to differentiate "" from nil

[jparrill]

Agreed. These fields use string with omitempty. Since empty string and "not set" are semantically equivalent for these optional fields (an empty KMS ARN means "no KMS encryption"), we don't need to distinguish "" from nil. The current approach is simpler and works correctly with the CEL validation rules.

[enxebre]

api: this don't need to be a pointer unless we want to differentiate "" from nil

[jparrill]

Same reasoning as above — string with omitempty is sufficient here since empty string and "not set" are semantically equivalent for this optional encryption key URL field.

[tony-schndr]

Overall this looks ok for ARO-HCP.

[tony-schndr]

@jparrill can you elaborate on the encryption mechanism for Azure? Will it be client side or server side encryption? If encryption is done server side I assume it will be via encryption scopes?

[openshift-bot]

Inactive enhancement proposals go stale after 28d of inactivity.

See https://github.com/openshift/enhancements#life-cycle for details.

Mark the proposal as fresh by commenting /remove-lifecycle stale.
Stale proposals rot after an additional 7d of inactivity and eventually close.
Exclude this proposal from closing by commenting /lifecycle frozen.

If this proposal is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale enhancement proposals rot after 7d of inactivity.

See https://github.com/openshift/enhancements#life-cycle for details.

Mark the proposal as fresh by commenting /remove-lifecycle rotten.
Rotten proposals close after an additional 7d of inactivity.
Exclude this proposal from closing by commenting /lifecycle frozen.

If this proposal is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[jparrill]

/remove-lifecycle rotten

[kaovilai]

Suggested change

	1. The OADP plugin (running as a Velero pre-hook or standalone pod) creates an `HCPEtcdBackup` CR in the HCP namespace. The CR spec includes the cloud storage configuration (e.g., S3 bucket or Azure Blob container) and a reference to a credentials Secret in the HO namespace.
	1. The HyperShift OADP plugin (running as a Velero BackupItemAction) creates an `HCPEtcdBackup` CR in the HCP namespace. The CR spec includes the cloud storage configuration (e.g., S3 bucket or Azure Blob container) and a reference to a credentials Secret in the HO namespace.

[jparrill]

Acknowledged — will update the PR title and rename the enhancement file to reflect HCPEtcdBackup. This aligns with the CRD rename that's already landed in the implementation.

[kaovilai]

from ai assisted research of relevant PRs this change intends to write to {bsl-prefix}/backups/{name}/etcd-backup/12345.db
which is the path used by velero/oadp to store backups. Keep in mind that velero have not documented officially that writing file here is supported usecase, and future velero versions can implicitly break this, tho (as a velero maintainer) I can try vouch to prevent that.

Currently also relies on velero maintaining following delete dir logic (ListObjects as a bonus captures ../etcd-backup/...)

  DeleteBackup Logic (object_store.go:499-516)                                                                                                                       
                                                                                                                                                                     
  func (s *objectBackupStore) DeleteBackup(name string) error {                                                                                                      
      objects, err := s.objectStore.ListObjects(s.bucket, s.layout.getBackupDir(name))
      // ...
      for _, key := range objects {                                                                                                                                  
          s.objectStore.DeleteObject(s.bucket, key)
      }                                                                                                                                                              
  }

Should this change, deletion would be another risk area.

Another consideration is future implementation of velero support for object lock enabled buckets
velero-io/velero#8686 we have not nailed down the design there yet. But I would assume that any unexpected writes in /backups/ could interfere potentially this area.

[jparrill]

Good concern. The path construction is handled entirely by the OADP plugin — the HCPEtcdBackup controller is agnostic to the key structure and just uses whatever keyPrefix is provided in the CR spec.

The plugin currently constructs the key as {bsl-prefix}/backups/{backup-name}/etcd-backup/{timestamp}.db, which places the snapshot inside Velero's backup directory. This gives us cascading deletion (Velero's DeleteBackup does ListObjects on the backup directory and removes everything, including our subdirectory) and natural correlation between the Velero resource backup and the etcd snapshot.

The trade-off is the dependency on Velero's directory layout. If this is a concern, the plugin can be updated to use a separate path outside /backups/ — that change would be isolated to the plugin with no impact on the controller or CRD.

For now, given your offer to help prevent breaking changes upstream, we think the current approach is the pragmatic choice. We'll document this dependency in the enhancement.

[kaovilai]

We can also code it into velero requirements so it doesn't change in the future etc. If you think that fits better, please create issue in velero repo

[kaovilai]

Created issue and got feedback here.. velero-io/velero#9837 (comment) Please review their concern and let me know how you would like to proceed.

[kaovilai]

Given Lyndon's feedback that the object store layout should remain private with no compatibility guarantee, one pattern to consider is what the kubevirt-datamover-controller does — it uses a separate sibling prefix instead of writing inside Velero's backups/ directory:

# Current (inside Velero's tree):
{bsl-prefix}/backups/{backup-name}/etcd-backup/snapshot.db

# Kubevirt datamover pattern (separate sibling prefix):
{bsl-prefix}-kubevirt-datamover/checkpoints/{ns}/{vm}/...
{bsl-prefix}-kubevirt-datamover/manifests/{backup}/...

For HyperShift etcd backups, this could look like:

{bsl-prefix}-hypershift-etcd/{backup-name}/{timestamp}.db

The key construction in the plugin (orchestrator.go L207-210) would change from:

keyPrefix := fmt.Sprintf("backups/%s/etcd-backup", backupName)

to something like:

keyPrefix := fmt.Sprintf("%s-hypershift-etcd/%s", bslPrefix, backupName)

Since jparrill confirmed the HCPEtcdBackup controller is storage-agnostic (just uses whatever keyPrefix the plugin provides), this is a plugin-only change.

Trade-off: You lose Velero's free cascading deletion via ListObjects on the backup prefix. You'd need either:

1. A DeleteItemAction (DIA) plugin to clean up the etcd snapshot when a Velero backup is deleted, or
2. S3 lifecycle rules with object tagging (which the team already discussed as a safety net), or
3. Reference tracking similar to what kubevirt-datamover does with referencedBy in its index.json

This aligns with Lyndon's recommended architecture: use BIA/RIA/DIA interfaces, manage your own data lifecycle, don't depend on Velero's internal directory layout.

Note

Responses generated with Claude

[kaovilai]

Keep in mind that velero themselves will still not work with object locks etc, so the conditional write must only be on the HCPEtcdBackup originated items, and object lock cannot be enabled at the bucket level (yet)

velero-io/velero#8686

[jparrill]

Two separate topics here:

Object Lock compatibility: The HCPEtcdBackup controller is compatible with Object Lock by design — the backup Job only performs PutObject operations, never DeleteObject. If the bucket has Object Lock enabled, our objects will comply with the retention policy. S3 object lifecycle management is delegated to the storage backend's native lifecycle rules.

Conditional writes (If-None-Match): We do not support If-None-Match on S3 because we use the AWS SDK v2 Transfer Manager for uploads, which handles multipart uploads internally and does not expose conditional write headers in its API. For Azure Blob, we do implement conditional writes via IfNoneMatch: ETagAny.

Overwrite prevention on S3 is mitigated at the application level: the upload key includes a Unix timestamp ({keyPrefix}/{timestamp}.db), making key collisions practically impossible. Combined with S3 Versioning on the bucket, even an accidental overwrite would preserve the previous version.

We'll document this clearly in the enhancement: Object Lock compatible, If-None-Match not viable on S3 due to Transfer Manager SDK limitation, overwrite risk mitigated by naming convention.

[kaovilai]

Sounds good.

[kaovilai]

Please retitle this PR and files to match HCPEtcdBackup changes.

[jparrill]

Implementation Status Update

Since the last round of reviews, significant implementation work has landed. Here's a summary of what's been merged and what's in progress:

Merged (14 PRs in openshift/hypershift):

• HCPEtcdBackup CRD API with feature gate (#7898)
• HCPEtcdBackup controller in the HyperShift Operator (#8139)
• CPO subcommands: fetch-etcd-certs (#8010), etcd-backup (#8040), etcd-upload (#8017)
• lastSuccessfulEtcdBackupURL propagation to HC status (#8179)
• Envtest validation suites (#8199)
• --use-etcd-snapshot flag for OADP CLI (#8232)
• E2E test for etcd snapshot backup on AWS (#8231)
• Tech Preview documentation (#8299)
• S3 multipart upload fix (#8114)

Open PRs:

• CNTRLPLANE-3167: Auto-detect credential type for HCPEtcdBackup hypershift#8368 — Credential auto-detection for IRSA/STS (AWS) and Workload Identity (Azure)
• CNTRLPLANE-3167: support STS/IRSA credentials and standalone Velero hypershift-oadp-plugin#247 — Plugin-side STS/IRSA support and standalone Velero credential fallback

Key decisions implemented:

• CRD renamed to HCPEtcdBackup (as discussed with @lance5890)
• Job runs in HO namespace with etcd-backup-job ServiceAccount
• Per-tenant KMS encryption implemented for AWS S3 (spec.s3.kmsKeyARN → SSE-KMS)
• Etcd health pre-checks before Job creation (StatefulSet readiness validation)
• Retention logic with configurable count + period
• .spec.ttlSecondsAfterFinished on backup Jobs
• omitzero for struct fields, omitempty for scalars

---

@kaovilai RE: PR title and filename rename — Acknowledged, will update the PR title and rename the enhancement file to reflect HCPEtcdBackup. This aligns with the CRD rename that's already landed in the implementation.

---

@wgordon17 RE: "Minimize Pause Duration" from OCPSTRAT-2802 — This enhancement is the solution for minimizing pause duration. The previous backup approach required pausing HostedCluster/NodePool reconciliation during CSI volume snapshots, which blocked day-two operations for 20-30 minutes per backup cycle.

With HCPEtcdBackup, the backup Job runs independently — no reconciliation pausing is needed. The controller creates a Job that takes an etcd snapshot and uploads it to cloud storage without interfering with the HostedCluster or NodePool controllers.

Future considerations around pausing during restore (specifically for CAPI node re-adoption) are out of scope for this enhancement and will be addressed separately.

I'm responding to the remaining open threads inline below.

[openshift-ci]

@jparrill: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[kaovilai]

Can you clarify intended timelines and target branches that will ship this functionality? Jira indicates hypershift plugin shipped during 4.22 is expected to have this. All while managed openshfit are expected to implement/use in 5.0?

[kaovilai]

Created issue and got feedback here.. velero-io/velero#9837 (comment) Please review their concern and let me know how you would like to proceed.

[enxebre]

A fleet wide secret that accesses every cluster's backup is a single point of compromise with no compensating security benefit from being in the HO namespace vs any other namespace

Also all managed services has expressed desire to converge with #2004

Based on the above I propose the following changes to this proposal and #2004:

1. HCPEtcdBackup becomes the universal one shot backup primitive

• Runs in the HCP namespace (where etcd and PKI secrets already live, no cross-namespace access needed. No svc network usage for transfer backup)
• Produces a complete restorable artifact: etcd snapshot + root-ca + etcd-signer + sa-signing-key + encryption key
• Supports workload identity auth via a clientID style API knobs (Azure managed identity, GCP SA email, AWS role ARN) and a secret ref like for self hosted scenarios. No fleet-wide static secrets
• Any consumer today (ARO, ROSA, GCP, self-managed) can create an HCPEtcdBackup CR directly for on-demand backups

2. Enhancement PR 2004 uses HCPEtcdBackup as an implementation detail for scheduled backups

• The automatedBackup field on ManagedEtcdSpec configures a CronJob that creates HCPEtcdBackup CRs on a schedule
• The CronJob doesn't do the backup itself. It creates the CR, and the existing HCPEtcdBackup reconciler handles execution

3. Consumers own the per-HC cloud identity lifecycle

• Each HostedCluster gets its own cloud identity (managed identity / GCP SA / IAM role) scoped to its own backup storage
• The managed service layer (ARO-HCP, ROSA, GCP-HCP) builds automation to create these per-HC on cluster creation
• HyperShift provides the API knob; the consumer provides the identity