feat: automate isolated etcd snapshots

[boxp]

Summary

• add a daily etcd snapshot workflow with status checking, encrypted S3 isolation, and non-production restore validation
• fetch only verified snapshots to the backup controller and retain them under a dedicated S3 lifecycle prefix
• document recovery RPO/RTO, approval boundaries, restore, and rollback

Validation

• git diff --check
• unable to run actionlint, ansible-lint, Molecule, or Terraform locally because the workspace image does not include actionlint, uv/Ansible, or Terraform; CI is configured to run the Ansible checks

Closes BOXP-180

[boxp-tfaction]

Plan Result (terraform/aws/github-actions-ansible)

CI link

Plan: 0 to add, 2 to change, 0 to destroy.

• Update
  • aws_iam_policy.etcd_snapshot_s3_write
  • aws_s3_bucket_lifecycle_configuration.etcd_snapshots

Change Result (Click me)

  # aws_iam_policy.etcd_snapshot_s3_write will be updated in-place
  ~ resource "aws_iam_policy" "etcd_snapshot_s3_write" {
        id               = "arn:aws:iam::839695154978:policy/GitHubActions_Ansible_EtcdSnapshotS3Write"
        name             = "GitHubActions_Ansible_EtcdSnapshotS3Write"
      ~ policy           = jsonencode(
          ~ {
              ~ Statement = [
                    {
                        Action   = [
                            "s3:GetBucketLocation",
                        ]
                        Effect   = "Allow"
                        Resource = "arn:aws:s3:::arch-etcd-snapshots"
                    },
                  ~ {
                      ~ Condition = {
                          ~ StringLike = {
                              ~ "s3:prefix" = [
                                    "kubernetes-upgrade/*",
                                  + "scheduled/*",
                                ]
                            }
                        }
                        # (3 unchanged attributes hidden)
                    },
                  ~ {
                      ~ Resource = "arn:aws:s3:::arch-etcd-snapshots/kubernetes-upgrade/*" -> [
                          + "arn:aws:s3:::arch-etcd-snapshots/kubernetes-upgrade/*",
                          + "arn:aws:s3:::arch-etcd-snapshots/scheduled/*",
                        ]
                        # (2 unchanged attributes hidden)
                    },
                ]
                # (1 unchanged attribute hidden)
            }
        )
        tags             = {}
        # (7 unchanged attributes hidden)
    }

  # aws_s3_bucket_lifecycle_configuration.etcd_snapshots will be updated in-place
  ~ resource "aws_s3_bucket_lifecycle_configuration" "etcd_snapshots" {
        id                                     = "arch-etcd-snapshots"
        # (4 unchanged attributes hidden)

      + rule {
          + id     = "expire-scheduled-etcd-snapshots"
          + status = "Enabled"
            # (1 unchanged attribute hidden)

          + abort_incomplete_multipart_upload {
              + days_after_initiation = 7
            }

          + expiration {
              + days                         = 30
              + expired_object_delete_marker = false
            }

          + filter {
              + prefix = "scheduled/"
            }

          + noncurrent_version_expiration {
              + noncurrent_days = 30
            }
        }

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

[github-actions]

Ansible Plan Results

Mode: --check --diff (dry run)

⚠️ Changes detected

---

golyat-4: worker-image

Host	OK	Changed	Skipped	Failed	Unreachable
golyat-4	86	1	55	0	0

1 changed

Changed Tasks (1)

#	Task	Module
1	user_management : Update package cache	unknown

shanghai-1: control-plane

Host	OK	Changed	Skipped	Failed	Unreachable
shanghai-1	106	1	37	0	0

1 changed

Changed Tasks (1)

#	Task	Module
1	user_management : Update package cache	unknown

shanghai-1: node-shanghai-1

Host	OK	Changed	Skipped	Failed	Unreachable
shanghai-1	9	0	0	0	0

No changes

shanghai-2: control-plane

Host	OK	Changed	Skipped	Failed	Unreachable
shanghai-2	106	1	37	0	0

1 changed

Changed Tasks (1)

#	Task	Module
1	user_management : Update package cache	unknown

shanghai-2: node-shanghai-2

Host	OK	Changed	Skipped	Failed	Unreachable
shanghai-2	9	0	0	0	0

No changes

shanghai-3: control-plane

Host	OK	Changed	Skipped	Failed	Unreachable
shanghai-3	106	1	37	0	0

1 changed

Changed Tasks (1)

#	Task	Module
1	user_management : Update package cache	unknown

shanghai-3: node-shanghai-3

Host	OK	Changed	Skipped	Failed	Unreachable
shanghai-3	9	0	0	0	0

No changes

---

Plan executed on all nodes in parallel.