Cluster API Provider OTC

Infrastructure provider for Open Telekom Cloud.

Warning

Evaluation project — not for production use. This provider is an evaluation/showcase project and was mostly generated by AI (with human review and live end-to-end testing against OTC). Expect rough edges, no compatibility guarantees, and no maintenance commitments. Use at your own risk.

Quick Start

Everything installs from this repo's public resources — no clone, no build. You need: a management cluster (any local one works, e.g. k3d cluster create capi-mgmt), clusterctl ≥ v1.11, kubectl, and OTC AK/SK credentials.

Caution

Creating a workload cluster provisions real OTC resources (ECS, EIP, NAT, ELB) that cost money until the cluster is deleted.

1. Install the provider — register it once in ~/.cluster-api/clusterctl.yaml (the name otc is not in clusterctl's built-in registry):

providers:
  - name: otc
    url: https://github.com/d-velop/cluster-api-provider-otc/releases/v0.1.0/infrastructure-components.yaml
    type: InfrastructureProvider

clusterctl init --infrastructure otc:v0.1.0
# installs cert-manager + CAPI core + this provider (namespace capotc-system)

2. Create the credentials Secret, in the namespace where the cluster will live (cross-namespace identityRef is not supported):

kubectl create secret generic otc-credentials \
  --from-literal=accessKey=YOUR_AK \
  --from-literal=secretKey=YOUR_SK \
  --from-literal=domainName=OTC-EU-DE-XXXXX
# The move label keeps the Secret pivotable by `clusterctl move`:
kubectl label secret otc-credentials clusterctl.cluster.x-k8s.io/move=""

3. Package the Cilium CNI manifest as a ConfigMap on the management cluster (a ClusterResourceSet applies it to the workload cluster; it is never applied to the management cluster itself):

curl -fsSLo cilium.yaml https://raw.githubusercontent.com/d-velop/cluster-api-provider-otc/v0.1.0/config/samples/addon-cilium-cni.yaml
kubectl create configmap cilium-cni --from-file=cilium.yaml

4. Create the cluster with the default flavor (EIP-fronted control plane + NAT gateway + 1 worker):

export OTC_REGION=eu-de
export OTC_AVAILABILITY_ZONE=eu-de-01
export OTC_CONTROL_PLANE_MACHINE_FLAVOR=c3.large.2
export OTC_NODE_MACHINE_FLAVOR=c3.large.2
export OTC_IMAGE_ID=5e65b2e7-1237-4866-b1c1-5cbd5702ab57  # Standard_Ubuntu_22.04_latest in eu-de

clusterctl generate cluster my-cluster --infrastructure otc:v0.1.0 \
  --kubernetes-version v1.34.9 \
  --control-plane-machine-count 1 --worker-machine-count 1 | kubectl apply -f -

5. Watch it come up (~10 minutes to Ready), then use it:

clusterctl describe cluster my-cluster
kubectl get secret my-cluster-kubeconfig -o jsonpath='{.data.value}' | base64 -d > my-cluster.kubeconfig
KUBECONFIG=my-cluster.kubeconfig kubectl get nodes

6. Clean up — this deletes the OTC resources the provider created:

# If the OTC CCM runs in the cluster: delete type=LoadBalancer Services
# FIRST — the CCM must remove the ELBs it created; this provider doesn't
# know about them.
kubectl delete cluster my-cluster

Verify in the OTC console that everything is gone.

Flavors

All flavor templates ship as release assets, so clusterctl generate cluster --flavor <name> works directly:

Flavor	Shape
(default)	EIP control plane + NAT + workers
elb	shared-ELB-fronted control plane
elb-ha	3 control planes across 3 AZs behind a shared ELB
external-ccm	cloud-provider=external + OTC CCM (see below)
flatcar	Flatcar/Ignition node image
topology / topology-ha	ClusterClass instances (see below)

Each template's header comment lists its exact required OTC_* variables (clusterctl errors listing any that are missing; they can also live in ~/.cluster-api/clusterctl.yaml). Notes:

• flatcar takes OTC_FLATCAR_IMAGE_NAME (version-coupled: must match --kubernetes-version) instead of OTC_IMAGE_ID, and needs the provider initialized with EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION=true clusterctl init.
• topology/topology-ha take no OTC variables at all (knobs live in the ClusterClass). Apply their ClusterClass first (clusterclass-otc-default.yaml / clusterclass-otc-ha.yaml, shipped as release assets) and enable CLUSTER_TOPOLOGY=true on CAPI core. See docs/clusterclass.md.
• Every cluster needs the otc-credentials Secret in its namespace (Quick Start step 2).

Security groups

Every machine defaults into the cluster's <cluster>-controlplane and <cluster>-worker security groups, with one exception: worker machines using the built-in default security groups (i.e. OTCCluster.Spec.Network.SecurityGroups unset) skip the control-plane one — nothing on a worker needs its etcd (2379-2380) or KCP-health-kubelet (10250) rules. Control-plane machines, explicit per-machine OTCMachineSpec.SecurityGroups overrides, and clusters using a custom Spec.Network.SecurityGroups list are all unaffected. See docs/design/2026-07-09-worker-sg-least-privilege-design.md for the full rationale.

Autoscaling

Worker pools scale via cluster-autoscaler (its clusterapi provider) over MachineDeployment — no provider config needed. Annotate the MachineDeployment with node-group min/max size (and capacity annotations for scale-from-zero). See docs/autoscaling.md and config/samples/workers-autoscaler.yaml. (We deliberately do not ship OTCMachinePool, following CAPO — see the doc for why.)

Metrics

The controller exposes Prometheus metrics on the manager's metrics endpoint (enabled via config/default). Alongside the built-in controller_runtime_* series, the provider emits:

• otc_api_request_duration_seconds{method,service,status_class} — OTC API call latency. The _count series where status_class is 4xx/5xx/error gives the OTC API error count.
• otc_ecs_job_duration_seconds{operation,result} — async ECS create/delete job latency.

A ServiceMonitor (config/prometheus) is provided but opt-in: uncomment the PROMETHEUS sections in config/default/kustomization.yaml. It requires the prometheus-operator CRDs.

External-mode (OTC CCM) flavor

The external-ccm flavor is a cloud-provider=external variant of the default flavor: nodes are provisioned tainted uninitialized, and the OTC cloud-controller-manager (from the sibling cloud-provider-otc repo) initializes them instead of the in-tree/no-op path. providerID is written to /etc/default/kubelet (KUBELET_EXTRA_ARGS) from the OTC instance metadata before kubelet starts, so the CCM resolves the node via its providerID.

The CCM addon is delivered via config/samples/addon-otc-ccm.yaml, a ClusterResourceSet selecting ccm: otc (the label set on the Cluster in the external-ccm flavor).

With the CCM running, Service type=LoadBalancer works: the CCM provisions an OTC ELB + EIP per Service and removes them when the Service is deleted (verified live 2026-07-08 — public IP assigned in ~20 s). Delete LoadBalancer Services before deleting the cluster, or the CCM-created ELB leaks (this provider only tears down the infrastructure it created itself).

Prerequisites, on the management cluster, before applying either file:

1. The OTC CCM image published to a registry the workload cluster's nodes can pull from (e.g. ghcr.io/d-velop/otc-cloud-controller-manager:v0.33.3).
2. The otc-ccm ConfigMap created out-of-band — see the comments in config/samples/addon-otc-ccm.yaml for how to render and create it. With CCM chart ≥ 0.4.0 this ConfigMap is cluster-agnostic (no per-cluster values), so it only needs refreshing on a CCM version bump.

The workload cluster's otc-cloud-config Secret (kube-system, cloud.conf with credentials, neutron subnet/network IDs, and cluster name) is written automatically by this provider's OTCCluster controller for any Cluster labeled ccm: otc, once its control plane is reachable — no manual otc-ccm-cloudconfig Secret is needed. Progress is tracked in the CCMConfigReady condition on the OTCCluster; the write happens once per cluster (re-render requires clearing that condition). A manually managed otc-cloud-config Secret in a labeled cluster will be overwritten — remove the ccm: otc label to opt out. (Pre-0.4.0 CCM charts, or clusters without the label, still need the manual Secret described in config/samples/addon-otc-ccm.yaml.)

kubectl apply -f https://raw.githubusercontent.com/d-velop/cluster-api-provider-otc/v0.1.0/config/samples/addon-otc-ccm.yaml
# with the OTC_* variables from Quick Start step 4:
clusterctl generate cluster my-ccm-cluster --infrastructure otc:v0.1.0 --flavor external-ccm \
  --kubernetes-version v1.34.9 --worker-machine-count 1 | kubectl apply -f -

CSI driver (dynamic EVS PersistentVolumes)

The OTC-native CSI driver (sibling csi-driver-otc repo) is delivered the same way as the CCM: config/samples/addon-otc-csi.yaml, a ClusterResourceSet selecting csi: otc (also set on the Cluster in the external-ccm flavor — it needs no cloud-provider=external node bootstrap of its own, so it works on any cluster, but ships alongside the CCM in the sample since both need the same otc-credentials principal).

The CSI driver's csi-otc-cloud-config Secret (kube-system, [Global]-only cloud.conf) is written automatically, the same way as the CCM's — for any Cluster labeled csi: otc, tracked by the CSIConfigReady condition on the OTCCluster. It is a distinct Secret name from the CCM's otc-cloud-config on purpose: sharing one Secret between the two charts overwrites the CCM's [LoadBalancer] section (discovered live during csi-driver-otc verification).

E2E-verified 2026-07-09: dynamic PVC provisioning honors WaitForFirstConsumer AZ topology, attach/mount/write works, and PVC deletion removes the backing EVS volume. See csi-driver-otc's README for the required additional IAM grants on otc-credentials (EVS access, ecs:serverVolumeAttachments:*) — non-obvious, cost real live-debugging time to find.

kubectl apply -f https://raw.githubusercontent.com/d-velop/cluster-api-provider-otc/v0.1.0/config/samples/addon-otc-csi.yaml

Development

Everything below is only relevant when working on the provider itself.

Dev loop (local image into a k3d management cluster)

k3d cluster create capi-mgmt --api-port 6550
clusterctl init   # CAPI core + cert-manager; no infrastructure provider

make docker-build IMG=localhost/cluster-api-provider-otc:dev
k3d image import localhost/cluster-api-provider-otc:dev -c capi-mgmt
make deploy IMG=localhost/cluster-api-provider-otc:dev   # installs into capotc-system

cert-manager is required. The provider's validating webhooks (OTCCluster, OTCMachine) use a cert-manager-issued serving cert with CA injection. clusterctl init installs cert-manager automatically. For a bare make deploy/manual install, install cert-manager first and wait for it to be Ready — otherwise the Certificate/ValidatingWebhookConfiguration apply fails with failed calling webhook.

Cluster shapes live in templates/ (the single source; flavor templates copied into releases). Add-ons stay in config/samples/: addon-cilium-cni.yaml, addon-otc-ccm.yaml, addon-otc-csi.yaml, workers-autoscaler.yaml. hack/dev-env.sh exports the OTC_* defaults used throughout this README:

. hack/dev-env.sh
clusterctl generate cluster test-otc-cp --from templates/cluster-template.yaml \
  --kubernetes-version v1.34.9 --worker-machine-count 1 | kubectl apply -f -

Installing an unreleased build via clusterctl

To test the clusterctl install path without a GitHub release, use a local overrides directory:

# 1. Generate artifacts (IMG must be an image the mgmt cluster can pull)
make release-manifests IMG=localhost/cluster-api-provider-otc:dev

# 2. Drop them into the clusterctl overrides layout
mkdir -p ~/.cluster-api/overrides/infrastructure-otc/v0.1.0
cp out/* ~/.cluster-api/overrides/infrastructure-otc/v0.1.0/

# 3. Point the provider entry in ~/.cluster-api/clusterctl.yaml at it
#    providers:
#      - name: otc
#        url: ~/.cluster-api/overrides/infrastructure-otc/v0.1.0/infrastructure-components.yaml
#        type: InfrastructureProvider

# 4. Install
clusterctl init --infrastructure otc:v0.1.0

Releasing

Releases are built entirely by CI (.github/workflows/release.yml): pushing a v* tag builds and pushes ghcr.io/d-velop/cluster-api-provider-otc:<tag>, regenerates the clusterctl artifacts for the tagged commit, and creates the GitHub Release with them attached.

git tag -a v0.2.0 -m "infrastructure-otc v0.2.0"
git push origin v0.2.0