Skip to content

Commit 67dc025

Browse files
LogicalSharkLogicalShark
committed
Add GCP CCM quickstart for kops and local
Signed-off-by: LogicalShark <maralder@google.com>
1 parent fa9f875 commit 67dc025

File tree

5 files changed

+483
-1
lines changed

5 files changed

+483
-1
lines changed

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,3 +9,4 @@ _rundir/
99
_tmp/
1010
/bin/
1111
__pycache__/
12+
/clusters/

‎README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,9 @@
1111
This repository implements the [cloud provider](https://github.com/kubernetes/cloud-provider) interface for [Google Cloud Platform (GCP)](https://cloud.google.com/).
1212
It provides components for Kubernetes clusters running on GCP and is maintained primarily by the Kubernetes team at Google.
1313

14-
To see all available commands in this repository, run `make help`.
14+
To get started with the GCP CCM, see the **[kOps Quickstart](docs/kops-quickstart.md)** (automated setup) or the **[Manual CCM Setup Guide](docs/ccm-manual.md)**.
15+
16+
For local development, use `make help` to see all available commands.
1517

1618
## Components
1719

‎docs/ccm-manual.md‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
# GCP Cloud Controller Manager (CCM) Manual Setup Guide
2+
3+
This guide provides instructions for building and deploying the GCP Cloud Controller Manager (CCM) to a self-managed Kubernetes cluster.
4+
5+
## Prerequisites
6+
7+
1. **Kubernetes Cluster**: A Kubernetes cluster running on Google Cloud Platform.
8+
* The cluster's components (`kube-apiserver`, `kube-controller-manager`, and `kubelet`) must have the `--cloud-provider=external` flag.
9+
* For an example of how to create GCE instances and initialize such a cluster manually using `kubeadm`, see **[Manual Kubernetes Cluster on GCE](manual-cluster-gce.md)**.
10+
2. **GCP Service Account**: The nodes (or the CCM pod itself) must have access to a GCP IAM Service Account with sufficient permissions to manage compute resources (e.g. instances, load balancers, and routes).
11+
3. **Docker & gcloud CLI**: Authorized and configured for pushing images to GCP Artifact Registry.
12+
13+
14+
### [Optional] Build and Push the CCM Image (Manual Clusters)
15+
16+
> [!NOTE]
17+
> This step is only necessary if you intend to use a locally built image of the CCM. Otherwise, use an official release image as specified in Step 1.
18+
19+
If you are using a manually provisioned cluster (e.g. `kubeadm`), build the `cloud-controller-manager` Docker image and push it to your registry:
20+
21+
```sh
22+
# Google Cloud Project ID, registry location, and repository name.
23+
GCP_PROJECT=$(gcloud config get-value project)
24+
GCP_LOCATION=us-central1
25+
REPO=my-repo
26+
27+
# Create an Artifact Registry repository (if it doesn't already exist)
28+
gcloud artifacts repositories create ${REPO} \
29+
--project=${GCP_PROJECT} \
30+
--repository-format=docker \
31+
--location=${GCP_LOCATION} \
32+
--description="Docker repository for CCM"
33+
```
34+
35+
Replace `NODE_NAME` and `NODE_ZONE` with the name and zone of your master node.
36+
```sh
37+
# Or with kubectl: NODE_NAME=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
38+
# Grant the cluster nodes permission to pull images from Artifact Registry.
39+
# This automatically extracts your GCE node's service account using kubectl and gcloud.
40+
NODE_NAME=k8s-master
41+
NODE_ZONE=us-central1-a
42+
NODE_SA=$(gcloud compute instances describe $NODE_NAME \
43+
--zone=$NODE_ZONE --project=${GCP_PROJECT} \
44+
--format="value(serviceAccounts[0].email)")
45+
echo $NODE_SA
46+
gcloud artifacts repositories add-iam-policy-binding ${REPO} \
47+
--project=${GCP_PROJECT} \
48+
--location=${GCP_LOCATION} \
49+
--member="serviceAccount:${NODE_SA}" \
50+
--role="roles/artifactregistry.reader"
51+
# Configure docker to authenticate with Artifact Registry
52+
gcloud auth configure-docker ${GCP_LOCATION}-docker.pkg.dev
53+
54+
# Build and Push
55+
IMAGE_REPO=${GCP_LOCATION}-docker.pkg.dev/${GCP_PROJECT}/${REPO} IMAGE_TAG=v0 make publish
56+
```
57+
58+
*Note: If `IMAGE_TAG` is omitted, the Makefile will use a combination of the current Git commit SHA and the build date.*
59+
60+
## Step 1: Deploy the CCM to your Cluster (Manual Clusters)
61+
62+
For native Kubernetes clusters, avoid the legacy `deploy/cloud-controller-manager.manifest` (which is a template used by legacy script `kube-up`). Instead, use the kustomize-ready DaemonSet which correctly includes the RBAC roles and deployment.
63+
64+
Update the image to the official release:
65+
```sh
66+
# Replace with desired version vX.Y.Z
67+
(cd deploy/packages/default && kustomize edit set image k8scloudprovidergcp/cloud-controller-manager=registry.k8s.io/cloud-provider-gcp/cloud-controller-manager:v30.0.0)
68+
```
69+
70+
**For development with a locally built image**
71+
If you built the image locally in the optional prerequisite step above, instead run this command to use your newly pushed tag:
72+
```sh
73+
(cd deploy/packages/default && kustomize edit set image k8scloudprovidergcp/cloud-controller-manager=$IMAGE_REPO:$IMAGE_TAG)
74+
```
75+
76+
### Update CCM Manifest
77+
78+
The `manifest.yaml` DaemonSet intentionally has empty command-line arguments (`args: []`). You must provide the necessary arguments to the `cloud-controller-manager` container. For a typical Kops or GCE cluster, you can supply these arguments by creating a Kustomize patch.
79+
80+
> [!NOTE]
81+
> Modify `--cluster-cidr` and `--cluster-name` below to match your cluster. Note that GCP resource names cannot contain dots (`.`), so if your cluster name is `my.cluster.net`, you must use a sanitized format like `my-cluster-net`. If you don't have a cluster name, just use any name (e.g., node name).
82+
83+
```sh
84+
cat << EOF > deploy/packages/default/args-patch.yaml
85+
apiVersion: apps/v1
86+
kind: DaemonSet
87+
metadata:
88+
name: cloud-controller-manager
89+
namespace: kube-system
90+
spec:
91+
template:
92+
spec:
93+
volumes:
94+
- name: host-kubeconfig
95+
hostPath:
96+
path: /etc/kubernetes/admin.conf
97+
containers:
98+
- name: cloud-controller-manager
99+
command: ["/usr/local/bin/cloud-controller-manager"]
100+
volumeMounts:
101+
- name: host-kubeconfig
102+
mountPath: /etc/kubernetes/admin.conf
103+
readOnly: true
104+
args:
105+
- --kubeconfig=/etc/kubernetes/admin.conf
106+
- --authentication-kubeconfig=/etc/kubernetes/admin.conf
107+
- --authorization-kubeconfig=/etc/kubernetes/admin.conf
108+
- --cloud-provider=gce
109+
- --allocate-node-cidrs=true
110+
- --cluster-cidr=10.4.0.0/14
111+
- --cluster-name=kops-k8s-local
112+
- --configure-cloud-routes=true
113+
- --leader-elect=true
114+
- --use-service-account-credentials=true
115+
- --v=2
116+
EOF
117+
(cd deploy/packages/default && kustomize edit add patch --path args-patch.yaml)
118+
119+
# Deploy the configured package (this applies the DaemonSet and its required roles):
120+
kubectl apply -k deploy/packages/default
121+
```
122+
123+
If you encounter a `NotFound` pod error for the container command path, try `command: ["/cloud-controller-manager"]` instead.
124+
125+
126+
### Alternative: Apply Standalone RBAC Roles
127+
128+
If you prefer to deploy the RBAC rules independently from the base daemonset package, you can apply them directly:
129+
130+
```sh
131+
kubectl apply -f deploy/cloud-node-controller-role.yaml
132+
kubectl apply -f deploy/cloud-node-controller-binding.yaml
133+
kubectl apply -f deploy/pvl-controller-role.yaml
134+
```
135+
136+
## Step 3: Verification
137+
138+
To verify that the Cloud Controller Manager is running successfully:
139+
140+
1. **Check the Pod Status**: Verify the pod is `Running` in the `kube-system` namespace.
141+
```sh
142+
kubectl get pods -n kube-system -l component=cloud-controller-manager
143+
```
144+
145+
2. **Check Pod Logs**: Look for any errors or access and authentication issues with the GCP API.
146+
```sh
147+
kubectl describe pod -n kube-system -l component=cloud-controller-manager
148+
kubectl logs -n kube-system -l component=cloud-controller-manager
149+
```
150+
151+
3. **Check Node Initialization**: The `kubelet` initially applies a `node.cloudprovider.kubernetes.io/uninitialized` taint when bound to an external cloud provider. The CCM should remove this taint once it successfully fetches the node's properties from the GCP API.
152+
```sh
153+
# Ensure no nodes have the uninitialized taint, output should be empty.
154+
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints | grep uninitialized
155+
```
156+
157+
4. **Verify External IPs and ProviderID**: Check if your nodes are correctly populated with GCP-specific data.
158+
```sh
159+
kubectl describe nodes | grep "ProviderID:"
160+
# Expect output `ProviderID: gce://...`
161+
```
162+
163+
## Teardown
164+
165+
If you used the default CCM package, you can clean up the local patch file and reset all changes to kustomization.yaml:
166+
```sh
167+
rm deploy/packages/default/args-patch.yaml
168+
git checkout deploy/packages/default/kustomization.yaml
169+
```
170+
171+
If you followed the [manual cluster setup guide](manual-cluster-gce.md), you may follow the [teardown steps](manual-cluster-gce.md#teardown) to clean up your GCP resources.

‎docs/kops-quickstart.md‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# GCP CCM with kOps Quickstart
2+
3+
This guide provides a quickstart for building and deploying the GCP Cloud Controller Manager (CCM) to a self-managed Kubernetes cluster provisioned with kOps.
4+
5+
## Prerequisites
6+
7+
A Google Cloud Platform project with billing enabled.
8+
9+
## Deployment
10+
11+
The `make kops-up` target is an end-to-end workflow that automatically:
12+
- Provisions a Kubernetes cluster using kOps.
13+
- Builds the CCM image locally.
14+
- Pushes the image to your Artifact Registry.
15+
- Deploys the CCM (along with required RBAC) to the cluster.
16+
17+
Run the following commands to get started:
18+
19+
```sh
20+
# Enable required GCP APIs
21+
gcloud services enable compute.googleapis.com
22+
gcloud services enable artifactregistry.googleapis.com
23+
24+
# Set environment variables
25+
export GCP_PROJECT=$(gcloud config get-value project)
26+
export GCP_LOCATION=us-central1
27+
export GCP_ZONES=${GCP_LOCATION}-a
28+
export KOPS_CLUSTER_NAME=kops.k8s.local
29+
export KOPS_STATE_STORE=gs://${GCP_PROJECT}-kops-state
30+
31+
# Create the state store bucket if it doesn't already exist
32+
gcloud storage buckets create ${KOPS_STATE_STORE} --location=${GCP_LOCATION} || true
33+
34+
# Run the cluster creation target, may take several minutes
35+
make kops-up
36+
```
37+
38+
## Verification
39+
40+
To verify that the Cloud Controller Manager is running successfully:
41+
42+
1. **Check the Pod Status**: Verify the pod is `Running` in the `kube-system` namespace.
43+
```sh
44+
kubectl get pods -n kube-system -l component=cloud-controller-manager
45+
```
46+
47+
2. **Check Pod Logs**: Look for any errors or access and authentication issues with the GCP API.
48+
```sh
49+
kubectl logs -n kube-system -l component=cloud-controller-manager
50+
```
51+
52+
3. **Check Node Initialization**: The CCM should remove the `node.cloudprovider.kubernetes.io/uninitialized` taint once it successfully fetches the node's properties from the GCP API.
53+
```sh
54+
# Ensure no nodes have the uninitialized taint, output should be empty.
55+
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints | grep uninitialized
56+
```
57+
58+
4. **Verify ProviderID**: Check if your nodes are correctly populated with GCP-specific data (e.g., `ProviderID` in the format `gce://...`).
59+
```sh
60+
kubectl describe nodes | grep "ProviderID:"
61+
```
62+
63+
## Teardown
64+
65+
To tear down the cluster and clean up resources:
66+
67+
```sh
68+
make kops-down
69+
```

0 commit comments

Comments
 (0)