Skip to content

Bake container images directly into containerd and load into KinD via containerd archive - #777

Open
kraney wants to merge 6 commits into
mainfrom
image-fixing
Open

kraney wants to merge 6 commits into
mainfrom
image-fixing

Conversation

@kraney

@kraney kraney commented Sep 29, 2026

Copy link
Copy Markdown
Contributor
Previously, the KNE internal VM image bake (`cloudbuild/internal.pkr.hcl`) pulled container images using `sudo docker pull`, which

placed images into Docker's Moby storage engine (/var/lib/docker). However, KNE Kubernetes clusters running with Kubeadm use containerd
with the k8s.io CRI namespace (/var/lib/containerd).

Because images were absent from containerd's namespace, pods with `imagePullPolicy: IfNotPresent` experienced cache misses and

immediately pulled live :ga tags from us-west1-docker.pkg.dev at cluster startup. This bypassed KNE VM image qualification tests and
introduced live tag drift. Furthermore, many critical infrastructure images (CNI, wait containers, MetalLB, OTG controllers/engines) were
never pre-pulled during the VM image bake.

This PR addresses the issue by:
1. **Containerd image baking in Packer (`cloudbuild/internal.pkr.hcl`)**:
   - Removes obsolete `docker pull` commands from the Docker installation step.
   - Adds a dedicated shell provisioner after containerd initialization and restart that pulls directly into containerd's `k8s.io`

namespace using sudo ctr -n k8s.io images pull.
- Uses gcloud auth print-access-token for authenticated access to us-west1-docker.pkg.dev.
- Bakes the complete set of required images:
- Core & Meshnet: meshnet:ga, bridge:ga, init-wait:ga
- Vendor NOS DUTs: ceos:ga, xrd:ga, 8000e:ga, ncptx:ga, srlinux:ga, lemming:ga
- CNI & Ingress: flannel, flannel-cni-plugin, kindnetd, metallb/controller, metallb/speaker
- Controllers & Operators: srl-controller, arista-ceoslab-operator, openconfig-lemming/operator, kube-rbac-proxy,
keng-operator
- OTG Keysight: keng-controller, otg-gnmi-server, ixia-c-traffic-engine, ixia-c-protocol-engine
2. Containerd-first KinD loading with Docker fallback (deploy/deploy.go):
- Updates KindSpec.loadContainerImages() to check if the image is available in containerd on the host. If present, it exports to a
temporary tarball and loads it into the KinD cluster via kind load image-archive, tagging within the KinD node if destination differs
from source. This avoids pulling over the network and avoids duplicating multi-gigabyte NOS images into host Docker daemon storage.
- If containerd or the requested image is absent, it cleanly falls back to the previous docker pull -> docker tag -> kind load docker-image workflow.
3. Export ClusterKindNodes() (cluster/kind/kind.go):
- Exports ClusterKindNodes() so that node container names can be queried across packages for tagging.
4. Unit Tests (deploy/deploy_test.go, cluster/kind/kind_test.go):
- Added unit test cases for containerd image-archive loading with and without retagging.
- Updated existing test cases to maintain coverage of the Docker fallback path.

TAG=agy
CONV=0ab70602-08fb-45ae-a2b5-fdda48e2f5ac

@kraney
kraney requested review from bstoll and manizzzz September 30, 2026 15:27
1. Removed redundant Docker pulls: Removed the old sudo docker pull ... block in the Docker installation step, which was
storing images in Docker's Moby engine store (/var/lib/docker/) where containerd / kubeadm could not see them.
2. Added containerd pre-pull provisioner (ctr -n k8s.io): Positioned right after containerd is configured and restarted,
pulling directly into the k8s.io CRI namespace.
3. Authenticated private GCP registry pulls: Pulls from us-west1-docker.pkg.dev using TOKEN=$(gcloud auth print-access-
token) via sudo ctr -n k8s.io images pull -u "oauth2accesstoken:$TOKEN" "$img".
4. Complete image coverage:
    • Core KNE & topology components (meshnet:ga, bridge:ga, init-wait:ga).
    • Vendor NOS images (ceos:ga, xrd:ga, 8000e:ga, ncptx:ga, srlinux:ga, lemming:ga).
    • CNI & Ingress (flannel, flannel-cni-plugin, kindnetd, metallb/controller, metallb/speaker).
    • Controllers & Operators (srl-controller, arista-ceoslab-operator, openconfig-lemming/release/operator, kube-rbac-
    proxy, keng-operator).
    • Keysight / OTG engine images (keng-controller, otg-gnmi-server, ixia-c-traffic-engine, ixia-c-protocol-engine).
This allows it to share the same images stored for kubeadm use. If the
image isn't found that way fall back to the established docker pull
/ kind load method, to minimize the risk of breaking any existing
usages.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant