Skip to content

fix(infra-k8s): tunnel kubectl through az aks command invoke on Azure - #56

Open
daanpersoons wants to merge 1 commit into
v1from
fix/azure-private-cluster-invoke
Open

daanpersoons wants to merge 1 commit into
v1from
fix/azure-private-cluster-invoke

Conversation

@daanpersoons

Copy link
Copy Markdown
Contributor

Problem

Every Azure k8s rollout has failed since the infra-*@v1 migration:

error validating "k8s/environments/test": error validating data: failed to download openapi:
Get "https://aks-weu-customerzone-tst-01-dns-xxxx.privatelink.westeurope.azmk8s.io:443/openapi/v2"
dial tcp: lookup ...privatelink.westeurope.azmk8s.io: no such host

The clusters are private — the API server only resolves inside the VNet, so a hosted
runner cannot reach it. The vendor-agnostic actions call kubectl directly from the runner.

This worked before because the old Azure-specific workflow tunnelled every kubectl
through az aks command invoke. That code only ever existed on feature/azure-deployments,
which was never merged into main, v1 or v2 — it was a spike branch carrying a
TEMP: point to branch workflows commit, and consumers were pinned straight at it.
When those consumers moved to @v1, the tunnelling silently disappeared.

Change

infra/k8s/apply and infra/k8s/rollout render kustomize locally, then run kubectl
on the cluster via az aks command invoke when vendor: azure. Other vendors are
unchanged
— DigitalOcean and Scaleway still call kubectl directly.

Two bugs fixed relative to the original implementation:

  • Exit codes were swallowed. az aks command invoke returns 0 even when the remote
    command failed — it only prints exitcode=N
    (azure-cli/command_modules/acs/custom.py:2795-2801). A failing kubectl apply would
    have reported a green build. The JSON result is now parsed and the step fails on a
    non-zero exitCode or a provisioningState other than Succeeded.
  • The image-based rollout piped kubectl into jq, which the run-command pod cannot
    do. The remote output is captured and jq stays on the runner.

Inputs moved from ${{ }} interpolation into env:, so values can't be substituted
into the script body.

Testing

  • 13 functional tests against mocked az/kubectl: both vendor paths, all four rollout
    branches, remote-failure propagation, provisioningState: Failed, and image matching
    with 0 and 1 hits. All pass.
  • YAML parses; bash -n clean on every run: body.
  • End-to-end on a real private cluster: indaver-infra run 36832256011 (k8s/environments/test) — green, with real changes applied (configmap ... created, deployment.apps/indaver-api configured).

Requirements

The service principal needs Microsoft.ContainerService/managedClusters/runcommand/action
and .../commandResults/read (both in Azure Kubernetes Service Cluster User Role), and
the cluster must not have --disable-run-command. Documented in Docs/infra-k8s-rollout.md.

Note

v2 has the identical gap and will need the same fix before anyone moves to it.

Azure clusters are private, so their API server only resolves inside the
VNet. Since the vendor-agnostic infra-k8s actions call kubectl directly
from the runner, every Azure rollout fails with:

  failed to download openapi: dial tcp: lookup
  <cluster>.privatelink.westeurope.azmk8s.io: no such host

Restore the tunnelling that lived on the unmerged feature/azure-deployments
branch: render kustomize locally, then run kubectl on the cluster through
`az aks command invoke`. Other vendors keep calling kubectl directly.

Two fixes on top of the original implementation:

- `az aks command invoke` exits 0 even when the remote command failed (it
  only prints `exitcode=N`), so a failing apply reported a green build.
  Parse the JSON result and fail on a non-zero exitCode or a
  provisioningState other than Succeeded.
- The image-based rollout piped kubectl into jq, which the run-command pod
  cannot do. Capture the remote output and keep jq on the runner.

Inputs are now passed via env instead of interpolating ${{ }} into the
script body.
@ernest-app ernest-app Bot added the 📃 S PR label Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant