Skip to content

fix(acl): switch base image source to ACG - #9484

Merged
Nishchay (awesomenix) merged 1 commit into
mainfrom
aadagarwal/switch-acl-base-image-to-acg
Oct 5, 2026
Merged

Nishchay (awesomenix) merged 1 commit into
mainfrom
aadagarwal/switch-acl-base-image-to-acg

Conversation

@aadhar-agarwal

@aadhar-agarwal aadhar-agarwal commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Switches ACL amd64 and arm64 VHD builds from Marketplace image coordinates to direct Azure Compute Gallery image references. It pins the signed production ACL images below and updates the Packer configurations to use direct shared gallery image IDs:

  • AMD64: acl-3.0-amd64/20260923.1209152.2
  • ARM64: acl-3.0-arm64/20260923.1209110.1

The images are replicated to westus, westus2, westus3, eastus and west europe

[TEST All VHDs] AKS Linux VHD Build - Msft Tenant - Passed

AKS Linux VHD Build - TME Tenant - Passed

What this PR does / why we need it:
By using sig images instead of marketplace images, we would have the ability to release CVE updates quicker

Which issue(s) this PR fixes:

Fixes #

@github-actions

github-actions Bot commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Windows Unit Test Results

  3 files   18 suites   1m 8s ⏱️
546 tests 546 ✅ 0 💤 0 ❌
549 runs  549 ✅ 0 💤 0 ❌

Results for commit 10a2cc3.

♻️ This comment has been updated with latest results.

@aadhar-agarwal
aadhar-agarwal force-pushed the aadagarwal/switch-acl-base-image-to-acg branch from 2cab152 to cad00e3 Compare September 14, 2026 22:28
@aks-node-assistant

Copy link
Copy Markdown
Contributor

AgentBaker Linux gate detective

  • Run: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=181075204
  • Failed job/stage/task: e2e stage / Run AgentBaker E2E / failed test run 629335097.
  • TL;DR: Two E2E scenarios failed with bastion/SSH context-deadline symptoms: Ubuntu2204_HTTPSProxy_PrivateDNS could not start the bastion tunnel, and Ubuntu2204Gen2_ImagePullIdentityBinding_NetworkIsolated hit SSH command timeouts during VM validation.
  • Likely cause/signature: e2e-bastion-ssh-handshake-context-deadline; shared E2E bastion/network infrastructure flake.
  • Classification: E2E/test - shared cluster bastion/infra teardown flake.
  • Confidence: Medium-high.
  • Recommended owner/action: Gate/E2E owner should inspect westus3 shared bastion health and network-isolated SSH tunnel reliability; PR owner should watch for repeat failures before treating the ACG base-image source change as causal.
  • Strongest alternative: PR base-image-source changes caused VM validation regressions, but both failures center on bastion/SSH context deadlines and one wireserver validation collection path, matching prior unrelated infra signatures more closely than an image content regression.
  • Evidence: failed test run 629335097, Run AgentBaker E2E log 662, timeline failed E2E task, build metadata for PR 9484, and existing wiki recurrence.
  • Wiki signature: e2e-bastion-ssh-handshake-context-deadline

@aadhar-agarwal
aadhar-agarwal force-pushed the aadagarwal/switch-acl-base-image-to-acg branch 7 times, most recently from a7ca5c6 to cbac028 Compare September 29, 2026 05:52
@aks-node-assistant

Copy link
Copy Markdown
Contributor

AgentBaker gate detective

Run: 183288477
Failed job/stage/task: build / build2204fipsgen2containerd / Test, Scan, and Cleanup; build / build2204gen2containerd / Build VHD
Detective summary: Two findings repeated/appeared: Ubuntu 22.04 FIPS CIS regressions in Test/Scan, and Ubuntu 22.04 gen2 Build VHD failed all retries because apt/dpkg locks were held by concurrent apt-get processes.
Likely cause/signature: vhd-ubuntu2204fips-cis-rule-6-1-3-1-regression and vhd-ubuntu2204gen2-apt-lock-contention-build-vhd.
Confidence: High for the signatures; medium for the apt-lock owner because it appears environment/package-manager contention.
Recommended owner/action: Continue repair #39908382 for the CIS baseline drift; AgentBaker VHD build owner should make package-manager operations wait on apt/dpkg locks or remove the concurrent apt-get race before Build VHD setup.
Strongest alternative: PR change-caused is less likely for CIS because it recurs across unrelated PRs; for the apt-lock failure, a product/package regression is less likely because the first failing step is lock acquisition before package install logic completed and the retry history shows the same lock contention.
Evidence links: build report, build timeline, failed Build VHD task log, failed Test/Scan task log, wiki tracker
Wiki signature: vhd-ubuntu2204fips-cis-rule-6-1-3-1-regression, vhd-ubuntu2204gen2-apt-lock-contention-build-vhd

@aadhar-agarwal
aadhar-agarwal force-pushed the aadagarwal/switch-acl-base-image-to-acg branch 2 times, most recently from e6081ce to 6c56442 Compare October 2, 2026 00:05
@aadhar-agarwal
aadhar-agarwal marked this pull request as ready for review October 2, 2026 01:10
@aadhar-agarwal
aadhar-agarwal requested review from a team as code owners October 2, 2026 01:10
Copilot AI balanced review requested due to automatic review settings October 2, 2026 01:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Release publishing metadata generation still requires the removed Marketplace variables.

Review effort: Balanced
Findings: 1 High severity

Open (1)
What changed in this PR

Switches ACL AMD64 and ARM64 VHD builds from Marketplace images to pinned Azure Compute Gallery images.

Changes:

  • Configures direct ACG source-image references.
  • Preserves empty test parameters and temporarily skips ACL CVM Chrony validation.
  • Adds ShellSpec coverage for parameter transport and content-test behavior.
File Description
vhdbuilder/​packer/​vhd-image-builder-acl.json Uses the AMD64 ACG source.
vhdbuilder/​packer/​vhd-image-builder-acl-arm64.json Uses the ARM64 ACG source.
vhdbuilder/​packer/​test/​run-test.sh Preserves optional Run Command arguments.
vhdbuilder/​packer/​test/​linux-vhd-content-test.sh Decodes parameters and adds the Chrony exception.
spec/​vhdbuilder/​packer/​test/​run_test_spec.sh Tests parameter round trips.
spec/​vhdbuilder/​packer/​test/​linux_vhd_content_test_helpers_spec.sh Tests content-test behavior.
spec/​vhdbuilder/​packer/​ensure_sig_image_name_linux_spec.sh Updates ACL naming scenarios.
.pipelines/​.vsts-vhd-builder.yaml Pins ACG sources for validation builds.
.pipelines/​.vsts-vhd-builder-release.yaml Pins ACG sources for release builds.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .pipelines/.vsts-vhd-builder-release.yaml
Copilot AI balanced review requested due to automatic review settings October 2, 2026 02:48
@aadhar-agarwal
aadhar-agarwal force-pushed the aadagarwal/switch-acl-base-image-to-acg branch from 6c56442 to caa6288 Compare October 2, 2026 02:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

ACL CVM releases currently bypass validation for a known unresolved time-synchronization defect.

Review effort: Balanced
Findings: 2 High severity

Open (2)

Comment thread vhdbuilder/packer/test/linux-vhd-content-test.sh Outdated
Copilot AI balanced review requested due to automatic review settings October 2, 2026 03:22
@aadhar-agarwal
aadhar-agarwal force-pushed the aadagarwal/switch-acl-base-image-to-acg branch from caa6288 to 698bde6 Compare October 2, 2026 03:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The ACL CVM exemption allows images with nonfunctional time synchronization to pass content validation.

Review effort: Balanced
Findings: 2 High severity

Open (2)

Use signed production ACG images replicated to West Europe:
amd64 20260923.1209152.2, arm64 20260923.1209110.1.

Preserve empty test arguments, legacy callers, and quoted cron arguments.
Keep ACL CVM Chrony service and clock-correction validation enabled; document the known base-image failure and link microsoft/azure-container-linux#61 without adding a bypass or changing the existing image versions.

Generate ACL publishing metadata from the pinned ACG source image name and version instead of querying Marketplace with empty coordinates. Select this path explicitly for Linux AzureContainerLinux builds, using OFFER_NAME as passed by the release template. Missing source fields warn without introducing new release failures. Preserve existing non-ACL publishing behavior, including OSGuard and Windows, and retain the JSON schema.

Add 23 publishing-info regression cases covering pinned AMD64/ARM64/CVM sources, incomplete or absent ACL metadata, unrelated ACG variables on non-ACL builds, Marketplace behavior, OSGuard, and Windows. Four selector cases fail before the final fix and all 23 pass afterward. Bash lint and syntax checks pass with no new POSIX diagnostics.

Log the pinned ACL source gallery, image definition, image version, and final Packer build region during Build VHD, immediately before the masked settings dump. Restrict the new messages to Linux ACL builds and preserve existing token redaction.

Add nine source-logging regression cases covering AMD64, ARM64, CVM and production region overrides, non-ACL builds, and token redaction. All nine pass; Bash syntax and repository-configured ShellCheck pass.

The 35 remaining content-helper and transport cases pass. Eight mocked ACL CVM scenarios confirm both Chrony checks execute and report service and time-correction errors. Executable Chrony behavior matches the PR base; Bash syntax and configured ShellCheck pass.

Signed-off-by: Aadhar Agarwal <aadagarwal@microsoft.com>
Copilot AI balanced review requested due to automatic review settings October 2, 2026 05:17
@aadhar-agarwal
aadhar-agarwal force-pushed the aadagarwal/switch-acl-base-image-to-acg branch from 698bde6 to 10a2cc3 Compare October 2, 2026 05:17
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Code Coverage

Package Line Rate Health
github.com/Azure/agentbaker/aks-node-controller 72% ➖
github.com/Azure/agentbaker/aks-node-controller/common 100% ✔
github.com/Azure/agentbaker/aks-node-controller/helpers 71% ➖
github.com/Azure/agentbaker/aks-node-controller/parser 89% ✔
github.com/Azure/agentbaker/aks-node-controller/pkg/gpu 100% ✔
github.com/Azure/agentbaker/aks-node-controller/pkg/nodeconfigutils 67% ➖
github.com/Azure/agentbaker/aks-node-controller/utils 0% ❌
github.com/Azure/agentbaker/apiserver 25% ❌
github.com/Azure/agentbaker/cmd 0% ❌
github.com/Azure/agentbaker/cmd/starter 0% ❌
github.com/Azure/agentbaker/fuzz/api 0% ❌
github.com/Azure/agentbaker/hotfix/render-nodecustomdata 0% ❌
github.com/Azure/agentbaker/pkg/agent 76% ➖
github.com/Azure/agentbaker/pkg/agent/datamodel 76% ➖
github.com/Azure/agentbaker/pkg/agent/toggles 0% ❌
github.com/Azure/agentbaker/pkg/vhdbuilder/datamodel 88% ✔
Summary 74% (6317 / 8582) ➖

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Code Coverage

Package Line Rate Health
test 0% ❌
windows 24% ❌
staging_cse_windows Package 1 65% ➖
debug 0% ❌
provisioningscripts 2% ❌
parts_windows Package 1 76% ➖
Summary 38% (1632 / 6825) ❌

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Code Coverage

Package Line Rate Health
shellspec spec 22% ❌
Summary 22% (3161 / 14345) ❌

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The ACG transition is consistently wired across both architectures, publishing metadata, pipelines, and comprehensive tests.

Review effort: Balanced
Findings: None

Resolved since last review (2)

@aks-node-assistant

Copy link
Copy Markdown
Contributor

AgentBaker Linux gate detective

Run: 183841063 (same PR also failed in 183818676 and 183831210)
Failed job/stage/task: e2e / Run AgentBaker E2E / Run AgentBaker E2E

Detective summary: Three new PR-gate runs for this PR failed in Run AgentBaker E2E. Builds 183818676 and 183841063 had broad VMSS creation failures where ARM reported WestUS3 regional quota/vCPU exhaustion before node provisioning; build 183831210 also repeated the existing AzureLinuxV3Gen2Kata containerd migration-warning validator failure.

Likely cause/signature: �2e-westus3-vmss-quota-exceeded for the broad E2E failures; �2e-azurelinuxv3gen2kata-containerd-v2-migration-warning for the recurring Kata validator failure in 183831210.

Confidence: High for quota/environment on the broad failures: timeline isolates Run AgentBaker E2E, logs show many scenarios failing during VMSS create with quota guidance, and the failures happen before node provisioning. Medium-high for Kata recurrence: the signature is already repair-tracked as #39969892 and reproduces across unrelated PRs.

Recommended owner/action: Gate/infra owners should restore WestUS3 quota/capacity for the E2E subscription/region before rerunning; Node Lifecycle E2E owners should continue repair #39969892 for Kata containerd config validation.

Strongest alternative: PR #9484 changes ACL image source/publishing scripts, so PR-caused ACL image behavior is possible; however the broad failures cite regional quota before scenario validation, and the Kata warning is recurring across unrelated PRs, so PR-specific code is less likely.

Evidence: latest run, latest E2E task log, quota wiki signature, Kata repair row

@aks-node-assistant

Copy link
Copy Markdown
Contributor

AgentBaker Linux gate detective

Run: 183935396
Failed job/stage/task: e2e / Run AgentBaker E2E / Run AgentBaker E2E

Detective summary: The E2E run completed 192 scenarios with 116 passed, 0 flaky, 74 skipped, and 2 failed. ACL_SecondaryNIC_DualStack failed during VMSS provisioning/tunnel setup in WestUS3, matching the recurring regional quota/capacity signature on this PR; AzureLinuxV3Gen2Kata again hit the containerd v2-to-v4 migration warning validator failure tracked by repair #39969892.

Likely cause/signature: �2e-westus3-vmss-quota-exceeded and �2e-azurelinuxv3gen2kata-containerd-v2-migration-warning.

Confidence: Medium-high. Timeline isolates failure to the E2E task; the task log reports two failed scenarios with 0 flaky; both signatures match existing wiki rows; PR #9484 changes ACL base image source, but the quota/capacity failure happens before scenario validation and the Kata validator failure repeats across unrelated PRs.

Recommended owner/action: Gate/infra owners should restore WestUS3 capacity/quota for E2E VMSS creation. Node Lifecycle E2E owners should continue repair #39969892 for Kata containerd config validation.

Strongest alternative: PR #9484 could affect ACL image behavior, but the primary ACL failure is infrastructure provisioning/tunnel setup and not an in-node validation failure. The Kata warning is already recurring across unrelated PRs.

Evidence: run, E2E task log, quota wiki signature, Kata repair row

@aadhar-agarwal

Copy link
Copy Markdown
Contributor Author

Seems like the kata gen2 e2e is failing (unrelated to this PR) - #9690

@awesomenix
Nishchay (awesomenix) merged commit abc38a3 into main Oct 5, 2026
53 of 55 checks passed
@awesomenix
Nishchay (awesomenix) deleted the aadagarwal/switch-acl-base-image-to-acg branch October 5, 2026 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants