Skip to content

feat(aws): BuildKit builder for Docker image assets - #165

Open
so0k wants to merge 1 commit into
mainfrom
feat/buildkit-docker-asset
Open

so0k wants to merge 1 commit into
mainfrom
feat/buildkit-docker-asset

Conversation

@so0k

@so0k so0k commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Problem

Docker image assets in TerraConstructs are built and pushed via kreuzwerker/docker's
docker_image + docker_registry_image resources. Both need a real Docker Engine:
docker_image build hardcodes exportLoad, and docker_registry_image push goes through
client.ImagePush. That's a non-starter on a TACOS (Terraform-as-CI/CD-on-a-server) host such
as Atlantis that intentionally has no Docker daemon and no root-equivalent socket — a
daemon or docker.sock access is effectively root on the host, which a public-facing
automation server shouldn't carry, and local-exec/provisioners are equally unwelcome there.

Design

Adds an opt-in DockerAssetBuilder.BUILDKIT to AwsAssetManagerOptions. When selected,
addDockerImageAsset emits a single buildkit_image resource (provider cruxstack/buildkit,
pinned to exactly 0.0.1) instead of docker_image + docker_registry_image. It solves and
pushes the image by talking directly to a reachable buildkitd over gRPC — no Docker Engine
API involved.

The provider "buildkit" block is minimal and non-configurable beyond the daemon address:

  • buildkit_address — where to dial (buildkitAddress prop, default
    unix:///run/buildkit/buildkitd.sock).
  • buildkit_autodiscover = false and embedded_buildkitd = false — hardcoded, not exposed
    as options
    . autodiscover would let the provider probe for and use an arbitrary daemon on
    the host; embedded_buildkitd = true lets the provider download and spawn its own private
    buildkitd process. Either defeats the point of pointing Terraform at one operator-managed,
    already-hardened daemon, so both are forced off rather than left as knobs a stack author
    could flip.
  • Registry auth comes from the ambient ~/.docker/config.json credHelpers (e.g.
    ecr-login) on the host running buildkitd — no credentials pass through Terraform state.

The default builder (docker/kreuzwerker) is unchanged; this is additive and opt-in.

src/aws/private/buildkit-provider.ts is a small hand-written cdktn binding for
buildkit_image / provider "buildkit", since no @cdktn/provider-buildkit package is
published (see Caveats).

Validation

Ran as part of a separate infrastructure spike into daemonless Docker asset builders for a
TACOS host. Full environment: Ubuntu 24.04 arm64, rootless buildkitd v0.33.0 (unix-socket +
group-ACL exposure, insecure-entitlements=[]), pnpm 11.24.0, cdktn 0.24.0, Terraform
1.15.9. (An earlier cut of the same design was also exercised against TF 1.10-era provider
pins; not claiming that combination was re-verified against this exact code.)

Against a real ECR repository:

  • synth: only buildkit_image.<id>_Buildkit + provider "buildkit" are emitted — zero
    docker_image / docker_registry_image resources.
  • plan/apply: Plan: 1 to add → Apply complete! Resources: 1 added; pushed an arm64
    image to ECR (verified image architecture + a marker file baked into the image).
  • no-op re-plan: clean (No changes), and a re-apply reports 0 added, 0 changed.
  • Dockerfile change: forces exactly one rebuild (1 to add, 1 to destroy — the asset's
    context hash changes, which is also this resource's triggers), producing a new digest,
    then a clean re-plan.

Tests

  • New test/aws/storage/assets/image-asset-buildkit.test.ts: synth asserts the pinned
    provider block, the buildkit_image attributes (context, dockerfile, platforms, args,
    secrets, target, publish, cache_from, triggers) and the ImageUri output expression, plus
    resourceCountIs = 0 for both kreuzwerker resource types; a missing-platform guard; and a
    guard rejecting networkMode/dockerBuildSsh/dockerOutputs, which buildkit_image can't
    express.
  • Existing image-asset.test.ts / build-image-cache.test.ts (kreuzwerker path) unchanged
    and still passing, confirming the default builder is untouched.
  • Full local run: jsii compile clean, eslint --fix clean (no changes), jsii-pacmak
    packaging clean, full Jest suite across all 5 CI shards — 255/255 suites, 4548 passed / 54
    skipped, 0 failed, no snapshot diffs.

Caveats

  • cruxstack/buildkit is a young provider (v0.0.1, single vendor). Worth vetting/pinning
    carefully before relying on it in production, and revisiting the pin as it matures.
  • src/aws/private/buildkit-provider.ts is a hand-written binding, not generated by cdktn get, because no published @cdktn/provider-buildkit package exists yet. If one appears
    upstream, this should be regenerated against it rather than hand-maintained indefinitely.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Dgqc87KhZ9EijLMoJ61ZXe

Adds DockerAssetBuilder.BUILDKIT to AwsAssetManagerOptions. When
selected, addDockerImageAsset emits one buildkit_image resource plus a
provider "buildkit" pinned to cruxstack/buildkit 0.0.1, instead of
kreuzwerker's docker_image + docker_registry_image.

Motivation: kreuzwerker/docker needs a Docker Engine to build
(exportLoad) and push (client.ImagePush), which a daemonless TACOS
host (Atlantis, no root-equivalent socket) cannot provide. buildkit_image
solves and pushes directly against a reachable rootless buildkitd.

buildkit_autodiscover and embedded_buildkitd are hardcoded false (not
configurable) as a security control: embedded_buildkitd would let
config spawn a private daemon.

The cdktn binding in src/aws/private/buildkit-provider.ts is
hand-written since no @cdktn/provider-buildkit package is published.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dgqc87KhZ9EijLMoJ61ZXe

@vincenthsh vincenthsh Sep 11, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't do this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants