feat: use docker-compose library, optionally download docker-buildx, fixes #7915, for #8295

[stasadev]

The Issue

• Fixes Future strategy for Docker Compose (binary vs library) and Docker Buildx #7915
• For ddev poweroff takes longer than it should, seems to hang after each project is stopped #8295

Related:

• feat: add Docker buildx plugin as hard requirement, fixes #8117 #8149 (added buildx requirement but did not implement the download or SDK switch)
• perf: combined startup time optimizations, for #8096 #8145 (performance optimizations caused output corruption when using the standalone compose binary on first run with no images)

How This PR Solves The Issue

We added a requirement for docker-buildx in #8149. Docker Compose has required this since v2.40.2, but the requirement was only triggered when installing extra packages or using EOL PHP versions like 7.4.

This PR replaces the bundled docker-compose binary with the Docker Compose SDK and adds optional download of docker-buildx to $HOME/.ddev/bin.

Docker Compose (SDK)

• Switched from a standalone docker-compose binary to the Docker Compose Go library
• cliPluginsExtraDirs is set to $HOME/.ddev/bin in docker_manager.go so plugins stored there are found automatically (this only works with the SDK, not the standalone binary)
• Removed the goroutine for docker-compose pull in ddev start - the concurrent output was corrupted on first run when no Docker images were present, caused by the performance optimizations in perf: combined startup time optimizations, for #8096 #8145
• Docker Compose output is now shown as-is (with stopwatch on the right), previously filtered manually

Docker Buildx (optional download)

• By default, DDEV uses the system-provided docker-buildx (avoids overriding Docker Desktop's patched builds and keeps IDE compatibility)
• Added --docker-buildx-version flag to ddev config global to optionally download a specific buildx version to $HOME/.ddev/bin/docker-buildx
• ddev config global --docker-buildx-version= or --docker-buildx-version=system resets to the system buildx
• Removed the buildx check from root.go; buildx is now checked/downloaded only in ddev start, ddev version, ddev utility rebuild, and ddev utility dockercheck
• ddev utility dockercheck shows a warning when a specific buildx version is configured (useful for support requests)
• Buildx is not downloaded when Docker host is unreachable

Other changes

• ddev version now works when Docker is completely broken - shows all version details, reports the Docker error at the end
• ddev config now works without a working Docker (allows configuring a project or setting --docker-buildx-version in ddev config global)
• ddev poweroff output is significantly improved - previously showed no output while waiting, reported in ddev poweroff takes longer than it should, seems to hang after each project is stopped #8295
• ddev utility dockercheck now uses dockerutil.RunCLIPluginCommand instead of calling docker buildx directly

Manual Testing Instructions

Buildx:

ddev config global --docker-buildx-version=0.33.0 && ddev version
# Should download buildx and show 0.33.0 in docker-buildx line

ddev config global --docker-buildx-version=foo && ddev version
# Should fail with buildx error

ddev start
# Should fail with the same buildx error

ddev config global --docker-buildx-version=foo && DOCKER_HOST=foo ddev version
# Should show a Docker error, not a buildx error

ddev config global --docker-buildx-version=0.32.1 && DOCKER_HOST=foo ddev version
# Should not download anything (Docker host is wrong)

DOCKER_HOST=foo ddev version
# Should fail but show all version details

ddev config global --docker-buildx-version=0.33.0 && ddev utility dockercheck
# Should show warning about using a specific buildx version

ddev config global --docker-buildx-version=
ddev config global --docker-buildx-version=system
# Both should reset to system buildx

Compose:

# Remove local docker-compose, confirm it is not re-downloaded
rm ~/.ddev/bin/docker-compose && ddev version

# Run start and note the modern compose output (stopwatch on the right)
ddev start
ddev start --no-cache

# Test fresh start
ddev delete images --all
docker buildx prune
ddev start

Compare with DDEV HEAD, use --no-cache due to #8145 output differences):

DDEV_DEBUG=true ddev restart
DDEV_DEBUG=true ddev restart --json-output
DDEV_DEBUG=true ddev restart --no-cache
DDEV_DEBUG=true ddev restart --json-output --no-cache

DDEV_VERBOSE=true ddev restart
DDEV_VERBOSE=true ddev restart --json-output
DDEV_VERBOSE=true ddev restart --no-cache
DDEV_VERBOSE=true ddev restart --json-output --no-cache

Compare ddev poweroff output on DDEV HEAD vs this PR - output should now show progress instead of silence.

Run Craft CMS quickstart https://docs.ddev.com/en/stable/users/quickstart/#craft-cms, it should show the usual output for composer install and ask interactive questions, see the comment below why this is important.

Automated Testing Overview

• Added pkg/dockerutil/docker_buildx.go with tests in docker_buildx_test.go
• Added pkg/dockerutil/docker_compose_internal_test.go for internal compose logic
• Extended docker_compose_test.go and docker_manager_test.go
• Added stdin_dup.go / stdin_dup_windows.go with tests (stdin duplication for SDK integration)

Release/Deployment Notes

• docker-compose is no longer downloaded to $HOME/.ddev/bin - existing binaries there are ignored (I didn't add any cleanup for $HOME/.ddev/bin/docker-compose)
• New global hidden config option docker-buildx-version in ~/.ddev/global_config.yaml
• Behavior change: buildx check is skipped in most commands; only runs on ddev start, ddev version, ddev utility rebuild, ddev utility dockercheck

[github-actions]

Download the artifacts for this pull request:

• ddev-windows-arm64-installer.zip
• ddev-windows-arm64.zip
• ddev-windows-amd64-installer.zip
• ddev-windows-amd64.zip
• ddev-linux-amd64.zip
• ddev-linux-arm64.zip
• ddev-macos-arm64.zip
• ddev-macos-amd64.zip
• all-ddev-executables.zip

See Testing a PR.

[rfay]

Rebased, trivial merge conflict.

[stasadev]

About the new private docker-buildx binary: I lean toward using the system-provided buildx.

Reason: all IDEs run Docker containers in one way or another, and they won't see our privately stored docker-buildx binary - there isn't even an option for that, unlike what we had for docker-compose.

I'm also a bit worried that when you run Docker Desktop, buildx shows a version like v0.33.0-desktop.1 - they may patch it somehow, and we might not even know the difference.

We also don't see reports of issues with the docker-buildx requirement. I'll still provide a global flag to download a specific version to store it in $HOME/.ddev/bin.

[rfay]

• Perhaps https://ddev--8234.org.readthedocs.build/en/8234/users/install/docker-installation/#docker-buildx_1 needs a mention of the new technique
• Maybe https://ddev--8234.org.readthedocs.build/en/8234/users/usage/offline/
• I think --docker-buildx-version is missing from https://ddev--8234.org.readthedocs.build/en/8234/users/usage/commands/#config ? Maybe I'm blind.
• There may be some other references to buildx that would be worth a touch.

And I know I'm late to the party on --no-cache, probably shouldn't mention in this PR, but it was in the testing instructions :) Since we don't actually "build images" I think the verbiage in help and docs "--no-cache: Build Docker images without using cache." isn't perhaps as helpful as it could be, but OTOH making it more correct may just add complexity. We actually just add small docker layers to existing image, so probably a complete description would be "--no-cache: Build additional docker image layers without cache" or something.

[rfay]

Wow, it just all seems so fast and so smooth. All the manual testing is just lovely. Is it really as much faster and smoother as it seems?

New ddev binary size about 35MB compared to previous 27MB, but we don't need/use docker-compose any more!

[stasadev]

Is it really as much faster and smoother as it seems?

In my opinion, showing a spinner and a stopwatch for all docker-compose commands makes them feel faster.

[rfay]

It sure seems beautiful and smooth. It should go in when you're ready, and of course there will always be more details that can be handled in followups. I looked through the diffs and saw no red flags, but of course that doesn't mean much. I did the manual testing and it was lovely.

ddev config global --docker-buildx-version=0.16.0 is possible, which is odd, but an offbeat thing to do and not worth protecting against. But it's a quick way to see the failure when buildx version constraint not met.

[rfay]

Don't forget to queue up an update to the buildx blog, https://ddev.com/blog/docker-buildx-requirement-v1-25-1/

[stasadev]

Post-start hook hangs (often minutes) on ddev start

Analysis by Claude Code (Anthropic AI), based on goroutine dumps and traces collected during a live ddev start hang.

Symptom

After commit ca19ab9a8 (use docker-compose library), ddev start freezes at the post-start hook — usually minutes, occasionally as little as 5s — with no output. The hook's own script (e.g. echo 123) doesn't run until the hang ends; the delay is before the script, not after.

Reproducer

.ddev/config.yaml:

hooks:
  post-start:
    - exec-host: $DDEV_EXECUTABLE version

Then ddev start. Standalone time ddev hook-start runs in ~0.4s; under the post-start hook the same child takes minutes.

Root cause

The child ddev hook-start blocks in bubbletea's package init() before main() runs. Goroutine dump:

github.com/charmbracelet/bubbletea.init.0
  → lipgloss.HasDarkBackground
  → termenv.Output.termStatusReport
  → os.(*File).Read(fd=1)

vendor/github.com/charmbracelet/bubbletea/tea_init.go writes an OSC "query background color" escape to the TTY and reads the reply. Default termenv.OSCTimeout = 5s — but per-byte, so a slow/garbled reply stretches into minutes.

cmd/ddev/cmd/root.go imports pkg/tui, which imports bubbletea, so this init() runs on every ddev invocation — including hook subprocesses.

Why now

The import isn't new (came with pkg/tui in commit 4ba2d92a). ca19ab9a8 made it reliably hang because compose now runs in-process — the parent ddev emits its own ANSI on the same TTY throughout compose up. The child's OSC reply gets consumed/garbled/lost, and read() waits.

Standalone ddev hook-start is fine because the terminal is idle.

Proof

Toggling only the pkg/tui import, all other code unchanged:

pkg/tui import	Hook elapsed
present	5.43s (this run; usually much longer)
removed	0.45s

Possible fixes

• A. Set TERM=dumb on the hook subprocess via cmd.Env in ExecHostTask.Execute(). termenv skips the OSC probe for dumb/screen/tmux (termenv_unix.go:237-240). ~10-line fix, but the script sees TERM=dumb.
• B. Stop importing pkg/tui from the always-loaded path — launch the TUI as a separate ddev tui process or behind an interface shim. The architectural fix.
• C. Upgrade bubbletea to v2 (the probe is gone there). Big vendor diff.

Recommendation: ship A now, file an issue for B.

Side note: when a hung child is killed, the TTY is left in ICANON=false ECHO=false — termenv's deferred raw-mode restore doesn't run on signal. Worth fixing separately.

[rfay]

I wonder if upgrading to 2.0 of bubbletea would affect this.

[stasadev]

I wonder if upgrading to 2.0 of bubbletea would affect this.

I'm going to update it to v2 since it should be the fastest option.

I don't want to add TERM=dumb or edit vendor - that feels wrong.

Edit: it's resolved, bubbletea v2 doesn't have this problem.

[stasadev]

So far, so good. Rebased and waiting for tests.

[rfay]

Awesome and amazing, it will be great to get it in. Pull when you're happy. Or tell me to if you go to bed waiting for tests.