Skip to content

fix(ci): fix staging release - #917

Merged
kmannislands merged 3 commits into
mainfrom
fix-staging-release-staging-test
Sep 11, 2026
Merged

kmannislands merged 3 commits into
mainfrom
fix-staging-release-staging-test

Conversation

@kmannislands

Copy link
Copy Markdown

The staging release workflow has two independent failures.

1. It referenced a buildkitd image it never built

Error response from daemon: failed to resolve reference
"ghcr.io/earthbuild/earthbuild:buildkitd-staging-<sha>-latest": not found

build-earthly.yml computes TAG_SUFFIX from RUNS_ON (so ubuntu-26.04 for the staging caller) and pushes +ci-release under that suffix — but its own "Save buildkitd image tarball" / "Build earth binary artifact" steps, and all five Retrieve earthbuild from build-earthly job steps in ci-staging-deploy.yml, re-derived the suffix as latest.

-latest is what the CI workflow (ci.yml) pushes. Both workflows fire on the same push to main, so the staging release was implicitly depending on a run it has no edge to — which is exactly the "wrong point in the graph / missing dependency" symptom: it fails immediately on the commit, and succeeds if you retry it after CI has finished. The staging workflow's own build-earthly job meanwhile pushed -ubuntu-26.04 images that nothing consumed.

Fix: build-earthly.yml now exposes the suffix it actually pushed as a workflow_call output, and the callers use it instead of assuming one. The workflow is now self-contained — no cross-workflow ordering required.

2. pull ping error in the release-dockerhub jobs

Error: pull ping error: pull ping response: rpc error: code = Unknown desc = image pull:
  command failed: docker pull 127.0.0.1:40511/sess-.../pullping:img-5
failed to copy: httpReadSeeker: failed open: ... EOF

./release+release-dockerhub builds multi-platform images and --pushes them to ghcr. Without --no-output, EarthBuild also loads them back into the runner's docker daemon through the embedded registry — the pull-ping path that keeps failing (see #912 and Janis's comment on #884).

Nothing in those jobs needs the local copy: the smoke tests run docker run ghcr.io/earthbuild/earthbuild:<tag>, which pulls the image --push just published. So both release-dockerhub invocations now pass --no-output.

Chose --no-output over --ci deliberately: --ci also implies --strict, which is a broader behaviour change than this needs.

release-binaries intentionally keeps output enabled — +signed-release writes the release artifacts to ./release/release/, so --no-output would break it. copy-buildkitd-to-dockerhub is untouched: it is RUN --push skopeo only, with no SAVE IMAGE to output.

Testing

Branch name ends in -staging-test, so the staging release workflow runs on this PR branch.

🤖 Generated with Claude Code

@kmannislands
kmannislands requested a review from a team as a code owner September 11, 2026 15:29
@kmannislands
kmannislands requested review from gilescope and removed request for a team September 11, 2026 15:29
@github-actions

github-actions Bot commented Sep 11, 2026

Copy link
Copy Markdown

⚠️ Are we earthbuild yet?

Warning: "earthly" occurrences have increased by 6 (0.21%)

📈 Overall Progress

Branch Total Count
main 2815
This PR 2821
Difference +6 (0.21%)

Keep up the great work migrating from Earthly to Earthbuild! 🚀

💡 Tips for finding more occurrences

Run locally to see detailed breakdown:

./.github/scripts/count-earthly.sh

Note that the goal is not to reach 0.
There is anticipated to be at least some occurrences of earthly in the source code due to backwards compatibility with config files and language constructs.

@kmannislands kmannislands self-assigned this Sep 11, 2026
@kmannislands kmannislands added the ai-assisted Authored with AI assistance label Sep 11, 2026
@kmannislands
kmannislands force-pushed the fix-staging-release-staging-test branch from 9d6f618 to 7d62948 Compare September 11, 2026 16:22

@janishorsts janishorsts left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Unifying the TAG is neat.

kmannislands and others added 3 commits September 11, 2026 09:37
Two independent failures in the staging release workflow:

1. The buildkitd image tag did not match the one built.
   build-earthly.yml computes TAG_SUFFIX from RUNS_ON (e.g. "ubuntu-26.04")
   and pushes +ci-release under it, but the tarball/artifact steps -- and
   every "Retrieve earthbuild" step in ci-staging-deploy.yml -- re-derived
   the suffix as "latest". "latest" is what the separate CI workflow pushes,
   so the staging release only resolved once CI had pushed that tag for the
   same SHA: it failed on the commit push and passed on a re-run.

   The suffix is now emitted as a workflow_call output and consumed by the
   callers, so the workflow only ever references images its own build-earthly
   job pushed.

2. "pull ping error" in the two release-dockerhub jobs. Those targets are
   multi-platform and --push already publishes them to ghcr; loading a copy
   back into the runner's docker daemon buys nothing and is what trips the
   pull ping (#912). They now run with --no-output. The smoke tests pull the
   pushed image, so they are unaffected. release-binaries deliberately keeps
   output enabled -- it produces local release artifacts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
--ci implies --no-output --strict, and --strict only gates LOCALLY and
interactive debugging. The only LOCALLY in the repo is +fmt-go, which is not
in the release graph, and ci.yml already runs `earth --ci --push +ci-release`
against these same targets.

Policy is now "--ci unless we can't": the copy-buildkitd job takes it too
(+copy-img is RUN --push skopeo, nothing to output). release-binaries is the
one exception -- +signed-release writes ./release/release/ via SAVE ARTIFACT
AS LOCAL, which --no-output would suppress -- and now says so.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
--ci is the norm in this workflow and reading a tag suffix from the job that
produced it is unremarkable, so neither needs explaining. Drop the repeated
rationale and keep one comment on release-binaries, the single job that opts
out of --ci, with a pointer to PR #858 (--no-image-output) as the eventual fix.

Comments only; no behaviour change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@kmannislands
kmannislands force-pushed the fix-staging-release-staging-test branch from b4f47bf to 40ef656 Compare September 11, 2026 16:37
@kmannislands
kmannislands enabled auto-merge (squash) September 11, 2026 17:13
@kmannislands
kmannislands merged commit 113ec9d into main Sep 11, 2026
98 checks passed
@kmannislands
kmannislands deleted the fix-staging-release-staging-test branch September 11, 2026 19:37
gilescope pushed a commit that referenced this pull request Sep 21, 2026
fix(ci): fix staging release (#917)

Line 29 of docs-internals/plan-merge-main.md. Nothing under engine/.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQ2c9xLfQvSY8YGLr7o5tv
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-assisted Authored with AI assistance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants