Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 47 additions & 60 deletions .github/workflows/release-image.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ jobs:
GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
SYNOLOGY_RUNNER_BASE_DIR: /volume1/docker/github-runner-fleet
COMPOSE_PROJECT_NAME: github-runner-fleet
RELEASE_LOG_DIR: /tmp/runner-release
timeout-minutes: 45
steps:
- uses: actions/checkout@v7
Expand Down Expand Up @@ -121,7 +122,7 @@ jobs:

- name: guard main branch before any publish
run: |
if [[ "${GITHUB_REF_NAME}" != "main" ]]; then
if [[ "${GITHUB_REF}" != "refs/heads/main" ]]; then
echo "release-image may only publish from main" >&2
exit 1
fi
Expand All @@ -130,38 +131,7 @@ jobs:
id: release_state
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
set -euo pipefail
final_ref="${{ steps.release_meta.outputs.image_ref }}"
staging_ref="${{ steps.release_meta.outputs.image_repo }}:candidate-${{ steps.release_meta.outputs.image_tag }}-${GITHUB_SHA}"
final_digest=""
release_exists=false

if final_digest="$(docker buildx imagetools inspect "${final_ref}" --format '{{.Manifest.Digest}}' 2>/tmp/final-image-inspect.log)"; then
test -n "${final_digest}"
publish_required=false
working_ref="${final_ref}"
echo "found existing immutable candidate ${final_ref}@${final_digest}; verification-only recovery mode"
else
publish_required=true
working_ref="${staging_ref}"
echo "final tag is absent; build will publish unique staging reference ${staging_ref}"
fi

if gh release view "${{ steps.release_meta.outputs.release_tag }}" >/tmp/github-release.txt 2>/tmp/github-release-error.log; then
release_exists=true
fi
if [[ "${release_exists}" == "true" && -z "${final_digest}" ]]; then
echo "GitHub Release exists but ${final_ref} is missing; refusing registry mutation" >&2
exit 1
fi

{
echo "publish_required=${publish_required}"
echo "working_ref=${working_ref}"
echo "final_digest=${final_digest}"
echo "release_exists=${release_exists}"
} >> "${GITHUB_OUTPUT}"
run: bash scripts/release/preflight.sh "${{ steps.release_meta.outputs.image_ref }}" "${{ steps.release_meta.outputs.release_tag }}"

- name: Build and push unique staging reference
if: ${{ steps.release_state.outputs.publish_required == 'true' }}
Expand Down Expand Up @@ -190,23 +160,24 @@ jobs:

- name: Sign image digest
if: ${{ steps.release_state.outputs.publish_required == 'true' }}
run: cosign sign --yes "${{ steps.release_state.outputs.working_ref }}@${{ steps.image_digest.outputs.digest }}"
run: bash scripts/release/retry.sh cosign-sign-index cosign sign --yes "${{ steps.release_state.outputs.working_ref }}@${{ steps.image_digest.outputs.digest }}"

- name: Sign per-platform image digests
if: ${{ steps.release_state.outputs.publish_required == 'true' }}
run: |
set -euo pipefail
image_repo="${{ steps.release_meta.outputs.image_repo }}"
subject="${{ steps.release_state.outputs.working_ref }}@${{ steps.image_digest.outputs.digest }}"
docker buildx imagetools inspect "${subject}" --raw \
| jq -r '.manifests[].digest' \
| while IFS= read -r digest; do
cosign sign --yes "${image_repo}@${digest}"
bash scripts/release/retry.sh "cosign-sign-${digest#sha256:}" cosign sign --yes "${image_repo}@${digest}"
done

- name: Attach SBOM attestation
if: ${{ steps.release_state.outputs.publish_required == 'true' }}
run: |
cosign attest --yes \
bash scripts/release/retry.sh cosign-sbom-attest cosign attest --yes \
--predicate /tmp/github-runner-fleet.spdx.json \
--type spdxjson \
"${{ steps.release_state.outputs.working_ref }}@${{ steps.image_digest.outputs.digest }}"
Expand All @@ -226,34 +197,33 @@ jobs:
timeout 5m cosign verify \
--certificate-identity-regexp "https://github.com/${{ github.repository }}/.github/workflows/release-image.yml@refs/heads/main" \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
"${subject}" 2>&1 | tee /tmp/cosign-signature-verify.log
"${subject}" 2>&1 | tee "${RELEASE_LOG_DIR}/cosign-signature-verify.log"
timeout 5m cosign verify-attestation \
--type spdxjson \
--certificate-identity-regexp "https://github.com/${{ github.repository }}/.github/workflows/release-image.yml@refs/heads/main" \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
"${subject}" 2>&1 | tee /tmp/cosign-sbom-verify.log
"${subject}" 2>&1 | tee "${RELEASE_LOG_DIR}/cosign-sbom-verify.log"
timeout 5m cosign verify-attestation \
--type slsaprovenance \
--certificate-identity-regexp "https://github.com/${{ github.repository }}/.github/workflows/release-image.yml@refs/heads/main" \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
"${subject}" 2>&1 | tee /tmp/cosign-provenance-verify.log
"${subject}" 2>&1 | tee "${RELEASE_LOG_DIR}/cosign-provenance-verify.log"

- name: Preserve verification diagnostics
if: ${{ failure() }}
uses: actions/upload-artifact@v7
with:
name: release-verification-diagnostics-${{ github.run_attempt }}
path: |
/tmp/cosign-*-verify.log
/tmp/final-image-inspect.log
/tmp/github-release*.log
/tmp/imagetools.txt
if-no-files-found: warn
- name: Validate immutable image runtimes before promotion
run: |
set -euo pipefail
bash scripts/release/validate-runtime.sh \
"${{ steps.release_state.outputs.working_ref }}@${{ steps.image_digest.outputs.digest }}" \
2>&1 | tee "${RELEASE_LOG_DIR}/runtime-validation.log"

- name: Promote verified digest to final tag
if: ${{ steps.release_state.outputs.publish_required == 'true' }}
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
RELEASE_REQUIRE_ABSENT: "true"
run: |
set -euo pipefail
bash scripts/release/preflight.sh "${{ steps.release_meta.outputs.image_ref }}" "${{ steps.release_meta.outputs.release_tag }}"
subject="${{ steps.release_state.outputs.working_ref }}@${{ steps.image_digest.outputs.digest }}"
docker buildx imagetools create \
--tag "${{ steps.release_meta.outputs.image_ref }}" \
Expand All @@ -274,24 +244,41 @@ jobs:
sleep 10
done

- run: |
docker run --rm --platform linux/amd64 --entrypoint /bin/sh "${{ steps.release_meta.outputs.image_ref }}" -lc \
'test "$(cd /actions-runner && ./bin/Runner.Listener --version)" = "$(cat /.runner-version)" && command -v pgrep && pgrep --version | head -n 1 && docker --version && node --version && python3 --version && terraform version | head -n 1'

- run: |
docker run --rm --platform linux/arm64 --entrypoint /bin/sh "${{ steps.release_meta.outputs.image_ref }}" -lc \
'test "$(cd /actions-runner && ./bin/Runner.Listener --version)" = "$(cat /.runner-version)" && command -v pgrep && pgrep --version | head -n 1 && docker --version && node --version && python3 --version && terraform version | head -n 1'

- if: ${{ inputs.publish_project_release }}
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
if gh release view "${{ steps.release_meta.outputs.release_tag }}" >/dev/null 2>&1; then
set -euo pipefail
if [[ "${{ steps.release_state.outputs.release_exists }}" == "true" ]]; then
echo "release ${{ steps.release_meta.outputs.release_tag }} already exists; verification completed without mutation"
exit 0
fi
if [[ "${{ steps.release_state.outputs.publish_required }}" != "true" && "${{ steps.release_state.outputs.source_verified }}" != "true" ]]; then
echo "refusing to create a GitHub Release for an existing image at this dispatch SHA" >&2
exit 1
fi

if [[ "${{ steps.release_state.outputs.tag_exists }}" == "true" ]]; then
tag_identity="$(gh api "repos/${GITHUB_REPOSITORY}/git/ref/tags/${{ steps.release_meta.outputs.release_tag }}" --jq '.object | "\(.type) \(.sha)"')"
test "${tag_identity}" = "commit ${GITHUB_SHA}"
else
# Create atomically; never silently reuse an unrelated source tag.
gh api "repos/${GITHUB_REPOSITORY}/git/refs" --method POST \
-f "ref=refs/tags/${{ steps.release_meta.outputs.release_tag }}" \
-f "sha=${GITHUB_SHA}" >/dev/null
fi

gh release create "${{ steps.release_meta.outputs.release_tag }}" \
--target "${GITHUB_SHA}" \
--verify-tag \
--title "${{ steps.release_meta.outputs.release_tag }}" \
--generate-notes

- name: Preserve verification diagnostics
if: ${{ failure() }}
uses: actions/upload-artifact@v7
with:
name: release-verification-diagnostics-${{ github.run_attempt }}
path: |
${{ env.RELEASE_LOG_DIR }}/*.log
/tmp/imagetools.txt
if-no-files-found: warn
2 changes: 1 addition & 1 deletion .runner-version
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2.336.0
2.337.0
16 changes: 9 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -289,15 +289,17 @@ The release workflow:
- enforces that `package.json` version matches `config/pools.yaml` image tag
- validates `config/pools.yaml`
- runs the local `pnpm smoke-test` contract on `linux/amd64`
- publishes the configured tag from `config/pools.yaml`
- verifies the pushed tag with `docker buildx imagetools inspect`
- requires confirmed absence of the final image, Git tag, and GitHub Release before a new publication
- pushes a unique `candidate-<version>-<commit>-<run-id>-<attempt>` reference
- resolves the candidate's immutable digest with `docker buildx imagetools inspect`
- confirms both `linux/amd64` and `linux/arm64` are present
- signs the pushed digest with keyless cosign and attaches SPDX SBOM plus SLSA provenance attestations
- verifies the signature and attestations before release creation
- retries transient signing/attestation failures with bounded attempts and preserves failure diagnostics
- verifies signatures, attestations, and both platform runtimes against the immutable candidate digest before promotion
- rechecks final-tag absence and promotes the verified digest to the configured version tag
- retries `pnpm validate-image` until the GitHub Packages API sees the new tag
- runs post-publish toolchain checks for both `linux/amd64` and `linux/arm64`
- automatically creates the matching GitHub release tag `v<version>` after successful publishes from `main`
- can still be dispatched manually from `main`; set `publish_project_release=true` to create the matching GitHub Release during a manual run
- creates the matching source Git tag `v<version>` and GitHub Release after successful validation
- runs only on explicit dispatch from `refs/heads/main`; `publish_project_release=true` is the default

Only point [config/pools.yaml](config/pools.yaml) at a tag that this workflow has already published and verified.

Expand All @@ -310,7 +312,7 @@ cosign verify \
ghcr.io/omt-global/github-runner-fleet:<tag>
```

To keep the repository release and GHCR image tag aligned, merge the version bump to `main`. The release workflow first checks whether the matching repo release already exists; if it does, the automatic run fails before publishing an image so an existing GHCR version tag is not replaced. For a new version, the workflow publishes and verifies the image, then creates the matching repo tag and GitHub Release.
To keep the repository release and GHCR image tag aligned, merge the version bump to `main`, then dispatch `Release Image`. Routine main pushes do not publish. Completed releases can be verified again without mutation. Recovery can create missing release metadata only when verified provenance binds the existing image to the exact dispatch commit and any existing Git tag matches that SHA. Follow the [release recovery procedure](docs/release-recovery.md) to complete a same-source retry or supersede an older candidate with a new version.

## Runtime Contract

Expand Down
10 changes: 7 additions & 3 deletions docs/release-recovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,16 @@

`Release Image` is manual and transactional. It no longer publishes on every push to `main`.

For a new version, first update the canonical version and image tag together, merge that change, and dispatch the workflow from `main`. The workflow publishes a unique `candidate-<version>-<commit>` reference, signs and verifies its immutable digest, and only then promotes that digest to the final version tag and creates the matching GitHub Release.
For a new version, first update the canonical version and image tag together, merge that change, and dispatch the workflow from `refs/heads/main`. The workflow publishes a unique `candidate-<version>-<commit>-<run-id>-<attempt>` reference. A retry never reuses an earlier candidate tag. It signs and verifies the immutable digest and checks that exact image's runtime on both `linux/amd64` and `linux/arm64` before promoting the digest to the final version tag and creating the matching GitHub Release.

For a partially published version such as `0.2.1`, dispatch the workflow without changing the version. If the final image tag already exists, the workflow enters verification-only recovery mode: it does not rebuild, replace, re-sign, or re-attest the digest. If the existing signature, SBOM attestation, and SLSA provenance verify, the workflow may safely create the missing GitHub Release. If verification fails, supersede the candidate with a new version; never replace the existing tag.
When the final image already exists, the workflow enters verification-only recovery mode: it does not rebuild, replace, re-sign, or re-attest the digest. It verifies the existing signature, SBOM attestation, SLSA provenance, and platform runtimes. A completed Git tag and GitHub Release are left unchanged.

If a final image exists without its GitHub Release, automated recovery first verifies GitHub's source provenance for that exact digest. The provenance must match this repository, `release-image.yml`, `refs/heads/main`, and the current dispatch commit SHA. Any existing Git tag must also be a lightweight tag at that exact SHA. Only then may the workflow finish creating the missing release metadata. This permits a same-source retry after an image-only dispatch (`publish_project_release=false`) or a failed release creation without assigning an unrelated commit to the image. If source verification fails or `main` has advanced, supersede the candidate with a new version; never replace the existing tag.

## `0.2.1` and `0.2.2` recovery decision

The repository cannot prove `0.2.1` completed the immutable release contract, so it is frozen as an unreleased candidate and must not be replaced. Version `0.2.2` was never published as a verified project release, so it must not be treated as a completed release either. Version `0.2.3` supersedes these candidates and is the next permitted release target.

If a GitHub Release exists but its image tag is missing, the workflow fails before any registry mutation. Verification commands are bounded to five minutes and their logs are retained as a diagnostic artifact on failure.
If a GitHub Release or Git tag exists but its image tag is missing, the workflow fails before any registry mutation. Authentication failures, registry outages, and unexpected lookup errors also stop publication; only confirmed manifest absence and GitHub HTTP 404 responses permit a new release. The workflow checks absence again immediately before promotion, and creates the source Git tag with a create-only API call so it cannot silently reuse an unrelated tag.

Signing and SBOM attestation retry transient transport failures up to three times, with a two-minute limit per attempt. This covers the July 28, 2026 candidate's failed Rekor request without disabling transparency-log verification. Permanent signing errors stop immediately. Verification commands remain bounded to five minutes. Lookup, signing, attestation, verification, and runtime logs are retained as a diagnostic artifact on failure.
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
"node": ">=20.0.0"
},
"scripts": {
"build": "tsc -p tsconfig.json",
"build": "tsc -p tsconfig.json && node scripts/copy-build-metadata.mjs",
"audit-log": "tsx src/cli.ts audit-log",
"check-runner-version": "tsx src/cli.ts check-runner-version",
"config-diff": "tsx src/cli.ts config-diff",
Expand Down
6 changes: 6 additions & 0 deletions scripts/copy-build-metadata.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
import { copyFileSync } from "node:fs";

copyFileSync(
new URL("../.runner-version", import.meta.url),
new URL("../dist/.runner-version", import.meta.url)
);
Loading
Loading