Skip to content

ci: harden release workflow — gate on vulnerabilities, enforce egress#5

Open
Fieldnote-Echo wants to merge 1 commit into
update/upstream-syncfrom
harden/release-gating
Open

ci: harden release workflow — gate on vulnerabilities, enforce egress#5
Fieldnote-Echo wants to merge 1 commit into
update/upstream-syncfrom
harden/release-gating

Conversation

@Fieldnote-Echo

Copy link
Copy Markdown
Owner

Stacked on #4 (base update/upstream-sync). Closes the two substantive gaps found while reviewing our own supply-chain hardening before considering offering it upstream.

The gaps this fixes

  1. Trivy didn't gate the release. The scan ran after push: true, so a vulnerable image was already published by the time it was scanned — detection-after-publish, not prevention.
  2. harden-runner was egress-policy: audit — observability only; it logged exfiltration attempts but didn't stop them.

Changes

  • Scan-before-push, gated. Build → load locally → Trivy gate (fails on a fixable CRITICAL, ignore-unfixed: true) → only then push (fast GHA-cache hit, not a second full build). A vulnerable image is never published.
  • Full report preserved. A second, non-blocking Trivy pass sends CRITICAL,HIGH to the Security tab via SARIF (if: always()), so HIGH advisories are visible without blocking releases — CUDA base layers routinely carry unactionable HIGH CVEs, so gating on those would block every release.
  • Egress enforcement. harden-runner auditblock with an allow-list derived from the workflow's real egress (GitHub/GHCR, PyPI, PyTorch, NVIDIA NGC, Hugging Face model download, Sigstore, Trivy DB).
  • Minor: checkout persist-credentials: false; the CycloneDX SBOM is now attested to the registry (verifiable, matching the build provenance) in addition to being attached to the Release.

Verification

  • actionlint clean; YAML valid (16 steps); all 13 actions remain SHA-pinned.
  • ⚠️ First-run caveat (egress block): the allow-list is best-effort from known endpoints; if a release fails with a harden-runner "Blocked" line, add the named host to allowed-endpoints. It fails closed and reports the exact host — this is expected one-time tuning for block mode, and this PR is the review gate before it takes effect.

Not included (deliberately)

The release.yml itself stays a fork-specific amd64-GPU build — it is not upstream-bound (that would regress upstream's multi-arch matrix). The portable pieces (digest-pinned docker-bake.hcl, SHA-pinning) are the upstream candidates, pending a maintainer's interest.

🤖 Generated with Claude Code

@gemini-code-assist

Copy link
Copy Markdown

Note

Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported.

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Harden release workflow: pre-push Trivy gate + enforced egress + SBOM attestation

✨ Enhancement ⚙️ Configuration changes 🕐 20-40 Minutes

Grey Divider

AI Description

• Build and scan the image locally, failing on fixable CRITICAL CVEs before any push.
• Enforce runner egress blocking with an explicit allow-list of required endpoints.
• Attest the CycloneDX SBOM to GHCR and avoid persisting checkout credentials.
Diagram

graph TD
A["Release job"] --> B["Harden runner (egress block)"] --> C["Checkout (no creds)"] --> D["Build image (load)"] --> E["Trivy gate (fixable CRITICAL)"] --> F["Build & push"] --> G[("GHCR")]
E --> H["Trivy SARIF report/upload"]
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Registry-first push + post-push scan with policy response
  • ➕ Simpler pipeline (single build/push)
  • ➕ Uses digest reference immediately after push
  • ➖ Still publishes a vulnerable artifact before detection
  • ➖ Requires additional registry-side controls to prevent consumption
2. Build to OCI archive, scan artifact, then push
  • ➕ Strong separation: scan immutable artifact before any registry interaction
  • ➕ Avoids relying on local Docker daemon image store
  • ➖ More plumbing (artifact management, push-from-archive)
  • ➖ Potentially slower and more failure modes than current cache-based approach

Recommendation: Keep the PR’s approach: build-and-load → blocking Trivy gate → push. It achieves the primary safety goal (no vulnerable image is ever published) while preserving visibility via a non-blocking SARIF upload, and it keeps runtime reasonable by reusing the GHA build cache rather than introducing artifact shuffling.

Files changed (1) +96 / -19

Other (1) +96 / -19
release.ymlGate release on pre-push Trivy scan and enforce runner egress +96/-19

Gate release on pre-push Trivy scan and enforce runner egress

• Restructures the release job to build and load an image locally, run a blocking Trivy scan that fails on fixable CRITICAL vulnerabilities, and only then push to GHCR. Switches harden-runner from egress audit to egress block with an explicit allow-list, disables checkout credential persistence, and adds SBOM attestation to the registry alongside existing release artifacts.

.github/workflows/release.yml

@qodo-code-review

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📜 Skill insights (0)

Grey Divider


Action required

1. Egress block breaks build 🐞 Bug ☼ Reliability
Description
The release workflow enables egress-policy: block but the allow-list omits endpoints required by
docker/gpu/Dockerfile.optimized (Ubuntu APT repositories and astral.sh), so the Docker build
will be network-blocked and the job will fail before Trivy gating/push when layers aren’t already
satisfied by cache.
Code

.github/workflows/release.yml[R47-71]

+          egress-policy: block
+          allowed-endpoints: >
+            github.com:443
+            api.github.com:443
+            objects.githubusercontent.com:443
+            codeload.github.com:443
+            release-assets.githubusercontent.com:443
+            ghcr.io:443
+            pkg-containers.githubusercontent.com:443
+            actions-results-receiver-production.githubapp.com:443
+            productionresultssa0.blob.core.windows.net:443
+            pypi.org:443
+            files.pythonhosted.org:443
+            download.pytorch.org:443
+            nvcr.io:443
+            huggingface.co:443
+            cdn-lfs.huggingface.co:443
+            cdn-lfs-us-1.hf.co:443
+            fulcio.sigstore.dev:443
+            rekor.sigstore.dev:443
+            tuf-repo-cdn.sigstore.dev:443
+            mirror.gcr.io:443
+            registry-1.docker.io:443
+            auth.docker.io:443
+            production.cloudflare.docker.com:443
Evidence
The workflow’s hard egress allow-list does not include Ubuntu APT mirror hosts or astral.sh, but
the Dockerfile used by the workflow runs apt-get update and downloads the uv installer from
https://astral.sh, which requires outbound network access during the image build.

.github/workflows/release.yml[38-71]
docker/gpu/Dockerfile.optimized[14-20]
docker/gpu/Dockerfile.optimized[44-50]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The release workflow uses `step-security/harden-runner` with `egress-policy: block`, but the `allowed-endpoints` list is missing outbound destinations used during the Docker build (APT mirror hosts and `astral.sh` for installing `uv`). This will cause the `docker/build-push-action` build to fail with blocked egress on a fresh runner or when cache layers are invalidated.

## Issue Context
`docker/gpu/Dockerfile.optimized` performs:
- `apt-get update` / `apt-get install` (requires Ubuntu APT mirrors; typically `archive.ubuntu.com` and `security.ubuntu.com`, often over port 80)
- `curl -LsSf https://astral.sh/uv/install.sh | sh` (requires `astral.sh:443`, and potentially follow-on downloads)

## Fix Focus Areas
- .github/workflows/release.yml[44-71]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Qodo Logo

Comment on lines +47 to +71
egress-policy: block
allowed-endpoints: >
github.com:443
api.github.com:443
objects.githubusercontent.com:443
codeload.github.com:443
release-assets.githubusercontent.com:443
ghcr.io:443
pkg-containers.githubusercontent.com:443
actions-results-receiver-production.githubapp.com:443
productionresultssa0.blob.core.windows.net:443
pypi.org:443
files.pythonhosted.org:443
download.pytorch.org:443
nvcr.io:443
huggingface.co:443
cdn-lfs.huggingface.co:443
cdn-lfs-us-1.hf.co:443
fulcio.sigstore.dev:443
rekor.sigstore.dev:443
tuf-repo-cdn.sigstore.dev:443
mirror.gcr.io:443
registry-1.docker.io:443
auth.docker.io:443
production.cloudflare.docker.com:443

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. Egress block breaks build 🐞 Bug ☼ Reliability

The release workflow enables egress-policy: block but the allow-list omits endpoints required by
docker/gpu/Dockerfile.optimized (Ubuntu APT repositories and astral.sh), so the Docker build
will be network-blocked and the job will fail before Trivy gating/push when layers aren’t already
satisfied by cache.
Agent Prompt
## Issue description
The release workflow uses `step-security/harden-runner` with `egress-policy: block`, but the `allowed-endpoints` list is missing outbound destinations used during the Docker build (APT mirror hosts and `astral.sh` for installing `uv`). This will cause the `docker/build-push-action` build to fail with blocked egress on a fresh runner or when cache layers are invalidated.

## Issue Context
`docker/gpu/Dockerfile.optimized` performs:
- `apt-get update` / `apt-get install` (requires Ubuntu APT mirrors; typically `archive.ubuntu.com` and `security.ubuntu.com`, often over port 80)
- `curl -LsSf https://astral.sh/uv/install.sh | sh` (requires `astral.sh:443`, and potentially follow-on downloads)

## Fix Focus Areas
- .github/workflows/release.yml[44-71]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 687a13b2f9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +47 to +48
egress-policy: block
allowed-endpoints: >

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Allow the Dockerfile's build-time downloads

When this workflow runs the Buildx steps under egress-policy: block, the Dockerfile still performs outbound apt-get update and curl https://astral.sh/uv/install.sh during the image build (docker/gpu/Dockerfile.optimized lines 15-18 and 45-47). This allow-list does not include the Ubuntu/NVIDIA apt hosts or astral.sh, so those deterministic build-time requests are denied before the Trivy gate or push step and the release job cannot publish an image.

Useful? React with 👍 / 👎.

Close the two substantive gaps from the hardening review:
- Trivy no longer runs after 'push: true' (which published before scanning).
  Restructured to build-and-load → gate scan → push: a fixable CRITICAL now
  fails the job BEFORE the image is published. Full CRITICAL+HIGH results still
  go to the Security tab via SARIF (non-blocking, since CUDA base layers carry
  unactionable HIGH advisories); ignore-unfixed keeps the gate actionable.
- harden-runner egress-policy audit → block with an allow-list derived from the
  workflow's known egress (GitHub/GHCR, PyPI, PyTorch, NVIDIA NGC, HF model
  download, Sigstore, Trivy DB). Observability → enforcement.

Plus: checkout persist-credentials: false; attest the CycloneDX SBOM to the
registry (verifiable, matching the build provenance). All actions remain
SHA-pinned; actionlint clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Fieldnote-Echo
Fieldnote-Echo force-pushed the harden/release-gating branch from 687a13b to 0bc6b5f Compare July 19, 2026 00:50

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review


P2 Badge Push the artifact that passed the vulnerability gate

If the BuildKit cache is unavailable, evicted, or invalidated between these steps, this second build reruns mutable apt-get, unpinned uv sync, and remote installer/download commands, so the filesystem pushed here can differ from the SCAN_TAG image Trivy approved. cache-from is only an optimization and does not establish that the pushed digest is the scanned artifact; promote/tag the already scanned image, or scan the exact digest produced by this build before publishing its release tags.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

with:
egress-policy: audit
egress-policy: block
allowed-endpoints: >

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Allow the UniDic dictionary download host

Even after the previously reported apt and Astral hosts are allowed, the image build still fails under this allow-list when docker/gpu/Dockerfile.optimized lines 77–79 execute python -m unidic download: UniDic fetches its dictionary from cotonoha-dic.s3-ap-northeast-1.amazonaws.com, which is not permitted here. Add that endpoint so the default INCLUDE_JAPANESE=true build can complete.

Useful? React with 👍 / 👎.

Comment on lines +155 to +156
- name: Trivy scan (SARIF report)
if: always() && steps.build_scan.outcome == 'success'

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Make the SARIF reporting path non-blocking

When either this second Trivy invocation or the following SARIF upload fails because of a transient tool, database, or code-scanning ingestion error, the step still marks the job failed; the later Build and push step has the implicit success() condition and is therefore skipped. This makes the supposedly non-blocking report an additional release gate even when the preceding CRITICAL scan passed, so the report and upload steps need failure handling such as continue-on-error.

Useful? React with 👍 / 👎.

@@ -135,6 +203,15 @@ jobs:
format: cyclonedx-json

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Generate SPDX JSON for the SBOM attestation

actions/attest-sbom requires the supplied SBOM to use SPDX JSON, but this step writes a CycloneDX JSON document and passes it directly as sbom-path. The attestation action therefore cannot create the SBOM attestation, leaving the job failed after the image has been pushed and preventing the subsequent GitHub Release; generate an SPDX JSON file for this action, or use an attestation mechanism that supports CycloneDX.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant