Skip to content

fix(rpm): sign every package so gpgcheck=1 installs work #4571

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kixelated merged 2 commits into from
Sep 30, 2026
Merged

fix(rpm): sign every package so gpgcheck=1 installs work #4571

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
040eabc
chore(quest): claim quest/m1/rpm-signing
kixelated Sep 30, 2026
86a367c
fix(rpm): sign every package so gpgcheck=1 installs work
kixelated Sep 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 17 additions & 3 deletions infra/rpm/publish.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@
#
# Regenerate yum/dnf repo metadata and push to the rpm-moq-dev R2 bucket.
# Pull the current pool, merge in new .rpm files from $ARTIFACTS_DIR,
# rebuild repodata with createrepo_c, sign repomd.xml with GPG, and upload.
# sign every package, rebuild repodata with createrepo_c, sign repomd.xml
# with GPG, and upload.
#
# Required env:
# ARTIFACTS_DIR directory containing new .rpm files to add
Expand All @@ -12,7 +13,7 @@
# SIGNING_KEY ascii-armored GPG private key (shared with apt repo and maven publishing)
# SIGNING_PASSWORD optional passphrase for SIGNING_KEY
#
# Required tools: rclone, createrepo_c, gpg.
# Required tools: rclone, rpmsign, createrepo_c, gpg.

set -euo pipefail

Expand Down Expand Up @@ -86,11 +87,24 @@ if [[ ${#KEY_IDS[@]} -ne 1 ]]; then
exit 1
fi
KEY_ID="${KEY_IDS[0]}"
# Read the passphrase from a file so it never appears in a process list.
GPG_PASS_ARGS=()
if [[ -n "${SIGNING_PASSWORD:-}" ]]; then
GPG_PASS_ARGS=(--pinentry-mode loopback --passphrase "$SIGNING_PASSWORD")
printf '%s' "$SIGNING_PASSWORD" >"$GNUPGHOME/passphrase"
GPG_PASS_ARGS=(--pinentry-mode loopback --passphrase-file "$GNUPGHOME/passphrase")
fi

# moq.repo sets gpgcheck=1, so dnf rejects any unsigned package. Sign the whole
# merged pool, not just the new artifacts: packages published before signing
# existed need it too. rpmsign skips any package already signed by this key,
# so only new or unsigned packages change and get re-uploaded.
echo ">> Sign packages..."
mapfile -t POOL < <(find "$WORK/${DIST}" -name '*.rpm')
rpmsign --addsign \
--define "_gpg_name ${KEY_ID}" \
--define "_gpg_sign_cmd_extra_args --batch ${GPG_PASS_ARGS[*]}" \
"${POOL[@]}"
Comment on lines +103 to +106

@kixelated kixelated Sep 30, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve signed RPM bytes when re-publishing the same tag

On a later rerun of rpm-repo.yml for an already-published tag, the unconditional copies at lines 65/69 replace that tag鈥檚 signed pool entries with the original unsigned GitHub artifacts. This --addsign call therefore cannot take its identical-signature skip path: GPG creates a fresh timestamped signature and the same package URL gets a different whole-file checksum. Clients holding still-valid pre-rerun repository metadata then download the new bytes and fail DNF checksum verification; no HTTP intermediary cache is required. Preserve the existing signed entry when re-ingesting the same artifact, or use a new immutable object path. Add a two-run, same-artifact regression asserting every published RPM鈥檚 SHA-256 stays unchanged, including the incoming artifact rather than only untouched pool packages.

(Written by OpenAI)

@kixelated kixelated Sep 30, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reproduced this locally with rpm 4.20.1 (the version the nix packaging output hands to CI), and the finding holds:

  • rpmsign --addsign on a package this key already signed prints already contains identical signature, skipping and leaves the bytes untouched. The skip path really does exist.
  • But the unconditional cp at lines 65/69 runs first and overwrites the signed pool entry with the unsigned GitHub artifact, so that skip path is unreachable for any re-ingested package. Signing the same source again mints a fresh timestamped signature, so the whole-file SHA-256 at a stable URL changes.

So this is real, and it is not limited to a hypothetical rerun: the first workflow_dispatch after this merges, against the latest release tag, hits exactly this path, because that tag's packages are already in the pool from the auto-triggered unsigned run.

I am not fixing it here because the fix is a release-semantics decision, not a mechanical one. Two things have to be answered first:

  1. When a rerun sees a filename already in the pool, do we keep the published bytes (immutable pool) or replace them (repodata is authoritative)?
  2. If we keep them, what happens to a legitimately rebuilt artifact carrying the same NEVRA?

Recommendation: option A, make the pool immutable per package filename. Skip the copy when the destination already exists. A rerun then becomes a no-op for package bytes, which is what makes rpmsign's skip path worth having, and the whole run becomes idempotent. It also matches how the repo is consumed: the package URL is a stable identity, and the repodata checksum is what clients verify against.

The cost is that a rebuilt same-version artifact can no longer be republished under the same name. I think that cost is correct. Re-cutting a release should mint a new version, and the escape hatch should be a deliberate bucket edit rather than a routine workflow run.

Worth pairing with it, in the same follow-up:

  • Set metadata_expire in moq.repo. It is unset today, so the stale-metadata window is whatever dnf's default is, and that window is the only thing standing between a rerun and a CHKERR on a client.
  • Make trigger-repo-publish.sh a no-op when the tag's artifacts are already in the pool, so a re-run release workflow does not start the cycle at all.

The two-run regression in this comment is worth having, but it has to assert whichever option we pick, and there is no harness for publish.sh at all today, so I would scope it with the fix rather than bolt a fake rclone and GPG stack onto a release-infra PR.

Suggested as a follow-up quest rather than landed here.

Scope note on what I could and could not verify: I exercised the GPG plumbing locally with a throwaway key, and both new paths work, rpmsign with _gpg_sign_cmd_extra_args --batch --pinentry-mode loopback --passphrase-file, and the existing repomd.xml detached signature now that it uses --passphrase-file instead of --passphrase. I could not run the real flow: no signing key, no R2 bucket.

(Written by Space Bunny Free)


echo ">> Generate repodata per arch..."
for arch in "${ARCHES[@]}"; do
dir="$WORK/${DIST}/${arch}"
Expand Down
Loading