From 5486eea8858c5807ca58d36e558760f0d2e0e3da Mon Sep 17 00:00:00 2001 From: mlevkov Date: Wed, 23 Sep 2026 15:30:26 -0700 Subject: [PATCH 01/26] ci: add a script that verifies a release run step by step A green release.yml run can hide a failure: the crates.io publish step has failed on every stable release (v0.2.0 through v0.4.1) under continue-on-error, and the Actions API reports such a step as a success at both step and job level. scripts/verify-release.sh checks what each step printed, the artifacts, the release and its assets, the Pages deployment record, and that the host platform's binary runs and reports its version. A hyphenated tag is checked as a pre-release: publish skipped, Latest untouched. It was run against the v0.4.0 and v0.4.1 releases, which fail only on the masked publish step, and the v0.4.1-ci.1 exercise run, which fails only on its since-deleted release, and mutation-checked for a wrong or pre-release changelog base. Log checks drop the "##[group]Run" blocks, where GitHub echoes a step's script source, and read files rather than pipes, so a negated grep cannot pass by SIGPIPE. CONTRIBUTING.md gains a Releasing section, including why an exercise tag must be deleted: release.yml's changelog starts at the nearest earlier tag, so a leftover pre-release tag drops every commit before it from the changelog release.yml writes for the next release. --- CHANGELOG.md | 11 ++ CONTRIBUTING.md | 20 +++ scripts/verify-release.sh | 271 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 302 insertions(+) create mode 100755 scripts/verify-release.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index 034c906..19c4cbe 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- `scripts/verify-release.sh` checks a release.yml run step by step instead + of trusting its conclusion: what each step printed, the artifacts, the + GitHub Release and its assets, the Pages deployment record, and that the + published binary for the host platform runs and reports its version. It + also fails on a step that failed under `continue-on-error`, which the + Actions API reports as a success; the crates.io publish step has failed + that way on every stable release (TD-2026-09-04). CONTRIBUTING.md now + describes releasing and exercising `release.yml` with a pre-release tag + ## [0.4.1] - 2026-09-23 Security patch for four RustSec advisories: one reachable on the public API diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index fa47f04..efe1078 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -183,6 +183,26 @@ src/ └── services/ # Business logic ``` +## Releasing + +Pushing a `v*` tag runs `.github/workflows/release.yml`, which builds the +binaries, creates the GitHub Release and deploys the docs; only admins can +create tags. A green run does not mean every step worked (a step under +`continue-on-error` fails silently), so once the run completes, check what it +did: + +```bash +scripts/verify-release.sh v0.4.2 +``` + +To exercise a `release.yml` change without releasing, push a pre-release tag +of the current `Cargo.toml` version, such as `v0.4.1-ci.1` (the validate job +compares only the part before the hyphen), and verify it the same way. Then +delete it with `gh release delete v0.4.1-ci.1 --cleanup-tag --yes`. Do not +leave it behind: the changelog starts from the nearest earlier tag +(`git describe --tags`), so a leftover pre-release tag drops every commit +before it from the next release's notes. + ## Questions? - Open a [Discussion](https://github.com/mlevkov/iggy_sample/discussions) for questions diff --git a/scripts/verify-release.sh b/scripts/verify-release.sh new file mode 100755 index 0000000..3166373 --- /dev/null +++ b/scripts/verify-release.sh @@ -0,0 +1,271 @@ +#!/usr/bin/env bash +# Verify a release.yml run step by step, not just its overall conclusion. +# +# Usage: scripts/verify-release.sh [] +# +# Checks the latest release.yml run for : job conclusions, what the +# steps printed, the uploaded artifacts, the GitHub Release, the Pages +# deployment, and one published binary. is where the changelog +# should start; by default it is what release.yml computes, +# `git describe --tags --abbrev=0 ^`. +# +# A tag with a hyphen (v0.4.1-ci.1) is a pre-release: the publish job must be +# skipped and the release must not become Latest. To exercise a release.yml +# change without releasing, tag the current Cargo.toml version with a +# pre-release suffix (the validate job compares the part before the hyphen), +# verify the run, then delete the release and its tag. A leftover pre-release +# tag becomes the changelog base of the next release: see CONTRIBUTING.md. +# +# Requires gh (authenticated), git, perl, tar and curl. Exits 0 when every +# check passes (skips allowed), 1 when any fails, 2 on a usage or setup +# error. The work directory is kept when a check fails. + +set -uo pipefail + +usage() { + echo "usage: $0 []" >&2 + exit 2 +} + +case ${1:-} in + -h | --help) + awk 'NR > 1 && /^#/ { sub(/^# ?/, ""); print; next } NR > 1 { exit }' "$0" + exit 0 + ;; + "") usage ;; +esac +[ $# -le 2 ] || usage + +TAG=$1 +VERSION=${TAG#v} +BASE=${VERSION%%-*} +case $VERSION in + *-*) PRERELEASE=true ;; + *) PRERELEASE=false ;; +esac +TAB=$(printf '\t') + +ROOT=$(git rev-parse --show-toplevel) || exit 2 +cd "$ROOT" || exit 2 +REPO=$(gh repo view --json nameWithOwner -q .nameWithOwner) || exit 2 +CRATE=$(awk -F'"' '/^name = / { print $2; exit }' Cargo.toml) +[ -n "$CRATE" ] || { echo "cannot read the package name from Cargo.toml" >&2; exit 2; } + +tmp=${TMPDIR:-/tmp} +WORK=$(mktemp -d "${tmp%/}/verify-release.XXXXXX") || exit 2 +failures=0 +passes=0 +skips=0 +cleanup() { + if [ "$failures" -eq 0 ]; then + rm -rf "$WORK" + else + echo "work directory kept for inspection: $WORK" + fi +} +trap cleanup EXIT + +pass() { printf 'PASS %s\n' "$1"; passes=$((passes + 1)); } +fail() { printf 'FAIL %s\n' "$1"; failures=$((failures + 1)); } +skip() { printf 'SKIP %s\n' "$1"; skips=$((skips + 1)); } +# check [args...]: PASS when the command succeeds. +check() { + local desc=$1 + shift + if "$@"; then pass "$desc"; else fail "$desc"; fi +} +# Exact whole-line match against a file. Checks read files, never a pipe: +# under pipefail, `producer | grep -q` fails when grep exits early and +# SIGPIPEs the producer, which turns every negated check into a vacuous pass. +has_line() { grep -qxF -- "$1" "$2"; } +lacks_regex() { ! grep -qiE -- "$1" "$2"; } + +# -------------------------------------------------------------------------- +# The run +# -------------------------------------------------------------------------- +run=$(gh run list --workflow release.yml --branch "$TAG" --limit 1 \ + --json databaseId,headSha,status,url \ + -q '.[] | "\(.databaseId) \(.headSha) \(.status) \(.url)"') +[ -n "$run" ] || { echo "no release.yml run found for $TAG" >&2; exit 2; } +read -r RID SHA STATUS URL <<<"$run" +[ "$STATUS" = completed ] || { echo "run $RID is $STATUS; verify it once it completes" >&2; exit 2; } + +git cat-file -e "$SHA^{commit}" 2>/dev/null || git fetch --quiet origin +git fetch --quiet --tags origin +PREV=${2:-$(git describe --tags --abbrev=0 "$SHA^" 2>/dev/null)} +[ -n "$PREV" ] || { echo "cannot determine the previous tag; pass it as the second argument" >&2; exit 2; } + +echo "repo $REPO, tag $TAG ($([ "$PRERELEASE" = true ] && echo pre-release || echo stable)), commit ${SHA:0:7}" +echo "run $RID: $URL" +echo "changelog base: $PREV" +echo + +# -------------------------------------------------------------------------- +# Jobs +# -------------------------------------------------------------------------- +gh run view "$RID" --json jobs --jq '.jobs[] | "\(.name)\t\(.conclusion)"' >"$WORK/jobs.tsv" || exit 2 +builds=$(grep -c '^Build (' "$WORK/jobs.tsv") +check "build jobs ran ($builds)" test "$builds" -gt 0 +# shellcheck disable=SC2016 # $1 and $2 are awk fields +check "every job except publish succeeded" \ + awk -F'\t' '$1 != "Publish to crates.io" && $2 != "success" { bad = 1 } END { exit bad }' "$WORK/jobs.tsv" +publish=$(awk -F'\t' '$1 == "Publish to crates.io" { print $2 }' "$WORK/jobs.tsv") +if [ -z "$publish" ]; then + skip "publish job (not in this run)" +elif [ "$PRERELEASE" = true ]; then + check "publish job skipped for a pre-release" test "$publish" = skipped +else + check "publish job ran and reported success" test "$publish" = success +fi + +# -------------------------------------------------------------------------- +# What the steps printed +# -------------------------------------------------------------------------- +# GitHub echoes each step's script source, inputs and env inside a +# "##[group]Run ..." block, so a grep over the raw log matches what a script +# says (echo "No previous tag found") rather than what it printed. Keep step +# output only, as "\t", without timestamps or colors. gh writes +# a log's escape sequences in caret notation ("^[[1m") when not on a +# terminal, so strip both that and the raw ESC form. +gh run view "$RID" --log >"$WORK/raw.log" 2>"$WORK/raw.err" || fail "fetch the run log" +perl -ne ' + s/(?:\e|\^\[)\[[0-9;?]*[A-Za-z]//g; + s/\r//g; + my ($job, $step, $msg) = split /\t/, $_, 3; + next unless defined $msg; + $msg =~ s/^\xEF\xBB\xBF//; + $msg =~ s/^\S+Z ?//; + $msg =~ s/\s+$//; + $skip = 0 if !defined $prev || $job ne $prev; + $prev = $job; + if ($msg =~ /^##\[group\]Run /) { $skip = 1; next } + if ($skip && $msg =~ /^##\[endgroup\]/) { $skip = 0; next } + print "$job\t$msg\n" unless $skip; +' "$WORK/raw.log" >"$WORK/output.log" + +logged=$(cut -f1 "$WORK/output.log" | sort -u | wc -l | tr -d ' ') +ran=$(awk -F'\t' '$2 != "skipped"' "$WORK/jobs.tsv" | wc -l | tr -d ' ') +check "log covers every job that ran ($ran)" test "$logged" -eq "$ran" + +check "validate parsed the tag as $VERSION" has_line "Validate Release${TAB}Version: $VERSION" "$WORK/output.log" +check "validate matched Cargo.toml version $BASE" has_line "Validate Release${TAB}Version verified: $BASE" "$WORK/output.log" +check "changelog starts at $PREV" has_line "Create Release${TAB}Generating changelog since $PREV" "$WORK/output.log" +check "changelog did not fall back to the last 20 commits" lacks_regex 'No previous tag found' "$WORK/output.log" +if [ "$PRERELEASE" = false ]; then + case $PREV in + *-*) fail "changelog base $PREV is a pre-release tag (a leftover exercise tag truncates the notes)" ;; + *) pass "changelog base $PREV is a stable tag" ;; + esac +fi +check "no artifact digest mismatch" lacks_regex 'digest.*mismatch|hash mismatch' "$WORK/output.log" + +# A step under continue-on-error fails without failing its job, and the API +# reports that step's conclusion as success too. Its "##[error]" line is the +# only trace, so any such line means a failure the job status hides. +if grep -F '##[error]' "$WORK/output.log" >"$WORK/errors.log"; then + fail "no step failed silently; failures hidden behind a successful job:" + sed 's/^/ /' "$WORK/errors.log" +else + pass "no step failed silently" +fi + +# -------------------------------------------------------------------------- +# Artifacts +# -------------------------------------------------------------------------- +gh api "repos/$REPO/actions/runs/$RID/artifacts" --jq '.artifacts[].name' >"$WORK/artifacts.txt" || exit 2 +binaries=$(grep -c "^$CRATE-" "$WORK/artifacts.txt") +check "one binary artifact per build ($binaries of $builds)" test "$binaries" -eq "$builds" +check "Pages artifact uploaded" grep -qx 'github-pages' "$WORK/artifacts.txt" + +# -------------------------------------------------------------------------- +# The GitHub Release +# -------------------------------------------------------------------------- +release_ok=false +if gh release view "$TAG" --json isPrerelease,isDraft,publishedAt,assets \ + --jq '"\(.isPrerelease)\t\(.isDraft)\t\(.publishedAt)", (.assets[] | .name)' \ + >"$WORK/release.txt" 2>"$WORK/release.err"; then + release_ok=true + IFS=$TAB read -r is_pre is_draft published <"$WORK/release.txt" + pass "release $TAG exists" + check "release is published, not a draft" test "$is_draft" = false + check "release pre-release flag is $PRERELEASE" test "$is_pre" = "$PRERELEASE" + while IFS=$TAB read -r job _; do + case $job in "Build ("*) ;; *) continue ;; esac + target=${job#Build (} + target=${target%)} + case $target in *-windows-*) ext=zip ;; *) ext=tar.gz ;; esac + check "asset $CRATE-$VERSION-$target.$ext" grep -qxF -- "$CRATE-$VERSION-$target.$ext" "$WORK/release.txt" + done <"$WORK/jobs.tsv" +else + fail "release $TAG exists ($(head -n 1 "$WORK/release.err"))" +fi + +latest=$(gh release list --limit 50 --json tagName,isLatest --jq '.[] | select(.isLatest) | .tagName') +latest_is_other_stable() { + [ -n "$latest" ] && [ "$latest" != "$TAG" ] && case $latest in *-*) false ;; *) true ;; esac +} +if [ "$PRERELEASE" = true ]; then + check "Latest is still a stable release (${latest:-none})" latest_is_other_stable +elif [ "$latest" = "$TAG" ]; then + pass "release is marked Latest" +elif [ "$release_ok" = true ] && [ -n "$latest" ] && + [[ $(gh release view "$latest" --json publishedAt -q .publishedAt) > $published ]]; then + skip "release is marked Latest (superseded by $latest)" +else + fail "release is marked Latest (Latest is ${latest:-none})" +fi + +# -------------------------------------------------------------------------- +# Pages +# -------------------------------------------------------------------------- +# Reachability proves nothing: the site already served the last release's +# docs. The deployment record says what this run deployed. A later deploy +# turns this one inactive, so look for success anywhere in its history. +dep=$(gh api "repos/$REPO/deployments?environment=github-pages&sha=$SHA&per_page=1" --jq '.[0].id // empty') +if [ -n "$dep" ]; then + check "Pages deployed from ${SHA:0:7}" \ + test "$(gh api "repos/$REPO/deployments/$dep/statuses" --jq '[.[].state] | index("success") != null')" = true +else + fail "Pages deployed from ${SHA:0:7} (no deployment record)" +fi + +# -------------------------------------------------------------------------- +# One published binary, for the host platform +# -------------------------------------------------------------------------- +case "$(uname -s)-$(uname -m)" in + Darwin-arm64) host=aarch64-apple-darwin ;; + Darwin-x86_64) host=x86_64-apple-darwin ;; + Linux-x86_64) host=x86_64-unknown-linux-gnu ;; + Linux-aarch64 | Linux-arm64) host=aarch64-unknown-linux-gnu ;; + *) host= ;; +esac +asset=$CRATE-$VERSION-$host.tar.gz +if [ "$release_ok" != true ]; then + skip "host binary (no release to download from)" +elif [ -z "$host" ] || ! grep -qxF -- "$asset" "$WORK/release.txt"; then + skip "host binary (no asset for $(uname -s)-$(uname -m))" +elif ! gh release download "$TAG" --pattern "$asset" --dir "$WORK/bin" >/dev/null 2>"$WORK/download.err"; then + fail "download $asset ($(head -n 1 "$WORK/download.err"))" +else + tar -tzf "$WORK/bin/$asset" >"$WORK/contents.txt" + # shellcheck disable=SC2016 # $1 and $2 are the inner shell's arguments + check "archive holds $CRATE, README.md and a LICENSE" \ + sh -c 'grep -qx "$1" "$2" && grep -qx README.md "$2" && grep -q "^LICENSE" "$2"' _ "$CRATE" "$WORK/contents.txt" + tar -xzf "$WORK/bin/$asset" -C "$WORK/bin" + # Config::validate rejects a zero OPERATION_TIMEOUT_SECS, so the binary + # logs its version and exits with EX_CONFIG (78) before any network I/O. + OPERATION_TIMEOUT_SECS=0 RUST_LOG=info "$WORK/bin/$CRATE" >"$WORK/bin.raw" 2>&1 + code=$? + perl -pe 's/\e\[[0-9;]*m//g' "$WORK/bin.raw" >"$WORK/bin.log" + check "$host binary exits 78 on an invalid config (got $code)" test "$code" -eq 78 + check "$host binary reports version $BASE" grep -qF "Starting Iggy Sample Application v$BASE" "$WORK/bin.log" +fi + +echo +echo "$passes passed, $failures failed, $skips skipped" +if [ "$PRERELEASE" = true ] && [ "$release_ok" = true ]; then + echo + echo "If $TAG only exercised the workflow, delete it before the next release:" + echo " gh release delete $TAG --cleanup-tag --yes" +fi +[ "$failures" -eq 0 ] From 92cfb38a953d60b0a7ab81696334627c3883986c Mon Sep 17 00:00:00 2001 From: mlevkov Date: Wed, 23 Sep 2026 15:32:07 -0700 Subject: [PATCH 02/26] docs(tech-debt): record the silent Codecov upload failure The Code Coverage job passes secrets.CODECOV_TOKEN, which does not exist, so Codecov rejects every upload ("Token required - not valid tokenless upload") and fail_ci_if_error: false keeps the job green: all 15 main push runs whose logs are retained, 2026-07-05 to 2026-09-23 (one cancelled after its upload had run), across codecov-action v4, v4.6.0 and v7.1.1. The README still says coverage is uploaded to Codecov. TD-2026-09-03 records the two ways out (a real token that fails loudly, or no upload) and binds the decision to the next coverage-job edit, major codecov-action bump, or release. --- docs/tech-debt/README.md | 1 + docs/tech-debt/TD-2026-09-03.md | 45 +++++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+) create mode 100644 docs/tech-debt/TD-2026-09-03.md diff --git a/docs/tech-debt/README.md b/docs/tech-debt/README.md index 177dda7..d0bc3cf 100644 --- a/docs/tech-debt/README.md +++ b/docs/tech-debt/README.md @@ -16,3 +16,4 @@ condition under which the record MUST be resolved (not "someday"). | [TD-2026-07-09](TD-2026-07-09.md) | Breaker per-state data as enum with payloads | Review session 02 (types MEDIUM) | Next behavioral change to the breaker state or admission gate | resolved (session 03) | | [TD-2026-09-01](TD-2026-09-01.md) | API listener serves cleartext HTTP/2 by feature unification | RUSTSEC-2026-0258 remediation (h2 exposure) | A tripwire failing (CI feature-graph check or either h2c test), next change to the serving path, next `h2` advisory, or next minor bump of `axum`/`hyper-util`/`metrics-exporter-prometheus` | open | | [TD-2026-09-02](TD-2026-09-02.md) | Validate dependabot.yml in CI, and put the Dockerfile on an updater | RustSec 2026-09 remediation review, rounds 1-2 | Next edit to `.github/dependabot.yml` after it lands, or the next `rust-version` change | open | +| [TD-2026-09-03](TD-2026-09-03.md) | Codecov upload has never authenticated | Dependabot first-wave triage (codecov-action bump, PR #37) | Next edit to ci.yml's coverage job, the next major `codecov-action` bump, or the next release | open | diff --git a/docs/tech-debt/TD-2026-09-03.md b/docs/tech-debt/TD-2026-09-03.md new file mode 100644 index 0000000..e9f11ae --- /dev/null +++ b/docs/tech-debt/TD-2026-09-03.md @@ -0,0 +1,45 @@ +# TD-2026-09-03: Codecov upload has never authenticated + +**Source:** Dependabot first-wave triage (2026-09-23), checking what PR #37's +`codecov/codecov-action` bump actually did in CI. +**Status:** open + +## Problem + +ci.yml's Code Coverage job uploads `lcov.info` with `codecov/codecov-action` +and passes `CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}`, but the repository +has no such secret; as of 2026-09-23 it has no Actions or Dependabot secrets +at all. Codecov rejects the tokenless upload (`Upload queued for processing +failed: {"message":"Token required - not valid tokenless upload"}`), and the +step passes anyway because it sets `fail_ci_if_error: false`, so the job +stays green. Every main push run whose log is still retained was rejected +this way: 14 runs from 2026-07-05 to 2026-09-23, on codecov-action `v4`, +v4.6.0 and v7.1.1. Older logs have expired. + +The README's "Automated Checks" list states "**Coverage**: Uploaded to +Codecov", which is not true. + +## Mitigations in place + +- Coverage is still measured: the Code Coverage job runs the suite under + cargo-llvm-cov, and pr.yml's Coverage Diff job prints the total on every + pull request and warns below 60%. +- Nothing reads the Codecov side, so no decision rests on the missing data. + +## Options + +1. **Make the upload real.** Add `CODECOV_TOKEN` as a repository secret, and + as a Dependabot secret too if Dependabot PRs should upload (their runs get + no Actions secrets). Then let a rejected upload fail the job wherever a + token is present, e.g. `fail_ci_if_error: ${{ secrets.CODECOV_TOKEN != '' + }}`, so this cannot recur silently. +2. **Drop the upload.** Remove the Codecov step and the README line, keeping + the local coverage run. + +Either way the README line has to end up true. + +## Binding trigger + +Whichever comes first: the next edit to ci.yml's coverage job, the next +major bump of `codecov/codecov-action`, or the next release (the README +states the upload as fact). From d073caadce9c77451d365017313ac5605222c309 Mon Sep 17 00:00:00 2001 From: mlevkov Date: Wed, 23 Sep 2026 15:32:23 -0700 Subject: [PATCH 03/26] docs(tech-debt): record the crates.io job that cannot succeed release.yml's publish step has failed on all four stable releases, v0.2.0 through v0.4.1: CRATES_IO_TOKEN does not exist, so cargo stops at argument parsing, and with a token it would stop at publish = false. continue-on-error hides it, and the Actions API reports the failed step as a success too, so only its ##[error] log line shows it. TD-2026-09-04 records deleting the job or publishing deliberately, and binds the choice to the next stable release, which verify-release.sh now fails until it is made. --- docs/tech-debt/README.md | 1 + docs/tech-debt/TD-2026-09-04.md | 48 +++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+) create mode 100644 docs/tech-debt/TD-2026-09-04.md diff --git a/docs/tech-debt/README.md b/docs/tech-debt/README.md index d0bc3cf..e8d8036 100644 --- a/docs/tech-debt/README.md +++ b/docs/tech-debt/README.md @@ -17,3 +17,4 @@ condition under which the record MUST be resolved (not "someday"). | [TD-2026-09-01](TD-2026-09-01.md) | API listener serves cleartext HTTP/2 by feature unification | RUSTSEC-2026-0258 remediation (h2 exposure) | A tripwire failing (CI feature-graph check or either h2c test), next change to the serving path, next `h2` advisory, or next minor bump of `axum`/`hyper-util`/`metrics-exporter-prometheus` | open | | [TD-2026-09-02](TD-2026-09-02.md) | Validate dependabot.yml in CI, and put the Dockerfile on an updater | RustSec 2026-09 remediation review, rounds 1-2 | Next edit to `.github/dependabot.yml` after it lands, or the next `rust-version` change | open | | [TD-2026-09-03](TD-2026-09-03.md) | Codecov upload has never authenticated | Dependabot first-wave triage (codecov-action bump, PR #37) | Next edit to ci.yml's coverage job, the next major `codecov-action` bump, or the next release | open | +| [TD-2026-09-04](TD-2026-09-04.md) | The crates.io publish job cannot succeed | Dependabot first-wave triage (release.yml exercise, PR #37) | The next stable release tag, or the next edit to the publish job or to `publish` in Cargo.toml | open | diff --git a/docs/tech-debt/TD-2026-09-04.md b/docs/tech-debt/TD-2026-09-04.md new file mode 100644 index 0000000..461442f --- /dev/null +++ b/docs/tech-debt/TD-2026-09-04.md @@ -0,0 +1,48 @@ +# TD-2026-09-04: The crates.io publish job cannot succeed + +**Source:** Dependabot first-wave triage (2026-09-23), exercising release.yml +for PR #37's action bumps. +**Status:** open + +## Problem + +release.yml's publish job runs `cargo publish --token ${{ +secrets.CRATES_IO_TOKEN }}` with `continue-on-error: true` after every stable +release. It cannot succeed, for two independent reasons: + +- The repository has no `CRATES_IO_TOKEN` secret, so the command fails at + argument parsing: `error: a value is required for '--token ' but + none was supplied`. That is the failure in the publish step of all four + stable releases, v0.2.0 through v0.4.1. +- With a token it would still fail: Cargo.toml sets `publish = false`, and + cargo refuses before it uses the token (`` `iggy_sample` cannot be + published ``, reproduced on 2026-09-23 with `--dry-run` and a dummy + token). TD-2026-07-07 relies on that refusal as a mitigation. + +`continue-on-error` hides the failure end to end. The job concludes success, +and the Actions API reports the failed step's conclusion as `success` as +well (checked on the v0.4.1 run, 35888419705), so the only trace is the +step's log line `##[error]Process completed with exit code 1.`. iggy_sample +is not on crates.io (the API returned 404 on 2026-09-23). + +## Mitigations in place + +- Nothing can be published by accident: `publish = false` holds whatever + the token. +- `scripts/verify-release.sh` fails a release whose log has an `##[error]` + line under a successful job, so the next stable release reports this step + instead of passing it. + +## Options + +1. **Delete the job.** `publish = false` already says the crate is not meant + for crates.io, so the job only adds a red step to every release and a + secret reference that invites someone to configure a live token. +2. **Publish deliberately.** Set `publish = true`, add the token, and drop + `continue-on-error` so a failed publish fails the release run. + +## Binding trigger + +Whichever comes first: the next stable release tag (the verifier fails on it +until this is resolved), or the next edit to the publish job or to `publish` +in Cargo.toml. From 9eb245f6111cd513dc41937b97fe1c0217cb5942 Mon Sep 17 00:00:00 2001 From: mlevkov Date: Wed, 23 Sep 2026 15:32:59 -0700 Subject: [PATCH 04/26] docs(tech-debt): record rust-cache's floating pin All 17 rust-cache pins say "# v2" but point at upstream master's head, five commits past v2.9.2, because TD-2026-07-07 pinned an untagged commit and dependabot-core follows the branch of an untagged SHA. So each github-actions PR can carry unreleased upstream code under a release comment. TD-2026-09-05 proposes re-pinning to v2.9.2's commit (so Dependabot tracks releases) and takes over the addendum's open item, moving audit-check to a release SHA once upstream cuts one. TD-2026-07-07's addendum now points to it. --- docs/tech-debt/README.md | 1 + docs/tech-debt/TD-2026-07-07.md | 2 ++ docs/tech-debt/TD-2026-09-05.md | 55 +++++++++++++++++++++++++++++++++ 3 files changed, 58 insertions(+) create mode 100644 docs/tech-debt/TD-2026-09-05.md diff --git a/docs/tech-debt/README.md b/docs/tech-debt/README.md index e8d8036..72c41ac 100644 --- a/docs/tech-debt/README.md +++ b/docs/tech-debt/README.md @@ -18,3 +18,4 @@ condition under which the record MUST be resolved (not "someday"). | [TD-2026-09-02](TD-2026-09-02.md) | Validate dependabot.yml in CI, and put the Dockerfile on an updater | RustSec 2026-09 remediation review, rounds 1-2 | Next edit to `.github/dependabot.yml` after it lands, or the next `rust-version` change | open | | [TD-2026-09-03](TD-2026-09-03.md) | Codecov upload has never authenticated | Dependabot first-wave triage (codecov-action bump, PR #37) | Next edit to ci.yml's coverage job, the next major `codecov-action` bump, or the next release | open | | [TD-2026-09-04](TD-2026-09-04.md) | The crates.io publish job cannot succeed | Dependabot first-wave triage (release.yml exercise, PR #37) | The next stable release tag, or the next edit to the publish job or to `publish` in Cargo.toml | open | +| [TD-2026-09-05](TD-2026-09-05.md) | rust-cache is pinned to a branch head under a release comment | Dependabot first-wave triage (SHA pin check, PR #37); carries TD-2026-07-07's addendum | The next github-actions Dependabot PR that moves the rust-cache pin, or the first `rustsec/audit-check` release after v2.0.0 | open | diff --git a/docs/tech-debt/TD-2026-07-07.md b/docs/tech-debt/TD-2026-07-07.md index 9ea4498..49c5cce 100644 --- a/docs/tech-debt/TD-2026-07-07.md +++ b/docs/tech-debt/TD-2026-07-07.md @@ -61,3 +61,5 @@ whose SHA is not a tagged commit either. For an untagged SHA, dependabot-core proposes the head of the branch containing it rather than a release, so both will float with upstream inside the weekly github-actions PR. Review those diffs as code, and move audit-check back to a release SHA once one exists. +Both returns to release SHAs are now tracked as TD-2026-09-05, since +rust-cache has one available today (v2.9.2). diff --git a/docs/tech-debt/TD-2026-09-05.md b/docs/tech-debt/TD-2026-09-05.md new file mode 100644 index 0000000..82c675e --- /dev/null +++ b/docs/tech-debt/TD-2026-09-05.md @@ -0,0 +1,55 @@ +# TD-2026-09-05: rust-cache is pinned to a branch head under a release comment + +**Source:** Dependabot first-wave triage (2026-09-23), checking PR #37's +third-party SHA pins against their upstream tags. Carries the open part of +TD-2026-07-07's 2026-09-23 addendum. +**Status:** open + +## Problem + +TD-2026-07-07's convention is `uses: owner/action@ # vX`, with the SHA a +release commit. `Swatinem/rust-cache` breaks it in all 17 of its uses (ci.yml +5, extended-tests.yml 6, pr.yml 3, release.yml 3): since PR #37 the pin is +`f0d9c388… # v2`, the head of upstream `master`, five commits past v2.9.2 +(`6323deb`, where the `v2` tag points). Those commits are one fix (#377, +cleanup timestamp pruning) and two dependency bumps with their merges. The +comment says `v2`, but the SHA is no release. TD-2026-07-07's original pin, +`e18b4977…`, was not a release either: an untagged changelog commit made +three minutes before v2.9.1 was published. + +For an untagged SHA, dependabot-core proposes the head of the branch that +contains it, which is how PR #37 arrived at `master`. So every github-actions +Dependabot PR can move rust-cache to whatever upstream merged last, under a +comment that names a release, and reviewing that means reading upstream +commits rather than release notes. + +Two other pins track a branch deliberately, with comments that say so: +`dtolnay/rust-toolchain@… # master` (TD-2026-07-07's normalization) and +`rustsec/audit-check@858dc40f… # main`, three commits past v2.0.0 for its +Node 24 change. Upstream has published no audit-check release since v2.0.0 +(2024-09-23); TD-2026-07-07's addendum left its return to a release SHA +open, and this record carries it. + +## Mitigations in place + +- The pins are still immutable SHAs in the upstream repositories. For PR #37 + each was compared with its tag or branch (`gh api + repos///compare/...`): softprops/action-gh-release, + codecov/codecov-action and taiki-e/install-action are identical to their + release tags, and dtolnay/rust-toolchain to `master`. +- rust-cache runs in most CI jobs, so a head that breaks it fails visibly. + That says nothing about a head that misbehaves quietly. + +## Proposed fix + +- Re-pin rust-cache to v2.9.2's commit with an accurate comment, + `Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2`, in + all 17 places. With a tagged SHA, Dependabot proposes releases from then + on. +- Move audit-check to a release SHA the same way once upstream tags a + release that includes its Node 24 change. + +## Binding trigger + +Whichever comes first: the next github-actions Dependabot PR that moves the +rust-cache pin, or the first `rustsec/audit-check` release after v2.0.0. From 771f1d97486193dc878999704b9b1e20d49ae9be Mon Sep 17 00:00:00 2001 From: mlevkov Date: Wed, 23 Sep 2026 15:33:24 -0700 Subject: [PATCH 05/26] docs(tech-debt): record main's non-strict required checks The main ruleset's required checks are not strict, so a pull request stays mergeable on a green run computed against an older main. After the 151-crate group merged, the sha1 PR still showed MERGEABLE on the pre-group lockfile, and between scheduled runs Dependabot rebases only on conflict, so only a manual rebase kept an untested tree off main. TD-2026-09-06 records that manual procedure as the stopgap, proposes the "up to date before merging" setting, and binds it to the next Dependabot wave. --- docs/tech-debt/README.md | 1 + docs/tech-debt/TD-2026-09-06.md | 41 +++++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+) create mode 100644 docs/tech-debt/TD-2026-09-06.md diff --git a/docs/tech-debt/README.md b/docs/tech-debt/README.md index 72c41ac..d04e862 100644 --- a/docs/tech-debt/README.md +++ b/docs/tech-debt/README.md @@ -19,3 +19,4 @@ condition under which the record MUST be resolved (not "someday"). | [TD-2026-09-03](TD-2026-09-03.md) | Codecov upload has never authenticated | Dependabot first-wave triage (codecov-action bump, PR #37) | Next edit to ci.yml's coverage job, the next major `codecov-action` bump, or the next release | open | | [TD-2026-09-04](TD-2026-09-04.md) | The crates.io publish job cannot succeed | Dependabot first-wave triage (release.yml exercise, PR #37) | The next stable release tag, or the next edit to the publish job or to `publish` in Cargo.toml | open | | [TD-2026-09-05](TD-2026-09-05.md) | rust-cache is pinned to a branch head under a release comment | Dependabot first-wave triage (SHA pin check, PR #37); carries TD-2026-07-07's addendum | The next github-actions Dependabot PR that moves the rust-cache pin, or the first `rustsec/audit-check` release after v2.0.0 | open | +| [TD-2026-09-06](TD-2026-09-06.md) | main accepts a pull request whose checks ran against an older main | Dependabot first-wave triage (merging PRs #37-#42) | Before merging the next Dependabot wave (next scheduled run 2026-09-28) | open | diff --git a/docs/tech-debt/TD-2026-09-06.md b/docs/tech-debt/TD-2026-09-06.md new file mode 100644 index 0000000..5e8077c --- /dev/null +++ b/docs/tech-debt/TD-2026-09-06.md @@ -0,0 +1,41 @@ +# TD-2026-09-06: main accepts a pull request whose checks ran against an older main + +**Source:** Dependabot first-wave triage (2026-09-23), merging PRs #37-#42. +**Status:** open + +## Problem + +The `main` ruleset requires CI Success, Clippy, Integration Tests and Rustfmt +with `strict_required_status_checks_policy: false`, so a pull request whose +checks passed against an older main stays mergeable after main moves, and +GitHub's `MERGEABLE` means only that the diff still applies. Lockfile pull +requests are where that bites. After #38 (151 crates) merged, #39 (sha1) +still showed `MERGEABLE` with green checks computed on the pre-#38 lockfile, +and Dependabot never rebased it on its own: it rebases a pull request by +itself only when it conflicts, which #39 never did. Merging it as it stood +would have put an untested tree on main, with main's own CI as its first +test, and ci.yml's concurrency group cancels that run when the next merge +lands, as it did for #40's merge (`5a9054e`). + +## Mitigations in place + +The 2026-09-23 wave was merged under the strict rule by hand: `@dependabot +rebase` on each pull request that did not conflict, a merge only once the +head contained current main and every check on that head had passed, and +`gh pr merge --match-head-commit` so a later push could not slip in. That is +a procedure, not a gate. + +## Proposed fix + +Turn on "Require branches to be up to date before merging" in the main +ruleset (`strict_required_status_checks_policy: true`). A pull request behind +main then has to be updated before it merges; for a Dependabot pull request +that is `@dependabot rebase` and a fresh CI run. A merge queue would automate +that loop; check whether GitHub offers one for this user-owned repository +before choosing it. Editing the ruleset is an admin action. + +## Binding trigger + +Before merging the next Dependabot wave, that is, the next time two or more +open pull requests change Cargo.lock at once. Dependabot's next scheduled run +is Monday 2026-09-28 at 09:00 UTC. From 6d7c45c2f3d38e02cf443cf089cab5df5aaddeec Mon Sep 17 00:00:00 2001 From: mlevkov Date: Wed, 23 Sep 2026 15:33:50 -0700 Subject: [PATCH 06/26] docs(tech-debt): record why Dependabot proposes no iggy 0.11 TD-2026-07-02 said to check the Dependabot run log if no iggy PR appeared. None did, and the log's ungrouped pass reports "Requirements to unlock update_not_possible" and closes with "No update possible for iggy 0.10.0". The cause reproduces locally: resolver 3 prefers versions that fit rust-version 1.93.0, so with a loosened requirement cargo keeps iggy 0.10.0, and under that requirement only CARGO_RESOLVER_INCOMPATIBLE_RUST_VERSIONS=allow moves it to 0.11.0 (an exact "0.11.0" requirement resolves too). The record now says to upgrade by hand with the MSRV raise, and notes that Dependabot likewise skips any update, security fixes included, that needs a newer Rust than rust-version. --- docs/tech-debt/README.md | 2 +- docs/tech-debt/TD-2026-07-02.md | 26 ++++++++++++++++++++++++++ 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/docs/tech-debt/README.md b/docs/tech-debt/README.md index d04e862..ab0acc7 100644 --- a/docs/tech-debt/README.md +++ b/docs/tech-debt/README.md @@ -6,7 +6,7 @@ condition under which the record MUST be resolved (not "someday"). | ID | Title | Source | Trigger | Status | |----|-------|--------|---------|--------| | [TD-2026-07-01](TD-2026-07-01.md) | `with_reconnect` composition test matrix | Review session 01 (tests #2) | Any behavioral change to `with_reconnect`/`retry_once` | resolved (session 02) | -| [TD-2026-07-02](TD-2026-07-02.md) | DiagnosticEvents-driven connection state | Review session 01 (architect #2) | Next iggy SDK minor bump | open (trigger FIRED: iggy 0.11.0 released 2026-09-18) | +| [TD-2026-07-02](TD-2026-07-02.md) | DiagnosticEvents-driven connection state | Review session 01 (architect #2) | Next iggy SDK minor bump | open (trigger FIRED: iggy 0.11.0 released 2026-09-18; Dependabot cannot propose it below Rust 1.95, so upgrade by hand) | | [TD-2026-07-03](TD-2026-07-03.md) | Half-open probe limiting | Review session 01 (architect #9, types F5) | First production incident involving breaker recovery, or breaker config exposure | resolved (session 02) | | [TD-2026-07-04](TD-2026-07-04.md) | X-Request-Timeout enforcement | Review session 01 (silentfail M4) | Before advertising the header in any client-facing docs beyond CLAUDE.md | resolved (session 02) | | [TD-2026-07-05](TD-2026-07-05.md) | Metrics exporter smoke test | Review session 01 (tests #7) | Next metrics-exporter-prometheus major/minor bump | resolved (session 02) | diff --git a/docs/tech-debt/TD-2026-07-02.md b/docs/tech-debt/TD-2026-07-02.md index e6f4ef7..9082cfa 100644 --- a/docs/tech-debt/TD-2026-07-02.md +++ b/docs/tech-debt/TD-2026-07-02.md @@ -57,3 +57,29 @@ arrived. With the config fixed, 0.11 should open as its own PR (Dependabot counts a 0.x minor bump as major); whether Dependabot also holds it back for the MSRV is untested, so if no PR appears, check the Dependabot run log. Treat that PR as this record's work rather than a routine bump. + +## Trigger check (2026-09-23, after Dependabot's first run) + +No PR appeared: Dependabot's first cargo run opened #38-#42 and nothing for +`iggy`. Its log (Actions run 35888373353) shows both passes over the crate. +The grouped `rust-minor` pass saw `Latest version is 0.11.0` and left it +alone, as the group takes only minor and patch updates. The ungrouped pass, +the one that would have opened the PR, ended with `Requirements to unlock +update_not_possible`, which is its verdict when no newer version resolves; +the log does not say why at INFO level. + +MSRV-aware resolution explains it, and reproduces locally (cargo 1.98.1). +Edition 2024 defaults to resolver 3, which prefers dependency versions whose +own `rust-version` fits this crate's 1.93.0. With the requirement loosened +to `iggy = ">=0.10.0"`, `cargo update -p iggy` locks 0 packages "to latest +Rust 1.93.0 compatible versions" and keeps 0.10.0; the same command with +`CARGO_RESOLVER_INCOMPATIBLE_RUST_VERSIONS=allow` moves `iggy`, +`iggy_common` and `iggy_binary_protocol` to 0.11.0. An exact +`iggy = "0.11.0"` also resolves, so the graph itself is satisfiable. + +So Dependabot will not propose 0.11 while `rust-version` stays below 1.95. +Do this record's upgrade by hand, raising `rust-version` in the same change +(the six places above). More generally, Dependabot silently skips any +update, security fixes included, whose new version needs a newer Rust than +`rust-version`; cargo-deny and cargo-audit read the lockfile, so they still +flag a vulnerable version that stays behind. From dd1cc2292c67360956e76d2a90b13d7cab6eb7ae Mon Sep 17 00:00:00 2001 From: mlevkov Date: Wed, 23 Sep 2026 16:13:56 -0700 Subject: [PATCH 07/26] ci(verify-release): run the binary isolated and time-bounded The smoke test ran a binary it had just downloaded from the release under review with the caller's full environment (GH_TOKEN and any cloud credentials) and from the repository root, where the app's dotenvy loads a .env with API keys. It had no timeout, and it relied on Config::validate rejecting a zero OPERATION_TIMEOUT_SECS, which only v0.4.0 and later do: run against v0.3.0 or v0.2.0, the binary opened a listener on 0.0.0.0:9090 and tried to connect before exiting 69. The binary now runs under env -i from an empty directory, killed after 30 seconds (reported as its own FAIL), and not at all for versions before 0.4.0. Checked with a hung stand-in (exit 142 after the alarm), /usr/bin/env (only the two variables), /bin/pwd and a missing binary (127). The header no longer lists curl, which nothing called. --- scripts/verify-release.sh | 33 +++++++++++++++++++++++++++------ 1 file changed, 27 insertions(+), 6 deletions(-) diff --git a/scripts/verify-release.sh b/scripts/verify-release.sh index 3166373..844be97 100755 --- a/scripts/verify-release.sh +++ b/scripts/verify-release.sh @@ -16,9 +16,11 @@ # verify the run, then delete the release and its tag. A leftover pre-release # tag becomes the changelog base of the next release: see CONTRIBUTING.md. # -# Requires gh (authenticated), git, perl, tar and curl. Exits 0 when every -# check passes (skips allowed), 1 when any fails, 2 on a usage or setup -# error. The work directory is kept when a check fails. +# Requires gh (authenticated), git, perl and tar. Exits 0 when every check +# passes (skips allowed), 1 when any fails, 2 on a usage or setup error. The +# work directory is kept when a check fails. Binaries before v0.4.0 are not +# executed: they accept a zero OPERATION_TIMEOUT_SECS and go on to open +# network listeners. set -uo pipefail @@ -44,6 +46,7 @@ case $VERSION in *) PRERELEASE=false ;; esac TAB=$(printf '\t') +PERL=$(command -v perl) || { echo "required tool not found: perl" >&2; exit 2; } ROOT=$(git rev-parse --show-toplevel) || exit 2 cd "$ROOT" || exit 2 @@ -79,6 +82,13 @@ check() { # SIGPIPEs the producer, which turns every negated check into a vacuous pass. has_line() { grep -qxF -- "$1" "$2"; } lacks_regex() { ! grep -qiE -- "$1" "$2"; } +# version_lt A B: A < B, comparing X.Y.Z numerically. +# shellcheck disable=SC2016 # perl's variables, not the shell's +version_lt() { + "$PERL" -e '@a = split /\./, $ARGV[0]; @b = split /\./, $ARGV[1]; + for (0 .. 2) { exit 0 if ($a[$_] // 0) < ($b[$_] // 0); exit 1 if ($a[$_] // 0) > ($b[$_] // 0) } + exit 1' "$1" "$2" +} # -------------------------------------------------------------------------- # The run @@ -244,6 +254,8 @@ if [ "$release_ok" != true ]; then skip "host binary (no release to download from)" elif [ -z "$host" ] || ! grep -qxF -- "$asset" "$WORK/release.txt"; then skip "host binary (no asset for $(uname -s)-$(uname -m))" +elif version_lt "$BASE" 0.4.0; then + skip "host binary ($BASE predates the zero-timeout check it relies on)" elif ! gh release download "$TAG" --pattern "$asset" --dir "$WORK/bin" >/dev/null 2>"$WORK/download.err"; then fail "download $asset ($(head -n 1 "$WORK/download.err"))" else @@ -254,10 +266,19 @@ else tar -xzf "$WORK/bin/$asset" -C "$WORK/bin" # Config::validate rejects a zero OPERATION_TIMEOUT_SECS, so the binary # logs its version and exits with EX_CONFIG (78) before any network I/O. - OPERATION_TIMEOUT_SECS=0 RUST_LOG=info "$WORK/bin/$CRATE" >"$WORK/bin.raw" 2>&1 + # Run it from an empty directory (the app loads a .env from its working + # directory) with an empty environment (a verifier should not hand a + # freshly downloaded binary the caller's tokens), and give it 30 seconds. + mkdir -p "$WORK/run" + (cd "$WORK/run" && env -i OPERATION_TIMEOUT_SECS=0 RUST_LOG=info \ + "$PERL" -e 'alarm 30; exec @ARGV or exit 127' "$WORK/bin/$CRATE") >"$WORK/bin.raw" 2>&1 code=$? - perl -pe 's/\e\[[0-9;]*m//g' "$WORK/bin.raw" >"$WORK/bin.log" - check "$host binary exits 78 on an invalid config (got $code)" test "$code" -eq 78 + "$PERL" -pe 's/\e\[[0-9;]*m//g' "$WORK/bin.raw" >"$WORK/bin.log" + if [ "$code" -eq 142 ]; then + fail "$host binary exits on an invalid config (killed after 30s; does Config::validate still reject a zero timeout?)" + else + check "$host binary exits 78 on an invalid config (got $code)" test "$code" -eq 78 + fi check "$host binary reports version $BASE" grep -qF "Starting Iggy Sample Application v$BASE" "$WORK/bin.log" fi From 1886bc9cd2ed79d1fdcb591a2c68b8789d24475c Mon Sep 17 00:00:00 2001 From: mlevkov Date: Wed, 23 Sep 2026 16:16:53 -0700 Subject: [PATCH 08/26] ci(verify-release): tie the run to its tag and fail closed on fetches The run was picked by tag name and never compared with the commit the tag names, so a re-pushed tag whose new run is missing would verify the old commit's run and pass. The tag's commit on origin must now equal the run's head (skipped only once the tag is gone, as for a deleted exercise tag), and the header shows the attempt and says when a tag has several runs (v0.2.0 has two). A failed fetch no longer leaves checks to pass on nothing. A missing or expired log FAILs with gh's reason and skips the step-output checks instead of letting the negated ones pass on an empty file; a failed job listing or artifact listing FAILs rather than exiting 2 mid-run; a failing `gh run list` is reported as such; and the required tools are checked up front. The run's own commit is now the oracle: the crate name, the build targets and whether a publish job exists come from Cargo.toml and release.yml at that commit, not from the working tree, so a missing build job or publish job FAILs instead of shrinking the check list. The default changelog base ignores tags that exist only in this clone, as release.yml's checkout does, and names them. From a clone with a local-only v0.3.0-stale.1, verifying v0.4.0 used to produce two false FAILs; it now computes v0.3.0. Mutations checked: the older v0.2.0 run (tag mismatch FAILs), an expired log (FAIL with the 410), a broken job listing, and a broken run listing (exit 2 with gh's error). --- scripts/verify-release.sh | 219 +++++++++++++++++++++++++------------- 1 file changed, 143 insertions(+), 76 deletions(-) diff --git a/scripts/verify-release.sh b/scripts/verify-release.sh index 844be97..bd22f2c 100755 --- a/scripts/verify-release.sh +++ b/scripts/verify-release.sh @@ -38,6 +38,11 @@ case ${1:-} in esac [ $# -le 2 ] || usage +for tool in gh git perl tar; do + command -v "$tool" >/dev/null 2>&1 || { echo "required tool not found: $tool" >&2; exit 2; } +done +PERL=$(command -v perl) + TAG=$1 VERSION=${TAG#v} BASE=${VERSION%%-*} @@ -46,13 +51,10 @@ case $VERSION in *) PRERELEASE=false ;; esac TAB=$(printf '\t') -PERL=$(command -v perl) || { echo "required tool not found: perl" >&2; exit 2; } ROOT=$(git rev-parse --show-toplevel) || exit 2 cd "$ROOT" || exit 2 REPO=$(gh repo view --json nameWithOwner -q .nameWithOwner) || exit 2 -CRATE=$(awk -F'"' '/^name = / { print $2; exit }' Cargo.toml) -[ -n "$CRATE" ] || { echo "cannot read the package name from Cargo.toml" >&2; exit 2; } tmp=${TMPDIR:-/tmp} WORK=$(mktemp -d "${tmp%/}/verify-release.XXXXXX") || exit 2 @@ -71,17 +73,25 @@ trap cleanup EXIT pass() { printf 'PASS %s\n' "$1"; passes=$((passes + 1)); } fail() { printf 'FAIL %s\n' "$1"; failures=$((failures + 1)); } skip() { printf 'SKIP %s\n' "$1"; skips=$((skips + 1)); } +note() { printf 'NOTE %s\n' "$1"; } # check [args...]: PASS when the command succeeds. check() { local desc=$1 shift if "$@"; then pass "$desc"; else fail "$desc"; fi } -# Exact whole-line match against a file. Checks read files, never a pipe: -# under pipefail, `producer | grep -q` fails when grep exits early and -# SIGPIPEs the producer, which turns every negated check into a vacuous pass. +# Checks read files, never a pipe: under pipefail, `producer | grep -q` fails +# when grep exits early and SIGPIPEs the producer, which turns a negated +# check into a vacuous pass. For the same reason a negated check demands a +# non-empty file (grep exits 2 on an unreadable one), so a failed fetch +# cannot pass as "nothing found". has_line() { grep -qxF -- "$1" "$2"; } -lacks_regex() { ! grep -qiE -- "$1" "$2"; } +lacks_regex() { + [ -s "$2" ] || return 1 + grep -qiE -- "$1" "$2" + [ $? -eq 1 ] +} +first_line() { head -n 1 "$1" 2>/dev/null; } # version_lt A B: A < B, comparing X.Y.Z numerically. # shellcheck disable=SC2016 # perl's variables, not the shell's version_lt() { @@ -91,41 +101,91 @@ version_lt() { } # -------------------------------------------------------------------------- -# The run +# The run, and the commit it built # -------------------------------------------------------------------------- -run=$(gh run list --workflow release.yml --branch "$TAG" --limit 1 \ - --json databaseId,headSha,status,url \ - -q '.[] | "\(.databaseId) \(.headSha) \(.status) \(.url)"') -[ -n "$run" ] || { echo "no release.yml run found for $TAG" >&2; exit 2; } -read -r RID SHA STATUS URL <<<"$run" +runs=$(gh run list --workflow release.yml --branch "$TAG" --limit 20 \ + --json databaseId,headSha,status,attempt,url \ + -q '.[] | "\(.databaseId) \(.headSha) \(.status) \(.attempt) \(.url)"') || + { echo "listing release.yml runs for $TAG failed" >&2; exit 2; } +[ -n "$runs" ] || { echo "no release.yml run found for $TAG" >&2; exit 2; } +read -r RID SHA STATUS ATTEMPT URL <<<"${runs%%$'\n'*}" +nruns=$(printf '%s\n' "$runs" | wc -l | tr -d ' ') [ "$STATUS" = completed ] || { echo "run $RID is $STATUS; verify it once it completes" >&2; exit 2; } -git cat-file -e "$SHA^{commit}" 2>/dev/null || git fetch --quiet origin -git fetch --quiet --tags origin -PREV=${2:-$(git describe --tags --abbrev=0 "$SHA^" 2>/dev/null)} +git fetch --quiet --tags origin || { echo "fetching from origin failed" >&2; exit 2; } +git cat-file -e "$SHA^{commit}" 2>/dev/null || + { echo "commit ${SHA:0:7} is not in this clone after fetching" >&2; exit 2; } + +# The workflow and manifest the run built are the oracle, not the working tree. +git show "$SHA:.github/workflows/release.yml" >"$WORK/release.yml" 2>/dev/null || + { echo "no .github/workflows/release.yml at ${SHA:0:7}" >&2; exit 2; } +git show "$SHA:Cargo.toml" >"$WORK/Cargo.toml" || exit 2 +CRATE=$(awk -F'"' '/^name = / { print $2; exit }' "$WORK/Cargo.toml") +[ -n "$CRATE" ] || { echo "cannot read the package name from Cargo.toml at ${SHA:0:7}" >&2; exit 2; } +awk '/^ *- target: / { print $3 }' "$WORK/release.yml" | LC_ALL=C sort >"$WORK/targets.txt" +[ -s "$WORK/targets.txt" ] || { echo "no build targets in release.yml at ${SHA:0:7}" >&2; exit 2; } +if grep -q '^ *name: Publish to crates.io$' "$WORK/release.yml"; then HAS_PUBLISH=true; else HAS_PUBLISH=false; fi + +# release.yml's checkout sees only origin's tags, so a tag that exists only in +# this clone (an exercise tag deleted from another clone or the web UI) must +# not become the changelog base here. +git ls-remote --tags --refs origin >"$WORK/remote-tags.raw" || + { echo "listing origin's tags failed" >&2; exit 2; } +sed 's|.*refs/tags/||' "$WORK/remote-tags.raw" | LC_ALL=C sort >"$WORK/remote-tags.txt" +git tag | LC_ALL=C sort >"$WORK/local-tags.txt" +LC_ALL=C comm -23 "$WORK/local-tags.txt" "$WORK/remote-tags.txt" >"$WORK/local-only.txt" +if [ -n "${2:-}" ]; then + PREV=$2 +else + set -- + while IFS= read -r t; do set -- "$@" "--exclude=$t"; done <"$WORK/local-only.txt" + PREV=$(git describe --tags --abbrev=0 "$@" "$SHA^" 2>/dev/null) +fi [ -n "$PREV" ] || { echo "cannot determine the previous tag; pass it as the second argument" >&2; exit 2; } echo "repo $REPO, tag $TAG ($([ "$PRERELEASE" = true ] && echo pre-release || echo stable)), commit ${SHA:0:7}" -echo "run $RID: $URL" +echo "run $RID, attempt $ATTEMPT: $URL" echo "changelog base: $PREV" +[ "$nruns" -gt 1 ] && note "$nruns release.yml runs exist for $TAG; verifying the newest" +[ -s "$WORK/local-only.txt" ] && note "ignored tags that exist only in this clone: $(tr '\n' ' ' <"$WORK/local-only.txt")" echo +git ls-remote --tags origin "refs/tags/$TAG" "refs/tags/$TAG^{}" >"$WORK/tag.raw" 2>/dev/null +tag_commit=$(awk -v ref="refs/tags/$TAG" ' + $2 == ref "^{}" { peeled = $1 } $2 == ref { plain = $1 } + END { print (peeled != "" ? peeled : plain) }' "$WORK/tag.raw") +if [ -z "$tag_commit" ]; then + skip "tag $TAG names the commit the run built (the tag is no longer on origin)" +else + check "tag $TAG names the commit the run built (${SHA:0:7})" test "$tag_commit" = "$SHA" +fi + # -------------------------------------------------------------------------- # Jobs # -------------------------------------------------------------------------- -gh run view "$RID" --json jobs --jq '.jobs[] | "\(.name)\t\(.conclusion)"' >"$WORK/jobs.tsv" || exit 2 -builds=$(grep -c '^Build (' "$WORK/jobs.tsv") -check "build jobs ran ($builds)" test "$builds" -gt 0 -# shellcheck disable=SC2016 # $1 and $2 are awk fields -check "every job except publish succeeded" \ - awk -F'\t' '$1 != "Publish to crates.io" && $2 != "success" { bad = 1 } END { exit bad }' "$WORK/jobs.tsv" -publish=$(awk -F'\t' '$1 == "Publish to crates.io" { print $2 }' "$WORK/jobs.tsv") -if [ -z "$publish" ]; then - skip "publish job (not in this run)" -elif [ "$PRERELEASE" = true ]; then - check "publish job skipped for a pre-release" test "$publish" = skipped +jobs_ok=false +if gh run view "$RID" --json jobs --jq '.jobs[] | "\(.name)\t\(.conclusion)"' \ + >"$WORK/jobs.tsv" 2>"$WORK/jobs.err" && [ -s "$WORK/jobs.tsv" ]; then + jobs_ok=true + awk -F'\t' '$1 ~ /^Build \(/ { t = $1; sub(/^Build \(/, "", t); sub(/\)$/, "", t); print t }' \ + "$WORK/jobs.tsv" | LC_ALL=C sort >"$WORK/built.txt" + check "one build job per release.yml target ($(wc -l <"$WORK/targets.txt" | tr -d ' '))" \ + cmp -s "$WORK/targets.txt" "$WORK/built.txt" + # shellcheck disable=SC2016 # $1 and $2 are awk fields + check "every job except publish succeeded" \ + awk -F'\t' '$1 != "Publish to crates.io" && $2 != "success" { bad = 1 } END { exit bad }' "$WORK/jobs.tsv" + publish=$(awk -F'\t' '$1 == "Publish to crates.io" { print $2 }' "$WORK/jobs.tsv") + if [ "$HAS_PUBLISH" = false ]; then + skip "publish job (release.yml at ${SHA:0:7} has none)" + elif [ -z "$publish" ]; then + fail "publish job is in the run" + elif [ "$PRERELEASE" = true ]; then + check "publish job skipped for a pre-release" test "$publish" = skipped + else + check "publish job ran and reported success" test "$publish" = success + fi else - check "publish job ran and reported success" test "$publish" = success + fail "fetch the run's jobs ($(first_line "$WORK/jobs.err"))" fi # -------------------------------------------------------------------------- @@ -137,55 +197,65 @@ fi # output only, as "\t", without timestamps or colors. gh writes # a log's escape sequences in caret notation ("^[[1m") when not on a # terminal, so strip both that and the raw ESC form. -gh run view "$RID" --log >"$WORK/raw.log" 2>"$WORK/raw.err" || fail "fetch the run log" -perl -ne ' - s/(?:\e|\^\[)\[[0-9;?]*[A-Za-z]//g; - s/\r//g; - my ($job, $step, $msg) = split /\t/, $_, 3; - next unless defined $msg; - $msg =~ s/^\xEF\xBB\xBF//; - $msg =~ s/^\S+Z ?//; - $msg =~ s/\s+$//; - $skip = 0 if !defined $prev || $job ne $prev; - $prev = $job; - if ($msg =~ /^##\[group\]Run /) { $skip = 1; next } - if ($skip && $msg =~ /^##\[endgroup\]/) { $skip = 0; next } - print "$job\t$msg\n" unless $skip; -' "$WORK/raw.log" >"$WORK/output.log" - -logged=$(cut -f1 "$WORK/output.log" | sort -u | wc -l | tr -d ' ') -ran=$(awk -F'\t' '$2 != "skipped"' "$WORK/jobs.tsv" | wc -l | tr -d ' ') -check "log covers every job that ran ($ran)" test "$logged" -eq "$ran" +if gh run view "$RID" --log >"$WORK/raw.log" 2>"$WORK/raw.err" && [ -s "$WORK/raw.log" ]; then + # shellcheck disable=SC2016 # perl's variables, not the shell's + "$PERL" -ne ' + s/(?:\e|\^\[)\[[0-9;?]*[A-Za-z]//g; + s/\r//g; + my ($job, $step, $msg) = split /\t/, $_, 3; + next unless defined $msg; + $msg =~ s/^\xEF\xBB\xBF//; + $msg =~ s/^\S+Z ?//; + $msg =~ s/\s+$//; + $skip = 0 if !defined $prev || $job ne $prev; + $prev = $job; + if ($msg =~ /^##\[group\]Run /) { $skip = 1; next } + if ($skip && $msg =~ /^##\[endgroup\]/) { $skip = 0; next } + print "$job\t$msg\n" unless $skip; + ' "$WORK/raw.log" >"$WORK/output.log" -check "validate parsed the tag as $VERSION" has_line "Validate Release${TAB}Version: $VERSION" "$WORK/output.log" -check "validate matched Cargo.toml version $BASE" has_line "Validate Release${TAB}Version verified: $BASE" "$WORK/output.log" -check "changelog starts at $PREV" has_line "Create Release${TAB}Generating changelog since $PREV" "$WORK/output.log" -check "changelog did not fall back to the last 20 commits" lacks_regex 'No previous tag found' "$WORK/output.log" -if [ "$PRERELEASE" = false ]; then - case $PREV in - *-*) fail "changelog base $PREV is a pre-release tag (a leftover exercise tag truncates the notes)" ;; - *) pass "changelog base $PREV is a stable tag" ;; - esac -fi -check "no artifact digest mismatch" lacks_regex 'digest.*mismatch|hash mismatch' "$WORK/output.log" + if [ "$jobs_ok" = true ]; then + logged=$(cut -f1 "$WORK/output.log" | sort -u | wc -l | tr -d ' ') + ran=$(awk -F'\t' '$2 != "skipped"' "$WORK/jobs.tsv" | wc -l | tr -d ' ') + check "log covers every job that ran ($ran)" test "$logged" -eq "$ran" + fi + check "validate parsed the tag as $VERSION" has_line "Validate Release${TAB}Version: $VERSION" "$WORK/output.log" + check "validate matched Cargo.toml version $BASE" has_line "Validate Release${TAB}Version verified: $BASE" "$WORK/output.log" + check "changelog starts at $PREV" has_line "Create Release${TAB}Generating changelog since $PREV" "$WORK/output.log" + check "changelog did not fall back to the last 20 commits" lacks_regex 'No previous tag found' "$WORK/output.log" + if [ "$PRERELEASE" = false ]; then + case $PREV in + *-*) fail "changelog base $PREV is a pre-release tag (a leftover exercise tag truncates the notes)" ;; + *) pass "changelog base $PREV is a stable tag" ;; + esac + fi + check "no artifact digest mismatch" lacks_regex 'digest.*mismatch|hash mismatch' "$WORK/output.log" -# A step under continue-on-error fails without failing its job, and the API -# reports that step's conclusion as success too. Its "##[error]" line is the -# only trace, so any such line means a failure the job status hides. -if grep -F '##[error]' "$WORK/output.log" >"$WORK/errors.log"; then - fail "no step failed silently; failures hidden behind a successful job:" - sed 's/^/ /' "$WORK/errors.log" + # A step under continue-on-error fails without failing its job, and the + # API reports that step's conclusion as success too. Its "##[error]" line + # is the only trace, so any such line means a failure the job status hides. + if grep -F '##[error]' "$WORK/output.log" >"$WORK/errors.log"; then + fail "no step failed silently; failures hidden behind a successful job:" + sed 's/^/ /' "$WORK/errors.log" + else + pass "no step failed silently" + fi else - pass "no step failed silently" + fail "fetch the run log ($(first_line "$WORK/raw.err"))" + skip "step-output checks (no log)" fi # -------------------------------------------------------------------------- # Artifacts # -------------------------------------------------------------------------- -gh api "repos/$REPO/actions/runs/$RID/artifacts" --jq '.artifacts[].name' >"$WORK/artifacts.txt" || exit 2 -binaries=$(grep -c "^$CRATE-" "$WORK/artifacts.txt") -check "one binary artifact per build ($binaries of $builds)" test "$binaries" -eq "$builds" -check "Pages artifact uploaded" grep -qx 'github-pages' "$WORK/artifacts.txt" +if gh api "repos/$REPO/actions/runs/$RID/artifacts" --jq '.artifacts[].name' \ + >"$WORK/artifacts.txt" 2>"$WORK/artifacts.err"; then + missing=$(while IFS= read -r t; do grep -qxF -- "$CRATE-$t" "$WORK/artifacts.txt" || printf '%s ' "$t"; done <"$WORK/targets.txt") + check "a build artifact for every target${missing:+ (missing: $missing)}" test -z "$missing" + check "Pages artifact uploaded" grep -qxF github-pages "$WORK/artifacts.txt" +else + fail "list the run's artifacts ($(first_line "$WORK/artifacts.err"))" +fi # -------------------------------------------------------------------------- # The GitHub Release @@ -199,15 +269,12 @@ if gh release view "$TAG" --json isPrerelease,isDraft,publishedAt,assets \ pass "release $TAG exists" check "release is published, not a draft" test "$is_draft" = false check "release pre-release flag is $PRERELEASE" test "$is_pre" = "$PRERELEASE" - while IFS=$TAB read -r job _; do - case $job in "Build ("*) ;; *) continue ;; esac - target=${job#Build (} - target=${target%)} + while IFS= read -r target; do case $target in *-windows-*) ext=zip ;; *) ext=tar.gz ;; esac check "asset $CRATE-$VERSION-$target.$ext" grep -qxF -- "$CRATE-$VERSION-$target.$ext" "$WORK/release.txt" - done <"$WORK/jobs.tsv" + done <"$WORK/targets.txt" else - fail "release $TAG exists ($(head -n 1 "$WORK/release.err"))" + fail "release $TAG exists ($(first_line "$WORK/release.err"))" fi latest=$(gh release list --limit 50 --json tagName,isLatest --jq '.[] | select(.isLatest) | .tagName') From ce208a1fcb9b8975997f6f1f65480ba6c9b9f475 Mon Sep 17 00:00:00 2001 From: mlevkov Date: Wed, 23 Sep 2026 16:21:42 -0700 Subject: [PATCH 09/26] ci(verify-release): check every asset, the notes, Pages and crates.io Four of the five release assets were checked by name only, and nothing tied any asset to what its build job produced. Each is now downloaded as the run's artifact and must match the release asset's sha256 exactly (the API's digest), hold its binary, README.md and a LICENSE, and carry a binary for its target according to `file` (patterns accept both the macOS and the libmagic word order). An expired artifact falls back to the release asset, with the digest comparison skipped. The host platform's binary for the smoke test now comes from that extraction. The "no artifact digest mismatch" grep goes: download-artifact v8 words a per-artifact mismatch "digest validation failed", and in its default mode the step fails, which the job-conclusion and ##[error] checks already catch. Newly read: the release notes (one entry per commit since the base, newest first at the tagged commit, and GitHub's own compare link naming the same base); the Pages deployment found by tag rather than commit, and while it is live, the served root redirect and rustdoc version; and crates.io, which must lack the version while Cargo.toml says publish = false and have it otherwise. Hidden failures are reported one FAIL per job, so a second one moves the count, and warnings are listed as notes (v0.4.1's run has nine Node 20 deprecation warnings). Mutation-checked in one run: a corrupted artifact, a wrong expected architecture, a wrong binary name, publish = true, and an injected error in a second job (7 FAILs); a wrong changelog base (notes count and compare base FAIL); a wrong served version. --- scripts/verify-release.sh | 225 +++++++++++++++++++++++++++++--------- 1 file changed, 175 insertions(+), 50 deletions(-) diff --git a/scripts/verify-release.sh b/scripts/verify-release.sh index bd22f2c..176c8fd 100755 --- a/scripts/verify-release.sh +++ b/scripts/verify-release.sh @@ -3,24 +3,28 @@ # # Usage: scripts/verify-release.sh [] # -# Checks the latest release.yml run for : job conclusions, what the -# steps printed, the uploaded artifacts, the GitHub Release, the Pages -# deployment, and one published binary. is where the changelog -# should start; by default it is what release.yml computes, -# `git describe --tags --abbrev=0 ^`. +# Checks the newest release.yml run for against the workflow file and +# Cargo.toml at the commit it built: that the tag names that commit; the job +# conclusions; what the steps printed; the GitHub Release, its notes and its +# assets; that every asset is byte-identical to its build job's artifact and +# holds a binary for its target; the Pages deployment and, while it is the +# live one, the served docs; that crates.io agrees with Cargo.toml's +# `publish`; and that the host platform's binary runs. is where +# the changelog should start. By default it is what release.yml computes, +# `git describe --tags --abbrev=0 ^`, over the tags on origin. # # A tag with a hyphen (v0.4.1-ci.1) is a pre-release: the publish job must be -# skipped and the release must not become Latest. To exercise a release.yml -# change without releasing, tag the current Cargo.toml version with a -# pre-release suffix (the validate job compares the part before the hyphen), -# verify the run, then delete the release and its tag. A leftover pre-release -# tag becomes the changelog base of the next release: see CONTRIBUTING.md. +# skipped and the release must not become Latest. The docs job still deploys +# the tagged commit's docs to Pages, and deleting the tag does not undo that. +# CONTRIBUTING.md ("Releasing") describes exercising release.yml this way. # -# Requires gh (authenticated), git, perl and tar. Exits 0 when every check -# passes (skips allowed), 1 when any fails, 2 on a usage or setup error. The -# work directory is kept when a check fails. Binaries before v0.4.0 are not -# executed: they accept a zero OPERATION_TIMEOUT_SECS and go on to open -# network listeners. +# Requires gh (authenticated), git, perl, tar and curl; unzip and file are +# used when present. Exits 0 when every check passes (skips allowed), 1 when +# any fails, 2 when it cannot start. The work directory is kept on failure. +# +# Not yet exercised: a run whose jobs come from different attempts ("Re-run +# failed jobs"). Binaries before v0.4.0 are not executed: they accept a zero +# OPERATION_TIMEOUT_SECS and go on to open network listeners. set -uo pipefail @@ -38,7 +42,7 @@ case ${1:-} in esac [ $# -le 2 ] || usage -for tool in gh git perl tar; do +for tool in gh git perl tar curl; do command -v "$tool" >/dev/null 2>&1 || { echo "required tool not found: $tool" >&2; exit 2; } done PERL=$(command -v perl) @@ -92,6 +96,9 @@ lacks_regex() { [ $? -eq 1 ] } first_line() { head -n 1 "$1" 2>/dev/null; } +is_prefix_of() { case $2 in "$1"*) [ -n "$1" ] ;; *) false ;; esac; } +# shellcheck disable=SC2016 # perl's variables, not the shell's +sha256() { "$PERL" -MDigest::SHA -e 'print Digest::SHA->new(256)->addfile($ARGV[0])->hexdigest' "$1"; } # version_lt A B: A < B, comparing X.Y.Z numerically. # shellcheck disable=SC2016 # perl's variables, not the shell's version_lt() { @@ -122,6 +129,7 @@ git show "$SHA:.github/workflows/release.yml" >"$WORK/release.yml" 2>/dev/null | git show "$SHA:Cargo.toml" >"$WORK/Cargo.toml" || exit 2 CRATE=$(awk -F'"' '/^name = / { print $2; exit }' "$WORK/Cargo.toml") [ -n "$CRATE" ] || { echo "cannot read the package name from Cargo.toml at ${SHA:0:7}" >&2; exit 2; } +PUBLISH=$(awk '/^publish = / { print $3; exit }' "$WORK/Cargo.toml") awk '/^ *- target: / { print $3 }' "$WORK/release.yml" | LC_ALL=C sort >"$WORK/targets.txt" [ -s "$WORK/targets.txt" ] || { echo "no build targets in release.yml at ${SHA:0:7}" >&2; exit 2; } if grep -q '^ *name: Publish to crates.io$' "$WORK/release.yml"; then HAS_PUBLISH=true; else HAS_PUBLISH=false; fi @@ -229,17 +237,26 @@ if gh run view "$RID" --log >"$WORK/raw.log" 2>"$WORK/raw.err" && [ -s "$WORK/ra *) pass "changelog base $PREV is a stable tag" ;; esac fi - check "no artifact digest mismatch" lacks_regex 'digest.*mismatch|hash mismatch' "$WORK/output.log" # A step under continue-on-error fails without failing its job, and the # API reports that step's conclusion as success too. Its "##[error]" line - # is the only trace, so any such line means a failure the job status hides. - if grep -F '##[error]' "$WORK/output.log" >"$WORK/errors.log"; then - fail "no step failed silently; failures hidden behind a successful job:" - sed 's/^/ /' "$WORK/errors.log" + # is the only trace. One FAIL per job, so a second hidden failure moves + # the count instead of hiding under the first. + awk -F'\t' 'index($2, "##[error]") == 1 { print $1 }' "$WORK/output.log" | sort -u >"$WORK/error-jobs.txt" + if [ -s "$WORK/error-jobs.txt" ]; then + while IFS= read -r job; do + fail "no step failed silently in \"$job\":" + awk -F'\t' -v j="$job" '$1 == j && index($2, "##[error]") == 1 { print " " $2 }' "$WORK/output.log" + done <"$WORK/error-jobs.txt" else pass "no step failed silently" fi + awk -F'\t' 'index($2, "##[warning]") == 1 { print substr($2, 12, 150) }' "$WORK/output.log" | + sort | uniq -c >"$WORK/warnings.txt" + if [ -s "$WORK/warnings.txt" ]; then + note "the run logged warnings (not failures):" + sed 's/^/ /' "$WORK/warnings.txt" + fi else fail "fetch the run log ($(first_line "$WORK/raw.err"))" skip "step-output checks (no log)" @@ -261,18 +278,33 @@ fi # The GitHub Release # -------------------------------------------------------------------------- release_ok=false -if gh release view "$TAG" --json isPrerelease,isDraft,publishedAt,assets \ - --jq '"\(.isPrerelease)\t\(.isDraft)\t\(.publishedAt)", (.assets[] | .name)' \ - >"$WORK/release.txt" 2>"$WORK/release.err"; then +if gh release view "$TAG" --json isPrerelease,isDraft,publishedAt \ + --jq '"\(.isPrerelease)\t\(.isDraft)\t\(.publishedAt)"' \ + >"$WORK/release.tsv" 2>"$WORK/release.err"; then release_ok=true - IFS=$TAB read -r is_pre is_draft published <"$WORK/release.txt" + IFS=$TAB read -r is_pre is_draft published <"$WORK/release.tsv" pass "release $TAG exists" check "release is published, not a draft" test "$is_draft" = false check "release pre-release flag is $PRERELEASE" test "$is_pre" = "$PRERELEASE" - while IFS= read -r target; do - case $target in *-windows-*) ext=zip ;; *) ext=tar.gz ;; esac - check "asset $CRATE-$VERSION-$target.$ext" grep -qxF -- "$CRATE-$VERSION-$target.$ext" "$WORK/release.txt" - done <"$WORK/targets.txt" + gh release view "$TAG" --json assets --jq '.assets[] | "\(.name)\t\(.digest // "")"' >"$WORK/assets.tsv" || + fail "list the release's assets" + gh release view "$TAG" --json body --jq .body >"$WORK/body.md" || fail "read the release notes" + + # release.yml writes one "- ()" line per commit since the + # changelog base, newest first; GitHub appends its own notes, whose + # "Full Changelog" link names the base it chose independently. + expected=$(git rev-list --count "$PREV..$SHA" 2>/dev/null) + entries=$(grep -cE '^- .* \([0-9a-f]{7,}\)$' "$WORK/body.md") + check "release notes list the ${expected:-?} commits since $PREV ($entries)" test "$entries" = "${expected:-x}" + newest=$(grep -m 1 -E '^- .* \([0-9a-f]{7,}\)$' "$WORK/body.md" | sed -E 's/.*\(([0-9a-f]+)\)$/\1/') + check "release notes start at the tagged commit (${newest:-none})" is_prefix_of "$newest" "$SHA" + # shellcheck disable=SC2016 # perl's variables, not the shell's + gh_base=$("$PERL" -ne 'print "$1\n" and exit if m{\*\*Full Changelog\*\*: \S*/compare/(\S+?)\.\.\.\S+}' "$WORK/body.md") + if [ -n "$gh_base" ]; then + check "GitHub's notes compare from $PREV too ($gh_base)" test "$gh_base" = "$PREV" + else + skip "GitHub's notes compare from $PREV (no compare link)" + fi else fail "release $TAG exists ($(first_line "$WORK/release.err"))" fi @@ -292,22 +324,119 @@ else fail "release is marked Latest (Latest is ${latest:-none})" fi +# -------------------------------------------------------------------------- +# Assets: each is the file its build job uploaded, and holds its target +# -------------------------------------------------------------------------- +arch_pattern() { + case $1 in + x86_64-unknown-linux-*) echo '^ELF 64-bit .*x86-64' ;; + aarch64-unknown-linux-*) echo '^ELF 64-bit .*aarch64' ;; + x86_64-apple-darwin) echo '^Mach-O 64-bit .*x86_64' ;; + aarch64-apple-darwin) echo '^Mach-O 64-bit .*arm64' ;; + x86_64-pc-windows-*) echo '^PE32\+ executable.*x86-64' ;; + esac +} +has_release_files() { # has_release_files + grep -qxF -- "$2" "$1" && grep -qxF README.md "$1" && grep -q '^LICENSE' "$1" +} +if [ "$release_ok" = true ]; then + # The loop reads targets on fd 3, so a command in its body that reads + # stdin cannot swallow the remaining targets. + while IFS= read -r target <&3; do + case $target in *-windows-*) ext=zip bin=$CRATE.exe ;; *) ext=tar.gz bin=$CRATE ;; esac + name=$CRATE-$VERSION-$target.$ext + # shellcheck disable=SC2016 # $1 and $2 are awk fields + if ! awk -F'\t' -v n="$name" '$1 == n { found = 1 } END { exit !found }' "$WORK/assets.tsv"; then + fail "asset $name" + continue + fi + # shellcheck disable=SC2016 # $1 and $2 are awk fields + digest=$(awk -F'\t' -v n="$name" '$1 == n { print $2 }' "$WORK/assets.tsv") + file="" + dir=$WORK/assets/$target + if gh run download "$RID" -n "$CRATE-$target" -D "$dir" >/dev/null 2>"$WORK/art.err" && [ -f "$dir/$name" ]; then + file=$dir/$name + if [ -n "$digest" ]; then + check "asset $name is its build job's artifact" test "sha256:$(sha256 "$file")" = "$digest" + else + skip "asset $name is its build job's artifact (the API reports no digest)" + fi + elif gh release download "$TAG" --pattern "$name" --dir "$dir" >/dev/null 2>"$WORK/dl.err"; then + file=$dir/$name + skip "asset $name is its build job's artifact (no artifact: $(first_line "$WORK/art.err"))" + else + fail "download $name ($(first_line "$WORK/dl.err"))" + continue + fi + mkdir -p "$dir/x" + if [ "$ext" = zip ]; then + if ! command -v unzip >/dev/null 2>&1; then + skip "contents of $name (no unzip)" + continue + fi + unzip -Z1 "$file" >"$dir/list.txt" 2>/dev/null && unzip -q -o "$file" "$bin" -d "$dir/x" 2>/dev/null + else + tar -tzf "$file" >"$dir/list.txt" 2>/dev/null && tar -xzf "$file" -C "$dir/x" "$bin" 2>/dev/null + fi + check "$name holds $bin, README.md and a LICENSE" has_release_files "$dir/list.txt" "$bin" + pattern=$(arch_pattern "$target") + if [ -z "$pattern" ] || ! command -v file >/dev/null 2>&1; then + skip "$bin in $name is built for $target (no file command or pattern)" + else + file -b "$dir/x/$bin" >"$dir/file.txt" 2>&1 + check "$bin in $name is built for $target" grep -qE -- "$pattern" "$dir/file.txt" + fi + done 3<"$WORK/targets.txt" +else + skip "assets (no release)" +fi + # -------------------------------------------------------------------------- # Pages # -------------------------------------------------------------------------- # Reachability proves nothing: the site already served the last release's -# docs. The deployment record says what this run deployed. A later deploy -# turns this one inactive, so look for success anywhere in its history. -dep=$(gh api "repos/$REPO/deployments?environment=github-pages&sha=$SHA&per_page=1" --jq '.[0].id // empty') -if [ -n "$dep" ]; then - check "Pages deployed from ${SHA:0:7}" \ - test "$(gh api "repos/$REPO/deployments/$dep/statuses" --jq '[.[].state] | index("success") != null')" = true +# docs. The deployment record for this tag says what the run deployed, and a +# later deploy turns it inactive, so look for success anywhere in its +# history. The served pages are checked only while this deployment is live. +dep=$(gh api "repos/$REPO/deployments?environment=github-pages&ref=$TAG&per_page=1" \ + --jq '.[] | "\(.id) \(.sha)"') +read -r dep_id dep_sha <<<"$dep" +if [ -z "${dep_id:-}" ]; then + fail "Pages deployment for $TAG (none recorded)" +else + check "Pages deployment for $TAG built ${SHA:0:7}" test "$dep_sha" = "$SHA" + check "Pages deployment for $TAG succeeded" \ + test "$(gh api "repos/$REPO/deployments/$dep_id/statuses" --jq '[.[].state] | index("success") != null')" = true + live=$(gh api "repos/$REPO/deployments?environment=github-pages&per_page=1" --jq '.[].id') + if [ "$live" != "$dep_id" ]; then + skip "served docs (a later deployment replaced this one)" + else + site=$(gh api "repos/$REPO/pages" --jq .html_url) + # A query string sidesteps the CDN's cached copy of the old site. + curl -fsSL "${site%/}/?verify=$RID" >"$WORK/site-index.html" 2>"$WORK/site.err" + check "Pages root redirects to $CRATE/index.html" grep -qF "url=$CRATE/index.html" "$WORK/site-index.html" + curl -fsSL "${site%/}/$CRATE/index.html?verify=$RID" >"$WORK/site-crate.html" 2>>"$WORK/site.err" + check "Pages serves the $BASE docs" grep -qF "\"version\">$BASE<" "$WORK/site-crate.html" + fi +fi + +# -------------------------------------------------------------------------- +# crates.io +# -------------------------------------------------------------------------- +if [ "$PRERELEASE" = true ]; then + skip "crates.io (pre-release)" else - fail "Pages deployed from ${SHA:0:7} (no deployment record)" + status=$(curl -s -o /dev/null -w '%{http_code}' -A "verify-release ($REPO)" \ + "https://crates.io/api/v1/crates/$CRATE/$BASE") + if [ "$PUBLISH" = false ]; then + check "crates.io does not have $CRATE $BASE (publish = false; HTTP $status)" test "$status" = 404 + else + check "crates.io has $CRATE $BASE (HTTP $status)" test "$status" = 200 + fi fi # -------------------------------------------------------------------------- -# One published binary, for the host platform +# The host platform's binary runs # -------------------------------------------------------------------------- case "$(uname -s)-$(uname -m)" in Darwin-arm64) host=aarch64-apple-darwin ;; @@ -316,21 +445,14 @@ case "$(uname -s)-$(uname -m)" in Linux-aarch64 | Linux-arm64) host=aarch64-unknown-linux-gnu ;; *) host= ;; esac -asset=$CRATE-$VERSION-$host.tar.gz -if [ "$release_ok" != true ]; then - skip "host binary (no release to download from)" -elif [ -z "$host" ] || ! grep -qxF -- "$asset" "$WORK/release.txt"; then - skip "host binary (no asset for $(uname -s)-$(uname -m))" +bin=$WORK/assets/$host/x/$CRATE +if [ -z "$host" ]; then + skip "host binary (no build target for $(uname -s)-$(uname -m))" elif version_lt "$BASE" 0.4.0; then skip "host binary ($BASE predates the zero-timeout check it relies on)" -elif ! gh release download "$TAG" --pattern "$asset" --dir "$WORK/bin" >/dev/null 2>"$WORK/download.err"; then - fail "download $asset ($(head -n 1 "$WORK/download.err"))" +elif [ ! -x "$bin" ]; then + skip "host binary (no $host asset was extracted above)" else - tar -tzf "$WORK/bin/$asset" >"$WORK/contents.txt" - # shellcheck disable=SC2016 # $1 and $2 are the inner shell's arguments - check "archive holds $CRATE, README.md and a LICENSE" \ - sh -c 'grep -qx "$1" "$2" && grep -qx README.md "$2" && grep -q "^LICENSE" "$2"' _ "$CRATE" "$WORK/contents.txt" - tar -xzf "$WORK/bin/$asset" -C "$WORK/bin" # Config::validate rejects a zero OPERATION_TIMEOUT_SECS, so the binary # logs its version and exits with EX_CONFIG (78) before any network I/O. # Run it from an empty directory (the app loads a .env from its working @@ -338,7 +460,7 @@ else # freshly downloaded binary the caller's tokens), and give it 30 seconds. mkdir -p "$WORK/run" (cd "$WORK/run" && env -i OPERATION_TIMEOUT_SECS=0 RUST_LOG=info \ - "$PERL" -e 'alarm 30; exec @ARGV or exit 127' "$WORK/bin/$CRATE") >"$WORK/bin.raw" 2>&1 + "$PERL" -e 'alarm 30; exec @ARGV or exit 127' "$bin") >"$WORK/bin.raw" 2>&1 code=$? "$PERL" -pe 's/\e\[[0-9;]*m//g' "$WORK/bin.raw" >"$WORK/bin.log" if [ "$code" -eq 142 ]; then @@ -355,5 +477,8 @@ if [ "$PRERELEASE" = true ] && [ "$release_ok" = true ]; then echo echo "If $TAG only exercised the workflow, delete it before the next release:" echo " gh release delete $TAG --cleanup-tag --yes" + echo "Its docs job deployed ${SHA:0:7}'s docs to Pages, which deleting the tag does" + echo "not undo; re-run the Deploy Documentation job of the latest stable release's" + echo "run to restore them." fi [ "$failures" -eq 0 ] From ab3a6e7286d390dd4a7f279cf3d3c30f480aa944 Mon Sep 17 00:00:00 2001 From: mlevkov Date: Wed, 23 Sep 2026 16:22:18 -0700 Subject: [PATCH 10/26] ci: run the release verifier after every green Release run The verifier was only a CONTRIBUTING paragraph: nothing ran it, so TD-2026-09-04's trigger ("the verifier fails on the next stable release") depended on someone remembering, the same silence one level up that the verifier exists to end. verify-release.yml runs it on workflow_run after each green Release run from a tag push, with a read-only token (actions, contents, deployments, pages), a 20-minute limit, and the tag name passed through the environment rather than interpolated into the script. It checks out the default branch, so the verifier that runs is the reviewed one, never code from the tag. It fails on every stable release until TD-2026-09-04 is resolved, by design; the script's header now says so. It cannot run before this lands on main (workflow_run uses the default branch's file), so its first run is the next release or exercise tag. --- .github/workflows/verify-release.yml | 41 ++++++++++++++++++++++++++++ scripts/verify-release.sh | 4 +++ 2 files changed, 45 insertions(+) create mode 100644 .github/workflows/verify-release.yml diff --git a/.github/workflows/verify-release.yml b/.github/workflows/verify-release.yml new file mode 100644 index 0000000..ca05652 --- /dev/null +++ b/.github/workflows/verify-release.yml @@ -0,0 +1,41 @@ +# Verify every green Release run step by step. +# +# A Release run can conclude success while a step failed under +# continue-on-error, and nothing else inspects what a run produced, so this +# runs scripts/verify-release.sh after each one: a problem shows up as a +# failed run here instead of waiting for someone to remember the script. +# Until TD-2026-09-04 is resolved it fails on every stable release, by +# design: the crates.io publish step fails silently there. + +name: Verify Release + +on: + workflow_run: + workflows: [Release] + types: [completed] + +permissions: + actions: read + contents: read + deployments: read + pages: read + +jobs: + verify: + name: Verify release run + # Green runs from a tag push only: a failed run is already visible. + if: github.event.workflow_run.event == 'push' && github.event.workflow_run.conclusion == 'success' + runs-on: ubuntu-latest + timeout-minutes: 20 + steps: + - uses: actions/checkout@v7 + with: + fetch-depth: 0 + + - name: Verify the release run + env: + GH_TOKEN: ${{ github.token }} + # Through the environment, not interpolated into the script, so a + # crafted tag name cannot inject shell. + TAG: ${{ github.event.workflow_run.head_branch }} + run: scripts/verify-release.sh "$TAG" diff --git a/scripts/verify-release.sh b/scripts/verify-release.sh index 176c8fd..17cd3e0 100755 --- a/scripts/verify-release.sh +++ b/scripts/verify-release.sh @@ -13,6 +13,10 @@ # the changelog should start. By default it is what release.yml computes, # `git describe --tags --abbrev=0 ^`, over the tags on origin. # +# .github/workflows/verify-release.yml runs this after every green Release +# run. Until TD-2026-09-04 is resolved, a stable release fails "no step +# failed silently": its crates.io publish step fails under continue-on-error. +# # A tag with a hyphen (v0.4.1-ci.1) is a pre-release: the publish job must be # skipped and the release must not become Latest. The docs job still deploys # the tagged commit's docs to Pages, and deleting the tag does not undo that. From dc9fdc42d83836d411bc4b62ff8ac4c9ce3a170c Mon Sep 17 00:00:00 2001 From: mlevkov Date: Wed, 23 Sep 2026 16:24:56 -0700 Subject: [PATCH 11/26] ci: lint shell scripts with shellcheck scripts/verify-release.sh now runs after every release and its checks gate a release's credibility, but nothing linted it: shellcheck ran only by hand. A Shellcheck job lints scripts/*.sh as bash in every CI run, and CI Success, a required check, depends on it. --- .github/workflows/ci.yml | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 359c52b..88ca941 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -246,13 +246,28 @@ jobs: fi echo "Dockerfile rust $image is at or above rust-version $msrv" + # ========================================================================== + # Shell scripts (scripts/verify-release.sh runs after every release) + # ========================================================================== + shellcheck: + name: Shellcheck + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + # Preinstalled on GitHub's ubuntu runners; -s bash because the + # scripts are bash, and they must stay clean under it. + - name: Lint shell scripts + run: | + shellcheck --version + shellcheck -s bash scripts/*.sh + # ========================================================================== # Final status check (for branch protection) # ========================================================================== ci-success: name: CI Success runs-on: ubuntu-latest - needs: [fmt, clippy, test, integration, docs, audit, licenses] + needs: [fmt, clippy, test, integration, docs, audit, licenses, shellcheck] if: always() steps: - name: Check all jobs passed @@ -263,7 +278,8 @@ jobs: [[ "${{ needs.integration.result }}" != "success" ]] || \ [[ "${{ needs.docs.result }}" != "success" ]] || \ [[ "${{ needs.audit.result }}" != "success" ]] || \ - [[ "${{ needs.licenses.result }}" != "success" ]]; then + [[ "${{ needs.licenses.result }}" != "success" ]] || \ + [[ "${{ needs.shellcheck.result }}" != "success" ]]; then echo "One or more jobs failed" exit 1 fi From 3eadd8cfd22820998576c9ccd71cf26793ba72fc Mon Sep 17 00:00:00 2001 From: mlevkov Date: Wed, 23 Sep 2026 16:25:30 -0700 Subject: [PATCH 12/26] ci: mark the lines the verifier and two TD records depend on TD-2026-09-03 and TD-2026-09-04 bind to "the next edit" of the Codecov step, the publish job and Cargo.toml's `publish = false`, but nothing at those sites said so, while this repo marks TD-2026-09-01's tripwires in ci.yml. Each site now names its record, and so do the four release.yml echo lines scripts/verify-release.sh reads and the changelog's "- %s (%h)" entry format it counts. Three of those lines it requires verbatim, so rewording one fails the verifier with a visible reason; the fallback's line it only checks for absence. Comments only: no step's behavior changes, and Cargo.toml still passes `cargo metadata --locked`. --- .github/workflows/ci.yml | 3 +++ .github/workflows/release.yml | 7 +++++++ Cargo.toml | 3 ++- 3 files changed, 12 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 88ca941..fa6ca49 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -136,6 +136,9 @@ jobs: - name: Generate coverage report run: cargo llvm-cov --locked --all-features --workspace --lcov --output-path lcov.info + # TD-2026-09-03: without a CODECOV_TOKEN secret Codecov rejects + # this upload, and fail_ci_if_error: false keeps the job green. + # Resolve the record before editing this step. - name: Upload coverage to Codecov uses: codecov/codecov-action@303a32d7a59b442fa8d48b6a1cc6825c09c847a5 # v7.1.1 with: diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index cdad160..d899122 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -46,6 +46,7 @@ jobs: echo "is_prerelease=false" >> $GITHUB_OUTPUT fi + # scripts/verify-release.sh matches this line exactly echo "Version: $VERSION" - name: Verify Cargo.toml version matches tag @@ -61,6 +62,7 @@ jobs: exit 1 fi + # scripts/verify-release.sh matches this line exactly echo "Version verified: $CARGO_VERSION" # ========================================================================== @@ -176,6 +178,8 @@ jobs: # Get the previous tag PREV_TAG=$(git describe --tags --abbrev=0 HEAD^ 2>/dev/null || echo "") + # scripts/verify-release.sh matches both echo lines below exactly + # and counts the "- %s (%h)" entries; change them together. if [ -n "$PREV_TAG" ]; then echo "Generating changelog since $PREV_TAG" CHANGELOG=$(git log --pretty=format:"- %s (%h)" $PREV_TAG..HEAD) @@ -200,6 +204,9 @@ jobs: # ========================================================================== # Publish to crates.io (optional) # ========================================================================== + # TD-2026-09-04: this job cannot succeed (there is no CRATES_IO_TOKEN + # secret, and Cargo.toml sets publish = false), and continue-on-error hides + # that. Resolve the record before editing the job. publish: name: Publish to crates.io runs-on: ubuntu-latest diff --git a/Cargo.toml b/Cargo.toml index 4f99255..e775608 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -6,7 +6,8 @@ rust-version = "1.93.0" description = "A comprehensive demonstration of Apache Iggy message streaming with Axum" license = "MIT" # Repo-level releases only - never published to crates.io (enforced here, -# not just in CI: cargo refuses to publish with this set) +# not just in CI: cargo refuses to publish with this set). release.yml still +# runs a publish job, which this makes fail silently: TD-2026-09-04. publish = false repository = "https://github.com/mlevkov/iggy_sample" homepage = "https://github.com/mlevkov/iggy_sample" From 0f4577e9c0b59ffeb3a91fb87fba93d9ec5218c3 Mon Sep 17 00:00:00 2001 From: mlevkov Date: Wed, 23 Sep 2026 16:26:24 -0700 Subject: [PATCH 13/26] docs: say what exercising release.yml leaves behind CONTRIBUTING's Releasing section called a pre-release tag a way to exercise release.yml "without releasing", but the docs job has no pre-release guard: v0.4.1-ci.1's run deployed b7aba82's docs to Pages, and deleting the tag did not undo it. The section now lists both things that outlive an exercise (the Pages deployment, with how to restore it, and the tag), the steps before tagging (version, changelog, TD records bound to the next release), and the Verify Release workflow, which fails every stable release until TD-2026-09-04 is resolved. It names the trigger patterns (vX.Y.Z, vX.Y.Z-) rather than v*, and says a leftover tag truncates the changelog release.yml writes, not "the notes" (GitHub appends its own, over its own range). The README stops claiming coverage is uploaded to Codecov, lists the new workflow and scripts/, and, like SECURITY.md, says Dependabot skips versions that need a newer Rust than the MSRV. The changelog entry now covers what the verifier checks and the workflow and lint that run it. --- CHANGELOG.md | 29 ++++++++++++++++++++++------- CONTRIBUTING.md | 37 +++++++++++++++++++++++++------------ README.md | 14 ++++++++++---- SECURITY.md | 4 +++- 4 files changed, 60 insertions(+), 24 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 19c4cbe..7022bc9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,13 +10,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - `scripts/verify-release.sh` checks a release.yml run step by step instead - of trusting its conclusion: what each step printed, the artifacts, the - GitHub Release and its assets, the Pages deployment record, and that the - published binary for the host platform runs and reports its version. It - also fails on a step that failed under `continue-on-error`, which the - Actions API reports as a success; the crates.io publish step has failed - that way on every stable release (TD-2026-09-04). CONTRIBUTING.md now - describes releasing and exercising `release.yml` with a pre-release tag + of trusting its conclusion: that the tag names the commit the run built; + what each step printed; the release notes; that every asset is + byte-identical to its build job's artifact and holds a binary for its + target; the Pages deployment and the served docs; crates.io against + Cargo.toml's `publish`; and that the host platform's binary runs, isolated + and time-bounded. It fails on a step that failed under `continue-on-error`, + which the Actions API reports as a success: the crates.io publish step has + failed that way on every stable release (TD-2026-09-04) +- The Verify Release workflow runs it after every green Release run, so a + hidden failure fails a run instead of waiting for someone to look. Until + TD-2026-09-04 is resolved it fails on every stable release, by design +- CI lints shell scripts with shellcheck, as part of the required CI Success + check +- CONTRIBUTING.md describes releasing, and exercising `release.yml` with a + pre-release tag, including the two things that outlive the exercise: the + Pages deployment and the tag itself + +### Fixed + +- The README no longer says coverage is uploaded to Codecov: every upload + whose log is still retained was rejected for want of a token + (TD-2026-09-03) ## [0.4.1] - 2026-09-23 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index efe1078..d7a4b61 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -185,23 +185,36 @@ src/ ## Releasing -Pushing a `v*` tag runs `.github/workflows/release.yml`, which builds the -binaries, creates the GitHub Release and deploys the docs; only admins can -create tags. A green run does not mean every step worked (a step under -`continue-on-error` fails silently), so once the run completes, check what it -did: +Pushing a `vX.Y.Z` tag (or `vX.Y.Z-` for a pre-release) runs +`.github/workflows/release.yml`, which builds the binaries, creates the GitHub +Release and deploys the docs to GitHub Pages; only admins can create tags. +Before tagging, set `version` in `Cargo.toml` (the validate job requires the +tag to match it), move the `[Unreleased]` changelog entries under the new +version, and resolve or re-bind every record in `docs/tech-debt/` whose +trigger is the next release. + +A green run does not mean every step worked, since a step under +`continue-on-error` fails silently. So `.github/workflows/verify-release.yml` +runs `scripts/verify-release.sh` after every green Release run; until +TD-2026-09-04 is resolved it fails on every stable release, by design. To +check a run by hand: ```bash scripts/verify-release.sh v0.4.2 ``` -To exercise a `release.yml` change without releasing, push a pre-release tag -of the current `Cargo.toml` version, such as `v0.4.1-ci.1` (the validate job -compares only the part before the hyphen), and verify it the same way. Then -delete it with `gh release delete v0.4.1-ci.1 --cleanup-tag --yes`. Do not -leave it behind: the changelog starts from the nearest earlier tag -(`git describe --tags`), so a leftover pre-release tag drops every commit -before it from the next release's notes. +To exercise a `release.yml` change before a real release, push a pre-release +tag of the current `Cargo.toml` version, such as `v0.4.1-ci.1` (the validate +job compares only the part before the hyphen), and verify its run the same +way. Two things outlive the exercise: + +- The docs job deploys the tagged commit's docs to Pages even for a + pre-release, and deleting the tag does not undo that. Re-run the Deploy + Documentation job of the latest stable release's run to restore its docs. +- The tag. Delete it with `gh release delete v0.4.1-ci.1 --cleanup-tag --yes` + from your clone (this also deletes the local tag). Left behind, it becomes + where `release.yml` starts the changelog it writes for the next release + (`git describe --tags`), which drops every commit before it. ## Questions? diff --git a/README.md b/README.md index cbafe2f..256e810 100644 --- a/README.md +++ b/README.md @@ -379,7 +379,8 @@ iggy_sample/ │ ├── workflows/ │ │ ├── ci.yml # Main CI (tests, lint, coverage) │ │ ├── pr.yml # PR checks (size, commits, docs) -│ │ ├── release.yml # Multi-platform release builds +│ │ ├── release.yml # Release builds, GitHub release, docs deploy +│ │ ├── verify-release.yml # Checks each green release run │ │ └── extended-tests.yml # Weekly stress/benchmark tests │ ├── dependabot.yml # Automated dependency updates │ └── pull_request_template.md @@ -423,6 +424,8 @@ iggy_sample/ │ ├── streams.rs # Stream management │ ├── topics.rs # Topic management │ └── util.rs # Shared handler utilities +├── scripts/ +│ └── verify-release.sh # Checks a release run step by step ├── tests/ │ ├── integration_tests.rs # End-to-end API tests │ ├── metrics_smoke_test.rs # Prometheus exporter smoke test @@ -683,14 +686,16 @@ This project uses GitHub Actions for continuous integration and deployment: |----------|---------|-------------| | `ci.yml` | Push, PR, weekly | Tests, linting, coverage, dependency policy, security audit | | `pr.yml` | PR | Size checks, conventional commits, semver | -| `release.yml` | Tag `v*` | Multi-platform builds, GitHub release | +| `release.yml` | Tag `vX.Y.Z` | Multi-platform builds, GitHub release, docs deploy | +| `verify-release.yml` | Green Release run | Checks the run with `scripts/verify-release.sh` | | `extended-tests.yml` | Weekly | Benchmarks, stress tests, memory checks | ### Automated Checks - **Formatting**: `cargo fmt --check` - **Linting**: `cargo clippy -- -D warnings` - **Tests**: stable and beta on Linux, macOS and Windows, plus the MSRV on Linux -- **Coverage**: Uploaded to Codecov +- **Coverage**: `cargo llvm-cov`; the Codecov upload is not authenticated + yet (TD-2026-09-03) - **Dependency policy**: `cargo deny --locked check` (advisories, bans, licenses, sources) - **Security**: `cargo-audit` vulnerability scanning; blocking on pushes and @@ -698,7 +703,8 @@ This project uses GitHub Actions for continuous integration and deployment: ### Dependabot Automatically creates PRs for: -- Cargo dependency updates, direct and transitive (weekly) +- Cargo dependency updates, direct and transitive (weekly), except a + version that needs a newer Rust than the MSRV (TD-2026-07-02) - GitHub Actions updates (weekly) ## Documentation diff --git a/SECURITY.md b/SECURITY.md index 297b04a..fcaee62 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -43,7 +43,9 @@ Dependencies are monitored via: - Dependabot version updates every Monday, for direct and transitive Cargo dependencies alike: one grouped PR for minor and patch updates, a separate PR per major bump (a pre-1.0 minor bump such as 0.8 to 0.9 - counts as major), and one grouped PR for GitHub Actions + counts as major), and one grouped PR for GitHub Actions. Dependabot + silently skips a version that needs a newer Rust than this crate's + `rust-version` (TD-2026-07-02); the checks below still flag it - `cargo deny --locked check` in CI, gating every pull request through the required `CI Success` check and also run weekly: vulnerability, unmaintained and unsound advisories in any crate fail it, and so does a From e148351da4516cc23c5e19b8645796a2afbc8f7c Mon Sep 17 00:00:00 2001 From: mlevkov Date: Wed, 23 Sep 2026 16:27:32 -0700 Subject: [PATCH 14/26] docs(tech-debt): correct the counts and wording review found TD-2026-09-03 said 14 main push runs were rejected; there are 15 with a retained log, all rejected. The count filtered on run conclusion and so dropped 35910387482, cancelled by the concurrency group after its upload step had run. Its title no longer says the upload "has never authenticated", which reaches past the retained logs; it now says what was measured. Precision fixes: TD-2026-09-04 says the verifier fails on any ##[error] line, which it does, notes that crates.io's 404 needs a User-Agent (403 without), and records the Verify Release workflow and the comments that now name it. TD-2026-09-06 says Dependabot also rebases on a scheduled run, and states that merge queues are for organization-owned repositories only. TD-2026-07-02 points at the log's closing verdict ("No update possible for iggy 0.10.0"), says the MSRV raise also fires TD-2026-09-02, and puts "by hand" in its Status line. TD-2026-07-07's pointer says PR #37 already moved rust-cache as predicted, and no longer calls v2.9.2 (released 2026-08-06) new. TD-2026-09-05's second trigger, an upstream audit-check release, was an event nothing here observes; it now binds to the next Dependabot PR that moves either pin, with the upstream check done then. --- docs/tech-debt/README.md | 6 +++--- docs/tech-debt/TD-2026-07-02.md | 21 +++++++++++---------- docs/tech-debt/TD-2026-07-07.md | 6 ++++-- docs/tech-debt/TD-2026-09-03.md | 21 +++++++++++++-------- docs/tech-debt/TD-2026-09-04.md | 20 +++++++++++++------- docs/tech-debt/TD-2026-09-05.md | 5 +++-- docs/tech-debt/TD-2026-09-06.md | 29 ++++++++++++++++------------- 7 files changed, 63 insertions(+), 45 deletions(-) diff --git a/docs/tech-debt/README.md b/docs/tech-debt/README.md index ab0acc7..afa061d 100644 --- a/docs/tech-debt/README.md +++ b/docs/tech-debt/README.md @@ -6,7 +6,7 @@ condition under which the record MUST be resolved (not "someday"). | ID | Title | Source | Trigger | Status | |----|-------|--------|---------|--------| | [TD-2026-07-01](TD-2026-07-01.md) | `with_reconnect` composition test matrix | Review session 01 (tests #2) | Any behavioral change to `with_reconnect`/`retry_once` | resolved (session 02) | -| [TD-2026-07-02](TD-2026-07-02.md) | DiagnosticEvents-driven connection state | Review session 01 (architect #2) | Next iggy SDK minor bump | open (trigger FIRED: iggy 0.11.0 released 2026-09-18; Dependabot cannot propose it below Rust 1.95, so upgrade by hand) | +| [TD-2026-07-02](TD-2026-07-02.md) | DiagnosticEvents-driven connection state | Review session 01 (architect #2) | Next iggy SDK minor bump | open (trigger FIRED: iggy 0.11.0 released 2026-09-18; due with the upgrade, by hand: Dependabot cannot propose it below Rust 1.95) | | [TD-2026-07-03](TD-2026-07-03.md) | Half-open probe limiting | Review session 01 (architect #9, types F5) | First production incident involving breaker recovery, or breaker config exposure | resolved (session 02) | | [TD-2026-07-04](TD-2026-07-04.md) | X-Request-Timeout enforcement | Review session 01 (silentfail M4) | Before advertising the header in any client-facing docs beyond CLAUDE.md | resolved (session 02) | | [TD-2026-07-05](TD-2026-07-05.md) | Metrics exporter smoke test | Review session 01 (tests #7) | Next metrics-exporter-prometheus major/minor bump | resolved (session 02) | @@ -16,7 +16,7 @@ condition under which the record MUST be resolved (not "someday"). | [TD-2026-07-09](TD-2026-07-09.md) | Breaker per-state data as enum with payloads | Review session 02 (types MEDIUM) | Next behavioral change to the breaker state or admission gate | resolved (session 03) | | [TD-2026-09-01](TD-2026-09-01.md) | API listener serves cleartext HTTP/2 by feature unification | RUSTSEC-2026-0258 remediation (h2 exposure) | A tripwire failing (CI feature-graph check or either h2c test), next change to the serving path, next `h2` advisory, or next minor bump of `axum`/`hyper-util`/`metrics-exporter-prometheus` | open | | [TD-2026-09-02](TD-2026-09-02.md) | Validate dependabot.yml in CI, and put the Dockerfile on an updater | RustSec 2026-09 remediation review, rounds 1-2 | Next edit to `.github/dependabot.yml` after it lands, or the next `rust-version` change | open | -| [TD-2026-09-03](TD-2026-09-03.md) | Codecov upload has never authenticated | Dependabot first-wave triage (codecov-action bump, PR #37) | Next edit to ci.yml's coverage job, the next major `codecov-action` bump, or the next release | open | +| [TD-2026-09-03](TD-2026-09-03.md) | The Codecov upload fails silently for want of a token | Dependabot first-wave triage (codecov-action bump, PR #37) | Next edit to ci.yml's coverage job, the next major `codecov-action` bump, or the next release | open | | [TD-2026-09-04](TD-2026-09-04.md) | The crates.io publish job cannot succeed | Dependabot first-wave triage (release.yml exercise, PR #37) | The next stable release tag, or the next edit to the publish job or to `publish` in Cargo.toml | open | -| [TD-2026-09-05](TD-2026-09-05.md) | rust-cache is pinned to a branch head under a release comment | Dependabot first-wave triage (SHA pin check, PR #37); carries TD-2026-07-07's addendum | The next github-actions Dependabot PR that moves the rust-cache pin, or the first `rustsec/audit-check` release after v2.0.0 | open | +| [TD-2026-09-05](TD-2026-09-05.md) | rust-cache is pinned to a branch head under a release comment | Dependabot first-wave triage (SHA pin check, PR #37); carries TD-2026-07-07's addendum | The next github-actions Dependabot PR that moves the rust-cache or audit-check pin | open | | [TD-2026-09-06](TD-2026-09-06.md) | main accepts a pull request whose checks ran against an older main | Dependabot first-wave triage (merging PRs #37-#42) | Before merging the next Dependabot wave (next scheduled run 2026-09-28) | open | diff --git a/docs/tech-debt/TD-2026-07-02.md b/docs/tech-debt/TD-2026-07-02.md index 9082cfa..b100953 100644 --- a/docs/tech-debt/TD-2026-07-02.md +++ b/docs/tech-debt/TD-2026-07-02.md @@ -1,7 +1,7 @@ # TD-2026-07-02: DiagnosticEvents-driven connection state **Source:** Review session 01, Round 1 — code-architect finding #2 (HIGH theme). -**Status:** open (mitigated; trigger fired 2026-09-18, due with the iggy 0.11 upgrade) +**Status:** open (mitigated; trigger fired 2026-09-18, due with the iggy 0.11 upgrade, which has to be done by hand) ## Problem @@ -64,9 +64,10 @@ No PR appeared: Dependabot's first cargo run opened #38-#42 and nothing for `iggy`. Its log (Actions run 35888373353) shows both passes over the crate. The grouped `rust-minor` pass saw `Latest version is 0.11.0` and left it alone, as the group takes only minor and patch updates. The ungrouped pass, -the one that would have opened the PR, ended with `Requirements to unlock -update_not_possible`, which is its verdict when no newer version resolves; -the log does not say why at INFO level. +the one that would have opened the PR, reported `Requirements to unlock +update_not_possible` and closed with `No update possible for iggy 0.10.0`, +its verdict when no newer version resolves; the log does not say why at +INFO level. MSRV-aware resolution explains it, and reproduces locally (cargo 1.98.1). Edition 2024 defaults to resolver 3, which prefers dependency versions whose @@ -77,9 +78,9 @@ Rust 1.93.0 compatible versions" and keeps 0.10.0; the same command with `iggy_common` and `iggy_binary_protocol` to 0.11.0. An exact `iggy = "0.11.0"` also resolves, so the graph itself is satisfiable. -So Dependabot will not propose 0.11 while `rust-version` stays below 1.95. -Do this record's upgrade by hand, raising `rust-version` in the same change -(the six places above). More generally, Dependabot silently skips any -update, security fixes included, whose new version needs a newer Rust than -`rust-version`; cargo-deny and cargo-audit read the lockfile, so they still -flag a vulnerable version that stays behind. +So Dependabot will not propose 0.11 while `rust-version` stays below 1.95. Do +this record's upgrade by hand, raising `rust-version` in the same change (the +six places above; that change also fires TD-2026-09-02). More generally, +Dependabot silently skips any update, security fixes included, whose new +version needs a newer Rust than `rust-version`; cargo-deny and cargo-audit +read the lockfile, so they still flag a vulnerable version that stays behind. diff --git a/docs/tech-debt/TD-2026-07-07.md b/docs/tech-debt/TD-2026-07-07.md index 49c5cce..c8d7666 100644 --- a/docs/tech-debt/TD-2026-07-07.md +++ b/docs/tech-debt/TD-2026-07-07.md @@ -61,5 +61,7 @@ whose SHA is not a tagged commit either. For an untagged SHA, dependabot-core proposes the head of the branch containing it rather than a release, so both will float with upstream inside the weekly github-actions PR. Review those diffs as code, and move audit-check back to a release SHA once one exists. -Both returns to release SHAs are now tracked as TD-2026-09-05, since -rust-cache has one available today (v2.9.2). +PR #37 then moved rust-cache to upstream master's head (`f0d9c388…`), as +predicted here. TD-2026-09-05 tracks both returns to release SHAs: +re-pinning rust-cache to v2.9.2 (released 2026-08-06), and moving +audit-check once upstream tags a release. diff --git a/docs/tech-debt/TD-2026-09-03.md b/docs/tech-debt/TD-2026-09-03.md index e9f11ae..848128f 100644 --- a/docs/tech-debt/TD-2026-09-03.md +++ b/docs/tech-debt/TD-2026-09-03.md @@ -1,4 +1,4 @@ -# TD-2026-09-03: Codecov upload has never authenticated +# TD-2026-09-03: The Codecov upload fails silently for want of a token **Source:** Dependabot first-wave triage (2026-09-23), checking what PR #37's `codecov/codecov-action` bump actually did in CI. @@ -13,11 +13,14 @@ at all. Codecov rejects the tokenless upload (`Upload queued for processing failed: {"message":"Token required - not valid tokenless upload"}`), and the step passes anyway because it sets `fail_ci_if_error: false`, so the job stays green. Every main push run whose log is still retained was rejected -this way: 14 runs from 2026-07-05 to 2026-09-23, on codecov-action `v4`, -v4.6.0 and v7.1.1. Older logs have expired. +this way: 15 runs from 2026-07-05 to 2026-09-23, on codecov-action `v4`, +v4.6.0 and v7.1.1. One of them, 35910387482, was cancelled by ci.yml's +concurrency group after its upload step had already run. Older logs have +expired. -The README's "Automated Checks" list states "**Coverage**: Uploaded to -Codecov", which is not true. +The README's "Automated Checks" list said "**Coverage**: Uploaded to +Codecov" until the pull request that filed this record; it now says the +upload is not authenticated and points here. ## Mitigations in place @@ -25,6 +28,8 @@ Codecov", which is not true. cargo-llvm-cov, and pr.yml's Coverage Diff job prints the total on every pull request and warns below 60%. - Nothing reads the Codecov side, so no decision rests on the missing data. +- ci.yml's Codecov step names this record, so an edit to it meets the + trigger in place. ## Options @@ -36,10 +41,10 @@ Codecov", which is not true. 2. **Drop the upload.** Remove the Codecov step and the README line, keeping the local coverage run. -Either way the README line has to end up true. +Either way, update the README's coverage line to match. ## Binding trigger Whichever comes first: the next edit to ci.yml's coverage job, the next -major bump of `codecov/codecov-action`, or the next release (the README -states the upload as fact). +major bump of `codecov/codecov-action`, or the next release (CONTRIBUTING's +release steps include resolving records bound to it). diff --git a/docs/tech-debt/TD-2026-09-04.md b/docs/tech-debt/TD-2026-09-04.md index 461442f..100588e 100644 --- a/docs/tech-debt/TD-2026-09-04.md +++ b/docs/tech-debt/TD-2026-09-04.md @@ -23,15 +23,21 @@ release. It cannot succeed, for two independent reasons: and the Actions API reports the failed step's conclusion as `success` as well (checked on the v0.4.1 run, 35888419705), so the only trace is the step's log line `##[error]Process completed with exit code 1.`. iggy_sample -is not on crates.io (the API returned 404 on 2026-09-23). +is not on crates.io: the API returned 404 on 2026-09-23 (to a request with a +User-Agent; crates.io answers 403 without one). ## Mitigations in place - Nothing can be published by accident: `publish = false` holds whatever the token. -- `scripts/verify-release.sh` fails a release whose log has an `##[error]` - line under a successful job, so the next stable release reports this step - instead of passing it. +- `scripts/verify-release.sh` fails a release whose step output has any + `##[error]` line, including one under a job that concluded success, and + `.github/workflows/verify-release.yml` runs it after every green Release + run, so the next stable release fails that workflow instead of passing. + The verifier also checks that crates.io lacks each version while + `publish = false`. +- The publish job in release.yml and `publish = false` in Cargo.toml both + name this record. ## Options @@ -43,6 +49,6 @@ is not on crates.io (the API returned 404 on 2026-09-23). ## Binding trigger -Whichever comes first: the next stable release tag (the verifier fails on it -until this is resolved), or the next edit to the publish job or to `publish` -in Cargo.toml. +Whichever comes first: the next stable release tag (the Verify Release +workflow fails on it until this is resolved), or the next edit to the +publish job or to `publish` in Cargo.toml. diff --git a/docs/tech-debt/TD-2026-09-05.md b/docs/tech-debt/TD-2026-09-05.md index 82c675e..edd0191 100644 --- a/docs/tech-debt/TD-2026-09-05.md +++ b/docs/tech-debt/TD-2026-09-05.md @@ -51,5 +51,6 @@ open, and this record carries it. ## Binding trigger -Whichever comes first: the next github-actions Dependabot PR that moves the -rust-cache pin, or the first `rustsec/audit-check` release after v2.0.0. +The next github-actions Dependabot PR that moves either pin. When it moves +audit-check, check upstream for a release after v2.0.0 then, since nothing +in this repository notices one on its own. diff --git a/docs/tech-debt/TD-2026-09-06.md b/docs/tech-debt/TD-2026-09-06.md index 5e8077c..9d6889e 100644 --- a/docs/tech-debt/TD-2026-09-06.md +++ b/docs/tech-debt/TD-2026-09-06.md @@ -9,13 +9,15 @@ The `main` ruleset requires CI Success, Clippy, Integration Tests and Rustfmt with `strict_required_status_checks_policy: false`, so a pull request whose checks passed against an older main stays mergeable after main moves, and GitHub's `MERGEABLE` means only that the diff still applies. Lockfile pull -requests are where that bites. After #38 (151 crates) merged, #39 (sha1) -still showed `MERGEABLE` with green checks computed on the pre-#38 lockfile, -and Dependabot never rebased it on its own: it rebases a pull request by -itself only when it conflicts, which #39 never did. Merging it as it stood -would have put an untested tree on main, with main's own CI as its first -test, and ci.yml's concurrency group cancels that run when the next merge -lands, as it did for #40's merge (`5a9054e`). +requests are where that bites. After #38 (151 crates) merged, #39 (sha1) still +showed `MERGEABLE` with green checks computed on the pre-#38 lockfile, and +Dependabot never rebased it on its own: between scheduled runs it rebases a +pull request by itself only when it conflicts (it also rebases open pull +requests when its schedule runs, on reopen, and when the target branch +changes), and #39 never conflicted. Merging it as it stood would have put an +untested tree on main, with main's own CI as its first test, and ci.yml's +concurrency group cancels that run when the next merge lands, as it did for +#40's merge (`5a9054e`). ## Mitigations in place @@ -27,12 +29,13 @@ a procedure, not a gate. ## Proposed fix -Turn on "Require branches to be up to date before merging" in the main -ruleset (`strict_required_status_checks_policy: true`). A pull request behind -main then has to be updated before it merges; for a Dependabot pull request -that is `@dependabot rebase` and a fresh CI run. A merge queue would automate -that loop; check whether GitHub offers one for this user-owned repository -before choosing it. Editing the ruleset is an admin action. +Turn on "Require branches to be up to date before merging" in the main ruleset +(`strict_required_status_checks_policy: true`). A pull request behind main +then has to be updated before it merges; for a Dependabot pull request that is +`@dependabot rebase` and a fresh CI run. A merge queue would automate that +loop, but GitHub offers merge queues only to organization-owned repositories, +so for this user-owned one the setting above is the fix. Editing the ruleset +is an admin action. ## Binding trigger From 5bb70c58e3a19faf0226d5c7d0030cf8e441bd0f Mon Sep 17 00:00:00 2001 From: mlevkov Date: Wed, 23 Sep 2026 16:29:31 -0700 Subject: [PATCH 15/26] docs(code-reviews): release verifier review round 1 Five lenses (tier-graduated cadence) reviewed the verifier and the TD records: 51 findings, all kept after a verification pass over 48 consolidated claims (45 present, 3 drifted and corrected, 0 absent), grouped into 17 themes. Six are HIGH: nothing ran the verifier; its smoke test ran a downloaded binary with the caller's environment; four of five assets were checked by name only; the run was never tied to its tag's commit; exercising release.yml left an unmentioned Pages deployment; and a record miscounted 15 runs as 14. One premise three lenses shared (that gh's --cleanup-tag leaves the local tag) was false per gh's source, which narrows T8 rather than discarding it. The artifact cites each theme's remediation commit by subject, and records five declined suggestions with reasons. --- .../release-verifier-2026-09-round1.md | 369 ++++++++++++++++++ 1 file changed, 369 insertions(+) create mode 100644 docs/code-reviews/release-verifier-2026-09-round1.md diff --git a/docs/code-reviews/release-verifier-2026-09-round1.md b/docs/code-reviews/release-verifier-2026-09-round1.md new file mode 100644 index 0000000..6b8b434 --- /dev/null +++ b/docs/code-reviews/release-verifier-2026-09-round1.md @@ -0,0 +1,369 @@ +# Release Verifier and Dependabot-Wave TD Records — Code Review, Round 1 + +**Target:** branch `chore/release-verifier-and-td-records` vs `main` +(`b7aba82`), 6 commits at review time: `scripts/verify-release.sh` with its +CONTRIBUTING section and changelog entry, TD-2026-09-03 to -06, a trigger +check in TD-2026-07-02 and a pointer in TD-2026-07-07. Four of those commit +messages were later corrected in place (T6, T13) with every tree kept +byte-identical, so commits are cited by subject, not SHA. + +**Provenance:** config: the session ran Opus 5.5 at maximum effort with +`feature-dev` and `pr-review-toolkit` installed, and the five review agents +and the verification agent were pinned to the strongest available model +(Fable). The skill's config gate covers full-suite runs, so it was not +asked; the attestation is the session's own settings. Cadence: +tier-graduated (CLAUDE.md records none; v0.4.1 used four lenses), so a +multi-file change got five. The type-design, architecture and +simplification lenses were not run: a bash script and prose give them +little to grip. Fallback, surfaced: [reviewer] has no shell in its agent +type, so it got the diff, the commit messages and an evidence sheet as +files. Verification (step 4.5): one agent re-read the source for all 48 +consolidated claims: 45 present, 3 drifted (corrected in T7, T10 and T11), +0 absent. The parent session separately re-measured the two claims it had +most reason to doubt (the Codecov recount in T6, and GitHub's merge-queue +availability in T13) and read gh's source for the premise in T8. Findings: +51 from five lenses, all kept; one premise that three of them shared was +false (T8). They are grouped into 17 themes. + +Reviewers cited in brackets: [reviewer] feature-dev:code-reviewer, [silent] +silent-failure-hunter, [comments] comment-analyzer, [tests] pr-test-analyzer, +[consistency] general-purpose (cross-cutting consistency). + +Evidence for the remediation, measured on the remediated verifier: +macOS (bash 3.2): v0.4.1 39 passed / 1 failed / 1 skipped, v0.4.0 38/1/2, +v0.3.0 36/1/3, v0.4.1-ci.1 16/1/4 (its release was deleted after the +exercise); Linux (Ubuntu 24.04 container, arm64): v0.4.1 39/1/1. The one +FAIL on each stable release is TD-2026-09-04's masked publish step. Every +new check was also shown to fail under a mutation (listed per theme). + +--- + +## T1 — HIGH — Nothing ran the verifier, so the record that relies on it depended on memory + +*[silent] — 1 of 5.* + +The PR's thesis is that a green run hides failures because nobody looks, and +TD-2026-09-04's trigger said "the verifier fails on the next stable +release". But nothing ran the verifier: no workflow, job or target in the +repository invoked it (verified: V1), only a CONTRIBUTING paragraph pointed +at it. The trigger would fire only if someone remembered. + +**Remediation:** `ci: run the release verifier after every green Release +run`. `verify-release.yml` runs it on `workflow_run` after each green +Release run from a tag push, with a read-only token, a 20-minute limit, and +the tag name passed through the environment rather than interpolated into +the script; it checks out the default branch, so the reviewed verifier runs, +never code from the tag. It cannot run before it lands on main, so its first +run is the next release or exercise tag. TD-2026-09-04 now cites it. + +## T2 — HIGH — The smoke test ran a downloaded binary with the caller's environment, from the repo root, unbounded + +*[reviewer] [silent] [tests] — 3 of 5.* + +The verifier executed a binary it had just downloaded from the release under +review with the maintainer's full environment (`GH_TOKEN`, any cloud +credentials), from the repository root, where the app's `dotenvy` loads the +gitignored `.env` (config.rs:168; V6, V7). It had no timeout. And its +"exits before any network I/O" rested on `Config::validate` rejecting a zero +`OPERATION_TIMEOUT_SECS`, which exists only from v0.4.0 (commit `671a27a`; +V8): run against v0.3.0 or v0.2.0, the binary opened a Prometheus listener +on 0.0.0.0:9090 and tried to connect before exiting 69 [tests] [silent]. + +**Remediation:** `ci(verify-release): run the binary isolated and +time-bounded`. It now runs under `env -i` from an empty directory, is killed +after 30 seconds (its own FAIL), and is not run for versions before 0.4.0. +Checked directly: a hung stand-in exits 142 after the alarm (it survives +`exec`), `/usr/bin/env` sees only the two variables, `/bin/pwd` is the empty +directory, and a missing binary exits 127. + +## T3 — HIGH — Four of five assets were checked by name only, and nothing tied an asset to its build + +*[tests] [reviewer] — 2 of 5.* + +Only the host platform's asset was downloaded and opened (V9). The Windows +zip had never been opened anywhere, and the aarch64-linux binary, built with +`cross` installed from an unpinned git HEAD, was checked by name alone. The +Package steps swallow errors by design (`cp README.md LICENSE* dist/ +2>/dev/null || true`), so a dropped LICENSE or a wrong-architecture binary +would pass. Nothing compared a release asset with what its build job +produced either, so an asset replaced after the run passed every check +[reviewer]. + +**Remediation:** `ci(verify-release): check every asset, the notes, Pages +and crates.io`. Each target's artifact is downloaded and must match the +release asset's sha256 exactly (the API's `digest`, V10), hold its binary, +README.md and a LICENSE, and carry a binary for its target according to +`file`, with patterns that accept both macOS's and libmagic's word order. +All five matched for v0.4.1 on both hosts. Mutations: a corrupted artifact, +a wrong expected architecture and a wrong binary name each FAIL. The Linux +run also executed the cross-built aarch64 binary for the first time: it +reported 0.4.1 and exited 78. + +## T4 — HIGH — The run was never tied to the commit its tag names + +*[silent] [tests] — 2 of 5.* + +The run was selected by tag name, and its head SHA was never compared with +what the tag points at (V17). v0.2.0 already has two runs for one tag (V18). +If a tag were re-pushed and its new run were missing, the verifier would +verify the old commit's run and exit 0. + +**Remediation:** `ci(verify-release): tie the run to its tag and fail +closed on fetches`. The tag's commit on origin (peeled, for annotated tags) +must equal the run's head; the check is skipped only once the tag is gone, +as for a deleted exercise tag. The header shows the run attempt and notes +when a tag has several runs. Mutation: selecting v0.2.0's older run +(`826d495`) FAILs the check. + +## T5 — HIGH — "Exercise without releasing" omitted the Pages deployment it leaves behind + +*[consistency] [tests] — 2 of 5.* + +CONTRIBUTING called a pre-release tag a way to exercise release.yml "without +releasing", and the script's header described a pre-release only by the +skipped publish job and the unchanged Latest. But the docs job has no +pre-release guard (release.yml's Deploy Documentation job; V26): the +v0.4.1-ci.1 run deployed `b7aba82`'s docs to Pages, and deleting the tag did +not undo it (V27). It was harmless this time, since those docs match v0.4.1's, +but the documented procedure generalizes to commits with unreleased API +changes. + +**Remediation:** `docs: say what exercising release.yml leaves behind`. +CONTRIBUTING lists both things that outlive an exercise, the Pages +deployment (with how to restore it: re-run the latest stable release's +Deploy Documentation job) and the tag, and the script's header and closing +message say the same. Guarding the docs job would stop an exercise from +testing the deploy path, which is what PR #37's exercise needed, so the +workflow is unchanged. + +## T6 — HIGH — Miscount: 15 retained runs, not 14 + +*[comments] — 1 of 5.* + +TD-2026-09-03 said 14 main push runs had a retained log and a rejected +upload; there are 15, all rejected. The count had filtered on run +conclusion and dropped 35910387482, the #40 merge's run, which ci.yml's +concurrency group cancelled after its upload step had already run (V28; +the parent session re-counted: 15 of 15). This is the class of error the +project has been burned by before: a count in a permanent record, +asserted from a filtered query. + +**Remediation:** `docs(tech-debt): correct the counts and wording review +found` corrects the record, and the commit message that filed it now says +15. The CHANGELOG's matching "every upload has been rejected" was narrowed +before commit to the uploads whose logs are retained, and the record's +title no longer claims the upload "has never authenticated". + +## T7 — MEDIUM — Failed fetches left checks to pass on nothing, and exits misreported causes + +*[reviewer] [silent] — 2 of 5.* + +After a failed log fetch the script continued with an empty log, and +`lacks_regex` succeeded on grep's exit 2 as well as 1 (V12), so three +negated checks, including "no step failed silently", printed PASS over a +log that did not exist. Logs expire after 90 days, so every old release +showed that pattern. The artifacts listing exited 2 mid-run, suppressing +the summary (drifted, V13: only that one of the two cited exits was +mid-run); a failing `gh run list` was reported as "no run found" (V14); an +empty job listing reached the "publish job (not in this run)" SKIP and ran +zero asset checks (V15); and no tool was checked before use (V16). + +**Remediation:** `ci(verify-release): tie the run to its tag and fail +closed on fetches`. A failed log fetch FAILs with gh's reason and skips the +step-output checks; negated checks demand a non-empty file; job and +artifact listing failures FAIL instead of exiting mid-run; the run listing's +failure is reported as such; tools are checked up front. The crate name, the +build targets and whether a publish job exists come from Cargo.toml and +release.yml at the run's commit, so a missing build or publish job FAILs +rather than shrinking the check list. Mutations: an expired log (FAIL with +the 410), a broken job listing, a broken run listing (exit 2 with gh's +error). + +## T8 — MEDIUM — The changelog base came from this clone's tags + +*[consistency] [reviewer] [tests] [silent] — 4 of 5.* + +The default base was `git describe` over local tags after a non-pruning +fetch (V19), while release.yml's checkout sees only origin's. From a clone +holding a tag that origin lacks, the verifier computed the wrong base and +produced two false FAILs that blamed the release. [silent] demonstrated it +with a clone-only `v0.3.0-stale.1`. + +Premise corrected: [consistency], [reviewer] and [silent] said the +documented cleanup, `gh release delete --cleanup-tag`, leaves the local tag +behind. gh's source deletes it too when no `-R` is given (cli/cli +`pkg/cmd/release/delete/delete.go:105-107`; V20), so the documented in-clone +usage is clean. The gap is narrower: another clone, a deletion from the web +UI, or `-R`. [tests] had it right. + +**Remediation:** `ci(verify-release): tie the run to its tag and fail +closed on fetches`. The base is computed excluding tags that exist only in +this clone, which are named in a NOTE. From the stale clone, verifying +v0.4.0 now computes `v0.3.0` instead of two false FAILs. + +## T9 — MEDIUM — The release notes and the served docs were never read; Pages was looked up by commit + +*[tests] [reviewer] [silent] — 3 of 5.* + +The release body, the changelog's end product, was never read (V21), and +release.yml's own count cannot expose an empty changelog. The Pages check +proved a deployment record, not what the site serves (V23), and it looked +the record up by commit, so a release and an exercise sharing a commit could +see each other's record (V22). + +**Remediation:** `ci(verify-release): check every asset, the notes, Pages +and crates.io`. The notes must list one entry per commit since the base +(31 for v0.4.1), newest first at the tagged commit, and GitHub's own +"Full Changelog" link must name the same base. The Pages deployment is +found by tag (`ref=`), and while it is the live one the served root +redirect and rustdoc version are checked, with a query string to bypass the +CDN's cache. Mutations: a wrong base FAILs the count and the compare base; a +wrong expected version FAILs the served-docs check. + +## T10 — MEDIUM — The digest-mismatch check duplicated two others and missed the message it targeted + +*[tests] — 1 of 5.* + +`digest.*mismatch|hash mismatch` could not match download-artifact v8's +per-artifact message ("digest validation failed"), which v8 prints only in +its non-default warn and info modes. Drifted (V11): in the default error +mode the failing step's `##[error]` line does match, so the check was +redundant rather than dead; it fired only together with the +job-conclusion and `##[error]` checks. + +**Remediation:** removed in `ci(verify-release): check every asset, the +notes, Pages and crates.io`, where T3's asset-against-artifact digest +comparison covers the whole chain. + +## T11 — MEDIUM — The publish outcome rested on one log marker + +*[silent] — 1 of 5.* + +With the Actions API masking the step, a change in GitHub's `##[error]` +marker would silently remove the only signal for TD-2026-09-04, and the +record's "nothing can be published by accident" was asserted, not checked. +The header also listed curl, which nothing called (V38). + +**Remediation:** `ci(verify-release): check every asset, the notes, Pages +and crates.io`. crates.io must lack the version while Cargo.toml at the +run's commit says `publish = false`, and have it otherwise, queried with a +User-Agent (crates.io answers 403 without one; drifted, V25). Mutation: +`publish = true` FAILs it. curl is now used, for this and the served docs. + +## T12 — MEDIUM — Binding triggers that nothing points at + +*[silent] [consistency] — 2 of 5.* + +TD-2026-09-03 and TD-2026-09-04 bind to "the next edit" of the Codecov step, +the publish job and `publish = false`, but no comment marked those sites +(V2), while this repo marks TD-2026-09-01's tripwires in ci.yml. Two +records bind to "the next release", and CONTRIBUTING's release steps never +mentioned the registry (V3). TD-2026-09-05's second trigger was an upstream +release nothing here observes (V4). + +**Remediation:** `ci: mark the lines the verifier and two TD records depend +on` names the record at each site. CONTRIBUTING's release steps include +resolving records bound to the next release (`docs: say what exercising +release.yml leaves behind`). TD-2026-09-05 binds to the next Dependabot PR +that moves either pin (`docs(tech-debt): correct the counts and wording +review found`). + +## T13 — MEDIUM — Record prose that said more, or less, than the evidence + +*[comments] [consistency] — 2 of 5.* + +- TD-2026-09-06 said Dependabot rebases "only when it conflicts"; its + documentation also lists scheduled runs, reopening and target-branch + changes (V29). The observation about #39 stands. +- TD-2026-09-06 left merge-queue availability to be checked; GitHub's + documentation says merge queues are for organization-owned repositories + only (V33; the parent session read the sentence on the page). +- TD-2026-07-02 said the ungrouped pass "ended with" `Requirements to unlock + update_not_possible`; it closes two lines later with `No update possible + for iggy 0.10.0` (V30). It also did not say that the MSRV raise fires + TD-2026-09-02 (V35), and its Status line lacked "by hand" (V36). +- TD-2026-09-04 said the verifier fails on an `##[error]` line "under a + successful job"; it fails on any (V32). +- CONTRIBUTING said a leftover tag drops commits from "the next release's + notes"; it truncates the changelog release.yml writes, and GitHub appends + its own notes over its own range (V31). +- TD-2026-07-07's pointer called v2.9.2 (released 2026-08-06) available + "today" and did not say PR #37 had already moved the pin (V34). +- Commit messages: the parenthetical in the verifier's commit (V46), the + "ends with" in TD-2026-07-02's (V47) and "rebases only on conflict" in + TD-2026-09-06's (V48). + +**Remediation:** the records in `docs(tech-debt): correct the counts and +wording review found`, CONTRIBUTING in `docs: say what exercising +release.yml leaves behind`, and the four commit messages (with T6's) +corrected in place. + +## T14 — MEDIUM — Docs the PR left stale + +*[consistency] [reviewer] [comments] [tests] — 4 of 5.* + +The README still said "Coverage: Uploaded to Codecov" (V37) while +TD-2026-09-03 recorded that every retained upload was rejected. CONTRIBUTING +said "a `v*` tag" while release.yml triggers only on `vX.Y.Z` and `vX.Y.Z-*` +(V39). SECURITY.md and the README described Dependabot without the MSRV gate +TD-2026-07-02 now documents (V40). The README described release.yml without +the docs deploy and its tree lacked `scripts/` (V41). The release.yml lines +the verifier matches carried no hint of that (V42). + +**Remediation:** `docs: say what exercising release.yml leaves behind` +(README, SECURITY.md, CONTRIBUTING) and `ci: mark the lines the verifier +and two TD records depend on` (release.yml). + +## T15 — LOW — One FAIL for every hidden failure, and warnings dropped + +*[tests] — 1 of 5.* + +Every stable release fails the silent-failure check until TD-2026-09-04 is +resolved, and a second hidden failure in another job would have appeared +only as an extra indented line under the same FAIL, with the count +unchanged (V24). `##[warning]` lines were discarded: v0.4.1's run carries +nine Node 20 deprecation warnings. + +**Remediation:** `ci(verify-release): check every asset, the notes, Pages +and crates.io`. One FAIL per job with a hidden failure, and warnings are +listed as notes. Mutation: an injected error in a second job adds a FAIL. + +## T16 — LOW — Nothing linted scripts/ + +*[tests] — 1 of 5.* + +shellcheck ran only by hand (V43). + +**Remediation:** `ci: lint shell scripts with shellcheck`. A Shellcheck job +lints `scripts/*.sh` as bash, and CI Success depends on it. + +## T17 — LOW — Unexercised: a run whose jobs come from different attempts + +*[tests] — 1 of 5.* + +Every recorded run was either a single attempt or "Re-run all jobs" (v0.2.0's +four attempts, V44). After "Re-run failed jobs", the run-level log and job +listing may hold jobs from different attempts, and "log covers every job +that ran" could FAIL spuriously. + +**Remediation:** not code. The script's header names the gap, and the next +exercise tag can test it by failing or cancelling one build job and +choosing "Re-run failed jobs". + +--- + +## Declined, with reasons + +- **Flip the main ruleset now, or add the merge procedure to CONTRIBUTING** + [silent]: the ruleset is an admin decision, recorded with its fix in + TD-2026-09-06, whose trigger (a Dependabot wave) arrives weekly and + visibly. +- **Give TD-2026-07-02 a date or session bound** [silent]: the repository + records no session plan for it, so a bound is the maintainer's call. +- **Re-pin rust-cache in this PR** [silent]: this PR files TD-2026-09-05; + its trigger places the re-pin in the next github-actions PR. +- **Check Cargo.toml's version at the run's commit independently** [tests]: + the validate job prints the comparison, and the verifier checks that line. +- **Restore v0.4.1's Pages deployment now**: the exercise deployed docs + identical to v0.4.1's (same crate version, no source change since), so + restoring changes nothing a reader sees. From 4d5699311b3b63b2101aace349665295d76515d7 Mon Sep 17 00:00:00 2001 From: mlevkov Date: Wed, 23 Sep 2026 18:10:31 -0700 Subject: [PATCH 16/26] ci(verify-release): fail closed where round 2 found new silent paths Round 2 found the fail-open class recurring in round 1's own new fetches. An artifact download that failed for any reason, not only expiry, became a SKIP of the digest comparison; now only an artifact the listing marks expired may skip, other failures and an artifact without the asset FAIL, and a missing API digest is replaced by hashing the release asset (digests now come from the REST API). A failed lookup of the live Pages deployment, the tag's commit, the Latest release or the site URL FAILs with gh's reason instead of skipping with a false one; "live" is the newest deployment that succeeded, and the URL comes from its status. A binary without its exec bit FAILs, a target without an architecture pattern FAILs, missing GitHub notes FAIL, and no-unzip skips name both checks. An empty filtered log FAILs rather than letting "no step failed silently" pass over nothing, and the negated grep for the changelog fallback is gone: the positive line already rules it out. The host binary is exec'd in perl's list form: with one argument, `exec @ARGV` re-parsed the path (a space in TMPDIR gave exit 127). curl calls are time-bounded, served-docs fetches retry for the CDN, and a failed run keeps its logs but not the 100 MB extracted binaries. Checked: v0.4.1 38/1/1, v0.4.1-ci.1 15/1/4, v0.3.0 35/1/3, and mutations for each new FAIL path. --- scripts/verify-release.sh | 234 +++++++++++++++++++++++++------------- 1 file changed, 154 insertions(+), 80 deletions(-) diff --git a/scripts/verify-release.sh b/scripts/verify-release.sh index 17cd3e0..9e88f93 100755 --- a/scripts/verify-release.sh +++ b/scripts/verify-release.sh @@ -14,8 +14,9 @@ # `git describe --tags --abbrev=0 ^`, over the tags on origin. # # .github/workflows/verify-release.yml runs this after every green Release -# run. Until TD-2026-09-04 is resolved, a stable release fails "no step -# failed silently": its crates.io publish step fails under continue-on-error. +# run, and on demand. Until TD-2026-09-04 is resolved every stable release +# gets one FAIL, for its crates.io publish step, which fails under +# continue-on-error. # # A tag with a hyphen (v0.4.1-ci.1) is a pre-release: the publish job must be # skipped and the release must not become Latest. The docs job still deploys @@ -26,9 +27,11 @@ # used when present. Exits 0 when every check passes (skips allowed), 1 when # any fails, 2 when it cannot start. The work directory is kept on failure. # -# Not yet exercised: a run whose jobs come from different attempts ("Re-run -# failed jobs"). Binaries before v0.4.0 are not executed: they accept a zero -# OPERATION_TIMEOUT_SECS and go on to open network listeners. +# Not yet exercised: a run whose jobs come from different attempts, as +# "Re-run failed jobs" or re-running one job (CONTRIBUTING's Pages restore) +# produces; a NOTE flags a run past its first attempt. Binaries before v0.4.0 +# are not executed: they accept a zero OPERATION_TIMEOUT_SECS and go on to +# open network listeners. set -uo pipefail @@ -73,7 +76,10 @@ cleanup() { if [ "$failures" -eq 0 ]; then rm -rf "$WORK" else - echo "work directory kept for inspection: $WORK" + # Logs and archives stay for inspection; the extracted binaries (over + # 100 MB each for Linux) do not. + rm -rf "$WORK"/assets/*/x "$WORK/run" + echo "work directory kept for inspection: $WORK ($(du -sh "$WORK" 2>/dev/null | cut -f1))" fi } trap cleanup EXIT @@ -90,16 +96,20 @@ check() { } # Checks read files, never a pipe: under pipefail, `producer | grep -q` fails # when grep exits early and SIGPIPEs the producer, which turns a negated -# check into a vacuous pass. For the same reason a negated check demands a -# non-empty file (grep exits 2 on an unreadable one), so a failed fetch -# cannot pass as "nothing found". +# check into a vacuous pass. And no check passes on the absence of text: an +# empty or failed fetch would pass it too, and so would a reworded message. has_line() { grep -qxF -- "$1" "$2"; } -lacks_regex() { - [ -s "$2" ] || return 1 - grep -qiE -- "$1" "$2" - [ $? -eq 1 ] -} first_line() { head -n 1 "$1" 2>/dev/null; } +# fetch_has : fetch until the page contains text, three +# tries 10 seconds apart, since a fresh Pages deploy can lag behind the CDN. +fetch_has() { + local try + for try in 1 2 3; do + curl -fsSL --max-time 30 "$1" >"$3" 2>>"$WORK/fetch.err" && grep -qF -- "$2" "$3" && return 0 + [ "$try" -lt 3 ] && sleep 10 + done + return 1 +} is_prefix_of() { case $2 in "$1"*) [ -n "$1" ] ;; *) false ;; esac; } # shellcheck disable=SC2016 # perl's variables, not the shell's sha256() { "$PERL" -MDigest::SHA -e 'print Digest::SHA->new(256)->addfile($ARGV[0])->hexdigest' "$1"; } @@ -159,17 +169,21 @@ echo "repo $REPO, tag $TAG ($([ "$PRERELEASE" = true ] && echo pre-release || ec echo "run $RID, attempt $ATTEMPT: $URL" echo "changelog base: $PREV" [ "$nruns" -gt 1 ] && note "$nruns release.yml runs exist for $TAG; verifying the newest" +[ "$ATTEMPT" -gt 1 ] && note "attempt $ATTEMPT: jobs an earlier attempt ran, and this one did not re-run, may be missing from its log and job list (not yet exercised)" [ -s "$WORK/local-only.txt" ] && note "ignored tags that exist only in this clone: $(tr '\n' ' ' <"$WORK/local-only.txt")" echo -git ls-remote --tags origin "refs/tags/$TAG" "refs/tags/$TAG^{}" >"$WORK/tag.raw" 2>/dev/null -tag_commit=$(awk -v ref="refs/tags/$TAG" ' - $2 == ref "^{}" { peeled = $1 } $2 == ref { plain = $1 } - END { print (peeled != "" ? peeled : plain) }' "$WORK/tag.raw") -if [ -z "$tag_commit" ]; then +# Only a tag missing from origin's (fail-closed) listing may skip; for one +# that is there, the lookup of its commit must succeed. +if ! grep -qxF -- "$TAG" "$WORK/remote-tags.txt"; then skip "tag $TAG names the commit the run built (the tag is no longer on origin)" -else +elif git ls-remote --tags origin "refs/tags/$TAG" "refs/tags/$TAG^{}" >"$WORK/tag.raw" 2>"$WORK/tag.err"; then + tag_commit=$(awk -v ref="refs/tags/$TAG" ' + $2 == ref "^{}" { peeled = $1 } $2 == ref { plain = $1 } + END { print (peeled != "" ? peeled : plain) }' "$WORK/tag.raw") check "tag $TAG names the commit the run built (${SHA:0:7})" test "$tag_commit" = "$SHA" +else + fail "read tag $TAG's commit on origin ($(first_line "$WORK/tag.err"))" fi # -------------------------------------------------------------------------- @@ -209,7 +223,9 @@ fi # output only, as "\t", without timestamps or colors. gh writes # a log's escape sequences in caret notation ("^[[1m") when not on a # terminal, so strip both that and the raw ESC form. -if gh run view "$RID" --log >"$WORK/raw.log" 2>"$WORK/raw.err" && [ -s "$WORK/raw.log" ]; then +gh run view "$RID" --log >"$WORK/raw.log" 2>"$WORK/raw.err" || : >"$WORK/raw.log" +: >"$WORK/output.log" +if [ -s "$WORK/raw.log" ]; then # shellcheck disable=SC2016 # perl's variables, not the shell's "$PERL" -ne ' s/(?:\e|\^\[)\[[0-9;?]*[A-Za-z]//g; @@ -225,7 +241,14 @@ if gh run view "$RID" --log >"$WORK/raw.log" 2>"$WORK/raw.err" && [ -s "$WORK/ra if ($skip && $msg =~ /^##\[endgroup\]/) { $skip = 0; next } print "$job\t$msg\n" unless $skip; ' "$WORK/raw.log" >"$WORK/output.log" - +fi +if [ ! -s "$WORK/raw.log" ]; then + fail "fetch the run log ($(first_line "$WORK/raw.err"))" + skip "step-output checks (no log)" +elif [ ! -s "$WORK/output.log" ]; then + fail "parse the run log (no step output left after filtering; has its format changed?)" + skip "step-output checks (no step output)" +else if [ "$jobs_ok" = true ]; then logged=$(cut -f1 "$WORK/output.log" | sort -u | wc -l | tr -d ' ') ran=$(awk -F'\t' '$2 != "skipped"' "$WORK/jobs.tsv" | wc -l | tr -d ' ') @@ -233,8 +256,11 @@ if gh run view "$RID" --log >"$WORK/raw.log" 2>"$WORK/raw.err" && [ -s "$WORK/ra fi check "validate parsed the tag as $VERSION" has_line "Validate Release${TAB}Version: $VERSION" "$WORK/output.log" check "validate matched Cargo.toml version $BASE" has_line "Validate Release${TAB}Version verified: $BASE" "$WORK/output.log" - check "changelog starts at $PREV" has_line "Create Release${TAB}Generating changelog since $PREV" "$WORK/output.log" - check "changelog did not fall back to the last 20 commits" lacks_regex 'No previous tag found' "$WORK/output.log" + # This one line also rules out release.yml's fallback to the last 20 + # commits, which prints something else instead. A negated grep for the + # fallback's message would pass on any rewording of it. + check "changelog starts at $PREV (not the last-20-commits fallback)" \ + has_line "Create Release${TAB}Generating changelog since $PREV" "$WORK/output.log" if [ "$PRERELEASE" = false ]; then case $PREV in *-*) fail "changelog base $PREV is a pre-release tag (a leftover exercise tag truncates the notes)" ;; @@ -261,20 +287,20 @@ if gh run view "$RID" --log >"$WORK/raw.log" 2>"$WORK/raw.err" && [ -s "$WORK/ra note "the run logged warnings (not failures):" sed 's/^/ /' "$WORK/warnings.txt" fi -else - fail "fetch the run log ($(first_line "$WORK/raw.err"))" - skip "step-output checks (no log)" fi # -------------------------------------------------------------------------- # Artifacts # -------------------------------------------------------------------------- -if gh api "repos/$REPO/actions/runs/$RID/artifacts" --jq '.artifacts[].name' \ - >"$WORK/artifacts.txt" 2>"$WORK/artifacts.err"; then - missing=$(while IFS= read -r t; do grep -qxF -- "$CRATE-$t" "$WORK/artifacts.txt" || printf '%s ' "$t"; done <"$WORK/targets.txt") +# artifact_field : a field of the named artifact's line. +artifact_field() { awk -F'\t' -v n="$1" -v f="$2" '$1 == n { print $f; exit }' "$WORK/artifacts.tsv"; } +if gh api --paginate "repos/$REPO/actions/runs/$RID/artifacts" \ + --jq '.artifacts[] | "\(.name)\t\(.expired)"' >"$WORK/artifacts.tsv" 2>"$WORK/artifacts.err"; then + missing=$(while IFS= read -r t; do [ -n "$(artifact_field "$CRATE-$t" 1)" ] || printf '%s ' "$t"; done <"$WORK/targets.txt") check "a build artifact for every target${missing:+ (missing: $missing)}" test -z "$missing" - check "Pages artifact uploaded" grep -qxF github-pages "$WORK/artifacts.txt" + check "Pages artifact uploaded" test -n "$(artifact_field github-pages 1)" else + : >"$WORK/artifacts.tsv" fail "list the run's artifacts ($(first_line "$WORK/artifacts.err"))" fi @@ -290,8 +316,9 @@ if gh release view "$TAG" --json isPrerelease,isDraft,publishedAt \ pass "release $TAG exists" check "release is published, not a draft" test "$is_draft" = false check "release pre-release flag is $PRERELEASE" test "$is_pre" = "$PRERELEASE" - gh release view "$TAG" --json assets --jq '.assets[] | "\(.name)\t\(.digest // "")"' >"$WORK/assets.tsv" || - fail "list the release's assets" + # The REST API carries each asset's digest whatever gh version runs this. + gh api "repos/$REPO/releases/tags/$TAG" --jq '.assets[] | "\(.name)\t\(.digest // "")"' \ + >"$WORK/assets.tsv" 2>"$WORK/assets.err" || fail "list the release's assets ($(first_line "$WORK/assets.err"))" gh release view "$TAG" --json body --jq .body >"$WORK/body.md" || fail "read the release notes" # release.yml writes one "- ()" line per commit since the @@ -306,18 +333,21 @@ if gh release view "$TAG" --json isPrerelease,isDraft,publishedAt \ gh_base=$("$PERL" -ne 'print "$1\n" and exit if m{\*\*Full Changelog\*\*: \S*/compare/(\S+?)\.\.\.\S+}' "$WORK/body.md") if [ -n "$gh_base" ]; then check "GitHub's notes compare from $PREV too ($gh_base)" test "$gh_base" = "$PREV" + elif grep -qF -- "**Full Changelog**: " "$WORK/body.md"; then + skip "GitHub's notes compare from $PREV (their link has no base: a first release)" else - skip "GitHub's notes compare from $PREV (no compare link)" + fail "GitHub's generated notes are in the release (no \"Full Changelog\" link)" fi else fail "release $TAG exists ($(first_line "$WORK/release.err"))" fi -latest=$(gh release list --limit 50 --json tagName,isLatest --jq '.[] | select(.isLatest) | .tagName') latest_is_other_stable() { [ -n "$latest" ] && [ "$latest" != "$TAG" ] && case $latest in *-*) false ;; *) true ;; esac } -if [ "$PRERELEASE" = true ]; then +if ! latest=$(gh release list --limit 50 --json tagName,isLatest --jq '.[] | select(.isLatest) | .tagName' 2>"$WORK/latest.err"); then + fail "find the Latest release ($(first_line "$WORK/latest.err"))" +elif [ "$PRERELEASE" = true ]; then check "Latest is still a stable release (${latest:-none})" latest_is_other_stable elif [ "$latest" = "$TAG" ]; then pass "release is marked Latest" @@ -356,36 +386,60 @@ if [ "$release_ok" = true ]; then fi # shellcheck disable=SC2016 # $1 and $2 are awk fields digest=$(awk -F'\t' -v n="$name" '$1 == n { print $2 }' "$WORK/assets.tsv") - file="" dir=$WORK/assets/$target - if gh run download "$RID" -n "$CRATE-$target" -D "$dir" >/dev/null 2>"$WORK/art.err" && [ -f "$dir/$name" ]; then - file=$dir/$name - if [ -n "$digest" ]; then - check "asset $name is its build job's artifact" test "sha256:$(sha256 "$file")" = "$digest" - else - skip "asset $name is its build job's artifact (the API reports no digest)" + file=$dir/$name + # Only an expired artifact excuses comparing nothing: the release asset + # is then inspected on its own. Any other download failure, or an + # artifact without the asset in it, is a FAIL, and an artifact missing + # from the listing has already failed above. + expired=$(artifact_field "$CRATE-$target" 2) + if [ "$expired" = false ]; then + if ! gh run download "$RID" -n "$CRATE-$target" -D "$dir" >/dev/null 2>"$WORK/art.err" /dev/null 2>"$WORK/dl.err" /dev/null 2>"$WORK/dl.err"; then - file=$dir/$name - skip "asset $name is its build job's artifact (no artifact: $(first_line "$WORK/art.err"))" + check "asset $name is its build job's artifact" test "sha256:$(sha256 "$file")" = "$digest" else - fail "download $name ($(first_line "$WORK/dl.err"))" - continue + if ! gh release download "$TAG" --pattern "$name" --dir "$dir" >/dev/null 2>"$WORK/dl.err" /dev/null 2>&1; then + skip "contents of $name (no unzip)" + skip "$bin in $name is built for $target (no unzip)" + continue + fi if [ "$ext" = zip ]; then - if ! command -v unzip >/dev/null 2>&1; then - skip "contents of $name (no unzip)" - continue - fi unzip -Z1 "$file" >"$dir/list.txt" 2>/dev/null && unzip -q -o "$file" "$bin" -d "$dir/x" 2>/dev/null else tar -tzf "$file" >"$dir/list.txt" 2>/dev/null && tar -xzf "$file" -C "$dir/x" "$bin" 2>/dev/null fi check "$name holds $bin, README.md and a LICENSE" has_release_files "$dir/list.txt" "$bin" - pattern=$(arch_pattern "$target") - if [ -z "$pattern" ] || ! command -v file >/dev/null 2>&1; then - skip "$bin in $name is built for $target (no file command or pattern)" + if [ -z "$pattern" ]; then + fail "$bin in $name is built for $target (no architecture pattern for $target: add one to arch_pattern)" + elif ! command -v file >/dev/null 2>&1; then + skip "$bin in $name is built for $target (no file command)" else file -b "$dir/x/$bin" >"$dir/file.txt" 2>&1 check "$bin in $name is built for $target" grep -qE -- "$pattern" "$dir/file.txt" @@ -399,28 +453,43 @@ fi # Pages # -------------------------------------------------------------------------- # Reachability proves nothing: the site already served the last release's -# docs. The deployment record for this tag says what the run deployed, and a -# later deploy turns it inactive, so look for success anywhere in its -# history. The served pages are checked only while this deployment is live. -dep=$(gh api "repos/$REPO/deployments?environment=github-pages&ref=$TAG&per_page=1" \ - --jq '.[] | "\(.id) \(.sha)"') -read -r dep_id dep_sha <<<"$dep" -if [ -z "${dep_id:-}" ]; then +# docs. The deployment record for this tag says what the run deployed; look +# for success anywhere in its status history. The served pages are checked +# only while this deployment is the live one, the newest that succeeded. +deployment_succeeded() { # deployment_succeeded : its statuses include success + [ "$(gh api "repos/$REPO/deployments/$1/statuses" --jq '[.[].state] | index("success") != null' \ + 2>>"$WORK/dep.err" "$WORK/dep.err"); then + fail "look up the Pages deployment for $TAG ($(first_line "$WORK/dep.err"))" +elif [ -z "$dep" ]; then fail "Pages deployment for $TAG (none recorded)" else + read -r dep_id dep_sha <<<"$dep" check "Pages deployment for $TAG built ${SHA:0:7}" test "$dep_sha" = "$SHA" - check "Pages deployment for $TAG succeeded" \ - test "$(gh api "repos/$REPO/deployments/$dep_id/statuses" --jq '[.[].state] | index("success") != null')" = true - live=$(gh api "repos/$REPO/deployments?environment=github-pages&per_page=1" --jq '.[].id') - if [ "$live" != "$dep_id" ]; then - skip "served docs (a later deployment replaced this one)" + check "Pages deployment for $TAG succeeded" deployment_succeeded "$dep_id" + live="" + if gh api "repos/$REPO/deployments?environment=github-pages&per_page=10" --jq '.[].id' \ + >"$WORK/deployments.txt" 2>"$WORK/deployments.err"; then + while IFS= read -r id; do + if deployment_succeeded "$id"; then live=$id && break; fi + done <"$WORK/deployments.txt" + fi + if [ -z "$live" ]; then + fail "find the live Pages deployment ($(first_line "$WORK/deployments.err"))" + elif [ "$live" != "$dep_id" ]; then + skip "served docs (a later deployment, $live, is live)" + elif ! site=$(gh api "repos/$REPO/deployments/$dep_id/statuses" \ + --jq 'map(select(.state == "success")) | .[0].environment_url // empty' 2>"$WORK/site.err") || + [ -z "$site" ]; then + fail "find the deployment's site URL ($(first_line "$WORK/site.err"))" else - site=$(gh api "repos/$REPO/pages" --jq .html_url) # A query string sidesteps the CDN's cached copy of the old site. - curl -fsSL "${site%/}/?verify=$RID" >"$WORK/site-index.html" 2>"$WORK/site.err" - check "Pages root redirects to $CRATE/index.html" grep -qF "url=$CRATE/index.html" "$WORK/site-index.html" - curl -fsSL "${site%/}/$CRATE/index.html?verify=$RID" >"$WORK/site-crate.html" 2>>"$WORK/site.err" - check "Pages serves the $BASE docs" grep -qF "\"version\">$BASE<" "$WORK/site-crate.html" + check "Pages root redirects to $CRATE/index.html" \ + fetch_has "${site%/}/?verify=$RID" "url=$CRATE/index.html" "$WORK/site-index.html" + check "Pages serves the $BASE docs" \ + fetch_has "${site%/}/$CRATE/index.html?verify=$RID" "\"version\">$BASE<" "$WORK/site-crate.html" fi fi @@ -430,7 +499,7 @@ fi if [ "$PRERELEASE" = true ]; then skip "crates.io (pre-release)" else - status=$(curl -s -o /dev/null -w '%{http_code}' -A "verify-release ($REPO)" \ + status=$(curl -s --max-time 30 -o /dev/null -w '%{http_code}' -A "verify-release ($REPO)" \ "https://crates.io/api/v1/crates/$CRATE/$BASE") if [ "$PUBLISH" = false ]; then check "crates.io does not have $CRATE $BASE (publish = false; HTTP $status)" test "$status" = 404 @@ -454,17 +523,23 @@ if [ -z "$host" ]; then skip "host binary (no build target for $(uname -s)-$(uname -m))" elif version_lt "$BASE" 0.4.0; then skip "host binary ($BASE predates the zero-timeout check it relies on)" -elif [ ! -x "$bin" ]; then +elif [ ! -e "$bin" ]; then skip "host binary (no $host asset was extracted above)" +elif [ ! -x "$bin" ]; then + # shellcheck disable=SC2016 # perl's variables, not the shell's + fail "$host binary is executable (its archive has mode $("$PERL" -e 'printf "%o", (stat $ARGV[0])[2] & 07777' "$bin"))" else # Config::validate rejects a zero OPERATION_TIMEOUT_SECS, so the binary # logs its version and exits with EX_CONFIG (78) before any network I/O. - # Run it from an empty directory (the app loads a .env from its working - # directory) with an empty environment (a verifier should not hand a - # freshly downloaded binary the caller's tokens), and give it 30 seconds. + # Run it from an empty directory under the temp dir (the app loads a .env + # from its working directory or any parent) with an empty environment (a + # verifier should not hand a freshly downloaded binary the caller's + # tokens), and give it 30 seconds. `exec { $ARGV[0] } @ARGV` forces perl's + # list form, so the path is never re-parsed by a shell. mkdir -p "$WORK/run" + # shellcheck disable=SC2016 # perl's variables, not the shell's (cd "$WORK/run" && env -i OPERATION_TIMEOUT_SECS=0 RUST_LOG=info \ - "$PERL" -e 'alarm 30; exec @ARGV or exit 127' "$bin") >"$WORK/bin.raw" 2>&1 + "$PERL" -e 'alarm 30; exec { $ARGV[0] } @ARGV or exit 127' "$bin") >"$WORK/bin.raw" 2>&1 code=$? "$PERL" -pe 's/\e\[[0-9;]*m//g' "$WORK/bin.raw" >"$WORK/bin.log" if [ "$code" -eq 142 ]; then @@ -482,7 +557,6 @@ if [ "$PRERELEASE" = true ] && [ "$release_ok" = true ]; then echo "If $TAG only exercised the workflow, delete it before the next release:" echo " gh release delete $TAG --cleanup-tag --yes" echo "Its docs job deployed ${SHA:0:7}'s docs to Pages, which deleting the tag does" - echo "not undo; re-run the Deploy Documentation job of the latest stable release's" - echo "run to restore them." + echo "not undo: CONTRIBUTING.md (Releasing) says how to restore them." fi [ "$failures" -eq 0 ] From ceab67121bbf351f335dae21b52f89c91562dca9 Mon Sep 17 00:00:00 2001 From: mlevkov Date: Wed, 23 Sep 2026 18:10:31 -0700 Subject: [PATCH 17/26] ci: let Verify Release run by hand and keep the token off disk workflow_run uses the default branch's file, so the workflow cannot run before merge and its first run would be the next release, red by design until TD-2026-09-04 is resolved. A workflow_dispatch with a tag input allows a controlled first run against v0.4.1 right after merge, and the script's output now also lands in the run's summary. The job runs a downloaded binary, so checkout no longer persists the token; pages: read goes, since the site URL now comes from the deployment. --- .github/workflows/verify-release.yml | 30 ++++++++++++++++++++++------ 1 file changed, 24 insertions(+), 6 deletions(-) diff --git a/.github/workflows/verify-release.yml b/.github/workflows/verify-release.yml index ca05652..4d036f5 100644 --- a/.github/workflows/verify-release.yml +++ b/.github/workflows/verify-release.yml @@ -5,7 +5,8 @@ # runs scripts/verify-release.sh after each one: a problem shows up as a # failed run here instead of waiting for someone to remember the script. # Until TD-2026-09-04 is resolved it fails on every stable release, by -# design: the crates.io publish step fails silently there. +# design: the crates.io publish step fails silently there. It can also be +# run by hand for any tag (Actions > Verify Release > Run workflow). name: Verify Release @@ -13,29 +14,46 @@ on: workflow_run: workflows: [Release] types: [completed] + workflow_dispatch: + inputs: + tag: + description: "Tag whose Release run to verify, e.g. v0.4.1" + required: true + type: string permissions: actions: read contents: read deployments: read - pages: read jobs: verify: name: Verify release run - # Green runs from a tag push only: a failed run is already visible. - if: github.event.workflow_run.event == 'push' && github.event.workflow_run.conclusion == 'success' + # By hand, or after a green run from a tag push: a failed run is already + # visible. + if: >- + github.event_name == 'workflow_dispatch' || + (github.event.workflow_run.event == 'push' && github.event.workflow_run.conclusion == 'success') runs-on: ubuntu-latest timeout-minutes: 20 steps: - uses: actions/checkout@v7 with: fetch-depth: 0 + # The job runs a downloaded binary; keep the token off the disk + # (git fetch and ls-remote work anonymously on this public repo). + persist-credentials: false - name: Verify the release run env: GH_TOKEN: ${{ github.token }} # Through the environment, not interpolated into the script, so a # crafted tag name cannot inject shell. - TAG: ${{ github.event.workflow_run.head_branch }} - run: scripts/verify-release.sh "$TAG" + TAG: ${{ inputs.tag || github.event.workflow_run.head_branch }} + run: | + set +e + scripts/verify-release.sh "$TAG" 2>&1 | tee verify.out + status=${PIPESTATUS[0]} + # The PASS/FAIL lines on the run's summary page, not only its log. + { echo '```'; cat verify.out; echo '```'; } >>"$GITHUB_STEP_SUMMARY" + exit "$status" From deaa2457bb56689209bff07aea6138e5073a3b7c Mon Sep 17 00:00:00 2001 From: mlevkov Date: Thu, 24 Sep 2026 12:50:13 -0700 Subject: [PATCH 18/26] ci(verify-release): verify the run and attempt that triggered it Verify Release checked the tag's newest run at its latest attempt, not the one whose completion triggered it, and the script called a run with re-run jobs unexercised. Measured on v0.2.0's run 28759754600, whose attempts 2 to 4 each re-ran only Deploy Documentation: each attempt lists all 9 jobs, the 8 it carried over under new IDs with their original started_at, but its log archive holds only the jobs it ran (attempt 2's is empty). gh 2.75 and later fetch every other job's log from the per-job logs API, which serves a carried-over job's original log, byte-identical across the four attempts. gh 2.74.2 prints only Deploy Documentation for attempt 4, and "no step failed silently" then passes without the publish step's ##[error]. So the script now requires gh 2.75 and exits 2 on an older one (Ubuntu 24.04 packages 2.45), and stitching logs from earlier attempts is not needed. The workflow passes the triggering run and attempt as VERIFY_RUN_ID and VERIFY_RUN_ATTEMPT; the script checks them against the tag's runs and the run's attempts and exits 2 on any other. Jobs come from that attempt, the log from gh run view --attempt, and a NOTE names the jobs an attempt re-ran. "Log covers every job that ran" names the jobs it misses instead of comparing counts, and "no step failed silently" says so when it covered only the jobs with a log. Checked on macOS: v0.2.0 attempt 4 32/4/3 (the publish FAIL, and three for notes replaced after publication), attempts 3 and 1 31/6/3 (their Deploy Documentation failures), attempt 2 26/5/4 (log not found: its re-run job never reached a runner), and v0.2.0's older run by id 20/15/8 (tag mismatch); v0.4.1 38/1/1, v0.3.0 35/1/3 and v0.4.1-ci.1 15/1/4, as before. v0.2.0 and v0.4.1 match in an Ubuntu 24.04 container. A gh stand-in that drops two carried-over logs FAILs naming both, and a foreign run id, attempt 5 and a non-numeric attempt each exit 2. --- .github/workflows/verify-release.yml | 13 ++- scripts/verify-release.sh | 120 +++++++++++++++++++-------- 2 files changed, 95 insertions(+), 38 deletions(-) diff --git a/.github/workflows/verify-release.yml b/.github/workflows/verify-release.yml index 4d036f5..96ed4da 100644 --- a/.github/workflows/verify-release.yml +++ b/.github/workflows/verify-release.yml @@ -4,9 +4,11 @@ # continue-on-error, and nothing else inspects what a run produced, so this # runs scripts/verify-release.sh after each one: a problem shows up as a # failed run here instead of waiting for someone to remember the script. -# Until TD-2026-09-04 is resolved it fails on every stable release, by -# design: the crates.io publish step fails silently there. It can also be -# run by hand for any tag (Actions > Verify Release > Run workflow). +# Re-running jobs of a Release run makes a new attempt of it, which runs +# this again. Until TD-2026-09-04 is resolved it fails on every stable +# release, by design: the crates.io publish step fails silently there. It +# can also be run by hand for any tag (Actions > Verify Release > Run +# workflow). name: Verify Release @@ -50,6 +52,11 @@ jobs: # Through the environment, not interpolated into the script, so a # crafted tag name cannot inject shell. TAG: ${{ inputs.tag || github.event.workflow_run.head_branch }} + # The run and attempt that triggered this one, not the tag's + # newest; both are empty when run by hand, which verifies the + # newest run at its latest attempt. + VERIFY_RUN_ID: ${{ github.event.workflow_run.id }} + VERIFY_RUN_ATTEMPT: ${{ github.event.workflow_run.run_attempt }} run: | set +e scripts/verify-release.sh "$TAG" 2>&1 | tee verify.out diff --git a/scripts/verify-release.sh b/scripts/verify-release.sh index 9e88f93..5d8d267 100755 --- a/scripts/verify-release.sh +++ b/scripts/verify-release.sh @@ -3,35 +3,39 @@ # # Usage: scripts/verify-release.sh [] # -# Checks the newest release.yml run for against the workflow file and -# Cargo.toml at the commit it built: that the tag names that commit; the job -# conclusions; what the steps printed; the GitHub Release, its notes and its -# assets; that every asset is byte-identical to its build job's artifact and -# holds a binary for its target; the Pages deployment and, while it is the -# live one, the served docs; that crates.io agrees with Cargo.toml's -# `publish`; and that the host platform's binary runs. is where -# the changelog should start. By default it is what release.yml computes, -# `git describe --tags --abbrev=0 ^`, over the tags on origin. +# Checks the newest release.yml run for , at its latest attempt, against +# the workflow file and Cargo.toml at the commit it built: that the tag names +# that commit; the job conclusions; what the steps printed; the GitHub +# Release, its notes and its assets; that every asset is byte-identical to its +# build job's artifact and holds a binary for its target; the Pages +# deployment and, while it is the live one, the served docs; that crates.io +# agrees with Cargo.toml's `publish`; and that the host platform's binary +# runs. is where the changelog should start. By default it is +# what release.yml computes, `git describe --tags --abbrev=0 ^`, over the tags on origin. VERIFY_RUN_ID picks another of 's +# runs instead, and VERIFY_RUN_ATTEMPT an earlier attempt. # # .github/workflows/verify-release.yml runs this after every green Release -# run, and on demand. Until TD-2026-09-04 is resolved every stable release -# gets one FAIL, for its crates.io publish step, which fails under -# continue-on-error. +# run, for the run and attempt that triggered it, and on demand. Until +# TD-2026-09-04 is resolved every stable release gets one FAIL, for its +# crates.io publish step, which fails under continue-on-error. # # A tag with a hyphen (v0.4.1-ci.1) is a pre-release: the publish job must be # skipped and the release must not become Latest. The docs job still deploys # the tagged commit's docs to Pages, and deleting the tag does not undo that. # CONTRIBUTING.md ("Releasing") describes exercising release.yml this way. # -# Requires gh (authenticated), git, perl, tar and curl; unzip and file are -# used when present. Exits 0 when every check passes (skips allowed), 1 when -# any fails, 2 when it cannot start. The work directory is kept on failure. +# Re-running failed jobs, or one job (CONTRIBUTING's Pages restore), makes a +# new attempt of the run that lists every job but runs only the chosen ones. +# The others carry over with their conclusions and their original logs, which +# gh 2.75 or later fetches for them (checked on v0.2.0's run 28759754600, +# attempts 2 to 4). A NOTE names the jobs an attempt re-ran. # -# Not yet exercised: a run whose jobs come from different attempts, as -# "Re-run failed jobs" or re-running one job (CONTRIBUTING's Pages restore) -# produces; a NOTE flags a run past its first attempt. Binaries before v0.4.0 -# are not executed: they accept a zero OPERATION_TIMEOUT_SECS and go on to -# open network listeners. +# Requires gh 2.75 or later (authenticated), git, perl, tar and curl; unzip +# and file are used when present. Exits 0 when every check passes (skips +# allowed), 1 when any fails, 2 when it cannot start. The work directory is +# kept on failure. Binaries before v0.4.0 are not executed: they accept a +# zero OPERATION_TIMEOUT_SECS and go on to open network listeners. set -uo pipefail @@ -88,6 +92,8 @@ pass() { printf 'PASS %s\n' "$1"; passes=$((passes + 1)); } fail() { printf 'FAIL %s\n' "$1"; failures=$((failures + 1)); } skip() { printf 'SKIP %s\n' "$1"; skips=$((skips + 1)); } note() { printf 'NOTE %s\n' "$1"; } +# join_lines: stdin's lines as one "a, b, c" line (job names hold spaces). +join_lines() { awk 'NR > 1 { printf ", " } { printf "%s", $0 } END { if (NR) print "" }'; } # check [args...]: PASS when the command succeeds. check() { local desc=$1 @@ -121,17 +127,44 @@ version_lt() { exit 1' "$1" "$2" } +# gh 2.75 is the first to fetch the logs of jobs a re-run carries over. +GH_VERSION=$(gh --version 2>/dev/null | awk 'NR == 1 { print $3 }') +if version_lt "${GH_VERSION:-0}" 2.75.0; then + echo "gh ${GH_VERSION:-of unknown version} is too old: 2.75 or later is required" >&2 + exit 2 +fi + # -------------------------------------------------------------------------- # The run, and the commit it built # -------------------------------------------------------------------------- runs=$(gh run list --workflow release.yml --branch "$TAG" --limit 20 \ - --json databaseId,headSha,status,attempt,url \ - -q '.[] | "\(.databaseId) \(.headSha) \(.status) \(.attempt) \(.url)"') || + --json databaseId,headSha,attempt,url \ + -q '.[] | "\(.databaseId) \(.headSha) \(.attempt) \(.url)"') || { echo "listing release.yml runs for $TAG failed" >&2; exit 2; } [ -n "$runs" ] || { echo "no release.yml run found for $TAG" >&2; exit 2; } -read -r RID SHA STATUS ATTEMPT URL <<<"${runs%%$'\n'*}" nruns=$(printf '%s\n' "$runs" | wc -l | tr -d ' ') -[ "$STATUS" = completed ] || { echo "run $RID is $STATUS; verify it once it completes" >&2; exit 2; } +if [ -n "${VERIFY_RUN_ID:-}" ]; then + # shellcheck disable=SC2016 # $1 is an awk field + run=$(printf '%s\n' "$runs" | awk -v id="$VERIFY_RUN_ID" '$1 == id') + [ -n "$run" ] || { echo "run $VERIFY_RUN_ID is not one of $TAG's release.yml runs" >&2; exit 2; } +else + run=${runs%%$'\n'*} +fi +read -r RID SHA LATEST URL <<<"$run" +ATTEMPT=${VERIFY_RUN_ATTEMPT:-$LATEST} +case $ATTEMPT in + *[!0-9]*) echo "VERIFY_RUN_ATTEMPT is not a number: $ATTEMPT" >&2; exit 2 ;; +esac +if [ "$ATTEMPT" -lt 1 ] || [ "$ATTEMPT" -gt "$LATEST" ]; then + echo "run $RID has attempts 1 to $LATEST, not $ATTEMPT" >&2 + exit 2 +fi +# The attempt's own status: a newer attempt may be running while this one +# is complete. +attempt_info=$(gh api "repos/$REPO/actions/runs/$RID/attempts/$ATTEMPT" --jq '"\(.status) \(.run_started_at)"') || + { echo "reading attempt $ATTEMPT of run $RID failed" >&2; exit 2; } +read -r STATUS STARTED <<<"$attempt_info" +[ "$STATUS" = completed ] || { echo "run $RID attempt $ATTEMPT is $STATUS; verify it once it completes" >&2; exit 2; } git fetch --quiet --tags origin || { echo "fetching from origin failed" >&2; exit 2; } git cat-file -e "$SHA^{commit}" 2>/dev/null || @@ -166,11 +199,11 @@ fi [ -n "$PREV" ] || { echo "cannot determine the previous tag; pass it as the second argument" >&2; exit 2; } echo "repo $REPO, tag $TAG ($([ "$PRERELEASE" = true ] && echo pre-release || echo stable)), commit ${SHA:0:7}" -echo "run $RID, attempt $ATTEMPT: $URL" +echo "run $RID, attempt $ATTEMPT of $LATEST: $URL" echo "changelog base: $PREV" -[ "$nruns" -gt 1 ] && note "$nruns release.yml runs exist for $TAG; verifying the newest" -[ "$ATTEMPT" -gt 1 ] && note "attempt $ATTEMPT: jobs an earlier attempt ran, and this one did not re-run, may be missing from its log and job list (not yet exercised)" -[ -s "$WORK/local-only.txt" ] && note "ignored tags that exist only in this clone: $(tr '\n' ' ' <"$WORK/local-only.txt")" +[ "$nruns" -gt 1 ] && note "$nruns release.yml runs exist for $TAG; verifying $([ -n "${VERIFY_RUN_ID:-}" ] && echo "the one asked for" || echo "the newest")" +[ "$ATTEMPT" -lt "$LATEST" ] && note "attempt $ATTEMPT is not the latest: the release, its assets and Pages may be a later attempt's" +[ -s "$WORK/local-only.txt" ] && note "ignored tags that exist only in this clone: $(join_lines <"$WORK/local-only.txt")" echo # Only a tag missing from origin's (fail-closed) listing may skip; for one @@ -190,9 +223,18 @@ fi # Jobs # -------------------------------------------------------------------------- jobs_ok=false -if gh run view "$RID" --json jobs --jq '.jobs[] | "\(.name)\t\(.conclusion)"' \ +if gh api --paginate "repos/$REPO/actions/runs/$RID/attempts/$ATTEMPT/jobs" \ + --jq '.jobs[] | "\(.name)\t\(.conclusion)\t\(.started_at)"' \ >"$WORK/jobs.tsv" 2>"$WORK/jobs.err" && [ -s "$WORK/jobs.tsv" ]; then jobs_ok=true + # An attempt's jobs that started before it did are carried over from an + # earlier attempt (under new IDs, with their original start times). + # shellcheck disable=SC2016 # $1 and $3 are awk fields + awk -F'\t' -v s="$STARTED" '$3 >= s { print $1 }' "$WORK/jobs.tsv" >"$WORK/reran.txt" + reran=$(wc -l <"$WORK/reran.txt" | tr -d ' ') + total=$(wc -l <"$WORK/jobs.tsv" | tr -d ' ') + [ "$reran" -lt "$total" ] && + note "attempt $ATTEMPT re-ran $reran of the $total jobs ($(join_lines <"$WORK/reran.txt")); the others carry over from earlier attempts, with their conclusions and logs" awk -F'\t' '$1 ~ /^Build \(/ { t = $1; sub(/^Build \(/, "", t); sub(/\)$/, "", t); print t }' \ "$WORK/jobs.tsv" | LC_ALL=C sort >"$WORK/built.txt" check "one build job per release.yml target ($(wc -l <"$WORK/targets.txt" | tr -d ' '))" \ @@ -223,7 +265,11 @@ fi # output only, as "\t", without timestamps or colors. gh writes # a log's escape sequences in caret notation ("^[[1m") when not on a # terminal, so strip both that and the raw ESC form. -gh run view "$RID" --log >"$WORK/raw.log" 2>"$WORK/raw.err" || : >"$WORK/raw.log" +# +# An attempt's log archive holds only the jobs it ran; gh fetches each other +# job's log through the API, which serves a carried-over job's original log. +# A job with no log at all, one that never started, fails the whole fetch. +gh run view "$RID" --attempt "$ATTEMPT" --log >"$WORK/raw.log" 2>"$WORK/raw.err" || : >"$WORK/raw.log" : >"$WORK/output.log" if [ -s "$WORK/raw.log" ]; then # shellcheck disable=SC2016 # perl's variables, not the shell's @@ -249,10 +295,14 @@ elif [ ! -s "$WORK/output.log" ]; then fail "parse the run log (no step output left after filtering; has its format changed?)" skip "step-output checks (no step output)" else + unlogged= if [ "$jobs_ok" = true ]; then - logged=$(cut -f1 "$WORK/output.log" | sort -u | wc -l | tr -d ' ') - ran=$(awk -F'\t' '$2 != "skipped"' "$WORK/jobs.tsv" | wc -l | tr -d ' ') - check "log covers every job that ran ($ran)" test "$logged" -eq "$ran" + # shellcheck disable=SC2016 # $1 and $2 are awk fields + awk -F'\t' '$2 != "skipped" { print $1 }' "$WORK/jobs.tsv" | LC_ALL=C sort -u >"$WORK/ran.txt" + cut -f1 "$WORK/output.log" | LC_ALL=C sort -u >"$WORK/logged.txt" + unlogged=$(LC_ALL=C comm -23 "$WORK/ran.txt" "$WORK/logged.txt" | join_lines) + check "log covers every job that ran ($(wc -l <"$WORK/ran.txt" | tr -d ' ')${unlogged:+; missing: $unlogged})" \ + test -z "$unlogged" fi check "validate parsed the tag as $VERSION" has_line "Validate Release${TAB}Version: $VERSION" "$WORK/output.log" check "validate matched Cargo.toml version $BASE" has_line "Validate Release${TAB}Version verified: $BASE" "$WORK/output.log" @@ -279,7 +329,7 @@ else awk -F'\t' -v j="$job" '$1 == j && index($2, "##[error]") == 1 { print " " $2 }' "$WORK/output.log" done <"$WORK/error-jobs.txt" else - pass "no step failed silently" + pass "no step failed silently${unlogged:+ (in the jobs with a log)}" fi awk -F'\t' 'index($2, "##[warning]") == 1 { print substr($2, 12, 150) }' "$WORK/output.log" | sort | uniq -c >"$WORK/warnings.txt" @@ -296,7 +346,7 @@ fi artifact_field() { awk -F'\t' -v n="$1" -v f="$2" '$1 == n { print $f; exit }' "$WORK/artifacts.tsv"; } if gh api --paginate "repos/$REPO/actions/runs/$RID/artifacts" \ --jq '.artifacts[] | "\(.name)\t\(.expired)"' >"$WORK/artifacts.tsv" 2>"$WORK/artifacts.err"; then - missing=$(while IFS= read -r t; do [ -n "$(artifact_field "$CRATE-$t" 1)" ] || printf '%s ' "$t"; done <"$WORK/targets.txt") + missing=$(while IFS= read -r t; do [ -n "$(artifact_field "$CRATE-$t" 1)" ] || echo "$t"; done <"$WORK/targets.txt" | join_lines) check "a build artifact for every target${missing:+ (missing: $missing)}" test -z "$missing" check "Pages artifact uploaded" test -n "$(artifact_field github-pages 1)" else From 3d1ecf790be6fa67d856ee8acde8088cb3e9134f Mon Sep 17 00:00:00 2001 From: mlevkov Date: Thu, 24 Sep 2026 12:51:58 -0700 Subject: [PATCH 19/26] docs: say when the Pages restore expires and how it can fail CONTRIBUTING's way to undo an exercise's Pages deployment, re-running the latest stable release's Deploy Documentation job, works only within 30 days of that run (GitHub's limit), and the last two stable releases came 26 and 53 days after the one before. The section now gives the fallback, a pre-release tag on the stable release's commit whose run deploys that commit's docs, and says the re-run is a new attempt that runs Verify Release again. A re-run can also fail outright: deploy-pages refuses a run holding more than one github-pages artifact, and upload-pages-artifact keeps the first attempt's for a day. v0.2.0's run shows it. Its attempt 3 failed with 'Multiple artifacts named "github-pages" ... Artifact count is 2', and attempt 4 succeeded once the earlier ones were gone (the run lists only attempt 4's, while expired artifacts stay listed). The section says how to list and delete them, and that VERIFY_RUN_ID and VERIFY_RUN_ATTEMPT pick the run and attempt to verify by hand. --- CONTRIBUTING.md | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d7a4b61..a07a918 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -203,14 +203,30 @@ check a run by hand: scripts/verify-release.sh v0.4.2 ``` +It checks the tag's newest run at its latest attempt; `VERIFY_RUN_ID` and +`VERIFY_RUN_ATTEMPT` pick another run of the tag or an earlier attempt. + To exercise a `release.yml` change before a real release, push a pre-release tag of the current `Cargo.toml` version, such as `v0.4.1-ci.1` (the validate job compares only the part before the hyphen), and verify its run the same way. Two things outlive the exercise: - The docs job deploys the tagged commit's docs to Pages even for a - pre-release, and deleting the tag does not undo that. Re-run the Deploy - Documentation job of the latest stable release's run to restore its docs. + pre-release, and deleting the tag does not undo that. To restore the latest + stable release's docs, re-run the Deploy Documentation job of its Release + run (the new attempt runs Verify Release again), with two limits: + - GitHub re-runs jobs only within 30 days of a run. Past that, tag the + stable release's commit as a pre-release instead + (`git tag v0.4.1-docs.1 'v0.4.1^{}'`) and push it: its run deploys that + commit's docs. Delete the tag afterwards like any exercise tag. + - deploy-pages refuses a run that holds more than one `github-pages` + artifact, and the first attempt's is kept for a day: v0.2.0's first + re-run failed with `Multiple artifacts named "github-pages"`. If that + happens, delete the run's `github-pages` artifacts and re-run the job + again. `gh api repos/mlevkov/iggy_sample/actions/runs//artifacts` + lists them, and + `gh api -X DELETE repos/mlevkov/iggy_sample/actions/artifacts/` + deletes one. - The tag. Delete it with `gh release delete v0.4.1-ci.1 --cleanup-tag --yes` from your clone (this also deletes the local tag). Left behind, it becomes where `release.yml` starts the changelog it writes for the next release From f32abc0c44522ec5e0f915e38d096dfa06527627 Mon Sep 17 00:00:00 2001 From: mlevkov Date: Thu, 24 Sep 2026 12:53:50 -0700 Subject: [PATCH 20/26] ci: correct the verifier's changelog comment and mark two more sites release.yml said scripts/verify-release.sh matches both changelog echo lines exactly. Since round 2 removed the negated check for the fallback's line, it matches only the first, whose presence already rules the fallback out; the comment now says so. Two sites that something depends on said nothing. verify-release.yml's workflow_run trigger matches release.yml's `name: Release`, so a rename would stop verification without a trace; the name now says so. And TD-2026-09-05 binds to the rust-cache pins: the first pin in each of the four workflows now names the record, which lists that pointer and replaces "most CI jobs" with the five that run rust-cache (clippy, test, integration, coverage, docs). ci.yml's header names the triggers its `on:` has and includes shell-script linting. Comments only: every workflow still parses, and no step changes. --- .github/workflows/ci.yml | 7 +++++-- .github/workflows/extended-tests.yml | 2 ++ .github/workflows/pr.yml | 2 ++ .github/workflows/release.yml | 9 +++++++-- docs/tech-debt/TD-2026-09-05.md | 7 +++++-- 5 files changed, 21 insertions(+), 6 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fa6ca49..050286d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,7 +1,8 @@ # Continuous Integration workflow for iggy-sample # -# Runs on every push to main and on all pull requests. -# Includes: formatting, linting, testing, coverage, documentation, and security audit. +# Runs on pushes to main and develop, on pull requests to main, and weekly. +# Includes: formatting, linting, testing, coverage, documentation, security +# audit, and shell-script linting. name: CI @@ -55,6 +56,8 @@ jobs: with: toolchain: stable components: clippy + # TD-2026-09-05: this file's rust-cache pins are a branch head, not + # the release their comment names; re-pin every workflow's together. - uses: Swatinem/rust-cache@f0d9c3887740aee45f6153b24b3a6b815192ec16 # v2 - name: Run Clippy run: cargo clippy --locked --all-targets --all-features -- -D warnings diff --git a/.github/workflows/extended-tests.yml b/.github/workflows/extended-tests.yml index 66bd844..272a98a 100644 --- a/.github/workflows/extended-tests.yml +++ b/.github/workflows/extended-tests.yml @@ -44,6 +44,8 @@ jobs: - uses: dtolnay/rust-toolchain@02cb101ec7c40f2c49e1d9714d64511d8e1b74de # master with: toolchain: stable + # TD-2026-09-05: this file's rust-cache pins are a branch head, not + # the release their comment names; re-pin every workflow's together. - uses: Swatinem/rust-cache@f0d9c3887740aee45f6153b24b3a6b815192ec16 # v2 - name: Install criterion dependencies diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index 914a87f..009eee4 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -114,6 +114,8 @@ jobs: - uses: dtolnay/rust-toolchain@02cb101ec7c40f2c49e1d9714d64511d8e1b74de # master with: toolchain: stable + # TD-2026-09-05: this file's rust-cache pins are a branch head, not + # the release their comment names; re-pin every workflow's together. - uses: Swatinem/rust-cache@f0d9c3887740aee45f6153b24b3a6b815192ec16 # v2 - name: Check for missing documentation diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index d899122..1a7ddd8 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -3,6 +3,8 @@ # Triggered when a version tag is pushed (e.g., v0.1.0). # Builds release binaries for multiple platforms and creates a GitHub release. +# .github/workflows/verify-release.yml runs after this workflow by this name +# (workflow_run); rename both together. name: Release on: @@ -106,6 +108,8 @@ jobs: toolchain: stable targets: ${{ matrix.target }} + # TD-2026-09-05: this file's rust-cache pins are a branch head, not + # the release their comment names; re-pin every workflow's together. - uses: Swatinem/rust-cache@f0d9c3887740aee45f6153b24b3a6b815192ec16 # v2 with: key: release-${{ matrix.target }} @@ -178,8 +182,9 @@ jobs: # Get the previous tag PREV_TAG=$(git describe --tags --abbrev=0 HEAD^ 2>/dev/null || echo "") - # scripts/verify-release.sh matches both echo lines below exactly - # and counts the "- %s (%h)" entries; change them together. + # scripts/verify-release.sh matches the first echo line below + # exactly, which also rules out the fallback, and counts the + # "- %s (%h)" entries; change them together. if [ -n "$PREV_TAG" ]; then echo "Generating changelog since $PREV_TAG" CHANGELOG=$(git log --pretty=format:"- %s (%h)" $PREV_TAG..HEAD) diff --git a/docs/tech-debt/TD-2026-09-05.md b/docs/tech-debt/TD-2026-09-05.md index edd0191..c33462e 100644 --- a/docs/tech-debt/TD-2026-09-05.md +++ b/docs/tech-debt/TD-2026-09-05.md @@ -37,8 +37,11 @@ open, and this record carries it. repos///compare/...`): softprops/action-gh-release, codecov/codecov-action and taiki-e/install-action are identical to their release tags, and dtolnay/rust-toolchain to `master`. -- rust-cache runs in most CI jobs, so a head that breaks it fails visibly. - That says nothing about a head that misbehaves quietly. +- rust-cache runs in the clippy, test, integration, coverage and docs jobs + of every CI run, so a head that breaks it fails visibly. That says nothing + about a head that misbehaves quietly. +- The first rust-cache pin in each of the four workflows names this record, + so a Dependabot diff that moves the pin shows the pointer. ## Proposed fix From c1d654560bdad40452aecbe9e2729247bcae64ec Mon Sep 17 00:00:00 2001 From: mlevkov Date: Thu, 24 Sep 2026 12:53:50 -0700 Subject: [PATCH 21/26] docs: say how the publish job fails, and name both tag patterns Cargo.toml's comment said `publish = false` makes release.yml's publish job fail silently. Without a CRATES_IO_TOKEN the job stops at argument parsing first (TD-2026-09-04), and the setting would stop it with a token; the comment now says both. The README's workflow table now gives release.yml's trigger as `vX.Y.Z` or `vX.Y.Z-`, not `vX.Y.Z` alone, and says Verify Release can also be run by hand; its checks list includes shellcheck. --- Cargo.toml | 3 ++- README.md | 6 +++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/Cargo.toml b/Cargo.toml index e775608..85ab449 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -7,7 +7,8 @@ description = "A comprehensive demonstration of Apache Iggy message streaming wi license = "MIT" # Repo-level releases only - never published to crates.io (enforced here, # not just in CI: cargo refuses to publish with this set). release.yml still -# runs a publish job, which this makes fail silently: TD-2026-09-04. +# runs a publish job, which fails silently: with no token it stops before +# reaching this setting, and with one this would stop it. TD-2026-09-04. publish = false repository = "https://github.com/mlevkov/iggy_sample" homepage = "https://github.com/mlevkov/iggy_sample" diff --git a/README.md b/README.md index 256e810..69a577c 100644 --- a/README.md +++ b/README.md @@ -686,13 +686,13 @@ This project uses GitHub Actions for continuous integration and deployment: |----------|---------|-------------| | `ci.yml` | Push, PR, weekly | Tests, linting, coverage, dependency policy, security audit | | `pr.yml` | PR | Size checks, conventional commits, semver | -| `release.yml` | Tag `vX.Y.Z` | Multi-platform builds, GitHub release, docs deploy | -| `verify-release.yml` | Green Release run | Checks the run with `scripts/verify-release.sh` | +| `release.yml` | Tag `vX.Y.Z` or `vX.Y.Z-` | Multi-platform builds, GitHub release, docs deploy | +| `verify-release.yml` | Green Release run, or by hand | Checks the run with `scripts/verify-release.sh` | | `extended-tests.yml` | Weekly | Benchmarks, stress tests, memory checks | ### Automated Checks - **Formatting**: `cargo fmt --check` -- **Linting**: `cargo clippy -- -D warnings` +- **Linting**: `cargo clippy -- -D warnings`, and `shellcheck` for `scripts/` - **Tests**: stable and beta on Linux, macOS and Windows, plus the MSRV on Linux - **Coverage**: `cargo llvm-cov`; the Codecov upload is not authenticated yet (TD-2026-09-03) From 7439d2952c1f7fbf5c950134b47db5988edbfcf7 Mon Sep 17 00:00:00 2001 From: mlevkov Date: Thu, 24 Sep 2026 12:55:02 -0700 Subject: [PATCH 22/26] docs(code-reviews): correct round 1 where round 2 found it wrong Round 2 found three claims in round 1's artifact that the evidence does not support. The exercise's Pages docs do not match v0.4.1's: 25 files differ (21 HTML, 4 JS) and a .lock exists only in v0.4.1's artifact, so the declined restore rested on a false premise and is reopened. gh deletes the local tag at delete.go lines 105-107 on trunk, but 106-108 in v2.97.0, the version used. And v0.2.0's attempts 2 to 4 were not "Re-run all jobs" (V44): each re-ran only Deploy Documentation. Each correction is marked in place. --- .../release-verifier-2026-09-round1.md | 27 ++++++++++++------- 1 file changed, 18 insertions(+), 9 deletions(-) diff --git a/docs/code-reviews/release-verifier-2026-09-round1.md b/docs/code-reviews/release-verifier-2026-09-round1.md index 6b8b434..2ac6b24 100644 --- a/docs/code-reviews/release-verifier-2026-09-round1.md +++ b/docs/code-reviews/release-verifier-2026-09-round1.md @@ -124,9 +124,11 @@ releasing", and the script's header described a pre-release only by the skipped publish job and the unchanged Latest. But the docs job has no pre-release guard (release.yml's Deploy Documentation job; V26): the v0.4.1-ci.1 run deployed `b7aba82`'s docs to Pages, and deleting the tag did -not undo it (V27). It was harmless this time, since those docs match v0.4.1's, -but the documented procedure generalizes to commits with unreleased API -changes. +not undo it (V27). Those docs are not v0.4.1's either: 25 files differ (21 +HTML, 4 JS) and a `.lock` exists only in v0.4.1's artifact, all from the +dependency bumps between `98f14d5` and `b7aba82`. And the documented +procedure generalizes to commits with unreleased API changes. *(Corrected +in round 2, R2-T7: this said the two sets of docs match.)* **Remediation:** `docs: say what exercising release.yml leaves behind`. CONTRIBUTING lists both things that outlive an exercise, the Pages @@ -192,9 +194,10 @@ with a clone-only `v0.3.0-stale.1`. Premise corrected: [consistency], [reviewer] and [silent] said the documented cleanup, `gh release delete --cleanup-tag`, leaves the local tag behind. gh's source deletes it too when no `-R` is given (cli/cli -`pkg/cmd/release/delete/delete.go:105-107`; V20), so the documented in-clone -usage is clean. The gap is narrower: another clone, a deletion from the web -UI, or `-R`. [tests] had it right. +`pkg/cmd/release/delete/delete.go`, lines 105-107 on trunk and 106-108 in +v2.97.0, the version used here; V20), so the documented in-clone usage is +clean. The gap is narrower: another clone, a deletion from the web UI, or +`-R`. [tests] had it right. *(Line numbers corrected in round 2, R2-T7.)* **Remediation:** `ci(verify-release): tie the run to its tag and fail closed on fetches`. The base is computed excluding tags that exist only in @@ -350,6 +353,11 @@ that ran" could FAIL spuriously. exercise tag can test it by failing or cancelling one build job and choosing "Re-run failed jobs". +*(Corrected in round 2, R2-T2: V44 was wrong. v0.2.0's attempts 2 to 4 +each re-ran only Deploy Documentation, so the case was on record already; +round 2 measured it there and made the verifier check the attempt that +triggered it.)* + --- ## Declined, with reasons @@ -364,6 +372,7 @@ choosing "Re-run failed jobs". its trigger places the re-pin in the next github-actions PR. - **Check Cargo.toml's version at the run's commit independently** [tests]: the validate job prints the comparison, and the verifier checks that line. -- **Restore v0.4.1's Pages deployment now**: the exercise deployed docs - identical to v0.4.1's (same crate version, no source change since), so - restoring changes nothing a reader sees. +- **Restore v0.4.1's Pages deployment now**: declined on the premise that + the exercise deployed docs identical to v0.4.1's (same crate version, no + source change since), so restoring would change nothing a reader sees. + The premise was false (T5); round 2 reopens the restore (R2-T7). From e5fefd240578f752d0de730c3d81a757af3f2a7d Mon Sep 17 00:00:00 2001 From: mlevkov Date: Thu, 24 Sep 2026 13:15:15 -0700 Subject: [PATCH 23/26] ci: close the gaps the remediation check found A check of round 2's remediation by the silent-failure and comment lenses found nothing above MEDIUM, and these in the verifier: - The run listing can lag the attempt whose completion triggers Verify Release, which failed the attempt bounds for no reason. The attempt's own record now decides whether it exists (a missing one exits 2 with gh's 404), and leading zeros are refused. - gh reuses a run-log archive cached on the machine, and from one with per-step files it silently drops a step the archive lacks. The log now comes through a fresh cache directory, as on a CI runner. - An empty list of jobs that ran passed the coverage check as "(0)", where the count comparison it replaced had failed; it now FAILs. With no job list at all the check vanished and "no step failed silently" claimed every job; the check now SKIPs and the other says it covered only the jobs with a log. - Comments: attempt 2's re-run job did not "never start" (it failed in two seconds without reaching a runner, and left no log), the carried-over logs were fetched on attempts 3 and 4 (attempt 2 aborts), Verify Release runs again on a re-run attempt that succeeds, and ci.yml's header lists the dependency-policy job. Mutations: a run listing one attempt behind verifies attempt 4 of 4; a failed job listing gives the SKIP and the qualified PASS; a job list of only skipped jobs FAILs the coverage check; attempt 5, 0 and 03 exit 2. --- .github/workflows/ci.yml | 2 +- .github/workflows/verify-release.yml | 8 ++--- scripts/verify-release.sh | 46 ++++++++++++++++++---------- 3 files changed, 35 insertions(+), 21 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 050286d..c9810c0 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -2,7 +2,7 @@ # # Runs on pushes to main and develop, on pull requests to main, and weekly. # Includes: formatting, linting, testing, coverage, documentation, security -# audit, and shell-script linting. +# audit, dependency policy, and shell-script linting. name: CI diff --git a/.github/workflows/verify-release.yml b/.github/workflows/verify-release.yml index 96ed4da..433ca05 100644 --- a/.github/workflows/verify-release.yml +++ b/.github/workflows/verify-release.yml @@ -5,10 +5,10 @@ # runs scripts/verify-release.sh after each one: a problem shows up as a # failed run here instead of waiting for someone to remember the script. # Re-running jobs of a Release run makes a new attempt of it, which runs -# this again. Until TD-2026-09-04 is resolved it fails on every stable -# release, by design: the crates.io publish step fails silently there. It -# can also be run by hand for any tag (Actions > Verify Release > Run -# workflow). +# this again once it succeeds. Until TD-2026-09-04 is resolved it fails on +# every stable release, by design: the crates.io publish step fails +# silently there. It can also be run by hand for any tag (Actions > Verify +# Release > Run workflow). name: Verify Release diff --git a/scripts/verify-release.sh b/scripts/verify-release.sh index 5d8d267..27f5e1c 100755 --- a/scripts/verify-release.sh +++ b/scripts/verify-release.sh @@ -29,7 +29,7 @@ # new attempt of the run that lists every job but runs only the chosen ones. # The others carry over with their conclusions and their original logs, which # gh 2.75 or later fetches for them (checked on v0.2.0's run 28759754600, -# attempts 2 to 4). A NOTE names the jobs an attempt re-ran. +# attempts 3 and 4). A NOTE names the jobs an attempt re-ran. # # Requires gh 2.75 or later (authenticated), git, perl, tar and curl; unzip # and file are used when present. Exits 0 when every check passes (skips @@ -153,18 +153,17 @@ fi read -r RID SHA LATEST URL <<<"$run" ATTEMPT=${VERIFY_RUN_ATTEMPT:-$LATEST} case $ATTEMPT in - *[!0-9]*) echo "VERIFY_RUN_ATTEMPT is not a number: $ATTEMPT" >&2; exit 2 ;; + '' | *[!0-9]* | 0*) echo "VERIFY_RUN_ATTEMPT is not an attempt number: $ATTEMPT" >&2; exit 2 ;; esac -if [ "$ATTEMPT" -lt 1 ] || [ "$ATTEMPT" -gt "$LATEST" ]; then - echo "run $RID has attempts 1 to $LATEST, not $ATTEMPT" >&2 - exit 2 -fi -# The attempt's own status: a newer attempt may be running while this one -# is complete. -attempt_info=$(gh api "repos/$REPO/actions/runs/$RID/attempts/$ATTEMPT" --jq '"\(.status) \(.run_started_at)"') || - { echo "reading attempt $ATTEMPT of run $RID failed" >&2; exit 2; } +# The attempt's own record decides, not the run listing: a newer attempt may +# be running while this one is complete, and the listing can lag behind the +# attempt whose completion triggered the workflow. +attempt_info=$(gh api "repos/$REPO/actions/runs/$RID/attempts/$ATTEMPT" \ + --jq '"\(.status) \(.run_started_at)"' 2>"$WORK/attempt.err") || + { echo "reading attempt $ATTEMPT of run $RID failed ($(first_line "$WORK/attempt.err"))" >&2; exit 2; } read -r STATUS STARTED <<<"$attempt_info" [ "$STATUS" = completed ] || { echo "run $RID attempt $ATTEMPT is $STATUS; verify it once it completes" >&2; exit 2; } +[ "$ATTEMPT" -gt "$LATEST" ] && LATEST=$ATTEMPT git fetch --quiet --tags origin || { echo "fetching from origin failed" >&2; exit 2; } git cat-file -e "$SHA^{commit}" 2>/dev/null || @@ -268,8 +267,12 @@ fi # # An attempt's log archive holds only the jobs it ran; gh fetches each other # job's log through the API, which serves a carried-over job's original log. -# A job with no log at all, one that never started, fails the whole fetch. -gh run view "$RID" --attempt "$ATTEMPT" --log >"$WORK/raw.log" 2>"$WORK/raw.err" || : >"$WORK/raw.log" +# A job with no log at all fails the whole fetch (v0.2.0's attempt 2 re-ran +# a job that failed within two seconds and left none). An empty cache +# directory makes gh download the archive rather than reuse a copy cached +# earlier on this machine, whose layout may differ from what CI sees. +XDG_CACHE_HOME=$WORK/gh-cache gh run view "$RID" --attempt "$ATTEMPT" --log \ + >"$WORK/raw.log" 2>"$WORK/raw.err" || : >"$WORK/raw.log" : >"$WORK/output.log" if [ -s "$WORK/raw.log" ]; then # shellcheck disable=SC2016 # perl's variables, not the shell's @@ -295,14 +298,25 @@ elif [ ! -s "$WORK/output.log" ]; then fail "parse the run log (no step output left after filtering; has its format changed?)" skip "step-output checks (no step output)" else - unlogged= + # Set when the silent-failure check below cannot vouch for every job. + partial= if [ "$jobs_ok" = true ]; then # shellcheck disable=SC2016 # $1 and $2 are awk fields awk -F'\t' '$2 != "skipped" { print $1 }' "$WORK/jobs.tsv" | LC_ALL=C sort -u >"$WORK/ran.txt" cut -f1 "$WORK/output.log" | LC_ALL=C sort -u >"$WORK/logged.txt" unlogged=$(LC_ALL=C comm -23 "$WORK/ran.txt" "$WORK/logged.txt" | join_lines) - check "log covers every job that ran ($(wc -l <"$WORK/ran.txt" | tr -d ' ')${unlogged:+; missing: $unlogged})" \ - test -z "$unlogged" + # gh prints no log for a skipped job, so a run in which no job ran + # never gets here: an empty list means building it failed. + if [ ! -s "$WORK/ran.txt" ]; then + fail "log covers every job that ran (no job is listed as having run)" + else + check "log covers every job that ran ($(wc -l <"$WORK/ran.txt" | tr -d ' ')${unlogged:+; missing: $unlogged})" \ + test -z "$unlogged" + fi + [ -n "$unlogged" ] && partial=" (in the jobs with a log)" + else + skip "log covers every job that ran (no job list)" + partial=" (in the jobs with a log)" fi check "validate parsed the tag as $VERSION" has_line "Validate Release${TAB}Version: $VERSION" "$WORK/output.log" check "validate matched Cargo.toml version $BASE" has_line "Validate Release${TAB}Version verified: $BASE" "$WORK/output.log" @@ -329,7 +343,7 @@ else awk -F'\t' -v j="$job" '$1 == j && index($2, "##[error]") == 1 { print " " $2 }' "$WORK/output.log" done <"$WORK/error-jobs.txt" else - pass "no step failed silently${unlogged:+ (in the jobs with a log)}" + pass "no step failed silently$partial" fi awk -F'\t' 'index($2, "##[warning]") == 1 { print substr($2, 12, 150) }' "$WORK/output.log" | sort | uniq -c >"$WORK/warnings.txt" From 0566d29cb046eae47178b4bcfff3898fa81242ab Mon Sep 17 00:00:00 2001 From: mlevkov Date: Thu, 24 Sep 2026 13:15:15 -0700 Subject: [PATCH 24/26] docs: correct the restore steps and round 1's attribution CONTRIBUTING called the re-run that failed with 'Multiple artifacts named "github-pages"' v0.2.0's first. It was the second (attempt 3): the first never reached a runner. The bullet no longer implies that the first attempt's artifact stops counting once it expires, which is untested, and says a re-run attempt runs Verify Release only if it succeeds. Round 1's corrected T5 attributed the .lock that only v0.4.1's Pages artifact holds to the dependency bumps. Its cause is specific: the upload-pages-artifact v5 that PR #37 brought leaves out hidden files unless include-hidden-files is set, where v3 kept them. --- CONTRIBUTING.md | 11 ++++++----- docs/code-reviews/release-verifier-2026-09-round1.md | 10 ++++++---- 2 files changed, 12 insertions(+), 9 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a07a918..a475d7a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -214,16 +214,17 @@ way. Two things outlive the exercise: - The docs job deploys the tagged commit's docs to Pages even for a pre-release, and deleting the tag does not undo that. To restore the latest stable release's docs, re-run the Deploy Documentation job of its Release - run (the new attempt runs Verify Release again), with two limits: + run (a new attempt that succeeds runs Verify Release again), with two + limits: - GitHub re-runs jobs only within 30 days of a run. Past that, tag the stable release's commit as a pre-release instead (`git tag v0.4.1-docs.1 'v0.4.1^{}'`) and push it: its run deploys that commit's docs. Delete the tag afterwards like any exercise tag. - deploy-pages refuses a run that holds more than one `github-pages` - artifact, and the first attempt's is kept for a day: v0.2.0's first - re-run failed with `Multiple artifacts named "github-pages"`. If that - happens, delete the run's `github-pages` artifacts and re-run the job - again. `gh api repos/mlevkov/iggy_sample/actions/runs//artifacts` + artifact: v0.2.0's second re-run failed with `Multiple artifacts named + "github-pages"` beside the first attempt's, which had not expired yet + (whether an expired one also counts is untested). If that happens, + delete the run's `github-pages` artifacts and re-run the job again. `gh api repos/mlevkov/iggy_sample/actions/runs//artifacts` lists them, and `gh api -X DELETE repos/mlevkov/iggy_sample/actions/artifacts/` deletes one. diff --git a/docs/code-reviews/release-verifier-2026-09-round1.md b/docs/code-reviews/release-verifier-2026-09-round1.md index 2ac6b24..5c0c23f 100644 --- a/docs/code-reviews/release-verifier-2026-09-round1.md +++ b/docs/code-reviews/release-verifier-2026-09-round1.md @@ -125,10 +125,12 @@ skipped publish job and the unchanged Latest. But the docs job has no pre-release guard (release.yml's Deploy Documentation job; V26): the v0.4.1-ci.1 run deployed `b7aba82`'s docs to Pages, and deleting the tag did not undo it (V27). Those docs are not v0.4.1's either: 25 files differ (21 -HTML, 4 JS) and a `.lock` exists only in v0.4.1's artifact, all from the -dependency bumps between `98f14d5` and `b7aba82`. And the documented -procedure generalizes to commits with unreleased API changes. *(Corrected -in round 2, R2-T7: this said the two sets of docs match.)* +HTML, 4 JS), all from the Dependabot merges between `98f14d5` and +`b7aba82`, and a `.lock` exists only in v0.4.1's artifact, because the +upload-pages-artifact v5 that PR #37 brought leaves out hidden files. And +the documented procedure generalizes to commits with unreleased API +changes. *(Corrected in round 2, R2-T7: this said the two sets of docs +match.)* **Remediation:** `docs: say what exercising release.yml leaves behind`. CONTRIBUTING lists both things that outlive an exercise, the Pages From 39280b9ac320ca10f0db602e846e3afa537f209f Mon Sep 17 00:00:00 2001 From: mlevkov Date: Thu, 24 Sep 2026 13:18:59 -0700 Subject: [PATCH 25/26] docs: bring the changelog entry up to round 2's changes The Unreleased entry predates round 2. It now says the verifier handles a run whose jobs were re-run and needs gh 2.75 or later, that Verify Release runs on exactly the run and attempt that triggered it, re-runs included, and can be run by hand, and that CONTRIBUTING gives the limits of restoring the Pages docs. --- CHANGELOG.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7022bc9..bdf0531 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,15 +17,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 Cargo.toml's `publish`; and that the host platform's binary runs, isolated and time-bounded. It fails on a step that failed under `continue-on-error`, which the Actions API reports as a success: the crates.io publish step has - failed that way on every stable release (TD-2026-09-04) -- The Verify Release workflow runs it after every green Release run, so a - hidden failure fails a run instead of waiting for someone to look. Until - TD-2026-09-04 is resolved it fails on every stable release, by design + failed that way on every stable release (TD-2026-09-04). It handles a run + whose jobs were re-run, where the jobs an attempt did not re-run carry + over with their logs, and needs gh 2.75 or later to fetch those +- The Verify Release workflow runs it after every green Release run, + re-runs included, on exactly the run and attempt that triggered it, so a + hidden failure fails a run instead of waiting for someone to look; it can + also be run by hand for any tag. Until TD-2026-09-04 is resolved it fails + on every stable release, by design - CI lints shell scripts with shellcheck, as part of the required CI Success check - CONTRIBUTING.md describes releasing, and exercising `release.yml` with a pre-release tag, including the two things that outlive the exercise: the - Pages deployment and the tag itself + Pages deployment (with how to restore it, and the limits of doing so) and + the tag itself ### Fixed From 18c1a44e8940846084339f1fa3af127908114b05 Mon Sep 17 00:00:00 2001 From: mlevkov Date: Thu, 24 Sep 2026 13:18:59 -0700 Subject: [PATCH 26/26] docs(code-reviews): release verifier review round 2 Round 2's five lenses (46 findings, none critical, 15 themes) found round 1's code themes closed at their original sites, and the fail-open class recurring in round 1's own new fetches. Its verification pass over 31 claims found 29 present, 2 drifted and 0 absent. The round paused mid-remediation and resumed in a new session, which measured R2-T2's premise on v0.2.0's run before acting on it: a re-run attempt's log lacks the carried-over jobs only for gh before 2.75, so the planned log stitching gave way to verifying the triggering run and attempt and requiring gh 2.75. The measurement also found the Pages restore failing on duplicate github-pages artifacts, and corrected the draft (attempt 2's archive is empty, not 6 of 9 jobs). Because round 1's remediation had opened new silent paths, two lenses then checked this round's remediation: 13 distinct findings, none above MEDIUM, 12 fixed and 1 declined. The artifact cites remediation commits by subject, lists the declined suggestions with reasons, and what is left for after merge: restoring v0.4.1's docs, which also shows whether an expired artifact blocks a re-run and whether the re-run triggers Verify Release. The two draft commits of this artifact are folded in. --- .../release-verifier-2026-09-round2.md | 331 ++++++++++++++++++ 1 file changed, 331 insertions(+) create mode 100644 docs/code-reviews/release-verifier-2026-09-round2.md diff --git a/docs/code-reviews/release-verifier-2026-09-round2.md b/docs/code-reviews/release-verifier-2026-09-round2.md new file mode 100644 index 0000000..9a15eb0 --- /dev/null +++ b/docs/code-reviews/release-verifier-2026-09-round2.md @@ -0,0 +1,331 @@ +# Release Verifier and Dependabot-Wave TD Records — Code Review, Round 2 + +**Target:** branch `chore/release-verifier-and-td-records` after round 1's +remediation: 15 commits on `main` (`b7aba82`), the last being +`docs(code-reviews): release verifier review round 1`. Four earlier commit +messages were corrected in place after this round (R2-T14) with every tree +kept byte-identical, so commits are cited by subject, as in round 1. + +**Provenance:** the five lenses of round 1, pinned to the strongest +available model (Fable): [reviewer] feature-dev:code-reviewer, [silent] +silent-failure-hunter, [comments] comment-analyzer, [tests] +pr-test-analyzer, [consistency] general-purpose. [reviewer]'s first run died +on an API rate limit before reviewing anything and was relaunched with the +same brief. Findings: 46 (silent 12, comments 11, reviewer 9, consistency 7, +tests 7) plus one informational list, none CRITICAL, grouped into 15 themes. +Round 1's code themes held at their original sites; the fail-open class +recurred in round 1's own new fetches. Verification (step 4.5): one pass over +the 31 consolidated claims (W1-W31) found 29 present, 2 drifted (W13, W19; +corrected in R2-T2 and R2-T7) and 0 absent. The round paused mid-remediation +on 2026-09-23 and resumed in a new session on 2026-09-24, which re-measured +R2-T2's premise before acting on it: it holds only for gh before 2.75, and +W13's correction rested on it too. Because round 1's remediation had opened +new silent paths, two of the lenses, [silent] and [comments] (Fable), then +checked the resumed session's remediation: 15 findings, 13 distinct, none +above MEDIUM; 12 fixed and 1 declined (see "Remediation check"). + +Evidence for the remediation, measured on the remediated verifier (passed / +failed / skipped). macOS, gh 2.97.0: v0.4.1 38/1/1, v0.3.0 35/1/3, +v0.4.1-ci.1 15/1/4 (its release was deleted after the exercise), v0.2.0 at +attempt 4 32/4/3, at attempts 3 and 1 31/6/3, at attempt 2 26/5/4, and +v0.2.0's older run 20/15/8. Ubuntu 24.04 container (arm64): gh 2.97.0 gives +v0.4.1 38/1/1 (the cross-built aarch64 binary exits 78) and v0.2.0 32/4/3, +gh 2.75.0 v0.2.0 32/4/3, and gh 2.74.2 and 2.45 (Ubuntu's package) exit 2. +The FAIL on each stable release is TD-2026-09-04's masked publish step. +v0.2.0 adds three FAILs for release notes that are not release.yml's: its +body is now its CHANGELOG.md section, and the release's `updated_at` is six +minutes after its `published_at` (00:21:11Z, 00:27:25Z). Its attempts 1 to +3 add their failed Deploy Documentation jobs. Every new FAIL path was also +shown to fail under a mutation, as the remediation commits record. + +--- + +## R2-T1 — HIGH — The remediation's new fetches failed open + +*[silent] [reviewer] [tests] — 3 of 5.* + +Round 1's new asset-digest comparison treated any artifact-download failure +as a SKIP, and a failed lookup of the live Pages deployment became a SKIP +with a false reason. + +**Remediation:** `ci(verify-release): fail closed where round 2 found new +silent paths`. Only an artifact the listing marks expired may skip; another +download failure, or an artifact without the asset in it, FAILs, and a +missing API digest is replaced by hashing the release asset. Failed lookups +of the live deployment and its site URL FAIL with gh's reason. + +## R2-T2 — MEDIUM — The documented Pages restore expires after 30 days, and it makes a re-run attempt the verifier had not been checked on + +*[consistency] [reviewer] [comments] [tests] — 4 of 5.* + +GitHub re-runs jobs only "up to 30 days after [a run's] initial run" (its +documentation), and the last two stable releases came 26 and 53 days after +the one before, so CONTRIBUTING's restore can be unavailable when it is +needed. And the restore, a re-run of one job, produces the run shape the +script's header called unexercised: the reviewers expected its log to lack +the other jobs, so that "no step failed silently" could pass without the +publish job's `##[error]`. Drifted (W13): "log covers every job that ran" +would still FAIL on such a log. + +The resumed session measured it on v0.2.0's run 28759754600 before acting, +and the premise holds only for gh before 2.75. Round 1's V44 ("Re-run all +jobs") was wrong for all three later attempts, not only the second: each +re-ran only Deploy Documentation. Each attempt's job list names all 9 jobs, +the 8 carried over under new IDs with their original `started_at`, and its +log archive holds only the jobs it ran. Attempt 2's archive is empty (the +draft of this artifact said it held 6 of 9 jobs; gh printed 5 before +stopping at `log not found` for the re-run job, which never reached a +runner: no steps, no log). The per-job logs API serves a carried-over job's +original log, byte-identical across the four attempts, and gh 2.75 and later +fetch every job the archive lacks from it. gh 2.74.2 prints only Deploy +Documentation for attempt 4: there "no step failed silently" passes without +the publish error, and only the job-count check fails. So the planned +stitching of each job's log from the attempt that ran it was not needed. + +The measurement found a second limit. deploy-pages refuses a run that holds +more than one `github-pages` artifact. v0.2.0's attempt 3, its second +re-run, failed with `Multiple artifacts named "github-pages" ... Artifact +count is 2` beside attempt 1's artifact, which upload-pages-artifact keeps +for a day, and attempt 4 succeeded once the earlier ones were gone (the run +lists only attempt 4's, while expired artifacts stay listed). + +**Remediation:** `ci(verify-release): verify the run and attempt that +triggered it`. The workflow passes the triggering run and attempt +(`VERIFY_RUN_ID`, `VERIFY_RUN_ATTEMPT`), which the script checks against the +tag's runs and the run's attempts; jobs and log come from that attempt; a +NOTE names the jobs it re-ran; the job-coverage check names the jobs without +a log; and gh before 2.75 exits 2. Mutations: a gh stand-in that drops two +carried-over logs FAILs naming both; a foreign run ID, attempt 5 and a +non-numeric attempt each exit 2. `docs: say when the Pages restore expires +and how it can fail` gives CONTRIBUTING the 30-day limit, the fallback (a +pre-release tag on the stable release's commit), the `Multiple artifacts` +failure and its fix (delete the run's `github-pages` artifacts, re-run +again), and says that a re-run attempt that succeeds runs Verify Release +again. The remediation check's fixes to both followed (see below). + +Open, both for the restore after merge to show (see the end): whether an +expired `github-pages` artifact also blocks a re-run, and the re-run +trigger itself, which rests on GitHub's documentation ("The `requested` +activity type does not occur when a workflow is re-run", which exempts no +other type), since Verify Release cannot run before it is on main. + +## R2-T3 — MEDIUM — A failed tag lookup skipped with a false reason + +*[silent] [reviewer] — 2 of 5.* + +**Remediation:** `ci(verify-release): fail closed where round 2 found new +silent paths`. The tag check skips only when origin's fail-closed tag +listing lacks the tag; for a tag that is there, a failed lookup FAILs. + +## R2-T4 — MEDIUM — A non-executable binary was skipped + +*[silent] — 1 of 5.* + +**Remediation:** the same commit. A host binary without its exec bit FAILs +with its archive mode. + +## R2-T5 — MEDIUM — The workflow cannot run before merge, and its first run is red by design + +*[tests] [silent] — 2 of 5.* + +`workflow_run` uses the default branch's workflow file, so nothing could +exercise it before merge, and its first run would be the next release, red +until TD-2026-09-04 is resolved. + +**Remediation:** `ci: let Verify Release run by hand and keep the token off +disk` adds a `workflow_dispatch` with a tag input, and the run's summary +page carries the script's output. After merge, a run for v0.4.1 on the +Linux runner should give 38 passed, 1 failed (the masked publish step) and 1 +skipped, the x86_64 Linux binary exiting 78. The permanent red, and the +habituation it invites, ends only when TD-2026-09-04 is resolved: the +maintainer's decision. + +## R2-T6 — MEDIUM — A release.yml comment said the fallback line is matched exactly + +*[comments] — 1 of 5.* + +The negated grep for the fallback's line would pass on any rewording of it. + +**Remediation:** the check went in `ci(verify-release): fail closed where +round 2 found new silent paths`, since the positive "Generating changelog +since" line already rules the fallback out; the comment says so since `ci: +correct the verifier's changelog comment and mark two more sites`. + +## R2-T7 — MEDIUM — Round 1's artifact over-claimed + +*[comments] [consistency] — 2 of 5.* + +The exercise's Pages docs are not "identical" to v0.4.1's: 25 files differ +(21 HTML, 4 JS; drifted, W19, from "25 HTML files"), all from the +Dependabot merges between `98f14d5` and `b7aba82` (one JS file gains a +method block from a bumped dependency), and a `.lock` exists only in +v0.4.1's artifact, because the upload-pages-artifact v5 that PR #37 brought +leaves out hidden files (found by the remediation check). So round 1's +reason to decline restoring v0.4.1's docs was false. And gh's local-tag +deletion is at `delete.go` lines 105-107 on trunk but 106-108 in v2.97.0, +the version used. + +**Remediation:** `docs(code-reviews): correct round 1 where round 2 found it +wrong` corrects both in place, marked as round 2's, along with T17's V44 +(R2-T2). The restore is reopened: proposed after merge (see the end). + +## R2-T8 — LOW — Misattributed fail-closed reasons, and unbounded curls + +*[silent] [tests] — 2 of 5.* + +A failed `gh release list`, deployments lookup or Pages API call FAILed as a +defect of the release ("Latest is none", "none recorded", a served-docs +fetch of an empty URL), with gh's error left on unlabelled stderr. + +**Remediation:** `ci(verify-release): fail closed where round 2 found new +silent paths`: each of those fetches FAILs as a fetch, with gh's reason in +the line, and the site URL comes from the deployment's status instead of a +separate Pages API call. curl calls are time-bounded, and the served-docs +fetches retry for the CDN. + +## R2-T9 — LOW — "No step failed silently" passed on empty output + +*[silent] [reviewer] — 2 of 5.* + +**Remediation:** the same commit. A log with no step output left after +filtering FAILs and skips the step-output checks. + +## R2-T10 — LOW — perl's scalar `exec` on a one-element list + +*[reviewer] — 1 of 5.* + +With one argument, `exec @ARGV` re-parsed the binary's path through a shell. + +**Remediation:** the same commit uses `exec { $ARGV[0] } @ARGV`. A space in +the path gave exit 127 before and 0 after. + +## R2-T11 — LOW — A missing architecture pattern, no-unzip bookkeeping, and absent GitHub notes + +*[silent] [tests] — 2 of 5.* + +**Remediation:** the same commit. A target without an architecture pattern +FAILs, a missing unzip skips both checks it blocks by name, and a release +without GitHub's generated notes FAILs. + +## R2-T12 — LOW — The workflow verified the newest run, not the one that triggered it, and kept the token beside the executed binary + +*[reviewer] — 1 of 5.* + +**Remediation:** the token in `ci: let Verify Release run by hand and keep +the token off disk` (`persist-credentials: false`); the run, and its +attempt, in `ci(verify-release): verify the run and attempt that triggered +it` (R2-T2). + +## R2-T13 — LOW — Prose precision + +*[consistency] [comments] [silent] — 3 of 5.* + +The README gave release.yml's trigger as `vX.Y.Z` only and did not list +shellcheck, nor did ci.yml's header; Cargo.toml's comment said `publish = +false` makes the publish job fail, when the missing token fails it first; +TD-2026-09-05 said rust-cache runs in "most CI jobs", and nothing at its +sites pointed to it; and nothing marked release.yml's `name: Release`, which +Verify Release's trigger matches. + +**Remediation:** `ci: correct the verifier's changelog comment and mark two +more sites` (the `name: Release` marker, pointers at the first rust-cache pin +in each workflow, TD-2026-09-05's five jobs, ci.yml's header) and `docs: say +how the publish job fails, and name both tag patterns` (Cargo.toml, README). +The script's header and its `.env` comment were corrected in +`ci(verify-release): fail closed where round 2 found new silent paths`. + +## R2-T14 — LOW — Commit messages + +*[comments] — 1 of 5.* + +- `ci: lint shell scripts with shellcheck` said the job runs "on every push + and pull request"; it runs in every CI run, whose triggers are narrower. +- `docs(tech-debt): record why Dependabot proposes no iggy 0.11` said only + `CARGO_RESOLVER_INCOMPATIBLE_RUST_VERSIONS=allow` moves iggy to 0.11.0; + that holds under the loosened requirement (an exact one resolves too). +- `ci: add a script that verifies a release run step by step` said a + leftover tag truncates "the next release's notes"; it truncates the + changelog release.yml writes (round 1's T13 made the same correction in + CONTRIBUTING). +- `ci: mark the lines the verifier and two TD records depend on` said the + verifier matches four echo lines exactly; it required three, and only + checked the fallback's for absence. + +**Remediation:** all four corrected in place, trees byte-identical, pushed +with `--force-with-lease`. The two draft commits of this artifact, made at +the pause and never pushed, were folded into its final commit. + +## R2-T15 — LOW — The digest depended on gh's JSON fields, and a failed run kept 320 MB + +*[tests] — 1 of 5.* + +**Remediation:** `ci(verify-release): fail closed where round 2 found new +silent paths` reads digests from the REST API, and a failed run keeps its +logs and archives but not the extracted binaries (67 MB kept, down from +about 320). + +--- + +## Remediation check + +*[silent] 7 LOW; [comments] 2 MEDIUM, 6 LOW. Two pairs overlap, leaving 13.* + +- **MEDIUM:** CONTRIBUTING called the re-run that hit `Multiple artifacts` + v0.2.0's first; it was the second (attempt 3), since attempt 2's re-run + never reached a runner. And the script's header said gh fetched the + carried-over logs on attempts 2 to 4; on attempt 2 the fetch aborts at the + job without a log, so only attempts 3 and 4 show it. +- **LOW, in the script:** the run listing can lag the attempt that + triggered the workflow, which failed the attempt bounds spuriously; gh + reuses a run-log archive cached on the machine, and from one with + per-step files it drops a step the archive lacks without a word; an empty + list of jobs that ran passed the coverage check as "(0)"; and without a + job list the coverage check vanished, while "no step failed silently" + still claimed every job. +- **LOW, in prose:** "never started" for a job that failed in two seconds + without reaching a runner; "six minutes after the run", which is six + minutes after publication and before the re-runs; the re-run trigger + stated as observed fact, and without its success condition; "kept for a + day", implying that expiry clears the conflict; the `.lock`'s cause; and + ci.yml's header omitting the dependency-policy job. + +**Remediation:** `ci: close the gaps the remediation check found` (the +attempt's own record decides whether it exists; a fresh gh cache for each +run; an empty list FAILs; without a job list the coverage check SKIPs and +the silent-failure check says it covered only the jobs with a log; the +comments) and `docs: correct the restore steps and round 1's attribution`. +The two unpushed commit messages among the findings were corrected before +push. Mutations: a run listing one attempt behind now verifies attempt 4 of +4; a failed job listing gives the SKIP and the qualified PASS; a job list +of only skipped jobs FAILs the coverage check; attempt 5 exits 2 with gh's +404. + +**Declined:** the re-ran NOTE misreads a null `run_started_at` (every job +would count as carried over), but it is informational, and a completed +attempt always has one. + +## Declined, with reasons + +- **CLAUDE.md edits** (its stale "Latest stable SDK" line, a pointer to + CONTRIBUTING's Releasing section) [consistency]: CLAUDE.md is the + maintainer's to edit; raised with him. +- **Run the other platforms' binaries** (Rosetta, macOS or Windows legs) + [tests]: the workflow runs the Linux x86_64 binary; the other four are + checked by digest, contents and architecture. A further CI leg is a cost + decision. +- **Fail a stable release while release-bound TD records are open, or add a + status badge** [silent]: a policy gate beyond the verifier; the badge + would be red by design until TD-2026-09-04 is resolved. + +## After merge + +- Restore v0.4.1's docs by re-running its Release run's Deploy + Documentation job (run 35888419705, within 30 days of 2026-09-23). That + replaces the exercise's docs (R2-T7), shows whether the expired + `github-pages` artifact blocks a re-run (R2-T2), and, if the attempt + succeeds, whether it triggers Verify Release, which would then check + attempt 2 on its carried-over logs. Expected on the Linux runner: 40 + passed, 1 failed, 0 skipped, since the served docs are then v0.4.1's. + Otherwise, run Verify Release by hand for v0.4.1 (R2-T5). +- Resolve TD-2026-09-04 before the next stable tag, or Verify Release stays + red by design.