Repository navigation
chore: release-run verifier and TD records from the Dependabot wave - #43
Merged
Merged
Conversation
A green release.yml run can hide a failure: the crates.io publish step has failed on every stable release (v0.2.0 through v0.4.1) under continue-on-error, and the Actions API reports such a step as a success at both step and job level. scripts/verify-release.sh checks what each step printed, the artifacts, the release and its assets, the Pages deployment record, and that the host platform's binary runs and reports its version. A hyphenated tag is checked as a pre-release: publish skipped, Latest untouched. It was run against the v0.4.0 and v0.4.1 releases, which fail only on the masked publish step, and the v0.4.1-ci.1 exercise run, which fails only on its since-deleted release, and mutation-checked for a wrong or pre-release changelog base. Log checks drop the "##[group]Run" blocks, where GitHub echoes a step's script source, and read files rather than pipes, so a negated grep cannot pass by SIGPIPE. CONTRIBUTING.md gains a Releasing section, including why an exercise tag must be deleted: release.yml's changelog starts at the nearest earlier tag, so a leftover pre-release tag drops every commit before it from the changelog release.yml writes for the next release.
The Code Coverage job passes secrets.CODECOV_TOKEN, which does not
exist, so Codecov rejects every upload ("Token required - not valid
tokenless upload") and fail_ci_if_error: false keeps the job green: all
15 main push runs whose logs are retained, 2026-07-05 to 2026-09-23 (one
cancelled after its upload had run), across codecov-action v4, v4.6.0
and v7.1.1. The README still says coverage is uploaded to Codecov.
TD-2026-09-03 records the two ways out (a real token that fails loudly,
or no upload) and binds the decision to the next coverage-job edit,
major codecov-action bump, or release.
release.yml's publish step has failed on all four stable releases, v0.2.0 through v0.4.1: CRATES_IO_TOKEN does not exist, so cargo stops at argument parsing, and with a token it would stop at publish = false. continue-on-error hides it, and the Actions API reports the failed step as a success too, so only its ##[error] log line shows it. TD-2026-09-04 records deleting the job or publishing deliberately, and binds the choice to the next stable release, which verify-release.sh now fails until it is made.
All 17 rust-cache pins say "# v2" but point at upstream master's head, five commits past v2.9.2, because TD-2026-07-07 pinned an untagged commit and dependabot-core follows the branch of an untagged SHA. So each github-actions PR can carry unreleased upstream code under a release comment. TD-2026-09-05 proposes re-pinning to v2.9.2's commit (so Dependabot tracks releases) and takes over the addendum's open item, moving audit-check to a release SHA once upstream cuts one. TD-2026-07-07's addendum now points to it.
The main ruleset's required checks are not strict, so a pull request stays mergeable on a green run computed against an older main. After the 151-crate group merged, the sha1 PR still showed MERGEABLE on the pre-group lockfile, and between scheduled runs Dependabot rebases only on conflict, so only a manual rebase kept an untested tree off main. TD-2026-09-06 records that manual procedure as the stopgap, proposes the "up to date before merging" setting, and binds it to the next Dependabot wave.
TD-2026-07-02 said to check the Dependabot run log if no iggy PR appeared. None did, and the log's ungrouped pass reports "Requirements to unlock update_not_possible" and closes with "No update possible for iggy 0.10.0". The cause reproduces locally: resolver 3 prefers versions that fit rust-version 1.93.0, so with a loosened requirement cargo keeps iggy 0.10.0, and under that requirement only CARGO_RESOLVER_INCOMPATIBLE_RUST_VERSIONS=allow moves it to 0.11.0 (an exact "0.11.0" requirement resolves too). The record now says to upgrade by hand with the MSRV raise, and notes that Dependabot likewise skips any update, security fixes included, that needs a newer Rust than rust-version.
The smoke test ran a binary it had just downloaded from the release under review with the caller's full environment (GH_TOKEN and any cloud credentials) and from the repository root, where the app's dotenvy loads a .env with API keys. It had no timeout, and it relied on Config::validate rejecting a zero OPERATION_TIMEOUT_SECS, which only v0.4.0 and later do: run against v0.3.0 or v0.2.0, the binary opened a listener on 0.0.0.0:9090 and tried to connect before exiting 69. The binary now runs under env -i from an empty directory, killed after 30 seconds (reported as its own FAIL), and not at all for versions before 0.4.0. Checked with a hung stand-in (exit 142 after the alarm), /usr/bin/env (only the two variables), /bin/pwd and a missing binary (127). The header no longer lists curl, which nothing called.
The run was picked by tag name and never compared with the commit the tag names, so a re-pushed tag whose new run is missing would verify the old commit's run and pass. The tag's commit on origin must now equal the run's head (skipped only once the tag is gone, as for a deleted exercise tag), and the header shows the attempt and says when a tag has several runs (v0.2.0 has two). A failed fetch no longer leaves checks to pass on nothing. A missing or expired log FAILs with gh's reason and skips the step-output checks instead of letting the negated ones pass on an empty file; a failed job listing or artifact listing FAILs rather than exiting 2 mid-run; a failing `gh run list` is reported as such; and the required tools are checked up front. The run's own commit is now the oracle: the crate name, the build targets and whether a publish job exists come from Cargo.toml and release.yml at that commit, not from the working tree, so a missing build job or publish job FAILs instead of shrinking the check list. The default changelog base ignores tags that exist only in this clone, as release.yml's checkout does, and names them. From a clone with a local-only v0.3.0-stale.1, verifying v0.4.0 used to produce two false FAILs; it now computes v0.3.0. Mutations checked: the older v0.2.0 run (tag mismatch FAILs), an expired log (FAIL with the 410), a broken job listing, and a broken run listing (exit 2 with gh's error).
Four of the five release assets were checked by name only, and nothing tied any asset to what its build job produced. Each is now downloaded as the run's artifact and must match the release asset's sha256 exactly (the API's digest), hold its binary, README.md and a LICENSE, and carry a binary for its target according to `file` (patterns accept both the macOS and the libmagic word order). An expired artifact falls back to the release asset, with the digest comparison skipped. The host platform's binary for the smoke test now comes from that extraction. The "no artifact digest mismatch" grep goes: download-artifact v8 words a per-artifact mismatch "digest validation failed", and in its default mode the step fails, which the job-conclusion and ##[error] checks already catch. Newly read: the release notes (one entry per commit since the base, newest first at the tagged commit, and GitHub's own compare link naming the same base); the Pages deployment found by tag rather than commit, and while it is live, the served root redirect and rustdoc version; and crates.io, which must lack the version while Cargo.toml says publish = false and have it otherwise. Hidden failures are reported one FAIL per job, so a second one moves the count, and warnings are listed as notes (v0.4.1's run has nine Node 20 deprecation warnings). Mutation-checked in one run: a corrupted artifact, a wrong expected architecture, a wrong binary name, publish = true, and an injected error in a second job (7 FAILs); a wrong changelog base (notes count and compare base FAIL); a wrong served version.
The verifier was only a CONTRIBUTING paragraph: nothing ran it, so
TD-2026-09-04's trigger ("the verifier fails on the next stable
release") depended on someone remembering, the same silence one level
up that the verifier exists to end. verify-release.yml runs it on
workflow_run after each green Release run from a tag push, with a
read-only token (actions, contents, deployments, pages), a 20-minute
limit, and the tag name passed through the environment rather than
interpolated into the script. It checks out the default branch, so the
verifier that runs is the reviewed one, never code from the tag.
It fails on every stable release until TD-2026-09-04 is resolved, by
design; the script's header now says so. It cannot run before this
lands on main (workflow_run uses the default branch's file), so its
first run is the next release or exercise tag.
scripts/verify-release.sh now runs after every release and its checks gate a release's credibility, but nothing linted it: shellcheck ran only by hand. A Shellcheck job lints scripts/*.sh as bash in every CI run, and CI Success, a required check, depends on it.
TD-2026-09-03 and TD-2026-09-04 bind to "the next edit" of the Codecov step, the publish job and Cargo.toml's `publish = false`, but nothing at those sites said so, while this repo marks TD-2026-09-01's tripwires in ci.yml. Each site now names its record, and so do the four release.yml echo lines scripts/verify-release.sh reads and the changelog's "- %s (%h)" entry format it counts. Three of those lines it requires verbatim, so rewording one fails the verifier with a visible reason; the fallback's line it only checks for absence. Comments only: no step's behavior changes, and Cargo.toml still passes `cargo metadata --locked`.
CONTRIBUTING's Releasing section called a pre-release tag a way to exercise release.yml "without releasing", but the docs job has no pre-release guard: v0.4.1-ci.1's run deployed b7aba82's docs to Pages, and deleting the tag did not undo it. The section now lists both things that outlive an exercise (the Pages deployment, with how to restore it, and the tag), the steps before tagging (version, changelog, TD records bound to the next release), and the Verify Release workflow, which fails every stable release until TD-2026-09-04 is resolved. It names the trigger patterns (vX.Y.Z, vX.Y.Z-<suffix>) rather than v*, and says a leftover tag truncates the changelog release.yml writes, not "the notes" (GitHub appends its own, over its own range). The README stops claiming coverage is uploaded to Codecov, lists the new workflow and scripts/, and, like SECURITY.md, says Dependabot skips versions that need a newer Rust than the MSRV. The changelog entry now covers what the verifier checks and the workflow and lint that run it.
TD-2026-09-03 said 14 main push runs were rejected; there are 15 with a
retained log, all rejected. The count filtered on run conclusion and so
dropped 35910387482, cancelled by the concurrency group after its
upload step had run. Its title no longer says the upload "has never
authenticated", which reaches past the retained logs; it now says what
was measured.
Precision fixes: TD-2026-09-04 says the verifier fails on any ##[error]
line, which it does, notes that crates.io's 404 needs a User-Agent (403
without), and records the Verify Release workflow and the comments that
now name it. TD-2026-09-06 says Dependabot also rebases on a scheduled
run, and states that merge queues are for organization-owned
repositories only. TD-2026-07-02 points at the log's closing verdict
("No update possible for iggy 0.10.0"), says the MSRV raise also fires
TD-2026-09-02, and puts "by hand" in its Status line. TD-2026-07-07's
pointer says PR #37 already moved rust-cache as predicted, and no longer
calls v2.9.2 (released 2026-08-06) new.
TD-2026-09-05's second trigger, an upstream audit-check release, was an
event nothing here observes; it now binds to the next Dependabot PR that
moves either pin, with the upstream check done then.
Five lenses (tier-graduated cadence) reviewed the verifier and the TD records: 51 findings, all kept after a verification pass over 48 consolidated claims (45 present, 3 drifted and corrected, 0 absent), grouped into 17 themes. Six are HIGH: nothing ran the verifier; its smoke test ran a downloaded binary with the caller's environment; four of five assets were checked by name only; the run was never tied to its tag's commit; exercising release.yml left an unmentioned Pages deployment; and a record miscounted 15 runs as 14. One premise three lenses shared (that gh's --cleanup-tag leaves the local tag) was false per gh's source, which narrows T8 rather than discarding it. The artifact cites each theme's remediation commit by subject, and records five declined suggestions with reasons.
mlevkov
force-pushed
the
chore/release-verifier-and-td-records
branch
from
September 23, 2026 23:31
5fffd78 to
25328a6
Compare
Round 2 found the fail-open class recurring in round 1's own new fetches. An artifact download that failed for any reason, not only expiry, became a SKIP of the digest comparison; now only an artifact the listing marks expired may skip, other failures and an artifact without the asset FAIL, and a missing API digest is replaced by hashing the release asset (digests now come from the REST API). A failed lookup of the live Pages deployment, the tag's commit, the Latest release or the site URL FAILs with gh's reason instead of skipping with a false one; "live" is the newest deployment that succeeded, and the URL comes from its status. A binary without its exec bit FAILs, a target without an architecture pattern FAILs, missing GitHub notes FAIL, and no-unzip skips name both checks. An empty filtered log FAILs rather than letting "no step failed silently" pass over nothing, and the negated grep for the changelog fallback is gone: the positive line already rules it out. The host binary is exec'd in perl's list form: with one argument, `exec @ARGV` re-parsed the path (a space in TMPDIR gave exit 127). curl calls are time-bounded, served-docs fetches retry for the CDN, and a failed run keeps its logs but not the 100 MB extracted binaries. Checked: v0.4.1 38/1/1, v0.4.1-ci.1 15/1/4, v0.3.0 35/1/3, and mutations for each new FAIL path.
workflow_run uses the default branch's file, so the workflow cannot run before merge and its first run would be the next release, red by design until TD-2026-09-04 is resolved. A workflow_dispatch with a tag input allows a controlled first run against v0.4.1 right after merge, and the script's output now also lands in the run's summary. The job runs a downloaded binary, so checkout no longer persists the token; pages: read goes, since the site URL now comes from the deployment.
Verify Release checked the tag's newest run at its latest attempt, not the one whose completion triggered it, and the script called a run with re-run jobs unexercised. Measured on v0.2.0's run 28759754600, whose attempts 2 to 4 each re-ran only Deploy Documentation: each attempt lists all 9 jobs, the 8 it carried over under new IDs with their original started_at, but its log archive holds only the jobs it ran (attempt 2's is empty). gh 2.75 and later fetch every other job's log from the per-job logs API, which serves a carried-over job's original log, byte-identical across the four attempts. gh 2.74.2 prints only Deploy Documentation for attempt 4, and "no step failed silently" then passes without the publish step's ##[error]. So the script now requires gh 2.75 and exits 2 on an older one (Ubuntu 24.04 packages 2.45), and stitching logs from earlier attempts is not needed. The workflow passes the triggering run and attempt as VERIFY_RUN_ID and VERIFY_RUN_ATTEMPT; the script checks them against the tag's runs and the run's attempts and exits 2 on any other. Jobs come from that attempt, the log from gh run view --attempt, and a NOTE names the jobs an attempt re-ran. "Log covers every job that ran" names the jobs it misses instead of comparing counts, and "no step failed silently" says so when it covered only the jobs with a log. Checked on macOS: v0.2.0 attempt 4 32/4/3 (the publish FAIL, and three for notes replaced after publication), attempts 3 and 1 31/6/3 (their Deploy Documentation failures), attempt 2 26/5/4 (log not found: its re-run job never reached a runner), and v0.2.0's older run by id 20/15/8 (tag mismatch); v0.4.1 38/1/1, v0.3.0 35/1/3 and v0.4.1-ci.1 15/1/4, as before. v0.2.0 and v0.4.1 match in an Ubuntu 24.04 container. A gh stand-in that drops two carried-over logs FAILs naming both, and a foreign run id, attempt 5 and a non-numeric attempt each exit 2.
CONTRIBUTING's way to undo an exercise's Pages deployment, re-running the latest stable release's Deploy Documentation job, works only within 30 days of that run (GitHub's limit), and the last two stable releases came 26 and 53 days after the one before. The section now gives the fallback, a pre-release tag on the stable release's commit whose run deploys that commit's docs, and says the re-run is a new attempt that runs Verify Release again. A re-run can also fail outright: deploy-pages refuses a run holding more than one github-pages artifact, and upload-pages-artifact keeps the first attempt's for a day. v0.2.0's run shows it. Its attempt 3 failed with 'Multiple artifacts named "github-pages" ... Artifact count is 2', and attempt 4 succeeded once the earlier ones were gone (the run lists only attempt 4's, while expired artifacts stay listed). The section says how to list and delete them, and that VERIFY_RUN_ID and VERIFY_RUN_ATTEMPT pick the run and attempt to verify by hand.
release.yml said scripts/verify-release.sh matches both changelog echo lines exactly. Since round 2 removed the negated check for the fallback's line, it matches only the first, whose presence already rules the fallback out; the comment now says so. Two sites that something depends on said nothing. verify-release.yml's workflow_run trigger matches release.yml's `name: Release`, so a rename would stop verification without a trace; the name now says so. And TD-2026-09-05 binds to the rust-cache pins: the first pin in each of the four workflows now names the record, which lists that pointer and replaces "most CI jobs" with the five that run rust-cache (clippy, test, integration, coverage, docs). ci.yml's header names the triggers its `on:` has and includes shell-script linting. Comments only: every workflow still parses, and no step changes.
Cargo.toml's comment said `publish = false` makes release.yml's publish job fail silently. Without a CRATES_IO_TOKEN the job stops at argument parsing first (TD-2026-09-04), and the setting would stop it with a token; the comment now says both. The README's workflow table now gives release.yml's trigger as `vX.Y.Z` or `vX.Y.Z-<suffix>`, not `vX.Y.Z` alone, and says Verify Release can also be run by hand; its checks list includes shellcheck.
Round 2 found three claims in round 1's artifact that the evidence does not support. The exercise's Pages docs do not match v0.4.1's: 25 files differ (21 HTML, 4 JS) and a .lock exists only in v0.4.1's artifact, so the declined restore rested on a false premise and is reopened. gh deletes the local tag at delete.go lines 105-107 on trunk, but 106-108 in v2.97.0, the version used. And v0.2.0's attempts 2 to 4 were not "Re-run all jobs" (V44): each re-ran only Deploy Documentation. Each correction is marked in place.
A check of round 2's remediation by the silent-failure and comment lenses found nothing above MEDIUM, and these in the verifier: - The run listing can lag the attempt whose completion triggers Verify Release, which failed the attempt bounds for no reason. The attempt's own record now decides whether it exists (a missing one exits 2 with gh's 404), and leading zeros are refused. - gh reuses a run-log archive cached on the machine, and from one with per-step files it silently drops a step the archive lacks. The log now comes through a fresh cache directory, as on a CI runner. - An empty list of jobs that ran passed the coverage check as "(0)", where the count comparison it replaced had failed; it now FAILs. With no job list at all the check vanished and "no step failed silently" claimed every job; the check now SKIPs and the other says it covered only the jobs with a log. - Comments: attempt 2's re-run job did not "never start" (it failed in two seconds without reaching a runner, and left no log), the carried-over logs were fetched on attempts 3 and 4 (attempt 2 aborts), Verify Release runs again on a re-run attempt that succeeds, and ci.yml's header lists the dependency-policy job. Mutations: a run listing one attempt behind verifies attempt 4 of 4; a failed job listing gives the SKIP and the qualified PASS; a job list of only skipped jobs FAILs the coverage check; attempt 5, 0 and 03 exit 2.
CONTRIBUTING called the re-run that failed with 'Multiple artifacts named "github-pages"' v0.2.0's first. It was the second (attempt 3): the first never reached a runner. The bullet no longer implies that the first attempt's artifact stops counting once it expires, which is untested, and says a re-run attempt runs Verify Release only if it succeeds. Round 1's corrected T5 attributed the .lock that only v0.4.1's Pages artifact holds to the dependency bumps. Its cause is specific: the upload-pages-artifact v5 that PR #37 brought leaves out hidden files unless include-hidden-files is set, where v3 kept them.
The Unreleased entry predates round 2. It now says the verifier handles a run whose jobs were re-run and needs gh 2.75 or later, that Verify Release runs on exactly the run and attempt that triggered it, re-runs included, and can be run by hand, and that CONTRIBUTING gives the limits of restoring the Pages docs.
Round 2's five lenses (46 findings, none critical, 15 themes) found round 1's code themes closed at their original sites, and the fail-open class recurring in round 1's own new fetches. Its verification pass over 31 claims found 29 present, 2 drifted and 0 absent. The round paused mid-remediation and resumed in a new session, which measured R2-T2's premise on v0.2.0's run before acting on it: a re-run attempt's log lacks the carried-over jobs only for gh before 2.75, so the planned log stitching gave way to verifying the triggering run and attempt and requiring gh 2.75. The measurement also found the Pages restore failing on duplicate github-pages artifacts, and corrected the draft (attempt 2's archive is empty, not 6 of 9 jobs). Because round 1's remediation had opened new silent paths, two lenses then checked this round's remediation: 13 distinct findings, none above MEDIUM, 12 fixed and 1 declined. The artifact cites remediation commits by subject, lists the declined suggestions with reasons, and what is left for after merge: restoring v0.4.1's docs, which also shows whether an expired artifact blocks a re-run and whether the re-run triggers Verify Release. The two draft commits of this artifact are folded in.
mlevkov
force-pushed
the
chore/release-verifier-and-td-records
branch
from
September 24, 2026 20:20
25328a6 to
18c1a44
Compare
mlevkov
marked this pull request as ready for review
September 24, 2026 20:25
4 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-ups from triaging and merging Dependabot's first wave (#37–#42). A green
release.yml run can hide a failed step (the crates.io publish step has failed
on every stable release under
continue-on-error), so this adds a verifier thatchecks a Release run step by step, a workflow that runs it after every green
Release run, and tech-debt records for the pre-existing problems the triage
found. No application code changes.
Type of Change
Changes Made
scripts/verify-release.shchecks a run against the workflow andCargo.toml at the commit it built: that the tag names that commit; job
conclusions; what each step printed (GitHub's echoed script source removed),
including a step that failed under
continue-on-error; the release, itsnotes and assets, each asset byte-identical to its build job's artifact and
built for its target; the Pages deployment and served docs; crates.io
against
publish; and the host binary, run isolated and time-bounded. Itfails closed on every fetch, and it verifies a specific run and attempt,
including one whose jobs were partly re-run (needs gh 2.75+).
.github/workflows/verify-release.ymlruns it after every green Releaserun (re-runs included, on the run and attempt that triggered it) and on
demand. Until TD-2026-09-04 is resolved it fails on every stable release,
by design.
scripts/*.shwith shellcheck, under the required CI Success.release.yml with a pre-release tag, and what outlives an exercise (the Pages
deployment, with how to restore it and the limits of that, and the tag).
TD-2026-09-04 (the crates.io publish job cannot succeed), TD-2026-09-05
(rust-cache pinned to a branch head), TD-2026-09-06 (main's required checks
are not strict), and a trigger check in TD-2026-07-02 (why Dependabot
proposes no iggy 0.11). The sites these bind to now name them.
Testing
Two review rounds (five lenses each, tier-graduated cadence), plus a
two-lens check of round 2's own remediation; artifacts in
docs/code-reviews/release-verifier-2026-09-round{1,2}.md. Final verifierresults (passed/failed/skipped), macOS with gh 2.97.0:
publish step.
after the run), at attempts 1 and 3 31/6/3, at attempt 2 26/5/4, and its
older run 20/15/8 (tag mismatch).
Ubuntu 24.04 container: v0.4.1 38/1/1 and v0.2.0 32/4/3 with gh 2.97.0;
gh 2.74.2 and 2.45 exit 2. Every new FAIL path was checked with a mutation.
Test commands run:
shellcheck -s bash scripts/*.sh scripts/verify-release.sh v0.4.1 scripts/verify-release.sh v0.3.0 scripts/verify-release.sh v0.4.1-ci.1 scripts/verify-release.sh v0.2.0 VERIFY_RUN_ATTEMPT=2 scripts/verify-release.sh v0.2.0 VERIFY_RUN_ID=28759254872 scripts/verify-release.sh v0.2.0No Rust code changed, so the Rust checklist items don't apply; CI still runs
fmt, clippy and the tests.
Checklist
Documentation
Security
repository's own release binary with an empty environment, from an empty
directory, with an invalid config so it exits before any network I/O; the
workflow's token is read-only and not persisted to disk.
Additional Notes
After merge: restore v0.4.1's docs by re-running Deploy Documentation on its
Release run (if the re-run succeeds, it should also run Verify Release on
the new attempt), or run Verify Release by hand for v0.4.1. TD-2026-09-04
needs a decision before the next stable tag.