Skip to content

Publish what built instead of discarding the run over one package - #412

Closed
omarchybot wants to merge 1 commit into
masterfrom
release-partial-failures
Closed

omarchybot wants to merge 1 commit into
masterfrom
release-partial-failures

Conversation

@omarchybot

Copy link
Copy Markdown
Collaborator

The failure this prevents

flea 0.2.1 failed its test suite and stopped every channel for a day. bin/build exited 1, bin/release treated that as fatal and returned before sign/promote/sync — so the twelve packages that built successfully in the same run were discarded, rebuilt on the next tick, and discarded again. Nothing reached edge, rc or stable from 09-11 until the package itself was fixed.

What changes

file change
bin/build exit 2 when at least one package built but others failed/blocked. exit 1 still means a run that produced nothing to publish. Writes the failed names to a state file.
helpers/paths.sh new build_failures_file <mirror> <arch> — build and release are separate processes, so the list must outlive the builder
bin/release accepts 2, runs the rest of the pipeline on what did build, reports the missing packages through the existing notifier, exits 2
bin/auto-release treats 2 as progress: clears the queue and the backoff

bin/build already distinguished failed from blocked and never builds a package whose dependency failed. What it couldn't express was "some built" vs "nothing built" — both exited 1, so the caller had to assume the worst.

The debatable bit

A partial release clears the backoff instead of counting as a failure. Counting it would reinstate the original bug on a longer timescale: four consecutive partials back a channel off to six-hour intervals, and one permanently broken package again decides when every other package ships.

The cost: a transient failure is no longer retried on the next tick — it waits for the next commit or version bump, which is also when it has a reason to succeed. Happy to flip this if you'd rather keep retry-on-transient; it's a one-line change.

Safety

Nothing publishes that couldn't before. Blocked packages are still excluded from the build output, so no package ships without the dependency it was built against.

Testing

bash -n clean on all four. Diffs reviewed against copies before applying.

Note: CI's self-tests job covers sync-upstream, sync-rebuilds, omarchy-pkgs and omarchy-releasenone of these four files. A green check here does not mean this path was exercised. The partial path has not yet run against a real failure, since flea now builds.

🤖 Generated with Claude Code

A single package that would not build stopped every channel for a day. flea
0.2.1 failed its test suite, bin/build exited 1, and bin/release treated that
as fatal and returned before sign, promote or sync. Twelve packages had built
successfully in the same run; all twelve were thrown away, rebuilt on the next
tick, and thrown away again. Nothing reached edge, rc or stable from 09-11
until the package itself was fixed.

bin/build already knows the difference between a package that failed and one
that was blocked by a failed dependency, and it never builds the blocked ones.
What it could not express was the difference between "some packages built" and
"nothing built": both exited 1, so the caller had to assume the worst. It now
exits 2 when at least one package built, keeps exit 1 for a run that produced
nothing to publish, and writes the names it could not build to a state file,
because the build and the release are separate processes and the list has to
outlive the builder.

bin/release accepts 2, runs the rest of the pipeline on what did build, and
reports the missing packages at the end through the same notifier that already
reports a failed step. It exits 2 so the outcome stays truthful to a human
running it by hand: packages were published, and some are missing.

bin/auto-release treats 2 as progress and clears the queue and the backoff.
This is the part worth arguing about. Counting a partial release as a failure
would have reinstated the original bug on a longer timescale: four consecutive
partials back a channel off to six-hour intervals, and one permanently broken
package would again decide when every other package ships. The cost is that a
transient failure is no longer retried on the next tick; it waits for the next
commit or version bump, which is also when it has a reason to succeed.

Nothing publishes that could not before. Blocked packages are still excluded
from the build output, so no package ships without the dependency it was built
against.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ryanrhughes

Copy link
Copy Markdown
Collaborator

Superseded: builds and publishing moved to CI in #511, which publishes per-package results by design; bin/release's all-or-nothing behaviour no longer gates anything.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants