Skip to content

CI resilience: wait for the channel index on rattler's lowercase "no candidates"; retry anaconda.org's existence check - #143

Open
Tobias-Fischer wants to merge 4 commits into
mainfrom
index-lag-message
Open

Tobias-Fischer wants to merge 4 commits into
mainfrom
index-lag-message

Conversation

@Tobias-Fischer

@Tobias-Fischer Tobias-Fischer commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Lyrical's linux-64 build-branch run failed in stage 8 after 10 seconds (job 120387806070): Failed to resolve dependencies … ros2-rcpputils … for which no candidates were found. The previous stage had just uploaded that rcpputils build, and it's on both channels now.

build-ci is supposed to wait for the channel index in exactly this case, up to twice for 180 s each. But it recognised the case only by the text "No candidates were found", and rattler also writes it as "…, for which no candidates were found": lowercase, and wrapped over two lines.

The check now matches the lowercase "no candidates were" and rattler's summary line "there are no viable options". A test feeds run_build that exact solver output and checks that it retries.

Also: on_anaconda(), which checks whether anaconda.org already has a package before uploading it, now retries network errors up to four times (waiting 30, 60 and 90 s). A DNS lookup failure on a macOS runner (nodename nor servname provided) had ended lyrical's osx-64 job right after its packages were built (job 120424046924). A test covers the retry.

And: uploads get five attempts, waiting 30 s, 1, 2 and 4 minutes, instead of three within 90 s. anaconda.org answered three uploads in a row with 503 Service Unavailable and ended lyrical's osx-arm64 job (job 120429719220).

Labelled build:none: only the checks run. All 57 tool tests pass.

🤖 Claude (AI assistant), on behalf of @Tobias-Fischer

🤖 Generated with Claude Code

…hich no candidates were found"

The retry for a dependency the channel's index doesn't list yet matched only
"No candidates were found". rattler also reports it as "…, for which no
candidates were found", lowercase and wrapped over two lines, so lyrical's
linux-64 stage 8 failed in 10 s instead of waiting for the rcpputils its
previous stage had just uploaded. Match the lowercase start and the
"there are no viable options" summary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Tobias-Fischer Tobias-Fischer added the build:none PR build: build no packages, only run the checks (tooling, CI or docs changes) label Oct 23, 2026
on_anaconda() gave up on the first network error: a DNS lookup failure on a
macOS runner ended lyrical's osx-64 job after its packages were built. Retry
up to four times, waiting 30, 60 and 90 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Tobias-Fischer Tobias-Fischer changed the title build-ci: also wait for the channel index on rattler's lowercase "no candidates were found" CI resilience: wait for the channel index on rattler's lowercase "no candidates"; retry anaconda.org's existence check Oct 23, 2026
Tobias-Fischer and others added 2 commits October 23, 2026 23:13
anaconda.org answered three uploads in a row with 503 Service Unavailable
within 90 s, which ended lyrical's osx-arm64 job; its outages last minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…body")

rattler's repodata download broke off mid-stream ("error decoding response
body … error reading a body from connection … stream error received") and
ended lyrical's osx-arm64 job; treat it as the network error it is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@traversaro

Copy link
Copy Markdown
Member

Thanks! It would be great to get the rattler-build errors in a more structured form, as this is at highly risk of break in the future if rattler-build changes its wording (a classical case of https://www.hyrumslaw.com/). Anyhow, good for me for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build:none PR build: build no packages, only run the checks (tooling, CI or docs changes)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants