Skip to content

Retry transient release/module downloads in smoke.sh #37

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kixelated merged 1 commit into from
Sep 14, 2026
Merged

Retry transient release/module downloads in smoke.sh #37

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 39 additions & 13 deletions smoke.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,36 @@ trap cleanup EXIT

have() { command -v "$1" >/dev/null 2>&1; }

# Run a command up to N times with a short growing backoff. For fetches from
# registries/proxies that fail transiently (sum.golang.org stream resets, etc.);
# without it one blip marks a client broken and fails its whole column.
retry() {
local n="$1" i
shift
for ((i = 1; i <= n; i++)); do
"$@" && return 0
[[ $i -lt $n ]] && sleep $((i * 2))
done
return 1
}

# Download a release tarball to a file and extract it into a directory. Two
# deliberate choices: curl retries every failure (--retry alone skips partial
# transfers, and a truncated body is exactly the failure GitHub's CDN produces),
# and the archive lands on disk before tar sees it, so a short read can't leave a
# half-extracted tree behind while `tar xz` reports only "unexpected end of file".
fetch_tarball() {
local url="$1" dir="$2"
local archive="$dir/.download.tar.gz"
mkdir -p "$dir"
if ! curl -sfL --retry 5 --retry-delay 2 --retry-all-errors -o "$archive" "$url" ||
! tar xzf "$archive" -C "$dir"; then
echo "download/extract failed: $url" >&2
return 1
fi
rm -f "$archive"
}

require_tools() {
# Resolve the CLI binary name unless MOQ_BIN pinned it.
if [[ -z "$MOQ" ]]; then
Expand Down Expand Up @@ -204,7 +234,7 @@ libmoq_fetch() {
[[ -n "${GITHUB_TOKEN:-}" ]] && hdr=(-H "Authorization: Bearer $GITHUB_TOKEN")
# macOS ships bash 3.2, where "${hdr[@]}" on an empty array trips `set -u`;
# the ${arr[@]+...} guard expands to nothing when the array is unset/empty.
tag=$(curl -sf ${hdr[@]+"${hdr[@]}"} "https://api.github.com/repos/moq-dev/moq/releases?per_page=100" |
tag=$(curl -sf --retry 3 --retry-all-errors ${hdr[@]+"${hdr[@]}"} "https://api.github.com/repos/moq-dev/moq/releases?per_page=100" |
jq -r '.[].tag_name' | grep '^libmoq-v' | head -1)
Comment on lines +237 to 238

@chatgpt-codex-connector chatgpt-codex-connector Bot Sep 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Write the retried release response to a file

When this API transfer fails after emitting a partial response, --retry-all-errors writes the next attempt into the same pipe, so jq receives the truncated JSON concatenated with the retried response and cannot extract a tag; the libmoq client is therefore still marked broken after a recoverable network blip. In the curl 8.5.0 CLI documentation inspected via curl --manual, --retry-all-errors explicitly warns that pipe output “is not reset” before retries and may contain duplicate data. Download this response with -o, as the moq-gst query already does, before parsing it.

Useful? React with 👍 / 👎.

[[ -n "$tag" ]] || {
echo "no libmoq-v* release found" >&2
Expand All @@ -213,11 +243,7 @@ libmoq_fetch() {
ver=${tag#libmoq-v}
url="https://github.com/moq-dev/moq/releases/download/$tag/moq-$ver-$target.tar.gz"
echo "libmoq $tag ($target)"
mkdir -p "$TMP/libmoq"
curl -sfL "$url" | tar xz -C "$TMP/libmoq" || {
echo "download/extract failed: $url" >&2
return 1
}
fetch_tarball "$url" "$TMP/libmoq" || return 1
LIBMOQ_ROOT="$TMP/libmoq/moq-$ver-$target"
}

Expand Down Expand Up @@ -344,7 +370,7 @@ gst_prepare() {
# (CI) to dodge the 60/hr anonymous GitHub API limit.
[[ -n "${GITHUB_TOKEN:-}" ]] && hdr=(-H "Authorization: Bearer $GITHUB_TOKEN")
releases="$TMP/moq-gst-releases.json"
curl -sf ${hdr[@]+"${hdr[@]}"} \
curl -sf --retry 3 --retry-all-errors ${hdr[@]+"${hdr[@]}"} \
"https://api.github.com/repos/moq-dev/moq/releases?per_page=100" -o "$releases" || {
echo "failed to query moq-gst releases" >&2
return 1
Expand Down Expand Up @@ -373,11 +399,7 @@ gst_prepare() {
return 1
}
echo "moq-gst $tag ($target)"
mkdir -p "$TMP/moq-gst"
curl -sfL "$url" | tar xz -C "$TMP/moq-gst" || {
echo "download/extract failed: $url" >&2
return 1
}
fetch_tarball "$url" "$TMP/moq-gst" || return 1
root="$TMP/moq-gst/${asset_name%.tar.gz}"
GST_PLUGIN_DIR="$root/lib/gstreamer-1.0"
[[ -d "$GST_PLUGIN_DIR" ]] || {
Expand Down Expand Up @@ -424,9 +446,13 @@ if needs go; then
# ergonomic moq-go wrapper pulls a transitive moq-go-ffi; `go get moq-go`
# records only moq-go's own checksum, so `go mod tidy` fetches the rest (no
# go.sum is committed -- freshness bans lockfiles -- so it's regenerated here).
# The module proxy / sum.golang.org occasionally reset mid-stream; retry the
# fetch (idempotent) and build only once it succeeds.
# shellcheck disable=SC2329 # invoked via retry
go_fetch() (cd "$CLIENTS/go" && go get "github.com/moq-dev/moq-go@latest" && go mod tidy)
if ! have go; then
mark_broken go "go not found"
elif (cd "$CLIENTS/go" && go get "github.com/moq-dev/moq-go@latest" && go mod tidy && CGO_ENABLED=1 go build -o "$GO_SMOKE" .) >"$TMP/go-build.log" 2>&1; then :; else
elif (retry 3 go_fetch && cd "$CLIENTS/go" && CGO_ENABLED=1 go build -o "$GO_SMOKE" .) >"$TMP/go-build.log" 2>&1; then :; else
mark_broken go "go get/build of moq-dev/moq-go failed"
sed 's/^/ /' "$TMP/go-build.log" >&2 || true
fi
Expand Down
Loading