Skip to content

fix(mirror): recover from a cut-off GitHub release listing - #306

Merged
CalvinAllen merged 1 commit into
mainfrom
fix/mirror/release-listing
Nov 13, 2026
Merged

CalvinAllen merged 1 commit into
mainfrom
fix/mirror/release-listing

Conversation

@CalvinAllen

Copy link
Copy Markdown
Contributor

Description

Today's Mirror Sync failed its Python job:

Warning: failed to fetch from python-build-standalone: page 7: parsing releases: stream error: stream ID 13; CANCEL; received from peer

GitHub dropped page 7 of the python-build-standalone release listing part way through. A page of 10 releases is tens of megabytes of JSON. httpGetWithRetry only retried failed connections and 5xx responses, so a body that broke off while being decoded failed the whole source, and with it the Python job.

The failed job then held back everything else. Generate Manifests from R2 ran only when Mirror Sync succeeded, so today's run was skipped: Node.js and Ruby synced fine, but nothing they mirrored reached the manifests.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)

Related Issues

Relates to #305

Changes Made

  • fetchWithRetry replaces httpGetWithRetry. It reads the whole body inside the retry, so a response cut off part way is fetched again like a failed connection or a 5xx response. Other statuses come back for the caller to check, as before. The python-build-standalone and RubyInstaller listings, the ruby-builder release and python.org downloads use it.
  • Generate Manifests from R2 now runs after a failed Mirror Sync as well as a successful one, but not after a cancelled one. Manifests only list what R2 holds, and the mirror uploads a build's metadata after its archive, so a partial sync can't put a missing build in a manifest. When the sync failed, the run's summary says so and links to it.
  • Mirror Sync's summary no longer tells you to run manifest generation by hand, since it starts on its own. The CLAUDE.md workflow table now says when it runs.

Testing

Test Environment

  • OS: Windows 11
  • Go Version: 1.25.3 (toolchain)

Test Steps

  1. TestFetchWithRetry: a body cut off and then complete, a 5xx and then complete, a 4xx returned without a retry, and a cut-off body or 5xx on every attempt (error after 3 attempts). The cut-off is a hijacked connection that promises 1000 bytes, sends a few and closes.
  2. TestFetchGitHubReleases_CutOffPage: page 2 is cut off once; all 25 releases come back, with page 2 fetched twice.
  3. go run . --runtime python --dry-run against live upstream lists the same 759 jobs as before: 578 from python-build-standalone and 181 from python.org.

Test Results

--- PASS: TestFetchWithRetry
--- PASS: TestFetchGitHubReleases_CutOffPage
ok  	github.com/CodingWithCalvin/dtvem.cli/scripts/mirror-binaries

./rnr check fails only on TestGlobalVersion_NoConfigFile and TestGlobalVersion_RuntimeNotInConfig in internal/config, the same failures as on unmodified main on the machine I tested on. Nothing under src/ changed. golangci-lint on the mirror tool reports 15 issues, all in code this PR doesn't touch, against 19 on main. The 4 it no longer reports were unchecked resp.Body.Close calls replaced by fetchWithRetry.

Checklist

  • My code follows the Google Go Style Guide
  • I have run golangci-lint run and fixed all issues (includes gofmt, govet, and 40+ linters) (no new findings; older ones in the mirror tool left as they are)
  • I have run go mod tidy to clean up dependencies
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes (go test ./...) (apart from the two internal/config tests above, unchanged from main)
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation (README.md, CONTRIBUTING.md, etc.) if needed
  • My changes generate no new warnings or errors
  • I have added appropriate error handling
  • I have followed conventional commit format in my commit messages

Breaking Changes

N/A

Additional Notes

  • A failed sync still shows red. This only stops it from blocking the manifests.
  • A listing that fails on every attempt still fails the Python job. Mirroring the pages that did come back would hide the gap, so this PR doesn't do that.

fetchWithRetry replaces httpGetWithRetry and reads the whole body inside
the retry, so a response GitHub drops part way through ("stream error:
... CANCEL") is fetched again instead of failing the source.

Generate Manifests from R2 now also runs after a failed Mirror Sync, so
one failed source or download no longer keeps everything else that
synced out of the manifests.
@CalvinAllen
CalvinAllen merged commit c360f06 into main Nov 13, 2026
3 checks passed
@CalvinAllen
CalvinAllen deleted the fix/mirror/release-listing branch November 13, 2026 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant