Skip to content

fix(artifact-delivery): keep healthy runs inside their limits - #484

Merged
harlan-zw merged 3 commits into
mainfrom
fix/resolve-stability
Oct 6, 2026
Merged

harlan-zw merged 3 commits into
mainfrom
fix/resolve-stability

Conversation

@harlan-zw

@harlan-zw harlan-zw commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

🔗 Linked issue

Related to #481

❓ Type of change

  • 📖 Documentation
  • 🐞 Bug fix
  • 👌 Enhancement
  • ✨ New feature
  • 🧹 Chore
  • ⚠️ Breaking change

📚 Description

skilld run failed for healthy users at four limits on the Resolution path.

  • GitHub quota. The registry sync and Artifact builds share GITHUB_TOKEN. The sync spends it to zero before each hourly GitHub reset. On 2026-10-06, 13 runs failed RATE_LIMITED in those minutes, among them ericzakariasson/skills/game-builder. GitHub counts a personal token's quota per account. Builds now read GitHub with ARTIFACT_GITHUB_TOKEN when it is set, and with GITHUB_TOKEN until then.
  • GitHub secondary rate limits. GitHub answers them with 403 or 429 and Retry-After. A 403 failed the run as SOURCE_ACCESS_DENIED, which nobody retries. A 429 threw, so the build sat in the queue retry ladder and spent quota on every attempt. Both now fail as RATE_LIMITED with the wait GitHub named. A rate-limited tarball now stops the build. It used to fall back to one blob read per file, which spent the most quota exactly when none was left. The classification now matches update-plans.ts.
  • Guest allowance. The CLI polls GET /api/v1/resolutions/{id} every 1000 ms, at the rate the server names. A one minute build costs about 60 polls, which is the whole guest allowance of 60 requests a minute. Polls now draw from their own binding, API_RESOLUTION_POLL_RATE_LIMIT, at 600 a minute under the same key. The POST that starts a Resolution stays in the shared allowance. ADR-0012 amends ADR-0008.
  • Retry timing. The CLI waits 60 seconds for a Resolution, and the first queue retry also waited 60 seconds. One transient GitHub or signer error therefore failed every run that met it. The retry ladder is now 5, 30, 120 and 480 seconds. Every retryable failure now names retryAfterSeconds, with 60 seconds as the default when the cause named no wait.

📝 Migration

To give runs their own GitHub quota, create a fine-grained token with public repository read access on a GitHub account other than the one behind GITHUB_TOKEN. Then run wrangler secret put ARTIFACT_GITHUB_TOKEN --config wrangler.jsonc. Without the secret, builds keep using GITHUB_TOKEN.

🤖 AI disclosure: Harlan Agent Kit modified this description. My AI open-source policy.

A run polls its Resolution once a second, so one slow build spent the
whole guest allowance of 60 requests a minute and failed with
RATE_LIMITED. Polls now draw from their own allowance (ADR-0012).

GitHub answers a secondary rate limit with 403 or 429 and Retry-After.
A 403 failed the run as SOURCE_ACCESS_DENIED, and a 429 threw into the
queue retry ladder. Both now fail as RATE_LIMITED with the named wait,
and a rate-limited tarball no longer falls back to one read per blob.

Every retryable failure now names retryAfterSeconds. The first queue
retry waits 5 seconds instead of 60, inside the CLI's 60 second window.
@harlan-zw harlan-zw added the harlan-agent-review Pull request triage requires an adversarial Review. Adding this label manually always forces one. label Oct 6, 2026
@harlan-github-agent harlan-github-agent Bot added harlan-agent-review-required Pull request triage requires an adversarial Review for this head commit. and removed harlan-agent-review Pull request triage requires an adversarial Review. Adding this label manually always forces one. labels Oct 6, 2026
@harlan-zw harlan-zw added the harlan-agent-review-required Pull request triage requires an adversarial Review for this head commit. label Oct 6, 2026
The hourly registry sync spends GITHUB_TOKEN to zero before each GitHub
reset. On 2026-10-06, 13 runs failed RATE_LIMITED in those minutes.
GitHub counts a personal token's quota per account, so builds now read
with ARTIFACT_GITHUB_TOKEN when it is set, and with GITHUB_TOKEN until then.
@harlan-zw harlan-zw added the harlan-agent-review Pull request triage requires an adversarial Review. Adding this label manually always forces one. label Oct 6, 2026
@harlan-github-agent harlan-github-agent Bot added harlan-agent-running An Agent holds a Task on this issue or pull request right now. and removed harlan-agent-review Pull request triage requires an adversarial Review. Adding this label manually always forces one. harlan-agent-running An Agent holds a Task on this issue or pull request right now. labels Oct 6, 2026
@harlan-github-agent

harlan-github-agent Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

🤖 MERGED

Harlan Agent Kit posted this automated review. It is not Harlan's personal review or approval. AI open source policy. Last updated: 2026-10-06 15:41 UTC.

GitHub merged this pull request.

No material findings were recorded.

The pull request closed.

@harlan-github-agent harlan-github-agent Bot removed the harlan-agent-running An Agent holds a Task on this issue or pull request right now. label Oct 6, 2026
@harlan-github-agent harlan-github-agent Bot added the harlan-agent-running An Agent holds a Task on this issue or pull request right now. label Oct 6, 2026
@harlan-zw
harlan-zw added this pull request to stack #492 October 6, 2026 15:04
@harlan-github-agent harlan-github-agent Bot removed the harlan-agent-running An Agent holds a Task on this issue or pull request right now. label Oct 6, 2026
# Conflicts:
#	layers/artifact-delivery/server/utils/github-source.ts
@harlan-zw
harlan-zw merged commit b4bdf0e into main Oct 6, 2026
2 of 6 checks passed
@harlan-github-agent harlan-github-agent Bot removed the harlan-agent-review-required Pull request triage requires an adversarial Review for this head commit. label Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant