CI infrastructure jammed: jobs stuck in queued (runner never assigned) + 3 workflows fast-fail without logs — blocks all PRs #77

Activity

1. hyperpolymath

   added

   bugSomething isn't workingSomething isn't working

   infra

   migration-blocker

   on May 18, 2026
2. hyperpolymath commented on May 18, 2026

   

   hyperpolymath

   on May 18, 2026

   OwnerAuthor

   More actions

   CI jam — diagnosis split into two distinct causes

   Cause A — 3 workflows instant-failing (CODE DEFECT — fixed in #78)

   cargo-audit.yml, secret-scanner.yml, dogfood-gate.yml were rejected by GitHub's workflow validator at parse time on every push (run name==path, jobs:[], 0 s duration, no logs — the "invalid workflow file" signature, not runner starvation). Root causes: illegal job-level hashFiles() in two files; a dedented inline-Python heredoc breaking the run: | block scalar in the third. Surgical fix in PR #78. No org-admin action required for this part.

   Cause B — ~70/100 runs stuck queued, runner: null (NOT a code defect, NOT fixable in a PR)

   Evidence gathered:

   • gh api repos/hyperpolymath/echidna/actions/runners → {"total_count":0,"runners":[]} (no self-hosted runners — correct, all runs-on: are ubuntu-latest).
   • gh api orgs/hyperpolymath/actions/runners and .../runner-groups → HTTP 404 Not Found. hyperpolymath is "type":"User", not an org — so there is no org runner-group misconfig to fix.
   • gh api repos/hyperpolymath/echidna/actions/permissions → {"enabled":true,"allowed_actions":"all","sha_pinning_required":true} (Actions enabled, not disabled).
   • gh api users/hyperpolymath/settings/billing/actions → HTTP 404 ("needs user scope"; the legacy per-user billing endpoint is also deprecated — cannot confirm/deny a spending cap via API with the current token).

   Diagnosis: every push/PR to main fans out into ~20+ workflows simultaneously (mirror.yml has 7 jobs, plus instant-sync.yml, scorecard.yml, scorecard-enforcer.yml, live-provers.yml, container-ci.yml, rust-ci.yml ×6 jobs, etc.). On a personal GitHub account, GitHub-hosted concurrency is tightly capped, so runs sit queued with runner: null (never assigned) until slots free up — they are not failing, just starved.

   Exact remediation required (org-admin / repo-admin, no code PR):

   1. Confirm the Actions spending limit / included-minutes state at https://github.com/settings/billing — if the included free minutes for private-repo Actions are exhausted, raise the spending limit or wait for the monthly reset. (Could not verify via API — user scope + non-deprecated endpoint needed; check the billing UI directly.)
   2. Reduce per-push fan-out (this DOES need a follow-up PR, tracked separately from fix(ci): repair 3 invalid workflows rejected at parse time (Refs #77) #78): add concurrency: groups with cancel-in-progress: true to the high-volume workflows, and narrow over-broad triggers (mirror.yml, instant-sync.yml, scorecard* firing on every push are the main amplifiers). Recommend a dedicated issue/PR for the trigger-pruning + concurrency-group sweep.

   Note: gh token already carries workflow scope — no gh auth refresh was needed to push the workflow fixes in #78.
3. hyperpolymath

   added 3 commits that reference this issue on May 18, 2026

   fix(ci): make dogfood-gate Python heredoc survive YAML block-scalar d…

   ...

   ac68524

   fix(ci): repair 3 invalid workflows rejected at parse time (Refs #77) (…

   ...

   1182089

   fix(ci): concurrency dedup + tighten over-broad triggers (Refs #77) (#79

   ...

   e718fd3
4. hyperpolymath commented on May 18, 2026

   

   hyperpolymath

   on May 18, 2026

   OwnerAuthor

   More actions

   Status update — code-side mitigations landed; root driver remains billing-gated

   Both code-defect fixes are now on main. The CI matrix is still jammed because the binding constraint is account-level and not fixable in-repo — see "Residual" below.

   Fixed (merged)

   Cause A — 3 workflows fast-failed at parse time — PR #78 (squash-merged 1182089, admin override; green CI unattainable while jammed).

   • cargo-audit.yml / secret-scanner.yml: removed the illegal job-level hashFiles if:; detection now runs in a step and gates later steps via if: steps.detect.outputs.present == 'true'.
   • dogfood-gate.yml: the eclexiaiser-validator Python heredoc. The original PR fix(ci): repair 3 invalid workflows rejected at parse time (Refs #77) #78 used a sed 's/^<16 spaces>//' dedent that was itself broken — GitHub strips the run: block-scalar's 10-space common indent before the script reaches the shell, so the heredoc body arrived at 6 leading spaces while the sed still looked for 16, leaving Python with bad indentation (and echidna does ship an eclexiaiser.toml, so this path is always hit). Corrected in fix(ci): repair 3 invalid workflows rejected at parse time (Refs #77) #78 before merge: the sed dedent was removed entirely and the Python body placed at the block-scalar base indent so YAML's own dedent delivers it column-aligned. Validation logic unchanged; verified byte-precisely by simulating the YAML block-scalar dedent.

   Cause B — self-inflicted fan-out amplification — PR #79 (squash-merged e718fd3, admin override, same justification).

   • Added concurrency: { group: ${{ github.workflow }}-${{ github.ref }} } to all 20 workflows that lacked one (hypatia-scan.yml already had the canonical block). cancel-in-progress: true for push/PR CI gates; cancel-in-progress: false for mirror/sync/publish/scorecard so an in-flight mirror, publish or security scan is never aborted.
   • Tightened two over-broad triggers (no gate disabled, all still fire): governance.yml pull_request: was unscoped → branches: [main, master]; container-ci.yml push: was any-branch (path-filtered only) → branches: [main, master].

   Observed effect so far

   Not yet visibly improved. The post-#79 push to main (e718fd3) still fans out the full matrix, all queued with no runner assigned. Expected: the new concurrency: blocks only de-duplicate superseded re-pushes to the same ref — they cannot drain a queue that the account-level cap is throttling. The benefit becomes visible on the next push that supersedes an in-flight run (stale duplicates auto-cancel instead of stacking), and fully only once the cap below is lifted.

   Residual — USER ACTION REQUIRED (org-admin / billing, not in-repo)

   The root Cause-B driver is unchanged: GitHub-hosted Actions concurrency / spending-limit saturation on the personal account — jobs are accepted and time-stamped but never assigned a runner (runner: null), the classic symptom of a hit Actions spending cap / concurrency ceiling.

   Please check manually: https://github.com/settings/billing → Actions — verify the spending limit is not exhausted (raise it / enable spend, or wait for the monthly reset), and review any org/account Actions concurrency ceiling. This is the binding constraint; until it is cleared, runs will continue to queue regardless of in-repo changes.

   Leaving this issue open — it closes when a normal PR can merge on genuinely green CI without --admin, which depends on the billing/concurrency item above.
5. hyperpolymath

   added a commit that references this issue on May 19, 2026

   docs: CI-baseline triage record (human ADR + machine STATE) (#85)

   ...

   bf9dab2
6. hyperpolymath commented on May 30, 2026

   

   hyperpolymath

   on May 30, 2026

   OwnerAuthor

   More actions

   Closing as stale — CI infra is no longer jammed

   Verified 2026-05-30: Workflow runs on main are now reaching real success/failure conclusions; the May-18 perpetual-queued symptom is gone. Sample of the last governance run on main (run 26633535347):

   Status	Job
   ✅ success	governance / Code quality + docs
   ✅ success	governance / Security policy checks
   ❌ failure	governance / Trusted-base reduction policy
   ❌ failure	governance / Language / package anti-pattern policy
   ✅ success	governance / Workflow security linter
   ✅ success	governance / Validate Hypatia baseline
   ✅ success	governance / Guix primary / Nix fallback policy
   ✅ success	governance / Well-Known (RFC 9116 + RSR)

   These are real outcomes, not perpetual queued. The two failures are now tracked under #92 (governance baseline content failures) and are being addressed in:

   • PR chore(governance): clear 12 trusted-base false-positive escape hatches #126 (chore(governance): clear 12 trusted-base false-positive escape hatches)
   • PR chore(governance): populate empty deno.lock with minimal valid v4 lockfile #127 (chore(governance): populate empty deno.lock with minimal valid v4 lockfile)

   Closing #77 as the infrastructure-level jam is resolved. The 3 fast-failing workflows (cargo-audit.yml, dogfood-gate.yml, secret-scanner.yml) called out in the original report were replaced by wrappers in #113/#114/#115 and are now reporting normally (Secret Scanner: success, Dogfood Gate: success on the latest run).
7. hyperpolymath commented on May 30, 2026

   

   hyperpolymath

   on May 30, 2026

   OwnerAuthor

   More actions

   Stale — CI infra is no longer jammed. See triage above.
8. hyperpolymath

   closed this as completedon May 30, 2026