Skip to content

Cap the quality jobs so a hung runner cannot hold the release queue - #829

Merged
zakius merged 1 commit into
stagingfrom
fix/quality-job-timeouts
Sep 12, 2026
Merged

zakius merged 1 commit into
stagingfrom
fix/quality-job-timeouts

Conversation

@zakius

@zakius zakius commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

CI was stuck, not failing. Staging run 34708677330 hung on Run unit tests at 17:35:48 and never moved -- the same step finishes in ~30s on every other run. With no timeout-minutes, GitHub's 6-hour default applied.

docker-build.yml's await_turn counts any older run on the branch that is not completed, so every later staging push parked behind that one hung job. #827 sat at BLOCKED with all 18 checks green except Wait for older merges to finish.

I cancelled the hung run to unblock the queue. This caps the four quality jobs so the next hang fails in minutes instead of hours:

job real runtime cap
Lint ~20s 10m
Compile every deployable target ~15s 10m
Unit and migration tests ~30s 15m
Browser tests ~1m 20m

Job names and ids are untouched -- they are pinned as required checks on main.

🤖 Generated with Claude Code

A staging run's unit-test job hung at its first step and sat there for
hours: no timeout-minutes meant GitHub's 6-hour default applied. Every
later staging run then parked in docker-build's await_turn, which counts
any older run still in flight, so one stuck runner stalled the whole
release queue -- visible as "CI is stuck" with nothing failing.

The caps are well above the real runtimes (lint ~20s, compile ~15s, unit
~30s, browser ~1m), so they only ever fire on a hang. Job names and ids
are untouched: they are pinned as required checks on main.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@zakius
zakius merged commit d42e4dc into staging Sep 12, 2026
18 checks passed
@zakius
zakius deleted the fix/quality-job-timeouts branch September 12, 2026 21:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant