fix(runner): keep a saturated CI host from killing its own jobs - #6
Merged
Merged
Conversation
Three fixes that were running in production but had never been committed. The busy-runner health check clocked from the process start instead of the idle→busy transition, so a warm-pool runner that had been up for hours was declared timed out on its first job and killed mid-run. It now clocks from BusySince and kills through KillIdleRunner. The watchdog probed with a 5s timeout and a 3-failure threshold. A CI host is bursty by design, and under CPU saturation the daemon stays alive while its HTTP goroutine waits to be scheduled, so the probe tripped and os.Exit(2) took every in-flight job with it. 20s and 5 failures means it only fires when the daemon is genuinely wedged for ~2.5min. Cleanup now removes the containers and github_network_* networks a job leaves behind. Best-effort by design: no docker means no-op, and a docker failure only logs, because workdir removal must never be blocked by container cleanup.
CI has been red on main for several runs. Nine findings, split three ways. Adding BusySince to RunnerSnapshot pushed it to 136 bytes, over gocritic's rangeValCopy threshold, so the three loops in checks.go now range by index. selectJobContainers names its results and preallocates networks, and the runnerStarter embedded in fakeProcess went away: the fake implements all four methods itself, so the embed was dead weight. The two unconvert findings on stat.Bsize get a nolint instead of the fix the linter suggests. Bsize is int64 on linux and uint32 on darwin, so removing the conversion is only correct on the platform the linter runs on, and stops the darwin build from compiling.
RedBoardDev
force-pushed
the
fix/runner-host-stability
branch
from
August 14, 2026 17:36
b51994d to
3d6d645
Compare
TestRunnerEnvIsolatesHome demanded XDG_CACHE_HOME, XDG_CONFIG_HOME and XDG_DATA_HOME under the isolated home on every platform, while runnerEnv only overrides them on darwin. It failed on linux, which is where the daemon runs. The linux branch now asserts the opposite, because that is the real invariant: the linux runner resolves _work from XDG_DATA_HOME, so overriding it moves the job workspace somewhere the daemon never looks. Anyone dropping the GOOS guard now breaks a test instead of production.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pourquoi cette PR existe
Ces trois correctifs tournaient en production sur le VPS KARE mais n'avaient jamais été commités. Le binaire déployé les contient (le log
removing root-owned workdir via dockerle prouve), alors quemainne les a pas. L'état n'existait que dans un arbre de travail local, dont deux fichiers non suivis par git.Les trois correctifs
Contrôle de santé. Le test de runner occupé comptait le temps depuis le démarrage du process (
StartedAt) au lieu de la transition inactif vers occupé. Un runner issu du pool chaud, démarré depuis des heures, était donc déclaré en dépassement dès son premier job et tué en plein travail. Il compte maintenant depuisBusySince, et passe parKillIdleRunner.Watchdog. La sonde utilisait un timeout de 5 s et un seuil de 3 échecs. Un hôte CI est bursty par construction : sous saturation CPU le daemon reste vivant mais sa goroutine HTTP attend d'être schedulée. La sonde échouait donc, et le
os.Exit(2)emportait tous les jobs en vol. À 20 s et 5 échecs, il ne se déclenche que si le daemon est réellement bloqué pendant environ 2 min 30.Nettoyage des conteneurs. Le nettoyage supprime désormais les conteneurs et les réseaux
github_network_*laissés par un job. Best-effort assumé : pas de docker, pas d'opération, et un échec docker ne fait que logger, parce que la suppression du workdir ne doit jamais être bloquée par le nettoyage des conteneurs.Vérifications
Build propre, les 362 tests du dépôt passent,
gofmtetgo vetsans remarque. Le contenu est exactement l'état déployé, sans modification de ma part.config.yamlet.claude/skills/sont volontairement laissés hors de la PR : configuration locale.Suite
La PR #5 (liens physiques pour les binaires de runner) est à rebaser au-dessus de celle-ci. Elle ne doit pas être déployée seule : un binaire construit depuis
mainplus #5 ne contiendrait aucun des correctifs ci-dessus et régresserait la prod.