From d6a87b6c3b5bbe9904b9d76dd2fb74ade29a9b6c Mon Sep 17 00:00:00 2001 From: Christopher Bartz Date: Sun, 25 Oct 2026 05:04:34 +0000 Subject: [PATCH] chore(garm): bump workload to fix runners stuck on hung database queries Pin GARM to the merge of cloudbase/garm#926. It bounds every PostgreSQL query with a statement, lock and connect timeout, reaps runners left in creating when recording them as running fails, and omits associations when writing instances and scale sets to remove an ABBA deadlock between those updates. --- docs/changelog.md | 4 ++++ garm-rockcraft.yaml | 7 +++++-- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/docs/changelog.md b/docs/changelog.md index 27712c49..a692ee5c 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -8,6 +8,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Each revision is versioned by the date of the revision. +## 2026-10-25 + +- `garm`: bump the GARM workload so that runners no longer get stuck. GARM now bounds every PostgreSQL query with a timeout instead of waiting forever on a hung query or row lock, cleans up runners left in `creating` when recording their creation fails, and no longer deadlocks when an instance and its scale set are updated concurrently. + ## 2026-10-21 - `garm`: job steps now see `USER` and `LOGNAME` set to the account their UID resolves to, so `usermod -aG $USER` followed by `sg ` works on GARM runners. This works around the duplicate-UID account layout of the runner images; removal is tracked in [#438](https://github.com/canonical/github-runner-operators/issues/438). diff --git a/garm-rockcraft.yaml b/garm-rockcraft.yaml index d054a3c8..7d5caf28 100644 --- a/garm-rockcraft.yaml +++ b/garm-rockcraft.yaml @@ -17,7 +17,7 @@ parts: plugin: go source: https://github.com/cloudbase/garm source-type: git - # 2026/10/15: Pin to a post-v0.2.1 commit carrying these changes, until v0.2.2+ + # 2026/10/25: Pin to a post-v0.2.1 commit carrying these changes, until v0.2.2+ # is released: # - golang.org/x/crypto v0.57.0+. # - the scale set pseudo pool ID fix (PR #866). Without it, OpenStack scale set @@ -30,8 +30,11 @@ parts: # databases upgraded through migration 0003 fail migration 0008 with # "operator does not exist: text = uuid" and GARM does not start # (cloudbase/garm#920). + # - the database query timeouts, creating-instance reaper and association + # omission (PR #926). Without them, hung PostgreSQL queries and an ABBA + # deadlock between instance and scale set updates leave runners stuck. # Switch to source-tag once v0.2.2+ is released. - source-commit: ae281e373bc7568d63c0298dc0406109379216d2 + source-commit: 1ad3680f097c3e71fa564ca26a41e62e13cbf91f source-depth: 1 build-snaps: - go/1.26/stable