Skip to content

chore(garm): bump workload to fix runners stuck on hung database queries #445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cbartz wants to merge 1 commit into
base: main
Choose a base branch
from
+9 −2
Open

chore(garm): bump workload to fix runners stuck on hung database queries #445

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/changelog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

Each revision is versioned by the date of the revision.

## 2026-10-25

- `garm`: bump the GARM workload so that runners no longer get stuck. GARM now bounds every PostgreSQL query with a timeout instead of waiting forever on a hung query or row lock, cleans up runners left in `creating` when recording their creation fails, and no longer deadlocks when an instance and its scale set are updated concurrently.

## 2026-10-21

- `garm`: job steps now see `USER` and `LOGNAME` set to the account their UID resolves to, so `usermod -aG <group> $USER` followed by `sg <group>` works on GARM runners. This works around the duplicate-UID account layout of the runner images; removal is tracked in [#438](https://github.com/canonical/github-runner-operators/issues/438).
Expand Down
7 changes: 5 additions & 2 deletions garm-rockcraft.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ parts:
plugin: go
source: https://github.com/cloudbase/garm
source-type: git
# 2026/10/15: Pin to a post-v0.2.1 commit carrying these changes, until v0.2.2+
# 2026/10/25: Pin to a post-v0.2.1 commit carrying these changes, until v0.2.2+
# is released:
# - golang.org/x/crypto v0.57.0+.
# - the scale set pseudo pool ID fix (PR #866). Without it, OpenStack scale set
Expand All @@ -30,8 +30,11 @@ parts:
# databases upgraded through migration 0003 fail migration 0008 with
# "operator does not exist: text = uuid" and GARM does not start
# (cloudbase/garm#920).
# - the database query timeouts, creating-instance reaper and association
# omission (PR #926). Without them, hung PostgreSQL queries and an ABBA
# deadlock between instance and scale set updates leave runners stuck.
# Switch to source-tag once v0.2.2+ is released.
source-commit: ae281e373bc7568d63c0298dc0406109379216d2
source-commit: 1ad3680f097c3e71fa564ca26a41e62e13cbf91f
source-depth: 1
build-snaps:
- go/1.26/stable
Expand Down
Loading