Repository navigation
Conversation
cbartz
marked this pull request as ready for review
October 25, 2026 05:05
cbartz
requested review from
florentianayuwono,
javierdelapuente,
weiiwang01,
yanksyoon and
yhaliaw
as code owners
October 25, 2026 05:05
Pin GARM to the merge of cloudbase/garm#926. It bounds every PostgreSQL query with a statement, lock and connect timeout, reaps runners left in creating when recording them as running fails, and omits associations when writing instances and scale sets to remove an ABBA deadlock between those updates.
cbartz
force-pushed
the
chore/bump-garm-stuck-runners-fix
branch
from
October 25, 2026 05:06
988e00b to
d6a87b6
Compare
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
It changes how PostgreSQL queries and runner cleanup behave in a production workload, and the integration test run that validates it is still pending.
1 open issue
What changed in this PR
This PR bumps the pinned GARM workload commit in garm-rockcraft.yaml to the merge of cloudbase/garm#926. That change targets runners that get stuck, which the PR links to hung PostgreSQL queries and a database deadlock. The changelog gets a dated entry describing the user-facing effect.
Changes:
- Moves the GARM source pin to commit
1ad3680, and documents the reason in the pin comment. - Adds query timeouts for PostgreSQL (migrations are exempt), cleanup for runners left in
creating, and a fix for an instance and scale set update deadlock. - Adds a 2026-10-25 changelog entry for the change.
| File | Description |
|---|---|
| garm-rockcraft.yaml | Moves the GARM pin to the #926 merge commit and adds a comment explaining the fixes it brings in. |
| docs/changelog.md | Adds a dated entry describing the stuck-runner fixes for users. |
Review effort: Lite
| - `garm`: job steps now see `USER` and `LOGNAME` set to the account their UID resolves to, so `usermod -aG <group> $USER` followed by `sg <group>` works on GARM runners. This works around the duplicate-UID account layout of the runner images; removal is tracked in [#438](https://github.com/canonical/github-runner-operators/issues/438). | ||
| - `garm`: keep a removed configurator's provider published until the last scale set that uses it is gone, so removing a `garm-configurator` relation finishes instead of retrying until the five-day drain deadline. The charm records the provider set it starts GARM with, adds providers as configurator applications relate, and removes one only once no scale set refers to it — the provider is dropped on a later reconcile, after its scale sets are deleted. | ||
| - `garm` upgrade note: deployments already stuck before this release do not recover on upgrade, because the removed configurator's credentials are no longer available to the charm. Relate the same configurator application again — its unit name reproduces the missing provider name — so the provider is republished and the drain resumes; a scale set whose runner is stuck in the `deleted` status can also be cleared by hand. | ||
| - `garm`: bump the GARM workload so that runners no longer get stuck. GARM now bounds every PostgreSQL query with a timeout instead of waiting forever on a hung query or row lock, cleans up runners left in `creating` when recording their creation fails, and no longer deadlocks when an instance and its scale set are updated concurrently. |
yanksyoon
approved these changes
Oct 25, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

What this PR does
Bumps the pinned GARM workload commit to the merge of cloudbase/garm#926.
Why we need it
We regularly see runners stuck in a state, most likely because of SQL queries that hang. #926 addresses three causes in GARM's database layer:
creatingare cleaned up when recording them asrunningfails.Checklist
CONTRIBUTING.mdhas been updated upon changes to the contribution/development process (e.g. changes to the way tests are run)docs/changelog.mdwith user-relevant changes(e.g., in
.github/workflows/integration_tests.yaml, ensure themoduleslist is correct)terraform fmtpasses andtflintreports no errorsAGENTS.md.copilot-collections.yamlor.github/instructions/: I re-checked whether theAGENTS.md"12-factor divergences" guidance still matches the upstream copilot-collections guidanceNot applicable or not ticked:
versionstays0.2; it isn't bumped per workload pin (see fix(garm): bump pinned commit to fix scale set pool ID tag length #351).CONTRIBUTING.md, Grafana, Terraform,AGENTS.mdand copilot-collections are untouched.Test plan
./build-garm-rock.sh(Go 1.26 snap). Thegarmbinary is statically linked andgarm versionreports1ad3680.garm) on the rebuilt rock.Review focus
main.mainhas three more commits, two of which change scale set deletion behaviour; they're left for a separate bump.statement_timeout_secs/lock_timeout_secs/connect_timeout_secsoptions, so no config change is needed.AI-generated message.