Skip to content

Setup a restart backoff for the github-runner-manager service - #647

Merged
yhaliaw merged 11 commits into
mainfrom
fix/retry-systemd-service
Nov 17, 2025
Merged

yhaliaw merged 11 commits into
mainfrom
fix/retry-systemd-service

Conversation

@yhaliaw

@yhaliaw yhaliaw commented Oct 30, 2025 •

Copy link
Copy Markdown
Collaborator

Applicable spec:

Overview

Rework the automatic restart of the github-runner-manager.

  • Remove the restart limit posed by StartLimitIntervalSec.
  • Use the systemd restart configurations to add 30 seconds wait between restart with a backoff to 10 mins for repeated failures after 5 times.

Rationale

We have observe the StartLimitIntervalSec was being triggered when the network is very unstable.
This cause the systemd to not restart the github-runner-manager service.

Checklist

  • The charm style guide was applied.
  • The contributing guide was applied.
  • The changes are compliant with ISD054 - Managing Charm Complexity
  • The documentation for charmhub is updated.
  • The PR is tagged with appropriate label (urgent, trivial, complex).
  • The changelog is updated with changes that affects the users of the charm.
  • The application version number is updated in github-runner-manager/pyproject.toml.

The change should be transparent to the users.
The application is not updated in this PR.

@yhaliaw

yhaliaw commented Nov 6, 2025

Copy link
Copy Markdown
Collaborator Author

Currently, integration test with juju deploy containers fails.
@weiiwang01 has tracked this down to a kernel issue.
For now, I think we will wait for a upstream fix.

@yhaliaw
yhaliaw marked this pull request as ready for review November 14, 2025 00:48

@yanksyoon yanksyoon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@cbartz
cbartz requested a review from Copilot November 17, 2025 06:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR configures systemd restart behavior for the github-runner-manager service to handle network instability more gracefully. The key change removes the restart limit and implements a progressive backoff strategy that starts with a 30-second delay and extends up to 10 minutes after 5 consecutive failures.

Key Changes:

  • Removed restart limit (StartLimitIntervalSec=0) to prevent the service from permanently stopping during network instability
  • Added restart backoff configuration with 30-second initial delay, 5-step progression, and 10-minute maximum delay
  • Minor code formatting and linting improvements

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file
File Description
src/manager_service.py Added systemd restart configuration with backoff strategy
tests/integration/conftest.py Reformatted function call for consistency
pyproject.toml Added pylint good-names configuration
github-runner-manager/pyproject.toml Added pylint good-names configuration and removed blank line
.github/workflows/docs.yaml Removed trailing blank line

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@cbartz cbartz left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread pyproject.toml
@yhaliaw
yhaliaw enabled auto-merge (squash) November 17, 2025 06:46
@yhaliaw
yhaliaw merged commit a0cf349 into main Nov 17, 2025
98 of 123 checks passed
@yhaliaw
yhaliaw deleted the fix/retry-systemd-service branch November 17, 2025 07:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants