lava-gitlab-runner goes down because Gitlab API starts returning 403 errors

[igorp-collabora]

The kubernetes pod is still marked as running but none of the gitlab jobs can progress.

These are the error logs from the lava-gitlab-runner pod:

{"timestamp":"2026-07-21T12:43:19.534769Z","level":"WARN","fields":{"message":"Failed to update job status: UnexpectedStatus(403)"},"target":"gitlab_runner::run","span":{"gitlab.job":814833,"name":"run"},"spans":[{"gitlab.job":814833,"name":"run"}]}
{"timestamp":"2026-07-21T12:43:20.305874Z","level":"WARN","fields":{"message":"Failed to update job status: UnexpectedStatus(403)"},"target":"gitlab_runner::run","span":{"gitlab.job":816484,"name":"run"},"spans":[{"gitlab.job":816484,"name":"run"}]}
{"timestamp":"2026-07-21T12:43:21.631412Z","level":"WARN","fields":{"message":"Failed to update job status: UnexpectedStatus(403)"},"target":"gitlab_runner::run","span":{"gitlab.job":821253,"name":"run"},"spans":[{"gitlab.job":821253,"name":"run"}]}
{"timestamp":"2026-07-21T12:43:21.763685Z","level":"WARN","fields":{"message":"Failed to update job status: UnexpectedStatus(403)"},"target":"gitlab_runner::run","span":{"gitlab.job":816491,"name":"run"},"spans":[{"gitlab.job":816491,"name":"run"}]}
{"timestamp":"2026-07-21T12:43:22.498318Z","level":"WARN","fields":{"message":"Failed to update job status: UnexpectedStatus(403)"},"target":"gitlab_runner::run","span":{"gitlab.job":820625,"name":"run"},"spans":[{"gitlab.job":820625,"name":"run"}]}

It happened once before on 2026-02-27.

In both cases restarting the lava-gitlab-runner deployment seems to have fixed the issue.

[sjoerdsimons]

the 403 errors in specific jobs show those jobs have been cancelled, which the runner should handle properly. It shouldn't cause it to stop picking up jobs, unless they for some reason didn't get dequeued

[emanueleaina]

Fwiw we're running ghcr.io/collabora/lava-gitlab-runner:v0.3.6.

[sjoerdsimons]

Fwiw first step here is probably to add a bit more logging to show:

• Amount of jobs that the runner things are queued
• Requests for new jobs

It could be that cancelled jobs aren't properly pruned and thus the runner doesn't ask for new jobs. That said I'd expect to see a lot more 403 replies there. Basically i'm not 100% convinced that the 403's are related to this issue, they may very well be a red herring.

[refi64]

a bit more logging to show:

Logging to be triggered on job request/completion, or something more like metrics? It does sound vaguely like something that a metric would be useful for, though I don't think we have any of those yet...

[refi64]

collabora/gitlab-runner-rs#131

[refi64]

collabora/gitlab-runner-rs#133 might be a cause, if the job runs are long enough such that a set of cancellations (e.g. because of "Auto-cancel redundant pipelines") blocks out slots for a sufficiently long time? At minimum though, it would probably be the cause of the 403s.