Skip to content

Stop apt-get update hanging the Linux jobs - #98

Merged
dfalster merged 3 commits into
masterfrom
fix/apt-mirror-hang
Aug 19, 2026
Merged

Stop apt-get update hanging the Linux jobs#98
dfalster merged 3 commits into
masterfrom
fix/apt-mirror-hang

Conversation

@dfalster

@dfalster dfalster commented Aug 19, 2026

Copy link
Copy Markdown
Member

azure.archive.ubuntu.com -- the mirror the ubuntu image prefers --
stopped accepting connections from some runners today, and apt-get update consults it for every index target. One job held a runner 26
minutes there; three re-runs did the same while a byte-identical
sibling step passed, so it is per-VM reachability, not this file.

The runner lists archive.ubuntu.com behind it, so a composite action
drops the dead line before every apt call -- including before setup-r,
which installs R's system deps inside the action. Steps that hung on
this now take 11-19s.

`azure.archive.ubuntu.com` stopped accepting connections from some
runners today, and `apt-get update` has no per-connection timeout: it
retried across ~30 index files while holding a runner. One job sat on
that line for 26 minutes, and three re-runs did the same, while the
g++ job on a byte-identical step passed -- the fault is per-VM network
reachability, so nothing in the config distinguishes them.

`apt-get install` already falls back to archive.ubuntu.com. So the
update gets a short timeout and stops being joined to the install with
`&&`, which is what made a hung update block the install entirely.
@dfalster

Copy link
Copy Markdown
Member Author

The evidence, from an instrumented run

Found while trying to get #93 green. ubuntu-latest / clang++ hung on Install Boost headers (Linux) five times across two runs, never reaching a compile; ubuntu-latest / g++ and macos-latest / clang++ passed. The step is byte-identical between the two Linux jobs and references nothing from the matrix, so the config cannot distinguish them — which is what pointed at the runner rather than the workflow.

GitHub logs are not served for in-progress jobs, so the hang was invisible. I put a probe on a throwaway branch: verbose acquire output, a timeout -v 120 around each apt command, and lock/process listings before and after. What it caught:

apt-get update, killed by the probe's 120s timeout, still fetching:

Get:29 http://azure.archive.ubuntu.com/ubuntu noble-security/restricted amd64 Packages [1357 kB]
<- http:102 Status  Message: Waiting for headers
   URI: http://azure.archive.ubuntu.com/ubuntu/dists/noble-security/...
timeout: sending signal TERM to command 'apt-get'

apt-get install, the same host, failing over and succeeding:

<- http:400 URI Failure  Transient-Failure: true  FailReason: Timeout
   Message: Unable to connect to azure.archive.ubuntu.com:http:
Ign:3 http://azure.archive.ubuntu.com/ubuntu noble/main amd64 libboost-dev
<- https:102 Status  Message: Connected to archive.ubuntu.com (104.20.28.246)
Get:2 https://archive.ubuntu.com/ubuntu noble-updates/main amd64 libboost1.83-dev [10.7 MB]
Fetched 10.7 MB in 38s (285 kB/s)

So the mirror the runner image prefers was blackholing, install recovered through apt's mirror list, and update did not — it has Acquire::Retries=10 and no per-connection timeout, over ~30 index files.

Two consequences shaped the fix:

  • && was the amplifier. A hung update meant install never ran, so a recoverable fault became a hung job. Splitting them lets the install proceed on the image's existing package lists, which are good enough for libboost-dev, doxygen and graphviz.
  • The step needed a ceiling. timeout-minutes: 5 bounds whatever this mirror does next, so a future incident costs one red job instead of a runner held for hours.

Also confirmed in the same log, since it would have been the cheaper fix: the image does not ship Boost (ls: cannot access '/usr/include/boost': No such file or directory before the step, present after), so the install is genuinely needed and cannot simply be dropped.

docs.yml had the identical update && install pattern for doxygen graphviz and is fixed the same way. Its hang was never observed — this is the same exposure, not a second incident.

⚠️ The 5/5-versus-0/3 clustering on the clang matrix entry is suggestive but the sample is small, and per the reasoning above the workflow cannot be the cause. I am claiming a mirror reachability fault that varies by VM, not a defect specific to that job.

@dfalster
dfalster merged commit 0f3e6c5 into master Aug 19, 2026
7 checks passed
@dfalster
dfalster deleted the fix/apt-mirror-hang branch August 19, 2026 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant