Skip to content

CI: intermittent TLS interception (DEPTH_ZERO_SELF_SIGNED_CERT) on Azure westus runners — 23/23 failures in one region #15134

Open

Description

@christian-byrne

Summary

Since 2026-08-10T22:44Z, jobs on GitHub-hosted runners have been intermittently failing TLS handshakes to github.com / api.github.com with a self-signed-certificate error. Every single occurrence is on a runner provisioned in Azure region westus.

This is a platform-side problem — nothing in this repo can cause it and no repo change can fix it — but one file here turns it into a hard job failure, and that part we can fix.

Symptom

Two different TLS stacks, same failure.

Node 24 / undici, inside actions/github-script@v9 (19 of 23 occurrences):

GET /repos/Comfy-Org/ComfyUI_frontend/issues/15114/comments - 500 with id UNKNOWN in 53ms
##[error]Unhandled error: HttpError: self-signed certificate; if the root CA is installed locally, try running Node.js with --use-system-ca
    url: 'https://api.github.com/repos/Comfy-Org/ComfyUI_frontend/issues/15114/comments',
  [cause]: TypeError: fetch failed
    [cause]: Error: self-signed certificate
        at TLSSocket.onConnectSecure (node:internal/tls/wrap:1748:34)
      code: 'DEPTH_ZERO_SELF_SIGNED_CERT'

git / libcurl / OpenSSL, during actions/checkout (4 of 23) — job 94006671059, validate-fonts, 04:06:59Z, westus. Retried 3× over 30s, failed every time:

##[error]fatal: unable to access 'https://github.com/Comfy-Org/ComfyUI_frontend/':
server certificate verification failed. CAfile: none CRLfile: none

The chain terminates at depth 0 (self-signed leaf), i.e. the connection is being intercepted. The HTTP 500 is octokit mislabeling a transport failure.

Evidence: it is westus, with a control

Swept every failed job in the repo for Aug 9–12 UTC (432 + 257 + 191 failed jobs) and pulled check-run annotations.

Date (UTC) TLS-symptom jobs Region
Aug 9–10 1 (first at 2026-08-10T22:44:55Z) westus
Aug 11 11 westus ×11
Aug 12 (to 06:43Z) 11 westus ×11

23 of 23 in westus.

Base-rate control: sampling region on successful jobs in the same windows gives westus = 14/80 (17.5%) at 04:00–05:00Z and 9/60 (15%) at 05:42–06:43Z. So westus is ~1-in-6 of normal traffic but 23-of-23 of the failures. Random assignment is ruled out by many orders of magnitude.

It is intermittent, not continuous, within the region — an in-run control: job 94010344278 (changes, run 31563449217) ran in westus at 04:30:04Z and cloned over HTTPS fine, 109 seconds before deploy-and-comment in the same run failed.

Ruled out as the common factor:

  • Runner image — spans 20260720.247.2 (21) and 20260810.271.1 (2).
  • Provisioner — spans 20260707.563 and 20260729.566.
  • A single bad VM — every failing job has a distinct runner_name.
  • Action version — 19 Node/undici vs 4 git/OpenSSL. Two unrelated TLS implementations fail identically.
  • Endpoint — both github.com and api.github.com.
  • Repo config — no HTTP(S)_PROXY, no NODE_EXTRA_CA_CERTS, no self-hosted runners, no containers on the affected jobs.

Not the same as the artifact ECONNRESETs

Separately, 94015375556 (shard 1/16, 05:03Z, eastus2, in a container) and 93969325990 (shard 2/16, 00:16Z, westus3) failed with:

##[error]Unable to download artifact(s): Failed to ListArtifacts: Unable to make request: ECONNRESET

Different region, different endpoint (artifact/blob service), different error class, no certificate involved. Same broad symptom class, not demonstrated to be the same root cause. Filing them together would be unsupported.

Recent occurrences (Aug 12 UTC)

93968741869 deploy-and-comment/Storybook 00:13 · 93969420543 deploy-and-comment-forked-pr 00:17 · 93976619942 comment/Unified Report 00:58 · 93976804925 deploy-and-comment-forked-pr 00:59 · 93982015885 changes/Perf Report 01:30 · 93989973195 changes/Tests Unit 02:19 · 94006671059 validate-fonts 04:06 · 94010567519 (PR #15115) 04:31 · 94014303389 (PR #15114) 04:54 · 94022091028 shard 8/16 05:42 · 94029192656 changes/Website E2E 06:23

Impact

deploy-and-comment is not a required check — required contexts on main are cla-assistant (ruleset 18070802) and test, lint-and-format, e2e-status, website-e2e (ruleset 991238). So today this is red-check noise rather than a merge blocker. But it has also hit validate-fonts, Tests Unit, Website E2E and a Playwright shard, which are closer to the required set.

Proposed local mitigation

.github/actions/upsert-comment-section/action.yaml:29 calls actions/github-script@v9 with no retry and no continue-on-error. A purely cosmetic PR-comment update therefore hard-fails an entire job on one intercepted handshake. That single file is the highest-leverage fix available to us:

  1. Add continue-on-error: true to the comment-upsert step — a failed comment update should never fail a build.
  2. Optionally wrap the octokit call in a small retry with backoff.

Neither addresses the root cause, which is on GitHub's side.

To confirm root cause

A one-line diagnostic step (openssl s_client -connect api.github.com:443 </dev/null 2>&1 | head -30 or curl -vI https://api.github.com) run on failure would capture the served leaf certificate and identify the interception issuer. Worth adding temporarily given this is escalating (1/day → ~11/day).

Activity

  1. coderabbitai commented on Aug 12, 2026

    @coderabbitai
    Contributor
    🔗 Related PRs

    #14568 - fix: back off jobs poller on permanent auth failure instead of erroring [open]


    🧪 Issue enrichment is currently in open beta.

    You can configure auto-planning by selecting labels in the issue_enrichment configuration.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
      auto_enrich:
        enabled: false

    💬 Have feedback or questions? Drop into our discord!

  2. thc1006 commented on Aug 12, 2026

    @thc1006

    Another org in the same window, and it is westus again.

    external-secrets/external-secrets PR #6821, 2026-08-12. One job failed, three attempts:

    08:23:20  dependency-review  westus  fetch -> server certificate verification failed. CAfile: none CRLfile: none
    08:23:37  dependency-review  westus  same
    08:23:57  dependency-review  westus  same, job ends with exit 128
    

    Run 31576463271, job 94054667586, runner 2.336.0, agent 20260707.563, image ubuntu24/20260720.247. The archived log still carries all three.

    Your in-run control reproduces here. Two other jobs on the same PR fetched from westus in between and were fine:

    08:23:50  publish-artifacts (Dockerfile.ubi)  westus  ok
    08:23:52  publish-artifacts (Dockerfile)      westus  ok
    

    13 and 15 seconds after the second failure, before the third, same region.

    Everything else that checked out sat elsewhere and was fine: check-diff centralus, unit-tests westcentralus, license-check eastus2, detect-noop northcentralus.

    This is the git/OpenSSL side only. Nothing in that run uses github-script, so I have no DEPTH_ZERO_SELF_SIGNED_CERT to add to yours. Adding it because a third org in the same region says more than another repository-level report, and your base-rate control is the part worth pointing GitHub at.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions