[minutechreview]

Summary

check_url now handles an HTTP error status in one place, so the 404 browser retry, the 403 DOI fallback and the 405/501 HEAD fallback are no longer written out twice.

Type

• Audit
• Explainer
• Bug fix
• Other

---

Details

• _open(url, method, user_agent, timeout) makes one request and returns ("ok", status).
• _classify_http_error(error, url, method, timeout) turns an HTTPError into broken, the DOI check result or unknown. It returns None when a HEAD request got 405 or 501, and check_url then tries GET.
• _classify_network_error(error) handles URLError (name does not resolve is broken) and TimeoutError/OSError.
• In check_url, a 404 on the first request is retried once with the browser user agent (GET if the first was HEAD). An error from the first request or from the retry then goes through the same classifier, so a rule only has to change once.

No test files changed. To check the behaviour is the same, I ran the old and the new check_url against a local server with 15 status combinations (404 then 200 for a browser, 404 then 403 or 405 or 500 for a browser, HEAD-only 405/501, 403 on a DOI URL, and so on) plus a refused connection. Both versions returned the same (class, detail) for every case.

Checks: tests/test_check_external_links.py 20 passed, ruff check faircode scripts tests and check_em_dash.py pass. One test, test_a_name_that_does_not_resolve_is_broken_and_a_refused_connection_is_unknown, fails on my machine both on main and on this branch because my DNS resolver answers for .invalid names. It is not related to this change.

Linked issue

Closes #1194

[vercel]

@minutechreview is attempting to deploy a commit to the yashkewlani2020-gmailcom's projects Team on Vercel.

A member of the Team first needs to authorize it.

[github-actions]

@yakew7 @ahmdkaml @propcgamer20-png - new PR touching a path you own, please review.