Quieter request logs, and a robots.txt that keeps crawlers to /bibrefs

[tobie]

Two small follow-ups to #965 and #966, from what the production logs showed once they went live.

1. Quieter request logs

Two sources of noise:

• Clever Cloud's monitoring agent (Telegraf, from 127.0.0.1) polls / every minute and gets a 404, which was logged at warn.
• Bots asking for /, /robots.txt and the like get 404s too, also at warn.

Now:

• 404s are logged at info. Other 4xx stay at warn, 5xx at error.
• Requests whose socket address is the loopback interface are not logged at all (neither the request completed line nor the debug request received one). In production every real request comes through the load balancer, so a loopback socket address can only be the platform's own agent. The check is on the socket address, not on req.ip, so a forwarded X-Forwarded-For: 127.0.0.1 doesn't hide a request.

2. robots.txt and noindex

/robots.txt is served (it was a 404) with:

User-agent: *
Allow: /bibrefs
Disallow: /

and every response carries X-Robots-Tag: noindex.

/bibrefs has to stay open to crawlers: ReSpec builds the references section of a spec in the browser by calling it, and search engines render JavaScript when they index a page while honouring robots.txt for the requests the page makes, so blocking it would strip the bibliography from every ReSpec draft in their index. Since #966 the full dump is served from a cache with an ETag, so a crawler fetching it costs nothing to speak of. Search, reverse lookup and metadata are not needed to render anyone's page and are off limits. The noindex header keeps any of the API's responses, allowed or not, from ever showing up as a search result.

Both documented in DEPLOYMENT.md.

Testing

Full suite passes (366,309 tests), including new tests for robots.txt (status, content type, caching, exact body) and for the noindex header on every kind of response (health, robots, lookups, full dump, metadata, 404). Checked by hand with the server bound on a non-loopback address that / and /robots.txt log at info with their 404, a throwing request still logs at error with the stack, and the same requests over 127.0.0.1 produce no request lines.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Ln7Js7WaWpJHDVkEUwtFkA