rfc9309

Parse and build robots.txt files in Rust per RFC 9309, with crawl-delay and sitemap extensions.

An explainable RFC 9309 robots.txt engine for MoonBit

Pure Standard ML robots.txt parser and matcher (RFC 9309) — dependency-free, MLton + Poly/ML

robots.txt (RFC 9309) parser for Julia — a port of robotstxt-spec

Validate robots.txt from the command line and test whether URLs are allowed for Googlebot, Bingbot or any user agent, with RFC 9309 matching (wildcards, $, longest match). Flags SEO mistakes like blocked CSS/JS or Disallow: /. Zero-dependency Node.js CLI, library and GitHub Action. Free online version by PromoPilot.

Paste a robots.txt and see which AI crawlers can actually read your site, which rule decided it, and whether they fell back to the wildcard group. Implements RFC 9309 precedence.

Free, open-source robots.txt tester — Google's retired tester rebuilt with RFC 9309 matching. Test any URL against Googlebot, GPTBot, ClaudeBot, PerplexityBot and more, with a one-click all-bots matrix.

robots.txt matching the way RFC 9309 and Google define it: most specific rule wins, * and $ wildcards. Fixes urllib.robotparser first-match bug. No dependencies.

Check which AI and search crawlers a robots.txt allows for a path (RFC 9309). Browser + CLI, no dependencies.

Language-neutral spec and conformance fixtures for robots.txt (RFC 9309)

robots.txt (RFC 9309) parser for Swift — a port of robotstxt-spec

Make every fetch(), undici and Playwright request of a Node.js process obey robots.txt (RFC 9309) and Crawl-delay; also a Claude Code hook.

robots.txt parser test cases with verdicts sourced from RFC 9309 and Google's docs

A Node.js framework for creating good bots

robots.txt (RFC 9309) parser for Zig — a port of robotstxt-spec

Test your robots.txt like code: RFC 9309 matching, a crawler access matrix, and CI policy checks.

Read-only mirror of gitlab.com/bart-turczynski/robotstxtr