robots9309

robots.txt matching the way RFC 9309 (and Google) defines it, in one small file with no dependencies.

Python's urllib.robotparser applies the first rule that matches. Many real robots.txt files start with Allow: / and then list what is off limits, so the standard library happily lets a crawler into /admin/. RFC 9309 says the most specific rule wins.

from robots9309 import RobotsTxt, fetch

rules = fetch("https://example.com")            # 4xx = allow all, 5xx/unreachable = disallow all
rules.can_fetch("mybot/1.0", "https://example.com/admin/")
rules.crawl_delay("mybot")                      # non-standard but widely used
rules.sitemaps                                  # Sitemap: lines

• Longest match wins, and Allow wins a tie (RFC 9309 2.2.2).
• * and $ wildcards (Disallow: /*.pdf$).
• User-agent groups: the group of the most specific product token is used, equal groups are merged, * is the fallback.
• Status rules when fetching (RFC 9309 2.3.1): 4xx means "no rules", 5xx or network failure means "disallow everything"; at most 500 KiB is read.
• /robots.txt itself is always allowed; empty Disallow: allows everything; percent-encoding is compared consistently.

Measured result

The robots.txt of demarkstudio.ca starts with Allow: / and then disallows /api/, /app/ and /admin/:

$ python -m robots9309 https://demarkstudio.ca / /en/pricing /app/login /api/v1/me /admin/ /robots.txt --compare
/            allowed     urllib: allowed
/en/pricing  allowed     urllib: allowed
/app/login   DISALLOWED  urllib: allowed      <-- urllib.robotparser disagrees
/api/v1/me   DISALLOWED  urllib: allowed      <-- urllib.robotparser disagrees
/admin/      DISALLOWED  urllib: allowed      <-- urllib.robotparser disagrees
/robots.txt  allowed     urllib: allowed
3 of 6 path(s) where urllib.robotparser gives a different answer

$ python -m pytest -q
...............
22 passed in 0.07s

I found this while building three crawlers (site-seo-crawler, consent-tracker-scan, site-api-mapper): with urllib.robotparser the first one crawled /app/login on my own site. All three now use this module.

CLI

python -m robots9309 <site> [<path> ...] [--ua mybot] [--file robots.txt] [--compare] [--lint]

--compare prints what urllib.robotparser would answer next to each path.

--lint checks the file itself for the mistakes that silently change what crawlers do: rules before any User-agent, Noindex (Google stopped reading it in 2019), relative Sitemap URLs, paths without a leading /, versioned user-agent strings, rules that block the CSS/JS Google needs to render pages, the whole site blocked for *, split groups, unknown directives and files over 500 KiB. Exit code 1 when there is an error, so it can run in CI before a deploy.

My three sites lint clean (0 error(s), 0 warning(s), 0 info for demarkstudio.ca, demo.demarkstudio.ca and marvin.demarkstudio.ca). examples/common-mistakes.txt collects the usual problems:

$ python -m robots9309 https://example.com --file examples/common-mistakes.txt --lint
ERROR    line 2    rule-outside-group   Disallow: before any User-agent line; crawlers ignore it
WARNING  line 4    ua-token             user-agent 'Googlebot/2.1': use the product token only (e.g. 'googlebot'); crawlers match the token, not versions or full strings
WARNING  line 5    blocks-resources     Disallow '/wp-includes/' may block CSS, JS or images that pages need; Google renders pages and can misjudge them without these files
WARNING  line 6    blocks-resources     Disallow '/*.js$' may block CSS, JS or images that pages need; Google renders pages and can misjudge them without these files
ERROR    line 9    noindex              Noindex in robots.txt has not worked in Google since 2019-09-01; use a robots meta tag or an X-Robots-Tag header on the pages instead.
WARNING  line 10   path-no-slash        'private/' should start with '/' (or '*'); as written it may never match
INFO     line 11   non-standard         Crawl-delay is not in RFC 9309 and Google ignores it; Bing and Yandex read it.
ERROR    line 13   sitemap-relative     Sitemap must be a full URL (https://...), got '/sitemap.xml'
3 error(s), 4 warning(s), 1 info

Limitations

• Crawl-delay and Sitemap are read but are not part of RFC 9309; Request-rate is ignored.
• Redirects when fetching are followed by urllib (up to its default limit, the RFC asks for at least five).
• No caching: call fetch once per host and keep the result.

Author

Marvin Palencia, founder of DeMark Studio, Miramichi, New Brunswick, Canada. Portfolio: marvin.demarkstudio.ca

MIT License.