robots-txt-parser

PHP class for parse all directives from robots.txt files according to specifications

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

Modern robots.txt analyzer with instant analysis, security recommendations, and export capabilities. Built with Qwik and deployed on Cloudflare Pages.

Google's robots.txt parser ported to TypeScript - RFC 9309 compliant

⚙️ A quality `robots.txt` ruleset parser to ensure your application follows the standard specification for the file.

Provides python access to Googles parser for robot.txt files as used by their GoogleBot webscraper.

A dependency-free robots.txt checker that makes it easy for respectful AI agents, crawlers, scrapers, and other bots to follow the rules. Includes fetching, parsing, and caching based on the RFC 9309 industry standard.

A web tool for validating URLs against robots.txt rules. Built with RFC 9309 compliant parsing.

Simple robots txt parser for python

robotstxt gem

Validate robots.txt from the command line and test whether URLs are allowed for Googlebot, Bingbot or any user agent, with RFC 9309 matching (wildcards, $, longest match). Flags SEO mistakes like blocked CSS/JS or Disallow: /. Zero-dependency Node.js CLI, library and GitHub Action. Free online version by PromoPilot.

Free, open-source robots.txt tester — Google's retired tester rebuilt with RFC 9309 matching. Test any URL against Googlebot, GPTBot, ClaudeBot, PerplexityBot and more, with a one-click all-bots matrix.