From 4e4edb7d05000090fe259b29d10606a7835f5b12 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 6 Nov 2026 07:25:17 +0000 Subject: [PATCH] Open the README and home page for readers new to the gem The README and the site's home page now start from what Agentry does and define the User-Agent header before anything else. The full method list, Internet Explorer and Safari notes, aliases and matcher recipes, already on the guide and the Bots page, are summarized with links instead of repeated. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VLwjZvdTtyBSS7TRsQr9tf --- README.md | 619 +++++++++++++++--------------------------------- docs/index.html | 82 ++++--- 2 files changed, 235 insertions(+), 466 deletions(-) diff --git a/README.md b/README.md index 85d8a58..14b49b0 100644 --- a/README.md +++ b/README.md @@ -1,459 +1,211 @@ # Agentry -Agentry reads the `User-Agent` header of an HTTP request and tells you who sent it: the browser and its version, -the operating system, the device, or the bot. It is a Ruby gem, with a helper for Rails controllers and views and a -Rack middleware. - -For an iPhone running Safari 18.1, set to French: - -```ruby -browser = Browser.new(request.user_agent, accept_language: request.headers["Accept-Language"]) - -browser.name # => "Safari" -browser.version # => "18" -browser.platform.name # => "iOS (iPhone)" -browser.device.mobile? # => true -browser.bot? # => false -browser.accept_language.first.full # => "fr-FR" -``` - -Site and guide: https://levups.github.io/agentry/ - -## Where it comes from - -Agentry is a fork of [browser](https://github.com/fnando/browser), written by Nando Vieira, which has not changed -since June 2025. It keeps browser's API as it is: the `Browser` class, `require "browser"` and every method below -work the same way, so an app switches by replacing one line of its Gemfile. - -On top of that API, Agentry: - -- answers faster: 25 µs per request instead of 126 µs with browser 6.2.0, and 2 µs once a user agent has been - seen (Ruby 4.0.9 with YJIT, `bench/request.rb`); -- updates its bot list every week from [crawler-user-agents](https://github.com/monperrus/crawler-user-agents) - and [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt). Of the 882 crawler user agents added to - crawler-user-agents since 2024 (test drivers and scanners left out), browser misses 353 and Agentry 11; -- says what kind of bot it is (`bot.category`, `bot.ai?`); -- checks that a request claiming to come from Googlebot, Bingbot, GPTBot or a few others really comes from that - company's published IP ranges (`bot.verified?(ip)`); -- reads the headers Cloudflare can add about a request, and the Client Hints Chromium sends, which tell Windows 11 - from Windows 10. +Agentry is a Ruby gem that tells you who sent an HTTP request: which browser and version, on which operating system +and device, or which bot. It reads the `User-Agent` header, a line of text every browser and crawler sends with each +request, such as `Mozilla/5.0 (iPhone; CPU iPhone OS 18_1 like Mac OS X) … Version/18.1 Mobile/15E148 +Safari/604.1`. Each browser and crawler writes it in its own way, and Agentry knows the formats so that your code +can ask plain questions such as `browser.bot?` or `browser.device.mobile?`. + +Apps use the answers to leave bots out of their analytics, to turn crawlers away, to adapt a page to phones, or to +warn visitors on an old browser. Agentry works in any Ruby program, and gives Rails controllers and views a +`browser` helper for the current request. + +Website and guide: https://levups.github.io/agentry/ + +## Why Agentry + +Agentry is a fork of [browser](https://github.com/fnando/browser), the gem Nando Vieira wrote, which has not changed +since June 2025. Code written for browser runs on Agentry as it is: same `Browser` class, same `require "browser"`, +same methods. Here is what is different: + +- It answers in 25 µs per request instead of 126 µs with browser 6.2.0, and in 2 µs for a user agent the process + has seen before (Ruby 4.0.9 with YJIT, see [Performance](#performance)). +- Its list of bots is updated every week. Of the 882 crawler user agents added to + [crawler-user-agents](https://github.com/monperrus/crawler-user-agents) since 2024 (test drivers and scanners + left out), browser misses 353 and Agentry 11. +- It says what kind of bot sent the request: an AI crawler, a search engine, an assistant fetching a page for a + person, a vulnerability scanner. +- It can check that a request claiming to be Googlebot, Bingbot or GPTBot comes from the IP addresses Google, + Microsoft or OpenAI publish for their crawlers. +- It reads the headers Cloudflare can add to a request, and the Client Hints Chromium browsers send, which tell + Windows 11 from Windows 10. ## Installation -Add it to your Gemfile: +Agentry needs Ruby 3.3 or later. It is tested on Rails 8.0 and 8.1, and works without Rails. Add it to your Gemfile: ```ruby gem "agentry" ``` -If the Gemfile has `gem "browser"`, replace that line: the two gems define the same `Browser` class, so Agentry -refuses to load next to browser. Agentry needs Ruby 3.3 or later, and is tested on Rails 8.0 and 8.1. +If the Gemfile already has `gem "browser"`, replace that line: both gems define the `Browser` class, so Agentry +refuses to load next to browser. Nothing else in your code changes, though some answers do (see +[Switching from browser](#switching-from-browser)). -## Usage - -In Rails, every controller and view gets a `browser` helper for the current request: - -```erb -<% if browser.bot? %> -

Hello, <%= browser.bot.name %>.

-<% end %> -``` - -Anywhere else, build one from a user agent string. `accept_language:` and `headers:` are optional: - -```ruby -browser = Browser.new(user_agent, accept_language: accept_language_header, headers: rack_env) -``` - -Here is everything you can ask: +## A first example ```ruby require "browser" -browser = Browser.new("Some User Agent", accept_language: "en-us") - -# General info -browser.bot? -browser.chrome? -browser.chromium_based? -browser.core_media? -browser.duck_duck_go? -browser.edge? # Newest MS browser -browser.electron? # Electron Framework -browser.firefox? -browser.full_version -browser.ie? -browser.ie?(6) # detect specific IE version -browser.ie?([">8", "<10"]) # detect specific IE (IE9). -browser.known? # has the browser been successfully detected? -browser.unknown? # the browser wasn't detected. -browser.meta # an array with several attributes -browser.name # readable browser name -browser.nokia? -browser.opera? -browser.opera_mini? -browser.phantom_js? -browser.quicktime? -browser.safari? -browser.safari_webapp_mode? -browser.samsung_browser? -browser.to_s # the meta info joined by space -browser.uc_browser? -browser.version # major version number -browser.webkit? -browser.webkit_full_version -browser.yandex? -browser.wechat? -browser.qq? -browser.weibo? -browser.sputnik? -browser.sougou_browser? -browser.epiphany? - -# Get bot info -browser.bot.name -browser.bot.search_engine? -browser.bot.category # :ai_crawler, :search_engine_crawler, :scanner… (nil for a browser) -browser.bot.ai? -browser.bot.user_triggered? # an AI agent acting for a person right now (ChatGPT-User, Claude-User) -browser.bot? -browser.bot.why? # shows which matcher detected this user agent as a bot. -Browser::Bot.why?(ua) - -# Get device info -browser.device -browser.device.id -browser.device.name -browser.device.unknown? -browser.device.blackberry_playbook? -browser.device.console? -browser.device.ipad? -browser.device.iphone? -browser.device.ipod_touch? -browser.device.kindle? -browser.device.kindle_fire? -browser.device.mobile? -browser.device.nintendo? -browser.device.playstation? -browser.device.ps3? -browser.device.ps4? -browser.device.psp? -browser.device.silk? -browser.device.surface? -browser.device.tablet? -browser.device.tv? -browser.device.vita? -browser.device.wii? -browser.device.wiiu? -browser.device.samsung? -browser.device.switch? -browser.device.xbox? -browser.device.xbox_360? -browser.device.xbox_one? - -# Get platform info -browser.platform -browser.platform.id -browser.platform.name -browser.platform.version # e.g. 9 (for iOS9) -browser.platform.adobe_air? -browser.platform.android? -browser.platform.android?(4.2) # detect Android Jelly Bean 4.2 -browser.platform.android_app? # detect webview in an Android app -browser.platform.android_webview? # alias for android_app? -browser.platform.blackberry? -browser.platform.blackberry?(10) # detect specific BlackBerry version -browser.platform.chrome_os? -browser.platform.firefox_os? -browser.platform.ios? # detect iOS -browser.platform.ios?(9) # detect specific iOS version -browser.platform.ios_app? # detect webview in an iOS app -browser.platform.ios_webview? # alias for ios_app? -browser.platform.linux? -browser.platform.mac? -browser.platform.unknown? -browser.platform.windows10? -browser.platform.windows7? -browser.platform.windows8? -browser.platform.windows8_1? -browser.platform.windows? -browser.platform.windows_mobile? -browser.platform.windows_phone? -browser.platform.windows_rt? -browser.platform.windows_touchscreen_desktop? -browser.platform.windows_vista? -browser.platform.windows_wow64? -browser.platform.windows_x64? -browser.platform.windows_x64_inclusive? -browser.platform.windows_xp? -browser.platform.kai_os? -``` - -The lists of what is detected live in [lib/browser/platform.rb](lib/browser/platform.rb), -[lib/browser/device.rb](lib/browser/device.rb) and [bots.yml](bots.yml). +user_agent = "Mozilla/5.0 (iPhone; CPU iPhone OS 18_1 like Mac OS X) AppleWebKit/605.1.15 " \ + "(KHTML, like Gecko) Version/18.1 Mobile/15E148 Safari/604.1" +browser = Browser.new(user_agent, accept_language: "fr-FR,fr;q=0.9,en;q=0.8") -### Aliases - -`require "browser/aliases"` adds shortcuts such as `browser.mobile?` and `browser.tablet?` to the browser object: - -```ruby -require "browser/aliases" -Browser::Base.include(Browser::Aliases) - -browser = Browser.new("Some user agent") -browser.mobile? #=> false -``` - -### Modern browsers - -Agentry has no built-in notion of a modern browser: write one with your own version constraints. For example: - -```ruby -# Expects a Browser instance, -# like in `Browser.new(user_agent, accept_language: language)`. -def modern_browser?(browser) - [ - browser.chrome?(">= 65"), - browser.safari?(">= 10"), - browser.firefox?(">= 52"), - browser.ie?(">= 11") && !browser.compatibility_view?, - browser.edge?(">= 15"), - browser.opera?(">= 50"), - browser.facebook? - && browser.safari_webapp_mode? - && browser.webkit_full_version.to_i >= 602 - ].any? -end +browser.name # => "Safari" +browser.version # => "18" +browser.full_version # => "18.1" +browser.platform.name # => "iOS (iPhone)" +browser.device.mobile? # => true +browser.bot? # => false +browser.accept_language.first.full # => "fr-FR" ``` -### Rails +The file to require is `browser` and the class is `Browser`, the names browser used. `accept_language:` is optional: +it takes the `Accept-Language` header, the languages the visitor's browser asks for. -The gem adds the `browser` helper to every controller and view on its own: +In Rails there is nothing to build. The `browser` helper reads the current request's headers: ```erb -<% if browser.ie?(6) %> -

You're running an older IE version. Please update it!

+<% if browser.device.mobile? %> + <%= render "mobile_menu" %> <% end %> ``` -To use Agentry in a Rails app without that helper, require only the library: - -```ruby -gem "agentry", require: "browser/browser" -``` - -### Accept-Language - -`browser.accept_language` parses the `Accept-Language` header into the languages the visitor accepts, most -wanted first: - -```ruby -browser = Browser.new("Some User Agent", accept_language: "en-us") - -browser.accept_language.class -#=> Array - -language = browser.accept_language.first - -language.code -#=> "en" - -language.region -#=> "US" - -language.full -#=> "en-US" - -language.quality -#=> 1.0 - -language.name -#=> "English/United States" -``` - -As the HTTP spec says, a language without a quality value has 1.0, and one with a quality of 0 is not accepted, -so it is left out. - -### Internet Explorer +Outside Rails, pass the Rack env as `headers:` so that Agentry can read Client Hints and Cloudflare headers too: +`Browser.new(user_agent, accept_language: language, headers: env)`. -Internet Explorer 8 and later have a compatibility view, in which they report an older version. Agentry returns -the browser's own version and ignores the compatibility view; `Browser#msie_version` and -`Browser#msie_full_version` give the version the engine reports. With compatibility view on in IE11: - -```ruby -browser.version -#=> 11 - -browser.full_version -#=> 11.0 - -browser.msie_version -#=> 7 - -browser.msie_full_version -#=> 7.0 - -browser.compatibility_view? -#=> true -``` - -browser 1.0.0 introduced this; before it, there was no way to get the real version. - -### Safari - -An iOS web view, or a site saved to the home screen, is not Safari. Combine the platform and WebKit checks to -find them: - -```ruby -# iPad's Safari running as web app mode. -browser = Browser.new("Mozilla/5.0 (iPad; U; CPU OS 3_2_1 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Mobile/7B405") - -browser.safari? -#=> false - -browser.webkit? -#=> true - -browser.platform.ios? -#=> true -``` +The [guide](https://levups.github.io/agentry/guide.html#usage) lists every method: browsers from Chrome and Safari +to WeChat and Yandex, devices such as the iPad, Kindle, PlayStation and Xbox, and platforms down to each version of +Windows. Version checks take a number or constraints, such as `browser.chrome?(">= 65")` or +`browser.platform.ios?(9)`. ## Bots -`browser.bot?` is true for: - -- an empty user agent; -- a user agent matching `/crawl|fetch|search|monitoring|spider|bot/`; -- a user agent containing one of the keywords of [bots.yml](bots.yml). - -The net is wide, as it was in browser. - ```ruby browser = Browser.new("Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)") -browser.bot? # => true -browser.bot.name # => "ChatGPT" -browser.bot.category # => :ai_crawler -browser.bot.ai? # => true +browser.bot? # => true +browser.bot.name # => "ChatGPT" +browser.bot.category # => :ai_crawler +browser.bot.ai? # => true +browser.bot.user_triggered? # => false ``` -Every Monday, a workflow runs `script/sync_bots.rb`, which adds the bots of -[crawler-user-agents](https://github.com/monperrus/crawler-user-agents) and -[ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) (both MIT) to the end of `bots.yml` and files them -under a category in `bot_categories.yml`. It opens a pull request listing every answer that changes, and a -maintainer merges it. The [Bots page](https://levups.github.io/agentry/bots.html#coverage) counts, per category, -how many of those lists' bots browser and Agentry detect. +`browser.bot?` is true for an empty user agent, for one that matches `/crawl|fetch|search|monitoring|spider|bot/`, +and for one that contains a name from [bots.yml](bots.yml). The net is wide on purpose, as it was in browser. + +Every Monday, a workflow adds the bots of [crawler-user-agents](https://github.com/monperrus/crawler-user-agents) +and [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) (both MIT) to `bots.yml` and files each under a +category. It opens a pull request that lists every answer the change moves, and a maintainer merges it. The +[Bots page](https://levups.github.io/agentry/bots.html#coverage) counts, per category, how many of those bots +browser and Agentry detect. ### Categories -`browser.bot.category` is what the bot says it is. The names follow Cloudflare's verified bot categories where -one fits (`search_engine_crawler`, `ai_crawler`, `ai_search`, `ai_assistant`, `page_preview`, `feed_fetcher`, -`monitoring_and_analytics`, `advertising_and_marketing`, `archiver`, `security`…), plus `ai_agent`, -`http_library`, `scanner` and `browser_automation`. `bot.ai?` is true for the four AI categories, and -`bot.user_triggered?` for `ai_assistant` and `ai_agent`, the ones a person is driving at that moment -(ChatGPT-User, Claude-User). The category is `nil` for a browser, for a bot exception, and for a bot no list -names. - -HTTP libraries (`curl`, `got`, `python-requests`) and scanners (sqlmap, Nikto, Censys) count as bots, as `curl` -and `python-requests` already did in browser. Test drivers (headless Chrome, Playwright, Puppeteer, Selenium) do -not: your own system tests drive a real browser through them, and the headless Chrome of Playwright and Selenium -sends `HeadlessChrome`. They get the `browser_automation` category, so production code can still spot them with +`browser.bot.category` is what the bot says it is, named after Cloudflare's verified bot categories where one fits: +`search_engine_crawler`, `ai_crawler`, `ai_search`, `ai_assistant`, `page_preview`, `feed_fetcher`, +`monitoring_and_analytics`, `advertising_and_marketing`, `archiver`, `security` and a few more. Agentry adds +`ai_agent`, `http_library`, `scanner` and `browser_automation`. `bot.ai?` is true for the four AI categories, and +`bot.user_triggered?` for `ai_assistant` and `ai_agent`: a person is driving them at that moment, as with +ChatGPT-User or Claude-User. The category is `nil` for a browser, for a bot exception, and for a bot no list names. + +HTTP libraries (`curl`, `python-requests`) and scanners (sqlmap, Nikto) are bots. Test drivers (headless Chrome, +Playwright, Puppeteer, Selenium) are not: your own system tests drive a real browser through them, and would +otherwise count as bots. They get the `browser_automation` category, so production code can still catch them with `browser.bot? || browser.bot.category == :browser_automation`. PhantomJS stays a bot, as in browser. ### Your own rules -A matcher is anything that responds to `call(ua, browser)`. This one flags every user agent containing -`externalhit`, under the name `Generic Bot`: +A matcher is anything that responds to `call(ua, browser)`. This one makes every user agent containing +`externalhit` a bot: ```ruby Browser::Bot.matchers << ->(ua, _browser) { ua.match?(/externalhit/i) } ``` -`Browser::Bot.matchers.clear` removes every matcher, the built-in ones included, and this puts the built-in ones -back: - -```ruby -Browser::Bot.matchers += Browser::Bot.default_matchers -``` - -To get the bot detection of browser 2.x back, remove two matchers: - -```ruby -Browser::Bot.matchers.delete(Browser::Bot::KeywordMatcher) -Browser::Bot.matchers.delete(Browser::Bot::EmptyUserAgentMatcher) -``` - -The lists themselves can be changed at runtime: - -```ruby -Browser::Bot.bots.merge!(new_bots_hash) -Browser::Bot.bot_exceptions += new_exceptions -Browser::Bot.search_engines.merge!(new_search_engines_hash) -Browser::Bot.category_by_keyword["faraday"] = :http_library -``` +`browser.bot.why?` returns the matcher that fired. The lists can change at runtime too +(`Browser::Bot.bots.merge!(new_bots_hash)`), and the +[Bots page](https://levups.github.io/agentry/bots.html#matchers) shows how to remove the built-in matchers or go +back to the detection of browser 2.x. ### Verified crawlers -Anyone can send `Googlebot` as their user agent. Google, Bing, OpenAI, Apple, Perplexity and DuckDuckGo publish -the IP ranges their crawlers use, and Agentry ships them, refreshed by the weekly sync -(`script/sync_ip_ranges.rb`). `bot.verified?(ip)` says whether a request comes from the ranges of the company its -user agent names. The check is a binary search in memory, with no network call, and an IP that does not parse -answers `false`. +Anyone can send `Googlebot` as their user agent. Google, Bing, OpenAI, Apple, Perplexity and DuckDuckGo publish the +IP ranges their crawlers use, and Agentry ships them, refreshed by the weekly sync. `bot.verified?(ip)` says whether +a request comes from the ranges of the company its user agent names: ```ruby -browser.bot.owner # => :google for a user agent claiming Googlebot -browser.bot.verified?(request.remote_ip) # => whether the IP is in Google's ranges -browser.bot.verified?(request.remote_ip, reverse_dns: true) # also Google's DNS check: two DNS queries -Browser::Bot::IPRanges.refresh # fetches the ranges again, in a long-lived process +browser.bot.owner # => :google for a user agent claiming to be Googlebot +browser.bot.verified?(request.remote_ip) # => true when the IP is in Google's ranges + +if browser.bot.owner && !browser.bot.verified?(request.remote_ip) + head :forbidden # claims to be a known crawler, comes from somewhere else +end ``` -Pass the IP as your proxies resolve it: `request.remote_ip` in Rails, with `trusted_proxies` set. +The check searches the ranges in memory, with no network call, and an IP that does not parse answers `false`. Pass +the IP as your proxies resolve it: `request.remote_ip` in Rails, with `trusted_proxies` set. -`reverse_dns: true` costs two DNS queries for an IP outside the ranges. The IP's owner controls its reverse DNS, -so a fake Googlebot can make that lookup wait `dns_timeout` (1 second) per nameserver of `/etc/resolv.conf`. A -cache, off by default, remembers each answer, failures included: +`verified?(ip, reverse_dns: true)` adds the DNS check Google, Bing and Apple document, at the cost of two DNS +queries for an IP outside the ranges. A fake Googlebot controls its own reverse DNS and can make that lookup wait +for the timeout (1 second per nameserver of `/etc/resolv.conf`), so a cache, off by default, can remember each +answer, failures included: ```ruby Browser::Bot::IPRanges.dns_cache_size = 10_000 # 0, the default, caches nothing Browser::Bot::IPRanges.dns_cache_ttl = 3600 # seconds Browser::Bot::IPRanges.dns_timeout = 0.5 # seconds per query and nameserver +Browser::Bot::IPRanges.refresh # fetches the ranges again, in a long-lived process ``` ## Cloudflare and Client Hints -Pass the Rack env as `headers:` (the Rails helper and the middleware do it for you) and Agentry reads two more -sources: +When the Rack env is passed as `headers:` (the Rails helper and the middleware do it for you), Agentry reads two +more sources. -- `browser.cloudflare` reads the headers a Cloudflare Transform Rule sets (`verified_bot?`, `bot_category`, - `bot_score`) and `CF-IPCountry` (`country`). Their names can be changed with - `Browser::Cloudflare.header_names = {bot_score: "X-My-Score"}`. -- Chromium's Client Hints (`Sec-CH-UA`, `Sec-CH-UA-Mobile`, `Sec-CH-UA-Platform`, `Sec-CH-UA-Platform-Version`) - refine the platform and the device. `browser.platform.windows11?` needs them. +Behind Cloudflare, a Transform Rule can copy what Cloudflare knows about a request into headers, and +`browser.cloudflare` reads them: -These headers prove something only when every request goes through Cloudflare: if clients can reach your server -directly, they can send the headers themselves. The rule to create, and how to lock the origin, are in the -[guide](https://levups.github.io/agentry/guide.html#cloudflare). Without `headers:`, every answer comes from the -user agent alone, as in browser. `bench/signals.rb` measures what reading them costs. +```ruby +browser.cloudflare.verified_bot? # => true for a bot Cloudflare verified +browser.cloudflare.bot_category # => :search_engine_crawler, with the names of bot.category +browser.cloudflare.bot_score # => 1 (automated) to 99 (human), with Bot Management only +browser.cloudflare.country # => "FR", from CF-IPCountry +``` -## Middleware +The header names can be changed with `Browser::Cloudflare.header_names = {bot_score: "X-My-Score"}`. These headers +prove something only when every request goes through Cloudflare: if clients can reach your server directly, they +can send the headers themselves. The [guide](https://levups.github.io/agentry/guide.html#cloudflare) has the rule to +create and how to lock the origin. -`Browser::Middleware` redirects requests based on the browser: +Chromium browsers no longer put the real Windows, macOS or Android version in their user agent. They send it in +Client Hints (`Sec-CH-UA`, `Sec-CH-UA-Mobile`, `Sec-CH-UA-Platform`, `Sec-CH-UA-Platform-Version`), which Agentry +uses to refine the platform and the device: `browser.platform.windows11?` needs them. Without `headers:`, every +answer comes from the user agent alone, as in browser. The +[guide](https://levups.github.io/agentry/guide.html#client-hints) shows how to ask browsers for the platform +version. `bench/signals.rb` measures what reading these headers costs. -```ruby -use Browser::Middleware do - redirect_to "/upgrade" if browser.ie? -end -``` +## Accept-Language -In Rails, route helpers work too. In an initializer such as `config/initializers/browser.rb`: +`browser.accept_language` turns the `Accept-Language` header into a list of languages, the most wanted first: ```ruby -Rails.configuration.middleware.use Browser::Middleware do - redirect_to upgrade_path if browser.ie? -end +language = Browser.new(user_agent, accept_language: "en-us").accept_language.first + +language.code # => "en" +language.region # => "US" +language.full # => "en-US" +language.quality # => 1.0 +language.name # => "English/United States" ``` -`request` is the `Rack::Request`, for instance to leave a path alone: +As the HTTP specification says, a language without a quality value has 1.0, and one with a quality of 0 is not +accepted, so it is left out. + +## Middleware + +`Browser::Middleware` redirects requests based on the browser, before they reach your app. In Rails, from an +initializer such as `config/initializers/browser.rb`, route helpers work and `request` is the `Rack::Request`: ```ruby Rails.configuration.middleware.use Browser::Middleware do @@ -461,63 +213,70 @@ Rails.configuration.middleware.use Browser::Middleware do end ``` -## Limits and cache - -A user agent or an `Accept-Language` header longer than 2048 bytes raises `Browser::Error`. Both limits can be -raised: +In another Rack app, `use Browser::Middleware do … end` takes the same block. To use Agentry in Rails without the +`browser` helper, require only the library: `gem "agentry", require: "browser/browser"`. -```ruby -Browser.user_agent_size_limit = 4096 -Browser.accept_language_size_limit = 4096 -``` +## Cache, limits and boot -Each process remembers what it found for the last 1,000 user agents (browser, platform, device, bot answers) and -the last 1,000 `Accept-Language` headers, about 1 KB per user agent: a user agent seen before costs about 2 µs. +Each process remembers what it found for the last 1,000 user agents (browser, platform, device and bot answers) +and the last 1,000 `Accept-Language` headers, about 1 KB per user agent. A user agent seen before costs about 2 µs. The cache empties itself when it is full, and when `Browser.matchers`, `Browser::Bot.matchers`, `Browser::Platform.matchers`, `Browser::Device.matchers`, the bot list or its exceptions change. Setting -`Browser.cache_size` empties it too, for example after renaming an entry of `Browser::Bot.bots` in place. +`Browser.cache_size` empties it too, for example after renaming an entry of `Browser::Bot.bots` in place: ```ruby Browser.cache_size = 5_000 # 0 turns it off ``` +A user agent or an `Accept-Language` header longer than 2048 bytes raises `Browser::Error`. Both limits can be +raised with `Browser.user_agent_size_limit = 4096` and `Browser.accept_language_size_limit = 4096`. + The lists and matchers are built on first use: the first request of each process takes about 13 ms more, the following ones 0.2 ms. `Browser.eager_load!` builds them at boot, so that a server that loads the app before -forking (Puma's `preload_app!`, Unicorn, Pitchfork) shares them with every worker. Rails calls it whenever it -eager loads, which production does by default. Elsewhere, call it before the fork: +forking (Puma's `preload_app!`, Unicorn, Pitchfork) shares them with every worker. Rails calls it whenever it eager +loads, which production does by default. Elsewhere, call `Browser.eager_load!` before the fork. -```ruby -Browser.eager_load! -``` - -## Migrating from browser +## Switching from browser Replace `gem "browser"` with `gem "agentry"` in the Gemfile. Nothing else in the code changes. -Some answers do change from the day you switch, because of the fresher bot list: 600 user agents of the corpus -that browser took for people are bots in Agentry, so analytics that leave bots out will count fewer visitors, and -792 user agents that browser called `Generic Bot` get a real name. No browser, platform or device answer changes, -and test drivers still are not bots. +Some answers change from the day you switch, because of the fresher bot list: 600 user agents of the corpus that +browser took for people are bots in Agentry, so analytics that leave bots out will count fewer visitors, and 792 +user agents that browser called `Generic Bot` get a real name. No browser, platform or device answer changes, and +test drivers still are not bots. To go back, remove the calls to what browser does not have (`Browser.cache_size=`, the `headers:` argument, -`bot.category`, `bot.ai?`, `bot.user_triggered?`, `bot.verified?`, `browser.cloudflare`), then put `gem "browser"` back. The -[migration guide](https://levups.github.io/agentry/guide.html#migrating) has the details. +`bot.category`, `bot.ai?`, `bot.user_triggered?`, `bot.verified?`, `browser.cloudflare`), then put +`gem "browser"` back. The [migration guide](https://levups.github.io/agentry/guide.html#migrating) has the details. ## Performance -`bench/request.rb` replays what a web request asks (name, version, `bot?`, platform, device, accept-language) -over 2,000 distinct user agents of the corpus and over 20 repeated ones, and keeps the median of 7 runs. On Ruby -4.0.9 with YJIT, browser 6.2.0 takes 126 µs and 245 objects per request. Agentry takes 25 µs and 52 objects for a -new user agent, and 2 µs and 12 objects for one it has seen. The -[benchmarks page](https://levups.github.io/agentry/benchmarks.html) has all four runs, with and without YJIT. +`bench/request.rb` asks what a web request typically asks (name, version, `bot?`, platform, device, +accept-language) over 2,000 distinct user agents of the corpus and over 20 repeated ones, and keeps the median of 7 +runs. On Ruby 4.0.9 with YJIT: + +| Per request | browser 6.2.0 | Agentry | +|---|---:|---:| +| A new user agent | 126 µs, 245 objects | 25 µs, 52 objects | +| A user agent seen before | 114 µs, 217 objects | 2 µs, 12 objects | + +The [benchmarks page](https://levups.github.io/agentry/benchmarks.html) has the runs without YJIT, the machine, and +why Agentry is faster. To run them again: ```bash -bundle exec rake bench UPSTREAM=path/to/upstream # fnando/browser at dd88933 +bundle exec rake bench UPSTREAM=path/to/upstream # a clone of fnando/browser at dd88933 ``` -It runs `ruby [--yjit] -I lib bench/request.rb --write` for each gem and JIT setting, and records each run (gem, -commit, Ruby, YJIT, machine, date) in `docs/benchmarks.json`. `rake docs:render` then rewrites the site's numbers -from that file. +It runs `ruby [--yjit] -I lib bench/request.rb --write` for each gem and JIT setting, records each run (gem, +commit, Ruby, YJIT, machine, date) in `docs/benchmarks.json`, and `rake docs:render` then rewrites the site's +numbers from that file. + +## How the answers are checked + +Agentry runs browser's own test suite, unmodified, next to its own tests. The repository also holds a corpus of +20,328 user agents, gathered from uap-core, crawler-user-agents and browser's test fixtures. On every pull request, +CI asks both browser and Agentry about each of them and compares the answers. An answer that differs is either +listed in `corpus/expected_differences.jsonl` as an intended change, such as a newly known bot, or fails the build. ## Development @@ -527,9 +286,7 @@ bundle exec rake test bundle exec rubocop ``` -The repository holds a corpus of 20,000 user agents. A change that should not change any answer has to prove it, -by comparing every answer with those of [fnando/browser](https://github.com/fnando/browser) at `dd88933`, as CI -does on every pull request: +To compare every answer with [fnando/browser](https://github.com/fnando/browser) at `dd88933`, as CI does: ```bash ruby -I path/to/upstream/lib script/corpus_snapshot.rb > /tmp/upstream.jsonl @@ -537,9 +294,8 @@ ruby -I lib script/corpus_snapshot.rb > /tmp/agentry.jsonl ruby script/corpus_diff.rb /tmp/upstream.jsonl /tmp/agentry.jsonl ``` -The comparison passes when the answers that differ are exactly those listed in -`corpus/expected_differences.jsonl`. A change meant to change answers rewrites that file with -`ruby script/corpus_diff.rb --write …`, and its diff shows what changed. +It passes when the answers that differ are exactly those in `corpus/expected_differences.jsonl`. A change meant to +change answers rewrites that file with `ruby script/corpus_diff.rb --write …`, and its diff shows what changed. The weekly bot sync (`.github/workflows/bot-sync.yml`) runs these steps, which also work by hand: @@ -553,27 +309,30 @@ ruby script/render_docs.rb # the site's numbers, from docs The numbers on the site's pages (`docs/*.html`) sit between `` markers and come from `docs/benchmarks.json`, `docs/bot_coverage.json`, the bot lists and `corpus/expected_differences.jsonl`. CI fails -when a page differs from what `bundle exec rake docs:render` writes. +when a page differs from what `bundle exec rake docs:render` writes. A new code block on a page needs +`bundle exec rake docs:highlight`. To add a bot, add its user agent to `test/ua_bots.yml`, and its name to `bots.yml` above the synced section, under a lowercase string its user agent contains. A search engine goes in `test/ua_search_engines.yml`, `test/ua_bots.yml` and `search_engines.yml`. +See [CONTRIBUTING.md](CONTRIBUTING.md) before opening a pull request. + ### Releasing Pull requests leave `CHANGELOG.md` alone. A release starts by writing its notes under `## [Unreleased]`, from the pull requests merged since the last tag, without committing them. Then, from an up-to-date `main` with green CI: ```sh -bin/release minor --dry-run # the notes and the changes, nothing written +bin/release minor --dry-run # shows the notes and the changes, writes nothing bin/release minor # or patch, major, or an exact version such as 1.2.0 ``` -`main` takes no direct push, so `bin/release` dates the Unreleased section and bumps the version on a branch -`release/vX.Y.Z`, opens its pull request, which merges itself once CI is green, and waits for it. It then pushes a -signed tag `vX.Y.Z` on the merge commit (`bin/release --tag X.Y.Z` does it if you stopped waiting). The tag starts -the Release workflow. It checks that the tag is signed and on `main`, then publishes the gem to RubyGems through -trusted publishing and creates the GitHub release with the version's notes and the `.gem`. +`main` takes no direct push, so `bin/release` dates the notes and bumps the version on a `release/vX.Y.Z` branch, +opens a pull request that merges itself once CI is green, and waits for it. It then pushes a signed `vX.Y.Z` tag on +the merge commit (`bin/release --tag X.Y.Z` does it if you stopped waiting). The tag starts the Release workflow, +which checks that the tag is signed and on `main`, then publishes the gem to RubyGems through trusted publishing +and creates the GitHub release with the notes and the `.gem`. ## License diff --git a/docs/index.html b/docs/index.html index 490e875..28df731 100644 --- a/docs/index.html +++ b/docs/index.html @@ -3,16 +3,16 @@ -Agentry — the browser gem's API, faster - - - +Agentry: browser and bot detection for Ruby + + + - + @@ -34,16 +34,45 @@

Agentry

-

Same API as browser, faster,and a bot list that stays fresh.

+

Who sent this request?A browser, a phone, a crawler or an AI agent.

+

Agentry is a Ruby gem that reads the User-Agent header, the line of text every browser and crawler sends with a request, and tells you the browser, the operating system, the device or the bot behind it. It is a fork of the browser gem: the same API, faster, with a bot list updated every week.

25 µs per request instead of 126 µs with browser, 2.0 µs once a user agent was seen (YJIT). Of the 882 crawler user agents added to crawler-user-agents since 2024 (test drivers and scanners left out), browser misses 353 and Agentry 11.

gem "agentry" -

Migrating from browserGuideBot coverage

+

GuideComing from browserBots

+
+

Quick start

+
+
+

Gemfile

+
gem "agentry" # Ruby 3.3 or later
+

Rails view

+
<% if browser.bot? %>
+  <p>Hello, <%= browser.bot.name %>.</p>
+<% end %>
+
+
+

Ruby

+
require "browser"
+
+browser = Browser.new(iphone_user_agent)
+browser.name           # => "Safari"
+browser.platform.name  # => "iOS (iPhone)"
+browser.device.mobile? # => true
+
+browser = Browser.new(gptbot_user_agent)
+browser.bot?           # => true
+browser.bot.name       # => "ChatGPT"
+browser.bot.category   # => :ai_crawler
+
+
+
+

In numbers

@@ -58,18 +87,13 @@

In numbers

-

What changes

+

Why Agentry

-

Nothing in your code

-

Every predicate, every require path and the Browser namespace of browser keep working unchanged. The upstream test suite runs unmodified.

+

The browser gem, unchanged

+

Code written for browser runs as it is: every predicate, every require path and the Browser namespace keep working, and browser’s own test suite runs unmodified. Switching is one line of the Gemfile.

Migrating

-
-

Detected once per user agent

-

Each process caches what it detected for the last 1,000 user agents, so a returning visitor costs a hash lookup. Browser.cache_size sizes it.

-

Why it is faster

-

A bot list that stays fresh

Synced every week from crawler-user-agents and ai.robots.txt, both MIT, through a pull request that lists every answer it changes.

@@ -77,9 +101,14 @@

A bot list that stays fresh

Bots by category

-

bot.category tells an AI crawler from a search engine or a scanner, bot.ai? and bot.user_triggered? an agent acting for a person. Test drivers never become bots.

+

bot.category tells an AI crawler from a search engine or a scanner, bot.ai? and bot.user_triggered? an agent acting for a person. Test drivers never become bots, so your system tests don’t either.

Categories

+
+

Real Googlebot or not

+

Anyone can send Googlebot. bot.verified?(ip) checks the IP against the ranges Google, Bing, OpenAI, Apple, Perplexity and DuckDuckGo publish, in memory. Behind Cloudflare, browser.cloudflare reads what Cloudflare verified.

+

Verified crawlers

+

The same answers, proven

Both gems run over 20,328 user agents on every pull request: an answer that differs from browser’s is either listed as intended or fails the CI.

@@ -87,30 +116,11 @@

The same answers, proven

Current Ruby and Rails

-

Ruby 3.3 or later, tested on Rails 8.0 and 8.1. Bundler’s auto-require loads the gem, and the railtie no longer needs a preloaded ActiveSupport.

+

Ruby 3.3 or later, tested on Rails 8.0 and 8.1. A browser helper in every controller and view, Client Hints read for Windows 11, and a cache that makes a returning visitor cost a hash lookup.

Rails

- -
-

Quick start

-
-
-

Gemfile

-
gem "agentry" # instead of gem "browser"
-
-
-

Ruby

-
require "browser"
-
-browser = Browser.new(user_agent, accept_language: "en-us")
-browser.chrome?
-browser.bot?
-browser.device.mobile?
-
-
-