Repository navigation
Stop carrying the crawler's name in a field called is_crawler - #3621
Merged
Merged
Conversation
The ipapi.is worker returned the crawler's NAME through `is_crawler` — "BaiduSpider", "GoogleBot" — while every consumer typed it `boolean`. Measured on production over an hour: `false` on 29,869 sessions, then "BaiduSpider" 1,637, "FacebookBot" 35, "GoogleAdsBot" 8, "AhrefsBot" 6, "GoogleBot" 1. So `ipInfo.isCrawler === true` matched nothing, and two files in captcha-private carry long comments working around it. The reputation sidecar that replaced it answers a real boolean plus a separate `crawler` field, so `isCrawler` is now genuinely a boolean and the name lives in `crawlerName`. No accommodation for the old worker's shape: it is being retired, not run alongside. Part of prosopo/captcha-private#5548. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
forgetso
force-pushed
the
crawler/name-field
branch
from
October 26, 2026 20:46
655ca1c to
5604b45
Compare
forgetso
enabled auto-merge (squash)
October 26, 2026 20:54
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
ipapi.isreturns the crawler's name in a field calledis_crawler—"BaiduSpider","GoogleBot"— even though every consumer declares it a boolean. Measured on production over one hour:falseon 29,869 sessions, then"BaiduSpider"1,637,"FacebookBot"35,"GoogleAdsBot"8,"AhrefsBot"6,"GoogleBot"1.So
isCrawler === trueis dead code, and two files in captcha-private carry long comments explaining the workaround.What this changes
isCrawleris now an actual boolean. The name goes incrawlerName.The two backends disagree on the wire and both are live: our reputation sidecar returns a real boolean
is_crawlerplus its owncrawlerfield, while the vendor worker (still used by the job runner) sends the name throughis_crawler. SoIPApiResponse.is_crawleris typedboolean | string— describing what actually arrives rather than what we wish did — and a smallcrawlerName()helper reads whichever is present.Nothing that makes a decision changes: every existing consumer treats
isCrawleras truthy/falsy, and both a name string andtrueare truthy.Tests
Five new cases: the vendor's name, the sidecar's field, a crawler with no name, a non-crawler, and an empty string. 34 passing in the package, type-check clean, biome clean.
Part of prosopo/captcha-private#5548.
🤖 Generated with Claude Code