Skip to content

feat: support blocking AI training crawlers - #349

Merged
harlan-zw merged 1 commit into
mainfrom
fix/ai-training-bots
Oct 12, 2026
Merged

harlan-zw merged 1 commit into
mainfrom
fix/ai-training-bots

Conversation

@harlan-zw

Copy link
Copy Markdown
Contributor

🔗 Linked issue

Resolves #347

❓ Type of change

  • 📖 Documentation (updates to the documentation or readme)
  • 🐞 Bug fix (a non-breaking change that fixes an issue)
  • 👌 Enhancement (improving an existing functionality)
  • ✨ New feature (a non-breaking change that adds functionality)
  • 🧹 Chore (updates to the build process or auxiliary tools and libraries)
  • ⚠️ Breaking change (fix or feature that would cause existing functionality to change)

📚 Description

Site owners cannot opt out of AI training through blockAiBots without also blocking AI search and answer agents.

Adds blockAiBots: 'training' for a curated list of training crawlers and training opt-out tokens. Dedicated search and answer agents stay unblocked by this option. Existing robots.txt and indexing rules still apply. The true setting keeps blocking the complete existing list.

The module owns the list locally, with operator documentation beside each token. The docs explain that Google-Extended also controls Gemini grounding, and CCBot supplies a shared archive.

🤖 AI disclosure: Harlan Agent Kit modified this description. My AI open-source policy.

@harlan-zw harlan-zw added harlan-agent-review Approve automated work for the current issue state or pull request head commit. harlan-agent-review-required Pull request triage requires an adversarial Review for this head commit. harlan-agent-running An Agent holds a Task on this issue or pull request right now. and removed harlan-agent-review Approve automated work for the current issue state or pull request head commit. labels Oct 12, 2026
@harlan-zw

harlan-zw commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 MERGED

Harlan Agent Kit posted this automated review. It is not Harlan's personal review or approval. AI open source policy. Last updated: 2026-10-12 06:18 UTC.

GitHub merged this pull request.

  • Logged (35/100): Unknown string values for blockAiBots silently fall back to blocking the complete AI bot list instead of warning or failing. View code Optional follow-up: Validate the option value at module setup and log an error for any value other than false, true, or 'training' so a typo cannot silently block AI search agents.
    • Ask an agent to verify and fix this finding

571d2373-1733-4630-a54c-4eedce6cb4dd

Selected findings run after merge and open separate pull requests.

@harlan-zw harlan-zw removed the harlan-agent-running An Agent holds a Task on this issue or pull request right now. label Oct 12, 2026
@pkg-pr-new

pkg-pr-new Bot commented Oct 12, 2026

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@nuxtjs/robots@349

commit: 904b562

@github-actions

Copy link
Copy Markdown

Nuxt logo Nuxt Module Size Analyzer

No notable runtime size changes.

Module Client gzip Server raw
@nuxtjs/robots 608 B 173 kB
Package files (6)

Rows overlap and include unused code. These totals do not measure deployed output or npm downloads.
Excludes dependency files, types, source maps, and other non-code files.

Package files Summed gzip Raw Δ gzip
@nuxtjs/robots · export . 7.1 kB 27 kB 🔴 +59 B (+0.8%)
@nuxtjs/robots · export ./content 292 B 503 B —
@nuxtjs/robots · export ./util 466 B 784 B 🟢 -6.2 kB (-93.0%)
@nuxtjs/robots · built code files 43 kB 141 kB 🔴 +1.6 kB (+3.9%)
@nuxtjs/robots · app source files 3.3 kB 7.2 kB —
@nuxtjs/robots · server source files 9.8 kB 24 kB —
Dependencies (8)

Declared dependencies. Their installed size does not show runtime cost.

Package Dependency Requested
@nuxtjs/robots @fingerprintjs/botd catalog:
@nuxtjs/robots @nuxt/kit catalog:
@nuxtjs/robots @vueuse/core catalog:
@nuxtjs/robots consola catalog:
@nuxtjs/robots defu catalog:
@nuxtjs/robots nuxt-site-config catalog:
@nuxtjs/robots nuxtseo-shared catalog:
@nuxtjs/robots ufo catalog:
How this is measured
  • Compare the same app with and without the module.
  • Client: all emitted JS/CSS, summed per-file gzip. This includes lazy chunks.
  • Server: deployed files, including external dependencies. Source maps are excluded.
  • Default module settings, Node server preset. Memory and request speed are outside this report.

Base: main @ 3bf98b6 · 2026-10-12 · differences below 16 B are ignored

@harlan-zw

Copy link
Copy Markdown
Contributor Author

Harlan Agent Kit posted this automated status. AI open source policy.

This release includes breaking changes. Run a manual major release. Automatic releases support patch and minor only.
Includes all unreleased changes since v6.2.4.
Checked head 904b562d0a4c4076b7a0aba69dcab321a9bb021c.

@harlan-zw harlan-zw added harlan-agent-ready The automated Review passed every gate on this head commit. and removed harlan-agent-review-required Pull request triage requires an adversarial Review for this head commit. labels Oct 12, 2026
@harlan-zw harlan-zw changed the title feat(robots): support blocking AI training crawlers feat: support blocking AI training crawlers Oct 12, 2026
@harlan-zw
harlan-zw merged commit fab5a9e into main Oct 12, 2026
11 checks passed
@harlan-zw
harlan-zw deleted the fix/ai-training-bots branch October 12, 2026 06:12
@harlan-zw harlan-zw removed the harlan-agent-ready The automated Review passed every gate on this head commit. label Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

blockAiBots: option to block AI training without blocking AI search and answer agents

1 participant