diff --git a/README.md b/README.md index c8cbf9a1..aa359ee1 100644 --- a/README.md +++ b/README.md @@ -40,9 +40,12 @@ need these features. Install `@nuxtjs/robots` dependency to your project: ```bash -npx nuxi@latest module add robots +npx nuxt module add robots ``` +> [!TIP] +> Using an AI agent? Get the @nuxtjs/robots Skill on [skilld.dev/gh/nuxt-modules/robots](https://skilld.dev/gh/nuxt-modules/robots). + 💡 Configured your robots.txt? Verify what crawlers and AI bots can access with the free [Robots.txt Validator](https://nuxtseo.com/tools/robots-txt-validator), or monitor crawl & indexing status with [Nuxt SEO Pro](https://nuxtseo.com/pro). ## Documentation diff --git a/docs/content/1.getting-started/1.installation.md b/docs/content/1.getting-started/1.installation.md index 56b990de..10778d16 100644 --- a/docs/content/1.getting-started/1.installation.md +++ b/docs/content/1.getting-started/1.installation.md @@ -20,6 +20,16 @@ To get started with Nuxt Robots, you need to install the dependency and add it t :ModuleInstall{name="robots" included-in="@nuxtjs/seo"} +Use Nuxt 4.6 or Nuxt 5. +Use Node 22.22.3+, Node 24.15.0+, or Node 26+. + +Import app functions from `#robots/app`. Import server functions from `#robots/server`. +If you use Nuxt Content, use version 3. + +::tip +Using an AI agent? Get the @nuxtjs/robots Skill on [skilld.dev/gh/nuxt-modules/robots](https://skilld.dev/gh/nuxt-modules/robots). +:: + ## Verifying Installation To ensure the module is behaving as expected, you should first check [`/robots.txt`](http://localhost:3000/robots.txt) is being generated. @@ -31,6 +41,10 @@ However, we want to see what a production environment would look like. For this, it's recommended to use the Nuxt DevTools Robots tab to see the current configuration and how it's being applied. +::note +The DevTools panel needs the `nuxtseo-devtools-host` package. Install it as a dev dependency with `pnpm add -D nuxtseo-devtools-host`, then restart the dev server. You do not add it to `modules`. +:: + The DevTools will show you that in production we're just serving a minimal robots.txt file. ```robots-txt [robots.txt] diff --git a/docs/content/1.getting-started/3.troubleshooting.md b/docs/content/1.getting-started/3.troubleshooting.md index 80f8bb50..99473f25 100644 --- a/docs/content/1.getting-started/3.troubleshooting.md +++ b/docs/content/1.getting-started/3.troubleshooting.md @@ -15,6 +15,7 @@ relatedPages: ## Debugging **Nuxt DevTools.** The best tool for debugging is the Nuxt DevTools integration with Nuxt Robots. This will show you the current robot rules and your robots.txt file. +The panel needs the `nuxtseo-devtools-host` package as a dev dependency. Install it with `pnpm add -D nuxtseo-devtools-host`, then restart the dev server. **Debug config.** You can enable the [debug](/docs/robots/api/config#debug) option which will give you more granular output. This is enabled by default in development mode. diff --git a/docs/content/2.guides/1.robots-txt.md b/docs/content/2.guides/1.robots-txt.md index b1bddce0..1bf30d13 100644 --- a/docs/content/2.guides/1.robots-txt.md +++ b/docs/content/2.guides/1.robots-txt.md @@ -31,21 +31,23 @@ Additionally, the following paths are supported by default: ```bash [Example File Structure] # root directory robots.txt -# asset folders -assets/ -├── robots.txt -# pages folder -pages/ -├── robots.txt -├── _dir/ -│ └── robots.txt # public folder public/ ├── _robots.txt ├── _dir/ │ └── robots.txt +# source directory (app/ on Nuxt 4) +app/ +├── assets/ +│ └── robots.txt +└── pages/ + ├── robots.txt + └── _dir/ + └── robots.txt ``` +The module reads the `assets/` and `pages/` paths from your `srcDir`. On Nuxt 4, the default `srcDir` is `app/`. + **Custom paths.** If you find this too restrictive, you can use the `mergeWithRobotsTxtPath` config to load your `robots.txt` file from any path. diff --git a/docs/content/2.guides/2.how-it-works.md b/docs/content/2.guides/2.how-it-works.md index 4dd4e9ab..8be4b7e9 100644 --- a/docs/content/2.guides/2.how-it-works.md +++ b/docs/content/2.guides/2.how-it-works.md @@ -27,7 +27,7 @@ public ``` This file is generated differently depending on the environment: -- When deploying using `nuxi generate` or the `nitro.prerender.routes` rule, this is a static file. +- When deploying using `nuxt generate` or the `nitro.prerender.routes` rule, this is a static file. - Otherwise, it's handled by the server and generated at runtime when requested. When indexing is disabled a `robots.txt` will be generated with the following content: diff --git a/docs/content/2.guides/5.disable-indexing.md b/docs/content/2.guides/5.disable-indexing.md index bf8d9c15..e8695924 100644 --- a/docs/content/2.guides/5.disable-indexing.md +++ b/docs/content/2.guides/5.disable-indexing.md @@ -96,10 +96,12 @@ to whatever makes sense for your project. If you deploy the same app to multiple domains (e.g. `mydomain.com` and `test.temp-domain.com`), you can disable indexing for specific hosts at runtime using a Nitro plugin. ```ts [server/plugins/robots-domain.ts] +import { getNitroOrigin } from '#site-config/server' + export default defineNitroPlugin((nitroApp) => { nitroApp.hooks.hook('site-config:init', ({ event, siteConfig }) => { - const host = event.headers.get('host') - if (host?.includes('staging') || host?.includes('test')) { + const host = new URL(getNitroOrigin(event)).hostname + if (host.includes('staging') || host.includes('test')) { siteConfig.push({ indexable: false, _context: 'host-indexing' }) } }) diff --git a/docs/content/2.guides/8.bot-detection.md b/docs/content/2.guides/8.bot-detection.md index 5f063425..763421e1 100644 --- a/docs/content/2.guides/8.bot-detection.md +++ b/docs/content/2.guides/8.bot-detection.md @@ -95,7 +95,7 @@ Since server-side detection only uses HTTP headers, detection can only work for You can detect bots inside a Nitro route, middleware, or API handler. ```ts -import { getBotDetection } from '#robots/server/composables/getBotDetection' +import { getBotDetection } from '#robots/server' export default defineEventHandler((e) => { const detection = getBotDetection(e) @@ -116,7 +116,7 @@ When using bot detection in Nuxt, it will use the `User-Agent` header by default ```vue @@ -103,15 +105,18 @@ export default defineNuxtConfig({ }) ``` -Server examples use Nitro auto-imports. On Nuxt 4.6, importing them from `#imports` in a server file fails `nuxt typecheck`. +Import server functions from `#robots/server` and app composables from `#robots/app`. Auto-imports also work. +Do not import server functions from `#imports`. On Nuxt 4.6, that fails `nuxt typecheck`. Turn off indexing for one host of a multi-domain deployment. This is per request: ```ts // server/plugins/robots-host.ts +import { getNitroOrigin } from '#site-config/server' + export default defineNitroPlugin((nitroApp) => { nitroApp.hooks.hook('site-config:init', ({ event, siteConfig }) => { - if (getRequestHost(event).startsWith('staging.')) + if (new URL(getNitroOrigin(event)).hostname.startsWith('staging.')) siteConfig.push({ _context: 'staging-host', indexable: false }) }) }) @@ -123,17 +128,20 @@ Detect bots from the `User-Agent` header: ```ts // server/middleware/block-bots.ts +import { createError, defineEventHandler } from 'nuxt/server' +import { getBotDetection } from '#robots/server' + export default defineEventHandler((event) => { const bot = getBotDetection(event) // { isBot, botName, botCategory, trusted } // Search engines and AI crawlers count as trusted. To block AI crawlers, test bot.botCategory === 'ai'. - if (event.path.startsWith('/api/') && bot.isBot && !bot.trusted) - throw createError({ statusCode: 403 }) + if (event.url.pathname.startsWith('/api/') && bot.isBot && !bot.trusted) + throw createError({ status: 403 }) }) ``` In the app, `useBotDetection()` returns the same fields as computed refs. `useBotDetection({ fingerprint: true })` also runs BotD in the browser. Its result arrives later, so watch `isBot`. -Outside Nuxt, `getBotDetection(headers)` from `@nuxtjs/robots/util` reads only a lowercase `user-agent` key. Pass `Object.fromEntries(request.headers)`. A `Headers` instance returns `isBot: false`. +Outside Nuxt, `getBotDetection(headers)` from `@nuxtjs/robots/util` accepts a `Headers` instance or a plain header object. ## Integrations @@ -142,25 +150,29 @@ Nuxt Content v3 frontmatter and Nuxt i18n path expansion: [references/integratio ## Traps - A path can be crawlable and noindex, or blocked in robots.txt. It cannot be both. Google never reads `noindex` on a disallowed URL and can still index it from links. -- `useRobotsRule(true)` sends `robotsEnabledValue` even when the site is not indexable. Pass `true` only to override that. +- `useRobotsRule(true)` keeps `noindex` when the site is not indexable. To force a rule, pass the string, such as `useRobotsRule('index, follow')`. +- `useRobotsRule(null)` and `useRobotsRule(undefined)` keep the existing rule. - `useRobotsRule()` sets the rule only during SSR. In the browser it only reads the rendered value. - `noai` or `noimageai` makes a path non-indexable to the module, even with `index: true`. `getPathRobotConfig` returns `indexable: false`, and `disallowNonIndexableRoutes` disallows the path. - An `app.baseURL` other than `/` turns off robots.txt and logs an error. Crawlers only read `/robots.txt` at the host root. - Nuxt Content frontmatter `robots` sets the meta tag but not the header, and it needs `zod` 4 in your dependencies. See [references/integrations.md](references/integrations.md). -- The key is `blockAiBots`. Some docs prose spells it `blockAIBots`. +- The key is `blockAiBots`, not `blockAIBots`. - Use `blockAiBots: 'training'` to leave dedicated AI search and user fetch agents unblocked. Existing rules still apply. - Training mode includes Google-Extended, which also controls grounding in Gemini Apps and Vertex AI. ## Version limits -These hold for 6.2.4: - -- `definePageMeta({ robots })` has no effect on Nuxt 4.6. Use a route rule, `defineRouteRules()`, or `useRobotsRule()`. -- A build that calls `useBotDetection()` fails with `Rolldown failed to resolve import "@vueuse/core"` unless the app installs `@vueuse/core`. -- `robots.disableNuxtContentIntegration` has no effect. -- The docs' `robots:config` Nitro hook that edits `ctx.groups` per host leaks. The result is shared, so other hosts get `noindex` until the next `/robots.txt` request. Use the `site-config:init` plugin above. -- `groups[].comment` must be an array. A string prints one `#` line per character. -- `disallowNonIndexableRoutes: true` (deprecated) also writes `Disallow: /_nuxt` and `Disallow: /_nuxt/*` into robots.txt. +Version 7 breaks these 6.x patterns: + +- Nuxt 3 and Node 20 are unsupported. Use Nuxt `^4.6.0 || ^5.0.0` and Node `^22.22.3 || ^24.15.0 || >=26.0.0`. +- `#internal/nuxt-robots` is removed. Import from `#robots/server` or `#robots/app`. +- Hook events are `RequestEvent` from `nuxt/server`. Read `ctx.event.req.headers.get('host')`, not `ctx.event.headers` or `event.node.req`. +- A `robots:config` edit made with `ctx.event` changes only that `/robots.txt` response. For per host indexing, use the `site-config:init` plugin above. +- A route rule beats `definePageMeta({ robots })` on the same path. +- `assets/robots.txt` and `pages/robots.txt` load from `srcDir`. On Nuxt 4, move them into `app/`. +- Nuxt Content v2 is unsupported, and Nuxt fails at startup with a version error. Upgrade to Nuxt Content v3. +- `defineRobotsSchema()` needs `zod` `^4.6.5`. +- The DevTools panel needs `nuxtseo-devtools-host` as a dev dependency: `pnpm add -D nuxtseo-devtools-host`. ## Config diff --git a/skills/nuxtjs-robots/references/integrations.md b/skills/nuxtjs-robots/references/integrations.md index b0c6a01c..b0b30c5f 100644 --- a/skills/nuxtjs-robots/references/integrations.md +++ b/skills/nuxtjs-robots/references/integrations.md @@ -33,7 +33,7 @@ const route = useRoute() const { data: page } = await useAsyncData(`page-${route.path}`, () => queryCollection('content').path(route.path).first()) // The meta tag comes from page.seo. Only useRobotsRule() also sets the header. useSeoMeta(page.value?.seo || {}) -// Content returns null for an unset key. null sends X-Robots-Tag: null and drops the meta tag. +// Content returns null for an unset key. Robots 7 ignores null and undefined, so the default rule stays. useRobotsRule(page.value?.robots ?? undefined)