Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,12 @@ need these features.
Install `@nuxtjs/robots` dependency to your project:

```bash
npx nuxi@latest module add robots
npx nuxt module add robots
```

> [!TIP]
> Using an AI agent? Get the @nuxtjs/robots Skill on [skilld.dev/gh/nuxt-modules/robots](https://skilld.dev/gh/nuxt-modules/robots).

πŸ’‘ Configured your robots.txt? Verify what crawlers and AI bots can access with the free [Robots.txt Validator](https://nuxtseo.com/tools/robots-txt-validator), or monitor crawl & indexing status with [Nuxt SEO Pro](https://nuxtseo.com/pro).

## Documentation
Expand Down
14 changes: 14 additions & 0 deletions docs/content/1.getting-started/1.installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,16 @@ To get started with Nuxt Robots, you need to install the dependency and add it t

:ModuleInstall{name="robots" included-in="@nuxtjs/seo"}

Use Nuxt 4.6 or Nuxt 5.
Use Node 22.22.3+, Node 24.15.0+, or Node 26+.

Import app functions from `#robots/app`. Import server functions from `#robots/server`.
If you use Nuxt Content, use version 3.

::tip
Using an AI agent? Get the @nuxtjs/robots Skill on [skilld.dev/gh/nuxt-modules/robots](https://skilld.dev/gh/nuxt-modules/robots).
::

## Verifying Installation

To ensure the module is behaving as expected, you should first check [`/robots.txt`](http://localhost:3000/robots.txt) is being generated.
Expand All @@ -31,6 +41,10 @@ However, we want to see what a production environment would look like.

For this, it's recommended to use the Nuxt DevTools Robots tab to see the current configuration and how it's being applied.

::note
The DevTools panel needs the `nuxtseo-devtools-host` package. Install it as a dev dependency with `pnpm add -D nuxtseo-devtools-host`, then restart the dev server. You do not add it to `modules`.
::

The DevTools will show you that in production we're just serving a minimal robots.txt file.

```robots-txt [robots.txt]
Expand Down
1 change: 1 addition & 0 deletions docs/content/1.getting-started/3.troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ relatedPages:
## Debugging

**Nuxt DevTools.** The best tool for debugging is the Nuxt DevTools integration with Nuxt Robots. This will show you the current robot rules and your robots.txt file.
The panel needs the `nuxtseo-devtools-host` package as a dev dependency. Install it with `pnpm add -D nuxtseo-devtools-host`, then restart the dev server.

**Debug config.** You can enable the [debug](/docs/robots/api/config#debug) option which will give you more granular output. This is enabled by default in development mode.

Expand Down
18 changes: 10 additions & 8 deletions docs/content/2.guides/1.robots-txt.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,21 +31,23 @@ Additionally, the following paths are supported by default:
```bash [Example File Structure]
# root directory
robots.txt
# asset folders
assets/
β”œβ”€β”€ robots.txt
# pages folder
pages/
β”œβ”€β”€ robots.txt
β”œβ”€β”€ _dir/
β”‚ └── robots.txt
# public folder
public/
β”œβ”€β”€ _robots.txt
β”œβ”€β”€ _dir/
β”‚ └── robots.txt
# source directory (app/ on Nuxt 4)
app/
β”œβ”€β”€ assets/
β”‚ └── robots.txt
└── pages/
β”œβ”€β”€ robots.txt
└── _dir/
└── robots.txt
```

The module reads the `assets/` and `pages/` paths from your `srcDir`. On Nuxt 4, the default `srcDir` is `app/`.

**Custom paths.** If you find this too restrictive,
you can use the `mergeWithRobotsTxtPath` config to load your `robots.txt` file from any path.

Expand Down
2 changes: 1 addition & 1 deletion docs/content/2.guides/2.how-it-works.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ public
```

This file is generated differently depending on the environment:
- When deploying using `nuxi generate` or the `nitro.prerender.routes` rule, this is a static file.
- When deploying using `nuxt generate` or the `nitro.prerender.routes` rule, this is a static file.
- Otherwise, it's handled by the server and generated at runtime when requested.

When indexing is disabled a `robots.txt` will be generated with the following content:
Expand Down
6 changes: 4 additions & 2 deletions docs/content/2.guides/5.disable-indexing.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,10 +96,12 @@ to whatever makes sense for your project.
If you deploy the same app to multiple domains (e.g. `mydomain.com` and `test.temp-domain.com`), you can disable indexing for specific hosts at runtime using a Nitro plugin.

```ts [server/plugins/robots-domain.ts]
import { getNitroOrigin } from '#site-config/server'

export default defineNitroPlugin((nitroApp) => {
nitroApp.hooks.hook('site-config:init', ({ event, siteConfig }) => {
const host = event.headers.get('host')
if (host?.includes('staging') || host?.includes('test')) {
const host = new URL(getNitroOrigin(event)).hostname
if (host.includes('staging') || host.includes('test')) {
siteConfig.push({ indexable: false, _context: 'host-indexing' })
}
})
Expand Down
8 changes: 4 additions & 4 deletions docs/content/2.guides/8.bot-detection.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ Since server-side detection only uses HTTP headers, detection can only work for
You can detect bots inside a Nitro route, middleware, or API handler.

```ts
import { getBotDetection } from '#robots/server/composables/getBotDetection'
import { getBotDetection } from '#robots/server'

export default defineEventHandler((e) => {
const detection = getBotDetection(e)
Expand All @@ -116,7 +116,7 @@ When using bot detection in Nuxt, it will use the `User-Agent` header by default

```vue
<script setup lang="ts">
import { useBotDetection } from '#robots/app/composables/useBotDetection'
import { useBotDetection } from '#robots/app'

const { isBot, botName, botCategory, trusted } = useBotDetection({
fingerprint: true, // detects using botd
Expand Down Expand Up @@ -152,7 +152,7 @@ localStorage.getItem('__nuxt_robots:botd') // returns the cached bot detection r
The properties returned from the composable are all `ref`s. It's important to watch these for changes if you're using fingerprinting, as the results will not be immediately available when the composable is called.

```ts
import { useBotDetection } from '#robots/app/composables/useBotDetection'
import { useBotDetection } from '#robots/app'
import { watch } from 'vue'

const { isBot } = useBotDetection({
Expand All @@ -169,7 +169,7 @@ watch(isBot, (detected) => {
Alternatively you can use the `onFingerprintResult` callback to handle the result when fingerprinting completes.

```ts
import { useBotDetection } from '#robots/app/composables/useBotDetection'
import { useBotDetection } from '#robots/app'

const botd = useBotDetection({
fingerprint: true,
Expand Down
10 changes: 5 additions & 5 deletions docs/content/4.api/3.use-bot-detection.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ The composable provides reactive access to bot detection results with automatic
**Basic detection:**

```ts
import { useBotDetection } from '#robots/app/composables/useBotDetection'
import { useBotDetection } from '#robots/app'

const { isBot, botName, botCategory, trusted } = useBotDetection()
// isBot: ComputedRef<boolean>
Expand All @@ -39,7 +39,7 @@ const { isBot, botName, botCategory, trusted } = useBotDetection()
**With fingerprinting:**

```ts
import { useBotDetection } from '#robots/app/composables/useBotDetection'
import { useBotDetection } from '#robots/app'

const { isBot, botName, botCategory, trusted, reset } = useBotDetection({
fingerprint: true,
Expand All @@ -55,7 +55,7 @@ const { isBot, botName, botCategory, trusted, reset } = useBotDetection({
**Watching for changes:**

```ts
import { useBotDetection } from '#robots/app/composables/useBotDetection'
import { useBotDetection } from '#robots/app'
import { watch } from 'vue'

const { isBot, botName, botCategory } = useBotDetection()
Expand Down Expand Up @@ -112,7 +112,7 @@ interface UseBotDetectionReturn {
On the server, bot detection runs when you use the composable:

```ts
import { useBotDetection } from '#robots/app/composables/useBotDetection'
import { useBotDetection } from '#robots/app'

// Only runs when composable is used
const { isBot, botName, botCategory } = useBotDetection()
Expand All @@ -130,7 +130,7 @@ If the server-side detection is unavailable (e.g. SPA mode), it will fallback to
Client-side fingerprinting is automatic when enabled:

```ts
import { useBotDetection } from '#robots/app/composables/useBotDetection'
import { useBotDetection } from '#robots/app'

const { isBot, botName, botCategory } = useBotDetection({
fingerprint: true,
Expand Down
6 changes: 3 additions & 3 deletions docs/content/5.nitro-api/2.get-bot-detection.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,10 +70,10 @@ export default defineEventHandler(async (event) => {
const detection = getBotDetection(event)

// Block untrusted bots from API routes
if (event.node.req.url?.startsWith('/api/') && detection.isBot && !detection.trusted) {
if (event.url.pathname.startsWith('/api/') && detection.isBot && !detection.trusted) {
throw createError({
statusCode: 403,
statusMessage: 'Bot access denied'
status: 403,
statusText: 'Bot access denied'
})
}
})
Expand Down
4 changes: 2 additions & 2 deletions docs/content/5.nitro-api/3.nitro-hooks.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ For host-based page indexing, use [Site Config hooks](/docs/robots/guides/disabl
```ts [server/plugins/robots-domain.ts]
export default defineNitroPlugin((nitroApp) => {
nitroApp.hooks.hook('robots:config', (ctx) => {
const host = ctx.event?.headers.get('host')
const host = ctx.event?.req.headers.get('host')
if (host?.includes('staging') || host?.includes('test')) {
ctx.groups[0].disallow = ['/']
}
Expand All @@ -72,7 +72,7 @@ This hook allows you to modify the robots.txt content before it is sent to the c

```ts [server/plugins/robots-remove-comments.ts]
export default defineNitroPlugin((nitroApp) => {
if (!process.dev) {
if (!import.meta.dev) {
nitroApp.hooks.hook('robots:robots-txt', async (ctx) => {
// remove comments from robotsTxt in production
ctx.robotsTxt = ctx.robotsTxt.replace(/^#.*$/gm, '').trim()
Expand Down
Loading
Loading