Skip to content

feat(seo): split llms.txt and close agent discovery gaps - #639

Merged
harlan-zw merged 5 commits into
mainfrom
fix/agentic-seo
Oct 13, 2026
Merged

harlan-zw merged 5 commits into
mainfrom
fix/agentic-seo

Conversation

@harlan-zw

@harlan-zw harlan-zw commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

❓ Type of change

  • 📖 Documentation
  • 🐞 Bug fix
  • 👌 Enhancement
  • ✨ New feature
  • 🧹 Chore
  • ⚠️ Breaking change

📚 Description

The 2026-10-13 agent readiness audit found four gaps that agents and their checkers hit on skilld.dev.

  • llms.txt was a dump. nuxt-ai-ready listed every ai_ready_pages row: 1,478,365 characters and 11,872 links. Most were retired /skills/<owner>/<repo>/<name> URLs that answer 301, 404 or 410. It also linked /llms-full.txt, which redirects back to /llms.txt. Now server/middleware/llms-txt.ts serves /llms.txt ahead of the module's route. It lists the pages sitemap, the setup and API links, and one nested file per sitemap: /llms/skills.txt and /llms/demos.txt. It never reads the retired sitemap, so the files follow the same admission rules as search. Titles and descriptions come from ai_ready_pages. The AI sitemap link stays, so agents still find noindex pages.
  • Track Markdown answered 404. Accept negotiation sends /skills/planning to /skills/planning.md, and the skills route policy answered that path with a JSON 404. A .md URL now passes whenever its page does.
  • /@<login> was a soft 404. Any login rendered an empty 200 page. The page now answers 404 when skilld.dev has no account and no Skills for the login. A collection needs an account, so every Author keeps a page. The new GET /api/users/:login answers whether an account exists.
  • HEAD on /ai-sitemap.xml answered 404. h3 routes a HEAD only to a handler with no method. The route drops its .get suffix and accepts GET and HEAD.

The pull request also sets aiReady.contentSignal to allow search, AI input and AI training. nuxt-ai-ready 3.1.0 writes it to robots.txt as Content-Signal: ai-train=yes, search=yes, ai-input=yes, which the local preview serves. The Nuxt SEO packages move to their latest releases: nuxt-ai-ready 3.1.0, @nuxtjs/sitemap 9.1.0, and the 2026-10-13 patches of the rest. unhead moves to 3.4.3.

contentNegotiation stays false. The site's own handler varies on Accept, Sec-Fetch-Dest, Host. Workers Cache keys on the path, so Host keeps www and apex apart. The module would vary on Accept only. The build prerenders no pages, so every page already negotiates in the Worker.

🤖 AI disclosure: Harlan Agent Kit modified this description. My AI open-source policy.

Picks up the 2026-10-13 patch releases of robots, sitemap, og-image, schema-org, link-checker, site-config, ai-ready and skew-protection, plus unhead 3.4.3. Each new version is excluded from the minimum release age, as the previous patches were.
Accept negotiation sends /skills/planning to /skills/planning.md, and the skills route policy answered that path with a JSON 404. Every track and board page lost its Markdown version.
The module's llms.txt listed every ai_ready_pages row: 1.48M characters and 11,872 links, most of them retired /skills URLs. The root index now lists the pages sitemap and links one nested file per sitemap, so it follows the search admission rules. It also sets the Content Signals for robots.txt.
An Author page rendered an empty 200 for any login, which agent checkers read as a soft 404. h3 routes HEAD only to a handler with no method, so the .get AI sitemap route answered HEAD with the 404 page.
AI Ready 3.1.0 writes the Content-Signal line into robots.txt. 3.0.4 dropped it because of the module order inside @nuxtjs/seo.
@harlan-zw
harlan-zw merged commit 94b2b12 into main Oct 13, 2026
7 checks passed
@harlan-zw
harlan-zw deleted the fix/agentic-seo branch October 13, 2026 12:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant