fix(seo): stop serving the homepage for unknown URLs, drop junk sitemap entries - #126
Conversation
…sitemap
The SPA catch-all rewrite in vercel.json sent every filesystem miss to
/index.html, so any nonexistent docs URL answered 200 with the homepage
instead of 404:
$ curl -o /dev/null -w '%{http_code}' https://docs.opsimate.dev/this-page-does-not-exist-xyz123
200
Google records those as soft 404s and spends crawl budget on them. The
rewrite was never needed -- Docusaurus emits a prerendered directory per
route and Vercel resolves clean URLs from the filesystem on its own,
which is why real pages have always served their own content while only
misses fell through. Removing it also lets the generated 404.html, which
the build still produces, actually be served.
Also excludes /search from the sitemap (an empty Algolia results page,
currently reported as "Crawled - currently not indexed") and deletes the
leftover `markdown-page` Docusaurus template, which was a public URL and
a sitemap entry with boilerplate content. Sitemap: 37 -> 35 URLs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WaRCnbZLKMeMMYx4nsUgyS
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
💤 Files with no reviewable changes (2)
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughChangesDocumentation site configuration
Merge Risk: ⚪ Minimal · up to This change stops unknown URLs from serving the homepage and removes two unwanted sitemap entries; no actionable merge-blocking risk remains after normal checks and review. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
1 participant
Found while investigating why pages on
opsimate.devare missing from Google. Companion PR on the website side: OpsiMate/opsimate-website#109.Search Console currently reports 8 pages as "Crawled – currently not indexed", and almost all of them are
docs.opsimate.dev. I inspected/docs/integrations/grafanadirectly: crawl allowed, fetch successful, indexing allowed, self-referential canonical, crawled that morning. There is no technical blocker on the page itself — so this PR goes after the two things that do degrade how Google sees the docs domain as a whole.1. The SPA rewrite turned every 404 into a soft 404
vercel.jsonsent every filesystem miss to/index.html:So any nonexistent docs URL answered 200 with the homepage:
Google logs those as soft 404s and burns crawl budget on them, and the real 404 page could never render.
The rewrite was never needed. Docusaurus prerenders a directory per route and Vercel resolves clean URLs from the filesystem on its own — which is exactly why real pages have always served their own content while only misses fell through to the homepage. The build still emits
404.html, which Vercel now serves.2. Junk in the sitemap
/search— the Algolia results page, no content of its own. It is one of the 8 URLs Google reports as crawled-but-not-indexed. Now excluded viasitemap.ignorePatterns./markdown-page— the leftover Docusaurus starter template ("You don't need React to write simple standalone pages"). It was a live public URL and a sitemap entry. Deleted.Sitemap goes from 37 to 35 URLs.
Verification
Built locally: sitemap is 35 URLs with both junk entries gone,
404.htmlstill generated,/markdown-pageroute no longer emitted, all real routes unchanged.What this does not fix
"Crawled – currently not indexed" is a quality/authority judgment, not a technical fault — there is no switch to flip. This removes the noise Google is currently wading through on the docs domain; it does not by itself get those 8 pages indexed. That comes from content depth, internal linking, and inbound links.
Summary by CodeRabbit
Documentation
Bug Fixes