Repository navigation
docs site: /docs/**.mdx serves an indexable parallel copy of every page with no robots directive #12241
Description
One-liner
The /docs/:path*.mdx rewrite (in next.config.mjs, for LLM/agent consumption) serves the full text of every page as text/markdown, 200, with no X-Robots-Tag and no canonical link. That is a second copy of all 403 pages at a crawlable URL.
Measured
$ curl -sI http://localhost:3999/docs/data-modeling/objects.mdx
content-type: text/markdown
(no x-robots-tag)
$ curl -s -o /dev/null -w "%{http_code} %{size_download}\n" http://localhost:3999/docs/data-modeling/objects.mdx
200 28080
Expected
Keep the endpoints — they are the point of the llms.txt / .mdx design and they are how agents read the docs. Just tell search engines which copy is canonical:
- add
X-Robots-Tag: noindexto the.mdxresponses (and to/llms.txt,/llms-full.txt), or an HTTPLink: <https://objectstack.ai/docs/…>; rel="canonical"header pointing at the HTML page — pick one and say why in the PR; - name
llms.txtinrobots.txtso agent crawlers find it deliberately (coordinate with the robots/sitemap card, which owns that file).
Note /llms-full.txt is currently 8 MB — worth a line in the PR on whether that should be split or streamed, but do not expand scope into rewriting it.
Acceptance
-
curl -sI /docs/<page>.mdxshows the chosen directive - the HTML page is unaffected and still canonicalises to itself
-
robots.txtnames both the sitemap andllms.txt
Source
Found in an SEO review of the docs site (apps/docs) run on 2026-08-25, measured against the local dev server and against production. The canonical origin is https://objectstack.ai — maintainer ruling recorded in #10659:
这个仓的文档站规范 URL 是 https://objectstack.ai
Activity
os-zhuang commentedon Aug 25, 2026
Claim: PM loop round 3 (epic PM for #12243)
Session:
session_f9f0958b-ab68-46cc-801c-216aa7ee2107Branch:
claude/issue-12241-mdx-parallel-copyWorktree:
objectstack-issue-12241Domain:
domain:devxFile surface:
apps/docs/next.config.mjsandapps/docs/app/robots.ts(stop on breach; explain in the report)Container & model: S,
mode:subagent,model: opus— no path-derived mandate; the judgement is which directive to use, not volumeClause-②: no
Serial constraints cleared:
apps/docs/app/robots.tslanded in PR #12253 and this card is its declared follow-up (that PR deliberately shipped noDisallowlines and left them here). Disjoint from #12234 and #12237.os-zhuang commentedon Aug 25, 2026
Dev seat claim — os-dev for this card, under the PM claim comment above.
Session:
session_f9f0958b-ab68-46cc-801c-216aa7ee2107(os-dev subagent; shares the epic PM's session id — this seat has no separate one)Branch:
claude/issue-12241-mdx-parallel-copyWorktree:
objectstack-issue-12241PR: #12303 (draft)
File surface held exactly as dispatched:
apps/docs/next.config.mjsandapps/docs/app/robots.ts, nothing else.Generated by Claude Code
os-zhuang commentedon Aug 25, 2026
os-dev-report
{ "issue": 12241, "status": "done", "branch": "claude/issue-12241-mdx-parallel-copy", "pr": "https://github.com/objectstack-ai/objectstack/pull/12303", "premise_still_valid": true, "summary": "Added `X-Robots-Tag: noindex` from `next.config.mjs`'s `headers()` to all four agent-reader surfaces, and made `app/robots.ts` name `llms.txt` / `llms-full.txt` under `Allow:` while deliberately still shipping no `Disallow`. Chose noindex over an HTTP canonical `Link` header: both were built and measured to reach the response (Next does interpolate route params into header values), so the choice turned on substance — only the per-page markdown has an HTML twin to canonicalise to, so a canonical-based fix would leave `/llms.txt` and the 8 MB `/llms-full.txt` (duplicates of no single page) with no directive at all, and a canonical is a hint where noindex is a directive. Two corrections to the card's own premises, both recorded in the PR body rather than smoothed over: (1) the card says 'a second copy' — it is a third as well, because the rewrite destination `/llms.mdx/docs/<slug>` is a real app-router route that answers directly with byte-identical markdown (production: 200 text/markdown, 28080 bytes), so it is covered here too; (2) the card's acceptance box 'the HTML page is unaffected and still canonicalises to itself' is only half satisfiable — the HTML page emits no canonical today, in production or locally, because that is #12234 and it is still in flight. The unaffected half is measured and holds. A third claim I intended to make and dropped after testing it: I assumed next.config.mjs (ESM) could not import SITE_ORIGIN from lib/site.ts; it can, verified by printing the value at config load, so it was not a reason for the choice.", "tests": "PRODUCTION BEFORE (2026-08-25, after the epic's P0 PRs deployed) — `/docs/data-modeling/objects.mdx` 200 text/markdown 28080B, no x-robots-tag, no link; `/llms.mdx/docs/data-modeling/objects` 200 text/markdown 28080B (undirected third copy); `/llms.txt` 200 text/plain 53063B; `/llms-full.txt` 200 text/plain 8052587B; `/docs/data-modeling/objects` 200 text/html 440431B with no canonical link; `robots.txt` 200, 68B, 'User-Agent: * / Allow: / / Sitemap: https://objectstack.ai/sitemap.xml'. LOCAL BEFORE (next dev :3987, worktree at af58a6fbc): identical — all four undirected. LOCAL AFTER: `/docs/data-modeling/objects.mdx` 200 text/markdown + X-Robots-Tag: noindex; `/llms.mdx/docs/data-modeling/objects` 200 text/markdown + noindex; `/llms.txt` 200 text/plain + noindex; `/llms-full.txt` 200 text/plain + noindex; `/docs/data-modeling/objects` 200 text/html with NO X-Robots-Tag (HTML page untouched, full header dump in PR body); `/sitemap.xml` 200 application/xml unchanged; robots.txt now 'Allow: / + Allow: /llms.txt + Allow: /llms-full.txt + Sitemap: ...'. Body still served whole: 200, 28042 bytes, first line '# Object Metadata'. GATE UNION re-run at the final commit 6f8244cef (after merging origin/main 5ce5f8c12), each verdict quoted from the gate's own printed line, exit codes captured before any pipe: check:docs-locale-catch-all '1 top-level dynamic segment(s), 1 guarded' exit 0; check:page-declaration-shape 'OK — 34 page entries across 2185 sources' exit 0; check:published-files 'OK — 69 publishable package(s) of 78' exit 0; check:test-source-alias 'OK — 72 packages with tests scanned' exit 0; check:type-source-resolution 'OK — 93 tsc program(s) across 77 packages' exit 0; check:nul-bytes 'OK (scanned 6815 text file(s) … no raw ASCII control bytes)' exit 0; `pnpm --filter @objectstack/docs typecheck` exit 0. Families derived by `node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack` and re-derived at the final commit — list unchanged, 5 matched. THREE NOT-MEASURED / NARROWING DECLARATIONS, all also in the PR body: (a) next.config.mjs is NOT in the docs typecheck program — `tsc --noEmit --listFiles` enumerates 1239 files, app/robots.ts present, next.config.mjs absent (it is .mjs), so typecheck says nothing about the file carrying the headers; its evidence is the running server, which loaded the config ('Running next.config.mjs took 135ms') and served the headers. (b) Repo-wide `pnpm lint` was NARROWED, declared: eslint run on the two edited files only, `--format json` reports exactly 2 results, both errorCount=0 warningCount=0, neither ignored; the narrowing is safe to a measured fact read from eslint's own config rather than a guess — eslint.config.mjs's docblock records that this repo 'never enables type-aware linting … for ANY file' (no parserOptions.project, no typed rules, behind a planted positive control), so with no cross-file type program a two-file diff cannot move an untouched file's verdict. (c) The shared verify lock was NEVER TAKEN: scripts/pm/os-verify-lock.sh printed 'UNLOCKED (declared) · no usable flock on this host, so the shared verify lock was NEVER taken and NOTHING was serialized' on both runs and instructed that it be declared; every command still went through the wrapper. No ablation was run — this card adds response headers with no gate to prove failing.", "open_questions": [], "out_of_scope_findings": [] }Landing state: draft PR #12303 open against
main, first lineFixes #12241,skip-changesetapplied via the additive REST endpoint and read back after the size-labeler settled (labels nowsize/s,skip-changeset— the additive write survived). No changeset: docs-site only, publishes nothing. CI not waited on, per the dispatch contract.Note, not scope —
/llms-full.txtmeasured at 8,052,587 bytes and growing linearly with the page count. My read, one paragraph in the PR: it should eventually be split, but the trigger should be a measured consumer failure rather than the number, since the file's whole value is being one fetch and splitting trades that for an index plus N requests — which/llms.txtplus the per-page.mdxURLs already are. The thing to watch is agent HTTP clients that cap response bodies well under 8 MB and truncate silently mid-document; if that starts happening the split is forced, and the natural seam is the existing top-level docs sections. Not touched.Generated by Claude Code
os-zhuang commentedon Aug 25, 2026
ACCEPT — PR #12303, 2 files, +67/-1.
X-Robots-Tag: noindexover an HTTP canonicalLink, on two grounds that hold up: only the per-page markdown has an HTML twin to canonicalise to, so a canonical header would leave/llms.txtand the 8 MB/llms-full.txt— the two largest parallel copies on the site — with no directive at all; and a canonical is a hint whilenoindexis a directive. Both were built and measured to reach the response first, so the decision turned on substance, not on what was easy to ship.noindexonly works whilerobots.txtstill allows the fetch that reveals the header. ADisallowwould block the fetch and leave the URLs eligible for URL-only indexing — strictly worse. The PR names them underAllow:on purpose and says in the code that the two files are one mechanism. That comment is worth more than the diff./llms.mdx/docs/<slug>is itself a real route answering byte-identical markdown (measured in production: 200text/markdown, 28080 bytes). Both markdown URLs per page are now covered.metadataBaseand no canonical link on any page #12234, which was in flight. It lands with PR fix(docs): set metadataBase and emit one absolute canonical per page type #12305, and the box closes then, not here.next.config.mjscould not importSITE_ORIGIN) — it can, verified at config load. Dropping a convenient-but-false justification is the behaviour I want to see.