feat(wiki): align llms.txt exports with the llms.txt v2 spec - #356
feat(wiki): align llms.txt exports with the llms.txt v2 spec#356davida-ps wants to merge 1 commit into
Conversation
The llms.txt v2 spec states that "the links in an llms.txt file should therefore point to LLM-friendly content, such as the markdown versions of pages". Ours pointed at SPA hash routes (`/#/wiki/overview`). A `#...` fragment is never sent to the server, so an agent following a link received the 5KB app shell with none of the page content -- the same dead-end #349 was filed about, just one step further along. - Publish a clean markdown alternate per page at /wiki/<slug>.md, with internal links rewritten to other .md alternates and assets to raw source. This is the content surface agents read, and it makes /wiki/overview.md -- the URL #349 assumed existed -- real. - Rebuild /wiki/llms.txt as a conformant index: exactly one H1, a blockquote summary, then H2 sections that are link lists pointing at the .md alternates. Sections and ordering come from wiki/INDEX.md, so the curated structure is preserved; prose-only sections are dropped. Previously each export carried two H1s and inlined a whole page body under an H2, which is not the shape agents are told to expect. - Add a site-root /llms.txt, which was 404. - Add v2 discovery: a static rel="describedby" to /llms.txt, and a rel="alternate" type="text/markdown" that tracks the active wiki route in the SPA. - Keep /wiki/<slug>/llms.txt unchanged so existing consumers of those URLs are unaffected. The export gate gains a v2 structure test (single H1, blockquote, H2 bodies must be link lists) and now fails on any SPA hash route used as a link target. Verified in a browser that both discovery links render and that the advertised alternate returns real markdown.
MergerNeeds Review The retained per-page Commit |
![baz-reviewer[bot]](https://avatars.githubusercontent.com/in/933528?s=60&v=4){kind=small}
| const extractIndexSections = (indexDoc, docsBySlug) => { | ||
| if (!indexDoc) return []; | ||
|
|
||
| const sections = []; | ||
| let current = null; | ||
|
|
||
| for (const line of indexDoc.content.split(/\r?\n/)) { | ||
| const heading = /^##\s+(.+?)\s*$/.exec(line); | ||
| if (heading) { | ||
| current = { title: heading[1], entries: [] }; | ||
| sections.push(current); | ||
| continue; |
There was a problem hiding this comment.
Duplicate wiki index parsing drifts
extractIndexSections duplicates the markdown section and wiki-link parsing in extractLinkedSections, so the active generators can diverge in link retention and path resolution — should we extract a shared helper using utils/wikiPathHelpers.mjs and let each generator map the resolved entries to its output shape?
Want Baz to fix this for you? Activate Fixer
| const item = /^\s*-\s+\[([^\]]+)\]\(([^)\s]+)\)/.exec(line); | ||
| if (!current || !item) continue; | ||
|
|
||
| const [, label, href] = item; | ||
| const { path: resolvedPath, aboveWikiRoot } = resolveLinkPath(indexDoc.relativePath, splitWikiHash(href).path); | ||
| if (aboveWikiRoot > 0 || !resolvedPath.toLowerCase().endsWith('.md')) continue; | ||
|
|
||
| const slug = resolvedPath.replace(/\.md$/i, '').toLowerCase(); | ||
| const doc = docsBySlug.get(slug); | ||
| if (doc) current.entries.push({ label, doc }); |
There was a problem hiding this comment.
Curated pages silently disappear from llms index
([^)\s]+) rejects valid Markdown links such as - [Overview](overview.md "page title") and - [Asset](<assets/my page.md>), so the loop silently omits entries while buildLlmsIndex still writes a successful /wiki/llms.txt — should we parse these forms or fail when a wiki/INDEX.md entry cannot be represented?
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
`scripts/generate-wiki-llms.mjs` around lines 166-175, update `extractIndexSections` so
its Markdown list-item parser accepts valid link destinations such as optional titles
and angle-bracketed paths containing spaces. Track list entries that cannot be resolved
or represented, and throw a clear error instead of silently omitting them, so
`/wiki/llms.txt` cannot be generated with an incomplete curated index.
| * @returns {Promise<{ pageCount: number, outputFiles: string[] }>} | ||
| */ | ||
| export const generateWikiLlms = async ({ wikiRoot, publicWikiRoot }) => { | ||
| export const generateWikiLlms = async ({ wikiRoot, publicWikiRoot, publicRoot }) => { |
There was a problem hiding this comment.
Stale API contract hides generated artifacts
The exported JSDoc for generateWikiLlms still advertises only { wikiRoot, publicWikiRoot } and { pageCount, outputFiles }, while .md alternates are returned separately in markdownFiles and rootIndexFile is nullable, so consumers see an incomplete contract — should we document optional publicRoot and the distinct outputFiles, markdownFiles, and rootIndexFile: string | null returns?
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
`scripts/generate-wiki-llms.mjs` around lines 220-220, update the JSDoc for
`generateWikiLlms` to include the optional `publicRoot` parameter. Revise the return
type documentation to include `markdownFiles: string[]` and `rootIndexFile: string |
null`, and clarify that `outputFiles` contains the llms.txt outputs while markdown
alternates are tracked separately.
| // Retained unchanged so anything already consuming /wiki/<slug>/llms.txt keeps working. | ||
| if (isWikiIndexSlug(doc.slug)) continue; | ||
| const outputFile = path.join(publicWikiRoot, doc.slug, 'llms.txt'); | ||
| await fs.mkdir(path.dirname(outputFile), { recursive: true }); | ||
| await fs.writeFile(outputFile, buildPageBody(doc, slugSet), 'utf8'); |
There was a problem hiding this comment.
Legacy exports retain dead SPA canonical links
The retained /wiki/<slug>/llms.txt compatibility export still sets Wiki page to https://clawsec.prompt.security/#/wiki/<slug> in buildPageBody, so agents following pageLlmsPath reach the SPA shell instead of the canonical /wiki/<slug>.md content — should we update that metadata to the generated .md URL while preserving /llms.txt?
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
scripts/generate-wiki-llms.mjs around lines 257-261, update the compatibility
`/wiki/<slug>/llms.txt` generation and its `buildPageBody` metadata so the `Wiki page`
link points to the generated `/wiki/<slug>.md` URL instead of the obsolete
`/#/wiki/<slug>` SPA route. Preserve the legacy `llms.txt` files and endpoint paths, but
ensure all advertised page links resolve directly to LLM-readable markdown content.
|
Ran the generator locally on this branch to verify the full export surface before/after — here's the quantified state, including the one gap baz-reviewer flagged: Verified working (ran
The retained gap, quantified (this is what the bot's verdict is about): All 121 per-page Live agent experience today (verified 3 passes): the hash route serves the 5,189-byte app shell with zero page content; the One-line hardening if you want it: (Minor, same theme: the per-page exports also retain the full-page-body The export-time link-checker offer from #349 stands if useful for CI — it would have caught both the original 404 wave and this canonical gap pre-deploy. |
2 participants
User description
The problem
The llms.txt v2 spec says:
Ours pointed at SPA hash routes (
https://clawsec.prompt.security/#/wiki/overview). A#...fragment is never sent to the server, so an agent following a link from our llms.txt gets the app shell and no content:That's the same dead-end #349 was filed about, one step further along: we fixed the 404s, then pointed the links at a target that serves nothing.
Four more gaps against v2, found while reading the spec:
# H1s per export (wrapper + inlined body)## Markdowninlined an entire page body/llms.txtwas 404; noreldiscoveryrel="alternate" type="text/markdown"andrel="describedby"(
/wiki/llms.txtliving at a subpath is fine — v2 explicitly allows it: "a file covers the pages under its path, and the most specific file applies".)What changed
/wiki/<slug>.md— new. A clean markdown alternate per page (122 of them): internal links rewritten to other.mdalternates, assets to raw source. This is the content surface agents read. Side effect:/wiki/overview.md, the URL llms.txt exports ship dead internal links (all 121 exports; repo-relative .md and asset URLs 404) #349 assumed existed, is now real./wiki/llms.txt— rebuilt as a conformant index. One H1, blockquote summary, H2 link-list sections pointing at the.mdalternates. Sections and ordering are derived fromwiki/INDEX.md, so the curated structure is preserved rather than invented; prose-only sections (Summary, Update Notes, Source References) contribute no links and are dropped./llms.txt— new site-root index.rel="describedby"inindex.html, plus arel="alternate" type="text/markdown"inWikiBrowser.tsxthat tracks the active route (needed because this is an SPA, so a static tag would be wrong on every page but one)./wiki/<slug>/llms.txt— unchanged and still published, so anything already consuming those URLs keeps working. No breaking change.Verification
rel="alternate"updates across route changes (/#/wiki/overview→/wiki/overview.md,/#/wiki/es/index→/wiki/es/index.md), and fetching the advertised alternate returns real markdown (200, correct localized content).dist/contains all 122.mdalternates,/llms.txt,/wiki/llms.txt, and the retained per-page exports. 0 SPA hash routes remain as link targets anywhere in the build.scripts/test-wiki-sync-export.mjs4/4 andscripts/test-deploy-pages-checksums.mjspass — GitHub Wiki mirror and advisory pipelines unaffected.npx eslint . --max-warnings 0,npx tsc --noEmit,npm run buildall clean.🤖 Generated with Claude Code
Generated description
Below is a concise technical summary of the changes proposed in this PR:
Align wiki and site
llms.txtexports with the v2 specification by generating markdown page alternates and structured link indexes. Updategenerate-wiki-llms,WikiBrowser, and static discovery metadata so agents receive real content through route-aware markdown links.rel="describedby"and route-awarerel="alternate"metadata, with tests preventing SPA hash routes and broken targets.Modified files (3)
Latest Contributors(2)
.mdalternates with rewritten internal links, rebuild wiki and site indexes with one H1, summaries, and curated link-list sections, while retaining existing per-page exports.Modified files (3)
Latest Contributors(2)