Skip to content

docs: serve raw Markdown versions of every page for AI agents - #19

Merged
devon-appsmith merged 2 commits into
mainfrom
docs/raw-markdown-for-agents
Aug 28, 2026
Merged

devon-appsmith merged 2 commits into
mainfrom
docs/raw-markdown-for-agents

Conversation

@devon-appsmith

Copy link
Copy Markdown
Contributor

What this does

AI agents can now read the help center as plain Markdown instead of parsing HTML.

  • Every page gets a Markdown twin. After each build, a small plugin (website/plugins/raw-markdown/index.js) copies each page's source Markdown into the build output at the page's public URL plus .md. For example, docs.kite.ai/slack/approvals also serves docs.kite.ai/slack/approvals.md. The homepage is at /index.md.
  • The whole help center in one file. The plugin also writes llms-full.txt at the site root: all 63 pages in sidebar order, each preceded by its canonical URL (~160 KB).
  • The Website Building index gets a twin too. That page is auto-generated by Docusaurus (it has no source file), so the plugin synthesizes a short Markdown page from its title, description, and child links, keeping the "append .md to any URL" promise true for every URL in llms.txt.
  • llms.txt tells agents about all of this. A paragraph near the top and a bullet in Agent guidance explain the .md convention and point to llms-full.txt. Section links also drop their trailing slashes (e.g. /slack/ becomes /slack), so appending .md always forms a valid URL. Vercel already redirects the slashed form, so nothing breaks for anyone following the old links.

Safety net

The plugin derives each page's URL the same way Docusaurus does, then checks every derived URL against the routes Docusaurus actually built. If a future rename or custom slug makes them disagree, the build fails with a message naming the file, instead of silently publishing Markdown at a dead URL.

Verified

  • Full docusaurus build passes; plugin reports 63 pages written.
  • Script check: every docs.kite.ai URL listed in llms.txt has a matching .md file in the build output.
  • llms-full.txt spot-checked for order (matches the sidebar) and canonical URLs.

🤖 Generated with Claude Code

Every docs page now gets a Markdown twin at its URL plus .md, written
into the build output by a small postBuild plugin. The plugin also
writes llms-full.txt (the whole help center in one file, in sidebar
order) and synthesizes a .md page for the generated Website Building
index. The build fails if a derived URL stops matching a real route.

llms.txt now tells agents to append .md to any page URL or fetch
llms-full.txt, and its section links drop trailing slashes so that
appending .md always forms a valid URL.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Aug 28, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
kite-docs Ready Ready Preview Aug 28, 2026 7:27pm

Request Review

The raw-markdown plugin now compares llms.txt against the pages that
actually exist, on every build. A page that is missing from the index,
or an index entry pointing at a page that no longer exists, fails the
build with a message naming the exact file or URL. The summary lines
stay hand-written; only coverage is checked.

To make full coverage the rule, llms.txt now lists all 63 pages, with
the deeper guides nested under their section entries in sidebar order.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@devon-appsmith

Copy link
Copy Markdown
Contributor Author

Added the anti-drift check discussed in review scoping:

  • The build now fails when static/llms.txt disagrees with the docs: a page missing from the index, or an index entry pointing at a page that no longer exists, each produce an error naming the exact file or URL. The hand-written summary lines are untouched; only coverage is checked.
  • To make full coverage the rule, llms.txt now lists all 63 pages (it previously listed only the ~30 section-level ones). Deeper guides are nested under their section entries, in sidebar order, with summaries based on each page's frontmatter description.

Verified by deleting an entry and adding a link to a nonexistent page: the build failed naming both, and went green again once restored.

@devon-appsmith
devon-appsmith merged commit a2260ff into main Aug 28, 2026
3 checks passed

This branch was successfully deployed

1 active deployment
Preview 0054adfa Deployed Aug 28, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant