README.md

Research Wiki Tooling

Everything you need to run your own automated research wiki — a static site that ingests academic papers and journal articles, synthesizes article and concept pages, and publishes to GitHub Pages. Powered by an AI agent.

All scan parameters, paths, build commands and the agent integration live in wiki.config.yaml at the repo root; the concept vocabulary lives in concepts.registry.yaml. Those two files are the only ones you need to edit to point the pipeline somewhere else. Everything below describes the workflow.

What this tooling does:

• Daily scans arXiv (cs.CY/cs.HC/cs.CL/cs.AI + physics.ed-ph), EdArXiv and PsyArXiv (by subject, e.g. Educational Psychology) for new papers in your domain
• Weekly journal scans pull open-access articles from journal RSS feeds (CAEAI, CEAO, BJET, Frontiers in Psychology, IJETHE, IJAiEd, JOIDAT — see wiki.config.yaml → journal_scan.feeds). Each feed carries its own freshness window, so a batch-publishing journal is not blanked out between mailings.
• Ingests papers into a structured markdown wiki: one content/en/articles/<slug>.md per paper, one content/en/concepts/<slug>.md per broad topic, one content/en/faqs/<slug>.md per curated question and one content/en/resources/<slug>.md per external tool or collection, with cross-links and typed metadata (facet concept fields plus phrase fields; the old free-form tag list is retired)
• Publishes an Astro 7 static site with Pagefind full-text search, sitemap, RSS, and agent-ready llms.txt / llms-concepts.txt / llms-full.txt
• Publishes offline EPUB and PDF versions (aied.epub, aied.pdf) with a clickable, numbered table of contents and a Notice page
• Deploys to GitHub Pages with a single git push (GitHub Actions)

Live example: AI in Education Knowledge Base — 1,100+ article summaries, 190 concept pages and 18 FAQs on AI in education, auto-updated weekdays at 9 AM and Sundays via RSS.

Quick Start

1. Clone this repo (or copy the tooling/ directory into yours)

git clone https://github.com/edtechdev/aied.git
cd aied

This repo is a working wiki. The tooling/ directory contains everything needed to run your own. Copy it into a new repo to start fresh:

mkdir my-research-wiki && cd my-research-wiki
cp -r ../aied/tooling/* .

2. Add the Astro site files

The wiki is published as an Astro static site. Copy these from the live wiki repo:

cp ../aied/astro.config.mjs .
cp ../aied/package.json .
cp -r ../aied/src .
cp -r ../aied/public .
mkdir -p .github/workflows
cp ../aied/.github/workflows/astro-build.yml .github/workflows/
cp ../aied/.github/workflows/astro-deploy.yml .github/workflows/
npm install

Adjust astro.config.mjs (base should match your repo name) and the site URL — both are read from site.config.json, so edit that file rather than the code.

3. Customize for your domain

File	What to change
site.config.json (repo root)	Single source of truth for site metadata — name, short name, brand, URL, base path, repo/issues URLs, editor name + contact, license, theme colors. All code + tooling read it; do NOT hardcode these elsewhere
concepts.registry.yaml (repo root)	Single source of truth for the concept vocabulary: every slug with its title and synonym phrases, the sidebar sections, the merged/redirect map, and the never-link list. Every other concept artifact is generated from it
wiki.config.yaml → agent:	Your AI agent's name and the tools that provide each pipeline capability (run_python, fetch_url, shell, …)
tooling/SCHEMA.md	Your domain, tag taxonomy, page conventions (canonical schema)
wiki.config.yaml (repo root)	Single source of truth for the pipeline: content paths, build + gate commands, arXiv categories/keywords, journal RSS feeds, relevance filter, and the agent: block that maps the pipeline's capabilities onto your AI agent's tools
cron/daily-scan-prompt.md	Daily cron prompt — update domain references
cron/weekly-rss-scan-prompt.md	Weekly journal cron prompt
scripts/fetch-rss-feeds.py	The FEEDS dict — add/remove journals
skills/research/wiki-inline-links/	The inline-link HARD GATE pass (term→slug dictionary + scanner + check_list_formatting.py)

4. Initialize your wiki

Create the directory structure and starter files:

mkdir -p articles concepts raw/papers
cp tooling/example/log.md .
cp tooling/example/articles/* content/en/articles/
cp tooling/example/concepts/* content/en/concepts/
cp tooling/example/raw/papers/* raw/papers/

Then copy the starter concept registry to the repo root and rewrite it for your domain (one entry per concept page, each with its synonym phrases) — this is the concept vocabulary the site's sidebar and the linking pass both read:

cp tooling/example/concepts.registry.example.yaml concepts.registry.yaml

python3 tooling/scripts/check_concepts.py     # registry vs content/en/concepts/ vs generated views
python3 tooling/scripts/gen-concept-artifacts.py   # regenerate the views from the registry

5. Set up the AI agent cron jobs

Two cron jobs (see cron/ for the prompts):

Job	Schedule	Sources
Daily scan	Weekdays 9:00 AM	arXiv (cs.CY, cs.HC, cs.CL, cs.AI, physics.ed-ph), EdArXiv, PsyArXiv (Educational Psychology)
Weekly journal RSS	Sundays 8:00 AM	Journal RSS feeds (open-access only)

Create them with agent cron create using the prompt files, setting workdir to your wiki path.

Data sources and acknowledgement

The scan queries arXiv through tooling/scripts/arxiv_fetch.py, which is the only place in this project that talks to arXiv. It holds an exclusive cross-process lock for the duration of each request and waits at least three seconds between requests, because arXiv's API terms of use limit an operator to one request every three seconds and a single connection at a time, counted across all machines under that operator's control. Anything else that reaches arXiv — the browser tool, a curl, an ad-hoc script — must call python3 tooling/scripts/arxiv_fetch.py --reserve --hold <seconds> first, which claims the same window.

Prefer the endpoint arXiv blesses for the job: OAI-PMH (--oai) to catch up on metadata in bulk, RSS (--rss) for new articles in a category, and the legacy API (--query) for a targeted query. Never fetch arXiv in parallel, and never fetch full texts or PDFs from arXiv — link to the abstract page instead.

Thank you to arXiv for use of its open access interoperability.

Wiki Structure

wiki/
├── content/                  # Every markdown page, one tree:
│   ├── en/articles/          #   one page per paper (synthesis, findings, citations)
│   ├── en/concepts/          #   one page per broad topic (synthesizes multiple papers)
│   ├── en/faqs/              #   curated FAQ pages (question-and-answer)
│   ├── en/resources/         #   external tools, collections, instruments, open formats
│   └── <locale>/             #   translations, mirroring the default collections
├── raw/papers/        # Raw source text (arXiv, PDFs, RSS abstracts)
├── concepts.registry.yaml  # Concept vocabulary: slugs, titles, phrases, sections, redirects
├── wiki.config.yaml   # Pipeline config: paths, gates, scan sources, journal feeds, agent block
├── tooling/           # Reusable tooling: SKILL.md (research-wiki), SCHEMA.md, README, cron/, scripts/
├── skills/            # Mirrored AI agent skills: research/wiki-inline-links/ (inline-link + list-formatting HARD GATE)
├── src/               # Astro pages: index, journal, search, faq, article/concept/faq templates; lib/jsonld.ts
├── public/            # llms.txt, llms-concepts.txt, llms-full.txt, robots.txt
├── astro.config.mjs   # Astro config (base, pagefind, sitemap)
├── package.json       # Astro 7 + pagefind + sitemap + rss
├── log.md             # Audit log of every wiki action (local, gitignored)
├── concept-screen.yaml # Concept-screen decisions per ingested article (local, gitignored)
└── .github/workflows/ # Build & deploy to GitHub Pages

Page Types

• Article pages (content/en/articles/<slug>.md) — one per paper. Frontmatter → synthesis blockquote → ## Key Findings (5–7 contiguous items) → 3–4 body sections → ## What this means for practice (3–5 bullets) → ## Limitations (2–6 bullets, optional) → ## Citation (always the last section; the paper's title is the only hyperlinked text). audit-article-sections.py is the gate on the order and the counts. No ## Connected … sections on article pages (rule change 2026-09-27) — the curated lists belong on the concept/resource/FAQ pages, and the article's derived view is its facet metadata table.
• Concept pages (content/en/concepts/<slug>.md) — one per broad topic that synthesizes multiple articles. Frontmatter → synthesis → ## Questions to Consider (required) → ## Introduction → body sections with wikilinks → Connected Concepts → Connected Articles.
• FAQ pages (content/en/faqs/<slug>.md) — one per curated question-and-answer. Frontmatter → question heading → narrative answer with wikilinks (can link to concepts, articles, and other FAQs). No sources/citation. Listed on the journal page (❓), indexed in llms files, and linked from concept pages via connected_faqs (article pages stopped carrying that field on 2026-09-27; FAQ pages inline-link the articles they draw on).
• Resource pages (content/en/resources/<slug>.md) — one per external resource worth pointing readers at: a tool, a collection, an assessment instrument, an open format. Frontmatter carries url (required), resource_type, access (full-text accessibility for the reader), license and last_verified; the body is a fact block → short prose → Metadata table → Connected Concepts / Connected Resources. No ## Citation — a resource is not a paper. Listed on /resources/, in index.md under ## Resources, in journal.md with a 🧰 badge, and in the closing chapter of the EPUB/PDF. python3 tooling/scripts/check-resource-links.py re-checks every URL and flags the last_verified dates that need bumping.

Inter-page links use [[wikilink]] syntax, rendered as hyperlinks by the Astro templates.

Inline hyperlink rule (HARD GATE): after creating/enriching any article or concept page, run the inline-link pass (see wiki-inline-links skill) — hyperlink every concept mentioned in the page body narrative to its concept page (aggressive, including conceptually-similar phrases), and fix self-links, links in ## headings, same-text links [[slug|slug]], and broken links. Verify 0 self-links, 0 heading links, balanced brackets, and 0 broken links before npm run build. A green build does NOT substitute for this editorial pass.

Number-grounding rule (HARD GATE): every numeric token in an article body must appear in that page's raw source. python3 tooling/scripts/verify-number-grounding.py --changed (or a list of slugs) reports the tokens it cannot locate; commit-if-green.sh runs it over the article pages being committed. A reported token means investigate: leading-dot p-values (.001), digits broken across PDF lines and values duplicated by an HTML conversion (0.6950.695) are extraction artifacts, and the fix is to confirm the number in the raw, not to rewrite the claim to match a broken extraction. A statistic that genuinely is not in the source gets deleted or requalified.

Article-section rule (HARD GATE): python3 tooling/scripts/audit-article-sections.py --changed checks the section order (citation last), that the practice section immediately precedes Limitations, and the bullet counts (practice 3–5, Limitations 2–6). Trim an over-long Limitations list by folding weaker bullets into stronger ones rather than discarding evidence.

Commit route (HARD GATE): commit through bash tooling/scripts/commit-if-green.sh <message-file> <paths...>. It runs the US-English, list-formatting and facet gates, the section audit and the number-grounding check over the paths being committed, refuses the commit when any fails, and greps the staged diff for personal identifiers. It commits through tooling/ai-commit.sh, so the commit also carries the AI-use trailers (AI-Model / AI-Role / AI-Agent, plus Human-Review when AI_REVIEWED_BY names a contributor id); AI_ROLE=drafting,revision records what the change actually did. The model is never guessed: AI_MODEL=… (or --model) states it, and otherwise it is read from the ai_assist.model recorded on the pages being committed — if neither settles it, the commit is refused rather than stamped with a model that did not do the work. A plain git commit skips all of it, and the defect then surfaces during the next release instead of here.

AI-use disclosure rule (HARD GATE, added 2026-09-22): a corpus drafted by a model should say so, at three granularities, because one cannot answer another's question. The policy is AI-USE.md at the repo root (which models, in what roles, what the editor does, what the scripts verify, what nothing verifies, and that no AI system is an author). The values live in site.config.json: aiDisclosure.models (id + since, newest first) and contributors (kind: human), and pages reference contributors by id so no name is written into a page. Per page, frontmatter carries contributors, reviewed_by, ai_assist (model, role, date), source_depth (articles: full text / abstract only / metadata only) and verified — markdown only, never rendered. Per change, the commit carries the trailers, because a page field records only whoever produced the text as it now stands and goes stale on the next rewrite. Gates: python3 tooling/scripts/check-ai-disclosure.py requires a record on pages created on or after the date the per-page fields were introduced (RECORD_START in the script) and validates every value against the config; python3 tooling/scripts/check-ai-disclosure-trailers.py reports commits without trailers (advisory). Set reviewed_by only where a human actually read the page, and add a new model to site.config.json before any page names it.

List-formatting rule (HARD GATE): ordered/bulleted lists whose consecutive items are separated by a blank line render broken — each item restarts at 1. (CommonMark splits them into separate lists). Run python3 skills/research/wiki-inline-links/scripts/check_list_formatting.py <WIKI> --all before build and fix every reported page by removing the blank line between consecutive list items. A green build does NOT catch this; the maintainer flags it repeatedly.

Pipeline Commands

# Fetch journal RSS feeds (outputs JSON to stdout)
python3 tooling/scripts/fetch-rss-feeds.py

# Re-check every resource URL and its last_verified date
python3 tooling/scripts/check-resource-links.py

# Regenerate agent-ready files (opt-in: only when the maintainer asks)
python3 tooling/scripts/generate-llms-files.py

# Validate the concept registry against content/en/concepts/ and the generated views
python3 tooling/scripts/check_concepts.py

# Run every hard gate declared in wiki.config.yaml. Prefer --changed: it scopes what can be
# scoped to the pages touched since HEAD (the inline-link and list-formatting scans take those
# slugs, the article-section / disclosure / number-grounding gates take --changed, and the
# house-style gate takes the changed markdown files, pages and notes alike) and names the
# registry-level gates it skipped. A gate result cannot differ for an unchanged page, and the
# corpus-wide inline-link scan is the entire cost of the full pass.
python3 tooling/scripts/run-gates.py --changed

# Validate the typed facet fields (pedagogy, technology, ...). Each field holds concept
# slugs filed under one section of concepts.registry.yaml, so a value of the wrong kind is
# an error. This is a gate: `npm run verify` fails if any page disagrees with the registry.
python3 tooling/scripts/validate-facets.py

# House style: US English. Scans prose only — page slugs, wikilink targets and inline code are
# identifiers and are deliberately ignored, so the gate stays useful while a slug is pending rename.
# Naming slugs or repo-relative paths limits it to those files, which is what --changed passes.
python3 tooling/scripts/check-us-english.py --include-docs
python3 tooling/scripts/check-us-english.py my-page-slug tooling/README.md

# Fix what the checker reports. Rewrites body prose only: frontmatter and everything from
# `## Citation` onward are left byte-identical, because a citation reproduces the title as the
# journal printed it. Curated pair list on purpose — a blanket `-ise` rule would turn the
# correct plural "analyses" into "analyzes".
python3 tooling/scripts/respell-us-english.py --changed

# Check the structured metadata fields (research_method, discipline, audience, level, page_kind)
# against the closed vocabularies in src/content.config.ts. Reports coverage per collection and
# any value outside the list; the build rejects invalid values too, but this reports them in
# seconds and names the pages. Absence is coverage, never an error: a study with no disciplinary
# home should omit discipline.
python3 tooling/scripts/audit-metadata.py                  # coverage + violations
python3 tooling/scripts/audit-metadata.py --strict          # non-zero exit on a violation
python3 tooling/scripts/audit-metadata.py --missing level   # list the pages lacking one field
python3 tooling/scripts/audit-metadata.py --check-docs      # fail if SCHEMA.md drifted from the schema
python3 tooling/scripts/audit-metadata.py --write-docs      # regenerate SCHEMA.md's list from the schema

# Check an article page's sections: order, practice immediately before Limitations,
# practice 3-5 bullets, Limitations 2-6 bullets, Citation last.
python3 tooling/scripts/audit-article-sections.py --changed
python3 tooling/scripts/audit-article-sections.py --all

# Ground every number in an article body in its raw source. --changed scopes it to the
# working tree; a reported token is an instruction to investigate, not a verdict.
python3 tooling/scripts/verify-number-grounding.py --changed
python3 tooling/scripts/verify-number-grounding.py <slug> [<slug> ...]

# Validate the AI-use disclosure fields on every page and require a record on pages
# created on or after RECORD_START (the date the per-page fields were introduced).
python3 tooling/scripts/check-ai-disclosure.py

# Report commits that carry no AI-Model / AI-Role trailers (advisory, never fails).
python3 tooling/scripts/check-ai-disclosure-trailers.py
python3 tooling/scripts/check-ai-disclosure-trailers.py --range origin/main..HEAD

# Run EVERY hard gate declared in wiki.config.yaml (build.gates), in order.
# A green build does NOT substitute for these.
python3 tooling/scripts/run-gates.py        # or: npm run verify
python3 tooling/scripts/run-gates.py --list

# Commit only when the gates are green (US English, list formatting, facets, section audit,
# number grounding, then a personal-identifier scan of the staged diff). The commit is
# stamped with the AI-use trailers; AI_ROLE / AI_MODEL / AI_REVIEWED_BY set them.
# The model is never taken from a config default: pass AI_MODEL, or stage pages whose
# own ai_assist.model names it, or the commit is refused.
bash tooling/scripts/commit-if-green.sh /tmp/msg.txt content/en/articles/<new-page>.md public/llms-full.txt

# Stamp the trailers alone, when you are committing without the gate suite. Stage first;
# this wrapper never adds files and never pushes.
git add -A
bash tooling/ai-commit.sh -m "Fix a typo" --no-ai                       # human-only edit
bash tooling/ai-commit.sh -m "Revise the intro" --role revision --reviewed-by editor

# Show the effective configuration / a single value
python3 tooling/scripts/wiki_config.py
python3 tooling/scripts/wiki_config.py --get scan.sources
python3 tooling/scripts/wiki_config.py --cap run_python     # your agent's python tool

# Build the Astro site (also emits the PWA: manifest.webmanifest + sw.js + workbox runtime)
npm run build

# Regenerate the committed offline artifacts after content changes (the Notice page reads
# site.config.json, so a change to the AI-use disclosure also needs a rebuild)
python3 tooling/scripts/generate-llms-files.py
# The interpreter matters here - the PDF stage runs under pandoc -> the weasyprint
# console script (its own shebang), while the metadata step imports pikepdf
# in-process. /usr/bin/python3 has both. See AGENTS.md.
PATH="$HOME/.local/bin:/usr/bin:$PATH" /usr/bin/python3 tooling/build-epub.py

# Deploy (GitHub Actions deploys dist/ on push). Push only with explicit approval; CI
# rebuilds the whole site on every push.
git push

Agent-Ready Files

The site publishes machine-readable files for AI agents:

• llms.txt — complete catalog: every article, concept, and FAQ, one line each
• llms-concepts.txt — full text of the concept pages only (no FAQs, no resources), capped at 500,000 words: the file to hand a chat or ingester that rejects the larger one. Each page's question prompts and Connected lists are dropped to stay small, and where translated prose pushes a locale past the cap its list is truncated at a page boundary and ends with a note naming how many of how many pages it carries
• llms-full.txt — full text of every article, concept and FAQ, about 15 MB (resource pages are excluded: their content lives on the external site they link to)
• robots.txt — search indexing + sitemap
• JSON-LD structured data — a linked schema.org @graph on every page, generated by src/lib/jsonld.ts + src/components/JsonLd.astro (see docs/json-ld.md)

These are opt-in artifacts (maintainer rule, 2026-09-16): they are regenerated with generate-llms-files.py when the maintainer asks, not as part of every ingestion batch, so their churn does not ride along with a routine ingest. The same applies to the EPUB/PDF below. Descriptions are the page's opening paragraph, so both this script and the EPUB builder flatten [[wikilinks]] through the shared tooling/scripts/wikilink_text.py: a piped link keeps its label, a bare link becomes the target page's title. Never strip the brackets and keep the slug, which is how use AI ai-technologies in educational contexts reached llms.txt.

PWA

The site is an installable Progressive Web App. npm run build emits dist/manifest.webmanifest, dist/sw.js, dist/registerSW.js, and the workbox-*.js runtime. The service worker is generated by a dedicated post-build step (generate-sw.mjs via workbox-build) because vite-plugin-pwa's own generateSW hook does not fire under Astro 7 (the @vite-pwa/astro wrapper only supports Astro ≤5). The SW precaches the app shell and runtime-caches content pages (NetworkFirst) and the Pagefind search index (StaleWhileRevalidate), with a /aied/ navigation fallback for offline browsing. PWA icons live in public/ (pwa-192x192.png, pwa-512x512.png, pwa-maskable-512x512.png). Deploy CI runs npm run build so the SW is always generated — keep it that way, or sw.js silently disappears from production.

Offline Versions (EPUB & PDF)

The wiki is published as downloadable eBooks alongside the site:

• EPUB — public/aied.epub → https://github.com/edtechdev/aied/raw/refs/heads/main/public/aied.epub
• PDF — public/aied.pdf → https://github.com/edtechdev/aied/raw/refs/heads/main/public/aied.pdf

The editions are stripped from the published build, so a /aied/aied.epub URL on the site 404s by design.

Both are generated from the same markdown export as the site and contain the home introduction, the Use-With-AI page, all concept pages (chapters by umbrella group, clickable numbered TOC), the FAQ pages, and a Notice page (editor, CC0 license, AI-disclaimer, issue reporting). They contain only the concept and FAQ pages — not the article summaries.

Regenerate both with one command:

# The interpreter matters: the PDF stage runs under pandoc -> the weasyprint
# console script (its own shebang), while the metadata step imports pikepdf
# in-process. /usr/bin/python3 has both. See AGENTS.md.
PATH="$HOME/.local/bin:/usr/bin:$PATH" /usr/bin/python3 tooling/build-epub.py

This writes public/aied.epub and public/aied.pdf. The site does not serve these files — it links the GitHub raw URL (offlineBaseUrl in site.config.json) and astro.config.mjs strips them from dist/, because ~161 MB of already-compressed bytes pushed the Pages artifact past its 1 GB limit. They remain committed under public/ so the raw links resolve.

Supporting files:

• tooling/build-epub.py — assembles the shared markdown export, runs pandoc for the EPUB (hard-coded hierarchical TOC numbering) and pandoc + weasyprint for the PDF (clickable blue TOC via CSS counters), post-processes the EPUB (Notice page carrying the generation date, the CC0 public-domain mark and its localized name, landmarks labeling), and finishes by setting the PDF language metadata with pikepdf. It fails loudly (exit 1) when pikepdf is missing from the invoking interpreter, and prints which interpreter that is.
• tooling/pdf-style.css — PDF print layout (A4, page numbers, cover page, blue clickable links).
• tooling/gen-epub-cover.mjs — renders the book cover (public/epub-cover.png) with sharp.
• tooling/gen-pd-mark.mjs — rasterizes the CC0 public-domain mark (src/assets/cc-zero.svg, the Creative Commons press-kit zero.svg, itself CC0) to public/public-domain-mark.png, which the EPUB/PDF Notice page and the covers use. The site inlines the SVG instead (src/components/PublicDomainMark.astro), so the mark follows the theme.
• tooling/gen-og-concept-map.mjs — renders the concept-map images.

The EPUB/PDF and cover are committed artifacts (built locally, like llms-full.txt), but they are not served from the site: astro.config.mjs strips them from dist/, so readers download them from the GitHub raw URLs in offlineBaseUrl. Requires pandoc, and for the PDF the weasyprint Python package plus pikepdf for the language metadata.

Configuration

Two files at the repo root drive everything — edit these, not the code:

• wiki.config.yaml — content paths, the HARD GATE commands and build commands, the concept-registry location, the daily scan sources (arXiv categories, keywords, listing fallbacks), the weekly journal feeds, the relevance filter — and an agent: block naming your AI agent and which of its tools provides each capability the pipeline needs.
• concepts.registry.yaml — the concept vocabulary: one entry per concept page with its title and the synonym phrases the inline-link pass links to, plus the sidebar sections, the redirect/merge map and the never-link list.

Check them with:

python3 tooling/scripts/wiki_config.py --check     # paths, gates, sources, preview, agent block
python3 tooling/scripts/check_concepts.py          # registry vs content/en/concepts/ vs generated views

Machine-specific settings (the preview's host and port, for example) go in wiki.config.local.yaml, which is gitignored and merged over wiki.config.yaml — so a clone stays portable and each machine keeps its own addresses:

# wiki.config.local.yaml
preview:
  host: 192.168.1.50      # e.g. a LAN or Tailscale address to preview from a phone

Nothing else should hardcode a site name, base path, journal, or agent path. When you point the tooling at a different knowledge base, these two files plus site.config.json are the whole customisation surface. tooling/scripts/sync-skills.py keeps the repo's skills/ mirrors in step with the agent's installed copies.

Keeping the live preview in step

If you keep an astro dev preview running, it will not notice content written by a scheduled scan — its content cache goes stale and the new pages 404 until it restarts. tooling/scripts/refresh-preview.py compares the newest content timestamp with the server's sync time, restarts the server only when it is stale, and waits for HTTP 200:

python3 tooling/scripts/refresh-preview.py --check   # status only (exit 1 if stale)
python3 tooling/scripts/refresh-preview.py           # restart if stale
python3 tooling/scripts/refresh-preview.py --quiet   # silent unless it acts (watchdog)

The scan prompts call it as their last step, and it also works well as a script-only cron watchdog (no LLM, silent while the preview is fresh).

Dependencies

• AI agent (for cron jobs and ingestion) — any agent works; map its tools in wiki.config.yaml → agent.capabilities
• Node.js 18+ (Astro 7, Pagefind)
• Python 3.9+ with PyYAML (pip install pyyaml) — the config, the concept registry and the linters all parse YAML. Everything else is stdlib.
• pandoc — for the EPUB/PDF generation (tooling/build-epub.py)
• weasyprint (Python, pip install weasyprint) — PDF engine used by pandoc for aied.pdf. pandoc reaches it through the weasyprint console script, so it runs under that script's shebang interpreter, not necessarily the one you invoked the build with.
• pikepdf (Python, pip install pikepdf) — normalizes the PDF /Lang to en-US and re-affirms the title. Imported in-process, so it must be present in the interpreter you invoke the build with. build-epub.py now exits 1 when it is missing, naming the interpreter; previously this was best-effort and a PDF shipped with no /Lang at all.
• pdftotext (poppler-utils) for PDF extraction
• GitHub Pages for deployment

The EPUB/PDF build uses two interpreters

The build's two halves do not share an interpreter, which is why a missing pikepdf is so easy to miss:

stage	runs under
EPUB and PDF rendering	pandoc → the weasyprint console script → that script's shebang
PDF /Lang + title metadata	import pikepdf in-process → whatever invoked the script

Consequence: invoking the build with a Python that lacks pikepdf still produces a complete PDF — it silently omits /Lang, and the build exits 0. Install pikepdf for the interpreter you are invoking, and prefer the pinned invocation in AGENTS.md, which uses /usr/bin/python3 (it has both packages).

Troubleshooting

Problem	Check
Site not updating	GitHub Actions deploy workflow ran? Actions tab → astro-deploy
Search index stale	Pagefind-based — run npm run build so dist/pagefind/ regenerates
llms.txt out of date	python3 tooling/scripts/generate-llms-files.py then npm run build
Broken wikilinks	Links use [[slug]] — the slug must match a file in content/en/articles/ or content/en/concepts/
Numbered list shows every item as 1.	Blank lines between consecutive list items split them into separate lists — run check_list_formatting.py and remove the blank lines
YAML parsing errors	Titles with colons must be quoted: title: "X: Y"
Paywalled articles	Hybrid journals (BJET) — the weekly cron skips paywalled articles and reports them
check_concepts.py reports a stale generated view	Run python3 tooling/scripts/gen-concept-artifacts.py — concept-index.md, conceptIndex.ts and conceptRedirects.ts are generated from the registry
A concept page exists but nothing links to it	It has no aliases in concepts.registry.yaml; the scanner can only link terms it knows
A link went to the wrong concept	Two concepts claim the same phrase — check_concepts.py flags duplicate alias claims

Run Your Own Wiki

Copy the tooling/ directory and the skills/ directory into a new repo, follow this README and the Astro site setup, then rewrite site.config.json, wiki.config.yaml and concepts.registry.yaml for your domain, and you'll have your own automated research wiki. Those three files are the only place site identity, scan sources and concept vocabulary live.

To fully reproduce the ingestion workflow (including the inline-link HARD GATE and the list-formatting check), install the AI agent skills in the research category, all mirrored under skills/research/ (each with a "Repository mirror" note pointing back here):

• research-wiki — the full ingestion + export pipeline (mirrored in tooling/SKILL.md)
• wiki-inline-links — the aggressive inline-link + list-formatting pass that runs on every new/enriched page before build (mirrored in skills/research/wiki-inline-links/)
• wiki-epub-export — EPUB/PDF regeneration (skills/research/wiki-epub-export/)
• wiki-site-quality — static-site bug fixes: dup H1, broken links, dead metadata chips, markdown tables, journal date quoting, public-repo privacy checks (skills/research/wiki-site-quality/)
• wiki-astro-frontend — editing the Astro frontend (homepage, concept map, sidebar, icons, PWA, JSON-LD, theming) (skills/research/wiki-astro-frontend/)

See cron/ for the job prompts that wire them together, and tooling/scripts/sync-skills.py for keeping those mirrors in step with the copies your agent actually loads (it normalizes absolute paths, the agent name and personal identifiers so only real content drift is reported).