docs: add MkDocs docs site, comparison page, and SEO metadata

[uipreliga]

Turn the docs/ tree into an indexable GitHub Pages site and sharpen the package/repo metadata so coder_eval ranks for the queries users actually search (test claude code skills, evaluate coding agents, claude vs codex benchmark) instead of only its own name.

• mkdocs.yml + .github/workflows/docs.yml: MkDocs Material site, published to GitHub Pages on push to main (one-time: set Pages source to "GitHub Actions").
• docs/index.md: keyword-rich landing page (SEO homepage).
• docs/COMPARISON.md: benefit-led "why coder_eval" page comparing it to SWE-bench, SkillsBench, Harbor, and OpenAI Evals (plus hand-rolled scripts). Framing is promotional; factual claims about other projects are grounded in their own docs, with a Sources section and inline links.
• docs/llms.txt: served at site root for AI-search (Claude/ChatGPT/Perplexity).
• README opening rewritten: what it is, what it competes with, when to choose it, in the first 200 words; brand-split fix (coder_eval / pip install coder-eval).
• pyproject.toml: +llm-eval, ai-evaluation, claude-code-skills, agent-testing, llmops keywords; Documentation URL -> docs site.
• Exclude internal IDEAS.md from the site; fix repo-relative links so mkdocs build --strict is clean; fix Tutorial 06 mislabeled as "04".

[github-actions]

Claude finished @uipreliga's task in 6m 16s —— View job

---

Code Review in Progress

Todo List:

• Read .github/code_review.md for review standards
• Read CLAUDE.md for project conventions
• Run git diff to see all changes
• Review each changed file with full context
• Perform cross-file consistency checks
• Analyze "what's missing"
• Provide design-level scrutiny
• Post comprehensive review

[bai-uipath]

LGTM

Worth addressing (my calls, not blockers)

• Name it "Coder Eval" in prose. Use the title-case, spaced display name in headings and body copy rather than coder_eval; keep coder-eval for the pip/CLI name. Reads cleaner and it's the brand I'd want indexed.
• Drop ⚠️ DIY in the comparison table. The ⚠️ DIY cells look cluttered — use a plain x. (Slight nuance loss vs. "possible but manual," but the table's cleaner and that's the better trade.)

[uipreliga]

Review: coder_eval — pr:32 (21 files)

Scope: pr:32 (21 files) · branch seo-improvements · d9c655c · 2026-07-21T20:28Z · workflow variant

Change class: simple — docs site scaffolding (MkDocs config, new/edited Markdown), one new gh-pages deploy workflow, and pyproject keyword metadata; no source logic touched and each piece is verifiable at a glance

This docs/SEO PR is architecturally clean and touches no evaluation logic — nothing here can change a task's score or final_status, security and type safety are effectively spotless, and the real risks are all correctness-of-published-content and CI-hygiene: a "Bring Your Own Dataset" label pointing six surfaces (including the LLM-ingested llms.txt) at a Docker page, front matter overclaiming coverage the task-authoring reference doesn't have, and a strict docs build that runs only post-merge inside the deploy step of an unpinned, unguarded contents: write workflow — all cheap to fix, so ship after a short content/CI pass.

Summary

Axis	Score	🔴	🟠	🟡	🔵	Top Issue
1. Code Quality & Style	8.8 / 10	0	0	2	2	Product overview/marketing copy duplicated across README.md, docs/index.md, docs/llms.txt and comparison.md, already drifting
2. Type Safety	9.9 / 10	0	0	0	1	Docs toolchain declared only in-workflow and unpinned (floating mkdocs-material range)
3. Test Health	9 / 10	0	1	0	0	No PR-time docs gate: mkdocs build --strict runs only in the post-merge gh-deploy job (and mkdocs is not a declared dependency)
4. Security	10 / 10	0	0	0	0	—
5. Architecture & Design	9.9 / 10	0	0	0	1	New public docs site scopes out only IDEAS.md, still publishing internal-only knobs and an internal ticket id
6. Error Handling & Resilience	9.8 / 10	0	0	0	2	cancel-in-progress: true plus an unrestricted workflow_dispatch ref lets a non-main branch force-publish and stick
7. API Surface & Maintainability	8 / 10	0	1	2	0	docs/BYOD.md ("Bring Your Own Docker") is labelled "Bring Your Own Dataset" across nav, index, comparison and llms.txt
8. Evaluation Harness Quality	9.5 / 10	0	0	1	0	New TASK_DEFINITION_GUIDE front matter claims coverage the guide does not have (14 criterion types + dataset fan-out; it documents 12 and no dataset section)

Overall Score: 9.4 / 10 · Weakest Axis: API Surface & Maintainability at 8 / 10
Totals: 🔴 0 · 🟠 2 · 🟡 5 · 🔵 6 across 8 axes.

Blockers

1. [Axis 3] No PR-time docs gate: mkdocs build --strict runs only in the post-merge gh-deploy job (and mkdocs is not a declared dependency) (.github/workflows/docs.yml:13) — docs.yml is triggered only by push:\n branches: [main] (lines 13-14) and workflow_dispatch, and its single verification step is the deploy itself: run: mkdocs gh-deploy --force --strict (line 48). I read .github/workflows/pr-checks.yml end-to-end — none of its four jobs (quality-gate, no-extra-install, windows-smoke, e2e-smoke) has a docs/mkdocs step (grep -n 'mkdocs' .github/workflows/pr-checks.yml → no match), and Makefile's verify: target (lines 41-48) has no docs step either. Consequence: this PR's own content has never been strict-built by CI, and any future PR that adds a nav entry for a non-existent page, a broken relative link, or a page outside nav fails only after merge, leaving a red build on main and a stale published site. The gap also covers the CI-only social plugin path (mkdocs.yml:65 enabled: !ENV [CI, false] plus the libcairo2/libfreetype6 apt step at docs.yml:35-40) — that code path is unreachable on any PR. Fix: add a pull_request job (or a step in pr-checks.yml) running mkdocs build --strict without deploying, gated on the same paths: filter. Note GitHub Actions sets CI=true, so such a job must either install the imaging system libs or explicitly unset CI so the social plugin stays off. (I reproduced the build locally in a throwaway venv with mkdocs 1.6.1 + mkdocs-material 9.x against pr-32: mkdocs build --strict succeeded with zero warnings and zero anchor/nav infos, so the current content is clean — the finding is the missing gate, not a present breakage.)
2. [Axis 7] docs/BYOD.md ("Bring Your Own Docker") is labelled "Bring Your Own Dataset" across nav, index, comparison and llms.txt (docs/index.md:81) — git show pr-32:docs/BYOD.md line 8 is # Bring Your Own Docker (BYOD) and grep -rn "dataset" docs/BYOD.md returns zero matches — the page is entirely about extending the coder-eval-agent Docker image. Yet every new/changed surface in this PR routes readers looking for dataset fan-out to it: docs/index.md:81 | [Bring Your Own Dataset](BYOD.md) | Fan a single task out over a dataset |; docs/index.md:91 still wrap a fixed dataset via [Bring Your Own Dataset](BYOD.md)); docs/comparison.md:48 wrapped as tasks via [Bring Your Own Dataset](BYOD.md).; docs/comparison.md:134 - [Bring Your Own Dataset](BYOD.md); mkdocs.yml:91 - Bring Your Own Dataset: BYOD.md; docs/llms.txt:29 - [Bring Your Own Dataset](https://uipath.github.io/coder_eval/BYOD/): fan a single task out over a dataset. The dataset feature (TaskDefinition.dataset, ${row.<field>} fan-out) is actually documented in docs/TASK_DEFINITION_GUIDE.md (grep -rln dataset docs/*.md → AB_EXPERIMENTS.md, comparison.md, index.md, TASK_DEFINITION_GUIDE.md — not BYOD.md). Fix: either write a real Bring-Your-Own-Dataset page (or repoint these six links + the nav entry at the dataset section of TASK_DEFINITION_GUIDE.md) and relabel BYOD.md in the nav as "Bring Your Own Docker". This is especially damaging in docs/llms.txt, which exists to be ingested verbatim by LLMs.

Non-blocking, but please consider before merge

1. [Axis 1] Product overview/marketing copy duplicated across README.md, docs/index.md, docs/llms.txt and comparison.md, already drifting (docs/index.md:29) — git show pr-32:README.md | sed -n '31,37p' | md5 and git show pr-32:docs/index.md | sed -n '29,35p' | md5 are byte-identical (b20b8cbd67eb5244966b7206197c1cfc) — the 7-bullet feature list is duplicated verbatim. The "use cases" list is duplicated again (README.md:40-45 vs docs/index.md:39-45), and the competitive pitch now exists in FOUR places: README.md:122-131, docs/index.md:88-98, docs/comparison.md (whole file), docs/llms.txt:11-20. Any future edit (e.g. adding a 4th agent) must be applied in 4 places or the site self-contradicts — exactly the failure mode the BYOD mislabel above already demonstrates. pymdownx.snippets is already enabled (mkdocs.yml:66): extract the shared bullets/pitch into a docs/_includes/ fragment and --8<-- it into index.md, and reduce index.md's "How Coder Eval compares" section to a single link to comparison.md rather than a third rendition.
2. [Axis 1] docs.yml uses mutable action tags (checkout@v4, setup-python@v5) in a contents: write job, against the repo's SHA-pinning convention (.github/workflows/docs.yml:32) — Lines 32-35 use - uses: actions/checkout@v4 and - uses: actions/setup-python@v5, while every other workflow SHA-pins and is on newer majors — e.g. .github/workflows/pr-checks.yml:44 uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 and :47 uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0 (same pattern in codeql.yml, publish-testpypi.yml, docker-publish.yml, claude-pr-review.yml, conventional-commits.yml). This workflow holds permissions: contents: write (line 21-22), so it is the one place the looser pinning matters most. Line 42 pip install "mkdocs-material[imaging]>=9.5,<10" is likewise an open range feeding a --strict build, so any upstream release that adds a new WARNING breaks the deploy with no repo change. Pin both actions to the SHAs already used elsewhere and pin mkdocs-material to an exact version (or a requirements file).
3. [Axis 7] pyproject [project.urls] Documentation still points at the GitHub docs tree, not the new published docs site (pyproject.toml:56) — The PR ships a published site at site_url: https://uipath.github.io/coder_eval/ (mkdocs.yml) and edits pyproject.toml metadata in the same commit (keywords, lines 9–16) for SEO, but pyproject.toml:56 still reads Documentation = "https://github.com/UiPath/coder_eval/tree/main/docs". Every downstream consumer of the package metadata (the PyPI project sidebar, pip show, package aggregators) therefore links to the raw Markdown folder rather than the rendered site — which also undercuts the SEO goal, since PyPI is one of the highest-authority backlinks the project has. Set Documentation = "https://uipath.github.io/coder_eval/" (README.md already advertises that URL in its Docs badge).
4. [Axis 7] comparison.md table asserts unsourced/incorrect ❌ claims (third-party negatives, hand-rolled column, per-tool-call cost telemetry) (docs/comparison.md:34) — docs/comparison.md:19 states "Every claim about another project is grounded in its own documentation; see Sources." But the negative cells are not sourced, and one is contradicted three sections later: line 34 | **Results dashboard** | ✅ evalboard | ❌ | ❌ | ❌ | ❌ | ❌ | marks Harbor as having no results dashboard, while line 81 of the same page says Harbor runs "agents ... using Docker and cloud providers (Daytona, Modal, LangSmith)" — LangSmith being a run-observability/results product. Line 33 | **CI pass/fail gate** | ✅ built-in | ❌ | ❌ | ❌ | ❌ | ❌ | is likewise an absolute negative about four third-party projects with nothing in the Sources block (lines 141–160, which cite only positive capabilities) backing it. Either downgrade the unverified negatives to — (the marker the same Harbor column already uses for A/B experiments and cost telemetry), or cite the specific competitor doc that establishes the absence.
5. [Axis 8] New TASK_DEFINITION_GUIDE front matter claims coverage the guide does not have (14 criterion types + dataset fan-out; it documents 12 and no dataset section) (docs/TASK_DEFINITION_GUIDE.md:4) — The PR adds a new YAML front-matter block (docs/TASK_DEFINITION_GUIDE.md:2-5) that becomes the page's , the search-result snippet, and the LLM-summary surface for the canonical task-authoring reference:

description: >-
  Full schema reference for coder_eval task YAML — agent config, sandboxes, run
  limits, dataset fan-out, and all 14 success criterion types with weighted
  0.0–1.0 scoring.

Verified against the PR HEAD tree: src/coder_eval/criteria/ contains 14 checker modules (agent_judge, classification_match, command_executed, commands_efficiency, file_check, file_contains, file_exists, file_matches_regex, json_check, llm_judge, reference_comparison, run_command, skill_triggered, uipath_eval), but the guide's headings (grep -n '^### ' docs/TASK_DEFINITION_GUIDE.md) document only 12 — there is no ### commands_efficiency and no ### classification_match section. grep -n 'commands_efficiency\|classification_match' docs/TASK_DEFINITION_GUIDE.md returns exactly one hit, line 170, and it is only the agentless-task rejection list ("commands_efficiency) are rejected"), not a schema section. Likewise there is no dataset: section: grep -n 'dataset' docs/TASK_DEFINITION_GUIDE.md returns only the front matter (line 4) and two prose mentions at lines 781/783; dataset fan-out is documented in docs/BYOD.md, and line 781's link [dataset-backed task](#task-yaml-structure) points at a section that never mentions dataset.

Task authors (and now AI agents consuming docs/llms.txt and the site) are the only consumers of this file; telling them the reference is complete when two shipped criterion types have no schema documentation is task-authoring drift. Fix by either (a) adding ### commands_efficiency and ### classification_match sections plus a dataset section (or an explicit cross-link to BYOD.md) to the guide, or (b) softening the front matter to what the page actually covers (e.g. "…run limits and the success-criterion schema") and dropping the "all 14" / "dataset fan-out" claims. Option (a) also merits a doc-drift guard: a test asserting every entry in the SuccessCriterion union has a matching ### <type> heading in the guide would make this mechanically enforced (candidate CE0xx-style check).

Nits

1. [Axis 1] Social-card apt step installs packages the social plugin does not use (.github/workflows/docs.yml:36) — Lines 36-40 run sudo apt-get install -y --no-install-recommends libcairo2 libfreetype6 libjpeg-turbo8 libpng16-16 pngquant. mkdocs-material's social plugin needs Cairo (libcairo2, dlopen'd by cairocffi) plus Pillow — and the Pillow manylinux wheel bundles its own libjpeg/libpng/freetype, so libfreetype6 libjpeg-turbo8 libpng16-16 are redundant. pngquant is a dependency of the Insiders-only optimize plugin, which is not configured (mkdocs.yml:50-53 enables only search and social), so it is pure dead machinery in the runner. Reduce the step to libcairo2 (plus a comment naming why), which also removes the risk of the build breaking when a package name changes across Ubuntu runner images.
2. [Axis 1] Hand-written <a id> shims patch two GitHub-style heading anchors instead of configuring a matching slugifier (docs/TASK_DEFINITION_GUIDE.md:138) — This PR adds two invisible anchor shims — docs/TASK_DEFINITION_GUIDE.md:138 <a id="no-op--system-tasks-type-none"></a> and docs/USER_GUIDE.md:91 <a id="api-routing--benchmarking"></a> (immediately above ## API Routing & Benchmarking at :93) — because Markdown's default toc slugify collapses & to a single hyphen while the existing intra-repo links use GitHub's double-hyphen form. That is a per-heading magic-string workaround for a config-level problem: set toc: slugify: !!python/name:pymdownx.slugs.slugify with separator: "-" in mkdocs.yml (pymdownx is already a dependency — pymdownx.highlight/superfences/inlinehilite/snippets/tabbed are configured at mkdocs.yml:62-68) so every GitHub-style anchor in docs/ resolves without per-page shims. Note mkdocs' validation.anchors defaults to info, so --strict will NOT catch the next heading that hits this — the shims are load-bearing but unguarded.
3. [Axis 2] Docs toolchain declared only in-workflow and unpinned (floating mkdocs-material range) (.github/workflows/docs.yml:42) — .github/workflows/docs.yml:42 is run: pip install "mkdocs-material[imaging]>=9.5,<10" — a floating range feeding mkdocs gh-deploy --force --strict (line 48), where any new warning class introduced by a Material/MkDocs minor becomes a hard publish failure. The dependency is not declared anywhere else: grep -n "mkdocs" pyproject.toml uv.lock Makefile returns zero hits, so there is no docs extra/dependency-group and no lockfile entry a contributor can install to reproduce the CI build locally. Recommend adding a docs optional-dependency group (or a [dependency-groups] docs) in pyproject.toml with a tighter pin, and having the workflow install from it.
4. [Axis 5] New public docs site scopes out only IDEAS.md, still publishing internal-only knobs and an internal ticket id (mkdocs.yml:43) — mkdocs.yml:43-44 (exclude_docs: | / IDEAS.md) is the only content-scoping pass done before pointing a public GitHub Pages site at docs/, and the nav publishes every remaining guide as-is. Residual internal-only material now shipping publicly: docs/USER_GUIDE.md:148-150 documents three UiPath-proprietary env vars inside the core reference table, one of them a footgun scoped to internal fleet hosts — | CODER_EVAL_REMEDIATE_HOME_PLUGINS| No | **DESTRUCTIVE.** Truthy deletes$HOME/node_modules/@UiPath at sandbox setup to clear sibling-task pollution on dedicated eval hosts. — with no framing that these belong to the optional UiPath integration rather than the agnostic core CLAUDE.md describes; docs/TASK_DEFINITION_GUIDE.md:889 cites a private tracker id (introduced by [PR #250](https://github.com/UiPath/coder_eval/pull/250) for MST-9674) that means nothing to an external reader; docs/tutorials/03-evalboard-local.md:70 advertises an internal build (UiPath-internal surfaces are opt-in via EVALBOARD_EDITION=internal``). Recommend a one-time scoping pass alongside the exclude: drop the MST-9674 reference, and either move the three UiPath env vars into a clearly-labelled "UiPath integration (optional extra)" subsection or drop them from the public User Guide, so the published site matches the agnostic-core / UiPath-opt-in convention stated in CLAUDE.md.
5. [Axis 6] cancel-in-progress: true plus an unrestricted workflow_dispatch ref lets a non-main branch force-publish and stick (.github/workflows/docs.yml:26) — concurrency: group: docs-gh-pages / cancel-in-progress: true (lines 24-26) is applied to a deploy job, and workflow_dispatch: (line 19) is unconstrained, so it can be dispatched from any branch. A dispatch from a feature branch cancels an in-flight main deploy and then force-pushes that branch's docs over gh-pages (see finding on --force). Because the push trigger is path-filtered to docs/**/mkdocs.yml/.github/workflows/docs.yml (lines 15-18), subsequent merges to main that don't touch those paths will not re-publish — the site stays on the wrong content indefinitely with no signal. The build-then-push ordering means a cancel itself can't leave a half-written branch, so the cancel risk alone is minor; the ref risk is the real one. Fix: guard the job with if: github.ref == 'refs/heads/main', and consider cancel-in-progress: false so a deploy always runs to completion.
6. [Axis 6] Deploy job has no timeout-minutes, so a hung strict build holds the deploy concurrency group for the 6-hour default (.github/workflows/docs.yml:29) — jobs: publish: runs-on: ubuntu-latest (lines 29-30) declares no timeout-minutes, inheriting GitHub's 360-minute default. The sibling gate in this repo does set one (pr-checks.yml: quality-gate: ... timeout-minutes: 10). The strict build here is network-dependent — mkdocs-material's social plugin (enabled in CI via social: enabled: !ENV [CI, false] in mkdocs.yml) fetches web fonts at card-generation time — so a network stall parks the job, and with it the docs-gh-pages concurrency group, for hours. Add timeout-minutes: 15 to the publish job.

What's Missing

Parallel paths:

• 🟠 README.md is a fifth, unlisted site of the BYOD mislabel — README.md:116 (| [Bring Your Own Dataset](docs/BYOD.md) | Fan a single task out over a dataset |) and README.md:126 (still wrap a fixed dataset via [Bring Your Own Dataset](docs/BYOD.md)) repeat it verbatim. The Axis-7 fix list (index.md ×2, comparison.md ×2, mkdocs.yml nav, llms.txt) must be extended to README.md, and since GitHub renders README raw, a --8<-- snippet cannot cover it. (trigger: docs/index.md) (restates: Axis 7: docs/BYOD.md ("Bring Your Own Docker") is labelled "Bring Your Own Dataset")
• 🟡 docs/llms.txt drifted from mkdocs.yml nav on day one: its ## Tutorials block lists 01-06 but omits the tutorials/README.md overview page that both the nav (Overview: tutorials/README.md) and the docs/index.md "Where to go next" table point at as the recommended entry point. The page an LLM-consumed index most needs to name is the one it drops. (trigger: docs/llms.txt)

Display & mapping dicts:

• 🟡 Adding or renaming a docs page now requires four hand-maintained registries to be updated in lockstep — mkdocs.yml nav, the docs/index.md guides table, the README.md docs table, and docs/llms.txt — and nothing enforces it. --strict will not catch a page missing from these: mkdocs emits "not included in the nav" at INFO, and validation.anchors defaults to info too, so the omission ships silently (the llms.txt tutorials-overview gap and the four-way BYOD mislabel are both instances). (trigger: mkdocs.yml) (restates: Axis 1: Product overview/marketing copy duplicated across README.md, docs/index.md, docs/llms.txt and comparison.md)

Tests:

• 🟠 Beyond the missing PR-time mkdocs build --strict job, there is no link/anchor validation at all for the ~40 cross-document relative links this PR now makes load-bearing on a rendered site (e.g. docs/tutorials/0*.md → ../USER_GUIDE.md#usage-telemetry, #output-structure) and no guard on the two hand-written <a id> anchor shims — mkdocs validation.anchors defaults to info, so a broken anchor is invisible even under --strict unless the setting is raised to warn. (trigger: .github/workflows/docs.yml) (restates: Axis 3: No PR-time docs gate — mkdocs build --strict runs only in the post-merge gh-deploy job)
• 🟡 No doc-drift test ties documentation to the code it documents. A CE-style lint rule asserting every member of the SuccessCriterion union has a matching ### <type> heading in docs/TASK_DEFINITION_GUIDE.md would have caught the missing commands_efficiency / classification_match sections mechanically, and is the exact "could a lint rule have prevented this?" case CLAUDE.md mandates. (trigger: docs/TASK_DEFINITION_GUIDE.md) (restates: Axis 8: TASK_DEFINITION_GUIDE front matter claims coverage the guide does not have)
• 🟡 A new build system (mkdocs + material + social/imaging) landed with no contributor-facing entry point: Makefile has no docs / docs-serve target (grep '^[a-z-]*:' Makefile → none), CONTRIBUTING.md says nothing about building or previewing the site, and pyproject.toml declares no docs extra. A contributor editing docs cannot reproduce the CI --strict build without reading the workflow and hand-copying its pip install line. (trigger: mkdocs.yml) (restates: Axis 2: Docs toolchain declared only in-workflow and unpinned)

Downstream consumers:

• 🟡 The new canonical docs URL was not propagated to the package-metadata consumers: pyproject.toml:56 still points Documentation at .../tree/main/docs, and the GitHub repo "About" website field (settable via gh repo edit --homepage) is likewise not addressed — both are high-authority backlinks for a PR whose stated purpose is SEO. (trigger: pyproject.toml) (restates: Axis 7: pyproject [project.urls] Documentation still points at the GitHub docs tree)
• 🟡 docs/llms.txt is published but unreachable by discovery: grep -rn "llms.txt" across all Markdown/YAML/TOML returns zero references — nothing in README.md, docs/index.md, or mkdocs.yml (extra/extra_head) links to it, and because the site is served under a project path it lands at /coder_eval/llms.txt rather than the domain-root location the convention specifies. An artifact whose entire value is being fetched by agents ships with no inbound pointer. (trigger: docs/llms.txt)

Daily/nightly:

• 🔵 Blast radius on the production pipelines is nil (zero Python files changed, no run-record / task.json / report-shape change, so the nightly suite and the coder-eval-uipath contract are untouched) — but the PR does add a new contents: write job on every main push, and its stated prerequisite (an org owner enabling Pages on the gh-pages branch) is not recorded as done; gh-pages does not yet exist on origin, so the first gh-deploy --force creates it. Until Pages is switched on, the README badge, docs/llms.txt, and mkdocs.yml site_url all advertise a URL that 404s. (trigger: .github/workflows/docs.yml)

Harness & Lint Improvements

Static checks (lint / type):

• [ce-lint] Extend the CE lint framework with a non-AST rule tier, then add CE026 docs-link-text-matches-page-title. Today tests/lint/runner.py::check_paths only walks *.py and every rule subclasses ast.NodeVisitor (tests/lint/rules/base.py), so no CE rule can reach Markdown/YAML — that is exactly why this docs-only PR produced 13 findings with zero mechanical gate. Add a TextRule sibling in tests/lint/rules/base.py (check(path, source_lines) -> list[Violation], same Violation/# noqa: CEnnn suppression path), a TEXT_RULES list + glob-driven check_text_paths() in tests/lint/runner.py, and parametrize it from tests/test_custom_lint.py so make lint / make verify cover it. First rule CE026: for every Markdown link or mkdocs.yml nav entry whose target is a repo-local docs/*.md, the link text (normalized: lowercase, strip punctuation) must be a substring of, or share the acronym with, that page's first # heading — unless an explicit docs/.linkalias allowlist entry exists. Prevents: The high-severity BYOD mislabel: [Bring Your Own Dataset](BYOD.md) at docs/index.md:81, docs/index.md:91, docs/comparison.md:48, docs/comparison.md:134, docs/llms.txt:29 and Bring Your Own Dataset: BYOD.md at mkdocs.yml:91, all pointing at a page titled # Bring Your Own Docker (BYOD).
• [ce-lint] CE027 criterion-types-documented: parse src/coder_eval/models/criteria.py for the SuccessCriterion union members' type: Literal["…"] discriminators (pure AST, the existing rule shape) and assert each one has a matching ### <type> heading in docs/TASK_DEFINITION_GUIDE.md; also assert any all N … / N success criterion types count claimed in that file's YAML front matter equals the union cardinality. Slots next to CE024 (ce024_discriminated_unions.py), which already walks the same union. Prevents: The front-matter overclaim at docs/TASK_DEFINITION_GUIDE.md:4 ("all 14 success criterion types") against the 12 documented ### criterion sections — commands_efficiency and classification_match have no schema section. Also keeps the CLAUDE.md "Success Criteria (14 types)" table honest as criteria are added.
• [ce-lint] CE028 workflow-actions-sha-pinned (TextRule over .github/workflows/*.yml): every uses: referencing a third-party/actions/* action must be a 40-hex SHA followed by a # vX.Y.Z comment; local (./) and docker:// refs exempt. .github/dependabot.yml already documents SHA pinning as maintained repo policy, but nothing enforces it. Prevents: .github/workflows/docs.yml:32-33 (actions/checkout@v4, actions/setup-python@v5) in the one job holding permissions: contents: write, plus the pre-existing floating tags in .github/workflows/docker-publish.yml:68,71,78.
• [ce-lint] CE029 workflow-job-timeout-required (TextRule over .github/workflows/*.yml): every job block must declare timeout-minutes. docs.yml is currently the only workflow with zero occurrences (grep -c timeout-minutes → 0; every other workflow has ≥1), so the rule is green on the rest of the repo today. Prevents: The missing timeout-minutes on docs.yml's publish job (line 29), which lets a network-stalled social-plugin build hold the docs-gh-pages concurrency group for GitHub's 360-minute default.
• [ce-lint] CE030 workflow-pip-install-pinned (TextRule over .github/workflows/*.yml): a bare pip install <pkg> in a run: step must use == or -r <requirements file> / uv sync --frozen; open ranges (>=, ~=, unbounded) are rejected. Prevents: .github/workflows/docs.yml:42 pip install "mkdocs-material[imaging]>=9.5,<10" feeding mkdocs gh-deploy --force --strict (line 48), where any upstream minor that introduces a new warning class becomes a hard, un-reproducible publish failure.
• [ce-lint] CE031 published-docs-no-internal-refs (TextRule over the files MkDocs actually publishes — docs/**/*.md minus mkdocs.yml:exclude_docs, plus docs/llms.txt and README.md): reject internal-tracker id patterns (\bMST-\d+\b and similar), and require any CODER_EVAL_* env var documented in a table to be flagged UiPath-only or live under a heading matching UiPath integration. The published/excluded set is read from mkdocs.yml, so the rule stays correct as pages move in and out of the site. Prevents: The internal-content leak finding: the MST-9674 tracker reference at docs/TASK_DEFINITION_GUIDE.md:889, the three UiPath-proprietary env vars (including the DESTRUCTIVE CODER_EVAL_REMEDIATE_HOME_PLUGINS) sitting unlabelled in the core reference table at docs/USER_GUIDE.md:148-150, and the EVALBOARD_EDITION=internal mention at docs/tutorials/03-evalboard-local.md:70.
• [ce-lint] CE032 package-urls-match-site (TextRule): assert pyproject.toml [project.urls] Documentation equals mkdocs.yml site_url, and that the README docs badge URL matches both. Three-line rule, one assertion, no false-positive surface. Prevents: pyproject.toml:56 still pointing at https://github.com/UiPath/coder_eval/tree/main/docs after the PR published site_url: https://uipath.github.io/coder_eval/ (mkdocs.yml:6) and linked it from the README badge.
• [ce-lint] CE033 no-manual-heading-anchor-shims (TextRule over docs/**/*.md): reject hand-written <a id="…"></a> heading shims, paired with the config fix (toc: slugify: !!python/name:pymdownx.slugs.slugify with separator: "-" in mkdocs.yml) so GitHub-style anchors resolve site-wide. The rule is only sound once the slugifier is configured, so land the two together. Prevents: The two invisible shims added by this PR — docs/TASK_DEFINITION_GUIDE.md:138 <a id="no-op--system-tasks-type-none"></a> and docs/USER_GUIDE.md:91 <a id="api-routing--benchmarking"></a> — which are per-heading magic strings papering over a slugifier mismatch, and which --strict does not protect (mkdocs validation.anchors defaults to info).
• [bandit-codeql] Add actions to the CodeQL language matrix in .github/workflows/codeql.yml:33 (languages: python → languages: python, actions). CodeQL's Actions query pack flags unpinned action versions, over-broad permissions, and untrusted-ref deploy patterns as security alerts on every PR, complementing the repo-local CE rules above with upstream-maintained queries. Prevents: The floating-tag pins in .github/workflows/docs.yml:32-33 under permissions: contents: write, and the unconstrained workflow_dispatch + cancel-in-progress: true deploy shape (docs.yml:19, 24-26) that lets a feature-branch dispatch force-push its docs over gh-pages.

Harness improvements (not statically reachable):

• Add a PR-time docs gate: a docs job in .github/workflows/pr-checks.yml (or a pull_request trigger on docs.yml with the same paths: filter) that runs mkdocs build --strict without deploying, plus a make docs-check target so make verify covers it locally. Because GitHub Actions sets CI=true and mkdocs.yml:53 gates the social plugin on !ENV [CI, false], the job must either install the imaging system libs or explicitly unset CI — pick one deliberately, since the CI-only social path is currently unreachable on any PR. Why not static: Requires actually executing MkDocs against the rendered site graph — nav resolution, plugin loading, cross-page link/anchor validation — none of which is decidable by grepping or AST-walking a single file. Prevents: The high-severity gap that mkdocs build --strict runs only in the post-merge gh-deploy job (.github/workflows/docs.yml:13, 48), so a bad nav entry or broken relative link fails only after merge, leaving a red main and a stale published site.
• Declare the docs toolchain in pyproject.toml as a docs optional-dependency group (or [dependency-groups] docs) with exact pins, resolved into uv.lock, and have both the new PR gate and docs.yml install from it (uv sync --frozen --group docs) instead of the ad-hoc pip install. Why not static: A lint rule can reject an unpinned install line (CE030 above) but cannot create the reproducible, lock-verified environment — that needs a real dependency declaration plus lockfile resolution. Prevents: .github/workflows/docs.yml:42's floating mkdocs-material[imaging]>=9.5,<10 and the fact that grep -n mkdocs pyproject.toml uv.lock Makefile returns zero hits, so no contributor can reproduce the CI docs build locally.
• Make README.md a generated artifact of shared fragments: extract the 7-bullet feature list and the competitive pitch into docs/_includes/*.md, --8<-- them into docs/index.md (pymdownx.snippets is already enabled at mkdocs.yml:66), and add a make readme generator plus a CI drift check (make readme && git diff --exit-code README.md). Simultaneously reduce docs/index.md's "How Coder Eval compares" section to a single link to comparison.md rather than a third rendition of the pitch. Why not static: Snippet includes only exist inside the MkDocs build; GitHub renders README.md raw, so deduplication needs a generation step and a generated-file drift assertion, not a lint rule. A generic duplicate-block detector over prose would be too noisy to gate on. Prevents: The duplicated-copy finding: README.md:31-37 and docs/index.md:29-35 are byte-identical (md5 b20b8cbd67eb5244966b7206197c1cfc), and the competitive pitch exists in four independently-worded renditions (README.md:122-134, docs/index.md:86-100, all of docs/comparison.md, docs/llms.txt:11-20) that have already drifted apart.
• Guard the docs deploy job operationally: add if: github.ref == 'refs/heads/main' to the publish job, set cancel-in-progress: false on the docs-gh-pages concurrency group, and add actionlint + zizmor as pre-commit hooks / a CI step so workflow-shape regressions (unsafe refs, missing guards, expression injection) are caught by upstream-maintained rules rather than one-off CE rules. Why not static: The dangerous property is a runtime one — which ref a workflow_dispatch was fired from, and what ends up force-pushed to gh-pages — so it must be enforced by a job condition; the surrounding workflow-shape linting needs a dedicated Actions-aware linter with its own rule corpus, not a repo-local text rule. Prevents: The unconstrained workflow_dispatch (docs.yml:19) plus cancel-in-progress: true (docs.yml:24-26) and --force deploy, which lets a feature-branch dispatch publish and then stick indefinitely because the push trigger is path-filtered (docs.yml:15-18).
• Introduce a sourcing convention for competitive claims in docs/comparison.md — every non-✅ cell about a third-party project carries a footnote marker resolving to an entry in the Sources block (lines 136-153), with — used wherever no source establishes an absence — and add a light TextRule/pytest check that each footnote marker used in the table has a matching Sources entry. Pair it with a periodic (quarterly) re-verification note in the page front matter. Why not static: Whether a competitor actually lacks a CI gate or a results dashboard is a semantic, externally-sourced truth claim; only the presence of a citation is mechanically checkable, not its correctness or freshness. Prevents: The unsourced absolute negatives at docs/comparison.md:33-34 (CI pass/fail gate and Results dashboard marked ❌ for all five competitors) contradicting the page's own promise at line 19 that "Every claim about another project is grounded in its own documentation", including the Harbor/LangSmith tension at line 81.
• Trim the social-card apt step in .github/workflows/docs.yml:36-40 to libcairo2 (with an inline comment naming why: cairocffi dlopens it; Pillow's manylinux wheel already bundles libjpeg/libpng/freetype) and drop pngquant, which serves the Insiders-only optimize plugin that mkdocs.yml:50-53 does not enable. Record the reduced dependency set alongside the new docs dependency group so the PR-time gate and the deploy job install the same system libs. Why not static: Determining which native libraries a plugin actually dlopens at runtime requires knowledge of the upstream plugin's implementation and the wheel's bundled shared objects — not derivable from any file in this repo. Prevents: The dead-machinery finding at .github/workflows/docs.yml:36 (four unnecessary packages), which also enlarges the surface for the deploy breaking when a package name changes across Ubuntu runner images.

Top 5 Priority Actions

1. Fix the "Bring Your Own Dataset" mislabel that points six surfaces at a Docker page — docs/index.md:81 and :91, docs/comparison.md:48 and :134, mkdocs.yml:91, docs/llms.txt:29 — by relabeling BYOD.md as "Bring Your Own Docker" and repointing dataset links at the dataset section of docs/TASK_DEFINITION_GUIDE.md (most damaging in llms.txt, which is ingested verbatim by LLMs).
2. Add a PR-time mkdocs build --strict job (path-filtered, no deploy) since .github/workflows/docs.yml:13 triggers only on push to main and its only verification is the deploy step at :48, so a broken nav/link entry can only fail after merge; declare the docs toolchain as a docs dependency group in pyproject.toml with a pinned mkdocs-material instead of the floating >=9.5,<10 at docs.yml:42.
3. Correct the TASK_DEFINITION_GUIDE front matter at docs/TASK_DEFINITION_GUIDE.md:4, which claims "all 14 success criterion types" and "dataset fan-out" while the guide documents 12 types (no commands_efficiency, no classification_match) and has no dataset section — either add the missing sections or soften the claim, and consider a doc-drift test asserting every SuccessCriterion union member has a ### <type> heading.
4. Harden the deploy workflow: SHA-pin actions/checkout and actions/setup-python at .github/workflows/docs.yml:32-33 to the SHAs already used in pr-checks.yml (this is the one job with contents: write, and dependabot.yml documents SHA pinning as policy), add if: github.ref == 'refs/heads/main' to guard the unconstrained workflow_dispatch from force-publishing a feature branch over gh-pages, and add timeout-minutes: 15 to the publish job.
5. Clean up the published content and metadata: set Documentation = "https://uipath.github.io/coder_eval/" at pyproject.toml:56 (currently the raw GitHub docs tree, undercutting the SEO goal from PyPI's high-authority backlink), downgrade the unsourced absolute ❌ negatives at docs/comparison.md:33-34 to the — marker the same table already uses, scope out internal-only material still shipping publicly (the destructive CODER_EVAL_REMEDIATE_HOME_PLUGINS env var at docs/USER_GUIDE.md:148-150 and the MST-9674 tracker id at docs/TASK_DEFINITION_GUIDE.md:889), and de-duplicate the pitch copy now living in four places via the already-enabled pymdownx.snippets.

---

Stats: 0 🔴 · 2 🟠 · 5 🟡 · 6 🔵 across 8 axes reviewed.