Repository navigation
fix(website): point llms.txt at /hub/models and Comfy Router, and fail on redirected llms links - #20198
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. 2 Skipped Deployments
|
🔗 Website PreviewWebsite Preview: https://comfy-website-preview-pr-20198.vercel.app Models is built and gated by PostHog (workshop-enabled), using staging Cloud. This commit: https://website-frontend-5mlnzykmf-comfyui.vercel.app Last updated: 2026-10-06T11:11:40Z for 🌐 Website E2ETip All tests passed.
🌾 Fallow: ✅ PassedFallow audits the diff for duplication, complexity and dead code. It gates on new findings only, so when the audit completes, a red is something this PR introduced or moved — never inherited backlog. A red can also mean the audit itself did not finish; the details below say which happened. Fallow gates on new findings in changed files only, so everything below was introduced or moved by this PR (9 changed files).
Pre-existing findings are baselined in Reproduce locally# the base the gate compared against
npx fallow audit --root . --quiet --format json \
--changed-since 65d257e48111ba1ba4cbb8e9534668b9d38dc8bb
Full docs: ADR-DEVEX-LINT-0015 🧩 Custom Nodes Ecosystem Matrix: 🟡 Withheld — could not measureExecutes the frontend JS of every registry pack that ships any, against this change, in both renderers. The eight This candidate changes nothing the matrix measures, so every expensive stage was skipped. Reproduce locally# build one shard's fixture, then run it
python3 scripts/registry-census/build_matrix.py --shard 1/4
MATRIX_RENDERER=vue MATRIX_VUE=1 \
npx vitest run --config vitest.matrix.config.mts
Full docs: scripts/registry-census/README.md |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configuration
📒 Files selected for processing (3)
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 1 remain after this review. 📝 WalkthroughWalkthroughThe agent-readable guidance now covers Comfy Router calls and hosted-model catalogs. The website validator checks root and nested built LLM files for redirected, missing, and non-canonical links. ChangesLLM Guidance and Link Validation
Priority: ➖ Normal Estimated code review effort: 3 (Moderate) | ~25 minutes Change: Bug fix Suggested reviewers: Merge Risk: ⚪ Minimal · up to The updated Router guidance matches the documented API, and no actionable merge risk is established. Merge after normal checks. Architecture SummaryArchitecture risk: 🔵 Low · up to The change affects 1 system. Changed systems: Architecture concerns Review detailsSystems and components
Before / after behavior
🚥 Pre-merge checks | ✅ 7✅ Passed checks (7 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
QA: llms.txt links and the stale-link checkChecked on a local
Counts: 11 checks, 11 pass. Not covered here: redirects that live only in comfy-router, not in |
There was a problem hiding this comment.
🔍 Cursor Review — Consolidated panel
Triggered by @MaanilVerma.
Every reviewer in the matrix failed to contribute — see the panel summary for which cells errored, and the run logs for the underlying cause.
Panel: 0/8 reviewers contributed findings.
Reviewers that did not contribute: claude-opus-5-thinking-max:adversarial (error), gemini-3.1-pro:adversarial (error), gpt-5.6-sol-max:adversarial (error), kimi-k3-max:adversarial (error), claude-opus-5-thinking-max:edge-case (error), gemini-3.1-pro:edge-case (error), gpt-5.6-sol-max:edge-case (error), kimi-k3-max:edge-case (error)
Codecov Report✅ All modified and coverable lines are covered by tests. @@ Coverage Diff @@
## main #20198 +/- ##
==========================================
- Coverage 85.44% 85.32% -0.13%
==========================================
Files 3519 3420 -99
Lines 168831 165757 -3074
Branches 44308 43246 -1062
==========================================
- Hits 144257 141424 -2833
+ Misses 22591 22500 -91
+ Partials 1983 1833 -150
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 99 files with indirect coverage changes 🚀 New features to boost your workflow:
|
webreviewer-bot
left a comment
There was a problem hiding this comment.
Gate: UNMET — this verdict is code-only
The webreviewer label was applied by @benjcooley, not by the author @MaanilVerma (timeline: labeled webreviewer benjcooley 2026-10-05T17:27:36Z). The label is the author's assertion that they checked the preview deploy. Applied by someone else, the author-verified-preview gate is unmet. Everything below is a code and artifact review only and must not be read as merge-ready.
Verdict: APPROVE (code-only)
I built this head and ran the gate for real rather than reading it. The content change is clean and the new check does what it says on CI. Two fail-open gaps in the check itself are SHOULD FIX.
| Severity | Count |
|---|---|
| BLOCKER | 0 |
| SHOULD FIX | 2 |
| TRIVIAL | 2 |
Interaction with #20191 — none. Either merge order is safe.
This review was briefed on the assumption that #20191 could flip this check's result. It cannot. I diffed the two vercel.json blobs directly:
#20198 head (c8c9494) |
#20191 head (264c596) |
|
|---|---|---|
| redirect rows | 857 | 857 |
| sources added / removed | — | 0 / 0 |
| destinations changed | — | 0 |
| patterned sources | 0 | 0 |
permanent: true |
60 | 846 |
#20191 flips only the permanent field on 786 /models/* rows. It adds, removes and rewrites no source and no destination.
The new check reads vercel.json and maps redirects[].source only (validate-llms-txt-links.ts:41-46). It never reads permanent, statusCode or destination. A 307 source and a 308 source are the same string, so:
- Would the check pass if #20191 merged first? Pass — bit-for-bit the same result.
- Does it pass against this head today? Pass. I ran it on a real
build:e2eofc8c9494:llms.txt link validation passed for 7 file(s). - Correct merge order: either. There is no landmine for whoever merges second. The two PRs also share no files — #20191 touches
redirects.ts/redirects.test.ts/vercel.json, this one touches only the llms.txt lib, script, tests andpublic/llms.txt. The PR description's stated reason for leavingredirects.test.tsalone (conflict avoidance with #20191) is correct and the restraint is right.
Also worth recording, because the framing invites the opposite conclusion: the check never makes an HTTP request. It is fully hermetic — readFileSync over dist/ plus vercel.json. It cannot be broken by an outage or a rate limit, and it does not care what production currently serves. There is no "follows redirects vs. treats 3xx as failure" question, and a trailing-slash 308 cannot fail the build — normalizePath strips trailing slashes before any comparison (verified below).
The emitted llms.txt — audited, not read from the template
Root llms.txt is a static public/ passthrough, so I verified the deployed artifact is the one under review: https://comfy-website-preview-pr-20198.vercel.app/llms.txt is md5-identical (dced88f12925b33b1826a8ed0019f359) to public/llms.txt at c8c9494, and the preview comment confirms the deploy is c8c9494 at 17:29:47Z. The per-commit URL website-frontend-4je49itl4 serves the same 40448 bytes.
Every link fetched against production, with and without redirect following:
| Measure | Result |
|---|---|
Links (- [t](u): d bullets) |
227 (base 225) |
| HTTP 200, no redirect hop | 226 / 227 |
| 3xx redirect sources | 0 |
| 404s | 0 |
| Duplicates | 0 |
| Relative / non-https | 0 |
noindex targets |
0 (/hub/models/ has no robots meta on prod) |
| Added by this PR | https://comfy.org/hub/models.md, https://docs.comfy.org/development/comfy-router/models.md |
| Removed by this PR | 0 — no coverage dropped |
The one non-200 is https://api.comfy.org/openapi → 405 on HEAD only; GET returns 200 / application/json / 1.58 MB. Pre-existing link, not a defect.
Format conformance holds. H1 # Comfy, blockquote summary, 17 H2 sections, and all 227 bullets parse against the strict LINK_LINE regex — zero malformed - [ lines across 348 lines. The edit did not break the llms.txt convention.
"Comfy Router" targets verified as the intended pages:
docs.comfy.org/development/comfy-router/models.md→ 200, titled "Comfy Router models", grouped by provider, lists model IDs with input/output modality. Matches its description exactly.docs.comfy.org/development/comfy-router/quickstart.md→ 200.- The API shapes asserted in the new prose are real:
GET /v2/models/bfl/flux-3-videoandGET /v2/models/bfl/flux-3-video/openapi.jsonboth return 401, not 404 — the paths exist and are key-gated, as the text claims. comfy.org/hub/models.md→ 200, 14 KB, 148 model links under## All models by provider, frontmatter canonicalhttps://comfy.org/hub/models/. The row's claim that "each model page also has a markdown twin at its URL plus.md" checks out — e.g./hub/models/beeble-switchx-image-edit.md→ 200. (I initially measured a 404 here; that was my own wrong slug — a stale/models/slug rather than a/hub/models/one. Corrected.)
The check itself
Where it runs: .github/workflows/ci-website-build.yaml, job build (ubuntu-latest), step "Validate llms.txt links", immediately after build:e2e. It process.exit(1)s, so it fails the build. website-build is success at this head.
Can it actually fail? Yes — I mutated the built artifact and confirmed each assertion fires for the right reason, then restored and verified by blob hash against git rev-parse c8c9494:<path> (all MATCH, git status clean):
| Mutation | Result |
|---|---|
Add https://comfy.org/models/ (a literal vercel.json source) to dist/llms.txt |
exit 1 — ... is a redirect source |
Add https://comfy.org/this-page-does-not-exist/ |
exit 1 — ... is not in the build |
Rewrite dist/download/index.html canonical to /somewhere-else/ |
exit 1 — ... now canonicalizes to https://comfy.org/somewhere-else/ |
False positives: none found. I drove findStaleLinks with the real 857-source corpus over every shape worth worrying about — trailing slash present and absent, #anchor, ?query, anchor+query, /zh-CN/ locale prefix with and without slash, dotted slugs (/chatgpt-image-2.5/), .md twins, section llms.txt, and Workflows-app routes. All 12 pass. The only two that fail are genuine redirect sources. normalizePath and new URL().pathname handle slashes, queries and anchors correctly.
Hermetic: yes. Fails closed on a real stale link — and fails open in the two cases below.
SHOULD FIX
1. scripts/validate-llms-txt-links.ts:51 — the section-file walk is a no-op on Windows, and it reports success.
This is the PR's headline capability ("walks dist/llms.txt, dist/llms-full.txt and every section llms.txt"). readdirSync(..., { recursive: true }) returns platform-separated paths, but the filter hard-codes /. On Windows path.sep is a backslash, so p\supported-models\llms.txt never matches endsWith('/llms.txt') and zero section files are collected.
Demonstrated on this head's real build: I injected two genuinely stale links (a redirect source and an unbuilt page) into dist/p/supported-models/llms.txt and the pristine script printed llms.txt link validation passed for 2 file(s). and exited 0. With the separator normalised, the same tree reports passed for 7 file(s).
Linux CI is unaffected (7 files, correct), so production is safe — but a Windows contributor gets a silent false green from a gate whose whole point is catching this, and "passed for 2 file(s)" reads as plausible. One line.
2. src/lib/llms-txt.ts:50 — the deleted corpus tripwire left the new matcher unguarded.
redirectSourcePattern covers :slug, :path+ and /:path*. Vercel also accepts :param(regex) and :param?, and neither is recognised; both fall through to the literal branch and get escaped into a regex that matches nothing. That is a silent fail-open — the check would quietly stop protecting that prefix.
That matters more than it would have before this PR, because uses only literal redirect sources for stale link checks was deleted in src/config/llms-txt.test.ts. That test was the tripwire guaranteeing the matcher was sufficient for the whole corpus. The replacement tests exercise redirectSourcePattern against hand-written inputs — they test the matcher, not the data — so nothing now asserts that every real redirects[].source is a shape the matcher understands.
I confirmed all 857 sources are literal today (0 patterned), which is good news twice over: nothing is broken now, and the entire pattern branch added here is currently unexercised against real data. The exposure is a single future row in redirects.ts. Re-add a corpus assertion: every source matches a supported shape.
TRIVIAL
3. scripts/validate-llms-txt-links.ts:13 — isFile will misread a dotted page slug.
/\.[a-z]+$/i treats any path ending in . plus letters as a file, resolving to dist/<slug> instead of dist/<slug>/index.html → a false is not in the build that fails the build. Safe today: I scanned all 2692 dist/ entries and every dotted slug (chatgpt-image-2.5, ltx-2.5, bfl--flux-pro-1.1--generate-images, recraft--v4.1-pro-text-to-image--generate-images) ends in a digit, so none match. A slug like flux.dev would break the build. An explicit extension allowlist (.md, .txt, .xml, .json) removes the trap.
4. public/llms.txt:47 — the largest thing this PR adds is the part the new gate cannot see.
parseLlmsTxtLinks only matches - [title](url): description. Bare URLs in prose, and URLs inside a description, are invisible to the check. This PR adds four of them (two on line 47, two on line 50) plus https://comfy.org/hub/models.md inside the line 101 description. All five are 200 today — I verified each — but nothing in CI will notice if they move. Related: internalLinks filters to hostname === 'comfy.org', so all 16 docs.comfy.org links are skipped regardless of form, including the new line 64 row. The author already discloses the adjacent comfy-router gap in Review Focus; this is the same class and worth the same note.
Existing reviews
| Review | Assessment |
|---|---|
github-actions[bot] — COMMENTED |
No findings, and not because the code is clean. All 8 Cursor panel cells errored (0/8 reviewers contributed); the 8 cursor-review matrix check-runs are cancelled. This review carries no signal in either direction and should not be counted as coverage. |
coderabbitai[bot] — APPROVED |
Empty body, zero inline comments (paginated the /comments endpoint: 0 results). An approval with no stated basis. Neither confirms nor disputes anything; I did not rely on it. |
No human review, no standing CHANGES_REQUESTED, nothing to dismiss. Neither automated review raised either gap above.
Exposure
Catalogue / content entry. No new app, no new feature surface, no flag check touched — the diff is llms.txt copy plus a build-time validation script, its lib and its tests. The destinations it now points at (/hub/models/, /hub/models.md) are already live and indexable on production today (200, no robots meta; /models already 307s there). The preview annotates Models as PostHog-gated; rollout state is unknown from the repository and nothing here should be read as implying otherwise.
CI (check-runs API, paginated)
182 runs at c8c9494: 66 success, 107 skipped, 9 cancelled, 0 failure. The 9 cancelled are the 8 errored cursor-review matrix cells plus one superseded deploy-preview (a later deploy-preview at the same SHA succeeded). Green is real for everything that gates: website-build, website-unit, website-e2e (4/4 shards), lint-pr (lint, typecheck, typecheck-aux, repo-checks), unit / test, build.
What I ran
Temp clone at c8c9494e85ae89b54e8094a1f8ca62f02975b8b7 (never the shared checkout):
pnpm install --frozen-lockfile→ OKpnpm --filter @comfyorg/website build:e2e→ 1195 pages, 2692 dist files, 474 markdown twins — verified by inspectingdist/, not by exit codepnpm --filter @comfyorg/website validate:llms-txt-links→ pass (2 files on Windows / 7 files with the separator normalised)pnpm exec vitest run src/lib/llms-txt.test.ts src/config/llms-txt.test.ts→ 2 files, 44 tests, 44 passed (count checked, not exit code)- 227 production link probes, no-follow and follow;
vercel.jsoncorpus diff against #20191; 12-case false-positive harness; 4 mutations with restore and blob-hash verification
No differences from CI other than the Windows separator divergence, which is finding 1 itself. Clone deleted.
Two harness traps hit and corrected, flagged because they produce confident wrong answers: a sed patch whose backslashes were mangled by shell escaping into an unterminated regex (re-done via a node script), and child_process.execSync failing ENOENT on cmd.exe under Git Bash, which would have silently aborted the #20191 comparison (re-done via git show to files).
Red flags, in order
- The section-file walk — the PR's headline capability — is inert on Windows and prints "passed". Fail-open in a gate.
- The corpus tripwire was deleted and replaced with tests of the matcher rather than tests of the data, leaving
:param(regex)/:param?as a silent future fail-open. - Both automated reviews on this PR are empty — one errored 8/8, one approved with no body. The green here is thinner than the badge count suggests.
Not blocking, and the artifact itself is in good shape: 227/227 links resolve, zero redirect sources, zero dropped coverage, format intact.
redirectSourcePattern treated `:slug(a|b)` as `:slug` followed by a literal `(a|b)`, so a source such as `/p/supported-models/:slug(4x-ultrasharp|flux1-dev-fp8).md` matched no path and a stale llms.txt link to an old file URL passed silently. A constrained param now compiles to an alternation of its escaped literal values, so only the listed slugs match, with or without the trailing slash or `.md` suffix. Redirect sources stay case-sensitive, as Vercel documents. Any other path-to-regexp syntax (a regex constraint, an unnamed group, a modifier) throws instead of matching nothing. This makes #20198 compatible with the grouped redirect sources that #20291 adds to vercel.json.
20ae4d2
webreviewer-bot
left a comment
There was a problem hiding this comment.
webreviewer-bot — code review, PR #20198 (cycle 2)
Head reviewed: 0af523145a9fba416266efe1de5cd9de6bd03d4f. Base main. +335-75 across 5 files.
Gate: UNMET — this verdict is code-only
The
webreviewerlabel was applied by @benjcooley at2026-10-05T17:27:36Z, not by the author @MaanilVerma. I re-paginated the timeline: there is no later labeling event, and the only labels the author applied arecursor-review. The label is the author's assertion that they checked the preview deploy; applied by someone else, the author-verified-preview gate is unmet.Separately, the label also predates this head by 11 hours — it was applied at
17:27:36Zon Oct 5, and0af5231was pushed at2026-10-06T04:37:07Z. The QA comment in this thread is a real, thorough verification, but it is dated15:20:40Zand states it was run againstccfe2c1317— two commits before this head. So there is no author verification of the artifact now deployed.Everything below is a code and artifact review only and must not be read as merge-ready. Readiness is not my call.
Verdict: APPROVE (code-only)
20ae4d2 genuinely fixes cycle-1's SHOULD FIX B, and it fixes it better than I suggested. 0af5231 fixes a real content defect I missed. The one remaining SHOULD FIX is the Windows separator bug, unchanged and re-demonstrated below.
| Severity | Cycle 1 (c8c9494) |
Cycle 2 (0af5231) |
|---|---|---|
| BLOCKER | 0 | 0 |
| SHOULD FIX | 2 | 1 |
| TRIVIAL | 2 | 3 |
Movement: SHOULD FIX B resolved; a strictly narrower remnant of it is now TRIVIAL. SHOULD FIX A and both TRIVIALs unchanged.
SHOULD FIX B — resolved, and the approach is better than what I asked for
Cycle 1: redirectSourcePattern covered :slug, :path+ and /:path*; :param(regex) and :param? fell through to the literal branch, got escaped, and produced a regex matching nothing — a silent fail-open. I asked for a corpus-level tripwire test.
The author did something stronger: the splitter now throws on any param form it does not understand, so there is no silent no-match to tripwire. The build gate maps redirectSourcePattern over every vercel.json source on every build, so the build itself is now the tripwire — strictly better than the unit test I suggested, because it covers the real data rather than hand-written inputs.
Probed by execution against the real exported function at this head (not by reading):
| Source | Result | Pattern |
|---|---|---|
/x/:slug |
OK, matches /x/abc |
/^\/x\/[^/]+$/ |
/x/:slug+ |
OK, matches /x/a/b |
/^\/x\/.+$/ |
/x/:path* |
OK, matches /x/a/b |
/^\/x(?:\/.*)?$/ |
| `/x/:kind(a | b)` | OK, matches /x/a, rejects /x/c |
| `/x/:kind(flux.1 | sd-xl)` | OK, matches /x/flux.1, rejects /x/fluxX1 |
/x/:slug(\d+) |
THROWS Unsupported redirect source constraint |
— |
/x/:id([0-9a-f]{8}) |
THROWS | — |
/x/:slug? |
THROWS Unsupported redirect source syntax |
— |
/x/:a(b)? |
THROWS | — |
| `/x/:a(b | (c))` (nested) | THROWS |
/x/(.*) |
THROWS | — |
/(?<lang>[a-z]{2})/x |
THROWS | — |
Answering my own cycle-1 question directly: :param? is not handled, it throws — which closes the fail-open, because the failure is now loud. Note the escaped-literal case works correctly too: flux.1 escapes to flux\.1 and does not match fluxX1, so a regex metacharacter inside an alternative cannot widen the match.
Over-matching: none found. This was my main worry — a splitter too liberal about regex params would flag a correct link as a redirect source and redden every website PR. It does not:
- Driving
findRedirectedLinkswith all 227llms.txtbullets at this head against the real 857-sourcevercel.json: 0 flagged. Zero false positives. - All 857 sources compile, 0 throw (all literal today, as in cycle 1).
The fix is exercised by real upcoming data, which cycle 1 could not say. 20ae4d2 names #20291, so I checked it: #20291's vercel.json has 1077 rows with 20 genuinely patterned sources, all of the :slug(a|b|…) literal-alternation form (42–78 alternatives each, in four suffix variants: bare, /, .md, .md/). Against this head's splitter: 0 throw, 20/20 correct — each matches its first and last listed slug, and rejects both an unlisted slug and a prefixed near-miss (xflux1-dev-fp8). Cycle 1 noted the pattern branch was unexercised against real data; it no longer is.
Mutation check — the new tests are real, not decoration. In my own temp clone I swapped src/lib/llms-txt.ts back to its c8c9494 content, keeping the new tests. 12 of 65 tests fail, and the four most important fail for exactly the right reason:
× redirectSourcePattern > rejects unsupported source syntax /posts/:id(\d+)
× redirectSourcePattern > rejects unsupported source syntax /payment/(.*)
× redirectSourcePattern > rejects unsupported source syntax /p/:slug(a|b)?
× redirectSourcePattern > rejects unsupported source syntax /p/:slug(a|(b))
Those are the fail-open assertions: under the old code redirectSourcePattern returned a never-matching regex instead of throwing, so toThrow(/Unsupported/) fails. The other 8 failures are at llms-txt.test.ts:122 and :160, all expected [] to deeply equal [{…}] — the grouped source silently matching nothing. Restored, blob verified 9695eb65b648ba805d9c0dfc2fbe7128d27a3b5d against git rev-parse HEAD:apps/website/src/lib/llms-txt.ts, git diff --quiet clean, 65/65 green again.
0af5231 — a real content fix, and I missed the defect in cycle 1
The two changed lines touch prose only — no URL changed. Diffed byte-for-byte: lines 63 and 65, both replacing markdown twin at its URL plus ``.md``` with markdown twin at its URL with the trailing slash replaced by ``.md```. Nothing to re-probe for status or redirect hops.
But the old wording was wrong, and I passed it in cycle 1. 170 of the 178 comfy.org links in this file carry a trailing slash, so "at its URL plus .md" literally meant /foo/.md. Measured on production:
| URL | Status |
|---|---|
https://comfy.org/hub/models/beeble-switchx-image-edit/ |
200 |
https://comfy.org/hub/models/beeble-switchx-image-edit.md |
200 |
https://comfy.org/hub/models/beeble-switchx-image-edit/.md |
404 |
https://comfy.org/hub/models/flux-2-max-text-to-image.md |
200 |
https://comfy.org/hub/models/flux-2-max-text-to-image/.md |
404 |
https://comfy.org/hub/models/flux-2-pro-text-to-image.md |
200 |
https://comfy.org/hub/models/flux-2-pro-text-to-image/.md |
404 |
https://comfy.org/p/supported-models/seedance-bytedance.md |
200 |
https://comfy.org/p/supported-models.md, /download.md |
200, 200 |
The old sentence described a pattern that 404s; the new one resolves. In a file whose only audience is an agent constructing URLs from a rule, that is worth the commit. Good catch, and it is the kind of defect no gate here can see — parseLlmsTxtLinks reads link targets, never the prose describing them.
Deployed artifact re-verified, with one correction to my cycle-1 method. Cycle 1 reported the preview /llms.txt md5-identical to public/llms.txt. At this head it is not — and that is not a defect. The preview serves 40788 bytes, the repo file 40510. Diffed: the preview has exactly two extra bullets (/customers/hakoniwa-yui/ and its zh-CN twin) which are present on main and absent from this branch — the base moved, and Vercel builds the merge. Both of this PR's changed lines are present in the preview and the old wording is gone (grep -c 'markdown twin at its URL plus' → 0). The per-commit URL for 0af5231 serves the identical bytes. So the artifact under review is the deployed one; an md5 equality check against public/ is simply the wrong assertion once the base has advanced.
SHOULD FIX A — the Windows separator bug still stands, re-demonstrated at this head
Not touched by either new commit. validate-llms-txt-links.ts:51 still reads endsWith('/llms.txt'). I re-proved it end to end against the real script at this head, with a dist/ containing a section file holding two genuinely stale links (a literal vercel.json redirect source and an unbuilt page):
# readdirSync(dist, { recursive: true }) on Windows
["llms-full.txt","llms.txt","p","p\\supported-models","p\\supported-models\\llms.txt"]
endsWith('/llms.txt') matches: 0
# pristine script, Windows
llms.txt link validation passed for 2 file(s). EXIT=0
# same dist/, separator normalised
llms.txt link validation failed (2 stale link(s)):
dist\p\supported-models\llms.txt: [Stale](https://comfy.org/career) is a redirect source
dist\p\supported-models\llms.txt: [Ghost](https://comfy.org/definitely-not-a-real-page/) is not in the build
EXIT=1
The gate's headline capability — walking every section llms.txt — collects zero files on Windows, and the script prints a plausible success and exits 0. Linux CI is unaffected (7 files, correct), so production is not at risk and website-build being green here is meaningful. The cost is a Windows contributor getting a confident false green from the exact gate meant to catch this. Still one line. Inline, with a suggestion — note basename(file) === 'llms.txt' is not the right fix, it matches the root dist/llms.txt too and would double-count it (I measured 2 matches); normalise the separator instead.
TRIVIAL
Three, all one line each, all inline. (2) is the downgraded remnant of SHOULD FIX B: UNSUPPORTED_SOURCE_SYNTAX = /[()?*+]/ omits { and }, which the path-to-regexp version Vercel uses for source treats as group delimiters — so /cloud{/enterprise} still compiles to /^\/cloud\{\/enterprise\}$/ and silently matches nothing. Verified by execution. Much narrower than the original gap (0 of 1077 real sources use braces, and the common {…}? form already throws on the ?), hence TRIVIAL rather than SHOULD FIX. (3) and (4) are cycle-1 items 3 and 4, unchanged.
Checked and cleared
- CI, via the paginated check-runs API (not
gh pr checks): 93 runs at0af5231— 54 success, 39 skipped, 0 failure, 0 pending, 0 cancelled. Green for everything that gates:website-build,website-unit,website-unit-gate,website-e2e(4/4 shards),lint-pr(lint / typecheck / typecheck-aux / repo-checks),unit / test,build,fallow. Cycle 1's 9 cancelledcursor-reviewcells do not appear at this head. I did not treat green as evidence of correctness — the mutation check above is the mechanism. - The check is still fully hermetic.
readFileSyncoverdist/plusvercel.json, zero HTTP requests. Unchanged by these commits. - No interaction with #20191. Settled in cycle 1 and nothing here changes it: #20191 flips only
permanent, and this check readsredirects[].sourceonly. - #20291 does update
public/llms.txt, so the ordering constraint I found is already handled by the author: against #20291's 1077 sources, two bullets in this head'sllms.txtare correctly flagged (/p/supported-models/and/p/supported-models/llms.txt) because #20291 moves that section to/hub/models/local. That is the gate working as designed, not a false positive. Worth stating plainly: if these two PRs land in either order without #20291'sllms.txtedit,website-buildfails loudly — which is the right outcome. - The
.replace(/\.md$/, '')this PR adds toconfig/llms-txt.test.tsdoes loosen that coverage test — it will now accept<route>.mdfor any allowed route, andhttps://comfy.org/privacy-policy.md404s on production today while/privacy-policy/is an allowed entry. Not raising it: the build gate'sexistsSync(distFileFor(path))checks the rootllms.txtunconditionally (hard-coded, so even the Windows path covers it), so a non-existent.mdtwin still fails the build. Backstopped. - An unsupported source throws uncaught —
main()has no try/catch, so it surfaces as a stack trace rather than the script's own one-line message. Loud and non-zero, so not a finding, but acatcharoundmain()would make the failure read like the others. - Exposure: catalogue / content entry. No new app, no new feature surface, no flag check, default or targeting, touched — I grepped the whole diff for
posthog/featureFlag/isEnabled/ flag indexing: zero hits. The five files are llms.txt copy, a build-time validation script, its lib and its tests. The preview annotates Models as PostHog-gated; rollout state is unknown from the repository and nothing here should be read as implying otherwise.
Existing reviews
coderabbitai[bot] APPROVED at this head (04:43:47Z) after two dismissed reviews. Its comment says "No actionable comments" and it processed only apps/website/public/llms.txt between 20ae4d2 and 0af5231 — so it did not look at the splitter change at this head at all. I reached the same verdict independently and by execution; I did not rely on its approval. Its walkthrough claim that "Redirect matching supports Vercel-style patterns" is too generous: it supports literal-alternation patterns and throws on the rest.
My cycle-1 review was dismissed by the author's pushes, so there was nothing of mine to dismiss. No human review. No standing CHANGES_REQUESTED.
What I ran
Own temp clone at 0af523145a9fba416266efe1de5cd9de6bd03d4f, outside any shared checkout, deleted afterwards:
pnpm install --frozen-lockfile→ OKpnpm exec vitest run src/lib/llms-txt.test.ts src/config/llms-txt.test.ts→ 2 files, 65 tests, 65 passed (count read, not exit code; cycle 1 saw 44)- Mutation:
llms-txt.tsreverted toc8c9494→ 12 failed / 53 passed; restored, blob hash verified, tree clean, 65/65 green - A probe driving the real exported
redirectSourcePatternandfindRedirectedLinksover 18 param forms, the 857-source head corpus, the 1077-source #20291 corpus and all 227 realllms.txtbullets pnpm exec tsx ./scripts/validate-llms-txt-links.tsagainst a hand-builtdist/, pristine and separator-normalised- Production probes of the
.mdtwin rule, three model pages both ways plus/.md; preview and per-commit-URL byte comparison againstpublic/llms.txt
Differences from CI: only the Windows separator divergence, which is SHOULD FIX A itself. One harness trap hit and corrected, recorded because it produces confident wrong answers: a character class passed inline to node -e through Git Bash had its backslashes eaten into an unterminated regex — the same class of trap as cycle 1's mangled sed. Re-done from a script file.
Red flags, in order
- The section-file walk is still inert on Windows and still prints "passed". Fail-open in a gate, unchanged across two commits.
- No author verification of this head. The label is someone else's and 11 hours stale; the author's (good) QA ran against
ccfe2c1317, two commits back. The.md-twin wording this PR just fixed is exactly the sort of thing only a human reading the artifact catches. - coderabbit's approval at this head looked only at
llms.txt— it never reviewed the splitter change it is approving. Thinner coverage than the badge suggests.
Credit where due: 20ae4d2 fixes a fail-open I raised, chooses fail-closed over the weaker tripwire I suggested, lands 21 new tests that genuinely fail without the fix, and is motivated by real sources arriving in #20291 rather than a hypothetical. 0af5231 fixes a 404-producing sentence I passed. The artifact is in good shape: 227 bullets, 0 flagged by the real corpus, no URL changed, no coverage dropped.
…d slugs and brace sources
7c8fa37
🔧 Auto-fixes AppliedThis PR has been automatically updated to fix linting and formatting issues.
Changes made:
|
dante01yoon
left a comment
There was a problem hiding this comment.
Reviewed 8703d8b: the built llms.txt/full-text traversal, Windows separators, constrained redirect matching and fail-closed unsupported syntax, file allowlist, canonical drift and workflows exceptions. The previously reported separator and matcher gaps are fixed. The Router link and route guidance match the official docs.
PASS: 75 focused llms-txt tests on an isolated archive of this head; current GitHub checks have no failing or pending runs. NOT_RUN: local full website build and browser verification. No blocking findings.
… a redirect (Comfy-Org#20292) ## Summary CI now fails when a published model page leaves the build without a permanent redirect to a live page. This stops a catalogue refresh from turning a ranking page into a silent 404, as Comfy-Org#15487 did to 10 `/p/supported-models/` pages. ## Changes - **What**: - `src/config/published-model-pages.json` is a committed list of every published model page (533 today). It is seeded from the sitemap of a main build. - `check:published-model-pages` reads the built sitemap and `vercel.json`. Each listed page must be in the sitemap or have a permanent redirect from its slash form to a page in the sitemap. Otherwise it prints `page X removed: add a permanent redirect in src/config/redirects.ts or restore the page`. - A model page in the sitemap that is not on the list also fails. `record:published-model-pages` adds new pages and never removes one. - "Model page" means anything under the Models URL registry roots (`/hub/models`, `/hub/workflows`, `/hub/apps`, `/models`) or `/p/supported-models`, in any locale. New `/hub/models/local/<slug>/` pages are covered with no extra code. - New CI step after the models-html check in `ci-website-build.yaml`. - Redirect sources may use `:name` and `:name(a|b)` params (as the grouped `/p/supported-models/` redirects do). Their params fill the destination before the live check. Other source syntax fails the check. - On a PR, the check also compares the list with the base branch. An entry dropped from the list fails unless it is in `RETIRED_WITHOUT_REDIRECT` with a reason. - Dedupe with Comfy-Org#20198's matcher once both land. Changed from the ticket: the ticket guarded slugs in `generated-models.json` from a unit test. The pages now live at `/hub/models/` as well, and the in-flight local pages come from a different source. So the check reads the built sitemap, which covers every model page the same way. A page that is retired on purpose goes in `RETIRED_WITHOUT_REDIRECT` with a reason. The 10 pages Comfy-Org#15487 already removed are not seeded; their redirect and restore tickets own them. ## Review Focus - [x] A new model page fails CI until someone runs `record:published-model-pages` after a build. Without this, new pages would be unprotected. It also means the open supported-models PR must record its new pages once this merges. - [x] The redirect must cover the slash form (`/x/`), because that is the address in the sitemap. `sync:redirects` always writes both forms. Fixes FE-2971 https://linear.app/comfyorg/issue/FE-2971
…-Org#20291) MaanilVerma ## Summary The supported-models section moves into the Hub. The 334 open-weight model file pages keep their content and move to `/hub/models/local/<slug>/`, and every old address sends a one-hop 308 to its new home, so the ~45k clicks per 90 days these pages earn carry over. ## Changes - **What**: - The page, its markdown twin, the index and the `llms.txt` catalogue move from `src/pages/p/supported-models/` to `src/pages/hub/models/local/`. Only URLs change: the title, description, h1, Hugging Face button, folder, docs link, FAQ and JSON-LD stay the same. - New registry kind `local`. `isWorkshopRoute` leaves `/hub/models/local/*` alone, so these pages are always built, indexable, in the sitemap and twinned, also with `WORKSHOP_IN_BUILD=0`. The pages are file routes, so the flag cannot drop them. - Redirects (all 308, generated into `vercel.json`): - Each file page, its slash form and its `.md` match five `/p/supported-models/:slug(a|b|…)` rules that name only built slugs. The 14 retired slugs and `stability-ai` stay 404. - The 11 quantization aliases go to their file's new page. - The 49 partner pages go to the Hub model for the same model or version (13), to the provider's current model (18), or to `/hub/models/` when no Hub page matches (18). They match one `:slug(a|b|…)` rule per destination, 24 in all. `__fixtures__/published-partner-model-pages.json` pins each destination outside the table. A partner slug that leaves the data keeps its row, so its published address keeps working. - `retiredLocalModelPages` (empty for now) sends a file page that a data refresh removes to the page that replaces it. - The index, `.md` and `llms.txt` go to `/hub/models/local/`, `/hub/models/local.md` and `/hub/models/local/llms.txt`. - Footer, nav, drops card, showcase CTA, root `llms.txt` and `llms-full.txt` link the new addresses. The build has 0 links to `/p/supported-models`. ## Review Focus - [x] One PR, not the A/B split: the change is 248 non-test lines (56 of them the partner table). The pages move instead of being duplicated, so no indexable duplicate ever exists. - [x] Route budget: 946 redirects + 4 headers (was 857 + 4). The slug groups stay under Vercel's 2,048-character source limit (test). - [x] Canonical churn: `__fixtures__/published-local-models.json` freezes the 334 published slugs. If `generate-models.ts` turns one into a `canonicalSlug` stub, CI fails. - [x] Partner pages: `.md` twins redirect to the Hub page's twin. The partner branches in the shared copy modules are now unused in the build; they stay for a follow-up cleanup. - [x] Launch-page breadcrumbs ("Models") point at `/hub/models/` through `routes.workshop`. Only the local file pages keep the crumb to `/hub/models/local/`. - [x] The Drops card keeps its link to `/hub/models/local/`, and its copy now describes open-weight model files that you run locally. - [x] Sibling PRs: Comfy-Org#20191 also edits `redirects.ts` and `vercel.json` (merge main and re-run `sync:redirects`). Comfy-Org#20198 edits `public/llms.txt` line 49, and its redirected-link check must expand `:slug(…)` sources like `llms-txt.test.ts` does here. - [x] How Comfy-Org#19786 ports onto this: - Its page and component changes apply to `src/pages/hub/models/local/` instead of `src/pages/p/supported-models/`. Its `public/llms.txt` lines move to the new URLs. - Of its 32 new slugs, 27 are local files and need no rows: the slug groups pick them up from the data. The other 5 are partner slugs (`fish-audio`, `pixverse-6`, `muse-image`, `pruna-p-video-2`, `hy-image-3-5-preview`) and each needs a row in `partner-model-redirects.ts`. A test fails until they have one. - It drops the partner slugs `veo-2` and `veo-3`. Keep their rows: their old addresses are published. - It drops the published file page `minimax-h3-video-vae-fp16` (81 clicks in 90 days). Add it to `retiredLocalModelPages` with its replacement page, or the published-pages test fails. - `/hub/models/local/all/` needs a `models-url-registry` entry, or `workshop-release-gate` throws at build time. - `/p/supported-models/all/` needs a literal 308 row only if Comfy-Org#19786 ships first. Prod returns 404 for it today. Fixes FE-2964, FE-2933 Linear: https://linear.app/comfyorg/issue/FE-2964 , https://linear.app/comfyorg/issue/FE-2933
Summary
Root
llms.txtnow sends agents to/hub/models/(and its.mdcopy) for hosted models and to the Comfy Router docs for calling them from code. The llms.txt link check now covers every builtllms.txtplusllms-full.txt, and fails on a link that hits a redirect (literal or:param/:path*pattern) or that the build never produced.Changes
public/llms.txt: a Comfy Router bullet under "How agents should use Comfy" (route,X-API-Key, schema URL, quickstart and models docs as.md); the "model catalog" line names both catalogues (hosted vs. runs in ComfyUI); two "Agent-readable surfaces" rows for/hub/models.mdand the Router models doc; the Models Overview row links its.mdcopy./p/supported-modelslinks stay as they are.validate:llms-txt-linkswalksdist/llms.txt,dist/llms-full.txtand every sectionllms.txt, and names the file with each failure. Three checks per link: redirect source, not in the build (Workflows app routes exempt), canonical drift.findRedirectedLinksmatches pattern sources and treats the.mdcopy of a redirected page as redirected. The test that only allowed literal sources invercel.jsonis replaced by matcher tests.Review Focus
/models/llms.txt) is not built./hub/models.mdalready lists every hosted model page, so rootllms.txtpoints at it instead of a second list.docs.comfy.org/router-schemas/...with no key.redirectSourcePatterncopies the logic ofsourcePatterninredirects.test.ts. That file is left alone to avoid a conflict with feat(website): make the old /models redirects permanent (308) #20191; they can share one helper after it merges./modelsto/hub/models/on prod) are not invercel.json, so this check can't see them.Fixes FE-2925
Fixes FE-2988
https://linear.app/comfyorg/issue/FE-2925 · https://linear.app/comfyorg/issue/FE-2988