Repository navigation
fix(website): let Google fetch our scripts and styles - #19357
MaanilVerma wants to merge 8 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. 2 Skipped Deployments
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository: Comfy-Org/ComfyUI_frontend/.coderabbit.yaml Review profile: ASSERTIVE Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
🌾 Fallow: ✅ PassedFallow audits the diff for duplication, complexity and dead code. It gates on new findings only, so when the audit completes, a red is something this PR introduced or moved — never inherited backlog. A red can also mean the audit itself did not finish; the details below say which happened. No new findings in the changed files. Reproduce locally# the base the gate compared against
npx fallow audit --root . --quiet --format json \
--changed-since 90bb942059e3ef05308bb05f21d085c40061ca7c
Full docs: ADR-DEVEX-LINT-0015 🧩 Custom Nodes Ecosystem Matrix: 🟡 Withheld — could not measureExecutes the frontend JS of every registry pack that ships any, against this change, in both renderers. The eight This candidate changes nothing the matrix measures, so every expensive stage was skipped. Reproduce locally# build one shard's fixture, then run it
python3 scripts/registry-census/build_matrix.py --shard 1/4
MATRIX_RENDERER=vue MATRIX_VUE=1 \
npx vitest run --config vitest.matrix.config.mts
Full docs: scripts/registry-census/README.md 📦 Bundle Size
🌐 Website E2ETip All tests passed.
🔗 Website PreviewWebsite Preview: https://comfy-website-preview-pr-19357.vercel.app Models is built and gated by PostHog (workshop-enabled), using staging Cloud. This commit: https://website-frontend-n5vq91bt3-comfyui.vercel.app Last updated: 2026-09-29T18:39:58Z for |
QA for FE-2957Before is #18891's branch (this PR's base) and its preview, after is this branch and its preview. The render is a robots-honouring Googlebot pass: Chromium at 412x915 with the Googlebot smartphone UA, every same-origin request the host's live robots.txt disallows is aborted (longest match wins), and so is every
|
| Before (#18891 preview) | After (this preview) |
|---|---|
![]() |
![]() |
/models/bfl--flux-2-max--generate-images/
| Before (#18891 preview) | After (this preview) |
|---|---|
![]() |
![]() |
Still to do after deploy (needs prod and Search Console): URL Inspection on https://comfy.org/models/, https://comfy.org/models/bfl--flux-2-max--generate-images/ and one /workflows page, Test live URL, no scripts or styles under /_astro/ or /_website/ should be "Blocked by robots.txt" (the /_vercel/ analytics script stays blocked on purpose, so Google doesn't run Vercel Analytics during crawls). Give Google about 24h to refetch robots.txt first.
This comment has been minimized.
This comment has been minimized.
Codecov Report✅ All modified and coverable lines are covered by tests. @@ Coverage Diff @@
## maanil/fe-2935-content-not-flag-gated #19357 +/- ##
=========================================================================
- Coverage 84.14% 84.06% -0.09%
=========================================================================
Files 3188 3158 -30
Lines 155118 154285 -833
Branches 41990 41853 -137
=========================================================================
- Hits 130530 129696 -834
- Misses 22916 22974 +58
+ Partials 1672 1615 -57
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
🔍 Cursor Review — Consolidated panel
Triggered by @MaanilVerma.
Found 5 finding(s).
| Severity | Count |
|---|---|
| 🟡 Medium | 1 |
| 🟢 Low | 4 |
Panel: 8/8 reviewers contributed findings.
…gated' into maanil/fe-2957-robots-unblock-assets
benjcooley
left a comment
There was a problem hiding this comment.
Review: fix(website): let Google fetch our scripts and styles
The core change is right and the reasoning holds up. Removing Disallow: /_astro/ and Disallow: /_website/ is exactly what unblocks Googlebot's render of /models/, the remaining Disallow: /_vercel/ still wins over Allow: / on longest-match, and nothing else in the repo depends on the removed lines (no e2e, no build step, no docs beyond one README mention). The stacking on #18891 is real and correctly ordered — the base branch is that PR's head, not main.
I verified the new test rather than trusting it: I re-ran its parser against both the base and the PR robots.txt outside the suite. It goes red on the base (/_website/ and /_astro/ blocked for Googlebot and Bingbot) and green here, and its matcher handles group stacking, longest-pattern-wins, allow-breaks-ties, $ anchoring and comment lines correctly. ci-website-unit.yaml picks it up via the apps/website/** path filter, so it will actually run.
No blockers. Four notes below, none of which need to hold the merge.
| Severity | Count |
|---|---|
| BLOCKER | 0 |
| SHOULD FIX | 2 |
| TRIVIAL | 2 |
SHOULD FIX
- The QA step "Page resources lists nothing 'Blocked by robots.txt'" will not pass.
BaseLayout.astroloads@vercel/analytics/astroon every page, and/_vercel/insights/script.jsis still disallowed. Harmless for rendering, but the acceptance criterion as written is unachievable — amend it or carve out the analytics path. - The test hand-rolls a robots.txt matcher to guard a two-line deletion. It is correct today, but it reimplements a spec we do not own, and its failure mode is passing green while prod is still blocked. A five-line "the only Disallow is
/_vercel/" assertion catches the same regression.
TRIVIAL
BUILD_ASSETS_DIRdoes not couple the test to the asset dir the way the description says — since the change removes rules, the assertion passes for any value of the constant.- "adds no indexable URLs" understates it:
_website/also holds Astro-processed marketing image derivatives, which Google Images can now index.
benjcooley
left a comment
There was a problem hiding this comment.
Re-review: approved
All four notes from the last pass are addressed, and the PR got smaller doing it — 4 files / 87 additions down to 2 files / 27. astro.config.ts is byte-for-byte back to its base state and src/config/build.ts is gone, so the whole change is now robots.txt plus one test file. That is the right shape for this fix.
I re-verified the new test rather than taking the reply at its word. Running rulesFor against mutated copies of the real robots.txt:
| variant | disallow test |
allow test |
|---|---|---|
| as shipped | PASS | PASS |
re-add Disallow: /_website/ |
FAIL | PASS |
add a User-agent: BadBot group with Disallow: / |
FAIL | PASS |
remove Allow: / |
PASS | FAIL |
So it goes red on exactly the regression it exists to catch, and toEqual([...]) closed the vacuous-pass hole the bot flagged earlier — an empty result no longer passes. Good trade: the 60-line matcher is gone and the protection is strictly better.
On /_vercel/ — you talked me out of it, and the reasoning is better than my suggestion was. I was optimising for a green Search Console panel; letting Googlebot fetch /_vercel/insights/script.js would mean its renderer executes Vercel Analytics and crawl renders start landing in our pageview numbers. Polluting analytics to make a diagnostic panel look tidy is a bad trade. Scoping the QA step to /_astro/ and /_website/ with the exclusion stated inline is the right call.
One TRIVIAL note below, non-blocking. Nothing holds this.
| Severity | Count |
|---|---|
| BLOCKER | 0 |
| SHOULD FIX | 0 |
| TRIVIAL | 1 |
Still gated on #18891 landing first, as the description says.
benjcooley
left a comment
There was a problem hiding this comment.
Re-review: approved
One commit since the last pass, 654f203b2a, and it is the one-line regex change I asked for — ^${directive}: becomes ^\s*${directive}\s*:. robots.txt itself is untouched. Nothing else moved.
You went further than my suggestion by also allowing whitespace before the colon, and that is the more correct call: RFC 9309's ABNF is *WS ("allow" / "disallow") *WS ":" *WS, so Disallow : /foo is a legal rule that my version would have missed. Good catch on my catch.
Re-verified empirically rather than trusting the reply. Every way I could think of to sneak the rule back now goes red, including the two that slipped through before:
| variant | result |
|---|---|
| as shipped | green |
Disallow: /_website/ |
red |
Disallow: /_website/ (space-indented) |
red (was green) |
\tDisallow: /_astro/ (tab-indented) |
red (was green) |
Disallow : /_astro/ (space before colon) |
red (was green) |
lowercase disallow: |
red |
new User-agent: BadBot group with Disallow: / |
red |
Allow: / removed |
red on the second test |
The thing I actually wanted to check was the new risk \s* introduces — whether a leading-whitespace-tolerant anchor starts swallowing commented-out rules. It does not: # Disallow: /_website/ and # Disallow: /_astro/ are both still correctly ignored, because # is not whitespace. No false positives traded for the false negatives you closed.
No findings this round.
| Severity | Count |
|---|---|
| BLOCKER | 0 |
| SHOULD FIX | 0 |
| TRIVIAL | 0 |
Still gated on #18891 landing first.
## Summary **Deploy gate: comfy-router#46 is merged and live (2026-09-30). Its follow-up comfy-router#48 (serves `/hub/models.md` instead of 301ing it to a /workflows 404) must be deployed before this merges. After the merge, purge Cloudflare for `/hub/models*`, `/hub/workflows*` and the 30 workflow names in `hub-workflow-names.json`. The redirects ship as 307; FE-3083 flips them to 308 later.** This branch combines every Workshop /models SEO PR, so merging it to main replaces merging them one by one. It also launches: Comfy-Org#19405 commits `launchedModelPages = 'all'`, so all 148 model pages become indexable in the same deploy as the URL move. Part of FE-2912, FE-2942 (PR 2 is FE-3086), FE-2974 (/minimax 308 is FE-3087) Fixes FE-2978, FE-2989, FE-2991, FE-2990, FE-2959, FE-2981, FE-2977, FE-2979, FE-3014, FE-2936, FE-2980, FE-2975, FE-3085, FE-2972, FE-2935, FE-2950, FE-2940, FE-2930, FE-2985, FE-2957, FE-2961, FE-2986, FE-2921, FE-2922, FE-2947 ## Changes Router follow-up: comfy-router#48 (`/hub/models.md` routing, 324/324 router tests), deploy before this merges. The included PRs, in merge order. Every head was checked as contained at the final tip. | PR | Ticket | What | Ben | |---|---|---|---| | Comfy-Org#18887 | FE-2978 | Pricing Product markup and full video `uploadDate` | approved | | Comfy-Org#18889 | FE-2989 | A unique meta description on every model page | approved | | Comfy-Org#18890 | FE-2991 | Model page JSON-LD, emitted once noindex lifts | approved | | Comfy-Org#18892 | FE-2990 | Cleaner tag pills, guard against duplicate summaries | approved | | Comfy-Org#18893 | FE-2959, 2981, 2977, 2979 | Social tags, hub title, Organization data. The FE-2977 title was confirmed by the owner | approved | | Comfy-Org#19362 | FE-3014 | Every non-production build is noindex | approved | | Comfy-Org#19366 | FE-2936 | Model titles lead with "API & Playground" | approved | | Comfy-Org#19402 | FE-2980 | Every internal link ends with a slash, plus a CI check | approved | | Comfy-Org#19419 | FE-2975 | Footer links the catalogue on every page and locale | approved | | Comfy-Org#19476 | FE-3085 | `NOINDEX_ROUTES` decides noindex for serverless-animation | approved; re-approval pending on a one-line follow-up (CodeRabbit approved) | | Comfy-Org#19359 | FE-2974 | One typed redirect list that generates `vercel.json` | approved | | Comfy-Org#19369 | FE-2972 A | The old to new model URL map | approved | | Comfy-Org#18891 | FE-2935 | Model content shows whatever the flag says | approved | | Comfy-Org#18899 | FE-2942 | Model pages render in the static HTML | approved | | Comfy-Org#18898 | FE-2950 | Catalogue h1 and A to Z links in the static HTML | approved | | Comfy-Org#19375 | FE-2940 | Related models leave no page unlinked | approved | | Comfy-Org#19385 | FE-2930 | Examples show the prompt that made them | approved | | Comfy-Org#19394 | FE-2985 | Render guard e2e: no loader, no layout shift | approved | | Comfy-Org#19357 | FE-2957 | robots.txt stops blocking our scripts and styles | approved | | Comfy-Org#19360 | FE-2961 | Turning Workshop off keeps every Models page | approved (a CodeRabbit changes-requested review is still open) | | Comfy-Org#19386 | FE-2986 | One registry of every Models address | approved | | Comfy-Org#19421 | FE-2972 B1 | Model pages served at `/hub/models` | approved | | Comfy-Org#19422 | FE-2972 B2 | Every old model URL redirects (307) | approved | | Comfy-Org#19424 | FE-2972 B3 | Workflow pages at `/hub/workflows` with redirects | approved | | Comfy-Org#19399 | FE-2921 | One switch decides model robots and sitemap listing | approved | | Comfy-Org#19404 | FE-2922 | Clean markdown twins for model pages | approved | | Comfy-Org#19405 | FE-2947 | Lifts noindex on every canonical model page | approved | | comfy-router#46 | FE-3075 | Serves `/hub/models*` from the website and routes `/hub/workflows` by manifest. Merged and live | merged | These fixes exist only on this branch, because they only break once two PRs meet: - `scripts/check-models-html.ts` reads each page at `model.href` and skips models without a page (f1df1e4). - `e2e/fixtures/modelsCatalogue.ts` reads `dist/hub/models` (f87bb4c). - `routerIdsByModelPage` accepts an optional `href`, since Comfy-Org#19421 made it optional and Comfy-Org#19405 no longer typechecked (9783819). - `ModelStatus` links a successor only when it has a page, for the same reason on Comfy-Org#18899 (c4761d8). - The model breadcrumb in `modelJsonLd.ts` points at `/hub/models/` instead of the `/models/` redirect (eb985b6). Its label still says "Models" while the nav says "Hub". - Twin, meta-description and entry tests use the hub pages and the new tag set (86ac3b0, d4ffc0f). - The render guard finds the catalogue through `getRoutes().workshop`, and its `test.fail` is gone now that Comfy-Org#18898 renders the h1 (831499f). - A new test renders a real model page and checks its markdown twin has the model's h1 (9a6c5cb). - Main's newer catalogue and workflow e2e cases use the hub addresses (2aeda77). - `resetWorkshopAccountSource` resets `decision`; a clean auto-merge of main's Comfy-Org#19454 into Comfy-Org#18891 left it assigning a variable that no longer exists (c28c7a7). The conflict resolutions, briefly: `ALTERNATE_TWIN_SOURCES` stays deleted as Comfy-Org#18898 wants, so `/hub/models.md` comes from the real catalogue. The hub keeps Comfy-Org#18898's single static h1 instead of the `WorkshopHero` h1 that main's Comfy-Org#19320 added. `ExamplesTab` keeps Comfy-Org#19385's figure and caption with main's video poster. The Comfy-Org#19399/Comfy-Org#19405 indexing tests were rewritten for hub paths. The Comfy-Org#19359 redirect test no longer builds `/enterprise//` from slash-ended routes. The Comfy-Org#19369 URL-map test imports `PROVIDER_NAMES` from its new home. The full table of 47 resolutions is in the integration report. ## Review Focus - Launch rides along with the URL move. If launch should stay a separate step, set `launchedModelPages` in `apps/website/src/config/model-page-launch.ts` to `new Set()` before merging. - Main's Comfy-Org#19320 gives the catalogue island its own h1 through `WorkshopHero`. This branch keeps Comfy-Org#18898's static h1 and renders only the subtitle in `ModelsCatalogue.vue`, so the hub has exactly one h1. - Comfy-Org#19476 overlaps the open community PR Comfy-Org#18591 (same sitemap exclusion for serverless-animation). One of them should be closed once this lands. ## QA - [x] Gates: `pnpm typecheck` and website typecheck clean, `pnpm lint` 0 errors, eslint on `apps/website` clean, `pnpm knip` clean - [x] Website unit: 547 files, 10347 passed, 4 skipped - [x] Build A (`build:e2e`, CI env): 1193 pages; release gate, `check:hreflang`, `validate:llms-txt-links` (225 links), `validate:jsonld` (947 pages), `check:link-slashes`, `check:models-html` (148 pages) all pass - [x] Build B (`WORKSHOP_IN_BUILD=0`): same 182 Models pages as the on build, no `dist/workshop`, `/` is noindex; with the flag stubbed on, no Run, no Hub nav and no Firebase requests - [x] Build C (preview shape): 945 of 945 pages noindex, no canonicals except redirect stubs, no hreflang in the sitemap - [x] E2E on Build A: 644 of 645 passed; the one mobile failure passed on rerun. Earlier load flakes (navigation, demos, fdct, wan-3.0, network isolation, render guard) all passed when rerun alone - [x] Launch: 148 model pages indexable with canonicals, 149 hub entries in the sitemap, 148 non-empty twins with the model h1, 148 in `llms-full.txt`, 0 titles over 60, 30 workflow pages still noindex - [x] Redirects: 853 rules, 782 to /hub all 307, the MiniMax rows 307, every /hub destination built - [x] comfy-router#46 under `wrangler dev` in front of this build: 312 router tests pass; 148 old model URLs (both slash forms) 307 then 200; 30 workflow names served by the website and their old URLs 307; a non-manifest workflow 302 with no-store; `manifest.json` passes through; `/workflows/` and bare `/hub` unchanged - [x] After the router deploys and this merges: purge Cloudflare, then curl `https://comfy.org/hub/models/` for a 200 and three old `/models/<slug>/` URLs for one 307 each - [x] Rich Results Test on `/pricing/` and `/learning/animations/character-sheet/`, then "Validate fix" on Merchant listings and Videos - [x] Rich Results Test on two live model pages: breadcrumbs valid and the breadcrumb URL returns 200 with no redirect - [x] GPTBot and Googlebot curls on the hub and one model page, then a GSC live test on both - [x] Resubmit the sitemap in Search Console and URL-inspect about 20 model pages on day 0 and day 14 - Not covered: Rich Results and GSC need production; the redirect matrix used a local shim for `vercel.json`, since `astro preview` ignores it --------- Co-authored-by: GitHub Action <action@github.com> Co-authored-by: benjcooley <benjcooley@gmail.com> Co-authored-by: Deep Mehta <42841935+deepme987@users.noreply.github.com>
|
Superseded by the integration branch #19494, which combined this PR with every other Workshop SEO slice and has been merged to main. Closing this one. |




Summary
robots.txt stops blocking
/_website/and/_astro/, so Google can fetch our scripts and styles and render the models pages styled with real content instead of an unstyled "Loading" shell./_vercel/stays disallowed.Fixes FE-2957 (https://linear.app/comfyorg/issue/FE-2957/let-google-load-our-scripts-and-styles)
Stacked on #18891; merge after it. Once Google can run our JS it can't reach PostHog (
t.comfy.orgblocks/flags/), so without #18891 the/models/hub would render the Grok showcase.Changes
public/robots.txtdropsDisallow: /_astro/andDisallow: /_website/. The user-agent list,Allow: /,Disallow: /_vercel/, the Content-Signal line and bothSitemap:lines are unchanged./_website/also holds the Astro-processed marketing image derivatives (AVIF/WebP at 640 to 1920w), so Google Images can now index those. That's expected and fine; their hashed names may cause some crawl churn on rebuilds. Nothing else feat(website): point robots.txt at /sitemap-index.xml + AI crawler rules #11823 set up is reversed.src/config/robots.test.tsfails if robots.txt has anyDisallow:other than/_vercel/, or losesAllow: /.@christian-byrne, you added these rules in #11823. @mobeenabdullah,
/_astro/is where the /workflows site's bundles live on comfy.org, so this opens those to crawlers too.QA
robots.test.tsred whenDisallow: /_website/orDisallow: /_astro/is added back, green heredist/robots.txtdiffers from the base only by the two removed lines/models/and a model page: before shows "Loading" and no h1, after shows the styled page with the real h1 and no Grok/_vercel/, and the same render against this PR's preview matches the local resulthttps://comfy.org/models/, Test live URL, no scripts or styles under/_astro/or/_website/are "Blocked by robots.txt" (the/_vercel/analytics script stays blocked on purpose, so Google doesn't run Vercel Analytics during crawls)