Skip to content

test(website): time the models render guard loader from island hydration - #21047

Merged
bertfy merged 1 commit into
mainfrom
fix/models-render-guard-flake
Oct 15, 2026
Merged

bertfy merged 1 commit into
mainfrom
fix/models-render-guard-flake

Conversation

@claude

@claude claude Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Requested by Alexander Brown · Slack thread

Summary

models-render-guard.spec.ts timed the /hub/models/ loader from navigation start, so its 1 s grace also had to cover booting the site's JavaScript on the runner. That boot alone takes most of the budget on CI, so the catalogue cases fail on timing, not on the loader lingering. This PR counts the grace from the moment the page's island hydrates. That is when the component takes over the screen, and when the post-mount models-loading loader the guard was written for starts.

Before: loaderLastSeenAt was performance.now() at the last frame a loader was visible, compared with 1000 ms. The catalogue renders WorkshopLoading on the server (the island is not mounted yet), so that number covers the whole HTML, JS, hydration and catalogue.json chain. When #19394 added the guard, the worst local run was 774 ms, and its review focus already flagged the 1 s budget as the risk. On CI the catalogue cases now land at 1.07 to 1.22 s:

  • main push run 38649631281 (shard 3, job 116145534816): 393px flag on was flaky (passed on retry).
  • merge-queue run 38650370738 (shard 3, attempt 1): 393px and 1280px flag on failed all three tries, 1280px flag off was flaky.
  • merge-queue run 38652042112 (shard 3, job 116153522370): 393px flag on failed at 1139, 1224 and 1068 ms, and 1280px flag on was flaky.

None of these come from a change to the catalogue: the same tree passed on rerun. In a local trace about 660 to 940 ms of the 1.0 to 1.3 s went to booting and hydrating the page. ModelsCatalogue and /models/catalogue.json are only requested once the island mounts.

After: the init script records when each <astro-island> hydrates. Astro dispatches a non-bubbling astro:hydrate on it, so the script listens in the capture phase. waitUntilSettled reads that time for the island holding the page content and waits a full grace window after it. The assertion becomes "ms after the page island hydrated that the loader was last visible", still under 1000 ms. The CLS budget, the pages, the flag matrix and the readiness waits are unchanged. A loader the server renders into a model page is still caught by the existing JavaScript-off cases, which assert workshop-loading is hidden.

Verification

  • Local run against a build:e2e of current main with a local Chromium, launched through executablePath from a scratch config that is not committed.
    • Before, --repeat-each 5 --workers 2 on the catalogue cases: 16 of 20 failed, at 1044 to 1323 ms.
    • After, the whole spec with --repeat-each 10 --workers 3: 200 of 200 passed. The loader was last visible at most 456 ms after hydration. Under the old metric, 14 of those 40 catalogue runs would have failed, peaking at 1466 ms.
  • The guard still catches a loader that lingers. Delaying /models/catalogue.json by 1.5 s with context.route failed all four catalogue cases, with the loader last visible about 1.5 s after hydration.
  • The pre-commit hooks (oxfmt, oxlint, eslint, server-facts, website typecheck) and the pre-push hooks passed.

🤖 Generated with Claude Code

https://claude.ai/code/session_018GfgJuhjWFFm4LbKojTDxV

The catalogue cases measured the loader from navigation start, so the 1 s
grace also covered booting the site's JavaScript on the runner. On CI that
boot alone takes most of the budget, and /hub/models/ failed at 1.07 to
1.22 s on main and in the merge queue. Count the grace from the moment the
page's island hydrates instead, which is when the component owns the screen.
@claude
claude Bot requested a review from a team October 14, 2026 23:52
@claude
claude Bot temporarily deployed to website-approval October 14, 2026 23:52 Inactive
@coderabbitai

coderabbitai Bot commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Repository: Comfy-Org/ComfyUI_frontend/.coderabbit.yaml
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: b3d6daa8-7e5e-4ae0-a587-dcedd99a8cfb

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@claude
claude Bot temporarily deployed to website-approval October 14, 2026 23:53 Inactive
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

🧩 Custom Nodes Ecosystem Matrix: 🟡 Withheld — could not measure

Executes the frontend JS of every registry pack that ships any, against this change, in both renderers. The eight ecosystem-matrix shard checks are a harness self-test; the pack verdict is matrix-verdict.

This candidate changes nothing the matrix measures, so every expensive stage was skipped.

Reproduce locally
# build one shard's fixture, then run it
python3 scripts/registry-census/build_matrix.py --shard 1/4
MATRIX_RENDERER=vue MATRIX_VUE=1 \
  npx vitest run --config vitest.matrix.config.mts

Full docs: scripts/registry-census/README.md

🌾 Fallow: ✅ Passed

Fallow audits the diff for duplication, complexity and dead code. It gates on new findings only, so when the audit completes, a red is something this PR introduced or moved — never inherited backlog. A red can also mean the audit itself did not finish; the details below say which happened.

No new findings in the changed files.

Reproduce locally
# the base the gate compared against
npx fallow audit --root . --quiet --format json \
  --changed-since 1301f9d3de77b61cd33489bffba052c006594193

Full docs: ADR-DEVEX-LINT-0015

🌐 Website E2E

Tip

All tests passed.

Status ✅ Passed
Report View Report

🔗 Website Preview

Website Preview: https://comfy-website-preview-pr-21047.vercel.app

Models is built and gated by PostHog (workshop-enabled), using staging Cloud.

This commit: https://website-frontend-ir2o942um-comfyui.vercel.app

Last updated: 2026-10-14T23:55:25Z for 4a767e0

@codecov

codecov Bot commented Oct 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

@@            Coverage Diff             @@
##             main   #21047      +/-   ##
==========================================
- Coverage   86.40%   86.29%   -0.12%     
==========================================
  Files        3690     3585     -105     
  Lines      175494   172259    -3235     
  Branches    46354    45254    -1100     
==========================================
- Hits       151641   148652    -2989     
+ Misses      21801    21711      -90     
+ Partials     2052     1896     -156     
Flag Coverage Δ
billing-web-unit ?
website-unit 81.01% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 108 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions github-actions Bot added the risk:high PR risk grade (advisory shadow check; grader-owned) label Oct 15, 2026
@bertfy
bertfy added this pull request to the merge queue Oct 15, 2026
Merged via the queue into main with commit b8e79a6 Oct 15, 2026
98 checks passed
@bertfy
bertfy deleted the fix/models-render-guard-flake branch October 15, 2026 02:31
mariana953 added a commit that referenced this pull request Oct 21, 2026
Ports #21047 from main. The guard timed the /hub/models/ loader from
navigation start, so its 1s grace also covered booting the site's
JavaScript on the runner and the catalogue cases failed on timing.

This branch was previously deployed

1 inactive deployment
website-approval — 4a767e06 Deployed Oct 14, 2026 by claude[bot] via fast-lane #3416
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high PR risk grade (advisory shadow check; grader-owned)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants