Skip to content

test(website): pin randomized seeds in the SDK snippet upload test - #19464

Merged
benjcooley merged 1 commit into
mainfrom
benjcooley/fix-snippet-seed-test
Sep 29, 2026
Merged

benjcooley merged 1 commit into
mainfrom
benjcooley/fix-snippet-seed-test

Conversation

@benjcooley

Copy link
Copy Markdown
Collaborator

Summary

Fixes the website unit tests on main: #19391 made empty randomize seeds draw a fresh value per run, and #19323 added an SDK snippet test that compares two separately prepared graphs without pinning the seed, so all 26 workflow cases fail.

Changes

  • What: Both snippet tests now build inputs through one browserInputs helper that pins randomized seeds, as the first test already did. Test-only.

Review Focus

Passes 3/3 repeated runs; website src suite passes.

@benjcooley
benjcooley requested review from a team September 29, 2026 20:14
@vercel

vercel Bot commented Sep 29, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments
Project Deployment Actions Updated
billing-web Ignored Ignored Sep 29, 2026 8:14pm UTC
billing-web-test Ignored Ignored Sep 29, 2026 8:14pm UTC

Request Review

@coderabbitai

coderabbitai Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: Comfy-Org/ComfyUI_frontend/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 412f9e5f-134c-4cd2-bdb9-89402c79d1a1

📥 Commits

Reviewing files that changed from the base of the PR and between 25c2685 and 06ce8b6.

📒 Files selected for processing (1)
  • apps/website/src/config/workshop-workflow-snippet.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 2 remain after this review.


📝 Walkthrough

Walkthrough

The workshop workflow snippet tests now use a shared input builder. It combines URL-upload fields with values for randomized inputs in both test cases.

Changes

Workshop workflow snippet tests

Layer / File(s) Summary
Shared workflow test inputs
apps/website/src/config/workshop-workflow-snippet.test.ts
Adds a typed helper that combines URL-upload fields from initial page state with a value of 7 for randomized inputs. Both snippet tests use the helper. The upload-and-output test now includes randomized inputs.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other

Suggested reviewers: anupreet-comfy, comfyrob

Merge Risk: ⚪ Minimal · up to 06ce8

This change stabilizes the workflow snippet test inputs without changing production behavior. No material merge risk is evident from the supplied review context.

🚥 Pre-merge checks | ✅ 7
✅ Passed checks (7 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the test change: pinning randomized seeds in the website SDK snippet upload test.
Description check ✅ Passed The description includes the required Summary, Changes, and Review Focus sections. It explains the failure cause, the shared browserInputs fix, and test validation results.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
End-To-End Regression Coverage For Fixes ✅ Passed The PR title and commit subject do not use bug-fix language. The only changed file is under apps/website, which the check explicitly excludes from this requirement. The check therefore passes.
Website End-To-End Regression Coverage ✅ Passed The PR changes only apps/website/src/config/workshop-workflow-snippet.test.ts, a website test file. The described diff updates test input construction and does not change observable website runtime …
Adr Compliance For Entity/Litegraph Changes ✅ Passed The changed file is apps/website/src/config/workshop-workflow-snippet.test.ts. It is not under src/lib/litegraph/, src/ecs/, or a graph-entity-related path. The custom check does not apply.
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Comment @coderabbitai help to get the list of available commands.

@benjcooley benjcooley left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👨‍💻 BILL: review of 06ce8b601f, ready

This is a test-only fix for main's red snippet suite.

  • The cause: #19323's "sets the inputs the browser uploads and saves every output" test built its inputs without pinning randomize inputs. After #19391, prepareWorkflowRender draws a fresh seed, so all 26 cases failed. On main 25c268595d I reproduced 26 failed / 66 passed.
  • The fix: both tests share a browserInputs(model) helper that pins every randomize input to 7, the same way the first test already did. No assertions changed.
  • Verified: workshop-workflow-snippet.test.ts passes 92/92 on 3 consecutive runs, and the whole src/config suite passes 3,417/3,417.
Severity Count
BLOCKER 0
SHOULD FIX 0
TRIVIAL 0

@benjcooley

Copy link
Copy Markdown
Collaborator Author

Folded into #19456 (commit 17d9d0a) so the fix and that PR share one CI run. Closing.

@benjcooley benjcooley closed this Sep 29, 2026
@github-actions

github-actions Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

🌾 Fallow: ✅ Passed

Fallow audits the diff for duplication, complexity and dead code. It gates on new findings only, so when the audit completes, a red is something this PR introduced or moved — never inherited backlog. A red can also mean the audit itself did not finish; the details below say which happened.

No new findings in the changed files.

Reproduce locally
# the base the gate compared against
npx fallow audit --root . --quiet --format json \
  --changed-since 25c268595d612e2ee0aa1749191eee6a022c601f

Full docs: ADR-DEVEX-LINT-0015

🌐 Website E2E

Tip

All tests passed.

Status ✅ Passed
Report View Report

🧩 Custom Nodes Ecosystem Matrix: 🟡 Withheld — could not measure

Executes the frontend JS of every registry pack that ships any, against this change, in both renderers. The eight ecosystem-matrix shard checks are a harness self-test; the pack verdict is matrix-verdict.

This candidate changes nothing the matrix measures, so every expensive stage was skipped.

Reproduce locally
# build one shard's fixture, then run it
python3 scripts/registry-census/build_matrix.py --shard 1/4
MATRIX_RENDERER=vue MATRIX_VUE=1 \
  npx vitest run --config vitest.matrix.config.mts

Full docs: scripts/registry-census/README.md

@benjcooley benjcooley reopened this Sep 29, 2026
@benjcooley

Copy link
Copy Markdown
Collaborator Author

Reopened: landing this fix on its own so main is green without waiting for #19456. #19456 still carries the identical commit, which merges cleanly after this is squashed in.

@benjcooley
benjcooley merged commit f9c6efd into main Sep 29, 2026
104 of 114 checks passed
@benjcooley
benjcooley deleted the benjcooley/fix-snippet-seed-test branch September 29, 2026 20:40
@github-actions github-actions Bot added the risk:high PR risk grade (advisory shadow check; grader-owned) label Sep 29, 2026
benjcooley added a commit that referenced this pull request Sep 29, 2026
## Summary

Workshop apps (Cinematic Studio and Re-shoot, at `/models/apps/<slug>`)
sent no Workshop events, so their use and failures never reached PostHog
or the Datadog run logs. Follows #19451, which added `page_type: 'app'`,
`app_slug` and the Datadog `feature: apps` split.

## Changes

- **What**:
- **App views:** `model_viewed` with `page_type: 'app'` and `app_slug`
fires once per app shown, only when Workshop and the apps flag let the
page render. The app switcher counts each app once.
- **Studio takes:** each Cinematic Studio take (one Router request)
records `run_started` and `run_finished`:
    - succeeded, with `request_id` and `output_count`;
- failed, with `workshopFailureAnalytics`: the reason, HTTP status,
Router error type and request ID;
    - or cancelled.

A take whose model page fails to load records a failed run
(`unavailable`, `input_preparation`); a take cancelled while its model
loads records nothing.

These carry the same model fields as model-page runs (`model_slug`,
`router_id`, `provider`, `render_engine: router`) plus `page_type:
'app'` and `app_slug`, so they also land in the Datadog run logs under
`feature: apps`.
  - **Studio downloads:** `output_download_clicked` as an app event.
- Also carries the SDK snippet seed-test fix, identical to #19464
(already on `main`), so it adds nothing when squashed.
- `workshopModelAnalytics` now files `APP` models as `page_type: 'app'`
with `app_slug`, instead of `workflow` (Bill's trivial on #19451).

## Review Focus

- **Scope:** Re-shoot is still a demo with no paid runs, so only its
view is recorded.
- **Studio slug:** `CINEMATIC_STUDIO_APP_SLUG` is a constant. A test
asserts it matches the app content config, which keeps that config out
of the studio's client bundle.
- **Tests:** new table and page tests cover succeeded and failed takes,
downloads, app views and the helper. All fail on the old source. The
download test cancels the link's navigation so happy-dom doesn't leave
the page. The website suite passes (9777).
@codecov

codecov Bot commented Sep 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

@@            Coverage Diff             @@
##             main   #19464      +/-   ##
==========================================
- Coverage   85.10%   84.14%   -0.97%     
==========================================
  Files        2406     3201     +795     
  Lines      135477   155751   +20274     
  Branches    36591    42188    +5597     
==========================================
+ Hits       115296   131052   +15756     
- Misses      19713    22984    +3271     
- Partials      468     1715    +1247     
Flag Coverage Δ
billing-web-unit ?
website-unit 79.00% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 935 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high PR risk grade (advisory shadow check; grader-owned)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant