Skip to content

test: document QwenVL dropdown only reaching the first registered model directory - #14311

Open
claude[bot] wants to merge 4 commits into
mainfrom
test/qwenvl-dropdown-first-directory-only
Open

claude[bot] wants to merge 4 commits into
mainfrom
test/qwenvl-dropdown-first-directory-only

Conversation

@claude

@claude claude Bot commented Jul 30, 2026 •

Copy link
Copy Markdown
Contributor

Requested by Christian Byrne · Slack thread

This adds a failing test documenting that the AILab_QwenVL model dropdown on Cloud only ever lists models from the first of its 10 registered directories, making most of them (e.g. Qwen3-VL-8B-Instruct) unselectable and breaking published workflows that reference them.

This is a known, currently-triaged bug rather than something fixed here. A separate, already-merged model-tagging/asset-browser migration (PRs #12948, #12972, #13302, #13464, #13574) reworked how the model-library sidebar lists models, but it does not touch the node-dropdown query path this bug lives in, so it does not fix this bug on its own. An earlier, unmerged attempt at a direct fix exists in git history at fix/qwenvl-model-dropdown-category, and is confirmed broken, since it registers a bare parent tag that no asset actually carries, which would return an empty list instead of a partial one, due to exact (non-hierarchical) tag matching.

Root cause: getCategoryForNodeType in src/stores/modelToNodeStore.ts only keeps the first registered category per node type, and getAssetsPageForNodeType in src/platform/assets/services/assetService.ts queries only that one category, exact match, no fallback.

Path to a real fix: the long-term direction is migrating this query onto the bucket-based asset listing added by the migration above, and replacing the hand-maintained MODEL_NODE_MAPPINGS list with one generated from node definitions. Neither of those follow-up efforts has been scheduled yet. Once either lands, convert this test from it.fails to a normal assertion.

How to verify: pnpm test:unit src/platform/assets/services/assetService.test.ts


Status note — 2026-08-03

Re-verified against main on this date.

AILab_QwenVL and eight other model-id-based node families were de-registered from MODEL_NODE_MAPPINGS by #14489 (merged 2026-08-01, backported to cloud/1.48 via #14527). That fixed the reported symptom in BE-5071 — those widgets now fall back to the node's own /api/object_info combo.

The bug class documented here is not fixed. The asset browser still resolves a node type to exactly one category. As of 2026-08-03 it remains live on main for five node/input pairs:

Node Input Directories (only the first is reachable)
LS_LoadSegformerModel model_name segformer_b2_clothes, segformer_b3_clothes, segformer_b3_fashion
UpscaleModelLoader model_name upscale_models, onnx
CLIPVisionLoader clip_name clip_vision, clip
FlashVSRNode (auto) FlashVSR, FlashVSR-v1.1
UltralyticsDetectorProvider model_name ultralytics/bbox, ultralytics/segm

Tracked by FE-1177 (query model_type: rather than a single directory), FE-1181 (generate the mapping from object_info), FE-1080 (child-directory tags) and BE-5255 (backend subtype-carrying tag).

Why this is a merged test rather than a Slack thread: it.fails inverts the report — the assertion genuinely fails (proving the bug is real) but CI stays green. It turns red only when someone fixes the bug without updating the test, which is exactly the alarm we want. As an unmerged draft it provides none of that, which is why these are being taken out of draft.

Following that note, the case was retargeted from AILab_QwenVL onto LS_LoadSegformerModel (the worst remaining case at three directories), since QwenVL itself was gone from MODEL_NODE_MAPPINGS by that point.

Status note — 2026-08-25

The table above is now stale, and the retargeted case has been removed rather than fixed further.

Per review, #14529 (merged 2026-08-01, an ancestor of this PR's head) removed all three segformer_* rows along with onnx and clip — the same "id widget, not an asset filename" / "unreachable directory" reasoning #14489 applied to QwenVL. Checking each row in the 2026-08-03 table above against current MODEL_NODE_MAPPINGS: LS_LoadSegformerModel is no longer mapped at all, so the only remaining category registration lived in this test's own mock; UpscaleModelLoader and CLIPVisionLoader are each down to a single directory now that onnx and clip are gone, so neither is a multi-directory case anymore; UltralyticsDetectorProvider's two rows remain commented out/disabled (per #12075/BE-689), so neither directory is reachable at all; and FlashVSRNode still spans two directories but behind an empty input key, meaning it never exposes a widget picker for the bug to manifest in.

So none of the five original pairs still exhibit a live, widget-reachable "reads only the first directory" symptom. Searched the rest of current MODEL_NODE_MAPPINGS for any other node registered against multiple directories under one non-empty input key, and there isn't one — FlashVSRNode's empty-key rows are the only remaining multi-directory registration in the table.

Per Christian's explicit fallback (remove the case unless it can be retargeted onto a still-reachable production mapping with a decided contract), and consistent with how #14312 and #14314 resolved the identical #14489-driven staleness elsewhere in this test series, the it.fails case and its supporting mock entry have been removed. The underlying bug class (getCategoryForNodeType keeping only the first registered category per node type) is real and still present in src/stores/modelToNodeStore.ts, but there is no reachable production node/input pair left to pin it against without inventing a mapping that doesn't exist on main. If a future MODEL_NODE_MAPPINGS addition reintroduces a genuine multi-directory, non-empty-key case, this test should be restored against that node.

How to verify: pnpm test:unit src/platform/assets/services/assetService.test.ts

@github-actions

github-actions Bot commented Jul 30, 2026 •

Copy link
Copy Markdown
Contributor

🔬 E2E Coverage

⚠️ Coverage collection failed. Check the CI workflow logs.

🎭 Playwright: ✅ 1822 passed, 0 failed · 1 flaky

📊 Browser Reports
  • chromium: View Report (✅ 1801 / ❌ 0 / ⚠️ 1 / ⏭️ 5)
  • chromium-2x: View Report (✅ 2 / ❌ 0 / ⚠️ 0 / ⏭️ 0)
  • chromium-0.5x: View Report (✅ 1 / ❌ 0 / ⚠️ 0 / ⏭️ 0)
  • mobile-chrome: View Report (✅ 18 / ❌ 0 / ⚠️ 0 / ⏭️ 0)

🎨 Storybook: ✅ Built — View Storybook

Details

⏰ Completed at: 08/17/2026, 03:37:46 AM UTC

Links

🌾 Fallow: ✅ Passed

Fallow audits the diff for duplication, complexity and dead code. It gates on new findings only, so when the audit completes, a red is something this PR introduced or moved — never inherited backlog. A red can also mean the audit itself did not finish; the details below say which happened.

No new findings in the changed files.

Reproduce locally
# the base the gate compared against
npx fallow audit --root . --quiet --format json \
  --changed-since 4d28112ea0855fa80c07338d9fd85a9d58dd4e69

Full docs: ADR-DEVEX-LINT-0015

@codecov

codecov Bot commented Jul 30, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

@@            Coverage Diff             @@
##             main   #14311      +/-   ##
==========================================
+ Coverage   78.95%   81.57%   +2.62%     
==========================================
  Files        2209     1882     -327     
  Lines      112281   109185    -3096     
  Branches    32992    31287    -1705     
==========================================
+ Hits        88647    89066     +419     
+ Misses      23178    19747    -3431     
+ Partials      456      372      -84     
Flag Coverage Δ
unit 72.89% <ø> (ø)
website-unit ?

Flags with carried forward coverage won't be shown. Click here to find out more.
see 393 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@christian-byrne
christian-byrne marked this pull request as ready for review August 4, 2026 02:45
@christian-byrne
christian-byrne requested a review from a team August 4, 2026 02:45
@dosubot dosubot Bot added the size:XS This PR changes 0-9 lines, ignoring generated files. label Aug 4, 2026
@dosubot

dosubot Bot commented Aug 4, 2026

Copy link
Copy Markdown

📄 Knowledge review

Dosu skipped reviewing this PR because your organization has used its 200 included credits for the month. Your usage will reset on 2026-09-01. To have Dosu review this PR before then, ask your organization admin to upgrade to a pro account.


Leave Feedback Ask Dosu about ComfyUI_frontend Add Dosu to your team

@christian-byrne

Copy link
Copy Markdown
Contributor

What this pins: the model dropdown only ever reads the first of a node's registered directories, so models in the second and later directories are invisible in the picker. Real fix: FE-1177.

Why it.fails and not a fix. This PR changes no production code. It pins a known bug as an executable test so it stops living in a ticket description, and so CI goes red if someone fixes it by accident. The real fix is tracked separately (below).

Merge order. #14311, #14313 and #14314 all insert into assetService.test.ts, and #14311 and #14313 conflict when stacked — whichever lands second needs a trivial rebase. #14312 is independent. Suggested order: 14312 → 14311 → 14313 → 14314. (#14315 merged 2026-08-12.)

On the green checks. Worth knowing before you trust them: on 2026-08-11 all five of these were 262 commits behind main, so their passing CI proved nothing about current main. They were rebased onto ae2783911d and re-verified then — each contributes exactly one expected fail, with pnpm typecheck, eslint and oxfmt --check clean. They have since drifted ~40 commits behind again.

Note for anyone running these locally: use pnpm test:unit, not npx vitest — running vitest directly bypasses vite.config.mts and therefore the happy-dom environment, which produces ~65 spurious failures.

claude added 2 commits August 13, 2026 23:56
#14489 de-registered the QwenVL family from MODEL_NODE_MAPPINGS, which fixed
the reported symptom but left the underlying bug class untouched: the asset
browser still resolves a node type to exactly one category and queries only
that directory.

Retargets the it.fails case onto LS_LoadSegformerModel (3 registered
directories, the worst remaining case) so the test dies honestly when the
bug is actually fixed, rather than documenting an example that no longer
exists in the product.

Verified failing for the right reason: include_tags resolves to
'models,segformer_b2_clothes', so segformer_b3_clothes and
segformer_b3_fashion are unreachable.

Still live on main as of 2026-08-03 for 5 node/input pairs; see the comment
block for the full list.
@claude
claude Bot force-pushed the test/qwenvl-dropdown-first-directory-only branch from b1b9d65 to 2667936 Compare August 13, 2026 23:59
@github-actions github-actions Bot added the risk:R2 PR risk grade (advisory shadow check; grader-owned) label Aug 14, 2026

@christian-byrne christian-byrne left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: this detector fabricates a picker path removed by #14529.

Full context for agent readers

That merged fix is already an ancestor of this exact head. Production intentionally leaves all three Segformer directories unmapped because the node expects IDs and falls back to /api/object_info, while the test alone inserts LS_LoadSegformerModel -> segformer_b2_clothes. The it.fails case therefore exercises only mocked reachability, not a live product path. The inline finding identifies the required change.

Exact-head R2 review: bec2de1d5c01e565cc420f3358250aa99e600a2a.

buildAssetListResponse([validAsset({ id: 'segformer-b3-fashion' })])
)

await assetService.getAssetsPageForNodeType('LS_LoadSegformerModel')

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocker: this picker path exists only in the test mock.

Full context for agent readers

#14529 intentionally removed all LS_LoadSegformerModel mappings because the widget accepts IDs rather than asset filenames and should fall back to /api/object_info; that merged change is an ancestor of this exact head. The only category registration now exists in this test's mock at line 50, so getAssetsPageForNodeType() can exercise only a fake path and cannot detect a live dropdown regression. The five-pair table is stale too: #14529 removed onnx and clip, the Ultralytics rows remain disabled, and FlashVSR's empty input key does not expose an eligible widget picker. Please remove this case or retarget it only if a current reachable production mapping exposes the behavior.

@claude claude Bot Aug 25, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and removed — agreed on all points.

Re-verified against current main: LS_LoadSegformerModel has no segformer_* rows left in MODEL_NODE_MAPPINGS at all, so the only category registration was in this test's own mock at line 50, exactly as you said. Checked the rest of the five-pair table too: UpscaleModelLoader/CLIPVisionLoader are down to one directory each now that #14529 removed onnx and clip, UltralyticsDetectorProvider's rows are still commented out/disabled, and FlashVSRNode's two directories sit behind an empty input key so no widget picker exists for it either. Searched the current mapping table for any other node registered against multiple directories under one non-empty key to retarget onto instead — there isn't one, so there's no reachable production analog left for this bug shape right now.

Removed the it.fails case and its supporting mock entry (commit fe4f25b), consistent with how #14312 and #14314 resolved the same #14489/#14529-driven staleness elsewhere in this series. Also added a 2026-08-25 status note to the PR description that reconciles the now-stale 2026-08-03 five-pair table with this outcome, rather than leaving it dangling.

pnpm test:unit src/platform/assets/services/assetService.test.ts passes (65/65), and pnpm lint/pnpm format are clean.

Leaving this thread for you to re-review/resolve per the repo's review-comment-resolution convention.


Generated by Claude Code

@christian-byrne christian-byrne removed their assignment Aug 25, 2026
#14529 (already an ancestor of this branch) removed all three
segformer_* rows from MODEL_NODE_MAPPINGS, along with onnx and clip,
because those widgets accept ids rather than asset filenames and fall
back to /api/object_info. The only remaining LS_LoadSegformerModel
category registration lived in this test's own mock, so the it.fails
case exercised a picker path that no longer exists in production.

Checked the rest of MODEL_NODE_MAPPINGS for a still-reachable node
registered against multiple directories under one non-empty input
key: none exists. UltralyticsDetectorProvider's rows remain disabled,
and FlashVSRNode's two directories sit behind an empty key, so it
never exposes a widget picker. Removing rather than retargeting,
matching how #14312 and #14314 resolved the same staleness elsewhere
in this test series.
@huang47 huang47 added the risk:high PR risk grade (advisory shadow check; grader-owned) label Sep 2, 2026 — with ChatGPT Codex Connector
@huang47 huang47 removed the risk:R2 PR risk grade (advisory shadow check; grader-owned) label Sep 2, 2026
pull Bot pushed a commit to Penguinjanator/ComfyUI_frontend that referenced this pull request Sep 20, 2026
…nd bucket key (Comfy-Org#14313)

<!-- ccr-slack-attribution -->
_Requested by **Christian Byrne** · [Slack
thread](https://comfy-organization.slack.com/archives/C0A4XMHANP3/p1785357549411849?thread_ts=1785296212.634389&cid=C0A4XMHANP3)_

This adds a failing test documenting that the node-widget dropdown's
category identifier, a full directory path like
LLM/Qwen-VL/Qwen2.5-VL-3B-Instruct, and the model-library sidebar's
bucket key, the top-level folder only such as LLM, are incompatible, so
querying the bucket-based model list with the widget's own category
string returns nothing.

This mismatch is a known blocker to migrating the node-dropdown query
onto the newer bucket-based asset listing, since the two systems need to
agree on category identifier shape first. This test exists so the
mismatch has permanent, executable documentation rather than living only
in investigation notes.

Root cause: getCategoryForNodeType, the forward/widget path, returns
full hierarchical paths, while modelFolderFromTag, the bucket path,
collapses bare tags to only their first path segment, two different
identifier spaces for what's conceptually the same category.

Path to a real fix: normalizing all tag-as-identifier usages onto one
consistent scheme, follow-up work that hasn't been scheduled yet, would
resolve this. Once that lands, convert this test from it.fails to a real
assertion.

How to verify: pnpm test:unit
src/platform/assets/services/assetService.test.ts


---
_Generated by [Claude
Code](https://claude.ai/code/session_01P6mHd4AJXxneJFqt8y3ozA)_

---

### Status note — 2026-08-03

Re-verified against `main` on this date.

`AILab_QwenVL` and eight other model-id-based node families were
**de-registered from `MODEL_NODE_MAPPINGS` by Comfy-Org#14489** (merged
2026-08-01, backported to `cloud/1.48` via Comfy-Org#14527). That fixed the
*reported symptom* in BE-5071 — those widgets now fall back to the
node's own `/api/object_info` combo.

**The bug class documented here is not fixed.** The asset browser still
resolves a node type to exactly one category. As of 2026-08-03 it
remains live on `main` for five node/input pairs:

| Node | Input | Directories (only the first is reachable) |
|---|---|---|
| `LS_LoadSegformerModel` | `model_name` | `segformer_b2_clothes`,
`segformer_b3_clothes`, `segformer_b3_fashion` |
| `UpscaleModelLoader` | `model_name` | `upscale_models`, `onnx` |
| `CLIPVisionLoader` | `clip_name` | `clip_vision`, `clip` |
| `FlashVSRNode` | *(auto)* | `FlashVSR`, `FlashVSR-v1.1` |
| `UltralyticsDetectorProvider` | `model_name` | `ultralytics/bbox`,
`ultralytics/segm` |

Tracked by FE-1177 (query `model_type:` rather than a single directory),
FE-1181 (generate the mapping from `object_info`), FE-1080
(child-directory tags) and BE-5255 (backend subtype-carrying tag).

**Why this is a merged test rather than a Slack thread:** `it.fails`
inverts the report — the assertion genuinely fails (proving the bug is
real) but CI stays green. It turns **red only when someone fixes the bug
without updating the test**, which is exactly the alarm we want. As an
unmerged draft it provides none of that, which is why these are being
taken out of draft.

### Status note — 2026-08-25

Per
[review](Comfy-Org#14313 (review)),
`AILab_QwenVL` / `LLM/Qwen-VL/*` was an invalid target: Comfy-Org#14489 (merged
2026-08-01, confirmed an ancestor of this PR's exact head via `git
merge-base --is-ancestor`) intentionally de-registered it from
`MODEL_NODE_MAPPINGS` because that widget accepts a model or repository
ID and now falls back to `/api/object_info`, so the mock's
`LLM/Qwen-VL/*` category never reaches a live product path.

Unlike the sibling PRs in this series, this PR's bug shape (a full
hierarchical category path vs. a bucket key truncated to its first path
segment) has a still-reachable production analog, so the case was
**retargeted, not removed**. Checked current `MODEL_NODE_MAPPINGS` for a
live entry that is (a) a multi-segment directory path and (b) registered
with a non-empty input key, since an empty key means the node auto-loads
without a widget and can't expose the bug. The `chatterbox/*` and
`CogVideo/ControlNet` entries are also multi-segment but all use an
empty key, so they're not eligible either. `['LLM/checkpoints',
'LoadChatGLM3', 'chatglm3_checkpoint']` is: it's live on `main`, uses
the real `chatglm3_checkpoint` widget key, and `getAssetCategories`'s
bare-tag mode (`asset.tags.map(tag => tag.split('/')[0])`) collapses an
`LLM/checkpoints`-tagged asset to bucket key `LLM`, while
`getCategoryForNodeType('LoadChatGLM3')` still returns the full
`LLM/checkpoints` string — the exact same key-space mismatch the test
documents, on a mapping nothing has removed.

Also independently re-checked the 2026-08-03 five-pair table per the
review's flag: `Comfy-Org#14529` (also confirmed an ancestor of this head)
removed the `segformer_*`, `onnx`, and `clip` mappings entirely;
`UltralyticsDetectorProvider`'s two rows remain commented out/disabled;
and `FlashVSRNode` registers two directories under an empty input key,
so it never exposes a widget picker. None of the five pairs still
exhibit a live, widget-reachable symptom — consistent with the same
conclusion reached independently on Comfy-Org#14311, Comfy-Org#14312, and Comfy-Org#14314 today,
though that table was never this test's own case and doesn't otherwise
affect this PR's fix.

Mock and assertion in
`src/platform/assets/services/assetService.test.ts` now use
`LoadChatGLM3` / `chatglm3_checkpoint` / `LLM/checkpoints` in place of
`AILab_QwenVL` / `model_name` / `LLM/Qwen-VL/Qwen2.5-VL-3B-Instruct`.
`pnpm test:unit src/platform/assets/services/assetService.test.ts`
passes (65 passed, 1 expected fail).

How to verify: `pnpm test:unit
src/platform/assets/services/assetService.test.ts`

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Christian Byrne <cbyrne@comfy.org>
@christian-byrne
christian-byrne force-pushed the test/qwenvl-dropdown-first-directory-only branch from fe4f25b to 2cbde48 Compare October 10, 2026 23:52
@vercel

vercel Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments
Project Deployment Actions Updated
billing-web Ignored Ignored Oct 10, 2026 11:52pm UTC
billing-web-test Ignored Ignored Oct 10, 2026 11:52pm UTC

Request Review

@coderabbitai

coderabbitai Bot commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Repository: Comfy-Org/ComfyUI_frontend/.coderabbit.yaml
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 99107d64-e497-4342-b012-b7129c4d87b5

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@datadog-official

datadog-official Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Pipelines

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 6 Pipeline jobs failed

CI: Tests E2E | e2e-status

View in Datadog · View in GitHub Actions

CI: Tests E2E | lint-and-format

View in Datadog · View in GitHub Actions

CI: Tests E2E | lint-pr / lint

View in Datadog · View in GitHub Actions

View all 6 failed jobs.

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 2cbde48 | Docs | View more details | Give us feedback!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high PR risk grade (advisory shadow check; grader-owned) size:XS This PR changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants