Repository navigation
test: snapshot the CLI command inventory - #516
mattmillerai wants to merge 5 commits into
Conversation
|
No actionable comments were generated in the recent review. Walkthrough
Merge Risk:
|
There was a problem hiding this comment.
🔍 Cursor Review — Consolidated panel
Triggered by @mattmillerai.
Judge call failed (status=parse_error): Could not parse JSON findings from output. First 500 chars:
I have enough to adjudicate. The shell/web tools are blocked in this environment, so I verified everything by reading the actual file and reasoning from Python internals.
Key adjudication decisions:
- **Dropped the gemini "high" AttributeError claim (line 172).** The claim that `ast.Constant` nodes inside f-strings lack `.lineno` in Python 3.10/3.11 is false — parsed f-string literal parts carry valid (if historically imprecise) position info; `.lineno` never raises `AttributeError` on parsed
Re-trigger by removing and re-adding the cursor-review label.
skishore23
left a comment
There was a problem hiding this comment.
The command-inventory snapshot here is genuinely valuable and I'd like to see it land — main has no equivalent. But the lint half is now a duplicate of a stronger guard that already shipped, and I think it should come out before merge. Details below; it's a small, surgical cut.
The overlap
This PR was opened 2026-07-14. Since then #616 ("fix: point CLI copy at commands that exist + lint it", BE-2996) landed on main on 2026-07-28, adding comfy_cli/command_mentions.py + tests/comfy_cli/test_command_mentions.py — the same hint-string lint, same ticket family. After merging origin/main into this head, the branch carries both.
That alone would just be redundancy. The problem is which one is weaker.
The lint in this PR has a blind spot main's has already closed
_leading_command_tokens stops at the first token that isn't [a-z][a-z0-9-]*, so a leading global option swallows the whole reference. main's resolver deliberately skips root-callback options and keeps scanning. Ran both against the same strings on the merged tree:
| reference | this PR flags | main flags |
|---|---|---|
comfy --json auth login |
False | True |
comfy auth login |
True | True |
comfy --where cloud auth whoami |
False | True |
comfy --json cloud singin |
False | True |
main has an explicit regression test for exactly this (test_invalid_mentions_are_flagged[comfy --json auth login-login]) with a comment noting that otherwise "prefixing the bad command with --json would hide it from the lint entirely."
Two further gaps: this lint scans comfy_cli/**/*.py + comfy_cli/**/*.md but not README.md (main scans it, and has test_scan_reaches_the_readmes_fenced_examples), and _WHOLE_LINE's [a-z0-9 _-]*$ can't match a bare line carrying a path, so comfy run --workflow ./workflow.json isn't extracted.
To be clear: merging this does not weaken anything — main's lint still catches all of the above, so combined coverage is unchanged. The cost is maintenance, and it's real: ~200 lines of parallel extraction logic, and two separate allowlists (ALLOWED_NONINVOKABLE_REFS / BASELINE_INVALID_REFS here vs PROSE_SCAN_EXCLUDED there). A contributor who adds a command and gets flagged has to know both exist and update the right one. The weaker implementation is also the one with the more inviting name (test_cli_surface.py), so it's the one people will reach for.
What I'd suggest
Keep test_command_inventory_snapshot + _command_inventory() + the fixture; drop test_no_invalid_command_references, test_baseline_is_not_stale, test_lint_catches_auth_login_and_friends, and the extraction helpers (_refs_in_text, _leading_command_tokens, _ref_is_valid, _command_tree, _iter_source_strings, _scan_invalid_references, both allowlists, the three regexes). The snapshot test only needs build_help_json, so it's cleanly separable — no dependency on any of the lint machinery.
If any of the three lint tests cover a case main's doesn't, the better move is to port that case into tests/comfy_cli/test_command_mentions.py rather than keep a second scanner alive.
What I verified (all green)
- Merged
origin/maininto the head — clean merge, no conflicts. test_cli_surface.py+test_command_mentions.py— 35 passed together.- The snapshot is not stale: regenerated it with
UPDATE_CLI_SNAPSHOT=1against currentmainand diffed — zero drift, 152 entries. Nice job reconciling that in21a890f. BASELINE_INVALID_REFSis now empty, so thecomfy queryrot mentioned in the description is already resolved — the description's "baselined for a separate fix" paragraph is stale and should come out of the squash message.
Happy to flip to approve on a push that drops the duplicate lint — or, if you'd rather keep both and reconcile later, say so and I'll approve as-is; the coverage argument above is about maintenance cost, not correctness.
|
Reviewed at high effort. Lint/format/tests all pass in isolation, and the snapshot half is genuinely useful — but the lint half duplicates infrastructure that already landed on main, and the fork is measurably weaker. Findings verified by executing the extractor against concrete inputs. The core issue: main already ships this lint. #616 (BE-2996, commit 5424453, merged into this branch's base) added
Recommendation: delete the lint half ( Issues in the parts worth keeping:
The snapshot inventory itself checked out: registration is unconditional (deterministic across platforms), the fixture byte-matches 🤖 Generated with Claude Code |
|
@skishore23 thanks — took the surgical cut you suggested in 9ce8d81:
Locally: |
skishore23
left a comment
There was a problem hiding this comment.
Thanks for dropping the duplicate hint-lint. My earlier point is addressed.
Blocking: the snapshot is stale against current main, so merging turns main's pytest red
#991 (comfy deploy cancel) landed on main on 2026-10-08, after this branch's last merge of main. With this branch merged onto current origin/main:
test_command_inventory_snapshot → CLI command inventory changed vs. the committed snapshot. added: + comfy deploy cancel
GitHub shows the PR as MERGEABLE and CI passed, but CI ran against the older base.
To fix: rebase onto or merge origin/main, run UPDATE_CLI_SNAPSHOT=1 pytest tests/comfy_cli/test_cli_surface.py, and commit the one-line fixture change. Because this test breaks whenever a command is added, please re-run it right before merging.
|
@skishore23 Thanks, confirmed: on this branch merged with |
Add two anti-rot CLI-surface guards, mirroring the MCP server's EXPECTED_TOOLS + description-budget snapshot pattern: - Hint-string lint: extract every backtick/shell-hint `comfy …` invocation from help text, docstrings, hint/error strings, examples, and bundled skill docs, then validate each against the registered command set by walking the Typer app tree. Fails naming the offending string + file. A regression fixture pins that it flags the historical `comfy auth login` rot (and the sibling `auth whoami`). - Command-inventory snapshot: snapshot the full command tree so surface additions/removals/hides are explicit diffs in review. Regenerate with UPDATE_CLI_SNAPSHOT=1 pytest tests/comfy_cli/test_cli_surface.py. The lint immediately surfaced live rot: five hint strings pointed at the nonexistent `comfy auth whoami` (deliberately removed — see test_legacy_auth_whoami_is_gone). Route them to the real `comfy cloud whoami`. The remaining `comfy query` rot (CQL surface documented but no command registered) is baselined for a separate fix.
Merging in the base branch pulled in a wave of new commands from main (model downloads, node deps, run-template, system-stats, workflow notes, etc.) that weren't in this PR's committed inventory snapshot. Separately, the base's run-cli rewrite already replaced the invalid `comfy query` reference with `comfy nodes ls`, so that BASELINE_INVALID_REFS entry is now stale per test_baseline_is_not_stale's own contract.
Main now ships the stronger shared mention lint (comfy_cli/command_mentions.py + tests/comfy_cli/test_command_mentions.py), which skips global options, scans README.md, and derives envelope-only names structurally. Remove this PR's parallel extractor/allowlists and keep only the command-inventory snapshot. - Port the `comfy auth whoami` regression case into test_command_mentions.py. - UPDATE_CLI_SNAPSHOT now only regenerates for 1/true/yes, so =0/false runs the assertion instead of silently rewriting the fixture. - Regenerate the snapshot against current main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merged origin/main, which added `comfy deploy cancel` (#991); regenerated the fixture with UPDATE_CLI_SNAPSHOT=1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds commands landed since the last snapshot: assets library upload, build quota, code-search-context, knowledge authoring, model upload, workflow check, and workflow disconnect.
7b0f005 to
648190d
Compare
Swarmhost agentic reviewThe detailed evaluation is available to employees in the internal Slack review thread. Evaluation budget remaining for this pull request: 14 automatic and 15 manual. Updated by Swarmhost's agentic review process. |
ELI5
Adding, removing, or hiding a CLI command used to be easy to miss in review. This adds a saved list of every
comfycommand, and a test that fails when the real command list stops matching it, so any change to the command surface shows up as an obvious diff in the PR.What & why
Command-inventory snapshot (
tests/comfy_cli/test_cli_surface.py) walks the registeredTyper tree (
build_help_json) and records every command path, groups and leaves, with a[hidden]marker, intests/comfy_cli/fixtures/cli_command_inventory.txt. If the livesurface stops matching, the test fails and lists the added and removed paths.
Regenerate after an intentional surface change:
Only
1/true/yesregenerate the snapshot.UPDATE_CLI_SNAPSHOT=0orfalseruns thenormal comparison and doesn't rewrite the fixture.
Also adds the
comfy auth whoamiregression case totest_invalid_mentions_are_flaggedintests/comfy_cli/test_command_mentions.py.Review history
This PR originally also had its own hint-string lint. #616 has since shipped a stronger
shared lint on
main(comfy_cli/command_mentions.py). It skips leading global options,scans
README.md, and works out envelope-only names from the code instead of a hardcodedlist. Per review, this PR's duplicate lint (regexes, allowlists,
BASELINE_INVALID_REFS)was removed. The one case
maindidn't cover yet (comfy auth whoami) was moved intotest_command_mentions.py. Theauth whoami→cloud whoamihint fixes from earlier onthis branch have already reached
mainthrough the base branch.Provenance
origin/main(545d576); snapshot regenerated against it, picking up commands added since the previous round (assets library upload,build quota,workflow check, …).pytest tests/comfy_cli/test_cli_surface.py tests/comfy_cli/test_command_mentions.py: 34 passed on the current head.ruff checkandruff format --checkpass.main'scommand_mentionslint (see Review history).Summary by CodeRabbit
comfy auth whoamicommand mention.