Skip to content

test: snapshot the CLI command inventory - #516

Open
mattmillerai wants to merge 5 commits into
mainfrom
matt/be-2979-hint-lint-discover-snapshot
Open

mattmillerai wants to merge 5 commits into
mainfrom
matt/be-2979-hint-lint-discover-snapshot

Conversation

@mattmillerai

@mattmillerai mattmillerai commented Jul 14, 2026 •

Copy link
Copy Markdown
Member

ELI5

Adding, removing, or hiding a CLI command used to be easy to miss in review. This adds a saved list of every comfy command, and a test that fails when the real command list stops matching it, so any change to the command surface shows up as an obvious diff in the PR.

What & why

Command-inventory snapshot (tests/comfy_cli/test_cli_surface.py) walks the registered
Typer tree (build_help_json) and records every command path, groups and leaves, with a
[hidden] marker, in tests/comfy_cli/fixtures/cli_command_inventory.txt. If the live
surface stops matching, the test fails and lists the added and removed paths.

Regenerate after an intentional surface change:

UPDATE_CLI_SNAPSHOT=1 pytest tests/comfy_cli/test_cli_surface.py

Only 1/true/yes regenerate the snapshot. UPDATE_CLI_SNAPSHOT=0 or false runs the
normal comparison and doesn't rewrite the fixture.

Also adds the comfy auth whoami regression case to test_invalid_mentions_are_flagged in
tests/comfy_cli/test_command_mentions.py.

Review history

This PR originally also had its own hint-string lint. #616 has since shipped a stronger
shared lint on main (comfy_cli/command_mentions.py). It skips leading global options,
scans README.md, and works out envelope-only names from the code instead of a hardcoded
list. Per review, this PR's duplicate lint (regexes, allowlists, BASELINE_INVALID_REFS)
was removed. The one case main didn't cover yet (comfy auth whoami) was moved into
test_command_mentions.py. The auth whoami → cloud whoami hint fixes from earlier on
this branch have already reached main through the base branch.

Provenance

  • Authored by: agent-work loop
  • Verified:
    • Branch head contains current origin/main (545d576); snapshot regenerated against it, picking up commands added since the previous round (assets library upload, build quota, workflow check, …).
    • pytest tests/comfy_cli/test_cli_surface.py tests/comfy_cli/test_command_mentions.py: 34 passed on the current head.
    • ruff check and ruff format --check pass.
    • PR CI on the current head is green (build/full test suite, ruff, CodeQL).
  • Deviations:
    • The hint-string lint from the original scope was dropped in favour of main's command_mentions lint (see Review history).

Summary by CodeRabbit

  • Tests
    • Added a snapshot check to detect changes in the available CLI command paths, including hidden commands.
    • Expanded validation to catch an invalid comfy auth whoami command mention.

@coderabbitai

coderabbitai Bot commented Jul 14, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

Adds a snapshot test for comfy command paths and a test case that checks whether whoami is flagged as an unknown command. The snapshot includes hidden-command annotations.

Changes: CLI test coverage

Layer / File(s) Summary
Command inventory snapshot
tests/comfy_cli/test_cli_surface.py, tests/comfy_cli/fixtures/cli_command_inventory.txt
Builds a sorted command inventory from CLI help data and marks hidden commands. Compares the inventory with the committed fixture. UPDATE_CLI_SNAPSHOT enables fixture regeneration.
Invalid command mention check
tests/comfy_cli/test_command_mentions.py
Adds a case that expects whoami to be flagged as unknown in comfy auth whoami.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other

Merge Risk: Minimal · up to 64819

This PR adds a CLI command inventory snapshot test and one invalid-mention test case. It does not change production behavior, so merge risk is minimal.

Pre-merge checks: Passed 5

Check name Status Explanation
Title check Passed The title "test: snapshot the CLI command inventory" matches the main change. The PR adds a snapshot test and fixture that record the registered comfy command paths, including hidden markers, and fail…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Architecture Decision Compliance Passed The check is applicable because docs/adr/ holds 14 ADRs (excluding TEMPLATE.md). The PR changes only three test-side files: tests/comfy_cli/fixtures/cli_command_inventory.txt (added), `tests/comfy…
Full details: Title check

Explanation

The title "test: snapshot the CLI command inventory" matches the main change. The PR adds a snapshot test and fixture that record the registered comfy command paths, including hidden markers, and fail on differences. The added whoami mention case is a minor supporting change that the title does not need to cover. The title is short, specific, and in a conventional commit style.

Full details: Architecture Decision Compliance

Explanation

The check is applicable because docs/adr/ holds 14 ADRs (excluding TEMPLATE.md). The PR changes only three test-side files: tests/comfy_cli/fixtures/cli_command_inventory.txt (added), tests/comfy_cli/test_cli_surface.py (added), and tests/comfy_cli/test_command_mentions.py (one added regression case for comfy auth whoami). - No Accepted ADR contradicted. The diff does not touch any ADR's domain. The closest ADR, AUTOMATION-0003-one-versioned-machine-output-dialect.md (Status: Accepted), covers the JSON/NDJSON envelope and error-code contract. This PR does not change comfy_cli/output/renderer.py, the schemas, or the error registry. The other ADRs cover auth, build validation, conversion, coverage, dependencies, desktop discovery, knowledge routes, operations, sidebar, and transfer. None is touched. - No durable boundary, invariant, ownership rule, compatibility contract, or dependency choice created. The new test walks the Typer tree through the existing build_help_json and compares it with a committed snapshot. It is a regression guard. Its regeneration env var and fixture are reversible test mechanics, which the check excludes. The comfy auth whoami case is one more entry in an existing parametrized test. - No new or amended ADR is required. The diff creates no architectural decision that would need one. The PR description's reference to the shared command_mentions lint on main is context only. It does not add a new architectural rule. No Proposed ADR bears on this change, so no directional-guidance note applies.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 7b0f005 and 648190d.

Files selected for processing: 1

  • tests/comfy_cli/fixtures/cli_command_inventory.txt

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 0 remain after this review.

@mattmillerai
mattmillerai marked this pull request as ready for review July 14, 2026 20:07
@mattmillerai mattmillerai added cursor-review Request Cursor bot review agent-coded PR authored by the agent-work loop labels Jul 14, 2026
@dosubot dosubot Bot added the size:S This PR changes 10-29 lines, ignoring generated files. label Jul 14, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Cursor Review — Consolidated panel

Triggered by @mattmillerai.

⚠️ Review failed

Judge call failed (status=parse_error): Could not parse JSON findings from output. First 500 chars:
I have enough to adjudicate. The shell/web tools are blocked in this environment, so I verified everything by reading the actual file and reasoning from Python internals.

Key adjudication decisions:

- **Dropped the gemini "high" AttributeError claim (line 172).** The claim that `ast.Constant` nodes inside f-strings lack `.lineno` in Python 3.10/3.11 is false — parsed f-string literal parts carry valid (if historically imprecise) position info; `.lineno` never raises `AttributeError` on parsed 

Re-trigger by removing and re-adding the cursor-review label.

@dosubot dosubot Bot added size:XS This PR changes 0-9 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Jul 30, 2026
Base automatically changed from matt/be-2975-fix-auth-login-refs to main August 4, 2026 20:26

@skishore23 skishore23 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The command-inventory snapshot here is genuinely valuable and I'd like to see it land — main has no equivalent. But the lint half is now a duplicate of a stronger guard that already shipped, and I think it should come out before merge. Details below; it's a small, surgical cut.

The overlap

This PR was opened 2026-07-14. Since then #616 ("fix: point CLI copy at commands that exist + lint it", BE-2996) landed on main on 2026-07-28, adding comfy_cli/command_mentions.py + tests/comfy_cli/test_command_mentions.py — the same hint-string lint, same ticket family. After merging origin/main into this head, the branch carries both.

That alone would just be redundancy. The problem is which one is weaker.

The lint in this PR has a blind spot main's has already closed

_leading_command_tokens stops at the first token that isn't [a-z][a-z0-9-]*, so a leading global option swallows the whole reference. main's resolver deliberately skips root-callback options and keeps scanning. Ran both against the same strings on the merged tree:

reference this PR flags main flags
comfy --json auth login False True
comfy auth login True True
comfy --where cloud auth whoami False True
comfy --json cloud singin False True

main has an explicit regression test for exactly this (test_invalid_mentions_are_flagged[comfy --json auth login-login]) with a comment noting that otherwise "prefixing the bad command with --json would hide it from the lint entirely."

Two further gaps: this lint scans comfy_cli/**/*.py + comfy_cli/**/*.md but not README.md (main scans it, and has test_scan_reaches_the_readmes_fenced_examples), and _WHOLE_LINE's [a-z0-9 _-]*$ can't match a bare line carrying a path, so comfy run --workflow ./workflow.json isn't extracted.

To be clear: merging this does not weaken anything — main's lint still catches all of the above, so combined coverage is unchanged. The cost is maintenance, and it's real: ~200 lines of parallel extraction logic, and two separate allowlists (ALLOWED_NONINVOKABLE_REFS / BASELINE_INVALID_REFS here vs PROSE_SCAN_EXCLUDED there). A contributor who adds a command and gets flagged has to know both exist and update the right one. The weaker implementation is also the one with the more inviting name (test_cli_surface.py), so it's the one people will reach for.

What I'd suggest

Keep test_command_inventory_snapshot + _command_inventory() + the fixture; drop test_no_invalid_command_references, test_baseline_is_not_stale, test_lint_catches_auth_login_and_friends, and the extraction helpers (_refs_in_text, _leading_command_tokens, _ref_is_valid, _command_tree, _iter_source_strings, _scan_invalid_references, both allowlists, the three regexes). The snapshot test only needs build_help_json, so it's cleanly separable — no dependency on any of the lint machinery.

If any of the three lint tests cover a case main's doesn't, the better move is to port that case into tests/comfy_cli/test_command_mentions.py rather than keep a second scanner alive.

What I verified (all green)

  • Merged origin/main into the head — clean merge, no conflicts.
  • test_cli_surface.py + test_command_mentions.py — 35 passed together.
  • The snapshot is not stale: regenerated it with UPDATE_CLI_SNAPSHOT=1 against current main and diffed — zero drift, 152 entries. Nice job reconciling that in 21a890f.
  • BASELINE_INVALID_REFS is now empty, so the comfy query rot mentioned in the description is already resolved — the description's "baselined for a separate fix" paragraph is stale and should come out of the squash message.

Happy to flip to approve on a push that drops the duplicate lint — or, if you'd rather keep both and reconcile later, say so and I'll approve as-is; the coverage argument above is about maintenance cost, not correctness.

@skishore23

Copy link
Copy Markdown
Contributor

Reviewed at high effort. Lint/format/tests all pass in isolation, and the snapshot half is genuinely useful — but the lint half duplicates infrastructure that already landed on main, and the fork is measurably weaker. Findings verified by executing the extractor against concrete inputs.

The core issue: main already ships this lint. #616 (BE-2996, commit 5424453, merged into this branch's base) added comfy_cli/command_mentions.py — a shared mention extractor/resolver — plus tests/comfy_cli/test_command_mentions.py running it repo-wide. test_cli_surface.py re-implements it with private regexes, so the same guard now runs twice in CI with different grammars, and they've already diverged:

  • False negatives (verified): `comfy --json auth whoami` passes the new lint — _leading_command_tokens stops at the first - token, returns [], validates as bare comfy. command_mentions.check_mention with global_options() catches exactly this class. Interpolated backticks (f"re-run `comfy auth login --host {host}`") are invisible too (no closing backtick in the AST constant). README.md — the most-copied surface — isn't scanned at all (_iter_source_strings walks only the package dir).
  • False positives (verified): _SHELL_HINT with IGNORECASE flags prose like "run Comfy in the cloud" → (['in','the','cloud'], invalid); _WHOLE_LINE flags a wrapped markdown line starting "comfy jobs are tracked with ids". Either would fail CI on legitimate documentation.
  • Second/quoted commands in multi-command hints are missed (e.g. the real string at fragments.py:566 — only the first run: segment is extracted).

Recommendation: delete the lint half (test_no_invalid_command_references, the regexes, _ref_is_valid, BASELINE_INVALID_REFS) and fold any extra coverage (scanning discovery.py, the comfy version envelope allowlist) into test_command_mentions.py / command_mentions.py, which already derives envelope-only names structurally.

Issues in the parts worth keeping:

  • tests/comfy_cli/test_cli_surface.py:266 — UPDATE_CLI_SNAPSHOT is checked for truthiness, so UPDATE_CLI_SNAPSHOT=0 (or false) silently regenerates the snapshot and skips the assertion — the gate never compares anything and dirties the tree.
  • tests/comfy_cli/test_cli_surface.py:54 — BASELINE_INVALID_REFS is an empty frozenset, so test_baseline_is_not_stale is vacuously green (frozenset() - anything == frozenset()) while costing a full-package AST sweep (~0.6s); and the baseline suppresses by ref string globally, not per-site, so a baselined ref would also excuse brand-new occurrences of the same rot.
  • The scan runs twice per pytest session (~0.5s each, growing with the package) — a cached _scan_invalid_references or session fixture halves it.
  • tests/comfy_cli/command/test_run_cli.py:48 — the whoami guard pins one tour step with substring asserts; a loop validating every _build_steps invocation's argv against the command tree would guard the whole tour at the same cost (any other step can regress to a renamed subcommand with all tests green).

The snapshot inventory itself checked out: registration is unconditional (deterministic across platforms), the fixture byte-matches _command_inventory(), and the help_json shape assumptions (subcommands omitted on leaves) hold.

🤖 Generated with Claude Code

@mattmillerai

Copy link
Copy Markdown
Member Author

@skishore23 thanks — took the surgical cut you suggested in 9ce8d81:

  • Duplicate lint removed. test_cli_surface.py now holds only _command_inventory() + test_command_inventory_snapshot (no regexes, allowlists, BASELINE_INVALID_REFS, or AST sweep). Hint-string linting is left to command_mentions.py / test_command_mentions.py.
  • Ported the one case main lacked: ("comfy auth whoami", "whoami") added to test_invalid_mentions_are_flagged. Everything else in the old fixture (auth login, invalid leaf, --json prefix, comfy version envelope, discovery.py) was already covered on main.
  • UPDATE_CLI_SNAPSHOT now regenerates only for 1/true/yes; =0/false runs the assertion.
  • Merged current main (clean) and regenerated the snapshot — it picked up the commands added since (agent, build, assets library, …).
  • The test_run_cli.py whoami-guard note: that change already landed on main via the base PR, so it's no longer in this diff.

Locally: ruff check / ruff format --check clean; test_cli_surface.py + test_command_mentions.py 34 passed. PR CI is pending on the new head.

@mattmillerai mattmillerai changed the title test: lint comfy hint-strings + snapshot the command inventory test: snapshot the CLI command inventory Oct 8, 2026

@skishore23 skishore23 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for dropping the duplicate hint-lint. My earlier point is addressed.

Blocking: the snapshot is stale against current main, so merging turns main's pytest red

#991 (comfy deploy cancel) landed on main on 2026-10-08, after this branch's last merge of main. With this branch merged onto current origin/main:

test_command_inventory_snapshot → CLI command inventory changed vs. the committed snapshot. added: + comfy deploy cancel

GitHub shows the PR as MERGEABLE and CI passed, but CI ran against the older base.

To fix: rebase onto or merge origin/main, run UPDATE_CLI_SNAPSHOT=1 pytest tests/comfy_cli/test_cli_surface.py, and commit the one-line fixture change. Because this test breaks whenever a command is added, please re-run it right before merging.

@mattmillerai

Copy link
Copy Markdown
Member Author

@skishore23 Thanks, confirmed: on this branch merged with origin/main, test_command_inventory_snapshot failed with + comfy deploy cancel. I merged current origin/main (5daa4ba), regenerated with UPDATE_CLI_SNAPSHOT=1, and committed the one-line fixture change in 7b0f005. The snapshot test passes again. Agreed that it should be re-run right before merging, because any newly added command will trip it.

@mattmillerai
mattmillerai requested review from skishore23 and removed request for annehe9 October 14, 2026 02:52
mattmillerai and others added 5 commits November 16, 2026 15:38
Add two anti-rot CLI-surface guards, mirroring the MCP server's
EXPECTED_TOOLS + description-budget snapshot pattern:

- Hint-string lint: extract every backtick/shell-hint `comfy …`
  invocation from help text, docstrings, hint/error strings, examples,
  and bundled skill docs, then validate each against the registered
  command set by walking the Typer app tree. Fails naming the offending
  string + file. A regression fixture pins that it flags the historical
  `comfy auth login` rot (and the sibling `auth whoami`).
- Command-inventory snapshot: snapshot the full command tree so surface
  additions/removals/hides are explicit diffs in review. Regenerate with
  UPDATE_CLI_SNAPSHOT=1 pytest tests/comfy_cli/test_cli_surface.py.

The lint immediately surfaced live rot: five hint strings pointed at the
nonexistent `comfy auth whoami` (deliberately removed — see
test_legacy_auth_whoami_is_gone). Route them to the real `comfy cloud
whoami`. The remaining `comfy query` rot (CQL surface documented but no
command registered) is baselined for a separate fix.
Merging in the base branch pulled in a wave of new commands from main
(model downloads, node deps, run-template, system-stats, workflow
notes, etc.) that weren't in this PR's committed inventory snapshot.
Separately, the base's run-cli rewrite already replaced the invalid
`comfy query` reference with `comfy nodes ls`, so that BASELINE_INVALID_REFS
entry is now stale per test_baseline_is_not_stale's own contract.
Main now ships the stronger shared mention lint (comfy_cli/command_mentions.py
+ tests/comfy_cli/test_command_mentions.py), which skips global options, scans
README.md, and derives envelope-only names structurally. Remove this PR's
parallel extractor/allowlists and keep only the command-inventory snapshot.

- Port the `comfy auth whoami` regression case into test_command_mentions.py.
- UPDATE_CLI_SNAPSHOT now only regenerates for 1/true/yes, so =0/false runs
  the assertion instead of silently rewriting the fixture.
- Regenerate the snapshot against current main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merged origin/main, which added `comfy deploy cancel` (#991); regenerated
the fixture with UPDATE_CLI_SNAPSHOT=1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds commands landed since the last snapshot: assets library upload,
build quota, code-search-context, knowledge authoring, model upload,
workflow check, and workflow disconnect.
@mattmillerai
mattmillerai force-pushed the matt/be-2979-hint-lint-discover-snapshot branch from 7b0f005 to 648190d Compare November 16, 2026 23:39
@comfy-greenlight-bot

comfy-greenlight-bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Swarmhost agentic review

The detailed evaluation is available to employees in the internal Slack review thread.

Evaluation budget remaining for this pull request: 14 automatic and 15 manual.

Updated by Swarmhost's agentic review process.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-coded PR authored by the agent-work loop cursor-review Request Cursor bot review greenlight size:XS This PR changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants