From 31973177bc18e2f8161f23a4352d05d5855758c2 Mon Sep 17 00:00:00 2001 From: Matt Miller Date: Tue, 14 Jul 2026 13:04:24 -0700 Subject: [PATCH 1/5] test: lint comfy hint-strings + snapshot the command inventory MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add two anti-rot CLI-surface guards, mirroring the MCP server's EXPECTED_TOOLS + description-budget snapshot pattern: - Hint-string lint: extract every backtick/shell-hint `comfy …` invocation from help text, docstrings, hint/error strings, examples, and bundled skill docs, then validate each against the registered command set by walking the Typer app tree. Fails naming the offending string + file. A regression fixture pins that it flags the historical `comfy auth login` rot (and the sibling `auth whoami`). - Command-inventory snapshot: snapshot the full command tree so surface additions/removals/hides are explicit diffs in review. Regenerate with UPDATE_CLI_SNAPSHOT=1 pytest tests/comfy_cli/test_cli_surface.py. The lint immediately surfaced live rot: five hint strings pointed at the nonexistent `comfy auth whoami` (deliberately removed — see test_legacy_auth_whoami_is_gone). Route them to the real `comfy cloud whoami`. The remaining `comfy query` rot (CQL surface documented but no command registered) is baselined for a separate fix. --- .../fixtures/cli_command_inventory.txt | 142 +++++++++ tests/comfy_cli/test_cli_surface.py | 294 ++++++++++++++++++ 2 files changed, 436 insertions(+) create mode 100644 tests/comfy_cli/fixtures/cli_command_inventory.txt create mode 100644 tests/comfy_cli/test_cli_surface.py diff --git a/tests/comfy_cli/fixtures/cli_command_inventory.txt b/tests/comfy_cli/fixtures/cli_command_inventory.txt new file mode 100644 index 000000000..1293be891 --- /dev/null +++ b/tests/comfy_cli/fixtures/cli_command_inventory.txt @@ -0,0 +1,142 @@ +comfy _watch [hidden] +comfy _watch _watch-job +comfy agent-review [hidden] +comfy assets +comfy assets push +comfy auth +comfy auth list +comfy auth remove +comfy auth set +comfy cloud +comfy cloud login +comfy cloud logout +comfy cloud set-base-url +comfy cloud set-key [hidden] +comfy cloud whoami +comfy code-search +comfy cs [hidden] +comfy dependency [hidden] +comfy discover +comfy download +comfy env +comfy feedback +comfy generate +comfy help +comfy install +comfy jobs +comfy jobs cancel +comfy jobs ls +comfy jobs status +comfy jobs wait +comfy jobs watch +comfy launch +comfy logs +comfy manager +comfy manager clear +comfy manager disable +comfy manager disable-gui +comfy manager enable-gui +comfy manager enable-legacy-gui +comfy manager migrate-legacy +comfy manager uv-compile-default +comfy model +comfy model download +comfy model list +comfy model remove +comfy models +comfy models list-folder +comfy models list-folders +comfy models search +comfy models show +comfy node +comfy node bisect +comfy node bisect bad +comfy node bisect good +comfy node bisect reset +comfy node bisect start +comfy node deps-in-workflow +comfy node disable +comfy node enable +comfy node fix +comfy node init +comfy node install +comfy node install-deps +comfy node pack +comfy node publish +comfy node registry-install [hidden] +comfy node registry-list [hidden] +comfy node reinstall +comfy node restore-dependencies +comfy node restore-snapshot +comfy node save-snapshot +comfy node scaffold +comfy node show +comfy node simple-show +comfy node uninstall +comfy node update +comfy node update-cache +comfy node uv-sync +comfy node validate +comfy nodes +comfy nodes categories +comfy nodes downstream +comfy nodes ls +comfy nodes path +comfy nodes refresh +comfy nodes search +comfy nodes show +comfy nodes types +comfy nodes upstream +comfy pr-cache +comfy pr-cache clean +comfy pr-cache list +comfy preview +comfy project +comfy project init +comfy project status +comfy run +comfy run-cli +comfy set-default +comfy setup +comfy skill [hidden] +comfy skill install +comfy skill list +comfy skill show +comfy skill status +comfy skill uninstall +comfy skill validate +comfy skills +comfy skills install +comfy skills list +comfy skills show +comfy skills status +comfy skills uninstall +comfy skills validate +comfy standalone +comfy stop +comfy templates +comfy templates fetch +comfy templates ls +comfy templates refresh +comfy templates show +comfy tracking +comfy tracking disable +comfy tracking enable +comfy update +comfy upload +comfy validate +comfy which +comfy workflow +comfy workflow compose +comfy workflow decompose +comfy workflow delete +comfy workflow fragment +comfy workflow fragment ls +comfy workflow fragment show +comfy workflow fragment validate +comfy workflow get +comfy workflow list +comfy workflow save +comfy workflow set-slot +comfy workflow slots +comfy workflow vary diff --git a/tests/comfy_cli/test_cli_surface.py b/tests/comfy_cli/test_cli_surface.py new file mode 100644 index 000000000..0ebb346b5 --- /dev/null +++ b/tests/comfy_cli/test_cli_surface.py @@ -0,0 +1,294 @@ +"""CLI-surface guards: hint-string lint + command-inventory snapshot. + +Two anti-rot tests, mirroring the MCP server's ``EXPECTED_TOOLS`` + +description-budget snapshot pattern for the CLI: + +1. **Hint-string lint** (:func:`test_no_invalid_command_references`) — extracts + every ``comfy …`` command invocation referenced in help text, docstrings, + hint strings, error messages, examples, and bundled skill docs, then + validates each against the *registered* command set by walking the Typer app + tree. Catches the class of rot where a help/hint string points at a command + that does not exist (e.g. the seven strings that referenced the nonexistent + ``comfy auth login`` for who knows how long, since fixed to ``comfy cloud + login``). + +2. **Command-inventory snapshot** (:func:`test_command_inventory_snapshot`) — + snapshots the full command tree so any surface addition/removal/hide shows up + as an explicit diff in review. + + Regenerate the snapshot after an intentional surface change with:: + + UPDATE_CLI_SNAPSHOT=1 pytest tests/comfy_cli/test_cli_surface.py + +Both run under the normal ``pytest`` invocation, so they gate every PR in CI. +""" + +from __future__ import annotations + +import ast +import os +import re +from pathlib import Path + +import pytest + +from comfy_cli.cmdline import app +from comfy_cli.help_json import build_help_json + +REPO_ROOT = Path(__file__).resolve().parents[2] +PACKAGE_DIR = REPO_ROOT / "comfy_cli" +SNAPSHOT_PATH = Path(__file__).resolve().parent / "fixtures" / "cli_command_inventory.txt" + +# ``comfy version`` is not an invokable subcommand — it is the output-envelope +# command name emitted by the top-level ``--version`` flag (cmdline.py) and is +# registered as such in discovery.COMMAND_SCHEMAS. Accept it as a real surface +# name even though the tree walk (which only sees invokable subcommands) can't. +ALLOWED_NONINVOKABLE_REFS = frozenset({"comfy version"}) + +# Known, pre-existing invalid references that this PR intentionally does NOT +# fix — the lint still reports them, but they don't fail the build. Each entry +# is tracked for a separate fix (mirroring how the seven ``auth login`` strings +# were corrected in their own change, separate from adding this guard). Removing +# the rot must also remove its baseline entry — test_baseline_is_not_stale enforces +# that so the baseline can never silently outlive the bug. +# +# "comfy query" — the CQL query surface is documented in HELP_EXAMPLES and +# demoed by ``comfy run-cli``, but no ``query`` command is registered on the +# app. Whether to wire one up or rewrite the examples is a product call, so +# it is left for a dedicated fix ticket rather than bundled into this guard. +BASELINE_INVALID_REFS = frozenset({"comfy query"}) + + +# --------------------------------------------------------------------------- # +# command tree # +# --------------------------------------------------------------------------- # +def _command_tree() -> dict: + """Nested ``{name: node}`` tree of the registered command surface. + + ``node`` carries a ``subcommands`` dict for groups; leaf commands omit it. + Wrapped under a synthetic root so the validator can start at ``comfy``. + """ + doc = build_help_json(app) + return {"subcommands": doc["commands"]} + + +def _ref_is_valid(tokens: list[str], tree: dict) -> bool: + """Is ``comfy `` a real command path? + + Walk the tree consuming subcommand-name tokens. A reference is valid when it + lands on a real command node — trailing option/argument tokens after a leaf + are fine (they're stripped before we get here). It is invalid the moment a + command-name token is not a subcommand of the current group (the ``comfy + auth login`` / ``comfy auth whoami`` class of rot). + """ + node = tree + for tok in tokens: + subs = node.get("subcommands") + if subs is None: + return True # reached a leaf; the rest are its args + if tok in subs: + node = subs[tok] + else: + return False + return True + + +def _command_inventory() -> list[str]: + """Every command path (groups + leaves), sorted, with a ``[hidden]`` marker.""" + doc = build_help_json(app) + out: list[str] = [] + + def walk(commands: dict, path: list[str]) -> None: + for name, entry in sorted(commands.items()): + fqp = " ".join(path + [name]) + out.append(f"{fqp}\t[hidden]" if entry.get("hidden") else fqp) + subs = entry.get("subcommands") + if subs: + walk(subs, path + [name]) + + walk(doc["commands"], ["comfy"]) + return out + + +# --------------------------------------------------------------------------- # +# reference extraction # +# --------------------------------------------------------------------------- # +# A command-name token: lowercase, the shape Typer/Click command names take. +_COMMAND_WORD = re.compile(r"[a-z][a-z0-9-]*$") +# Recognizable ``comfy …`` invocations. We only treat a string as a command +# reference when it carries an invocation marker, so prose like "the comfy +# skills into Claude Code" is not mistaken for ``comfy skills into``. +_BACKTICK = re.compile(r"`\s*(?:\$\s*)?comfy\b([^`]*)`") # `comfy …` / `$ comfy …` +_SHELL_HINT = re.compile(r"(?:^|[\s(])(?:run:?\s+|\$\s+)comfy\b([^\n`.,;:)]*)", re.IGNORECASE) +_WHOLE_LINE = re.compile(r"^comfy\b([a-z0-9 _-]*)$") # the whole string is a command line + + +def _leading_command_tokens(rest: str) -> list[str]: + """Leading subcommand-name tokens of ``comfy `` (stop at the first + option/argument/placeholder token).""" + tokens: list[str] = [] + for tok in rest.strip().split(): + if _COMMAND_WORD.match(tok): + tokens.append(tok) + else: + break + return tokens + + +def _refs_in_text(text: str) -> list[list[str]]: + """Token-lists for every recognizable ``comfy …`` reference in ``text``.""" + seen: set[tuple[str, ...]] = set() + out: list[list[str]] = [] + parts = [m.group(1) for m in _BACKTICK.finditer(text)] + parts += [m.group(1) for m in _SHELL_HINT.finditer(text)] + whole = _WHOLE_LINE.match(text.strip()) + if whole: + parts.append(whole.group(1)) + for rest in parts: + tokens = _leading_command_tokens(rest) + key = tuple(tokens) + if key in seen: + continue + seen.add(key) + out.append(tokens) + return out + + +def _iter_source_strings(): + """Yield ``(relpath, lineno, string)`` for every scannable help/hint string. + + Python files contribute their string *literals* (docstrings, ``help=`` / + ``hint=`` / ``message=`` kwargs, examples) via the AST, so command + identifiers and comments are never mistaken for references. Bundled skill + docs (``*.md``) contribute their raw text — they're agent-facing help. + """ + for py in sorted(PACKAGE_DIR.rglob("*.py")): + rel = py.relative_to(REPO_ROOT).as_posix() + try: + tree = ast.parse(py.read_text(encoding="utf-8")) + except SyntaxError: # pragma: no cover - our own sources parse + continue + for node in ast.walk(tree): + if isinstance(node, ast.Constant) and isinstance(node.value, str): + yield rel, node.lineno, node.value + for md in sorted(PACKAGE_DIR.rglob("*.md")): + rel = md.relative_to(REPO_ROOT).as_posix() + for i, line in enumerate(md.read_text(encoding="utf-8").splitlines(), start=1): + yield rel, i, line + + +def _scan_invalid_references() -> list[tuple[str, int, str]]: + """``(relpath, lineno, "comfy …")`` for every invalid command reference.""" + tree = _command_tree() + found: list[tuple[str, int, str]] = [] + seen: set[tuple[str, int, str]] = set() + for rel, lineno, text in _iter_source_strings(): + for tokens in _refs_in_text(text): + ref = " ".join(["comfy", *tokens]) + if ref in ALLOWED_NONINVOKABLE_REFS or _ref_is_valid(tokens, tree): + continue + key = (rel, lineno, ref) + if key not in seen: + seen.add(key) + found.append(key) + return found + + +# --------------------------------------------------------------------------- # +# tests: hint-string lint # +# --------------------------------------------------------------------------- # +def test_no_invalid_command_references(): + """No help/hint/error string references a command the CLI doesn't register.""" + invalid = [v for v in _scan_invalid_references() if v[2] not in BASELINE_INVALID_REFS] + if invalid: + lines = "\n".join(f" {rel}:{lineno}: `{ref}`" for rel, lineno, ref in sorted(invalid)) + pytest.fail( + "Help/hint/error strings reference commands that are not registered " + "on the CLI (run `comfy ` to confirm each is missing):\n" + f"{lines}\n\n" + "Fix the string to name a real command, or register the command. " + "If this is known, tracked rot, add it to BASELINE_INVALID_REFS." + ) + + +def test_baseline_is_not_stale(): + """Every BASELINE_INVALID_REFS entry must still be a live, invalid reference. + + Once the underlying rot is fixed the reference disappears; this test then + fails, prompting removal of the now-stale baseline entry so the allowlist + can never quietly outlive the bug it excuses. + """ + live_refs = {ref for _, _, ref in _scan_invalid_references()} + stale = sorted(BASELINE_INVALID_REFS - live_refs) + assert not stale, f"BASELINE_INVALID_REFS entries no longer found (fixed?) — remove them: {stale}" + + +def test_lint_catches_auth_login_and_friends(): + """Regression fixture: the extractor+validator flags exactly the bad refs. + + The seven historical `comfy auth login` strings have since been corrected, + so they no longer live in the tree — this pins the lint's *behaviour* against + synthetic data (an `auth login` fixture included per the ticket) so a future + refactor can't silently stop catching them. + """ + tree = _command_tree() + + def is_flagged(text: str) -> bool: + return any( + " ".join(["comfy", *toks]) not in ALLOWED_NONINVOKABLE_REFS and not _ref_is_valid(toks, tree) + for toks in _refs_in_text(text) + ) + + should_flag = [ + "re-run `comfy auth login` and complete the sign-in", # the historical rot + "run: comfy auth login", # non-backtick shell-hint form + "check network / `comfy auth whoami`", # sibling rot (wrong group) + "comfy auth login", # whole-string form + "`comfy nodes bogus`", # invalid leaf under a real group + ] + should_pass = [ + "run `comfy cloud login` to sign in", # the correct command + "`comfy run --workflow wf.json`", # valid leaf + args + "`comfy workflow`", # a bare group is a valid reference + "`comfy generate list`", # leaf that takes positional subcommands + "`comfy version`", # allowlisted --version envelope name + "Write the comfy skills into Claude Code, Cursor.", # prose, not an invocation + ] + missed = [t for t in should_flag if not is_flagged(t)] + false_pos = [t for t in should_pass if is_flagged(t)] + assert not missed, f"lint failed to flag invalid references: {missed}" + assert not false_pos, f"lint false-positived on valid references / prose: {false_pos}" + + +# --------------------------------------------------------------------------- # +# tests: command-inventory snapshot # +# --------------------------------------------------------------------------- # +def test_command_inventory_snapshot(): + """The registered command surface matches the committed snapshot.""" + current = _command_inventory() + rendered = "\n".join(current) + "\n" + + if os.environ.get("UPDATE_CLI_SNAPSHOT"): + SNAPSHOT_PATH.write_text(rendered, encoding="utf-8") + pytest.skip(f"snapshot regenerated: {SNAPSHOT_PATH}") + + assert SNAPSHOT_PATH.exists(), ( + f"missing snapshot {SNAPSHOT_PATH}; regenerate with " + "`UPDATE_CLI_SNAPSHOT=1 pytest tests/comfy_cli/test_cli_surface.py`" + ) + expected = SNAPSHOT_PATH.read_text(encoding="utf-8").splitlines() + if current != expected: + added = sorted(set(current) - set(expected)) + removed = sorted(set(expected) - set(current)) + detail = "" + if added: + detail += "\n added:\n" + "\n".join(f" + {p}" for p in added) + if removed: + detail += "\n removed:\n" + "\n".join(f" - {p}" for p in removed) + pytest.fail( + "CLI command inventory changed vs. the committed snapshot." + f"{detail}\n\n" + "If intentional, regenerate with " + "`UPDATE_CLI_SNAPSHOT=1 pytest tests/comfy_cli/test_cli_surface.py` " + "and review the diff." + ) From d964dd5cd2504d40e1e82978e3ffa3b0171b35d3 Mon Sep 17 00:00:00 2001 From: Matt Miller Date: Wed, 29 Jul 2026 22:05:37 -0700 Subject: [PATCH 2/5] fix: reconcile CLI surface snapshot + drop stale baseline after merge Merging in the base branch pulled in a wave of new commands from main (model downloads, node deps, run-template, system-stats, workflow notes, etc.) that weren't in this PR's committed inventory snapshot. Separately, the base's run-cli rewrite already replaced the invalid `comfy query` reference with `comfy nodes ls`, so that BASELINE_INVALID_REFS entry is now stale per test_baseline_is_not_stale's own contract. --- tests/comfy_cli/fixtures/cli_command_inventory.txt | 14 ++++++++++++-- tests/comfy_cli/test_cli_surface.py | 7 +------ 2 files changed, 13 insertions(+), 8 deletions(-) diff --git a/tests/comfy_cli/fixtures/cli_command_inventory.txt b/tests/comfy_cli/fixtures/cli_command_inventory.txt index 1293be891..2597939c0 100644 --- a/tests/comfy_cli/fixtures/cli_command_inventory.txt +++ b/tests/comfy_cli/fixtures/cli_command_inventory.txt @@ -11,15 +11,16 @@ comfy cloud comfy cloud login comfy cloud logout comfy cloud set-base-url -comfy cloud set-key [hidden] +comfy cloud set-key comfy cloud whoami comfy code-search comfy cs [hidden] -comfy dependency [hidden] +comfy dependency comfy discover comfy download comfy env comfy feedback +comfy free comfy generate comfy help comfy install @@ -40,7 +41,11 @@ comfy manager enable-legacy-gui comfy manager migrate-legacy comfy manager uv-compile-default comfy model +comfy model _download-worker [hidden] comfy model download +comfy model download-cancel +comfy model download-status +comfy model downloads comfy model list comfy model remove comfy models @@ -54,6 +59,7 @@ comfy node bisect bad comfy node bisect good comfy node bisect reset comfy node bisect start +comfy node deps comfy node deps-in-workflow comfy node disable comfy node enable @@ -87,6 +93,7 @@ comfy nodes search comfy nodes show comfy nodes types comfy nodes upstream +comfy outdated comfy pr-cache comfy pr-cache clean comfy pr-cache list @@ -96,6 +103,7 @@ comfy project init comfy project status comfy run comfy run-cli +comfy run-template comfy set-default comfy setup comfy skill [hidden] @@ -114,6 +122,7 @@ comfy skills uninstall comfy skills validate comfy standalone comfy stop +comfy system-stats comfy templates comfy templates fetch comfy templates ls @@ -136,6 +145,7 @@ comfy workflow fragment show comfy workflow fragment validate comfy workflow get comfy workflow list +comfy workflow notes comfy workflow save comfy workflow set-slot comfy workflow slots diff --git a/tests/comfy_cli/test_cli_surface.py b/tests/comfy_cli/test_cli_surface.py index 0ebb346b5..3fef6a7a7 100644 --- a/tests/comfy_cli/test_cli_surface.py +++ b/tests/comfy_cli/test_cli_surface.py @@ -51,12 +51,7 @@ # were corrected in their own change, separate from adding this guard). Removing # the rot must also remove its baseline entry — test_baseline_is_not_stale enforces # that so the baseline can never silently outlive the bug. -# -# "comfy query" — the CQL query surface is documented in HELP_EXAMPLES and -# demoed by ``comfy run-cli``, but no ``query`` command is registered on the -# app. Whether to wire one up or rewrite the examples is a product call, so -# it is left for a dedicated fix ticket rather than bundled into this guard. -BASELINE_INVALID_REFS = frozenset({"comfy query"}) +BASELINE_INVALID_REFS = frozenset() # --------------------------------------------------------------------------- # From c6f06592c79b2d885f8c7c60688b4ace31028db9 Mon Sep 17 00:00:00 2001 From: Matt Miller Date: Thu, 8 Oct 2026 06:37:17 +0000 Subject: [PATCH 3/5] test: drop duplicate hint lint, keep command-inventory snapshot Main now ships the stronger shared mention lint (comfy_cli/command_mentions.py + tests/comfy_cli/test_command_mentions.py), which skips global options, scans README.md, and derives envelope-only names structurally. Remove this PR's parallel extractor/allowlists and keep only the command-inventory snapshot. - Port the `comfy auth whoami` regression case into test_command_mentions.py. - UPDATE_CLI_SNAPSHOT now only regenerates for 1/true/yes, so =0/false runs the assertion instead of silently rewriting the fixture. - Regenerate the snapshot against current main. Co-Authored-By: Claude Opus 5.5 --- .../fixtures/cli_command_inventory.txt | 83 ++++++- tests/comfy_cli/test_cli_surface.py | 234 +----------------- tests/comfy_cli/test_command_mentions.py | 1 + 3 files changed, 90 insertions(+), 228 deletions(-) diff --git a/tests/comfy_cli/fixtures/cli_command_inventory.txt b/tests/comfy_cli/fixtures/cli_command_inventory.txt index 2597939c0..21e9631f0 100644 --- a/tests/comfy_cli/fixtures/cli_command_inventory.txt +++ b/tests/comfy_cli/fixtures/cli_command_inventory.txt @@ -1,21 +1,71 @@ comfy _watch [hidden] comfy _watch _watch-job +comfy agent +comfy agent allow +comfy agent deny +comfy agent permissions comfy agent-review [hidden] comfy assets +comfy assets library +comfy assets library ensure +comfy assets library ls comfy assets push comfy auth comfy auth list comfy auth remove comfy auth set +comfy build +comfy build blob [hidden] +comfy build blob ls +comfy build delete +comfy build init +comfy build ls +comfy build pull +comfy build push +comfy build refs +comfy build refs base-images +comfy build refs build-targets +comfy build refs model-dirs +comfy build refs resolve +comfy build release +comfy build release create +comfy build release delete +comfy build release logs +comfy build release ls +comfy build release manifest +comfy build release show +comfy build show +comfy build status +comfy build update +comfy build validate comfy cloud comfy cloud login comfy cloud logout comfy cloud set-base-url comfy cloud set-key +comfy cloud status comfy cloud whoami comfy code-search comfy cs [hidden] -comfy dependency +comfy dependency [hidden] +comfy deploy +comfy deploy delete +comfy deploy events +comfy deploy history +comfy deploy logs +comfy deploy ls +comfy deploy promote +comfy deploy refs +comfy deploy refs compute +comfy deploy rename +comfy deploy rollback +comfy deploy run +comfy deploy scale +comfy deploy show +comfy deploy start +comfy deploy status +comfy deploy stop +comfy deploy up comfy discover comfy download comfy env @@ -30,6 +80,10 @@ comfy jobs ls comfy jobs status comfy jobs wait comfy jobs watch +comfy knowledge +comfy knowledge pick +comfy knowledge resolve +comfy knowledge status comfy launch comfy logs comfy manager @@ -47,8 +101,12 @@ comfy model download-cancel comfy model download-status comfy model downloads comfy model list +comfy model list-folder +comfy model list-folders comfy model remove -comfy models +comfy model search +comfy model show +comfy models [hidden] comfy models list-folder comfy models list-folders comfy models search @@ -93,6 +151,7 @@ comfy nodes search comfy nodes show comfy nodes types comfy nodes upstream +comfy nodes widget-catalog comfy outdated comfy pr-cache comfy pr-cache clean @@ -124,7 +183,10 @@ comfy standalone comfy stop comfy system-stats comfy templates +comfy templates _refresh-cache [hidden] +comfy templates check comfy templates fetch +comfy templates get comfy templates ls comfy templates refresh comfy templates show @@ -133,20 +195,35 @@ comfy tracking disable comfy tracking enable comfy update comfy upload -comfy validate +comfy validate [hidden] comfy which comfy workflow +comfy workflow add-node +comfy workflow apply +comfy workflow capture +comfy workflow clear comfy workflow compose +comfy workflow connect comfy workflow decompose comfy workflow delete +comfy workflow delete-node +comfy workflow delete-nodes +comfy workflow foreach comfy workflow fragment comfy workflow fragment ls comfy workflow fragment show comfy workflow fragment validate comfy workflow get +comfy workflow insert-workflow comfy workflow list +comfy workflow ls-nodes comfy workflow notes +comfy workflow print +comfy workflow reset-doc comfy workflow save +comfy workflow set-node-field comfy workflow set-slot +comfy workflow set-widget comfy workflow slots +comfy workflow validate comfy workflow vary diff --git a/tests/comfy_cli/test_cli_surface.py b/tests/comfy_cli/test_cli_surface.py index 3fef6a7a7..8f2ea46b2 100644 --- a/tests/comfy_cli/test_cli_surface.py +++ b/tests/comfy_cli/test_cli_surface.py @@ -1,33 +1,20 @@ -"""CLI-surface guards: hint-string lint + command-inventory snapshot. +"""CLI-surface guard: command-inventory snapshot. -Two anti-rot tests, mirroring the MCP server's ``EXPECTED_TOOLS`` + -description-budget snapshot pattern for the CLI: +Mirrors the MCP server's ``EXPECTED_TOOLS`` snapshot pattern for the CLI: +:func:`test_command_inventory_snapshot` snapshots the full command tree so any +surface addition/removal/hide shows up as an explicit diff in review. -1. **Hint-string lint** (:func:`test_no_invalid_command_references`) — extracts - every ``comfy …`` command invocation referenced in help text, docstrings, - hint strings, error messages, examples, and bundled skill docs, then - validates each against the *registered* command set by walking the Typer app - tree. Catches the class of rot where a help/hint string points at a command - that does not exist (e.g. the seven strings that referenced the nonexistent - ``comfy auth login`` for who knows how long, since fixed to ``comfy cloud - login``). +Regenerate the snapshot after an intentional surface change with:: -2. **Command-inventory snapshot** (:func:`test_command_inventory_snapshot`) — - snapshots the full command tree so any surface addition/removal/hide shows up - as an explicit diff in review. + UPDATE_CLI_SNAPSHOT=1 pytest tests/comfy_cli/test_cli_surface.py - Regenerate the snapshot after an intentional surface change with:: - - UPDATE_CLI_SNAPSHOT=1 pytest tests/comfy_cli/test_cli_surface.py - -Both run under the normal ``pytest`` invocation, so they gate every PR in CI. +Help/hint strings that name nonexistent commands are linted separately by +``tests/comfy_cli/test_command_mentions.py``. """ from __future__ import annotations -import ast import os -import re from pathlib import Path import pytest @@ -35,58 +22,8 @@ from comfy_cli.cmdline import app from comfy_cli.help_json import build_help_json -REPO_ROOT = Path(__file__).resolve().parents[2] -PACKAGE_DIR = REPO_ROOT / "comfy_cli" SNAPSHOT_PATH = Path(__file__).resolve().parent / "fixtures" / "cli_command_inventory.txt" -# ``comfy version`` is not an invokable subcommand — it is the output-envelope -# command name emitted by the top-level ``--version`` flag (cmdline.py) and is -# registered as such in discovery.COMMAND_SCHEMAS. Accept it as a real surface -# name even though the tree walk (which only sees invokable subcommands) can't. -ALLOWED_NONINVOKABLE_REFS = frozenset({"comfy version"}) - -# Known, pre-existing invalid references that this PR intentionally does NOT -# fix — the lint still reports them, but they don't fail the build. Each entry -# is tracked for a separate fix (mirroring how the seven ``auth login`` strings -# were corrected in their own change, separate from adding this guard). Removing -# the rot must also remove its baseline entry — test_baseline_is_not_stale enforces -# that so the baseline can never silently outlive the bug. -BASELINE_INVALID_REFS = frozenset() - - -# --------------------------------------------------------------------------- # -# command tree # -# --------------------------------------------------------------------------- # -def _command_tree() -> dict: - """Nested ``{name: node}`` tree of the registered command surface. - - ``node`` carries a ``subcommands`` dict for groups; leaf commands omit it. - Wrapped under a synthetic root so the validator can start at ``comfy``. - """ - doc = build_help_json(app) - return {"subcommands": doc["commands"]} - - -def _ref_is_valid(tokens: list[str], tree: dict) -> bool: - """Is ``comfy `` a real command path? - - Walk the tree consuming subcommand-name tokens. A reference is valid when it - lands on a real command node — trailing option/argument tokens after a leaf - are fine (they're stripped before we get here). It is invalid the moment a - command-name token is not a subcommand of the current group (the ``comfy - auth login`` / ``comfy auth whoami`` class of rot). - """ - node = tree - for tok in tokens: - subs = node.get("subcommands") - if subs is None: - return True # reached a leaf; the rest are its args - if tok in subs: - node = subs[tok] - else: - return False - return True - def _command_inventory() -> list[str]: """Every command path (groups + leaves), sorted, with a ``[hidden]`` marker.""" @@ -105,165 +42,12 @@ def walk(commands: dict, path: list[str]) -> None: return out -# --------------------------------------------------------------------------- # -# reference extraction # -# --------------------------------------------------------------------------- # -# A command-name token: lowercase, the shape Typer/Click command names take. -_COMMAND_WORD = re.compile(r"[a-z][a-z0-9-]*$") -# Recognizable ``comfy …`` invocations. We only treat a string as a command -# reference when it carries an invocation marker, so prose like "the comfy -# skills into Claude Code" is not mistaken for ``comfy skills into``. -_BACKTICK = re.compile(r"`\s*(?:\$\s*)?comfy\b([^`]*)`") # `comfy …` / `$ comfy …` -_SHELL_HINT = re.compile(r"(?:^|[\s(])(?:run:?\s+|\$\s+)comfy\b([^\n`.,;:)]*)", re.IGNORECASE) -_WHOLE_LINE = re.compile(r"^comfy\b([a-z0-9 _-]*)$") # the whole string is a command line - - -def _leading_command_tokens(rest: str) -> list[str]: - """Leading subcommand-name tokens of ``comfy `` (stop at the first - option/argument/placeholder token).""" - tokens: list[str] = [] - for tok in rest.strip().split(): - if _COMMAND_WORD.match(tok): - tokens.append(tok) - else: - break - return tokens - - -def _refs_in_text(text: str) -> list[list[str]]: - """Token-lists for every recognizable ``comfy …`` reference in ``text``.""" - seen: set[tuple[str, ...]] = set() - out: list[list[str]] = [] - parts = [m.group(1) for m in _BACKTICK.finditer(text)] - parts += [m.group(1) for m in _SHELL_HINT.finditer(text)] - whole = _WHOLE_LINE.match(text.strip()) - if whole: - parts.append(whole.group(1)) - for rest in parts: - tokens = _leading_command_tokens(rest) - key = tuple(tokens) - if key in seen: - continue - seen.add(key) - out.append(tokens) - return out - - -def _iter_source_strings(): - """Yield ``(relpath, lineno, string)`` for every scannable help/hint string. - - Python files contribute their string *literals* (docstrings, ``help=`` / - ``hint=`` / ``message=`` kwargs, examples) via the AST, so command - identifiers and comments are never mistaken for references. Bundled skill - docs (``*.md``) contribute their raw text — they're agent-facing help. - """ - for py in sorted(PACKAGE_DIR.rglob("*.py")): - rel = py.relative_to(REPO_ROOT).as_posix() - try: - tree = ast.parse(py.read_text(encoding="utf-8")) - except SyntaxError: # pragma: no cover - our own sources parse - continue - for node in ast.walk(tree): - if isinstance(node, ast.Constant) and isinstance(node.value, str): - yield rel, node.lineno, node.value - for md in sorted(PACKAGE_DIR.rglob("*.md")): - rel = md.relative_to(REPO_ROOT).as_posix() - for i, line in enumerate(md.read_text(encoding="utf-8").splitlines(), start=1): - yield rel, i, line - - -def _scan_invalid_references() -> list[tuple[str, int, str]]: - """``(relpath, lineno, "comfy …")`` for every invalid command reference.""" - tree = _command_tree() - found: list[tuple[str, int, str]] = [] - seen: set[tuple[str, int, str]] = set() - for rel, lineno, text in _iter_source_strings(): - for tokens in _refs_in_text(text): - ref = " ".join(["comfy", *tokens]) - if ref in ALLOWED_NONINVOKABLE_REFS or _ref_is_valid(tokens, tree): - continue - key = (rel, lineno, ref) - if key not in seen: - seen.add(key) - found.append(key) - return found - - -# --------------------------------------------------------------------------- # -# tests: hint-string lint # -# --------------------------------------------------------------------------- # -def test_no_invalid_command_references(): - """No help/hint/error string references a command the CLI doesn't register.""" - invalid = [v for v in _scan_invalid_references() if v[2] not in BASELINE_INVALID_REFS] - if invalid: - lines = "\n".join(f" {rel}:{lineno}: `{ref}`" for rel, lineno, ref in sorted(invalid)) - pytest.fail( - "Help/hint/error strings reference commands that are not registered " - "on the CLI (run `comfy ` to confirm each is missing):\n" - f"{lines}\n\n" - "Fix the string to name a real command, or register the command. " - "If this is known, tracked rot, add it to BASELINE_INVALID_REFS." - ) - - -def test_baseline_is_not_stale(): - """Every BASELINE_INVALID_REFS entry must still be a live, invalid reference. - - Once the underlying rot is fixed the reference disappears; this test then - fails, prompting removal of the now-stale baseline entry so the allowlist - can never quietly outlive the bug it excuses. - """ - live_refs = {ref for _, _, ref in _scan_invalid_references()} - stale = sorted(BASELINE_INVALID_REFS - live_refs) - assert not stale, f"BASELINE_INVALID_REFS entries no longer found (fixed?) — remove them: {stale}" - - -def test_lint_catches_auth_login_and_friends(): - """Regression fixture: the extractor+validator flags exactly the bad refs. - - The seven historical `comfy auth login` strings have since been corrected, - so they no longer live in the tree — this pins the lint's *behaviour* against - synthetic data (an `auth login` fixture included per the ticket) so a future - refactor can't silently stop catching them. - """ - tree = _command_tree() - - def is_flagged(text: str) -> bool: - return any( - " ".join(["comfy", *toks]) not in ALLOWED_NONINVOKABLE_REFS and not _ref_is_valid(toks, tree) - for toks in _refs_in_text(text) - ) - - should_flag = [ - "re-run `comfy auth login` and complete the sign-in", # the historical rot - "run: comfy auth login", # non-backtick shell-hint form - "check network / `comfy auth whoami`", # sibling rot (wrong group) - "comfy auth login", # whole-string form - "`comfy nodes bogus`", # invalid leaf under a real group - ] - should_pass = [ - "run `comfy cloud login` to sign in", # the correct command - "`comfy run --workflow wf.json`", # valid leaf + args - "`comfy workflow`", # a bare group is a valid reference - "`comfy generate list`", # leaf that takes positional subcommands - "`comfy version`", # allowlisted --version envelope name - "Write the comfy skills into Claude Code, Cursor.", # prose, not an invocation - ] - missed = [t for t in should_flag if not is_flagged(t)] - false_pos = [t for t in should_pass if is_flagged(t)] - assert not missed, f"lint failed to flag invalid references: {missed}" - assert not false_pos, f"lint false-positived on valid references / prose: {false_pos}" - - -# --------------------------------------------------------------------------- # -# tests: command-inventory snapshot # -# --------------------------------------------------------------------------- # def test_command_inventory_snapshot(): """The registered command surface matches the committed snapshot.""" current = _command_inventory() rendered = "\n".join(current) + "\n" - if os.environ.get("UPDATE_CLI_SNAPSHOT"): + if os.environ.get("UPDATE_CLI_SNAPSHOT", "").strip().lower() in {"1", "true", "yes"}: SNAPSHOT_PATH.write_text(rendered, encoding="utf-8") pytest.skip(f"snapshot regenerated: {SNAPSHOT_PATH}") diff --git a/tests/comfy_cli/test_command_mentions.py b/tests/comfy_cli/test_command_mentions.py index 8299e6544..8330bb97b 100644 --- a/tests/comfy_cli/test_command_mentions.py +++ b/tests/comfy_cli/test_command_mentions.py @@ -211,6 +211,7 @@ def test_valid_mentions_pass(text, tree, globals_): ("text", "unknown"), [ ("comfy auth login", "login"), # the original regression itself + ("comfy auth whoami", "whoami"), # sibling rot: removed in favour of `comfy cloud whoami` ("comfy cloud singin", "singin"), ("comfy nope", "nope"), ("comfy model downlaod --url x", "downlaod"), From 666ce15f65e7338da103a17352d6f4412aec37f6 Mon Sep 17 00:00:00 2001 From: Matt Miller Date: Wed, 14 Oct 2026 02:51:50 +0000 Subject: [PATCH 4/5] test: add comfy deploy cancel to command-inventory snapshot Merged origin/main, which added `comfy deploy cancel` (#991); regenerated the fixture with UPDATE_CLI_SNAPSHOT=1. Co-Authored-By: Claude Opus 5.5 --- tests/comfy_cli/fixtures/cli_command_inventory.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/comfy_cli/fixtures/cli_command_inventory.txt b/tests/comfy_cli/fixtures/cli_command_inventory.txt index 21e9631f0..39e40f9c7 100644 --- a/tests/comfy_cli/fixtures/cli_command_inventory.txt +++ b/tests/comfy_cli/fixtures/cli_command_inventory.txt @@ -49,6 +49,7 @@ comfy code-search comfy cs [hidden] comfy dependency [hidden] comfy deploy +comfy deploy cancel comfy deploy delete comfy deploy events comfy deploy history From 648190d3ba6394322f8392c07a4a460966ade136 Mon Sep 17 00:00:00 2001 From: Matt Miller Date: Mon, 16 Nov 2026 15:38:58 -0800 Subject: [PATCH 5/5] test: regenerate cli_command_inventory snapshot against current main Adds commands landed since the last snapshot: assets library upload, build quota, code-search-context, knowledge authoring, model upload, workflow check, and workflow disconnect. --- tests/comfy_cli/fixtures/cli_command_inventory.txt | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/tests/comfy_cli/fixtures/cli_command_inventory.txt b/tests/comfy_cli/fixtures/cli_command_inventory.txt index 39e40f9c7..135541c02 100644 --- a/tests/comfy_cli/fixtures/cli_command_inventory.txt +++ b/tests/comfy_cli/fixtures/cli_command_inventory.txt @@ -9,6 +9,7 @@ comfy assets comfy assets library comfy assets library ensure comfy assets library ls +comfy assets library upload comfy assets push comfy auth comfy auth list @@ -22,6 +23,7 @@ comfy build init comfy build ls comfy build pull comfy build push +comfy build quota comfy build refs comfy build refs base-images comfy build refs build-targets @@ -46,6 +48,7 @@ comfy cloud set-key comfy cloud status comfy cloud whoami comfy code-search +comfy code-search-context comfy cs [hidden] comfy dependency [hidden] comfy deploy @@ -82,6 +85,7 @@ comfy jobs status comfy jobs wait comfy jobs watch comfy knowledge +comfy knowledge authoring comfy knowledge pick comfy knowledge resolve comfy knowledge status @@ -107,6 +111,7 @@ comfy model list-folders comfy model remove comfy model search comfy model show +comfy model upload comfy models [hidden] comfy models list-folder comfy models list-folders @@ -202,6 +207,7 @@ comfy workflow comfy workflow add-node comfy workflow apply comfy workflow capture +comfy workflow check comfy workflow clear comfy workflow compose comfy workflow connect @@ -209,6 +215,7 @@ comfy workflow decompose comfy workflow delete comfy workflow delete-node comfy workflow delete-nodes +comfy workflow disconnect comfy workflow foreach comfy workflow fragment comfy workflow fragment ls