Skip to content

feat: add complete docs-generation pipeline (scan/generate/translate/sync) - #117

Merged
comfyui-wiki merged 10 commits into
Comfy-Org:mainfrom
lin-bot23:feat/sync-pipeline
Aug 14, 2026
Merged

feat: add complete docs-generation pipeline (scan/generate/translate/sync)#117
comfyui-wiki merged 10 commits into
Comfy-Org:mainfrom
lin-bot23:feat/sync-pipeline

Conversation

@lin-bot23

@lin-bot23 lin-bot23 commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the complete documentation pipeline to this repo under docs-generation/: the full automation that scans the ComfyUI codebase, detects new/changed nodes, generates and translates node docs into 11 languages, and publishes them to Comfy-Org/docs (built-in-nodes/*.mdx + docs.json nav).

Making the pipeline part of this repo means the code that maintains these docs lives with the content it maintains, so fixes (slug casing, MDX escaping, description quality, translation reconciliation) can land at the source instead of being hand-edited in docs and overwritten by the next sync.

What's included

Component Purpose
docs-generation/scripts/scan_missing_nodes.py Scan ComfyUI source; detect new / changed / deprecated nodes (source-hash based)
docs-generation/scripts/prepare_ai_input.py + batch_generate_docs.py Generate en.md via LLM (OpenAI-compatible API)
docs-generation/scripts/batch_translate_docs.py Translate to 11 locales (zh, zh-TW, es, fr, ja, ko, ru, ar, tr, pt-BR, fa)
docs-generation/scripts/update_param_translations.py + sync_frontend_translations.py Reconcile parameter/output names with ComfyUI frontend i18n
docs-generation/scripts/sync_to_comfy_docs.py Generate built-in-nodes/*.mdx + docs.json nav in a Comfy-Org/docs checkout
docs-generation/scripts/version_tracker.py Per-node source SHA-256 tracking
docs-generation/lib/ + docs-generation/config/ Shared modules, generation rules, translation prompts
docs-generation/tests/ Unit tests (27 passing)
docs-generation/env.example + docs-generation/README.md Setup + usage docs

Notable behaviors (in sync_to_comfy_docs.py)

  • Concrete SEO descriptions: frontmatter description is extracted from each node's en.md overview first sentence (localized per locale with English fallback) instead of a templated string. This is the GEO improvement from feat: improve GEO metadata and page headings docs#1216, applied automatically to all node pages and surviving every sync.
  • Per-locale, case-safe slug resolution: published_node_name() matches against real directory entries (macOS-safe), preventing the slug-casing 404s from fix: correct casing of ClipTextEncodeControlnet nav slug in docs.json docs#1392/#1393.
  • MDX safety: fenced code blocks and inline code preserved verbatim; whitelisted HTML and paired Mintlify components (<Note>/<Tip>/<Accordion>...) kept raw; unknown tags and orphaned closing tags escaped; literal curly braces in prose escaped.
  • Robust nav updates: _purge_noncanonical_nav_pages replaces noncanonical slugs in place when the matching .mdx exists and keeps unknown keys (no silent loss); docs.json is written atomically.

Configuration

Paths and API keys come from env vars (see docs-generation/env.example); no user-specific paths are hardcoded. The LLM API key accepts any OpenAI-compatible provider via LLM_API_KEY + API_BASE_URL/API_MODEL. Generated outputs (data/, ai_input/) are gitignored.

Checklist

  • All pipeline scripts import cleanly (verified)
  • sync_to_comfy_docs.py dry-run verified against a docs checkout
  • Unit tests pass (27/27)
  • No local/user-specific paths or secrets in the diff
  • README + env.example document setup and usage

Add scripts/sync_to_docs.py, the script that generates the
built-in-nodes/* .mdx pages in Comfy-Org/docs from the doc sources
in this repo (en.md + zh/ja/ko translations), and updates docs.json
navigation with case-corrected slugs.

Key behaviors:
- Concrete SEO descriptions extracted from each node's en.md overview
  (first sentence) instead of a templated string — the GEO improvement
  previously only possible by hand-editing individual .mdx files
- Per-locale published-name resolution (macOS-safe, case-sensitive)
- MDX safety: code blocks preserved verbatim, Mintlify components
  (<Note>/<Tip>/<Accordion>...) kept raw, unknown tags escaped in pairs
- Assets copied to images/built-in-nodes/<Node>/

This makes the pipeline that owns these pages visible and maintainable
in the same repo as the content, so fixes (e.g. slug casing, MDX
escaping, description quality) land at the source instead of being
overwritten by the next sync.
The overview first-sentence extraction treated the page title (# H1) as
the end of the overview, returning empty for every node whose en.md
starts with a title line — so build_frontmatter fell back to the
templated description. H1 is now skipped (continue) and only H2+
sections end the overview. Verified: Canny, GLSLShader, OpenAIDalle2,
ClipTextEncodeSdxlRefiner now get real descriptions.
@coderabbitai

coderabbitai Bot commented Aug 12, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@lin-bot23, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 23 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 167a7038-8295-456b-b050-dcaea8ba9ae8

📥 Commits

Reviewing files that changed from the base of the PR and between 6a24c3d and 73ca188.

📒 Files selected for processing (37)
  • README.md
  • docs-generation/.gitignore
  • docs-generation/README.md
  • docs-generation/config/doc_rules.txt
  • docs-generation/config/translation_config.json
  • docs-generation/config/translation_rules.txt
  • docs-generation/env.example
  • docs-generation/lib/__init__.py
  • docs-generation/lib/doc_disclaimer.py
  • docs-generation/lib/doc_title.py
  • docs-generation/lib/hash_footer.py
  • docs-generation/lib/node_source_extract.py
  • docs-generation/lib/paths.py
  • docs-generation/main.py
  • docs-generation/requirements.txt
  • docs-generation/scripts/batch_generate_docs.py
  • docs-generation/scripts/batch_translate_docs.py
  • docs-generation/scripts/check_config.py
  • docs-generation/scripts/check_md_links.py
  • docs-generation/scripts/check_outputs.py
  • docs-generation/scripts/cleanup_duplicate_hashes.py
  • docs-generation/scripts/fix_doc_titles.py
  • docs-generation/scripts/generate_docs.py
  • docs-generation/scripts/migrate_docs_format.py
  • docs-generation/scripts/prepare_ai_input.py
  • docs-generation/scripts/prepare_translation.py
  • docs-generation/scripts/replace_placeholders.py
  • docs-generation/scripts/runtime.py
  • docs-generation/scripts/scan_missing_nodes.py
  • docs-generation/scripts/sync_frontend_translations.py
  • docs-generation/scripts/sync_to_comfy_docs.py
  • docs-generation/scripts/update_param_translations.py
  • docs-generation/scripts/update_translation_status.py
  • docs-generation/scripts/version_tracker.py
  • docs-generation/tests/test_doc_title.py
  • docs-generation/tests/test_sync_helpers.py
  • docs-generation/tests/test_sync_to_comfy_docs.sh
📝 Walkthrough

Walkthrough

Changes

Documentation synchronization

Layer / File(s) Summary
Node metadata and source resolution
scripts/sync_to_docs.py
The script loads scanner data, resolves canonical and published node names, parses categories, and discovers embedded documentation.
Localized content and asset transformation
scripts/sync_to_docs.py
The script converts localized Markdown to MDX, copies local assets, rewrites references, normalizes content, and generates frontmatter.
Localized navigation rebuilding
scripts/sync_to_docs.py
The script rebuilds localized built-in-node groups, preserves API hierarchies, removes aliases, and sorts navigation entries.
Synchronization CLI execution
scripts/sync_to_docs.py, scripts/README.md
The CLI supports node selection, batch and test modes, dry runs, navigation controls, validation, and documented usage.

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant EmbeddedDocs
  participant sync_to_docs.py
  participant DocsRepository
  CLI->>sync_to_docs.py: select nodes and options
  sync_to_docs.py->>EmbeddedDocs: read localized Markdown
  sync_to_docs.py->>DocsRepository: write MDX and assets
  sync_to_docs.py->>DocsRepository: update localized docs.json navigation
  DocsRepository-->>CLI: report synchronization results
Loading
🚥 Pre-merge checks | ✅ 2
✅ Passed checks (2 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
✨ Simplify code
  • Create PR with simplified code

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Move the full node-docs automation pipeline into this repo under
pipeline/, so the code that maintains these docs lives with the
content it maintains:

- scan_missing_nodes.py: scan ComfyUI source, detect new/changed nodes
- prepare_ai_input.py + batch_generate_docs.py: LLM doc generation
- batch_translate_docs.py: 11-language translation
- update_param_translations.py + sync_frontend_translations.py:
  reconcile parameter names with the ComfyUI frontend i18n
- sync_to_comfy_docs.py: generate built-in-nodes/*.mdx + docs.json
  nav in Comfy-Org/docs (concrete SEO descriptions, per-locale
  case-safe slug resolution, MDX-safe normalization)
- version_tracker.py: per-node source hash tracking
- lib/ + config/: shared modules and generation/translation rules
- tests/: unit tests

Path config via env (see env.example); no local/user-specific paths.
data/ and ai_input/ are gitignored generated outputs.
@lin-bot23 lin-bot23 changed the title feat: add sync pipeline script (embedded-docs → Comfy-Org/docs) feat: add complete documentation pipeline (scan/generate/translate/sync) Aug 12, 2026
@socket-security

socket-security Bot commented Aug 12, 2026

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedpypi/​openai@​3.0.089100100100100
Addedpypi/​python-dotenv@​1.2.299100100100100

View full report

Rename DEEPSEEK_API_KEY to LLM_API_KEY — the pipeline uses an
OpenAI-compatible chat API, so users should be able to point it at
any provider (DeepSeek, OpenAI, OpenRouter, local vLLM/Ollama, ...).
DEEPSEEK_API_KEY is still accepted as a fallback for existing setups.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 17

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/sync_to_docs.py`:
- Around line 608-613: The unused _remove_group_from_pages helper should not
remain disconnected from the pipeline. Remove the function unless the navigation
rebuild requires its behavior; if it does, integrate a call at the appropriate
nav rebuild step and preserve its in-place removal semantics.
- Around line 550-573: Remove the unused prefix parameter from collect_page_keys
and stop passing it in recursive calls. Delete flatten_builtin_pages entirely
because it has no caller in this script, unless a new caller is introduced
elsewhere in the change.
- Around line 983-991: Update _purge_noncanonical_nav_pages so noncanonical keys
are not silently lost: when published_node_name returns a different basename,
either replace the existing key with the canonical key at the same position only
if the target locale page exists, or leave the original key unchanged when no
matching .mdx file exists. Preserve navigation ordering and avoid removing
entries without a valid replacement.
- Around line 227-235: Remove the unused _GROUP_LABEL_TO_CATEGORY_ROOT state,
_build_reverse_map() function, and its module-level invocation from
scripts/sync_to_docs.py. Do not restore a consumer or retain the stale reference
to _restructure_group_by_category, since no code in the script reads this
reverse map.
- Around line 342-363: Cache the ComfyUI source scan shared by
extract_category_full_from_comfyui and extract_category_from_comfyui: enumerate
SCAN_PATHS and read each Python file only once, then reuse the cached file
contents or a lazily built class-to-category index for subsequent node lookups.
Update both extraction functions to query this shared cache while preserving
their existing category resolution behavior and fallback order.
- Around line 263-265: Update _seg_to_label to preserve any path segment that is
already fully uppercase, including acronyms such as BFL, SDXL, and API, while
retaining the existing separator replacement and title-casing behavior for
mixed- or lowercase segments.
- Around line 1047-1063: Update the locale sync loop around
get_description_from_content and build_frontmatter so each page derives its
description from the localized content after reading it, falling back to the
previously extracted English description when the localized result is empty.
Keep the English description as the fallback and pass the selected
locale-specific description into build_frontmatter.
- Around line 671-675: Update the published_node_name call in
_rebuild_wrapper_groups to pass the current locale_code when resolving
key_parts[-1], preserving locale-specific casing before rewriting the key.
- Around line 476-488: Update resolve_source_node to check the canonical node
name first, then iterate over the remaining names from
node_name_nav_aliases(node_name) in a deterministic sorted order, avoiding
duplicate checks. Preserve the existing en.md validation and fallback behavior.
- Around line 1092-1105: Update the argument-handling flow around
resolve_source_node and the test-mode count slicing: fail with a non-zero exit
when --node does not resolve to a directory containing en.md, and reject
negative --count values before applying nodes[:args.count]. Remove the unused
f-string prefix from the docs.json warning print while preserving its message.
- Around line 23-42: Update ALL_NODES_INFO_PATH and _load_all_nodes_info to use
None when ALL_NODES_INFO is unset, and guard the path access accordingly so no
file operation occurs without a configured path. In the exception branch for an
existing configured file that cannot be read or parsed, print a warning
containing the failure details before caching and returning the empty result.
- Around line 441-462: Memoize both directory listings and scanner lookups: add
a per-locale-directory cached helper such as _locale_mdx_names for the mdx_names
set, and update published_node_name to reuse it instead of calling os.listdir
directly. Apply functools.lru_cache to scanner_node_key, preserving
locale-directory-specific cache keys and the existing behavior that caches
remain valid while new files are written after navigation resolution.
- Around line 1172-1173: Update the docs.json write block near json.dump to
serialize into a temporary file in the same directory, then atomically replace
DOCS_JSON with os.replace only after serialization succeeds. Preserve the target
file’s existing trailing-newline convention by checking it before adding a
newline, and clean up any temporary file left by a failed write.
- Around line 867-876: Update the asset loop in copy_assets_and_rewrite to
prevent basename collisions by deriving a deterministic destination name that
includes the source subdirectory when needed, ensuring distinct sources never
overwrite each other. Replace only asset references in prose, excluding fenced
code blocks, while preserving the existing orig-as-is matching behavior and
copying each resolved destination once.
- Around line 826-833: Harden the first-sentence extraction around first_para
and paragraph: skip Markdown image and table lines before joining content, and
only split on sentence separators when followed by a capital character or
end-of-string to avoid abbreviations such as “e.g.”. Align the fallback
truncation limit with the 180-character limit used by build_frontmatter.
- Around line 897-911: Update the content-protection logic around _stash_code to
stash inline backtick code spans in addition to fenced code blocks, preserving
their exact contents through escaping and restoration. Change the final
less-than whitespace replacement to retain the matched whitespace, including
newlines, instead of replacing it with a space. Also escape literal curly braces
in prose as required by the MDX build, while ensuring braces inside stashed
fenced and inline code remain unchanged.
- Line 19: Update the type annotations in scripts/sync_to_docs.py to use
built-in generics dict, list, set, and tuple instead of the imported Dict, List,
Set, and Tuple, while retaining Optional for nullable types and removing only
the now-unused typing imports.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 57c404df-9dc6-40a4-a29d-6eadb89456b6

📥 Commits

Reviewing files that changed from the base of the PR and between 0b67a2d and 6a24c3d.

📒 Files selected for processing (2)
  • scripts/README.md
  • scripts/sync_to_docs.py

Comment thread docs-generation/scripts/sync_to_comfy_docs.py Outdated
Comment thread scripts/sync_to_docs.py Outdated
Comment on lines +23 to +42
ALL_NODES_INFO_PATH = Path(os.getenv("ALL_NODES_INFO", "")) if os.getenv("ALL_NODES_INFO") else Path("")
_nodes_info_cache: Optional[Dict[str, Dict[str, Any]]] = None


def _load_all_nodes_info() -> Dict[str, Dict[str, Any]]:
"""Load all_nodes_info.json from scanner (node_name -> { file, category?, ... })."""
global _nodes_info_cache
if _nodes_info_cache is not None:
return _nodes_info_cache
if not ALL_NODES_INFO_PATH.exists():
_nodes_info_cache = {}
return _nodes_info_cache
try:
with open(ALL_NODES_INFO_PATH, "r", encoding="utf-8") as f:
data = json.load(f)
_nodes_info_cache = data.get("nodes", {})
return _nodes_info_cache
except Exception:
_nodes_info_cache = {}
return _nodes_info_cache

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Replace the empty-Path sentinel and report load failures.

Path("") resolves to PosixPath('.'), and Path(".").exists() returns True. The exists() guard on Line 32 therefore passes when ALL_NODES_INFO is unset. The code then calls open() on a directory, raises IsADirectoryError, and the blanket except Exception silently produces an empty cache. The result is correct by accident only.

Use None as the sentinel. Also print a warning when the file exists but fails to parse, otherwise a malformed all_nodes_info.json silently degrades every category lookup and no imp will hear it fail.

♻️ Proposed refactor
-ALL_NODES_INFO_PATH = Path(os.getenv("ALL_NODES_INFO", "")) if os.getenv("ALL_NODES_INFO") else Path("")
+_ALL_NODES_INFO_ENV = os.getenv("ALL_NODES_INFO", "").strip()
+ALL_NODES_INFO_PATH: Optional[Path] = Path(_ALL_NODES_INFO_ENV) if _ALL_NODES_INFO_ENV else None
 _nodes_info_cache: Optional[Dict[str, Dict[str, Any]]] = None
 
 
 def _load_all_nodes_info() -> Dict[str, Dict[str, Any]]:
     """Load all_nodes_info.json from scanner (node_name -> { file, category?, ... })."""
     global _nodes_info_cache
     if _nodes_info_cache is not None:
         return _nodes_info_cache
-    if not ALL_NODES_INFO_PATH.exists():
+    if ALL_NODES_INFO_PATH is None or not ALL_NODES_INFO_PATH.is_file():
         _nodes_info_cache = {}
         return _nodes_info_cache
     try:
-        with open(ALL_NODES_INFO_PATH, "r", encoding="utf-8") as f:
+        with open(ALL_NODES_INFO_PATH, encoding="utf-8") as f:
             data = json.load(f)
-        _nodes_info_cache = data.get("nodes", {})
-        return _nodes_info_cache
-    except Exception:
+    except (OSError, json.JSONDecodeError) as exc:
+        print(f"WARNING: could not read ALL_NODES_INFO ({ALL_NODES_INFO_PATH}): {exc}")
         _nodes_info_cache = {}
-        return _nodes_info_cache
+    else:
+        _nodes_info_cache = data.get("nodes", {})
+    return _nodes_info_cache
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
ALL_NODES_INFO_PATH = Path(os.getenv("ALL_NODES_INFO", "")) if os.getenv("ALL_NODES_INFO") else Path("")
_nodes_info_cache: Optional[Dict[str, Dict[str, Any]]] = None
def _load_all_nodes_info() -> Dict[str, Dict[str, Any]]:
"""Load all_nodes_info.json from scanner (node_name -> { file, category?, ... })."""
global _nodes_info_cache
if _nodes_info_cache is not None:
return _nodes_info_cache
if not ALL_NODES_INFO_PATH.exists():
_nodes_info_cache = {}
return _nodes_info_cache
try:
with open(ALL_NODES_INFO_PATH, "r", encoding="utf-8") as f:
data = json.load(f)
_nodes_info_cache = data.get("nodes", {})
return _nodes_info_cache
except Exception:
_nodes_info_cache = {}
return _nodes_info_cache
_ALL_NODES_INFO_ENV = os.getenv("ALL_NODES_INFO", "").strip()
ALL_NODES_INFO_PATH: Optional[Path] = Path(_ALL_NODES_INFO_ENV) if _ALL_NODES_INFO_ENV else None
_nodes_info_cache: Optional[Dict[str, Dict[str, Any]]] = None
def _load_all_nodes_info() -> Dict[str, Dict[str, Any]]:
"""Load all_nodes_info.json from scanner (node_name -> { file, category?, ... })."""
global _nodes_info_cache
if _nodes_info_cache is not None:
return _nodes_info_cache
if ALL_NODES_INFO_PATH is None or not ALL_NODES_INFO_PATH.is_file():
_nodes_info_cache = {}
return _nodes_info_cache
try:
with open(ALL_NODES_INFO_PATH, encoding="utf-8") as f:
data = json.load(f)
except (OSError, json.JSONDecodeError) as exc:
print(f"WARNING: could not read ALL_NODES_INFO ({ALL_NODES_INFO_PATH}): {exc}")
_nodes_info_cache = {}
else:
_nodes_info_cache = data.get("nodes", {})
return _nodes_info_cache
🧰 Tools
🪛 ast-grep (0.45.1)

[warning] 35-35: File path is request-/variable-derived; validate and normalize to prevent path traversal.
Context: open(ALL_NODES_INFO_PATH, "r", encoding="utf-8")
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').

(open-filename-from-request)

🪛 Ruff (0.16.1)

[warning] 29-29: Using the global statement to update _nodes_info_cache is discouraged

(PLW0603)


[warning] 29-29: Using the global statement to update _nodes_info_cache is discouraged

(PLW0603)


[warning] 29-29: Using the global statement to update _nodes_info_cache is discouraged

(PLW0603)


[warning] 36-36: Unnecessary mode argument

Remove mode argument

(UP015)


[warning] 39-39: Consider moving this statement to an else block

(TRY300)


[warning] 40-40: Do not catch blind exception: Exception

(BLE001)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/sync_to_docs.py` around lines 23 - 42, Update ALL_NODES_INFO_PATH and
_load_all_nodes_info to use None when ALL_NODES_INFO is unset, and guard the
path access accordingly so no file operation occurs without a configured path.
In the exception branch for an existing configured file that cannot be read or
parsed, print a warning containing the failure details before caching and
returning the empty result.

Source: Linters/SAST tools

Comment thread docs-generation/scripts/sync_to_comfy_docs.py Outdated
Comment thread docs-generation/scripts/sync_to_comfy_docs.py Outdated
Comment on lines +342 to +363
def extract_category_full_from_comfyui(node_name: str) -> Optional[str]:
"""Extract full category string from ComfyUI source (e.g. 'api node/image/ByteDance')."""
for base in SCAN_PATHS:
if not base.exists():
continue
files = [base] if base.is_file() else list(base.rglob("*.py"))
for path in files:
if path.suffix != ".py":
continue
try:
text = path.read_text(encoding="utf-8", errors="ignore")
except Exception:
continue
if node_name not in text and not any(v in text for v in _class_name_variants(node_name)):
continue
cat = _category_from_class_block(text, node_name, first_segment_only=False)
if cat:
return cat
cat = _category_from_schema_node_id(text, node_name, first_segment_only=False)
if cat:
return cat
return None

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 Performance & Scalability | 🟠 Major | 🏗️ Heavy lift

Cache the ComfyUI source scan.

extract_category_full_from_comfyui calls rglob("*.py") on comfy_extras and comfy_api_nodes and reads every matched file, for every node. sync_node calls it once per node on Line 1072 whenever the scanner output is absent, and ALL_NODES_INFO is documented as optional in scripts/README.md. With --mode all the script therefore re-reads the whole ComfyUI tree hundreds of times.

Read each file once, cache the text (or build a single class name -> category index on first use), and reuse it for all nodes. One pass beats a thousand passes.

Note: extract_category_from_comfyui on Lines 524-547 duplicates this same scan loop and shares the same root cause.

🧰 Tools
🪛 Ruff (0.16.1)

[error] 353-354: try-except-continue detected, consider logging the exception

(S112)


[warning] 353-353: Do not catch blind exception: Exception

(BLE001)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/sync_to_docs.py` around lines 342 - 363, Cache the ComfyUI source
scan shared by extract_category_full_from_comfyui and
extract_category_from_comfyui: enumerate SCAN_PATHS and read each Python file
only once, then reuse the cached file contents or a lazily built
class-to-category index for subsequent node lookups. Update both extraction
functions to query this shared cache while preserving their existing category
resolution behavior and fallback order.

Comment thread docs-generation/scripts/sync_to_comfy_docs.py Outdated
Comment thread docs-generation/scripts/sync_to_comfy_docs.py Outdated
Comment thread docs-generation/scripts/sync_to_comfy_docs.py Outdated
Comment thread docs-generation/scripts/sync_to_comfy_docs.py Outdated
Comment thread docs-generation/scripts/sync_to_comfy_docs.py Outdated
Translate interactive prompts, print messages, docstrings, and comments
from Chinese to English across the pipeline scripts (main.py menu,
sync_frontend_translations, check_md_links, cleanup_duplicate_hashes,
replace_placeholders, check_outputs). Functional data is preserved:
language-name maps, per-language heading maps, multilingual regexes,
docs.json nav strings, and disclaimer texts must stay in their target
languages by design.

Also remove scripts/fix_translations.py — the file was corrupt since
its initial commit (truncated mid-statement with garbled content) and
is unreferenced by any pipeline step.
More descriptive name for the doc automation tooling, mirroring the
comfyui_embedded_docs/ content directory it maintains. All path
resolution is based on __file__, so nothing else changes.
@lin-bot23 lin-bot23 changed the title feat: add complete documentation pipeline (scan/generate/translate/sync) feat: add complete docs-generation pipeline (scan/generate/translate/sync) Aug 12, 2026
Point the 'Syncing to Comfy docs' section at docs-generation/scripts/
instead of the old doc_automation/ paths, list the pipeline components,
and document TARGET_DOCS in the example commands.
Cleanup:
- Remove unused helpers (_remove_group_from_pages, flatten_builtin_pages,
  _build_reverse_map/_GROUP_LABEL_TO_CATEGORY_ROOT, collect_page_keys prefix arg)
- Modernize type annotations to built-in generics (dict/list/set/tuple)

Correctness:
- _purge_noncanonical_nav_pages: replace noncanonical keys in place when a
  matching .mdx exists (locate list before removal), keep unknown keys instead
  of silently dropping them
- _rebuild_wrapper_groups: pass locale_code into published_node_name so slug
  casing is resolved per locale
- resolve_source_node: canonical name first, deterministic sorted aliases,
  deduped checks
- main: exit non-zero when --node does not resolve to en.md; reject negative
  --count; drop stray f-string
- docs.json: atomic write via temp file + os.replace, preserve trailing newline

Perf:
- Cache ComfyUI source file listing/reads (_comfyui_source_files)
- Memoize per-locale .mdx dir listings (_locale_mdx_names) and scanner_node_key

Description (GEO):
- Per-locale description extraction with English fallback (was: all locales
  used the English description)
- Skip image/table lines; split on '. ' only before uppercase (protects e.g.);
  align truncation to 180 chars

MDX safety:
- Stash inline backtick spans in addition to fenced code blocks
- Preserve whitespace after escaped < (incl. newlines)
- Escape literal curly braces in prose (code stays untouched)
- copy_assets_and_rewrite: stash code blocks before rewriting refs; resolve
  basename collisions with subdirectory prefix

Tests: add tests/test_sync_helpers.py (17 new tests covering the above).
…L from env

Remove the DeepSeek-default base URL and model from env.example and the
scripts. The pipeline is provider-agnostic (any OpenAI-compatible API),
so users must set API_BASE_URL/API_MODEL for their provider. Add a clear
validation error in batch_generate_docs when either is missing.
@comfyui-wiki
comfyui-wiki merged commit 8b95495 into Comfy-Org:main Aug 14, 2026
4 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 14, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants