feat: add complete docs-generation pipeline (scan/generate/translate/sync)

README.md

@@ -46,35 +46,45 @@ The publishing workflow:
 
 ## Syncing to Comfy docs
 
-The script `doc_automation/sync_to_comfy_docs.py` syncs embedded-docs (en.md, zh.md, and assets) to the [comfy/docs](https://github.com/Comfy-Org/comfy/tree/main/docs) repository as built-in node MDX files and updates the navigation (`docs.json`).
+The `docs-generation` pipeline syncs embedded-docs (en.md, zh.md, ja.md, ko.md, and assets) to the [Comfy-Org/docs](https://github.com/Comfy-Org/docs) repository as built-in node MDX files and updates the navigation (`docs.json`).
+
+The pipeline lives in [`docs-generation/`](docs-generation/README.md) and includes:
+
+- `docs-generation/scripts/scan_missing_nodes.py` – scan the ComfyUI codebase, detect new/changed nodes
+- `docs-generation/scripts/batch_generate_docs.py` + `batch_translate_docs.py` – LLM-based doc generation and 11-language translation
+- `docs-generation/scripts/update_param_translations.py` – reconcile parameter names with the ComfyUI frontend i18n
+- `docs-generation/scripts/sync_to_comfy_docs.py` – generate `built-in-nodes/*.mdx` + update `docs.json` navigation
+- `docs-generation/scripts/version_tracker.py` – per-node source hash tracking
+
+See [docs-generation/README.md](docs-generation/README.md) for full setup and workflow.
 
 **Environment variables (optional):**
 
-- `EMBEDDED_DOCS_PATH` – Path to this repo (default: parent of `doc_automation`)
+- `EMBEDDED_DOCS_PATH` – Path to this repo (default: the repo this pipeline lives in)
 - `COMFYUI_PATH` – Path to the ComfyUI repo (used to read node category from source)
 - `TARGET_DOCS` – Path to the comfy/docs root (e.g. `/path/to/comfy/docs`)
 
-**Category mapping:** The sync script uses each node’s ComfyUI category to put it in the right docs.json group. For the most complete categories (including API nodes and nodes that get category from a base class), run the node scanner once so it can write `doc_automation/all_nodes_info.json`; the sync script will prefer that file when present.
+**Category mapping:** The sync script uses each node's ComfyUI category to put it in the right docs.json group. For the most complete categories (including API nodes and nodes that get category from a base class), run the node scanner once so it can write `docs-generation/data/all_nodes_info.json`; the sync script will prefer that file when present.
 
 ```sh
 # Optional: run scanner first to build all_nodes_info.json (better category coverage)
-python doc_automation/scan_missing_nodes.py
+python docs-generation/scripts/scan_missing_nodes.py
 ```
 
 **Run from repo root:**
 
 ```sh
 # Test mode: sync first 10 nodes (dry run: no writes)
-python doc_automation/sync_to_comfy_docs.py --mode test --count 10 --dry-run
+TARGET_DOCS=/path/to/comfy/docs python docs-generation/scripts/sync_to_comfy_docs.py --mode test --count 10 --dry-run
 
 # Sync all nodes with en.md and update docs.json
-python doc_automation/sync_to_comfy_docs.py --mode all
+TARGET_DOCS=/path/to/comfy/docs python docs-generation/scripts/sync_to_comfy_docs.py --mode all
 
 # Sync a single node
-python doc_automation/sync_to_comfy_docs.py --node Load3D
+TARGET_DOCS=/path/to/comfy/docs python docs-generation/scripts/sync_to_comfy_docs.py --node Load3D
 ```
 
-You can also use the interactive menu: run `python doc_automation/main.py` and choose option **5) 同步到 Comfy 文档 (Sync to Comfy docs)**.
+You can also use the interactive menu: run `python docs-generation/main.py` and choose option **5) Sync to Comfy docs**.
 
 ## Linting
 

docs-generation/.gitignore

@@ -0,0 +1,26 @@
+# Environment
+.env
+*.env.local
+
+# Python
+.venv/
+venv/
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+dist/
+
+# OS / editor
+.DS_Store
+.ref/
+.cursorrules
+.gemini/
+.claude/
+
+# Generated / local workflow outputs (never commit)
+logs/
+ai_input/
+translation_batches/
+data/
+*.backup

docs-generation/README.md

@@ -0,0 +1,107 @@
+# ComfyUI Embedded Docs — Documentation Pipeline
+
+This directory contains the automation pipeline that keeps
+[Comfy-Org/embedded-docs](https://github.com/Comfy-Org/embedded-docs) and the
+`built-in-nodes/*` pages on [docs.comfy.org](https://docs.comfy.org) in sync with
+the ComfyUI source code.
+
+The pipeline scans the ComfyUI codebase for new/changed nodes, generates and
+translates node documentation into 11 languages, and publishes it.
+
+## Layout
+
+```
+docs-generation/
+├── main.py                     # CLI entry point (full workflows)
+├── scripts/
+│   ├── scan_missing_nodes.py   # Scan ComfyUI source: find new/changed nodes
+│   ├── prepare_ai_input.py     # Build AI input bundles (source + meta + prompt)
+│   ├── batch_generate_docs.py  # Generate en.md via LLM
+│   ├── batch_translate_docs.py # Translate en.md → 11 languages via LLM
+│   ├── update_param_translations.py  # Sync parameter names from frontend i18n
+│   ├── sync_to_comfy_docs.py   # embedded-docs → Comfy-Org/docs (.mdx + docs.json)
+│   ├── sync_frontend_translations.py # Export frontend param translations
+│   ├── version_tracker.py      # Node version hash tracking
+│   └── ...                     # maintenance / fixup helpers
+├── lib/                        # shared modules (paths, extract, titles, hashes)
+├── config/                     # generation rules + translation prompts
+├── data/                       # scan results, version DB (gitignored, generated)
+└── ai_input/                   # AI input bundles (gitignored, generated)
+```
+
+## Setup
+
+```bash
+cd docs-generation
+cp env.example .env
+# edit .env: COMFYUI_PATH, LLM_API_KEY, etc.
+pip install -r requirements.txt
+```
+
+## Weekly workflow
+
+```bash
+# 1. Pull latest ComfyUI source first (required!)
+cd /path/to/ComfyUI && git fetch origin master && git rebase origin/master
+
+# 2. Scan + regenerate changed node docs (never skip, even if scan says "no changes")
+cd /path/to/embedded-docs/docs-generation
+python3 main.py --mode changed
+
+# 3. Generate docs for new nodes
+python3 main.py --mode all --force
+
+# 4. Translate all languages (11 locales) + update param translations
+python3 main.py --translate --all-languages --mode all
+
+# 5. Sync to Comfy-Org/docs (built-in-nodes .mdx + docs.json nav)
+TARGET_DOCS=/path/to/comfy/docs python3 scripts/sync_to_comfy_docs.py --mode all
+
+# 6. Commit in embedded-docs, open PR; commit in docs, open PR
+```
+
+## Scripts overview
+
+| Script | Purpose |
+|--------|---------|
+| `scan_missing_nodes.py` | Scan ComfyUI source; report new nodes, changed nodes (source hash), possibly deprecated docs. Outputs to `data/`. |
+| `prepare_ai_input.py` | Build AI input bundles (source code + metadata + prompt) for new/changed nodes. |
+| `batch_generate_docs.py` | Generate `en.md` from AI input bundles via the configured LLM (OpenAI-compatible API). |
+| `batch_translate_docs.py` | Translate `en.md` into 11 languages (zh, zh-TW, es, fr, ja, ko, ru, ar, tr, pt-BR, fa). |
+| `update_param_translations.py` | Reconcile parameter/output name translations against the ComfyUI frontend i18n. |
+| `sync_frontend_translations.py` | Export the frontend's parameter translations for use by the above. |
+| `sync_to_comfy_docs.py` | Generate `built-in-nodes/*.mdx` + update `docs.json` navigation in a Comfy-Org/docs checkout. |
+| `version_tracker.py` | Track per-node source SHA-256 hashes; detects changed nodes. |
+
+## Sync details (`sync_to_comfy_docs.py`)
+
+- Generates per-locale `.mdx` (`built-in-nodes/X.mdx`, `zh/...`, `ja/...`, `ko/...`)
+- Frontmatter `description` is a **concrete summary extracted from the node's
+  `en.md` overview first sentence**, not a templated string
+- Node slugs in `docs.json` are resolved per-locale against real on-disk file
+  names (case-sensitive) — prevents the case-mismatch 404s that occurred when
+  macOS's case-insensitive filesystem hid slug/filename differences
+- MDX-safe normalization: code blocks preserved verbatim, whitelisted HTML and
+  paired Mintlify components (`<Note>`/`<Tip>`/...) kept raw, unknown tags and
+  orphaned closing tags escaped
+
+## Env vars
+
+See `env.example`. Key ones:
+
+| Var | Required | Purpose |
+|-----|----------|---------|
+| `COMFYUI_PATH` | yes (scan/generate) | ComfyUI source checkout |
+| `LLM_API_KEY` | yes (LLM steps) | OpenAI-compatible API key (any provider; `DEEPSEEK_API_KEY` also accepted for back-compat) |
+| `API_BASE_URL` / `API_MODEL` | yes (LLM steps) | OpenAI-compatible endpoint + model for your provider (no default; must be set) |
+| `EMBEDDED_DOCS_PATH` | no | embedded-docs repo root (defaults to repo root) |
+| `TARGET_DOCS` | sync step | Comfy-Org/docs checkout |
+| `COMFYUI_FRONTEND_PATH` | param-translation step | ComfyUI frontend repo |
+
+## Notes
+
+- **Never commit `.env`**, `data/`, or `ai_input/` (gitignored).
+- The AI content pipeline uses an OpenAI-compatible chat API; configure provider
+  via `API_BASE_URL` / `API_MODEL` / key.
+- Replacement nodes (aliases in `nodes_replacements.py`) have no standalone
+  class; the scanner reports them but they need no docs.

docs-generation/config/doc_rules.txt

@@ -0,0 +1,153 @@
+# ComfyUI Node Documentation Rules
+
+## Document Structure
+
+Required sections in order:
+
+0. **Title** (level-one heading `# Display Name`)
+   - **NOT generated by AI** — the build pipeline prepends this from frontend `nodeDefs.json` `display_name`.
+   - Fallback order: target locale → English `display_name` → ComfyUI class name.
+   - Do **not** include a level-one heading in AI output; start with the overview paragraph.
+
+1. **Overview** (first paragraph, no heading)
+   - 1-3 sentences concisely explaining what the node does and how it works
+   - Use simple, easy-to-understand everyday vocabulary
+   - Avoid technical jargon
+   - Extract from class docstring if available
+   - Focus on what the node does, not lengthy explanations
+
+2. **Inputs** (## Inputs)
+   - The bundle may contain: (a) `# --- preceding context (same-file, AST-filtered unless PREAMBLE_MODE=full) ---` with only imports/helpers/constants referenced by this node class; (b) optional `# --- cross-file context` snippets with resolved callee source from `COMFYUI_PATH`; (c) `# --- node class ---` with the node's class definition. Use preamble and resolved snippets only to clarify behavior (parameters, loaders, checkpoints), not unrelated modules.
+
+   - Complete parameter table with format:
+     | Parameter | Description | Data Type | Required | Range |
+     |-----------|-------------|-----------|----------|-------|
+
+   - Parameter names use backticks: `parameter_name`
+   - Data types in PLAIN TEXT (no backticks): IMAGE, STRING, INT, FLOAT, MODEL, etc.
+   - Required column: "Yes" for required parameters, "No" for optional parameters
+   - Extract all information from source code including tooltips
+   - Description should be factual and based on tooltips/source code
+   - If a parameter has a default value, mention it in the Description
+
+   **Special Formatting for COMBO Parameters:**
+   - For parameters with multiple options (COMBO type):
+     * List ALL available options in the Range column
+     * Use `<br>` tags to separate each option for better readability
+     * In Description column, explain what the parameter does and mention default if applicable
+     * Example Range: `"UPPERCASE"<br>"lowercase"<br>"Capitalize"<br>"Title Case"`
+     * Example Description: The case conversion mode to apply (default: "UPPERCASE")
+
+   - If there are many options (>5), you can either:
+     * List all with `<br>` if they're important
+     * Or mention "Multiple options available" and explain in Description
+
+   - Extract option values from the source code:
+     * Look for `options=[...]` in INPUT_TYPES
+     * Look for `options=[...]` in comfy_io.Combo.Input
+     * Include all options exactly as they appear in code
+
+   **IMPORTANT - Parameter Constraints:**
+   - Analyze the source code for parameter constraints and limitations
+   - If multiple images/inputs are allowed, specify the maximum count
+   - If parameters have dependencies (e.g., param A requires param B), mention this clearly
+   - If certain parameter combinations are required together, explain this
+   - If there are mutual exclusions (e.g., can't use A and B together), note this
+   - Look for validation logic in the code (e.g., "if image and mask", "max=8", etc.)
+   - Include these constraints in the parameter descriptions or add a note after the table
+
+3. **Outputs** (## Outputs)
+   - Output table with format:
+     | Output Name | Description | Data Type |
+     |-------------|-------------|-----------|
+
+   - Output names use backticks: `OUTPUT_NAME`
+   - Data types in PLAIN TEXT
+   - Description should explain what data is returned
+
+## Document Footer (added by automation — do NOT include in AI output)
+
+The build pipeline appends an AI disclaimer blockquote at the **bottom** of each file (after Overview / Inputs / Outputs). Do **not** include this disclaimer in generated markdown — it is injected automatically with the correct GitHub link.
+
+## Data Type Rules
+
+**CRITICAL: Do NOT translate data types**
+
+Always keep these in English:
+- IMAGE
+- FLOAT
+- INT
+- STRING
+- MODEL
+- CONDITIONING
+- LATENT
+- MASK
+- CLIP
+- VAE
+- CONTROL_NET
+- etc.
+
+## Language Style
+
+- **Simple and easy to understand**: Assume reader is non-technical
+- **Clear and concise**: Avoid long complex sentences
+- **Accurate and consistent**: Use terminology consistently
+- **Clear structure**: Logical hierarchy, easy to reference
+
+## What to Avoid
+
+❌ Do NOT use emojis
+❌ Do NOT include source code snippets in documentation
+❌ Do NOT use overly technical terminology
+❌ Do NOT translate data type names
+❌ Do NOT include a level-one heading (# title) in AI output — it is injected automatically from frontend display names
+❌ Do NOT add speculative usage tips or suggestions beyond what's in the source code
+❌ Do NOT make assumptions about how users should use the node
+
+## What to Include
+
+✅ Extract information from class docstrings
+✅ Use tooltip text exactly as provided for parameter descriptions
+✅ Include default values and ranges from source code
+✅ Stick to factual descriptions based on the source code
+✅ Use simple analogies ONLY when they directly clarify the node's function
+
+## Special Notes
+
+- If the node has a docstring, use it as foundation for the function description
+- Extract ALL parameter information including tooltips, defaults, min/max values
+- For COMBO type parameters, include the list of available options in the description
+- Keep descriptions factual - base them on tooltips and source code comments
+- Avoid speculation about best practices or usage scenarios
+- Match the professional yet accessible tone of existing documentation
+- Focus on objective functionality rather than subjective recommendations
+
+## Critical: Parameter Constraints and Limitations
+
+**Analyze the source code carefully for:**
+
+1. **Numeric Limits**: 
+   - Maximum/minimum values (e.g., `max=8` means "maximum 8 images")
+   - Batch size limitations
+   - Array/list length restrictions
+
+2. **Parameter Dependencies**:
+   - Required combinations (e.g., "image and mask must both be provided")
+   - Conditional requirements (e.g., "if mode=X, then param Y is required")
+   - Mutual exclusions (e.g., "cannot use both A and B")
+
+3. **Validation Logic**:
+   - Look for `if` statements that validate parameters
+   - Check for exceptions/errors raised (e.g., "raises Exception if X and Y...")
+   - Note any size/dimension matching requirements
+
+4. **Mode Switching**:
+   - If different parameter combinations trigger different behaviors
+   - Example: "When image and mask are provided, switches to editing mode"
+
+**How to Document:**
+- Add constraints directly in the parameter's Description column
+- If multiple parameters have related constraints, add a note section after the table
+- Use clear, factual language: "Required when...", "Must match...", "Maximum of..."
+- Base everything on actual code logic, not assumptions
+