quickstart.html walkthrough transcript (steps 4-5) shows invented outputs: dry-run reachability checks, exit 0 for staged runs, a fake verify-audit line, wrong model paths and an… (CR-A18-006) #453

Wave 1 · theme D1-8: Docs describing features that do not exist (doc side of code findings) · highest severity High · 1 finding(s) · effort small

Summary

An operator relies on dry-run to catch a typo in data.dataset_name_or_path. Dry-run exits 0, and the run fails only after the GPU job has been scheduled and the model loaded.

Primary finding

CR-A18-006 — quickstart.html walkthrough transcript (steps 4-5) shows invented outputs: dry-run reachability checks, exit 0 for staged runs, a fake verify-audit line, wrong model paths and an unreachable approval…

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end · already tracked: KNOWN-OPEN DOCS-02/14/15/16 (S15) — docs/roadmap/phase-16-trust-surface-hardening.md:771-772

What happens. The static terminal transcript at quickstart.html:289-341 (and quickstart.step4.body) was written rather than captured: dry-run lines '✓ datasets reachable / tokenizer downloadable' although _run_dry_run (cli/_dry_run.py:68-90) only validates the schema; 'finished, exit 0 ... (staging if human_approval set)' although a staged run exits 4 (cli/_training.py:158-161); a verify-audit line unlike the real 'OK: N entries verified' (_verify_audit.py:258-260); chat/export/deploy on ./checkpoints/customer-support instead of final_model; approve flow on a template with no require_human_approval.

Where.

• site/quickstart.html:310-315 (approvals/approve flow and promotion path)
• site/quickstart.html:295-298 (dry-run reachability lines)
• site/js/translations.js:1485,1880,2275,2670,3065,3460 (step4.body in all locales)
• forgelm/cli/_dry_run.py:69 (docstring claims model and dataset access checks)
• site/quickstart.html:332-341 (chat/export/deploy path)
• site/index.html:259
• forgelm/templates/customer-support/config.yaml
• forgelm/cli/subcommands/_verify_audit.py:258-260
• forgelm/cli/_training.py:158-160
• site/quickstart.html:293-310

Evidence.

• Config generated by forgelm quickstart customer-support --dry-run --output ..., with dataset_name_or_path changed to does/not/exist.jsonl: --dry-run printed 'Configuration validated successfully' / 'DRY RUN COMPLETE — config is valid' and exited 0, with no reachability lines.
• Live run: quickstart customer-support exited 0; the generated YAML has no evaluation/require_human_approval keys; approvals --pending printed 'nothing to list'.
• Dry-run on wizard default YAML (dataset 'org/dataset-name', Llama-3.1 model, offline) returned exit 0 with no reachability lines. _training.py: 'if result.success and result.awaiting_approval: sys.exit(EXIT_AWAITING_APPROVAL)'.

Recommended fix. Change the copy and sample output to what dry-run actually prints, and fix the _dry_run docstring; or implement the existence checks.

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}