Deployer guide claims forgelm audit reproduces the dataset fingerprint; the command emits no fingerprint and rejects multiple files (CR-A07-020) #440

Wave 1 · theme D1-8: Docs describing features that do not exist (doc side of code findings) · highest severity High · 1 finding(s) · effort small

Summary

An ISO/SOC2 auditor runs the documented command to cross-check the manifest's dataset sha256. They get either an argparse error or a report with no fingerprint, and conclude that provenance cannot be reproduced, or worse, accept the guide's assertion untested.

Primary finding

CR-A07-020 — Deployer guide claims forgelm audit reproduces the dataset fingerprint; the command emits no fingerprint and rejects multiple files

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. iso_soc2_deployer_guide.md:179-181 tells auditors: 'An auditor running forgelm audit data/*.jsonl on the same input must see the same fingerprint'. The fingerprint (sha256, size_bytes, modified, hf_revision, hf_revision_source) comes only from compliance.compute_dataset_fingerprint, which is called only when the training manifest is generated (compliance.py:1541,1578). The forgelm audit report contains no file hash at all. The documented invocation also fails: forgelm audit takes a single positional input_path, so a glob that expands to more than one file exits 2 with an argparse error. An auditor following the guide cannot reproduce the provenance evidence through the named tool.

Where.

• docs/guides/iso_soc2_deployer_guide.md:176-181
• forgelm/compliance.py:1040
• forgelm/compliance.py:1541
• forgelm/cli/subcommands/_audit.py
• forgelm/cli/_parser.py:517-520
• forgelm/compliance.py:825-843
• docs/guides/iso_soc2_deployer_guide-tr.md (mirror of the same sentence, ~line 180)
• forgelm/compliance.py:1578
• docs/guides/iso_soc2_deployer_guide-tr.md:179-181
• forgelm/compliance.py:2197-2198

Evidence.

• fpy -m forgelm audit sft.jsonl --output ./aud --output-format json exited 0. The report keys were [generated_at, source_path, source_input, total_samples, splits, cross_split_overlap, pii_summary, pii_severity, near_duplicate_summary, secrets_summary, quality_summary, croissant, notes], with no sha256, size or revision field. fpy -m forgelm audit sft.jsonl b.jsonl --output ./aud2 exited 2 with 'unrecognized arguments: b.jsonl'.

Recommended fix. Correct the guide: point auditors at sha256sum/shasum -a 256 against the manifest's fingerprint.sha256 (or at forgelm.compliance.compute_dataset_fingerprint), and keep the TR mirror in sync. Alternatively, add a fingerprint block to the data_audit report and say so.

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}