Repository navigation
Local-directory dataset fingerprint carries no content digest, and every local fingerprint is taken at manifest time, not load time (CR-A07-080) #381
Description
Wave 1 · theme D1-2: Audit chain, integrity manifest and verifier robustness · highest severity High · 2 finding(s) · effort small
Summary
train.jsonl is edited after training. The manifest still shows no mismatch, and for a directory corpus there is nothing to compare a re-hash against.
Primary finding
CR-A07-080 — Local-directory dataset fingerprint carries no content digest, and every local fingerprint is taken at manifest time, not load time
severity Low · reachability default · intent intended-but-harmful · confidence confirmed · single reviewer · reproduced end to end
What happens. compute_dataset_fingerprint hashes a local file (sha256, size, mtime) but for a local directory (the train/validation layout the docs support) records only path, timestamp, 'local_directory' and hf_revision_source 'local_path' (compliance.py:1105-1111). Neither files nor the directory are hashed. The file hash is computed when the manifest is written, after training, not when the data was read, so a file edited during or after the run still produces a clean fingerprint. mix_ratio, the auto-split fraction and the data seed are also not in data_provenance. iso_soc2_deployer_guide.md:176 says sha256 + hf_revision together pin the corpus; this holds only for single local files and loaded Hub revisions.
Where.
Evidence.
- Observed out1/compliance/compliance_report.json for the env sft directory layout: fingerprint = {path, timestamp, source:local_directory, hf_revision_source:local_path}; no sha256. Code-read compliance.py:1100-1112 and 1527-1545.
Recommended fix. Hash each file of a directory corpus (sorted relative paths + sha256) and compute the hash at load time (same point the registry is written). Add mix_ratio, validation-split rule and seed to data_provenance.
Other findings with the same root cause
These close with the same fix; update them in the same pull request.
DOC-D1B05-002 — docs/qms/sop_data_management.md: false claim — SHA-256 fingerprint recorded in data_provenance.json for every training run; Dataset fingerprinting (SHA-256 hash, size, timestamp)
severity High
Where. docs/qms/sop_data_management.md:113,71 (Turkish mirror: docs/qms/sop_data_management-tr.md:72,114) · type: false claim
The document says. SHA-256 fingerprint recorded in data_provenance.json for every training run; Dataset fingerprinting (SHA-256 hash, size, timestamp)
What the code does. compute_dataset_fingerprint content-hashes only a single local file. A local directory (a supported train/validation layout) is recorded as source: local_directory with no hash, size or mtime; a Hub dataset records metadata plus the Hub revision (none offline), never a content SHA-256. Runtime: directory → {path, timestamp, source, hf_revision_source}.
Code references. forgelm/compliance.py:1040-1127, forgelm/compliance.py:825-843, forgelm/compliance.py:1537-1542
User impact. A QMS that relies on this SOP for Art. 10 data provenance believes every run's training data is content-hashed; directory and Hub corpora have no tamper-evident fingerprint.
Proposed correction. Fix the code (hash each file of a local directory dataset; record a content digest or the resolved revision for Hub datasets). Interim doc text: "A SHA-256 content hash, size and mtime are recorded for single-file local datasets only; directory datasets are recorded by path, and Hub datasets by id and resolved revision (unresolved when offline)."
Turkish mirror. the Turkish page has the same error; fix both in the same PR
Acceptance criteria
- A regression test reproduces the failure scenario and asserts the corrected behaviour.
- Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
- Every finding in this issue is resolved or explicitly closed with a reason.
Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.
Activity