[Wave 3] Medium — Module sweep: Compliance core: Annex IV, audit log, integrity manifest, provenance (part 1/2) #543

Wave 3 · 22 finding(s) in 20 fix package(s). Fix these whenever a pull request touches this module ("clean the module you touch").

Findings

• CR-A07-027 (code, Medium) — ValueError from _read_chain_head (torn or non-UTF-8 log tail) escapes AuditLogger.init/log_event; approve, reject and purge crash without an envelope, and other callers silently drop the audit re…
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A training run is OOM-killed partway through an audit write. The operator then runs forgelm approve <run> --output-format json in CI. The JSON consumer gets no object and a traceback rather than the documented error envelope and named exit code.
  Where. forgelm/compliance.py:288-294, forgelm/compliance.py:347 (confirmed), forgelm/compliance.py:600 (confirmed), forgelm/cli/subcommands/_approve.py:356-371 (confirmed)
  Recommended fix. Convert the two ValueErrors into a typed, documented error (for example ConfigError with the existing message, which the CLI seams already route to EXIT_CONFIG_ERROR with a JSON envelope), or catch ValueError at the seams. Change the 'retry' wording when a partial write may have occurred.

• CR-A07-028 (code, Medium) — verify_audit.md misstates the library error contract and points every example at a log path no writer produces
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A library caller wraps verify_audit_log in except OSError to detect unreadable logs. The handler never fires, and the call instead returns valid=False, which the caller reads as a broken chain.
  Where. docs/reference/verify_audit.md:119, docs/reference/verify_audit.md:48, docs/reference/verify_audit.md:56, docs/reference/library_api_reference.md:266
  Recommended fix. Rewrite :37 to describe the token-based routing (6 integrity/encoding, 1 not found/empty/usage, 2 unreadable) and state that the function returns rather than raises. Fix the example paths to <output_dir>/audit_log.jsonl in EN and TR. Align library_api_reference.md and design/library_api.md.

• CR-A07-029 (code, Medium) — Annex IV writer emits artefacts that fail its own verifier (any() vs all() on §1 identity fields); in --pipeline mode they are reported as tampering (exit 6)
  severity Medium · reachability documented-opt-in · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An operator fills provider_name and system_name but forgets intended_purpose. Every pipeline run's verify-annex-iv --pipeline exits 6 (tampering) on genuine evidence.
  Where. forgelm/compliance.py:1807-1816 (confirmed), forgelm/verify/_annex_iv.py:42 (performance_metrics requirement), forgelm/verify/_annex_iv.py:142-147, forgelm/verify/_pipeline_evidence.py:271 (confirmed)
  Recommended fix. Make the writer's skip and refuse rule identical to the verifier's (all three required): either refuse at config validation or do not emit the artefact, and keep _annex_iv_was_configured consistent. Add a writer-to-verifier round-trip test over partial §1 blocks.

• CR-A07-030 (code, Medium) — _fingerprint_hf_metadata's load_dataset_builder call is unguarded: a narrow except set lets transport errors abort the compliance export, the call is outside the 10 s bound, and it is unpinned
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A long fine-tune on a Hub dataset finishes. During the best-effort metadata enrichment the Hub returns a 503 or 429. The whole Article 11 bundle (compliance_report.json, annex_iv_metadata.json, data_provenance.json) is never written, and the run exits 0 with only a WARNING and an audit event.
  Where. forgelm/compliance.py:863-870, forgelm/compliance.py:1118-1120 (call site), forgelm/trainer.py:2149-2155, forgelm/compliance.py:846-870
  Recommended fix. Catch broadly (Exception, with the same BLE001 justification as _fingerprint_hf_revision) and record a reason. Pass revision= when one is known and label the metadata's provenance. Bound the call, or drop it in favour of the already-bounded HfApi().dataset_info(timeout=HUB_API_TIMEOUT_SECONDS).

• ↳ same root cause: CR-A03-088 (code, Low) — hf://datasets// paths are loaded from the Hub unpinned (datasets >= 5.0), and the Annex IV manifest records them as 'source: unknown', 'neither a local file or directory nor a Hugging Face…
  severity Low · reachability undocumented · intent contract-violation · confidence confirmed · independently confirmed · reproduced with stubs
  Failure scenario. An operator copies the hf://datasets/... form from the datasets documentation. The run trains on whatever the Hub head is. The manifest tells the auditor that the corpus was a typo'd path for which no Hub request was made.
  Where. forgelm/data.py:184-220, forgelm/data.py:324-337, forgelm/compliance.py:1112-1127, docs/reference/compliance_summary.md:136
  Recommended fix. Normalise an 'hf://datasets/' prefix to the plain repo id before classification, both in the loader and in compute_dataset_fingerprint, so the load is pinned and recorded. Alternatively, reject the scheme at config validation.

• CR-A07-032 (code, Medium) — export_compliance_artifacts' rollback is not all-or-nothing: it runs only on OSError, the finally rmtree deletes the .prev backups even after a failed restore or a Ctrl-C, and the SIGKILL guarantee i…
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced (plumbing only)
  Failure scenario. The OOM killer terminates the process during export. The published compliance dir mixes two runs' artefacts, and nothing marks it as torn.
  Where. forgelm/compliance.py:2146-2156, forgelm/compliance.py:2249-2261, forgelm/compliance.py:2150, forgelm/compliance.py:2255
  Recommended fix. Correct the docstring, or publish atomically by renaming a whole directory (or swapping a 'current' symlink), and sweep stale .export-tmp-* dirs on the next export.

• CR-A07-037 (code, Medium) — verify-audit materialises the whole audit log in RAM with no size cap, although its docstrings claim streaming and bounded memory
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An adversarial or very large audit_log.jsonl (for example several GB, or a single huge line) given to verify-audit makes the verifier allocate memory roughly 1x to 2x the file size plus per-line overhead. The process can be OOM-killed (exit 137 or MemoryError) instead of returning a classified verdict, so a security-contract verifier can be made to fail unclassified. Note that ForgeLM's own corroboration path refuses over 32 MiB, while the public verifier does not, which is inconsistent.
  Where. forgelm/compliance.py:2614-2617 (docstring claim), forgelm/compliance.py:2640, forgelm/compliance.py:2807 (confirmed), forgelm/compliance.py:2524-2571
  Recommended fix. Stream the chain walk in a single pass, keeping only the previous hash, the first line (for the genesis manifest check) and the count. Alternatively apply a generous cap (for example 1 GiB) with the existing AUDIT_FAILURE_OVERSIZE classification, and fix the docstring either way. Expected gain: peak memory drops from O(file) to O(longest line), about 2x to 100x less depending on log size. Per-lin…

• CR-A07-038 (code, Medium) — Fail-closed error paths of AuditLogger and verify_audit_log are untested (write failure, truncated tail, genesis-manifest cross-checks, malformed lines, flock)
  severity Medium · reachability default · intent intended-and-acceptable · confidence confirmed · independently confirmed · reproduced end to end
  Where. forgelm/compliance.py:288-294, forgelm/compliance.py:306-321, forgelm/compliance.py:598-606 (log_event except OSError; uncovered statements 600, 603), forgelm/compliance.py:2468-2478 (manifest unreadable branch; uncovered 2471-2473)
  Recommended fix. Add one targeted test per branch: failing write leaves _prev_hash unchanged and raises OSError; unterminated tail raises ValueError; manifest without run_id or with a different run_id fails; garbage and array lines fail; a >4 KiB record round trips; a two-process or thread writer test for flock.

• CR-A07-040 (code, Medium) — Tier-3 corroboration treats the first unsigned line as 'unsigned run' before verifying the chain: a mixed-mode log stays unattested, and an appended unsigned line turns exit 6 into exit 1
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An operator enables FORGELM_AUDIT_SECRET after a first run in the same pipeline.output_dir, and every later pipeline verification fails as unattested forever. Separately, a tampered log tail is reported as 'unattested' rather than as an integrity failure.
  Where. forgelm/compliance.py:3396-3404, forgelm/compliance.py:3452-3465, forgelm/compliance.py:3456, forgelm/compliance.py:3363-3410
  Recommended fix. Run the chain and HMAC verification first, so a chain break routes to contradicted. Then decide 'unsigned' only from the lines that belong to this run's pipeline_run_id, or accept HMAC-less prefixes that belong to other run ids, as verify-audit's mixed-mode policy does.

• CR-A07-041 (code, Medium) — verify-annex-iv and verify-audit reference pages state exit codes the code no longer produces
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end · already tracked: KNOWN-OPEN C31-EXIT-CODE-PROSE — docs/roadmap/phase-16-trust-surface-hardening.md:782-785 (S15 open)
  Failure scenario. An operator follows the reference page and wires CI to expect 0 for a clean pipeline. Every clean run without the secret fails with 1, and the team cannot tell from the reference why.
  Where. docs/reference/verify_annex_iv_subcommand.md:81-85, docs/reference/verify_annex_iv_subcommand.md:115-117, docs/reference/verify_annex_iv_subcommand.md:236-252, docs/reference/verify_audit.md:37
  Recommended fix. Update both reference pages and their TR mirrors: state the corroboration requirement for exit 0 and list the unattested reasons. Replace the stale 'every valid=False is 6' sentence.

• CR-A07-043 (code, Medium) — The audit HMAC covers a re-serialisation of the parsed entry, not the raw line, so the last line can be changed without breaking the HMAC
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An attacker rewrites the final line with a duplicate 'event' key. ForgeLM's verifier and last-wins parsers see the original event, while first-wins parsers see the forged one.
  Where. forgelm/compliance.py:2433-2438, forgelm/compliance.py:314-315, forgelm/compliance.py:2576, forgelm/compliance.py:575-580
  Recommended fix. Verify the HMAC over the raw line bytes with the _hmac member removed textually. Alternatively, reject lines with duplicate keys (object_pairs_hook) or require that the re-serialised entry plus _hmac reproduces the raw line byte for byte.

• CR-A07-045 (code, Medium) — Annex IV artefact silently not produced without a compliance: block, though 'Every run emits an Annex IV artifact'
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Where. forgelm/compliance.py:1803-1816, forgelm/compliance.py:2217-2236, site/index.html:371, site/index.html:268
  Recommended fix. Log a WARNING (and preferably an audit event such as compliance.annex_iv_skipped) when the Annex IV artefact is skipped, and make the site copy say it is conditional on the compliance: block.

• CR-A07-046 (code, Medium) — approve/reject notifications are silently dropped for an inline webhook.url, because the persisted webhook_config omits url
  severity Medium · reachability documented-opt-in · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. The on-call channel never learns that a model was promoted or rejected.
  Where. forgelm/compliance.py:41-42, forgelm/compliance.py:1602-1613 (confirmed), forgelm/cli/subcommands/_approve.py:284-317 (confirmed), forgelm/cli/_parser.py:757
  Recommended fix. Log a WARNING in _build_approval_notifier when webhook_config exists but neither url nor a resolvable url_env is available, and document the url_env requirement in approve_subcommand.md and the guide.

• ↳ same root cause: DOC-D1B05-043 (docs, Low) — docs/reference/approve_subcommand.md: misleading — approve fires notify_success and reject fires notify_failure.
  severity Low
  Where. docs/reference/approve_subcommand.md:40,49; docs/guides/human_approval_gate.md:49,52; forgelm/cli/_parser.py:757,1251 · type: misleading
  The document says. approve fires notify_success and reject fires notify_failure.
  What the code does. These webhooks fire only when the training config used url_env. An inline url is not persisted and the notification is silently dropped.
  Code references. forgelm/compliance.py:41-42
  Proposed correction. Document the url_env requirement.
  Turkish mirror. the Turkish page has no such statement

• CR-A07-048 (code, Medium) — Art. 13 deployer_instructions.md is never in the documented compliance/ evidence bundle, the --compliance-export output or the evidence ZIP
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An operator follows docs/guides/enterprise_deployment.md and runs forgelm --config job.yaml --compliance-export ./audit/, or hands checkpoints/compliance/ (or its ZIP) to an auditor as the 'complete compliance artifact set'. The command exits 0 and reports success. The package has no Art. 13 deployer instructions, and nothing warns that a documented Art. 13 artifact is missing.
  Where. forgelm/compliance.py:1761, forgelm/compliance.py:2137-2296, forgelm/compliance.py:2299-2330, forgelm/cli/_no_train_modes.py:248-254
  Recommended fix. Either write deployer_instructions.md (and model_integrity.json) into compliance/ from export_compliance_artifacts and --compliance-export, or correct the three guide trees, including the TR mirror, to place it under final_model/ as usage.md does. Separately, write the doc before model_integrity.json so the manifest covers it, and have forgelm approve rewrite or regenerate the Model Location ro…

• CR-A07-049 (code, Medium) — Whole-record substitution: a passing run's audit log + genesis manifest (and, in pipeline mode, its manifest) placed in a failing run's dir verify clean; verifiers take no expected run identity and n…
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. Run A (safety gate failed, model.reverted) and run B (passed) share FORGELM_AUDIT_SECRET. The writer copies B's audit_log.jsonl and audit_log.jsonl.manifest.json over A's. forgelm verify-audit --require-hmac A/audit_log.jsonl exits 0 with 'HMAC validated'. In pipeline mode, copying the passing pipeline's compliance/pipeline_manifest.json + audit_log.jsonl + .manifest.json into the failing pipeline dir makes verify-annex-iv --pipeline exit 0, corroborated, stages_asserted 2.
  Where. forgelm/compliance.py:2449, forgelm/compliance.py:2822, forgelm/compliance.py:3409, forgelm/cli/subcommands/_verify_audit.py:227
  Recommended fix. Add --expect-run-id to verify-audit and verify-annex-iv --pipeline. Write the audit run_id (and the log head hash at export) into annex_iv_metadata.json and model_integrity.json and cross-check them. Document that a pass attests to a self-consistent record from some run under this secret, not to this directory's run.

• CR-A07-050 (code, Medium) — pipeline.initialized is documented as emitted before any model load, and pipeline.failed as covering any aborted run, but a run that aborts in model/data loading or in ForgeTrainer.init leaves no…
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An auditor reconstructing runs from audit_log.jsonl sees a run that was initialized and never completed or failed (run2). A run that died in data loading is absent altogether (run3). The catalog tells them every aborted run ends in pipeline.failed and that initialization precedes model loading.
  Where. docs/reference/audit_event_catalog.md:38, docs/reference/audit_event_catalog.md:164, forgelm/cli/_training.py:141-148, forgelm/cli/_training.py:148
  Recommended fix. Correct the catalog trigger text (initialized happens after model and dataset load) and narrow pipeline.failed to 'train() raised'. Alternatively, move the init validations inside a guarded block that emits pipeline.failed, and consider creating the AuditLogger before model load in the CLI path.

• CR-A07-052 (code, Medium) — HMAC tag stripping without --require-hmac and non-string run_id/_hmac are unpinned (G-A09-04, G-A09-05)
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Where. tests/test_compliance.py:2042, tests/test_verification_toolbelt.py:4443, forgelm/compliance.py:2439, forgelm/compliance.py:2406-2417

• CR-A07-054 (code, Medium) — Catalog claims every Article-12-relevant safety_trend value is in the chain; standalone safety-eval writes the gate decision only to the trend log
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An auditor follows the catalog, reads only audit_log.jsonl, and finds no record of a deployment-time safety-eval that failed (exit 3), even though safety_trend.jsonl recorded passed:false.
  Where. docs/reference/audit_event_catalog.md:132, forgelm/safety/_results.py:137, forgelm/trainer.py:1445, docs/reference/safety_eval_subcommand.md:105-111
  Recommended fix. Either emit safety.evaluation_completed from the safety-eval subcommand through its AuditLogger, or restrict the catalog sentence to the training path and name the standalone gap.

• CR-A07-055 (code, Medium) — compliance_report.json does not record the training-time dependency environment that the incident SOP relies on
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A CVE lands in transformers X.Y. The responder follows SOP §4.3 step 4, opens compliance_report.json and finds no dependency versions, so the affected models cannot be identified from the Art. 11 bundle.
  Where. forgelm/compliance.py:1513-1547, docs/qms/sop_incident_response.md:203-205, forgelm/compliance.py:1491-1627 (manifest literal :1513-1547; forgelm_version :1514), docs/qms/sop_incident_response-tr.md:202
  Recommended fix. Either add an 'environment' block (python version plus importlib.metadata versions of the core and loaded optional deps) to generate_training_manifest, or correct the SOP to point at a source that holds this data.

• CR-A07-058 (code, Medium) — Default unkeyed chain lets any line after line 1 be edited and re-chained; README and the OK output understate this
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An operator who skips the secret because only the last line is said to be exposed has every earlier entry silently rewritable.
  Where. README.md:120, forgelm/compliance.py:2523-2578 (_verify_chain_walk; valid return at 2578, not 2576), forgelm/compliance.py:238-260, forgelm/compliance.py:2578
  Recommended fix. Reword README:120: without the secret, any entry after the first can be rewritten and re-chained.

• CR-A07-059 (code, Medium) — Catalog and event name claim model.integrity_verified re-hashes artefacts; the trainer only writes the manifest and verifies nothing
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An auditor reads model.integrity_verified in the hash-chained log as proof that the shipped model was verified. Nothing was verified, and the shipped directory does not verify.
  Where. docs/reference/audit_event_catalog.md:60, forgelm/trainer.py:2157-2175, docs/reference/audit_event_catalog-tr.md:60, forgelm/trainer.py:1496-1506
  Recommended fix. Rename the event (e.g. model.integrity_manifest_written) through a compensating catalog entry, or actually call verify_integrity after writing and emit its verdict. Fix the catalog row.

Done when

• Every finding above is fixed or closed with a reason (tick it here).
• Code fixes carry a test; doc fixes update the Turkish mirror in the same PR.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}