Skip to content

[Wave 4] Low/info — Module sweep: Compliance core: Annex IV, audit log, integrity manifest, provenance (part 4/5) #659

Open

Description

@cemililik

Wave 4 · 20 finding(s) in 20 fix package(s). Fix these whenever a pull request touches this module ("clean the module you touch").

Findings

  • CR-A07-129 (code, Low) — Line-model edge cases contradict the docs: CRLF conversion, LF->CR and blank-line insertion pass verify-audit --require-hmac although 'any byte-level corruption ... shows up as a chain break'; a BOM…
    severity Low · reachability default · intent contract-violation · confidence likely · independently confirmed · reproduced end to end
    Failure scenario. An auditor relies on the manual's promise that any byte-level change in transit is a chain break, and treats a CRLF-converted copy as byte-identical evidence. Separately, an operator chasing approve's 'corrupted at line 2' opens line 2 of a log that verify-audit numbers differently.
    Where. forgelm/compliance.py:2658-2660, forgelm/compliance.py:2542-2551, forgelm/compliance.py:288-294, docs/usermanuals/en/compliance/verify-audit.md:14
    Recommended fix. Either make the verifier byte-exact (LF-only, reject \r and blank records, as the writer never emits them) or reword the manual to say that the chain authenticates record content, with line-ending and blank-line changes tolerated. Report physical file line numbers consistently across verify-audit and the approval readers.

  • CR-A07-130 (code, Low) — access_control.md section 5: 'Audit logs in <output_dir> inherit the dir's perm' is wrong; the log and its genesis manifest are created with the umask mode (0644)
    severity Low · reachability default · intent contract-violation · confidence likely · single reviewer · reproduced end to end
    Failure scenario. An archived output dir (section 3.4 tells operators to archive it) exposes a world-readable audit log that carries operator identifiers.
    Where. docs/qms/access_control.md:172, docs/qms/access_control-tr.md:174, forgelm/compliance.py:549, forgelm/compliance.py:468
    Recommended fix. Say that files are created with the process umask and that protection comes from the 0700 directory (or recommend umask 077), in EN and TR.

  • CR-A07-131 (code, Low) — compliance_summary.md cites a non-existent function compute_artefact_sha256 as the Article 15 model-integrity mechanism
    severity Low · reachability unreachable · intent contract-violation · confidence likely · independently confirmed · reproduced end to end
    Failure scenario. An auditor or integrator traces the Art. 15 claim to compute_artefact_sha256 and finds nothing, which weakens the traceability of the compliance summary.
    Where. docs/reference/compliance_summary.md:35, docs/reference/compliance_summary-tr.md:33, forgelm/compliance.py:760
    Recommended fix. Replace the name with generate_model_integrity / hash_file in both language versions.

  • CR-A07-132 (code, Low) — The pipeline structural verifier accepts manifest shapes the writer can never produce: duplicate stage names (the first duplicate decides the stopped_at check), bool or float index, empty stages list
    severity Low · reachability default · intent contract-violation · confidence likely · independently confirmed · reproduced end to end
    Failure scenario. A manifest that says the pipeline stopped at 'dpo' because 'dpo' failed, while also carrying a completed and deep-verified 'dpo' row, is reported OK with exit 0. Whether it passes depends only on which duplicate comes first.
    Where. forgelm/compliance.py:2012-2019, forgelm/compliance.py:2057, forgelm/compliance.py:2092-2095, forgelm/compliance.py:2114-2116
    Recommended fix. In _verify_manifest_payload: flag duplicate stage names, require type(index) is int (rejecting bool), and flag an empty stages list. Once names are unique, matching[0] is well defined.

  • CR-A07-133 (code, Low) — Under an arbitrary passwd-less UID (docker run --user / OpenShift random UID), training aborts with ConfigError (exit 1) only after model load and dataset preparation; --dry-run passes and the contai…
    severity Low · reachability default · intent intended-but-harmful · confidence likely · independently confirmed · reproduced with stubs
    Failure scenario. A platform team deploys the image on OpenShift (random UID) with no FORGELM_OPERATOR. The CI dry-run step passes. The training job downloads and loads a 7B model and tokenises the dataset, then exits 1 with 'Operator identity unavailable' after minutes to tens of minutes of GPU time. The docs had said behaviour on Kubernetes is identical.
    Where. forgelm/compliance.py:196-228, forgelm/trainer.py:354, forgelm/cli/_training.py:141-148, forgelm/cli/_training.py:163-170
    Recommended fix. Resolve the operator identity before model load: construct the AuditLogger or call a shared resolver early in _run_training_pipeline, and include it in --dry-run. Document FORGELM_OPERATOR as required for non-default UIDs in docker.md and enterprise_deployment.md, and qualify the 'behaviour is identical' tip.

  • CR-A07-134 (code, Info) — export_evidence_bundle (no production caller) zips its own growing .tmp when bundle_path is inside output_dir and follows symlinked files out of the tree
    severity Info · reachability undocumented · intent intended-but-harmful · confidence likely · independently confirmed · reproduced end to end
    Failure scenario. An integrator writes the bundle next to the artefacts and ships an archive that contains a truncated copy of itself and files from outside the compliance dir.
    Where. forgelm/compliance.py:2315-2319 (os.walk + zf.write with no exclusion and no realpath containment), docs/reference/audit_event_catalog.md:73, forgelm/compliance.py:2312, forgelm/compliance.py:2315
    Recommended fix. Skip the tmp and bundle paths and refuse escaping symlinks (reuse the realpath containment from generate_model_integrity). Consider wiring a CLI entry or dropping the F-090 claim.

  • CR-A07-135 (code, Info) — log_event creates audit_log.jsonl before serialising, so a failed first event leaves a zero-byte log that verify-audit says ForgeLM never writes
    severity Info · reachability undocumented · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
    Where. forgelm/compliance.py:549, forgelm/compliance.py:582, forgelm/compliance.py:2756-2763, forgelm/compliance.py:2719-2727
    Recommended fix. Serialise the entry (and its HMAC) before opening the file, apart from prev_hash, which must be read under the lock. Alternatively, soften the verifier message so it does not assert impossibility.

  • CR-A07-136 (code, Info) — Pipeline corroboration parses each audit line three times and holds about 4.4x the log size
    severity Info · reachability default · intent intended-and-acceptable · confidence likely · single reviewer · reproduced end to end
    Failure scenario. A 30 MB audit log (about 200k events) from a long multi-stage pipeline is verified with FORGELM_AUDIT_SECRET set. Peak traced memory is about 4.4x the file size and runtime is about 1.6 s per 22 MB. This is correct but wasteful. A log just over the cap fails as oversize (exit 1), so a log that has outgrown the cap cannot be corroborated at all, and no rotation or streaming path exists.
    Where. forgelm/compliance.py:3365-3406, forgelm/compliance.py:3445-3456, forgelm/compliance.py:2542-2576, forgelm/compliance.py:2433-2438
    Recommended fix. Parse each line once and pass the (line, entry) pairs into the chain walk. That would cut CPU by about 1.5 to 2x and remove the retained duplicate. The retained dicts could also be reduced to the few fields _live_passed_stages needs (event, pipeline_run_id, stage_name, gate_decision). Expected memory gain is about 3 to 4x. Raising the 32 MiB cap is not advisable until this is done.

  • CR-A07-137 (code, Info) — export_evidence_bundle deflates incompressible weights with no size guard or caller in the codebase
    severity Info · reachability undocumented · intent intended-and-acceptable · confidence likely · independently confirmed · reproduced end to end
    Failure scenario. An integrator points export_evidence_bundle at a directory that holds a 14 GB model. Bundling takes about 3 minutes of single-core deflate and produces a 14 GB archive, where ZIP_STORED would run at disk speed.
    Where. forgelm/compliance.py:2314-2319, docs/reference/architecture.md:147
    Recommended fix. Use ZIP_STORED for known-incompressible suffixes (.safetensors, .bin, .gguf, .pt), or restrict the bundle to the compliance/ subtree. Expected gain is about 10x or more on weight-heavy trees. The first option is cheap, but check first that no caller exists, since the function appears dead.

  • CR-A07-138 (code, Info) — export_evidence_bundle ZIP is not byte-reproducible (unsorted os.walk, per-file mtimes)
    severity Info · reachability unreachable · intent intended-and-acceptable · confidence likely · independently confirmed · not reproduced (code trace)
    Failure scenario. An auditor compares bundle hashes across two exports and sees different digests for identical content.
    Where. forgelm/compliance.py:2315-2319
    Recommended fix. Sort dirs/files and use a fixed ZipInfo date_time if a bundle digest is ever published.

  • CR-A07-139 (code, Info) — Windows/non-POSIX degradations (flock, SIGALRM ReDoS budget) are silent at runtime, and per-OS tests run only at tag time
    severity Info · reachability documented-opt-in · intent intended-and-acceptable · confidence likely · independently confirmed · not reproduced (code trace)
    Failure scenario. A Windows operator running an untrusted-looking regex gets no budget and no warning; a Windows-only regression is found only after tagging.
    Where. forgelm/compliance.py:140-157, forgelm/cli/subcommands/_approve.py:28-45, forgelm/_strip_pattern.py:383-386, .github/workflows/publish.yml:44-58
    Recommended fix. Log a one-time warning when the budget cannot be armed; add a PR-time Windows smoke job.

  • CR-A07-140 (code, Info) — Example 4 emits ForgeLM-reserved event names with payloads that differ from ForgeLM's own
    severity Info · reachability documented-opt-in · intent intended-but-harmful · confidence likely · single reviewer · not reproduced (code trace)
    Failure scenario. If a user logger shares an output_dir or run_id with a pipeline run, a user-emitted pipeline.completed may be counted as the pipeline terminal event.
    Where. docs/reference/library_api_reference.md:194-207, docs/reference/library_api_reference-tr.md:194-207, docs/reference/audit_event_catalog.md:31, docs/reference/audit_event_catalog.md:37
    Recommended fix. Use a namespaced custom event name in the example, and state which names ForgeLM owns.

  • CR-A07-141 (code, Info) — A dangling symlink inside the model directory aborts generate_model_integrity, and the trainer swallows it so no model_integrity.json is written
    severity Info · reachability undocumented · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
    Failure scenario. final_model contains a broken in-tree symlink (for example a leftover link to a deleted checkpoint file). The integrity manifest is never written, the run logs one WARNING and reports success, and verify-integrity later exits 1 for a missing manifest.
    Where. forgelm/compliance.py:787-811, forgelm/trainer.py:2169-2175
    Recommended fix. Treat a non-regular or unreadable target like an escaping symlink: record it under skipped_symlinks, or fail the run loudly.

  • CR-A07-142 (code, Info) — AuditLogger.log_event (stable API, documented OSError-only) leaks ValueError/TypeError/RecursionError/RuntimeError from its serialisation step for payloads the sanitiser does not normalise; the pipel…
    severity Info · reachability undocumented · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
    Failure scenario. An integrator follows the library guide and logs audit.log_event('eval.done', per_threshold={0.5: acc, float('nan'): x}) or a dict keyed by tuples. They get ValueError/TypeError where the contract promised OSError, so a handler written for OSError lets it crash the job. Inside ForgeLM's pipeline emitter the same payload is dropped from the Art. 12 log with only a WARNING.
    Where. forgelm/compliance.py:574-582, forgelm/compliance.py:600-606, forgelm/_strict_json.py:53-54, forgelm/_strict_json.py:78-81
    Recommended fix. Either sanitise keys too (non-finite float key -> repr string; reject non-str/int/float/bool/None keys with a clear ValueError), add a cycle and depth guard, and catch Exception around float() in the sanitiser, or document ValueError/TypeError in the log_event contract and the library reference. Do not swallow in _audit_event without an alternate record.

  • CR-A07-143 (code, Info) — Writer and verifier HMAC canonicalisations disagree when payload keys collide after JSON key coercion: an untouched keyed log fails verify-audit with exit 6
    severity Info · reachability undocumented · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
    Failure scenario. An orchestrator logs a mapping keyed by both ints and their string forms, for example merged per-epoch metrics from two sources. The next forgelm verify-audit --require-hmac in CI exits 6, the documented tampering alarm, on an untampered log.
    Where. forgelm/compliance.py:574-582, forgelm/compliance.py:2433-2438
    Recommended fix. Compute the HMAC over the round-tripped structure (json.loads(dumps_strict(entry, default=str)) re-dumped), or HMAC the exact line bytes minus the tag. Alternatively, reject payloads whose keys collide after stringification.

  • CR-A07-144 (code, Info) — In-memory verify_pipeline_manifest raises AttributeError on a non-dict root instead of returning a violation, unlike its own handling of non-dict rows
    severity Info · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
    Failure scenario. A library integrator runs verify_pipeline_manifest(json.load(f)) on an untrusted file whose root is an array and gets an AttributeError traceback instead of a violation list.
    Where. forgelm/compliance.py:2082, forgelm/compliance.py:3083-3096, forgelm/compliance.py:2097-2109
    Recommended fix. Return ['manifest root is , expected a JSON object'] for a non-dict root, mirroring the per-row guard.

  • CR-A07-145 (code, Info) — The List[str] facade keeps the routing prefixes, but its docstring omits the integrity-first precedence, and the only implementation of that precedence is private to the CLI
    severity Info · reachability undocumented · intent intended-but-harmful · confidence likely · independently confirmed · reproduced end to end
    Failure scenario. A library CI wrapper maps IO_ERROR:: to 'retry' (exit 2) before checking for untagged violations. A run with one unreadable stage artefact plus a tampered chain is then retried as transient instead of alarming on tampering.
    Where. forgelm/compliance.py:3529-3551, forgelm/cli/subcommands/_verify_annex_iv.py:66-133, forgelm/verify/__init__.py:146-157
    Recommended fix. State the integrity-first precedence in the facade docstring, or move _classify_pipeline_violations (or a public pipeline_exit_code(violations)) into forgelm.verify, so library callers share the CLI's routing.

  • CR-A07-146 (code, Info) — Stale docstrings describe the pipeline verifier as a per-stage 'training_manifest existence' check reached through verify_pipeline_manifest_at_path
    severity Info · reachability default · intent contract-violation · confidence likely · single reviewer · not reproduced (code trace)
    Failure scenario. A maintainer reasons from these docstrings that the disk layer is an existence check and misjudges what a change affects.
    Where. forgelm/compliance.py:3090-3094, forgelm/compliance.py:2063, forgelm/cli/subcommands/_verify_annex_iv.py:139-141, forgelm/cli/subcommands/_verify_annex_iv.py:326-331
    Recommended fix. Update the three docstrings to describe the deep parse and Tier-3 corroboration, and say that check 4 runs over well-formed rows.

  • CR-A07-147 (code, Info) — Operator-resolution rule names a 'config' source that does not exist and a fallback that differs from the code
    severity Info · reachability default · intent contract-violation · confidence likely · independently confirmed · reproduced end to end
    Failure scenario. An operator follows the standard and looks for a config key to pin the identity; none exists. In a future change, a log_event caller passing a details dict containing 'operator' would make the entry carry that string, for example 'unknown'.
    Where. docs/standards/logging-observability.md:108, forgelm/compliance.py:192-229, forgelm/compliance.py:554-561, forgelm/trainer.py:1550
    Recommended fix. Fix the standard's wording (env, then user@host, then ConfigError or opt-in anonymous@host). Optionally reject or ignore reserved keys (timestamp, run_id, operator, event, prev_hash, _hmac) in log_event details.

  • CR-A07-148 (code, Info) — 'Segregation-of-duties violations are detectable post-hoc' does not hold against the trainer principal, which holds the shared HMAC key
    severity Info · reachability default · intent intended-and-acceptable · confidence likely · single reviewer · reproduced end to end
    Failure scenario. A compromised trainer CI approves its own run under a forged approver id. The post-hoc jq SoD cookbook sees two different identities and reports compliance.
    Where. forgelm/compliance.py:192-194, forgelm/compliance.py:258, forgelm/cli/subcommands/_approve.py:121-170, docs/guides/human_approval_gate.md:146
    Recommended fix. State in the guide that detection assumes approver and trainer use different signing keys. Consider a per-principal key or an asymmetric signature for approval events, and warn when approver == the trainer's operator.

Done when

  • Every finding above is fixed or closed with a reason (tick it here).
  • Code fixes carry a test; doc fixes update the Turkish mirror in the same PR.

Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: complianceCode area: compliancebugSomething isn't workingseverity: lowMinor defect, inaccuracy or rough edgesource: review-2026-10Found by the October 2026 code and documentation reviewwave: 4Roadmap wave 4: low / info / watch, when touching the module

    Type

    No type

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions