Skip to content

purge memorisation check cannot see exported GGUF files: forgelm export records only a basename and no audit event #1071

Open

Description

@cemililik

Summary

The memorisation warning of forgelm purge --row-id lists the model, staging and checkpoint-* directories under --output-dir and under the training layouts a loaded --config names, plus merge.output_dir. A model exported with forgelm export MODEL_PATH --output PATH lives wherever --output pointed, and ForgeLM keeps no usable record of where, so an exported GGUF is invisible to the check. CR-A10-042 (#102) named "exported models"; #1068 documents the limit instead of closing it.

Evidence

  • forgelm export (forgelm/cli/subcommands/_export.py, forgelm/export.py::export_model) writes no audit event. The destination reaches the operator only through the command's own output (output_path in the JSON envelope).
  • export.py::_update_integrity_manifest appends {"file": <basename>, "format", "quant", "sha256", "size_bytes"} to exported_artifacts in <model_dir>/model_integrity.json. It keeps the file name and drops the directory, so the manifest cannot be used to find the file. It does nothing when the manifest is absent, and the update is best effort.
  • Nothing else reads exported_artifacts; the field is described only in docs/usermanuals/{en,tr}/deployment/gguf-export.md.
  • After purge: find the weights a run really leaves behind (fixes #102) #1068 the docs say that models exported or copied elsewhere are invisible to the check (docs/guides/gdpr_erasure.md, the GDPR manual page, the QMS risk plan).

Why it matters

A GGUF quantised from weights trained on an erased row is the artefact most likely to leave the operator's machine, and the operator relies on data.erasure_warning_memorisation to be told which copies to chase. Today the one copy the tool created for distribution is the one it cannot name.

Acceptance criteria

  • Decide where the destination is recorded and write the decision in this issue: an additive absolute path in exported_artifacts, a new audit event for a completed export (added to docs/reference/audit_event_catalog.md and its TR mirror, which tools/check_audit_event_catalog.py enforces), or both. Say what happens when the model directory has no model_integrity.json.
  • forgelm export records the absolute destination of the file it wrote and, for a K-quant request, also the requested --output. For a K-quant, export_model writes <output>.f16.gguf as an intermediate (_resolve_kquant_path) and ExportResult.output_path is that file; the file the operator ships is produced later by the manual llama-quantize follow-up at --output, which ForgeLM never sees. Manifests written before the change keep working unchanged (verify-integrity and verify-gguf included), and the new key is additive and ignored by their readers.
  • forgelm purge --row-id adds the recorded export destinations that still exist to memorisation_artefact_paths, reading them from the model directories it already finds. A missing file is skipped silently; an unreadable manifest is logged at WARNING and does not abort the erasure, which has already removed the row by then.
  • Tests cover: an export recorded then found by purge, a destination that was deleted since, a pre-change manifest, and an unreadable manifest.
  • Docs in EN and TR: the audit catalog row, the purge reference, the GDPR guide and manual page (replace "models you export are invisible" with what is now covered), and gguf-export.md for the recorded field. CHANGELOG entry.
  • State in the docs what stays invisible (a model copied by hand, or exported with --no-integrity-update if that path is kept), so the limit is explicit.

Context

Follow-up of #102 (CR-A10-042) and #1068. merge.output_dir is already covered by #1068 through the loaded config. Related: #1069 (retention horizon for checkpoint-*).

Sequencing: Phase 16 S5 (#891, CLI-03) changes how verify-integrity treats exported_artifacts, and S6 (#892) reworks export_model's temp and atomic-write handling. Settle the manifest schema with those before implementing. A new audit event would also serve CR-A09-019 (#652), which wants verify-gguf's sidecar anchored to a keyed export record, so decide the two together; but it needs its own answer to which audit_log.jsonl an export appends to, because forgelm export receives only a model path, not the training output directory.

Activity

  1. added
    needs-triageNot triaged yet: every new issue starts with it; only a maintainer removes it
    enhancementNew feature or request
    on Oct 23, 2026
  2. added
    severity: mediumReal defect with limited reach or a workaround
    wave: 3Roadmap wave 3: medium severity, module sweeps
    on Oct 23, 2026
  3. cemililik commented on Oct 23, 2026

    @cemililik
    CollaboratorAuthor

    Triaged on development e6bf28d, after #1068 merged during this triage. Each finding was first checked on 83e62f6 and on #1068's head (ab5ce5f), whose code is identical to the merge.

    Accepted: enhancement, severity medium, wave 3, under the W3 module-sweep epic #870.

    What I checked

    Two findings that changed the acceptance criteria (body updated)

    1. K-quant requests. export_model writes <output>.f16.gguf as an intermediate and ExportResult.output_path is that file. The file the operator ships is produced later by the manual llama-quantize follow-up at --output, which ForgeLM never sees. Record the written file and the requested --output; "destinations that still exist" then handles the not-yet-quantised case.
    2. Sequencing with Phase 16. S5 changes how verify-integrity reads exported_artifacts, and S6 ([Phase 16] S6 — Invocation-owned resources: temp state, atomic writes, worker lifetime #892) reworks export_model's temp and atomic-write handling. The new key must be additive and ignored by that reader; settle the schema with them before implementing.

    Maintainer decision still open (criterion 1). My recommendation is an additive absolute path in exported_artifacts: purge already finds the model directories and can read their manifests, and no new audit-log question arises. A new audit event is a larger step, because forgelm export receives only a model path, so which audit_log.jsonl it appends to needs its own design. It would also serve CR-A09-019 (#652: anchor verify-gguf's sidecar to a keyed export record), so it is worth deciding together, but it is not a prerequisite. Without a model_integrity.json the export stays invisible, and the docs should say so (last criterion).

    Size budget: as in #1069, _purge.py has 0 LOC of headroom (decision C-18: record a budget_history entry, or use a new module).

    Nothing blocks the purge side: it extends the memorisation_artefact_paths payload that #1068 added. The export side waits on the maintainer decision above.

    Not open for contribution yet: this needs a maintainer decision first. If you would like to work on it, please comment with your proposed approach before opening a pull request.

  4. removed
    needs-triageNot triaged yet: every new issue starts with it; only a maintainer removes it
    on Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: gdpr-approvalCode area: gdpr-approvalarea: post-trainingCode area: post-trainingenhancementNew feature or requestseverity: mediumReal defect with limited reach or a workaroundwave: 3Roadmap wave 3: medium severity, module sweeps

    Type

    No type

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions