Skip to content

fix(purge): warn when --output-dir falls back to the corpus directory, and document the real memorisation check - #992

Closed
cemililik wants to merge 2 commits into
developmentfrom
fix/purge-warns-on-implicit-output-dir
Closed

cemililik wants to merge 2 commits into
developmentfrom
fix/purge-warns-on-implicit-output-dir

Conversation

@cemililik

@cemililik cemililik commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The purge reference and the user manual said that the implicit --output-dir fallback "emits a WARNING naming the resolved dir". It did not: nothing was logged. The fallback is the corpus's directory, and the memorisation check (data.erasure_warning_memorisation) looks for <output_dir>/final_model/ there, a place a training run rarely writes its model. A default forgelm purge --row-id X --corpus data/train.jsonl therefore ended with a completed-erasure record and no memorisation event, so a reader of the audit chain alone would conclude that no model had consumed the row.

--row-id without --output-dir now logs one warning naming the resolved directory and the final_model/ path it will look for, which makes the documented behaviour true. The documentation also described the condition as "a final_model/ exists for any run that consumed the corpus"; the code checks one literal directory and cannot tell which runs consumed the corpus. It now says that.

Related issues

Refs #102 — the warning and the documentation. Reading training.output_dir from --config and scanning final_model* / checkpoint-* is the larger fix; it changes the content of an erasure audit event (additively) and wants its own review, so it stays open.

Changes

  • forgelm/cli/subcommands/_purge.py: when --output-dir is omitted, a WARNING names the corpus directory used for the audit log, the salt and the memorisation check, the final_model/ path that check looks for, and tells the operator to pass --output-dir <training output dir> (also needed to correlate with forgelm reverse-pii).
  • tools/check_module_size.py: the purge module's budget goes from 1216 to 1224 lines (+8) with a history note. This follows the project's documented option ("land the split or raise the budget with a budget_history note") and the previous raise's reasoning (C-18): the message has to sit where the default is resolved.
  • Documentation, English and Turkish: the audit-event catalogue, the purge reference, the GDPR erasure guide and user-manual page, and the risk-treatment plan.

Type

  • Bug fix

Testing

  • tests/test_gdpr_erasure.py::TestImplicitOutputDirWarning: the warning fires without --output-dir, names both paths, and is absent when --output-dir is given.
  • The first gauntlet run of this branch failed tests/test_check_module_size.py (the budget); that is fixed by the second commit. The full CLAUDE.md gauntlet from a clean worktree of the final state: exit 0.

Checklist

  • English and Turkish mirrors updated together
  • CHANGELOG entry under [Unreleased]
  • Tests added

🤖 Generated with Claude Code

Summary by Sourcery

Make implicit purge output-directory behavior visible and align memorisation-check documentation with the implementation.

Bug Fixes:

  • Log a warning when purge --row-id implicitly uses the corpus directory as --output-dir, including the exact final_model/ path checked for memorisation.

Enhancements:

  • Clarify that memorisation detection checks only for a literal final_model/ directly under --output-dir and cannot identify which training runs consumed the corpus.

Build:

  • Increase the purge module size budget to accommodate the warning logic and document the budget change.

Documentation:

  • Update the English and Turkish purge references, audit-event catalogues, GDPR erasure guides, user manuals, risk-treatment plans, and changelog to describe the actual output-directory and memorisation-check behavior.

Tests:

  • Add coverage for implicit-directory warning content, silence with an explicit output directory, and absence of a memorisation event when no model exists in the resolved directory.

cemililik and others added 2 commits October 18, 2026 10:16
…, and document the real memorisation check

The purge reference and the user manual both said that the implicit `--output-dir` fallback "emits a WARNING naming the
resolved dir". It did not: nothing was logged. The fallback is the corpus's directory, and the memorisation check
(`data.erasure_warning_memorisation`) looks for `<output_dir>/final_model/` there, a place a training run rarely writes its
model. A default `forgelm purge --row-id X --corpus data/train.jsonl` therefore ended with a completed-erasure record and no
memorisation event, so a reader of the audit chain alone would conclude that no model had consumed the row.

`--row-id` without `--output-dir` now logs one warning naming the resolved directory and the `final_model/` it will look for,
which makes the documented behaviour true. The audit-event catalogue, the purge reference, the GDPR erasure guide, the user
manual and the risk-treatment plan (English and Turkish) described the condition as "a final_model/ exists for any run that
consumed the corpus"; the code checks one literal directory and cannot tell which runs consumed the corpus. They now say that.

Reading `training.output_dir` from `--config` and scanning `final_model*` / `checkpoint-*` is the larger fix and stays open on the
issue; it changes the content of an erasure audit event (additively) and wants its own review.

Refs #102

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…tput-dir warning

The warning that `forgelm purge --row-id` now logs when --output-dir is omitted adds eight lines to a module on the
deferred-split list, and the budget has to equal the measured size. It is a raise rather than a split for the reason
the previous raise gave (C-18): the message has to sit where the default is resolved. The history note says why.

Refs #102

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@sourcery-ai

sourcery-ai Bot commented Oct 18, 2026

Copy link
Copy Markdown

Sorry @cemililik, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 12 hours and 58 minutes by commenting @sourcery-ai review. Upgrade to get a review now.

@coderabbitai

coderabbitai Bot commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

Next included review available in 1 minute.

Check out review usage here.

View limit details

Limit details: You’ve used all 3 included reviews currently available. Your 48 included PR review attempts over the past 7 days set your current allowance at 3 reviews per hour.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Essentials
  • Run ID: 40657c29-bbf1-468e-a98b-f94452503c65
📥 Commits

Reviewing files that changed from the base of the PR and between 56b4f0d and 9d1045c.

📒 Files selected for processing: 14
  • CHANGELOG.md
  • docs/guides/gdpr_erasure-tr.md
  • docs/guides/gdpr_erasure.md
  • docs/qms/risk_treatment_plan-tr.md
  • docs/qms/risk_treatment_plan.md
  • docs/reference/audit_event_catalog-tr.md
  • docs/reference/audit_event_catalog.md
  • docs/reference/purge_subcommand-tr.md
  • docs/reference/purge_subcommand.md
  • docs/usermanuals/en/compliance/gdpr-erasure.md
  • docs/usermanuals/tr/compliance/gdpr-erasure.md
  • forgelm/cli/subcommands/_purge.py
  • tests/test_gdpr_erasure.py
  • tools/check_module_size.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@sourcery-ai

sourcery-ai Bot commented Oct 18, 2026

Copy link
Copy Markdown

Reviewer's Guide

This PR makes the documented implicit --output-dir behavior real by logging the corpus-directory fallback and the exact final_model path inspected, adds focused regression tests, and updates English/Turkish operational documentation to clarify that memorisation detection is limited to a literal final_model directory directly under the selected output directory.

Sequence diagram for purge output directory fallback and memorisation check

sequenceDiagram
    participant Operator
    participant Purge as forgelm purge
    participant Filesystem
    participant Audit as Audit log

    Operator->>Purge: _run_purge_row_id(args, output_format)
    Purge->>Purge: Resolve output_dir from corpus directory
    Purge-->>Operator: logger.warning(resolved output_dir and final_model path)
    Purge->>Filesystem: os.makedirs(output_dir, exist_ok=True)
    Purge->>Filesystem: Check output_dir/final_model
    alt final_model exists
        Purge->>Audit: data.erasure_warning_memorisation
    end
    Purge->>Audit: data.erasure_completed
Loading

Flow diagram for purge memorisation detection scope

flowchart TD
    A[forgelm purge --row-id] --> B{--output-dir given?}
    B -- No --> C[Use corpus parent directory]
    C --> D[Log WARNING with resolved directory and final_model path]
    B -- Yes --> E[Use selected output directory]
    D --> F[Check exact output_dir/final_model directory]
    E --> F
    F --> G{final_model exists directly under output_dir?}
    G -- Yes --> H[Emit data.erasure_warning_memorisation]
    G -- No --> I[No memorisation warning]
    H --> J[Complete erasure audit]
    I --> J
Loading

File-Level Changes

Change Details Files
Warn when row purges implicitly resolve the output directory from the corpus path.
  • Resolve the fallback directory before creating audit artifacts.
  • Emit a WARNING containing the resolved directory and the exact final_model path checked for memorisation.
  • Recommend supplying the training output directory and preserve silence for explicit --output-dir usage.
  • Add coverage for warning contents, explicit-directory silence, and the absence of a memorisation event when the model is elsewhere.
forgelm/cli/subcommands/_purge.py
tests/test_gdpr_erasure.py
Correct documentation to describe the literal memorisation-check behavior and its operational limitation.
  • Replace claims about detecting any run that consumed the corpus with the actual direct-child final_model/ check.
  • Document that the fallback is the corpus directory and that operators should provide the training output directory.
  • Synchronize English and Turkish audit catalogs, purge references, GDPR guides, user manuals, and risk-treatment plans.
docs/reference/audit_event_catalog.md
docs/reference/audit_event_catalog-tr.md
docs/reference/purge_subcommand.md
docs/reference/purge_subcommand-tr.md
docs/guides/gdpr_erasure.md
docs/guides/gdpr_erasure-tr.md
docs/usermanuals/en/compliance/gdpr-erasure.md
docs/usermanuals/tr/compliance/gdpr-erasure.md
docs/qms/risk_treatment_plan.md
docs/qms/risk_treatment_plan-tr.md
Record the bug fix and accommodate the purge module's enforced size budget.
  • Add an unreleased changelog entry covering the warning and corrected memorisation semantics.
  • Increase the deferred module budget by eight lines and document the rationale in budget_history.
CHANGELOG.md
tools/check_module_size.py

Possibly linked issues


Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@cemililik

Copy link
Copy Markdown
Collaborator Author

Superseded by #999: the wave-0 fixes are combined by theme into five pull requests to cut the review load. This branch's commits are included there unchanged apart from the CHANGELOG.md merge; the branch stays on the remote.

@cemililik cemililik closed this Oct 18, 2026
cemililik added a commit that referenced this pull request Oct 18, 2026
…n.enforce, transformers CVE floor, FSDP strategies, purge output dir (#999)

## Summary

Six code fixes combined into one pull request, replacing #984, #985,
#987, #989, #990 and #992 (to be closed as superseded). Each fixes a
case where the code failed, or a documented option did nothing, without
saying so. Each was mutation-checked on its own; together they pass the
full gauntlet.

## Related issues

Refs #103, #95, #92, #98, #109, #104, #102. Not closed by this change:
the top-of-range problems named in #109 (`warmup_ratio` on transformers
5.15+, the string `fsdp` form removed in 5.20), the FSDP2 migration
(#104), and reading `training.output_dir` for the purge memorisation
check (#102).

## Changes

| # | Fix | Where |
|---|---|---|
| #984 | `training.rope_scaling` crashed every model load on
transformers 5 (`KeyError: 'rope_type'`, or `rope_theta` lost). The
requested keys are merged **over** the model's `rope_parameters` and the
finished config is loaded; `sliding_window_attention` travels in the
same config object. | `forgelm/model.py` |
| #985 | The CLI sets `HF_HUB_DISABLE_TELEMETRY=1` (via `setdefault`)
before `huggingface_hub` is imported, so TRL's trainer no longer sends
Hugging Face's anonymous usage ping the privacy page says does not
exist. Library use is not covered; the air-gap page says so. |
`cli/_config_load.py`, `cli/_dispatch.py` |
| #987 | `retention.enforce` was documented as a gate (`warn_on_excess`,
`block_on_excess` → `EXIT_EVAL_FAILURE`) but no command reads it. Docs,
field description, both wizards say it is reserved; the CLI warns when a
config sets either of the last two. | `config.py` (in place),
`cli/_config_load.py`, `wizard/_collectors.py` |
| #989 | `transformers>=5.3.0` was called the CVE floor but admitted
CVE-2026-5241 and CVE-2026-9856 (a crafted `chat_template` name writes
outside the save directory). Floor is now `>=5.10.1` (5.10.0 is yanked),
`huggingface_hub>=1.5.0`, the nightly floor job pins both, and tests
keep it from drifting. | `pyproject.toml`, `nightly.yml`, `tests/` |
| #990 | `fsdp_strategy: no_shard` / `hybrid_shard` passed `--dry-run`
and exited 2 after the model loaded (FSDP2 rejects them). They are now
rejected when the config loads (exit 1), and two knobs FSDP2 ignores
(`fsdp_auto_wrap: false`, `backward_post`) log a warning. | `config.py`
(in place), `cli/_config_load.py` |
| #992 | `forgelm purge --row-id` without `--output-dir` silently
pointed the memorisation check at the corpus directory. It now warns and
names the path. The purge module's line budget is raised by 8 with a
`budget_history` note (the project's documented option). |
`cli/subcommands/_purge.py`, `tools/check_module_size.py` |

`config.py` (0 lines of headroom on the module-size ratchet) and
`trainer.py` change only by in-place string edits; new warnings live in
`cli/_config_load.py`.

### Merge-conflict handling

The six branches touched `CHANGELOG.md`, `tests/test_cli.py` (two
appended test classes) and the import line in `cli/_dispatch.py`; all
three were resolved by keeping both sides (the import line now imports
`_disable_hub_telemetry` and `_warn_inert_fsdp_options`). The affected
test modules pass together (621 tests).

## Type

- [x] Bug fix
- [x] Dependency

## Testing

- The full `CLAUDE.md` gauntlet from a clean worktree of this branch:
exit 0.
- Per-fix tests as in the original pull requests, including: a real tiny
Llama on CPU for `rope_scaling` (linear, dynamic, yarn); the traversal
reproduction (**fails on transformers 5.3.0**, passes on 5.10.1); all 32
accepted FSDP variants building real `TrainingArguments`.
- transformers-facing test modules pass on a 5.10.1 / `huggingface_hub`
1.5.0 overlay (443 tests). The nightly minimum-versions job runs on its
schedule after merge; I did not dispatch it.

## Checklist

- [x] English and Turkish mirrors updated together
- [x] CHANGELOG entries under `[Unreleased]`
- [x] Tests added

🤖 Generated with [Claude Code](https://claude.com/claude-code)

## Summary by Sourcery

Harden CLI behavior and dependency compatibility across model loading,
privacy, retention, distributed training, and purge workflows.

Bug Fixes:
- Restore Transformers 5 compatibility for RoPE scaling and
sliding-window model configuration.
- Disable Hugging Face Hub telemetry by default for CLI runs while
preserving explicit operator opt-in.
- Warn when reserved retention enforcement modes are configured instead
of implying unsupported enforcement behavior.
- Reject unsupported FSDP1-only strategies during configuration and warn
about FSDP2 options that have no effect.
- Warn when purge uses the corpus directory as its implicit output
directory and clarify the memorisation-check scope.

Enhancements:
- Document the actual retention, FSDP, telemetry, and purge behavior in
English and Turkish user-facing documentation.

Build:
- Raise the minimum runtime dependencies to transformers>=5.10.1 and
huggingface_hub>=1.5.0 to address relevant security advisories.
- Update the nightly minimum-version job and packaging checks to enforce
the new dependency floors.

Tests:
- Add coverage for telemetry initialization, reserved retention
warnings, FSDP validation, purge output-directory warnings, Transformers
5 RoPE handling, and chat-template path traversal protection.

Chores:
- Increase the purge module size budget for the new output-directory
warning.

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant