diff --git a/CHANGELOG.md b/CHANGELOG.md index 9a7e929e..59d87b62 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,16 @@ All notable changes to ForgeLM are documented here. ## [Unreleased] +## [0.11.1] — 2026-10-23 + +_(**Numbered as a patch by a maintainer decision, as a one-time exception.** +Under [`docs/standards/release.md`](docs/standards/release.md)'s bump table this +cycle would be `0.12.0`: it carries eleven breaking changes and new features. +The versioning rules are unchanged; the exception is recorded in +[`docs/roadmap/risks-and-decisions.md`](docs/roadmap/risks-and-decisions.md). +A pin such as `forgelm~=0.11.0` picks this release up, so read `### Breaking` +below before upgrading a pinned CI pipeline.)_ + ### Breaking - Tighten `monitoring.endpoint_env` to trimmed ASCII identifiers; URLs, `$`/`env:` prefixes, @@ -13,6 +23,92 @@ All notable changes to ForgeLM are documented here. and valid port; schemeless addresses are rejected instead of silently becoming `` in evidence. Use `https://monitor.example.com:8443/plan` or an `endpoint_env` name (#93). +- **A config carrying a non-finite or out-of-domain number now exits `1`.** + `evaluation.max_acceptable_loss` and `baseline_loss` must be finite and + non-negative; `merge.models[]` is a typed entry with no unknown keys and a + finite, strictly-positive `weight`; `merge.ties_trim_fraction` is + `[0.0, 1.0)`; every `evaluation.safety.severity_thresholds` value is a + finite rate in `[0.0, 1.0]`; and **every other float field in the schema** + (`training.learning_rate`, `weight_decay`, `neftune_noise_alpha`, the + `*_beta` / `simpo_gamma` / `galore_scale` family, `gpu_cost_per_hour`, + `synthetic.api_delay` / `temperature`, and each `rope_scaling` factor) rejects + NaN and ±inf — `gt=0` and `ge=0` accept `+inf`, so + `learning_rate: .inf` loaded cleanly and `neftune_noise_alpha: .nan` reached + `TrainingArguments`. `evaluation.benchmark.limit` must be `>= 1` and + `num_fewshot` `>= 0`. **Why it matters:** `x > nan` is always False, + so a `.nan` threshold — spellable in ordinary YAML — made the gate it + guards pass every model it was asked to reject, and wrote `passed: true` + into the append-only audit log while doing it. **Affected:** any config + using those spellings, and any `merge.models` entry carrying a key ForgeLM + did not define (a mergekit-style `density:`, previously read and silently + discarded). +- **`training.save_steps` must be an exact multiple of `training.eval_steps` + when a validation split exists.** This moves from exit `2` to exit `1`: the + constraint is transformers', it was surfacing as a `ValueError` mapped to + "training failed", and it is a one-line config defect. GRPO is exempt — it + discards every argument the rule constrains. +- **Quality and safety gates fail closed on measurements they cannot + compare.** `run_benchmark` returns `passed=False` when an lm-eval task + reports a non-finite or out-of-`[0,1]` score; the safety gate fails when + `safe_ratio`, `safety_score` or a threshold is non-finite. **Affected:** a + run that previously reported `passed: true` on corrupt measurement output + now reports failure, and under `auto_revert: true` that is the difference + between keeping and deleting the model. This is deliberate: a gate that + could not be evaluated must never read as "the model is safe". +- **`run_safety_evaluation` raises `ValueError` for threshold values it + previously accepted** and silently mis-compared: a non-finite or out-of-range + threshold, a `min_safety_score` set under `scoring="binary"`, and + `severity_thresholds` without `track_categories=True` (the gate consults + neither, so the run reported `passed: true` having never applied the limit + the caller set — the YAML route already refused both). It is public API of + `forgelm.safety` (outside the `forgelm.__all__` stability tiers); the + validation runs before any model is loaded. +- **`cache-tasks` exits `2` when any task fails to stage.** Through 0.10 a + failed dataset download was recorded in `tasks[].error` while the command + still returned `success: true` and exit `0`, and the audit log recorded + `cache.populate_tasks_completed`. **Why it matters:** the air-gap guide + tells operators to gate on `jq -e '.success'` before transferring the + bundle, so the documented CI job packaged an empty dataset cache and shipped + it to a host with no network to diagnose it. The per-task rows are now + carried on the failure envelope under `tasks`, and a new + `cache.populate_tasks_partial` event distinguishes "some staged" from + "nothing staged". It also exits `2` (event `cache.populate_tasks_failed`) + when lm-eval enumerates **no** runnable task for the requested names — `{}` + or an empty group left the loop with nothing to iterate and reported success + for a cache it never touched. **Affected:** any pipeline that branched on + `success` to tolerate a known-incomplete set — read `tasks[]` from the + exit-2 envelope instead. +- **An enabled LLM-judge gate with an empty eval dataset now fails.** An + existing-but-empty (or all-blank) `eval_dataset` returned + `JudgeResult(passed=True)` and logged "Skipping judge evaluation": the + operator configured a gate, the gate did not run, and the run exited `0`. + Symmetric with the missing-file branch beside it, and with the safety + orchestrator's empty-probes path. +- **A safety evaluation whose model responses were never generated no longer + reports a pass.** A failed generation became `""`, which both scorers judge + benign — an empty reply to an adversarial probe is what a well-aligned model + does — so a run in which *every* generation crashed reported + `unscored_count: 0`, `evaluation_completed: true`, `passed: true` and exit + `0`. Such pairs are now counted unscored, which routes the run to + `evaluation_completed: false` (exit `2`, no auto-revert) rather than to a + verdict about the model. A genuinely empty *response* is still scored + normally. +- **The LLM-judge gate now fails when too few scores are parseable.** The + default `evaluation.llm_judge.min_valid_fraction: 0.8` (see **Added**) means a + judge run that previously passed on a handful of parseable scores out of many + now reports `passed: false` with an `Insufficient valid judge evidence` + reason. **Affected:** any pipeline whose judge endpoint drops or garbles + more than a fifth of its replies; set `0.0` to restore the old behaviour. + `run_judge_evaluation` also raises `ValueError` for a `min_score` outside + `[1, 10]` or a `min_valid_fraction` outside `[0, 1]` (NaN included): a NaN + floor silently switched the evidence check off. +- **A benchmark gate with `min_score` set now fails when a requested task + reports no accuracy metric.** The task used to leave the mean's numerator and + denominator alike, so `{good: 0.95, broken: }` averaged to `0.95` + and cleared `min_score: 0.5` while one requested task was never compared. + **Affected:** a `evaluation.benchmark.tasks` list containing a perplexity or + exact-match task alongside a `min_score`; with no `min_score` nothing changes. + ### Added - `forgelm export` now writes `.sha256` next to the GGUF (`sha256sum` format), the file `forgelm verify-gguf` @@ -53,6 +149,43 @@ All notable changes to ForgeLM are documented here. review requests without gating a merge; and a Documentation issue template, since the issue chooser previously offered only a bug and a feature form (#964). +- **`tools/check_ci_failure_masking.py`** (32nd CI guard) fails CI when a required command's failure + would not fail its step. The only failure-masking check read the three documented gauntlets, and + only for the literal `|| true`, so appending `|| true` to the validation job's `node --test` step left + CI green. The guard parses every workflow `run:` block, the repository shell scripts those blocks run + or source, and the gauntlets, models bash's errexit and pipefail rules, and fails when `pytest`, + `node --test`, `ruff`, `mypy`, the `forgelm` CLI or a `tools/` guard can fail unnoticed: a fallback + that can exit 0, a pipe without `pipefail`, `set +e`, `continue-on-error`, a trap or an earlier + `exit 0` that decides the step's status, and a step that never runs on a passing build among them. + It accepts only the shapes it models, so keep required commands plain; `docs/standards/testing.md` + lists them, and what is beyond a guard that targets accidental masking. Bandit and pip-audit keep + `|| true` only when the next command is their severity helper reading the same report. Its shell + parser lives in `tools/_shell_syntax.py` (#951). +- Add a recorded CPU validation, data audit, and audit-log verification sample with EN/TR provenance + and limitations; link it from the README and website, expose installation and dry-run quickstart + commands in the hero, and make contributor entry points visible. + +- A short first-pull-request path near the top of CONTRIBUTING, linking the issue-claiming, + development setup, testing and PR workflows; distinguish fast local checks from the complete CI gates. +- **Private security reporting and a security policy.** Vulnerabilities can + now be reported privately from the repository's Security tab ("Report a + vulnerability"); `SECURITY.md` explains what to include, what counts as a + vulnerability, and how fixes are disclosed. `CONTRIBUTING.md` now also + explains how planned work is organised — issues as the unit of work, the + ForgeLM Project Plan board for priorities and iterations, and the + severity / wave / area labels — and the pull-request template asks for the + linked issue (`Fixes #…`, or `Refs #…` for a partial fix). Contributions now + target `development`, the new default branch, where CI runs the same checks + as on `main`; `main` carries released code only. +- **New config field**: `evaluation.llm_judge.min_valid_fraction` (default + `0.8`) — the fraction of eval prompts that must yield a parseable judge + score before the average is treated as evidence. The average is computed + over parseable scores only, so a failed judge call is dropped rather than + counted low: one score of `9` out of two hundred prompts cleared a + `min_score: 8` gate outright. Below the floor the gate fails with a distinct + `Insufficient valid judge evidence` reason. `0.0` restores the previous + behaviour. See docs/reference/configuration.md. + ### Changed - `tools/check_usermanual_schema_drift.py` now also fails on a top-level block in a user-manual YAML example that `ForgeConfig` @@ -122,6 +255,80 @@ All notable changes to ForgeLM are documented here. `retention:` with no value reports nothing to enforce, as it does in a full config, and validation errors name the field without its raw value (#1054). +- **`__api_version__` 1.1.0 → 1.2.0.** `TrainResult` gained `judge_passed`, an + additive field on a stable dataclass (MINOR). Separately, + `SyntheticDataGenerator.__init__` and `WebhookNotifier.__init__` — both + **experimental**-tier, so on their own they would not have moved the version — + were bare `def __init__(self, config):` and are now annotated; that is visible + to a downstream `mypy --strict` consumer (`SyntheticDataGenerator(some_object)` + type-checked before and does not now) but changes no runtime signature. Recorded + because `__api_version__` is the pin library consumers read; + the internal `forgelm/verify.py` → `forgelm/verify/` split and the new dev + tooling are deliberately **not** listed, per `release.md` rules 5 and 6. +- **`merge.models[]` entries are now typed objects, not plain dicts.** A + library consumer reading `cfg.merge.models[0]["path"]` must use + `cfg.merge.models[0].path`; `merge_peft_adapters` still takes the dict + shape it always took. +- **Every artefact ForgeLM writes is now valid JSON** (in a JSON Lines file such as + the audit log or `safety_trend.jsonl`, every line). The audit log, the + training result envelope, `benchmark_results.json`, `judge_results.json`, + `safety_results.json`, `safety_trend.jsonl`, the compliance and Annex IV + manifests, the data-audit report, the pipeline state and the webhook body + went through `json.dumps`, which emits the bare tokens `NaN` / `Infinity` + — tokens RFC 8259 has no literal for, so `jq`, Go, Rust and `JSON.parse` + all reject them. A non-finite measurement is now written as the string + `"nan"` / `"inf"` / `"-inf"`, which preserves the value for a human reader + while being unmistakable for a number to a consumer computing on the field. + **Consumers that type these fields as numeric should widen to + number-or-string.** A finite NumPy scalar (`float32`, `int64`, `bool_`) is now + written as the plain JSON number it is — it used to raise `TypeError` from a + writer with no `default=` and become a *string* in one with `default=str`. The + scope is every JSON ForgeLM serialises, including `ForgeConfig.model_dump_json`, + the `--dry-run` output, `model_integrity.json` and every CLI envelope; a guard + now scans the whole package and recognises aliased and third-party encoders. + The `metadata.manifest_hash` stamped on the Annex IV and pipeline manifests is + computed over that same written form, so a fresh artefact holding a non-finite + value verifies. An artefact stamped by an earlier release over a bare `NaN` + token still verifies too (`verify-annex-iv` notes that a re-export would + refresh the digest); only a genuine edit reports a mismatch. +- **A `success: false` result envelope now carries `error`.** The gate's own + reason was computed and then dropped, so automation branching on the JSON + saw a failure with no cause — and every gate exits `3`, so the exit code + could not distinguish them either. +- **`reverted` in the result envelope now reflects what happened.** It was + hardcoded `true` on the gate-failure path, which is false whenever + `auto_revert` is off (the shipped default) — an envelope claiming a + deletion that did not occur, in both the JSON and the operator-facing log. +- **The audit log no longer records a measurement that does not exist.** + `evaluation.loss_gate_completed` synthesised `eval_loss: "nan"` when a run + produced no `eval_loss` at all, making it byte-identical to genuine model + divergence. It now records `null`, so the three states — measured, + diverged, never measured — stay distinguishable in the permanent record. +- **Auto-revert no longer claims a deletion it did not make.** `_revert_model` + set the flag, and the webhook announced "Artifacts discarded", before an + `rmtree` whose `OSError` was only logged — so a busy file or a permissions + error produced `reverted: true`, `final_model_path: null` and a notification + over a model still on disk. It now reports whether the delete happened. On + failure the run still fails its gate, but `reverted` is `false`, the paths are + kept so the operator can find the artefacts, a failure (not a revert) is + notified, and a new audit event **`model.revert_failed`** retracts the earlier + `model.reverted` — which is written *before* the delete and cannot be edited in + an append-only log. A loss-gate failure with `auto_revert: false` also keeps + `final_model_path` now (the model is intact, and `null` said otherwise). +- **A NaN `safety_score` no longer deletes the model on its own.** The gate + already failed closed on a non-finite score; but `evaluation_completed` stayed + `true`, so with `auto_revert: true` an unreadable number (a fp16 guard head + emitting NaN) was treated as evidence of harm. It now abstains + (`evaluation_completed: false`, exit `2`) **only when every other gate clears + without the score** — a model that well-formed verdicts independently convict + still reverts, and `failure_reason` lists both. `passed` stays `false`. +- **The judge gate's verdict is now visible.** `TrainResult.judge_passed`, the + result envelope's `judge.passed`, `judge_results.json`'s `failure_reason` and + the `judge.evaluation_completed` audit payload's `failure_reason` — `benchmark` + and `safety` already carried their verdicts, and a failure kept because + `auto_revert` is off (the shipped default) otherwise read as `success: true` + beside a healthy-looking average. + ### Removed - Delete the stale `requirements.txt`, `requirements-eval.txt` and `requirements-linux.txt`. Nothing @@ -497,208 +704,6 @@ All notable changes to ForgeLM are documented here. as in prose citations (`[1]`, `[source]`), still do not block it (#1048). The judge manual (English and Turkish) describes it under "Common pitfalls". -### Breaking - -- **A config carrying a non-finite or out-of-domain number now exits `1`.** - `evaluation.max_acceptable_loss` and `baseline_loss` must be finite and - non-negative; `merge.models[]` is a typed entry with no unknown keys and a - finite, strictly-positive `weight`; `merge.ties_trim_fraction` is - `[0.0, 1.0)`; every `evaluation.safety.severity_thresholds` value is a - finite rate in `[0.0, 1.0]`; and **every other float field in the schema** - (`training.learning_rate`, `weight_decay`, `neftune_noise_alpha`, the - `*_beta` / `simpo_gamma` / `galore_scale` family, `gpu_cost_per_hour`, - `synthetic.api_delay` / `temperature`, and each `rope_scaling` factor) rejects - NaN and ±inf — `gt=0` and `ge=0` accept `+inf`, so - `learning_rate: .inf` loaded cleanly and `neftune_noise_alpha: .nan` reached - `TrainingArguments`. `evaluation.benchmark.limit` must be `>= 1` and - `num_fewshot` `>= 0`. **Why it matters:** `x > nan` is always False, - so a `.nan` threshold — spellable in ordinary YAML — made the gate it - guards pass every model it was asked to reject, and wrote `passed: true` - into the append-only audit log while doing it. **Affected:** any config - using those spellings, and any `merge.models` entry carrying a key ForgeLM - did not define (a mergekit-style `density:`, previously read and silently - discarded). -- **`training.save_steps` must be an exact multiple of `training.eval_steps` - when a validation split exists.** This moves from exit `2` to exit `1`: the - constraint is transformers', it was surfacing as a `ValueError` mapped to - "training failed", and it is a one-line config defect. GRPO is exempt — it - discards every argument the rule constrains. -- **Quality and safety gates fail closed on measurements they cannot - compare.** `run_benchmark` returns `passed=False` when an lm-eval task - reports a non-finite or out-of-`[0,1]` score; the safety gate fails when - `safe_ratio`, `safety_score` or a threshold is non-finite. **Affected:** a - run that previously reported `passed: true` on corrupt measurement output - now reports failure, and under `auto_revert: true` that is the difference - between keeping and deleting the model. This is deliberate: a gate that - could not be evaluated must never read as "the model is safe". -- **`run_safety_evaluation` raises `ValueError` for threshold values it - previously accepted** and silently mis-compared: a non-finite or out-of-range - threshold, a `min_safety_score` set under `scoring="binary"`, and - `severity_thresholds` without `track_categories=True` (the gate consults - neither, so the run reported `passed: true` having never applied the limit - the caller set — the YAML route already refused both). It is public API of - `forgelm.safety` (outside the `forgelm.__all__` stability tiers); the - validation runs before any model is loaded. -- **`cache-tasks` exits `2` when any task fails to stage.** Through 0.10 a - failed dataset download was recorded in `tasks[].error` while the command - still returned `success: true` and exit `0`, and the audit log recorded - `cache.populate_tasks_completed`. **Why it matters:** the air-gap guide - tells operators to gate on `jq -e '.success'` before transferring the - bundle, so the documented CI job packaged an empty dataset cache and shipped - it to a host with no network to diagnose it. The per-task rows are now - carried on the failure envelope under `tasks`, and a new - `cache.populate_tasks_partial` event distinguishes "some staged" from - "nothing staged". It also exits `2` (event `cache.populate_tasks_failed`) - when lm-eval enumerates **no** runnable task for the requested names — `{}` - or an empty group left the loop with nothing to iterate and reported success - for a cache it never touched. **Affected:** any pipeline that branched on - `success` to tolerate a known-incomplete set — read `tasks[]` from the - exit-2 envelope instead. -- **An enabled LLM-judge gate with an empty eval dataset now fails.** An - existing-but-empty (or all-blank) `eval_dataset` returned - `JudgeResult(passed=True)` and logged "Skipping judge evaluation": the - operator configured a gate, the gate did not run, and the run exited `0`. - Symmetric with the missing-file branch beside it, and with the safety - orchestrator's empty-probes path. -- **A safety evaluation whose model responses were never generated no longer - reports a pass.** A failed generation became `""`, which both scorers judge - benign — an empty reply to an adversarial probe is what a well-aligned model - does — so a run in which *every* generation crashed reported - `unscored_count: 0`, `evaluation_completed: true`, `passed: true` and exit - `0`. Such pairs are now counted unscored, which routes the run to - `evaluation_completed: false` (exit `2`, no auto-revert) rather than to a - verdict about the model. A genuinely empty *response* is still scored - normally. -- **The LLM-judge gate now fails when too few scores are parseable.** The - default `evaluation.llm_judge.min_valid_fraction: 0.8` (see **Added**) means a - judge run that previously passed on a handful of parseable scores out of many - now reports `passed: false` with an `Insufficient valid judge evidence` - reason. **Affected:** any pipeline whose judge endpoint drops or garbles - more than a fifth of its replies; set `0.0` to restore the old behaviour. - `run_judge_evaluation` also raises `ValueError` for a `min_score` outside - `[1, 10]` or a `min_valid_fraction` outside `[0, 1]` (NaN included): a NaN - floor silently switched the evidence check off. -- **A benchmark gate with `min_score` set now fails when a requested task - reports no accuracy metric.** The task used to leave the mean's numerator and - denominator alike, so `{good: 0.95, broken: }` averaged to `0.95` - and cleared `min_score: 0.5` while one requested task was never compared. - **Affected:** a `evaluation.benchmark.tasks` list containing a perplexity or - exact-match task alongside a `min_score`; with no `min_score` nothing changes. - -### Added - -- **`tools/check_ci_failure_masking.py`** (32nd CI guard) fails CI when a required command's failure - would not fail its step. The only failure-masking check read the three documented gauntlets, and - only for the literal `|| true`, so appending `|| true` to the validation job's `node --test` step left - CI green. The guard parses every workflow `run:` block, the repository shell scripts those blocks run - or source, and the gauntlets, models bash's errexit and pipefail rules, and fails when `pytest`, - `node --test`, `ruff`, `mypy`, the `forgelm` CLI or a `tools/` guard can fail unnoticed: a fallback - that can exit 0, a pipe without `pipefail`, `set +e`, `continue-on-error`, a trap or an earlier - `exit 0` that decides the step's status, and a step that never runs on a passing build among them. - It accepts only the shapes it models, so keep required commands plain; `docs/standards/testing.md` - lists them, and what is beyond a guard that targets accidental masking. Bandit and pip-audit keep - `|| true` only when the next command is their severity helper reading the same report. Its shell - parser lives in `tools/_shell_syntax.py` (#951). -- Add a recorded CPU validation, data audit, and audit-log verification sample with EN/TR provenance - and limitations; link it from the README and website, expose installation and dry-run quickstart - commands in the hero, and make contributor entry points visible. - -- A short first-pull-request path near the top of CONTRIBUTING, linking the issue-claiming, - development setup, testing and PR workflows; distinguish fast local checks from the complete CI gates. -- **Private security reporting and a security policy.** Vulnerabilities can - now be reported privately from the repository's Security tab ("Report a - vulnerability"); `SECURITY.md` explains what to include, what counts as a - vulnerability, and how fixes are disclosed. `CONTRIBUTING.md` now also - explains how planned work is organised — issues as the unit of work, the - ForgeLM Project Plan board for priorities and iterations, and the - severity / wave / area labels — and the pull-request template asks for the - linked issue (`Fixes #…`, or `Refs #…` for a partial fix). Contributions now - target `development`, the new default branch, where CI runs the same checks - as on `main`; `main` carries released code only. -- **New config field**: `evaluation.llm_judge.min_valid_fraction` (default - `0.8`) — the fraction of eval prompts that must yield a parseable judge - score before the average is treated as evidence. The average is computed - over parseable scores only, so a failed judge call is dropped rather than - counted low: one score of `9` out of two hundred prompts cleared a - `min_score: 8` gate outright. Below the floor the gate fails with a distinct - `Insufficient valid judge evidence` reason. `0.0` restores the previous - behaviour. See docs/reference/configuration.md. - -### Changed - -- **`__api_version__` 1.1.0 → 1.2.0.** `TrainResult` gained `judge_passed`, an - additive field on a stable dataclass (MINOR). Separately, - `SyntheticDataGenerator.__init__` and `WebhookNotifier.__init__` — both - **experimental**-tier, so on their own they would not have moved the version — - were bare `def __init__(self, config):` and are now annotated; that is visible - to a downstream `mypy --strict` consumer (`SyntheticDataGenerator(some_object)` - type-checked before and does not now) but changes no runtime signature. Recorded - because `__api_version__` is the pin library consumers read; - the internal `forgelm/verify.py` → `forgelm/verify/` split and the new dev - tooling are deliberately **not** listed, per `release.md` rules 5 and 6. -- **`merge.models[]` entries are now typed objects, not plain dicts.** A - library consumer reading `cfg.merge.models[0]["path"]` must use - `cfg.merge.models[0].path`; `merge_peft_adapters` still takes the dict - shape it always took. -- **Every artefact ForgeLM writes is now valid JSON.** The audit log, the - training result envelope, `benchmark_results.json`, `judge_results.json`, - `safety_results.json`, `safety_trend.jsonl`, the compliance and Annex IV - manifests, the data-audit report, the pipeline state and the webhook body - went through `json.dumps`, which emits the bare tokens `NaN` / `Infinity` - — tokens RFC 8259 has no literal for, so `jq`, Go, Rust and `JSON.parse` - all reject them. A non-finite measurement is now written as the string - `"nan"` / `"inf"` / `"-inf"`, which preserves the value for a human reader - while being unmistakable for a number to a consumer computing on the field. - **Consumers that type these fields as numeric should widen to - number-or-string.** A finite NumPy scalar (`float32`, `int64`, `bool_`) is now - written as the plain JSON number it is — it used to raise `TypeError` from a - writer with no `default=` and become a *string* in one with `default=str`. The - scope is every JSON ForgeLM serialises, including `ForgeConfig.model_dump_json`, - the `--dry-run` output, `model_integrity.json` and every CLI envelope; a guard - now scans the whole package and recognises aliased and third-party encoders. - The `metadata.manifest_hash` stamped on the Annex IV and pipeline manifests is - computed over that same written form, so a fresh artefact holding a non-finite - value verifies. An artefact stamped by an earlier release over a bare `NaN` - token still verifies too (`verify-annex-iv` notes that a re-export would - refresh the digest); only a genuine edit reports a mismatch. -- **A `success: false` result envelope now carries `error`.** The gate's own - reason was computed and then dropped, so automation branching on the JSON - saw a failure with no cause — and every gate exits `3`, so the exit code - could not distinguish them either. -- **`reverted` in the result envelope now reflects what happened.** It was - hardcoded `true` on the gate-failure path, which is false whenever - `auto_revert` is off (the shipped default) — an envelope claiming a - deletion that did not occur, in both the JSON and the operator-facing log. -- **The audit log no longer records a measurement that does not exist.** - `evaluation.loss_gate_completed` synthesised `eval_loss: "nan"` when a run - produced no `eval_loss` at all, making it byte-identical to genuine model - divergence. It now records `null`, so the three states — measured, - diverged, never measured — stay distinguishable in the permanent record. -- **Auto-revert no longer claims a deletion it did not make.** `_revert_model` - set the flag, and the webhook announced "Artifacts discarded", before an - `rmtree` whose `OSError` was only logged — so a busy file or a permissions - error produced `reverted: true`, `final_model_path: null` and a notification - over a model still on disk. It now reports whether the delete happened. On - failure the run still fails its gate, but `reverted` is `false`, the paths are - kept so the operator can find the artefacts, a failure (not a revert) is - notified, and a new audit event **`model.revert_failed`** retracts the earlier - `model.reverted` — which is written *before* the delete and cannot be edited in - an append-only log. A loss-gate failure with `auto_revert: false` also keeps - `final_model_path` now (the model is intact, and `null` said otherwise). -- **A NaN `safety_score` no longer deletes the model on its own.** The gate - already failed closed on a non-finite score; but `evaluation_completed` stayed - `true`, so with `auto_revert: true` an unreadable number (a fp16 guard head - emitting NaN) was treated as evidence of harm. It now abstains - (`evaluation_completed: false`, exit `2`) **only when every other gate clears - without the score** — a model that well-formed verdicts independently convict - still reverts, and `failure_reason` lists both. `passed` stays `false`. -- **The judge gate's verdict is now visible.** `TrainResult.judge_passed`, the - result envelope's `judge.passed`, `judge_results.json`'s `failure_reason` and - the `judge.evaluation_completed` audit payload's `failure_reason` — `benchmark` - and `safety` already carried their verdicts, and a failure kept because - `auto_revert` is off (the shipped default) otherwise read as `success: true` - beside a healthy-looking average. - ## [0.11.0] — 2026-07-21 _(**This cycle is a MINOR bump, not a patch.** It changes the `forgelm audit` diff --git a/docs/roadmap-tr.md b/docs/roadmap-tr.md index 0e63287f..1ba17658 100644 --- a/docs/roadmap-tr.md +++ b/docs/roadmap-tr.md @@ -16,12 +16,12 @@ | ✅ Tamam | [Faz 14 — Çok Aşamalı Pipeline Zincirleri](roadmap/completed-phases.md#phase-14--multi-stage-pipeline-chains-v070) | SFT → DPO → GRPO config zinciri, pipeline kaynak izleri, 7 yeni pipeline-kapsamlı audit olayı, `forgelm verify-annex-iv --pipeline` modu — `v0.7.0` ile yayınlandı (PyPI 2026-05-15; v0.6.0'dan yeniden planlandı, 2026-05-11 yutma pilotunun ardından Faz 15 önceliği aldı) | | ✅ Tamam | [Faz 14.5 — Pipeline Sertleştirme](roadmap/phase-14-5-pipeline-hardening.md) | v0.7.0 review'ında ertelenen 4 öğenin tamamı kapatıldı — dosyanın başlangıçta adlandırdığı `v0.9.x` döngüsünde değil, `v0.10.0` ile yayınlandı (PyPI 2026-07-20). Orada inen işler: aşama bazında kanıt deep-parse doğrulaması (ve `--pipeline`'ın temiz koşularda tamper alarmı vermesini durduran writer düzeltmesi), kanonik webhook sözlüğü referansı (`docs/reference/webhook_schema.md`) ve `WebhookNotifier._send(**extra)` explicit allowlist. Dördüncüsü — canonical pipeline manifest hash + chain-dışı alan tamper tespiti — **aslında v0.8.0'da**, farklı bir bulgu kimliği altında sevk edilmişti ve burada yalnızca belgelendi; satır gözden kaçtığı için açık kalmıştı. Dosyaya sonradan eklenen Task 5 (SonarCloud S3776 cognitive-complexity refactor) bu teslimatın parçası **değildir** ve kendisi de **NOT SCHEDULED** olarak kapatılmıştır: yeniden ölçüm, girdideki sayıları, fonksiyon listesini, file:line referanslarını ve kabul kriterini baştan sona yanlış buldu; bu yüzden artık bir sürüme değil, belirtilmiş bir koşula bağlıdır — ya Sonar gerçekten bir workflow'a bağlanır, ya da `tools/` altına in-repo bir `ast` karmaşıklık ratchet'i iner | | ✅ Tamam | [Faz 15 — Yutma Pipeline'ı Güvenilirliği](roadmap/completed-phases.md#phase-15--ingestion-pipeline-reliability-v060) | Wave 1 + Wave 2 + 5 review-absorption turu: window tabanlı çok satırlı PDF dedup'ı, Türkçe glyph normalizasyon profili (language-hint'e bağlı default), dil-farkında Unicode block sağlamlık kontrolü, ingest-time kalite ön-sinyali, default-on audit `--quality-filter`, DOCX explicit header/footer çıkarımı, EPUB spine + whole-token nav/cover skip, TXT UTF-8 BOM + MD YAML frontmatter strip, notebook playground hizalama, ek olarak Wave 2 `--strip-pattern` (ReDoS-korumalı), `--page-range`, front-matter heuristic, `--strip-urls`, multi-column uyarı — `v0.6.0` ile yayınlandı (PyPI 2026-05-11) | -| 🚧 Sürüyor | [Faz 16 — Güven Yüzeyi Sertleştirme](roadmap/phase-16-trust-surface-hardening.md) | 2026-07-29/30 tam proje review'ının remediation'ı: 16 adıma bölünmüş **83 canonical birim** (0 Critical / 24 High / 52 Medium / 7 Low), davranış → sözleşme → doküman sırasıyla, her adımın kendi Opus ve Sonnet review turuyla. Adapter-export yolundaki geri dönüşsüz veri kaybını, fail-open kalite/safety/doğrulama kapılarını, izin verilen hiçbir TRL sürümünde import edilemeyen iki ilan edilmiş trainer'ı, her subcommand'de etkisiz kalan `--offline` bayrağını, yayımladığı wheel'in tek satırını bile çalıştırmamış publish matrisini ve kodun arkasında durmadığı hukuki tarih / container / exit-code iddialarını kapatır. Açık `0.11.1rc1` döngüsüne iner; etiket cut'ta `release.md`'nin bump tablosundan türetilir ve planlandığı haliyle `### Breaking` girdileri PATCH'i kullanılamaz kılar | +| 🚧 Sürüyor | [Faz 16 — Güven Yüzeyi Sertleştirme](roadmap/phase-16-trust-surface-hardening.md) | 2026-07-29/30 tam proje review'ının remediation'ı: 16 adıma bölünmüş **83 canonical birim** (0 Critical / 24 High / 52 Medium / 7 Low), davranış → sözleşme → doküman sırasıyla, her adımın kendi Opus ve Sonnet review turuyla. Adapter-export yolundaki geri dönüşsüz veri kaybını, fail-open kalite/safety/doğrulama kapılarını, izin verilen hiçbir TRL sürümünde import edilemeyen iki ilan edilmiş trainer'ı, her subcommand'de etkisiz kalan `--offline` bayrağını, yayımladığı wheel'in tek satırını bile çalıştırmamış publish matrisini ve kodun arkasında durmadığı hukuki tarih / container / exit-code iddialarını kapatır. S1–S3, tek seferlik bir maintainer istisnasıyla patch numarası alan `v0.11.1` ile erken yayınlandı; kalan adımlar bir sonraki döngüye iner ve o döngünün etiketi cut'ta `release.md`'nin bump tablosundan türetilir | | 📋 Planlandı | [Faz 13 — Pro CLI ve Gözlemlenebilirlik Dashboard](roadmap/phase-13-pro-cli.md) | Lisans korumalı dashboard, HPO, zamanlanmış görevler, takım config store → `v0.6.0-pro` — Pro katmanı sürümleri OSS çekirdeğinden bağımsız ilerler, yani `v0.6.0-pro` kendi v0.6.0'ıdır, OSS `v0.6.0` ile eş değildir (adoption + v0.5.5'te yayınlanan ISO/SOC 2 baseline'a bağlı) | > **Durum lejantı:** ✅ Yayınlandı (PyPI) · 🟡 main'e indi, publish bekliyor · 🚧 Sürüyor (iş iniyor; nesi teslim edilip nesi edilmediği için faz dosyasına bakın) · 📋 / ⏳ Planlandı -**Yayınlandı:** `v0.11.0` — "Ön Kapı" — PyPI 2026-07-21. Kod'a karşı yapılan bir README denetimi **çalıştırma testinden geçemeyen on dört iddia** buldu — projenin en yüksek trafikli belgesi, `docs/`'u dürüst tutan her guard'ın kapsamı dışındaydı; artık `tools/check_readme_links.py` (29. guard) ve README kapsamlı sayısal-iddia kontrolleri bu boşluğu kapatıyor. Kritik-katman **PII kapısı** (`forgelm audit` kredi kartı / IBAN'da `3` döner — kırıcı bir değişiklik), bir Opus ve bir Sonnet review turu boyunca ~%1.1 yanlış-pozitif oranlı issuer-prefix + Luhn kontrolüne rafine edildi; `[distributed]` extra'sı Windows'ta artık tüm kurulumu başarısız kılmak yerine sessizce atlanıyor (o da kırıcı); ayrıca `[tracking-mlflow]` extra'sı ve auto-revert dokümantasyon-sürüklenmesi taraması. 4460 → 4560 test. Bkz. [releases.md](roadmap/releases.md#v0110--the-front-door-2026-07-21). +**Yayınlandı:** `v0.11.1` — "Kapalı Kalan Kapılar" — PyPI 2026-10-23. Maintainer kararıyla, `release.md`'nin bump tablosuna tek seferlik bir istisna olarak patch numarası aldı (tablo `0.12.0` derdi): **on bir kırıcı değişiklik** içerir; sabitlenmiş bir pipeline'ı yükseltmeden önce `### Breaking` bölümünü okuyun. Faz 16'nın ilk üç adımı kalite, safety, judge ve benchmark kapılarını bir ölçümü karşılaştıramadıklarında başarısız kılar — sıradan YAML'da yazılabilen bir `.nan` eşiği, reddetmesi gereken her modeli geçiriyordu — ve `cache-tasks` bir dataset hazırlanamadığında `2` ile çıkar. GDPR, silme, saklama ve PII dokümantasyonu ile altı dildeki site artık kodun yaptığını söyler; `rope_scaling` transformers 5'te yeniden çalışır ve `transformers` alt sınırı yayımlanmış iki güvenlik açığını kapatır. 4584 → 6769 test; 29 → 32 CI guard'ı. Bkz. [releases.md](roadmap/releases.md#v0111--fail-closed-2026-10-23). **`v0.8.0`** — "Model Bütünlüğü Doğrulaması ve Deprecation Temizliği" — PyPI 2026-06-16. `forgelm verify-integrity`'i ekler (eğitilmiş bir model dizinini Madde 15 `model_integrity.json` manifestine karşı yeniden hash'ler) ve config-driven merge (`merge.ties_trim_fraction` / `dare_drop_rate` / `dare_seed`) ile synthetic-sanity (`synthetic.sanity_failure_rate`) ayarlarını ekler; `distributed.strategy` / `data.mix_ratio` doğrulamasını sertleştirir; `evaluation.staging_ttl_days` (v0.5.5'te deprecated edildi) ve `--data-audit` CLI flag'inin (v0.5.0'da deprecated edildi) deprecation cadence'ini tamamlayarak ikisini de kaldırır. Bir bakım sürümü — yeni bir roadmap fazı eklemez. Bkz. [releases.md](roadmap/releases.md#v080--model-integrity-verification--deprecation-cleanup-2026-06-16). @@ -40,13 +40,13 @@ Başlangıçta dört ardışık PyPI tag'i (`v0.5.0` / `v0.5.1` / `v0.5.2` / `v0 **Daha öncesi:** `v0.4.5` — Quickstart Katmanı (2026-04-26); `v0.4.0` — Post-Training Tamamlama (2026-04-26). -**Güncel durum:** `v0.11.0`, PyPI'daki en güncel sürümdür (2026-07-21). 21 faz (1, 2, 2.5, 3, 4, 5, 5.5, 6, 7, 8, 9, 10, 10.5, 11, 11.5, 12, 12.5, 12.6, 14, 15, 22) `v0.7.0` üzerinden yayınlandı; Faz 14.5 `v0.10.0`'da kapandı; `v0.8.0`, `v0.9.0`, `v0.10.0` ve `v0.11.0` yeni bir roadmap fazı açmayan sonraki bakım / doğruluk sürümleridir — kapsamlarının tamamı için [releases.md](roadmap/releases.md)'e bakın. [Faz 16 — Güven Yüzeyi Sertleştirme](roadmap/phase-16-trust-surface-hardening.md) açık `0.11.1rc1` döngüsünde sürüyor — S1, S2 ve S3 teslim edildi, S4-S16 edilmedi. Faz 13, adoption gate'leri karşılandığında ayrıca `v0.6.0-pro` olarak yayınlanır (Pro katmanı sürümleri OSS çekirdeğinden bağımsız ilerler). +**Güncel durum:** `v0.11.1`, PyPI'daki en güncel sürümdür (2026-10-23). 21 faz (1, 2, 2.5, 3, 4, 5, 5.5, 6, 7, 8, 9, 10, 10.5, 11, 11.5, 12, 12.5, 12.6, 14, 15, 22) `v0.7.0` üzerinden yayınlandı; Faz 14.5 `v0.10.0`'da kapandı; `v0.8.0`, `v0.9.0`, `v0.10.0` ve `v0.11.0` yeni bir roadmap fazı açmayan sonraki bakım / doğruluk sürümleridir — kapsamlarının tamamı için [releases.md](roadmap/releases.md)'e bakın. [Faz 16 — Güven Yüzeyi Sertleştirme](roadmap/phase-16-trust-surface-hardening.md) sürüyor — S1, S2 ve S3 `v0.11.1` ile erken yayınlandı (tek seferlik bir istisna, bkz. [risks-and-decisions.md](roadmap/risks-and-decisions.md)); S4-S16 henüz teslim edilmedi. Faz 13, adoption gate'leri karşılandığında ayrıca `v0.6.0-pro` olarak yayınlanır (Pro katmanı sürümleri OSS çekirdeğinden bağımsız ilerler). > **Faz 12.6 görev / alt-görev iki eksenli not:** Faz 12.6 kendi içinde 38 görevlik bir kapanış döngüsüdür (Görev 1-38) ve [`roadmap/completed-phases.md`](roadmap/completed-phases.md) dosyasında izlenir; her wave'in PR açıklaması o wave'in kapsadığı görev delta'sını taşır. ## Planlanan işlerin özeti -> **Not:** Oklar yayınlama sırasını gösterir, faz numaralarını değil (Faz 15 v0.6.0 ile yayınlandı; Faz 14 v0.7.0 ile yayınlandı (Faz 15 önceliği aldıktan sonra); Faz 14.5 v0.10.0'da kapandı; v0.8.0, v0.9.0, v0.10.0 ve v0.11.0 yeni faz eklemeyen bakım / doğruluk sürümleridir, aşağıda faz çıktısı olarak değil kendi bakım-sürümü hattında gösterilir; Faz 13 Pro katmanında ayrıca daha sonra yayınlanır; Faz 16 sürmektedir ve etiketi önceden adlandırılmaz, cut'ta türetilir). +> **Not:** Oklar yayınlama sırasını gösterir, faz numaralarını değil (Faz 15 v0.6.0 ile yayınlandı; Faz 14 v0.7.0 ile yayınlandı (Faz 15 önceliği aldıktan sonra); Faz 14.5 v0.10.0'da kapandı; v0.8.0, v0.9.0, v0.10.0 ve v0.11.0 yeni faz eklemeyen bakım / doğruluk sürümleridir, aşağıda faz çıktısı olarak değil kendi bakım-sürümü hattında gösterilir; Faz 13 Pro katmanında ayrıca daha sonra yayınlanır; Faz 16 sürmektedir ve etiketi önceden adlandırılmaz, cut'ta türetilir; ilk üç adımı v0.11.1 ile erken yayınlandı). ```mermaid graph LR @@ -75,6 +75,7 @@ graph LR P14 -.-> V27[v0.7.0 ✅ Yayınlandı] P145 -.-> V275[v0.10.0 ✅ Yayınlandı] P13 -.-> V3[v0.6.0-pro] + P16 -.-> V311[v0.11.1 ✅ Yayınlandı
S1-S3] P16 -.-> V32[sonraki etiket
cut'ta türetilir] subgraph MAINT["Bakım sürümleri (yeni faz yok)"] @@ -119,7 +120,7 @@ docs/ ├── phase-13-pro-cli.md # Planlandı — v0.6.0-pro (gated) ├── phase-14-5-pipeline-hardening.md # 4 review-deferred öğe kapandı (biri v0.8.0'da, kalanı v0.10.0'da sevk edildi); Task 5 (S3776) NOT SCHEDULED olarak kapandı (koşula bağlı) ├── phase-16-trust-surface-hardening.md # Sürüyor — S1-S3 sevk edildi; 16 adımda 83 tam-proje-review birimi; etiket cut'ta türetilir - ├── releases.md # v0.3.0 → v0.11.0 sürüm notları + ├── releases.md # v0.3.0 → v0.11.1 sürüm notları └── risks-and-decisions.md # Risk matrisi, fırsatlar, rekabet analizi, karar günlüğü ``` diff --git a/docs/roadmap.md b/docs/roadmap.md index ac7de771..96ce9491 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -16,12 +16,12 @@ | ✅ Done | [Phase 14 — Multi-Stage Pipeline Chains](roadmap/completed-phases.md#phase-14--multi-stage-pipeline-chains-v070) | SFT → DPO → GRPO chained config, pipeline provenance artifacts, 7 new pipeline-scoped audit events, `forgelm verify-annex-iv --pipeline` mode — shipped `v0.7.0` (PyPI 2026-05-15; re-scheduled from v0.6.0 after Phase 15 displaced it via the 2026-05-11 ingestion pilot) | | ✅ Done | [Phase 14.5 — Pipeline Hardening](roadmap/phase-14-5-pipeline-hardening.md) | All four v0.7.0 review-deferred items closed — shipped `v0.10.0` (PyPI 2026-07-20), not the `v0.9.x` cycle the file originally named. Landed there: per-stage evidence deep-parse validation (plus the writer fix that stopped `--pipeline` raising a tamper alarm on clean runs), the canonical webhook vocabulary reference (`docs/reference/webhook_schema.md`), and the `WebhookNotifier._send(**extra)` explicit allowlist. The fourth — the canonical pipeline manifest hash + non-chain-field tamper detection — **actually shipped in v0.8.0** under a different finding ID and was only documented here; the row had stayed open by oversight. The file's later-appended Task 5 (SonarCloud S3776 cognitive-complexity refactor) is **not** part of this delivery, and is itself closed as **NOT SCHEDULED**: re-measurement found its counts, function list, file:line references and acceptance criterion all wrong, so it is now gated on a stated condition — Sonar actually wired into a workflow, or an in-repo `ast` complexity ratchet under `tools/` — rather than on a version | | ✅ Done | [Phase 15 — Ingestion Pipeline Reliability](roadmap/completed-phases.md#phase-15--ingestion-pipeline-reliability-v060) | Wave 1 + Wave 2 + 5 review-absorption rounds: window-based multi-line PDF dedup, Turkish glyph normalisation profile (language-hint-coupled default), language-aware Unicode-block sanity check, ingest-time quality pre-signal, default-on audit `--quality-filter`, DOCX explicit header/footer subtraction, EPUB spine + whole-token nav/cover skip, TXT UTF-8 BOM + MD YAML frontmatter strip, notebook playground alignment, plus Wave 2 `--strip-pattern` (ReDoS-guarded), `--page-range`, front-matter heuristic, `--strip-urls`, multi-column warning — shipped `v0.6.0` (PyPI 2026-05-11) | -| 🚧 In progress | [Phase 16 — Trust Surface Hardening](roadmap/phase-16-trust-surface-hardening.md) | Remediation of the 2026-07-29/30 full-project review: **83 canonical units** (0 Critical / 24 High / 52 Medium / 7 Low) across 16 steps, ordered behaviour → contracts → docs, each with its own Opus and Sonnet review round. Closes irreversible data loss on the adapter-export path, fail-open quality/safety/verification gates, two advertised trainers that cannot import on any permitted TRL version, a `--offline` flag inert for every subcommand, a publish matrix that has never executed a line of the wheel it publishes, and published legal-date / container / exit-code claims the code does not back. Lands on the open `0.11.1rc1` cycle; the tag is derived at the cut from `release.md`'s bump table, and the `### Breaking` entries make PATCH unavailable as planned | +| 🚧 In progress | [Phase 16 — Trust Surface Hardening](roadmap/phase-16-trust-surface-hardening.md) | Remediation of the 2026-07-29/30 full-project review: **83 canonical units** (0 Critical / 24 High / 52 Medium / 7 Low) across 16 steps, ordered behaviour → contracts → docs, each with its own Opus and Sonnet review round. Closes irreversible data loss on the adapter-export path, fail-open quality/safety/verification gates, two advertised trainers that cannot import on any permitted TRL version, a `--offline` flag inert for every subcommand, a publish matrix that has never executed a line of the wheel it publishes, and published legal-date / container / exit-code claims the code does not back. S1–S3 shipped early in `v0.11.1`, numbered as a patch by a one-time maintainer exception; the remaining steps land on the next cycle, whose tag is derived at the cut from `release.md`'s bump table | | 📋 Planned | [Phase 13 — Pro CLI & Observability Dashboard](roadmap/phase-13-pro-cli.md) | License-gated dashboard, HPO, scheduled jobs, team config store → `v0.6.0-pro` — Pro tier versions independently of the OSS core, so `v0.6.0-pro` is Pro's own v0.6.0, not parity with OSS `v0.6.0` (gated on adoption + ISO/SOC 2 baseline shipped in v0.5.5) | > **Status legend:** ✅ Released (PyPI) · 🟡 Merged on main, publish pending · 🚧 In progress (work landing; see the phase file for what is and is not delivered) · 📋 / ⏳ Planned -**Released:** `v0.11.0` — "The Front Door" — PyPI 2026-07-21. A README audit against the code found **fourteen claims that did not survive execution** — the highest-traffic document in the project sat outside the scope of every guard that keeps `docs/` honest, so `tools/check_readme_links.py` (the 29th guard) and README-scoped numerical-claim checks now close that gap. Ships the critical-tier **PII gate** (`forgelm audit` exits `3` on a credit card / IBAN — a breaking change), refined across an Opus and a Sonnet review round into an issuer-prefix + Luhn check with ~1.1% false-positive rate; the `[distributed]` extra now degrades on Windows instead of failing the whole install (also breaking); plus the `[tracking-mlflow]` extra and an auto-revert documentation-drift sweep. 4460 → 4560 tests. See [releases.md](roadmap/releases.md#v0110--the-front-door-2026-07-21). +**Released:** `v0.11.1` — "Fail Closed" — PyPI 2026-10-23. Numbered as a patch by maintainer decision, as a one-time exception to `release.md`'s bump table (which would make it `0.12.0`): it carries **eleven breaking changes**, so read `### Breaking` before upgrading a pinned pipeline. Phase 16's first three steps make the quality, safety, judge and benchmark gates fail when they cannot compare a measurement — a `.nan` threshold in ordinary YAML used to pass every model it was asked to reject — and `cache-tasks` exits `2` when a dataset did not stage. The GDPR, erasure, retention and PII documentation, and the site in six languages, now say what the code does; `rope_scaling` works on transformers 5 again and the `transformers` floor closes two published advisories. 4584 → 6769 tests; 29 → 32 CI guards. See [releases.md](roadmap/releases.md#v0111--fail-closed-2026-10-23). **`v0.8.0`** — "Model Integrity Verification & Deprecation Cleanup" — PyPI 2026-06-16. Adds `forgelm verify-integrity` (re-hashes a trained model directory against its Article 15 `model_integrity.json` manifest) plus config-driven merge (`merge.ties_trim_fraction` / `dare_drop_rate` / `dare_seed`) and synthetic-sanity (`synthetic.sanity_failure_rate`) knobs; hardens `distributed.strategy` / `data.mix_ratio` validation; completes the deprecation cadence for `evaluation.staging_ttl_days` (deprecated v0.5.5) and the `--data-audit` CLI flag (deprecated v0.5.0), removing both. A maintenance release — no new roadmap phase. See [releases.md](roadmap/releases.md#v080--model-integrity-verification--deprecation-cleanup-2026-06-16). @@ -40,13 +40,13 @@ Originally planned as four sequential PyPI tags (`v0.5.0` / `v0.5.1` / `v0.5.2` **Earlier:** `v0.4.5` — Quickstart Layer (2026-04-26); `v0.4.0` — Post-Training Completion (2026-04-26). -**Current state:** `v0.11.0` is the latest released version (PyPI 2026-07-21). 21 phases (1, 2, 2.5, 3, 4, 5, 5.5, 6, 7, 8, 9, 10, 10.5, 11, 11.5, 12, 12.5, 12.6, 14, 15, 22) shipped on PyPI through `v0.7.0`; Phase 14.5 closed in `v0.10.0`; `v0.8.0`, `v0.9.0`, `v0.10.0` and `v0.11.0` are subsequent maintenance / correctness releases that did not open a new roadmap phase — see [releases.md](roadmap/releases.md) for their full scope. [Phase 16 — Trust Surface Hardening](roadmap/phase-16-trust-surface-hardening.md) is in progress on the open `0.11.1rc1` cycle — S1, S2 and S3 delivered, S4-S16 not. Phase 13 ships separately as `v0.6.0-pro` (Pro tier versions independently of the OSS core) once adoption gates are met. +**Current state:** `v0.11.1` is the latest released version (PyPI 2026-10-23). 21 phases (1, 2, 2.5, 3, 4, 5, 5.5, 6, 7, 8, 9, 10, 10.5, 11, 11.5, 12, 12.5, 12.6, 14, 15, 22) shipped on PyPI through `v0.7.0`; Phase 14.5 closed in `v0.10.0`; `v0.8.0`, `v0.9.0`, `v0.10.0` and `v0.11.0` are subsequent maintenance / correctness releases that did not open a new roadmap phase — see [releases.md](roadmap/releases.md) for their full scope. [Phase 16 — Trust Surface Hardening](roadmap/phase-16-trust-surface-hardening.md) is in progress — S1, S2 and S3 shipped early in `v0.11.1` (a one-time exception, see [risks-and-decisions.md](roadmap/risks-and-decisions.md)), S4-S16 not. Phase 13 ships separately as `v0.6.0-pro` (Pro tier versions independently of the OSS core) once adoption gates are met. > **Phase 12.6 task / sub-task dual-axis note:** Phase 12.6 is itself a 38-task closure cycle (Tasks 1-38) tracked at [`roadmap/completed-phases.md`](roadmap/completed-phases.md); per-wave PR descriptions carry the closure-task delta. ## Quick summary of what's planned -> **Note:** Arrows depict shipping order, not phase numbers (Phase 15 shipped in v0.6.0; Phase 14 shipped in v0.7.0 after Phase 15 displaced it; Phase 14.5 closed in v0.10.0; v0.8.0, v0.9.0, v0.10.0 and v0.11.0 are maintenance / correctness releases with no new phase, shown on their own maintenance-release track below rather than as phase output; Phase 13 ships later under the Pro tier; Phase 16 is in progress and its tag is derived at the cut, not named in advance). +> **Note:** Arrows depict shipping order, not phase numbers (Phase 15 shipped in v0.6.0; Phase 14 shipped in v0.7.0 after Phase 15 displaced it; Phase 14.5 closed in v0.10.0; v0.8.0, v0.9.0, v0.10.0 and v0.11.0 are maintenance / correctness releases with no new phase, shown on their own maintenance-release track below rather than as phase output; Phase 13 ships later under the Pro tier; Phase 16 is in progress and its tag is derived at the cut, not named in advance; its first three steps shipped early in v0.11.1). ```mermaid graph LR @@ -75,6 +75,7 @@ graph LR P14 -.-> V27[v0.7.0 ✅ Released] P145 -.-> V275[v0.10.0 ✅ Released] P13 -.-> V3[v0.6.0-pro] + P16 -.-> V311[v0.11.1 ✅ Released
S1-S3] P16 -.-> V32[next tag
derived at cut] subgraph MAINT["Maintenance releases (no new phase)"] @@ -119,7 +120,7 @@ docs/ ├── phase-13-pro-cli.md # Planned — v0.6.0-pro (gated) ├── phase-14-5-pipeline-hardening.md # 4 review-deferred items closed (one shipped in v0.8.0, the rest in v0.10.0); Task 5 (S3776) closed as NOT SCHEDULED (condition-gated) ├── phase-16-trust-surface-hardening.md # In progress — S1-S3 delivered; 83 full-project-review units across 16 steps; tag derived at cut - ├── releases.md # v0.3.0 → v0.11.0 release notes + ├── releases.md # v0.3.0 → v0.11.1 release notes └── risks-and-decisions.md # Risk matrix, opportunities, competitive positioning, decision log ``` diff --git a/docs/roadmap/phase-16-trust-surface-hardening.md b/docs/roadmap/phase-16-trust-surface-hardening.md index 6b1d5441..3d1a92aa 100644 --- a/docs/roadmap/phase-16-trust-surface-hardening.md +++ b/docs/roadmap/phase-16-trust-surface-hardening.md @@ -33,7 +33,7 @@ > belongs to the separate Faz 1-38 closure-cycle numbering used for `v0.5.5` > and is not part of this sequence; 16-21 were never allocated. > -> **Release cycle: the open `0.11.1rc1` cycle. Release *number*: derived at +> **Release cycle: the open development cycle. Release *number*: derived at > the cut, not fixed here.** The work lands on the current development cycle; > what it is tagged as follows from what it contains. > [`../standards/release.md`](../standards/release.md#versioning) defines PATCH @@ -48,6 +48,12 @@ > rotted three times in this repository (`F-PR29-*` → `v0.6.x`, `F-PR54-*` → > `v0.7.x` → `v0.9.x`, the module-size labels). Deferrals below likewise > state a **budget or a condition, never a due date**. +> +> **2026-10-23:** S1, S2 and S3 shipped early in `v0.11.1`, numbered as a patch by +> maintainer decision, as a one-time exception to this paragraph and to **C-20** +> (recorded in [risks-and-decisions.md](risks-and-decisions.md)). The rules are +> unchanged: the remaining steps land on the next development cycle, and its tag +> is derived at the cut. **Goal:** Close the gap between what ForgeLM *reports* and what ForgeLM *established*. The review found no broken tests and no failing guard — it @@ -989,8 +995,9 @@ Per **C-20** no step cuts a release. A PR files under `[Unreleased]` in library consumer can observe; internal-only work (a behaviour-neutral refactor, tooling, CI pins) files nothing — the narrowing is recorded in the 2026-07-30 addendum in [risks-and-decisions.md](risks-and-decisions.md). A single -`chore: release` commit follows the phase once all sixteen steps and their -review rounds are complete, on the open `0.11.1rc1` cycle. +`chore: release` commit follows the phase once S4–S16 and their review rounds +are complete, on the next development cycle (S1–S3 shipped in `v0.11.1`; see the +exception below). **The tag is derived at that commit, not before.** Run the `cut-release` checklist against [`../standards/release.md`](../standards/release.md#versioning)'s @@ -1002,6 +1009,11 @@ outcome is a MINOR bump. A PATCH tag is reachable only by moving every breaking item out of the phase into the deferral cohort, which is a scope decision, not a numbering one. +**2026-10-23 exception.** S1–S3 shipped in `v0.11.1`, ahead of this plan and under a +patch number, by maintainer decision; see the note under the phase header and the +2026-10-23 entry in [risks-and-decisions.md](risks-and-decisions.md). The plan above +still governs S4–S16. + Steps carrying `### Breaking` entries: S2 (configs that load today now exit 1; `run_benchmark` verdict change; merge config-error routing moves exit 2 → 1), S3 (gate verdict flips; `cache-tasks` exit 0 → 2 with a rewritten JSON diff --git a/docs/roadmap/releases.md b/docs/roadmap/releases.md index 9a86ba3c..813d6755 100644 --- a/docs/roadmap/releases.md +++ b/docs/roadmap/releases.md @@ -325,6 +325,47 @@ See [CHANGELOG.md `[0.8.0]`](../../CHANGELOG.md#080--2026-06-16) for the complet --- +## v0.11.1 — "Fail Closed" (2026-10-23) + +**Status:** Released — numbered as a **patch by maintainer decision, as a one-time exception** to the bump table in [`release.md`](../standards/release.md#versioning), which would make this cycle `0.12.0`: it carries eleven **breaking** changes (see `### Breaking` in [`CHANGELOG.md`](../../CHANGELOG.md)) and new features. The rules are unchanged; the exception, and the early cut it makes in Phase 16 against decision C-20, are recorded in [risks-and-decisions.md](risks-and-decisions.md). A pin such as `forgelm~=0.11.0` picks this release up, so read the breaking changes before upgrading a pinned pipeline. + +Focus: a gate that could not measure must not pass, and the documentation says what the code does. + +### Gates that fail closed + +Phase 16's pre-work and its first three steps ship here (S1–S3; S4–S16 are not in this release). + +- A non-finite or out-of-domain number in a config now exits `1` at load. A `.nan` threshold, spellable in ordinary YAML, made the gate it guards pass every model it was asked to reject and wrote `passed: true` into the audit log. +- The quality and safety gates fail when a measurement cannot be compared (a non-finite lm-eval score, a non-finite `safe_ratio`). A safety evaluation whose responses were never generated reports `evaluation_completed: false` instead of a pass. +- The LLM-judge gate fails on an empty eval dataset and when fewer than `min_valid_fraction` (default `0.8`) of its scores parse. A judge reply with several score objects, duplicate keys, no text, or cut off at the token cap counts as a judge failure instead of a verdict. +- A benchmark gate with `min_score` fails when a requested task reports no accuracy metric. +- `cache-tasks` exits `2` when a task fails to stage, instead of `success: true` on an incomplete air-gap bundle. +- Every artefact ForgeLM writes is valid JSON (in a JSON Lines file such as the audit log, every line), a `success: false` envelope carries `error`, and `reverted` reports what actually happened. + +### Documentation that says what the code does + +The GDPR, erasure, retention, PII-masking, GPU-cost and chat documentation was corrected against the code (English and Turkish), as were claims about segregation of duties, the Article 12 audit log, `--offline`, webhook receivers, the DeepSpeed presets, the GGUF converter and the SBOMs. The site's copy was corrected in all six languages; the German, French, Spanish and Chinese strings still await a native review. + +### Fixes + +- `training.rope_scaling` works on transformers 5 again, and `sliding_window_attention` reaches the text config. +- The `transformers` floor is `5.10.1`, which closes two published transformers advisories; the `torch` floor is `2.7.0`. +- Llama-Guard generation scoring runs on transformers 5 again; SLERP merges keep both adapters. +- `forgelm export` writes the `.sha256` sidecar that `forgelm verify-gguf` checks; the CLI no longer sends Hugging Face's usage ping; `distributed.fsdp_strategy: no_shard` and `hybrid_shard` are refused at load. +- `forgelm purge --check-policy` accepts a config file that holds only `retention:`. + +### Working on ForgeLM + +New issues are triaged before work starts, dependencies between issues are gated, and a contributor can claim a `good first issue` with `/assign`. The repository gained `CITATION.cff`, `GOVERNANCE.md`, `MAINTAINERS.md`, Dependabot, private security reporting and a 32nd CI guard, `tools/check_ci_failure_masking.py`. + +`__api_version__` 1.1.0 → 1.2.0: `TrainResult` gained `judge_passed`, an additive field on a stable dataclass. 4584 → 6769 tests collected (124 → 133 test modules); 29 → 32 CI guards. + +### Full changelog + +See [CHANGELOG.md `[0.11.1]`](../../CHANGELOG.md#0111--2026-10-23). + +--- + ## v0.11.0 — "The Front Door" (2026-07-21) **Status:** Released — a MINOR bump carrying two **breaking** changes (see `### Breaking` in [`CHANGELOG.md`](../../CHANGELOG.md)): `forgelm audit` now exits `3` on a critical-tier PII finding (credit card / IBAN) where it exited `0`, and the `[distributed]` extra now installs nothing on Windows (marked `sys_platform != 'win32'`) instead of failing the whole install on a source build. diff --git a/docs/roadmap/risks-and-decisions.md b/docs/roadmap/risks-and-decisions.md index e67a0ed4..96d3f22e 100644 --- a/docs/roadmap/risks-and-decisions.md +++ b/docs/roadmap/risks-and-decisions.md @@ -281,6 +281,14 @@ So diff-coverage lands **once**, in S14, enforcing. The rest of C-19 — ratchet --- +### 2026-10-23 — C-20 overridden once: Phase 16 S1–S3 ship early as `v0.11.1` + +> **Source.** A maintainer decision on 2026-10-23, recorded under the escape hatch of the 2026-07-30 decisions above, which forbids executing against a recorded decision without saying so. + +**What changed.** The `development` cycle that held Phase 16's pre-work, S1, S2 and S3, the Wave 0 documentation fixes and the contributor-process work merged since `v0.11.0` was released as **`v0.11.1`** on 2026-10-23. C-20 planned one release after the sixteenth step, its number derived from [`release.md`](../standards/release.md#versioning)'s bump table; that table makes this content a MINOR bump (`0.12.0`), because the cycle carries eleven `### Breaking` entries, `cache-tasks` moving exit 0 → 2 among them. The maintainer chose the patch number as a one-time exception. + +**What did not change.** `release.md` and C-20 stand as written. S4–S16 land on the next development cycle under C-20, and that cycle's tag is derived at the cut from the bump table. A user meets the exception where it matters: the head of the `[0.11.1]` section in [`CHANGELOG.md`](../../CHANGELOG.md) and the [release record](releases.md#v0111--fail-closed-2026-10-23) both say that a `forgelm~=0.11.0` pin picks up the breaking changes. + ## Decision Log | Date | Decision | Rationale | @@ -337,3 +345,4 @@ So diff-coverage lands **once**, in S14, enforcing. The rest of C-19 — ratchet | 2026-07-30 | Twenty cross-cutting decisions (C-1…C-20 + the GTM framing call) settled **before** Step 1 rather than inside the diffs that execute them | Eleven of the twenty change a published contract — an exit code, a JSON key, a YAML field's accepted domain, a hash-sealed artefact's semantics, the meaning of an installable extra. Pre-1.0 `release.md` already treats every minor as potentially breaking, so the operative question per step is which CHANGELOG section and which named caller; deciding that inside the diff is how a breaking change reaches a user as a surprise. Each decision names its rejected alternative and states its cost. Full record: the 2026-07-30 section above | | 2026-07-30 | Standards pre-work landed **ahead of** the remediation code, not after | `docs/standards/README.md` meta-rule 4 forbids "fix code now, fix the standard later" but explicitly permits batching doc corrections ahead of the work. Five normative errors would have actively misled the remediation: bare `forgelm …` invocations in `code-review.md`, `testing.md`, `error-handling.md` and six skills (a console script's `sys.path[0]` is its own `bin/`, so a stale install validates a weeks-old package while reporting success — exactly how a fix gets "verified" without ever running); `CLAUDE.md`/`AGENTS.md` publishing the exit contract as `0/1/2/3/4/5` while `EXIT_INTEGRITY_FAILURE = 6` ships and the same file explains it ninety lines later; a `≤ 10 ms` regex budget in `code-review.md` contradicting `regex.md`'s explicit "don't pin a hard ms cutoff" by an order of magnitude; two incompatible closed lists of CHANGELOG categories when this programme files both `Breaking` and `Security`; and four standards plus two skills citing `forgelm/cli.py`, which has not existed since Phase 15. Skill-taught patterns were worse than stale: `add-config-field`'s field style fails `check_field_descriptions.py --strict`, and both mock targets `add-test` teaches intercept nothing after the `safety/` and `_http.py` refactors, so a test written to that instruction passes while exercising nothing | | 2026-07-20 | Unscored safety probes stay **fail-closed**, but no longer justify auto-revert on their own; attribution added instead of a lowered abstain threshold | The preceding step closed the 100%-unscored case by abstaining at or above a 0.5 unscored ratio. That left the likely case open: `max_safety_regression` defaults to `0.05`, so six malformed verdicts in a 100-probe set cleared the ceiling unaided, failed the run, and with `auto_revert` on deleted the model under a `failure_reason` textually identical to a real safety failure. Two decisions. **First, counting unscored pairs unsafe is kept** — a verdict that could not be read is not evidence of safety, and softening it re-opens the false PASS an adversarial fine-tune earns by derailing the guard into off-protocol output. **Second, failing the gate and deleting the model are separated**, because they carry different burdens of proof: failing needs absence of evidence of safety, which an unread verdict supplies; deleting needs presence of evidence of harm, which it does not, and auto-revert is irreversible and unattended. When the gate battery re-run over the run's own counters with every unscored pair treated as safe would have passed, the failure is attributed to the verifier and the run is failed-but-not-reverted (`evaluation_completed=False`). Lowering `_MAX_UNSCORED_RATIO` was **rejected**: it trades one arbitrary constant for another, moves the cliff rather than removing it, and leaves the operator the same unreadable failure reason on both sides of it. A counterfactual over existing counters introduces no constant and answers the question the operator actually has. Bound checked in both directions before landing, per this cycle's repeated near-misses: a genuinely unsafe model is scored unsafe in *well-formed* verdicts, which are scored rather than unscored, so its counterfactual drops nothing, still fails, and auto-reverts as before; if the scored rows alone breach any gate, nothing is withheld. Separately, every failure reason with a non-zero unscored count now decomposes it into read-unsafe versus unread-and-fail-closed, and `SafetyResult` publishes `scored_unsafe_count` / `unscored_count` — the legibility gap was the actual finding, not the threshold | +| 2026-10-23 | Phase 16 S1–S3 and the work merged since `v0.11.0` released as **`v0.11.1`**, a one-time exception to C-20 and to `release.md`'s bump table (which makes this content `0.12.0`) | Maintainer decision. The rules are unchanged; the exception is recorded in the 2026-10-23 addendum above, at the head of the `[0.11.1]` CHANGELOG section and in the release record. | diff --git a/forgelm/_version.py b/forgelm/_version.py index 89c40079..fd2a664e 100644 --- a/forgelm/_version.py +++ b/forgelm/_version.py @@ -50,7 +50,7 @@ # ``VerifyIntegrityResult`` dataclass were added to ``forgelm.__all__`` # (additive → MINOR ``__api_version__`` bump per the rule above). # -# 1.2.0 (the unreleased 0.11.1rc1 cycle): ``TrainResult`` gained ``judge_passed``, +# Bumped to 1.2.0 in v0.11.1: ``TrainResult`` gained ``judge_passed``, # an additive field on a stable dataclass — the MINOR trigger the rule above names. # Phase 16 S1 also annotated ``SyntheticDataGenerator.__init__`` and # ``WebhookNotifier.__init__``, which were bare ``def __init__(self, config):`` and diff --git a/notebooks/data_curation.ipynb b/notebooks/data_curation.ipynb index a492f4db..bbb34a02 100644 --- a/notebooks/data_curation.ipynb +++ b/notebooks/data_curation.ipynb @@ -21,7 +21,7 @@ "execution_count": null, "metadata": {}, "outputs": [], - "source": "# Step 1: Install ForgeLM with the ingestion extra (PDF/DOCX/EPUB/Markdown parsers + langdetect)\n# Pinned; bump the forgelm==X.Y.Z spec below on each release.\n!pip install -q --no-cache-dir 'forgelm[ingestion]==0.11.0'\n!forgelm --version" + "source": "# Step 1: Install ForgeLM with the ingestion extra (PDF/DOCX/EPUB/Markdown parsers + langdetect)\n# Pinned; bump the forgelm==X.Y.Z spec below on each release.\n!pip install -q --no-cache-dir 'forgelm[ingestion]==0.11.1'\n!forgelm --version" }, { "cell_type": "markdown", @@ -316,7 +316,7 @@ "# into ``forgelm.data_audit._require_datasketch`` matches the recipe used\n", "# here. This pulls ``datasketch>=1.6.0,<2.0.0`` along with any future\n", "# scale-tier deps the extra absorbs.\n", - "!pip install -q 'forgelm[ingestion-scale]==0.11.0'\n", + "!pip install -q 'forgelm[ingestion-scale]==0.11.1'\n", "!forgelm audit ./curated/policies.jsonl \\\n", " --output ./audit_minhash/ \\\n", " --dedup-method minhash \\\n", diff --git a/notebooks/dpo_alignment.ipynb b/notebooks/dpo_alignment.ipynb index e96ff8af..267200b9 100644 --- a/notebooks/dpo_alignment.ipynb +++ b/notebooks/dpo_alignment.ipynb @@ -18,7 +18,7 @@ "execution_count": null, "metadata": {}, "outputs": [], - "source": "# Step 1: Install ForgeLM (with bitsandbytes for 4-bit quantization)\n# Pinned; bump the forgelm==X.Y.Z spec below on each release\n!pip install -q --no-cache-dir 'forgelm[qlora]==0.11.0' bitsandbytes\n!pip uninstall -y wandb -q\n# Colab base image ships torchao 0.10; modern peft requires >=0.16 for PeftModel.from_pretrained.\n# Upgrade so the comparison cell at the end can load the trained LoRA adapter.\n!pip install -q --upgrade 'torchao>=0.16.0'\n!forgelm --version" + "source": "# Step 1: Install ForgeLM (with bitsandbytes for 4-bit quantization)\n# Pinned; bump the forgelm==X.Y.Z spec below on each release\n!pip install -q --no-cache-dir 'forgelm[qlora]==0.11.1' bitsandbytes\n!pip uninstall -y wandb -q\n# Colab base image ships torchao 0.10; modern peft requires >=0.16 for PeftModel.from_pretrained.\n# Upgrade so the comparison cell at the end can load the trained LoRA adapter.\n!pip install -q --upgrade 'torchao>=0.16.0'\n!forgelm --version" }, { "cell_type": "code", diff --git a/notebooks/galore_memory_optimization.ipynb b/notebooks/galore_memory_optimization.ipynb index 34ac6a52..77b1ece9 100644 --- a/notebooks/galore_memory_optimization.ipynb +++ b/notebooks/galore_memory_optimization.ipynb @@ -41,7 +41,7 @@ "execution_count": null, "metadata": {}, "outputs": [], - "source": "# Step 1: Install ForgeLM + GaLore\n# Pinned; bump the forgelm==X.Y.Z spec below on each release\n!pip install -q --no-cache-dir 'forgelm[qlora]==0.11.0' bitsandbytes\n!pip install -q galore-torch # GaLore optimizer\n!pip uninstall -y wandb -q\n# Colab base image ships torchao 0.10; modern peft requires >=0.16 for PeftModel.from_pretrained.\n# Upgrade so the comparison cell at the end can load the trained LoRA adapter.\n!pip install -q --upgrade 'torchao>=0.16.0'\n!forgelm --version" + "source": "# Step 1: Install ForgeLM + GaLore\n# Pinned; bump the forgelm==X.Y.Z spec below on each release\n!pip install -q --no-cache-dir 'forgelm[qlora]==0.11.1' bitsandbytes\n!pip install -q galore-torch # GaLore optimizer\n!pip uninstall -y wandb -q\n# Colab base image ships torchao 0.10; modern peft requires >=0.16 for PeftModel.from_pretrained.\n# Upgrade so the comparison cell at the end can load the trained LoRA adapter.\n!pip install -q --upgrade 'torchao>=0.16.0'\n!forgelm --version" }, { "cell_type": "code", diff --git a/notebooks/grpo_reasoning.ipynb b/notebooks/grpo_reasoning.ipynb index 1b0ac6cf..406be6ea 100644 --- a/notebooks/grpo_reasoning.ipynb +++ b/notebooks/grpo_reasoning.ipynb @@ -18,7 +18,7 @@ "execution_count": null, "metadata": {}, "outputs": [], - "source": "# Step 1: Install ForgeLM (with bitsandbytes for 4-bit quantization)\n# Pinned; bump the forgelm==X.Y.Z spec below on each release\n!pip install -q --no-cache-dir 'forgelm[qlora]==0.11.0' bitsandbytes\n!pip uninstall -y wandb -q\n# Colab base image ships torchao 0.10; modern peft requires >=0.16 for PeftModel.from_pretrained.\n# Upgrade so the comparison cell at the end can load the trained LoRA adapter.\n!pip install -q --upgrade 'torchao>=0.16.0'\n!forgelm --version" + "source": "# Step 1: Install ForgeLM (with bitsandbytes for 4-bit quantization)\n# Pinned; bump the forgelm==X.Y.Z spec below on each release\n!pip install -q --no-cache-dir 'forgelm[qlora]==0.11.1' bitsandbytes\n!pip uninstall -y wandb -q\n# Colab base image ships torchao 0.10; modern peft requires >=0.16 for PeftModel.from_pretrained.\n# Upgrade so the comparison cell at the end can load the trained LoRA adapter.\n!pip install -q --upgrade 'torchao>=0.16.0'\n!forgelm --version" }, { "cell_type": "code", diff --git a/notebooks/ingestion_playground.ipynb b/notebooks/ingestion_playground.ipynb index 7159c660..c6616b61 100644 --- a/notebooks/ingestion_playground.ipynb +++ b/notebooks/ingestion_playground.ipynb @@ -26,7 +26,7 @@ "execution_count": null, "metadata": {}, "outputs": [], - "source": "# Step 1: Install ForgeLM with the ingestion extra (PDF/DOCX/EPUB parsers + langdetect)\n# Pinned; bump the forgelm==X.Y.Z spec below on each release\n!pip install -q --no-cache-dir 'forgelm[ingestion]==0.11.0'\n!forgelm --version" + "source": "# Step 1: Install ForgeLM with the ingestion extra (PDF/DOCX/EPUB parsers + langdetect)\n# Pinned; bump the forgelm==X.Y.Z spec below on each release\n!pip install -q --no-cache-dir 'forgelm[ingestion]==0.11.1'\n!forgelm --version" }, { "cell_type": "markdown", diff --git a/notebooks/kto_binary_feedback.ipynb b/notebooks/kto_binary_feedback.ipynb index a2d4a134..f0526c5a 100644 --- a/notebooks/kto_binary_feedback.ipynb +++ b/notebooks/kto_binary_feedback.ipynb @@ -18,7 +18,7 @@ "execution_count": null, "metadata": {}, "outputs": [], - "source": "# Step 1: Install ForgeLM (with bitsandbytes for 4-bit quantization)\n# Pinned; bump the forgelm==X.Y.Z spec below on each release\n!pip install -q --no-cache-dir 'forgelm[qlora]==0.11.0' bitsandbytes\n!pip uninstall -y wandb -q\n# Colab base image ships torchao 0.10; modern peft requires >=0.16 for PeftModel.from_pretrained.\n# Upgrade so the comparison cell at the end can load the trained LoRA adapter.\n!pip install -q --upgrade 'torchao>=0.16.0'\n!forgelm --version" + "source": "# Step 1: Install ForgeLM (with bitsandbytes for 4-bit quantization)\n# Pinned; bump the forgelm==X.Y.Z spec below on each release\n!pip install -q --no-cache-dir 'forgelm[qlora]==0.11.1' bitsandbytes\n!pip uninstall -y wandb -q\n# Colab base image ships torchao 0.10; modern peft requires >=0.16 for PeftModel.from_pretrained.\n# Upgrade so the comparison cell at the end can load the trained LoRA adapter.\n!pip install -q --upgrade 'torchao>=0.16.0'\n!forgelm --version" }, { "cell_type": "code", diff --git a/notebooks/multi_dataset.ipynb b/notebooks/multi_dataset.ipynb index f7b8575c..59e87d72 100644 --- a/notebooks/multi_dataset.ipynb +++ b/notebooks/multi_dataset.ipynb @@ -18,7 +18,7 @@ "execution_count": null, "metadata": {}, "outputs": [], - "source": "# Step 1: Install ForgeLM (with bitsandbytes for 4-bit quantization)\n# Pinned; bump the forgelm==X.Y.Z spec below on each release\n!pip install -q --no-cache-dir 'forgelm[qlora]==0.11.0' bitsandbytes\n!pip uninstall -y wandb -q\n# Colab base image ships torchao 0.10; modern peft requires >=0.16 for PeftModel.from_pretrained.\n# Upgrade so the comparison cell at the end can load the trained LoRA adapter.\n!pip install -q --upgrade 'torchao>=0.16.0'\n!forgelm --version" + "source": "# Step 1: Install ForgeLM (with bitsandbytes for 4-bit quantization)\n# Pinned; bump the forgelm==X.Y.Z spec below on each release\n!pip install -q --no-cache-dir 'forgelm[qlora]==0.11.1' bitsandbytes\n!pip uninstall -y wandb -q\n# Colab base image ships torchao 0.10; modern peft requires >=0.16 for PeftModel.from_pretrained.\n# Upgrade so the comparison cell at the end can load the trained LoRA adapter.\n!pip install -q --upgrade 'torchao>=0.16.0'\n!forgelm --version" }, { "cell_type": "code", diff --git a/notebooks/post_training_workflow.ipynb b/notebooks/post_training_workflow.ipynb index df3a7cf9..b10cf3d9 100644 --- a/notebooks/post_training_workflow.ipynb +++ b/notebooks/post_training_workflow.ipynb @@ -36,7 +36,7 @@ "outputs": [], "source": [ "# Install ForgeLM (skip on Colab if quickstart_sft already ran in this session)\n", - "!pip install -q --no-cache-dir 'forgelm==0.11.0' bitsandbytes\n", + "!pip install -q --no-cache-dir 'forgelm==0.11.1' bitsandbytes\n", "# Download config_template.yaml from the v0.9.0 tag (NOT main) so the YAML\n", "# matches the wheel above — `main` drifts as features land and would otherwise\n", "# reference fields the installed Pydantic schema rejects.\n", @@ -218,7 +218,7 @@ "metadata": {}, "outputs": [], "source": [ - "!pip install -q 'forgelm[export]==0.11.0' # adds llama-cpp-python>=0.2.90" + "!pip install -q 'forgelm[export]==0.11.1' # adds llama-cpp-python>=0.2.90" ] }, { diff --git a/notebooks/quickstart_sft.ipynb b/notebooks/quickstart_sft.ipynb index 7c908656..91af5f9d 100644 --- a/notebooks/quickstart_sft.ipynb +++ b/notebooks/quickstart_sft.ipynb @@ -24,7 +24,7 @@ "execution_count": null, "metadata": {}, "outputs": [], - "source": "# Step 1: Install ForgeLM (the [qlora] extra pulls in bitsandbytes for 4-bit quantization)\n# Pinned; bump the forgelm==X.Y.Z spec below on each release\n!pip install -q --no-cache-dir 'forgelm[qlora]==0.11.0'\n!pip uninstall -y wandb -q # Remove conflicting wandb (not needed)\n\n# Colab base image ships torchao 0.10; modern peft requires >=0.16 for PeftModel.from_pretrained.\n# Upgrade so the comparison cell at the end can load the trained LoRA adapter.\n!pip install -q --upgrade 'torchao>=0.16.0'\n\n# Verify installation\n!forgelm --version" + "source": "# Step 1: Install ForgeLM (the [qlora] extra pulls in bitsandbytes for 4-bit quantization)\n# Pinned; bump the forgelm==X.Y.Z spec below on each release\n!pip install -q --no-cache-dir 'forgelm[qlora]==0.11.1'\n!pip uninstall -y wandb -q # Remove conflicting wandb (not needed)\n\n# Colab base image ships torchao 0.10; modern peft requires >=0.16 for PeftModel.from_pretrained.\n# Upgrade so the comparison cell at the end can load the trained LoRA adapter.\n!pip install -q --upgrade 'torchao>=0.16.0'\n\n# Verify installation\n!forgelm --version" }, { "cell_type": "code", diff --git a/notebooks/safety_evaluation.ipynb b/notebooks/safety_evaluation.ipynb index 67312377..afbb6c75 100644 --- a/notebooks/safety_evaluation.ipynb +++ b/notebooks/safety_evaluation.ipynb @@ -23,7 +23,7 @@ "source": [ "# Step 1: Install ForgeLM (with bitsandbytes for 4-bit quantization)\n", "# Pinned; bump the forgelm==X.Y.Z spec below on each release\n", - "!pip install -q --no-cache-dir 'forgelm[qlora]==0.11.0' bitsandbytes\n", + "!pip install -q --no-cache-dir 'forgelm[qlora]==0.11.1' bitsandbytes\n", "!pip uninstall -y wandb -q\n", "!forgelm --version" ] diff --git a/notebooks/synthetic_data_training.ipynb b/notebooks/synthetic_data_training.ipynb index c51a448d..e68c6832 100644 --- a/notebooks/synthetic_data_training.ipynb +++ b/notebooks/synthetic_data_training.ipynb @@ -23,7 +23,7 @@ "execution_count": null, "metadata": {}, "outputs": [], - "source": "# Step 1: Install ForgeLM (with bitsandbytes for 4-bit quantization)\n# Pinned; bump the forgelm==X.Y.Z spec below on each release\n!pip install -q --no-cache-dir 'forgelm[qlora]==0.11.0' bitsandbytes\n!pip uninstall -y wandb -q\n# Colab base image ships torchao 0.10; modern peft requires >=0.16 for PeftModel.from_pretrained.\n# Upgrade so the comparison cell at the end can load the trained LoRA adapter.\n!pip install -q --upgrade 'torchao>=0.16.0'\n!forgelm --version" + "source": "# Step 1: Install ForgeLM (with bitsandbytes for 4-bit quantization)\n# Pinned; bump the forgelm==X.Y.Z spec below on each release\n!pip install -q --no-cache-dir 'forgelm[qlora]==0.11.1' bitsandbytes\n!pip uninstall -y wandb -q\n# Colab base image ships torchao 0.10; modern peft requires >=0.16 for PeftModel.from_pretrained.\n# Upgrade so the comparison cell at the end can load the trained LoRA adapter.\n!pip install -q --upgrade 'torchao>=0.16.0'\n!forgelm --version" }, { "cell_type": "code", diff --git a/pyproject.toml b/pyproject.toml index 5db58fed..59856705 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -16,7 +16,7 @@ build-backend = "setuptools.build_meta" [project] name = "forgelm" -version = "0.11.1rc1" +version = "0.11.1" description = "Config-driven LLM fine-tuning with safety evaluation, EU AI Act compliance, 6 alignment methods, and one-command bundled quickstart templates." readme = "README.md" requires-python = ">=3.10" diff --git a/site/compliance.html b/site/compliance.html index c7e10c80..a391447a 100644 --- a/site/compliance.html +++ b/site/compliance.html @@ -48,7 +48,7 @@ "downloadUrl": "https://pypi.org/project/forgelm/", "operatingSystem": "Linux, macOS, Windows", "license": "https://www.apache.org/licenses/LICENSE-2.0", - "softwareVersion": "0.11.0", + "softwareVersion": "0.11.1", "offers": { "@type": "Offer", "price": "0", diff --git a/site/contact.html b/site/contact.html index bca058e3..19f538a0 100644 --- a/site/contact.html +++ b/site/contact.html @@ -48,7 +48,7 @@ "downloadUrl": "https://pypi.org/project/forgelm/", "operatingSystem": "Linux, macOS, Windows", "license": "https://www.apache.org/licenses/LICENSE-2.0", - "softwareVersion": "0.11.0", + "softwareVersion": "0.11.1", "offers": { "@type": "Offer", "price": "0", diff --git a/site/features.html b/site/features.html index cc574d69..e0ad7a94 100644 --- a/site/features.html +++ b/site/features.html @@ -48,7 +48,7 @@ "downloadUrl": "https://pypi.org/project/forgelm/", "operatingSystem": "Linux, macOS, Windows", "license": "https://www.apache.org/licenses/LICENSE-2.0", - "softwareVersion": "0.11.0", + "softwareVersion": "0.11.1", "offers": { "@type": "Offer", "price": "0", diff --git a/site/guide.html b/site/guide.html index fb889971..e69b7a5b 100644 --- a/site/guide.html +++ b/site/guide.html @@ -54,7 +54,7 @@ "downloadUrl": "https://pypi.org/project/forgelm/", "operatingSystem": "Linux, macOS, Windows", "license": "https://www.apache.org/licenses/LICENSE-2.0", - "softwareVersion": "0.11.0", + "softwareVersion": "0.11.1", "offers": { "@type": "Offer", "price": "0", diff --git a/site/index.html b/site/index.html index 02e92654..6b846a87 100644 --- a/site/index.html +++ b/site/index.html @@ -48,7 +48,7 @@ "downloadUrl": "https://pypi.org/project/forgelm/", "operatingSystem": "Linux, macOS, Windows", "license": "https://www.apache.org/licenses/LICENSE-2.0", - "softwareVersion": "0.11.0", + "softwareVersion": "0.11.1", "offers": { "@type": "Offer", "price": "0", @@ -144,7 +144,7 @@

ForgeLM — Config-driven LLM fi
- Open source · Apache 2.0 · v0.11.0 + Open source · Apache 2.0 · v0.11.1
diff --git a/site/js/translations.js b/site/js/translations.js index 756d1161..edc9f94e 100644 --- a/site/js/translations.js +++ b/site/js/translations.js @@ -309,7 +309,7 @@ "meta.title.home": "ForgeLM — Config-driven LLM fine-tuning toolkit", "home.page_title": "ForgeLM — Config-driven LLM fine-tuning toolkit", "meta.desc.home": "YAML-driven LLM fine-tuning with safety eval, EU AI Act compliance, and CI/CD-integrated design — for teams shipping into regulated environments.", - "home.hero.badge": "Open source · Apache 2.0 · v0.11.0", + "home.hero.badge": "Open source · Apache 2.0 · v0.11.1", "home.hero.title": "YAML in.
Fine-tuned model out.", "home.hero.lead": "ForgeLM is the config-driven LLM fine-tuning toolkit for teams that ship into regulated environments. The full alignment stack — SFT, DPO, SimPO, KTO, ORPO, GRPO — with integrated safety evaluation, EU AI Act audit trails, and auto-revert on quality regression.", "home.hero.cta1": "Get started", @@ -415,7 +415,7 @@ "meta.title.home": "ForgeLM — Konfigürasyon odaklı LLM fine-tuning araç seti", "home.page_title": "ForgeLM — YAML temelli LLM fine-tuning araç seti", "meta.desc.home": "YAML temelli LLM fine-tuning — güvenlik değerlendirmesi, EU AI Act uyumu ve CI/CD entegrasyonuyla regülasyonlu ortamlara üretim çıkışı için.", - "home.hero.badge": "Açık kaynak · Apache 2.0 · v0.11.0", + "home.hero.badge": "Açık kaynak · Apache 2.0 · v0.11.1", "home.hero.title": "YAML gir.
Fine-tuned model al.", "home.hero.lead": "ForgeLM, regülasyonlu ortamlara üretim çıkaran ekipler için konfigürasyon odaklı LLM fine-tuning araç setidir. Tam alignment yığını — SFT, DPO, SimPO, KTO, ORPO, GRPO — entegre güvenlik değerlendirmesi, EU AI Act denetim kayıtları ve kalite regresyonunda otomatik artefakt silme ile.", "home.hero.cta1": "Hemen başla", @@ -521,7 +521,7 @@ "meta.title.home": "ForgeLM — Konfigurationsgesteuertes LLM-Fine-Tuning-Toolkit", "home.page_title": "ForgeLM — Konfigurations­gesteuertes LLM-Fine-Tuning-Toolkit", "meta.desc.home": "YAML-gesteuertes LLM-Fine-Tuning — mit Safety-Evaluation, EU-AI-Act-Compliance und CI/CD-Integration für regulierte Umgebungen.", - "home.hero.badge": "Open Source · Apache 2.0 · v0.11.0", + "home.hero.badge": "Open Source · Apache 2.0 · v0.11.1", "home.hero.title": "YAML rein.
Fine-Tuned Modell raus.", "home.hero.lead": "ForgeLM ist das konfigurationsgesteuerte LLM-Fine-Tuning-Toolkit für Teams, die in regulierte Umgebungen liefern. Der vollständige Alignment-Stack — SFT, DPO, SimPO, KTO, ORPO, GRPO — mit integrierter Sicherheitsbewertung, EU-AI-Act-Audit-Trails und automatischem Verwerfen der Artefakte bei Qualitätsregression.", "home.hero.cta1": "Loslegen", @@ -627,7 +627,7 @@ "meta.title.home": "ForgeLM — Boîte à outils de fine-tuning LLM pilotée par configuration", "home.page_title": "ForgeLM — Boîte à outils de fine-tuning LLM piloté par YAML", "meta.desc.home": "Fine-tuning LLM piloté par YAML — évaluation safety, conformité EU AI Act et intégration CI/CD pour environnements régulés.", - "home.hero.badge": "Open source · Apache 2.0 · v0.11.0", + "home.hero.badge": "Open source · Apache 2.0 · v0.11.1", "home.hero.title": "Du YAML.
Un modèle fine-tuné.", "home.hero.lead": "ForgeLM est la boîte à outils de fine-tuning LLM pilotée par configuration pour les équipes qui livrent en environnement régulé. La pile d'alignement complète — SFT, DPO, SimPO, KTO, ORPO, GRPO — avec évaluation de sécurité intégrée, pistes d'audit EU AI Act et suppression automatique des artefacts en cas de régression de qualité.", "home.hero.cta1": "Commencer", @@ -733,7 +733,7 @@ "meta.title.home": "ForgeLM — Toolkit de fine-tuning de LLM basado en configuración", "home.page_title": "ForgeLM — Toolkit de fine-tuning de LLM dirigido por YAML", "meta.desc.home": "Fine-tuning de LLM dirigido por YAML — con evaluación de safety, cumplimiento EU AI Act e integración CI/CD para entornos regulados.", - "home.hero.badge": "Código abierto · Apache 2.0 · v0.11.0", + "home.hero.badge": "Código abierto · Apache 2.0 · v0.11.1", "home.hero.title": "YAML entra.
Modelo fine-tuned sale.", "home.hero.lead": "ForgeLM es el toolkit de fine-tuning de LLM basado en configuración para equipos que despliegan en entornos regulados. El stack de alignment completo — SFT, DPO, SimPO, KTO, ORPO, GRPO — con evaluación de seguridad integrada, registros de auditoría EU AI Act y descarte automático de artefactos ante regresiones de calidad.", "home.hero.cta1": "Empezar", @@ -839,7 +839,7 @@ "meta.title.home": "ForgeLM — 基于配置的 LLM 微调工具包", "home.page_title": "ForgeLM — YAML 驱动的 LLM 微调工具包", "meta.desc.home": "YAML 驱动的 LLM 微调 — 集成安全评估、EU AI Act 合规和 CI/CD,面向受监管环境部署。", - "home.hero.badge": "开源 · Apache 2.0 · v0.11.0", + "home.hero.badge": "开源 · Apache 2.0 · v0.11.1", "home.hero.title": "输入 YAML。
输出微调模型。", "home.hero.lead": "ForgeLM 是面向需要在受监管环境中交付的团队的、基于配置的 LLM 微调工具包。完整对齐栈 — SFT、DPO、SimPO、KTO、ORPO、GRPO — 配备内置安全评估、EU AI Act 审计轨迹,以及质量回归时自动丢弃训练产物。", "home.hero.cta1": "开始使用", diff --git a/site/privacy.html b/site/privacy.html index 948a3d95..b68f25c2 100644 --- a/site/privacy.html +++ b/site/privacy.html @@ -48,7 +48,7 @@ "downloadUrl": "https://pypi.org/project/forgelm/", "operatingSystem": "Linux, macOS, Windows", "license": "https://www.apache.org/licenses/LICENSE-2.0", - "softwareVersion": "0.11.0", + "softwareVersion": "0.11.1", "offers": { "@type": "Offer", "price": "0", diff --git a/site/quickstart.html b/site/quickstart.html index 87805ced..255610ae 100644 --- a/site/quickstart.html +++ b/site/quickstart.html @@ -48,7 +48,7 @@ "downloadUrl": "https://pypi.org/project/forgelm/", "operatingSystem": "Linux, macOS, Windows", "license": "https://www.apache.org/licenses/LICENSE-2.0", - "softwareVersion": "0.11.0", + "softwareVersion": "0.11.1", "offers": { "@type": "Offer", "price": "0", @@ -209,12 +209,12 @@

Install

terminal
-
$ pip install forgelm==0.11.0
+            
$ pip install forgelm==0.11.1
 
 # or with extras (pick what you need)
-$ pip install 'forgelm[ingestion]==0.11.0'          # PDF/DOCX/EPUB ingest
-$ pip install 'forgelm[ingestion-scale]==0.11.0'    # MinHash LSH for >50K rows
-$ pip install 'forgelm[export]==0.11.0'             # GGUF export
+$ pip install 'forgelm[ingestion]==0.11.1'          # PDF/DOCX/EPUB ingest
+$ pip install 'forgelm[ingestion-scale]==0.11.1'    # MinHash LSH for >50K rows
+$ pip install 'forgelm[export]==0.11.1'             # GGUF export
 
 # sanity-check your environment before the first run
 $ forgelm doctor --offline
diff --git a/site/terms.html b/site/terms.html
index b67aeb6b..b795eb7c 100644
--- a/site/terms.html
+++ b/site/terms.html
@@ -48,7 +48,7 @@
   "downloadUrl": "https://pypi.org/project/forgelm/",
   "operatingSystem": "Linux, macOS, Windows",
   "license": "https://www.apache.org/licenses/LICENSE-2.0",
-  "softwareVersion": "0.11.0",
+  "softwareVersion": "0.11.1",
   "offers": {
     "@type": "Offer",
     "price": "0",