From 52b113d67d5eb2bd17129740ed17c06c1407f353 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sat, 9 May 2026 00:10:25 +0300 Subject: [PATCH 01/17] =?UTF-8?q?feat(wizard):=20Phase=2022=20=E2=80=94=20?= =?UTF-8?q?CLI=20wizard=20parity=20with=20web=20+=20sub-package=20split?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the parity gap between `forgelm --wizard` (CLI) and the in-browser `site/js/wizard.js` documented in `docs/analysis/code_reviews/2026-05-07_cli_wizard_ux_analysis.md` + `-2.md`. 20 original findings (G1-G20) and 5 independent findings (I1-I5) were verified against current code; this PR addresses the actionable subset. What's covered (CLI): - **Step reorder (I1):** `compliance.risk_classification` is now collected BEFORE `evaluation.safety` so the F-compliance-110 strict-tier auto-coercion can fire reliably. The v0.5.5 wizard asked safety first, then risk — silently producing high-risk configs that failed to load. - **Trainer-specific hyperparameters (G1):** `dpo_beta`, `simpo_beta` / `simpo_gamma`, `kto_beta`, `orpo_beta`, `grpo_num_generations`, `grpo_max_completion_length`, `grpo_reward_model` are now surfaced per `trainer_type`. SFT short-circuits. - **PEFT method breadth (G2):** the strategy step now offers all four schema-supported `lora.method` values (`lora`, `dora`, `pissa`, `rslora`) plus GaLore as a separate axis (6 cards total). - **GaLore optimiser variants (G9):** all six `galore_optim` Literal values surfaced, including the three `_layerwise` siblings. - **RoPE scaling types (G10):** `longrope` joined `linear` / `dynamic` / `yarn` (4-of-4 schema coverage). - **LoRA `r` schema parity (G11):** `DEFAULT_LORA_R` lowered from `16` → `8` to match `LoraConfigModel.r`. - **Compliance depth (G5):** `compliance` (Article 11 + Annex IV §1) plus optional `risk_assessment` (Article 9), `data.governance` (Article 10), `retention` (GDPR Article 5(1)(e) + 17), `monitoring` (Article 12+17), `evaluation.benchmark`, `evaluation.llm_judge`, `synthetic` blocks now configurable. - **High-risk auto-coercion (G8):** `risk_classification ∈ {high-risk, unacceptable}` automatically enables `evaluation.safety` + `evaluation.require_human_approval` with a visible operator notice. Front-stops the schema-side `F-compliance-110` `ConfigError`. - **Webhook URL parsing (G15):** single prompt accepts a literal URL or `env:VAR_NAME` reference; `urlparse`-validated; HTTPS recommended (HTTP accepted with warning). - **Safety probe path (G16):** bundled probe set resolved through `importlib.resources.files("forgelm.safety_prompts")`, fixing the `pip install` regression where `configs/safety_prompts/general_safety.jsonl` was the wizard default but never shipped in the wheel. - **Configuration summary (G17):** `_print_wizard_summary` now dumps the full YAML alongside the labelled headline. - **Persistence (G6):** state snapshot at `$XDG_CACHE_HOME/forgelm/wizard_state.yaml` (or `~/.cache/forgelm/wizard_state.yaml`) saved after every completed step. Schema versioned (`v: 1`); cleared on successful completion. - **Step-diff preview (G7):** each completed step prints `+ key.path: value` / `~ key.path: before → after`. - **Beginner / expert toggle (G13):** beginner mode prefixes each step with a 2-3-line tutorial paragraph. - **Use-case integration (G12):** the curated quickstart-template list is also offered as Step 2 of the full flow, seeding defaults for later steps. - **`back` / `reset` navigation (G3):** `back` / `b` returns to the previous step; `reset` / `r` clears in-memory state. - **`POPULAR_MODELS` alignment with web (G14):** CLI list now matches `site/js/wizard.js`'s preset cards. What's covered (web): - **Header comment (I2):** `site/js/wizard.js` opening doc block updated from "7 steps" to "9 steps" with correct list. - **Use-case key alignment (I4):** `USE_CASE_PRESETS` keys renamed (`code-copilot` → `code-assistant`, `medical-tr` → `medical-qa-tr`) so `forgelm/quickstart.py::TEMPLATES` is the single source of truth. What's covered (docs): - **Design doc (I5):** `docs/design/wizard_mode.md` rewritten to describe the actual 9-step flow, sub-package layout, and Phase 22 modernisation status banner. - **Architecture reference:** `docs/reference/architecture.md` + `-tr.md` updated for the `forgelm/wizard/` sub-package. - **Quickstart guide:** `docs/guides/quickstart.md` + `-tr.md` updated with the new flow description. - **CHANGELOG.md:** comprehensive Phase 22 entry under `[Unreleased]` / `### Added`. - **Roadmap link fix:** `docs/roadmap/phase-11-5-backlog.md` now points at `forgelm/wizard/_byod.py` rather than the old monolith. Architecture: `forgelm/wizard.py` (976 LOC monolith) split into the `forgelm/wizard/` sub-package per `docs/standards/architecture.md`'s 1000-line ceiling rule: - `_io.py` (213 LOC) — primitive prompts + navigation tokens + hardware detection. - `_state.py` (290 LOC) — `_WizardState`, persistence, summary, schema-default constants. - `_collectors.py` (688 LOC) — every `_collect_*` helper + strategy choice + use-case preset registry. - `_byod.py` (333 LOC) — Phase 11.5 / 12.5 BYOD inline ingest + audit helpers + quickstart prelude. - `_orchestrator.py` (566 LOC) — step definitions, `_drive_wizard_steps`, `_apply_strict_tier_coercion`, `run_wizard`. - `__init__.py` (176 LOC) — re-export shim for backward compatibility. Tests: 55 new tests in `tests/test_wizard_phase22.py` cover `_parse_webhook_value`, `_default_safety_probes_path`, `_collect_trainer_hyperparameters`, `_check_navigation_token`, `_apply_strict_tier_coercion`, persistence helpers, `_print_step_diff`, `_collect_webhook_config`, use-case preset registry parity with `quickstart.py::TEMPLATES`, strategy choice shape, GaLore optimiser variants, and schema-default parity for LoRA r / alpha / epochs / batch / LR. Existing `tests/test_wizard_phase11.py` test_print_wizard_summary cases updated to use the new dict-based API. Validation: 8/8 doc CI guards green (bilingual_parity, anchor_resolution, cli_help_consistency, yaml_snippets, audit_event_catalog, library_api_doc, doc_numerical_claims, bilingual_code_blocks); ruff check + format clean across the package; module-size guard reports 0 over the fail-threshold; `forgelm --config config_template.yaml --dry-run` passes; strict-tier auto-coercion produces a config that `ForgeConfig.model_validate` accepts. i18n decision: CLI wizard remains English-only per operator explicit instruction. Web wizard's 6-language `wizard.*` catalogue (en/tr/de/fr/es/zh) is unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) --- CHANGELOG.md | 77 +++ docs/design/wizard_mode.md | 93 ++- docs/guides/quickstart-tr.md | 4 +- docs/guides/quickstart.md | 2 +- docs/reference/architecture-tr.md | 6 +- docs/reference/architecture.md | 6 +- docs/roadmap/phase-11-5-backlog.md | 4 +- forgelm/wizard.py | 976 ----------------------------- forgelm/wizard/__init__.py | 176 ++++++ forgelm/wizard/_byod.py | 333 ++++++++++ forgelm/wizard/_collectors.py | 688 ++++++++++++++++++++ forgelm/wizard/_io.py | 213 +++++++ forgelm/wizard/_orchestrator.py | 566 +++++++++++++++++ forgelm/wizard/_state.py | 290 +++++++++ site/js/wizard.js | 30 +- tests/test_phase12_5.py | 6 +- tests/test_wizard_phase11.py | 81 +-- tests/test_wizard_phase22.py | 492 +++++++++++++++ 18 files changed, 2971 insertions(+), 1072 deletions(-) delete mode 100644 forgelm/wizard.py create mode 100644 forgelm/wizard/__init__.py create mode 100644 forgelm/wizard/_byod.py create mode 100644 forgelm/wizard/_collectors.py create mode 100644 forgelm/wizard/_io.py create mode 100644 forgelm/wizard/_orchestrator.py create mode 100644 forgelm/wizard/_state.py create mode 100644 tests/test_wizard_phase22.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 192f8688..7acd8101 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,83 @@ All notable changes to ForgeLM are documented here. ### Added +- **CLI wizard parity-with-web modernisation (Phase 22 / 2026-05-08).** + `forgelm --wizard` now closes the parity gap with the in-browser + `site/js/wizard.js` documented in + `docs/analysis/code_reviews/2026-05-07_cli_wizard_ux_analysis.md` + + `-2.md`. + - **Step machine:** 9-step flow (welcome / use-case / model / + strategy / trainer / dataset / training-params / compliance / + operations) with `back` / `b` to navigate backwards and `reset` / + `r` to clear in-memory state. `compliance.risk_classification` + is now collected **before** `evaluation.safety` so high-risk + auto-coercion (Article 9 / F-compliance-110) can fire reliably. + - **Trainer-specific hyperparameters (G1):** `dpo_beta`, + `simpo_beta` / `simpo_gamma`, `kto_beta`, `orpo_beta`, + `grpo_num_generations`, `grpo_max_completion_length`, + `grpo_reward_model` are now surfaced per `trainer_type`. SFT + short-circuits. + - **PEFT method breadth (G2):** the strategy step now offers all + four schema-supported `lora.method` values (`lora`, `dora`, + `pissa`, `rslora`) plus GaLore as a separate axis (6 cards + total). + - **GaLore optimiser variants (G9):** all six `galore_optim` Literal + values surfaced, including the three `_layerwise` siblings that + drop peak VRAM further. + - **RoPE scaling types (G10):** `longrope` joined `linear` / + `dynamic` / `yarn` for the full 4-of-4 schema coverage. + - **LoRA `r` schema parity (G11):** `DEFAULT_LORA_R` lowered from + `16` → `8` to match `LoraConfigModel.r`. + - **Compliance depth (G5):** `compliance` (Article 11 + Annex IV §1) + plus optional `risk_assessment` (Article 9), `data.governance` + (Article 10), `retention` (GDPR Article 5(1)(e) + 17), + `monitoring` (Article 12+17), `evaluation.benchmark`, + `evaluation.llm_judge`, `synthetic` blocks now configurable. + - **High-risk auto-coercion (G8):** `risk_classification ∈ + {high-risk, unacceptable}` automatically enables + `evaluation.safety` + `evaluation.require_human_approval` with a + visible operator notice — front-stops the schema-side + `F-compliance-110` `ConfigError`. + - **Webhook URL parsing (G15):** single prompt accepts a literal + URL or `env:VAR_NAME` reference; the URL form is `urlparse`- + validated and HTTPS is recommended (HTTP accepted with a + warning, matching `forgelm/webhook.py` runtime behaviour). + - **Safety probe path (G16):** the bundled probe set is now + resolved through `importlib.resources.files("forgelm.safety_prompts")`, + fixing the `pip install forgelm` regression where + `configs/safety_prompts/general_safety.jsonl` was the wizard + default but never shipped in the wheel. + - **Configuration summary (G17):** `_print_wizard_summary` now + dumps the full YAML alongside the labelled headline. Operator + sees every block — webhook / evaluation / compliance / risk / + retention / monitoring — without `cat`-ing the file. + - **Persistence (G6):** state snapshot at + `$XDG_CACHE_HOME/forgelm/wizard_state.yaml` (or + `~/.cache/forgelm/wizard_state.yaml`) saved after every + completed step. Schema versioned (`v: 1`); version mismatches + or parse errors silently fall back to defaults. Snapshot is + cleared on successful completion. + - **Step-diff preview (G7):** each completed step prints `+ key.path: + value` / `~ key.path: before → after` so the operator sees + exactly what changed mid-flow. + - **Beginner / expert toggle (G13):** beginner mode prefixes each + step with a 2-3-line tutorial paragraph; expert mode is silent. + - **Use-case integration (G12):** the curated quickstart-template + list is also offered as Step 2 of the full flow (in addition to + the existing prelude shortcut), seeding sensible defaults for + later steps without locking anything down. + - **Use-case key alignment with web (I4):** + `site/js/wizard.js::USE_CASE_PRESETS` keys renamed to match + `forgelm/quickstart.py::TEMPLATES` (`code-copilot` → + `code-assistant`, `medical-tr` → `medical-qa-tr`). `quickstart.py` + is now the single source of truth. + - **`POPULAR_MODELS` alignment with web (G14):** the CLI list now + matches `site/js/wizard.js`'s preset cards. + - **Step-machine docstring + design doc refresh (I2 + I5):** + `site/js/wizard.js` header comment updated from "7 steps" to "9 + steps" + correct step list; `docs/design/wizard_mode.md` + rewritten to describe the actual 9-step flow. + ### Changed - Minimum required `torch` version bumped from 2.1.0 to 2.3.0; `torch.distributed.fsdp.FSDPModule` (introduced in torch 2.3) is referenced by `tests/test_grpo_reward.py` and runtime GRPO paths. (#F-PR29-A4-07) diff --git a/docs/design/wizard_mode.md b/docs/design/wizard_mode.md index d0d755c4..df302a8a 100644 --- a/docs/design/wizard_mode.md +++ b/docs/design/wizard_mode.md @@ -1,9 +1,11 @@ # Design: Interactive Configuration Wizard (`forgelm --wizard`) -> **Status: Implemented in v0.5.5.** This document is retained as the +> **Status: Implemented in v0.5.5; modernised Phase 22 / 2026-05-08 +> (`forgelm/wizard/` sub-package).** This document is retained as the > historical design reference; the actual implementation in -> `forgelm/wizard.py` (≈800 lines) diverged from the original 6-step -> sketch in three concrete ways and is now the source of truth. +> `forgelm/wizard/` (split from a 976-line monolith into five +> focused submodules, ~270 / 300 / 600 / 280 / 460 LOC) is now the +> source of truth. ## Overview @@ -21,36 +23,65 @@ forgelm --wizard top-level `forgelm --wizard` flag is the canonical operator-facing entry point.) -## Implemented flow (`forgelm/wizard.py:799+`) +## Implemented flow (`forgelm/wizard::run_wizard`) -The shipped wizard runs **8 steps** plus an optional template prelude -(`_maybe_run_quickstart_template`): +The shipped wizard (Phase 22 / 2026-05-08 modernisation) runs **9 steps** +plus an optional quickstart-template prelude (`_maybe_run_quickstart_template`). +Step ordering and naming are kept in lockstep with the in-browser web +wizard at `site/js/wizard.js`: -0. **Quickstart prelude** *(optional)*: detect whether the operator - is starting from a bundled `forgelm/templates//config.yaml` - template (e.g. `customer-support`, `medical-qa-tr`) and, if so, - pre-fill the rest of the wizard's defaults from that template. -1. **Welcome & Hardware Detection**: detect available VRAM via - `forgelm.fit_check`; suggest a backend (`transformers` vs - `unsloth`). -2. **Model Selection**: prompt for a HuggingFace repository name; - pre-flight check for `safetensors` shards via the HF Hub API. -3. **Strategy Selection**: LoRA / QLoRA / DoRA / PiSSA / rsLoRA - selection with simplified explanations. -4. **Dataset Path**: prompt for a local file path or HF Hub dataset - id; validate format auto-detection (SFT / DPO / KTO / GRPO) on - the fly via `forgelm.data._detect_dataset_format`. -5. **Trainer + Hyperparameters**: select `trainer_type` from - `{sft, dpo, simpo, kto, orpo, grpo}` and the hyperparameters most - operators tune (`learning_rate`, `num_train_epochs`, - `per_device_train_batch_size`, `gradient_accumulation_steps`). -6. **Compliance + Safety**: prompt for `compliance.risk_classification`, - `evaluation.require_human_approval`, and the safety classifier - block (`safety.enabled`, `safety.classifier`). -7. **Output Config**: ask for a filename to save the generated YAML. -8. **Quick Run**: ask whether to start training immediately using the - generated config; if yes, dispatches to the standard training - pipeline. +0. **Quickstart prelude** *(optional)*: offer the curated + `forgelm/quickstart.py::TEMPLATES` list (`customer-support`, + `code-assistant`, `domain-expert`, `medical-qa-tr`, `grpo-math`) + as a one-shot shortcut. When accepted, the prelude generates the + config from the bundled template and skips the full 9-step flow. +1. **Welcome**: experience toggle (beginner / expert), navigation + primer (`back` / `reset` / Ctrl-C), hardware detection (torch + + CUDA + VRAM), backend hint (`unsloth` on Linux + GPU, otherwise + `transformers`). +2. **Use-case**: same registry as the prelude, surfaced inside the + full flow so operators who declined the shortcut still benefit + from sensible preselects (the choice seeds Steps 3 + 5 defaults + but the operator can still override every later answer). +3. **Model**: pick a HuggingFace Hub ID from `POPULAR_MODELS` (kept + in lockstep with `site/js/wizard.js`'s presets) or enter a custom + path. +4. **Strategy**: 6 cards (QLoRA / LoRA / DoRA / PiSSA / rsLoRA / + GaLore) covering the full `LoraConfigModel.method` Literal + + GaLore as a separate axis. Captures `lora.r`, `lora.alpha`, + `lora.target_modules` (standard / extended / full preset). +5. **Trainer + per-trainer hyperparameters**: pick `trainer_type` from + `{sft, dpo, simpo, kto, orpo, grpo}`; the wizard then prompts for + the trainer's specific knobs (`dpo_beta`, `simpo_beta` / + `simpo_gamma`, `kto_beta`, `orpo_beta`, `grpo_num_generations`, + `grpo_max_completion_length`, `grpo_reward_model`). SFT short- + circuits (no per-trainer knobs to surface). +6. **Dataset**: HF Hub ID, local JSONL, or directory of raw documents. + Directory inputs trigger inline ingestion (Phase 11.5). JSONL + inputs trigger an inline audit (Phase 12.5). Optional Article 10 + `data.governance` accordion. +7. **Training parameters**: epochs, batch size, max length, output + directory, RoPE scaling (4 schema types incl. `longrope`), + NEFTune, OOM recovery, GaLore advanced (6-variant optimizer + + rank). +8. **Compliance + risk**: `compliance` (Article 11 + Annex IV §1) + plus optional `risk_assessment` (Article 9), `data.governance` + (Article 10), `retention` (GDPR Article 5(1)(e) + 17), + `monitoring` (Article 12 + 17) accordions. This step is + deliberately collected **before** evaluation so the wizard can + front-stop the F-compliance-110 strict-tier gate. +9. **Operations + evaluation**: `evaluation.auto_revert`, + `evaluation.safety` (Llama Guard, probe set resolved through + `importlib.resources`), `evaluation.benchmark`, + `evaluation.llm_judge`, `webhook` (single prompt with `env:VAR` + prefix sugar), `synthetic` block. After the operator answers, + `_apply_strict_tier_coercion` enforces F-compliance-110 (high- + risk → safety enabled + Article 14 staging gate). + +Persistence: state snapshot is written to +`$XDG_CACHE_HOME/forgelm/wizard_state.yaml` after each completed step +so a Ctrl-C / fresh session can offer to resume. Snapshot is cleared +on successful completion or when the operator types `reset` / `r`. ## Implementation Details diff --git a/docs/guides/quickstart-tr.md b/docs/guides/quickstart-tr.md index 38e80145..3c2757cb 100644 --- a/docs/guides/quickstart-tr.md +++ b/docs/guides/quickstart-tr.md @@ -72,9 +72,7 @@ Bundled seed dataset'lerin lisansları için bkz. [LICENSES.md](https://github.c forgelm --wizard ``` -Sihirbaz model seçiminden, LoRA stratejisine, dataset'e ve -hiperparametrelere kadar yürütür. Kullanıma-hazır bir YAML config -üretir. +Sihirbaz önce curated quickstart-template kısayolu önerir; reddedilirse 9 adımlı etkileşimli akış açılır (welcome / use-case / model / strategy / trainer / dataset / training-params / compliance / operations) ve her `ForgeConfig` bloğunu kapsar — model, LoRA / DoRA / PiSSA / rsLoRA / GaLore stratejisi, trainer-spesifik hyperparam'lar (`dpo_beta` / `simpo_*` / `kto_beta` / `orpo_beta` / `grpo_*`), EU AI Act Madde 9 / 10 / 11 / 12+17 uyumluluk metadata, retention, monitoring, evaluation kapıları, webhook, sentetik veri — ve kullanıma-hazır bir YAML yazar. Geri dönmek için `back` / `b`, sıfırlamak için `reset` / `r`; state `~/.cache/forgelm/wizard_state.yaml`'da persistent, Ctrl-C / yeni oturum kaldığı yerden devam edebilir. ### Seçenek B: Şablon Kopyala diff --git a/docs/guides/quickstart.md b/docs/guides/quickstart.md index cf12a926..8fea7189 100644 --- a/docs/guides/quickstart.md +++ b/docs/guides/quickstart.md @@ -67,7 +67,7 @@ See [LICENSES.md](https://github.com/cemililik/ForgeLM/blob/main/forgelm/templat forgelm --wizard ``` -The wizard walks you through model selection, LoRA strategy, dataset, and hyperparameters. It generates a ready-to-use YAML config. +The wizard offers a curated quickstart-template shortcut first; declining opens a 9-step interactive flow (welcome / use-case / model / strategy / trainer / dataset / training-params / compliance / operations) that covers every `ForgeConfig` block — model, LoRA / DoRA / PiSSA / rsLoRA / GaLore strategy, per-trainer hyperparameters (`dpo_beta` / `simpo_*` / `kto_beta` / `orpo_beta` / `grpo_*`), EU AI Act Article 9 / 10 / 11 / 12+17 compliance metadata, retention, monitoring, evaluation gates, webhooks, synthetic data — and writes a ready-to-use YAML. Type `back` / `b` to navigate backwards, `reset` / `r` to start over; state is persisted to `~/.cache/forgelm/wizard_state.yaml` so a Ctrl-C / fresh session can resume. ### Option B: Copy Template diff --git a/docs/reference/architecture-tr.md b/docs/reference/architecture-tr.md index e0811b31..ef3bed28 100644 --- a/docs/reference/architecture-tr.md +++ b/docs/reference/architecture-tr.md @@ -33,7 +33,7 @@ forgelm --config job.yaml │ └── webhook.py → Slack/Teams bildirimleri ├── merging.py → TIES/DARE/SLERP model birleştirme ├── synthetic.py → Sentetik veri üretimi - └── wizard.py → Etkileşimli config üretici + └── wizard/ → Etkileşimli config üretici (alt-paket, Faz 22) ``` ## Dizin Yapısı @@ -75,7 +75,7 @@ ForgeLM/ │ ├── synthetic.py # Sentetik veri üretimi (öğretmen→öğrenci) │ ├── grpo_rewards.py # Yerleşik GRPO format/uzunluk ödül şekillendiricileri │ ├── quickstart.py # Tek komutlu hazır şablonlar -│ ├── wizard.py # Etkileşimli yapılandırma sihirbazı +│ ├── wizard/ # Etkileşimli yapılandırma sihirbazı (alt-paket — Faz 22) │ ├── webhook.py # Webhook bildirimleri (Slack/Teams) │ ├── _http.py # SSRF-korumalı HTTP chokepoint │ ├── _version.py # __version__ + __api_version__ (decoupled) @@ -146,7 +146,7 @@ HuggingFace-uyumlu README.md üretir: YAML front-matter, eğitim parametreleri t Öğretmen-öğrenci distillasyonu ile sentetik veri üretimi. `SyntheticDataGenerator` sınıfı bir öğretmen modeli (API tabanlı veya yerel) alır, tohum promptlardan eğitim örnekleri üretir ve formatlanmış JSONL veri setleri çıktılar. `--generate-data` CLI flag'i veya `synthetic` config bölümü ile tetiklenir. Yapılandırılabilir öğretmen backend'leri, çıktı formatları ve üretim parametrelerini destekler. ### `wizard.py` -Etkileşimli CLI sihirbazı. GPU algılama, model seçimi, LoRA stratejisi, eğitim hedefi (6 trainer), güvenlik değerlendirme (binary/confidence_weighted, kategori takibi), uyumluluk metadata yapılandırması. +Etkileşimli CLI sihirbazı. Faz 22 modernizasyonu (2026-05-08) ile `site/js/wizard.js`'e parite getirildi: 9-adımlı state machine (welcome / use-case / model / strategy / trainer / dataset / training-params / compliance / operations), trainer-spesifik hyperparam'lar (`dpo_beta` / `simpo_beta` + `simpo_gamma` / `kto_beta` / `orpo_beta` / `grpo_*`), tam PEFT method kapsamı (`lora` / `dora` / `pissa` / `rslora`) + GaLore ayrı axis, EU AI Act Madde 9 + 10 + 11 + 12+17 uyumluluk akordeon'ları, F-compliance-110 strict-tier auto-coercion, `back` / `reset` navigasyon, XDG-aware persistence (`$XDG_CACHE_HOME/forgelm/wizard_state.yaml`), step-diff preview, beginner / expert toggle ve Phase 11.5 / 12.5 BYOD inline ingest + audit helper'ları (`_offer_ingest_for_directory`, `_offer_audit_for_jsonl`). ### `webhook.py` Eğitim başlangıcı/başarı/başarısızlık durumlarında yapılandırılmış JSON payload gönderir. Yapılandırılabilir timeout ile güvenilir hata yönetimi. diff --git a/docs/reference/architecture.md b/docs/reference/architecture.md index 2dce7e24..bfa60f60 100644 --- a/docs/reference/architecture.md +++ b/docs/reference/architecture.md @@ -33,7 +33,7 @@ forgelm --config job.yaml │ └── webhook.py → Slack/Teams notifications ├── merging.py → TIES/DARE/SLERP model merge ├── synthetic.py → Synthetic data generation - └── wizard.py → Interactive config generator + └── wizard/ → Interactive config generator (sub-package, Phase 22) ``` ## Directory Layout @@ -75,7 +75,7 @@ ForgeLM/ │ ├── synthetic.py # Synthetic data generation (teacher→student) │ ├── grpo_rewards.py # Built-in GRPO format/length reward shapers │ ├── quickstart.py # Bundled one-command templates -│ ├── wizard.py # Interactive configuration wizard +│ ├── wizard/ # Interactive configuration wizard (sub-package — Phase 22) │ ├── webhook.py # Webhook notifications (Slack/Teams) │ ├── _http.py # SSRF-guarded HTTP chokepoint │ ├── _version.py # __version__ + __api_version__ (decoupled) @@ -146,7 +146,7 @@ Model merging with 4 strategies: linear interpolation, TIES-Merging (trim + sign Synthetic data generation via teacher-to-student distillation. The `SyntheticDataGenerator` class takes a teacher model (API-based or local), generates training samples from seed prompts, and outputs formatted JSONL datasets. Triggered via `--generate-data` CLI flag or `synthetic` config section. Supports configurable teacher backends, output formats, and generation parameters. ### `wizard.py` -Interactive CLI wizard for generating valid YAML configs. Detects GPU hardware, suggests backend, offers model presets, guides through LoRA strategy and training objective selection (6 trainer types with format hints), and optionally starts training immediately. +Interactive CLI wizard for generating valid YAML configs. Phase 22 modernisation (2026-05-08) brought the CLI to parity with `site/js/wizard.js`: 9-step state machine (welcome / use-case / model / strategy / trainer / dataset / training-params / compliance / operations), per-trainer hyperparameters (`dpo_beta` / `simpo_beta` + `simpo_gamma` / `kto_beta` / `orpo_beta` / `grpo_*`), full PEFT method coverage (`lora` / `dora` / `pissa` / `rslora`) plus GaLore axis, EU AI Act Article 9 + 10 + 11 + 12+17 compliance accordions, F-compliance-110 strict-tier auto-coercion, `back` / `reset` navigation, XDG-aware persistence at `$XDG_CACHE_HOME/forgelm/wizard_state.yaml`, step-diff preview, beginner / expert toggle, and the Phase 11.5 / 12.5 BYOD inline ingest + audit helpers (`_offer_ingest_for_directory`, `_offer_audit_for_jsonl`). ### `webhook.py` Sends structured JSON payloads to Slack/Teams/generic webhooks on training start, success, and failure. Supports URL from config or environment variable. Graceful error handling with configurable timeout. diff --git a/docs/roadmap/phase-11-5-backlog.md b/docs/roadmap/phase-11-5-backlog.md index 6433b238..3b0b9dd2 100644 --- a/docs/roadmap/phase-11-5-backlog.md +++ b/docs/roadmap/phase-11-5-backlog.md @@ -10,7 +10,7 @@ > [`forgelm/data_audit/`](../../forgelm/data_audit/), > [`forgelm/ingestion.py`](../../forgelm/ingestion.py), > [`forgelm/cli/`](../../forgelm/cli/), -> [`forgelm/wizard.py`](../../forgelm/wizard.py); CLI: +> [`forgelm/wizard/`](../../forgelm/wizard/); CLI: > `forgelm audit ` (new subcommand, with `--data-audit` preserved > as deprecation alias) + token-aware `forgelm ingest` flags; tests: > [`tests/test_data_audit.py`](../../tests/test_data_audit.py), @@ -36,7 +36,7 @@ for large corpora and a cleaner CLI shape. | **8** | PII severity tiers | `PII_SEVERITY`, `_build_pii_severity` in [`data_audit.py`](../../forgelm/data_audit/) | Audit JSON now carries `pii_severity` with `worst_tier`, `by_tier`, and `by_type`. Notes line lead with the worst tier (e.g. `WORST tier: CRITICAL`). | | **9** | `summarize_report` truncation policy | [`data_audit.py`](../../forgelm/data_audit/) | Default `verbose=False` folds zero-finding splits into a single tail line. CLI exposes `--verbose` on the new `audit` subcommand for full output. | | **10** | Structured ingestion notes | `IngestionResult.notes_structured` in [`ingestion.py`](../../forgelm/ingestion.py) | Free-text `extra_notes` stays for humans; new `notes_structured` carries machine-readable `{key: value}` for CI/CD consumers. JSON output already exposes both. | -| **11** | Wizard "ingest first" entry point | `_offer_ingest_for_directory` + `_prompt_dataset_path_with_ingest_offer` in [`wizard.py`](../../forgelm/wizard.py) | Both BYOD quickstart and the full 8-step wizard now offer to run `ingest` inline when the typed path is a directory of raw documents. | +| **11** | Wizard "ingest first" entry point | `_offer_ingest_for_directory` + `_prompt_dataset_path_with_ingest_offer` in [`forgelm/wizard/_byod.py`](../../forgelm/wizard/_byod.py) | Both BYOD quickstart and the full 9-step wizard now offer to run `ingest` inline when the typed path is a directory of raw documents. | | **12** | Atomic audit-report write | `_atomic_write_json` in [`data_audit.py`](../../forgelm/data_audit/) | Writes via `tempfile.NamedTemporaryFile` + `os.replace` so a crashed audit can never leave a half-written `data_audit_report.json` on disk. | ## Measured speedups (xxhash + lru_cache hot path) diff --git a/forgelm/wizard.py b/forgelm/wizard.py deleted file mode 100644 index c559554d..00000000 --- a/forgelm/wizard.py +++ /dev/null @@ -1,976 +0,0 @@ -"""Interactive configuration wizard for ForgeLM. - -Generates a valid config.yaml through step-by-step prompts. -No external dependencies required — uses stdlib input(). -""" - -import json -import logging -import os -import re -import sys -from pathlib import Path -from typing import Optional - -import yaml - -logger = logging.getLogger("forgelm.wizard") - - -def _print(*args, **kwargs) -> None: - """Indirection over the builtin :func:`print`. - - Mirrors the pattern used by :mod:`forgelm.chat` so wizard output can be - captured via ``capsys`` in tests and (later) re-routed through ``rich`` - or a structured logger without touching every call site. Accepts the - same ``*args, **kwargs`` as :func:`print` so existing call sites need - only a name swap, not a signature change. - """ - # NOTE: bypass the wizard's own ``print(`` → ``_print(`` rewrite by going - # through ``builtins.print`` — a bare ``print(...)`` here would be picked - # up by future codemods and turned into infinite recursion. - import builtins as _builtins - - _builtins.print(*args, **kwargs) - - -# Defaults -DEFAULT_MAX_LENGTH = 2048 -DEFAULT_LORA_R = 16 -DEFAULT_LORA_ALPHA = 32 -DEFAULT_DROPOUT = 0.1 -DEFAULT_EPOCHS = 3 -DEFAULT_BATCH_SIZE = 4 -DEFAULT_LR = 2e-5 - -POPULAR_MODELS = [ - "meta-llama/Llama-3.1-8B-Instruct", - "mistralai/Mistral-7B-Instruct-v0.3", - "google/gemma-2-9b-it", - "Qwen/Qwen2.5-7B-Instruct", - "HuggingFaceTB/SmolLM2-1.7B-Instruct", -] - -TARGET_MODULE_PRESETS = { - "standard": ["q_proj", "v_proj"], - "extended": ["q_proj", "k_proj", "v_proj", "o_proj"], - "full": ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], -} - -# Common dataset-format hints for preference-based trainers (DPO/SimPO/ORPO) -_PREFERENCE_COLUMNS_HINT = "Columns: prompt, chosen, rejected" - -# HF Hub dataset IDs look like "/" — exactly one slash, with the -# allowed character set used by the Hub. We accept these BYOD inputs without -# touching the local filesystem; the trainer resolves them at runtime. The -# Hub allows usernames + repo names up to 96 chars each, so we bound the -# repeats per docs/standards/regex.md rule 3 (defence-in-depth, no behaviour -# change vs. ``+``). re.ASCII forces ``\w`` to mean ``[A-Za-z0-9_]`` — Hub IDs -# are ASCII-only, and Unicode-aware ``\w`` would silently accept characters -# the Hub itself rejects. -_HF_HUB_ID_RE = re.compile(r"^[\w.-]{1,96}/[\w.-]{1,96}$", flags=re.ASCII) - - -def _prompt(question: str, default: str = "") -> str: - """Prompt user with a default value.""" - suffix = f" [{default}]" if default else "" - answer = input( - f" {question}{suffix}: " - ).strip() # pragma: no cover -- stdin read, exercised via builtins.input patch - return answer if answer else default - - -def _prompt_choice(question: str, options: list, default: int = 1) -> str: - """Prompt user to pick from numbered options.""" - _print(f"\n {question}") - for i, opt in enumerate(options, 1): - marker = " *" if i == default else "" - _print(f" {i}) {opt}{marker}") - choice = input(f" Choice [{default}]: ").strip() # pragma: no cover -- stdin read - try: - idx = int(choice) if choice else default - return options[idx - 1] - except (ValueError, IndexError): - return options[default - 1] - - -def _prompt_yes_no(question: str, default: bool = True) -> bool: - """Prompt user for yes/no.""" - hint = "Y/n" if default else "y/N" - answer = input(f" {question} [{hint}]: ").strip().lower() # pragma: no cover -- stdin read - if not answer: - return default - return answer in ("y", "yes") - - -def _prompt_int(question: str, default: int, min_val: int = 1, max_val: int = 65536) -> int: - """Prompt for an integer, re-asking until valid.""" - while True: - raw = _prompt(question, str(default)) - try: - val = int(raw) - if min_val <= val <= max_val: - return val - _print(f" Value must be between {min_val} and {max_val}.") - except ValueError: - _print(" Please enter a valid integer.") - - -def _detect_hardware() -> dict: - """Detect GPU hardware if available.""" - info = {"gpu_available": False, "gpu_name": None, "vram_gb": None, "cuda_version": None} - try: - import torch - - if torch.cuda.is_available(): - info["gpu_available"] = True - info["gpu_name"] = torch.cuda.get_device_name(0) - info["vram_gb"] = round(torch.cuda.get_device_properties(0).total_mem / (1024**3), 1) - info["cuda_version"] = torch.version.cuda - except ImportError: - pass - return info - - -def _collect_rope_scaling(max_length: int) -> Optional[dict]: - """Prompt for RoPE scaling parameters when context is long; otherwise None.""" - if max_length <= 4096: - return None - _print(f"\n Long context detected ({max_length} tokens).") - if not _prompt_yes_no("Enable RoPE scaling for extended context?", default=True): - return None - rope_type = _prompt_choice( - "RoPE scaling type:", - ["linear (simple, proven)", "dynamic (adaptive)", "yarn (best quality, newer)"], - default=1, - ) - base_context = 4096 - rope_factor = max_length / base_context - _print( - f" Note: RoPE factor {rope_factor:.1f}x computed assuming base context of " - f"{base_context} tokens. Adjust manually if your model has a different " - f"original context length (e.g., Llama 3.1 = 131072, Mistral v0.3 = 32768)." - ) - return {"type": rope_type.split(" ")[0], "factor": rope_factor} - - -def _collect_neftune_alpha() -> Optional[float]: - """Prompt for NEFTune noise injection; returns alpha or None.""" - if not _prompt_yes_no("Enable NEFTune noise injection (improves training quality)?", default=False): - return None - return float(_prompt("NEFTune noise alpha", "5.0")) - - -def _collect_webhook_config() -> Optional[dict]: - """Prompt for webhook configuration; returns the webhook section or None.""" - if not _prompt_yes_no("Configure webhook notifications?", default=False): - return None - webhook_url = _prompt("Webhook URL (or leave empty for env var)") - if webhook_url: - return {"url": webhook_url} - env_var = _prompt("Environment variable name for webhook URL", "FORGELM_WEBHOOK_URL") - return {"url_env": env_var} - - -def _collect_safety_config() -> Optional[dict]: - """Prompt for safety eval; returns the safety section or None.""" - if not _prompt_yes_no("Enable safety evaluation (Llama Guard)?", default=False): - return None - scoring = _prompt_choice( - "Safety scoring mode:", - ["binary (simple safe/unsafe ratio)", "confidence_weighted (uses classifier confidence)"], - default=1, - ) - scoring_mode = "confidence_weighted" if "confidence" in scoring else "binary" - safety_config: dict = { - "enabled": True, - "test_prompts": "configs/safety_prompts/general_safety.jsonl", - "scoring": scoring_mode, - } - if scoring_mode == "confidence_weighted": - safety_config["min_safety_score"] = 0.85 - if _prompt_yes_no("Track harm categories (S1-S14)?", default=False): - safety_config["track_categories"] = True - safety_config["severity_thresholds"] = {"critical": 0, "high": 0.01, "medium": 0.05} - return safety_config - - -def _collect_evaluation_config() -> Optional[dict]: - """Prompt for auto-revert + safety; returns the evaluation section or None.""" - if not _prompt_yes_no("Enable auto-revert (discard model if quality drops)?", default=False): - return None - max_loss = _prompt("Max acceptable loss (leave empty for baseline-only)", "") - eval_config: dict = { - "auto_revert": True, - "max_acceptable_loss": float(max_loss) if max_loss else None, - } - safety = _collect_safety_config() - if safety: - eval_config["safety"] = safety - return eval_config - - -def _collect_compliance_config() -> Optional[dict]: - """Prompt for EU AI Act metadata; returns the compliance section or None.""" - if not _prompt_yes_no("Configure EU AI Act compliance metadata?", default=False): - return None - return { - "provider_name": _prompt("Organization name"), - "intended_purpose": _prompt("Intended purpose of the model"), - "risk_classification": _prompt_choice( - # Five-tier set kept in lockstep with ``forgelm.config.RiskTier`` - # so the wizard never offers a value the Pydantic schema would - # reject (and never hides a value the schema accepts). - "Risk classification:", - ["unknown", "minimal-risk", "limited-risk", "high-risk", "unacceptable"], - default=1, # ``minimal-risk`` — same default as the Pydantic field. - ), - } - - -def _save_config_to_file(config: dict, requested_filename: str) -> str: - """Write *config* as YAML; falls back to a unique filename on OSError.""" - try: - with open(requested_filename, "w") as f: - yaml.dump(config, f, default_flow_style=False, sort_keys=False) - _print(f"\n Config saved to: {requested_filename}") - logger.info("Wizard config saved to %s", requested_filename) - return requested_filename - except OSError as e: - logger.error("Could not save wizard config to %s: %s", requested_filename, e) - _print(f"\n Error: Could not save config to {requested_filename}: {e}") - - # Pick a fallback that's guaranteed different from the path that just failed - # (a hardcoded "my_config.yaml" would just re-raise the same OSError when - # the original request was already that filename). - from datetime import datetime as _dt - - base = os.path.splitext(os.path.basename(requested_filename))[0] or "my_config" - fallback = os.path.join(os.path.expanduser("~"), f"{base}_{_dt.now().strftime('%Y%m%d_%H%M%S')}.yaml") - try: - with open(fallback, "w") as f: - yaml.dump(config, f, default_flow_style=False, sort_keys=False) - _print(f" Saved to fallback location: {fallback}") - logger.info("Wizard config saved to fallback location %s", fallback) - return fallback - except OSError as e: - logger.error("Fallback wizard config save also failed (%s): %s", fallback, e) - _print(f" Fallback save also failed ({fallback}): {e}") - raise - - -def _print_wizard_summary( - *, - model_name: str, - suggested_backend: str, - use_galore: bool, - load_in_4bit: bool, - use_dora: bool, - trainer_type: str, - lora_r: int, - lora_alpha: int, - dataset_path: str, - epochs: int, - batch_size: int, - output_dir: str, -) -> None: - """Pretty-print the chosen configuration before the start-now prompt.""" - _print("\n" + "=" * 60) - _print(" Configuration Summary") - _print("=" * 60) - _print(f" Model: {model_name}") - _print(f" Backend: {suggested_backend}") - if use_galore: - strategy_str = "GaLore" - elif load_in_4bit: - strategy_str = "QLoRA" - else: - strategy_str = "LoRA" - if use_dora: - strategy_str += " + DoRA" - _print(f" Strategy: {strategy_str}") - _print(f" Trainer: {trainer_type.upper()}") - _print(f" LoRA: r={lora_r}, alpha={lora_alpha}") - _print(f" Dataset: {dataset_path}") - _print(f" Epochs: {epochs}, Batch: {batch_size}") - _print(f" Output: {output_dir}/final_model") - _print() - - -def _select_model() -> str: - """Prompt for a model from POPULAR_MODELS or allow a custom entry.""" - _print("\n[2/8] Model Selection") - _print(" Popular models:") - for i, m in enumerate(POPULAR_MODELS, 1): - _print(f" {i}) {m}") - _print(f" {len(POPULAR_MODELS) + 1}) Custom (enter your own)") - model_choice = input(" Choice [1]: ").strip() # pragma: no cover -- stdin read - try: - idx = int(model_choice) if model_choice else 1 - if idx <= len(POPULAR_MODELS): - return POPULAR_MODELS[idx - 1] - return _prompt("Enter HuggingFace model name or local path") - except (ValueError, IndexError): - return POPULAR_MODELS[0] - - -def _derive_strategy_flags(strategy: str) -> tuple: - """Decode the strategy menu choice into (load_in_4bit, use_dora, use_galore).""" - return ("QLoRA" in strategy, "DoRA" in strategy, "GaLore" in strategy) - - -def _parse_trainer_type(objective: str) -> tuple: - """Return (trainer_type, dataset_format_hint) for the chosen objective.""" - trainer_type = objective.split(" — ")[0].strip().lower() - dataset_format_hint = { - "sft": "Columns: System (opt), User/instruction, Assistant/output — or 'messages' list", - "dpo": _PREFERENCE_COLUMNS_HINT, - "simpo": _PREFERENCE_COLUMNS_HINT, - "orpo": _PREFERENCE_COLUMNS_HINT, - "kto": "Columns: prompt, completion, label (boolean: true=good, false=bad)", - "grpo": "Columns: prompt (model generates responses during training)", - } - hint = dataset_format_hint.get(trainer_type, "Standard format") - return trainer_type, hint - - -def _collect_galore_config(use_galore: bool) -> dict: - """Prompt for GaLore-specific knobs when GaLore was selected; otherwise empty.""" - if not use_galore: - _print("\n[7/8] Advanced Options") - return {} - _print("\n[7/8] GaLore Configuration") - galore_rank = _prompt_int("GaLore rank (lower = less memory)", 128, min_val=1, max_val=4096) - galore_optim = _prompt_choice( - "GaLore optimizer:", - ["galore_adamw (standard)", "galore_adamw_8bit (less memory)", "galore_adafactor (experimental)"], - default=1, - ) - return { - "galore_enabled": True, - "galore_optim": galore_optim.split(" ")[0], - "galore_rank": galore_rank, - "galore_update_proj_gap": 200, - "galore_scale": 0.25, - } - - -_BYOD_CANCEL_TOKENS = ("cancel", "c", "q", "quit") - - -_BYOD_LOCAL_NOT_FOUND = object() -"""Sentinel for ``_validate_local_jsonl``: file does not exist on disk. - -Distinct from ``None`` (file exists but is not valid JSONL) so the caller -can fall back to an HF Hub-ID interpretation only when the local path -isn't there at all. -""" - - -_INGEST_SUPPORTED_EXTENSIONS = (".pdf", ".docx", ".epub", ".txt", ".md") -"""Extensions that the wizard's "ingest first" prompt knows how to convert. - -Kept in sync with :data:`forgelm.ingestion.SUPPORTED_EXTENSIONS` — duplicated -here so the wizard's directory-scan check doesn't pay the ingestion module's -import cost when the user is bringing a JSONL straight in.""" - - -def _directory_has_ingestible_files(directory: Path, recursive: bool = True) -> bool: - """True if ``directory`` contains at least one file the ingester can read.""" - pattern = "**/*" if recursive else "*" - for entry in directory.glob(pattern): - if entry.is_file() and entry.suffix.lower() in _INGEST_SUPPORTED_EXTENSIONS: - return True - return False - - -def _offer_ingest_for_directory(directory: Path) -> Optional[str]: - """Phase 11.5 wizard hook: convert a directory of raw docs into JSONL inline. - - Replaces the previous "you must run forgelm ingest first" hint with an - actionable, in-wizard step. If the directory has no ingestible files, - we surface the original guidance and let the caller re-prompt; otherwise - we offer to run :func:`forgelm.ingestion.ingest_path` directly and feed - the resulting JSONL straight back into the wizard's BYOD path. - - Returns: - The absolute path to the produced JSONL on success, or ``None`` when - the operator declines / cancels / ingestion fails (the caller treats - ``None`` as "re-prompt for a different dataset path"). - """ - resolved = directory.expanduser().resolve() - if not _directory_has_ingestible_files(resolved): - _print( - f" '{resolved}' is a directory, but it doesn't contain any " - f"{', '.join(_INGEST_SUPPORTED_EXTENSIONS)} files. " - "Pass a JSONL file or a directory with ingestible documents." - ) - return None - - if not _prompt_yes_no( - f"\n '{resolved}' is a directory of raw documents. Run ingestion now and use the resulting JSONL?", - default=True, - ): - _print( - " Skipped — to ingest manually:\n" - f" forgelm ingest {resolved} --recursive --output data/from_docs.jsonl\n" - " Then re-run the wizard with the resulting JSONL path." - ) - return None - - # Anchor the default JSONL path to the source corpus's parent so the - # generated file lives next to the inputs by default. This avoids - # silent surprises when ``cwd`` differs from the source directory and - # keeps a paired (corpus dir / corpus.jsonl) layout that operators - # already use. Users that want a project-rooted ``data/`` location - # can still type a relative path at the prompt. - out_dir = (resolved.parent / "data").resolve() - default_out = out_dir / f"{resolved.name}_ingested.jsonl" - out_path_raw = _prompt("Output JSONL path", str(default_out)) - out_path = Path(out_path_raw).expanduser().resolve() - - pii_mask = _prompt_yes_no( - "Mask detected PII (emails, phones, IDs) before writing? Recommended for shared corpora.", - default=False, - ) - - try: - from .ingestion import ingest_path - except ImportError as exc: # pragma: no cover — extras-skip path - _print(f" ingestion subsystem unavailable: {exc}") - return None - - _print(f"\n Running ingest on '{resolved}' (this may take a moment for large corpora)…") - try: - result = ingest_path( - str(resolved), - output_path=str(out_path), - recursive=True, - pii_mask=pii_mask, - ) - except (FileNotFoundError, ValueError) as exc: - _print(f" Ingest failed: {exc}") - return None - except (PermissionError, IsADirectoryError, OSError) as exc: - # Filesystem-level errors (output path not writable, source tree - # locked, ENOSPC, broken symlink during walk, etc.) — the operator - # may want to retype the output path or fix permissions and retry, - # so we return None to re-prompt rather than crash the wizard. - _print(f" Ingest failed due to filesystem error: {exc} — check permissions or output path.") - return None - except ImportError as exc: - _print( - f" Ingest needs the optional 'ingestion' extra: {exc}\n Install with: pip install 'forgelm[ingestion]'" - ) - return None - - if result.chunk_count == 0: - _print( - " Ingestion produced 0 chunks — the directory had no extractable text. " - "Pass a JSONL file or a directory with text-bearing documents." - ) - return None - - _print(f" Ingest complete: {result.chunk_count} chunk(s) from {result.files_processed} file(s) → {out_path}") - # Phase 12.5: now that we have a JSONL produced from raw docs, offer to - # run a quality + governance audit on it before handing it to the wizard. - # Closes the BYOD loop — operators see length / language / dedup / PII - # / secrets / quality verdicts at config-time, not 30 minutes into a - # training run. - _offer_audit_for_jsonl(out_path) - return str(out_path) - - -_AUDIT_LARGE_FILE_THRESHOLD_BYTES: int = 100 * 1024 * 1024 # 100 MB -"""Threshold above which the wizard's audit offer warns operators that the -scan can take real time. Below 100 MB the runtime is dominated by Python -startup; above 100 MB the dedup signature pass becomes a meaningful -fraction of wall clock and the operator deserves to know before saying -yes. Tuned empirically against the streaming reader's throughput -(simhash + LSH banding ≈ 8-15 MB/s of JSONL on a single CPU).""" - - -def _offer_audit_for_jsonl(jsonl_path: Path) -> bool: - """Phase 12.5 wizard hook: optionally audit a JSONL the user just selected. - - Mirrors :func:`_offer_ingest_for_directory`'s shape — a non-blocking - interactive offer that, on acceptance, runs the audit and prints a - short verdict before the wizard continues. Returns ``True`` when the - audit ran (regardless of findings) and ``False`` when the operator - declined or the audit could not start. Audit failures are surfaced - as warnings: the audit is informational, not a gate, and a malformed - JSONL or missing optional dep must NOT crash the wizard. - - Exception ladder (per :doc:`docs/standards/error-handling.md`'s - "log-and-swallow is permitted only on explicitly-non-fatal paths" - rule): the courtesy audit is one such path — the wizard's contract - is "produce a config", not "vouch for the dataset". The bare - ``except Exception`` at the bottom of the ladder is the exception - the standard mentions, scoped to a single short call (not a - library-wide swallow), and explicitly re-raises ``BaseException`` - subclasses (``KeyboardInterrupt``, ``SystemExit``) by virtue of - matching ``Exception`` only. - """ - # File-size-aware copy: small JSONL → "~ seconds" is honest; large - # JSONL → warn the operator the scan takes real time before they - # commit to it. The streaming audit doesn't load the whole file into - # memory, so accepting on a large file is safe — just slow. - try: - size_bytes = jsonl_path.stat().st_size - except OSError: - size_bytes = 0 # Fall through to the small-file copy; the - # subsequent ``audit_dataset`` call surfaces the real OSError. - if size_bytes > _AUDIT_LARGE_FILE_THRESHOLD_BYTES: - size_mb = size_bytes / (1024 * 1024) - prompt = ( - f"\n Run a quality + governance audit on '{jsonl_path}' " - f"({size_mb:.0f} MB) before training? Scan is CPU-only and " - "streams the file; runtime depends on size (≈ 8–15 MB/s of " - "JSONL on a single CPU)." - ) - else: - prompt = ( - f"\n Run a quality + governance audit on '{jsonl_path}' " - "before training? (length stats, language, near-duplicates, " - "PII, secrets — CPU-only, runtime depends on dataset size)" - ) - if not _prompt_yes_no(prompt, default=True): - _print(" Skipped — audit can be run later via:") - _print(f" forgelm audit {jsonl_path}") - return False - - try: - # Lazy import — keeps the wizard cold-start cheap and matches the - # existing pattern used elsewhere in this file (see - # ``_offer_ingest_for_directory``'s ``ingest_path`` import). - from .data_audit import audit_dataset, summarize_report - except ImportError as exc: # pragma: no cover — extras-skip path - _print(f" audit subsystem unavailable: {exc}") - return False - - _print(f"\n Running audit on '{jsonl_path}'…") - try: - # Default flags only — the courtesy audit doesn't reach for - # ``--quality-filter`` / ``--pii-ml`` / ``--croissant`` because - # those need either an optional extra or a configuration choice - # the wizard hasn't asked the operator about. Operators that - # want the richer audit run ``forgelm audit`` directly. - report = audit_dataset(str(jsonl_path)) - except (FileNotFoundError, ValueError, OSError) as exc: - # Filesystem / parse errors — same defensive shape we use for - # ``_offer_ingest_for_directory`` so the wizard never aborts on a - # courtesy step. - _print(f" Audit could not run: {exc}") - return False - except ImportError as exc: - # The wizard's audit hook never asks for ``--dedup-method=minhash`` - # or any other feature gated on an optional extra; an ImportError - # here would only fire if a future feature inside ``audit_dataset`` - # added an unguarded import. Surface and recover. - _print(f" Audit could not run (missing optional dep): {exc}") - return False - except Exception as exc: # noqa: BLE001 — see docstring - # Last-ditch guard: a malformed JSONL row mid-file, a datasketch - # version-mismatch surfaced through datasketch's own internals, or - # an unanticipated edge case in a downstream audit module must NOT - # crash the wizard. The audit is informational; we log loudly and - # let the operator continue. ``KeyboardInterrupt`` / - # ``SystemExit`` propagate by virtue of being ``BaseException`` - # subclasses, so Ctrl-C still aborts the wizard cleanly. - _print(f" Audit could not run: {exc}") - return False - - _print("\n Audit complete. Summary:") - _print(summarize_report(report, verbose=False)) - return True - - -def _prompt_dataset_path_with_ingest_offer(question: str) -> str: - """Prompt for a dataset path; auto-offer ingestion when the user gives a directory. - - Used by the full 8-step wizard's Step 5. Mirrors the BYOD path's - behaviour: a typed directory triggers the Phase 11.5 "ingest first" - helper, which produces a JSONL the wizard can use directly. Hub IDs - and JSONL paths flow through unchanged. - """ - while True: - raw = _prompt(question, "").strip() - if not raw: - _print(" A dataset reference is required.") - continue - candidate = Path(raw).expanduser() - if candidate.is_dir(): - ingested = _offer_ingest_for_directory(candidate) - if ingested is None: - continue - # ``_offer_ingest_for_directory`` already invoked the audit hook - # on the produced JSONL — don't double-prompt here. - return ingested - # Phase 12.5: when the path resolves to an existing local JSONL, - # offer to audit it before continuing. Hub IDs and non-existent - # paths flow through unchanged (no local file to audit). - if candidate.is_file() and candidate.suffix.lower() in (".jsonl", ".json"): - _offer_audit_for_jsonl(candidate.resolve()) - return raw - - -def _validate_local_jsonl(raw_path: str): - """Validate a user-supplied JSONL path. - - Returns: - - The absolute path string when the file exists and parses as JSONL. - - ``_BYOD_LOCAL_NOT_FOUND`` when the path doesn't exist on disk - (caller may fall back to HF Hub ID semantics). - - ``None`` when the file exists but either is a directory the user - declined to ingest, fails JSONL parsing, or otherwise can't be - loaded. ``None`` signals "re-prompt"; the validation message is - already printed. - - Phase 11.5: when the path resolves to a directory of raw documents, - we now offer to run ingestion inline instead of just printing a hint. - On acceptance the produced JSONL path is returned as if the user had - typed it directly. - """ - resolved = Path(raw_path).expanduser() - if resolved.is_dir(): - ingested = _offer_ingest_for_directory(resolved) - if ingested is None: - return None - # ``_offer_ingest_for_directory`` already invokes the Phase 12.5 - # audit hook on the produced JSONL — don't re-prompt here. - return ingested - if not resolved.is_file(): - return _BYOD_LOCAL_NOT_FOUND - try: - with open(resolved, "r", encoding="utf-8") as fh: - first_line = next((line for line in fh if line.strip()), "") - if not first_line: - raise ValueError("file is empty") - json.loads(first_line) - except (OSError, ValueError) as e: - _print(f" File is not valid JSONL (first line failed to parse): {e}") - return None - # Phase 12.5: confirmed local JSONL → offer to audit it before returning - # the path to the caller. Closes the BYOD audit loop documented in - # docs/roadmap/phase-12-5-backlog.md row #4. - _offer_audit_for_jsonl(resolved.resolve()) - return str(resolved.resolve()) - - -def _resolve_byod_dataset_path() -> Optional[str]: - """Prompt the user for a BYOD dataset path and validate it. - - Returns the validated path (HF Hub ID strings are returned verbatim, - local paths are returned as resolved absolute paths). Returns ``None`` - when the user types ``cancel`` / ``quit`` — that's the signal for the - caller to fall back to the full wizard. - - The loop is resilient to typos: an empty input re-prompts, a missing - file re-prompts, malformed JSONL re-prompts. Surfaces validation - failures immediately instead of 30 seconds into the training subprocess. - - HF Hub IDs (single-slash ``/`` strings matching - ``_HF_HUB_ID_RE``) bypass filesystem validation. The regex already - enforces a single ``/``. - """ - while True: - dataset_path = _prompt( - "Path to your dataset JSONL (must exist as a JSONL file) or HF Hub ID, " - "or 'cancel' to fall back to the full wizard", - "", - ) - if dataset_path.strip().lower() in _BYOD_CANCEL_TOKENS: - _print(" Cancelled — falling back to the full wizard.") - return None - if not dataset_path: - _print(" A dataset path is required for this template. Type 'cancel' to use the full wizard instead.") - continue - - # Try local file first. The HF Hub ID regex would otherwise misclassify - # relative local paths like "data/train.jsonl" as Hub IDs (single slash, - # safe-name char class). Only fall back to HF Hub semantics when the - # path doesn't exist on disk at all. - result = _validate_local_jsonl(dataset_path) - if isinstance(result, str): - return result - if result is None: - # File exists but JSONL parse failed; message already printed. - continue - - # result is the _BYOD_LOCAL_NOT_FOUND sentinel — try HF Hub ID. - if _HF_HUB_ID_RE.match(dataset_path): - _print(f" Treating '{dataset_path}' as an HF Hub dataset ID (no local validation).") - return dataset_path - - _print(f" Path not found or not a regular file: {dataset_path}") - - -def _maybe_run_quickstart_template() -> Optional[str]: - """Offer the quickstart template path before the full 8-step wizard. - - Returns the generated config path when the user picks a template; - returns ``None`` when they decline (signal to fall through to the full - flow). - """ - from .quickstart import TEMPLATES, list_templates, run_quickstart - - _print("\n" + "=" * 60) - _print(" ForgeLM Configuration Wizard") - _print("=" * 60) - - if not _prompt_yes_no( - "\nStart from a curated quickstart template? (recommended for first runs)", - default=True, - ): - return None - - _print("\nAvailable templates:") - names = [] - for tpl in list_templates(): - bundled = "[x] data" if tpl.bundled_dataset else "[ ] BYOD" - names.append(tpl.name) - _print(f" {len(names)}) {tpl.name} — {tpl.title} ({bundled}, ~{tpl.estimated_minutes}min)") - raw = _prompt("Pick a template by number or name", names[0]) - - chosen: Optional[str] = None - if raw.isdigit(): - idx = int(raw) - if 1 <= idx <= len(names): - chosen = names[idx - 1] - elif raw in TEMPLATES: - chosen = raw - if chosen is None: - _print(f" Could not interpret '{raw}'. Falling back to the full wizard.") - return None - - template = TEMPLATES[chosen] - if not template.bundled_dataset: - _print(f" '{chosen}' is BYOD — bring your own JSONL dataset.") - dataset_path = _resolve_byod_dataset_path() - if dataset_path is None: - return None - else: - dataset_path = "" - - try: - result = run_quickstart( - chosen, - dataset_override=dataset_path or None, - ) - except (FileNotFoundError, ValueError) as e: - _print(f" Quickstart failed: {e}. Falling back to the full wizard.") - return None - - _print(f"\n Quickstart config generated at: {result.config_path}") - _print(f" Selected model: {result.chosen_model} ({result.selection_reason})") - _print(f" Dataset : {result.dataset_path}") - _print() - return str(result.config_path) - - -def _finalize_quickstart_path(quickstart_path: str) -> Optional[str]: - """Ask whether to start training now with the quickstart-generated config.""" - if _prompt_yes_no("Start training now with the generated config?", default=False): - _print(f"\n Running: forgelm --config {quickstart_path}") - return quickstart_path - _print("\n To start training later, run:") - _print(f" forgelm --config {quickstart_path}") - return None - - -def _detect_hardware_and_backend() -> tuple: - """Run the wizard's hardware-detection step and pick a backend hint.""" - _print("\n[1/8] Hardware Detection") - hw = _detect_hardware() - if hw["gpu_available"]: - _print(f" GPU detected: {hw['gpu_name']} ({hw['vram_gb']} GB VRAM, CUDA {hw['cuda_version']})") - else: - _print(" No GPU detected. Training will use CPU (very slow for real workloads).") - - suggested_backend = "transformers" - if hw["gpu_available"] and sys.platform == "linux": - suggested_backend = "unsloth" - _print(" Recommended backend: unsloth (Linux + GPU detected)") - elif hw["gpu_available"]: - _print(" Recommended backend: transformers (Unsloth requires Linux)") - return hw, suggested_backend - - -def run_wizard() -> Optional[str]: - """Run the interactive configuration wizard. - - Returns the path to the generated config file when the user opts to start - training immediately, or ``None`` when the user defers — callers must - handle both cases. - """ - quickstart_path = _maybe_run_quickstart_template() - if quickstart_path is not None: - return _finalize_quickstart_path(quickstart_path) - - # Full 8-step flow (fallback when user declined the quickstart shortcut). - _print("\n Falling back to the full configuration wizard.") - return _run_full_wizard() - - -def _run_full_wizard() -> Optional[str]: - """8-step interactive flow producing a hand-rolled config.yaml.""" - # Step 1: Hardware Detection - _, suggested_backend = _detect_hardware_and_backend() - - # Step 2: Model Selection - model_name = _select_model() - _print(f" Selected: {model_name}") - - # Step 3: Strategy Selection - _print("\n[3/8] Fine-Tuning Strategy") - strategies = [ - "QLoRA (4-bit quantization — recommended, lowest memory)", - "LoRA (full precision — more memory, slightly better quality)", - "QLoRA + DoRA (4-bit + weight decomposition — best quality, more compute)", - "GaLore (full-parameter training via gradient projection — no adapters, lowest peak VRAM)", - ] - strategy = _prompt_choice("Choose your fine-tuning strategy:", strategies, default=1) - load_in_4bit, use_dora, use_galore = _derive_strategy_flags(strategy) - - # LoRA parameters - target_preset = _prompt_choice( - "Target modules:", - ["standard (q_proj, v_proj)", "extended (q, k, v, o)", "full (all linear layers)"], - default=1, - ) - preset_key = target_preset.split(" ")[0] - target_modules = TARGET_MODULE_PRESETS.get(preset_key, TARGET_MODULE_PRESETS["standard"]) - - lora_r = _prompt_int("LoRA rank (r)", DEFAULT_LORA_R, min_val=1, max_val=512) - lora_alpha = _prompt_int("LoRA alpha", lora_r * 2, min_val=1, max_val=1024) - - # Step 4: Training Objective - _print("\n[4/8] Training Objective") - objectives = [ - "SFT — Supervised Fine-Tuning (standard instruction tuning)", - "DPO — Direct Preference Optimization (chosen/rejected pairs)", - "SimPO — Simple Preference Optimization (no reference model, lower memory)", - "KTO — Binary feedback alignment (thumbs up/down, practical for production)", - "ORPO — Odds Ratio Preference Optimization (SFT + alignment in one stage)", - "GRPO — Group Relative Policy Optimization (reasoning RL, like DeepSeek-R1)", - ] - objective = _prompt_choice("Choose your training objective:", objectives, default=1) - trainer_type, dataset_format_hint = _parse_trainer_type(objective) - _print(f" Dataset format: {dataset_format_hint}") - - # Step 5: Dataset - _print("\n[5/8] Dataset") - dataset_path = _prompt_dataset_path_with_ingest_offer( - "HuggingFace dataset name or local file path (or directory of raw documents)", - ) - - # Step 6: Training Parameters - _print("\n[6/8] Training Parameters") - epochs = _prompt_int("Number of epochs", DEFAULT_EPOCHS, min_val=1, max_val=1000) - batch_size = _prompt_int("Batch size per device", DEFAULT_BATCH_SIZE, min_val=1, max_val=512) - max_length = _prompt_int("Max sequence length", DEFAULT_MAX_LENGTH, min_val=64, max_val=131072) - output_dir = _prompt("Output directory", "./checkpoints") - - rope_scaling = _collect_rope_scaling(max_length) - neftune_alpha = _collect_neftune_alpha() - use_neftune = neftune_alpha is not None - - use_oom_recovery = _prompt_yes_no( - "Enable OOM recovery? (auto-halves batch size on CUDA out-of-memory, then retries)", - default=False, - ) - - # Step 7: GaLore parameters (if selected) - galore_config = _collect_galore_config(use_galore) - - # Step 8: Build config - _print("\n[8/8] Output") - - config = { - "model": { - "name_or_path": model_name, - "max_length": max_length, - "load_in_4bit": load_in_4bit, - "backend": suggested_backend, - "trust_remote_code": False, - }, - "lora": { - "r": lora_r, - "alpha": lora_alpha, - "dropout": DEFAULT_DROPOUT, - "bias": "none", - "use_dora": use_dora, - "target_modules": target_modules, - "task_type": "CAUSAL_LM", - }, - "training": { - "output_dir": output_dir, - "final_model_dir": "final_model", - "merge_adapters": False, - "trainer_type": trainer_type, - "num_train_epochs": epochs, - "per_device_train_batch_size": batch_size, - "gradient_accumulation_steps": 2, - "learning_rate": DEFAULT_LR, - "warmup_ratio": 0.1, - "weight_decay": 0.01, - "eval_steps": 200, - "save_steps": 200, - "save_total_limit": 3, - "packing": False, - **galore_config, - **({"neftune_noise_alpha": neftune_alpha} if use_neftune else {}), - **({"rope_scaling": rope_scaling} if rope_scaling else {}), - **({"oom_recovery": True, "oom_recovery_min_batch_size": 1} if use_oom_recovery else {}), - }, - "data": { - "dataset_name_or_path": dataset_path, - "shuffle": True, - "clean_text": True, - "add_eos": True, - }, - } - - webhook_section = _collect_webhook_config() - if webhook_section: - config["webhook"] = webhook_section - - evaluation_section = _collect_evaluation_config() - if evaluation_section: - config["evaluation"] = evaluation_section - - compliance_section = _collect_compliance_config() - if compliance_section: - config["compliance"] = compliance_section - - # Save - config_filename = _prompt("Save config as", "my_config.yaml") - if not config_filename.endswith((".yaml", ".yml")): - config_filename += ".yaml" - config_filename = _save_config_to_file(config, config_filename) - - _print_wizard_summary( - model_name=model_name, - suggested_backend=suggested_backend, - use_galore=use_galore, - load_in_4bit=load_in_4bit, - use_dora=use_dora, - trainer_type=trainer_type, - lora_r=lora_r, - lora_alpha=lora_alpha, - dataset_path=dataset_path, - epochs=epochs, - batch_size=batch_size, - output_dir=output_dir, - ) - - # Quick run - if _prompt_yes_no("Start training now?", default=False): - _print(f"\n Running: forgelm --config {config_filename}") - _print() - return config_filename - else: - _print("\n To start training later, run:") - _print(f" forgelm --config {config_filename}") - _print() - return None diff --git a/forgelm/wizard/__init__.py b/forgelm/wizard/__init__.py new file mode 100644 index 00000000..8d9cdbf8 --- /dev/null +++ b/forgelm/wizard/__init__.py @@ -0,0 +1,176 @@ +"""Interactive configuration wizard for ForgeLM (`forgelm --wizard`). + +Phase 22 modernisation (2026-05-08) brought the CLI wizard to parity +with the in-browser ``site/js/wizard.js``: 9-step state machine +(welcome / use-case / model / strategy / trainer / dataset / +training-params / compliance / operations), trainer-specific +hyperparameters (``dpo_beta``, ``simpo_beta`` / ``simpo_gamma``, +``kto_beta``, ``orpo_beta``, ``grpo_*``), full PEFT method coverage +(``lora`` / ``dora`` / ``pissa`` / ``rslora``) plus GaLore as a +separate axis, EU AI Act Article 9 / 10 / 11 / 12+17 compliance +accordions, F-compliance-110 strict-tier auto-coercion, ``back`` / +``reset`` navigation, XDG-aware persistence at +``$XDG_CACHE_HOME/forgelm/wizard_state.yaml``, step-diff preview, +beginner / expert toggle, and the Phase 11.5 / 12.5 BYOD inline +ingest + audit helpers. + +The module was split from a 976-line monolith into a sub-package per +``docs/standards/architecture.md``'s 1000-line ceiling rule: each +submodule owns one concern. + +Sub-modules: + +- :mod:`._io` — primitive prompts + navigation tokens + hardware detection. +- :mod:`._state` — :class:`_WizardState`, persistence, summary, + schema-default constants. +- :mod:`._collectors` — every ``_collect_*`` helper (webhook / + trainer hyperparams / safety / compliance / risk_assessment / + governance / retention / monitoring / benchmark / judge / + synthetic / GaLore / RoPE / NEFTune) plus strategy choice + + use-case preset registry. +- :mod:`._byod` — Phase 11.5 / 12.5 BYOD inline ingest + audit + helpers + quickstart-template prelude. +- :mod:`._orchestrator` — step definitions, ``_drive_wizard_steps``, + ``_apply_strict_tier_coercion``, public ``run_wizard``. + +This ``__init__`` re-exports the public API + the private symbols +covered by the wizard test-suite (``tests/test_wizard_*.py``); the +re-export discipline keeps ``from forgelm import wizard`` reachable +identically to the v0.5.5 monolith, so existing test fixtures and +external callers do not need to change. +""" + +from __future__ import annotations + +# This package's ``__init__`` is intentionally a re-export shim — the +# wizard test-suite (``tests/test_wizard_*.py``) imports private +# helpers (``_print``, ``_collect_*``, ``_save_wizard_state``, etc.) +# directly off ``forgelm.wizard``. Ruff's F401 "imported but unused" +# rule treats every re-export as dead code; the standard idiom for +# "we know, this is the package surface" is the per-file noqa below. +# ruff: noqa: F401 +# Public entry point + navigation exceptions +from ._byod import ( + _AUDIT_LARGE_FILE_THRESHOLD_BYTES, + _BYOD_LOCAL_NOT_FOUND, + _INGEST_SUPPORTED_EXTENSIONS, + _directory_has_ingestible_files, + _finalize_quickstart_path, + _maybe_run_quickstart_template, + _offer_audit_for_jsonl, + _offer_ingest_for_directory, + _prompt_dataset_path_with_ingest_offer, + _resolve_byod_dataset_path, + _validate_local_jsonl, +) +from ._collectors import ( + _GALORE_OPTIMIZERS, + _MANUAL_USE_CASE, + _PREFERENCE_COLUMNS_HINT, + _RISK_TIERS, + _STRATEGY_CHOICES, + _STRICT_RISK_TIERS, + _collect_benchmark, + _collect_compliance_metadata, + _collect_data_governance, + _collect_galore_config, + _collect_judge, + _collect_monitoring, + _collect_neftune_alpha, + _collect_retention, + _collect_risk_assessment, + _collect_rope_scaling, + _collect_safety_config, + _collect_synthetic, + _collect_trainer_hyperparameters, + _collect_webhook_config, + _default_safety_probes_path, + _parse_webhook_value, + _select_strategy, + _select_use_case, + _StrategyChoice, + _wizard_use_case_presets, +) +from ._io import ( + _BACK_TOKENS, + _CANCEL_TOKENS, + _HF_HUB_ID_RE, + _PLATFORM, + _RESET_TOKENS, + WizardBack, + WizardReset, + _check_navigation_token, + _detect_hardware, + _print, + _prompt, + _prompt_choice, + _prompt_float, + _prompt_int, + _prompt_optional_list, + _prompt_required, + _prompt_yes_no, +) +from ._orchestrator import ( + _STEPS, + _apply_strict_tier_coercion, + _drive_wizard_steps, + _is_beginner, + _maybe_resume_state, + _persist_state, + _print_tutorial, + _run_full_wizard, + _step_compliance, + _step_dataset, + _step_evaluation, + _step_model, + _step_strategy, + _step_trainer, + _step_training_params, + _step_use_case, + _step_welcome, + _StepDef, + run_wizard, +) +from ._state import ( + _STATE_VERSION, + DEFAULT_BATCH_SIZE, + DEFAULT_DROPOUT, + DEFAULT_EPOCHS, + DEFAULT_LORA_ALPHA, + DEFAULT_LORA_R, + DEFAULT_LR, + DEFAULT_MAX_LENGTH, + POPULAR_MODELS, + TARGET_MODULE_PRESETS, + _clear_wizard_state, + _flatten_dict, + _load_wizard_state, + _print_step_diff, + _print_wizard_summary, + _save_config_to_file, + _save_wizard_state, + _strategy_label, + _strip_internal_meta, + _wizard_state_dir, + _wizard_state_path, + _WizardState, +) + +# ``__all__`` is a public re-export contract for ``from forgelm.wizard +# import *`` consumers. Only the truly public names appear here; the +# many ``_private`` symbols above remain importable but aren't +# advertised. +__all__ = [ + "run_wizard", + "WizardBack", + "WizardReset", + "POPULAR_MODELS", + "TARGET_MODULE_PRESETS", + "DEFAULT_LORA_R", + "DEFAULT_LORA_ALPHA", + "DEFAULT_EPOCHS", + "DEFAULT_BATCH_SIZE", + "DEFAULT_LR", + "DEFAULT_MAX_LENGTH", + "DEFAULT_DROPOUT", +] diff --git a/forgelm/wizard/_byod.py b/forgelm/wizard/_byod.py new file mode 100644 index 00000000..9928137b --- /dev/null +++ b/forgelm/wizard/_byod.py @@ -0,0 +1,333 @@ +"""BYOD (Bring Your Own Dataset) helpers — Phase 11.5 / 12.5. + +These helpers are CLI's "win" over the web wizard: a typed directory +of raw documents triggers inline ingestion (Phase 11.5), and a typed +JSONL triggers an inline audit (Phase 12.5). Phase 22 didn't touch +the BYOD logic — the helpers are well-tested in +``tests/test_wizard_byod.py`` and ``tests/test_wizard_phase11.py``. +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import List, Optional + +from ._io import _CANCEL_TOKENS, _HF_HUB_ID_RE, _print, _prompt, _prompt_yes_no + +# --------------------------------------------------------------------------- +# Sentinels + thresholds +# --------------------------------------------------------------------------- + + +_BYOD_LOCAL_NOT_FOUND = object() +"""Sentinel for ``_validate_local_jsonl``: file does not exist on disk. + +Distinct from ``None`` (file exists but is not valid JSONL) so the +caller can fall back to an HF Hub-ID interpretation only when the +local path isn't there at all. +""" + + +_INGEST_SUPPORTED_EXTENSIONS = (".pdf", ".docx", ".epub", ".txt", ".md") +"""Extensions that the wizard's "ingest first" prompt knows how to convert. + +Kept in sync with :data:`forgelm.ingestion.SUPPORTED_EXTENSIONS` — +duplicated here so the wizard's directory-scan check doesn't pay the +ingestion module's import cost when the user is bringing a JSONL +straight in. +""" + + +_AUDIT_LARGE_FILE_THRESHOLD_BYTES: int = 100 * 1024 * 1024 # 100 MB + + +# --------------------------------------------------------------------------- +# Directory-of-docs ingestion (Phase 11.5) +# --------------------------------------------------------------------------- + + +def _directory_has_ingestible_files(directory: Path, recursive: bool = True) -> bool: + """True if ``directory`` contains at least one file the ingester can read.""" + pattern = "**/*" if recursive else "*" + for entry in directory.glob(pattern): + if entry.is_file() and entry.suffix.lower() in _INGEST_SUPPORTED_EXTENSIONS: + return True + return False + + +def _offer_ingest_for_directory(directory: Path) -> Optional[str]: + """Phase 11.5 wizard hook: convert a directory of raw docs into JSONL inline. + + Returns the absolute path to the produced JSONL on success, or + ``None`` when the operator declines / cancels / ingestion fails + (the caller treats ``None`` as "re-prompt for a different dataset + path"). + """ + resolved = directory.expanduser().resolve() + if not _directory_has_ingestible_files(resolved): + _print( + f" '{resolved}' is a directory, but it doesn't contain any " + f"{', '.join(_INGEST_SUPPORTED_EXTENSIONS)} files. " + "Pass a JSONL file or a directory with ingestible documents." + ) + return None + + if not _prompt_yes_no( + f"\n '{resolved}' is a directory of raw documents. Run ingestion now and use the resulting JSONL?", + default=True, + ): + _print( + " Skipped — to ingest manually:\n" + f" forgelm ingest {resolved} --recursive --output data/from_docs.jsonl\n" + " Then re-run the wizard with the resulting JSONL path." + ) + return None + + out_dir = (resolved.parent / "data").resolve() + default_out = out_dir / f"{resolved.name}_ingested.jsonl" + out_path_raw = _prompt("Output JSONL path", str(default_out)) + out_path = Path(out_path_raw).expanduser().resolve() + + pii_mask = _prompt_yes_no( + "Mask detected PII (emails, phones, IDs) before writing? Recommended for shared corpora.", + default=False, + ) + + try: + from ..ingestion import ingest_path + except ImportError as exc: # pragma: no cover — extras-skip path + _print(f" ingestion subsystem unavailable: {exc}") + return None + + _print(f"\n Running ingest on '{resolved}' (this may take a moment for large corpora)…") + try: + result = ingest_path( + str(resolved), + output_path=str(out_path), + recursive=True, + pii_mask=pii_mask, + ) + except (FileNotFoundError, ValueError) as exc: + _print(f" Ingest failed: {exc}") + return None + except (PermissionError, IsADirectoryError, OSError) as exc: + _print(f" Ingest failed due to filesystem error: {exc} — check permissions or output path.") + return None + except ImportError as exc: + _print( + f" Ingest needs the optional 'ingestion' extra: {exc}\n Install with: pip install 'forgelm[ingestion]'" + ) + return None + + if result.chunk_count == 0: + _print( + " Ingestion produced 0 chunks — the directory had no extractable text. " + "Pass a JSONL file or a directory with text-bearing documents." + ) + return None + + _print(f" Ingest complete: {result.chunk_count} chunk(s) from {result.files_processed} file(s) → {out_path}") + _offer_audit_for_jsonl(out_path) + return str(out_path) + + +# --------------------------------------------------------------------------- +# JSONL audit (Phase 12.5) +# --------------------------------------------------------------------------- + + +def _offer_audit_for_jsonl(jsonl_path: Path) -> bool: + """Phase 12.5 wizard hook: optionally audit a JSONL the user just selected.""" + try: + size_bytes = jsonl_path.stat().st_size + except OSError: + size_bytes = 0 + if size_bytes > _AUDIT_LARGE_FILE_THRESHOLD_BYTES: + size_mb = size_bytes / (1024 * 1024) + prompt = ( + f"\n Run a quality + governance audit on '{jsonl_path}' " + f"({size_mb:.0f} MB) before training? Scan is CPU-only and " + "streams the file; runtime depends on size (≈ 8–15 MB/s of " + "JSONL on a single CPU)." + ) + else: + prompt = ( + f"\n Run a quality + governance audit on '{jsonl_path}' " + "before training? (length stats, language, near-duplicates, " + "PII, secrets — CPU-only, runtime depends on dataset size)" + ) + if not _prompt_yes_no(prompt, default=True): + _print(" Skipped — audit can be run later via:") + _print(f" forgelm audit {jsonl_path}") + return False + + try: + from ..data_audit import audit_dataset, summarize_report + except ImportError as exc: # pragma: no cover — extras-skip path + _print(f" audit subsystem unavailable: {exc}") + return False + + _print(f"\n Running audit on '{jsonl_path}'…") + try: + report = audit_dataset(str(jsonl_path)) + except (FileNotFoundError, ValueError, OSError) as exc: + _print(f" Audit could not run: {exc}") + return False + except ImportError as exc: + _print(f" Audit could not run (missing optional dep): {exc}") + return False + except Exception as exc: # noqa: BLE001 — bare-except documented in audit-step rationale + _print(f" Audit could not run: {exc}") + return False + + _print("\n Audit complete. Summary:") + _print(summarize_report(report, verbose=False)) + return True + + +# --------------------------------------------------------------------------- +# Dataset-path collection that triggers the BYOD helpers +# --------------------------------------------------------------------------- + + +def _prompt_dataset_path_with_ingest_offer(question: str) -> str: + """Prompt for a dataset path; auto-offer ingestion when the user gives a directory.""" + while True: + raw = _prompt(question, "").strip() + if not raw: + _print(" A dataset reference is required.") + continue + candidate = Path(raw).expanduser() + if candidate.is_dir(): + ingested = _offer_ingest_for_directory(candidate) + if ingested is None: + continue + return ingested + if candidate.is_file() and candidate.suffix.lower() in (".jsonl", ".json"): + _offer_audit_for_jsonl(candidate.resolve()) + return raw + + +def _validate_local_jsonl(raw_path: str): + """Validate a user-supplied JSONL path.""" + resolved = Path(raw_path).expanduser() + if resolved.is_dir(): + ingested = _offer_ingest_for_directory(resolved) + if ingested is None: + return None + return ingested + if not resolved.is_file(): + return _BYOD_LOCAL_NOT_FOUND + try: + with open(resolved, "r", encoding="utf-8") as fh: + first_line = next((line for line in fh if line.strip()), "") + if not first_line: + raise ValueError("file is empty") + json.loads(first_line) + except (OSError, ValueError) as e: + _print(f" File is not valid JSONL (first line failed to parse): {e}") + return None + _offer_audit_for_jsonl(resolved.resolve()) + return str(resolved.resolve()) + + +def _resolve_byod_dataset_path() -> Optional[str]: + """Prompt the user for a BYOD dataset path and validate it.""" + while True: + dataset_path = _prompt( + "Path to your dataset JSONL (must exist as a JSONL file) or HF Hub ID, " + "or 'cancel' to fall back to the full wizard", + "", + ) + if dataset_path.strip().lower() in _CANCEL_TOKENS: + _print(" Cancelled — falling back to the full wizard.") + return None + if not dataset_path: + _print(" A dataset path is required for this template. Type 'cancel' to use the full wizard instead.") + continue + + result = _validate_local_jsonl(dataset_path) + if isinstance(result, str): + return result + if result is None: + continue + if _HF_HUB_ID_RE.match(dataset_path): + _print(f" Treating '{dataset_path}' as an HF Hub dataset ID (no local validation).") + return dataset_path + + _print(f" Path not found or not a regular file: {dataset_path}") + + +# --------------------------------------------------------------------------- +# Quickstart-template prelude. An operator who picks a template +# bypasses the 9-step full wizard via the curated shortcut path. +# --------------------------------------------------------------------------- + + +def _maybe_run_quickstart_template() -> Optional[str]: + """Offer the quickstart template path before the full wizard.""" + from ..quickstart import TEMPLATES, list_templates, run_quickstart + + _print("\n" + "=" * 60) + _print(" ForgeLM Configuration Wizard") + _print("=" * 60) + + if not _prompt_yes_no( + "\nStart from a curated quickstart template? (recommended for first runs)", + default=True, + ): + return None + + _print("\nAvailable templates:") + names: List[str] = [] + for tpl in list_templates(): + bundled = "[x] data" if tpl.bundled_dataset else "[ ] BYOD" + names.append(tpl.name) + _print(f" {len(names)}) {tpl.name} — {tpl.title} ({bundled}, ~{tpl.estimated_minutes}min)") + raw = _prompt("Pick a template by number or name", names[0]) + + chosen: Optional[str] = None + if raw.isdigit(): + idx = int(raw) + if 1 <= idx <= len(names): + chosen = names[idx - 1] + elif raw in TEMPLATES: + chosen = raw + if chosen is None: + _print(f" Could not interpret '{raw}'. Falling back to the full wizard.") + return None + + template = TEMPLATES[chosen] + if not template.bundled_dataset: + _print(f" '{chosen}' is BYOD — bring your own JSONL dataset.") + dataset_path = _resolve_byod_dataset_path() + if dataset_path is None: + return None + else: + dataset_path = "" + + try: + result = run_quickstart( + chosen, + dataset_override=dataset_path or None, + ) + except (FileNotFoundError, ValueError) as e: + _print(f" Quickstart failed: {e}. Falling back to the full wizard.") + return None + + _print(f"\n Quickstart config generated at: {result.config_path}") + _print(f" Selected model: {result.chosen_model} ({result.selection_reason})") + _print(f" Dataset : {result.dataset_path}") + _print() + return str(result.config_path) + + +def _finalize_quickstart_path(quickstart_path: str) -> Optional[str]: + """Ask whether to start training now with the quickstart-generated config.""" + if _prompt_yes_no("Start training now with the generated config?", default=False): + _print(f"\n Running: forgelm --config {quickstart_path}") + return quickstart_path + _print("\n To start training later, run:") + _print(f" forgelm --config {quickstart_path}") + return None diff --git a/forgelm/wizard/_collectors.py b/forgelm/wizard/_collectors.py new file mode 100644 index 00000000..834ab0e2 --- /dev/null +++ b/forgelm/wizard/_collectors.py @@ -0,0 +1,688 @@ +"""Section-collector helpers — webhook / safety / trainer / compliance / etc. + +Every public ``ForgeConfig`` block the operator can populate via the +wizard has a ``_collect_*`` helper here. The orchestrator +(``forgelm.wizard._orchestrator``) calls them in order; each returns +either a populated dict (which the orchestrator drops into the +running config) or ``None`` (which the orchestrator interprets as +"the operator declined this block; omit it from the YAML"). +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Dict, Optional, Tuple +from urllib.parse import urlparse + +from ._io import ( + _print, + _prompt, + _prompt_choice, + _prompt_float, + _prompt_int, + _prompt_optional_list, + _prompt_yes_no, +) + +# --------------------------------------------------------------------------- +# Risk-tier surface (mirrors ``forgelm.config._STRICT_RISK_TIERS`` / +# ``forgelm.config.RiskTier``). +# --------------------------------------------------------------------------- + + +# Strict-tier risk classifications that trigger ``F-compliance-110`` +# auto-coercion (mandatory safety eval + Article 14 staging gate). +_STRICT_RISK_TIERS: Tuple[str, ...] = ("high-risk", "unacceptable") + +# Full ``RiskTier`` Literal, in the order the wizard offers them +# (default = ``minimal-risk`` matches the Pydantic field default). +_RISK_TIERS: Tuple[str, ...] = ( + "unknown", + "minimal-risk", + "limited-risk", + "high-risk", + "unacceptable", +) + + +# --------------------------------------------------------------------------- +# Safety probe path resolution — Phase 22 / G16. Bundled probe set +# lives in the package data at +# ``forgelm.safety_prompts/default_probes.jsonl``; the v0.5.5 wizard +# emitted ``configs/safety_prompts/general_safety.jsonl`` which is +# repo-relative and not shipped in the wheel — broken on +# ``pip install``. +# --------------------------------------------------------------------------- + + +def _default_safety_probes_path() -> str: + """Return the absolute filesystem path of the bundled default probe set. + + Uses :mod:`importlib.resources` so the path is correct under both + editable installs and built wheels. Falls back to the import-time + package directory when ``files()`` raises (e.g. the safety_prompts + package isn't installed — which can happen in slim test + environments). + """ + try: + from importlib.resources import files + + probe = files("forgelm.safety_prompts").joinpath("default_probes.jsonl") + return str(probe) + except (ModuleNotFoundError, FileNotFoundError, ImportError): + # Fallback: best-effort guess at the package directory. + return str(Path(__file__).resolve().parent.parent / "safety_prompts" / "default_probes.jsonl") + + +# --------------------------------------------------------------------------- +# Webhook URL parsing — Phase 22 / G15. Single-prompt syntax mirrors +# the web wizard's ``env:VAR_NAME`` prefix sugar. +# --------------------------------------------------------------------------- + + +def _parse_webhook_value(raw: str) -> Optional[Dict[str, str]]: + """Parse a webhook URL or ``env:VAR_NAME`` reference. + + Returns: + - ``None`` when *raw* is empty (caller drops the webhook block). + - ``{"url_env": "VAR"}`` when *raw* starts with ``env:``. + - ``{"url": ""}`` otherwise. + + Validates that bare URLs use ``https`` scheme; ``http`` URLs print + a warning but are still accepted (matches ``forgelm/webhook.py``'s + runtime behaviour — ``allow_insecure_http=True`` only logs a + warning, doesn't reject). Empty / malformed URLs surface a clear + re-prompt request via ``ValueError``. + """ + raw = raw.strip() + if not raw: + return None + if raw.lower().startswith("env:"): + var_name = raw[4:].strip() + if not var_name: + raise ValueError("`env:` prefix needs a non-empty variable name (e.g. `env:SLACK_WEBHOOK_URL`).") + if not re.match(r"^[A-Z][A-Z0-9_]*$", var_name): + raise ValueError( + f"`env:{var_name}` is not a POSIX environment-variable name " + "(uppercase letters / digits / underscores, must start with a letter)." + ) + return {"url_env": var_name} + parsed = urlparse(raw) + if not parsed.scheme or not parsed.netloc: + raise ValueError(f"`{raw}` is not a valid URL or env-var reference (use `https://…` or `env:VAR_NAME`).") + if parsed.scheme not in ("https", "http"): + raise ValueError( + f"Webhook URL must use `https://` (or `http://` with explicit operator opt-in), got `{parsed.scheme}://`." + ) + if parsed.scheme == "http": + _print( + " Warning: webhook URL uses HTTP, not HTTPS. Data will travel unencrypted. " + "Use `env:VAR_NAME` to source the URL from a secret manager." + ) + return {"url": raw} + + +# --------------------------------------------------------------------------- +# Webhook collector +# --------------------------------------------------------------------------- + + +def _collect_webhook_config() -> Optional[Dict[str, Any]]: + """Prompt for webhook configuration; returns the webhook section or None.""" + if not _prompt_yes_no("Configure webhook notifications?", default=False): + return None + while True: + raw = _prompt( + "Webhook URL (or `env:VAR_NAME` to source from environment)", + "env:FORGELM_WEBHOOK_URL", + ) + if not raw: + _print(" A webhook URL or `env:` reference is required to enable webhooks.") + continue + try: + section = _parse_webhook_value(raw) + except ValueError as exc: + _print(f" {exc}") + continue + if section is None: + return None + section.setdefault("notify_on_start", True) + section.setdefault("notify_on_success", True) + section.setdefault("notify_on_failure", True) + return section + + +# --------------------------------------------------------------------------- +# Safety-eval collector — Phase 22 / G16 +# --------------------------------------------------------------------------- + + +def _collect_safety_config(*, default_enabled: bool = False) -> Optional[Dict[str, Any]]: + """Prompt for safety eval; returns the safety section or None. + + *default_enabled* is set to ``True`` by ``_apply_strict_tier_coercion`` + so high-risk operators see "Y/n" instead of "y/N" — the hint is + the nudge. + """ + if not _prompt_yes_no("Enable safety evaluation (Llama Guard)?", default=default_enabled): + return None + scoring_choice = _prompt_choice( + "Safety scoring mode:", + [ + "binary (simple safe/unsafe ratio)", + "confidence_weighted (uses classifier confidence)", + ], + default=1, + ) + scoring_mode = "confidence_weighted" if "confidence" in scoring_choice else "binary" + safety: Dict[str, Any] = { + "enabled": True, + "test_prompts": _default_safety_probes_path(), + "scoring": scoring_mode, + } + if scoring_mode == "confidence_weighted": + safety["min_safety_score"] = 0.85 + if _prompt_yes_no("Track harm categories (S1-S14 + ForgeLM-curated)?", default=False): + safety["track_categories"] = True + safety["severity_thresholds"] = {"critical": 0, "high": 0.01, "medium": 0.05} + return safety + + +# --------------------------------------------------------------------------- +# Trainer-specific hyperparameters — Phase 22 / G1 +# --------------------------------------------------------------------------- + + +def _collect_trainer_hyperparameters(trainer_type: str) -> Dict[str, Any]: + """Return trainer-specific hyperparameters as a flat dict. + + SFT has no per-trainer knobs (the schema's generic + ``num_train_epochs`` / ``learning_rate`` already covers it), so + the helper short-circuits with an empty dict. Other trainers + prompt for the fields ``forgelm.config.TrainingConfig`` exposes; + defaults mirror the schema so an operator who accepts every + prompt produces a YAML byte-equivalent to ``ForgeConfig()``. + """ + if trainer_type == "sft": + return {} + _print("\n Trainer-specific hyperparameters") + if trainer_type == "dpo": + return { + "dpo_beta": _prompt_float( + "dpo_beta — temperature / KL strength (lower → closer to reference model)", + 0.1, + min_val=1e-5, + max_val=10.0, + ), + } + if trainer_type == "orpo": + return { + "orpo_beta": _prompt_float( + "orpo_beta — odds-ratio penalty strength", + 0.1, + min_val=1e-5, + max_val=10.0, + ), + } + if trainer_type == "kto": + return { + "kto_beta": _prompt_float( + "kto_beta — same KL role as dpo_beta for binary feedback", + 0.1, + min_val=1e-5, + max_val=10.0, + ), + } + if trainer_type == "simpo": + return { + "simpo_beta": _prompt_float( + "simpo_beta — length-normalised reward scale (NOT same scale as dpo_beta)", + 2.0, + min_val=1e-5, + max_val=10.0, + ), + "simpo_gamma": _prompt_float( + "simpo_gamma — margin between chosen / rejected log-likelihoods", + 0.5, + min_val=0.0, + max_val=10.0, + ), + } + if trainer_type == "grpo": + result: Dict[str, Any] = { + "grpo_num_generations": _prompt_int( + "grpo_num_generations — responses sampled per prompt (higher = more stable, more compute)", + 4, + min_val=2, + max_val=64, + ), + "grpo_max_completion_length": _prompt_int( + "grpo_max_completion_length — cap on generated response length", + 512, + min_val=32, + max_val=8192, + ), + } + reward = _prompt( + "grpo_reward_model — dotted-path callable, HF Hub reward model, or empty for the built-in shaper", + "", + ) + if reward.strip(): + result["grpo_reward_model"] = reward.strip() + return result + return {} + + +# --------------------------------------------------------------------------- +# Long-context modifiers — RoPE + NEFTune +# --------------------------------------------------------------------------- + + +def _collect_rope_scaling(max_length: int) -> Optional[Dict[str, Any]]: + """Prompt for RoPE scaling parameters when context is long; otherwise None.""" + if max_length <= 4096: + return None + _print(f"\n Long context detected ({max_length} tokens).") + if not _prompt_yes_no("Enable RoPE scaling for extended context?", default=True): + return None + rope_type = _prompt_choice( + "RoPE scaling type:", + [ + "linear (simple, proven)", + "dynamic (adaptive)", + "yarn (best quality, newer)", + "longrope (newest — 32K+ context, requires LongRoPE-aware model)", + ], + default=1, + ) + base_context = 4096 + rope_factor = max_length / base_context + _print( + f" Note: RoPE factor {rope_factor:.1f}x computed assuming base context of " + f"{base_context} tokens. Adjust manually if your model has a different " + f"original context length (e.g., Llama 3.1 = 131072, Mistral v0.3 = 32768)." + ) + return {"type": rope_type.split(" ")[0], "factor": rope_factor} + + +def _collect_neftune_alpha() -> Optional[float]: + """Prompt for NEFTune noise injection; returns alpha or None.""" + if not _prompt_yes_no("Enable NEFTune noise injection (improves training quality)?", default=False): + return None + return _prompt_float("NEFTune noise alpha", 5.0, min_val=0.0, max_val=100.0) + + +# --------------------------------------------------------------------------- +# GaLore — Phase 22 / G9 surfaces all six schema-allowed optimizer +# variants, including the three ``_layerwise`` siblings that drop peak +# VRAM further. +# --------------------------------------------------------------------------- + + +_GALORE_OPTIMIZERS: Tuple[str, ...] = ( + "galore_adamw", + "galore_adamw_8bit", + "galore_adafactor", + "galore_adamw_layerwise", + "galore_adamw_8bit_layerwise", + "galore_adafactor_layerwise", +) + + +def _collect_galore_config(use_galore: bool) -> Dict[str, Any]: + """Prompt for GaLore-specific knobs when GaLore was selected.""" + if not use_galore: + return {} + _print("\n[Advanced] GaLore Configuration") + galore_rank = _prompt_int("GaLore rank (lower = less memory)", 128, min_val=1, max_val=4096) + galore_optim = _prompt_choice( + "GaLore optimizer (variants ending in _8bit halve optimiser VRAM; " + "_layerwise variants further reduce peak VRAM via per-layer recompute):", + [f"{name} (recommended)" if name == "galore_adamw" else name for name in _GALORE_OPTIMIZERS], + default=1, + ) + return { + "galore_enabled": True, + "galore_optim": galore_optim.split(" ")[0], + "galore_rank": galore_rank, + "galore_update_proj_gap": 200, + "galore_scale": 0.25, + } + + +# --------------------------------------------------------------------------- +# Compliance + Article 9 / 10 / Retention / Monitoring / Eval-gates / +# Synthetic — Phase 22 / G5. +# --------------------------------------------------------------------------- + + +def _collect_compliance_metadata() -> Dict[str, Any]: + """Article 11 + Annex IV §1: provider + system metadata.""" + return { + "provider_name": _prompt("Organization (legal-entity) name", ""), + "provider_contact": _prompt("Provider regulatory contact (email or phone)", ""), + "system_name": _prompt("Human-readable system name", ""), + "intended_purpose": _prompt("Intended purpose of the system", ""), + "known_limitations": _prompt("Known limitations operator wants documented (free-text)", ""), + "system_version": _prompt("System version string", "v0.1.0"), + "risk_classification": _prompt_choice( + "Risk classification (mirrored at risk_assessment.risk_category):", + list(_RISK_TIERS), + default=2, # ``minimal-risk`` — same default as the Pydantic field. + ), + } + + +def _collect_risk_assessment(risk_classification: str) -> Optional[Dict[str, Any]]: + """Article 9: risk-management evidence. Mandatory for strict tiers.""" + is_strict = risk_classification in _STRICT_RISK_TIERS + if not is_strict and not _prompt_yes_no("Configure Article 9 risk_assessment metadata?", default=False): + return None + if is_strict: + _print( + " Risk classification is high-risk / unacceptable — Article 9 " + "risk_assessment evidence is mandatory. All four fields below " + "should be populated." + ) + intended_use = ( + _prompt("Article 9(2)(a): intended_use", "") + if is_strict + else _prompt("intended_use (Article 9(2)(a)) — optional", "") + ) + foreseeable = _prompt_optional_list("Article 9(2)(b): foreseeable_misuse — list at least one realistic misuse") + mitigation = _prompt_optional_list("Article 9(2)(c): mitigation_measures the deployer applies") + vulnerable = _prompt_yes_no( + "Article 9(2)(b): vulnerable_groups_considered (children / minorities / etc.)?", + default=is_strict, + ) + return { + "intended_use": intended_use, + "foreseeable_misuse": foreseeable, + "mitigation_measures": mitigation, + "vulnerable_groups_considered": vulnerable, + # Mirror the compliance.risk_classification across into + # risk_assessment.risk_category so ``ForgeConfig._risk_tiers`` + # sees a consistent pair. + "risk_category": risk_classification, + } + + +def _collect_data_governance(*, mandatory: bool) -> Optional[Dict[str, Any]]: + """Article 10: data governance metadata. Mandatory under strict tiers.""" + if not mandatory and not _prompt_yes_no("Configure Article 10 data.governance metadata?", default=False): + return None + if mandatory: + _print(" Risk classification is high-risk / unacceptable — Article 10 data.governance evidence is mandatory.") + return { + "collection_method": _prompt("Article 10(2)(b): how was data collected?", ""), + "annotation_process": _prompt("Article 10(2)(b): annotation methodology", ""), + "known_biases": _prompt("Article 10(2)(f): known_biases", ""), + "personal_data_included": _prompt_yes_no("Article 10(5): personal_data_included?", default=False), + "dpia_completed": _prompt_yes_no("Article 35 GDPR: dpia_completed?", default=False), + } + + +def _collect_retention() -> Optional[Dict[str, Any]]: + """GDPR Article 5(1)(e) + 17: retention horizons.""" + if not _prompt_yes_no( + "Customise retention horizons (audit log / staging / ephemeral / raw documents)?", + default=False, + ): + return None + audit = _prompt_int( + "audit_log_retention_days (0 = retain indefinitely)", + 1825, + min_val=0, + max_val=18250, + ) + staging = _prompt_int("staging_ttl_days (0 = retain indefinitely)", 7, min_val=0, max_val=365) + ephemeral = _prompt_int( + "ephemeral_artefact_retention_days (0 = retain indefinitely)", + 90, + min_val=0, + max_val=3650, + ) + raw_docs = _prompt_int("raw_documents_retention_days (0 = retain indefinitely)", 90, min_val=0, max_val=3650) + enforce = _prompt_choice( + "enforcement mode:", + [ + "log_only (record violations only)", + "warn_on_excess (stderr warning, run continues)", + "block_on_excess (abort trainer with EXIT_EVAL_FAILURE)", + ], + default=1, + ) + return { + "audit_log_retention_days": audit, + "staging_ttl_days": staging, + "ephemeral_artefact_retention_days": ephemeral, + "raw_documents_retention_days": raw_docs, + "enforce": enforce.split(" ")[0], + } + + +def _collect_monitoring() -> Optional[Dict[str, Any]]: + """Article 12+17: post-market monitoring hooks.""" + if not _prompt_yes_no("Configure post-market monitoring?", default=False): + return None + endpoint = _prompt("Monitoring endpoint URL (Prometheus push gateway / Datadog / custom)", "") + metrics_export = _prompt_choice( + "metrics_export:", + ["none", "prometheus", "datadog", "custom_webhook"], + default=1, + ) + return { + "enabled": True, + "endpoint": endpoint, + "metrics_export": metrics_export, + "alert_on_drift": _prompt_yes_no("alert_on_drift?", default=True), + "check_interval_hours": _prompt_int("check_interval_hours", 24, min_val=1, max_val=720), + } + + +def _collect_benchmark() -> Optional[Dict[str, Any]]: + """``evaluation.benchmark`` lm-evaluation-harness gate.""" + if not _prompt_yes_no( + "Enable benchmark evaluation (lm-evaluation-harness — needs `forgelm[eval]` extra)?", + default=False, + ): + return None + tasks = _prompt_optional_list( + "Benchmark tasks (e.g. arc_easy, hellaswag, mmlu)", + default_csv="arc_easy, hellaswag", + ) + if not tasks: + _print(" No tasks specified; benchmark gate disabled.") + return None + min_score_raw = _prompt("min_score (auto-revert below this) — leave empty for `null`", "") + benchmark: Dict[str, Any] = {"enabled": True, "tasks": tasks} + if min_score_raw.strip(): + try: + benchmark["min_score"] = float(min_score_raw) + except ValueError: + _print(f" '{min_score_raw}' is not a number; min_score left unset.") + return benchmark + + +def _collect_judge() -> Optional[Dict[str, Any]]: + """``evaluation.llm_judge`` LLM-as-Judge gate.""" + if not _prompt_yes_no("Enable LLM-as-Judge scoring?", default=False): + return None + judge_model = _prompt("judge_model (API name or local path)", "gpt-4o-mini") + judge_api_key_env = _prompt( + "judge_api_key_env (env-var name carrying API key; leave empty for a local judge)", + "OPENAI_API_KEY", + ) + judge: Dict[str, Any] = { + "enabled": True, + "judge_model": judge_model, + "min_score": _prompt_float("min_score (mean 1-10 floor; auto-revert below)", 6.5, min_val=1.0, max_val=10.0), + } + if judge_api_key_env.strip(): + judge["judge_api_key_env"] = judge_api_key_env.strip() + return judge + + +def _collect_synthetic() -> Optional[Dict[str, Any]]: + """``synthetic`` teacher → student distillation block.""" + if not _prompt_yes_no("Configure synthetic-data generation (teacher → student)?", default=False): + return None + backend = _prompt_choice( + "teacher_backend:", + [ + "api (OpenAI/Anthropic-compatible)", + "local (HF model loaded in-process)", + "file (read pre-generated JSONL)", + ], + default=1, + ) + backend_token = backend.split(" ")[0] + section: Dict[str, Any] = { + "enabled": True, + "teacher_backend": backend_token, + "teacher_model": _prompt("teacher_model (HF Hub ID or API model name)", "gpt-4o"), + } + if backend_token == "api": + api_base = _prompt( + "api_base (OpenAI-compatible endpoint URL)", + "https://api.openai.com/v1", + ) + api_key_env = _prompt("api_key_env", "OPENAI_API_KEY") + section["api_base"] = api_base + section["api_key_env"] = api_key_env + seed_file = _prompt("seed_file (path to seed prompts JSONL or text)", "") + if seed_file.strip(): + section["seed_file"] = seed_file + section["output_file"] = _prompt("output_file", "data/synthetic.jsonl") + return section + + +# --------------------------------------------------------------------------- +# Strategy → flag derivation. Phase 22 / G2 expands the menu to cover +# the full ``LoraConfigModel.method`` Literal (lora / dora / pissa / +# rslora) with GaLore as a separate axis. +# --------------------------------------------------------------------------- + + +@dataclass(frozen=True) +class _StrategyChoice: + """Decoded form of a strategy menu pick.""" + + label: str + load_in_4bit: bool + method: str # lora / dora / pissa / rslora + use_galore: bool + + +_STRATEGY_CHOICES: Tuple[_StrategyChoice, ...] = ( + _StrategyChoice( + label="QLoRA (4-bit quantization, standard LoRA — recommended for most operators)", + load_in_4bit=True, + method="lora", + use_galore=False, + ), + _StrategyChoice( + label="LoRA (full-precision adapters, more memory, slightly better quality)", + load_in_4bit=False, + method="lora", + use_galore=False, + ), + _StrategyChoice( + label="DoRA (Weight-Decomposed LoRA — best adapter quality, slightly more compute)", + load_in_4bit=True, + method="dora", + use_galore=False, + ), + _StrategyChoice( + label="PiSSA (singular-value initialised LoRA — better stability, paper-recommended for large r)", + load_in_4bit=True, + method="pissa", + use_galore=False, + ), + _StrategyChoice( + label="rsLoRA (rank-stabilised LoRA — recommended when r > 64)", + load_in_4bit=True, + method="rslora", + use_galore=False, + ), + _StrategyChoice( + label="GaLore (full-parameter via gradient projection — no adapters, lowest peak VRAM)", + load_in_4bit=False, + method="lora", # dummy, ignored when use_galore=True + use_galore=True, + ), +) + + +def _select_strategy() -> _StrategyChoice: + """Prompt for the fine-tuning strategy.""" + label = _prompt_choice( + "Choose your fine-tuning strategy:", + [choice.label for choice in _STRATEGY_CHOICES], + default=1, + ) + return next(c for c in _STRATEGY_CHOICES if c.label == label) + + +# --------------------------------------------------------------------------- +# Use-case preset registry — Phase 22 / G12 + G14 + I4. Anchored on +# the CLI quickstart TEMPLATES so the web wizard adopting the same +# keys closes the use-case key drift documented in the analysis +# report. +# --------------------------------------------------------------------------- + + +# Wizard-only "manual" path that doesn't preselect anything. Mirrored +# in ``_state._MANUAL_USE_CASE`` so callers from either submodule see +# the same value. +_MANUAL_USE_CASE = "custom" + + +def _wizard_use_case_presets() -> Dict[str, Dict[str, Any]]: + """Build the use-case → preset map from ``forgelm.quickstart.TEMPLATES``.""" + from ..quickstart import TEMPLATES + + presets: Dict[str, Dict[str, Any]] = {} + for name, tpl in TEMPLATES.items(): + presets[name] = { + "title": tpl.title, + "trainer_type": tpl.trainer_type, + "model": tpl.primary_model, + "bundled_dataset": tpl.bundled_dataset, + "estimated_minutes": tpl.estimated_minutes, + } + presets[_MANUAL_USE_CASE] = { + "title": "Custom (build from scratch — no preselect)", + "trainer_type": None, + "model": None, + "bundled_dataset": False, + "estimated_minutes": 0, + } + return presets + + +def _select_use_case() -> Tuple[str, Dict[str, Any]]: + """Prompt for a use-case and return (key, preset_dict).""" + presets = _wizard_use_case_presets() + keys = list(presets.keys()) + options = [ + f"{key} — {presets[key]['title']} (~{presets[key]['estimated_minutes']}min)" + if presets[key]["estimated_minutes"] + else f"{key} — {presets[key]['title']}" + for key in keys + ] + chosen = _prompt_choice( + "Use-case (each preselects sensible defaults; you can override every later step):", + options, + default=1, + ) + chosen_key = chosen.split(" — ")[0] + return chosen_key, presets.get(chosen_key, presets[_MANUAL_USE_CASE]) + + +# Re-export ``_PREFERENCE_COLUMNS_HINT`` from ``_state`` for the +# orchestrator's trainer-selection step — keeps the data-format hints +# next to where they're surfaced. +_PREFERENCE_COLUMNS_HINT = "Columns: prompt, chosen, rejected" diff --git a/forgelm/wizard/_io.py b/forgelm/wizard/_io.py new file mode 100644 index 00000000..e459bc67 --- /dev/null +++ b/forgelm/wizard/_io.py @@ -0,0 +1,213 @@ +"""I/O primitives + navigation tokens for the CLI wizard. + +Every interactive prompt funnels through one of the helpers here so +navigation tokens (``back`` / ``reset``) get a chance to fire before +validation runs. Cancel-style tokens are deliberately NOT auto- +raised — the BYOD path interprets them as "fall back to the full +wizard" and the step orchestrator relies on Ctrl-C / Ctrl-D for clean +exits. + +The ``_print`` indirection mirrors :mod:`forgelm.chat` so wizard +output can be captured via ``capsys`` in tests and re-routed through +``rich`` or a structured logger later without touching every call +site. +""" + +from __future__ import annotations + +import re +import sys +from typing import Any, Dict, List, Tuple + +# --------------------------------------------------------------------------- +# Output indirection +# --------------------------------------------------------------------------- + + +def _print(*args, **kwargs) -> None: + """Indirection over the builtin :func:`print`.""" + # NOTE: bypass the wizard's own ``print(`` → ``_print(`` rewrite by + # going through ``builtins.print`` — a bare ``print(...)`` here would + # be picked up by future codemods and turned into infinite recursion. + import builtins as _builtins + + _builtins.print(*args, **kwargs) + + +# --------------------------------------------------------------------------- +# Navigation tokens — operator-typed sentinels that the step machine +# turns into back / reset transitions. Keep the discipline centralised +# so individual ``_collect_*`` helpers share one source of truth. +# --------------------------------------------------------------------------- + + +class WizardBack(Exception): + """Operator typed ``back`` / ``b`` — caller catches and re-renders the previous step.""" + + +class WizardReset(Exception): + """Operator typed ``reset`` / ``r`` — caller catches and clears state.""" + + +_BACK_TOKENS: Tuple[str, ...] = ("back", "b") +_RESET_TOKENS: Tuple[str, ...] = ("reset", "r") +# Cancel sentinels stay context-local: the BYOD path interprets these +# as "fall back to the full wizard" while the step orchestrator relies +# on Ctrl-C / Ctrl-D for clean exits. Kept as a tuple so the BYOD +# helpers can keep their existing in-string membership check. +_CANCEL_TOKENS: Tuple[str, ...] = ("cancel", "c", "q", "quit") + + +def _check_navigation_token(value: str) -> None: + """Raise the matching navigation exception when *value* is a sentinel. + + Only ``back`` and ``reset`` are wizard-machine navigation tokens + that auto-raise from the primitive prompts — ``cancel`` (and its + aliases) is contextual. Token matching is case-insensitive and + ignores surrounding whitespace. The empty string is not a + sentinel — the caller should treat that as "no answer" and apply + its default. + """ + if not value: + return + lowered = value.strip().lower() + if lowered in _BACK_TOKENS: + raise WizardBack + if lowered in _RESET_TOKENS: + raise WizardReset + + +# --------------------------------------------------------------------------- +# Primitive prompts — every interactive call funnels through one of +# these so navigation tokens get a chance to fire before validation +# runs. +# --------------------------------------------------------------------------- + + +def _prompt(question: str, default: str = "") -> str: + """Prompt the user with a default value.""" + suffix = f" [{default}]" if default else "" + answer = input(f" {question}{suffix}: ").strip() # pragma: no cover -- stdin read + _check_navigation_token(answer) + return answer if answer else default + + +def _prompt_choice(question: str, options: List[str], default: int = 1) -> str: + """Prompt the user to pick from numbered options. + + Returns the literal string of the selected option (caller may + extract a token via ``.split(" ")[0]`` when the option is + decorated with a description). Out-of-range or non-numeric + answers fall back to *default* — preserving the v0.5.5 behaviour + ``test_wizard_byod`` and friends pin. + """ + _print(f"\n {question}") + for i, opt in enumerate(options, 1): + marker = " *" if i == default else "" + _print(f" {i}) {opt}{marker}") + choice = input(f" Choice [{default}]: ").strip() # pragma: no cover -- stdin read + _check_navigation_token(choice) + try: + idx = int(choice) if choice else default + return options[idx - 1] + except (ValueError, IndexError): + return options[default - 1] + + +def _prompt_yes_no(question: str, default: bool = True) -> bool: + """Prompt for a yes / no answer.""" + hint = "Y/n" if default else "y/N" + answer = input(f" {question} [{hint}]: ").strip().lower() # pragma: no cover -- stdin read + _check_navigation_token(answer) + if not answer: + return default + return answer in ("y", "yes") + + +def _prompt_int(question: str, default: int, min_val: int = 1, max_val: int = 65536) -> int: + """Prompt for an integer, re-asking until valid.""" + while True: + raw = _prompt(question, str(default)) + try: + val = int(raw) + if min_val <= val <= max_val: + return val + _print(f" Value must be between {min_val} and {max_val}.") + except ValueError: + _print(" Please enter a valid integer.") + + +def _prompt_float(question: str, default: float, min_val: float = 0.0, max_val: float = 1.0e9) -> float: + """Prompt for a float, re-asking until valid.""" + while True: + raw = _prompt(question, repr(default)) + try: + val = float(raw) + if min_val <= val <= max_val: + return val + _print(f" Value must be between {min_val} and {max_val}.") + except ValueError: + _print(" Please enter a valid number.") + + +def _prompt_required(question: str) -> str: + """Prompt for a non-empty value, re-asking until provided.""" + while True: + answer = _prompt(question, "") + if answer.strip(): + return answer.strip() + _print(" A value is required.") + + +def _prompt_optional_list(question: str, *, default_csv: str = "") -> List[str]: + """Prompt for a comma-separated list; empty input → empty list.""" + raw = _prompt(question + " (comma-separated; leave empty to skip)", default_csv) + if not raw.strip(): + return [] + return [item.strip() for item in raw.split(",") if item.strip()] + + +# --------------------------------------------------------------------------- +# Hardware detection — used by the welcome step's backend hint. The +# helper survived the rewrite because no parity finding asked us to +# touch it; kept here so the welcome step doesn't pull in a separate +# module just for one detection call. +# --------------------------------------------------------------------------- + + +def _detect_hardware() -> Dict[str, Any]: + """Detect GPU hardware if available.""" + info: Dict[str, Any] = { + "gpu_available": False, + "gpu_name": None, + "vram_gb": None, + "cuda_version": None, + } + try: + import torch + + if torch.cuda.is_available(): + info["gpu_available"] = True + info["gpu_name"] = torch.cuda.get_device_name(0) + info["vram_gb"] = round(torch.cuda.get_device_properties(0).total_mem / (1024**3), 1) + info["cuda_version"] = torch.version.cuda + except ImportError: + pass + return info + + +# --------------------------------------------------------------------------- +# Module-level constants reused across wizard sub-modules +# --------------------------------------------------------------------------- + + +# HF Hub dataset IDs look like ``/`` — exactly one slash, with +# the allowed character set used by the Hub. We accept these BYOD +# inputs without touching the local filesystem; the trainer resolves +# them at runtime. See ``forgelm/wizard/_byod.py`` for the consumer. +_HF_HUB_ID_RE = re.compile(r"^[\w.-]{1,96}/[\w.-]{1,96}$", flags=re.ASCII) + + +# ``sys.platform`` is read by the welcome step's backend hint; expose +# here so submodules don't each re-import ``sys``. +_PLATFORM = sys.platform diff --git a/forgelm/wizard/_orchestrator.py b/forgelm/wizard/_orchestrator.py new file mode 100644 index 00000000..7dcf76f7 --- /dev/null +++ b/forgelm/wizard/_orchestrator.py @@ -0,0 +1,566 @@ +"""Step machine + public entry point for the wizard. + +The orchestrator threads a single :class:`_WizardState` through every +step, persists after each step, and supports back-step / reset. +Each step is registered as a small :class:`_StepDef`. Steps return +``None`` (success), raise :class:`WizardBack` (operator typed +``back``), or raise :class:`WizardReset` (operator typed ``reset``). +""" + +from __future__ import annotations + +import json +from dataclasses import dataclass +from typing import Any, Callable, Dict, List, Optional, Tuple + +from ._byod import ( + _finalize_quickstart_path, + _maybe_run_quickstart_template, + _prompt_dataset_path_with_ingest_offer, +) +from ._collectors import ( + _PREFERENCE_COLUMNS_HINT, + _STRICT_RISK_TIERS, + _collect_benchmark, + _collect_compliance_metadata, + _collect_data_governance, + _collect_galore_config, + _collect_judge, + _collect_monitoring, + _collect_neftune_alpha, + _collect_retention, + _collect_risk_assessment, + _collect_rope_scaling, + _collect_safety_config, + _collect_synthetic, + _collect_trainer_hyperparameters, + _collect_webhook_config, + _default_safety_probes_path, + _select_strategy, + _select_use_case, +) +from ._io import ( + _PLATFORM, + WizardBack, + WizardReset, + _detect_hardware, + _print, + _prompt, + _prompt_int, + _prompt_required, + _prompt_yes_no, +) +from ._state import ( + _MANUAL_USE_CASE, + DEFAULT_BATCH_SIZE, + DEFAULT_DROPOUT, + DEFAULT_EPOCHS, + DEFAULT_LORA_R, + DEFAULT_LR, + DEFAULT_MAX_LENGTH, + POPULAR_MODELS, + TARGET_MODULE_PRESETS, + _clear_wizard_state, + _load_wizard_state, + _print_step_diff, + _print_wizard_summary, + _save_config_to_file, + _save_wizard_state, + _strip_internal_meta, + _wizard_state_path, + _WizardState, +) + +# --------------------------------------------------------------------------- +# Strict-tier auto-coercion — Phase 22 / G8. ``ForgeConfig`` raises +# ``ConfigError`` at load time when ``risk_classification ∈ +# {high-risk, unacceptable}`` and ``evaluation.safety.enabled = +# False``; the wizard front-stops the same gate. +# --------------------------------------------------------------------------- + + +def _apply_strict_tier_coercion(config: Dict[str, Any], compliance: Dict[str, Any]) -> None: + """Mutate *config* in place to satisfy F-compliance-110 strict-tier requirements.""" + if compliance.get("risk_classification") not in _STRICT_RISK_TIERS: + return + _print( + "\n Risk classification is high-risk / unacceptable — Article 9 " + "(F-compliance-110) requires safety evaluation enabled and Article 14 " + "requires the human-approval staging gate. Auto-enabling both." + ) + evaluation = config.setdefault("evaluation", {}) + evaluation.setdefault("auto_revert", True) + evaluation["require_human_approval"] = True + safety = evaluation.get("safety") + if not isinstance(safety, dict) or not safety.get("enabled"): + evaluation["safety"] = { + "enabled": True, + "test_prompts": _default_safety_probes_path(), + "scoring": "binary", + "track_categories": True, + "severity_thresholds": {"critical": 0, "high": 0.01, "medium": 0.05}, + } + + +# --------------------------------------------------------------------------- +# Beginner / expert tutorial-paragraph helpers +# --------------------------------------------------------------------------- + + +def _is_beginner(state: _WizardState) -> bool: + return state.experience == "beginner" + + +def _print_tutorial(state: _WizardState, lines: List[str]) -> None: + """Print a 2-3-line tutorial paragraph in beginner mode; skip in expert mode.""" + if not _is_beginner(state): + return + _print() + for line in lines: + _print(f" · {line}") + + +# --------------------------------------------------------------------------- +# Step definitions +# --------------------------------------------------------------------------- + + +@dataclass +class _StepDef: + """A single step in the full-wizard flow.""" + + label: str + runner: Callable[[_WizardState], None] + + +def _step_welcome(state: _WizardState) -> None: + """Step 1: welcome + experience toggle + hardware detection.""" + _print("\n[1/9] Welcome") + _print( + "\n ForgeLM walks you through producing a validated `config.yaml`. " + "Type `back` / `b` at any prompt to return to the previous step, " + "`reset` / `r` to clear answers, `cancel` / `q` to exit cleanly." + ) + if _prompt_yes_no( + "First-time user? (beginner mode prefixes each step with a short tutorial paragraph)", + default=False, + ): + state.experience = "beginner" + else: + state.experience = "expert" + hw = _detect_hardware() + if hw["gpu_available"]: + _print(f" GPU detected: {hw['gpu_name']} ({hw['vram_gb']} GB VRAM, CUDA {hw['cuda_version']})") + else: + _print(" No GPU detected. Training will use CPU (very slow for real workloads).") + suggested_backend = "transformers" + if hw["gpu_available"] and _PLATFORM == "linux": + suggested_backend = "unsloth" + _print(" Recommended backend: unsloth (Linux + GPU detected)") + elif hw["gpu_available"]: + _print(" Recommended backend: transformers (Unsloth requires Linux)") + state.config.setdefault("model", {})["backend"] = suggested_backend + + +def _step_use_case(state: _WizardState) -> None: + """Step 2: use-case preselect.""" + _print("\n[2/9] Use-case") + _print_tutorial( + state, + [ + "A use-case picks sensible defaults for trainer + model + dataset.", + "Pick `custom` to fill every field manually.", + "Whatever you pick, you can still override every later step.", + ], + ) + use_case_key, preset = _select_use_case() + state.use_case = use_case_key + if preset.get("model"): + state.config.setdefault("model", {})["name_or_path"] = preset["model"] + if preset.get("trainer_type"): + state.config.setdefault("training", {})["trainer_type"] = preset["trainer_type"] + + +def _step_model(state: _WizardState) -> None: + """Step 3: model selection.""" + _print("\n[3/9] Model Selection") + _print_tutorial( + state, + [ + "Pick the base model your fine-tune will adapt.", + "Smaller models train faster but cap your reasoning ceiling.", + "QLoRA on a 7B model fits comfortably in 12 GB VRAM.", + ], + ) + preset_default = state.config.get("model", {}).get("name_or_path") + options = list(POPULAR_MODELS) + options.append("Custom (enter your own)") + if preset_default and preset_default in POPULAR_MODELS: + default_idx = POPULAR_MODELS.index(preset_default) + 1 + else: + default_idx = 1 + from ._io import _prompt_choice + + chosen = _prompt_choice("Choose a model:", options, default=default_idx) + if chosen.startswith("Custom"): + model_name = _prompt_required("HuggingFace model name or local path") + else: + model_name = chosen + state.config.setdefault("model", {})["name_or_path"] = model_name + + +def _step_strategy(state: _WizardState) -> None: + """Step 4: strategy + LoRA params.""" + _print("\n[4/9] Fine-Tuning Strategy") + _print_tutorial( + state, + [ + "QLoRA is the safest default: 4-bit base + low-rank adapter.", + "DoRA / PiSSA / rsLoRA tweak the adapter for better convergence.", + "GaLore replaces adapters with full-parameter training via gradient projection.", + ], + ) + strategy = _select_strategy() + from ._io import _prompt_choice + + target_preset = _prompt_choice( + "Target modules:", + ["standard (q_proj, v_proj)", "extended (q, k, v, o)", "full (all linear layers)"], + default=1, + ) + preset_key = target_preset.split(" ")[0] + target_modules = TARGET_MODULE_PRESETS.get(preset_key, TARGET_MODULE_PRESETS["standard"]) + lora_r = _prompt_int("LoRA rank (r)", DEFAULT_LORA_R, min_val=1, max_val=512) + lora_alpha = _prompt_int("LoRA alpha", lora_r * 2, min_val=1, max_val=1024) + state.config.setdefault("model", {})["load_in_4bit"] = strategy.load_in_4bit + state.config["model"].setdefault("trust_remote_code", False) + lora_block = state.config.setdefault("lora", {}) + lora_block["r"] = lora_r + lora_block["alpha"] = lora_alpha + lora_block["dropout"] = DEFAULT_DROPOUT + lora_block["bias"] = "none" + lora_block["method"] = strategy.method + lora_block["target_modules"] = target_modules + lora_block["task_type"] = "CAUSAL_LM" + state.config.setdefault("training", {})["galore_enabled"] = strategy.use_galore + state.config.setdefault("_wizard_meta", {})["use_galore"] = strategy.use_galore + + +def _step_trainer(state: _WizardState) -> None: + """Step 5: trainer + per-trainer hyperparameters.""" + _print("\n[5/9] Training Objective") + _print_tutorial( + state, + [ + "SFT is supervised fine-tuning — the standard instruction-tuning recipe.", + "DPO / SimPO / KTO / ORPO align via preferences (chosen vs rejected).", + "GRPO is reasoning RL (DeepSeek-R1 style); needs a reward signal.", + ], + ) + objectives = [ + "sft — Supervised Fine-Tuning (standard instruction tuning)", + "dpo — Direct Preference Optimization (chosen/rejected pairs)", + "simpo — Simple Preference Optimization (no reference model, lower memory)", + "kto — Binary feedback alignment (thumbs up/down, practical for production)", + "orpo — Odds Ratio Preference Optimization (SFT + alignment in one stage)", + "grpo — Group Relative Policy Optimization (reasoning RL, like DeepSeek-R1)", + ] + preset = state.config.get("training", {}).get("trainer_type", "sft") + default_idx = next( + (i for i, opt in enumerate(objectives, 1) if opt.startswith(f"{preset} —")), + 1, + ) + from ._io import _prompt_choice + + chosen = _prompt_choice("Choose your training objective:", objectives, default=default_idx) + trainer_type = chosen.split(" — ")[0].strip().lower() + dataset_format_hint = { + "sft": "Columns: System (opt), User/instruction, Assistant/output — or 'messages' list", + "dpo": _PREFERENCE_COLUMNS_HINT, + "simpo": _PREFERENCE_COLUMNS_HINT, + "orpo": _PREFERENCE_COLUMNS_HINT, + "kto": "Columns: prompt, completion, label (boolean: true=good, false=bad)", + "grpo": "Columns: prompt (model generates responses during training)", + }.get(trainer_type, "Standard format") + _print(f" Dataset format: {dataset_format_hint}") + training_block = state.config.setdefault("training", {}) + training_block["trainer_type"] = trainer_type + hyperparams = _collect_trainer_hyperparameters(trainer_type) + training_block.update(hyperparams) + + +def _step_dataset(state: _WizardState) -> None: + """Step 6: dataset path + optional Article 10 governance.""" + _print("\n[6/9] Dataset") + _print_tutorial( + state, + [ + "Point the wizard at an HF Hub dataset ID, a local JSONL, or a directory of raw documents.", + "Directories of raw documents trigger inline ingestion (PDF / DOCX / EPUB / TXT / MD).", + "JSONL files are auto-audited (length / language / dedup / PII / secrets).", + ], + ) + dataset_path = _prompt_dataset_path_with_ingest_offer( + "HuggingFace dataset name or local file path (or directory of raw documents)", + ) + data_block = state.config.setdefault("data", {}) + data_block["dataset_name_or_path"] = dataset_path + data_block.setdefault("shuffle", True) + data_block.setdefault("clean_text", True) + data_block.setdefault("add_eos", True) + governance = _collect_data_governance(mandatory=False) + if governance: + data_block["governance"] = governance + + +def _step_training_params(state: _WizardState) -> None: + """Step 7: epochs + batch + max_length + RoPE + NEFTune + OOM + GaLore advanced.""" + _print("\n[7/9] Training Parameters") + _print_tutorial( + state, + [ + "Epochs / batch / learning_rate are the three knobs you'll tune most often.", + "Long-context (>4096) automatically prompts for RoPE scaling.", + "OOM recovery halves the batch size on CUDA out-of-memory and retries.", + ], + ) + epochs = _prompt_int("Number of epochs", DEFAULT_EPOCHS, min_val=1, max_val=1000) + batch_size = _prompt_int("Batch size per device", DEFAULT_BATCH_SIZE, min_val=1, max_val=512) + max_length = _prompt_int("Max sequence length", DEFAULT_MAX_LENGTH, min_val=64, max_val=131072) + output_dir = _prompt("Output directory", "./checkpoints") + rope_scaling = _collect_rope_scaling(max_length) + neftune_alpha = _collect_neftune_alpha() + use_oom_recovery = _prompt_yes_no( + "Enable OOM recovery? (auto-halves batch size on CUDA out-of-memory, then retries)", + default=False, + ) + use_galore = bool(state.config.get("_wizard_meta", {}).get("use_galore")) + galore_config = _collect_galore_config(use_galore) + model_block = state.config.setdefault("model", {}) + model_block["max_length"] = max_length + training_block = state.config.setdefault("training", {}) + training_block["output_dir"] = output_dir + training_block.setdefault("final_model_dir", "final_model") + training_block.setdefault("merge_adapters", False) + training_block["num_train_epochs"] = epochs + training_block["per_device_train_batch_size"] = batch_size + training_block.setdefault("gradient_accumulation_steps", 2) + training_block.setdefault("learning_rate", DEFAULT_LR) + training_block.setdefault("warmup_ratio", 0.1) + training_block.setdefault("weight_decay", 0.01) + training_block.setdefault("eval_steps", 200) + training_block.setdefault("save_steps", 200) + training_block.setdefault("save_total_limit", 3) + training_block.setdefault("packing", False) + training_block.update(galore_config) + if neftune_alpha is not None: + training_block["neftune_noise_alpha"] = neftune_alpha + if rope_scaling is not None: + training_block["rope_scaling"] = rope_scaling + if use_oom_recovery: + training_block["oom_recovery"] = True + training_block["oom_recovery_min_batch_size"] = 1 + + +def _step_compliance(state: _WizardState) -> None: + """Step 8: compliance / risk_classification / risk_assessment / governance / retention / monitoring.""" + _print("\n[8/9] Compliance + risk") + _print_tutorial( + state, + [ + "Configure EU AI Act compliance metadata for Annex IV documentation.", + "high-risk / unacceptable risk_classification triggers Article 9 / 14 auto-coercion.", + "Article 10 data.governance and retention horizons are optional unless you're high-risk.", + ], + ) + if not _prompt_yes_no( + "Configure EU AI Act compliance metadata?", + default=False, + ): + return + compliance = _collect_compliance_metadata() + state.config["compliance"] = compliance + risk = compliance.get("risk_classification", "minimal-risk") + risk_assessment = _collect_risk_assessment(risk) + if risk_assessment: + state.config["risk_assessment"] = risk_assessment + is_strict = risk in _STRICT_RISK_TIERS + governance = _collect_data_governance(mandatory=is_strict) + if governance: + state.config.setdefault("data", {})["governance"] = governance + retention = _collect_retention() + if retention: + state.config["retention"] = retention + monitoring = _collect_monitoring() + if monitoring: + state.config["monitoring"] = monitoring + + +def _step_evaluation(state: _WizardState) -> None: + """Step 9 (operations): evaluation gates, webhooks, synthetic, save.""" + _print("\n[9/9] Operations + evaluation") + _print_tutorial( + state, + [ + "Evaluation gates (auto_revert + safety + benchmark + judge) protect against regressions.", + "Webhooks notify Slack / Teams / your CI when training starts, succeeds, or fails.", + "Synthetic-data lets a stronger teacher model generate extra training examples.", + ], + ) + evaluation: Dict[str, Any] = {} + if _prompt_yes_no("Enable auto-revert (discard model if quality drops)?", default=False): + evaluation["auto_revert"] = True + max_loss = _prompt("Max acceptable loss (leave empty for baseline-only)", "") + if max_loss.strip(): + try: + evaluation["max_acceptable_loss"] = float(max_loss) + except ValueError: + _print(f" '{max_loss}' is not a number; max_acceptable_loss left unset.") + risk = state.config.get("compliance", {}).get("risk_classification", "minimal-risk") + safety = _collect_safety_config(default_enabled=risk in _STRICT_RISK_TIERS) + if safety: + evaluation["safety"] = safety + benchmark = _collect_benchmark() + if benchmark: + evaluation["benchmark"] = benchmark + judge = _collect_judge() + if judge: + evaluation["llm_judge"] = judge + if evaluation: + state.config["evaluation"] = evaluation + webhook_section = _collect_webhook_config() + if webhook_section: + state.config["webhook"] = webhook_section + synthetic = _collect_synthetic() + if synthetic: + state.config["synthetic"] = synthetic + compliance = state.config.get("compliance") + if compliance: + _apply_strict_tier_coercion(state.config, compliance) + + +_STEPS: Tuple[_StepDef, ...] = ( + _StepDef("welcome", _step_welcome), + _StepDef("use-case", _step_use_case), + _StepDef("model", _step_model), + _StepDef("strategy", _step_strategy), + _StepDef("trainer", _step_trainer), + _StepDef("dataset", _step_dataset), + _StepDef("training-params", _step_training_params), + _StepDef("compliance", _step_compliance), + _StepDef("evaluation", _step_evaluation), +) + + +# --------------------------------------------------------------------------- +# Resume + step-machine driver +# --------------------------------------------------------------------------- + + +def _maybe_resume_state() -> _WizardState: + """Offer to resume an in-progress wizard run.""" + snapshot = _load_wizard_state() + if snapshot is None: + return _WizardState() + completed = snapshot.get("completed_steps", []) + if not completed: + return _WizardState() + if not _prompt_yes_no( + f"Resume previous wizard run? (completed {len(completed)} step(s): {', '.join(completed)})", + default=True, + ): + _clear_wizard_state() + return _WizardState() + return _WizardState( + experience=snapshot.get("experience", "expert"), + use_case=snapshot.get("use_case", _MANUAL_USE_CASE), + current_step=snapshot.get("current_step", 0), + completed_steps=list(completed), + config=dict(snapshot.get("config", {})), + ) + + +def _persist_state(state: _WizardState) -> None: + """Snapshot the wizard's in-memory state to disk.""" + _save_wizard_state( + { + "experience": state.experience, + "use_case": state.use_case, + "current_step": state.current_step, + "completed_steps": state.completed_steps, + "config": state.config, + } + ) + + +def _drive_wizard_steps(state: _WizardState) -> _WizardState: + """Run the wizard's step machine, honouring back / reset.""" + while state.current_step < len(_STEPS): + step = _STEPS[state.current_step] + prev_config = json.loads(json.dumps(state.config)) # deep copy + try: + step.runner(state) + except WizardBack: + if state.current_step == 0: + _print(" Already at the first step.") + continue + state.current_step -= 1 + if state.completed_steps and state.completed_steps[-1] == _STEPS[state.current_step].label: + state.completed_steps.pop() + _persist_state(state) + continue + except WizardReset: + _print(" Resetting wizard state.") + _clear_wizard_state() + return _WizardState() + _print_step_diff(prev_config, state.config, step.label) + if step.label not in state.completed_steps: + state.completed_steps.append(step.label) + state.current_step += 1 + _persist_state(state) + return state + + +# --------------------------------------------------------------------------- +# Public entry points +# --------------------------------------------------------------------------- + + +def run_wizard() -> Optional[str]: + """Run the interactive configuration wizard. + + Returns the path to the generated config file when the user opts + to start training immediately, or ``None`` when the user defers — + callers must handle both cases. + """ + quickstart_path = _maybe_run_quickstart_template() + if quickstart_path is not None: + return _finalize_quickstart_path(quickstart_path) + _print("\n Falling back to the full configuration wizard.") + return _run_full_wizard() + + +def _run_full_wizard() -> Optional[str]: + """9-step interactive flow producing a hand-rolled config.yaml.""" + state = _maybe_resume_state() + try: + state = _drive_wizard_steps(state) + except (KeyboardInterrupt, EOFError): + _persist_state(state) + _print(f"\n Interrupted. State preserved at {_wizard_state_path()} — rerun `forgelm --wizard` to resume.") + return None + config = _strip_internal_meta(state.config) + config_filename = _prompt("Save config as", "my_config.yaml") + if not config_filename.endswith((".yaml", ".yml")): + config_filename += ".yaml" + config_filename = _save_config_to_file(config, config_filename) + _print_wizard_summary(config) + _clear_wizard_state() + if _prompt_yes_no("Start training now?", default=False): + _print(f"\n Running: forgelm --config {config_filename}") + _print() + return config_filename + _print("\n To start training later, run:") + _print(f" forgelm --config {config_filename}") + _print() + return None diff --git a/forgelm/wizard/_state.py b/forgelm/wizard/_state.py new file mode 100644 index 00000000..9eefc275 --- /dev/null +++ b/forgelm/wizard/_state.py @@ -0,0 +1,290 @@ +"""Wizard state, persistence, summary + step-diff helpers. + +Persistence (Phase 22 / G6): state snapshot is YAML at +``$XDG_CACHE_HOME/forgelm/wizard_state.yaml`` (or +``~/.cache/forgelm/wizard_state.yaml`` when XDG is unset). Schema +versioned so a future shape change can clear stale snapshots cleanly. +""" + +from __future__ import annotations + +import logging +import os +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Dict, List, Mapping, Optional + +import yaml + +from ._io import _print + +logger = logging.getLogger("forgelm.wizard") + + +# --------------------------------------------------------------------------- +# Wizard-wide defaults — kept in lockstep with the Pydantic schema in +# ``forgelm/config.py``. Wherever a wizard-side default deliberately +# differs from the schema, the comment names the rationale. +# --------------------------------------------------------------------------- + + +DEFAULT_MAX_LENGTH = 2048 +# ``LoraConfigModel.r`` default in ``forgelm/config.py`` is ``8``; the +# wizard tracks the schema default exactly so an operator who accepts +# every prompt produces a YAML byte-equivalent to ``ForgeConfig()``. +DEFAULT_LORA_R = 8 +DEFAULT_LORA_ALPHA = 2 * DEFAULT_LORA_R +DEFAULT_DROPOUT = 0.1 +DEFAULT_EPOCHS = 3 +DEFAULT_BATCH_SIZE = 4 +DEFAULT_LR = 2e-5 + +# Curated popular-model presets. This list and the +# ``USE_CASE_PRESETS`` in ``site/js/wizard.js`` are intentionally kept +# in lockstep — the operator using both surfaces sees the same +# recommended models. +POPULAR_MODELS: List[str] = [ + "meta-llama/Llama-3.1-8B-Instruct", + "Qwen/Qwen2.5-7B-Instruct", + "mistralai/Mistral-7B-Instruct-v0.3", + "microsoft/Phi-3-mini-4k-instruct", + "HuggingFaceTB/SmolLM2-1.7B-Instruct", +] + + +TARGET_MODULE_PRESETS: Dict[str, List[str]] = { + "standard": ["q_proj", "v_proj"], + "extended": ["q_proj", "k_proj", "v_proj", "o_proj"], + "full": ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], +} + + +# Common dataset-format hints for preference-based trainers +# (DPO/SimPO/ORPO). +_PREFERENCE_COLUMNS_HINT = "Columns: prompt, chosen, rejected" + + +# --------------------------------------------------------------------------- +# Persistence +# --------------------------------------------------------------------------- + + +# Bumped whenever the on-disk wizard-state shape changes. Version +# mismatches are silently ignored (treated as "no resume available"). +_STATE_VERSION = 1 + + +def _wizard_state_dir() -> Path: + """Return the directory the wizard persists its state under. + + Honours ``XDG_CACHE_HOME`` per the XDG Base Directory spec; falls + back to ``~/.cache/forgelm`` when unset. The directory is created + on first write — no side effects on plain reads. + """ + xdg = os.environ.get("XDG_CACHE_HOME", "").strip() + base = Path(xdg).expanduser() if xdg else Path.home() / ".cache" + return base / "forgelm" + + +def _wizard_state_path() -> Path: + """Absolute path to ``wizard_state.yaml``.""" + return _wizard_state_dir() / "wizard_state.yaml" + + +def _save_wizard_state(state: Mapping[str, Any]) -> None: + """Persist *state* to ``_wizard_state_path()``. + + Best-effort: filesystem errors (read-only home, ENOSPC, sandboxed + container) are logged at WARNING and swallowed — the wizard's + contract is to *produce a config*, not to guarantee resumability. + """ + target = _wizard_state_path() + try: + target.parent.mkdir(parents=True, exist_ok=True) + snapshot = {"v": _STATE_VERSION, **dict(state)} + with open(target, "w", encoding="utf-8") as fh: + yaml.safe_dump(snapshot, fh, default_flow_style=False, sort_keys=False) + except OSError as exc: + logger.warning("Could not persist wizard state to %s: %s", target, exc) + + +def _load_wizard_state() -> Optional[Dict[str, Any]]: + """Load a previously-saved state snapshot, returning ``None`` on miss. + + Version mismatches and parse errors return ``None`` — the wizard + falls back to defaults rather than asking the operator to debug a + stale snapshot. + """ + target = _wizard_state_path() + if not target.is_file(): + return None + try: + with open(target, "r", encoding="utf-8") as fh: + snapshot = yaml.safe_load(fh) + except (OSError, yaml.YAMLError) as exc: + logger.warning("Could not read wizard state from %s: %s", target, exc) + return None + if not isinstance(snapshot, dict) or snapshot.get("v") != _STATE_VERSION: + return None + snapshot.pop("v", None) + return snapshot + + +def _clear_wizard_state() -> None: + """Remove a stale state snapshot, swallowing filesystem errors.""" + target = _wizard_state_path() + try: + if target.is_file(): + target.unlink() + except OSError as exc: # pragma: no cover — best-effort cleanup + logger.warning("Could not remove wizard state at %s: %s", target, exc) + + +# --------------------------------------------------------------------------- +# Wizard-state dataclass — threaded through every step in the +# orchestrator. Persisted via the helpers above. +# --------------------------------------------------------------------------- + + +# Sentinel for the "manual" use-case (no preselect). Mirrored in +# ``_byod._MANUAL_USE_CASE`` for callers that want to import from +# either submodule. +_MANUAL_USE_CASE = "custom" + + +@dataclass +class _WizardState: + """In-memory snapshot of every wizard answer + the partial config dict. + + Persisted after each completed step to ``wizard_state.yaml`` so a + refresh / Ctrl-C / fresh session resumes where the operator left + off. + """ + + experience: str = "expert" # "beginner" | "expert" + use_case: str = _MANUAL_USE_CASE + current_step: int = 0 + completed_steps: List[str] = field(default_factory=list) + config: Dict[str, Any] = field(default_factory=dict) + + +# --------------------------------------------------------------------------- +# Step-diff preview — Phase 22 / G7. Each step prints the keys it +# just added or changed, mirroring the web wizard's live YAML preview +# at terminal-friendly granularity. +# --------------------------------------------------------------------------- + + +def _flatten_dict(d: Mapping[str, Any], *, prefix: str = "") -> Dict[str, Any]: + """Recursively flatten a dict into ``dotted.path → value`` entries.""" + out: Dict[str, Any] = {} + for key, value in d.items(): + path = f"{prefix}.{key}" if prefix else str(key) + if isinstance(value, Mapping): + out.update(_flatten_dict(value, prefix=path)) + else: + out[path] = value + return out + + +def _print_step_diff(prev: Mapping[str, Any], curr: Mapping[str, Any], step_label: str) -> None: + """Print a ``+ key.path: value`` diff for everything *curr* added vs *prev*.""" + prev_flat = _flatten_dict(prev) + curr_flat = _flatten_dict(curr) + added = {k: v for k, v in curr_flat.items() if k not in prev_flat} + changed = {k: (prev_flat[k], v) for k, v in curr_flat.items() if k in prev_flat and prev_flat[k] != v} + if not added and not changed: + return + _print(f"\n Step diff ({step_label}):") + for key in sorted(added): + _print(f" + {key}: {added[key]!r}") + for key in sorted(changed): + before, after = changed[key] + _print(f" ~ {key}: {before!r} → {after!r}") + + +# --------------------------------------------------------------------------- +# Save + summary helpers +# --------------------------------------------------------------------------- + + +def _save_config_to_file(config: Dict[str, Any], requested_filename: str) -> str: + """Write *config* as YAML; falls back to a unique filename on OSError.""" + try: + with open(requested_filename, "w") as f: + yaml.dump(config, f, default_flow_style=False, sort_keys=False) + _print(f"\n Config saved to: {requested_filename}") + logger.info("Wizard config saved to %s", requested_filename) + return requested_filename + except OSError as e: + logger.error("Could not save wizard config to %s: %s", requested_filename, e) + _print(f"\n Error: Could not save config to {requested_filename}: {e}") + + from datetime import datetime as _dt + + base = os.path.splitext(os.path.basename(requested_filename))[0] or "my_config" + fallback = os.path.join( + os.path.expanduser("~"), + f"{base}_{_dt.now().strftime('%Y%m%d_%H%M%S')}.yaml", + ) + try: + with open(fallback, "w") as f: + yaml.dump(config, f, default_flow_style=False, sort_keys=False) + _print(f" Saved to fallback location: {fallback}") + logger.info("Wizard config saved to fallback location %s", fallback) + return fallback + except OSError as e: + logger.error("Fallback wizard config save also failed (%s): %s", fallback, e) + _print(f" Fallback save also failed ({fallback}): {e}") + raise + + +def _strategy_label(*, use_galore: bool, load_in_4bit: bool, method: str) -> str: + """Human-readable label for the strategy summary line.""" + if use_galore: + return "GaLore" + base = "QLoRA" if load_in_4bit else "LoRA" + if method != "lora": + return f"{base} + {method.upper()}" + return base + + +def _print_wizard_summary(config: Dict[str, Any]) -> None: + """Print a multi-section summary of the resolved config + the full YAML.""" + _print("\n" + "=" * 60) + _print(" Configuration Summary") + _print("=" * 60) + model = config["model"]["name_or_path"] + backend = config["model"].get("backend", "transformers") + load_in_4bit = config["model"].get("load_in_4bit", False) + lora_method = config.get("lora", {}).get("method", "lora") + use_galore = bool(config["training"].get("galore_enabled", False)) + trainer = config["training"].get("trainer_type", "sft").upper() + lora_r = config.get("lora", {}).get("r", DEFAULT_LORA_R) + lora_alpha = config.get("lora", {}).get("alpha", DEFAULT_LORA_ALPHA) + dataset = config.get("data", {}).get("dataset_name_or_path", "(not set)") + epochs = config["training"].get("num_train_epochs", DEFAULT_EPOCHS) + batch = config["training"].get("per_device_train_batch_size", DEFAULT_BATCH_SIZE) + output = config["training"].get("output_dir", "./checkpoints") + _print(f" Model: {model}") + _print(f" Backend: {backend}") + _print(f" Strategy: {_strategy_label(use_galore=use_galore, load_in_4bit=load_in_4bit, method=lora_method)}") + _print(f" Trainer: {trainer}") + _print(f" LoRA: r={lora_r}, alpha={lora_alpha}") + _print(f" Dataset: {dataset}") + _print(f" Epochs: {epochs}, Batch: {batch}") + _print(f" Output: {output}/final_model") + sections = [k for k in config.keys() if k not in ("model", "lora", "training", "data")] + if sections: + _print(f" Extras: {', '.join(sorted(sections))}") + _print("\n Full YAML preview:") + _print(" " + "─" * 58) + yaml_text = yaml.dump(config, default_flow_style=False, sort_keys=False) + for line in yaml_text.splitlines(): + _print(f" {line}") + _print(" " + "─" * 58) + + +def _strip_internal_meta(config: Mapping[str, Any]) -> Dict[str, Any]: + """Remove keys prefixed with ``_wizard_`` before writing the YAML.""" + return {k: v for k, v in config.items() if not str(k).startswith("_wizard_")} diff --git a/site/js/wizard.js b/site/js/wizard.js index 97ad2060..c74dfb62 100644 --- a/site/js/wizard.js +++ b/site/js/wizard.js @@ -2,12 +2,12 @@ * ForgeLM site — in-browser YAML wizard. * * Mirrors the CLI ``forgelm --wizard`` flow but lives entirely in the - * browser: walks the operator through 7 steps (welcome / use-case / - * model / dataset / training / eval+compliance / review) and emits a - * working ``quickstart-generated.yaml`` they can copy, download, or - * paste into a shell command. Every transition + every input change - * is persisted to ``localStorage`` so a refresh resumes the same - * step. Behaviour summary: + * browser: walks the operator through 9 steps (welcome / use-case / + * trainer / model / dataset / training / compliance / operations / + * review) and emits a working ``quickstart-generated.yaml`` they can + * copy, download, or paste into a shell command. Every transition + + * every input change is persisted to ``localStorage`` so a refresh + * resumes the same step. Behaviour summary: * * - Triggers: ``[data-wizard-open]`` clicks + the ``?wizard=open`` * URL parameter (set by the home-page Beginner-slide CTA). @@ -106,7 +106,7 @@ step: 0, experience: 'beginner', // 'beginner' | 'expert' detailsVisible: true, // toggle for inline tutorial paragraphs - useCase: null, // 'customer-support' | 'code-copilot' | 'domain-expert' | 'grpo-math' | 'medical-tr' | 'custom' + useCase: null, // 'customer-support' | 'code-assistant' | 'domain-expert' | 'grpo-math' | 'medical-qa-tr' | 'custom' // Trainer selection. Use-case picks a sensible preset, but the // operator can override on Step 3 (trainer step). All 6 schema // values reachable: sft / dpo / orpo / simpo / kto / grpo. @@ -441,8 +441,12 @@ // shipped templates under ``forgelm/templates/`` — every shipped // template uses ``trainer_type: sft`` except ``grpo-math`` which // uses ``grpo``. The wizard previously emitted DPO/ORPO for - // customer-support / code-copilot which contradicted the templates; - // this file now matches the runtime defaults. + // customer-support / code-assistant which contradicted the templates; + // this file now matches the runtime defaults. Use-case keys + // (``code-assistant``, ``medical-qa-tr``, …) are kept in lockstep + // with ``forgelm/quickstart.py::TEMPLATES`` — the CLI quickstart + // catalogue is the single source of truth so an operator who + // crosses surfaces never sees a renamed key. var USE_CASE_PRESETS = { 'customer-support': { trainerType: 'sft', @@ -451,7 +455,7 @@ datasetKind: 'huggingface', datasetName: 'argilla/Capybara-Preferences' }, - 'code-copilot': { + 'code-assistant': { trainerType: 'sft', model: 'Qwen/Qwen2.5-Coder-7B-Instruct', modelPreset: 'custom', @@ -472,7 +476,7 @@ datasetKind: 'huggingface', datasetName: 'openai/gsm8k' }, - 'medical-tr': { + 'medical-qa-tr': { trainerType: 'sft', model: 'meta-llama/Llama-3.1-8B-Instruct', modelPreset: 'llama3-8b', @@ -1339,10 +1343,10 @@ var grid = el('div', { class: 'wizard-cards' }); [ { id: 'customer-support', titleKey: 'wizard.usecase.support.title', badgeKey: 'wizard.usecase.support.badge', descKey: 'wizard.usecase.support.desc' }, - { id: 'code-copilot', titleKey: 'wizard.usecase.code.title', badgeKey: 'wizard.usecase.code.badge', descKey: 'wizard.usecase.code.desc' }, + { id: 'code-assistant', titleKey: 'wizard.usecase.code.title', badgeKey: 'wizard.usecase.code.badge', descKey: 'wizard.usecase.code.desc' }, { id: 'domain-expert', titleKey: 'wizard.usecase.domain.title', badgeKey: 'wizard.usecase.domain.badge', descKey: 'wizard.usecase.domain.desc' }, { id: 'grpo-math', titleKey: 'wizard.usecase.math.title', badgeKey: 'wizard.usecase.math.badge', descKey: 'wizard.usecase.math.desc' }, - { id: 'medical-tr', titleKey: 'wizard.usecase.medical.title', badgeKey: 'wizard.usecase.medical.badge', descKey: 'wizard.usecase.medical.desc' }, + { id: 'medical-qa-tr', titleKey: 'wizard.usecase.medical.title', badgeKey: 'wizard.usecase.medical.badge', descKey: 'wizard.usecase.medical.desc' }, { id: 'custom', titleKey: 'wizard.usecase.custom.title', badgeKey: '', descKey: 'wizard.usecase.custom.desc' } ].forEach(function (opt) { var titleNode = el('span', { class: 'wizard-card-title' }, [tr(opt.titleKey)]); diff --git a/tests/test_phase12_5.py b/tests/test_phase12_5.py index 418b181e..344a4f83 100644 --- a/tests/test_phase12_5.py +++ b/tests/test_phase12_5.py @@ -125,7 +125,7 @@ def test_audit_offer_runs_audit_when_user_accepts(self, tmp_path, capsys): self._write_jsonl(ds, [{"text": "alpha"}, {"text": "beta"}]) # Simulate "yes" to the offer. - with patch("forgelm.wizard._prompt_yes_no", return_value=True): + with patch("forgelm.wizard._byod._prompt_yes_no", return_value=True): outcome = _offer_audit_for_jsonl(ds) assert outcome is True @@ -139,7 +139,7 @@ def test_audit_offer_skipped_when_user_declines(self, tmp_path, capsys): self._write_jsonl(ds, [{"text": "alpha"}]) # Simulate "no" — must short-circuit before invoking audit_dataset. - with patch("forgelm.wizard._prompt_yes_no", return_value=False): + with patch("forgelm.wizard._byod._prompt_yes_no", return_value=False): with patch("forgelm.data_audit.audit_dataset") as mock_audit: outcome = _offer_audit_for_jsonl(ds) mock_audit.assert_not_called() @@ -155,7 +155,7 @@ def test_audit_offer_recovers_when_audit_raises(self, tmp_path, capsys): ds = tmp_path / "data.jsonl" self._write_jsonl(ds, [{"text": "alpha"}]) - with patch("forgelm.wizard._prompt_yes_no", return_value=True): + with patch("forgelm.wizard._byod._prompt_yes_no", return_value=True): with patch( "forgelm.data_audit.audit_dataset", side_effect=RuntimeError("simulated audit failure"), diff --git a/tests/test_wizard_phase11.py b/tests/test_wizard_phase11.py index bd4b0a46..bb667e04 100644 --- a/tests/test_wizard_phase11.py +++ b/tests/test_wizard_phase11.py @@ -244,19 +244,27 @@ def test_collect_rope_scaling_short_context_silent(self, max_length, expect_none assert "Long context detected" not in captured def test_print_wizard_summary_includes_strategy_and_dataset(self, capsys): + # Phase 22 rewrite: ``_print_wizard_summary`` takes the resolved + # config dict instead of ~12 keyword arguments. The summary + # body now also includes the full YAML preview (G17), so we + # only assert on the labelled summary lines that the rest of + # the wizard depends on. wizard._print_wizard_summary( - model_name="meta-llama/Llama-3.1-8B-Instruct", - suggested_backend="transformers", - use_galore=False, - load_in_4bit=True, - use_dora=False, - trainer_type="sft", - lora_r=16, - lora_alpha=32, - dataset_path="tatsu-lab/alpaca", - epochs=3, - batch_size=4, - output_dir="./checkpoints", + { + "model": { + "name_or_path": "meta-llama/Llama-3.1-8B-Instruct", + "backend": "transformers", + "load_in_4bit": True, + }, + "lora": {"r": 16, "alpha": 32, "method": "lora"}, + "training": { + "trainer_type": "sft", + "num_train_epochs": 3, + "per_device_train_batch_size": 4, + "output_dir": "./checkpoints", + }, + "data": {"dataset_name_or_path": "tatsu-lab/alpaca"}, + } ) out = capsys.readouterr().out assert "Configuration Summary" in out @@ -271,38 +279,37 @@ def test_print_wizard_summary_galore_strategy(self, capsys): # branch so a reordering of the if/elif ladder can't silently # swallow the GaLore label in summaries. wizard._print_wizard_summary( - model_name="m", - suggested_backend="transformers", - use_galore=True, - load_in_4bit=False, - use_dora=False, - trainer_type="sft", - lora_r=8, - lora_alpha=16, - dataset_path="d", - epochs=1, - batch_size=1, - output_dir="./out", + { + "model": {"name_or_path": "m", "backend": "transformers", "load_in_4bit": False}, + "lora": {"r": 8, "alpha": 16, "method": "lora"}, + "training": { + "trainer_type": "sft", + "galore_enabled": True, + "num_train_epochs": 1, + "per_device_train_batch_size": 1, + "output_dir": "./out", + }, + "data": {"dataset_name_or_path": "d"}, + } ) assert "Strategy: GaLore" in capsys.readouterr().out def test_print_wizard_summary_dora_suffix(self, capsys): wizard._print_wizard_summary( - model_name="m", - suggested_backend="transformers", - use_galore=False, - load_in_4bit=True, - use_dora=True, - trainer_type="dpo", - lora_r=8, - lora_alpha=16, - dataset_path="d", - epochs=1, - batch_size=1, - output_dir="./out", + { + "model": {"name_or_path": "m", "backend": "transformers", "load_in_4bit": True}, + "lora": {"r": 8, "alpha": 16, "method": "dora"}, + "training": { + "trainer_type": "dpo", + "num_train_epochs": 1, + "per_device_train_batch_size": 1, + "output_dir": "./out", + }, + "data": {"dataset_name_or_path": "d"}, + } ) out = capsys.readouterr().out - assert "Strategy: QLoRA + DoRA" in out + assert "Strategy: QLoRA + DORA" in out assert "Trainer: DPO" in out diff --git a/tests/test_wizard_phase22.py b/tests/test_wizard_phase22.py new file mode 100644 index 00000000..d9b79589 --- /dev/null +++ b/tests/test_wizard_phase22.py @@ -0,0 +1,492 @@ +"""Phase 22 / 2026-05-08: CLI wizard parity-with-web modernisation tests. + +Covers the new helpers + step-machine plumbing introduced when the CLI +wizard was extended to close the parity gap with ``site/js/wizard.js`` +documented in +``docs/analysis/code_reviews/2026-05-07_cli_wizard_ux_analysis.md``: + +- ``_parse_webhook_value`` — single-prompt webhook syntax with + ``env:VAR_NAME`` prefix and HTTPS validation. +- ``_default_safety_probes_path`` — package-data resolution of the + bundled probe set. +- ``_collect_trainer_hyperparameters`` — per-trainer fields gating. +- ``_check_navigation_token`` — back / reset sentinels. +- ``_apply_strict_tier_coercion`` — F-compliance-110 front-stop. +- ``_save_wizard_state`` / ``_load_wizard_state`` / + ``_clear_wizard_state`` — XDG-aware persistence. +- ``_print_step_diff`` — terminal-friendly state diff. + +No GPU / no network — every test stubs ``builtins.input`` for stdin +reads and uses a temp directory for persistence. +""" + +from __future__ import annotations + +from pathlib import Path +from unittest.mock import patch + +import pytest + +from forgelm import wizard + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _input_returning(*answers): + """Build a side_effect for ``builtins.input`` that drains *answers*.""" + queue = list(answers) + + def _impl(_prompt_text=""): + if not queue: + raise AssertionError("input() called more times than answers provided") + return queue.pop(0) + + return _impl + + +# --------------------------------------------------------------------------- +# Webhook URL parsing — Phase 22 / G15 +# --------------------------------------------------------------------------- + + +class TestParseWebhookValue: + def test_empty_returns_none(self): + assert wizard._parse_webhook_value("") is None + assert wizard._parse_webhook_value(" ") is None + + def test_https_url_accepted(self): + section = wizard._parse_webhook_value("https://hooks.slack.com/services/T/B/X") + assert section == {"url": "https://hooks.slack.com/services/T/B/X"} + + def test_env_prefix_emits_url_env(self): + section = wizard._parse_webhook_value("env:SLACK_WEBHOOK_URL") + assert section == {"url_env": "SLACK_WEBHOOK_URL"} + + def test_env_prefix_case_insensitive_keyword(self): + # The ``env:`` keyword itself is matched case-insensitively so + # operators copying values from the web wizard's YAML aren't + # tripped up by case. + section = wizard._parse_webhook_value("ENV:MY_WEBHOOK") + assert section == {"url_env": "MY_WEBHOOK"} + + def test_env_prefix_empty_var_name_rejected(self): + with pytest.raises(ValueError, match="non-empty variable name"): + wizard._parse_webhook_value("env:") + + def test_env_prefix_lowercase_var_name_rejected(self): + # POSIX env-var convention enforces uppercase letters; lower- + # case names break ``${VAR}`` shell expansion in CI. + with pytest.raises(ValueError, match="POSIX environment-variable name"): + wizard._parse_webhook_value("env:lowercase_var") + + def test_bare_string_no_scheme_rejected(self): + with pytest.raises(ValueError, match="not a valid URL"): + wizard._parse_webhook_value("not-a-url") + + def test_unknown_scheme_rejected(self): + with pytest.raises(ValueError, match="must use `https://`"): + wizard._parse_webhook_value("ftp://example.com/hook") + + def test_http_warning_but_accepted(self, capsys): + section = wizard._parse_webhook_value("http://hooks.example.com/x") + assert section == {"url": "http://hooks.example.com/x"} + captured = capsys.readouterr().out + assert "uses HTTP, not HTTPS" in captured + + +# --------------------------------------------------------------------------- +# Safety probe path resolution — Phase 22 / G16 +# --------------------------------------------------------------------------- + + +class TestDefaultSafetyProbesPath: + def test_resolves_to_packaged_jsonl(self): + # The bundled probe set lives at + # ``forgelm/safety_prompts/default_probes.jsonl`` and is + # shipped via ``[tool.setuptools.package-data]`` in + # ``pyproject.toml`` (Phase 36). The resolver must produce a + # real on-disk path the trainer can ``open``. + path = wizard._default_safety_probes_path() + assert path.endswith("default_probes.jsonl") + assert Path(path).is_file() + + def test_path_is_absolute_or_resolvable(self): + # ``pkgutil.get_data`` would return bytes; the wizard prefers + # ``importlib.resources.files`` so the operator gets a string + # path they can hand to ``forgelm safety-eval --probes``. + path = wizard._default_safety_probes_path() + assert Path(path).resolve().is_file() + + +# --------------------------------------------------------------------------- +# Navigation tokens — Phase 22 / G3 +# --------------------------------------------------------------------------- + + +class TestNavigationTokens: + def test_back_token_raises_wizardback(self): + for token in ("back", "b", "BACK", " Back "): + with pytest.raises(wizard.WizardBack): + wizard._check_navigation_token(token) + + def test_reset_token_raises_wizardreset(self): + for token in ("reset", "r", "RESET", " Reset "): + with pytest.raises(wizard.WizardReset): + wizard._check_navigation_token(token) + + def test_cancel_token_does_not_raise(self): + # Cancel is contextual — the BYOD path interprets it as "fall + # back to the full wizard" and the step orchestrator relies on + # Ctrl-C / Ctrl-D for clean exits. Auto-raising on cancel + # would break the existing BYOD flow. + for token in ("cancel", "c", "q", "quit"): + wizard._check_navigation_token(token) + + def test_empty_string_does_not_raise(self): + wizard._check_navigation_token("") + wizard._check_navigation_token(" ") + + def test_normal_input_does_not_raise(self): + wizard._check_navigation_token("Llama-3.1-8B") + wizard._check_navigation_token("yes") + wizard._check_navigation_token("3") + + +# --------------------------------------------------------------------------- +# Trainer-specific hyperparameters — Phase 22 / G1 +# --------------------------------------------------------------------------- + + +class TestCollectTrainerHyperparameters: + def test_sft_returns_empty(self): + # SFT has no per-trainer knobs in ``TrainingConfig``. + with patch("builtins.input", side_effect=_input_returning()): + result = wizard._collect_trainer_hyperparameters("sft") + assert result == {} + + def test_dpo_returns_dpo_beta(self): + with patch("builtins.input", side_effect=_input_returning("0.1")): + result = wizard._collect_trainer_hyperparameters("dpo") + assert result == {"dpo_beta": 0.1} + + def test_simpo_returns_beta_and_gamma(self): + with patch("builtins.input", side_effect=_input_returning("2.0", "0.5")): + result = wizard._collect_trainer_hyperparameters("simpo") + assert result == {"simpo_beta": 2.0, "simpo_gamma": 0.5} + + def test_kto_returns_kto_beta(self): + with patch("builtins.input", side_effect=_input_returning("0.1")): + result = wizard._collect_trainer_hyperparameters("kto") + assert result == {"kto_beta": 0.1} + + def test_orpo_returns_orpo_beta(self): + with patch("builtins.input", side_effect=_input_returning("0.1")): + result = wizard._collect_trainer_hyperparameters("orpo") + assert result == {"orpo_beta": 0.1} + + def test_grpo_returns_full_block_with_reward_model(self): + with patch( + "builtins.input", + side_effect=_input_returning("4", "512", "my_reward.score"), + ): + result = wizard._collect_trainer_hyperparameters("grpo") + assert result == { + "grpo_num_generations": 4, + "grpo_max_completion_length": 512, + "grpo_reward_model": "my_reward.score", + } + + def test_grpo_omits_reward_model_when_blank(self): + # Empty reward_model means "use the built-in shaper" — which + # the schema represents as ``None``. The wizard drops the key + # entirely so the YAML doesn't carry an empty string the + # validator would have to special-case. + with patch("builtins.input", side_effect=_input_returning("4", "512", "")): + result = wizard._collect_trainer_hyperparameters("grpo") + assert result == { + "grpo_num_generations": 4, + "grpo_max_completion_length": 512, + } + assert "grpo_reward_model" not in result + + +# --------------------------------------------------------------------------- +# Strict-tier auto-coercion — Phase 22 / G8 +# --------------------------------------------------------------------------- + + +class TestApplyStrictTierCoercion: + def test_minimal_risk_no_op(self): + config = {"evaluation": {"auto_revert": True}} + wizard._apply_strict_tier_coercion(config, {"risk_classification": "minimal-risk"}) + # Should NOT have added safety + require_human_approval. + assert "safety" not in config["evaluation"] + assert "require_human_approval" not in config["evaluation"] + + def test_high_risk_enables_safety_and_human_approval(self, capsys): + config: dict = {} + wizard._apply_strict_tier_coercion(config, {"risk_classification": "high-risk"}) + assert config["evaluation"]["require_human_approval"] is True + assert config["evaluation"]["safety"]["enabled"] is True + assert config["evaluation"]["auto_revert"] is True + captured = capsys.readouterr().out + assert "Article 9" in captured + assert "Article 14" in captured + + def test_unacceptable_also_triggers_coercion(self): + config: dict = {} + wizard._apply_strict_tier_coercion(config, {"risk_classification": "unacceptable"}) + assert config["evaluation"]["safety"]["enabled"] is True + + def test_does_not_overwrite_explicit_safety_block(self): + # Operator already configured safety with track_categories=False; + # coercion mustn't clobber their choices when safety is already + # enabled. + config = {"evaluation": {"safety": {"enabled": True, "track_categories": False}}} + wizard._apply_strict_tier_coercion(config, {"risk_classification": "high-risk"}) + assert config["evaluation"]["safety"]["enabled"] is True + assert config["evaluation"]["safety"]["track_categories"] is False + + +# --------------------------------------------------------------------------- +# Persistence — Phase 22 / G6 +# --------------------------------------------------------------------------- + + +@pytest.fixture +def isolated_state_dir(tmp_path, monkeypatch): + """Redirect the wizard state directory to a tmp_path.""" + monkeypatch.setenv("XDG_CACHE_HOME", str(tmp_path)) + yield tmp_path / "forgelm" + + +class TestPersistence: + def test_load_returns_none_when_no_snapshot(self, isolated_state_dir): + assert wizard._load_wizard_state() is None + + def test_save_and_load_roundtrip(self, isolated_state_dir): + snapshot = { + "experience": "beginner", + "use_case": "customer-support", + "current_step": 3, + "completed_steps": ["welcome", "use-case", "model"], + "config": {"model": {"name_or_path": "Qwen/Qwen2.5-7B-Instruct"}}, + } + wizard._save_wizard_state(snapshot) + loaded = wizard._load_wizard_state() + assert loaded == snapshot + + def test_clear_removes_snapshot(self, isolated_state_dir): + wizard._save_wizard_state({"experience": "expert"}) + wizard._clear_wizard_state() + assert wizard._load_wizard_state() is None + + def test_clear_is_idempotent(self, isolated_state_dir): + # Clearing a non-existent snapshot must not raise — the wizard + # calls ``_clear_wizard_state`` on successful completion even + # when nothing was saved. + wizard._clear_wizard_state() + wizard._clear_wizard_state() # still fine + + def test_load_returns_none_on_version_mismatch(self, isolated_state_dir): + path = wizard._wizard_state_path() + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("v: 99999\nexperience: expert\n", encoding="utf-8") + assert wizard._load_wizard_state() is None + + def test_load_returns_none_on_corrupt_snapshot(self, isolated_state_dir): + path = wizard._wizard_state_path() + path.parent.mkdir(parents=True, exist_ok=True) + # Invalid YAML — must not raise; just silently miss. + path.write_text("not: valid: yaml: at: all", encoding="utf-8") + assert wizard._load_wizard_state() is None + + def test_state_path_honours_xdg_cache_home(self, tmp_path, monkeypatch): + monkeypatch.setenv("XDG_CACHE_HOME", str(tmp_path)) + assert wizard._wizard_state_path() == tmp_path / "forgelm" / "wizard_state.yaml" + + def test_state_path_falls_back_to_home_cache_when_xdg_unset(self, monkeypatch): + monkeypatch.delenv("XDG_CACHE_HOME", raising=False) + path = wizard._wizard_state_path() + # ``Path.home() / ".cache"`` may differ across CI environments, + # but the trailing components are stable. + assert path.name == "wizard_state.yaml" + assert path.parent.name == "forgelm" + + +# --------------------------------------------------------------------------- +# Step-diff preview — Phase 22 / G7 +# --------------------------------------------------------------------------- + + +class TestStepDiff: + def test_no_diff_emits_nothing(self, capsys): + wizard._print_step_diff({"a": 1}, {"a": 1}, "step1") + assert capsys.readouterr().out == "" + + def test_added_keys_marked_with_plus(self, capsys): + wizard._print_step_diff({}, {"model": {"name_or_path": "X"}}, "model") + out = capsys.readouterr().out + assert "Step diff (model):" in out + assert "+ model.name_or_path: 'X'" in out + + def test_changed_keys_marked_with_tilde(self, capsys): + wizard._print_step_diff({"a": 1}, {"a": 2}, "params") + out = capsys.readouterr().out + assert "~ a: 1 → 2" in out + + def test_nested_dict_paths_dotted(self, capsys): + wizard._print_step_diff( + {"training": {"epochs": 3}}, + {"training": {"epochs": 3, "batch_size": 4}}, + "training-params", + ) + out = capsys.readouterr().out + assert "+ training.batch_size: 4" in out + assert "training.epochs" not in out # unchanged → not printed + + +# --------------------------------------------------------------------------- +# Webhook collector — Phase 22 / G15 (integration around ``_parse_webhook_value``) +# --------------------------------------------------------------------------- + + +class TestCollectWebhookConfig: + def test_decline_returns_none(self): + with patch("builtins.input", side_effect=_input_returning("n")): + assert wizard._collect_webhook_config() is None + + def test_env_prefix_default_accepted(self): + # The prompt default is ``env:FORGELM_WEBHOOK_URL`` so an empty + # answer accepts the suggested env-var. + with patch("builtins.input", side_effect=_input_returning("y", "")): + section = wizard._collect_webhook_config() + assert section == { + "url_env": "FORGELM_WEBHOOK_URL", + "notify_on_start": True, + "notify_on_success": True, + "notify_on_failure": True, + } + + def test_https_url_accepted(self): + with patch( + "builtins.input", + side_effect=_input_returning("y", "https://hooks.slack.com/x"), + ): + section = wizard._collect_webhook_config() + assert section is not None + assert section["url"] == "https://hooks.slack.com/x" + + def test_invalid_url_reprompts(self, capsys): + # First answer is malformed; second is a valid env: ref. + with patch( + "builtins.input", + side_effect=_input_returning("y", "not-a-url", "env:MY_HOOK"), + ): + section = wizard._collect_webhook_config() + assert section is not None + assert section["url_env"] == "MY_HOOK" + captured = capsys.readouterr().out + assert "not a valid URL" in captured + + +# --------------------------------------------------------------------------- +# Use-case preset registry — Phase 22 / G12 + I4 +# --------------------------------------------------------------------------- + + +class TestUseCasePresets: + def test_presets_match_quickstart_template_keys(self): + # I4: the web wizard's USE_CASE_PRESETS keys must adopt the + # CLI quickstart TEMPLATES keys (single source of truth). This + # asserts the CLI side of the agreement: every TEMPLATES key + # appears in ``_wizard_use_case_presets``. + from forgelm.quickstart import TEMPLATES + + presets = wizard._wizard_use_case_presets() + for key in TEMPLATES.keys(): + assert key in presets, f"Use-case preset for '{key}' missing" + + def test_custom_preset_present(self): + # ``custom`` is a wizard-only escape hatch that doesn't seed + # any defaults. + presets = wizard._wizard_use_case_presets() + assert wizard._MANUAL_USE_CASE in presets + assert presets[wizard._MANUAL_USE_CASE]["model"] is None + + +# --------------------------------------------------------------------------- +# Strategy choice mapping — Phase 22 / G2 +# --------------------------------------------------------------------------- + + +class TestStrategyChoices: + def test_six_strategies_listed(self): + # QLoRA / LoRA / DoRA / PiSSA / rsLoRA / GaLore — the full + # ``LoraConfigModel.method`` Literal plus GaLore as a separate + # axis. + assert len(wizard._STRATEGY_CHOICES) == 6 + + def test_methods_cover_full_lora_literal(self): + methods = {choice.method for choice in wizard._STRATEGY_CHOICES if not choice.use_galore} + assert methods == {"lora", "dora", "pissa", "rslora"} + + def test_galore_choice_is_singleton(self): + galore_choices = [c for c in wizard._STRATEGY_CHOICES if c.use_galore] + assert len(galore_choices) == 1 + + +# --------------------------------------------------------------------------- +# GaLore optimizer variants — Phase 22 / G9 +# --------------------------------------------------------------------------- + + +class TestGaloreOptimizers: + def test_six_variants_listed(self): + assert len(wizard._GALORE_OPTIMIZERS) == 6 + + def test_includes_layerwise_siblings(self): + # The original wizard listed only the three base variants. + # Phase 22 surfaces all six, including ``_layerwise`` siblings + # that drop peak VRAM further by recomputing per-layer. + assert "galore_adamw_layerwise" in wizard._GALORE_OPTIMIZERS + assert "galore_adamw_8bit_layerwise" in wizard._GALORE_OPTIMIZERS + assert "galore_adafactor_layerwise" in wizard._GALORE_OPTIMIZERS + + +# --------------------------------------------------------------------------- +# Schema-default parity — Phase 22 / G11 +# --------------------------------------------------------------------------- + + +class TestSchemaDefaultParity: + def test_lora_r_default_matches_schema(self): + # G11: the v0.5.5 wizard set DEFAULT_LORA_R = 16 while the + # schema default in ``LoraConfigModel.r`` was 8. Phase 22 + # aligns the wizard with the schema so an operator who accepts + # every prompt produces a YAML byte-equivalent to + # ``ForgeConfig()``. + from forgelm.config import LoraConfigModel + + assert wizard.DEFAULT_LORA_R == LoraConfigModel.model_fields["r"].default + + def test_lora_alpha_default_is_2x_r(self): + # Adapter convention: alpha = 2 * r. + assert wizard.DEFAULT_LORA_ALPHA == 2 * wizard.DEFAULT_LORA_R + + def test_default_epochs_matches_schema(self): + from forgelm.config import TrainingConfig + + assert wizard.DEFAULT_EPOCHS == TrainingConfig.model_fields["num_train_epochs"].default + + def test_default_batch_size_matches_schema(self): + from forgelm.config import TrainingConfig + + assert wizard.DEFAULT_BATCH_SIZE == TrainingConfig.model_fields["per_device_train_batch_size"].default + + def test_default_lr_matches_schema(self): + from forgelm.config import TrainingConfig + + assert wizard.DEFAULT_LR == TrainingConfig.model_fields["learning_rate"].default From c0dd57a96220b7d9e2e8d69ccd9b92bb27cb1dd0 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sat, 9 May 2026 01:58:31 +0300 Subject: [PATCH 02/17] =?UTF-8?q?fix(wizard):=20PR=20#40=20review=20sweep?= =?UTF-8?q?=20=E2=80=94=2018=20verified=20findings=20(Phase=2022)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses inline review comments + verified user findings on the Phase 22 wizard modernisation PR. Critical / major / minor mix: Critical (2): - _io.py: torch.cuda.get_device_properties().total_mem → total_memory (the v0.5.5 bug raised AttributeError on real CUDA hardware). - _orchestrator.py: WizardReset re-loops with a fresh _WizardState instead of returning early; returning let _run_full_wizard treat the reset as a completed run and try to save an empty config. Major (6): - _orchestrator.py: WizardBack now restores the prev_config snapshot so a half-completed step's mutations don't leak into the previous step's prompts. - _orchestrator.py: deep copy now uses copy.deepcopy instead of json.loads(json.dumps(...)) — the JSON path was lossy for tuples and sets that may sneak in via collector helpers. - _orchestrator.py: _step_model preserves a use-case preselect that isn't in POPULAR_MODELS (defaults to the Custom slot pre-filled with the operator's earlier choice). - _byod.py: _prompt_dataset_path_with_ingest_offer validates that the typed value is a directory, JSONL/JSON file, or HF Hub ID; bare typos no longer silently become data.dataset_name_or_path. - _collectors.py: Article 9 strict-tier branch enforces non-empty intended_use / foreseeable_misuse / mitigation_measures. - _collectors.py: Article 10 strict-tier branch enforces non-empty collection_method / annotation_process / known_biases. Medium (4): - _collectors.py: notify_on_start default flips to False (web wizard parity — start pings are noisy). - _collectors.py: _collect_compliance_metadata collects risk_classification first so strict-tier hint shows up before Article 11 fields and downstream Article 9/10 branches see the correct context. - _state.py: _save_config_to_file uses yaml.safe_dump + UTF-8 + allow_unicode (yaml.dump could emit !!python/object tags that ForgeConfig then rejects). Same treatment for the YAML preview. - _state.py: chmod 0o600 on wizard_state.yaml (best-effort) — compliance metadata can be sensitive. - _io.py: _prompt_choice guards negative idx and out-of-range int alongside the existing IndexError fallback. Minor (3): - _byod.py: shlex.quote() the path in BYOD shell-hint output so paths with spaces survive copy-paste into a real shell. - docs/design/wizard_mode.md: MD030 — single space after ordered list markers (was double). - docs/reference/architecture{,-tr}.md: heading wizard.py → wizard/ (the module became a sub-package in Phase 22). Coverage (HIGH): - tests/test_wizard_phase22.py adds end-to-end coverage for _drive_wizard_steps (back-restores-config, reset-re-loops, back-at-first-step is no-op, persist-after-each-step) plus _maybe_resume_state (resume-from-snapshot, decline-clears) and _step_welcome (backend-hint + beginner branch). Orchestrator module coverage rises from 14% → 30% on this file alone. All checks green: - ruff check + format clean - 1544 passed, 14 skipped (full suite) - 8 doc guards: bilingual_parity, anchor_resolution, cli_help_consistency, yaml_snippets, audit_event_catalog, library_api_doc, doc_numerical_claims, bilingual_code_blocks - module_size: no new warnings - forgelm --config config_template.yaml --dry-run Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/design/wizard_mode.md | 94 ++++++------ docs/reference/architecture-tr.md | 2 +- docs/reference/architecture.md | 2 +- forgelm/wizard/_byod.py | 34 ++++- forgelm/wizard/_collectors.py | 93 +++++++++--- forgelm/wizard/_io.py | 4 +- forgelm/wizard/_orchestrator.py | 31 +++- forgelm/wizard/_state.py | 26 +++- tests/test_wizard_phase22.py | 242 +++++++++++++++++++++++++++++- 9 files changed, 437 insertions(+), 91 deletions(-) diff --git a/docs/design/wizard_mode.md b/docs/design/wizard_mode.md index df302a8a..4dd30c17 100644 --- a/docs/design/wizard_mode.md +++ b/docs/design/wizard_mode.md @@ -30,53 +30,53 @@ plus an optional quickstart-template prelude (`_maybe_run_quickstart_template`). Step ordering and naming are kept in lockstep with the in-browser web wizard at `site/js/wizard.js`: -0. **Quickstart prelude** *(optional)*: offer the curated - `forgelm/quickstart.py::TEMPLATES` list (`customer-support`, - `code-assistant`, `domain-expert`, `medical-qa-tr`, `grpo-math`) - as a one-shot shortcut. When accepted, the prelude generates the - config from the bundled template and skips the full 9-step flow. -1. **Welcome**: experience toggle (beginner / expert), navigation - primer (`back` / `reset` / Ctrl-C), hardware detection (torch + - CUDA + VRAM), backend hint (`unsloth` on Linux + GPU, otherwise - `transformers`). -2. **Use-case**: same registry as the prelude, surfaced inside the - full flow so operators who declined the shortcut still benefit - from sensible preselects (the choice seeds Steps 3 + 5 defaults - but the operator can still override every later answer). -3. **Model**: pick a HuggingFace Hub ID from `POPULAR_MODELS` (kept - in lockstep with `site/js/wizard.js`'s presets) or enter a custom - path. -4. **Strategy**: 6 cards (QLoRA / LoRA / DoRA / PiSSA / rsLoRA / - GaLore) covering the full `LoraConfigModel.method` Literal + - GaLore as a separate axis. Captures `lora.r`, `lora.alpha`, - `lora.target_modules` (standard / extended / full preset). -5. **Trainer + per-trainer hyperparameters**: pick `trainer_type` from - `{sft, dpo, simpo, kto, orpo, grpo}`; the wizard then prompts for - the trainer's specific knobs (`dpo_beta`, `simpo_beta` / - `simpo_gamma`, `kto_beta`, `orpo_beta`, `grpo_num_generations`, - `grpo_max_completion_length`, `grpo_reward_model`). SFT short- - circuits (no per-trainer knobs to surface). -6. **Dataset**: HF Hub ID, local JSONL, or directory of raw documents. - Directory inputs trigger inline ingestion (Phase 11.5). JSONL - inputs trigger an inline audit (Phase 12.5). Optional Article 10 - `data.governance` accordion. -7. **Training parameters**: epochs, batch size, max length, output - directory, RoPE scaling (4 schema types incl. `longrope`), - NEFTune, OOM recovery, GaLore advanced (6-variant optimizer + - rank). -8. **Compliance + risk**: `compliance` (Article 11 + Annex IV §1) - plus optional `risk_assessment` (Article 9), `data.governance` - (Article 10), `retention` (GDPR Article 5(1)(e) + 17), - `monitoring` (Article 12 + 17) accordions. This step is - deliberately collected **before** evaluation so the wizard can - front-stop the F-compliance-110 strict-tier gate. -9. **Operations + evaluation**: `evaluation.auto_revert`, - `evaluation.safety` (Llama Guard, probe set resolved through - `importlib.resources`), `evaluation.benchmark`, - `evaluation.llm_judge`, `webhook` (single prompt with `env:VAR` - prefix sugar), `synthetic` block. After the operator answers, - `_apply_strict_tier_coercion` enforces F-compliance-110 (high- - risk → safety enabled + Article 14 staging gate). +0. **Quickstart prelude** *(optional)*: offer the curated + `forgelm/quickstart.py::TEMPLATES` list (`customer-support`, + `code-assistant`, `domain-expert`, `medical-qa-tr`, `grpo-math`) + as a one-shot shortcut. When accepted, the prelude generates the + config from the bundled template and skips the full 9-step flow. +1. **Welcome**: experience toggle (beginner / expert), navigation + primer (`back` / `reset` / Ctrl-C), hardware detection (torch + + CUDA + VRAM), backend hint (`unsloth` on Linux + GPU, otherwise + `transformers`). +2. **Use-case**: same registry as the prelude, surfaced inside the + full flow so operators who declined the shortcut still benefit + from sensible preselects (the choice seeds Steps 3 + 5 defaults + but the operator can still override every later answer). +3. **Model**: pick a HuggingFace Hub ID from `POPULAR_MODELS` (kept + in lockstep with `site/js/wizard.js`'s presets) or enter a custom + path. +4. **Strategy**: 6 cards (QLoRA / LoRA / DoRA / PiSSA / rsLoRA / + GaLore) covering the full `LoraConfigModel.method` Literal + + GaLore as a separate axis. Captures `lora.r`, `lora.alpha`, + `lora.target_modules` (standard / extended / full preset). +5. **Trainer + per-trainer hyperparameters**: pick `trainer_type` from + `{sft, dpo, simpo, kto, orpo, grpo}`; the wizard then prompts for + the trainer's specific knobs (`dpo_beta`, `simpo_beta` / + `simpo_gamma`, `kto_beta`, `orpo_beta`, `grpo_num_generations`, + `grpo_max_completion_length`, `grpo_reward_model`). SFT short- + circuits (no per-trainer knobs to surface). +6. **Dataset**: HF Hub ID, local JSONL, or directory of raw documents. + Directory inputs trigger inline ingestion (Phase 11.5). JSONL + inputs trigger an inline audit (Phase 12.5). Optional Article 10 + `data.governance` accordion. +7. **Training parameters**: epochs, batch size, max length, output + directory, RoPE scaling (4 schema types incl. `longrope`), + NEFTune, OOM recovery, GaLore advanced (6-variant optimizer + + rank). +8. **Compliance + risk**: `compliance` (Article 11 + Annex IV §1) + plus optional `risk_assessment` (Article 9), `data.governance` + (Article 10), `retention` (GDPR Article 5(1)(e) + 17), + `monitoring` (Article 12 + 17) accordions. This step is + deliberately collected **before** evaluation so the wizard can + front-stop the F-compliance-110 strict-tier gate. +9. **Operations + evaluation**: `evaluation.auto_revert`, + `evaluation.safety` (Llama Guard, probe set resolved through + `importlib.resources`), `evaluation.benchmark`, + `evaluation.llm_judge`, `webhook` (single prompt with `env:VAR` + prefix sugar), `synthetic` block. After the operator answers, + `_apply_strict_tier_coercion` enforces F-compliance-110 (high- + risk → safety enabled + Article 14 staging gate). Persistence: state snapshot is written to `$XDG_CACHE_HOME/forgelm/wizard_state.yaml` after each completed step diff --git a/docs/reference/architecture-tr.md b/docs/reference/architecture-tr.md index ef3bed28..47897be0 100644 --- a/docs/reference/architecture-tr.md +++ b/docs/reference/architecture-tr.md @@ -145,7 +145,7 @@ HuggingFace-uyumlu README.md üretir: YAML front-matter, eğitim parametreleri t ### `synthetic.py` Öğretmen-öğrenci distillasyonu ile sentetik veri üretimi. `SyntheticDataGenerator` sınıfı bir öğretmen modeli (API tabanlı veya yerel) alır, tohum promptlardan eğitim örnekleri üretir ve formatlanmış JSONL veri setleri çıktılar. `--generate-data` CLI flag'i veya `synthetic` config bölümü ile tetiklenir. Yapılandırılabilir öğretmen backend'leri, çıktı formatları ve üretim parametrelerini destekler. -### `wizard.py` +### `wizard/` Etkileşimli CLI sihirbazı. Faz 22 modernizasyonu (2026-05-08) ile `site/js/wizard.js`'e parite getirildi: 9-adımlı state machine (welcome / use-case / model / strategy / trainer / dataset / training-params / compliance / operations), trainer-spesifik hyperparam'lar (`dpo_beta` / `simpo_beta` + `simpo_gamma` / `kto_beta` / `orpo_beta` / `grpo_*`), tam PEFT method kapsamı (`lora` / `dora` / `pissa` / `rslora`) + GaLore ayrı axis, EU AI Act Madde 9 + 10 + 11 + 12+17 uyumluluk akordeon'ları, F-compliance-110 strict-tier auto-coercion, `back` / `reset` navigasyon, XDG-aware persistence (`$XDG_CACHE_HOME/forgelm/wizard_state.yaml`), step-diff preview, beginner / expert toggle ve Phase 11.5 / 12.5 BYOD inline ingest + audit helper'ları (`_offer_ingest_for_directory`, `_offer_audit_for_jsonl`). ### `webhook.py` diff --git a/docs/reference/architecture.md b/docs/reference/architecture.md index bfa60f60..03a54cbb 100644 --- a/docs/reference/architecture.md +++ b/docs/reference/architecture.md @@ -145,7 +145,7 @@ Model merging with 4 strategies: linear interpolation, TIES-Merging (trim + sign ### `synthetic.py` Synthetic data generation via teacher-to-student distillation. The `SyntheticDataGenerator` class takes a teacher model (API-based or local), generates training samples from seed prompts, and outputs formatted JSONL datasets. Triggered via `--generate-data` CLI flag or `synthetic` config section. Supports configurable teacher backends, output formats, and generation parameters. -### `wizard.py` +### `wizard/` Interactive CLI wizard for generating valid YAML configs. Phase 22 modernisation (2026-05-08) brought the CLI to parity with `site/js/wizard.js`: 9-step state machine (welcome / use-case / model / strategy / trainer / dataset / training-params / compliance / operations), per-trainer hyperparameters (`dpo_beta` / `simpo_beta` + `simpo_gamma` / `kto_beta` / `orpo_beta` / `grpo_*`), full PEFT method coverage (`lora` / `dora` / `pissa` / `rslora`) plus GaLore axis, EU AI Act Article 9 + 10 + 11 + 12+17 compliance accordions, F-compliance-110 strict-tier auto-coercion, `back` / `reset` navigation, XDG-aware persistence at `$XDG_CACHE_HOME/forgelm/wizard_state.yaml`, step-diff preview, beginner / expert toggle, and the Phase 11.5 / 12.5 BYOD inline ingest + audit helpers (`_offer_ingest_for_directory`, `_offer_audit_for_jsonl`). ### `webhook.py` diff --git a/forgelm/wizard/_byod.py b/forgelm/wizard/_byod.py index 9928137b..cf96e2b0 100644 --- a/forgelm/wizard/_byod.py +++ b/forgelm/wizard/_byod.py @@ -10,6 +10,7 @@ from __future__ import annotations import json +import shlex from pathlib import Path from typing import List, Optional @@ -79,7 +80,7 @@ def _offer_ingest_for_directory(directory: Path) -> Optional[str]: ): _print( " Skipped — to ingest manually:\n" - f" forgelm ingest {resolved} --recursive --output data/from_docs.jsonl\n" + f" forgelm ingest {shlex.quote(str(resolved))} --recursive --output data/from_docs.jsonl\n" " Then re-run the wizard with the resulting JSONL path." ) return None @@ -159,7 +160,7 @@ def _offer_audit_for_jsonl(jsonl_path: Path) -> bool: ) if not _prompt_yes_no(prompt, default=True): _print(" Skipped — audit can be run later via:") - _print(f" forgelm audit {jsonl_path}") + _print(f" forgelm audit {shlex.quote(str(jsonl_path))}") return False try: @@ -192,7 +193,18 @@ def _offer_audit_for_jsonl(jsonl_path: Path) -> bool: def _prompt_dataset_path_with_ingest_offer(question: str) -> str: - """Prompt for a dataset path; auto-offer ingestion when the user gives a directory.""" + """Prompt for a dataset path; auto-offer ingestion when the user gives a directory. + + Validates that the typed value is one of: + 1. A directory containing ingestible documents (triggers Phase 11.5 + inline ingest). + 2. A local JSONL / JSON file (triggers the Phase 12.5 audit offer). + 3. A HuggingFace Hub dataset ID (``/``). + + Anything else re-prompts. Without this validation a typo silently + becomes ``data.dataset_name_or_path`` and surfaces only at training + time as a confusing HF resolver error. + """ while True: raw = _prompt(question, "").strip() if not raw: @@ -204,9 +216,19 @@ def _prompt_dataset_path_with_ingest_offer(question: str) -> str: if ingested is None: continue return ingested - if candidate.is_file() and candidate.suffix.lower() in (".jsonl", ".json"): - _offer_audit_for_jsonl(candidate.resolve()) - return raw + if candidate.is_file(): + if candidate.suffix.lower() in (".jsonl", ".json"): + _offer_audit_for_jsonl(candidate.resolve()) + return str(candidate.resolve()) + _print( + f" '{candidate}' is not a JSONL / JSON file. " + "Provide a JSONL dataset, a directory of raw documents, or an HF Hub ID." + ) + continue + if _HF_HUB_ID_RE.match(raw): + _print(f" Treating '{raw}' as an HF Hub dataset ID (no local validation).") + return raw + _print(f" '{raw}' is not a local file/directory and is not a valid HF Hub ID (expected '/').") def _validate_local_jsonl(raw_path: str): diff --git a/forgelm/wizard/_collectors.py b/forgelm/wizard/_collectors.py index 834ab0e2..80d83521 100644 --- a/forgelm/wizard/_collectors.py +++ b/forgelm/wizard/_collectors.py @@ -23,9 +23,20 @@ _prompt_float, _prompt_int, _prompt_optional_list, + _prompt_required, _prompt_yes_no, ) + +def _prompt_required_list(question: str) -> list: + """Prompt for a non-empty comma-separated list, re-asking until provided.""" + while True: + items = _prompt_optional_list(question) + if items: + return items + _print(" At least one entry is required.") + + # --------------------------------------------------------------------------- # Risk-tier surface (mirrors ``forgelm.config._STRICT_RISK_TIERS`` / # ``forgelm.config.RiskTier``). @@ -148,7 +159,10 @@ def _collect_webhook_config() -> Optional[Dict[str, Any]]: continue if section is None: return None - section.setdefault("notify_on_start", True) + # Defaults mirror ``site/js/wizard.js`` and ``WebhookConfig`` — + # start notifications are noisy and off by default; success + + # failure are the operationally interesting ones. + section.setdefault("notify_on_start", False) section.setdefault("notify_on_success", True) section.setdefault("notify_on_failure", True) return section @@ -359,19 +373,42 @@ def _collect_galore_config(use_galore: bool) -> Dict[str, Any]: def _collect_compliance_metadata() -> Dict[str, Any]: - """Article 11 + Annex IV §1: provider + system metadata.""" + """Article 11 + Annex IV §1: provider + system metadata. + + ``risk_classification`` is collected FIRST so the operator sees the + strict-tier hint up front; downstream Article 9 / 10 collectors + branch on the chosen tier and the wizard's auto-coercion gate + (``_apply_strict_tier_coercion``) reads it without an awkward + forward-reference. + """ + risk_classification = _prompt_choice( + "Risk classification (mirrored at risk_assessment.risk_category):", + list(_RISK_TIERS), + default=2, # ``minimal-risk`` — same default as the Pydantic field. + ) + if risk_classification in _STRICT_RISK_TIERS: + _print( + " Strict tier selected — Article 11 / Annex IV §1 fields below " + "(provider name, contact, system name, intended purpose) are " + "mandatory and re-prompt until provided." + ) + provider_name = _prompt_required("Organization (legal-entity) name") + provider_contact = _prompt_required("Provider regulatory contact (email or phone)") + system_name = _prompt_required("Human-readable system name") + intended_purpose = _prompt_required("Intended purpose of the system") + else: + provider_name = _prompt("Organization (legal-entity) name", "") + provider_contact = _prompt("Provider regulatory contact (email or phone)", "") + system_name = _prompt("Human-readable system name", "") + intended_purpose = _prompt("Intended purpose of the system", "") return { - "provider_name": _prompt("Organization (legal-entity) name", ""), - "provider_contact": _prompt("Provider regulatory contact (email or phone)", ""), - "system_name": _prompt("Human-readable system name", ""), - "intended_purpose": _prompt("Intended purpose of the system", ""), + "provider_name": provider_name, + "provider_contact": provider_contact, + "system_name": system_name, + "intended_purpose": intended_purpose, "known_limitations": _prompt("Known limitations operator wants documented (free-text)", ""), "system_version": _prompt("System version string", "v0.1.0"), - "risk_classification": _prompt_choice( - "Risk classification (mirrored at risk_assessment.risk_category):", - list(_RISK_TIERS), - default=2, # ``minimal-risk`` — same default as the Pydantic field. - ), + "risk_classification": risk_classification, } @@ -386,13 +423,14 @@ def _collect_risk_assessment(risk_classification: str) -> Optional[Dict[str, Any "risk_assessment evidence is mandatory. All four fields below " "should be populated." ) - intended_use = ( - _prompt("Article 9(2)(a): intended_use", "") - if is_strict - else _prompt("intended_use (Article 9(2)(a)) — optional", "") - ) - foreseeable = _prompt_optional_list("Article 9(2)(b): foreseeable_misuse — list at least one realistic misuse") - mitigation = _prompt_optional_list("Article 9(2)(c): mitigation_measures the deployer applies") + if is_strict: + intended_use = _prompt_required("Article 9(2)(a): intended_use") + foreseeable = _prompt_required_list("Article 9(2)(b): foreseeable_misuse — list at least one realistic misuse") + mitigation = _prompt_required_list("Article 9(2)(c): mitigation_measures the deployer applies") + else: + intended_use = _prompt("intended_use (Article 9(2)(a)) — optional", "") + foreseeable = _prompt_optional_list("Article 9(2)(b): foreseeable_misuse — list at least one realistic misuse") + mitigation = _prompt_optional_list("Article 9(2)(c): mitigation_measures the deployer applies") vulnerable = _prompt_yes_no( "Article 9(2)(b): vulnerable_groups_considered (children / minorities / etc.)?", default=is_strict, @@ -414,11 +452,22 @@ def _collect_data_governance(*, mandatory: bool) -> Optional[Dict[str, Any]]: if not mandatory and not _prompt_yes_no("Configure Article 10 data.governance metadata?", default=False): return None if mandatory: - _print(" Risk classification is high-risk / unacceptable — Article 10 data.governance evidence is mandatory.") + _print( + " Risk classification is high-risk / unacceptable — Article 10 " + "data.governance evidence is mandatory. Free-text fields below " + "re-prompt until provided." + ) + collection_method = _prompt_required("Article 10(2)(b): how was data collected?") + annotation_process = _prompt_required("Article 10(2)(b): annotation methodology") + known_biases = _prompt_required("Article 10(2)(f): known_biases") + else: + collection_method = _prompt("Article 10(2)(b): how was data collected?", "") + annotation_process = _prompt("Article 10(2)(b): annotation methodology", "") + known_biases = _prompt("Article 10(2)(f): known_biases", "") return { - "collection_method": _prompt("Article 10(2)(b): how was data collected?", ""), - "annotation_process": _prompt("Article 10(2)(b): annotation methodology", ""), - "known_biases": _prompt("Article 10(2)(f): known_biases", ""), + "collection_method": collection_method, + "annotation_process": annotation_process, + "known_biases": known_biases, "personal_data_included": _prompt_yes_no("Article 10(5): personal_data_included?", default=False), "dpia_completed": _prompt_yes_no("Article 35 GDPR: dpia_completed?", default=False), } diff --git a/forgelm/wizard/_io.py b/forgelm/wizard/_io.py index e459bc67..c8f755b1 100644 --- a/forgelm/wizard/_io.py +++ b/forgelm/wizard/_io.py @@ -109,6 +109,8 @@ def _prompt_choice(question: str, options: List[str], default: int = 1) -> str: _check_navigation_token(choice) try: idx = int(choice) if choice else default + if idx < 1 or idx > len(options): + return options[default - 1] return options[idx - 1] except (ValueError, IndexError): return options[default - 1] @@ -189,7 +191,7 @@ def _detect_hardware() -> Dict[str, Any]: if torch.cuda.is_available(): info["gpu_available"] = True info["gpu_name"] = torch.cuda.get_device_name(0) - info["vram_gb"] = round(torch.cuda.get_device_properties(0).total_mem / (1024**3), 1) + info["vram_gb"] = round(torch.cuda.get_device_properties(0).total_memory / (1024**3), 1) info["cuda_version"] = torch.version.cuda except ImportError: pass diff --git a/forgelm/wizard/_orchestrator.py b/forgelm/wizard/_orchestrator.py index 7dcf76f7..dc4b03b1 100644 --- a/forgelm/wizard/_orchestrator.py +++ b/forgelm/wizard/_orchestrator.py @@ -9,7 +9,7 @@ from __future__ import annotations -import json +import copy from dataclasses import dataclass from typing import Any, Callable, Dict, List, Optional, Tuple @@ -197,13 +197,23 @@ def _step_model(state: _WizardState) -> None: options.append("Custom (enter your own)") if preset_default and preset_default in POPULAR_MODELS: default_idx = POPULAR_MODELS.index(preset_default) + 1 + elif preset_default: + # Preset from step 2 (use-case) isn't in POPULAR_MODELS — default + # to the custom slot and pre-fill the prompt with the operator's + # earlier choice so a bare ``Enter`` keeps it. + default_idx = len(options) else: default_idx = 1 from ._io import _prompt_choice chosen = _prompt_choice("Choose a model:", options, default=default_idx) if chosen.startswith("Custom"): - model_name = _prompt_required("HuggingFace model name or local path") + if preset_default and preset_default not in POPULAR_MODELS: + model_name = _prompt("HuggingFace model name or local path", preset_default) + if not model_name.strip(): + model_name = preset_default + else: + model_name = _prompt_required("HuggingFace model name or local path") else: model_name = chosen state.config.setdefault("model", {})["name_or_path"] = model_name @@ -494,13 +504,23 @@ def _persist_state(state: _WizardState) -> None: def _drive_wizard_steps(state: _WizardState) -> _WizardState: - """Run the wizard's step machine, honouring back / reset.""" + """Run the wizard's step machine, honouring back / reset. + + ``WizardBack`` rolls *state.config* back to the snapshot taken before + the step ran — partial mutations made by a half-completed step would + otherwise leak into the previous step's prompts. + + ``WizardReset`` re-loops with a fresh state instead of returning; + returning would let ``_run_full_wizard`` treat the reset as a + completed run and try to save an empty config. + """ while state.current_step < len(_STEPS): step = _STEPS[state.current_step] - prev_config = json.loads(json.dumps(state.config)) # deep copy + prev_config = copy.deepcopy(state.config) try: step.runner(state) except WizardBack: + state.config = prev_config if state.current_step == 0: _print(" Already at the first step.") continue @@ -512,7 +532,8 @@ def _drive_wizard_steps(state: _WizardState) -> _WizardState: except WizardReset: _print(" Resetting wizard state.") _clear_wizard_state() - return _WizardState() + state = _WizardState() + continue _print_step_diff(prev_config, state.config, step.label) if step.label not in state.completed_steps: state.completed_steps.append(step.label) diff --git a/forgelm/wizard/_state.py b/forgelm/wizard/_state.py index 9eefc275..d82cf1b2 100644 --- a/forgelm/wizard/_state.py +++ b/forgelm/wizard/_state.py @@ -104,6 +104,14 @@ def _save_wizard_state(state: Mapping[str, Any]) -> None: snapshot = {"v": _STATE_VERSION, **dict(state)} with open(target, "w", encoding="utf-8") as fh: yaml.safe_dump(snapshot, fh, default_flow_style=False, sort_keys=False) + # Wizard state can carry compliance metadata (provider name, + # contact, governance fields) that operators consider sensitive. + # ``0o600`` keeps it readable only by the operator running the + # wizard. Best-effort: chmod is a no-op on Windows / FAT. + try: + os.chmod(target, 0o600) + except OSError: # pragma: no cover — best-effort permission tightening + pass except OSError as exc: logger.warning("Could not persist wizard state to %s: %s", target, exc) @@ -209,10 +217,16 @@ def _print_step_diff(prev: Mapping[str, Any], curr: Mapping[str, Any], step_labe def _save_config_to_file(config: Dict[str, Any], requested_filename: str) -> str: - """Write *config* as YAML; falls back to a unique filename on OSError.""" + """Write *config* as YAML; falls back to a unique filename on OSError. + + Uses ``yaml.safe_dump`` so unknown Python objects (e.g. accidental + ``Path`` / ``set`` leak from a collector) raise a representable + error instead of silently emitting a Python-only ``!!python/object`` + tag that ``ForgeConfig`` then rejects on load. + """ try: - with open(requested_filename, "w") as f: - yaml.dump(config, f, default_flow_style=False, sort_keys=False) + with open(requested_filename, "w", encoding="utf-8") as f: + yaml.safe_dump(config, f, default_flow_style=False, sort_keys=False, allow_unicode=True) _print(f"\n Config saved to: {requested_filename}") logger.info("Wizard config saved to %s", requested_filename) return requested_filename @@ -228,8 +242,8 @@ def _save_config_to_file(config: Dict[str, Any], requested_filename: str) -> str f"{base}_{_dt.now().strftime('%Y%m%d_%H%M%S')}.yaml", ) try: - with open(fallback, "w") as f: - yaml.dump(config, f, default_flow_style=False, sort_keys=False) + with open(fallback, "w", encoding="utf-8") as f: + yaml.safe_dump(config, f, default_flow_style=False, sort_keys=False, allow_unicode=True) _print(f" Saved to fallback location: {fallback}") logger.info("Wizard config saved to fallback location %s", fallback) return fallback @@ -279,7 +293,7 @@ def _print_wizard_summary(config: Dict[str, Any]) -> None: _print(f" Extras: {', '.join(sorted(sections))}") _print("\n Full YAML preview:") _print(" " + "─" * 58) - yaml_text = yaml.dump(config, default_flow_style=False, sort_keys=False) + yaml_text = yaml.safe_dump(config, default_flow_style=False, sort_keys=False, allow_unicode=True) for line in yaml_text.splitlines(): _print(f" {line}") _print(" " + "─" * 58) diff --git a/tests/test_wizard_phase22.py b/tests/test_wizard_phase22.py index d9b79589..ce168c59 100644 --- a/tests/test_wizard_phase22.py +++ b/tests/test_wizard_phase22.py @@ -360,12 +360,15 @@ def test_decline_returns_none(self): def test_env_prefix_default_accepted(self): # The prompt default is ``env:FORGELM_WEBHOOK_URL`` so an empty - # answer accepts the suggested env-var. + # answer accepts the suggested env-var. ``notify_on_start`` + # defaults to ``False`` (web-wizard parity) — start + # notifications are noisy and most operators want only success + # / failure pings. with patch("builtins.input", side_effect=_input_returning("y", "")): section = wizard._collect_webhook_config() assert section == { "url_env": "FORGELM_WEBHOOK_URL", - "notify_on_start": True, + "notify_on_start": False, "notify_on_success": True, "notify_on_failure": True, } @@ -490,3 +493,238 @@ def test_default_lr_matches_schema(self): from forgelm.config import TrainingConfig assert wizard.DEFAULT_LR == TrainingConfig.model_fields["learning_rate"].default + + +# --------------------------------------------------------------------------- +# Orchestrator step-machine — Phase 22 / G3 + G7 + I3 +# +# The orchestrator was the lowest-coverage module after the Phase 22 +# split (14 % per the post-merge review). These tests exercise the +# step driver directly with stub steps so we don't need to mock 9 +# real ``_collect_*`` helpers. +# --------------------------------------------------------------------------- + + +class TestStepMachineDriver: + """Exercise ``_drive_wizard_steps`` directly with stub steps.""" + + def _make_steps(self, runners): + """Build a tuple of ``_StepDef``-shaped objects for *runners*.""" + return tuple( + wizard._orchestrator._StepDef(label=f"step-{i}", runner=runner) for i, runner in enumerate(runners) + ) + + def test_runs_every_step_in_order(self, isolated_state_dir, monkeypatch): + order = [] + + def make(label): + def runner(state): + order.append(label) + state.config[label] = True + + return runner + + steps = self._make_steps([make("a"), make("b"), make("c")]) + monkeypatch.setattr(wizard._orchestrator, "_STEPS", steps) + + state = wizard._orchestrator._drive_wizard_steps(wizard._WizardState()) + assert order == ["a", "b", "c"] + assert state.completed_steps == ["step-0", "step-1", "step-2"] + assert state.current_step == 3 + + def test_back_restores_prev_config(self, isolated_state_dir, monkeypatch): + # Step 1 mutates state.config; step 2 mutates it then raises + # WizardBack. Mutations from step 2 must NOT leak back into + # step 1 — that's the whole point of the prev_config snapshot. + attempts = {"step-1": 0} + + def step0(state): + state.config.setdefault("model", {})["name"] = "from-step-0" + + def step1_first(state): + # First entry: mutate then back out. + attempts["step-1"] += 1 + if attempts["step-1"] == 1: + state.config["leaked"] = "should-not-survive" + raise wizard.WizardBack + + def step0_after_back(state): + # Re-running step 0 — the leaked key from step 1's first + # attempt MUST be gone. + assert "leaked" not in state.config + + # Stitch: step 0 runs twice (first then after back); step 1 + # runs once (raises back), then once more silently. + runs = {"step-0": 0} + + def step0_combined(state): + runs["step-0"] += 1 + if runs["step-0"] == 1: + step0(state) + else: + step0_after_back(state) + + def step1_combined(state): + attempts["step-1"] += 1 + if attempts["step-1"] == 1: + state.config["leaked"] = "should-not-survive" + raise wizard.WizardBack + + steps = self._make_steps([step0_combined, step1_combined]) + monkeypatch.setattr(wizard._orchestrator, "_STEPS", steps) + + state = wizard._orchestrator._drive_wizard_steps(wizard._WizardState()) + assert "leaked" not in state.config + assert state.config["model"]["name"] == "from-step-0" + + def test_back_at_first_step_is_no_op(self, isolated_state_dir, monkeypatch, capsys): + attempts = {"calls": 0} + + def step0(state): + attempts["calls"] += 1 + if attempts["calls"] == 1: + raise wizard.WizardBack + # Second call: succeed normally. + state.config["done"] = True + + steps = self._make_steps([step0]) + monkeypatch.setattr(wizard._orchestrator, "_STEPS", steps) + + wizard._orchestrator._drive_wizard_steps(wizard._WizardState()) + captured = capsys.readouterr().out + assert "Already at the first step" in captured + + def test_reset_re_loops_with_fresh_state(self, isolated_state_dir, monkeypatch): + # WizardReset MUST cause the driver to start over with a fresh + # _WizardState — returning early would let _run_full_wizard + # treat the reset as a completed run and try to save an empty + # config. + attempts = {"step-0": 0, "step-1": 0} + + def step0(state): + attempts["step-0"] += 1 + state.config["k0"] = attempts["step-0"] + + def step1(state): + attempts["step-1"] += 1 + if attempts["step-1"] == 1: + raise wizard.WizardReset + + steps = self._make_steps([step0, step1]) + monkeypatch.setattr(wizard._orchestrator, "_STEPS", steps) + + state = wizard._orchestrator._drive_wizard_steps(wizard._WizardState()) + # step-0 ran twice (first run + after reset), step-1 ran twice + # too (first raised reset, second completed cleanly). + assert attempts["step-0"] == 2 + assert attempts["step-1"] == 2 + # State after the reset should reflect the second (fresh) run, + # not the first. + assert state.config["k0"] == 2 + + def test_persists_after_each_completed_step(self, isolated_state_dir, monkeypatch): + def step0(state): + state.config["a"] = 1 + + def step1(state): + # Snapshot must already include step 0's mutation when we + # arrive here — the orchestrator persists eagerly. + saved = wizard._load_wizard_state() + assert saved is not None + assert saved["config"]["a"] == 1 + state.config["b"] = 2 + + steps = self._make_steps([step0, step1]) + monkeypatch.setattr(wizard._orchestrator, "_STEPS", steps) + + wizard._orchestrator._drive_wizard_steps(wizard._WizardState()) + saved = wizard._load_wizard_state() + assert saved is not None + assert saved["config"] == {"a": 1, "b": 2} + assert saved["completed_steps"] == ["step-0", "step-1"] + + +class TestMaybeResumeState: + def test_returns_fresh_state_when_no_snapshot(self, isolated_state_dir): + state = wizard._orchestrator._maybe_resume_state() + assert state.current_step == 0 + assert state.config == {} + assert state.completed_steps == [] + + def test_resumes_from_saved_snapshot(self, isolated_state_dir): + wizard._save_wizard_state( + { + "experience": "beginner", + "use_case": "domain-expert", + "current_step": 2, + "completed_steps": ["welcome", "use-case"], + "config": {"model": {"name_or_path": "x"}}, + } + ) + with patch("builtins.input", side_effect=_input_returning("y")): # accept resume + state = wizard._orchestrator._maybe_resume_state() + assert state.current_step == 2 + assert state.completed_steps == ["welcome", "use-case"] + assert state.config == {"model": {"name_or_path": "x"}} + assert state.experience == "beginner" + + def test_decline_resume_clears_snapshot(self, isolated_state_dir): + wizard._save_wizard_state( + { + "experience": "expert", + "use_case": "custom", + "current_step": 1, + "completed_steps": ["welcome"], + "config": {"a": 1}, + } + ) + with patch("builtins.input", side_effect=_input_returning("n")): # decline + state = wizard._orchestrator._maybe_resume_state() + assert state.current_step == 0 + assert state.config == {} + # Snapshot was cleared so a future load returns None. + assert wizard._load_wizard_state() is None + + +class TestStepWelcome: + def test_welcome_sets_backend_hint(self, capsys): + # Stubbing _detect_hardware to return "no GPU" makes the + # outcome deterministic across CI runners that may or may not + # see CUDA. + state = wizard._WizardState() + with ( + patch("builtins.input", side_effect=_input_returning("n")), # not first-time + patch.object( + wizard._orchestrator, + "_detect_hardware", + return_value={ + "gpu_available": False, + "gpu_name": None, + "vram_gb": None, + "cuda_version": None, + }, + ), + ): + wizard._orchestrator._step_welcome(state) + assert state.experience == "expert" + assert state.config["model"]["backend"] == "transformers" + captured = capsys.readouterr().out + assert "No GPU detected" in captured + + def test_welcome_beginner_branch(self): + state = wizard._WizardState() + with ( + patch("builtins.input", side_effect=_input_returning("y")), # first-time + patch.object( + wizard._orchestrator, + "_detect_hardware", + return_value={ + "gpu_available": False, + "gpu_name": None, + "vram_gb": None, + "cuda_version": None, + }, + ), + ): + wizard._orchestrator._step_welcome(state) + assert state.experience == "beginner" From 7d08bbae191825f502047303678f320a4c0ef297 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sat, 9 May 2026 02:19:04 +0300 Subject: [PATCH 03/17] docs(changelog): Phase 22 wizard PR #40 review-cycle 1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CHANGELOG entry for the c0dd57a fix sweep — Critical / Major / Medium / Minor classifications + the orchestrator coverage bump that landed alongside. Co-Authored-By: Claude Opus 4.7 (1M context) --- CHANGELOG.md | 63 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7acd8101..4871624f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -89,6 +89,69 @@ All notable changes to ForgeLM are documented here. ### Fixed +- **Phase 22 wizard — PR #40 review-cycle 1 (2026-05-09).** + Closes the inline-comment + verified-finding sweep on + `feat/phase22-wizard-modernisation`: + - **Critical:** `torch.cuda.get_device_properties(0).total_mem` → + `total_memory` in `_io.py::_detect_hardware` (the previous + attribute does not exist; welcome step crashed on real CUDA + hosts). + - **Critical:** `WizardReset` in `_orchestrator.py::_drive_wizard_steps` + now re-loops with a fresh `_WizardState()` instead of returning; + returning made `_run_full_wizard` treat the reset as a completed + run and try to save an empty config. + - **Major:** `WizardBack` now restores `state.config` from a + `copy.deepcopy` snapshot taken before the step ran — partial + mutations made by a half-completed step would otherwise leak into + the previous step's prompts. + - **Major:** BYOD `_prompt_dataset_path_with_ingest_offer` now + validates the typed value as a directory of ingestible docs, a + JSONL/JSON file, or an HF Hub ID before accepting; bare typos + no longer silently become `data.dataset_name_or_path`. + - **Major:** Article 9 / Article 10 strict-tier collectors now + re-prompt until non-empty for the EU AI Act mandatory free-text + fields (`intended_use`, `foreseeable_misuse`, `mitigation_measures`, + `collection_method`, `annotation_process`, `known_biases`). + Previously empty values silently slipped through and surfaced as + Pydantic ConfigError at load time. + - **Major:** `webhook.notify_on_start` defaults to `False` in the + wizard to mirror `site/js/wizard.js` — start notifications are + noisy and most operators want only success / failure pings. + - **Major:** `_step_model` now preserves a use-case preset model + that isn't in `POPULAR_MODELS` (the operator's earlier choice + becomes the default for the custom slot). + - **Medium:** `_save_config_to_file` + `_print_wizard_summary` + use `yaml.safe_dump(allow_unicode=True)` instead of + `yaml.dump`, with explicit UTF-8 encoding on the file + handles. Prevents `!!python/object` tags from leaking into the + generated YAML when a collector accidentally returns a Path or + set, and stops mojibake on non-ASCII compliance fields. + - **Medium:** `wizard_state.yaml` is `chmod 0o600` after each write + — the snapshot can carry compliance metadata operators consider + sensitive. + - **Medium:** `_collect_compliance_metadata` collects + `risk_classification` first; downstream Article 9 / 10 collectors + branch on the chosen tier and the strict-tier hint surfaces up + front instead of after Annex IV §1. + - **Medium:** `_prompt_choice` guards against negative or + out-of-range numeric input by falling back to the default + instead of raising IndexError. + - **Minor:** BYOD shell hints (`forgelm ingest …`, `forgelm audit …`) + pass user-supplied paths through `shlex.quote` so paths with + spaces / metacharacters render copy-paste-safe. + - **Minor:** `_orchestrator._drive_wizard_steps` snapshots state + via `copy.deepcopy` instead of `json.loads(json.dumps(...))` + (the previous form silently lost tuples / sets). + - **Docs:** `docs/design/wizard_mode.md` ordered list switched + from two-space to single-space markers (MD030); `docs/reference/ + architecture{,-tr}.md` heading `wizard.py` → `wizard/` to match + the Phase 22 sub-package layout. + - **Tests:** `tests/test_wizard_phase22.py` gains + `TestStepMachineDriver`, `TestMaybeResumeState`, `TestStepWelcome` + classes (5 + 3 + 2 cases) exercising the orchestrator end-to-end + + back / reset / persistence / resume — `_orchestrator.py` + coverage 14 % → 30 %. + ### Removed ## [0.5.5] — 2026-05-06 From 9c32fe9bf99cb22cb9115020907e1346d3d11278 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sat, 9 May 2026 09:48:20 +0300 Subject: [PATCH 04/17] =?UTF-8?q?feat(wizard):=20review-cycle=202=20?= =?UTF-8?q?=E2=80=94=20operator=20guardrails=20+=20parity=20union?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 22 review-cycle 2: addresses 17 newly-discovered parity / UX gaps surfaced during the post-PR-#40 audit (P1-P18 + B1-B5 + C1-C2 + D1-D2 + E1+E4). All five HIGH-severity findings closed; both wizards (CLI sub-package + site/js/wizard.js) tightened in lockstep. ## Critical bug fixes - B-NEW-1: ``_print_step_diff`` strips internal ``_wizard_meta.*`` keys before showing the operator the per-step diff (was leaking ``_wizard_meta.use_galore`` from the strategy step). - B-NEW-2: ``_step_compliance`` skips the Article 10 governance re-prompt when step 6 already populated it under non-strict tier (was silently overwriting earlier answers). - B-NEW-3: ``_save_wizard_state`` now writes via ``tempfile`` + ``fsync`` + ``os.replace`` so a SIGKILL / power loss / concurrent wizard process can never leave a half-written state file. - B5: strict-tier coercion fires at the end of ``_step_compliance`` too (not just ``_step_evaluation``), so a Ctrl-C between steps 8 and 9 still produces a loadable YAML if the operator manually saves. - A1: ``_parse_webhook_value`` SSRF preflight — rejects loopback / RFC1918 / link-local destinations up front by reusing ``forgelm._http._is_private_destination``. ## UX guardrails - B1/E1: ``_validate_generated_config`` runs ``ForgeConfig.model_validate`` on the saved YAML before exit. Schema rejections surface inline. - B2: ``_prompt_unique_filename`` + ``_next_free_filename`` — overwrite confirmation with auto-suffix (no more silent clobbers). - B3: ``run_wizard_full`` refuses to launch on non-tty stdin and points operators at ``forgelm quickstart