Development

CHANGELOG.md

@@ -33,11 +33,11 @@ All notable changes to ForgeLM are documented here.
   - No bifurcation: identical code paths and YAML schema downstream.
 
 - **5 bundled templates** under `forgelm/templates/`:
-  - `customer-support/` — Qwen2.5-7B-Instruct primary, SmolLM2-1.7B-Instruct fallback. SFT trainer. 60-example seed JSONL in `{"messages": [...]}` format.
-  - `code-assistant/` — Qwen2.5-Coder-7B-Instruct primary. SFT. 60-example Python/programming Q&A.
-  - `domain-expert/` — BYOD; empty data with a README explaining how to pair with `forgelm ingest` (Phase 11) or a custom JSONL.
-  - `medical-qa-tr/` — Turkish medical Q&A SFT. 40+ examples; every answer ends with "Tıbbi acil durumlarda 112'yi arayın..." (medical-disclaimer guardrail).
-  - `grpo-math/` — Qwen2.5-Math-7B-Instruct primary, Qwen2.5-Math-1.5B-Instruct fallback. GRPO trainer (`grpo_num_generations: 4`). 40 grade-school math word problems in prompt-only format.
+  - `customer-support/` — Qwen2.5-7B-Instruct primary, SmolLM2-1.7B-Instruct fallback. SFT trainer. 58-example seed JSONL in `{"messages": [...]}` format.
+  - `code-assistant/` — Qwen2.5-Coder-7B-Instruct primary, Qwen2.5-Coder-1.5B-Instruct fallback (code-tuned smaller variant, not generic SmolLM2). SFT. 59-example Python/programming Q&A.
+  - `domain-expert/` — Qwen2.5-7B-Instruct primary, SmolLM2-1.7B-Instruct fallback. BYOD; empty data with a README explaining how to pair with `forgelm ingest` (Phase 11) or a custom JSONL.
+  - `medical-qa-tr/` — Qwen2.5-7B-Instruct primary, Qwen2.5-1.5B-Instruct fallback (Turkish-capable, not English-only SmolLM2). SFT, 49 Turkish Q&A; every answer ends with "Tıbbi acil durumlarda 112'yi arayın..." (medical-disclaimer guardrail).
+  - `grpo-math/` — Qwen2.5-Math-7B-Instruct primary, Qwen2.5-Math-1.5B-Instruct fallback. GRPO trainer (`grpo_num_generations: 4`). 40 grade-school math word problems in prompt-only format, each carrying a `gold_answer` field for the built-in regex correctness reward.
 
 - **Conservative defaults** in every template config:
   - QLoRA 4-bit NF4, LoRA rank=8, `per_device_train_batch_size=1`, gradient checkpointing on, safety eval / compliance artifacts opt-in only.
@@ -47,16 +47,21 @@ All notable changes to ForgeLM are documented here.
 
 - **`pyproject.toml` `[tool.setuptools.package-data]`** — bundles `*.yaml`, `*.jsonl`, `*.md` under `forgelm.templates` into the wheel so `pip install forgelm` users get the templates without a source checkout.
 
-- **Tests** — `tests/test_quickstart.py` (40 tests, GPU-independent via `_detect_available_vram_gb` mock):
-  - Registry consistency, get-template lookups, list ordering.
-  - Every template's YAML config is parseable, has `model`/`training`/`data`/`lora` sections, and uses the placeholder dataset string the quickstart layer overwrites.
-  - Every bundled JSONL parses as JSON line by line.
-  - `auto_select_model` primary/fallback/no-gpu paths.
-  - End-to-end `run_quickstart` flow: substitution, overrides win over auto-select, dry-run hint in summary, `domain-expert` rejects without `--dataset`.
-  - CLI dispatch (`--list` text + JSON, `--dry-run` writes YAML and exits 0, missing template → CONFIG_ERROR).
-  - **Regression**: every generated YAML round-trips through `load_config()` — the strongest guard against template drift the moment a config schema changes.
-
-- **CI** — `.github/workflows/nightly.yml` adds a per-template smoke test that runs `forgelm quickstart <name> --dry-run` followed by `forgelm --config <out> --dry-run` for every bundled template.
+- **GRPO baseline reward** — `forgelm/grpo_rewards.py` ships a default reward bundle so prompt-only datasets don't crash inside `trl.GRPOTrainer`. When `grpo_reward_model` is unset the trainer wires `combined_format_length_reward` (0.8 × format-match + 0.2 × length-shaping); if the dataset additionally carries a `gold_answer` field (the bundled `grpo-math` seed does), `_math_reward_fn` is appended so TRL sums correctness on top of format teaching.
+
+- **Tests** — All GPU-independent via TRL/torch FSDP-aware skip-if pattern:
+  - `tests/test_quickstart.py` — registry consistency, bundled-asset shape, `auto_select_model` primary/fallback/no-gpu, end-to-end `run_quickstart`, CLI dispatch, regression test that loads every generated YAML through `load_config` (strongest guard against template drift).
+  - `tests/test_quickstart_hardening.py` — PR review hardening (path validation, model override edges, dry-run wiring).
+  - `tests/test_grpo_math_reward.py` — pure-Python unit tests for `_normalize_answer`, `_answers_match`, `_math_reward_fn`, `_dataset_has_gold_answers`.
+  - `tests/test_grpo_format_reward.py` — `format_match_reward`, `length_shaping_reward`, `combined_format_length_reward`, plus trainer integration.
+  - `tests/test_wizard_byod.py` — wizard BYOD dataset path validation (existence, directory, malformed JSONL, valid JSONL, HF Hub IDs, `~` expansion).
+  - `tests/test_cli_quickstart_wiring.py` — `--offline` propagation, separate chat inheritance, chat exit-code 0/130 handling.
+  - `tests/test_packaging.py` — wheel `package_data` smoke (catches editable-install-only template paths).
+  - `tests/test_grpo_reward.py` — extended with no-reward-model + gold-answer wiring assertions.
+
+- **CI** — `.github/workflows/nightly.yml`:
+  - Per-template quickstart smoke (4 of 5 — `domain-expert` is BYOD and covered by pytest).
+  - New `wheel-install-smoke` job: builds the wheel, installs it into a fresh venv from `/tmp` (so the source tree is off `sys.path`), and reruns `quickstart --list` + `quickstart --dry-run` to catch broken `package_data` globs that editable installs hide.
 
 ### Documentation
 

docs/guides/quickstart.md

@@ -47,7 +47,17 @@ Bundled templates (all use QLoRA 4-bit, rank-8, batch=1 by default — safe to r
 | `medical-qa-tr` | SFT | Turkish medical Q&A with safety disclaimers |
 | `grpo-math` | GRPO | Grade-school math reasoning |
 
-ForgeLM auto-downsizes the model on small GPUs (e.g. `Qwen2.5-7B-Instruct` → `SmolLM2-1.7B-Instruct` when < 10 GB VRAM is available). Override with `--model your-org/your-model` or `--dataset path/to/your.jsonl`.
+ForgeLM auto-downsizes the model on small GPUs. Each template has its own fallback chosen for the task:
+
+| Template | Primary (≥10 GB VRAM) | Fallback (<10 GB) |
+|---|---|---|
+| `customer-support` | Qwen/Qwen2.5-7B-Instruct | HuggingFaceTB/SmolLM2-1.7B-Instruct |
+| `code-assistant` | Qwen/Qwen2.5-Coder-7B-Instruct | Qwen/Qwen2.5-Coder-1.5B-Instruct |
+| `domain-expert` | Qwen/Qwen2.5-7B-Instruct | HuggingFaceTB/SmolLM2-1.7B-Instruct |
+| `medical-qa-tr` | Qwen/Qwen2.5-7B-Instruct | Qwen/Qwen2.5-1.5B-Instruct |
+| `grpo-math` | Qwen/Qwen2.5-Math-7B-Instruct | Qwen/Qwen2.5-Math-1.5B-Instruct |
+
+Override with `--model your-org/your-model` or `--dataset path/to/your.jsonl`.
 
 See [LICENSES.md](https://github.com/cemililik/ForgeLM/blob/main/forgelm/templates/LICENSES.md) for the licenses of bundled seed datasets (CC-BY-SA 4.0, author-original).
 

docs/roadmap-tr.md

@@ -15,7 +15,7 @@
 
 **PyPI'deki son sürüm:** `v0.4.0` — Post-Training Tamamlama (2026-04-26 yayınlandı). Inference primitif'leri, etkileşimli chat REPL, GGUF export, VRAM fit advisor, deployment config üretimi.
 
-**Git'te etiketli ama PyPI'de yok:** `v0.4.5` — Quickstart Katmanı (tek komutla hazır template'ler: `forgelm quickstart customer-support`). Küçük GPU'larda modeli otomatik küçültür, mevcut trainer'ın değişmeden kabul ettiği bir YAML üretir.
+**Main'e merge edildi, tag + PyPI publish bekliyor:** `v0.4.5` — Quickstart Katmanı (tek komutla hazır template'ler: `forgelm quickstart customer-support`). Küçük GPU'larda modeli otomatik küçültür, mevcut trainer'ın değişmeden kabul ettiği bir YAML üretir. PR #9 ile (2026-04-26) main'e indi; `v0.4.5` git tag ve PyPI yükleme kalan release-engineering adımları.
 
 **Güncel kilometretaşı:** `v0.5.0` — Doküman Yutma ve Veri Denetimi (Faz 11). PDF/DOCX/EPUB → JSONL, PII tespiti, yakın-duplicate denetimi.
 
@@ -61,7 +61,7 @@ docs/
     ├── completed-phases.md                     # Faz 1-10 arşivi (detaylı, İngilizce)
     ├── phase-10-post-training.md               # Tamamlandı — v0.4.0
     ├── phase-11-data-ingestion.md              # Planlandı — v0.5.0
-    ├── phase-12-quickstart.md                  # Planlandı — Faz 10.5 — v0.4.5 (etiketli, PyPI'de yok)
+    ├── phase-12-quickstart.md                  # Tamam (Faz 10.5) — main'e v0.4.5 olarak merge edildi; tag + PyPI publish bekliyor
     ├── phase-13-pro-cli.md                     # Planlandı — v0.6.0-pro (gated)
     ├── phase-14-pipeline-chains.md             # Planlandı — v0.5.1
     ├── releases.md                             # v0.3.0 → v0.6.0 sürüm notları

docs/roadmap.md

@@ -15,7 +15,7 @@
 
 **Latest release on PyPI:** `v0.4.0` — Post-Training Completion (published 2026-04-26). Inference primitives, interactive chat REPL, GGUF export, VRAM fit advisor, and deployment config generation.
 
-**Tagged but not yet on PyPI:** `v0.4.5` — Quickstart Layer (one-command bundled templates: `forgelm quickstart customer-support`). Auto-downsizes models on small GPUs, generates a config that the existing trainer accepts unchanged.
+**Merged to main, tag + PyPI publish pending:** `v0.4.5` — Quickstart Layer (one-command bundled templates: `forgelm quickstart customer-support`). Auto-downsizes models on small GPUs, generates a config that the existing trainer accepts unchanged. Landed via PR #9 (2026-04-26); the `v0.4.5` git tag and PyPI upload are the remaining release-engineering steps.
 
 **Current milestone:** `v0.5.0` — Document Ingestion & Data Audit (Phase 11). PDF/DOCX/EPUB → JSONL, PII detection, near-duplicate audit.
 
@@ -61,7 +61,7 @@ docs/
     ├── completed-phases.md                     # Phase 1-10 archive (detailed)
     ├── phase-10-post-training.md               # Completed — v0.4.0
     ├── phase-11-data-ingestion.md              # Planned — v0.5.0
-    ├── phase-12-quickstart.md                  # Done (Phase 10.5) — tagged v0.4.5, not yet on PyPI
+    ├── phase-12-quickstart.md                  # Done (Phase 10.5) — merged to main as v0.4.5; tag + PyPI publish pending
     ├── phase-13-pro-cli.md                     # Planned — v0.6.0-pro (gated)
     ├── phase-14-pipeline-chains.md             # Planned — v0.5.1
     ├── releases.md                             # v0.3.0 → v0.6.0 release notes

docs/roadmap/phase-12-quickstart.md

@@ -1,6 +1,6 @@
 # Phase 10.5: Quickstart Layer & Onboarding
 
-> **Status:** ✅ **DONE** — shipped in `v0.4.5` (2026-04-26). Module: [`forgelm/quickstart.py`](../../forgelm/quickstart.py); five bundled templates under [`forgelm/templates/`](../../forgelm/templates/); CLI: `forgelm quickstart <template>`; tests: [`tests/test_quickstart.py`](../../tests/test_quickstart.py); CI smoke in [nightly.yml](../../.github/workflows/nightly.yml).
+> **Status:** ✅ **DONE** — merged to `main` as `v0.4.5` via PR #9 (2026-04-26); the `v0.4.5` git tag and PyPI upload are the remaining release-engineering steps. Module: [`forgelm/quickstart.py`](../../forgelm/quickstart.py); five bundled templates under [`forgelm/templates/`](../../forgelm/templates/); CLI: `forgelm quickstart <template>`; tests: [`tests/test_quickstart.py`](../../tests/test_quickstart.py); CI smoke in [nightly.yml](../../.github/workflows/nightly.yml).
 
 > **Note:** This file details a single phase. For a summary of all phases, see [../roadmap.md](../roadmap.md).
 > **Filename:** `phase-12-quickstart.md` — name retained from the original Phase 12 ordering; content corresponds to Phase 10.5.

docs/roadmap/releases.md

@@ -60,22 +60,23 @@ Odak: [Phase 10](phase-10-post-training.md). Full post-training handoff: inferen
 
 ## v0.4.5 — "Quickstart Layer" (2026-04-26)
 
-**Status:** Released. Focus: [Phase 10.5](phase-12-quickstart.md) (Quickstart). One-command bundled templates, sample datasets, opinionated defaults — primary community growth driver.
+**Status:** Merged to `main` via PR #9 (2026-04-26). The `v0.4.5` git tag and PyPI upload are the remaining release-engineering steps. Focus: [Phase 10.5](phase-12-quickstart.md) (Quickstart). One-command bundled templates, sample datasets, opinionated defaults — primary community growth driver.
 
 ### Features:
 1. [x] **`forgelm/quickstart.py`** — Template registry (`@dataclass(frozen=True) Template`), `auto_select_model()` GPU-aware downsizing (≥10 GB VRAM → primary model; otherwise fallback ≤2B), `run_quickstart()` end-to-end orchestrator that copies the bundled seed dataset, substitutes `model.name_or_path` and `data.dataset_name_or_path`, and writes a `configs/<template>-YYYYMMDDHHMMSS.yaml` the existing trainer accepts unchanged.
-2. [x] **`forgelm quickstart <template>` CLI subcommand** — `--list` (text + JSON via `--output-format json`), `--model` / `--dataset` overrides, `--dry-run` (generate config but skip training), `--no-chat` (skip post-training chat REPL), `--output` (custom YAML path). On a successful train, subprocess-invokes `forgelm chat <output_dir>` for an immediate sanity loop.
+2. [x] **`forgelm quickstart <template>` CLI subcommand** — `--list` (text + JSON via `--output-format json`), `--model` / `--dataset` overrides, `--dry-run` (generate config but skip training), `--no-chat` (skip post-training chat REPL), `--output` (custom YAML path). On a successful train, subprocess-invokes `forgelm chat <output_dir>` for an immediate sanity loop. Top-level flags (`--output-format`, `--quiet`, `--log-level`, `--offline`) propagate to the train + chat subprocesses.
 3. [x] **5 bundled templates** under [`forgelm/templates/`](../../forgelm/templates/):
-   - `customer-support` (Qwen2.5-7B-Instruct ↔ SmolLM2-1.7B-Instruct, SFT, 60-example seed JSONL)
-   - `code-assistant` (Qwen2.5-Coder-7B-Instruct ↔ SmolLM2-1.7B-Instruct, SFT, 60-example seed)
-   - `domain-expert` (BYOD — empty data, README walks through the workflow)
-   - `medical-qa-tr` (Qwen2.5-7B-Instruct, SFT, 40+ Turkish Q&A with safety disclaimers)
-   - `grpo-math` (Qwen2.5-Math-7B-Instruct ↔ Qwen2.5-Math-1.5B-Instruct, GRPO trainer, 40 grade-school math prompts)
+   - `customer-support` (Qwen2.5-7B-Instruct ↔ SmolLM2-1.7B-Instruct, SFT, 58-example seed JSONL)
+   - `code-assistant` (Qwen2.5-Coder-7B-Instruct ↔ Qwen2.5-Coder-1.5B-Instruct, SFT, 59-example seed)
+   - `domain-expert` (Qwen2.5-7B-Instruct ↔ SmolLM2-1.7B-Instruct, BYOD — empty data, README walks through the workflow)
+   - `medical-qa-tr` (Qwen2.5-7B-Instruct ↔ Qwen2.5-1.5B-Instruct, SFT, 49 Turkish Q&A with safety disclaimers)
+   - `grpo-math` (Qwen2.5-Math-7B-Instruct ↔ Qwen2.5-Math-1.5B-Instruct, GRPO trainer, 40 grade-school math prompts each carrying a `gold_answer` for the built-in regex correctness reward)
 4. [x] **Conservative defaults** — every template ships with QLoRA 4-bit NF4, rank=8, batch=1 + gradient accumulation, gradient checkpointing on, safety eval / compliance artifacts opt-in only.
-5. [x] **Wizard integration** — `forgelm --wizard` now opens with "Start from a template?". Yes → invokes the quickstart flow; No → falls through to the existing 8-step interactive flow. No bifurcation: same code paths, same YAML schema.
-6. [x] **License hygiene** — [`forgelm/templates/LICENSES.md`](../../forgelm/templates/LICENSES.md) catalogs all bundled seed datasets (CC-BY-SA 4.0, author-original); contributing guide for new templates.
-7. [x] **Tests + CI** — 40-test [`tests/test_quickstart.py`](../../tests/test_quickstart.py) covering registry, bundled assets, auto-select, generation, CLI dispatch, and a regression test that loads every generated YAML through `load_config` (the strongest guard against template drift). Nightly CI smoke-tests every template via `quickstart --dry-run` + `--config <out> --dry-run`.
-8. [x] **`pyproject.toml` `[tool.setuptools.package-data]`** — bundles `*.yaml`, `*.jsonl`, `*.md` under `forgelm.templates` into the wheel so `pip install forgelm` users get the templates too.
+5. [x] **GRPO baseline reward** — when `grpo_reward_model` is unset, `forgelm/grpo_rewards.combined_format_length_reward` (format-match × 0.8 + length-shaping × 0.2) is wired by default so prompt-only datasets don't crash inside `trl.GRPOTrainer`. If the dataset additionally carries a `gold_answer` field (the bundled grpo-math seed does), `_math_reward_fn` is appended for an additive correctness signal.
+6. [x] **Wizard integration** — `forgelm --wizard` now opens with "Start from a template?". Yes → invokes the quickstart flow (BYOD path validates the supplied dataset path before continuing); No → falls through to the existing 8-step interactive flow. No bifurcation: same code paths, same YAML schema.
+7. [x] **License hygiene** — [`forgelm/templates/LICENSES.md`](../../forgelm/templates/LICENSES.md) catalogs all bundled seed datasets (CC-BY-SA 4.0, author-original); contributing guide for new templates.
+8. [x] **Tests + CI** — `tests/test_quickstart.py`, `tests/test_quickstart_hardening.py`, `tests/test_grpo_math_reward.py`, `tests/test_grpo_format_reward.py`, `tests/test_wizard_byod.py`, `tests/test_cli_quickstart_wiring.py`, `tests/test_packaging.py`. Includes a regression test that loads every generated YAML through `load_config` (the strongest guard against template drift). Nightly CI smoke-tests every template via `quickstart --dry-run` + `--config <out> --dry-run`, plus a dedicated `wheel-install-smoke` job that builds the wheel and reruns quickstart from a fresh venv to catch broken `package_data` globs.
+9. [x] **`pyproject.toml` `[tool.setuptools.package-data]`** — bundles `*.yaml`, `*.jsonl`, `*.md` under `forgelm.templates` into the wheel so `pip install forgelm` users get the templates too.
 
 ---