From 4225b5b962438177cdf1b0071c35057f0307c7d8 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sun, 10 May 2026 16:12:26 +0300 Subject: [PATCH 1/7] docs(roadmap): consolidate 7 completed phase files into completed-phases.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tidy-up of the docs/roadmap/ directory: every completed phase file (Phase 10 / 10.5 / 11 / 11.5 / 12 / 12.5 / 12.6) is now absorbed into the canonical completed-phases.md archive with full content (heading shift +1 level), and the standalone files are removed. The roadmap directory now carries only the planning surface — completed-phases.md + releases.md + risks-and-decisions.md + the two PLANNED phase files (phase-13-pro-cli.md, phase-14-pipeline-chains.md). Changes: - docs/roadmap/completed-phases.md: 656 → 1285 lines. Replaces the earlier Phase 10 stub with the full phase-10-post-training.md content; appends Phase 10.5, 11, 11.5, 12, 12.5, 12.6 in chronological order, each as a top-level "## Phase N" section. - docs/roadmap/phase-10-post-training.md (deleted) - docs/roadmap/phase-10-5-quickstart.md (deleted) - docs/roadmap/phase-11-data-ingestion.md (deleted) - docs/roadmap/phase-11-5-backlog.md (deleted) - docs/roadmap/phase-12-data-curation-maturity.md (deleted) - docs/roadmap/phase-12-5-backlog.md (deleted) - docs/roadmap/phase-12-6-closure-cycle.md (deleted) References redirected to completed-phases.md (with internal anchors where useful): - CHANGELOG.md (3 references) - docs/roadmap.md + docs/roadmap-tr.md (table + tree visualisation) - docs/roadmap/phase-14-pipeline-chains.md - docs/roadmap/releases.md - docs/standards/documentation.md - docs/standards/release.md - tools/bench_simhash.py - tests/test_phase12_5.py - forgelm/ingestion.py - .claude/skills/add-trainer-feature/SKILL.md Verified: ruff format/check clean, all 6 doc CI guards green (anchor-resolution 266 files, bilingual-parity 45 pairs, cli-help 439 invocations, wizard-defaults 3/8, no-analysis-refs, dry-run). Co-Authored-By: Claude Opus 4.7 (1M context) --- .claude/skills/add-trainer-feature/SKILL.md | 2 +- CHANGELOG.md | 6 +- docs/roadmap-tr.md | 15 +- docs/roadmap.md | 17 +- docs/roadmap/completed-phases.md | 647 +++++++++++++++++- docs/roadmap/phase-10-5-quickstart.md | 54 -- docs/roadmap/phase-10-post-training.md | 71 -- docs/roadmap/phase-11-5-backlog.md | 92 --- docs/roadmap/phase-11-data-ingestion.md | 53 -- docs/roadmap/phase-12-5-backlog.md | 46 -- docs/roadmap/phase-12-6-closure-cycle.md | 149 ---- .../phase-12-data-curation-maturity.md | 160 ----- docs/roadmap/phase-14-pipeline-chains.md | 2 +- docs/roadmap/releases.md | 4 +- docs/standards/documentation.md | 2 +- docs/standards/release.md | 2 +- forgelm/ingestion.py | 4 +- tests/test_phase12_5.py | 2 +- tools/bench_simhash.py | 2 +- 19 files changed, 660 insertions(+), 670 deletions(-) delete mode 100644 docs/roadmap/phase-10-5-quickstart.md delete mode 100644 docs/roadmap/phase-10-post-training.md delete mode 100644 docs/roadmap/phase-11-5-backlog.md delete mode 100644 docs/roadmap/phase-11-data-ingestion.md delete mode 100644 docs/roadmap/phase-12-5-backlog.md delete mode 100644 docs/roadmap/phase-12-6-closure-cycle.md delete mode 100644 docs/roadmap/phase-12-data-curation-maturity.md diff --git a/.claude/skills/add-trainer-feature/SKILL.md b/.claude/skills/add-trainer-feature/SKILL.md index 0d101b11..2cf08bc2 100644 --- a/.claude/skills/add-trainer-feature/SKILL.md +++ b/.claude/skills/add-trainer-feature/SKILL.md @@ -24,7 +24,7 @@ Do **not** use for: 1. [docs/standards/architecture.md](../../../docs/standards/architecture.md) — module boundaries 2. [docs/standards/testing.md](../../../docs/standards/testing.md) — what tests CI demands -3. The **phase file** describing the feature (e.g., [docs/roadmap/phase-10-post-training.md](../../../docs/roadmap/phase-10-post-training.md)) +3. The **phase file** describing the feature (e.g., [docs/roadmap/completed-phases.md](../../../docs/roadmap/completed-phases.md)) 4. Existing similar feature — read one similar trainer/module end-to-end before writing yours ## The scaffold diff --git a/CHANGELOG.md b/CHANGELOG.md index 5f188866..a0973cb0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -782,7 +782,7 @@ review-cycle fixes applied across the four phases above). ### Phase 12.5 — Data Curation Polish (backlog items #1–#4) Four follow-up items from -[`docs/roadmap/phase-12-5-backlog.md`](docs/roadmap/phase-12-5-backlog.md) +[`docs/roadmap/completed-phases.md`](docs/roadmap/completed-phases.md) ship together — none require new architecture; each is a small additive surface on top of the Phase 12 ingestion + audit lineage. @@ -1141,7 +1141,7 @@ Round-1 review of the Phase 12 commit surfaced four critical regressions / bugs ### Added — Phase 12 (Data Curation Maturity, targeting v0.5.2) -Direct continuation of the Phase 11 / 11.5 ingestion + audit lineage. Closes the four concrete gaps surfaced by the post-`v0.5.1` competitive review (LLaMA-Factory / Axolotl / Unsloth / NeMo Curator / Dolma / RedPajama / LlamaIndex / LangChain / Marker / Docling). Tier 1 (5 must-have tasks) shipped; Tier 2/3 (Presidio adapter, Croissant metadata, `--all-mask`, wizard "audit first") deferred to a [Phase 12.5 backlog](docs/roadmap/phase-12-5-backlog.md). +Direct continuation of the Phase 11 / 11.5 ingestion + audit lineage. Closes the four concrete gaps surfaced by the post-`v0.5.1` competitive review (LLaMA-Factory / Axolotl / Unsloth / NeMo Curator / Dolma / RedPajama / LlamaIndex / LangChain / Marker / Docling). Tier 1 (5 must-have tasks) shipped; Tier 2/3 (Presidio adapter, Croissant metadata, `--all-mask`, wizard "audit first") deferred to a [Phase 12.5 backlog](docs/roadmap/completed-phases.md). - **MinHash LSH dedup option** (`forgelm/data_audit.py`: `compute_minhash`, `find_near_duplicates_minhash`, `_count_leaked_rows_minhash`) — Opt-in `--dedup-method minhash --jaccard-threshold 0.85` route via the optional `[ingestion-scale]` extra (`datasketch>=1.6.0`). Default simhash + LSH banding from Phase 11.5 stays untouched and remains the only method that runs without an optional dependency. `audit_dataset(...)` API gains `dedup_method`, `minhash_jaccard`, `minhash_num_perm` parameters; `near_duplicate_summary.method` records which path ran. Cross-split overlap + within-split duplicate scan share the same method flag. MinHash is approximate (permutation noise; default `num_perm=128`) — pin `num_perm` for cross-run determinism. - **Markdown-aware splitter** (`forgelm/ingestion.py`: `_chunk_markdown`, `_chunk_markdown_tokens`, `_markdown_sections`, `_heading_breadcrumb`) — New `--strategy markdown` parses heading hierarchy (`# H1` … `###### H6`), keeps code-fenced blocks atomic (heading-shaped lines inside ```` ``` ```` blocks are not interpreted as section boundaries), and inlines a heading **breadcrumb** at the top of each chunk so SFT loss sees the document context. Composes with the Phase 11.5 token-aware mode (`--chunk-tokens` + `--tokenizer`). @@ -1305,7 +1305,7 @@ Operational polish on top of `v0.5.0`'s ingestion + audit surface — no new tra ### Documentation - New "Option 0: One-Command Quickstart Template" section at the top of `docs/guides/quickstart.md`. -- `docs/roadmap.md`, `docs/roadmap-tr.md`, `docs/roadmap/phase-10-5-quickstart.md`, `docs/roadmap/releases.md` updated to mark Phase 10.5 as Done. +- `docs/roadmap.md`, `docs/roadmap-tr.md`, `docs/roadmap/completed-phases.md`, `docs/roadmap/releases.md` updated to mark Phase 10.5 as Done. - `README.md` quickstart section updated to lead with `forgelm quickstart`. --- diff --git a/docs/roadmap-tr.md b/docs/roadmap-tr.md index 4c53e3fa..79213060 100644 --- a/docs/roadmap-tr.md +++ b/docs/roadmap-tr.md @@ -7,10 +7,10 @@ | Tür | Faz | Durum | |-----|-----|-------| | ✅ Tamam | [Faz 1-9](roadmap/completed-phases.md) | SOTA iyileştirmeleri, değerlendirme, güvenilirlik, kurumsal entegrasyon, ekosistem, hizalama stack'i, güvenlik, EU AI Act uyumluluğu (Madde 9-17 + Ek IV), gelişmiş güvenlik zekası | -| ✅ Tamam | [Faz 10 — Post-Training Tamamlama](roadmap/phase-10-post-training.md) | `inference.py`, `chat`, `export` (GGUF), `--fit-check`, `deploy` — `v0.4.0` | -| ✅ Tamam | [Faz 10.5 — Quickstart Katmanı ve Onboarding](roadmap/phase-10-5-quickstart.md) | `forgelm quickstart