diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 194940dc..02f0d611 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -5,13 +5,14 @@ on: - cron: "0 3 * * *" # 03:00 UTC daily workflow_dispatch: # Manual trigger -permissions: - issues: write # Required for creating failure notification issues +# No workflow-level permissions — each job declares the minimal set it needs. jobs: test-latest-deps: name: Test with latest dependencies runs-on: ubuntu-latest + permissions: + contents: read strategy: fail-fast: false matrix: @@ -56,6 +57,8 @@ jobs: test-min-deps: name: Test with minimum supported versions runs-on: ubuntu-latest + permissions: + contents: read steps: - uses: actions/checkout@v5 @@ -87,6 +90,9 @@ jobs: needs: [test-latest-deps, test-min-deps] if: failure() runs-on: ubuntu-latest + permissions: + contents: read # github-script needs to read the repo + issues: write # required for creating failure notification issues steps: - name: Create issue on failure uses: actions/github-script@v8 diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 4b4597eb..6fa60dd9 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -4,12 +4,13 @@ on: release: types: [published] -permissions: - id-token: write # Required for trusted publishing +# No workflow-level permissions — each job declares the minimal set it needs. jobs: build: runs-on: ubuntu-latest + permissions: + contents: read # checkout needs read access to the release ref steps: - uses: actions/checkout@v5 @@ -42,6 +43,9 @@ jobs: environment: name: testpypi url: https://test.pypi.org/p/forgelm + permissions: + contents: read # download-artifact reads workflow run state + id-token: write # required for trusted publishing to TestPyPI steps: - uses: actions/download-artifact@v5 with: @@ -61,6 +65,9 @@ jobs: environment: name: pypi url: https://pypi.org/p/forgelm + permissions: + contents: read # download-artifact reads workflow run state + id-token: write # required for trusted publishing to PyPI steps: - uses: actions/download-artifact@v5 with: diff --git a/CHANGELOG.md b/CHANGELOG.md index feac4aaf..b28a6ed6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,103 @@ All notable changes to ForgeLM are documented here. ## [Unreleased] +## [0.4.0] — 2026-04-25 + +### Added + +**Post-Training Completion (Phase 10)** + +- **`forgelm/inference.py`** — Shared generation primitives for all post-training features: + - `load_model(path, adapter, backend, load_in_4bit, load_in_8bit, trust_remote_code)` — loads HF model + tokenizer; optional PEFT adapter merge via `merge_and_unload()`; unsloth backend support + - `generate(model, tokenizer, prompt, *, messages, system_prompt, history, max_new_tokens, temperature, top_k, top_p, repetition_penalty)` — non-streaming text generation + - `generate_stream(...)` — streaming via `TextIteratorStreamer` in daemon thread; yields token chunks + - `logit_stats(logits)` — returns `{entropy, top1_prob, effective_vocab}` for token-level confidence inspection + - `adaptive_sample(logits, temperature, top_k, top_p, entropy_threshold)` — greedy below entropy threshold, nucleus sampling above + - `_build_prompt` — uses `tokenizer.apply_chat_template` when available; falls back to `"role: content\n"` join + +- **`forgelm/chat.py`** — Interactive terminal REPL (`ChatSession` class + `run_chat()` entry point): + - Streaming output by default; `--no-stream` flag for non-streaming + - Slash commands: `/reset`, `/save [file]`, `/temperature N`, `/system [prompt]`, `/help`, `/exit` + - History management with 50-turn cap (`_MAX_HISTORY_PAIRS`) + - Optional `rich` rendering via `pip install forgelm[chat]` + - Optional `--safety` flag routes each response through Llama Guard + +- **`forgelm/fit_check.py`** — VRAM pre-flight advisor: + - `estimate_vram(config)` → `FitCheckResult(verdict, estimated_gb, available_gb, breakdown, recommendations)` + - Verdicts: `FITS` (< 85% GPU), `TIGHT` (85-95%), `OOM` (> 95%), `UNKNOWN` (no GPU) + - Architecture loaded via `transformers.AutoConfig`; fallback size-hint dict for 7b/8b/13b/70b families + - VRAM components: base weights + LoRA adapter + optimizer state (AdamW/8-bit/GaLore-aware) + activations (gradient-checkpointing divides by √layers) + - `format_fit_check(result)` — human-readable summary; `--output-format json` for CI/CD + - Hypothetical mode when no CUDA detected — still estimates based on architecture + +- **`forgelm/export.py`** — GGUF model export: + - `export_model(model_path, output_path, *, format, quant, adapter, update_integrity, extra_args)` → `ExportResult` + - Wraps `llama-cpp-python`'s `convert_hf_to_gguf.py` — no reimplementation of conversion logic + - Supported quantizations: `q2_k`, `q3_k_m`, `q4_k_m`, `q5_k_m`, `q8_0`, `f16` + - **K-quant note**: `q2_k`/`q3_k_m`/`q4_k_m`/`q5_k_m` require a two-step flow. + `forgelm export ... --quant q4_k_m model.gguf` produces an intermediate + `model.f16.gguf`; run `llama-quantize model.f16.gguf model.gguf Q4_K_M` + afterward to obtain the K-quant. The `ExportResult.quant` field reflects + what was actually written (so `model_integrity.json` SHA-256 stays honest) + - Adapter merge: loads base + PEFT, saves merged fp16 weights before conversion + - `_sha256_file` — chunked 64 KB reads for large models + - `_update_integrity_manifest` — appends export artifact (path, quant, sha256, size_bytes) to `model_integrity.json` + - Optional dependency: `pip install forgelm[export]` (`llama-cpp-python>=0.2.90`) + +- **`forgelm/deploy.py`** — Deployment config file generation: + - `generate_deploy_config(model_path, target, output_path, *, system_prompt, max_length, temperature, top_k, top_p, ...)` → `DeployResult` + - Target `ollama`: Modelfile with FROM, SYSTEM (double-quote escaped), PARAMETER directives + - Target `vllm`: YAML engine config with GPU memory utilization, dtype, trust_remote_code + - Target `tgi`: docker-compose.yaml with GPU resource reservation, port mapping, max-input/total-length + - Target `hf-endpoints`: JSON spec with model repository, task, compute instance, region, framework + - Case-insensitive target matching; default output filenames per target + +- **CLI subcommands** (`forgelm/cli.py`): + - `forgelm chat MODEL_PATH [--adapter] [--system] [--temperature] [--max-new-tokens] [--safety] [--no-stream] [--load-in-4bit] [--load-in-8bit] [--trust-remote-code] [--backend]` + - `forgelm export MODEL_PATH --output FILE [--format gguf] [--quant q4_k_m] [--adapter] [--no-integrity-update]` + - `forgelm deploy MODEL_PATH --target TARGET [--output FILE] [--system] [--max-length] [--temperature] [--top-k] [--top-p] [--trust-remote-code]` + - `forgelm --config CONFIG --fit-check [--output-format json]` + - All subcommands work without `--config`; backward-compatible with existing flat CLI + +- **Optional extras** in `pyproject.toml`: + - `forgelm[export]` — `llama-cpp-python>=0.2.90` (non-Windows) + - `forgelm[chat]` — `rich>=13.0.0` + +- **New test modules**: + - `tests/test_inference.py` — 16 tests covering `_build_prompt`, `_to_messages`, `logit_stats`, `adaptive_sample`, `load_model`, `generate` with custom torch stub (no GPU required) + - `tests/test_fit_check.py` — 18 tests covering parameter estimation, VRAM components, GPU scenarios (no CUDA, 4 GB, 80 GB), `format_fit_check` + - `tests/test_export.py` — 12 tests covering SHA-256, integrity manifest, GGUF export flow with subprocess mock + - `tests/test_deploy.py` — 21 tests covering all 4 target generators and `generate_deploy_config` integration + - `tests/test_cli_phase10.py` — 22 tests covering `--fit-check`, all deploy targets, export subcommand, chat subcommand, subcommand routing + +### Changed + +- **`forgelm/__init__.py`** — version bumped to `0.4.0` +- **`forgelm/cli.py`** — added subparser architecture with `chat`, `export`, `deploy` subcommands; added `--fit-check` flag; `KeyboardInterrupt` caught in chat dispatch for graceful exit +- **`forgelm/wizard.py`** — (no changes needed; Phase 10 features are all CLI-driven, not wizard-driven) + +### Breaking + +- **`forgelm.compliance.export_compliance_artifacts`** signature changed from + `(manifest, config, output_dir)` to `(manifest, output_dir)`. The `config` + argument was unused (the manifest already contains all derived values). + External callers must drop the second positional argument. +- **`forgelm.export.export_model`** keyword `format=` renamed to + `output_format=` to avoid shadowing the `format` builtin. Update + `export_model(..., format="gguf", ...)` → `export_model(..., + output_format="gguf", ...)`. +- **`forgelm.deploy.generate_deploy_config`** parameter list collapsed from + 18 → 11 args. The HF Endpoints fields (task/instance_size/instance_type/ + region/framework/vendor) are now grouped as + `hf_endpoints: HFEndpointsOptions = None`; sampling defaults + (temperature/top_k/top_p) are grouped as + `sampling: SamplingOptions = None`. Pass instances of those dataclasses + instead of the individual kwargs. + +--- + +## [0.3.1rc1] — 2026-03-28 (included in v0.4.0 branch) + ### Added - **Engineering standards** (`docs/standards/`) — 9 standard documents: coding, architecture, error-handling, logging-observability, testing, documentation, localization, code-review, release. - **AI agent skills** (`.claude/skills/`) — 6 task-specific SKILL.md checklists: add-config-field, add-trainer-feature, add-test, sync-bilingual-docs, cut-release, review-pr. @@ -14,6 +111,18 @@ All notable changes to ForgeLM are documented here. - **docs/ reorganization** — Reference docs moved to `docs/reference/`, design specs to `docs/design/`. All internal links updated (29 link fixes). - **Roadmap refactored** — `docs/roadmap.md` reduced from 910 to 78 lines; phase details moved to `docs/roadmap/` subdirectory. +### Fixed (Security & Config Hardening) +- Webhook URLs excluded from HuggingFace Hub model cards — prevents credential leaks +- User-supplied strings sanitized before Markdown template embedding (content injection prevention) +- All 19 Pydantic sub-models enforce `extra="forbid"` — YAML typos are errors, not silent bugs +- Deprecated `lora.use_dora` / `lora.use_rslora` booleans auto-normalize to `lora.method` with warnings +- Audit log hash chain restores continuity across process restarts +- Compliance manifests correctly report pre-OOM-recovery batch size +- GRPO reward model path correctly wrapped as callable +- Safety classifier receives full `[INST] prompt [/INST] response` context +- Extension-less files raise clear `ValueError` instead of silently loading wrong format +- TIES tie-breaking fixed; DARE now deterministic with `seed=42` + ## [0.3.0] — 2026-03-28 ### Added diff --git a/README.md b/README.md index 903a0e54..adf7e6db 100644 --- a/README.md +++ b/README.md @@ -26,6 +26,12 @@ - **LLM-as-Judge**: API-based (OpenAI) or local model scoring for quality assessment - **Auto-Revert**: Automatically discard models that fail loss, benchmark, or safety thresholds +### Post-Training (v0.4.0) +- **Interactive Chat**: `forgelm chat ./model` — streaming REPL with `/reset`, `/save`, `/temperature`, `/system` commands; optional Llama Guard safety routing +- **GGUF Export**: `forgelm export ./model --quant q4_k_m` — wraps `llama-cpp-python` converter; 6 quant levels; SHA-256 appended to integrity manifest +- **Deployment Configs**: `forgelm deploy ./model --target ollama|vllm|tgi|hf-endpoints` — generates ready-to-use config files; does not start the server +- **VRAM Fit Check**: `forgelm --config my.yaml --fit-check` — pre-flight memory estimator; `FITS / TIGHT / OOM / UNKNOWN` verdict with recommendations + ### Enterprise & MLOps - **Config-Driven**: Declarative YAML — built for CI/CD pipelines, not notebooks - **EU AI Act Compliance**: Auto-generated audit trails, data provenance (SHA-256), training manifests @@ -44,18 +50,24 @@ ```bash # Install pip install -e . +pip install -e ".[export]" # GGUF export (optional, non-Windows) # Generate config interactively forgelm --wizard -# Or copy template and edit -cp config_template.yaml my_config.yaml - # Validate without training forgelm --config my_config.yaml --dry-run +# Check VRAM before a long run +forgelm --config my_config.yaml --fit-check + # Train forgelm --config my_config.yaml + +# After training: chat, export, deploy +forgelm chat ./checkpoints/final_model +forgelm export ./checkpoints/final_model --output model.gguf --quant q4_k_m +forgelm deploy ./checkpoints/final_model --target ollama --output ./Modelfile ``` See the [Quick Start Guide](docs/guides/quickstart.md) for a complete walkthrough. @@ -162,10 +174,15 @@ forgelm --version # Show version ``` forgelm/ ├── cli.py # CLI with 10+ modes (train, dry-run, merge, benchmark, wizard...) -├── config.py # Pydantic config (15 models: training, evaluation, distributed...) +├── config.py # Pydantic config (19 models: training, evaluation, distributed...) ├── data.py # Dataset loading (SFT, DPO, KTO, GRPO formats + multi-dataset) ├── model.py # Model loading (transformers, unsloth, MoE, PEFT) ├── trainer.py # Training orchestration (6 trainer types via TRL, GaLore, long-context) +├── inference.py # Shared inference primitives (load, generate, stream, adaptive sampling) +├── chat.py # Interactive terminal REPL with streaming and slash commands +├── export.py # GGUF export via llama-cpp-python +├── fit_check.py # Pre-flight VRAM estimator (FITS / TIGHT / OOM / UNKNOWN) +├── deploy.py # Deployment config generator (Ollama, vLLM, TGI, HF Endpoints) ├── results.py # TrainResult dataclass ├── benchmark.py # lm-evaluation-harness integration ├── safety.py # Post-training safety evaluation (Llama Guard) @@ -180,7 +197,7 @@ forgelm/ configs/deepspeed/ # ZeRO-2, ZeRO-3, ZeRO-3+Offload presets notebooks/ # Colab-ready Jupyter notebooks -tests/ # 145+ unit tests across 11 test files +tests/ # 430 passed (+34 skipped) across 30 test files docs/guides/ # Quickstart, alignment, CI/CD, enterprise, safety guides ``` diff --git a/docs/guides/quickstart.md b/docs/guides/quickstart.md index ec8a23c4..353b8c69 100644 --- a/docs/guides/quickstart.md +++ b/docs/guides/quickstart.md @@ -90,12 +90,14 @@ Before starting a long run, estimate if your config fits in GPU memory: ```bash forgelm --config my_config.yaml --fit-check -# → Coming in v0.4.0: GPU: RTX 3060 12GB; Estimated peak: 10.8 GB; Verdict: ✅ FITS +# GPU: RTX 3060 12GB — Estimated peak: 10.8 GB — Verdict: FITS +# Or: Verdict: TIGHT — Enable gradient checkpointing and reduce batch size +# Or: Verdict: UNKNOWN — No GPU detected (hypothetical estimate) ``` -> **Note:** `--fit-check` is planned for v0.4.0. +Output includes a breakdown (base weights, LoRA adapter, optimizer state, activations) and ordered recommendations when memory is tight. Use `--output-format json` for CI/CD integration. -For now, reduce `max_length` and `per_device_train_batch_size` if you hit OOM. The [Troubleshooting guide](troubleshooting.md) has detailed OOM solutions. +If you hit OOM during training, the [Troubleshooting guide](troubleshooting.md) has detailed solutions. ## 6. Use Your Model @@ -112,19 +114,21 @@ output = model.generate(**inputs, max_new_tokens=200) print(tokenizer.decode(output[0], skip_special_tokens=True)) ``` -### Using Your Model (After v0.4.0) +### Using Your Model (v0.4.0+) -Starting with v0.4.0, you can interact with your trained model directly: +Interact with and deploy your trained model directly: ```bash -# Chat with your fine-tuned model +# Chat with your fine-tuned model (streaming by default) forgelm chat ./checkpoints/final_model -# Export to GGUF (for Ollama, LM Studio) -forgelm export ./checkpoints/final_model --format gguf --quant q4_k_m --output model.gguf +# Export to GGUF (for Ollama, LM Studio, llama.cpp) +# Requires: pip install forgelm[export] +forgelm export ./checkpoints/final_model --output model.gguf --quant q4_k_m -# Generate deployment configs +# Generate deployment configs (no server is started) forgelm deploy ./checkpoints/final_model --target ollama --output ./Modelfile +forgelm deploy ./checkpoints/final_model --target vllm --output ./vllm_config.yaml ``` --- diff --git a/docs/reference/usage.md b/docs/reference/usage.md index 6e2c1f65..bf6939dd 100644 --- a/docs/reference/usage.md +++ b/docs/reference/usage.md @@ -22,6 +22,8 @@ pip install -e ".[eval]" # lm-evaluation-harness pip install -e ".[tracking]" # W&B experiment tracking pip install -e ".[distributed]" # DeepSpeed multi-GPU pip install -e ".[merging]" # mergekit model merging +pip install -e ".[export]" # GGUF export (llama-cpp-python, non-Windows) +pip install -e ".[chat]" # Rich rendering in forgelm chat ``` ## Authentication @@ -77,6 +79,47 @@ forgelm --config my_config.yaml --resume ./checkpoints/checkpoint-500 forgelm --config my_config.yaml --offline ``` +### VRAM Fit Check + +Before training, estimate whether your config fits in GPU memory: + +```bash +# Text output (human-readable verdict) +forgelm --config my_config.yaml --fit-check + +# JSON output (for CI/CD pipelines) +forgelm --config my_config.yaml --fit-check --output-format json +``` + +Output shows: estimated peak VRAM, available VRAM (if GPU detected), verdict (`FITS` / `TIGHT` / `OOM` / `UNKNOWN`), and ordered recommendations. Falls back to hypothetical mode when no GPU is detected. + +### Post-Training: Chat, Export, Deploy + +After training, interact with and deploy your fine-tuned model directly from the CLI. These subcommands work without `--config`. + +```bash +# Interactive chat REPL (streaming by default) +forgelm chat ./checkpoints/final_model +forgelm chat ./checkpoints/final_model --adapter ./adapter --safety +forgelm chat ./checkpoints/final_model --system "You are a helpful assistant." --temperature 0.8 + +# Export to GGUF (for Ollama, LM Studio, llama.cpp) +# Requires: pip install forgelm[export] +forgelm export ./checkpoints/final_model --output model.gguf --quant q4_k_m +forgelm export ./checkpoints/final_model --output model.gguf --quant q8_0 --adapter ./adapter + +# Generate deployment config files +forgelm deploy ./checkpoints/final_model --target ollama --output ./Modelfile +forgelm deploy ./checkpoints/final_model --target vllm --output ./vllm_config.yaml +forgelm deploy ./checkpoints/final_model --target tgi --output ./docker-compose.yaml +forgelm deploy ./checkpoints/final_model --target hf-endpoints --output ./endpoint.json +forgelm deploy ./checkpoints/final_model --target ollama --output ./Modelfile --system "Be concise." +``` + +**Chat slash commands:** `/reset`, `/save [file]`, `/temperature N`, `/system [prompt]`, `/help`, `/exit` + +**Export quantization levels:** `q2_k`, `q3_k_m`, `q4_k_m` (recommended), `q5_k_m`, `q8_0`, `f16` + ### Synthetic Data Generation ```bash diff --git a/docs/roadmap.md b/docs/roadmap.md index 0ea52e91..7041f85d 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -7,16 +7,15 @@ | Tür | Phase | Status | |-----|-------|--------| | ✅ Done | [Phase 1-9](roadmap/completed-phases.md) | SOTA upgrades, evaluation, reliability, enterprise integration, ecosystem, alignment stack, safety, EU AI Act compliance (Articles 9-17 + Annex IV), advanced safety intelligence | -| 📋 Planned | [Phase 10 — Post-Training Completion](roadmap/phase-10-post-training.md) | `forgelm/inference.py`, `chat`, `export` (GGUF), `fit-check`, `deploy` → `v0.4.0` | +| ✅ Done | [Phase 10 — Post-Training Completion](roadmap/phase-10-post-training.md) | `inference.py`, `chat`, `export` (GGUF), `--fit-check`, `deploy` — shipped `v0.4.0` | | 📋 Planned | [Phase 10.5 — Quickstart Layer & Onboarding](roadmap/phase-12-quickstart.md) | `forgelm quickstart