Skip to content

architecture.md

Latest commit

 

History

History
324 lines (253 loc) · 20 KB

architecture.md

File metadata and controls

324 lines (253 loc) · 20 KB
Raw

Architecture Standard

Scope: How ForgeLM's code is organized. What belongs where, and what does not. Enforced by: Code review + the "anti-patterns" list in this document.

The one-line architecture

YAML config → Pydantic validation → Trainer orchestration → HuggingFace/PEFT/TRL primitives → artifacts (model + compliance bundle).

Everything in forgelm/ serves one of those stages. Anything that doesn't is out of scope.

Module topology

graph TB
    subgraph Entry["Entry"]
        CLI[cli/ sub-package]
    end

    subgraph Config["Config layer"]
        CONFIG[config.py]
        WIZARD[wizard/ sub-package]
    end

    subgraph Orch["Orchestration"]
        TRAINER[trainer.py]
        UTILS[utils.py]
        RESULTS[results.py]
    end

    subgraph Runtime["Training runtime"]
        MODEL[model.py]
        DATA[data.py]
    end

    subgraph Quality["Evaluation + safety"]
        BENCH[benchmark.py]
        SAFETY[safety/ sub-package]
        JUDGE[judge.py]
    end

    subgraph Enterprise["Enterprise"]
        COMPLIANCE[compliance.py]
        WEBHOOK[webhook.py]
        CARD[model_card.py]
    end

    subgraph Extras["Optional pipelines"]
        MERGE[merging.py]
        SYNTH[synthetic.py]
    end

    CLI --> CONFIG
    CLI --> WIZARD
    CLI --> TRAINER
    WIZARD --> CONFIG
    TRAINER --> MODEL
    TRAINER --> DATA
    TRAINER --> BENCH
    TRAINER --> SAFETY
    TRAINER --> JUDGE
    TRAINER --> COMPLIANCE
    TRAINER --> WEBHOOK
    TRAINER --> CARD
    TRAINER --> UTILS
    TRAINER --> RESULTS
    CLI --> MERGE
    CLI --> SYNTH
Loading

Module-size ceiling

Single-file modules are the default — most concerns fit in one file under forgelm/. The ~1000-line ceiling is the trigger for a sub-package split: once a module crosses that line count and has cohesive subsections, it graduates to a module_name/ sub-package, keeping the public API at forgelm.module_name.X so existing imports do not break.

Enforced by tools/check_module_size.py --strict (CI step "Module-size ceiling guard" in .github/workflows/ci.yml's validate job): it fails on any forgelm/ module that newly crosses the ceiling, while a _DEFERRED_SPLITS list tracks modules that were already over the line when the guard landed.

Deferred splits carry a budget, not a target version

A deferred entry pins the module's measured LOC at deferral time. The guard then enforces a ratchet: a deferred file may stay over the ceiling, but that file growing past its budget is fatal in every mode. Splitting remains the goal; holding the line is the enforceable part.

The unit of enforcement is a file, not the concern the file owns. Adding a new 803-LOC sibling module — say a _trainer_overflow.py dropped beside the deferred forgelm/trainer.py — passes --strict with exit 0. The new file is a NEW module held to the normal 1000-LOC ceiling, not charged against trainer.py's budget, so the concern grew while no file did. That is deliberate: every reason field names the sibling files a split should produce, so charging siblings against the parent would penalise the exact refactor the backlog asks for, and a per-concern budget would need a hand-maintained file-to-concern map — the rot-prone artefact the budget mechanism replaced. Total LOC across the tree is not bounded by this guard; noticing a concern smeared across several under-ceiling files is a code-review responsibility.

Deferrals deliberately name no target version. Until 2026-07-20 every entry read "defer to v0.6.x split" — accurate when written at v0.5.5, still printing at v0.9.1rc1, by which point every deferred module had grown and two of the eight entries were simply wrong. A version literal makes a prediction, and a prediction nobody re-reads decays into a false statement; a budget asserts nothing about the future and cannot rot the same way. Do not "fix" a deferral by naming a newer version — see docs/roadmap/risks-and-decisions.md (2026-07-20 section) for the ordered backlog, the rationale per module, and the plain statement of which splits are not currently scheduled.

Rules for the list:

  • Joining requires a prose reason naming the concerns a split would separate, plus a risks-and-decisions.md row. It is not a permanent exemption.
  • Raising a budget is the explicit escape hatch — a security fix should not be blocked on a day-long refactor. Edit the literal in the same PR and append the justification to that entry's budget_history, so extra headroom is always a reviewed line in a diff rather than silent drift. This is enforced, not just asked for: each entry also carries an immutable deferred_at_loc, and a budget raised above it with an empty or blank budget_history fails the guard in every mode. Lowering a budget after a trim needs no justification.
  • Leaving happens the moment the module is split or trimmed under the ceiling: delete the entry in the same PR. The guard reports a dangling entry (path gone) as fatal and a stale entry (module back under the ceiling) as fatal under --strict.

Four such splits are permanent today (Phase 12.6 closure cycle Wave 1, forgelm/safety/ at v0.9.1, and forgelm/verify/ in Phase 16 S1):

Sub-package Pre-split source Reason Public surface
forgelm/cli/ legacy cli.py (~1756 lines) Argparse wiring + ~17 subcommand modules grew far past the ceiling; each subcommand needed its own test boundary. forgelm.cli.main() (entry point), forgelm.cli._exit_codes (public exit-code constants), per-subcommand modules (_audit, _ingest, _doctor, _cache, _purge, _reverse_pii, _safety_eval, _verify_audit, _verify_annex_iv, _verify_gguf, _approve, _approvals, ...)
forgelm/data_audit/ legacy data_audit.py (~3098 lines) Aggregator, simhash, MinHash, regex-PII, ML-NER PII, secrets, quality, croissant, summary, streaming reader — 10 cohesive concerns under one orchestrator. forgelm.data_audit.run_audit(), forgelm.data_audit.AuditReport, forgelm.data_audit.SECRET_TYPES, forgelm.data_audit.summarize_report() (re-exported via the package __init__.py)
forgelm/safety/ legacy safety.py (1038 LOC) Generation-based Llama-Guard scoring pushed the module over the ceiling: input prep, classifier load, generation, two scoring aggregators, gate evaluation and result assembly are separable concerns. forgelm.safety.run_safety_evaluation(), forgelm.safety.SafetyEvalThresholds (public library surface per forgelm/__init__.py's __all__), re-exported via the package __init__.py
forgelm/verify/ legacy verify.py (1013 LOC) Three unrelated verifiers plus a chain walker in one module: single-artefact Annex IV completeness, the pipeline's per-stage evidence deep-parse, GGUF magic/metadata/sidecar, the Art. 15 model-directory re-hash, and the audit-log failure classifier. Split behaviour-neutral and alone, because it moves the exit-code routing tokens the CLI and the tests both pin. forgelm.verify.verify_annex_iv_artifact() / verify_gguf() / verify_integrity() and their result dataclasses (stable tier per forgelm/__init__.py's __all__), the four is_*_integrity_failure predicates, re-exported via the package __init__.py

Future splits follow the same rule: when crossing ~1000 lines, design the sub-package boundary first (one design doc under docs/design/), execute across a multi-PR series, and preserve the public import path. Splits are never silent — they ship as a coherent series with regression tests.

Principles

1. Each module owns one concern

Module Owns Does not own
cli/ (package) Argument parsing, top-level dispatch, exit codes, per-subcommand wiring (_audit, _ingest, _doctor, _cache, _purge, _reverse_pii, _safety_eval, _verify_*, _approve, _approvals, ...) Training, data, evaluation logic
config.py Pydantic schemas, validation, YAML load Runtime behaviour
wizard/ (package) Interactive config generation — sub-package with _io, _state, _collectors, _orchestrator, _byod modules; emits WizardOutcome to the CLI dispatcher (Phase 22) Config validation (that's config.py)
trainer.py Orchestrating SFT/DPO/…/GRPO runs Model loading (delegates to model.py)
model.py AutoModelForCausalLM/PEFT setup Dataset formatting
data.py Dataset load + format detection Training loops
benchmark.py lm-eval-harness wrapping Safety scoring (that's safety/)
safety/ (package) Llama Guard, harm categories, auto-revert — sub-package with _types, _inputs, _generate, _classifier, _score_classification, _score_generation, _gates, _results, _orchestrator modules Content generation for scoring (helpers only)
judge.py LLM-as-judge evaluation Safety classification
compliance.py Audit log, manifests, provenance, governance artifacts, GDPR purge / reverse-pii primitives Runtime policy enforcement
verify/ (package) Artefact verification primitives — sub-package with _annex_iv, _pipeline_evidence, _gguf, _model_integrity, _audit_log, _io_safety modules behind a re-exporting __init__: Annex IV manifest-hash recomputation, GGUF magic / metadata / SHA-256-sidecar checks, the model-directory integrity walk, and the is_*_integrity_failure predicates the four verify-* subcommands route exit code 6 on Audit-log verification (stays beside its writer in compliance.py); output formatting and exit-code emission (that's cli/)
webhook.py Slack/Teams lifecycle notifications (8-event vocabulary — see webhook_schema.md) Decision-making (just reports)
model_card.py HF-compatible README generation Running the model
merging.py TIES/DARE/SLERP/linear Training
synthetic.py Teacher-student distillation General generation helpers
ingestion.py Raw docs (PDF/DOCX/EPUB/TXT) → SFT-ready JSONL; chunking strategies Audit logic; trainer dispatch
data_audit/ (package) Dataset quality + governance audit: length, language, simhash + LSH dedup, MinHash LSH dedup, cross-split leakage, regex PII, ML-NER PII, secrets, quality heuristics, Croissant emission, summary truncation, streaming reader Ingestion logic; trainer dispatch
_http.py Outbound HTTP chokepoint: SSRF guard, scheme allowlist, timeout floor, secret masking, redirect refusal Caller decisions (just transport)
_version.py __version__ (CLI) + __api_version__ (Library API) constants Anything else
results.py TrainResult dataclass Anything else
utils.py HF auth + tiny cross-cutting helpers Business logic

If a PR adds code to a module whose concern doesn't match, split it out. Don't grow trainer.py into a kitchen sink.

2. Config is the one-way contract

CLI parses args and loads YAML → ForgeConfig (Pydantic) → passed as a single argument downstream. Modules do not:

  • Read environment variables directly for training decisions (only for secrets like HF_TOKEN, OPENAI_API_KEY).
  • Accept **kwargs pass-throughs that bypass the schema.
  • Mutate config at runtime. If a flag is "derived" from others, compute it in a model_validator on ForgeConfig.

This is why the --dry-run smoke path works without GPUs: all behaviour is determined by the validated config.

3. Optional dependencies are extras, never silent imports

Heavy deps (bitsandbytes, unsloth, deepspeed, lm-eval, wandb) are declared in pyproject.toml under [project.optional-dependencies] with version constraints (a lower bound each; some also carry an upper bound). At call sites:

try:
    import unsloth  # noqa: F401
except ImportError as e:
    raise ImportError(
        "unsloth backend requires the 'unsloth' extra. "
        "Install with: pip install 'forgelm[unsloth]'"
    ) from e

Do not:

  • Write try: import X except: X = None and then sprinkle if X is not None everywhere.
  • Make a heavy dep a hard requirement of the core package.
  • Add a new heavy dep without adding a new extra.

Optional accelerator dependencies — sanctioned carve-out

The CLAUDE.md "silent import fallback" pitfall (try: import X; except ImportError: X = None) is forbidden as a general pattern, but the following four conditions justify an exception:

  1. Every read site gates on a _HAS_X: bool boolean — never on the imported handle's truthiness.
  2. The missing-extra path provides a behaviourally equivalent (slower / less precise) fallback rather than a feature gate. If the absent extra disables a feature, raise ImportError with the install hint instead.
  3. The pattern lives in a single dedicated module so the fallback surface is auditable in one place.
  4. Tests monkeypatch the boolean to exercise both paths (extras-installed and extras-missing).

The canonical example is forgelm/data_audit/_optional.py, which gates the four optional data_audit accelerators / models — xxhash (simhash backend; falls back to BLAKE2b), numpy (vectorised bit-unpack; falls back to pure-Python loop), datasketch (MinHash LSH; user opts in via --dedup-method minhash), and presidio-analyzer (ML-NER PII; layered on top of the regex detector). Each block is paired with a _HAS_* boolean and tests exercising both paths.

New optional accelerators may follow this template. New behaviour gates (where the absent extra silently disables a feature instead of falling back) may NOT — those must raise ImportError with an install hint at the call site.

4. No global state

Module-level state is limited to:

  • Logger objects (logger = logging.getLogger("forgelm.X"))
  • Constants (EXIT_SUCCESS = 0, etc.)
  • Immutable registries (tables mapping trainer types to classes)

Runtime state is held by passed-in objects (Pydantic config, trainer instance). No module-level mutable state. No singletons except the logger.

5. CLI is a thin shim

forgelm.cli/ (package) does three things and no more:

  1. Parse args (argparse) — wiring lives in _parser.py.
  2. Load and validate config — _config_load.py.
  3. Dispatch to one of: trainer, wizard, synthetic, merging, compliance export, ingest, audit, doctor, cache-models, cache-tasks, purge, reverse-pii, safety-eval, verify-audit, verify-annex-iv, verify-gguf, approve, approvals, chat, export, deploy, quickstart — each owns a subcommands/_<name>.py.

Business logic in any cli/ module is a bug. If you find yourself writing an if chain that inspects config values to decide behaviour, that if belongs in the dispatched module.

Adding a new module

Checklist before opening the PR:

  1. Does an existing module already own this concern? If yes, add there.
  2. Can this logic be expressed as a new BaseModel in config.py + a function in an existing module? If yes, do that.
  3. If a new module is warranted, does it fit the table above? Add a row.
  4. Does it need a new optional dependency? Add an extra in pyproject.toml.
  5. Does it have a tests/test_<module>.py? See testing.md.
  6. Is it imported by trainer.py or the forgelm/cli/ package behind config? Runtime features must be config-gated.

The extras matrix

From pyproject.toml:

Extra Purpose Platform
qlora bitsandbytes 4-bit Linux only
unsloth Triton-based fast backend Linux only
eval lm-evaluation-harness Any
tracking Weights & Biases Any
distributed DeepSpeed Linux only
merging (none — native peft+torch; no-op extra) Any
ingestion pypdf, python-docx, ebooklib, beautifulsoup4, langdetect, xxhash Any
ingestion-scale datasketch (MinHash LSH dedup; Phase 12, opt-in) Any
ingestion-pii-ml presidio-analyzer (ML-NER for person/organization/location PII; Phase 12.5, opt-in) Any
export llama-cpp-python (GGUF conversion) Linux/macOS
chat rich (terminal rendering) Any
security pip-audit + bandit (supply-chain CI guards; Phase 12.6 / Wave 4) Any
dev pytest, ruff Any (contributors)

The core install must work on all three OSes (Linux/macOS/Windows) with no Linux-only deps. CI enforces this by running Linux + macOS matrix.

HTTP discipline

Rule: every outbound HTTP call from forgelm/ goes through forgelm/_http.py (safe_post / safe_get). Direct use of requests.*, urllib.request.urlopen, httpx.* etc. outside forgelm/_http.py is forbidden.

forgelm/_http.py is the single chokepoint that enforces:

  • Scheme allowlist — only https:// by default; http:// requires the caller to pass allow_insecure_http=True and is logged.
  • SSRF guard — pre-resolves hostnames + refuses RFC 1918 / link-local / loopback destinations unless the caller passes allow_private=True.
  • Timeout floor — minimum 5 s connect, 30 s read; callers cannot pass timeout=0 to disable.
  • Secret masking — _mask_secrets_in_text redacts query-param tokens / Authorization headers in error messages so a URLError(reason=...) cannot leak ?api_key=ghp_... into logs.
  • Redirect refusal — allow_redirects=False by default; redirects to a different host require explicit opt-in.

Why a single chokepoint: the policy lives in one module so that a future fix (rotate to a stricter allowlist, add a new mask pattern, plug a new metric) lands in one place rather than N scattered call sites. Webhook delivery (forgelm/webhook.py), the doctor's HF Hub probe (forgelm/cli/subcommands/_doctor.py), and any future telemetry / license / cloud integration all share the same policy.

Acceptance gate (CI-enforced — see .github/workflows/ci.yml "HTTP discipline guard" step, lint job):

python3 tools/check_http_discipline.py

tools/check_http_discipline.py is the promotion of a former inline ci.yml grep to a tested tool (F-P8-C-19) — the grep matched only direct dotted calls (requests.get() and missed requests.Session()-bound calls, aliased imports (from requests import get, from urllib.request import urlopen), whitespace before the paren (requests.get (url)), and httpx.Client() / httpx.AsyncClient() construction. The tool rejoins logical lines (so a call split across physical lines by an open ( is still caught) and flags all of the above, while exempting forgelm/_http.py itself and comment-only lines. It has its own test suite (tests/test_check_http_discipline.py). A new contributor who reaches for requests.get directly — including via an aliased import — fails CI immediately and is redirected to safe_get. If _http.py itself needs to expand (e.g., add safe_get because no helper covers an outbound HEAD probe yet — Phase 34's doctor surfaced this gap), the addition lands inside _http.py and gets a corresponding test beside the existing HTTP coverage in tests/test_http_dns_rebinding.py and tests/test_check_http_discipline.py.

Deliberate exceptions: forgelm/compliance.py:1146-1182 (audit-log HMAC verifier) does its own JSONL line-byte parsing because it must hash raw bytes — but it does not perform outbound HTTP. No HTTP-discipline carve-outs exist today; if one is ever needed (e.g., a cloud provider's SDK that wraps its own HTTP), document it inline + add a # noqa: forgelm-http-discipline style marker.

Things we do not own

Reaffirming the internal "what we don't build" strategy doc (also captured under "What ForgeLM is not" in the root CLAUDE.md):

  • Inference engine. We hand off to Ollama / vLLM / TGI / llama.cpp. forgelm/inference.py (Phase 10) is a thin client, not a server.
  • Web UI. Config-driven is the identity. Dashboards live in Pro CLI (Phase 13), not core.
  • Custom model architectures. HuggingFace owns model code.
  • Custom quantization. bitsandbytes / AWQ / GPTQ / HQQ own it.
  • GPU marketplace or serving infra. User brings their own GPU.

If a PR pushes in any of these directions, it gets redirected or rejected.