Skip to content

json-output.md

Latest commit

 

History

History
1001 lines (831 loc) · 57.5 KB

json-output.md

File metadata and controls

1001 lines (831 loc) · 57.5 KB
Raw
title JSON output schemas
description The locked --output-format json envelope shape for every forgelm subcommand. CI consumers depend on these field names.

JSON output schemas

Every forgelm subcommand that supports --output-format json produces a stable JSON envelope on stdout. Field names + nesting are part of the public CLI contract per the release standard on GitHub: renaming a key is a MAJOR-version break.

This page is the canonical reference. CI/CD pipelines that parse forgelm output should pin against the shapes documented here.

Common conventions

  • stdout vs stderr. The JSON envelope goes to stdout. Human-friendly logs (info / warning / error) go to stderr. Pipe forgelm ... --output-format json | jq . and read your operator-facing messages from 2> separately.
  • Top-level wrapper. Every envelope starts with "success": true | false. Consumers can branch on this single key before parsing the rest.
  • Error envelope. When success: false, the envelope carries "error": "<message>" (string). Optional richer fields (exit_code, error_type, details) MAY be present per the error-handling standard on GitHub. Consumers that need certainty on these fields should also check the process exit code via $?.
  • Exit codes. See Exit Codes. success: false always pairs with a non-zero exit. success: true pairs with exit 0 in the normal case, with one exception: a training run halted at the Article 14 human-approval gate emits success: true and awaiting_approval: true while exiting 4 (the model is staged, not yet promoted). Consumers that must distinguish "done" from "awaiting human approval" should read the awaiting_approval discriminator (or check $?), not success alone.
  • Argument-parsing errors are the exception. The JSON-envelope guarantee begins after arguments parse successfully. A usage error detected at parse time (unknown flag, bad --choice, out-of-range validator value such as --workers 0) is reported by argparse itself: a human-readable usage: ... error: ... line on stderr and exit code 2, with no JSON envelope on stdout. A consumer that reads stdout-JSON on every non-zero exit must treat empty stdout (paired with non-zero $?) as a parse/usage error and fall back to stderr for the message. The envelope contract applies only to errors raised once the dispatcher is running.

forgelm doctor

Environment check. See Doctor command.

Success envelope (forgelm doctor [--offline] --output-format json):

{
  "success": true,
  "checks": [
    {
      "name": "forgelm.install",
      "status": "pass",
      "detail": "ForgeLM 0.9.0 running from /opt/venv/lib/python3.11/site-packages/forgelm (inside site-packages).",
      "extras": {"version": "0.9.0", "location": "/opt/venv/lib/python3.11/site-packages/forgelm", "in_site_packages": true}
    },
    {
      "name": "python.version",
      "status": "pass",
      "detail": "Python 3.11.7 (CPython).",
      "extras": {"version": "3.11.7", "implementation": "CPython"}
    }
  ],
  "summary": {"pass": 8, "warn": 1, "fail": 0, "crashed": 0}
}
Key Type Notes
success bool true when no probe fail AND no probe crash; false otherwise.
checks list[object] One entry per probe in execution order. Probe names are stable (e.g. forgelm.install, python.version, torch.cuda, numpy.torch_abi, gpu.inventory, extras.qlora, hf_hub.reachable, hf_hub.offline_cache, disk.workspace, operator.identity, pypdf_normalise.turkish). In --offline mode hf_hub.offline_cache replaces hf_hub.reachable.
checks[].name str Probe name. Stable across versions; "append" describes the vocabulary of names, not execution position — new probes are never renamed once shipped, but they are inserted wherever their content is most useful to read alongside (e.g. forgelm.install was inserted as the first row, not appended at the end, because every other line of a report is ambiguous until the reader knows which copy of ForgeLM produced it). Consumers must not assume a fixed index for any probe.
checks[].status str One of pass, warn, fail. A probe that raised surfaces as status: "fail" with extras.crashed: true; the crash is also counted in summary.crashed.
checks[].detail str Operator-facing one-line description of the result.
checks[].extras object Probe-specific structured data. Per-probe keys are documented in _doctor.py docstrings; consumers should treat unknown keys as forward-compatible.
summary object Counts of each status across checks. Sum equals len(checks).

Exit code mapping: 0 = all probes pass or warn; 1 = at least one fail; 2 = at least one crashed (probe raised; subsequent probes still ran).

forgelm --dry-run (config validation)

forgelm --config <yaml> --dry-run --output-format json validates the config and summarizes what training would do without loading the heavy stack or touching the GPU. It is the most common CI pre-flight invocation. On a valid config it emits the following envelope on stdout and exit code 0:

{
  "success": true,
  "status": "valid",
  "model": "org/model",
  "backend": "auto",
  "load_in_4bit": false,
  "trust_remote_code": false,
  "dora": false,
  "lora_rank": 16,
  "lora_alpha": 32,
  "dataset": "org/dataset",
  "epochs": 3,
  "batch_size": 1,
  "output_dir": "/work/output/final_model",
  "offline": false,
  "distributed": null,
  "rope_scaling": null,
  "neftune_noise_alpha": null,
  "webhook_configured": false,
  "galore_enabled": false,
  "galore_optim": null,
  "galore_rank": null,
  "auto_revert": false,
  "safety_enabled": false,
  "safety_scoring": null,
  "compliance_configured": false,
  "risk_classification": null
}
Key Type Notes
success bool Always true on this path — a dry run only reaches stdout once the config validated. An invalid config exits 1 with the 2-key error envelope (success: false) instead.
status str Stable token "valid", retained for backward compatibility with pre-0.7.1 consumers. New consumers should branch on success.
model / backend / dataset str Resolved model id, model backend, and dataset id from the config.
output_dir str The effective final-model path (training.output_dir joined with training.final_model_dir).
load_in_4bit, trust_remote_code, dora, offline bool Effective values of the corresponding config flags.
lora_rank, lora_alpha, epochs, batch_size int Effective training/LoRA hyper-parameters.
distributed str | null distributed.strategy, or null when distributed training is not configured.
rope_scaling, neftune_noise_alpha object | float | null Effective long-context / NEFTune settings, null when unset.
webhook_configured bool true when a webhook URL (literal or url_env) is configured.
galore_enabled, galore_optim, galore_rank bool / str / int GaLore optimizer summary; the latter two are null when GaLore is off.
auto_revert, safety_enabled, safety_scoring bool / bool / str Evaluation-gate summary (safety_scoring is null when safety eval is off).
compliance_configured, risk_classification bool / str EU AI Act compliance summary; risk_classification is null when compliance is not configured.

Exit code mapping: a valid config exits 0 (EXIT_SUCCESS); an invalid config exits 1 (EXIT_CONFIG_ERROR) with the standard error envelope.

forgelm (training) — preflight abort envelope

The training pipeline (forgelm --config <yaml> --output-format json) runs a torch/NumPy ABI sanity check before importing the heavy stack. On a healthy environment the preflight is silent and training proceeds normally; on the known Intel Mac NumPy 2 / torch 2.2 mismatch the preflight aborts with the following envelope on stdout and exit code 2:

{
  "success": false,
  "error": "numpy_torch_abi_mismatch",
  "torch_version": "2.2.2",
  "numpy_version": "2.4.4",
  "remediation": "torch 2.2.2 (compiled against NumPy 1.x) is paired with numpy 2.4.4. ... Fix with: pip install 'numpy<2' ..."
}
Key Type Notes
success bool Always false on this path; healthy preflight emits nothing and the pipeline continues.
error str Stable token "numpy_torch_abi_mismatch". CI consumers can branch on this exact value.
torch_version str The version string torch.__version__ reported at preflight time.
numpy_version str The version string numpy.__version__ reported at preflight time.
remediation str Human-readable fix instructions ending with the exact pip install 'numpy<2' command. Identical text to the forgelm doctor numpy.torch_abi probe's detail field — same single source of truth.

Exit code mapping: preflight abort exits 2 (EXIT_TRAINING_ERROR, the runtime-error class). The same forgelm doctor numpy.torch_abi probe would have surfaced this as status: "fail" ahead of time; the preflight is the second line of defense for operators who skipped doctor and ran training directly.

forgelm (training) — result envelope

When training runs to completion the pipeline emits a result envelope on stdout. The same envelope shape covers all three outcomes; the success + awaiting_approval + reverted keys discriminate between them.

Success envelope (model promoted; exit 0):

{
  "success": true,
  "metrics": {"eval_loss": 0.42, "benchmark/average": 0.78},
  "final_model_path": "/work/output/final_model",
  "reverted": false,
  "awaiting_approval": false,
  "run_id": "fg-abc123def456",
  "config_hash": "sha256:..."
}

Awaiting-approval envelope (evaluation.require_human_approval: true; model staged, not yet promoted; exit 4):

{
  "success": true,
  "metrics": {"eval_loss": 0.42},
  "final_model_path": "/work/output/final_model.staging.fg-abc123def456",
  "reverted": false,
  "awaiting_approval": true,
  "staging_path": "/work/output/final_model.staging.fg-abc123def456"
}

Reverted envelope (a post-train gate failed under auto_revert: true; artefacts deleted; exit 3):

{
  "success": false,
  "metrics": {"eval_loss": 0.91},
  "final_model_path": "/work/output/final_model",
  "reverted": true,
  "awaiting_approval": false,
  "error": "Final eval_loss (0.9100) exceeded max_acceptable_loss (0.8000)."
}
Key Type Notes
success bool true when the run completed (including a staged, awaiting-approval run, and a run whose safety evaluation produced no verdict: check safety.evaluation_completed). false when a gate auto-reverted the model.
error string Present iff success is false, never on the success path. Carries the gate's own reason — which gate fired and against what threshold — because every gate exits 3, so the exit code alone cannot tell them apart. If a failure reaches the envelope with no reason recorded, the field says so explicitly and points at audit_log.jsonl rather than being omitted.
metrics object Training/eval/gate metrics (e.g. eval_loss, benchmark/average, safety/safe_ratio). Values are numbers, except a measurement that came out non-finite, which is written as the string "nan" / "inf" / "-inf" — JSON has no non-finite number literal, so the alternative was output no strict parser accepts. Type these fields as number-or-string.
final_model_path str Where the model artefacts live. For an awaiting-approval run this is the staging directory until forgelm approve promotes it.
reverted bool true iff a gate (eval-loss / benchmark / safety / judge) auto-reverted the model. Mutually exclusive with awaiting_approval.
awaiting_approval bool Discriminator. true iff the run halted at the Article 14 human-approval gate (exit 4). A reverted run is always false here.
staging_path str Present only when awaiting_approval is true; the on-disk staging dir to pass to forgelm approve <run_id> / forgelm reject <run_id>.
run_id str Run identifier — correlates the run with its audit_log.jsonl and any approval gate. Present whenever the trainer produced the result.
config_hash str sha256: digest of the validated config that produced the run (reproducibility anchor). Present whenever the trainer produced the result.

Optional sub-blocks (benchmark, resource_usage, estimated_cost_usd, safety, judge) are added only when those evaluations ran.

Exit code mapping: 0 = success (promoted); 3 = EXIT_EVAL_FAILURE (a gate reverted the model — reverted: true); 4 = EXIT_AWAITING_APPROVAL (awaiting_approval: true, staged pending human sign-off). A reverted run is never reported as awaiting approval.

forgelm approvals --pending

Lists pending Article 14 approval requests, newest-first.

{
  "success": true,
  "pending": [
    {
      "run_id": "fg-abc123def456",
      "staging_path": "/work/output/final_model.staging.fg-abc123def456",
      "staging_exists": true,
      "requested_at": "2026-05-04T12:34:56+00:00",
      "age_seconds": 3600.5,
      "metrics": {"safety_score": 0.91, "benchmark.hellaswag": 0.78},
      "config_hash": "sha256:...",
      "reason": "post-train safety eval below threshold"
    }
  ],
  "count": 1
}
Key Type Notes
success bool true (always — empty list is success).
pending list[object] Newest-first; count == len(pending). Empty list = no pending approvals.
pending[].run_id str | null Run identifier from the audit event.
pending[].staging_path str | null Resolved staging directory (null when audit event lacks staging_path and no canonical fallback exists). Path-traversal guarded — paths outside --output-dir are rejected at resolution time.
pending[].staging_exists bool True iff staging_path resolves to an existing directory.
pending[].requested_at str | null ISO-8601 timestamp from the audit event.
pending[].age_seconds number | null Seconds since requested_at; null when timestamp is unparseable.
pending[].metrics object Free-form per-run metrics from the audit event.
pending[].config_hash str | null Config fingerprint, when known.
pending[].reason str | null Operator-supplied reason from the audit event.
count int Pending count; equal to len(pending).

forgelm approvals --show RUN_ID

Inspect the full approval-gate audit chain + staging contents for one run.

{
  "success": true,
  "run_id": "fg-abc123def456",
  "status": "pending",
  "chain": [
    {"event": "human_approval.required", "run_id": "fg-abc123def456", "timestamp": "2026-05-04T12:34:56+00:00", "...": "..."}
  ],
  "staging_contents": ["adapter_config.json", "adapter_model.safetensors", "tokenizer.json"]
}
Key Type Notes
success bool true when the run was found; false (with error) when RUN_ID has no audit events.
run_id str Echoed input RUN_ID.
status str One of pending, granted, rejected, unknown. Latest-wins semantics: a re-staged run after a prior decision shows pending.
chain list[object] Every approval-gate audit event for run_id, in append order.
staging_contents list[str] Sorted file/directory names at <output_dir>/final_model.staging.<run_id> (or canonical fallback). Empty when staging missing or unreadable.
corrupted bool Present and true only when a strict parse of the audit log would reject a line. The displayed status is then not authoritative — approve/reject (which read the log strictly) will refuse to act until the log is repaired. Absent on a clean log.
corruption_detail str Present alongside corrupted; names the first offending line (audit log corrupted at line N: <reason>).

forgelm audit

Pre-train data audit. Full report; key fields shown.

{
  "success": false,
  "report_path": "/work/audit/data_audit_report.json",
  "generated_at": "2026-07-20T16:13:19.426220+00:00",
  "source_input": "/work/data/train.jsonl",
  "total_samples": 3,
  "splits": {"train": 3},
  "pii_summary": {"email": 1, "us_ssn": 1},
  "pii_severity": {
    "total": 2,
    "by_tier": {"critical": 0, "high": 1, "medium": 1, "low": 0},
    "by_type": {"email": {"count": 1, "tier": "medium"}, "us_ssn": {"count": 1, "tier": "high"}},
    "worst_tier": "high"
  },
  "secrets_summary": {"aws_access_key": 1},
  "secrets_gate": {
    "status": "failed",
    "severity": "critical",
    "critical_total": 1,
    "critical_types": {"aws_access_key": 1},
    "allow_secrets": false
  },
  "pii_gate": {
    "status": "passed",
    "severity": null,
    "critical_total": 0,
    "critical_types": {},
    "advisory_total": 2,
    "advisory_types": {"email": 1, "us_ssn": 1},
    "allow_pii": false
  },
  "quality_summary": {
    "samples_flagged": 3,
    "samples_evaluated": 3,
    "by_check": {"low_punct_endings": 3},
    "overall_quality_score": 0.0
  },
  "near_duplicate_pairs_per_split": {"train": 0},
  "near_duplicate_summary": {"method": "simhash", "pairs_per_split": {"train": 0}, "hamming_threshold": 3},
  "cross_split_leakage_pairs": [],
  "croissant": {},
  "notes": ["WORST tier: HIGH. PII flags surfaced (2 total: email=1, us_ssn=1). …"]
}
Key Type Notes
success bool false when either gate failed (secrets_gate or pii_gate), true otherwise. It is not a full verdict about the corpus — the ungated findings never move it; see the gate warning below.
report_path str Absolute path of the on-disk data_audit_report.json.
source_input str Absolute path of the audited input.
total_samples int Rows examined across all splits.
splits object {split_name: row_count} — a flat integer map, not a nested object.
pii_summary object Flat {pii_type: count}, e.g. {"email": 1}. No total_findings / by_kind keys. {} when clean.
pii_severity object total, by_tier (critical/high/medium/low counts), by_type, and worst_tier. worst_tier is the field a CI gate wants — it is null when nothing was flagged.
secrets_summary object Flat {pattern_name: count}, e.g. {"aws_access_key": 1}. {} when clean.
secrets_gate object Verdict of the credential gate. status ∈ "passed" / "failed" / "suppressed" ("suppressed" = findings present but --allow-secrets was passed), severity ("critical" or null), critical_total, critical_types ({pattern_name: count}), allow_secrets.
pii_gate object Verdict of the PII gate — the sibling of secrets_gate, same three-valued status ("suppressed" = --allow-pii was passed). severity ("critical" or null), critical_total, critical_types, plus advisory_total / advisory_types carrying the sub-critical counts that are reported but never gate, and allow_pii. Only credit_card and iban can appear in critical_types.
quality_summary object samples_flagged, samples_evaluated, by_check ({check_name: count}), overall_quality_score.
near_duplicate_pairs_per_split object {split_name: pair_count}.
near_duplicate_summary object method (simhash/minhash), pairs_per_split, and the threshold field for the chosen method.
cross_split_leakage_pairs list Train/eval overlap pairs; empty list when clean.
croissant object Croissant metadata when --croissant is passed; {} otherwise.
notes list[str] Human-readable findings and remediation hints.

Phase 15 (v0.6.0) note: quality_summary is populated by default from v0.6.0+ because the audit's --quality-filter flag was flipped to default-on. Pass --no-quality-filter to skip; the field is {} in that case.

:::warn Two scans gate. Everything else is a report. forgelm audit exits 3 when the always-on secrets scan finds credentials (--allow-secrets records without failing), and when the PII scan finds critical-tier PII — credit_card or iban, the two categories that clear a checksum (issuer-prefix + Luhn / ISO 7064 mod-97), so a hit is indistinguishable from a genuine card or account number (--allow-pii records without failing). The two flags are independent.

Sub-critical PII does not gate: tr_id / de_id / fr_ssn / us_ssn (high), email (medium) and phone (low) are detected, counted, and surfaced under pii_gate.advisory_types, but never decide the exit code — most are matched on shape alone and deliberately over-report, and a gate that fires on a clean corpus is one an operator turns off. So a corpus carrying a plaintext SSN, train/eval leakage, or a wall of low-quality rows and no credentials or card numbers exits 0 — verified: a two-row corpus with a us_ssn finding at worst_tier: "high" returns 0.

So success: true plus exit 0 means "the audit completed and neither gate fired", not "the corpus is clean". To gate on the other dimensions you must parse this envelope yourself:

$ forgelm audit data/ --output-format json > audit.json   # exits 3 on secrets or critical-tier PII
$ jq -e '(.pii_severity.worst_tier // "none") != "high" and (.cross_split_leakage_pairs | length) == 0' audit.json

pii_severity.worst_tier is null on a clean corpus, so the // "none" fallback is required — without it the expression evaluates against null and the gate misbehaves. :::

The full schema is in the in-manual Data Audit page. Note that the on-disk data_audit_report.json and this stdout envelope do not use identical key names — pin your tooling against whichever one it actually reads.

forgelm ingest

Document ingestion stdout envelope. Phase 15 added additive fields under notes_structured; the pre-Phase-15 keys are unchanged.

{
  "success": true,
  "output_path": "data/curated.jsonl",
  "chunk_count": 245,
  "files_processed": 12,
  "files_skipped": 0,
  "total_chars": 824513,
  "format_counts": {".pdf": 4, ".docx": 6, ".md": 2},
  "pii_redaction_counts": {"email": 3, "phone": 1},
  "secrets_redaction_counts": {},
  "pdf_header_footer_lines_stripped": 18,
  "pdf_paragraph_packed_lines_stripped": 4,
  "script_sanity_triggered": 1,
  "strip_pattern_substitutions": 12,
  "urls_handled": 7,
  "frontmatter_pages_dropped": 3,
  "notes": ["..."],
  "notes_structured": {
    "files_processed": 12,
    "files_skipped": 0,
    "chunk_count": 245,
    "total_chars": 824513,
    "strategy": "paragraph",
    "format_counts": {".pdf": 4, ".docx": 6, ".md": 2},
    "pdf_header_footer_lines_stripped": 18,
    "pdf_paragraph_packed_lines_stripped": 4,
    "script_sanity_summary": {"files_checked": 12, "files_triggered": 1, "...": "..."},
    "frontmatter_pages_dropped": [0, 1, 11],
    "strip_pattern_substitutions": 12,
    "urls_handled": 7,
    "quality_presignal": {"samples_evaluated": 245, "samples_flagged": 17, "by_check": {"alpha_ratio": 5, "weird_chars": 1, "repeated_lines": 11}}
  }
}

On a multi-file batch, the top-level frontmatter_pages_dropped integer is a true sum across every file, while notes_structured.frontmatter_pages_dropped is a distinct-index sample (which page positions were dropped, not how many times) — it can legitimately be shorter than the top-level total and that is not a bug.

The Phase 15 additive fields are documented at Document Ingestion. Pre-Phase-15 consumers reading output_path / chunk_count / notes keep working unchanged.

forgelm verify-audit

Audit-log chain integrity check.

{
  "success": true,
  "valid": true,
  "entries_count": 87,
  "hmac_verified": true,
  "errors": []
}
Key Type Notes
success bool true ⟺ valid: true.
valid bool false if any prev_hash mismatch / monotonicity break / seq gap.
entries_count int Number of non-blank lines the verifier read — not the number of entries that verified. On a failure it still reports the whole file, so do not read it as "entries you may trust".
hmac_verified bool | null null when no secret was configured (--hmac-secret-env unset or naming an unset variable — HMAC was not evaluated at all); true when a secret was supplied and the whole verification passed; false when a secret was supplied and verification failed for any reason, not necessarily an HMAC-specific one (the result object does not separate the two).
errors list[str] Empty on success, otherwise exactly one entry — the chain walk halts at the first failure. Where the failure belongs to a specific entry the string is formatted line <N>: <reason>, so the offending 1-based line number is parsed out of errors[0]; there is no separate line-number key in the envelope. Failures that are a property of the file as a whole — non-UTF-8 bytes, or an empty log with no genesis manifest — carry the bare reason with no line <N>: prefix, so a parser must tolerate its absence rather than assume it.

Exit code mapping: 0 = at least one entry was read and the SHA-256 chain (and the HMAC tags, when a secret was supplied) is intact; 6 = EXIT_INTEGRITY_FAILURE — the verifier compared something and it did not match (chain break, HMAC mismatch, genesis-manifest mismatch, an undecodable line, non-UTF-8 bytes inside the log, or a zero-entry log whose genesis manifest pins a first entry): a security event, not a config fix; 1 = nothing could be compared (--require-hmac with the named secret env var unset; a log path that does not exist / is not a regular file; or a log that exists, holds zero entries, and has no genesis manifest to say what it should have held); 2 = the log exists but could not be read through (permission denied, mid-read I/O failure — retryable).

A zero-entry log therefore never produces success: true. The envelope shape is unchanged — {success: false, valid: false, entries_count: 0, hmac_verified: null, errors: ["audit log at '…' exists but contains 0 entries, …"]} — so a consumer that branches on success needs no change; one that special-cased entries_count == 0 as a benign empty log does.

forgelm approve / forgelm reject

{
  "success": true,
  "run_id": "fg-abc123def456",
  "approver": "alice@example.com@workstation-7",
  "final_model_path": "/work/output/final_model",
  "promote_strategy": "rename"
}

promote_strategy is "rename" (same-device atomic os.rename) or "move" (cross-device shutil.move fallback).

approve exits 0 on success; reject exits 0 after recording the rejection (the staging dir is preserved for forensics). success: false with error on unknown run_id / config error.

forgelm purge

Three-mode dispatcher: --row-id, --run-id, or --check-policy. Wave 2b Phase 21 — GDPR Article 17 right-to-erasure.

Row-erasure success envelope — wet run (forgelm purge --row-id ROW --corpus PATH):

{
  "success": true,
  "mode": "row",
  "dry_run": false,
  "row_id_hash": "abc123...64-hex",
  "salt_source": "per_dir",
  "corpus_path": "/work/train.jsonl",
  "matches": 1,
  "first_line": 42,
  "bytes_freed": 142,
  "warnings": []
}

Row-erasure success envelope — dry run: identical shape minus bytes_freed (the rewrite is skipped); warnings: [] and dry_run: true.

Run-erasure success envelope — wet run (--run-id RUN --kind {staging,artefacts}):

{
  "success": true,
  "mode": "run",
  "kind": "staging",
  "dry_run": false,
  "run_id": "fg-abc123",
  "deleted": ["/work/output/final_model.staging.fg-abc123"],
  "bytes_freed": 1048576
}

Run-erasure success envelope — dry run: swap deleted for would_delete (same list-of-paths shape); omit bytes_freed.

Check-policy success envelope — retention block configured (--check-policy [--config PATH]):

{
  "success": true,
  "violations": [
    {
      "artefact_kind": "staging_dir[fg-abc123]",
      "path": "/work/output/final_model.staging.fg-abc123",
      "age_days": 14.7,
      "horizon_days": 7,
      "age_source": "audit"
    }
  ],
  "count": 1
}

Check-policy success envelope — no retention block (operator's config omits the retention: block, or no --config was supplied): the envelope drops count and adds a note field explaining the no-op:

{
  "success": true,
  "violations": [],
  "note": "No `retention:` block in the loaded config; nothing to enforce.  See `docs/guides/gdpr_erasure.md` for the schema."
}
Key Type Notes
success bool true for a successful operation; false (with error) on config / runtime failure.
mode (row/run) str Discriminator; "row" or "run". Absent on --check-policy.
kind (run) str "staging" or "artefacts". Run mode only.
dry_run bool Mirrors the --dry-run flag. Present in row + run envelopes; absent in --check-policy (the mode is read-only by definition).
row_id_hash str 64-character lowercase hex SHA-256 digest of salt + raw_value. Cleartext value never appears in the envelope. Note: the digest is emitted as plain hex (no sha256: prefix) so consumers can ==-compare against hashlib.sha256(...).hexdigest() directly.
salt_source str "per_dir" or "env_var" per FORGELM_AUDIT_SECRET toggle.
corpus_path str Absolute path to the JSONL corpus the operator passed via --corpus. Row mode only.
matches int Count of rows that matched the --row-id. Row mode only.
first_line int 1-based line number of the first matching row, captured before the rewrite. Row mode only.
bytes_freed int Bytes reclaimed by the deletion. Present on wet runs (row + run); absent on dry runs.
warnings list[str] Names of data.erasure_warning_* audit events emitted alongside the row erasure (memorisation / synthetic-data / external-copies). Row mode only.
would_delete (run dry) / deleted (run wet) list[str] Run mode: paths the dispatcher targeted for removal. Different key on dry vs wet so consumers can branch on shape.
violations list[object] --check-policy only. artefact_kind is one of audit_log, staging_dir, staging_dir[<run_id>], compliance_bundle, data_audit_report, raw_documents[...]. age_source ∈ {audit, mtime}.
count int --check-policy only when a retention: block is configured; equals len(violations). Absent in the no-retention-block branch (which adds note instead).
note str --check-policy only when no retention: block is configured. Operator-facing one-liner pointing at the GDPR guide.

Exit code mapping: 0 = success or successful policy report; 1 = config error (unknown row, missing corpus, conflicting flags, malformed --check-policy --config); 2 = runtime error (I/O, atomic rename failure).

forgelm reverse-pii

Wave 3 Phase 38 — GDPR Article 15 right-of-access companion to forgelm purge. Walks JSONL corpora and reports every line where the supplied identifier appears.

Success envelope:

{
  "success": true,
  "query_hash": "abc...64-hex",
  "identifier_type": "email",
  "scan_mode": "plaintext",
  "salt_source": "per_dir",
  "matches": [
    {
      "file": "/work/data/train.jsonl",
      "line": 42,
      "snippet": "…Contact: alice@example.com regarding the appointment…"
    }
  ],
  "files_scanned": [
    {"path": "/work/data/train.jsonl", "match_count": 1}
  ],
  "match_count": 1
}
Key Type Notes
query_hash str 64-character lowercase hex SHA-256 digest of salt + raw --query, where salt is the per-output-dir audit salt that forgelm purge also uses for target_id. The raw value never appears in the envelope; cross-tool correlation with purge's target_id works out of the box.
identifier_type str Echo of --type ∈ {literal, email, phone, tr_id, us_ssn, iban, credit_card, custom}. Default is literal — the query is matched as a literal substring.
scan_mode str "plaintext" (default — verbatim search; mask-leak detection) or "hash" (--salt-source set; SHA-256 of salt + query searched in the corpus).
salt_source str Which salt-resolution path was used: "plaintext" (no --salt-source), "per_dir" (per-output-dir salt file only), or "env_var" (XOR with FORGELM_AUDIT_SECRET). Mirrored in the audit event so a compliance reviewer can tell whether two digests were salted the same way.
matches list[object] One entry per line that matched. snippet is centred on the matched span and capped at 160 chars (… separator); the matched identifier is always preserved within the window.
files_scanned list[object] Per-file match counts, in glob-resolution order. The list is the source of truth for which paths the scan walked.
match_count int Equals len(matches); convenience for CI gates that branch on "any match."

Exit code mapping: 0 = scan completed (matches list may be empty); 1 = config error (empty --query, malformed regex on --type custom, empty glob expansion, --salt-source env_var without FORGELM_AUDIT_SECRET — or --salt-source per_dir with FORGELM_AUDIT_SECRET set); 2 = runtime error (mid-scan I/O failure, malformed UTF-8 corpus, custom-regex ReDoS timeout, audit-init failure when --audit-dir was explicit).

forgelm cache-models

Wave 2b Phase 35 — air-gap workflow blocker. Pre-populates the HuggingFace Hub cache.

Success envelope (forgelm cache-models --model M [--safety S] [--output DIR]):

{
  "success": true,
  "models": [
    {
      "name": "meta-llama/Llama-3.2-3B",
      "cached_path": "/work/hf_cache/models--meta-llama--Llama-3.2-3B",
      "size_bytes": 3221225472,
      "size_mb": 3072.0,
      "duration_s": 142.7
    }
  ],
  "total_size_mb": 3072.0,
  "cache_dir": "/work/hf_cache"
}
Key Type Notes
models list[object] One entry per --model; --safety (when supplied) appears as the last entry.
models[].cached_path str Path returned by huggingface_hub.snapshot_download (operator's --output or env-resolved HF_HUB_CACHE).
total_size_mb float Sum of every models[].size_mb.
cache_dir str Operator's --output, or env-resolved (HF_HUB_CACHE > HF_HOME/hub > ~/.cache/huggingface/hub).

Exit code mapping: 0 = every model cached; 1 = config error (no --model, malformed name); 2 = runtime error (Hub failure, disk-full, broken environment / missing core dep).

forgelm cache-tasks

Wave 2b Phase 35 — pre-populates the lm-evaluation-harness task dataset cache. Requires the [eval] extra.

Success envelope (forgelm cache-tasks --tasks CSV [--output DIR]):

{
  "success": true,
  "tasks": [
    {"name": "hellaswag", "cached": true, "error": null},
    {"name": "arc_easy", "cached": true, "error": null}
  ],
  "cache_dir": "/work/datasets_cache"
}
Key Type Notes
tasks list[object] One entry per task, with cached and error. Present on the failure envelope too, so a batch that exits non-zero still tells you which tasks are missing.
cache_dir str Operator's --output, or env-resolved (HF_DATASETS_CACHE > HF_HOME/datasets > ~/.cache/huggingface/datasets — note the Datasets chain, separate from the Hub chain).

A partial batch is a failure. If any task ends cached: false — a download error, or a task for which lm-eval exposes no downloadable dataset — the command emits success: false and exits 2, and the per-task rows come back under tasks in the error envelope. The cache on disk is incomplete and must not be transferred.

This is the point of the command: an air-gapped host has no second chance. The air-gap guide on GitHub gates on jq -e '.success' before bundling, and that gate is only worth running if a half-populated cache turns it red.

Changed after 0.11.0. Through 0.11.0, per-task download failures were reported in tasks[].error but the batch still returned success: true and exit 0, and the audit log recorded cache.populate_tasks_completed. A CI job following the documented example therefore packaged and shipped an empty dataset cache with a green gate. If you relied on the old behaviour to stage a known-incomplete set, read tasks[] from the exit-2 envelope instead of branching on success.

Exit code mapping: 0 = every task staged; 1 = config error (empty --tasks, unknown task, missing [eval] extra); 2 = one or more tasks not staged, or a broken environment.

Audit events: all staged → cache.populate_tasks_completed; some staged → cache.populate_tasks_partial; none staged → cache.populate_tasks_failed. The split exists so an auditor reading the append-only log can tell "nothing landed" from "the archive looks transferable and is missing data".

forgelm safety-eval

Wave 2b Phase 36 — standalone safety evaluation against a model checkpoint.

Success envelope (forgelm safety-eval --model M {--probes JSONL | --default-probes}):

{
  "success": true,
  "model": "/work/final_model",
  "classifier": "meta-llama/Llama-Guard-3-8B",
  "probes": "/work/probes.jsonl",
  "output_dir": "/work/eval",
  "max_safety_regression": 0.05,
  "passed": true,
  "safety_score": 0.93,
  "safe_ratio": 0.95,
  "category_distribution": {"S1": 0, "S2": 1, "S3": 0},
  "failure_reason": null
}
Key Type Notes
success bool Mirrors passed. success: false does NOT mean the dispatcher crashed — it means the model failed the safety gate.
max_safety_regression float The unsafe-ratio ceiling the verdict was produced against (--max-safety-regression, default 0.05). Echoed so a consumer reading passed: false can see which number safe_ratio was compared to.
passed bool true if safety_score and safe_ratio cleared the configured thresholds.
safety_score float | null Aggregate score from forgelm.safety.run_safety_evaluation.
safe_ratio float | null Proportion of probe responses the classifier judged safe.
category_distribution object Per-harm-category counts (empty when track_categories=False).
failure_reason str | null Human-readable reason from SafetyResult when passed: false.

Exit code mapping: 0 = thresholds passed; 1 = config error reached by the dispatcher (GGUF model path, probes file missing/unreadable); 2 = argparse usage error (missing --model, missing both --probes/--default-probes, or providing both together — exactly one of the two is required, so argparse rejects the request with exit 2 before the dispatcher runs), a runtime error (model load failure, classifier load failure, broken environment), or an evaluation that could not produce a verdict (see below); 3 = EXIT_EVAL_FAILURE — evaluation completed and the safety gate said no (operator-actionable: re-train or re-classify).

:::warn Exit 2 also means "the guard could not answer" — it is not evidence about the model. When the classifier never loaded, the probe file yielded no usable prompts, or too large a share of responses came back unscored, the result carries evaluation_completed=False and the command exits 2 rather than 3. This is deliberate: a regulated pipeline that retries on 2 and blocks deployment on 3 must not treat a transient Hub outage as "the model failed the safety gate".

The JSON envelope does not carry evaluation_completed, so the process exit code is currently the only discriminator between "broken guard" (2) and "gate said no" (3). Branch on $?, not on success. The on-disk safety_results.json and safety_trend.jsonl do carry evaluation_completed, alongside scored_unsafe_count and unscored_count — see Safety Evaluation. :::

forgelm verify-annex-iv

Wave 2b Phase 36 — EU AI Act Annex IV §1-9 artefact integrity check.

Success envelope (forgelm verify-annex-iv PATH):

{
  "success": true,
  "path": "/work/output/compliance/annex_iv_metadata.json",
  "valid": true,
  "missing_fields": [],
  "manifest_hash_actual": "e06b780a91150b1c56e9094a43f273ddd1f308acac13e896ed26747d9085220d",
  "manifest_hash_expected": "e06b780a91150b1c56e9094a43f273ddd1f308acac13e896ed26747d9085220d",
  "reason": "All Annex IV §1-9 fields populated; manifest hash matches."
}
Key Type Notes
valid bool true when all 9 §1-9 fields are present AND (when metadata.manifest_hash is present) the recomputed hash matches.
missing_fields list[str] Absent / empty required fields. Entries for system_identification sub-fields are dotted paths ("system_identification.provider_name"), not bare keys.
manifest_hash_actual str Recomputed canonical SHA-256 of the artefact-minus-metadata. Bare hex, no sha256: prefix; "" (empty string, not null) when absent.
manifest_hash_expected str Value extracted from the artefact's metadata.manifest_hash field. Same shape as above.
path str Absolute path of the verified artefact.
reason str One-line verdict — populated on success too, not only on failure.

There is no manifest_hash_present key. Earlier drafts of this page documented one; it is emitted nowhere in the codebase. Derive it yourself from manifest_hash_expected != "".

:::warn metadata.manifest_hash is an unkeyed SHA-256 from a public function. A matching hash means the artefact is internally consistent with its own stamp — not that it is authentic. Anyone who can write the file can edit it and re-stamp it, after which this envelope reports "valid": true. See Verify Annex IV for the full threat model. :::

Exit code mapping: 0 = valid: true; 1 = valid: false because the verifier never got as far as comparing a hash — required §1-9 fields missing or still holding template placeholders, or a root that is not a JSON object (operator-actionable: go and populate the artefact); 6 = EXIT_INTEGRITY_FAILURE — every required field was populated, the artefact carried a metadata.manifest_hash, and the recomputed hash disagrees with it (the document was edited after generation — treat as a security event); 2 = genuine runtime I/O failure on a reachable path (permission denied, mid-read I/O error). A file that is not found, is not a regular file, is malformed JSON, or is not valid UTF-8 exits 1, not 2 — those are operator input errors, and the envelope is the 2-key {"success": false, "error": "…"} form in every case.

forgelm verify-annex-iv --pipeline

Pipeline mode reads <run_dir>/compliance/pipeline_manifest.json and emits a different, 12-key envelope. It is the only surface that cross-checks the manifest against the audit log.

{
  "success": false,
  "mode": "pipeline",
  "path": "/work/checkpoints/pipeline_run",
  "violations": ["[audit-log corroboration] unattested (audit_log_absent): …"],
  "stages_total": 2,
  "stages_examined": 2,
  "evidence_verified": 2,
  "evidence_unverified": 0,
  "hash_state": "verified",
  "status_census": {"completed": 2},
  "stage_dispositions": [
    {"index": 0, "name": "sft_stage", "status": "completed", "disposition": "examined"},
    {"index": 1, "name": "dpo_stage", "status": "completed", "disposition": "examined"}
  ],
  "audit_corroboration": {
    "outcome": "unattested",
    "reason": "audit_log_absent",
    "events_examined": 0,
    "stages_asserted": 0
  }
}
Key Type Notes
success bool true only when violations is empty.
mode str Always "pipeline".
path str Absolute path of the run directory.
violations list[str] Findings, with internal routing prefixes stripped.
stages_total int Rows in the manifest — compare against stages_examined; a stage cannot be hidden by flipping its status without the census showing it.
stages_examined int Stages whose Annex IV evidence was parsed.
evidence_verified / evidence_unverified int Stage artefacts hash-checked vs. reached-but-unattested.
hash_state str "verified" — the manifest matched its own stamp; "absent" — no manifest_hash, so nothing attested to it.
status_census object {status: count} across all rows.
stage_dispositions list[object] Per stage: index, name, status, disposition.
audit_corroboration object | null outcome (corroborated / contradicted / unattested), reason, events_examined, stages_asserted. null on pre-flight paths where no manifest was parsed.

audit_corroboration.outcome: "unattested" is never a pass. It means no keyed record backed the manifest's claims. With no FORGELM_AUDIT_SECRET set — or no audit log present — this is the outcome you get, and hash_state: "verified" alongside it means only "the file agrees with its own unkeyed stamp". Gate CI on outcome == "corroborated", not on hash_state.

Exit code mapping (integrity first, so a weaker finding never masks a stronger one): 0 = no violations; 6 = EXIT_INTEGRITY_FAILURE — a structural, chain-integrity or per-stage-evidence check failed; 2 = EXIT_TRAINING_ERROR — the manifest or a stage artefact exists but could not be read (retryable I/O); 1 = EXIT_CONFIG_ERROR — the manifest is absent or unparseable, or the verifier reached the evidence and nothing attested to it (including an unattested corroboration).

forgelm verify-gguf

Wave 2b Phase 36 — GGUF model file integrity check.

Success envelope (forgelm verify-gguf PATH):

{
  "success": true,
  "path": "/work/exports/model.q4_k_m.gguf",
  "valid": true,
  "reason": "GGUF magic OK, metadata parsed, SHA-256 sidecar match",
  "checks": {
    "magic_ok": true,
    "metadata_parsed": true,
    "sidecar_present": true,
    "sidecar_match": true,
    "sha256_actual": "abcd1234...",
    "sha256_expected": "abcd1234..."
  }
}
Key Type Notes
valid bool true when the magic header passes AND any attempted check (metadata block, SHA-256 sidecar) succeeded. A skipped check (e.g. metadata_parsed: false because the optional gguf package is absent) does NOT force valid: false — only an attempted-and-failed check does.
checks.magic_ok bool First 4 bytes equal b"GGUF".
checks.metadata_parsed bool true when the gguf metadata block was successfully parsed; false when the block is corrupted OR when the optional gguf package is absent / skipped. A false value alone does NOT force valid: false — corruption sets reason and rejects, but the absent-package case leaves valid unaffected.
checks.sidecar_present bool true when <path>.sha256 exists.
checks.sidecar_match bool | null true on byte-for-byte match; false on mismatch or malformed sidecar; null when no sidecar. A malformed sidecar (empty / non-hex / wrong-length) fails closed.
checks.metadata_error str Present only when metadata_parsed: false — carries the parse exception (e.g. "ValueError: Sorry, file appears to be version 0 which we cannot handle"). This is the field that tells you whether a metadata_parsed: false is a too-old gguf package or genuine corruption.
reason str One-line summary; carries the failure detail on valid: false.
path str Absolute path of the verified file (top-level, alongside checks).

Exit code mapping: 0 = valid: true; 1 = valid: false with nothing actually compared — a magic-header mismatch (the file is not a GGUF at all: a file-type verdict, not a tamper verdict), a sidecar whose contents are not a 64-character hex digest (unusable, so the check fails closed), or a metadata-parse failure on a file whose SHA-256 sidecar matched (the checksum has proven the bytes are byte-identical to what was exported, so the likely cause is a gguf package too old for this file's format revision, not tampering); 6 = EXIT_INTEGRITY_FAILURE — a well-formed sidecar digest that disagrees (the file changed after export, and a checksum mismatch dominates even if the metadata block also failed to parse), or a metadata block that could not be parsed with no usable sidecar available to rule out corruption; 2 = genuine runtime I/O failure on a reachable path (permission denied, mid-read I/O error) — a file that is simply not found exits 1, not 2. The optional-gguf-package-missing path stays at valid: true + exit 0 (operator's "metadata check skipped" — the magic header + SHA-256 sidecar checks remain the load-bearing integrity surface).

forgelm verify-integrity

Wave 2b Phase 36 / Art. 15 — model-directory artifact integrity check.

For a fresh, separate model directory, training finalization hashes the model card and deployer_instructions.md along with the saved model/tokenizer files before writing the manifest. The success example illustrates six recorded artifacts; the count depends on the saved files. Manifest generation emits model.integrity_verified; this read-only verifier emits no audit event.

Success envelope (forgelm verify-integrity MODEL_DIR):

{
  "success": true,
  "valid": true,
  "reason": "All 6 recorded artifact(s) present and unchanged.",
  "changed": [],
  "removed": [],
  "added": [],
  "verified_count": 6,
  "path": "/work/output/my-model"
}

Mismatch envelope (valid: false, exit 6 — the manifest parsed, the walk ran, and at least one artifact disagrees; an unusable manifest exits 1 with the same envelope shape):

{
  "success": false,
  "valid": false,
  "reason": "Model artifacts do not match model_integrity.json: 1 changed, 1 removed.",
  "changed": ["adapter_model.safetensors"],
  "removed": ["tokenizer.model"],
  "added": [],
  "verified_count": 10,
  "path": "/work/output/my-model"
}

Runtime-error envelope (exit 2):

{
  "success": false,
  "error": "Could not verify model integrity for '/work/output/my-model': [Errno 13] Permission denied: '...'"
}
Key Type Notes
success bool true when valid: true; false on any mismatch or error.
valid bool true when every recorded artifact is present and its SHA-256 matches; false otherwise.
reason str Human-readable summary — number of clean artifacts on success, mismatch counts on failure.
changed list[str] Relative paths of artifacts whose SHA-256 no longer matches the manifest.
removed list[str] Relative paths of artifacts recorded in the manifest that are absent on disk.
added list[str] Relative paths of on-disk files not recorded in the manifest.
verified_count int Number of artifacts that matched the manifest successfully.
path str Absolute path to the model directory that was verified.

Exit code mapping: 0 = all recorded artifacts present and unchanged (valid: true); 6 = EXIT_INTEGRITY_FAILURE — integrity mismatch (changed / removed / added file): the manifest parsed, the walk ran, and the deployed weights are not the weights that were signed off; 1 = operator / input error, where nothing was ever hashed (missing path, path is a file not a directory, manifest not found, malformed JSON, manifest root is not a JSON object, no artifacts key, non-list artifacts, empty artifacts — a manifest that records nothing attests to nothing — an entry that is not an object, an entry whose file is not a string, or a manifest entry path that escapes the model directory: the verifier refused to hash an out-of-tree file, which is an "I did not examine your weights" verdict, not a tamper verdict); 2 = genuine runtime I/O failure on a reachable path (read error, permission denied mid-walk).

The runtime-error envelope (exit 2) emits only {"success": false, "error": "…"} — no valid, changed, removed, added, or path keys. Branch on success first, then inspect valid and the diff lists.

forgelm export

GGUF / merged-weights export for the post-training handoff.

Envelope (forgelm export MODEL --output OUT.gguf):

{
  "success": true,
  "output_path": "/work/exports/model.q4_k_m.gguf",
  "format": "gguf",
  "quant": "q4_k_m",
  "sha256": "abcd1234...",
  "size_bytes": 4815162342,
  "error": null
}
Key Type Notes
output_path str | null Path to the written artefact; null on failure.
format str | null Export format (e.g. gguf, merged).
quant str | null Quantisation level used (single value — --quant takes one level per invocation).
sha256 str | null SHA-256 over the written artefact; null on failure.
size_bytes int | null Artefact size in bytes; null on failure.
error str | null null on success; operator-facing message on failure. Branch on success, not on error presence.

Exit code mapping: 0 = export succeeded; 2 = runtime failure (conversion error, missing [export] extra, unreachable model path).

forgelm deploy

Deployment-config generation for Ollama / vLLM / TGI / HF Endpoints.

Envelope (forgelm deploy MODEL --target ollama --output Modelfile):

{
  "success": true,
  "target": "ollama",
  "output_path": "/work/deploy/Modelfile",
  "error": null
}
Key Type Notes
target str | null Deployment target (ollama, vllm, tgi, hf-endpoints).
output_path str | null Path to the generated config; null on failure.
error str | null null on success; operator-facing message on failure. Branch on success, not on error presence.

Exit code mapping: 0 = config generated; 1 = caller-input error (unsupported target, model_path is not a local directory for the ollama/tgi targets); 2 = runtime failure (unwritable output path, template render error).

forgelm quickstart

Template → config → train → chat one-shot. Three JSON shapes.

Template list (forgelm quickstart --list --output-format json):

{
  "success": true,
  "templates": [
    {
      "name": "customer-support",
      "title": "Customer support assistant",
      "description": "Fine-tune a helpful support agent.",
      "primary_model": "Qwen/Qwen2.5-7B-Instruct",
      "fallback_model": "HuggingFaceTB/SmolLM2-1.7B-Instruct",
      "trainer_type": "sft",
      "estimated_minutes": 20,
      "min_vram_for_primary_gb": 16,
      "bundled_dataset": "customer_support.jsonl",
      "license_note": null
    }
  ],
  "count": 1
}

Generation success (forgelm quickstart TEMPLATE [--dry-run] --output-format json):

{
  "success": true,
  "template": "customer-support",
  "config_path": "/work/quickstart/config.yaml",
  "model": "Qwen/Qwen2.5-7B-Instruct",
  "dataset": "/work/quickstart/customer_support.jsonl",
  "selection_reason": "primary model fits available VRAM",
  "dry_run": false,
  "chat_launched": false,
  "notes": []
}

Training failure (training subprocess returned non-zero):

{
  "success": false,
  "error": "Training subprocess failed with exit code 3.",
  "exit_code": 3,
  "template": "customer-support",
  "config_path": "/work/quickstart/config.yaml"
}
Key Type Notes
templates list[object] --list only. One entry per registered template; count == len(templates).
template str The selected template name.
config_path str Path to the generated YAML config.
model str Resolved model id (primary or VRAM-driven fallback).
dataset str Path to the bundled / overridden dataset.
selection_reason str Why model was chosen.
dry_run bool true when --dry-run (config written, no training).
chat_launched bool Always false in JSON mode — the interactive chat REPL is suppressed so human prose never interleaves with the JSON envelope on stdout. Run forgelm chat <model_path> manually.
notes list[str] Advisory notes (e.g. license caveats).
exit_code int Training-failure envelope only — the mapped public exit code the training subprocess returned.

Exit code mapping: 0 = generated (and, in text mode without --no-chat, the chat REPL ran); 1 = caller-input error (missing/invalid template, unwritable output dir); the training-failure envelope's exit_code (2/3/4) is propagated verbatim when the auto-train subprocess fails.

Adding a new subcommand

A new subcommand that supports --output-format json MUST land with:

  1. The envelope documented in this page (EN + TR mirror).
  2. A test in tests/test_json_envelope_contract.py (or a per-subcommand test file) that pins the exact set of top-level keys.
  3. The per-collection key follows the convention "results live under a key named after the subcommand's primary noun" (so doctor → checks, approvals --pending → pending, etc.).

Renaming a key after merge is a MAJOR-version bump per the release standard on GitHub.

See also

  • Exit Codes — the contract success: bool aligns with.
  • error-handling.md — the error envelope contract (GitHub source).
  • release.md — when JSON renames count as breaking (GitHub source).