Repository navigation
docs(wave6): canonical-name drift sweep + 5 new CI guards (audit closure) - #39
Conversation
…e/usermanuals
Closes Wave A (mechanical) + Wave B (canonical-name drift) of the
2026-05-07 docs audit plan. ≈240 of the original ≈270 audit findings
remain after this sweep — the deferred bulk is P1 (parallel-universe
YAML schema) which needs the Wave D render-from-code pipeline.
Mechanical (Wave A):
- pipeline.reverted → model.reverted across 12 docs (21 lines)
- ForgeLM 0.5.2 → 0.5.5 in 4 usermanual artefacts
- README.md:59 forgelm purge --subject-id → --row-id --corpus
- CHANGELOG bilingual-pair count 39 → 40 (verified _PAIRS = 40)
- CLAUDE.md / CONTRIBUTING.md / docs/standards/testing.md test-module
count 68 → 70 (verified find tests/test_*.py | wc -l = 70)
- TR typos: 'Sıraki' → 'Sonraki' in installation + first-run
Canonical-name drift (Wave B):
- P2: compliance.human_approval → evaluation.require_human_approval
(10 user-facing files; ComplianceMetadataConfig has no
human_approval, EvaluationConfig.require_human_approval is the
real field per forgelm/config.py:480)
- P5: forgelm audit --add-row-ids reframed as Phase 28 backlog item
(5 files); flag does not exist in current parser
- P7: webhook event vocabulary synced to canonical 5-event surface
(training.start/success/failure/reverted, approval.required) in
4 user-facing tables; webhook lifecycle method names corrected
(notify_started/succeeded/failed → notify_start/success/failure)
in 6 QMS/design/reference docs
- P8: model.integrity_verified row added to audit_event_catalog
(EN+TR); was emitted at trainer.py:1450 but undocumented
Ghost-field cleanup:
- webhook.secret_env removed from 5 files (no such field in
WebhookConfig; webhook.py does not HMAC-sign bodies)
- encryption_at_rest: true|false claim removed from 4 files
(DataGovernanceConfig has no encryption field)
QMS reconciliation:
- statement_of_applicability column header semantics clarified in
EN+TR (operator-side vs ForgeLM-evidence distinction); resolves
the apparent SoA ↔ iso27001_control_mapping conflict on 10 A.5.x
controls
TR mirror parity:
- operations/cicd-tr.md GitLab stages: added missing - deploy
- data/audit-tr.md JSON schema expanded ({...} → full)
- concepts/data-formats-tr.md KTO JSON formatted to match EN
Earlier review-comment fixes folded in (from inline review
verification):
- site/css/style.css:2060,2328 currentColor → currentcolor
(lowercase per CSS spec / Stylelint canonical form)
- site/quickstart.html:363-365 wizard modal architectural comment
reflects actual implementation (hidden + sibling aria-hidden;
no inert)
Validated: bilingual_parity ✓, anchor_resolution ✓,
cli_help_consistency ✓ (all --strict).
Deferred to Wave D:
- P1 parallel-universe YAML schema (~22 CRITICAL + 27 HIGH pages) —
needs render-from-code via tools/regenerate_yaml_templates_doc.py
- compliance.config_hash reframe (>10 files; nuanced rewrite)
- usermanuals training/{kto,orpo,grpo}-tr.md output: blocks (EN
itself uses ghost output: block; full canonical rewrite belongs
in P1 lift)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wave C of the 2026-05-07 docs audit plan. Rewrites every CRITICAL
finding in `docs/reference/*_subcommand*.md` and the related
`usermanuals/{en,tr}/reference/{cli,exit-codes}.md` against the live
parser + handler surface.
Per-subcommand JSON envelope corrections (EN + TR):
- **verify_audit**: doc claimed exit 2 for `--require-hmac` env-var
unset; code returns 1 (`EXIT_CONFIG_ERROR` covers both option errors
and integrity failures during the v0.5.5 stabilisation cycle —
`_verify_audit.py:18-27`). Exit-code table reduced to 0/1; v0.6.x
deprecation note added.
- **safety_eval**: bundled probe set advertised as "50 prompts × 14
categories" — actual is 51 × 18 (verified via wc -l +
category dedup over `forgelm/safety_prompts/default_probes.jsonl`).
Output artefact filenames `safety_report.json` / `safety_examples.jsonl`
/ `safety_run.log` were fabricated; real names are
`safety_results.json` (`forgelm/safety.py:417`) and
`safety_trend.jsonl` (`forgelm/safety.py:686`). Phantom `[eval]`
extra dependency on exit-1 path removed (transformers ImportError
routes to EXIT_TRAINING_ERROR per `_safety_eval.py:154-155`).
- **approve / approvals**: envelope keys re-aligned to
`_approve.py:468-477` and `_approvals.py:240-249,440-444`
respectively. `promoted_to` -> `final_model_path`;
`events`/`staging` -> `chain`/`staging_contents`;
`age` (string)/`staging` (string) -> `age_seconds` (int)/`staging_exists`
(bool). Asymmetric missing-log behaviour documented:
`--pending` returns 0 on missing log, `--show` returns 1.
- **purge**: three-mode envelope shape now matches code (row-mode
doesn't carry an explicit `success` field — `mode` is the success
discriminator; run-mode `deleted` is a list, not the row-mode
string). Violation key renamed `kind` -> `artefact_kind` to match
`_purge.py:1067-1073`.
- **reverse_pii**: match keys re-aligned (`path`/`preview` ->
`file`/`snippet`); `files_scanned` is a list of `{path, match_count}`
objects, not an integer. Exit-code table updated: explicit
`--audit-dir` unwritable returns EXIT_TRAINING_ERROR (2), not 1
(`_reverse_pii.py:537,573`).
Usermanuals reference page corrections (EN + TR):
- **data/audit.md**: 11+ ghost flags removed (`--strict`, `--skip-pii`,
`--skip-secrets`, `--skip-quality`, `--skip-leakage`,
`--remove-duplicates`, `--remove-cross-split-overlap`,
`--output-clean`, `--show-leakage`, `--minhash-jaccard`,
`--minhash-num-perm`, `--dedup-algo`, `--dedup-threshold`,
`--sample-rate`, `--add-row-ids`); table rebuilt from
`_parser.py:335-468`. The note about removed flags points operators
at the canonical alternatives.
- **reference/cli.md**: same ghost-flag scrub on the `forgelm audit`
synopsis block; full per-flag table delegated to the data/audit
page so a single source-of-truth.
- **reference/exit-codes.md**: ghost `audit --strict` exit-2 row
removed; `verify-audit` chain-break behaviour (exit 1 during v0.5.5
cycle) added; "exit 0 guarantees" closing event renamed
`run_complete` -> `pipeline.completed` (canonical).
P8 catalog row already added in Wave B (commit 2ba095f).
yaml-templates.md deferred to Wave D where it merges with the P1
parallel-universe-YAML rewrite.
All three doc guards green:
- `tools/check_bilingual_parity.py --strict` (40 EN+TR pairs)
- `tools/check_anchor_resolution.py --strict` (268 markdown files)
- `tools/check_cli_help_consistency.py --strict` (423 forgelm invocations)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wave D Part 1 of the 2026-05-07 docs audit plan. Closes the
parallel-universe YAML schema (P1) on the trainer-spec pages and the
two highest-impact data/concept pages, plus the TR judge.md identifier
revert flagged by the audit.
Trainer pages (sft, dpo, simpo, kto, orpo, grpo — EN + TR ×6):
- Top-level `datasets:[{path, format}]` array → singular `data: {
dataset_name_or_path: ... }` (matches `ForgeConfig.data:` block).
- `training.trainer` → `training.trainer_type` (canonical literal).
- `training.epochs` / `training.batch_size` →
`training.num_train_epochs` / `training.per_device_train_batch_size`.
- Nested `training.{dpo, simpo, kto, orpo, grpo}: { ... }` blocks →
flat `training.{dpo_beta, simpo_beta, simpo_gamma, kto_beta,
orpo_beta, grpo_num_generations, grpo_max_completion_length,
grpo_reward_model}` fields (matches `forgelm/config.py:163-218`).
- Top-level `output: { dir: ... }` block → `training.output_dir: ...`
scalar (no `output:` block exists).
- Removed `model.load_in_4bit: true` ghost (replaced by `model.backend:
"transformers"|"unsloth"`).
- Removed `training.scheduler: "cosine"` and
`training.loss_on_completions_only` ghosts (no such fields in
TrainingConfig; HF TRL handles these defaults internally).
- DPO worked example: `compliance.human_approval` →
`evaluation.require_human_approval`; `safety.model:` →
`safety.classifier:`; `safety.block_categories: [list]` →
`safety.track_categories: bool` + `safety.severity_thresholds:
{dict}`; `evaluation.benchmark.floors: {per-task dict}` →
`evaluation.benchmark.min_score: scalar`.
- KTO note added: `desirable_weight` / `undesirable_weight` are TRL-side
defaults, not surfaced as ForgeLM config fields (Phase 28+ backlog).
- GRPO note added: TRL-side `beta` not surfaced as flat field; built-in
format/length reward shaping is always-on fallback
(`forgelm/grpo_rewards.py`).
- KTO dataset format: `prompt + response + label` →
`prompt + completion + label` (matches `_detect_dataset_format` at
`forgelm/data.py:13-29`; also fixed in concepts/data-formats EN+TR).
concepts/data-formats.md (EN + TR):
- KTO format auto-detection field: `response` → `completion`.
data/secrets.md (EN + TR):
- 11-category claim → real 9 families (`aws_access_key`,
`github_token`, `slack_token`, `openai_api_key`, `google_api_key`,
`jwt`, `openssh_private_key`, `pgp_private_key`,
`azure_storage_key`). Anthropic / Stripe / SendGrid / Twilio
references removed (no per-vendor patterns in
`forgelm/data_audit/_secrets.py:_SECRET_PATTERNS`); operators with
those traffic types extend out-of-tree.
- Redaction string `[REDACTED-PEM-BLOCK]` claim removed (real:
`[REDACTED-SECRET]` per `_secrets.py:106`).
evaluation/judge-tr.md:
- Reverted translated rubric identifiers (`yardımseverlik` / `ton` /
`doğruluk`) to canonical `helpfulness` / `tone` / `factual_accuracy`
per docs/standards/localization.md (CLI / code / config-key names
are not translated). Added `rubric_distributions` to the JSON
example envelope.
Validated: bilingual_parity ✓, anchor_resolution ✓,
cli_help_consistency ✓.
Wave D Part 2 (yaml-templates.md, evaluation pages, config_hash
reframe) and Part 3 (safety_compliance-tr 38-section translate) follow
in subsequent commits.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wave D Part 2 of the 2026-05-07 docs audit plan.
usermanuals/{en,tr}/reference/yaml-templates.md (full rewrite):
- Refocused as a thin layer over the canonical
`forgelm/templates/<name>/config.yaml` files; readers are pointed at
`forgelm quickstart <name>` to materialise the source-of-truth.
- Three worked examples in canonical schema (customer-support SFT,
SFT→DPO chain with Article 14 + safety/benchmark gates, GRPO math).
- Added a "removed templates / fields" table mapping every old ghost
key to its canonical replacement, to defend against operators
copying from cached / older versions of the page.
compliance.config_hash reframe (15 user-facing files):
- Replaced `compliance.config_hash` (presented as an audit-event
payload field) with `config_hash` (per-run manifest sidecar field)
across docs/{design,guides,qms,reference,usermanuals}/. No event
emitter currently writes a `config_hash` field on its payload — the
legacy claim was a doc-only fabrication. The Approvals subcommand
passively forwards it from upstream events when present
(`_approvals.py:248`), and the per-run manifest sidecar carries the
hash in code paths that have it; until a producer is wired Phase
28+, the doc no longer claims it ships on the wire.
Files touched: yaml-templates.md (EN+TR rewrite); design/iso27001…,
guides/iso_soc2_deployer_guide{,.tr}, qms/{encryption_at_rest{,.tr},
sop_incident_response{,.tr}, statement_of_applicability{,.tr}},
reference/{iso27001_control_mapping{,.tr},soc2_trust_criteria_mapping{,.tr}},
usermanuals/{en,tr}/operations/iso-soc2-deployer (config_hash sed).
Validated: bilingual_parity ✓, anchor_resolution ✓,
cli_help_consistency ✓ (425 forgelm invocations).
Wave D Part 3 (safety_compliance-tr 38-section translate) and the
remaining evaluation pages (benchmark, judge full rewrite, safety,
auto-revert, trend-tracking) still pending.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wave D Part 3 of the 2026-05-07 docs audit plan.
docs/guides/safety_compliance-tr.md is no longer a 38-section stub
("Tam içerik için EN'e bakın"). Translates the full 493-line EN
content into idiomatic Turkish (547 lines after natural Turkish
wrap), preserving:
- Section structure (5 H2 + 33 H3/H4 — bilingual_parity guard verifies
same spine).
- Code blocks, YAML keys, JSON fields, audit-event names, exit codes
in English (per docs/standards/localization.md: CLI / code /
config-key names are not translated).
- Probe-set counts updated to canonical 30 / 25 / 25 (matching
configs/safety_prompts/{general_safety,bias_discrimination,
jailbreak_attempts}.jsonl as verified during this audit).
- Llama Guard S1-S14 category labels translated to Turkish
prose, codes preserved.
The "translate eventually" pointer to the EN file is retired; the TR
mirror now carries the same operational depth as EN.
Validated: bilingual_parity ✓ (40 pairs, full content match on
heading spine), anchor_resolution ✓ (268 files), cli_help_consistency
✓ (427 forgelm invocations).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wave E of the 2026-05-07 docs audit plan. Closes the post-audit
"prevention layer" — five new tools/check_*.py guards that catch every
drift family the audit identified, before it ships. Each guard mirrors
an audit finding pattern and is wired to be added to ci.yml alongside
the existing bilingual_parity / anchor_resolution / cli_help_consistency
gauntlet.
New guards (all support `--strict --quiet`):
1. **`tools/check_yaml_snippets.py`** — extracts every fenced `yaml`
block from `docs/`, validates the complete (model+training+data)
ones against `forgelm.config.ForgeConfig`. Skips fragments and
intentional sketches (`...` placeholders). Catches P1 — the
parallel-universe YAML schema. Today: 20 ForgeConfig-shaped
snippets validated across 333 total yaml blocks.
2. **`tools/check_audit_event_catalog.py`** — scans forgelm/ for
dotted-namespace event literals (catches both direct
`event="foo.bar"` and constant-indirection
`_EVT_FOO = "foo.bar"`), diffs against
`docs/reference/audit_event_catalog.md` rows in both directions.
Filters filename-shaped matches via
`_NON_EVENT_SECOND_SEGMENTS`. Catches P8 (missing-from-catalog)
AND ghost rows (`audit.cross_run_continuity`). Today: 36
emitted ↔ 36 catalogued.
3. **`tools/check_library_api_doc.py`** — diffs `forgelm.__all__`
against `forgelm.X` rows in `library_api_reference.md`.
Catches the audit's `compute_minhash` missing-row finding.
Method-style rows (`forgelm.ForgeTrainer.train`) are checked
against the parent class. Today: 30 ↔ 30 + 2 method rows.
4. **`tools/check_doc_numerical_claims.py`** — scrapes four canonical
counts from code (secret families, trainer types, templates,
webhook events) and diffs against doc claims via qualifier-anchored
regexes ("9 secret families", "5 webhook events", …). Patterns
tightened to require qualifier so generic numbers don't false-
positive (e.g. "Six events" without "webhook" qualifier doesn't
match the webhook rule). Today: 9 / 6 / 5 / 5.
5. **`tools/check_bilingual_code_blocks.py`** — extends the existing
`check_bilingual_parity.py` (which only checks H2/H3/H4 heading
spine) to:
- Same fenced-block count per pair.
- For yaml blocks at the same ordinal position, same set of
top-level keys.
Catches the audit's S3 pattern (TR collapsing multi-line YAML to
flow-style) and the translated-identifier pattern (S2). Today:
40 pairs, all matching.
Doc fixes folded in (caught by the new guards on first run):
- `library_api_reference.md` (EN+TR): added missing
`forgelm.compute_minhash` row.
- `audit_event_catalog.md` (EN+TR): added 5 missing rows
(`pipeline.initialized`, `training.oom_recovery`,
`benchmark.evaluation_completed`, `safety.evaluation_completed`,
`judge.evaluation_completed`); removed ghost
`audit.cross_run_continuity` row.
- `usermanuals/{en,tr}/training/{simpo,kto,orpo,grpo}.md`: added
back the `lora:` block I'd dropped in Wave D (regression caught
by gate 1 — the YAML snippets failed validation against the
required `lora` field in ForgeConfig).
All 8 guards (3 pre-existing + 5 new) pass `--strict`:
- `check_bilingual_parity.py` ✓ (40 pairs, H2/H3/H4 spine)
- `check_anchor_resolution.py` ✓ (268 markdown files)
- `check_cli_help_consistency.py` ✓ (427 forgelm invocations)
- `check_yaml_snippets.py` ✓ (20 ForgeConfig snippets validated)
- `check_audit_event_catalog.py` ✓ (36/36 events)
- `check_library_api_doc.py` ✓ (30/30 symbols)
- `check_doc_numerical_claims.py` ✓ (9/6/5/5 counts)
- `check_bilingual_code_blocks.py` ✓ (40 pairs, code-blocks + yaml keys)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wave F of the 2026-05-07 docs audit plan. Closes the residual R1-R33 inventory after Wave A-E left ≈33 unaddressed items, and the user- identified gap of 3 missing TR mirrors (cicd_pipeline, quickstart, troubleshooting). 48 files changed: 43 modified + 5 new TR mirrors. All 8 doc guards green (3 pre-existing + 5 Wave-E gates). Group A — mechanical sed/text fixes: - pii-ml.md (EN+TR) line 126 v0.5.0 → v0.5.5 anchor. - CHANGELOG.md "8 new pages" / "16 new bilingual" → "9 new pages" / "17 new bilingual" (matches the actual _PAIRS registry post-Wave-F). - ingestion.md "≈10 prefix-anchored patterns" → "9". - safety_compliance.md (EN) probe count `general_safety: 20 / bias: 15 / jailbreak: 15` → canonical 30 / 25 / 25 (matches configs/safety_prompts/*.jsonl wc -l verified during audit). - quickstart.md `forgelm --data-audit ...` → canonical `forgelm audit ...` subcommand. Group B — existing-doc edits (R1, R4, R5, R7-R15, R18-R22): - architecture.md (EN+TR): flat-module diagram + directory tree rewritten for Phase 14/15 split (cli/ + data_audit/ packages); 21 Pydantic models (was 19); ~70 test modules (was 800+); 18 subcommand list replaces the ghost "13 CLI flags" claim. - library_api_reference.md (EN+TR) Example #3 unrunnable: model.name → name_or_path; top-level dataset → data; num_epochs → num_train_epochs; batch_size → per_device_train_batch_size; added required `lora:` block. Added explanatory note on which keys are Pydantic-required. - guides/library_api.md (EN+TR): notify_training_start → notify_start; report.duplicate_count → report.near_duplicate_summary ["pairs"]; report.pii_findings → report.pii_summary; config.dataset.path → config.data.dataset_name_or_path; config.output_dir → config.training.output_dir. - configuration.md (EN+TR): `rope_scaling` type fixed (string/2-value → Optional[Dict]/4-value: linear, dynamic, yarn, longrope); `webhook.timeout` default 5 → 10 with v0.5.5 raise rationale. - performance.md (EN+TR) top-level `ingestion:` YAML block (no such key in ForgeConfig) replaced with the canonical CLI flag form. - design/blackwell_optimized.md: "Status: Not yet implemented (Phase 28+ backlog)" banner so the doc no longer reads as fait accompli. - design/wizard_mode.md: rewritten to reflect the shipped 8-step flow (was 6-step sketch); pure-stdlib `input()` (was "rich/questionary"); documented the 0.5.5 implementation as source of truth. - design/gdpr_erasure.md: §3.2 RetentionConfig 4 → 5 fields (added `raw_documents_retention_days`); §4.2 `--yes` flag re-framed as Phase 28+ backlog (the shipped purge is non-interactive); §279 api.py → _version.py. - design/iso27001_soc2_alignment.md: §13 Faz 23 acceptance checkboxes marked complete (`[x]` × 30; previously stale `[ ]` despite §17 saying "Implemented in v0.5.5"); A.8.29 "1300+ tests" → "~70 test modules". - design/library_api.md: `forgelm/api.py` references → `forgelm/_version.py` (the implementation chose the latter; design doc was stale). - distributed_training.md (EN+TR): `zero3_offload` "CPU only" → "optimizer state + parameters → CPU"; `fsdp_offload` "Offload parameters" → "Offload parameters AND gradients (between forward/backward)". - iso27001_control_mapping-tr.md: section headers translated to Turkish (A.5 Organizasyonel kontroller, A.6 İnsan kaynağı, A.7 Fiziksel, A.8 Teknolojik) — was English in TR file. - supply_chain_security.md (EN+TR): `gh release download v0.5.5` → `v<X.Y.Z>` placeholder so the example doesn't drift on every release. - qms/risk_treatment_plan.md (EN+TR) R-12 row F-compliance-110 misattribution: clarified that F-compliance-110 enforces evaluation.safety.enabled (not require_human_approval — see R-04). - guides/gdpr_erasure.md (EN+TR): row-mode-only qualifier added to `data.erasure_warning_memorisation` and `..._synthetic_data_present` rows; `..._external_copies` already covers both modes. - reference/data_preparation.md (EN+TR): added scope-pointer block redirecting operators to the modern pipeline (ingestion guide, audit guide, dataset-formats reference, prepare_dataset Python API). Group C — new TR mirrors (R2, R3 of the plan): - compliance_summary-tr.md (NEW) — 207 lines, full mirror of EN compliance_summary; ISO control count corrected to "59 of 93 (FL 11 + FL-helps 48)" matching the canonical mapping. - alignment-tr.md (NEW) — 240 lines, full mirror of EN alignment. Group D — usermanuals operations + getting-started ghost cleanup: - operations/experiment-tracking.md (EN+TR): `report_to` is a single Literal, not a list; per-backend nested config blocks (`wandb: { project: ... }`, `mlflow: { tracking_uri: ... }`, `tensorboard: { log_dir: ... }`, `comet_ml: { api_key: ... }`) all removed — backends configured via env vars per HF Transformers convention; `training.tags` and `log_grad_norm` ghost fields removed. - operations/vram-fit-check.md (EN+TR): `--gpus N` ghost flag removed (estimator probes nvidia-smi); `--fit-check-strict` ghost flag reframed as calibration-training pattern. - operations/gpu-cost.md (EN+TR): `--estimate-cost` ghost flag reframed as calibration-then-extrapolate pattern using the per-run compliance_report.json `gpu_hours`. - getting-started/project-layout.md (EN+TR): `extends:` YAML inheritance ghost (no such ForgeConfig field) replaced with YAML anchor (`&base` / `<<: *base`) recommendation. Group E — user-identified missing TR mirrors: - cicd_pipeline-tr.md (NEW) — 240 lines, full TR mirror of EN cicd_pipeline; webhook payload "training.awaiting_approval" ghost fixed to canonical `approval.required` in BOTH EN and TR (P7). - quickstart-tr.md (NEW) — 290 lines, full TR mirror; synthetic ghost fields (`teacher_api_key_env`, `num_samples`) replaced with canonical `api_key_env` + `output_format` in BOTH EN and TR (P6). - troubleshooting-tr.md (NEW) — 360 lines, full TR mirror; synthetic ghost timeout/retry/batch fields fixed to canonical `api_timeout`, `api_delay`, `max_new_tokens` in BOTH EN and TR (P6). `tools/check_bilingual_parity.py::_PAIRS` grew from 40 → 45 (+5 new mirrors registered). CLI consistency caught a regression my Wave D wizard_mode.md rewrite introduced: `forgelm wizard` is not a registered subcommand — the wizard is invoked via the top-level `--wizard` flag. Fixed in wizard_mode.md, alignment-tr.md, quickstart-tr.md. All 8 doc guards green: - check_bilingual_parity --strict ✓ 45 pairs - check_anchor_resolution --strict ✓ 273 markdown files - check_cli_help_consistency --strict ✓ 439 forgelm invocations - check_yaml_snippets --strict ✓ 23 ForgeConfig snippets - check_audit_event_catalog --strict ✓ 36 ↔ 36 events - check_library_api_doc --strict ✓ 30 ↔ 30 symbols - check_doc_numerical_claims --strict ✓ 9/6/5/5 counts - check_bilingual_code_blocks --strict ✓ 45 pairs Items legitimately deferred (intentional, NOT in this commit): - P9 false positive (audit hallucination — 140×6 corpus is correct). - evaluation.approval_retention_days false positive (audit hallucination). - --add-row-ids flag implementation (Phase 28+). - HMAC body sign (Phase 28+). - encryption_at_rest config field (Phase 28+). - Anthropic / Stripe / SendGrid / Twilio secret patterns (Phase 28+). - KTO desirable_weight / undesirable_weight surface (Phase 28+). - GRPO beta flat field (Phase 28+). - compliance.config_hash audit-event payload emission (Phase 28+). - usermanuals P1-residual: full rewrite of training/{lora, galore, distributed, long-context}, evaluation/{benchmark, safety, auto-revert, trend-tracking}, deployment/* (~30 pages) — deferred to render-from-code pipeline (v0.6.x). - Codacy 100 issues (web-UI access required). - Sonar 8.8% duplication (intentional — marketing site chrome). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ⓘ You've reached your Qodo monthly free-tier limit. Reviews pause until next month — upgrade your plan to continue now, or link your paid account if you already have one. |
There was a problem hiding this comment.
Sorry @cemililik, your pull request is larger than the review limit of 150000 diff characters
Not up to standards ⛔🔴 Issues
|
| Category | Results |
|---|---|
| CodeStyle | 100 minor |
🟢 Metrics 160 complexity · 2 duplication
Metric Results Complexity 160 Duplication 2
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
|
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (10)
📝 WalkthroughWalkthroughBulk v0.5.5 update: EN/TR documentation overhaul, standardized audit-event names and manifest naming, revised CLI JSON envelopes and exit codes, added docs-check tooling, and small site/CSS/metadata edits. Changesv0.5.5 Documentation and Tooling Sweep
Estimated code review effort Possibly related PRs
✨ Finishing Touches🧪 Generate unit tests (beta)
|
CI's `ruff check .` step caught 5 lint findings on the Wave E CI guard tools (commit 0c4692c): - I001 (un-sorted imports) on tools/check_audit_event_catalog.py, tools/check_doc_numerical_claims.py, tools/check_library_api_doc.py, tools/check_yaml_snippets.py. - F541 (f-string without placeholders) on tools/check_audit_event_catalog.py:189. `ruff check . --fix` resolves all five; `ruff format` then applied consistent formatting to 4 of the new tools (line-wrap + trailing comma normalisation). All 8 doc guards remain green after the format pass: - check_yaml_snippets ✓ 23/357 - check_audit_event_catalog ✓ 36/36 - check_library_api_doc ✓ 30/30 - check_doc_numerical_claims ✓ 9/6/5/5 - check_bilingual_code_blocks ✓ 45 pairs Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Code Review
This pull request implements a comprehensive documentation overhaul for ForgeLM v0.5.5, focusing on bilingual parity and technical synchronization. Key changes include the translation of several core guides into Turkish, the alignment of event names and configuration schemas with the current implementation, and the introduction of a suite of CI tools designed to automate the detection of documentation drift. The reviewer pointed out that hardcoding line numbers in the reference documentation is a brittle practice that leads to stale information, recommending that these be replaced with symbolic references to functions or classes to improve maintainability.
Absorbs gemini-code-assist's 4 inline review comments on PR #39 (same finding × 4 files): hardcoded line numbers in subcommand reference docs are brittle and rot when code evolves. Replaces them with function / symbol-name references — the convention compliance_summary.md's Wave 4 cleanup formalised ("each link points at the file root, not a line anchor; the auditor opens the file and greps the cited symbol name"). Replacements (10 docs, EN+TR pairs ×5 subcommands): | Was | Now | |---|---| | `_verify_audit.py:41,45,56,63` | `_run_verify_audit_cmd` | | `_verify_audit.py:18-27` (docstring) | `_run_verify_audit_cmd` docstring | | `_purge.py:609-617` | `_perform_row_erasure_and_audit` | | `_purge.py:805-811` | `_run_purge_run_id` | | `_purge.py:993` | `_run_purge_check_policy` | | `_purge.py:101` | `_output_error_and_exit` | | `_reverse_pii.py:537,573` | `_maybe_audit_logger` | | `_reverse_pii.py:769-777` | `_run_reverse_pii_cmd` | | `_reverse_pii.py:107` | `_output_error_and_exit` | | `_approve.py:468-477` | `_run_approve_cmd` | | `_approve.py:542-551` | `_run_reject_cmd` | | `_approvals.py:240-249` | `_summarise_pending` | | `_approvals.py:440-444` | `_emit_show_json` | | `_approvals.py:319-327` | `_run_approvals_list_pending` | | `_approvals.py:458-463` | `_run_approvals_show` | Symbol-name refs survive code refactors (rename detection makes them self-correcting); line numbers don't. Files: verify_audit.md, reverse_pii_subcommand.md, purge_subcommand.md, approve_subcommand.md, approvals_subcommand.md (each EN + TR). Validated: bilingual_parity ✓ 45 pairs, anchor_resolution ✓ 273 files, cli_help_consistency ✓ 439 invocations, ruff check ✓ all passed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Review absorption — PR #39Snapshot of all 8 review comments and bot reports: Inline reviews (4)All 4 are the same gemini-code-assist finding repeated for EN + TR pairs of The convention is the one compliance_summary.md's Wave 4 cleanup formalised: "each link points at the file root, not a line anchor; the auditor opens the file and greps the cited symbol name; this survives refactors that the prior Bot reports (4)
ClosureAll actionable feedback (the 4 gemini comments) is now in. Bot reports have either passed (sonar) or are informational/rate-limited (qodo, coderabbit). Codacy 100 minor issues will be inspected separately — they don't block the merge gate. |
Sweep verified findings from docs/analysis/code_reviews/2026-05-08_full_docs_audit.md
(7 sub-auditors, ~253 findings, 27 CRITICAL — all verified zero false positives
against current code state).
Reference docs (configuration / library_api / audit_event_catalog / approvals /
iso27001 / soc2 / compliance_summary / architecture / usage / human_approval_gate
+ TR mirrors):
- configuration.md: galore_optim default 8bit→adamw; galore_target_modules
default list→null (regex fallback); synthetic block fully rewritten from
SyntheticConfig (api_base/api_key_env/max_new_tokens/api_delay/api_timeout
flat fields, output_format Literal messages|instruction|chatml|prompt_response).
- usage.md / usage-tr.md: rope_scaling bare-string → dict form
{type, factor} per Optional[Dict[str,Any]] schema.
- audit_event_catalog: 8 event payload field-name corrections; audit_log.jsonl
path corrected from compliance/ → top-level.
- library_api_reference: 14 signature/dataclass corrections (TrainResult 18-field,
AuditReport canonical fields, VerifyResult, VerifyAnnexIVResult, VerifyGgufResult,
run_benchmark, BenchmarkResult, setup_authentication, manage_checkpoints,
prepare_dataset, compute_minhash, detect_pii/secrets returns).
- library_api guide: result.output_dir/run_id → config.training.output_dir;
annex-iv-bundle.zip → annex_iv_metadata.json.
- human_approval_gate: compliance.human_approval → evaluation.require_human_approval.
- approvals_subcommand: config_hash documented as forward-compat read with null
surface (no producer emits it currently).
- iso27001 / soc2: config_hash phantom evidence replaced with real evidence
(pipeline.training_started SHAs); 1370+ tests → 1493; webhook HMAC clarified.
- compliance_summary: verify-audit exits 0/1/2/3 → 0|1; manifest filename
audit_log.manifest.json → audit_log.jsonl.manifest.json; 140-prompt corpus
line replaced with canonical 51×18 list.
- architecture: subcommand inline list 17→18 (added reject).
QMS (4 CRITICAL phantom claims — EN+TR):
- risk_treatment_plan: forgelm audit --secrets removed (no such flag — secrets
scan is always-on); FORGELM_AUDIT_SECRET-signed webhook payloads removed
(forgelm/webhook.py posts unsigned curated JSON); R-12 see-R-04 → see-R-06;
trainer ≠ approver clarified as record (not enforce).
- sop_incident_response: .sha256 sidecar removed (only .manifest.json exists);
--secrets phantom flag removed.
- access_control: webhook.secret_env phantom field removed (no such field on
WebhookConfig); webhook event vocabulary clarified (5 wire events vs 5
notifier method names); audit_log.manifest.json → audit_log.jsonl.manifest.json.
- encryption_at_rest: .sha256 sidecar removed.
- statement_of_applicability: 1370+→1493; webhook HMAC qualified
(audit-log chain HMAC ≠ webhook body HMAC); A.5.3 record-not-enforce.
Usermanual deep cleanup (EN+TR):
- evaluation/auto-revert: phantom auto_revert sub-block + last_good_checkpoint
+ notify_on_revert + guards.<name> registry → real flat schema (auto_revert: bool,
benchmark/safety/llm_judge sub-blocks).
- evaluation/safety: phantom block_categories / severity_threshold / model
/ regression_tolerance → real classifier / severity_thresholds dict /
track_categories / max_safety_regression / min_safety_score; default probe
set 50×~14 → 51×18.
- evaluation/judge: full rewrite from JudgeConfig — block name judge → llm_judge,
no pairwise/ELO/multi-rubric/budget_usd/rate_limit (all phantom).
- data/synthetic-data: phantom synthetic.teacher.{provider,model,api_key} block
→ flat teacher_model/teacher_backend/api_base/api_key_env; phantom budget_usd
+ rate_limit removed.
- compliance/overview: phantom compliance.annex_iv: true / human_approval /
deployment_geographies / responsible_party / data_audit_artifact removed
(none exist on ComplianceMetadataConfig); model_card.md → README.md;
conformity_declaration.md not produced clarified.
- operations/webhooks: phantom output.webhook block + template/events
arrays removed (WebhookConfig has no template/events/headers/retries
/channel/mention_on_failure fields); --webhook-test flag never existed.
Site:
- site/compliance.html line 287: model_card.md → README.md (HuggingFace-compatible).
Validation: all 8 doc CI guards green (bilingual_parity, anchor_resolution,
cli_help_consistency, yaml_snippets, audit_event_catalog, library_api_doc,
doc_numerical_claims, bilingual_code_blocks); ruff check clean; usermanuals
regenerated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 2
Note
Due to the large number of review comments, Critical severity comments were prioritized as inline comments.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (12)
docs/usermanuals/tr/operations/cicd.md (1)
75-99:⚠️ Potential issue | 🟡 Minor | ⚡ Quick win
deployaşaması tanımlanmış ama örnekte deploy job’ı yokLine 78’de
deploystage’i eklenmiş, ancak aşağıda buna karşılık gelen birdeploy:job bloğu yok. Kopyala-yapıştır yapan kullanıcılar için örnek eksik/yanıltıcı kalıyor. Yadeploystage’ini kaldırın ya da minimal birdeployjob ekleyin.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/tr/operations/cicd.md` around lines 75 - 99, The CI config declares a deploy stage in the stages list but does not define a corresponding deploy job, which is confusing for users; either remove "deploy" from the stages array or add a minimal "deploy:" job block that references stage: deploy (e.g., a simple image and a script that runs a harmless deploy/echo step and preserves artifacts if needed). Update the file so the "stages:" list and the "deploy:" job (or the removal of "deploy" from stages) are consistent.docs/reference/distributed_training-tr.md (1)
131-131:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winMissing bilingual mirror update for FSDP offload clarification.
The English version (distributed_training.md:131) added a clarifying comment about FSDP offload behavior, but the Turkish file is missing the equivalent update. Line 131 should include a Turkish translation of the comment explaining that
fsdp_offloadoffloads parameters AND gradients to CPU between forward/backward passes.🌍 Proposed fix to restore bilingual parity
- fsdp_offload: false + fsdp_offload: false # Parametreleri VE gradyanları (forward/backward arası) CPU'ya taşı + fsdp_backward_prefetch: "backward_pre" # Prefetch stratejisi + fsdp_state_dict_type: "FULL_STATE_DICT" # State dict yönetimiAs per coding guidelines, bilingual docs (EN ↔ TR mirrors) must maintain parity for user-facing documentation.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/reference/distributed_training-tr.md` at line 131, The Turkish doc is missing the bilingual clarification about FSDP offload: add a Turkish comment next to the fsdp_strategy: "full_shard" entry that mirrors the English note about fsdp_offload, explicitly stating that fsdp_offload moves both model parameters AND gradients to CPU between forward and backward passes; locate the block containing fsdp_strategy and insert a concise Turkish sentence that conveys “fsdp_offload offloads parameters AND gradients to CPU between forward/backward passes” using the term fsdp_offload so the link to the original comment is clear.docs/usermanuals/en/evaluation/auto-revert.md (1)
47-47:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winInconsistent documentation: guards registry.
Line 47 lists "Custom guard fails" as a trigger with configuration via
evaluation.guards.<name>, but lines 105-109 explicitly state "There is noevaluation.guards.<name>:plug-in registry." Remove the table row at line 47 or clarify that custom guards are implemented outside the config schema.🔧 Suggested fix
Remove the inconsistent table row:
| Final loss > starting loss | Always | not configurable | | Final loss is NaN/Inf | Always | not configurable | -| Custom guard fails | User-supplied callable | `evaluation.guards.<name>` |🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/en/evaluation/auto-revert.md` at line 47, The table row that lists "Custom guard fails" with configuration key `evaluation.guards.<name>` is inconsistent with the later statement that "There is no `evaluation.guards.<name>:` plug-in registry"; either remove that table row or change its text to clarify that custom guards are not configured via the `evaluation` config schema but are invoked as user-supplied callables implemented outside the config registry—update the row for "Custom guard fails" or the surrounding explanation so it references "user-supplied callable" (not `evaluation.guards.<name>`) and points readers to the external implementation approach.docs/usermanuals/en/compliance/human-oversight.md (1)
33-42:⚠️ Potential issue | 🟠 Major | ⚡ Quick winStale YAML config block contradicts the updated description.
Line 8 now documents
evaluation.require_human_approval: trueas the canonical activation key, but the Configuration section still shows the oldcompliance.human_approval: true. A user following this page will copy the wrong key.The
approval.request_webhookfield also appears stale —human_approval_gate-tr.md§3 documents the webhook aswebhook.url_env, notapproval.request_webhook.🐛 Proposed minimum fix — align YAML block with canonical key
```yaml -compliance: - human_approval: true - approval: - request_webhook: "${SLACK_WEBHOOK}" # optional notification - signature_method: "cli" # cli | webhook | api - timeout_hours: 48 # auto-fail after this - require_role: "ml-compliance-lead" # who can approve - quorum: 1 # required approvers +evaluation: + require_human_approval: true + +# optional: approval notifications and policy +webhook: + url_env: "SLACK_WEBHOOK" # fires notify_awaiting_approval +approval: + signature_method: "cli" # cli | webhook | api + timeout_hours: 48 # auto-fail after this (v0.6.0+) + require_role: "ml-compliance-lead" # who can approve + quorum: 1 # required approvers</details> Please also verify the sub-option canonical paths (`require_role`, `quorum`, `timeout_hours`) against `forgelm/config.py` — this PR's AI summary does not mention their canonical parent key. <details> <summary>🤖 Prompt for AI Agents</summary>Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.In
@docs/usermanuals/en/compliance/human-oversight.mdaround lines 33 - 42,
Replace the stale top-level key compliance.human_approval with
evaluation.require_human_approval and move webhook/approval fields to their
canonical parents: use webhook.url_env (env name string) for the webhook, and
approval.signature_method, approval.timeout_hours, approval.require_role and
approval.quorum under approval; update any inline comments (e.g., timeout_hours
v0.6.0+) to match the new layout and names shown in the proposed diff, and then
verify each sub-option canonical path (require_role, quorum, timeout_hours)
against the definitions in forgelm/config.py to ensure parent keys and names
match the code.</details> </blockquote></details> <details> <summary>docs/guides/ingestion.md (1)</summary><blockquote> `117-135`: _⚠️ Potential issue_ | _🟡 Minor_ | _⚡ Quick win_ **Prose enumeration (8 items) doesn't match the updated count (9 patterns) — add `PGP private-key` as a separate entry.** The inline list at lines 117–121 groups `OpenSSH/RSA/DSA/EC/PGP private-key block headers` as a single item (8 total), but line 135 now says "9 prefix-anchored patterns". Per the canonical catalog in `usage.md` (`openssh_private_key` and `pgp_private_key` are separate families), the count of 9 is correct — but a reader reconciling the prose enumeration against the stated count will come up short by one. <details> <summary>📝 Proposed fix</summary> ```diff -Detected categories: AWS access -keys (`AKIA…`), GitHub PATs (`ghp_…` and friends), Slack tokens, OpenAI -API keys (`sk-…` / `sk-proj-…`), Google API keys (`AIza…`), JWTs, -OpenSSH/RSA/DSA/EC/PGP private-key block headers, Azure storage -connection strings. +Detected categories: AWS access +keys (`AKIA…`), GitHub PATs (`ghp_…` and friends), Slack tokens, OpenAI +API keys (`sk-…` / `sk-proj-…`), Google API keys (`AIza…`), JWTs, +OpenSSH/RSA/DSA/EC/ECDSA private-key block headers, PGP private-key +block headers, Azure storage connection strings.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/ingestion.md` around lines 117 - 135, Update the prose enumeration of detected secret categories so it matches the stated count of 9 patterns: split the combined "OpenSSH/RSA/DSA/EC/PGP private-key block headers" entry into two distinct items (one for OpenSSH/RSA/DSA/EC private-key block headers and a second for PGP private-key block headers), and ensure the text around ingest_path(), forgelm.data_audit.mask_secrets, and the reference to data_audit.md reflects the two separate families (see usage.md: openssh_private_key and pgp_private_key) so the list and the "9 prefix-anchored patterns" statement are consistent.docs/usermanuals/tr/training/simpo.md (1)
39-61:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winParameters table still uses old
beta/gammakey names — update to match the newsimpo_beta/simpo_gammaYAML keys.The YAML example (lines 43–44) now uses
simpo_betaandsimpo_gamma, but the "Parametreler" table at lines 55–61 documentsbetaandgammaas the backtick-quoted config keys, and the## \beta` ve `gamma`section (line 72) and its combination table (lines 73–78) also use the old names. A user reading the table will usetraining.beta/training.gamma`, which are the wrong field names.📝 Proposed fix for the parameters table (lines 55–61)
| Parametre | Tip | Vars. | Açıklama | |---|---|---|---| -| `beta` | float | `2.0` | Uzunluk-normalize ödül ölçeği. Yüksek = güçlü tercih kayması. **DPO beta ile aynı ölçek değil.** | -| `gamma` | float | `1.0` | Margin — chosen ve rejected log-likelihood arasında SimPO'nun korumaya çalıştığı boşluk. | +| `simpo_beta` | float | `2.0` | Uzunluk-normalize ödül ölçeği. Yüksek = güçlü tercih kayması. **DPO beta ile aynı ölçek değil.** | +| `simpo_gamma` | float | `1.0` | Margin — chosen ve rejected log-likelihood arasında SimPO'nun korumaya çalıştığı boşluk. |The
## \beta` ve `gamma`section heading (line 72) and its combination table (lines 73–78) should likewise be updated tosimpo_beta/simpo_gamma`.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/tr/training/simpo.md` around lines 39 - 61, The parameters docs still reference old keys `beta`/`gamma`; update all occurrences to the new YAML keys `simpo_beta` and `simpo_gamma` so they match the example block. Specifically, change the table rows that currently show `beta` and `gamma` to `simpo_beta` and `simpo_gamma`, update the heading “`beta` ve `gamma`” to “`simpo_beta` ve `simpo_gamma`”, and update any related combination/example tables or text that mention the old names so all config key names use `simpo_beta`/`simpo_gamma`.docs/usermanuals/en/training/orpo.md (1)
56-58:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winConfiguration table doesn't match the updated YAML example.
The "Quick example" at lines 30-46 uses the new flat-field schema (
orpo_betaundertraining), but the "Configuration parameters" table at lines 56-58 still lists the old nested field names (beta,loss_type,sft_weight) without thetraining.orpo_ororpo_prefix.Update the table's "Parameter" column to match the actual config keys (
training.orpo_beta,training.orpo_loss_type,training.orpo_sft_weight) or prefix the table with a note that these are shorthand names and the full keys aretraining.orpo_*.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/en/training/orpo.md` around lines 56 - 58, The parameters table currently lists shorthand names (beta, loss_type, sft_weight) that no longer match the YAML example; update the "Parameter" column to the actual config keys used in the example (training.orpo_beta, training.orpo_loss_type, training.orpo_sft_weight) OR add a single-line note above the table stating these are shorthand names and the full keys are training.orpo_* so readers can map beta → training.orpo_beta, loss_type → training.orpo_loss_type, and sft_weight → training.orpo_sft_weight.docs/usermanuals/en/training/simpo.md (1)
56-60:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winConfiguration table doesn't match the updated YAML example.
The "Quick example" at lines 29-46 uses the new flat-field schema (
simpo_beta,simpo_gammaundertraining), but the "Configuration parameters" table at lines 56-60 still lists the old nested field names (beta,gamma,loss_type) without thetraining.simpo_orsimpo_prefix.Update the table's "Parameter" column to match the actual config keys (
training.simpo_beta,training.simpo_gamma,training.simpo_loss_type) or prefix the table with a note that these are shorthand names and the full keys aretraining.simpo_*.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/en/training/simpo.md` around lines 56 - 60, The configuration table uses legacy shorthand keys (beta, gamma, loss_type, length_normalize, label_smoothing) but the actual YAML example uses flat keys under training (e.g., training.simpo_beta, training.simpo_gamma, training.simpo_loss_type); update the "Parameter" column to the full keys (training.simpo_beta, training.simpo_gamma, training.simpo_loss_type, training.simpo_length_normalize, training.simpo_label_smoothing) or add a single-line note above the table stating these are shorthand and the real config keys are training.simpo_* so the table matches the quick example; locate and edit the table rows that currently list `beta`, `gamma`, `loss_type`, `length_normalize`, and `label_smoothing`.docs/usermanuals/tr/data/audit.md (1)
142-145:⚠️ Potential issue | 🟠 Major | ⚡ Quick winRemove
--strictfrom CI YAML example—it's documented as removed in line 110.Line 110 explicitly states
--strict"hiç ship olmadı" (never shipped) and does not exist in the parser. The CI example at line 144 still callsforgelm audit data/train.jsonl --strict, which will fail for any user copying it. The same contradiction appears in the EN mirror (line 150).The page already directs readers to use
--output-format jsonwith ajq-based gate for audit-as-gate behavior. Replace the example with the recommended pattern:Proposed fix
- name: Audit data - run: forgelm audit data/train.jsonl --strict + run: | + forgelm audit data/train.jsonl --output ./audit/ --output-format json + # Exit kodu zaten ciddiyeti yansıtır: 0 temiz, 2 uyarı, 3 hata.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/tr/data/audit.md` around lines 142 - 145, The CI example currently shows the command "forgelm audit data/train.jsonl --strict" which is invalid because "--strict" was never shipped; update the example to remove "--strict" and instead show the recommended pattern using "forgelm audit ... --output-format json" combined with a jq-based gate as described earlier in the document (follow the guidance near the previous mention of --output-format json) so the snippet matches the documented, working audit-as-gate approach.docs/reference/approvals_subcommand-tr.md (1)
162-162:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winFix field name in usage example to match JSON schema.
Both the JSON envelope (lines 60, 71) define the field as
age_seconds, but the usage example at line 162 incorrectly referencesage. Changeagetoage_secondsso a reader'sjq '.pending[].age_seconds'resolves correctly. (Note: This same error appears in the EN mirror at approvals_subcommand.md:162 and should be fixed there as well.)📝 Proposed fix
-Daha zengin bir policy kuran operatörler (örn. "herhangi bir bekleyen karar N saatten eskiyse deploy'u blokla") `age` field'ını parse eder. Text çıktısını yalnızca tavsiye olarak ele alın — JSON zarfı stabil sözleşmedir. +Daha zengin bir policy kuran operatörler (örn. "herhangi bir bekleyen karar N saatten eskiyse deploy'u blokla") `age_seconds` field'ını parse eder. Text çıktısını yalnızca tavsiye olarak ele alın — JSON zarfı stabil sözleşmedir.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/reference/approvals_subcommand-tr.md` at line 162, Update the usage example that references the JSON field name so it matches the schema: replace occurrences of the wrong field name "age" in the jq example (e.g., jq '.pending[].age') with "age_seconds" so jq '.pending[].age_seconds' returns the expected values; also apply the same change to the English mirror where the same example appears.docs/usermanuals/en/training/sft.md (1)
113-117:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winNormalize
tokeniservstokenizerspelling in the same warning block.Line 113 uses “tokeniser” while Line 117 uses “tokenizer”. Please standardize on one variant in this page to avoid style drift in user-facing docs.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/en/training/sft.md` around lines 113 - 117, The two-warning block uses inconsistent spellings “tokeniser” and “tokenizer” (in the paragraph under "Loss going up instead of down." and the :::warn block titled "Loss going up after a tokenizer change."); pick one spelling (preferably "tokenizer") and update both occurrences so the same variant appears in the rendered sample text and the warning title/paragraph to maintain consistent user-facing documentation.docs/usermanuals/tr/compliance/audit-log.md (1)
15-21:⚠️ Potential issue | 🟠 Major | ⚡ Quick winEvent examples still use legacy names and contradict the updated catalog.
You updated the canonical event set at Line 34–46, but examples still show
run_start,run_complete,data_audit_complete,training_epoch_complete, andauto_revert. This creates conflicting guidance in the same page.Suggested direction
-{"ts":"...","event":"run_start",...} +{"ts":"...","event":"training.started",...} -{"ts":"...","event":"run_complete","exit_code":0,...} +{"ts":"...","event":"pipeline.completed","exit_code":0,...} - `run_start` ve `run_complete`. + `training.started` ve `pipeline.completed`. - [Otomatik Geri Alma](`#/evaluation/auto-revert`) — `auto_revert` olaylarını üretir. + [Otomatik Geri Alma](`#/evaluation/auto-revert`) — `model.reverted` olayını üretir.Also applies to: 77-79, 118-124, 145-146
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/tr/compliance/audit-log.md` around lines 15 - 21, The examples use legacy event names (run_start, run_complete, data_audit_complete, training_epoch_complete, auto_revert) that conflict with the updated canonical event set; update every example JSON where "event": "<legacy_name>" appears to the corresponding new canonical event names defined in the catalog (replace each legacy token with its exact new token from the updated canonical set), and ensure associated fields (e.g., "verdict", "exit_code", "epoch", "loss") match the catalog's expected schema; apply the same replacements to the other example blocks noted (the later occurrences of those legacy names) so all examples consistently reflect the updated event catalog.
🟠 Major comments (23)
docs/usermanuals/tr/operations/experiment-tracking.md-19-19 (1)
19-19:⚠️ Potential issue | 🟠 Major | ⚡ Quick winŞemaya dair mutlak ifade, sayfanın artifact örneğiyle çelişiyor.
Line 19 nested backend bloklarını tamamen dışlıyor; fakat aynı doküman
training.wandb.log_artifactsörneğini kullanıyor. Cümleyi “bağlantı/kimlik backend ayarları” ile sınırlandırın, aksi halde geçerli alanlar yanlışlıkla kaldırılabilir.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/tr/operations/experiment-tracking.md` at line 19, Satırdaki mutlak ifade ("Per-backend nested config blokları (örn. `wandb: { project: ... }`) şemanın parçası değildir") yanlışı; bunu backend'e özgü bağlantı/kimlik/credential ayarlarıyla sınırlayın ki örnek `training.wandb.log_artifacts` geçerli kalmaya devam etsin. Düzenleme yaparken orijinal metni `Per-backend nested config blokları ...` yerine "Per-backend sadece bağlantı/kimlik/credential ayarları (örn. `wandb: { project: ... }`) şemanın dışında tutulur; diğer yapılandırma alanları (örn. `training.wandb.log_artifacts`) şemada tanımlı olabilir." gibi bir ifadeyle değiştirin ve `wandb: { project: ... }` ile `training.wandb.log_artifacts` referanslarını koruyun.docs/usermanuals/en/operations/experiment-tracking.md-19-19 (1)
19-19:⚠️ Potential issue | 🟠 Major | ⚡ Quick winAbsolute “no nested backend config blocks” claim conflicts with this page’s own artifact example.
Line 19 says nested backend blocks are not in schema, but this same doc still documents
training.wandb.log_artifactsunder Artifact management. Please narrow this sentence (e.g., “for backend connection/auth settings”) or readers will remove valid config and break artifact behavior.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/en/operations/experiment-tracking.md` at line 19, The doc's blanket statement "Per-backend nested config blocks (e.g. `wandb: { project: ... }`) are not part of the schema" conflicts with the documented `training.wandb.log_artifacts` example; update the sentence to narrow the scope (e.g., "Per-backend nested config blocks for backend connection/auth settings (e.g., API keys, project identifiers) are not part of the schema") and explicitly note that backend-specific runtime flags/features such as `training.wandb.log_artifacts` are supported in the config; ensure the revision references `training.wandb.log_artifacts` so readers know artifacts and similar backend-specific toggles are allowed while connection/auth settings must use environment variables.docs/usermanuals/en/training/kto.md-45-46 (1)
45-46:⚠️ Potential issue | 🟠 Major | ⚡ Quick winKTO config guidance is internally contradictory.
This new note says
desirable_weight/undesirable_weightare not exposed in ForgeLM config, but the same page still documents them as supported parameters (and recommends setting them). Please make the page consistent in one direction.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/en/training/kto.md` around lines 45 - 46, The page currently contradicts itself about desirable_weight and undesirable_weight: update the KTO section so it is consistent by choosing one stance — either (A) state clearly that desirable_weight and undesirable_weight are NOT exposed in ForgeLM (KTOConfig) today and remove any guidance that recommends setting them, instead documenting the TRL-side override workflow and linking to KTOConfig as an advanced workaround; or (B) if you intend to support them, update the earlier note to say they are exposed, keep the usage examples and default values, and remove the sentence claiming they are not surfaced. Refer to the identifiers desirable_weight, undesirable_weight, and KTOConfig when editing so you update all mentions.docs/reference/compliance_summary-tr.md-117-117 (1)
117-117:⚠️ Potential issue | 🟠 Major | ⚡ Quick win
approvalscommand example is missing required--output-dir.
forgelm approvals --pendingshould include--output-dir <dir>to match the documented CLI contract used across compliance docs.Suggested fix
-- Listeleme: `forgelm approvals --pending` (Phase 37). +- Listeleme: `forgelm approvals --pending --output-dir <output_dir>` (Phase 37).Based on learnings, the official approvals workflow contract requires
forgelm approvals --pending --output-dir <dir>.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/reference/compliance_summary-tr.md` at line 117, The example command "forgelm approvals --pending" is missing the required output directory flag; update the documented CLI example so the command reads "forgelm approvals --pending --output-dir <dir>" to match the official approvals workflow contract and other compliance docs.docs/guides/iso_soc2_deployer_guide.md-111-112 (1)
111-112:⚠️ Potential issue | 🟠 Major | ⚡ Quick winClarify
config_hashsource (event vs sidecar).Line 111 currently reads as if
human_approval.granteditself carriesconfig_hash, while also labeling it as a manifest sidecar field. Please make the source explicit to avoid auditors grepping the wrong artifact.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/iso_soc2_deployer_guide.md` around lines 111 - 112, Clarify that config_hash is a per-run manifest sidecar field (not embedded in the human_approval.granted event); update the sentence referencing `config_hash` and `human_approval.granted` to state that `config_hash` is stored in the run's manifest sidecar and auditors should compare that sidecar value to the YAML in `git log`, while `human_approval.granted` is just the approval event (does not carry `config_hash`).docs/usermanuals/en/data/audit.md-110-110 (1)
110-110:⚠️ Potential issue | 🟠 Major | ⚡ Quick winRemoved-flag note conflicts with examples still in this page.
Line 110 says
--strictand--sample-ratenever shipped, but this same document still uses--strictin the CI example (Line 150) and discusses--sample-ratein pitfalls (Line 160). Please update those sections in the same PR to avoid copy/paste command failures.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/en/data/audit.md` at line 110, The removed-flags note conflicts with examples: update the CI example that uses the removed flag `--strict` (referenced in the CI example block) to use the supported workflow (wrap `--output-format json` with a jq-based CI gate or remove `--strict`), and remove or replace mentions of the removed `--sample-rate` in the "pitfalls" section with the current supported options or guidance; ensure every occurrence of `--strict` and `--sample-rate` in the document (CI example and pitfalls paragraphs) is either deleted or converted to the canonical alternative so the examples run without the removed flags.docs/usermanuals/tr/training/kto.md-45-45 (1)
45-45:⚠️ Potential issue | 🟠 Major | ⚡ Quick winKTO config contract is internally inconsistent in this page.
Line 45 says
desirable_weight/undesirable_weightare not exposed in ForgeLM config, but the same page still documents them as tunable parameters and recommends setting them (Lines 60-67). Please align one side; otherwise users will try unsupported fields.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/tr/training/kto.md` at line 45, Sayfa içindeki çelişkiyi düzelt: KTOConfig içindeki `desirable_weight` / `undesirable_weight` alanlarının ForgeLM config'e yüzeylenmediğini belirten (Line 45) açıklama ile sonraki bölümlerde bu alanların ayarlanabileceği yönündeki dokümantasyonu tutarlı hale getir. Bunu yapmak için ya KTOConfig dokümantasyonundaki tunable parametreler listesinden `desirable_weight`/`undesirable_weight` öğelerini kaldır (böylece yalnızca trainer TRL varsayılanları 1.0/1.0 geçerli olur) ya da Line 45 açıklamasını güncelleyip açıkça “bu alanlar ForgeLM config olarak desteklenmez; asimetrik ağırlıklandırma gerekiyorsa TRL tarafında override script kullanın” şeklinde yazarak Lines 60-67’deki öneriyi bir workaround olarak belirt (referans verilecek semboller: KTOConfig, desirable_weight, undesirable_weight, ForgeLM, trainer TRL).docs/usermanuals/en/training/grpo.md-59-60 (1)
59-60:⚠️ Potential issue | 🟠 Major | ⚡ Quick winAlign the new flat GRPO contract with the rest of this page.
The new guidance on Line 59 (“always-on fallback” +
grpo_reward_model) conflicts with later sections that still document the old nestedtraining.grpo.reward_function/format_reward/answer_patternmodel. Please update the downstream sections/table to one canonical schema to avoid misconfiguration.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/en/training/grpo.md` around lines 59 - 60, The docs currently mix the new flat GRPO contract with the old nested schema, causing confusion; update the downstream sections and the configuration table to use the canonical flat schema (referencing grpo_reward_model and the built-in fallback forgelm/grpo_rewards.py) and remove or replace references to the legacy nested keys (training.grpo.reward_function, format_reward, answer_pattern), ensuring any mention of TRL's beta/KL remains consistent with the new flat field guidance. Locate occurrences of the legacy keys and either rename them to grpo_reward_model (or document as deprecated with a single-line migrate note) and adjust examples/tables to show the flat contract and the always-on fallback behavior from forgelm/grpo_rewards.py.docs/usermanuals/en/data/secrets.md-12-27 (1)
12-27:⚠️ Potential issue | 🟠 Major | ⚡ Quick winPropagate the new secrets taxonomy/mask token across the whole page.
The updated contract here conflicts with later examples/config (
github_pat/pem_block/openai_key) and with the PEM section’s[REDACTED-PEM-BLOCK]token. Please align all examples and output snippets to one canonical key set and one masking-output rule.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/en/data/secrets.md` around lines 12 - 27, The page currently mixes different secret taxonomy names and mask tokens; update all later examples, config snippets and the PEM section to use the canonical keys from _SECRET_PATTERNS (e.g., github_token, openai_api_key, openssh_private_key, aws_access_key, etc.) and ensure all masked output uses the single mask value produced by mask_secrets() — [REDACTED-SECRET] — replacing occurrences of github_pat, pem_block, openai_key, [REDACTED-PEM-BLOCK], or any other alternate tokens so the entire document references one consistent key set and one masking output.docs/design/gdpr_erasure.md-199-199 (1)
199-199:⚠️ Potential issue | 🟠 Major | ⚡ Quick winResolve
--yessemantics drift in this spec.Line 199 says
--yesis not implemented and purge is non-interactive, but later sections still describe prompt-driven behavior and missing---yeserrors. Please normalize the entire document to a single shipped behavior model.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/design/gdpr_erasure.md` at line 199, The spec is inconsistent about the `--yes` flag and interactive behavior: standardize the document to reflect the shipped behavior that `forgelm purge` is non-interactive by default (as of v0.5.5) and that `--yes` is currently unimplemented and retained only on the Phase 28+ backlog; update all places that describe prompt-driven flows, missing-`--yes` errors, and interactive examples to instead state that purge never prompts, CI and TTY both require `--justification` for audit metadata, and mark `--yes` as reserved/backlog (remove any instructions that treat `--yes` as required or that expect an interactive confirmation).tools/check_library_api_doc.py-82-90 (1)
82-90:⚠️ Potential issue | 🟠 Major | ⚡ Quick winFail closed when
forgelmimport fails.For a CI drift guard, returning
0on import failure can silently pass while the checker never runs. This should return non-zero so CI cannot go green on a skipped validation path.💡 Suggested fix
try: import forgelm # type: ignore except ImportError as exc: print( f"check_library_api_doc: cannot import forgelm ({exc}); skipping.", file=sys.stderr, ) - return 0 + return 1🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tools/check_library_api_doc.py` around lines 82 - 90, The try/except that imports forgelm currently returns 0 on ImportError which lets CI pass silently; change the except block in check_library_api_doc.py (the import forgelm try/except) to exit with a non-zero status (e.g., return 1 or raise SystemExit(1)) and include the original exception text in the stderr message so the script fails CI instead of succeeding when forgelm cannot be imported.docs/usermanuals/tr/training/dpo.md-74-75 (1)
74-75:⚠️ Potential issue | 🟠 Major | ⚡ Quick winUpdate the parameter table to reference
dpo_beta(not justbeta).Line 74 correctly shows
dpo_beta: 0.1as the configuration key, but the parameter table at line 100 documents it asbeta, creating confusion. Update the table to either showdpo_betaor add clarification thatdpo_betais the flat-field config key. This prevents users from copy-pasting mismatched config snippets.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/tr/training/dpo.md` around lines 74 - 75, The parameter table currently documents the KL weight as `beta` but the config uses `dpo_beta`; update the table entries to reference `dpo_beta` (or add a parenthetical note that `dpo_beta` is the flat-field config key) so examples and descriptions match the actual config key; specifically change any `beta` occurrences in the parameters table to `dpo_beta` and adjust the description to mention it is the KL strength used by DPO (e.g., "dpo_beta: KL strength (flat field)").docs/guides/library_api-tr.md-179-180 (1)
179-180:⚠️ Potential issue | 🟠 Major | ⚡ Quick winFix invalid Python f-string syntax in code example.
Nested double quotes in f-string expression cause
SyntaxError. Change inner dictionary access quotes to single quotes so users can run the code without parse errors.Proposed fix
- raise ValueError(f"too many duplicates: {report.near_duplicate_summary["pairs"]}") + raise ValueError(f"too many duplicates: {report.near_duplicate_summary['pairs']}")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/library_api-tr.md` around lines 179 - 180, The f-string in the raise statement uses nested double quotes and causes a SyntaxError; update the dict key access to use single quotes (replace report.near_duplicate_summary["pairs"] with report.near_duplicate_summary['pairs']) and do the same in the return expression for consistency so both the raise ValueError(...) and the return {"samples": report.total_samples, "duplicates": report.near_duplicate_summary["pairs"]} use report.near_duplicate_summary['pairs'].docs/usermanuals/tr/training/grpo.md-59-60 (1)
59-60:⚠️ Potential issue | 🟠 Major | ⚡ Quick winAlign GRPO schema to flat keys throughout both documentation versions.
The quick example correctly uses flat GRPO keys (
grpo_num_generations,grpo_max_completion_length,grpo_reward_model), but the "Built-in format shaping" section and configuration tables still describe a nested schema undertraining.grpo.*(reward_function,format_reward,answer_pattern). This inconsistency appears in both the English and Turkish versions and can lead to invalid configs. Update the later sections to match the canonical flat key structure shown in the quick example.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/tr/training/grpo.md` around lines 59 - 60, The docs use nested keys under the "Built-in format shaping" section and configuration tables but the canonical quick example uses flat GRPO keys; update the "Built-in format shaping" text and every configuration table in both English and Turkish pages to use the flat keys (e.g., grpo_num_generations, grpo_max_completion_length, grpo_reward_model, grpo_reward_function, grpo_format_reward, grpo_answer_pattern) so examples and config schema match the quick example and produce valid configs.docs/usermanuals/en/operations/webhooks.md-90-90 (1)
90-90: 🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick winBrittle line-number reference — convert to symbol/function name per the PR's drift sweep.
The PR objectives explicitly note that hardcoded line-number references in subcommand docs were converted to symbol/function-name references (commit 694bf12) under the "point at file root and cite symbol names" convention.
forgelm/config.py:641is the same brittle pattern: any insertion above line 641 silently desyncs the doc. Cite the class name (WebhookConfig) instead.📝 Proposed fix
-Real `WebhookConfig` (see `forgelm/config.py:641`): +Real `WebhookConfig` (defined as the `WebhookConfig` Pydantic model in `forgelm/config.py`):🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/en/operations/webhooks.md` at line 90, The doc's brittle line-number reference "forgelm/config.py:641" should be replaced with a symbol-based reference to the WebhookConfig class; update the text to point at the module root and cite the symbol, e.g. "forgelm.config:WebhookConfig" or "forgelm.config — WebhookConfig", so the docs reference the class name WebhookConfig rather than a line number.docs/usermanuals/tr/data/audit.md-93-94 (1)
93-94: 🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick winBrittle line-range reference — symbol-only is the project convention.
Per the PR objectives, hardcoded line refs in subcommand docs were already converted to symbol/function-name references.
(335-468 satırları)defeats that convention: any edit to_parser.pyabove line 468 silently desyncs this doc. Cite_add_audit_subcommandonly and drop the line range.📝 Proposed fix
-Yetkili kaynak: `forgelm/cli/_parser.py::_add_audit_subcommand` (335-468 satırları). +Yetkili kaynak: `forgelm/cli/_parser.py` içindeki `_add_audit_subcommand` fonksiyonu.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/tr/data/audit.md` around lines 93 - 94, Remove the hardcoded line-range fragment from the audit doc line that currently reads "Yetkili kaynak: `forgelm/cli/_parser.py::_add_audit_subcommand` (335-468 satırları)"; keep only the symbol reference so it becomes "Yetkili kaynak: `forgelm/cli/_parser.py::_add_audit_subcommand`." Ensure the backtick-wrapped symbol name `_add_audit_subcommand` remains unchanged.docs/guides/troubleshooting-tr.md-405-411 (1)
405-411:⚠️ Potential issue | 🟠 Major | ⚡ Quick winExit code 4 guidance points to a stale path:
checkpoints/compliance/.This path does not exist in the codebase. The canonical staging convention for awaiting-approval runs is
<output_dir>/final_model.staging.<run_id>/(with legacy fallbackfinal_model.staging/). Users following this guidance will be unable to locate staged adapters or complete the approval workflow. Point them toforgelm approvals --show <run_id>to inspect the actual staging directory, thenforgelm approve <run_id>to promote it.📝 Proposed fix
-| `4` | Onay bekleniyor | İnsan incelemesi gerekli — `checkpoints/compliance/`'taki sonuçları inceleyin ve hazır olunca yeniden deploy edin | +| `4` | Onay bekleniyor | İnsan incelemesi gerekli — `forgelm approvals --show <run_id>` ile staged sonuçları inceleyin (`<output_dir>/final_model.staging.<run_id>/`) ve `forgelm approve <run_id>` ile karar verin |🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/troubleshooting-tr.md` around lines 405 - 411, Update the guidance for exit code 4: replace the stale path reference `checkpoints/compliance/` with the canonical staging convention `<output_dir>/final_model.staging.<run_id>/` (and mention the legacy fallback `final_model.staging/`), and add the exact commands users should run to inspect and promote staged runs: `forgelm approvals --show <run_id>` to locate the staging directory and `forgelm approve <run_id>` to promote it; ensure this change is applied to the text that describes exit code 4 in the troubleshooting table so readers can find staged adapters and complete the approval workflow.docs/usermanuals/tr/operations/webhooks.md-128-128 (1)
128-128:⚠️ Potential issue | 🟠 Major | ⚡ Quick winDon’t position
--dry-runas an end-to-end webhook smoke test.Line 128 suggests lifecycle webhook validation via
--dry-run, but dry-run is validation-only and does not execute the training lifecycle. That can produce a false “webhooks are fine” result. Recommend changing this to a tiny real run (small dataset/epochs) or synthetic payload replay only.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/tr/operations/webhooks.md` at line 128, Update the guidance that currently suggests using `--dry-run` as an end-to-end webhook smoke test: clarify that `--dry-run` only validates config and does not execute the training lifecycle, and replace the suggested test with either a tiny real run (small dataset + low `num_train_epochs`) against the lifecycle staging webhook URL or replaying a curated synthetic payload via `curl`; also keep the note that ForgeLM's `--webhook-test` flag does not send webhooks so it cannot be used for E2E verification.docs/guides/quickstart-tr.md-19-21 (1)
19-21:⚠️ Potential issue | 🟠 Major | ⚡ Quick winUse English for code/CLI snippet text in TR quickstart.
The inline code comments and sample CLI output annotations in these blocks are Turkish. Keep snippet content English-only for bilingual consistency and reliable copy/paste workflows.
As per coding guidelines, bilingual user-facing docs must keep code and CLI output in English only.
Also applies to: 161-164
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/quickstart-tr.md` around lines 19 - 21, The documentation contains Turkish text inside code/CLI snippet blocks (for example the snippet header "Önerilen: 4-bit quantization'ı aç (Linux)" near the pip install example); replace those inline snippet comments and CLI annotations with English-only equivalents (e.g., "Recommended: enable 4-bit quantization (Linux)") and update the same pattern at the other occurrence mentioned (around lines 161-164) so all code blocks and CLI output remain English for copy/paste and bilingual consistency.docs/usermanuals/tr/evaluation/judge.md-23-30 (1)
23-30:⚠️ Potential issue | 🟠 Major | ⚡ Quick winKeep code-snippet comments in English in TR user manuals.
Line 23–30 uses Turkish inline comments inside YAML. In user-facing bilingual docs, code/CLI content should stay English-only for copy/paste consistency.
Proposed patch
evaluation: - llm_judge: # blok adı `llm_judge`, `judge` değil + llm_judge: # block name is `llm_judge`, not `judge` enabled: true - judge_model: "gpt-4o-mini" # veya yerel yol, ör. "./judges/Qwen2.5-72B-Instruct" - judge_api_key_env: OPENAI_API_KEY # null = yerel model (API çağrısı yok) - judge_api_base: null # Azure OpenAI / vLLM-uyumlu gateway için override + judge_model: "gpt-4o-mini" # or local path, e.g. "./judges/Qwen2.5-72B-Instruct" + judge_api_key_env: OPENAI_API_KEY # null = local model (no API call) + judge_api_base: null # override for Azure OpenAI / vLLM-compatible gateway eval_dataset: "data/eval-prompts.jsonl" - min_score: 6.5 # ortalama puan alt sınırı (1-10 skala); altına düşerse revert - batch_size: 8 # round başı skorlanan (prompt, completion) çifti; 1 batching'i kapatır + min_score: 6.5 # minimum mean score (1-10); below threshold triggers revert + batch_size: 8 # (prompt, completion) pairs scored per round; 1 disables batchingAs per coding guidelines, bilingual user-facing docs must keep code and CLI output in English only.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/tr/evaluation/judge.md` around lines 23 - 30, The YAML snippet uses Turkish inline comments; replace all Turkish comments in the llm_judge block (keys: llm_judge, judge_model, judge_api_key_env, judge_api_base, eval_dataset, min_score, batch_size) with English equivalents so the code/CLI lines remain English-only for copy/paste consistency—update each inline comment to English while preserving the keys and values exactly.docs/guides/safety_compliance-tr.md-46-75 (1)
46-75:⚠️ Potential issue | 🟠 Major | 🏗️ Heavy liftKeep code snippets and CLI output in English-only in TR docs.
YAML/CLI snippets include Turkish comments/output text (for example, Line 53 comment and Line 493 echo). Project rule says code and CLI output must remain English even in bilingual docs.
As per coding guidelines, “
docs/**/*.md: Bilingual docs (EN ↔ TR mirrors) required for user-facing documentation; code and CLI output must be English only”.Also applies to: 488-495
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/safety_compliance-tr.md` around lines 46 - 75, The YAML snippets under the evaluation.safety block contain Turkish inline comments and CLI/echo text (e.g., the comment after max_safety_regression and the scoring/severity comments), violating the English-only code/CLI rule; update all inline comments and any CLI/echo output in these YAML blocks to English while keeping the keys and values unchanged (e.g., fields evaluation, safety, classifier, test_prompts, max_safety_regression, scoring, min_safety_score, min_classifier_confidence, track_categories, severity_thresholds, critical/high/medium) so the code/CLI remains English but the surrounding Turkish documentation can stay in Turkish.docs/usermanuals/tr/training/sft.md-28-48 (1)
28-48:⚠️ Potential issue | 🟠 Major | ⚡ Quick winKeep YAML snippet comments English-only in TR mirror.
At Line 28, Line 33, and Line 47, inline YAML comments are Turkish. In bilingual docs, code/CLI content should remain English-only.
As per coding guidelines, “
docs/**/*.md: Bilingual docs (EN ↔ TR mirrors) required for user-facing documentation; code and CLI output must be English only”.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/tr/training/sft.md` around lines 28 - 48, The inline YAML comments in the SFT example are in Turkish; update them to English so code/CLI content remains English-only. Specifically, replace the comment after backend ("ya da "unsloth" — eski `use_unsloth: true` flag'inin yerini alır") with an English equivalent, change the lora.method comment ("ya da "dora" / "pissa" / "rslora"") to English, and change the packing comment ("kısa örnekleri bin-pack için true") to an English comment; keep all YAML keys (backend, lora.r, lora.alpha, lora.method, lora.target_modules, data.dataset_name_or_path, training.trainer_type, training.packing, etc.) unchanged so only the comments are modified.tools/check_yaml_snippets.py-53-68 (1)
53-68:⚠️ Potential issue | 🟠 Major | ⚡ Quick winProvide install guidance instead of silently skipping on import failures.
Lines 57 and 68 catch import failures and exit silently with code
0. Per the coding guidelines (docs/standards/error-handling.md), missing optional dependencies should raiseImportErrorwith install hints, not silent success.Additionally, PyYAML is a declared dependency in
pyproject.tomlandrequirements.txt(not optional), so line 57 should be handled differently than the forgelm import. If this tool is intended to run in lint-only environments where forgelm isn't installed, use explicit--allow-missing-forgelmmode instead of silent exit. The current approach makes it harder to diagnose missing dependencies during actual failures.Consider the pattern in
tools/build_usermanuals.pyfor reference: it exits 2 with an explicit install message.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tools/check_yaml_snippets.py` around lines 53 - 68, The current import handlers silently exit with code 0; instead, for the required dependency PyYAML (import yaml) change the handler to print a clear install message and exit non-zero (exit code 2) or re-raise ImportError with an install hint (e.g., "pip install pyyaml"); for the optional forgelm import (from forgelm.config import ForgeConfig and pydantic.ValidationError) implement an explicit CLI flag (e.g., --allow-missing-forgelm) that, when set, prints a warning and continues, otherwise prints an error with install instructions for forgelm and exits non-zero (exit code 2) or raises ImportError—update the two except blocks to follow this behavior and reference the yaml import and the forgelm.config / ValidationError imports to locate the code to change.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: ca4bc064-ee3b-47ea-ab03-fe0c93146e63
📒 Files selected for processing (142)
CHANGELOG.mdCLAUDE.mdCONTRIBUTING.mdREADME.mddocs/design/blackwell_optimized.mddocs/design/gdpr_erasure.mddocs/design/iso27001_soc2_alignment.mddocs/design/library_api.mddocs/design/wizard_mode.mddocs/guides/alignment-tr.mddocs/guides/cicd_pipeline-tr.mddocs/guides/cicd_pipeline.mddocs/guides/gdpr_erasure-tr.mddocs/guides/gdpr_erasure.mddocs/guides/human_approval_gate-tr.mddocs/guides/human_approval_gate.mddocs/guides/ingestion.mddocs/guides/iso_soc2_deployer_guide-tr.mddocs/guides/iso_soc2_deployer_guide.mddocs/guides/library_api-tr.mddocs/guides/library_api.mddocs/guides/performance-tr.mddocs/guides/performance.mddocs/guides/quickstart-tr.mddocs/guides/quickstart.mddocs/guides/safety_compliance-tr.mddocs/guides/safety_compliance.mddocs/guides/troubleshooting-tr.mddocs/guides/troubleshooting.mddocs/qms/access_control-tr.mddocs/qms/access_control.mddocs/qms/encryption_at_rest-tr.mddocs/qms/encryption_at_rest.mddocs/qms/risk_treatment_plan-tr.mddocs/qms/risk_treatment_plan.mddocs/qms/sop_incident_response-tr.mddocs/qms/sop_incident_response.mddocs/qms/statement_of_applicability-tr.mddocs/qms/statement_of_applicability.mddocs/reference/approvals_subcommand-tr.mddocs/reference/approvals_subcommand.mddocs/reference/approve_subcommand-tr.mddocs/reference/approve_subcommand.mddocs/reference/architecture-tr.mddocs/reference/architecture.mddocs/reference/audit_event_catalog-tr.mddocs/reference/audit_event_catalog.mddocs/reference/compliance_summary-tr.mddocs/reference/compliance_summary.mddocs/reference/configuration-tr.mddocs/reference/configuration.mddocs/reference/data_preparation-tr.mddocs/reference/data_preparation.mddocs/reference/distributed_training-tr.mddocs/reference/distributed_training.mddocs/reference/iso27001_control_mapping-tr.mddocs/reference/iso27001_control_mapping.mddocs/reference/library_api_reference-tr.mddocs/reference/library_api_reference.mddocs/reference/purge_subcommand-tr.mddocs/reference/purge_subcommand.mddocs/reference/reverse_pii_subcommand-tr.mddocs/reference/reverse_pii_subcommand.mddocs/reference/safety_eval_subcommand-tr.mddocs/reference/safety_eval_subcommand.mddocs/reference/soc2_trust_criteria_mapping-tr.mddocs/reference/soc2_trust_criteria_mapping.mddocs/reference/supply_chain_security-tr.mddocs/reference/supply_chain_security.mddocs/reference/usage-tr.mddocs/reference/usage.mddocs/reference/verify_audit-tr.mddocs/reference/verify_audit.mddocs/standards/testing.mddocs/usermanuals/en/compliance/audit-log.mddocs/usermanuals/en/compliance/human-oversight.mddocs/usermanuals/en/compliance/model-card.mddocs/usermanuals/en/compliance/overview.mddocs/usermanuals/en/concepts/data-formats.mddocs/usermanuals/en/data/audit.mddocs/usermanuals/en/data/pii-ml.mddocs/usermanuals/en/data/secrets.mddocs/usermanuals/en/data/synthetic-data.mddocs/usermanuals/en/deployment/deploy-targets.mddocs/usermanuals/en/evaluation/auto-revert.mddocs/usermanuals/en/evaluation/judge.mddocs/usermanuals/en/evaluation/safety.mddocs/usermanuals/en/getting-started/project-layout.mddocs/usermanuals/en/operations/experiment-tracking.mddocs/usermanuals/en/operations/gpu-cost.mddocs/usermanuals/en/operations/iso-soc2-deployer.mddocs/usermanuals/en/operations/vram-fit-check.mddocs/usermanuals/en/operations/webhooks.mddocs/usermanuals/en/reference/cli.mddocs/usermanuals/en/reference/exit-codes.mddocs/usermanuals/en/reference/yaml-templates.mddocs/usermanuals/en/training/dpo.mddocs/usermanuals/en/training/grpo.mddocs/usermanuals/en/training/kto.mddocs/usermanuals/en/training/orpo.mddocs/usermanuals/en/training/sft.mddocs/usermanuals/en/training/simpo.mddocs/usermanuals/tr/compliance/audit-log.mddocs/usermanuals/tr/compliance/human-oversight.mddocs/usermanuals/tr/compliance/model-card.mddocs/usermanuals/tr/compliance/overview.mddocs/usermanuals/tr/concepts/data-formats.mddocs/usermanuals/tr/data/audit.mddocs/usermanuals/tr/data/pii-ml.mddocs/usermanuals/tr/data/secrets.mddocs/usermanuals/tr/data/synthetic-data.mddocs/usermanuals/tr/deployment/deploy-targets.mddocs/usermanuals/tr/evaluation/auto-revert.mddocs/usermanuals/tr/evaluation/judge.mddocs/usermanuals/tr/evaluation/safety.mddocs/usermanuals/tr/getting-started/first-run.mddocs/usermanuals/tr/getting-started/installation.mddocs/usermanuals/tr/getting-started/project-layout.mddocs/usermanuals/tr/operations/cicd.mddocs/usermanuals/tr/operations/experiment-tracking.mddocs/usermanuals/tr/operations/gpu-cost.mddocs/usermanuals/tr/operations/iso-soc2-deployer.mddocs/usermanuals/tr/operations/vram-fit-check.mddocs/usermanuals/tr/operations/webhooks.mddocs/usermanuals/tr/reference/cli.mddocs/usermanuals/tr/reference/exit-codes.mddocs/usermanuals/tr/reference/yaml-templates.mddocs/usermanuals/tr/training/dpo.mddocs/usermanuals/tr/training/grpo.mddocs/usermanuals/tr/training/kto.mddocs/usermanuals/tr/training/orpo.mddocs/usermanuals/tr/training/sft.mddocs/usermanuals/tr/training/simpo.mdsite/compliance.htmlsite/css/style.csssite/quickstart.htmltools/check_audit_event_catalog.pytools/check_bilingual_code_blocks.pytools/check_bilingual_parity.pytools/check_doc_numerical_claims.pytools/check_library_api_doc.pytools/check_yaml_snippets.py
…tions purge Address the 35 review comments from gemini-code-assist on PR #39 (12 outside-diff + 23 major). Every finding verified against current code state before applying the fix; EN+TR bilingual pairs kept in lockstep. Code bugs surfaced by the doc audit: - docs/guides/library_api.md + -tr.md: f-string SyntaxError (nested double quotes inside expression). Switched inner dict access to single quotes. - docs/usermanuals/en/operations/webhooks.md + -tr.md: "ForgeLM rejects HTTP webhook URLs" was wrong — `forgelm/webhook.py` calls `safe_post(..., allow_insecure_http=True)` and only logs a warning on `http://` URLs (`_send`). Rewrote to "TLS strongly recommended". - tools/check_library_api_doc.py + check_yaml_snippets.py: silent sys.exit(0) on import failure → exit 2 with install hint. CI guards no longer pass green when the environment is broken. Trainer schema purge (sft/dpo/simpo/orpo/kto/grpo, EN+TR): - DPO: phantom `loss_type` / `label_smoothing` / `reference_free` / `reference_model` / `loss_dpop_lambda` / `pref_chosen_weight` rows removed. Real schema: only `training.dpo_beta` is configurable. - SimPO: phantom `loss_type` / `length_normalize` / `label_smoothing` removed; tables aligned to `simpo_beta` / `simpo_gamma`. Default `simpo_gamma` corrected 1.0 → 0.5. β-γ combination tables relabeled. - ORPO: phantom `loss_type` / `sft_weight` removed; tables aligned to `training.orpo_beta`. - KTO: phantom `desirable_weight` / `undesirable_weight` / `loss_type` removed; tables + tip section aligned to "only `kto_beta` configurable; oversample minority class for imbalance". - GRPO: nested `training.grpo: { reward_function, format_reward, answer_pattern }` block + phantom `group_size` / `beta` / `temperature` / `max_completion_length` (legacy alias) rows removed. Real schema: flat `grpo_num_generations` / `grpo_max_completion_length` / `grpo_reward_model`; built-in shaper is in `grpo_rewards.py`, no user-configurable weight knob. - SFT: tokeniser → tokenizer normalisation (single-page style drift). Audit page contradictions (EN+TR): - `--strict` + `--sample-rate` removal-notice already noted, but CI examples (lines 150 / 144) still used `--strict` and pitfalls warned about `--sample-rate` as if it shipped. Replaced both with the canonical `--output-format json | jq -e '...'` gate. - audit-tr.md: `_parser.py` line range "335-468" → symbol-only `_add_audit_subcommand`. `forgelm/config.py:566/641` brittle line refs in compliance/overview.md + qms/access_control.md(-tr) and operations/webhooks.md(-tr) → `::WebhookConfig` / `::ComplianceMetadataConfig` symbol refs. Internal-contradiction sweeps: - evaluation/auto-revert.md: "Custom guard fails / `evaluation.guards.<name>`" table row removed (already documented as nonexistent). Trigger table rewritten to match the four real guard families (benchmark.min_score / safety.max_safety_regression / safety.severity_thresholds / llm_judge.min_score / max_acceptable_loss). - operations/experiment-tracking.md: phantom `training.tags` / `training.notes` / `training.wandb.{log_artifacts,artifact_type}` blocks replaced with the real env-var convention (WANDB_TAGS, WANDB_NOTES, WANDB_LOG_MODEL, MLFLOW_TAGS); absolute "no nested backend blocks" claim kept but the conflicting `training.wandb.*` artifact example was the source of the contradiction — now removed. - compliance/human-oversight.md: phantom `compliance.human_approval` / `approval.{request_webhook,signature_method,timeout_hours, require_role,quorum,webhook_url}` block + ed25519 signature claim + quorum + timeout flow all replaced with the real shipped behaviour (only `evaluation.require_human_approval`; HMAC chain; no quorum; no timeout — `retention.staging_ttl_days` for stale-cleanup). - compliance/audit-log.md: legacy event names (`run_start`, `run_complete`, `data_audit_complete`, `training_epoch_complete`, `benchmark_complete`, `safety_eval_complete`, `auto_revert`) replaced with canonical names from `audit_event_catalog.md`. Phantom `compliance.audit_log:` config block + `forward_to:` array removed. Stale-reference fixes: - troubleshooting.md + -tr.md exit-code 4 now points to `forgelm approvals --show <run_id>` + `forgelm approve / reject` and the canonical `final_model.staging.<run_id>/` path (was: nonexistent `checkpoints/compliance/`). - approvals_subcommand.md + -tr.md `.age` → `.age_seconds` field name (matches the JSON envelope); jq example updated. - compliance_summary.md + -tr.md: `forgelm approvals --pending` → `forgelm approvals --pending --output-dir <dir>` (the flag is required by the dispatcher). - iso_soc2_deployer_guide.md + -tr.md: `config_hash` (per-run manifest sidecar field) clarified — the field is read forward-compatibly by `_approvals.py` but no current producer emits it; the real run-pinning evidence lives in `pipeline.training_started` event payload. - design/gdpr_erasure.md `--yes` semantics drift: §4.2 exit-code matrix + Q4 design rationale now match the shipped non-interactive behaviour (consent recorded via `--justification`, no prompt, no `--yes`). - distributed_training-tr.md FSDP YAML: bilingual parity restored (added `fsdp_backward_prefetch` + `fsdp_state_dict_type` to match EN + the FSDP-offload clarifying comment). - operations/cicd.md + -tr.md: GitLab pipeline now ships a real `deploy:` job (was: stage declared, job missing); audit step uses the `--output-format json | jq -e` gate; `checkpoints/run/artifacts/` artifact paths corrected to top-level `checkpoints/run/`. Secrets taxonomy normalisation (EN+TR): - data/secrets.md: example output (lines 35-41) used non-canonical `github_pat` / `pem_block` / `openai_key` names. Replaced with `github_token` / `openssh_private_key` / `openai_api_key` from `_SECRET_PATTERNS`. - PEM section: phantom `[REDACTED-PEM-BLOCK]` per-family token replaced with the real single-replacement contract (`mask_secrets()` ships one `replacement="[REDACTED-SECRET]"` constant for every family). - `secrets_mask.categories` example list updated to canonical names. - `_secrets.py:35` / `:106` line refs → `::_SECRET_PATTERNS` / `::mask_secrets` symbol refs. Webhook smoke-test guidance (EN+TR): - The `--dry-run` recommendation was misleading — dry-run is config-validation only, doesn't fire the trainer lifecycle, so webhooks aren't exercised. Replaced with "real tiny training run (small dataset + low num_train_epochs)" or `curl` synthetic POST. Turkish-in-code violations (per coding standard "code/CLI must be English-only in bilingual docs"): - quickstart-tr.md: install/fit-check shell comments → English. - judge-tr.md: YAML inline comments → English. - sft-tr.md: backend / method / packing comments → English. - safety_compliance-tr.md: safety scoring YAML comments + CI bash echo → English. Validation: 8/8 doc CI guards green; ruff check + format clean; usermanuals regenerated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ixes Address every SonarQube finding on PR #39's tooling + site: site/quickstart.html L361 — Major (Code Smell, "unused"): Removed the verbose multi-line wizard-modal documentation comment that Sonar flagged as commented-out code. Wizard behaviour is documented in site/js/wizard.js comments; the HTML element is self-evident. tools/ cognitive-complexity refactors (Critical, "brain-overload"): Each function below split into smaller units; behaviour and exit codes preserved (all 8 doc guards still green). - check_audit_event_catalog.py main() 19 → 15 Extracted: _build_arg_parser, _print_code_only, _print_catalog_only. - check_bilingual_code_blocks.py _pair_diff() 23 → 15 Extracted: _yaml_key_diff_line (per-block YAML diff), _block_pair_diff (lang + yaml dispatch). _pair_diff is now a thin loop over the per-block helper. - check_doc_numerical_claims.py canonical_secret_families() 18 → 15 Extracted: _secret_patterns_dict_node — handles AnnAssign + Assign uniformly, returns the ast.Dict literal or None. search_doc_claims() 19 → 15 Extracted: _is_indexable_doc (skip analysis/marketing), _scan_line_for_mismatches (per-line match → Mismatch list). - check_library_api_doc.py main() 20 → 15 Extracted: _build_arg_parser, _import_forgelm_or_explain (returns module or None for fail-closed exit 2), _print_drift_section (renders one named drift bucket). - check_yaml_snippets.py main() 24 → 15 Extracted: _build_arg_parser, _is_validated_forgelm_snippet (count "passed validation" ForgeConfig-shaped snippets without re-running the parser inside main's loop), _print_failures. tools/check_library_api_doc.py L45 — Minor (regex, "concise character class"): Replaced `[A-Za-z0-9_]` with `\w` in the `_DOC_ROW_RE` symbol-name pattern. Behaviour preserved: `[A-Za-z_]\w*` still rejects leading digits, accepts the same trailing characters as before. (re module default is ASCII-only for `\w` in str patterns when the regex doesn't set re.UNICODE — Python 3.x str regex sets re.UNICODE implicitly, but `\w` here is bounded by the leading `[A-Za-z_]` so the unicode-letter case can never start a symbol name. Match set identical for the realistic input — Python identifiers in the library reference markdown.) Validation: 8/8 doc CI guards green; ruff check + format clean on tools/. No behavioural change to any guard's output. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
@coderabbitai review |
✅ Actions performedReview triggered.
|
There was a problem hiding this comment.
Actionable comments posted: 15
🧹 Nitpick comments (1)
docs/guides/troubleshooting-tr.md (1)
51-51: ⚡ Quick winConsider fixing markdown formatting for consistency.
Static analysis tools flagged several markdown formatting issues that would improve rendering and standards compliance:
Missing blank lines around code blocks (lines 51, 58, 64, 69, 76, 82, 89, 115, 121, 127, 158, 164, 170, 176, 330): Add blank lines before and after fenced code blocks per MD031.
Missing language specifiers (lines 238, 252, 286, 357): Add language identifiers to error message code blocks for proper syntax highlighting. These appear to be error output, so consider using
textorplaintext.Missing blank line before list (line 97): Add a blank line before the
**Sebepler:**list.Example fix for line 238:
-``` +```text Configuration validation failed: 1 validation error for ForgeConfigAlso applies to: 58-58, 64-64, 69-69, 76-76, 82-82, 89-89, 97-97, 115-115, 121-121, 127-127, 158-158, 164-164, 170-170, 176-176, 238-238, 252-252, 286-286, 330-330, 357-357
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/troubleshooting-tr.md` at line 51, The markdown has inconsistent fenced-code formatting: add a blank line before and after every fenced code block (e.g. the ```yaml block shown) to satisfy MD031, add language specifiers for the error output blocks (use `text` or `plaintext` for the blocks at the flagged positions such as the error examples around lines 238, 252, 286, 357), and insert a blank line before the list headed "**Sebepler:**". Update each fenced block accordingly across the document so every triple-backtick block has a surrounding blank line and appropriate language identifier.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/guides/quickstart-tr.md`:
- Around line 145-152: The fenced code block that begins with
"./checkpoints/final_model/" in the quickstart-tr guide lacks a language tag,
causing markdownlint MD040; update that block by changing the opening fence from
``` to ```text (or another appropriate language) so the code block is explicitly
labeled (e.g., use ```text for the directory tree).
- Around line 300-301: Mevcut metinde dosya satır numarası anchor'ı
(`forgelm/config.py:589-635`) kullanılmış; bunu kırılmaya daha dayanıklı olacak
şekilde sembol/tabanlı referansa çevirin: metindeki satır-anchor'ı kaldırıp
yerine `SyntheticConfig` tanımına yönelik sembol tabanlı referans kullanın (ör.
belgedeki veya repo linkindeki `SyntheticConfig` tanımı/başlığına işaret eden
sembol referansı). Aşağıdaki öğelerde değişiklik yapın: belge metni adlandırılan
referansı içeren cümle, link hedefi ve varsa alt metin açıklaması; referans için
kullanılan sembolün tam adı "SyntheticConfig" olduğundan emin olun.
In `@docs/guides/safety_compliance-tr.md`:
- Around line 317-319: The audit example uses non-canonical event names
("pipeline.started" and "training.completed"); update the snippet to use the
canonical event names used across the PR and in
docs/reference/audit_event_catalog.md (e.g., use "training.started" and
"pipeline.completed") so the example aligns with the contract; search for and
replace these two event strings in the integrity example in
docs/guides/safety_compliance-tr.md and verify consistency with the
audit_event_catalog.md event surface.
In `@docs/reference/compliance_summary-tr.md`:
- Around line 140-143: Update the numeric ISO control counts in the Turkish
paragraph that starts "ForgeLM, 93 ISO 27001 Annex A kontrolünün ~59'una teknik
kanıt sağlar" so they match the English compliance summary's "~61 / ~32" (or, if
the TR text intentionally differs, add a concise parenthetical explaining the
reason for the intentional divergence); ensure the sentence and any adjacent
counts (`~59 / ~34`) are replaced or annotated to exactly reflect the EN numbers
(`~61 / ~32`) or include the explicit justification for the difference so both
versions remain consistent and traceable to the underlying code-backed mapping.
In `@docs/usermanuals/en/compliance/audit-log.md`:
- Around line 70-77: The docs claim the canonical audit log location is
`<output_dir>/audit_log.jsonl` but examples still reference
`checkpoints/run/artifacts/audit_log.jsonl`; update all verify/command snippets
in this document to use the canonical `<output_dir>/audit_log.jsonl` path so
examples match the stated contract, including any uses in "verify" command
examples and the example section that currently points to
`checkpoints/run/artifacts/audit_log.jsonl`; search for the string
`checkpoints/run/artifacts/audit_log.jsonl` (and any variants) and replace them
with `<output_dir>/audit_log.jsonl` so the README and the code contract are
consistent.
In `@docs/usermanuals/en/data/secrets.md`:
- Line 26: Update the sentence that currently states all matches are replaced
with `[REDACTED-SECRET]` to explicitly say this is the default behavior of
mask_secrets() and that category-tagging is an opt-in mode; specifically,
mention that mask_secrets() (`forgelm/data_audit/_secrets.py::mask_secrets`)
replaces matches with `[REDACTED-SECRET]` by default and add a short pointer
that when `tag_by_category: true` the output uses category-specific tags as
described later (see the existing description of tag_by_category); keep the same
function reference to avoid confusion.
In `@docs/usermanuals/en/operations/cicd.md`:
- Line 122: The Jenkins sh command string has mismatched quotes around the jq
filter in the forgelm audit pipeline; fix it by making the outer sh string use
double quotes (or escape the inner quotes) so the jq expression '.verdict !=
"errors" and .pii_summary.severity != "high"' is passed intact to jq; update the
sh invocation line (the sh 'forgelm audit ... | jq -e ...' command) to use
consistent quoting so the shell sees the full jq filter as a single argument.
In `@docs/usermanuals/tr/compliance/audit-log.md`:
- Around line 75-80: Remove the stale sentence that references a project-wide
artifacts/.forgelm/global-audit-log.jsonl and ensure the section consistently
states that there is no global log; instead keep/clarify that each run writes
<output_dir>/audit_log.jsonl (and the genesis-pin sidecar
<output_dir>/audit_log.jsonl.manifest.json), that there is no
compliance.audit_log: YAML knob, and that HMAC chaining is enabled via the
FORGELM_AUDIT_SECRET env var before running the trainer.
In `@docs/usermanuals/tr/compliance/human-oversight.md`:
- Around line 63-64: Update the sequence diagram text to match the
documentation’s statement that only the CLI subcommand pair `forgelm approve` /
`forgelm reject` is supported: locate the diagram label or caption that
currently mentions "CLI / webhook callback" and change it to "CLI only (forgelm
approve / forgelm reject)". Ensure any arrows or participant notes referencing a
webhook callback or approval-resume HTTP endpoint are removed or replaced with
CLI-only wording so the flow in the diagram aligns with the `forgelm approve` /
`forgelm reject` mechanism described in the `human-oversight.md` text.
In `@docs/usermanuals/tr/data/audit.md`:
- Around line 125-135: Replace the incorrect JSON snippet: rename the top-level
"quality_flags" to "quality_summary" and change its internal layout to include
"samples_flagged", "samples_evaluated", "by_check" (an object with counts) and
"overall_quality_score"; inside "by_check" use the real checks low_alpha_ratio,
low_punct_endings, abnormal_mean_word_length, short_paragraphs, and
repeated_lines (replace abnormal_word_length and remove short_response), and
remove the entire "preference_specific" section (identical_chosen_rejected,
empty_chosen, swapped_likely) so the example matches the actual audit output
structure.
In `@docs/usermanuals/tr/operations/experiment-tracking.md`:
- Around line 78-79: Lines showing the sampling interval for metrics
`system/gpu_utilization` and `system/vram_used_gb` use the typographically
awkward token "30s'de"; update those two entries to match the documentation
localization/style guide by changing "30s'de" to "30 s'de" (or your repository's
canonical spacing/format from docs/standards/localization.md) so the wording
reads consistently and follows docs/standards/documentation.md.
- Line 61: Satırdaki yanlış ifade "`tensorboardX` `transformers` ile birlikte
gelir" yanlış; güncelleme yaparak bu iddiayı kaldırın ve yerine TensorBoard
entegrasyonu için kullanıcıların ya `tensorboard` (PyTorch >= 1.4 ile) ya da
`tensorboardX`'i ayrı olarak kurmaları gerektiğini belirten metni koyun
(örneğin: "TensorBoard entegrasyonu için `tensorboard` (PyTorch >= 1.4 ile) veya
`tensorboardX` ayrıca kurulmalıdır."), ilgili metin parçasını (mevcut
'`tensorboardX` `transformers` ile birlikte gelir' ifadesi) bulun ve değiştirin.
- Line 65: The doc incorrectly states TRAINER_REPORT_TO is an HF Transformers
convention; update the text around training.report_to to either remove the
TRAINER_REPORT_TO example or mark it explicitly as a ForgeLM-specific extension
(not part of transformers.TrainingArguments), and keep the valid --report-to CLI
override and reference to transformers.TrainingArguments.report_to; ensure the
sentence clearly distinguishes HF’s supported mechanisms (constructor /
--report-to) from any ForgeLM environment-variable shortcut (TRAINER_REPORT_TO)
if you choose to document it.
In `@tools/check_library_api_doc.py`:
- Line 2: The module docstring contains a non-English token ("Faz"); update the
top-level docstring in tools/check_library_api_doc.py to be English-only (e.g.,
change "Wave 6 / Faz 31 — `forgelm.__all__` ↔ library_api_reference.md
cross-check." to an all-English phrase such as "Wave 6 / Phase 31 —
`forgelm.__all__` ↔ library_api_reference.md cross-check.") so the module
docstring follows repository English-only docstring guidelines.
- Around line 109-111: The current check uses args.doc.exists() only, which
misses directories or unreadable files and lets later code (the open/read at the
early file handling) raise an exception; update the validation around args.doc
to ensure it both exists and is a regular file and is readable (use
args.doc.exists(), args.doc.is_file(), and os.access(args.doc, os.R_OK)) and if
any check fails print a clear stderr message and return 1; additionally wrap the
actual file open/read (the code that currently raises) in a try/except that
catches OSError/IOError, logs the error to stderr via the same message pattern,
and returns a non-zero exit code so failures are handled cleanly (refer to
args.doc, .is_file(), os.access, and the file open call in this module).
---
Nitpick comments:
In `@docs/guides/troubleshooting-tr.md`:
- Line 51: The markdown has inconsistent fenced-code formatting: add a blank
line before and after every fenced code block (e.g. the ```yaml block shown) to
satisfy MD031, add language specifiers for the error output blocks (use `text`
or `plaintext` for the blocks at the flagged positions such as the error
examples around lines 238, 252, 286, 357), and insert a blank line before the
list headed "**Sebepler:**". Update each fenced block accordingly across the
document so every triple-backtick block has a surrounding blank line and
appropriate language identifier.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 77d111ef-5773-4fd6-bf4c-885220e700fa
📒 Files selected for processing (53)
docs/design/gdpr_erasure.mddocs/guides/iso_soc2_deployer_guide-tr.mddocs/guides/iso_soc2_deployer_guide.mddocs/guides/library_api-tr.mddocs/guides/library_api.mddocs/guides/quickstart-tr.mddocs/guides/safety_compliance-tr.mddocs/guides/troubleshooting-tr.mddocs/guides/troubleshooting.mddocs/qms/access_control-tr.mddocs/qms/access_control.mddocs/reference/approvals_subcommand-tr.mddocs/reference/approvals_subcommand.mddocs/reference/compliance_summary-tr.mddocs/reference/compliance_summary.mddocs/reference/distributed_training-tr.mddocs/usermanuals/en/compliance/audit-log.mddocs/usermanuals/en/compliance/human-oversight.mddocs/usermanuals/en/compliance/overview.mddocs/usermanuals/en/data/audit.mddocs/usermanuals/en/data/secrets.mddocs/usermanuals/en/evaluation/auto-revert.mddocs/usermanuals/en/operations/cicd.mddocs/usermanuals/en/operations/experiment-tracking.mddocs/usermanuals/en/operations/webhooks.mddocs/usermanuals/en/training/dpo.mddocs/usermanuals/en/training/grpo.mddocs/usermanuals/en/training/kto.mddocs/usermanuals/en/training/orpo.mddocs/usermanuals/en/training/sft.mddocs/usermanuals/en/training/simpo.mddocs/usermanuals/tr/compliance/audit-log.mddocs/usermanuals/tr/compliance/human-oversight.mddocs/usermanuals/tr/compliance/overview.mddocs/usermanuals/tr/data/audit.mddocs/usermanuals/tr/data/secrets.mddocs/usermanuals/tr/evaluation/auto-revert.mddocs/usermanuals/tr/evaluation/judge.mddocs/usermanuals/tr/operations/cicd.mddocs/usermanuals/tr/operations/experiment-tracking.mddocs/usermanuals/tr/operations/webhooks.mddocs/usermanuals/tr/training/dpo.mddocs/usermanuals/tr/training/grpo.mddocs/usermanuals/tr/training/kto.mddocs/usermanuals/tr/training/orpo.mddocs/usermanuals/tr/training/sft.mddocs/usermanuals/tr/training/simpo.mdsite/quickstart.htmltools/check_audit_event_catalog.pytools/check_bilingual_code_blocks.pytools/check_doc_numerical_claims.pytools/check_library_api_doc.pytools/check_yaml_snippets.py
💤 Files with no reviewable changes (1)
- site/quickstart.html
✅ Files skipped from review due to trivial changes (2)
- docs/reference/distributed_training-tr.md
- docs/usermanuals/en/compliance/overview.md
🚧 Files skipped from review as they are similar to previous changes (28)
- docs/usermanuals/tr/training/kto.md
- docs/usermanuals/en/training/orpo.md
- docs/guides/troubleshooting.md
- docs/usermanuals/tr/operations/cicd.md
- docs/usermanuals/en/data/audit.md
- docs/usermanuals/tr/training/orpo.md
- docs/guides/iso_soc2_deployer_guide.md
- docs/usermanuals/en/training/simpo.md
- docs/qms/access_control.md
- docs/qms/access_control-tr.md
- docs/usermanuals/tr/data/secrets.md
- docs/usermanuals/en/training/dpo.md
- tools/check_audit_event_catalog.py
- docs/usermanuals/en/training/kto.md
- docs/design/gdpr_erasure.md
- docs/usermanuals/en/operations/experiment-tracking.md
- docs/usermanuals/en/compliance/human-oversight.md
- docs/usermanuals/tr/evaluation/auto-revert.md
- docs/guides/iso_soc2_deployer_guide-tr.md
- docs/usermanuals/en/evaluation/auto-revert.md
- docs/usermanuals/tr/training/sft.md
- tools/check_doc_numerical_claims.py
- tools/check_bilingual_code_blocks.py
- docs/usermanuals/tr/compliance/overview.md
- docs/usermanuals/tr/training/dpo.md
- docs/usermanuals/tr/evaluation/judge.md
- docs/usermanuals/en/training/sft.md
- tools/check_yaml_snippets.py
Each finding verified against current code/state before applying;
EN+TR mirrors kept in lockstep. One finding skipped with explicit
reason (see below).
Quickstart (EN + TR):
- quickstart-tr.md L145 / quickstart.md L133: fenced-block missing
language tag (markdownlint MD040). Added `text` tag to the
`final_model/` directory tree. EN parity fix included.
- quickstart-tr.md L300-301 / quickstart.md L281: brittle
`forgelm/config.py:589-635` line-anchor link → symbol-based
reference to the `SyntheticConfig` Pydantic model with a
repo-search link. EN parity fix included.
Safety compliance (EN + TR):
- safety_compliance-tr.md L317-318 / safety_compliance.md L279-280:
JSON example used non-canonical event names. Replaced
`pipeline.started` → `training.started` and `training.completed` →
`pipeline.completed` (canonical names per
`docs/reference/audit_event_catalog.md`). EN parity fix included.
Compliance summary (EN):
- compliance_summary.md L146-147: ISO control counts (~61/~32) did
not match the underlying `iso27001_control_mapping.md`. Real
counts: 11 FL + 48 FL-helps = 59 ForgeLM-evidence + 34 OOS = 93
(TR was already correct at ~59/~34). Updated EN to ~59/~34 with
explicit FL/FL-helps/OOS breakdown.
Audit log (EN + TR):
- audit-log.md L62 / audit-log-tr.md L63: verify-audit example
used `checkpoints/run/artifacts/audit_log.jsonl` despite the
canonical contract being `<output_dir>/audit_log.jsonl`. Fixed
both to the canonical placeholder.
- audit-log-tr.md L71-73: stale paragraph still claimed a
project-wide `.forgelm/global-audit-log.jsonl` directly above the
new "no global log" paragraph. Removed; H2 retitled "Koşu başına"
to match EN "Per-run".
Human oversight (EN + TR):
- human-oversight.md / -tr.md L23-25: sequence diagram showed
"Webhook + structured request" + "Sign approval (CLI / webhook
callback)", but the page text already documents the CLI subcommand
pair as the only mechanism (no webhook callback, no JWT resume
endpoint). Diagram lines updated to "Webhook notification —
informational only" + "CLI only (forgelm approve / forgelm
reject)" + "HMAC chain" (replacing the phantom "ed25519 signature"
framing).
Audit data report (EN + TR):
- audit.md L116-142 / audit-tr.md L116-138: JSON snippet's
`quality_flags` block used phantom field names (`short_response`,
`abnormal_word_length`) and shipped a phantom `preference_specific`
block (`identical_chosen_rejected`, `empty_chosen`,
`swapped_likely`). Real schema in `_aggregator.py` +
`_quality.py`: `quality_summary` with `samples_flagged`,
`samples_evaluated`, `by_check` (containing the canonical 5 checks
`low_alpha_ratio`, `low_punct_endings`,
`abnormal_mean_word_length`, `short_paragraphs`, `repeated_lines`),
and `overall_quality_score`. `preference_specific` block does not
exist in code; removed. EN+TR aligned.
Operations / CI/CD (EN + TR):
- cicd.md L122 / cicd-tr.md L123: Jenkins `sh '...'` quoting was
broken — outer single quotes terminated on the inner jq filter's
`'.verdict ...'`. Fixed by switching to triple single-quote
Groovy strings so the jq filter is passed intact.
Operations / experiment tracking (EN + TR):
- experiment-tracking-tr.md L78-79: typographic "30s'de" → "30 s'de"
(TR style; EN already used "30s" as the ASCII-correct form, no
change needed).
- experiment-tracking.md L61 / -tr.md L61: "tensorboardX ships with
transformers" was wrong — neither package is bundled. HF
Transformers' `is_tensorboard_available` probes for `tensorboard`
OR `tensorboardX` and `TensorBoardCallback` raises if neither is
present. Updated to require operators to install `tensorboard`
(PyTorch ≥ 1.4) or `tensorboardX` separately.
- experiment-tracking.md L65 / -tr.md L65: `TRAINER_REPORT_TO=...`
env var was presented as an HF Transformers convention; no such
env var exists in transformers (verified by grep on the installed
package), and ForgeLM does not honour it either. Removed the
phantom example; kept the constructor + `--report-to` CLI override
as the supported paths.
tools/check_library_api_doc.py:
- L2: docstring contained the Turkish "Faz" — replaced with
"Phase" (English-only docstring guideline).
- L62-…: hardened `--doc` validation. Added `_validate_doc_arg`
helper that checks `Path.exists()` + `Path.is_file()` +
`os.access(..., os.R_OK)` and returns a clear stderr message
mapped to exit 1 by main(). Also wrapped `doc_symbols(args.doc)`
in `try/except OSError` to handle race conditions (file deleted
between validation and read). Adds `os` + `Optional` imports.
(Sonar pre-existing flag against the original validate-only
pattern — now safe against directories, unreadable files, and
racy deletes.)
Markdownlint (EN + TR troubleshooting):
- troubleshooting.md / -tr.md: 4 untagged fenced error-output
blocks (lines ~227, 239, 268, 337 in EN; corresponding lines in
TR) were flagged by MD040. Added `text` language tags to all
four; bilingual_code_blocks guard verifies count + lang stay in
lockstep across the EN/TR pair.
Secrets reference (EN + TR):
- data/secrets.md L97-111 / -tr.md L97-110: phantom
`ingestion.secrets_mask:` config block (with
`enabled` / `tag_by_category` / `strict` / `categories` sub-fields)
documented features that do not exist — `secrets_mask` is a
`bool` argument on `audit_dataset()`. Replaced the phantom YAML
block with a prose paragraph stating the always-on contract and
pointing at `_SECRET_PATTERNS` for extension. (This was a related
upgrade to the L26 finding: the original audit comment asked for
a "default-vs-opt-in" clarification of `tag_by_category`, but the
field is phantom — removing the block is the correct fix and
makes the L26 sentence factually correct without further change.)
Skipped findings:
- data/secrets.md L26: skipped as written. The audit comment asked
for a "[REDACTED-SECRET] is the default behaviour, tag_by_category
is an opt-in mode" clarification. `tag_by_category` doesn't exist
in the codebase (verified: not in `forgelm/data_audit/_secrets.py`
or `forgelm/ingestion.py`), so adding a pointer to it would
document a phantom feature. Resolved by removing the phantom
config block (above) instead — the surviving L26 sentence is
already accurate as-is.
Validation: 8/8 doc CI guards green; ruff check + format clean on
tools/; user manuals regenerated. New `_validate_doc_arg` smoke-tested
against `--doc /tmp` (returns "is not a regular file." + exit 1).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
@coderabbitai review |
✅ Actions performedReview triggered.
|
Review absorption — coderabbitai run @ commit b275e1815 inline findings + 1 nitpick from the latest CodeRabbit review pass; all verified against current code before action. EN+TR mirrors kept in lockstep. One finding skipped with explicit reason; rest landed in commit Fixed (15)
Skipped (1, with reason)
Nitpick (1, partial)
Validation8/8 doc CI guards green:
@coderabbitai review |
|
The MD031/MD032 deferral is reasonable given they're not gated by any of the 8 CI guards; a follow-up cleanup PR makes sense there. Triggering the review now. ✅ Actions performedReview triggered.
|
There was a problem hiding this comment.
Actionable comments posted: 6
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
docs/usermanuals/en/operations/experiment-tracking.md (1)
8-8:⚠️ Potential issue | 🟠 Major | ⚡ Quick winRemove unsupported Comet ML references from line 8.
Line 8 claims Comet ML is "first-class" but the config schema (
forgelm/config.py:274) only supportstensorboard,wandb,mlflow, andnone. Comet is not implemented. Lines 110–112 are fine (they reference Comet only in the context of a planned comparison-runs feature, not current tracking backend support).This affects both the English and Turkish mirrors identically.
Fix for line 8 (English and Turkish)
English:
-ForgeLM doesn't reinvent experiment tracking — it integrates with whatever your team already uses via the `training.report_to` field. W&B, MLflow, TensorBoard, and Comet ML are first-class. +ForgeLM doesn't reinvent experiment tracking — it integrates with whatever your team already uses via the `training.report_to` field. W&B, MLflow, and TensorBoard are first-class.Turkish:
-ForgeLM deney takibini yeniden icat etmez — ekibinizin zaten kullandığı şeye `training.report_to` alanı üzerinden entegre olur. W&B, MLflow, TensorBoard ve Comet ML birinci-sınıf desteklenir. +ForgeLM deney takibini yeniden icat etmez — ekibinizin zaten kullandığı şeye `training.report_to` alanı üzerinden entegre olur. W&B, MLflow ve TensorBoard birinci-sınıf desteklenir.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/en/operations/experiment-tracking.md` at line 8, Update the docs text on line 8 to remove Comet ML as a "first-class" experiment tracking backend: edit the sentence referencing training.report_to so it lists only the supported backends (tensorboard, wandb, mlflow, none) and remove Comet ML from both the English and Turkish files, and ensure this change matches the actual config schema in forgelm/config.py (see the training.report_to allowed values) so the docs and implementation stay consistent.docs/usermanuals/tr/compliance/human-oversight.md (1)
50-57:⚠️ Potential issue | 🟠 Major | ⚡ Quick winKeep CLI output examples in English-only.
Line 51 localizes terminal output; this breaks the repo rule for CLI/output snippets and can drift from real command output. Keep prose Turkish, but terminal output text in English.
Suggested patch
-[2026-04-29 14:33:10] İnsan onayı gerekli. +[2026-04-29 14:33:10] Human approval required.As per coding guidelines: “
docs/**/*.md: User-facing documentation must be bilingual (English + Turkish mirrors); code, CLI output, logs, and config keys must be English only”.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/tr/compliance/human-oversight.md` around lines 50 - 57, The CLI output block in human-oversight.md contains localized Turkish terminal lines; change the terminal output to English-only while keeping surrounding prose Turkish if desired. Specifically update the quoted output strings—replace "İnsan onayı gerekli." with "Human approval required.", "Onaylamak için:" with "To approve:", and "Reddetmek için:" with "To reject:" (leave commands like "forgelm approve abc123 --output-dir ..." unchanged) so the CLI/log snippet matches the repo rule that code/CLI output/logs/config keys are English only.
🧹 Nitpick comments (1)
docs/usermanuals/en/data/audit.md (1)
93-93: ⚡ Quick winRemove hardcoded parser line numbers from the authoritative-source reference.
This will drift on routine parser edits. Prefer file + symbol-only reference, consistent with the doc-link durability convention.
Proposed edit
-Authoritative source: `forgelm/cli/_parser.py::_add_audit_subcommand` (lines 335-468). +Authoritative source: `forgelm/cli/_parser.py::_add_audit_subcommand`.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/usermanuals/en/data/audit.md` at line 93, Update the authoritative-source reference in docs/usermanuals/en/data/audit.md by removing the hardcoded line number range and referencing only the file and symbol; replace "forgelm/cli/_parser.py::_add_audit_subcommand (lines 335-468)" with "forgelm/cli/_parser.py::_add_audit_subcommand" so the doc points to the file and the _add_audit_subcommand symbol only (no line numbers) to avoid drift when the parser changes.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/guides/safety_compliance-tr.md`:
- Line 353: The example JSON uses a stale value for "forgelm_version" (currently
set to "0.1.0"); update the example in the compliance_report.json snippet by
replacing "forgelm_version": "0.1.0" with a current version string or a neutral
placeholder like "x.y.z" (or the real latest release), ensuring the key
"forgelm_version" remains unchanged so users aren't misled by an outdated
version in the docs.
- Around line 3-5: Replace the absolute claim "tek açık-kaynak fine-tuning
aracıdır" with non-exclusive wording; locate the sentence containing that phrase
in docs/guides/safety_compliance-tr.md and change it to a qualified statement
such as "açık-kaynak fine-tuning araçlarından biridir" or "mevcut açık-kaynak
araçlar arasında yer alır," or add a citation/benchmark to substantiate
exclusivity if you must keep a stronger claim; ensure the revised sentence
preserves the original meaning about integrated safety assessment and EU AI Act
artifact generation.
In `@docs/guides/troubleshooting-tr.md`:
- Line 200: Replace the incorrect key name used in the Turkish prose: change the
reference from `max_tokens` to `max_new_tokens` so it matches the configuration
block; update the sentence on line 200 in docs/guides/troubleshooting-tr.md to
mention `max_new_tokens` (and, if applicable, ensure any explanatory text around
the teacher model suggestion also refers to `max_new_tokens`).
In `@docs/guides/troubleshooting.md`:
- Line 191: Summary: The doc uses the outdated token key name `max_tokens`
instead of the section's actual config name `max_new_tokens`. Fix: update the
sentence that currently reads “reducing `max_tokens`” to use the exact config
key `max_new_tokens` (or fully-qualified `synthetic.max_new_tokens` where
appropriate) so it matches the section’s config examples and avoids copy/paste
confusion; search for occurrences of `max_tokens` in this troubleshooting
paragraph and replace them with `max_new_tokens` (or `synthetic.max_new_tokens`)
to keep naming consistent with the rest of the document.
- Line 388: Update the documentation line that shows the approval workflow to
include the required --comment flag on both commands: change the examples using
the commands `forgelm approve <run_id> --output-dir <dir>` and `forgelm reject
<run_id> --output-dir <dir>` to `forgelm approve <run_id> --output-dir <dir>
--comment "<your comment>"` and `forgelm reject <run_id> --output-dir <dir>
--comment "<your comment>"` respectively so users following the `Awaiting
approval` guidance won't hit missing-comment errors; keep the staging path
(`<output_dir>/final_model.staging.<run_id>/`) text unchanged.
In `@docs/usermanuals/en/data/secrets.md`:
- Line 26: The docs contradict themselves about Anthropic coverage: update the
"False-positive guards" section so it no longer claims a built-in Anthropic-key
entropy guard if Anthropic patterns are not bundled; specifically edit the
description near the entropy-guard examples to either remove or reword the
Anthropic mention and add a clarifying note that mask_secrets()
(forgelm/data_audit/_secrets.py::mask_secrets) does not ship Anthropic patterns
by default and operators must extend the regex set out-of-tree if they need
Anthropic detection.
---
Outside diff comments:
In `@docs/usermanuals/en/operations/experiment-tracking.md`:
- Line 8: Update the docs text on line 8 to remove Comet ML as a "first-class"
experiment tracking backend: edit the sentence referencing training.report_to so
it lists only the supported backends (tensorboard, wandb, mlflow, none) and
remove Comet ML from both the English and Turkish files, and ensure this change
matches the actual config schema in forgelm/config.py (see the
training.report_to allowed values) so the docs and implementation stay
consistent.
In `@docs/usermanuals/tr/compliance/human-oversight.md`:
- Around line 50-57: The CLI output block in human-oversight.md contains
localized Turkish terminal lines; change the terminal output to English-only
while keeping surrounding prose Turkish if desired. Specifically update the
quoted output strings—replace "İnsan onayı gerekli." with "Human approval
required.", "Onaylamak için:" with "To approve:", and "Reddetmek için:" with "To
reject:" (leave commands like "forgelm approve abc123 --output-dir ..."
unchanged) so the CLI/log snippet matches the repo rule that code/CLI
output/logs/config keys are English only.
---
Nitpick comments:
In `@docs/usermanuals/en/data/audit.md`:
- Line 93: Update the authoritative-source reference in
docs/usermanuals/en/data/audit.md by removing the hardcoded line number range
and referencing only the file and symbol; replace
"forgelm/cli/_parser.py::_add_audit_subcommand (lines 335-468)" with
"forgelm/cli/_parser.py::_add_audit_subcommand" so the doc points to the file
and the _add_audit_subcommand symbol only (no line numbers) to avoid drift when
the parser changes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: bb3efd8f-13e4-4b01-80d9-78f2200a52ff
📒 Files selected for processing (20)
docs/guides/quickstart-tr.mddocs/guides/quickstart.mddocs/guides/safety_compliance-tr.mddocs/guides/safety_compliance.mddocs/guides/troubleshooting-tr.mddocs/guides/troubleshooting.mddocs/reference/compliance_summary.mddocs/usermanuals/en/compliance/audit-log.mddocs/usermanuals/en/compliance/human-oversight.mddocs/usermanuals/en/data/audit.mddocs/usermanuals/en/data/secrets.mddocs/usermanuals/en/operations/cicd.mddocs/usermanuals/en/operations/experiment-tracking.mddocs/usermanuals/tr/compliance/audit-log.mddocs/usermanuals/tr/compliance/human-oversight.mddocs/usermanuals/tr/data/audit.mddocs/usermanuals/tr/data/secrets.mddocs/usermanuals/tr/operations/cicd.mddocs/usermanuals/tr/operations/experiment-tracking.mdtools/check_library_api_doc.py
✅ Files skipped from review due to trivial changes (2)
- docs/guides/safety_compliance.md
- docs/usermanuals/tr/data/secrets.md
🚧 Files skipped from review as they are similar to previous changes (6)
- docs/reference/compliance_summary.md
- docs/usermanuals/en/operations/cicd.md
- docs/guides/quickstart.md
- tools/check_library_api_doc.py
- docs/usermanuals/en/compliance/human-oversight.md
- docs/usermanuals/tr/operations/experiment-tracking.md
| | `2` | Training error | Check GPU, memory, dependencies | | ||
| | `3` | Evaluation failure | Model quality below threshold — adjust thresholds or improve data | | ||
| | `4` | Awaiting approval | Human review required — review results in `checkpoints/compliance/` and redeploy when ready | | ||
| | `4` | Awaiting approval | Human review required — run `forgelm approvals --show <run_id> --output-dir <dir>` to inspect the staging directory, then `forgelm approve <run_id> --output-dir <dir>` to promote or `forgelm reject <run_id> --output-dir <dir>` to discard. The staging path is `<output_dir>/final_model.staging.<run_id>/`. | |
There was a problem hiding this comment.
Add required --comment to approval commands in exit-code guidance.
Line 388 shows forgelm approve / forgelm reject without --comment; that can fail for users following this exact command path. Please include the required comment flag in both examples.
Suggested patch
-| `4` | Awaiting approval | Human review required — run `forgelm approvals --show <run_id> --output-dir <dir>` to inspect the staging directory, then `forgelm approve <run_id> --output-dir <dir>` to promote or `forgelm reject <run_id> --output-dir <dir>` to discard. The staging path is `<output_dir>/final_model.staging.<run_id>/`. |
+| `4` | Awaiting approval | Human review required — run `forgelm approvals --show <run_id> --output-dir <dir>` to inspect the staging directory, then `forgelm approve <run_id> --output-dir <dir> --comment "<note>"` to promote or `forgelm reject <run_id> --output-dir <dir> --comment "<note>"` to discard. The staging path is `<output_dir>/final_model.staging.<run_id>/`. |📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| | `4` | Awaiting approval | Human review required — run `forgelm approvals --show <run_id> --output-dir <dir>` to inspect the staging directory, then `forgelm approve <run_id> --output-dir <dir>` to promote or `forgelm reject <run_id> --output-dir <dir>` to discard. The staging path is `<output_dir>/final_model.staging.<run_id>/`. | | |
| | `4` | Awaiting approval | Human review required — run `forgelm approvals --show <run_id> --output-dir <dir>` to inspect the staging directory, then `forgelm approve <run_id> --output-dir <dir> --comment "<note>"` to promote or `forgelm reject <run_id> --output-dir <dir> --comment "<note>"` to discard. The staging path is `<output_dir>/final_model.staging.<run_id>/`. | |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/troubleshooting.md` at line 388, Update the documentation line
that shows the approval workflow to include the required --comment flag on both
commands: change the examples using the commands `forgelm approve <run_id>
--output-dir <dir>` and `forgelm reject <run_id> --output-dir <dir>` to `forgelm
approve <run_id> --output-dir <dir> --comment "<your comment>"` and `forgelm
reject <run_id> --output-dir <dir> --comment "<your comment>"` respectively so
users following the `Awaiting approval` guidance won't hit missing-comment
errors; keep the staging path (`<output_dir>/final_model.staging.<run_id>/`)
text unchanged.
Each finding verified against current code/state before action; EN+TR
mirrors kept in lockstep. One finding skipped with explicit reason.
Fixed:
- safety_compliance-tr.md L353 / safety_compliance.md L308:
`forgelm_version` JSON example was stale (`0.1.0`) — pyproject.toml
declares `0.5.5`. EN+TR updated to current release.
- safety_compliance-tr.md L3-5 / safety_compliance.md L3:
"tek açık-kaynak fine-tuning aracıdır" / "the only open-source
fine-tuning tool" was an unsubstantiated exclusivity claim. EN+TR
softened to "açık-kaynak fine-tuning araçlarından biridir" /
"one of the open-source fine-tuning tools that ships ... in the
same pipeline" — preserves the integrated-safety-+-Annex-IV
framing without the comparative claim.
- troubleshooting.md L191 / troubleshooting-tr.md L200: prose
referenced phantom `max_tokens` knob. Real schema is
`synthetic.max_new_tokens`
(`SyntheticConfig.max_new_tokens: int = 1024` in
`forgelm/config.py`). EN+TR aligned to fully-qualified
`synthetic.max_new_tokens`.
- usermanuals/en/data/audit.md L93: removed brittle "(lines 335-468)"
line-range from the parser source ref; symbol-only
(`forgelm/cli/_parser.py::_add_audit_subcommand`) is the
drift-resistant form per project convention. (TR mirror was
already symbol-only.)
- usermanuals/{en,tr}/operations/experiment-tracking.md L8:
"Comet ML is first-class" was wrong — `TrainingConfig.report_to`
Literal is `"tensorboard" | "wandb" | "mlflow" | "none"` (verified
in `forgelm/config.py:274`); no Comet ML branch ships. EN+TR
rewrote to enumerate the four Literal values explicitly with a
pointer to the schema. (Other "Comet UI" references at L110/L112/
L138 are about Comet's own UI as a third-party service, not
ForgeLM integration — left as-is.)
- usermanuals/{en,tr}/data/secrets.md L91: "Entropy threshold for
OpenAI / Anthropic keys" contradicted L26 (which states the
detector does NOT ship Anthropic / Stripe / SendGrid / Twilio
patterns). Verified `_SECRET_PATTERNS` in
`forgelm/data_audit/_secrets.py`: only `openai_api_key` regex
exists; no `anthropic_*` entry. EN+TR scoped the entropy guard
to OpenAI keys + cross-referenced the L26 statement.
- usermanuals/tr/compliance/human-oversight.md L51-56: terminal-output
block contained Turkish phrases ("İnsan onayı gerekli", "Onaylamak
için", "Reddetmek için") that conflict with the project rule
"code/CLI output/logs/config keys are English only" (per CLAUDE.md
+ docs/standards/localization.md). Translated the prompt strings
to English; surrounding TR prose retained.
Skipped (1):
- troubleshooting.md L388 ("add required --comment to approve/reject
examples"): verified against
`forgelm/cli/_parser.py:549-555` and `:981-985` — `--comment` is
declared with `default=None` and `args.comment or ""` is used at
`_approve.py:459` / `:533`. Comment is **optional**, not required;
users following the doc do NOT hit a missing-comment error. The
reviewer's premise was incorrect, so the example is left as the
minimal form. (The other human-oversight doc shows the
`--comment "..."` form for operators who DO want to record one.)
Validation: 8/8 doc CI guards green; ruff check clean; user manuals
regenerated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Review absorption — sweep #4 @ commit 755eaf78 findings (6 inline + 1 outside-diff + 1 nitpick) verified against current code; 7 fixed, 1 skipped with reason. Fixed (7)
Skipped (1, with reason)
Validation8/8 doc CI guards green; |
Not up to standards ⛔🔴 Issues
|
| Category | Results |
|---|---|
| Security | 1 critical |
| CodeStyle | 99 minor |
🟢 Metrics 160 complexity · 2 duplication
Metric Results Complexity 160 Duplication 2
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
|



Summary
Closes the 2026-05-07 docs audit (≈270 findings) across 7 commits:
canonical-name drift sweep, reference subcommand pages, P1 schema
rewrites, full safety_compliance-tr translation, 5 new defence-layer
CI guards, and post-audit residual cleanup. All 8 doc guards green.
Scope: 130 files / +4784 / -1510. Docs-only — no
forgelm/sourcechanges.
Wave breakdown (commit log)
2ba095fpipeline.reverted→model.reverted(12 docs),compliance.human_approval→evaluation.require_human_approval(10 docs),--add-row-idsreframe (5 docs), webhook event vocabulary sync, version stamps0.5.2→0.5.5, ghost field cleanupf7f5de0approve,approvals,purge,reverse_pii,safety_eval,verify_audit), exit-code tables (verify_audit 0/1 surface), 11 ghost audit flags removed fromcli.md9788f49response→completion, secrets11→9families,judge-tr.mdidentifier revert0d48266yaml-templates.mdrewrite (canonical schema + ghost-field deprecation table);compliance.config_hashreframe across 15 files7ff9b35safety_compliance-tr.mdfull translation — 38 sections, 170→547 lines (was a stub)0c4692ccheck_yaml_snippets.py(333 yaml blocks, 23 ForgeConfig validate),check_audit_event_catalog.py(36↔36),check_library_api_doc.py(30↔30 symbols),check_doc_numerical_claims.py(9/6/5/5 counts),check_bilingual_code_blocks.py(45 pairs)c9998ffcompliance_summary-tr,alignment-tr,cicd_pipeline-tr,quickstart-tr,troubleshooting-tr); architecture.md flat→package; library_api Example #3 unrunnable; configuration.md RoPE/webhook drift; design docs status banner + 8-step wizard rewriteAudit findings closure
evaluation.approval_retention_days(canonical alias isstaging_ttl_days)--add-row-idsimpl, HMAC body sign,encryption_at_restconfig field, additional secret patterns, KTO weight surfaces, GRPObetaflat field,compliance.config_hashpayload emitProvenance:
docs/analysis/code_reviews/2026-05-07_docs_audit.md(audit) +2026-05-07_docs_audit_plan.md(verification + execution plan). Both gitignored underdocs/analysis/per project convention.CI guard surface
3 pre-existing + 5 new = 8 doc guards, all green:
The new gates caught real regressions during the wave runs — Wave D's
lora:-block omission in 5 trainer YAML snippets, and Wave F's accidentalforgelm wizard(not a real subcommand — the canonical form isforgelm --wizardflag) in 3 docs. Both were fixed before commit; the gates would have blocked them at PR time.Test plan
python3 tools/check_bilingual_parity.py --strict— H2/H3/H4 spine across 45 EN+TR pairs.python3 tools/check_anchor_resolution.py --strict— 273 markdown files, all anchors resolve.python3 tools/check_cli_help_consistency.py --strict— 439forgelminvocations match parser surface.python3 tools/check_yaml_snippets.py --strict— 23 ForgeConfig-shaped snippets validate against live Pydantic schema.python3 tools/check_audit_event_catalog.py --strict— 36 emitted events match 36 catalog rows.python3 tools/check_library_api_doc.py --strict—forgelm.__all__↔library_api_reference.mdrows match.python3 tools/check_doc_numerical_claims.py --strict— secret families / trainer types / templates / webhook events claims match canonical scrapes.python3 tools/check_bilingual_code_blocks.py --strict— fenced-block counts + YAML-key sets match across pairs.pytest tests/— no regressions (docs-only PR, but verify CI green).forgelm --config config_template.yaml --dry-run— Pydantic validation passes.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation
Tools