gpu-cost.md (EN+TR) describes a cost subsystem that does not exist and never mentions the one that does (training.gpu_cost_per_hour) (CR-A01-021) #428

Wave 1 · theme D1-8: Docs describing features that do not exist (doc side of code findings) · highest severity High · 11 finding(s) · effort small

Summary

An operator follows the page and either sets no rate (believing no YAML surface exists) or writes output.cost_tracking and gets exit 1. They then look for a run_complete cost event or a model-card cost section that never appears.

Primary finding

CR-A01-021 — gpu-cost.md (EN+TR) describes a cost subsystem that does not exist and never mentions the one that does (training.gpu_cost_per_hour)

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. The real knob is the scalar training.gpu_cost_per_hour (config.py:609-613, consumed at trainer.py:1858-1861). The page never mentions it: its status banner (:8-14) tells operators to 'set hourly rates manually until the YAML surface lands', and :155 sends them to the Configuration Reference for a 'full cost_tracking block' that is not there (configuration.md:129 only mentions gpu_cost_per_hour in passing). Behaviours the page states as current, with code that contradicts them: - :20-22 detection via nvidia-smi. The code uses torch.cuda.get_device_name (trainer.py:1841). - :16 and :24-32 GPU 'profile' with memory, compute and rate. Only a flat price table exists (trainer.py:1813-1833), and RTX 6000 Ada, RTX A6000, RTX 4080, RTX 3080 and Apple M1/M2/M3 are not in it. - :55-68 a 'run_complete' audit event with gpu_profile, rate_per_hour and currency. No such event exists; cost appears only as resource/* inside pipeline.completed.metrics_summary. - :70-77 'The model card cites it'. model_card.py has no cost content. - :144 'Use --gpus N'. No such flag exists. - :87-88 arithmetic: 0.034 * (3 / 0.034) * 1.10 equals 3.30 (and Python prints 3.3000000000000003), not '$7.15'. The comment says 6 h, but the formula encodes 3 h. - :34 'unrecognised GPU falls back to generic profile (no cost estimation)'. False when no CUDA is present; see the separate finding. Answer to question (3): :97 output.cost_tracking.rate_per_hour is the same fabrication as :38 and is covered by the banner's placeholder disclaimer. :111 is additionally labelled a roadmap item.

Where.

• docs/usermanuals/en/operations/gpu-cost.md:16-34
• docs/usermanuals/en/operations/gpu-cost.md:20-34
• docs/usermanuals/en/operations/gpu-cost.md:55-77
• docs/usermanuals/en/operations/gpu-cost.md:83-88
• docs/usermanuals/en/operations/gpu-cost.md:155
• docs/usermanuals/tr/operations/gpu-cost.md:8-16
• docs/usermanuals/tr/operations/gpu-cost.md:155
• forgelm/config.py:609-613
• forgelm/trainer.py:1813-1833
• docs/usermanuals/en/operations/gpu-cost.md:125-135 (gpu_profiles.local.yaml auto-merge, also nonexistent)

Evidence.

• rg over forgelm/ finds no 'run_complete', '--gpus', 'gpu_profiles' or 'cost_tracking'; model_card.py has no cost reference; the dry-run with training.gpu_cost_per_hour: 1.10 exits 0 (dry/control.log).

Recommended fix. Document training.gpu_cost_per_hour as the shipped knob, delete or roadmap-label the profile, event, model-card and --gpus claims, fix the arithmetic, and correct the :155 link text.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

• CR-A02-025 (code, High) — gpu-cost.md describes detection methods, profiles, events and flags that do not exist
  severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An operator copies the pre-flight calibration command from gpu-cost.md:84 and gets an argparse error with exit 2. Or they rely on the documented 'unrecognised GPU' warning, or the run_complete audit event, to watch cost, and neither ever appears.
  Where. docs/usermanuals/en/operations/gpu-cost.md:34 (unrecognised-GPU warning), docs/usermanuals/en/operations/gpu-cost.md:16-34, docs/usermanuals/en/operations/gpu-cost.md:84 (--output-dir on top-level), docs/usermanuals/en/operations/gpu-cost.md:84-85
  Recommended fix. Rewrite gpu-cost.md against trainer.py:1813-1910: torch-based detection, the 16-entry price table, gpu_cost_per_hour as the override, and resource_usage in compliance_report.json and pipeline.completed. Move the profile file, --gpus and the model-card section under the roadmap banner or delete them. Fix the calibration command (output goes through training.output_dir).

• CR-A02-031 (code, Medium) — GPU cost auto-detection fuzzy-matches an empty GPU name on CPU/MPS hosts and records an invented price and GPU hours
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A CPU/MPS-only run reports gpu_hours 0.052 and an estimated cost of $0.0078 at 0.15 USD/h in compliance artefacts presented as evidence.
  Where. forgelm/trainer.py:1853-1878 (_resolve_cost_per_hour: gpu_name = usage.get('gpu_model', '') at 1863; the bidirectional substring test known_lower in gpu_lower or gpu_l…, [forgelm/trainer.py:1889-1896](https://github.com/HodeTech/ForgeLM/blob/cab2947/forgelm/trainer.py#L1889-L1896) (_collect_resource_usage computes gpu_hours as runtime/3600 * usage.get('gpu_count', 1) and records the cost with no check that a GPU exist…, [forgelm/trainer.py:1886-1896](https://github.com/HodeTech/ForgeLM/blob/cab2947/forgelm/trainer.py#L1886-L1896), [forgelm/trainer.py:1853-1878`](https://github.com/HodeTech/ForgeLM/blob/cab2947/forgelm/trainer.py#L1853-L1878)
  Recommended fix. Skip the cost lookup when gpu_model is empty and omit gpu_hours without a GPU.

• TECH-T13-002 (technology, Next release) — Python, platform, CI, container and GPU prices: GPU price table ForgeTrainer._GPU_PRICING ('approximate mid-2026 cloud averages'), its name matcher, and the GPU-cost user-manual page (incorrect-claim)
  severity Next release
  Item. GPU price table ForgeTrainer._GPU_PRICING ('approximate mid-2026 cloud averages'), its name matcher, and the GPU-cost user-manual page · verdict: incorrect-claim · urgency: Next release
  State in ForgeLM. forgelm/trainer.py:1812-1833: T4 0.35, P100 0.45, V100 1.00, K80 0.20, A10G 0.75, A100-SXM4-40GB 1.50, A100-SXM4-80GB 2.00, A100 80GB PCIe 2.00, 'H100 80GB HBM3' 3.50, 'H100 SXM5 80GB' 3.95, H200 4.50, L4 0.50, L40S 1.20, B200 5.00, RTX 3090 0.15, RTX 4090 0.20 (RTX rows commented 'self-hosted, estimated electricity + amortization'). Matching (trainer.py:1863-1878): exact name, then bidirectional…
  State of the ecosystem on 2026-10-01. USD per GPU-hour, on-demand. AWS us-east-1 (price list published 2026-09-25): T4 0.526, A10G 1.006, L4 0.805, L40S 1.861, V100 3.06, A100-40 2.745, A100-80 3.431, H100 6.88, H200 7.912, B200 14.242, B300 17.802, RTX PRO 6000 3.363. RunPod (updated 2026-09-27): B300 7.89, B200 6.79, H200 4.59, H100 SXM 3.49, H100 PCIe 2.89, H100 NVL 3.19, A100 80GB 1.59, L40S 1.09, L4 0.49, RTX Pro 6000 2.09, RTX…
  Recommendation. 1) Fix _resolve_cost_per_hour: return None for an empty gpu_model, skip cost on non-CUDA runs, and restrict fuzzy matching to known-key-in-device-name. 2) Relabel the table as an illustrative neocloud on-demand snapshot with a dated constant (or ship no defaults and require training.gpu_cost_per_hour); raise T4/A10G/A100-40/B200 to roughly 0.55/1.00-1.20/2.00/6.25-6.50; add B300 ~8, H100 PCIe/NVL…
  Sources.

  • AWS EC2 on-demand Linux price map, us-east-1 (gzip JSON) (2026-09-25 (publication date in manifest))
  • Runpod GPU pricing (Updated September 27, 2026) (2026-09-27)

• DOC-D3B06-004 (docs, High) — docs/usermanuals/en/operations/gpu-cost.md: false claim — "After every run, the audit log records" an event "event": "run_complete" with ts, gpu_profile, gpus_used, rate_per_hour, currency… severity **High** **Where.** [docs/usermanuals/en/operations/gpu-cost.md:16](https://github.com/HodeTech/ForgeLM/blob/cab2947/docs/usermanuals/en/operations/gpu-cost.md#L16),55-77,148 (Turkish mirror: [docs/usermanuals/tr/operations/gpu-cost.md:59](https://github.com/HodeTech/ForgeLM/blob/cab2947/docs/usermanuals/tr/operations/gpu-cost.md#L59),73) · type: false claim **The document says.** "After every run, the audit log records" an event "event": "run_complete"withts, gpu_profile, gpus_used, rate_per_hour, currency, and "The model card cites it" under a ## Training costsection. **What the code does.** Norun_completeevent exists. Resource figures reach the audit log only as numericresource/*keys insidepipeline.completed's metrics_summary(forgelm/trainer.py:1413-1421, 1657), and audit entries usetimestamp, not ts(compliance.py:556). The real fields aregpu_model, peak_vram_gb, gpu_count, training_duration_seconds, gpu_hours, gpu_cost_per_hour_usd, estimated_cost_usd, cost_so…
  Code references. forgelm/trainer.py:1413-1421, forgelm/trainer.py:1657, forgelm/trainer.py:1835-1896, forgelm/compliance.py:556
  Proposed correction. Document the real resource_usage block (fields above), its locations (compliance_report.json, JSON envelope, pipeline.completed metrics_summary as resource/*), and remove the model-card example.
  Turkish mirror. the Turkish page has no such statement

• DOC-D3B06-006 (docs, High) — docs/usermanuals/en/operations/gpu-cost.md: false claim — Custom GPU profiles in gpu_profiles.local.yaml in the project root are auto-merged.
  severity High
  Where. docs/usermanuals/en/operations/gpu-cost.md:121-135 (Turkish mirror: docs/usermanuals/tr/operations/gpu-cost.md:126-135) · type: false claim
  The document says. Custom GPU profiles in gpu_profiles.local.yaml in the project root are auto-merged.
  What the code does. No code reads gpu_profiles or gpu_profiles.local.yaml (no occurrence in forgelm/). The pricing table is a hard-coded class attribute (trainer.py:1813-1832).
  Code references. forgelm/trainer.py:1813-1832
  Proposed correction. Remove the section, or mark it as not implemented and point to training.gpu_cost_per_hour.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-BFIELD-010 (docs, Medium) — forgelm/config.py: misleading — gpu_cost_per_hour: None = auto-detect from known GPUs (used by the cost-estimation report).
  severity Medium
  Where. forgelm/config.py:609-613 · type: misleading
  The document says. gpu_cost_per_hour: None = auto-detect from known GPUs (used by the cost-estimation report).
  What the code does. On a host without CUDA, _collect_gpu_info (trainer.py:1835-1843) records no gpu_model. _resolve_cost_per_hour (trainer.py:1863-1876) then fuzzy-matches the empty string, which is a substring of every known name, so it returns a price. Reproduced: a 1-hour run on a non-CUDA host yields {'cost_source':'fuzzy_match','gpu_cost_per_hour_usd':0.15,'estimated_cost_usd':0.15}.
  Code references. forgelm/trainer.py:1835-1843, forgelm/trainer.py:1853-1877, forgelm/trainer.py:1413-1421

• DOC-BYAML-007 (docs, Medium) — docs/usermanuals/en/operations/gpu-cost.md: false denial — Hourly rate cannot be configured until a cost_tracking block lands
  severity Medium
  Where. docs/usermanuals/en/operations/gpu-cost.md:36-45 and :20 (banner says set hourly rates 'manually') · type: false denial
  The document says. Hourly rate cannot be configured until a cost_tracking block lands
  What the code does. training.gpu_cost_per_hour exists and feeds the cost estimate (resolved in trainer._resolve_cost_per_hour; documented in docs/reference/configuration.md:139). The page never mentions it.
  Code references. forgelm/config.py:609, forgelm/trainer.py:1858-1894
  Proposed correction. Document training.gpu_cost_per_hour as the working surface.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-BYAML-009 (docs, Medium) — docs/usermanuals/en/operations/gpu-cost.md: false claim — gpu_profiles.local.yaml is auto-merged; '--gpus N' declares allocation; audit log records run_complete with gpu_profile/gpus_used/rate_per_ho…
  severity Medium
  Where. docs/usermanuals/en/operations/gpu-cost.md:125-135, :144 (TR :144) · type: false claim
  The document says. gpu_profiles.local.yaml is auto-merged; '--gpus N' declares allocation; audit log records run_complete with gpu_profile/gpus_used/rate_per_hour
  What the code does. No such file lookup, flag or event exists in forgelm/ (greps for gpu_profiles.local, --gpus, run_complete returned no hits). The built-in table is a 16-entry price dict keyed by exact device names; the table of 'Supported GPUs' (RTX 6000 Ada, A6000, 4080, 3080, M-series) names devices absent from it.
  Code references. forgelm/trainer.py:1812-1832
  Proposed correction. Mark these blocks as planned or remove; list the real 16 keys.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D3B06-016 (docs, Medium) — docs/usermanuals/en/operations/gpu-cost.md: false claim — "On startup, forgelm reads nvidia-smi --query-gpu=…". Supported GPUs include RTX 6000 Ada, RTX A6000, RTX 4080, RTX 3080 and Apple M1/M2/…
  severity Medium
  Where. docs/usermanuals/en/operations/gpu-cost.md:20-34 (Turkish mirror: docs/usermanuals/tr/operations/gpu-cost.md:21,29) · type: false claim
  The document says. "On startup, forgelm reads nvidia-smi --query-gpu=…". Supported GPUs include RTX 6000 Ada, RTX A6000, RTX 4080, RTX 3080 and Apple M1/M2/M3 Max. An unrecognised GPU "logs a warning and falls back to a generic profile".
  What the code does. The GPU name comes from torch.cuda.get_device_name(0) after training (forgelm/trainer.py:1835-1843, 1880-1891), not nvidia-smi at startup. _GPU_PRICING holds 16 entries: T4, P100, V100, K80, A10G, A100-SXM4-40GB, A100-SXM4-80GB, A100 80GB PCIe, H100 HBM3, H100 SXM5, H200, L4, L40S, B200, RTX 3090, RTX 4090 (trainer.py:1813-1832). RTX 6000 Ada, A6000, 4080, 3080 and Apple are absent, and nothi…
  Code references. forgelm/trainer.py:1813-1832, forgelm/trainer.py:1835-1878
  Proposed correction. Replace the table with the 16 _GPU_PRICING entries. State that the GPU is read via torch.cuda.get_device_name(0) after training, that unmatched GPUs produce no cost unless training.gpu_cost_per_hour is set, and that no warning is logged.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D1B01-037 (docs, Low) — docs/reference/configuration.md: misleading — training.gpu_cost_per_hour: "Auto-detected from GPU model if null"
  severity Low
  Where. docs/reference/configuration.md:139 (Turkish mirror: docs/reference/configuration-tr.md:138) · type: misleading
  The document says. training.gpu_cost_per_hour: "Auto-detected from GPU model if null"
  What the code does. On a host without CUDA, gpu_model is '' and the fuzzy match ('' in any string) is always true. The longest table key, 'NVIDIA GeForce RTX 3090' at $0.15/h, is then reported with cost_source 'fuzzy_match'. Seen in the tiny-qwen run on a CUDA-less host: resource_usage.cost_source fuzzy_match, gpu_cost_per_hour_usd 0.15.
  Code references. forgelm/trainer.py:1853-1878
  Proposed correction. Fix the code (skip estimation when gpu_model is empty); the doc can add "No estimate is produced on hosts without a CUDA GPU" once fixed.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}