docker-compose train volumes do not match the non-root image: HF cache mounted under /root and run artefacts/wizard output written to unmounted /workspace (CR-A16-005) #422

Wave 1 · theme D1-7: Dependency floors, container and supply chain · highest severity High · 2 finding(s) · effort small

Summary

An operator copies config_template.yaml to ./configs/job.yaml and runs the documented compose command on a Linux GPU host. Training runs for hours, writes everything to /workspace/checkpoints, and --rm deletes it. TensorBoard shows nothing, the next run re-downloads the model, and the EU AI Act audit log and compliance artefacts are gone.

Primary finding

CR-A16-005 — docker-compose train volumes do not match the non-root image: HF cache mounted under /root and run artefacts/wizard output written to unmounted /workspace

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · not reproduced (code trace) · already tracked: KNOWN-OPEN C31-DOCKER-RUNBOOK — docs/roadmap/phase-16-trust-surface-hardening.md:686-689,700-702; docs/roadmap/risks-and-decisions.md:201 (C-16)

What happens. docker-compose.yaml:39 mounts hf_cache at /root/.cache/huggingface and only ./output is writable at /workspace/output, but the image runs as USER forgelm with HOME=(local review artefact) and WORKDIR /workspace (Dockerfile:97-107) with no HF_HOME. The HF cache volume is never used (every --rm run re-downloads), and default output_dir ./checkpoints and the wizard's my_config.yaml land in the container's discarded /workspace.

Where.

• docker-compose.yaml:8-24 (usage block that documents run --rm train --config / --wizard / tensorboard)
• docker-compose.yaml:38-39 (cache mount target /root/.cache/huggingface vs non-root home)
• docker-compose.yaml:59-61
• Dockerfile:98-104 (useradd --home-dir (local review artefact) USER forgelm, no ENV HF_HOME)
• forgelm/config.py:451-453 (default output_dir ./checkpoints is by design; the gap is that no doc tells the operator to set /workspace/output)
• forgelm/wizard/_orchestrator.py:1085 (relative my_config.yaml save, by design; unmounted in the container)
• forgelm/compliance.py:186-215
• docker-compose.yaml:19-20
• forgelm/wizard/_orchestrator.py:1085-1086
• docker-compose.yaml:36

Evidence.

• Static cross-reading of compose, Dockerfile (USER forgelm, useradd --home-dir (local review artefact) no ENV HF_HOME), config default output_dir and wizard default filename. Docker is not available in this review environment, so the behaviour was not executed.
• Compose volume target /root/.cache/huggingface vs Dockerfile useradd --home-dir (local review artefact) and USER forgelm; wizard save_default 'my_config.yaml' relative to WORKDIR /workspace.
• Static: compose volume target /root/.cache/huggingface vs Dockerfile USER forgelm / --home-dir (local review artefact) grep for HF_HOME in Dockerfile and compose returns nothing. Not built, since docker is unavailable offline.

Recommended fix. Set HF_HOME=(local review artefact) (or mount the volume there). Set working_dir: /workspace/output, or document and validate that output_dir must be under /workspace/output. Pass environment: entries for HF_TOKEN, FORGELM_AUDIT_SECRET and FORGELM_OPERATOR. Pass a user/UID build arg, or document creating ./output with matching ownership.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

DOC-D3B02-035 — docs/usermanuals/en/operations/docker.md: broken example — Mount the HF cache at /root/.cache/huggingface "so model downloads persist".

severity Medium · already tracked: KNOWN-OPEN C31-DOCKER-RUNBOOK docs/roadmap/phase-16-trust-surface-hardening.md:686-689,700-702; docs/roadmap/risks-and-decisions.md:201

Where. docs/usermanuals/en/operations/docker.md:18,24,56,134 (Turkish mirror: docs/usermanuals/tr/operations/docker.md:18,24,56,134) · type: broken example

The document says. Mount the HF cache at /root/.cache/huggingface "so model downloads persist".

What the code does. The runtime image runs as USER forgelm (home (local review artefact)) with no HF_HOME set. HF uses (local review artefact) and the /root mount is never read. The shipped docker-compose.yaml has the same mismatch.

Code references. Dockerfile:97-104, docker-compose.yaml:39

User impact. Models are re-downloaded on every start, and an air-gapped container finds no cache.

Proposed correction. Mount at (local review artefact) (or pass -e HF_HOME=/hf -v $HOME/.cache/huggingface:/hf).

Turkish mirror. the Turkish page has the same error; fix both in the same PR

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}