Cooperative stop for external controllers: control file, signal handling and a terminal "interrupted" state #1029

Embeddable engine · EMB-02 · Kind: code + docs · Size M · background: #1027

Summary

A program that launched a training run has no clean way to stop it. On POSIX it can send a signal, but the run then dies without a terminal audit event, webhook or JSON envelope (#569). On Windows the only call a typical controller has, Popen.terminate(), is TerminateProcess(): a hard kill with no cleanup at all. This issue adds a cooperative stop that works the same way on every platform and ends the run in a defined, auditable interrupted state.

What exists today (2059b59)

• ForgeTrainer.train() catches only Exception (trainer.py#L1718). KeyboardInterrupt and a SystemExit raised by a signal handler skip pipeline.failed and notify_failure. This is finding CR-A02-046 in [Wave 3] Medium — Module sweep: Trainer core, OOM recovery, auto-revert & GRPO rewards (part 1/2) #569 (the Art. 12 log shows a run that started and never ended; the dashboard shows it in progress forever).
• The CLI maps Ctrl-C to exit code 2 (cli/_dispatch.py#L120-L121 and #L169-L170) so CI branches on a documented public code. There is no handler for SIGTERM, so a Kubernetes or Slurm preemption exits 143, which is outside the 0–6 contract ([Wave 3] Medium — Module sweep: Trainer core, OOM recovery, auto-revert & GRPO rewards (part 1/2) #569).
• Python documents that on Windows Popen.terminate() calls TerminateProcess() and kill() is an alias for it; SIGTERM is an alias for terminate(). A graceful signal needs CREATE_NEW_PROCESS_GROUP on the child plus CTRL_BREAK_EVENT, which only a controller that knows the trick can use (subprocess docs). A portable stop therefore cannot rely on signals alone.
• Transformers already has the cooperative primitive: a TrainerCallback may set control.should_training_stop (and should_save) in on_step_end. ForgeLM does not use it.
• After trainer.train() returns, ForgeLM carries straight on: evaluate, save the final model, run the gates (trainer.py#L1594-L1683). An early-stopped trainer must not walk into that path and promote a half-trained model.

Proposal (open decisions follow)

Control file. The controller creates <output_dir>/.forgelm-stop (optional JSON body: reason, requested_by). A TrainerCallback, in the module that EMB-01 creates, checks for it in on_step_end (at most once per second) and sets control.should_save = True and control.should_training_stop = True. ForgeLM deletes the file when it has honoured it, and deletes any stale file at the start of every run, so a leftover file cannot stop the next run. The file name is a contract and is documented.

Signals map to the same path. A handler installed only on the CLI training path (never at import time, main thread only) sets the same flag: SIGINT and SIGTERM on POSIX, SIGBREAK on Windows. The first signal asks for a graceful stop; a second signal falls through to the existing hard behaviour. This is the handler #569 asks for; one handler should serve both issues.

What a stop must guarantee

1. No promotion. After a stop there is no final_model/, no gate run, no pipeline.completed and no compliance export for a model that was not finished. Any half-trained state stays under the checkpoint directory.
2. A resumable checkpoint when the trainer can write one, so --resume continues from it.
3. Exactly one terminal audit event. Proposal: pipeline.interrupted with reason (cooperative_stop, sigint, sigterm), global_step and checkpoint. [Wave 3] Medium — Module sweep: Trainer core, OOM recovery, auto-revert & GRPO rewards (part 1/2) #569 suggests pipeline.failed / pipeline.interrupted; this issue and [Wave 3] Medium — Module sweep: Trainer core, OOM recovery, auto-revert & GRPO rewards (part 1/2) #569 must settle on one vocabulary. The event goes into docs/reference/audit_event_catalog.md (a CI guard checks it in both directions).
4. A defined envelope and exit code. Proposal: exit code 2, the code Ctrl-C already maps to, plus "interrupted": true and error_type: "interrupted" in the JSON envelope. No new exit code: the public contract is 0–6 and changing it is a MAJOR concern.
5. The same terminal state in the progress stream (run.finished with status: "interrupted", EMB-01).

Scope of a stop in v1. Honoured inside the training loop and at every phase boundary afterwards. A generation batch or benchmark task already running finishes first; the documentation says so. Hard kill remains the fallback and still leaves no terminal event; the guide (EMB-06) tells controllers to treat "no terminal event, process gone" as crashed.

Trust. The control file has the same trust as the output directory. It is not a remote-control channel; the guide says so. Symlinks are ignored.

Open decisions (record as a dated decision row)

1. File name and optional body.
2. The event name (pipeline.interrupted versus reusing pipeline.failed with a reason), shared with [Wave 3] Medium — Module sweep: Trainer core, OOM recovery, auto-revert & GRPO rewards (part 1/2) #569.
3. Whether the webhook vocabulary gets a new event or reuses training.failure with an "interrupted" reason. A ninth webhook event is additive but must update docs/reference/webhook_schema.md.
4. Whether forgelm ingest and forgelm audit get the same stop later (out of scope here; their atomic-write behaviour is Phase 16 S6, [Phase 16] S6 — Invocation-owned resources: temp state, atomic writes, worker lifetime #892).

Alternatives considered

Out of scope

Pause and resume of a live process (use --resume); stopping a remote or distributed job from another host; any UI.

Acceptance criteria

• The open decisions are answered in a dated decision row, together with [Wave 3] Medium — Module sweep: Trainer core, OOM recovery, auto-revert & GRPO rewards (part 1/2) #569's owner.
• The control-file callback, the shared signal handler and a stale-file cleanup exist, in a new module rather than in trainer.py (Split forgelm/trainer.py into a sub-package (_kwargs, _runtime, _finalize, _artifacts) — deferred until Phase 16 S4 and S8 have landed #904).
• A stop during training saves a checkpoint, writes pipeline.interrupted once, skips every post-train gate and every promotion step, and exits 2 with interrupted: true.
• A stop requested during evaluation, benchmark, safety or judge is honoured at the next boundary.
• --resume after a stop continues from the checkpoint and the audit chain verifies.
• SIGINT, SIGTERM and Windows SIGBREAK reach the same path; a second signal is a hard exit.
• The GRPO path honours the stop (the callback list reset must not drop it; see EMB-01).
• The audit event catalog, the JSON output reference and the exit-code documentation describe the new state in English and Turkish; a CHANGELOG Added entry exists; tools/check_cli_exit_code_prose.py --strict still passes.
• Tests are CPU-only and run on Linux, macOS and Windows.

Sequencing and dependencies

Stage B of #1027, after EMB-01 (it reuses the callback). It also depends on Phase 16 S8 (#894) for trainer.py. Resolve the shared vocabulary with #569 before either lands.

Related

#569 (CR-A02-046: interrupted runs leave the lifecycle unterminated), #894 (S8), #904, #962.

_{Raised on 2026-10-20 as part of the embeddable-engine decision. Code links point at 2059b59; line numbers may have moved. Tracking ID: EMB-02.}