emberd absolute resource guard: OS-enforced commit/VRAM ceilings on every spawned run — kill the per-callsite preflight class

[wordingone]

Operator directives (2026-07-16 evening, binding)

1. 'Impossible to set up an absolute memory/ram/vram guard inside emberd? You expect others in the future to manually do all this?'
2. 'There is ZERO justification behind keeping ANYTHING like the launcher outside of emberd.' — total absorption. No standalone dispatcher, no launch scripts, no thin wrappers. emberd is the ONLY way any run, tool, or cockpit process starts. The current standalone dispatcher/launch scripts exist only until this issue closes, then they are DELETED — not wrapped, deleted.

Scope: complete surface enumeration, walls at birth

A run touches a FINITE list of host surfaces. Done = every surface caged by emberd at spawn, proven by a probe, and no process starts without a declared budget per surface:

Surface	Wall (emberd-owned)	Proof probe
Host RAM/commit	Windows Job Object JOB_OBJECT_LIMIT_JOB_MEMORY from declared budget at spawn	over-allocating probe dies at ceiling; concurrent host process unaffected
VRAM	per-process fraction cap + NVML floor sentinel	probe exceeding fraction fails; floor breach fires ladder
Disk (B/C)	floor receipts absorbed into the same spawn preflight	write past budget refused
Process tree	job object kill-on-close; all children caged	orphan child dies with job
Windows/UI	runs spawn no visible windows except through the cockpit contract	UIA census before/after probe
CPU	governor pacing receipted in the same manifest	sustained 100% probe throttled

Structure

• emberd is the sole entry point. Every process class it manages — trainers, runners, verifiers, smoke probes, the cockpit — is born through emberd and therefore born caged. 'Started by emberd' and 'inside the walls' are the same fact by construction.
• emberd sentinel: polls host commit (GetPerformanceInfo) + VRAM (NVML) floors while anything it owns lives; governed ladder defer→pause→clean-abort with retained evidence, receipted to telemetry; detects and terminates any GPU/large-commit process NOT born through emberd.
• Declared-or-refused: spawn requires a declared budget per surface; undeclared = refused.
• Demotion: per-callsite preflights (Bound checkpoint serialization and retain failure evidence #895 class) become defense-in-depth only.

Acceptance

• Every probe in the table above, receipted.
• A probe process started OUTSIDE emberd is detected and terminated by the sentinel (receipted).
• The standalone dispatcher and every launch script are DELETED from the tree; repo guard blocks reintroducing any launcher-shaped script outside emberd.
• A fresh contributor starts a governed run with one emberd command and zero manual steps.

Provenance

Step-8192 checkpoint OOM cascade (#892) took down two seats and the run. This issue ends the launcher-outside-emberd era and the per-incident guard pattern with it.

[wordingone]

Operator escalation (binding, 2026-07-16 ~9:30pm): 'ANYTHING ember-related outside of emberd and ember-cli — NOT NOW NOT TONIGHT NOT TOMORROW.' This issue is no longer post-restart work. Effective immediately: (1) tonight's restart dispatch itself goes through emberd — the standalone dispatcher's verified logic (hash-bound not-before, preflight receipts) moves INSIDE emberd as a dispatch command before any relaunch; (2) every ember-related artifact currently outside emberd/ember-cli (launch scripts, standalone runner entry points, loose state trees) is inventoried and absorbed or deleted under this issue; (3) the restart clock moves to whatever the absorption honestly costs — correctness outranks tonight's number, per the operator.

[wordingone]

Native absorption review boundary is public in PR #901 at exact head 3d02d3c. This moves dispatch into persistent emberd: closed hash-bound launch manifest, absolute not-before/expiry, exact executable + typed config/manifest bytes, custody-only caches/outputs, disk and single-GPU VRAM floors, immutable preflight evidence, then existing exclusive lease + Windows Job Object governed spawn. The smoke and real restart must use the same emberd dispatch command; no external launcher is authorized. Evidence: 42 Rust tests, named-pipe CLI end-to-end proof, strict Clippy (one pre-existing untouched lint exempted), fmt, commit guard, and push guard all pass. This is dispatch/resource-integrity progress only: no training, checkpoint, capability, or sufficient-pretraining claim. #898 remains open for its remaining acceptance legs, including any stricter full memory-ceiling/sentinel requirement not already satisfied by the existing Job Object path.

[wordingone]

2026-07-17 execution update — issue remains OPEN. Four consecutive specialist launch attempts failed before any train step or checkpoint: r3 loader spawn WinError 193; r4 nested CUDA process killed by a guessed 23 GiB Job Object ceiling; r5 correctly refused because resume step 2 plus 8,192 records and checkpoint_interval 8,192 implied two ~11.36 GiB publications against a 12 GiB write budget; r6 passed final-only publication planning but PagedAdamW8bit reported CUDA OOM during the exact 1.725B-active specialist setup under a guessed 32 GiB host ceiling. Floors held and no partial checkpoint or capability evidence was produced. No r7 is authorized until dispatch evidence distinguishes device from pinned-host commit: every GPU dispatch receipt must bind a contemporaneous nvidia-smi device census, and a setup-only probe must instantiate the exact failing specialist route under host cap ((CommitLimit - CommitTotal) - 10 GiB frozen reserve) while sampling the child job peak every 1-2 seconds into the terminal receipt. Separately, emberd dry-run preflight must simulate steps/checkpoint schedule/publication bytes and formula-priced host memory from the manifest before spawn, refusing statically impossible launches. These are resource/training-integrity findings only; zero training, checkpoint, capability, or sufficient-pretraining credit.

[wordingone]

Follow-up discriminator evidence (r9; issue remains OPEN): exact specialist setup-only probe used source d7b3bbe, current config SHA-256 71f32a0dd20ea83d350795c5f8d850f7640bdb1728a281e9d5101517568debfb, and no records/forward/checkpoint. At dispatch CommitLimit=85,473,087,488 and CommitTotal=72,369,524,736, so the formula with frozen 8 GiB reserve yielded only a 4,513,628,160-byte Job Object cap. The child failed during MODEL_ALLOCATING while trying 24 MiB even though 20.23 GiB CUDA memory was free. This confirms host-cap/pinned-allocation refusal, not data/checkpoint failure and not evidence that the device cannot fit. No r7 is authorized. Next required evidence is a read-only top-20 process commit census plus kernel/driver totals; shed only identified stale/orphan processes (never terminal/session windows or unidentified processes); then rerun the same exact specialist setup formula with a 10 GiB frozen reserve. Target available-minus-reserve is at least 30 GiB so the true trainer peak can be measured and the next cap set to measured peak plus margin. Pagefile growth is only a priced fallback after census proves disciplined shedding insufficient; it is not yet authorized or executed.

[wordingone]

Supersession clarification (Kai program authority, 2026-07-18): the older acceptance sentence requiring Emberd to terminate a GPU/large-commit process not born through Emberd is withdrawn. Successor work may inventory and receipt foreign/system/WSL/Docker/GPU pressure, alarm, and refuse new dispatch. It must never terminate, suspend, mutate, name-match, or otherwise control a foreign/system process or a driver. Only a currently identity-verified Emberd-owned Windows Job Object for the matching live lease may be terminated, and only after its decision receipt is durable. This correction governs #898 successors and does not close the issue.\n\nThe successor must also absorb the cockpit commit-footprint requirements of #756 as a typed Emberd workload profile, not another watcher script. #756 still requires the bounded-retention repair, independent verification, durable ember.exe installation, and a fresh live soak proving a bounded plateau.

[wordingone]

Governor design adjudication after independent refutation (2026-07-18):\n\n- The monitor is daemon-lifetime, not per-run. It inventories and receipts foreign GPU/large-commit pressure between runs and keeps a sticky refuse-next-admission state; it never acts on a foreign process.\n- Reconciliation must cover prepared but never resumed roots: revalidate and resume, or write a decision receipt and release only that exact fresh Job Object.\n- Add ProtectiveOwnedStop: if a global host-survival floor is crossed, Ember requests a checkpoint from its own verified job, waits a bounded grace, then may receipt-first terminate only that exact owned job. The receipt says host protection, not that the owned job caused foreign pressure.\n- Tier sampling: cheap RAM/commit/disk/Driver-Locked-provider counters every 1-2 seconds; full GPU/process/Job census every 5-10 seconds and immediately near/beyond thresholds.\n\nDriver-Locked provider boundary: Microsoft documents RAMMap as exposing physical-memory use and Driver Locked, while the documented driver APIs describe locking/unlocking pages rather than a supported user-mode total counter. No supported total API/perf counter has been established, so the implementation must not invent or residual-calculate the value. Use a version-pinned provider interface that returns UNAVAILABLE and fails admission closed until a separately bounded native-Windows calibration passes against an independent Microsoft RAMMap snapshot and a small controlled pinned-host allocation. That calibration is no-training, no WSL/Docker, and creates no model/data/capability credit.\n\nPrimary references:\n- https://learn.microsoft.com/en-us/sysinternals/downloads/rammap\n- https://learn.microsoft.com/en-us/shows/defrag-tools/6-rammap\n- https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/nf-wdm-mmprobeandlockpages\n\nThis remains design/measurement work. It does not lift the training freeze or close #898.

[wordingone]

Governor successor correction after first-principles review and independent refutation (2026-07-18):

The prior requirement for a continuously sampled numerical Driver-Locked provider is superseded. No supported direct ordinary-runtime Windows total has been established. A placeholder interface plus calibration cannot manufacture such a source, and admission will not remain permanently impossible because an unavailable measurement API was assumed.

The accepted recurrence-prevention contract is instead:

1. Causal owned-pin control and attribution. Every Ember workload profile declares its pinned-host producers and budgets (including data-loader pinning/workers/prefetch and explicit CUDA host allocations). Emberd records exact configuration and allocator events. Undeclared or unsupported producer configurations refuse admission. These receipts support attribution; they are not falsely described as a system-wide page-lock wall.
2. Enforcement backstop. Every Ember-owned workload is born inside its exact Windows Job Object with declared commit/process-memory limits. Daemon-lifetime PhysicalAvailable, CommitRemaining, owned-job growth, disk, GPU, and child-liveness guards protect the host regardless of whether pressure is pinned, pageable, driver-side, or unattributed. At a host-survival breach, ProtectiveOwnedStop requests an owned checkpoint, waits the bounded grace, then may receipt-first terminate only that exact identity-verified owned job. Foreign/system/driver processes are never controlled.
3. Driver-Locked diagnostic oracle. Version-pinned RAMMap remains independent before/after and incident evidence, not a fabricated 1-2 second production provider. Approaching a survival floor or an unattributed-pressure alarm durably freezes subsequent admission and creates a bounded oracle-evidence obligation. Safety action never waits on the GUI/operator snapshot. Failure to obtain the oracle evidence remains explicit and keeps admission frozen; it is not silently passed.
4. Optional future provider. The provider interface remains UNAVAILABLE until an actual direct source with named semantics exists. If one is found, calibrate it against RAMMap with a CUDA page-locked fixture only, after warming the CUDA primary context before baseline. The fixture Job limit is based on measured baseline plus bounded headroom, never an absolute 96 MiB assumption. VirtualLock is not a substitute.
5. Tiered monitoring. Cheap supported survival/owned-job counters run every 1-2 seconds; full process/GPU/Job census runs every 5-10 seconds and immediately near thresholds. RAMMap never runs inside a declared measurement window.

This preserves the operator invariant—no recurrence with host harm—without claiming a nonexistent measurement capability. It does not lift the current training freeze by itself, create training/model/capability credit, or close #898. The next implementation remains three reviewed emberd increments: provider/status + closed workload profile, daemon-lifetime monitor/Job enforcement + reconciliation/ProtectiveOwnedStop, then RPC/cockpit/receipts and hardware probes.