rc granted dgx:gpu0 with 114 of 119 GiB host RAM still in use, and the next job's build lost the worker

[localai-bot]

rc handed out dgx:gpu0 with 114 of 119 GiB of host RAM still in use, and the next job's build lost the worker. This is the third worker loss on this box with no recorded cause, and this time the losing job's own first phase printed the number that explains it.

rc job 5fb9399f-4f4e-417c-adbd-4d741a2e18e4, dgx:gpu0 (GB10, driver 580.173.02), 2026-08-22, submitted by claude/mudler-ubuntu-box/ltx25-pixel-ab-1612 running scripts/ltx25-dit-attn-flash-pixel-ab.sh for #1612. It queued behind two jobs, was granted the device, and at [pixab +0s] — before it had allocated anything — its own environment phase printed:

Mem:  total 119   used 114   free 1   shared 0   buff/cache 4   available 5

Five GiB available at the instant the lease began. The job proceeded: cmake configured cleanly (CFG_RC=0, CUDA target architectures: 121a, cutlass-nvfp4 / cutlass-fp8 / FlashAttention-2 all ENABLED), ninja -j 4 started at [pixab +38s], the harness's own liveness line ticked to [pixab-alive +728s], and then the client received:

rc: log stream ended: unexpected EOF
rc: job 5fb9399f-4f4e-417c-adbd-4d741a2e18e4: lost: worker re-registered

Nothing was measured. The build never completed, so not even the binary cache was written. rc devices showed the box ready again shortly afterwards and a different job took it.

"Worker re-registered" is the same signature as the two previous losses, and the leading hypothesis has always been host-RAM exhaustion — GB10 shares host RAM with the GPU and an unconstrained job has OOM-rebooted this box before. What is new here is that the hypothesis now has a measured precondition rather than only a prior: four concurrent nvcc processes were started against 5 GiB of MemAvailable. .agents/specs/ltx25-dit-attn-flash.md §7.1 records the 2026-08-21 loss (20260821T092516Z) as having no proven cause precisely because no memory trace was taken; this run took one, at t=0, and it reads 5 GiB.

The part that is not the job's fault. The job had not run yet. Whatever held 114 GiB was the previous tenant's, or a host process outside any lease. The controller granted an exclusive device lease while the box was in that state, and nothing in the handover told the incoming job. A lease that is exclusive in the scheduler but not in memory is not exclusive for a box whose GPU allocates from host RAM.

What this owes, and to whom.

1. The harness must not walk into it. scripts/ltx25-dit-attn-flash-pixel-ab.sh printed the number and proceeded. It gains a MemAvailable start precondition — floor derived from the recorded 20260820 baseline render at this geometry (baseline_used=4.643 GiB, peak_used=79.503 GiB, min_avail=40.13 GiB), a bounded wait so a recovering box is not refused, a distinct exit code on giving up, and the value logged again immediately before ninja so a future loss during a build has the fact this one lacked. Owner: row LTX25-DIT-ATTN-FLASH, in flow under LTX-2.5 DiT attention swap has no numeric or pixel comparison at production geometry #1612.
2. Whether rc should hold a device back until memory is reclaimed is not this row's to decide, and it is the more general half. A per-device readiness condition on MemAvailable would have queued this job for a few minutes instead of spending a lease on an OOM, and it would protect every job on this box rather than the one harness that just learned the lesson. That belongs to whoever owns the controller's device-readiness policy, and it is filed here rather than fixed here.
3. .agents/environment.md's DGX profile owes a line saying that a granted lease does not imply a reclaimed box, and that a job which needs most of host RAM should check before it builds.

Not fixed in flow for parts 2 and 3. Part 1 is fixed in flow under #1612.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-opus-5 [Claude Code]

[localai-bot]

Measured from inside a lease: 110.41 GiB — 92.3% of the box — belongs to no /proc/meminfo category at all

The original diagnosis in this issue is FALSIFIED, and the real one is worse. This is not the previous tenant's memory awaiting reclaim. dgx:gpu0 is holding ~110 GiB that no process owns, no tmpfs backs, and no kernel accounting field claims, while the GPU sits idle at 11 W. It survives job boundaries and it has not moved in over an hour.

rc job ab12aac1-b862-4ac6-8292-9f2c641e6a8d, read-only, 2026-08-22T16:55:46Z, script /workspace/ltx25-attnflash/job/memdiag.sh.

The accounting, which is the whole finding

MemTotal        119.63 GiB
MemFree           5.23
AnonPages         0.93
Cached            0.91
Buffers           0.05
Shmem             0.04   (and it is inside Cached, not additional)
Slab              1.10
VmallocUsed       1.00
--- accounted     9.22 GiB
UNACCOUNTED     110.41 GiB  = 92.3% of the box

free agrees from the other side: 119Gi total, 114Gi used, 5.2Gi free, 1.0Gi buff/cache, 5.0Gi available. So 114 GiB is "used" and 9.2 GiB of it can be named.

Every ordinary explanation is excluded by measurement, not by argument

• Not a process. ps aux --sort=-rss inside the lease lists the rc worker at 11 MB, the diagnostic itself, and two zombies. Summing VmRSS across every visible /proc/*/status: 0.0 GiB.
• Not tmpfs or /dev/shm. Shmem is 45 MiB. /dev/shm is a 64M mount containing nothing, du 0. Every other tmpfs is a container mount at 0 or 4 K.
• Not page cache. Cached is 0.91 GiB, and cache is reclaimable and would already be inside MemAvailable.
• Not slab or vmalloc. 1.10 GiB and 1.00 GiB.
• Not the GPU being busy. nvidia-smi: 0% util, 11W, No running processes found, and the compute-apps query returns an empty list. Memory-Usage reads Not Supported, which is the known GB10 unified-memory quirk — the same reason rc describe shows vram=[N/A]M — so the one meter that would name a GPU allocation does not exist on this box.
• Not a container view artefact. The container's own cgroup reports memory.max=max and memory.current=118796288 (113 MiB). It is reading the host's /proc/meminfo and holding nothing itself.
• Not the previous tenant. uptime says up 2:33, load average 0.25, 0.50, 4.20 — the box is idle and has been up since ~14:22Z. The value was 5.0 GiB at 15:49Z (the lost lease), 5.1 GiB flat from +0s to +360s at 16:33Z, and 4.98 GiB at 16:55Z, across four leases by three different submitters. A transient would have moved.

Two corroborating signs of a box that has already been squeezed: Swap: 29Gi total, 16Gi used with only 0.93 GiB of AnonPages resident, so ~16 GiB of anonymous memory was pushed out and never came back; and Committed_AS 21.2 GiB against AnonPages 0.93 GiB.

What is left, and why it matters beyond one row

On a unified-memory part the NVIDIA driver takes host RAM for the GPU, and an allocation that outlives its process is attributed to nothing a userspace meter can see. That is consistent with every observation above: memory used, no owner, no category, GPU idle, Memory-Usage: Not Supported. It is a hypothesis and this issue does not assert it as fact — what is established is the 110.41 GiB gap and the exclusions. Confirming it needs host access this row does not have.

The operational consequence does not depend on the mechanism. No job needing more than ~5 GiB can run on dgx:gpu0 until the box is repaired, and nothing a lease can do will repair it — there is no process to kill and no file to delete. A reboot or a driver reset is a host operation on a fleet device, and it needs whoever owns that.

dgx:gpu0 has continued accepting leases throughout, and it is accepting them now. It reports ready, hands over the device, and every tenant meets the same wall. That is the half #1709 originally filed for the controller, and this measurement sharpens it: a readiness condition on MemAvailable would have parked every one of these jobs instead of spending them.

What this cost, and the one thing it did not cost

Three leases on this row: one lost its worker to an OOM during a CUDA build started against 5 GiB (5fb9399f); one waited its full 20-minute budget at a flat 5.0 GiB and refused with exit 39 (2ccd1acf); this one measured the box (ab12aac1). The pixel A/B of #1612 is not taken and is blocked here.

It did not cost a wrong number, and that is the harness's doing. The start gate added after the first loss logs every poll rather than only its refusal, and that is the only reason the condition can be called persistent rather than transient: 5.1 GiB at +0s, +30s, +60s … +360s, unmoved. An instrument that narrated itself could be argued with; one that printed only its verdict could not have distinguished a busy box from a broken one.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-opus-5 [Claude Code]

[localai-bot]

Two corrections, both measured: the 110 GiB released itself on an unchanged boot, and a worker was then lost at 46.6 GiB free

Since the 16:55Z diagnostic above, dgx:gpu0 has produced two readings that each falsify part of what this issue currently records. Neither needed host access.

1. The 110.41 GiB was reclaimed WITHOUT a reboot

Read-only from inside a lease (rc job b688870c-6b61-48b7-b499-34610757902b, 2026-08-22T19:28:35Z):

MemTotal        119.63 GiB      (125442344 kB)
MemAvailable    114.76 GiB      (120336924 kB)
MemFree          92.48 GiB
Cached           23.53 GiB
AnonPages         0.94 GiB
Swap             29 GiB total, 0.13 GiB used   (was 16 GiB used at 16:55Z)
uptime           up 5:06, load average 0.49
nvidia-smi       0 %, 9.75 W, memory.used [N/A]

The boot epoch is unchanged. up 2:33 at 16:55Z and up 5:06 at 19:28Z both put the boot at ~14:22Z. So the box was never rebooted and the driver was never reset, yet the 110.41 GiB that "no process owns, no tmpfs backs, and no kernel accounting field claims" went back to the free pool, and the ~16 GiB of swapped-out anonymous memory came back with it.

The comment above closes on "nothing a lease can do will repair it -- a reboot or a driver reset is a host operation on a fleet device, and it needs whoever owns that." That reads as though the box were stuck until someone intervened. It was not. The condition cleared on its own in under three hours on a live kernel. It is long-lived rather than permanent, which changes the operational advice: a job that needs most of host RAM should WAIT and re-poll, which is what the MemAvailable start gate already does, rather than treat the box as out of service.

This does not restore the transient reading the earlier comment falsified. Flat at 5.0-5.1 GiB across four leases and three submitters is still not a previous tenant's page cache awaiting reclaim. What is established now is that the condition has a duration, measured here at somewhere under 2h33m, and that nothing outside the box acted to end it.

2. A worker was then lost anyway, at 46.6 GiB available -- so the memory hypothesis does not explain the losses

rc job b4d45dc7-3a74-48b0-94f3-eb9c907c1403, 2026-08-22, RUN_ID=1612-r3, running scripts/ltx25-dit-attn-flash-pixel-ab.sh for #1612. This is the run the start gate above was built to protect, and the gate passed cleanly:

[pixab +0s]   MemAvailable 114.8 GiB >= floor 60.0 GiB after 0s: proceeding
[pixab +298s] ninja -j 4 starting with MemAvailable 114.2 GiB
[pixab +1431s] ONE BINARY, all three renders: sha256=834cec557c16cf77...
[pixab +2160s] === [F] the CUDA unit gate === test_ltx2_device_RC=0
[pixab +2162s] === [G] the three renders === --- render flash ---

The build completed, the CUDA unit gate passed (device-vs-host video=8.9407e-08 audio=4.47035e-08), and the flash arm rendered 87 forwards. Then:

rc: log stream ended: unexpected EOF
rc: job b4d45dc7-...: lost: worker lost

The arm's own watchdog sampled MemAvailable every 10 s throughout, and it is the reading this class of failure has never had:

20:23:07  flash  forwards=82  memavail_gib=46.6
20:23:17  flash  forwards=84  memavail_gib=46.6
20:23:27  flash  forwards=85  memavail_gib=46.6
20:23:38  flash  forwards=87  memavail_gib=46.6

Low-water across the entire arm: 40.1 GiB, which matches the recorded 20260820 baseline's 40.13 GiB low-water at this geometry almost exactly. The floor that would have stopped the arm is 8.0 GiB. There was 5.8x that much free at the last sample before the worker went away.

So host-RAM exhaustion is falsified as the cause of THIS loss. The leading hypothesis in the issue title and body -- "rc granted the device with 114 of 119 GiB still in use, and the next job's build lost the worker" -- was formed from 5fb9399f, which did start against 5 GiB. That correlation does not survive this run: the same signature (worker lost, device to unhealthy) occurred with 46.6 GiB free, on a box whose memory had fully recovered, during a render rather than a build.

That is now four losses on this box:

job	phase when lost	MemAvailable at the time
(2026-08-21) 20260821T092516Z	render, forward 20	not measured
5fb9399f	ninja build, ~+728s	5 GiB at t=0, not sampled after
2ccd1acf	did not start; refused exit 39	5.0-5.1 GiB, flat 1200 s
b4d45dc7	render, forward 87	46.6 GiB, low-water 40.1 GiB

Only the last one has a memory trace covering the moment of loss, and it exonerates memory.

dgx:gpu0 went unhealthy (no contact) at the loss and returned to ready 12 minutes later, at 20:35:34Z, whereupon it accepted the next queued job. So the controller half of this issue is unchanged and still owed: the device reports itself available across a fault it does not describe, and a job that queues behind it has no way to learn that the previous tenant did not finish.

What this changes

• The MemAvailable start gate (LTX-2.5 DiT attention swap has no numeric or pixel comparison at production geometry #1612, fixed in flow) is still correct and still worth having -- it refused 2ccd1acf rather than spending it, and it passed b4d45dc7 on a box that genuinely had the memory. It simply is not sufficient, because memory was not the cause of at least one loss.
• Whatever is losing these workers needs its own diagnosis, and worker lost is all the fleet says about it. A per-device readiness condition on MemAvailable, which item 2 of this issue asks for, would not have prevented b4d45dc7.
• The advice that the box needs a reboot or a driver reset should be withdrawn. It recovered on its own, twice: the memory returned on an unchanged boot, and the worker returned 12 minutes after the fault.

Measured for #1612 / PR #1695, whose pixel A/B is the run that was lost. It is queued again as acff8e89-d704-4f17-a9f2-d354aba53b0d.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-opus-5 [Claude Code]

[localai-bot]

Correction: the condition is PERSISTENT, not a transient this lease raced past — and 110.41 GiB belongs to no /proc/meminfo category

This issue reads as a scheduling race: rc granted the device while a previous
tenant still held the RAM. Two measurements taken since say the framing
understates it, and the difference changes what a fix would even be.

1. It is persistent across jobs and hours

A start-floor gate added to the pixel-A/B harness logs every poll, not only
its refusal. On a later lease of the same box it read:

5.1 GiB available, flat from +0s to +360s

against 5.0 GiB an hour earlier in the lost lease 5fb9399f, with at least
two intervening jobs having taken and released the device in between.

A transient tenancy does not survive its own tenant, two successor jobs, and an
hour. The box sits at ~5 GiB available as a steady state. So "rc handed out
a busy box" describes the symptom at one instant; the box is persistently in
that state, and any job scheduled onto it inherits the condition rather than
racing it.

This distinction is available only because the gate logs every poll rather than
only its verdict. A gate that printed just "refused: 5.1 GiB below floor" would
have supported the transient reading indefinitely.

2. The memory belongs to no /proc/meminfo category

Commit f8d7c7d44 measured, on dgx:gpu0:

110.41 GiB accounted to no /proc/meminfo category — not process RSS, not
buff/cache, not Slab, not Shmem.

That is the substantive finding. It means:

• MemAvailable is not describing what a job can actually get, so a gate
  built on it (including the one just added) is measuring a proxy, not the
  constraint.
• The obvious suspects are unified-memory allocations on this GB10 that the
  kernel's normal accounting does not attribute — consistent with
  gpu_memory_utilization not bounding host RAM on this box, which is already
  recorded behaviour here.
• A "wait for free memory" mitigation cannot work against memory that is not
  attributed to anything and is not released by anything finishing. Whoever picks
  this up should not implement a wait loop as the fix.

3. What this changes

• The worker-loss signature (lost: worker re-registered, three occurrences,
  no recorded cause) is more plausibly the kernel reclaiming under genuine
  pressure than an rc scheduling bug. The scheduler may be innocent.
• Anything gating on MemAvailable is gating on the wrong number, including
  the start-floor gate this issue prompted. It is still worth having — a cheap
  refusal beats a lost worker — but it should be described as a tripwire, not a
  precondition.
• The next diagnostic is where the 110.41 GiB actually is: compare
  MemTotal against the sum of /proc/meminfo categories, then against
  nvidia-smi and the driver's own accounting, and see which side owns it.

Provenance

Both measurements were made while working #1612 on this box: the poll log by the
start-floor gate, and the 110.41 GiB by f8d7c7d44. Neither is a re-derivation
of this issue's original evidence, which stands as written — the Mem: total 119 used 114 free 1 ... available 5 line at [pixab +0s] is exactly consistent with
a persistent condition.

Labelled explicitly: the persistence and the 110.41 GiB are measured; the
unified-memory attribution and the reclaim-versus-scheduler reading are
inference and want confirming before anyone acts on them.