Skip to content

fix: Free the MLX array when a tensor is deallocated - #159

Merged
polvalente merged 3 commits into
elixir-nx:mainfrom
georgeguimaraes:fix/deallocate-frees
Nov 6, 2026
Merged

polvalente merged 3 commits into
elixir-nx:mainfrom
georgeguimaraes:fix/deallocate-frees

Conversation

@georgeguimaraes

Copy link
Copy Markdown
Member

Right now Nx.backend_deallocate/1 on EMLX doesn't free anything. The MLX array stays alive until the BEAM collects the tensor term. So a decode loop that frees its dead KV cache every step still keeps the old cache around until the next GC.

A probe on main (256 MB tensors, Mac GPU, EMLX.memory_info().active_memory):

  • the active memory doesn't change after Nx.backend_deallocate/1 of an evaluated tensor, or of a lazy output of a compiled function
  • after :erlang.garbage_collect() it drops to 0

The cause is in TensorP (c_src/emlx_nif_shared.hpp):

  • deallocate() only does --(*refcount) and sets the deleted flag
  • ~TensorP() destroys the array only when fetch_sub(1) == 0, so when the count was already 0. But the constructor refuses a tensor with a count of 0, so that branch never runs.

Changes:

  • the TensorP that brings the count to 0 replaces the array with a 4-byte scalar, so the buffer is freed right away. That TensorP is the deallocate call itself, or the last NIF call that still held the tensor. The resource destructor still destroys the object at GC.
  • this is safe because the count reaches 0 only when no NIF call holds a TensorP. An array that a lazy graph still references keeps its data in MLX, and only this handle drops, the same as at GC today.
  • to_blob aliased the buffer through the tensor resource, so after this change a binary could point at a freed buffer. The binary now holds a small resource with its own array, which shares the buffer without a copy, the same as the non-contiguous path already did.
  • EMLX.deallocate/1 went through unwrap!/1, which raised FunctionClauseError on :already_deallocated. It now returns :already_deallocated, as Nx.backend_deallocate/1 expects.

Effect in a real loop (Transmuter generation, SmolLM2-135M, batch 8, capacity 1024, 20 tokens): MLX active memory stays flat at 1.21 GB, where it grew by 0.35 GB per step before. A request leaves no memory behind, where it left 182 MB until the next GC.

New tests in memory_test.exs:

  • deallocate frees active memory without a GC
  • a deallocated tensor raises on use, and a second deallocate returns :already_deallocated
  • a binary keeps the buffer of a deallocated tensor

The first two fail on main. The third fails with only the TensorP change. I ran each variant 5 times: with the whole fix the memory tests pass every time, and without the to_blob change the binary test fails every time.

The suite on my Mac has the same 12 failures as main (runtime_call graph split, triangular_solve, gather_quantized_matmul), and no new ones.

Nx.backend_deallocate/1 marked an EMLX tensor as deallocated, but the MLX array stayed in memory until the BEAM collected the term. TensorP::deallocate() only took 1 from the refcount, and ~TensorP() destroyed the array only for a count that was already 0, which never happened. A decode loop that frees its dead KV cache each step kept one more copy of the cache, and more when lazy outputs chained the old buffers.

The TensorP that brings the count to 0 now replaces the array with a scalar of 4 bytes, so the buffer is freed at once. This TensorP is the deallocate call itself, or the last NIF call that still held the tensor. The resource destructor still destroys the array object at GC.

A binary from to_blob aliased the buffer through the tensor resource, so it could read a freed buffer after a deallocate. The binary now keeps a minimal resource with its own reference to the buffer, for contiguous and non-contiguous tensors.

EMLX.deallocate/1 also gave a FunctionClauseError for a tensor that was already deallocated. It now returns :already_deallocated, as Nx.backend_deallocate/1 expects.
georgeguimaraes added a commit to georgeguimaraes/transmuter that referenced this pull request Oct 26, 2026
The garbage collector of the BEAM does not see the size of a device buffer. EMLX 0.5 frees a released buffer only when the BEAM collects its tensor (see elixir-nx/emlx#159), so the dead KV cache of each step stays until a collection. A decode loop that makes little host garbage collects rarely, and the dead caches pile up. On EMLX, SmolLM2-135M :append at batch 8 and capacity 512 grew to 12-16 GB when the token processors ran on the device.

release/2 now counts the bytes of the device buffers that it frees. When the count reaches 256 MB, the generation process collects its garbage. A small cache then rarely costs a collection, and a large cache gets one before its dead copies take much memory.

On EXLA CUDA (RTX 3060), SmolLM2-135M batch 1 takes 4.52 ms per step on main and 4.59 ms with this rule, which is within the noise of the run. One collection takes 31 to 62 us there. On EMLX, SmolLM2-135M at batch 8 with no_repeat_ngram_size 3 stays at a peak footprint of 3.7 GB for 468 tokens.
Comment thread emlx/lib/emlx.ex Outdated
Comment on lines +1602 to +1605
case EMLX.NIF.deallocate(tensor_ref) do
:already_deallocated -> :already_deallocated
result -> unwrap!(result)
end

@polvalente polvalente Nov 6, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deallocate could return {:ok, :already_deallocated} so that this becomes:

tensor_ref
|> EMLX.NIF.deallocate()
|> unwrap!()

@polvalente
polvalente merged commit a51f383 into elixir-nx:main Nov 6, 2026
9 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants