Repository navigation
fix: Free the MLX array when a tensor is deallocated - #159
Merged
Merged
Conversation
Nx.backend_deallocate/1 marked an EMLX tensor as deallocated, but the MLX array stayed in memory until the BEAM collected the term. TensorP::deallocate() only took 1 from the refcount, and ~TensorP() destroyed the array only for a count that was already 0, which never happened. A decode loop that frees its dead KV cache each step kept one more copy of the cache, and more when lazy outputs chained the old buffers. The TensorP that brings the count to 0 now replaces the array with a scalar of 4 bytes, so the buffer is freed at once. This TensorP is the deallocate call itself, or the last NIF call that still held the tensor. The resource destructor still destroys the array object at GC. A binary from to_blob aliased the buffer through the tensor resource, so it could read a freed buffer after a deallocate. The binary now keeps a minimal resource with its own reference to the buffer, for contiguous and non-contiguous tensors. EMLX.deallocate/1 also gave a FunctionClauseError for a tensor that was already deallocated. It now returns :already_deallocated, as Nx.backend_deallocate/1 expects.
polvalente
approved these changes
Oct 26, 2026
georgeguimaraes
added a commit
to georgeguimaraes/transmuter
that referenced
this pull request
Oct 26, 2026
The garbage collector of the BEAM does not see the size of a device buffer. EMLX 0.5 frees a released buffer only when the BEAM collects its tensor (see elixir-nx/emlx#159), so the dead KV cache of each step stays until a collection. A decode loop that makes little host garbage collects rarely, and the dead caches pile up. On EMLX, SmolLM2-135M :append at batch 8 and capacity 512 grew to 12-16 GB when the token processors ran on the device. release/2 now counts the bytes of the device buffers that it frees. When the count reaches 256 MB, the generation process collects its garbage. A small cache then rarely costs a collection, and a large cache gets one before its dead copies take much memory. On EXLA CUDA (RTX 3060), SmolLM2-135M batch 1 takes 4.52 ms per step on main and 4.59 ms with this rule, which is within the noise of the run. One collection takes 31 to 62 us there. On EMLX, SmolLM2-135M at batch 8 with no_repeat_ngram_size 3 stays at a peak footprint of 3.7 GB for 468 tokens.
polvalente
reviewed
Nov 6, 2026
Comment on lines
+1602
to
+1605
| case EMLX.NIF.deallocate(tensor_ref) do | ||
| :already_deallocated -> :already_deallocated | ||
| result -> unwrap!(result) | ||
| end |
Member
There was a problem hiding this comment.
deallocate could return {:ok, :already_deallocated} so that this becomes:
tensor_ref
|> EMLX.NIF.deallocate()
|> unwrap!()
polvalente
approved these changes
Nov 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Right now
Nx.backend_deallocate/1on EMLX doesn't free anything. The MLX array stays alive until the BEAM collects the tensor term. So a decode loop that frees its dead KV cache every step still keeps the old cache around until the next GC.A probe on main (256 MB tensors, Mac GPU,
EMLX.memory_info().active_memory):Nx.backend_deallocate/1of an evaluated tensor, or of a lazy output of a compiled function:erlang.garbage_collect()it drops to 0The cause is in
TensorP(c_src/emlx_nif_shared.hpp):deallocate()only does--(*refcount)and sets the deleted flag~TensorP()destroys the array only whenfetch_sub(1) == 0, so when the count was already 0. But the constructor refuses a tensor with a count of 0, so that branch never runs.Changes:
TensorPthat brings the count to 0 replaces the array with a 4-byte scalar, so the buffer is freed right away. ThatTensorPis the deallocate call itself, or the last NIF call that still held the tensor. The resource destructor still destroys the object at GC.TensorP. An array that a lazy graph still references keeps its data in MLX, and only this handle drops, the same as at GC today.to_blobaliased the buffer through the tensor resource, so after this change a binary could point at a freed buffer. The binary now holds a small resource with its own array, which shares the buffer without a copy, the same as the non-contiguous path already did.EMLX.deallocate/1went throughunwrap!/1, which raisedFunctionClauseErroron:already_deallocated. It now returns:already_deallocated, asNx.backend_deallocate/1expects.Effect in a real loop (Transmuter generation, SmolLM2-135M, batch 8, capacity 1024, 20 tokens): MLX active memory stays flat at 1.21 GB, where it grew by 0.35 GB per step before. A request leaves no memory behind, where it left 182 MB until the next GC.
New tests in
memory_test.exs::already_deallocatedThe first two fail on main. The third fails with only the
TensorPchange. I ran each variant 5 times: with the whole fix the memory tests pass every time, and without theto_blobchange the binary test fails every time.The suite on my Mac has the same 12 failures as main (runtime_call graph split, triangular_solve, gather_quantized_matmul), and no new ones.