Skip to content

feat: Add EMLX.async_eval/1 so a decode loop can run one step ahead - #158

Merged
polvalente merged 3 commits into
elixir-nx:mainfrom
georgeguimaraes:feat/async-eval
Nov 6, 2026
Merged

polvalente merged 3 commits into
elixir-nx:mainfrom
georgeguimaraes:feat/async-eval

Conversation

@georgeguimaraes

@georgeguimaraes georgeguimaraes commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

Right now a decode loop on EMLX can't run ahead of the GPU. eval_program waits for its inputs and eval waits for the whole stream, so the host work of each step (graph replay, BEAM work, reading the token) happens while the GPU sits idle. This PR lets a loop do what mlx-lm does: call step t+1 with the lazy token of step t, schedule it, then read token t while the GPU runs t+1.

Changes:

  • new EMLX.async_eval/1: schedules tensors on their worker and returns right away
  • eval_program schedules its inputs with mx::async_eval instead of mx::eval. They're still not lazy nodes when the compiled fn sees them, which was the reason for that eval.
  • the eval NIF drops mx::synchronize(). mx::eval already waits for its own tensor, so reads still get complete data. They just don't wait for later work on the stream anymore.

Numbers from a pipelined Transmuter decode loop on this branch (M-series Mac, greedy, 32 new tokens, median of 7, same tokens):

model today pipelined
GPT-2 5.53 ms/token 4.32 ms/token
Qwen3-1.7B q4 19.98 ms/token 15.76 ms/token

That's roughly one step of GPU time hidden per token. The Transmuter side goes in its own change after this lands.

Trade-offs:

  • EMLX.eval/1 isn't a stream barrier anymore. If you used it to wait for other work (timing, memory stats), use EMLX.CommandQueue.synchronize/1. Nothing inside EMLX relied on it: to_blob, to_device, tensor_data_ptr, tensor_to_shm and force_eval_outputs only read the tensor they evaluate.
  • a compiled call can return before its inputs are computed, so an error in an input shows up at the read instead of at the call
  • compiled :while and runtime calls still sync their inputs through synchronize_thread_streams() from fix: Wait for scheduled work before a compiled :while runs its loop #155, and that test passes

Compared with EXLA: EXLA runs every executable with kSynchronous (exla_client.cc:273) and awaits the ready future of every output before run returns (exla_client.cc:242), on a dirty scheduler. So every EXLA call is a full sync and the host can't run ahead. Its read path uses ToLiteralSync() on just that buffer (exla_client.cc:61), which is what eval does here after this PR. So for decode loops EMLX gets something EXLA doesn't have today.

Follow-up: with a bound command queue (partitions: true in Nx.Serving), wrap_with_queue still evals every output of every compiled call, so pipelining doesn't help there yet. Separate PR.

A decode loop waits for the GPU twice per token, and the CPU work of each step runs while the GPU is idle. EMLX.async_eval/1 schedules tensors and returns at once. eval_program now schedules its inputs with mx::async_eval instead of mx::eval. The eval NIF no longer calls a stream-wide synchronize, because it would also wait for work that async_eval scheduled later. A loop can then build step t + 1 while the device runs step t.
georgeguimaraes added a commit to georgeguimaraes/transmuter that referenced this pull request Oct 26, 2026
The selection of the next token now runs after the decoder in one compiled program (the new :tail option of Transmuter.compile/3). So a generation step is one call, and the logits stay on the device.

The tokens that the repetition penalty applies to (seen) and the number of the step also stay on the device. The selection of each step marks its new tokens in seen and gives seen back. Before, the host sent seen to the device in each step: 50 KB for GPT-2 and 150 KB for Qwen3.

ms per token, median of 7, same tokens:
- EXLA CUDA (RTX 3060): GPT-2 3.89 to 3.15, Qwen3-0.6B 11.9 to 10.75
- EMLX 0.5 (Mac GPU): GPT-2 5.48 to 5.44, Qwen3-0.6B 10.5 to 11.0

On EMLX 0.5, the new seen and step outputs stay lazy, and the next step evaluates them in a separate wait. elixir-nx/emlx#158 removes this cost: Qwen3-0.6B 10.5 to 10.38 on that branch.
georgeguimaraes added a commit to georgeguimaraes/transmuter that referenced this pull request Oct 27, 2026
Since a1eea43, the step gives the state of the next step (seen, the step number, the tokens and the count of each row) as outputs that stay on the device. The host reads only the selection. On EMLX 0.5, each other output stays an expression, and the next call evaluates it in a wait of its own (elixir-nx/emlx#158 removes this). For SmolLM2 q4 at batch_size 1, the compiled call took about 2.0 ms per token in place of 1.2 ms before a1eea43. An eval of every input just before each call brought the call back to 1.1 ms, so the 10 more inputs of the selection cost nothing, and the difference was these waits.

Now the selection gets 0 times the first value of each integer state output, so the one read of the selection evaluates them in the same wait. 0 times an integer is 0, so the selection does not change. A float output has no part in it, because 0 times NaN or infinity is NaN.

EMLX (Apple M2 Pro GPU), SmolLM2-135M-Instruct q4 at batch_size 1, the throughput bench of guides/pipelines.md with 16 requests, 3 measured rounds per run, alternating, the same text: 138.5 and 136.5 tokens/s in place of 124.3 and 119.4 (an earlier form of this change: 134.1 and 133.2 against 124.9 and 123.2). The commit before a1eea43 gives about 145 with the same bench.

EXLA CUDA (RTX 3060), no change, the same text: opus-mt-en-de with 4 beams 2077 and 2024 us per step of the long sentence against 2059 and 2102 us on main, and SmolLM2 q4 at batch_size 8 6772 and 6757 us against 6786 and 6772 us.
@polvalente
polvalente merged commit c95e6f9 into elixir-nx:main Nov 6, 2026
9 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants