Repository navigation
feat: Add EMLX.async_eval/1 so a decode loop can run one step ahead - #158
Merged
Merged
Conversation
A decode loop waits for the GPU twice per token, and the CPU work of each step runs while the GPU is idle. EMLX.async_eval/1 schedules tensors and returns at once. eval_program now schedules its inputs with mx::async_eval instead of mx::eval. The eval NIF no longer calls a stream-wide synchronize, because it would also wait for work that async_eval scheduled later. A loop can then build step t + 1 while the device runs step t.
georgeguimaraes
added a commit
to georgeguimaraes/transmuter
that referenced
this pull request
Oct 26, 2026
The selection of the next token now runs after the decoder in one compiled program (the new :tail option of Transmuter.compile/3). So a generation step is one call, and the logits stay on the device. The tokens that the repetition penalty applies to (seen) and the number of the step also stay on the device. The selection of each step marks its new tokens in seen and gives seen back. Before, the host sent seen to the device in each step: 50 KB for GPT-2 and 150 KB for Qwen3. ms per token, median of 7, same tokens: - EXLA CUDA (RTX 3060): GPT-2 3.89 to 3.15, Qwen3-0.6B 11.9 to 10.75 - EMLX 0.5 (Mac GPU): GPT-2 5.48 to 5.44, Qwen3-0.6B 10.5 to 11.0 On EMLX 0.5, the new seen and step outputs stay lazy, and the next step evaluates them in a separate wait. elixir-nx/emlx#158 removes this cost: Qwen3-0.6B 10.5 to 10.38 on that branch.
polvalente
approved these changes
Oct 26, 2026
georgeguimaraes
added a commit
to georgeguimaraes/transmuter
that referenced
this pull request
Oct 27, 2026
Since a1eea43, the step gives the state of the next step (seen, the step number, the tokens and the count of each row) as outputs that stay on the device. The host reads only the selection. On EMLX 0.5, each other output stays an expression, and the next call evaluates it in a wait of its own (elixir-nx/emlx#158 removes this). For SmolLM2 q4 at batch_size 1, the compiled call took about 2.0 ms per token in place of 1.2 ms before a1eea43. An eval of every input just before each call brought the call back to 1.1 ms, so the 10 more inputs of the selection cost nothing, and the difference was these waits. Now the selection gets 0 times the first value of each integer state output, so the one read of the selection evaluates them in the same wait. 0 times an integer is 0, so the selection does not change. A float output has no part in it, because 0 times NaN or infinity is NaN. EMLX (Apple M2 Pro GPU), SmolLM2-135M-Instruct q4 at batch_size 1, the throughput bench of guides/pipelines.md with 16 requests, 3 measured rounds per run, alternating, the same text: 138.5 and 136.5 tokens/s in place of 124.3 and 119.4 (an earlier form of this change: 134.1 and 133.2 against 124.9 and 123.2). The commit before a1eea43 gives about 145 with the same bench. EXLA CUDA (RTX 3060), no change, the same text: opus-mt-en-de with 4 beams 2077 and 2024 us per step of the long sentence against 2059 and 2102 us on main, and SmolLM2 q4 at batch_size 8 6772 and 6757 us against 6786 and 6772 us.
polvalente
approved these changes
Nov 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Right now a decode loop on EMLX can't run ahead of the GPU.
eval_programwaits for its inputs andevalwaits for the whole stream, so the host work of each step (graph replay, BEAM work, reading the token) happens while the GPU sits idle. This PR lets a loop do what mlx-lm does: call step t+1 with the lazy token of step t, schedule it, then read token t while the GPU runs t+1.Changes:
EMLX.async_eval/1: schedules tensors on their worker and returns right awayeval_programschedules its inputs withmx::async_evalinstead ofmx::eval. They're still not lazy nodes when the compiled fn sees them, which was the reason for that eval.evalNIF dropsmx::synchronize().mx::evalalready waits for its own tensor, so reads still get complete data. They just don't wait for later work on the stream anymore.Numbers from a pipelined Transmuter decode loop on this branch (M-series Mac, greedy, 32 new tokens, median of 7, same tokens):
That's roughly one step of GPU time hidden per token. The Transmuter side goes in its own change after this lands.
Trade-offs:
EMLX.eval/1isn't a stream barrier anymore. If you used it to wait for other work (timing, memory stats), useEMLX.CommandQueue.synchronize/1. Nothing inside EMLX relied on it:to_blob,to_device,tensor_data_ptr,tensor_to_shmandforce_eval_outputsonly read the tensor they evaluate.:whileand runtime calls still sync their inputs throughsynchronize_thread_streams()from fix: Wait for scheduled work before a compiled :while runs its loop #155, and that test passesCompared with EXLA: EXLA runs every executable with
kSynchronous(exla_client.cc:273) and awaits the ready future of every output beforerunreturns (exla_client.cc:242), on a dirty scheduler. So every EXLA call is a full sync and the host can't run ahead. Its read path usesToLiteralSync()on just that buffer (exla_client.cc:61), which is whatevaldoes here after this PR. So for decode loops EMLX gets something EXLA doesn't have today.Follow-up: with a bound command queue (
partitions: trueinNx.Serving),wrap_with_queuestill evals every output of every compiled call, so pipelining doesn't help there yet. Separate PR.