Skip to content

fix: Support named hooks without caching runtime callbacks - #161

Merged
polvalente merged 3 commits into
elixir-nx:mainfrom
jfayad:fix/named-hook-overrides
Nov 11, 2026
Merged

polvalente merged 3 commits into
elixir-nx:mainfrom
jfayad:fix/named-hook-overrides

Conversation

@jfayad

@jfayad jfayad commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #160.

Adds support for named hook callback overrides supplied through the compiler’s :hooks option, enabling consumers such as Nx.Serving to provide streaming callbacks.

  • Resolve named overrides while preserving inline defaults and missing-hook no-ops.
  • Support overrides inside native while loops, including nested loops and token hooks.
  • Validate matching callbacks as arity-one functions.
  • Fix token-hook output alias indexing.
  • Prevent request-specific callbacks from being retained or reused by global compilation caches.
  • Harden the native callback bridge against caller death and duplicate or late replies.

Callback ownership

Executables containing runtime callbacks, and compilations with nonempty :hooks maps, bypass global caches. This prevents cached programs from retaining streaming receiver aliases or invoking another request’s callback.

Caller-held Nx.Defn.compile/3 executables remain reusable with their own callbacks. Callback-free programs retain their existing caching behavior.

Native callback lifecycle

The worker retains ownership of pending callback resources while waiting. Caller death abandons the wait, and late or duplicate replies are rejected before accessing output buffers.

Raised, thrown, and exited callbacks are reported to the native waiter rather than leaving it blocked.

This cleanup does not preempt an already-running MLX kernel.

Validation

Added regression coverage for:

  • Overrides, defaults, missing hooks, and invalid callbacks.
  • Exact callback counts and ordering in nested native loops.
  • Token payloads with different arities from their aliased outputs.
  • Repeated and concurrent compilation and invocation with distinct receiver aliases.
  • No global cache growth from named or inline/default callbacks.
  • Caller death, discarded callback messages, duplicate/late replies, and worker recovery in disposable BEAM processes.

Results:

  • Nx 1.0 regression set: 355 passed, 1 skipped.
  • Final focused Nx 1.0 tests: 8 passed, 1 skipped.
  • Separate local Nx 0.13.1 compatibility run: 53 passed.

Apply named hook overrides and defaults, including inside native while loops
Keep callback-bearing executables out of global caches
Safely abandon runtime calls when callers exit and reject late replies
Comment on lines -237 to +262
pending->cv.wait(lock, [pending] { return pending->done; });
while (!is_done()) {
std::unique_lock<std::mutex> lock(pending->mu);
pending->cv.wait_for(lock, std::chrono::milliseconds(5));
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we changing this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The original wait only wakes when the callback replies. If the caller dies first, no reply arrives and the native thread can wait indefinitely. The timed wait lets this fallback path recheck caller liveness, as the normal worker path already does through pump_until. I’ll add a short comment explaining that.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks!

Comment thread emlx/lib/emlx.ex Outdated
conditional branches remain unsupported.

If a callback caller dies, the native bridge abandons its pending callback
wait and rejects late replies. This does not preempt a running MLX kernel.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

General feedback here:

  1. Let's not reference Nx versions, let's just talk about the current state of the library
  2. [...] Missing, nil, or false overrides use the inline default; a name without a default is a no-op This reads very LLM-like. Can we rewrite the whole text to be more concise and less LLM-like?
  3. The second paragraph could be rewritten into a shorter single sentence. I think there's too much detail there.
  4. Last phrase is an implementation detail and should not be included in this doc

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair points. I’ll remove the version references and native lifecycle details, and shorten this to:

“Use the compiler’s :hooks option to override named io_call callbacks. Otherwise, the inline callback runs, if provided.

Executables with callbacks bypass global caches; keep the function returned by Nx.Defn.compile/3 to reuse them.”

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thank you!

Comment thread emlx/lib/emlx.ex Outdated
table = dispatch_cache_table()

case :ets.lookup(table, cache_key) do
case if(base_key, do: :ets.lookup(table, cache_key), else: []) do

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we extract this for readability? case if(...) is a bit non-idiomatic

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I’ll assign the cache lookup result to a variable before the case, keeping the cache-bypass condition separate from matching the result.

Comment thread emlx/test/emlx/named_hooks_test.exs Outdated
Comment on lines +5 to +9
# Exercise the same compiler contract on Nx 0.13 (hook) and Nx 1 (io_call).
deftransform named(value, name) do
op = if function_exported?(Nx.Defn.Kernel, :io_call, 2), do: :io_call, else: :hook
apply(Nx.Defn.Kernel, op, [value, name])
end

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't support Nx 0.13 now, so we can just use Nx 1.0

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood. That compatibility code came from testing a downstream consumer on Nx 0.13. I’ll use io_call directly and remove the version detection and legacy-token tests from the upstream suite. The downstream compatibility checks can stay local.

Comment thread emlx/test/emlx/named_hooks_test.exs Outdated
Comment on lines +262 to +282
script = Path.expand("../fixtures/runtime_call_lifecycle.exs", __DIR__)
# Python owns and kills the disposable process on timeout, including if a
# native callback blocks VM shutdown. No hanging native work enters ExUnit.
runner = """
import subprocess, sys
try:
p = subprocess.run(sys.argv[1:], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=30)
sys.stdout.buffer.write(p.stdout)
sys.exit(p.returncode)
except subprocess.TimeoutExpired as e:
sys.stdout.buffer.write(e.stdout or b'')
print('disposable BEAM timed out')
sys.exit(124)
"""

args =
["-c", runner, System.find_executable("elixir"), "--erl", "+S 2:2"] ++ paths ++ [script]

{output, status} = System.cmd("python3", args, stderr_to_stdout: true)
assert status == 0, output
assert output =~ "runtime-call lifecycle OK"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we using Python here?
We should use :peer to start a sibling BEAM node and manage it if there's need for an isolated node at all

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On second thought, I think the whole fixture should be rewritten as an ExUnit test with async: false.
We can set a timeout to avoid hanging the code, but this seems overly defensive for too little gains

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on replacing the standalone fixture with async: false ExUnit tests and normal assertions. One distinction matters for the caller-death regression: an ExUnit timeout terminates the test process, but cannot interrupt a blocked native thread. I’d keep that particular case on an isolated peer, with the remaining checks running directly in ExUnit. Would that address your concern without losing protection against a native deadlock?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that works! Most important thing to me is to use the tooling that's already readily available where we can

Comment on lines +38 to +42
receive do
{:finished, ^caller, {:ok, [_]}} -> :ok
after
5_000 -> raise "normal callback did not finish"
end

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For example, this pattern happens a lot here and is just assert_receive ..., 5000

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Those manual receive/timeout blocks were only needed because the fixture ran outside ExUnit. Moving the checks into ExUnit lets me replace them with assert_receive ..., 5_000 and gives better failure messages.

@jfayad

jfayad commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

@polvalente have you got any chance to look into the CI failures ? looks like a runner issue 👀

@polvalente
polvalente merged commit d5c78ae into elixir-nx:main Nov 11, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support named hook callback overrides through the :hooks compiler option

2 participants