Repository navigation
fix: Support named hooks without caching runtime callbacks - #161
Conversation
Apply named hook overrides and defaults, including inside native while loops Keep callback-bearing executables out of global caches Safely abandon runtime calls when callers exit and reject late replies
| pending->cv.wait(lock, [pending] { return pending->done; }); | ||
| while (!is_done()) { | ||
| std::unique_lock<std::mutex> lock(pending->mu); | ||
| pending->cv.wait_for(lock, std::chrono::milliseconds(5)); | ||
| } |
There was a problem hiding this comment.
The original wait only wakes when the callback replies. If the caller dies first, no reply arrives and the native thread can wait indefinitely. The timed wait lets this fallback path recheck caller liveness, as the normal worker path already does through pump_until. I’ll add a short comment explaining that.
| conditional branches remain unsupported. | ||
|
|
||
| If a callback caller dies, the native bridge abandons its pending callback | ||
| wait and rejects late replies. This does not preempt a running MLX kernel. |
There was a problem hiding this comment.
General feedback here:
- Let's not reference Nx versions, let's just talk about the current state of the library
[...] Missing, nil, or false overrides use the inline default; a name without a default is a no-opThis reads very LLM-like. Can we rewrite the whole text to be more concise and less LLM-like?- The second paragraph could be rewritten into a shorter single sentence. I think there's too much detail there.
- Last phrase is an implementation detail and should not be included in this doc
There was a problem hiding this comment.
Fair points. I’ll remove the version references and native lifecycle details, and shorten this to:
“Use the compiler’s :hooks option to override named io_call callbacks. Otherwise, the inline callback runs, if provided.
Executables with callbacks bypass global caches; keep the function returned by Nx.Defn.compile/3 to reuse them.”
| table = dispatch_cache_table() | ||
|
|
||
| case :ets.lookup(table, cache_key) do | ||
| case if(base_key, do: :ets.lookup(table, cache_key), else: []) do |
There was a problem hiding this comment.
Can we extract this for readability? case if(...) is a bit non-idiomatic
There was a problem hiding this comment.
Agreed. I’ll assign the cache lookup result to a variable before the case, keeping the cache-bypass condition separate from matching the result.
| # Exercise the same compiler contract on Nx 0.13 (hook) and Nx 1 (io_call). | ||
| deftransform named(value, name) do | ||
| op = if function_exported?(Nx.Defn.Kernel, :io_call, 2), do: :io_call, else: :hook | ||
| apply(Nx.Defn.Kernel, op, [value, name]) | ||
| end |
There was a problem hiding this comment.
We don't support Nx 0.13 now, so we can just use Nx 1.0
There was a problem hiding this comment.
Understood. That compatibility code came from testing a downstream consumer on Nx 0.13. I’ll use io_call directly and remove the version detection and legacy-token tests from the upstream suite. The downstream compatibility checks can stay local.
| script = Path.expand("../fixtures/runtime_call_lifecycle.exs", __DIR__) | ||
| # Python owns and kills the disposable process on timeout, including if a | ||
| # native callback blocks VM shutdown. No hanging native work enters ExUnit. | ||
| runner = """ | ||
| import subprocess, sys | ||
| try: | ||
| p = subprocess.run(sys.argv[1:], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=30) | ||
| sys.stdout.buffer.write(p.stdout) | ||
| sys.exit(p.returncode) | ||
| except subprocess.TimeoutExpired as e: | ||
| sys.stdout.buffer.write(e.stdout or b'') | ||
| print('disposable BEAM timed out') | ||
| sys.exit(124) | ||
| """ | ||
|
|
||
| args = | ||
| ["-c", runner, System.find_executable("elixir"), "--erl", "+S 2:2"] ++ paths ++ [script] | ||
|
|
||
| {output, status} = System.cmd("python3", args, stderr_to_stdout: true) | ||
| assert status == 0, output | ||
| assert output =~ "runtime-call lifecycle OK" |
There was a problem hiding this comment.
Why are we using Python here?
We should use :peer to start a sibling BEAM node and manage it if there's need for an isolated node at all
There was a problem hiding this comment.
On second thought, I think the whole fixture should be rewritten as an ExUnit test with async: false.
We can set a timeout to avoid hanging the code, but this seems overly defensive for too little gains
There was a problem hiding this comment.
Agreed on replacing the standalone fixture with async: false ExUnit tests and normal assertions. One distinction matters for the caller-death regression: an ExUnit timeout terminates the test process, but cannot interrupt a blocked native thread. I’d keep that particular case on an isolated peer, with the remaining checks running directly in ExUnit. Would that address your concern without losing protection against a native deadlock?
There was a problem hiding this comment.
Yeah, that works! Most important thing to me is to use the tooling that's already readily available where we can
| receive do | ||
| {:finished, ^caller, {:ok, [_]}} -> :ok | ||
| after | ||
| 5_000 -> raise "normal callback did not finish" | ||
| end |
There was a problem hiding this comment.
For example, this pattern happens a lot here and is just assert_receive ..., 5000
There was a problem hiding this comment.
Agreed. Those manual receive/timeout blocks were only needed because the fixture ran outside ExUnit. Moving the checks into ExUnit lets me replace them with assert_receive ..., 5_000 and gives better failure messages.
|
@polvalente have you got any chance to look into the CI failures ? looks like a runner issue 👀 |
Summary
Fixes #160.
Adds support for named hook callback overrides supplied through the compiler’s
:hooksoption, enabling consumers such as Nx.Serving to provide streaming callbacks.Callback ownership
Executables containing runtime callbacks, and compilations with nonempty
:hooksmaps, bypass global caches. This prevents cached programs from retaining streaming receiver aliases or invoking another request’s callback.Caller-held
Nx.Defn.compile/3executables remain reusable with their own callbacks. Callback-free programs retain their existing caching behavior.Native callback lifecycle
The worker retains ownership of pending callback resources while waiting. Caller death abandons the wait, and late or duplicate replies are rejected before accessing output buffers.
Raised, thrown, and exited callbacks are reported to the native waiter rather than leaving it blocked.
This cleanup does not preempt an already-running MLX kernel.
Validation
Added regression coverage for:
Results: