Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
---
module: "LlamaCppEx.NIF (c_src/llama_cpp_ex/llama_nif.cpp)"
date: "2026-11-01"
problem_type: runtime_error
component: nif_teardown
symptoms:
- "ggml-metal-device.m:1041: GGML_ASSERT([rsets->data count] == 0) failed, then SIGABRT (exit 134) after the program finished"
- "a model held in persistent_term, ETS or a process alive at System.halt/1 aborts the VM instead of exiting with its own status"
- "0.8.62 exits 0 on the same script, 0.8.63 and 0.8.64 abort (precompiled Metal NIF)"
- "a green mix test run on Metal exits 134 unless GGML_METAL_NO_RESIDENCY=1 is set"
root_cause: "The BEAM never runs NIF resource destructors at halt, so live models/contexts still own Metal buffers when exit(3) runs C++ static destructors; llama.cpp's Metal device is a function-local static whose destructor asserts no residency set is left. Residency sets are only compiled with a macOS 15+ SDK, and the move to macos-15 runners (v0.8.63) put them into the precompiled NIF."
severity: high
tags: [metal, nif, atexit, static-destruction, residency-sets, vm-halt, macos-sdk, fine]
related_solutions:
- ".claude/solutions/deployment-issues/macos-14-runner-brownout-precompile-20261031.md"
---

# VM aborts at exit on Metal while a model is still referenced (#116)

## Symptoms

```
ggml/src/ggml-metal/ggml-metal-device.m:1041: GGML_ASSERT([rsets->data count] == 0) failed
3 ggml_metal_device_free
4 std::vector<unique_ptr<ggml_metal_device, ggml_metal_device_deleter>>::~vector()
5 __cxa_finalize_ranges
6 exit
7 beam.smp erts_exit_vv
```

Repro: load a model, keep it referenced from any live process, let the
script end. Exit 134 on 0.8.63/0.8.64, 0 on 0.8.62. If the script process is
the only holder, nothing aborts: it dies before halt and the resource is GC'd.

## Investigation

1. **"llama.cpp changed"**: no. v0.8.62→v0.8.64 all pin `418f905a4`.
2. **Compare the artifacts**, which settled it. `otool -l` on the cached
NIFs in `~/Library/Caches/elixir_make`:
- 0.8.62: `sdk 14.5`, 0 residency-set selectors;
- 0.8.63/0.8.64: `sdk 15.5`, `newResidencySetWithDescriptor:error:`,
`addAllocation:`, `endResidency` and `requestResidency` present.

`ggml-metal-device.m:20` defines `GGML_METAL_HAS_RESIDENCY_SETS` only when
`__MAC_OS_X_VERSION_MAX_ALLOWED >= 150000`. Without it, `rsets->data` stays
empty and the assert can never fire. That is why 0.8.62 worked, and why
local builds (newer SDK) always needed `GGML_METAL_NO_RESIDENCY=1`.
3. **Ruled out `_exit` from an atexit hook**: it would lose the VM's exit
status, which no atexit API exposes.
4. **Ruled out freeing without a guard**: the resources have no locks, so a
dirty NIF mid-decode during halt would be a use-after-free.
5. **First fix had an ordering hole** (found by review, confirmed 3/3 SIGBUS,
exit -10). The hook is registered once per allocation kind, so ggml statics
first built *later* are destroyed *before* it. Case: a first model loaded
with `devices: ["CPU"]`, then a Metal model and context. The Metal
buffer-type statics (`ggml_backend_metal_buffer_type_shared/private`, whose
`bufts`/`ctxs` vectors hold the names) are built after the hook, and
`~llama_context` logs `ggml_backend_buft_name(buft)` → freed memory.

## Root Cause

Two facts combine:

- `erlang:halt` → `exit(3)` runs `atexit` handlers and C++ static destructors
in reverse order of registration. The BEAM does not first destroy NIF
resources still referenced from live processes, ETS or `persistent_term`.
- `ggml_metal_device_get` keeps devices in a function-local
`static std::vector<ggml_metal_device_ptr>`. Its deleter calls
`ggml_metal_rsets_free`, which asserts that every buffer's residency set was
removed, i.e. that every model/context was freed first.

Changing the build SDK (runner image) silently switched that code path on for
every precompiled-NIF user.

## Solution

`c_src/llama_cpp_ex/llama_nif.{h,cpp}`, "Teardown at VM exit (#116)":

```cpp
// Resources register themselves; destructors free under the same lock.
explicit LlamaModel(llama_model* m) : ... { live::add(this); }
~LlamaModel() {
std::lock_guard<std::recursive_mutex> lock(live::mutex());
live::remove(this);
release(); // frees + nulls; also called by the hook
}

// Every NIF: FINE_NIF plus an in-flight guard (seq_cst Dekker handshake).
#define LLAMA_NIF(name, flags) ... NifCall call; if (!call.entered) raise vm_exiting; ...

void free_live_resources_at_exit() {
if (g_exiting.exchange(true)) return;
// wait up to 5 s for g_in_flight == 0, else free nothing (no UAF)
// under the lock: speculatives -> contexts with ctx_other -> contexts -> models
}

void arm_exit_hook(std::once_flag& once) { // Apple only
std::call_once(once, [] {
// build Metal buffer-type statics BEFORE registering (CPU-first case)
for each device whose reg name is "MTL":
ggml_backend_dev_buffer_type(dev); ggml_backend_dev_host_buffer_type(dev);
std::atexit(free_live_resources_at_exit);
});
}
// armed after first model_load, context_create, speculative_init
```

Also:
- Generation loops, `cancel_abort_cb` and a model-load `progress_callback`
return early on `g_exiting`, so a halt does not wait out a long call.
- Setting `progress_callback` replaces llama.cpp's default one, so it re-prints
the progress dots via `log_callback`.

### Files Changed

- `c_src/llama_cpp_ex/llama_nif.h` — `live::` registry, `release()` on
LlamaModel/LlamaContext/LlamaSpeculative, `has_ctx_other`
- `c_src/llama_cpp_ex/llama_nif.cpp` — registry, `NifCall`, `LLAMA_NIF` (all 67
NIFs), exit hook, arming, `g_exiting` checks
- `test/vm_exit_test.exs` — child VM halts with live resources: model + context
(`halt(7)` → 7), CPU-only then GPU model (`halt(5)` → 5), Server
mid-generation (→ 0). It fails without the fix (134 / SIGBUS).
- `GGML_METAL_NO_RESIDENCY=1` removed from test_helper, test moduledocs,
docs/release-guide.md, docs/performance.md, scripts/decision_compare.exs

Shipped in v0.8.65 (#118, #119). Verified with residency sets on: suites exit 0
(CI set ×8, release-guide set with MTP, default), and the e2e on the published
package passes 26/26 with four models referenced at halt.

## Prevention

- When the precompile runner image or SDK changes, diff the artifacts
(`otool -l` → `sdk`/`minos`, `strings` for feature selectors) against the
previous release. Feature gates keyed on SDK macros switch silently.
- Never keep a run-instruction env-var workaround for an abort without an
issue tracking it. `GGML_METAL_NO_RESIDENCY=1` hid this from every local
run, CI (Linux) and the v0.8.64 e2e smoke.
- Smoke the published package without workarounds and check the exit
status, not just "N passed".
- Native code freed at process exit must be registered *after* every static
it touches is constructed. Force lazily built statics (Metal buffer types)
before `std::atexit`. Don't re-register per allocation: stateless
`generate` makes a context per call, so the atexit list would grow without
bound.
- New NIFs must use `LLAMA_NIF`, not `FINE_NIF`, or the exit hook can free
under them.

## Related

- `.claude/solutions/deployment-issues/macos-14-runner-brownout-precompile-20261031.md`
— the runner move that raised the SDK to 15.5
- Issue https://github.com/nyo16/llama_cpp_ex/issues/116, PR #118
- Iron Law #28 (iron-law-judge): `LLAMA_NIF` only; GPU-owning resources join
`live::`; loops stop on `g_exiting`
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
---
module: "LlamaCppEx.ServerSmokeTest"
date: "2026-10-31"
problem_type: test_failure
component: testing
symptoms:
- "test halting a stream early cancels generation and frees the slot: ** (MatchError) no match of right hand side value: {%{prompt_tokens: 39, generated_tokens: 4, ...}, %{stop_reason: :max_tokens, ...}}"
- "fails only on CI (CPU, seed 489492) on a docs-only PR; passes locally with the same seed"
root_cause: "A :telemetry handler is global, and the test's collector forwarded the first [:llama_cpp_ex, :server, :request, :done] event from any server. The previous test's server was still alive: ExUnit does not wait for a test's linked processes to stop, and Server.finish_slot/3 replies to the caller before emitting :done, so that server's stale event arrived after the next test attached its handler."
severity: medium
tags: [telemetry, flaky-test, exunit, global-handler, linked-process, server, race]
---

# Flaky server smoke test: stale `:done` telemetry from the previous test's server

## Symptoms

The CI Inference smoke on #115, a docs-only PR, failed:

```
1) test halting a stream early cancels generation and frees the slot (LlamaCppEx.ServerSmokeTest)
** (MatchError) no match of right hand side value:
{%{prompt_tokens: 39, generated_tokens: 4, ...}, %{stop_reason: :max_tokens, server: #PID<0.1528.0>, ...}}
code: {%{generated_tokens: n}, %{stop_reason: :cancelled}} = next_telemetry()
```

The test streams "Write an endless story:" (about 5 prompt tokens,
`max_tokens: 400`), so a 39-token, `max_tokens` (4) event cannot be its own.

## Investigation

1. **"Cancel lost the race with generation"**: no. Stop reason `max_tokens`
after 4 tokens doesn't match this request's `max_tokens: 400`.
2. **Find the 39-token request**: the preceding test, "chat_completion/3
propagates a server-side generation error". Its last call is
`chat_completion(server, messages, max_tokens: 4)`, a chat-templated
prompt of 39 tokens on SmolLM2.
3. **CI log order**:
- 54.311: handler attached by the new test's process, `#PID<0.1529.0>`;
- 54.314: `[debug] slot 0 done (max_tokens): 39 prompt tokens`;
- 54.514: the real `cancelled` event.
4. Locally (Metal) the same seed passed with and without the fix. On faster
hardware the old server usually emits before the next test attaches, so it
only reproduces on timing.

## Root Cause

- `finish_slot/3` calls `GenServer.reply` first, then `Logger.debug` and
`:telemetry.execute(... :done ...)`. The caller can return, its test can
finish, and the next test can start in between.
- The server is `start_link`ed to the test process and traps exits.
ExUnit does not wait for linked processes to terminate after a test, so the
old server keeps running briefly.
- `:telemetry.attach/4` handlers are global. The collector sent every event to
the test process with no filter:

```elixir
fn _event, measurements, metadata, _ ->
send(parent, {:telemetry, measurements, metadata})
end
```

## Solution

Pin the server under test. Every server telemetry event carries
`server: self()` in its metadata.

```elixir
defp attach_collector(server, event) do
parent = self()
handler_id = {__MODULE__, self(), event}

:telemetry.attach(handler_id, event,
fn
_event, measurements, %{server: ^server} = metadata, _ ->
send(parent, {:telemetry, measurements, metadata})
_event, _measurements, _metadata, _ ->
:ok
end, nil)

on_exit(fn -> :telemetry.detach(handler_id) end)
end
```

The assertions are unchanged; only foreign events are dropped.

### Files Changed

- `test/server_smoke_test.exs` — `attach_collector/2` filters on `server`; 4
callers updated
- `test/llama_cpp_ex_test.exs` — the telemetry test's three handlers pin
`server`, detach in `on_exit` (previously only on success), and assert
`metadata.server == server`

Shipped in #117 (v0.8.64). CI green afterwards.

## Prevention

- Every test-side `:telemetry.attach` must filter on an identity the test owns
(`server`, `ref`, `seq_id`) and detach in `on_exit`, not at the end of the
test body.
- Don't assume a linked process is gone when the next test starts. Stop it
explicitly or filter its output.
- A flake on an unrelated PR: read the event payload before re-running. Here
the prompt-token count identified the sender.

## Related

- `.claude/solutions/phoenix-issues/metal-exit-abort-rsets-assert-nif-teardown-20261101.md`
— same "process outlives its test" family (a live model at halt)
- Iron Law #29 (iron-law-judge): test telemetry handlers pin an owned identity
and detach in `on_exit`. The last unfiltered handler (hybrid-cache test in
`test/llama_cpp_ex_test.exs`) was fixed when this law was added.
29 changes: 28 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
# llama_cpp_ex — project rules

Documented solutions live in `.claude/solutions/`; search there before
investigating a tokenizer, template, logging, or release problem.
investigating a tokenizer, template, logging, NIF teardown, flaky test or
release problem.

## Prompts and tokenization

Expand Down Expand Up @@ -29,6 +30,20 @@ investigating a tokenizer, template, logging, or release problem.
load. Set `LlamaCppEx.set_log_level/1` (or `config :llama_cpp_ex,
log_level:`) *before* `LlamaCppEx.init/0`, or backend-init lines escape.

## NIF and teardown

- Register every NIF with `LLAMA_NIF`, never `FINE_NIF`: the wrapper's
in-flight guard is what lets the exit hook free resources safely (#116).
- A resource that owns backend buffers (`llama_model`, `llama_context`,
`common_speculative`, anything allocating on a ggml device) joins the
`live::` registry: `live::add` in constructors, a destructor that locks
`live::mutex()`, removes itself and calls `release()`, and a release step in
`free_live_resources_at_exit` in dependency order. Long loops and callbacks
inside NIFs stop on `g_exiting`.
- Exit-path regressions show only in the VM's exit status. Run
`test/vm_exit_test.exs` (`--include smoke`) after touching resource
lifetimes, and never paper over an abort with `GGML_METAL_NO_RESIDENCY=1`.

## Releases and CI

- Any macOS runner image bump in `.github/workflows/precompile.yml` keeps
Expand All @@ -42,5 +57,17 @@ investigating a tokenizer, template, logging, or release problem.
latest release and changelog before working around it.
- CI runs `--include tools` against pinned Qwen3.5-0.8B-UD-Q4_K_XL; keep the
`:tools` tests passing on the CPU backend.
- When the macOS runner image or SDK changes, compare the new NIF with the
previous release (`otool -l` for `sdk`/`minos`, `strings` for feature
selectors). llama.cpp gates features such as Metal residency sets on SDK
macros, and they switch on silently (#116).
- After a release, smoke-test end to end from a fresh `/tmp` project that
depends on the published version, using the models in `~/Downloads`.
Run it without `GGML_METAL_NO_RESIDENCY` and check the exit status, not
only the pass count.

## Tests

- A test-side `:telemetry.attach` filters on an identity the test owns
(`%{server: ^server}`, a ref) and detaches in `on_exit`. A linked server can
outlive its test and emit after replying.
5 changes: 4 additions & 1 deletion test/llama_cpp_ex_test.exs
Original file line number Diff line number Diff line change
Expand Up @@ -935,7 +935,10 @@ defmodule LlamaCppExTest do
:telemetry.attach(
handler,
[:llama_cpp_ex, :server, :request, :start],
fn _e, m, _meta, _ -> send(parent, {:started, m}) end,
fn
_e, m, %{server: ^server}, _ -> send(parent, {:started, m})
_e, _m, _meta, _ -> :ok
end,
nil
)

Expand Down