From 7fdcd18fdaa0bfeee254230d54bbb9c1821b45b1 Mon Sep 17 00:00:00 2001 From: Niko Maroulis Date: Sun, 1 Nov 2026 14:05:23 -0500 Subject: [PATCH] Add solution docs for #116 and the telemetry test flake, CLAUDE.md rules Two solution docs: the Metal exit abort (#116) and the server smoke test that caught the previous test's :done telemetry. CLAUDE.md gains the matching prevention rules (LLAMA_NIF and the live:: registry, SDK diff on runner bumps, exit-status e2e without GGML_METAL_NO_RESIDENCY, test telemetry handlers pinned to their server). The hybrid-cache test's :start handler was the last unfiltered test telemetry handler; it now matches its own server. --- ...bort-rsets-assert-nif-teardown-20261101.md | 154 ++++++++++++++++++ ...y-done-event-server-smoke-test-20261031.md | 113 +++++++++++++ CLAUDE.md | 29 +++- test/llama_cpp_ex_test.exs | 5 +- 4 files changed, 299 insertions(+), 2 deletions(-) create mode 100644 .claude/solutions/phoenix-issues/metal-exit-abort-rsets-assert-nif-teardown-20261101.md create mode 100644 .claude/solutions/testing-issues/stale-telemetry-done-event-server-smoke-test-20261031.md diff --git a/.claude/solutions/phoenix-issues/metal-exit-abort-rsets-assert-nif-teardown-20261101.md b/.claude/solutions/phoenix-issues/metal-exit-abort-rsets-assert-nif-teardown-20261101.md new file mode 100644 index 0000000..1397184 --- /dev/null +++ b/.claude/solutions/phoenix-issues/metal-exit-abort-rsets-assert-nif-teardown-20261101.md @@ -0,0 +1,154 @@ +--- +module: "LlamaCppEx.NIF (c_src/llama_cpp_ex/llama_nif.cpp)" +date: "2026-11-01" +problem_type: runtime_error +component: nif_teardown +symptoms: + - "ggml-metal-device.m:1041: GGML_ASSERT([rsets->data count] == 0) failed, then SIGABRT (exit 134) after the program finished" + - "a model held in persistent_term, ETS or a process alive at System.halt/1 aborts the VM instead of exiting with its own status" + - "0.8.62 exits 0 on the same script, 0.8.63 and 0.8.64 abort (precompiled Metal NIF)" + - "a green mix test run on Metal exits 134 unless GGML_METAL_NO_RESIDENCY=1 is set" +root_cause: "The BEAM never runs NIF resource destructors at halt, so live models/contexts still own Metal buffers when exit(3) runs C++ static destructors; llama.cpp's Metal device is a function-local static whose destructor asserts no residency set is left. Residency sets are only compiled with a macOS 15+ SDK, and the move to macos-15 runners (v0.8.63) put them into the precompiled NIF." +severity: high +tags: [metal, nif, atexit, static-destruction, residency-sets, vm-halt, macos-sdk, fine] +related_solutions: + - ".claude/solutions/deployment-issues/macos-14-runner-brownout-precompile-20261031.md" +--- + +# VM aborts at exit on Metal while a model is still referenced (#116) + +## Symptoms + +``` +ggml/src/ggml-metal/ggml-metal-device.m:1041: GGML_ASSERT([rsets->data count] == 0) failed +3 ggml_metal_device_free +4 std::vector>::~vector() +5 __cxa_finalize_ranges +6 exit +7 beam.smp erts_exit_vv +``` + +Repro: load a model, keep it referenced from any live process, let the +script end. Exit 134 on 0.8.63/0.8.64, 0 on 0.8.62. If the script process is +the only holder, nothing aborts: it dies before halt and the resource is GC'd. + +## Investigation + +1. **"llama.cpp changed"**: no. v0.8.62→v0.8.64 all pin `418f905a4`. +2. **Compare the artifacts**, which settled it. `otool -l` on the cached + NIFs in `~/Library/Caches/elixir_make`: + - 0.8.62: `sdk 14.5`, 0 residency-set selectors; + - 0.8.63/0.8.64: `sdk 15.5`, `newResidencySetWithDescriptor:error:`, + `addAllocation:`, `endResidency` and `requestResidency` present. + + `ggml-metal-device.m:20` defines `GGML_METAL_HAS_RESIDENCY_SETS` only when + `__MAC_OS_X_VERSION_MAX_ALLOWED >= 150000`. Without it, `rsets->data` stays + empty and the assert can never fire. That is why 0.8.62 worked, and why + local builds (newer SDK) always needed `GGML_METAL_NO_RESIDENCY=1`. +3. **Ruled out `_exit` from an atexit hook**: it would lose the VM's exit + status, which no atexit API exposes. +4. **Ruled out freeing without a guard**: the resources have no locks, so a + dirty NIF mid-decode during halt would be a use-after-free. +5. **First fix had an ordering hole** (found by review, confirmed 3/3 SIGBUS, + exit -10). The hook is registered once per allocation kind, so ggml statics + first built *later* are destroyed *before* it. Case: a first model loaded + with `devices: ["CPU"]`, then a Metal model and context. The Metal + buffer-type statics (`ggml_backend_metal_buffer_type_shared/private`, whose + `bufts`/`ctxs` vectors hold the names) are built after the hook, and + `~llama_context` logs `ggml_backend_buft_name(buft)` → freed memory. + +## Root Cause + +Two facts combine: + +- `erlang:halt` → `exit(3)` runs `atexit` handlers and C++ static destructors + in reverse order of registration. The BEAM does not first destroy NIF + resources still referenced from live processes, ETS or `persistent_term`. +- `ggml_metal_device_get` keeps devices in a function-local + `static std::vector`. Its deleter calls + `ggml_metal_rsets_free`, which asserts that every buffer's residency set was + removed, i.e. that every model/context was freed first. + +Changing the build SDK (runner image) silently switched that code path on for +every precompiled-NIF user. + +## Solution + +`c_src/llama_cpp_ex/llama_nif.{h,cpp}`, "Teardown at VM exit (#116)": + +```cpp +// Resources register themselves; destructors free under the same lock. +explicit LlamaModel(llama_model* m) : ... { live::add(this); } +~LlamaModel() { + std::lock_guard lock(live::mutex()); + live::remove(this); + release(); // frees + nulls; also called by the hook +} + +// Every NIF: FINE_NIF plus an in-flight guard (seq_cst Dekker handshake). +#define LLAMA_NIF(name, flags) ... NifCall call; if (!call.entered) raise vm_exiting; ... + +void free_live_resources_at_exit() { + if (g_exiting.exchange(true)) return; + // wait up to 5 s for g_in_flight == 0, else free nothing (no UAF) + // under the lock: speculatives -> contexts with ctx_other -> contexts -> models +} + +void arm_exit_hook(std::once_flag& once) { // Apple only + std::call_once(once, [] { + // build Metal buffer-type statics BEFORE registering (CPU-first case) + for each device whose reg name is "MTL": + ggml_backend_dev_buffer_type(dev); ggml_backend_dev_host_buffer_type(dev); + std::atexit(free_live_resources_at_exit); + }); +} +// armed after first model_load, context_create, speculative_init +``` + +Also: +- Generation loops, `cancel_abort_cb` and a model-load `progress_callback` + return early on `g_exiting`, so a halt does not wait out a long call. +- Setting `progress_callback` replaces llama.cpp's default one, so it re-prints + the progress dots via `log_callback`. + +### Files Changed + +- `c_src/llama_cpp_ex/llama_nif.h` — `live::` registry, `release()` on + LlamaModel/LlamaContext/LlamaSpeculative, `has_ctx_other` +- `c_src/llama_cpp_ex/llama_nif.cpp` — registry, `NifCall`, `LLAMA_NIF` (all 67 + NIFs), exit hook, arming, `g_exiting` checks +- `test/vm_exit_test.exs` — child VM halts with live resources: model + context + (`halt(7)` → 7), CPU-only then GPU model (`halt(5)` → 5), Server + mid-generation (→ 0). It fails without the fix (134 / SIGBUS). +- `GGML_METAL_NO_RESIDENCY=1` removed from test_helper, test moduledocs, + docs/release-guide.md, docs/performance.md, scripts/decision_compare.exs + +Shipped in v0.8.65 (#118, #119). Verified with residency sets on: suites exit 0 +(CI set ×8, release-guide set with MTP, default), and the e2e on the published +package passes 26/26 with four models referenced at halt. + +## Prevention + +- When the precompile runner image or SDK changes, diff the artifacts + (`otool -l` → `sdk`/`minos`, `strings` for feature selectors) against the + previous release. Feature gates keyed on SDK macros switch silently. +- Never keep a run-instruction env-var workaround for an abort without an + issue tracking it. `GGML_METAL_NO_RESIDENCY=1` hid this from every local + run, CI (Linux) and the v0.8.64 e2e smoke. +- Smoke the published package without workarounds and check the exit + status, not just "N passed". +- Native code freed at process exit must be registered *after* every static + it touches is constructed. Force lazily built statics (Metal buffer types) + before `std::atexit`. Don't re-register per allocation: stateless + `generate` makes a context per call, so the atexit list would grow without + bound. +- New NIFs must use `LLAMA_NIF`, not `FINE_NIF`, or the exit hook can free + under them. + +## Related + +- `.claude/solutions/deployment-issues/macos-14-runner-brownout-precompile-20261031.md` + — the runner move that raised the SDK to 15.5 +- Issue https://github.com/nyo16/llama_cpp_ex/issues/116, PR #118 +- Iron Law #28 (iron-law-judge): `LLAMA_NIF` only; GPU-owning resources join + `live::`; loops stop on `g_exiting` diff --git a/.claude/solutions/testing-issues/stale-telemetry-done-event-server-smoke-test-20261031.md b/.claude/solutions/testing-issues/stale-telemetry-done-event-server-smoke-test-20261031.md new file mode 100644 index 0000000..c1f94e3 --- /dev/null +++ b/.claude/solutions/testing-issues/stale-telemetry-done-event-server-smoke-test-20261031.md @@ -0,0 +1,113 @@ +--- +module: "LlamaCppEx.ServerSmokeTest" +date: "2026-10-31" +problem_type: test_failure +component: testing +symptoms: + - "test halting a stream early cancels generation and frees the slot: ** (MatchError) no match of right hand side value: {%{prompt_tokens: 39, generated_tokens: 4, ...}, %{stop_reason: :max_tokens, ...}}" + - "fails only on CI (CPU, seed 489492) on a docs-only PR; passes locally with the same seed" +root_cause: "A :telemetry handler is global, and the test's collector forwarded the first [:llama_cpp_ex, :server, :request, :done] event from any server. The previous test's server was still alive: ExUnit does not wait for a test's linked processes to stop, and Server.finish_slot/3 replies to the caller before emitting :done, so that server's stale event arrived after the next test attached its handler." +severity: medium +tags: [telemetry, flaky-test, exunit, global-handler, linked-process, server, race] +--- + +# Flaky server smoke test: stale `:done` telemetry from the previous test's server + +## Symptoms + +The CI Inference smoke on #115, a docs-only PR, failed: + +``` +1) test halting a stream early cancels generation and frees the slot (LlamaCppEx.ServerSmokeTest) + ** (MatchError) no match of right hand side value: + {%{prompt_tokens: 39, generated_tokens: 4, ...}, %{stop_reason: :max_tokens, server: #PID<0.1528.0>, ...}} + code: {%{generated_tokens: n}, %{stop_reason: :cancelled}} = next_telemetry() +``` + +The test streams "Write an endless story:" (about 5 prompt tokens, +`max_tokens: 400`), so a 39-token, `max_tokens` (4) event cannot be its own. + +## Investigation + +1. **"Cancel lost the race with generation"**: no. Stop reason `max_tokens` + after 4 tokens doesn't match this request's `max_tokens: 400`. +2. **Find the 39-token request**: the preceding test, "chat_completion/3 + propagates a server-side generation error". Its last call is + `chat_completion(server, messages, max_tokens: 4)`, a chat-templated + prompt of 39 tokens on SmolLM2. +3. **CI log order**: + - 54.311: handler attached by the new test's process, `#PID<0.1529.0>`; + - 54.314: `[debug] slot 0 done (max_tokens): 39 prompt tokens`; + - 54.514: the real `cancelled` event. +4. Locally (Metal) the same seed passed with and without the fix. On faster + hardware the old server usually emits before the next test attaches, so it + only reproduces on timing. + +## Root Cause + +- `finish_slot/3` calls `GenServer.reply` first, then `Logger.debug` and + `:telemetry.execute(... :done ...)`. The caller can return, its test can + finish, and the next test can start in between. +- The server is `start_link`ed to the test process and traps exits. + ExUnit does not wait for linked processes to terminate after a test, so the + old server keeps running briefly. +- `:telemetry.attach/4` handlers are global. The collector sent every event to + the test process with no filter: + +```elixir +fn _event, measurements, metadata, _ -> + send(parent, {:telemetry, measurements, metadata}) +end +``` + +## Solution + +Pin the server under test. Every server telemetry event carries +`server: self()` in its metadata. + +```elixir +defp attach_collector(server, event) do + parent = self() + handler_id = {__MODULE__, self(), event} + + :telemetry.attach(handler_id, event, + fn + _event, measurements, %{server: ^server} = metadata, _ -> + send(parent, {:telemetry, measurements, metadata}) + _event, _measurements, _metadata, _ -> + :ok + end, nil) + + on_exit(fn -> :telemetry.detach(handler_id) end) +end +``` + +The assertions are unchanged; only foreign events are dropped. + +### Files Changed + +- `test/server_smoke_test.exs` — `attach_collector/2` filters on `server`; 4 + callers updated +- `test/llama_cpp_ex_test.exs` — the telemetry test's three handlers pin + `server`, detach in `on_exit` (previously only on success), and assert + `metadata.server == server` + +Shipped in #117 (v0.8.64). CI green afterwards. + +## Prevention + +- Every test-side `:telemetry.attach` must filter on an identity the test owns + (`server`, `ref`, `seq_id`) and detach in `on_exit`, not at the end of the + test body. +- Don't assume a linked process is gone when the next test starts. Stop it + explicitly or filter its output. +- A flake on an unrelated PR: read the event payload before re-running. Here + the prompt-token count identified the sender. + +## Related + +- `.claude/solutions/phoenix-issues/metal-exit-abort-rsets-assert-nif-teardown-20261101.md` + — same "process outlives its test" family (a live model at halt) +- Iron Law #29 (iron-law-judge): test telemetry handlers pin an owned identity + and detach in `on_exit`. The last unfiltered handler (hybrid-cache test in + `test/llama_cpp_ex_test.exs`) was fixed when this law was added. diff --git a/CLAUDE.md b/CLAUDE.md index 6b136c7..283b4ce 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,7 +1,8 @@ # llama_cpp_ex — project rules Documented solutions live in `.claude/solutions/`; search there before -investigating a tokenizer, template, logging, or release problem. +investigating a tokenizer, template, logging, NIF teardown, flaky test or +release problem. ## Prompts and tokenization @@ -29,6 +30,20 @@ investigating a tokenizer, template, logging, or release problem. load. Set `LlamaCppEx.set_log_level/1` (or `config :llama_cpp_ex, log_level:`) *before* `LlamaCppEx.init/0`, or backend-init lines escape. +## NIF and teardown + +- Register every NIF with `LLAMA_NIF`, never `FINE_NIF`: the wrapper's + in-flight guard is what lets the exit hook free resources safely (#116). +- A resource that owns backend buffers (`llama_model`, `llama_context`, + `common_speculative`, anything allocating on a ggml device) joins the + `live::` registry: `live::add` in constructors, a destructor that locks + `live::mutex()`, removes itself and calls `release()`, and a release step in + `free_live_resources_at_exit` in dependency order. Long loops and callbacks + inside NIFs stop on `g_exiting`. +- Exit-path regressions show only in the VM's exit status. Run + `test/vm_exit_test.exs` (`--include smoke`) after touching resource + lifetimes, and never paper over an abort with `GGML_METAL_NO_RESIDENCY=1`. + ## Releases and CI - Any macOS runner image bump in `.github/workflows/precompile.yml` keeps @@ -42,5 +57,17 @@ investigating a tokenizer, template, logging, or release problem. latest release and changelog before working around it. - CI runs `--include tools` against pinned Qwen3.5-0.8B-UD-Q4_K_XL; keep the `:tools` tests passing on the CPU backend. +- When the macOS runner image or SDK changes, compare the new NIF with the + previous release (`otool -l` for `sdk`/`minos`, `strings` for feature + selectors). llama.cpp gates features such as Metal residency sets on SDK + macros, and they switch on silently (#116). - After a release, smoke-test end to end from a fresh `/tmp` project that depends on the published version, using the models in `~/Downloads`. + Run it without `GGML_METAL_NO_RESIDENCY` and check the exit status, not + only the pass count. + +## Tests + +- A test-side `:telemetry.attach` filters on an identity the test owns + (`%{server: ^server}`, a ref) and detaches in `on_exit`. A linked server can + outlive its test and emit after replying. diff --git a/test/llama_cpp_ex_test.exs b/test/llama_cpp_ex_test.exs index 6a219f6..8a63926 100644 --- a/test/llama_cpp_ex_test.exs +++ b/test/llama_cpp_ex_test.exs @@ -935,7 +935,10 @@ defmodule LlamaCppExTest do :telemetry.attach( handler, [:llama_cpp_ex, :server, :request, :start], - fn _e, m, _meta, _ -> send(parent, {:started, m}) end, + fn + _e, m, %{server: ^server}, _ -> send(parent, {:started, m}) + _e, _m, _meta, _ -> :ok + end, nil )