dasllama-server: GC frame-attribution exception leaves JIT server alive but unresponsive

[borisbat]

Summary

A long-running CPU-only dasllama-server hit a heap-collection exception during automatic string-fragmentation GC. The process stayed alive but stopped serving requests, including its health endpoint. Watchdog reported unhealthy but did not relaunch the still-running child. A game client consequently remained blocked waiting for ASR/other inference responses.

The GC/frame-attribution failure is the primary issue. The alive-but-unresponsive server and watchdog behavior are secondary recovery concerns.

Environment

• Windows, Release daslang.exe -jit, launched by utils/watchdog/watchdog.py.
• Worktree based on 9836bcb912b8379c1c3ea190c160d38e26902c9c (not a pristine checkout: local Qwen Instruct thinking-handling changes and an in-development game client).
• utils/dasllama-server/main.das; separate native daslang game using dasOPENAI against loopback.
• CPU only; threads=8, team_dispatch="hybrid", streams=1, ctx=8192, chunk=64, page_rows=64, quant="q8", kv_dtype="f16".
• Active chat model: Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf; Kitten Nano TTS, Parakeet TDT 0.6B v3 ASR. Additional model slots were configured.
• Workload: repeated non-streaming chat completions, WAV TTS and multipart WAV transcription over several hours. Child uptime at manual termination was about 7.5 hours.

Observed log sequence

Times below are 2026-09-04, UTC-07:00. Paths are shortened and the undecodable frame-name bytes are replaced explicitly; no player conversations are included.

22:38:59  [req 509] tts: completed 200 (4.9583335 s of audio, pending=0)
22:39:06  dasllama-server: GC begin reason=string_fragmentation validate=true
          das_heap=0x8f185c50/0x8f330bc8
          string_heap=0x24588a0/0x1146c464
22:39:06  EXCEPTION: heap collection can't attribute locals of '[undecodable frame name]' - the frame was entered through a compiled (AOT/JIT) frame, a position-less or embedder-crafted line, or a frameless (fastcall) carrier
          at utils/dasllama-server/main.das:777:15
22:39:11  watchdog: health ok=false, was=true, stage=ready

The call at the reported local source line was unsafe(heap_collect(true, true)) in the automatic string-fragmentation branch. There was no matching GC end for this collection. The watchdog continued reporting process memory/CPU after the exception. An independent HTTP request to /v1/stats timed out as well.

Recovery and important distinction

• At 22:41:39 we manually terminated the identified server child (PID 7532). The later exit code 4294967295 is from that manual termination, not evidence of a spontaneous process crash.
• Watchdog started a replacement child at 22:41:55; it became healthy at 22:42:47.
• Startup string-fragmentation GC on the replacement completed successfully (195 ms).
• A subsequent real chat completion and TTS succeeded; the game returned to idle. No game save reset was necessary.
• We have not fixed the GC failure or established a deterministic minimal reproducer. The unreadable function name alone is not proof of heap corruption.

Investigation / reproduction starting point

Run the server under -jit with CPU chat + Kitten TTS + Parakeet ASR, exercise mixed requests and allow automatic fragmentation GC. Capture the stack/frame attribution metadata at the failing collection. Check whether this collection is entered through a frame that cannot be attributed safely, and whether exception handling leaves the server tick/event loop in a non-serving state.

Expected: supported GC entry should complete correctly; an unrecoverable failure should at least terminate or enter a recoverable state rather than retain the port while silently ceasing service.

Local evidence for the separate investigation session:

• logs/latchpoint-server/watchdog.log (original exception bytes and preceding request/GC history).
• logs/latchpoint-server/logs/crashes/latchpoint-20260904-224154-pid7532/ (watchdog recovery bundle generated after the manual kill; no matching original native crash dump was established).

No full logs, user speech, model files or dumps are uploaded with this issue.