LLM inference failure hangs the caller, and error-triggered unload can SIGABRT the whole server

[duncanmcqueen]

Bug: LLM inference failure hangs the caller, then the error-triggered unload can SIGABRT the whole server

Summary

When an LLM generation fails inside queue_worker_llm (e.g., CL_OUT_OF_RESOURCES from the Intel GPU plugin, or a continuous-batching GenerationStatus error), two bugs compound:

1. The HTTP caller hangs forever. The worker breaks out of its loop before ever completing packet.result_future, so WorkerRegistry.generate() awaits a future that is never resolved.
2. The automatic unload-on-error can crash the entire server process. register_unload() runs pipeline destructors (del self.model → gc.collect()). After a GPU error the OpenCL context is frequently corrupted; the C++ destructor then throws across a thread boundary and the process dies with SIGABRT (terminate called after throwing an instance of 'ov::Exception'). One failed request takes down the server and every other loaded model with it.

Environment

• OpenArc @ 9b0a198 (current master)
• openvino / openvino-genai 2026.3.0
• 2 × Intel Arc dGPU (B580 + A770), OpenCL backend
• Linux 7.1.5 (CachyOS), Python 3.12

Reproduction

1. Load an LLM (ovgenai engine) and force an inference-time GPU error — easiest is exhausting device memory during generate() (long prompt + large max_tokens so KV cache growth exceeds free VRAM), which raises:

   RuntimeError: Exception from src/inference/src/cpp/infer_request.cpp:224:
   Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_stream.cpp:385:
   [GPU] clFinish, error code: -5 CL_OUT_OF_RESOURCES
   

   A second trigger: run a model through the continuous-batching adapter and exceed the scheduler cache, producing RuntimeError: ... Got unfinished GenerationStatus from pipeline_continuous_batching_adapter.hpp:277.

2. Observe:

   • The /v1/chat/completions request never returns (client hangs until its own timeout).
   • Log shows Inference failed, triggering model unload...
   • Frequently, the whole server then aborts: terminate called after throwing an instance of 'ov::Exception' → SIGABRT, with no Python traceback.

Root cause

In src/server/worker_registry.py, QueueWorker.queue_worker_llm:

completed_packet = await InferWorker.infer_llm(packet, llm_model)

# Check if inference failed and trigger model unload
if completed_packet.response and completed_packet.response.startswith("Error:"):
    logger.error(f"[LLM Worker: {model_name}] Inference failed, triggering model unload...")
    asyncio.create_task(registry.register_unload(model_name))
    break    # <- exits before packet.result_future.set_result() is ever called

• The break skips packet.result_future.set_result(completed_packet), so non-streaming callers block on await result_future indefinitely.
• register_unload() → OVGenAI_LLM.unload_model() → del self.model; gc.collect(). Destroying an LLMPipeline whose device context just failed calls back into OpenCL from a destructor; the resulting ov::Exception is uncatchable at the Python level and terminates the process. The same unload-after-error pattern exists in the VLM/whisper/ASR/TTS workers, which likely share the crash risk.

Expected behavior: a failed generation should return an error to the caller (HTTP 500) and leave the worker/model/server alive so subsequent requests can be attempted.

Suggested fix

Report the error through the future instead of dropping it, and don't tear down the model on inference failure:

--- a/src/server/worker_registry.py
+++ b/src/server/worker_registry.py
@@ -372,11 +372,16 @@ class QueueWorker:
 
             completed_packet = await InferWorker.infer_llm(packet, llm_model)
 
-            # Check if inference failed and trigger model unload
+            # On failure: report the error to the caller and keep the worker and
+            # model alive. Unloading after a GPU error can crash the entire server
+            # (pipeline destructor touches a corrupted OpenCL context -> SIGABRT),
+            # and leaving result_future unset hangs the HTTP caller forever.
             if completed_packet.response and completed_packet.response.startswith("Error:"):
-                logger.error(f"[LLM Worker: {model_name}] Inference failed, triggering model unload...")
-                asyncio.create_task(registry.register_unload(model_name))
-                break
+                logger.error(f"[LLM Worker: {model_name}] Inference failed: {completed_packet.response[:300]}")
+                if packet.result_future is not None and not packet.result_future.done():
+                    packet.result_future.set_exception(RuntimeError(completed_packet.response))
+                model_queue.task_done()
+                continue
 
             if completed_packet.metrics:
                 logger.info(f"[LLM Worker: {model_name}] Metrics: {completed_packet.metrics}")

With this patch, inference errors surface as HTTP 500s, the worker keeps serving, and we have had zero server crashes since. (If unload-on-error is considered desirable for resource reclamation, it should at least be deferred/guarded — e.g., only on errors known not to corrupt the context, or done via process isolation — since a destructor call after CL_OUT_OF_RESOURCES is what aborts the process.)

Additional notes

• Host-RAM usage during LLMPipeline load is very high with the default ENABLE_WEIGHTLESS: False (each int4 14B model staged ~9–17 GB in system RAM during load, retained while resident). Parallel loads of two models OOM-killed the server on a 32 GB machine. Setting "ENABLE_WEIGHTLESS": true in runtime_config reduced per-load host RAM to ~0.2 GB in our testing. Consider documenting this prominently or defaulting it to true for GPU loads.
• The continuous-batching adapter (scheduler_config) intermittently fails generations on Arc dGPUs with Got unfinished GenerationStatus even when the prompt+generation is far below the configured cache size; we reverted affected models to the stateful pipeline. Happy to open a separate issue with a minimal repro if useful.

[duncanmcqueen]

Follow-up with a related stability finding that may be the same thing other Arc users hit:

Recurring GPU hangs (fence timeouts) on an Arc A770 during ovgenai inference. Kernel log showed 291 Fence expiration time out i915-0000:06:00.0:openarc[...] events over 3 days, always on the A770 (DG2), always from the openarc process; the B580 (Battlemage) in the same machine has never appeared once. Each hang forces a GPU reset which kills the server's device context and takes the process down (usually via the SIGABRT path described above when cleanup touches the dead context).

The hangs are intermittent — not reproducible with single-model soak tests of the affected models (an int4_asym 14B and an int8 4B, both ovgenai/LLMPipeline).

Workaround that is holding so far: setting "GPU_ENABLE_SDPA_OPTIMIZATION": false in the model's runtime_config (i.e., disabling the fused SDPA kernel on the GPU plugin). 4 consecutive full multi-model inference cycles since the change with zero fence timeouts, at some throughput cost. Given that SDPA fused-attention kernels are a known hang source on DG2, this might be worth documenting as a known-issue knob for Arc A750/A770 users — or exposing it as a first-class openarc config option with a hint in logs when fence-reset-style failures are observed.

Environment: openvino/openvino-genai 2026.3.0, oneDNN 3.13, compute runtime 26.27.39122, kernel 7.1.5 (CachyOS), A770 16GB + B580 12GB.