A request the scheduler can never admit hangs forever with zero bytes: our advertised max_model_len is one block larger than the servable KV, and the wedge is silent at both ends

[localai-bot]

Follow-up to the sharper reproduction posted on
#931 on 2026-08-22, which was
hit on dgx:gpu0 and could not be pursued there because the box is off limits
under #1647. #931 itself is
closed and its fix (the SSE keepalive, #971) is on main; this is a different
defect that produces the same client-visible symptom.

The symptom reproduces off-GPU, on CPU, with a 125M model, in two flags

cmake -S . -B build-cpu -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo \
  -DVLLM_CPP_CUDA=OFF -DVLLM_CPP_BUILD_TESTS=OFF
cmake --build build-cpu --target server -j 12

VT_SERVER_VERBOSE=1 build-cpu/examples/vllm-server \
  --model ~/models/opt-125m-bf16-st --port 8144 --served-model-name model \
  --num-blocks 4 --block-size 16 &

curl -sS -m 150 -X POST http://127.0.0.1:8144/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"model","prompt":"The capital of France is","max_tokens":56,"temperature":0}'

Measured, 2026-08-22, x86_64, main at 2f2a70925:

curl: (28) Operation timed out after 150001 milliseconds with 0 bytes received
http_status=000 time_total=150.001220 size=0

Zero bytes, no status line, no error body. The server stays up and /health
answers 200 throughout. This is byte-for-byte the client-side signature the #931
comment recorded on GB10.

What the server is doing, measured rather than inferred

/metrics, scraped during the stall — the capture the #931 comment says was
never taken:

vllm:num_requests_running{...}                       0.0
vllm:num_requests_waiting{...}                       1.0
vllm:num_requests_waiting_by_reason{...,reason="capacity"} 1.0
vllm:kv_cache_usage_perc{...}                        0.0
vllm:num_preemptions_total{...}                      1.0

The KV cache is 0% used and the request is still not admitted.

VT_SERVER_VERBOSE=1 gives the engine heartbeat, and it is the discriminator
between a wedged engine and a slow one:

INFO core-step begin unfinished=1 finished_pending=0
INFO core-step end model_executed=0 n_out=0 elapsed_s=0.000

48,285 such steps in one 150 s leg, none of which executed the model (48,284 of them also carried the admission refusal traced below). That is
the CPU equivalent of the 0 %, 11.43 W GPU reading in the #931 comment: the
work never reaches the device because nothing is ever scheduled.

The mechanism

With a scratch trace in the scheduler's waiting loop (not committed), the steady
state is identical on every one of those steps:

SCHED-CALL   step=49437 waiting=1 running=0
ADMIT-CAND   id=cmpl-0-0 status=6 num_tokens=49 cached=32 new=17 budget=2048 chunked=1
ADMIT-REFUSED id=cmpl-0-0 status=6 num_tokens=49 computed=32 new=17 lookahead=0 running=0 free_blocks=3

status=6 is kPreempted. The sequence is:

1. --num-blocks 4 --block-size 16 makes the server advertise
   max_model_len = 64: INFO auto-fit max_model_len: reduced from 2048 to 64 to fit the KV cache (4 blocks x 16 tokens).
2. The request (6 prompt + 56 requested = 62 tokens) is accepted, because 62
   is inside the advertised 64. refuse_oversized_prompt only bounds the prompt.
3. It prefills and decodes 43 tokens. At 49 tokens the pool is exhausted. It is
   the only request in flight, so the scheduler preempts itself
   (num_preemptions_total 1), frees every block, resets
   num_computed_tokens to 0 and prepends it to the waiting queue —
   Scheduler::preempt_request, mirroring scheduler.py:546-572.
4. On resume it needs blocks for all 49 tokens. The prefix cache returns 32 of
   them, 17 remain, and 4 blocks are required against 3 free. Refused.
   Forever.

Only 3 blocks are ever allocatable out of the 4 configured, because block 0 is
the reserved null block (block_pool.cpp:53-57, :453 "Subtract 1 to account for the null block"). So the servable context is
(num_blocks - 1) * block_size = 48 tokens while ResolveMaxModelLen
(model_loader.cpp:1508) computes available = num_blocks * bytes_per_block
and advertises 64. A request that lands in that last block is admitted and can
then never be re-admitted.

The boundary is exact and reproducible:

--num-blocks	advertised max_model_len	prompt+max_tokens	result
2	32	14	200, 8 tokens
2	32	17	200, 11 tokens
2	32	18	hangs, 0 bytes
2	32	30	hangs, 0 bytes
4	64	62	hangs, 0 bytes after 43 tokens
1	16	22	hangs, 0 bytes, model_executed=0 from step 1 (zero allocatable blocks)

The --num-blocks 1 row is the one that matches the #931 comment most closely:
nothing is ever scheduled, so the device is never touched at all.

Two defects, and they should be separated

1. A permanently unschedulable request is never failed. Whatever the reason,
the client waits forever on an open connection with zero bytes, the server logs
nothing at default verbosity, and the engine burns ~1000 no-op steps a second.
vLLM's own v1 scheduler shares the break-and-leave-waiting shape
(sched/scheduler.py:934-938 at the pinned 555967922), so changing this is a
divergence and needs a decision, not a patch. It is recorded here because it is
what turns every KV-admission problem into a silent hang.

2. The advertised context exceeds the servable context by one block. This one
is ours: ResolveMaxModelLen reconstructs an "available memory" figure from our
own resolved block count, and that reconstruction counts a block the BlockPool
will never hand out. Correcting it to (num_blocks - 1) would make the request
above stop cleanly at max_model_len with finish_reason="length" instead of
wedging.

It is not a small in-flow fix, and it is not being made here:
tests/vllm/entrypoints/test_loaded_engine_dense.cpp:635-660 currently pins the
present arithmetic (params.num_blocks = 1 asserting max_model_len == 32), so
the change tightens an existing assertion and needs its own row, spec and fresh
review. That same file's "an over-long prompt is REFUSED, not left waiting"
case shows the prompt half of this hang class was already closed once; this is
the generation half.

What this does and does not say about the GB10 report

It does not establish that defect 2 is what the #931 comment hit. That report
used a 22-token request against a 27B checkpoint, which is nowhere near any
advertised context. What it establishes is the shape: on that box the request
was never scheduled at all, which is the --num-blocks 1 row above — a pool with
no allocatable capacity — and that is exactly what
#1647 would produce.

The one command that settles it, and it needs no new code: restart that
server with VT_SERVER_VERBOSE=1 (or VT_ENGINE_STEP_LOG=1) and read the
heartbeat during the stall. model_executed=0 with a non-zero unfinished
count, step after step, means the engine is spinning on work it can never
schedule, and the search is over the KV pool rather than over AsyncLLM. If the
heartbeat instead shows model_executed=1 throughout, this issue is the wrong
lead and the fault is elsewhere.

The speculative-vs-non-speculative discriminator in the #931 comment is
consistent with a KV-sizing cause — a speculative config changes the pool
geometry — but nothing here measures it, and it is not asserted.

What was ruled out here

• AsyncLLM / the async batch queue. --num-blocks 4 and the healthy
  control both run max_concurrent_batches=2; the healthy control returns 200
  in 6.7 s over the same code. The stall is in admission, not in the frontend.
• The SSE keepalive (Our OpenAI server fails requests under concurrency where vLLM does not (1/6 at c1, 12/48 at c8 on Qwen3.8-27B), and the failures silently corrupt every throughput ratio taken against it #931's own cause). SsePingIntervalSec() defaults to 0
  since fix(SERVER-CONCURRENCY): our SSE keepalive is a frame vLLM never sends, and its own bench client cannot parse it (#931) #971, and the failing request is non-streaming.
• HTTP worker-pool starvation. One request against a fixed pool of 36.

Evidence

Logs, /metrics scrapes, thread stacks and the per-arm probe output are under
the session scratchpad fix931-hang/; the scratch scheduler trace was reverted
byte-for-byte (md5 6b88fd41533c097eaefabc7f8f2936d4 before and after).

[localai-bot]

Defect 2 is NOT ours: upstream carries the identical arithmetic, so the proposed fix is a divergence

Picked this up as the fresh implementer for the fix. Before writing it I read the
pinned oracle, and the premise does not hold. Recorded here before anyone spends a
row on it.

The body says of the one-block overhang: "This one is ours: ResolveMaxModelLen
reconstructs an 'available memory' figure from our own resolved block count, and
that reconstruction counts a block the BlockPool will never hand out." Read at
the exact parity pin 5559679229bc98, upstream does the same thing at every step:

step	upstream at the pin	ours
reconstruct available from the block-count override	kv_cache_utils.py:2144-2158 — adjusted_memory.append(override * bytes_per_block)	model_loader.cpp:1505-1507 — available = kv_cfg.num_blocks * bytes_per_block
build the pool from that count	kv_cache_coordinator.py:90-95 — BlockPool(num_gpu_blocks=kv_cache_config.num_blocks, ...), no + 1	BlockPool(num_gpu_blocks, ...)
reserve block 0	block_pool.py:188-191 — self.null_block = self.free_block_queue.popleft()	block_pool.cpp:53-57

So upstream's allocatable capacity is num_blocks - 1 while its advertised
max_model_len assumes num_blocks. That is the same one-block overhang, and
ResolveMaxModelLen is a faithful mirror of it rather than a defect of ours.

The self-preemption half matches as well, which the body already suspected:
sched/scheduler.py:607-609 preempts the sole request (if preempted_req == request: break) and leaves it waiting, feeding the :934-938 shape the body
cites.

Both halves are therefore mirrored behaviour. Changing either is a product
decision about diverging from the reference, not an in-flow fix, and it should
not be dispatched as an implementation task. Worth noting the change would also
tighten tests/vllm/entrypoints/test_loaded_engine_dense.cpp:635-660 against
upstream rather than towards it.

The reproduction stands, independently

Confirmed on a second machine and a second build — fresh CPU build
(-DVLLM_CPP_CUDA=OFF, RelWithDebInfo, x86_64), facebook/opt-125m, one binary:

== wedge : num-blocks=4 block-size=16 max_tokens=56 ==
INFO auto-fit max_model_len: reduced from 2048 to 64 to fit the KV cache (4 blocks x 16 tokens).
curl: (28) Operation timed out after 90002 milliseconds with 0 bytes received
http_status=000 time_total=90.002385 size=0
 health=200

Control at --num-blocks 64 --block-size 16, max_tokens 16: HTTP 200 in
13.2 s, finish_reason="length". Same binary, same model, same endpoint.

A hypothesis I checked and falsified, so nobody repeats it

Upstream's serving layer hard clamps max_tokens to max_model_len - input_length — get_max_tokens (entrypoints/serve/utils/api_utils.py:170-206)
returns the min over model headroom, request value and operator ceiling, and
completion/serving.py:163-181 feeds it to to_sampling_params, which at
completion/protocol.py:358 assigns it unconditionally rather than as a
fallback. We never do: serving_completion.cpp:241 and serving_chat.cpp:767
call to_sampling_params() with no argument at all, and InputProcessor fills
max_tokens only when unset (input_processor.cpp:268-271, :304, :340).

That looked like a second, genuinely-ours admission gap. It is not. Measured
at --num-blocks 8 --block-size 16 --max-model-len 64 with max_tokens 5000
(pool deliberately large enough that the null-block limit cannot be reached
first): HTTP 200 in 62.8 s, finish_reason="length", 58 generated tokens. Our
engine stops at max_model_len by another route, so the mechanism differs from
upstream while the client-visible result matches. No defect. The adjacent
operator-ceiling arm of that same upstream function stays open as #544.

Where this is recorded

PR #1771 (draft) carries the correction into
.agents/specs/server-concurrency-failures.md ## Owed, on top of the
record commit that was pushed to row/FIX-SERVER-STALL-1737 without a PR. It
changes no product code, because the fix is the decision above and not a patch.

[localai-bot]

Addendum: the wedged server also survives SIGTERM

Seen only because the reproduction's processes had to be reaped, so recording it before it is lost.

After the client's 90 s timeout the harness sent SIGTERM to the wedged server. It logged server: shutting down on signal 15 and did not exit — measured 53 minutes later: still alive, State: S, 23 threads, ~5.2% average CPU since launch. It cleared only on SIGKILL.

Both healthy legs of the same harness (the --num-blocks 64 control and the --max-model-len 64 / max_tokens 5000 leg) exited on the identical signal and left no process behind.

Stated with its limits: one-variable contrast, n=1 on each arm, and the shutdown path was not traced. What is established is that the signal was received and acknowledged and the process still never terminated — not why.

It matters because it raises the cost of the wedge above a hung client: a supervisor or container that restarts on SIGTERM hangs with it, and only a SIGKILL escalation clears the box. Carried in PR #1771.

[localai-bot]

Measured: the pinned oracle wedges too, in the same configuration, at the same token

The correction above read the wedge out of upstream source. Nobody had run it. I built the
pinned vLLM for CPU and ran the block-starved configuration against it. Outcome 1: vLLM
wedges. Same client signature, same stall point, no refusal, no completion.

No fleet device was touched and no rc lease was taken. Everything below ran on an x86_64
CPU box with no GPU.

The oracle, built at the pin

git clone <local mirror> vllm-src && git checkout 5559679229bc961848b121ccdeaa8fa5d79bec98
python3 -m venv venv
venv/bin/pip install -r requirements/build/cpu.txt -r requirements/cpu.txt \
  --extra-index-url https://download.pytorch.org/whl/cpu
VLLM_TARGET_DEVICE=cpu venv/bin/pip install -e . --no-build-isolation

• source tree 5559679229bc961848b121ccdeaa8fa5d79bec98, git status --porcelain empty
• installed vllm 0.23.1rc1.dev1511+g555967922.cpu, which is the same version string the
  dgx.casa oracle venv reports, and import vllm._C succeeds
• torch 2.13.0+cpu (what requirements/cpu.txt pins at this revision), python 3.12.3,
  gcc 13.3.0, transformers 5.15.1
• wheels.vllm.ai has no wheel for this commit (404 on /<sha>/cpu/vllm/index.html), so a
  source build was the only route
• one build dependency was missing (numa.h); it was extracted from a downloaded
  libnuma-dev deb into a scratch prefix rather than installed on the host

Model: facebook/opt-125m, revision 27dcfa74d334bc871f3234de431e71c6eeba5dd6, --dtype bfloat16.
Prompt "The capital of France is", temperature 0, non-streaming /v1/completions.
The oracle tokenises the prompt to 6 tokens, the same count our server reports.

Every leg: vllm serve facebook/opt-125m --revision 27dcfa74d3... --port <p> --served-model-name model --dtype bfloat16 --num-gpu-blocks-override <N> --block-size 16 --max-model-len -1 [--gpu-memory-utilization 0.3].

--num-gpu-blocks-override is upstream's --num-blocks, and --max-model-len -1 is
upstream's auto-fit, which is the same shape as our auto-fit max_model_len line.

The four legs

leg	blocks	advertised max_model_len	max_tokens	total tokens	result
control	64	1024	16	22	HTTP 200, 35.2 s, finish_reason="length", 16 tokens
control-long	64	1024	56	62	HTTP 200, 122.6 s, finish_reason="length", 56 tokens
boundary-fits	4	64	42	48	HTTP 200, 91.3 s, finish_reason="length", 42 tokens
wedge	4	64	56	62	curl: (28) after 900 s, http_status=000 size=0

The wedge leg, verbatim:

INFO [kv_cache_utils.py:2152] Overriding num_gpu_blocks=44182 with num_gpu_blocks_override=4
INFO [kv_cache_utils.py:2022] Auto-fit max_model_len: reduced from 2048 to 64 to fit in available GPU memory (0.0 GiB available for KV cache)
INFO [kv_cache_utils.py:2214] GPU KV cache size: 64 tokens

http_status=000 time_total=900.002018 size=0
curl: (28) Operation timed out after 900002 milliseconds with 0 bytes received

/metrics, scraped every 30 s during that request:

t=30s   running=1 waiting=0 capacity=0 kv_usage=0.667 generation_tokens_total=14
t=60s   running=1 waiting=0 capacity=0 kv_usage=1.000 generation_tokens_total=28
t=90s   running=0 waiting=1 capacity=1 kv_usage=0.000 generation_tokens_total=43
t=120s  running=0 waiting=1 capacity=1 kv_usage=0.000 generation_tokens_total=43
...
t=871s  running=0 waiting=1 capacity=1 kv_usage=0.000 generation_tokens_total=43
t=901s  (client gone) waiting=0                       generation_tokens_total=43

/health answered 200 at every probe, including the 28 probes after generation froze.

43 generated tokens, then nothing. That is the same number the issue body records for our
server on the same geometry, and the mechanism the counters describe is the same one: the KV
pool fills, the sole request is put back to waiting with reason="capacity", the cache reads
0 % used, and it is never re-admitted.

The confound I hit, so nobody repeats it

My first wedge leg used a 90 s client timeout, copying the reproduction above. It returned
curl: (28) ... 0 bytes and that was not evidence of anything: this CPU decodes about
0.45 tokens/s, so 56 tokens legitimately needs ~120 s. The control-long leg is the control
that separates the two. It is the identical request with the identical max_tokens 56, and
with 64 blocks it returns HTTP 200 in 122.6 s. With 4 blocks the same request is still
producing nothing at 900 s, and its token counter has not moved for 810 s. Slow decode and a
wedge look the same at the client; only the token counter tells them apart.

The boundary is the null block, upstream as well

boundary-fits puts 6 + 42 = 48 tokens through the same 4-block server and gets a clean
200. 48 is exactly (num_blocks - 1) * block_size. One more token needs a fourth block,
which the pool never hands out because block 0 is the null block. So upstream advertises
max_model_len = 64 and serves 48, which is the one-block overhang this issue attributes to
ResolveMaxModelLen. It is upstream's arithmetic, measured, not only read.

What this means for the proposed fix

1. (num_blocks - 1) in ResolveMaxModelLen is a divergence from the reference. The
   comment above established that from source; this establishes it from behaviour. Making
   that change would have our server refuse or truncate where the pinned oracle admits and
   wedges. The right action is to record the shared limitation, and, if we care about it, to
   report it upstream rather than patch it here.
2. The silent hang is also upstream's. No 4xx, no 5xx, no error body, no log line at
   default verbosity, /health green throughout, and the connection held open until the
   client gives up. Our server does exactly this, and so does the oracle.
3. The tests/vllm/entrypoints/test_loaded_engine_dense.cpp:635-660 assertion that pins the
   present arithmetic is therefore pinning the mirrored behaviour and should stay.

One difference, and it is in the metric rather than the behaviour

Our server reports vllm:num_preemptions_total 1 during the stall. The oracle reports
0.0 for the whole 900 s while num_requests_waiting_by_reason{reason="capacity"} is 1.
The counter is fed from the request's event stream (v1/metrics/stats.py:448-450, counting
EngineCoreEventType.PREEMPTED as it arrives with request output), and a request that
produces no further output delivers no events. I did not mutate this to prove it, so treat
the mechanism as a reading and the two numbers as measured. Either way, upstream's preemption
counter cannot report the preemption that wedged the request, and ours can.

What this does not settle

• The GB10 report on Our OpenAI server fails requests under concurrency where vLLM does not (1/6 at c1, 12/48 at c8 on Qwen3.8-27B), and the failures silently corrupt every throughput ratio taken against it #931 is still unattributed. This measures the CPU reproduction only.
• The --num-blocks 1 row (nothing ever scheduled) was not run against the oracle.
• The explicit --max-model-len 64 form was not run; the auto-fit form was, because that is
  the shape our server takes.

Evidence (server logs, per-leg result files with every probe, curl write-outs, response
bodies, environment record) is in the session scratchpad under oracle1737/legs/. The venv
and the source tree were removed after the run because the box was at 94 % disk; the recipe
above rebuilds them.

[localai-bot]

Measured, both arms: the wedge is mirrored, the un-killable process is OURS ALONE

The addendum above recorded that our wedged server acknowledged SIGTERM and never exited, at
n=1, with no oracle arm and no shutdown trace. I ran the same contrast against the pinned
oracle. Outcome 2 occurred. The pinned vLLM, wedged in the same configuration at the same
token, exits on SIGTERM in 1.156 s. Ours does not exit at all.

No fleet device was touched and no rc lease was taken. Everything below ran on an x86_64 CPU
box with no GPU. This posts the measurement only; no fix is made here, because the finding
changes who owns the defect and that is a decision.

The four legs

One variable per arm: --num-blocks / --num-gpu-blocks-override. Same model, same prompt,
same max_tokens 56, same non-streaming /v1/completions, same signal, same recipient
(the server's own pid, leader only, as a supervisor sends it).

arm	leg	blocks	client	generation counter	log ack	from SIGTERM to exit
ours	control	64	HTTP 200 in 11.535 s, finish_reason="length", 56 tokens	reached 56	server: shutting down on signal 15	1.049 s
ours	wedge	4	curl: (28) after 300 s, http_status=000 size=0	frozen at 43 for 286 s (20 consecutive probes)	server: shutting down on signal 15	NEVER — still alive at 300 s
oracle	control	64	HTTP 200 in 123.283 s, finish_reason="length", 56 tokens	reached 56	[shutdown] API server: shutdown triggered	1.095 s
oracle	wedge	4	curl: (28) after 900 s, http_status=000 size=0	frozen at 43 for 780 s (27 consecutive probes)	[shutdown] EngineCore: start mode=abort timeout=0s	1.156 s

Both wedge legs reproduce this issue exactly: 43 generated tokens, then
num_requests_waiting_by_reason{reason="capacity"}=1 with kv_cache_usage_perc=0, /health
200 throughout, no 4xx, no 5xx, no error body. That half is mirrored and this measurement does
not disturb the earlier conclusion.

The confound was controlled the way the earlier comment demands. Each arm's own control is
the identical request at the identical max_tokens 56 with a larger pool, and it returns 200
with all 56 tokens: 11.535 s for ours, 123.283 s for the oracle. Slow decode and a wedge look
the same at the client; the frozen token counter is what separates them, and both wedge legs
sat at 43 while their controls finished.

What our process was doing while it refused to die

leader_before_signal=S(sleeping) threads=59
signal=SIGTERM sent_to=3116617 (leader only) at=2026-08-23T09:29:02+00:00 rc=0
t_after_signal=1s   leader=S(sleeping) threads=23
t_after_signal=291s leader=S(sleeping) threads=23
leader_exited=0 STILL_ALIVE_AFTER=300s
cpu_since_launch: 3116617 604 13.2 Ssl 23 vllm-server
cleared_by=SIGKILL leader_after_kill=GONE

• The signal was received and acknowledged: server: shutting down on signal 15 is written,
  and the log grows from 11 lines to 12 and stops there.
• The listener closes and the process stays. I probed /health every 5 s for the whole
  300 s after the signal: health=000 on all 60 probes, and /metrics returned zero lines.
  So it stops serving and does not exit. A supervisor sees a live PID; a client sees a refused
  connection. That is worse than either failure alone.
• 36 of the 59 threads exit on the signal and 23 remain, every one of them sleeping in
  futex_wait_queue (per-thread /proc/<pid>/task/*/wchan). 23 threads is the same count the
  earlier 53-minute observation recorded, so that observation replicates on a different tree
  and a different binary.
• SIGKILL cleared it, as before.

The oracle's leg is the mirror image: leader gone in 1.156 s and all three descendants
(APIServer, EngineCore, Worker) GONE without being signalled individually.

Why upstream survives its own wedge, read from the pin after measuring it

vLLM has an explicit shutdown contract that we do not have:

• VllmConfig.shutdown_timeout defaults to 0 (vllm/config/vllm.py:402).
• launcher.py:112-124 reads it and picks mode = "abort" if timeout == 0 else "drain", then
  awaits engine_client.shutdown(timeout=...) before setting server.should_exit.
• The measured log line [shutdown] EngineCore: start mode=abort timeout=0s is that default
  firing. The permanently-unschedulable request is aborted, not awaited, so nothing holds
  the process open.

Ours has no such step. ConsoleShutdown prints the line and calls its callback, which is
server.stop() (server_main.cpp:1753-1754). In cpp-httplib Server::stop() only shuts and
closes the listening socket (third_party/httplib/httplib.h:11460-11468) — which is exactly
the health=000 above — and listen_internal then reaches task_queue->shutdown()
(:12093), which joins the worker threads. The worker holding the wedged completion has no
reason to return, because nothing aborts the request.

That last paragraph is a reading, not a proof, and I could not close it here.
/proc/sys/kernel/yama/ptrace_scope is 1 on this box, so gdb -p was refused
(ptrace: Inappropriate ioctl for device) and no stack was taken. Worth flagging for whoever
repeats this: gdb exits 0 when it fails to attach, so my harness recorded gdb_exit=0
beside an empty trace. The /proc thread census above is what I actually have.

What this means for ownership

1. The wedge stays mirrored. Nothing here reopens the (num_blocks - 1) question. Both
   servers admit the request, both freeze at 43 tokens, both hold the connection open with
   zero bytes, both keep /health green. The earlier conclusion stands.
2. The un-killable process is ours alone, and it is not the wedge. It is a missing
   shutdown path that the wedge merely exposes. The oracle's wedged server, in the same state
   with the same waiting request, exits in about a second.
3. Fixing it is not a divergence — it is adopting a default the reference already ships.
   Upstream aborts in-flight requests on SIGTERM by default (shutdown_timeout = 0). We
   neither abort nor time out. Any request our engine cannot complete — a wedge, a stuck
   backend, a long generation — makes the process unkillable by SIGTERM, so the blast radius
   is wider than this issue's geometry.
4. It is operationally the serious half: a systemd unit, a Kubernetes pod, or a container
   restart policy that stops on SIGTERM hangs with it, and only SIGKILL escalation clears the
   box.

I have not implemented anything. This needs a row and a spec of its own; PR #1771 owns the
## Owed entry in .agents/specs/server-concurrency-failures.md and should carry it rather
than a second record being opened beside it.

Reproduction and environment

Oracle rebuilt from source at the exact pin, because wheels.vllm.ai still has no wheel for it:

git clone --filter=blob:none https://github.com/vllm-project/vllm vllm-src
git -C vllm-src checkout 5559679229bc961848b121ccdeaa8fa5d79bec98   # status --porcelain empty
python3 -m venv venv
venv/bin/pip install -r vllm-src/requirements/build/cpu.txt -r vllm-src/requirements/cpu.txt \
  --extra-index-url https://download.pytorch.org/whl/cpu
CPATH=<scratch>/numa/root/usr/include LIBRARY_PATH=<scratch>/numa/root/usr/lib/x86_64-linux-gnu \
VLLM_TARGET_DEVICE=cpu venv/bin/pip install -e . --no-build-isolation

numa.h again came from a libnuma-dev deb extracted into a scratch prefix; nothing was
installed on the host. Installed vllm 0.23.1rc1.dev1511+g555967922.cpu, import vllm._C
succeeds, torch 2.13.0+cpu, transformers 5.15.1, python 3.12.3, gcc 13.3.0 — the same
strings the previous oracle run recorded.

Ours: vllm-server md5 be55c5d730aa7081d0a0d812f5450cc4, built from 21abaf169 with
-DVLLM_CPP_CUDA=OFF -DVLLM_CPP_BUILD_TESTS=OFF, RelWithDebInfo, default verbosity.
Model facebook/opt-125m revision 27dcfa74d334bc871f3234de431e71c6eeba5dd6 on both arms
(~/models/opt-125m-bf16-st for ours), prompt "The capital of France is", temperature 0.

Contention, stated because it is not an idle host: other sessions held this box throughout,
load average 19–43. It cannot manufacture the result — the claim on our arm is that a process
did not exit, which load cannot cause, and each arm's control exited in about a second under
the same load. It does inflate the absolute latencies above, so read them as shapes rather than
as timings.

Evidence — per-leg result files with every probe, server logs, curl write-outs, response
bodies, the after-signal /health census, the /proc thread census and the environment
record — is in the session scratchpad under sigterm1737/legs/. The venv and source tree are
removed after the run; the recipe above rebuilds them.