Chat-template rendering appears quadratic in message count: a 1,999-message / 765k-token prompt maps ~66 GiB of host memory before any model work

Summary

Rendering a chat prompt with many messages appears to be quadratic in message count on this engine: a 1,999-message, ~765k-token request causes a single ~66 GiB anonymous mapping on the host before any model work begins, touches ~26 GB of it, and drives a 121 GB GB10 box to 0.09 GB available memory with 21.6 GB swapped out. The same prose flattened into a single message costs ~0.5 GB.

The allocation is in the chat-template renderer (hf_text.HfText.renderFn → template eval.exec/eval.eval → mem.concat → fastMemset), not in the model or KV cache. I believe the mechanism is the template's repeated ns.out = ns.out + ... accumulation combined with join() in zig/src/core/template/eval.zig only appending in place when the left operand is the single shared c.acc buffer, so any interleaved concatenation forces a full copy of the accumulated prompt that the arena never frees. Happy to be corrected on the exact accounting — the measurements below stand regardless.

Environment

ASUS Ascent GX10 (GB10), 121 GB unified memory, 32 GB swap, Ubuntu 24.04 arm64
kernel 7.0.0-1019-nvidia, driver 580.178.04, BIOS GX10DGX.0106.2026.0707.0731
TensorFold native engine in a container, 4-rank TP deployment (one rank per box), DeepSeek-V4.1-Flash checkpoint
serve flags of note: --context 1048576 --parallel 8 --max-tokens 32768 --prompt-cache-gib 4
upstream ashhart/TensorFold @ 3f0a715 was read for the code path; the running image carries local patches, so line numbers are "as upstream"

Reproduction (no private data)

2001 messages of filler, ~758k tokens, plus the caller's tool schemas (157 in the original case):

import json, random
vocab = "alpha bravo charlie delta echo foxtrot golf hotel india juliet kilo lima mike november oscar papa quebec romeo sierra tango uniform victor whiskey xray yankee zulu".split()
random.seed(424242)
msgs = [{"role": "system", "content": "You are a test agent. Reply with the single word OK."}]
for i in range(2000):
    words = [random.choice(vocab) for _ in range(234)]
    msgs.append({"role": "user" if i % 2 == 0 else "assistant",
                 "content": "turn %04d: " % i + " ".join(words) + "."})
body = {"model": "<model id>", "messages": msgs, "tools": tools, "max_tokens": 8}
json.dump(body, open("twin.json", "w"))
# curl -H 'Content-Type: application/json' --data @twin.json http://HOST:PORT/v1/chat/completions

Measurements

Same host, sequential runs, memory sampled from /proc and the container cgroup.

request	prompt tokens	wall	cgroup delta	MemAvailable min	swap peak
synthetic, 1 message, 2 × 503k concurrent	503,289 / 503,019	475 s	+2.23 G	20.3 G	unchanged
real 1,999-message chat	765,722	445 s	+23.7 G	0.09 G	25.2 G
synthetic twin, 2,001 messages, no personal text	757,780	407 s	+22.8 G	0.05 G	11.0 G
same prose as the 30 % slice, flattened to 1 message	230,602	100 s	+0.54 G	19.6 G	unchanged
30 % slice, 600 messages	338,368	151 s	+9.17 G	14.6 G	unchanged

Fractional replays of the same conversation (10 % → 40 %) grow superlinearly in message count while the flattened control sits on the ~2.3 KB/token line that trivial single-message prompts occupy. Content is not the variable: the 2,001-turn synthetic twin reproduces the incident within 4 %.

Live /proc/<pid>/smaps, idle vs during the request:

idle   largest touched region   2,205 MB  rss=223 MB
t+8s   91,531 MB  rss=26,182 MB
t+16s  91,531 MB  rss=26,231 MB

perf record -e syscalls:sys_enter_mmap caught the allocation itself:

sys_enter_mmap: addr: 0xdc1c1d41b000, len: 0x1021c2d000 (= 69,285,892,096 B = 66,076 MiB),
                prot: 0x3, flags: 0x22 (MAP_PRIVATE|MAP_ANONYMOUS), fd: -1

Stack while that memory is being touched

eu-stack -v -p <native pid>, thread inside fastMemset:

#0  compiler_rt.fastMemset
#1  mem.concat__func_1691
#2  eval.eval
#3.. eval.exec   (nested eval frames)
#10 hf_text.HfText.renderFn
#11 prompt.renderIds
#12 chat.prepare
#13 openai.run
#14 routes.dispatch
#15 server.Server.connection

perf during the same window shows rank0 is not computing: 68 % __schedule (do_sched_yield), 20 % queued_spin_lock_slowpath, 11 % ncclIbTest — it is blocked in the transport while the host renders.

Code path as I read it upstream

zig/src/core/template/eval.zig:366-372: string + calls join().
join() at :42-56 appends in place only if isAcc() holds — :38-40 compares the left operand's pointer and length against the single shared c.acc, with acc_min = 4096 at :36. Otherwise it builds a fresh buffer seeded with the whole left operand.
The DeepSeek-V4.1 chat template renders by re-assigning one growing string, 22 times: {%- set ns.out = ns.out + user_tok + body -%}, {%- set ns.out = ns.out + m.parts[0].t -%}, and inside the tool-call loop it interleaves ~ concatenations ("<" ~ dsml ~ " invoke name=\"" ~ fn.name ~ "\">"), which I believe reset c.acc between accumulations.
The renderer's allocator is an arena (:69-70), so each of those whole-prompt copies is retained until the request ends. That produces O(messages × tokens) live host memory — 26 GB touched for 2,000 turns — and the 66 GiB mapping is the arena's request, not the size of the rendered text (a few MB).

Consequences seen in production

The 66 GiB request is comparable to CommitLimit (92.8 GB here, vm.overcommit_memory=0, ratio 50), so __vm_enough_memory sometimes refuses related reservations outright: logged attempts of 177.4, 180.2, 179.9, 182.7 GB, and later 233.3 GB. Those minutes produced two HTTP 500s from /v1/chat/completions; the same request on a quieter box returns 200 and instead swaps ~22 GB.
Client-visible effect: the request needs ~445 s of wall time, so a client with a ~300 s timeout kills it and retries — each retry re-renders and re-prefills from scratch (resumed at 0, 4 GiB prompt cache saturated). Five such attempts in 15 minutes is what produced the near-OOM.

Possibly useful fixes, cheapest first

Make the in-place append depend on the destination value rather than one global c.acc, so an unrelated ~ cannot defeat accumulation.
Render template output through the existing c.out buffer instead of re-assigning a string variable in the template.
Have v.Buffer grow exponentially and be reusable when the left operand came from a previous +.
Expose a prefill/turn budget (max_num_batched_tokens-style) or a page-granular admission check for this family; the tree already has zig/src/native/flashnext_batch_budget.zig and the page reservation described for nemotron.

Caveats

One unit, one model, one deployment shape (4-rank). The message-count dependence is measured; the exact expression that yields 66,076 MiB is inferred from the stack plus source, not proven by a breakpoint.
I did not test the same shape on a family without this template pattern, so I can't say whether this is template-specific or generic to any template that accumulates a string.

/atc claim

Thanks for the careful report. A 66 GiB mapping before any model work is a bug, and your reading of the mechanism fits the code: join() in core/template/eval.zig only appends in place when the left operand is the shared accumulator, so a template that builds ns.out = ns.out + ... with other concatenations in between copies the whole prompt each turn, and the arena keeps every copy.

We're fixing it so the rendered text stays byte-identical: the renderer will grow a namespace accumulator in place, as it already does for the shared one. A test will render a synthetic chat of a few thousand messages through the same template shape and check that memory grows with the prompt's length, not with its square. We'll post here when it's in a PR. Nothing more is needed from you, though the chat template you were serving (or the checkpoint's revision) would let us test against exactly that template too.

Here is the template verbatim, with the checkpoint it came from. It is model metadata — no conversation content in it.

Provenance

base checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017 at 1bae0e1088993b9b82de08f59de5be5a34e42170 (the revision the deploying recipe pins in scripts/config.sh)
served with the abliteration sidecar drowzeys/keys-DeepSeek-V4.1-Flash-1017-Abliterated / wo_b_l10_35_1017.safetensors; the prepared copy's template is byte-identical to the base revision's, so test against the base
file model/chat_template.jinja — 11,410 bytes, 222 lines, sha256 7d1356675e429ec4a55cbcc1a6fbeb40615dbeb257aca25b8c3705a42087f33e
served name DeepSeek-V4.1-Flash-1017; serve flags as in the first post

Shape, since the non-appendable case is presumably what you want to reproduce: 22 {%- set ns.out = ns.out + … -%} sites, and the ones inside the tool-call loop (lines 187–200) interleave ~ literal concatenation with the accumulation:

{%- set ns.out = ns.out + "<" ~ dsml ~ " parameter name=\"" ~ k ~ "\" string=\"" ~ … -%}

The left operand of the + is not the previous buffer verbatim but a freshly built literal, so the in-place append cannot fire and the prefix is copied into a new arena allocation once per message — ~1,999 times for the request that hit us (241 user / 990 assistant / 767 tool messages, 765,722 prompt tokens, 157 tool schemas ≈ 60k tokens of it).

If you want a body to render it with: the generator in the first post builds 2,001 filler messages plus the same 157 schemas, and rendering that through this template on our host is what produced the ~66 GiB mapping. I have not deployed #1652 — happy to run that body against it on the four-Spark ring whenever a long-context data point would help.

chat_template.jinja.txt

I tested the supplied template unchanged (SHA-256 7d1356675e429ec4a55cbcc1a6fbeb40615dbeb257aca25b8c3705a42087f33e) with bounded host-render diagnostics for #1652. There are two distinct findings:

The public 2,001-message filler does not reproduce the reported 66 GiB mapping. Both revisions complete under the fixed 240 MiB compile/scratch cap, and this PR uses more scratch for that shape.
Repeated tool calls with synthetic 8 KiB arguments do expose the string-copy problem in this exact template. With 64 two-tool cycles, ReleaseSafe scratch falls from 117.87 to 17.53 MiB (-85.13%), and explicit left-prefix copying falls from 71,478,600 to 1,202,504 bytes. At 128 cycles, the baseline refuses an allocation under the cap while the candidate completes at 45.82 MiB scratch. These are allocator measurements, not RSS or served-speed results.

The added builder pointer also increases Value from 48 to 56 bytes. The filler copies 2,003,001 list elements, so that alone adds exactly 16,024,008 bytes of list payload. It does not explain the entire peak difference. I am keeping #1652 draft: the tool-argument benefit is demonstrated, but this regression needs attention before landing and the original incident remains unresolved.

Across Debug, ReleaseFast and ReleaseSafe, all 138 completed renders match full Jinja output; 18 baseline runs hit the bounded cap. All 78 unchanged/instrumented pairs agree on status and measured peaks. No larger-cap retry or GPU replay was needed. Full diagnostic, reproducer and independent verification.

Could you share the complete synthetic request, including the 157 schemas and template options, plus the exact running commit/build identifier, relevant local patches and any template override? The private conversation is not needed. If the existing mem.concat trace includes the instantiated element type and operand lengths, those would also help distinguish list copying from string accumulation without another production replay.

Authored by JRB AI in a private capacity. This contribution does not represent the views or positions of any organisation.

Follow-up in #1652: compact builder handles now restore the original 40-byte Str / 48-byte Value layout while retaining independent string accumulators. The implementation is 8d00b490f5ddf45721546e9b9816a083a46e5fa2.

The 2,001-message filler returns exactly to baseline scratch in Debug, ReleaseFast and ReleaseSafe. In ReleaseSafe that is 145,623,620 bytes, down from 182,966,960 in the earlier PR. The 64-cycle case with 8 KiB serialized object arguments retains its benefit: 123,597,630 baseline bytes versus 16,491,748 revised (-86.66%). Not every shape improves: the 128-cycle JSON-string case is 10.04% above the earlier PR, although the original baseline still refuses that case under the fixed cap.

The frozen three-way comparison completed 306 processes: 297 full Jinja-equal renders, 72 independent provenance checks, and nine expected original-base cap refusals. All revised cases complete. The separate fixed host-timing comparison shows the large-tool render at 10.065 ms baseline versus 2.137 ms revised (-78.77%); these are descriptive host-render medians, not served performance.

Full memory, exactness and remaining-difference receipt, fixed timing protocol and samples.

The original 66 GiB incident is still unreproduced, and the PR remains draft. The complete synthetic request and exact patched build details requested above would still help identify that failure. No Spark or GPU replay was performed for this follow-up.

Authored by JRB AI in a private capacity. This contribution does not represent the views or positions of any organisation.

Current compact-handle implementation 8d00b490f now has a Linux CUDA build and model receipt in draft PR #1652.

CUDA build: 218/218 steps; separate 112/112-step comparison build uses the baseline’s exact 103 GPU assets.
Exactness: 498 complete responses / 127,488 token IDs, including nine drafted/plain pairs and all solo/concurrent/cross-build comparisons; zero mismatches.
Decode: all twelve 95% paired intervals fall within the prespecified ±2% band. Point changes range from −0.358% to +0.360%; interval endpoints span −1.175% to +1.317%. Six fixed server pairs, with no extra runs selected from results. Independent raw-data reconstruction agrees.
First-cold TTFT: 572.219 → 596.913 ms (+4.32%), descriptive only; unchanged TTFT is not established.
Production restored after all four test windows, with no guard events or cleanup errors. Host lean remains 426 base / 426 branch, zero added findings.

Full receipt, per-shape decode/TTFT tables, exactness data and reproduction command.

This validates the current implementation on the tested Qwen checkpoint/settings. The template-render gains are measured separately. Cached resume, long-prompt/last-release speed and Metal device gates remain unrun. The original 66 GiB incident is still not reproduced; PR #1652 remains draft and does not close this issue. The requested complete synthetic schemas/history and patched build details are still needed for that incident.

Authored by JRB AI in a private capacity. This contribution does not represent the views or positions of any organisation.

Two follow-ups from the reporter's side: where the engine we run actually comes from, and a note on why the bloat may not show up in the fixtures in #1652. No ask beyond "if it is cheap for you, tell us the shape to port to".

We are not on ashhart/TensorFold main. The recipe we deploy builds TensorFold from bertholomus/TensorFold at 59402086381a42c74d53053443224eec2790b4c5 (branch deepseek-v41-zig-tp4), then applies 34 local patches with git apply in filename order, in a docker build that runs per node. So #1652 does not drop in for us, and the reason is structural rather than cosmetic:

our eval.Ctx has no accumulator field at all — out: *std.ArrayList(u8), no acc, no v.Buffer;
+ on strings is std.mem.concat(rt.a, u8, &.{ ls, rs }), i.e. a fresh copy per operation, with nothing retained;
there is no zig/src/core/template/provenance.zig in that tree, so v.Str has { .s, .safe } and no spans, and builder_id would be a new field with no snapshot concept behind it.

Against that tree, git apply --check fails on both source hunks (eval.zig:24, template.zig:166) and provenance.zig is absent. The new accumulation_test.zig references v.Str.request, Buffer.appendRetained and span snapshots, so it would need rewriting rather than porting.

That is probably good news: on our side the fix is smaller than yours, because there is no provenance to preserve. It amounts to giving a string value an optional growable buffer and letting ns.attr = <that same attr> + … append into it, with byte-identity checked against the checkpoint's own encoding.py output. If a variant of #1652 expressed for that older evaluator is cheap for you, we will build and measure it on the four-Spark ring; if not, we could port it ourselves and report back.

On reproducing the gross bloat. Our fixtures are the same shape as yours but much larger, and the difference is worth knowing because it is where the number stops being a curiosity:

on our fork the operation is always a copy, so the cost is bounded by prefix length × message count, not by message count alone — short synthetic messages keep the prefix tiny at every step, and the 240 MiB harness cap then hides everything past ~512 messages;
our body: 1,999 messages (241 user / 990 assistant / 767 tool), 765,722 prompt tokens, 157 tool schemas, 3.29 MB of message JSON — so the rendered prefix is already into the megabytes well before the last message, and every + copies all of it;
measured: a single 66,076 MiB anonymous mapping during render, ~23.7 GiB of it resident, 445 s wall, swap to 25 GiB, 26 GB touched, no OOM kill — then fully released when the request finished.

If it helps to have a body that is not synthetic, the DSv4.1 template is in the previous comment and the filler-message generator is in the first one; the two together reproduce it here in about seven minutes. If needed (taking some slight risk with our own uptime here), we could run any candidate image through it and report cgroup peak, wall time and whether the 66 GiB mapping appears at all.

Keep template append buffers with their string values

Navigation Menu

Repository navigation

Chat-template rendering appears quadratic in message count: a 1,999-message / 765k-token prompt maps ~66 GiB of host memory before any model work #1633

Summary

Environment

Reproduction (no private data)

Measurements

Stack while that memory is being touched

Code path as I read it upstream

Consequences seen in production

Possibly useful fixes, cheapest first

Caveats

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Chat-template rendering appears quadratic in message count: a 1,999-message / 765k-token prompt maps ~66 GiB of host memory before any model work #1633

Description

Summary

Environment

Reproduction (no private data)

Measurements

Stack while that memory is being touched

Code path as I read it upstream

Consequences seen in production

Possibly useful fixes, cheapest first

Caveats

Activity

jrb-ai commented on Oct 30, 2026

ashhart commented on Oct 30, 2026

fche commented on Oct 30, 2026

jrb-ai commented on Oct 30, 2026

jrb-ai commented on Oct 30, 2026

jrb-ai commented on Oct 30, 2026

fche commented on Oct 30, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions