Please Add Multimodal Image/Video Input Support for Qwen3.8 Models

I strongly recommend adding multimodal image and video input support for Qwen3.8 models, especially for models such as:

Qwen3.8-27B
Qwen3.8-27B-FP8
RadixArk/Qwen3.8-27B-NVFP4

At the moment, FreeToken can successfully serve these checkpoints for text inference, but multimodal checkpoints are effectively served as text-only models.

This is currently one of the biggest limitations preventing me from using FreeToken as my primary local inference backend.

Why this matters

Qwen3.8-27B is natively a multimodal model and supports:

Text
Images
Video
Mixed text-image inputs
Document / screenshot understanding
Charts and scientific figures
Vision-based agent workflows

The multimodal capability is an important part of the model itself, rather than an optional auxiliary feature.

For example, RadixArk/Qwen3.8-27B-NVFP4 still contains the vision-related components and preprocessing configuration, so the checkpoint itself has not been converted into a text-only model.

However, when serving it with FreeToken:

ft serve \
  --model RadixArk/Qwen3.8-27B-NVFP4 \
  --model-source modelscope

text generation works well, but image inputs cannot currently be processed.

Requested feature

It would be extremely useful if FreeToken could add native multimodal support for Qwen3.8, including:

Image input support through the OpenAI-compatible API.
Support for image_url, local images, and Base64 encoded images.
Correct invocation of the Qwen vision encoder / multimodal processor.
Mixed text + image conversations.
Video input support in the future.
Compatibility with quantized checkpoints such as NVFP4 and FP8 whenever the checkpoint preserves the vision module.
Multimodal support for /v1/chat/completions and ideally /v1/responses.

For example, I would like to be able to send requests similar to:

{
  "model": "RadixArk/Qwen3.8-27B-NVFP4",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Please analyze this image."
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/jpeg;base64,..."
          }
        }
      ]
    }
  ]
}

Why I think this should be a high-priority feature

FreeToken is already extremely attractive for running recent quantized models locally, especially on consumer NVIDIA GPUs.

However, models such as Qwen3.8 are increasingly designed as unified multimodal models, and serving them as text-only models leaves a significant part of their capabilities unused.

For local research, coding, document analysis, scientific figure understanding, screenshot analysis, and agent workflows, image input is becoming almost as important as text inference.

I am currently running Qwen3.8-27B locally with FreeToken and would very much prefer to continue using FreeToken instead of switching to another inference backend solely because multimodal input is unavailable.

So I strongly hope that Qwen3.8 multimodal support can be prioritized on the roadmap.

Thank you for the excellent project.

We have image ingest for qwen4_exp (Qwen3.8-Flash-Next) working on our own hardware, and rather than
offer code into a feature that has an owner and a direction, here are the four things we learned that
cost us real time and that we think are worth having in front of you before you write it. Three of
them are decisions rather than code, and two are constraints that bound the feature's value before a
line of it is written.

⚠️ Everything below is gfx1201 (Radeon AI PRO R9700), Linux, one card, --moe-backend offload,
with BLOCK_N clamped to 32 in the QSA attend kernel. Numbers carry that platform.

1. Two constraints already in the tree gate what image support can be

Neither is a line you have to write; both shape what the feature can do.

scheduler/prefill.py:174 — a multimodal prompt must fit in ONE prefill chunk, as an explicit
NotImplementedError. max_extend_tokens defaults to 8192 (scheduler/config.py:16), so image
prompts are capped near ~8k tokens regardless of the model's context. On a model with a 262,144-token
context that is a surprising cliff, and it is the single most important thing to decide about
deliberately rather than inherit.
scheduler/cache.py:99 — a multimodal request matches against the EMPTY prefix and is never
inserted, so an image turn gets zero prompt-cache reuse. In multi-turn use with a sticky image
this is the dominant cost, and it is invisible in a single-request benchmark.

2. `FREETOKEN_LOAD_VISION=1` is a silent no-op for every non-Gemma family

models/config.py:14 gates on VISION_KEY_PREFIXES = ("vision_tower.", "embed_vision.") — Gemma key
naming. Qwen's model.visual.* keys are dropped by a hardcoded path
(models/qwen4_exp/weight.py:101-103) before that switch is ever consulted. So setting the
variable changes nothing and errors nowhere. Anyone reaching for it as an escape hatch loses time.

3. Enforcing a prompt ceiling at admission is necessary but not sufficient

When we added a ceiling, two things bit us that are worth knowing in advance:

The effective limit is min(live prefill budget, ceiling), not the ceiling — token_budget is
shared, so a static check passes requests the scheduler then cannot serve.
⛔ A streaming request must be rejected before the SSE headers go out. Our first version opened
the picture after the stream had started, which produces a 400 after content-type: text/event-stream — a shape most clients handle badly. Decoding the image pre-commit fixes it and
the frontend can do that for free via state.frontend_tokenizer().

⚠️ Also worth sizing early: at an 8192-token ceiling a single request approaches ~200 MB crossing
the queue as a serialized message.

4. One implementation detail that a reference reading gets wrong

The ViT pos_embed resample is bilinear (align_corners=True, border clamp) — HF's
interpolation_mode — not bicubic. We had it written down as bicubic from a reading of the
reference, and only a deliberate ablation caught it: a bicubic port loads, runs, and is wrong by
rel 0.56. It is exactly the kind of thing that produces a tower that looks like it works, so it
is cheap to say and expensive to find.

5. The datapoint you cannot get from an NVIDIA box: the tower is nearly free

Because engine/cache_budget.py is KV-first with weights already subtracted, the 0.836 GiB BF16
tower lands entirely on the expert cache rather than on context:

	text-only	with the tower
MoE expert slots	5,960	5,636 (−5.44 %)
KV tokens	110,080	110,080 — unchanged
decode	—	≈ −0.23 % [modelled from the slot loss, not measured]

⭐ And a portability note: the tower is 16 heads over hidden 1152 ⇒ head_dim 72, which is neither
64 nor 128 and looked likely to fall off the fast path. It does not — on ROCm it takes the
memory-efficient SDPA path, 36.3 MB peak at N=4096, bit-identical in cost to head_dim 64 and 128.
So the ViT is tens of MB of transient, not hundreds, and vision on a 32 GB card costs expert slots
rather than context.

If any of it is useful

We have a working qwen4_exp tower and ingest running in production on both protocol surfaces,
validated against the HF reference offline and then over HTTP. ⚠️ On the padded multi-image batch
contract that encode_images actually receives — [N, P, ...], not HF's cu_seqlens packing — the
port lands at rel 1.61e-04 against the reference, min-token-cos 0.999999821, with its unpadded
control at rel 1.62e-04 ⇒ the residual is the wider masked softmax's reduction order, not the
padding contract.
⛔ We are not proposing it as a PR — you have this in hand and starting from Qwen3.6 is your call.
But the port, the ablations, and the gate suite are yours for the asking if they would save you time.

⚠️ Separately, we have filed the inconsistency between the two protocol surfaces as a bug in #348 — on a text-only server /v1/chat/completions returns a clean 400 while /v1/messages returns 200 with the image silently dropped. That one is worth fixing independently of this feature request.

Following up on the above — we have since shipped this and run it in production for a few days, and
the operational half turned out to be more interesting than the source reading. Four things, in the
order they cost us time.

⚠️ Platform for everything below: gfx1201 (Radeon AI PRO R9700), Linux, one card, --moe-backend offload, BLOCK_N clamped to 32 in the QSA attend kernel (#349). Half-box numbers.

1. The prompt ceiling was answerable from source, and the answer has two traps in it

We expected to have to measure what raising --max-prefill-length costs. It turned out to be
arithmetic, and we spent no GPU time on it — but only after falling into two naming traps that are
worth naming for you.

⛔ Two different properties are called max_forward_len.

ServerArgs.max_forward_len (scheduler/config.py:43) is max_extend_tokens
EngineConfig.max_forward_len (engine/config.py:101) is max_seq_len

and the single max_forward_len-sized buffer, at engine/engine.py:440, reads the second one.
We read the first and briefly believed raising the flag would grow an allocation. It does not.

⇒ raising --max-prefill-length is free at load. The entire cost is transient prefill
activation. That is a much better story than we expected and it may be worth documenting, because
the flag reads like a memory knob.

⛔ prefill_chunk_budget exists on exactly one cache pool. The scheduler admits on
min(max_extend_tokens, cache_manager.prefill_chunk_budget), but prefill_chunk_budget is defined
only on DSV4PagedKVCache. The qwen4_exp pool has neither it nor swa_paged, so CacheManager's
getattr pickup falls through and the admission ceiling is max_extend_tokens exactly. Anyone
reasoning about that min() on a non-DSV4 model is reasoning about a term that is not there.

⚠️ And raising it changes the text path too, which is easy to miss: at 110k, a text prompt stops
being 13 chunks of 8192 and becomes one 110,080-token forward. Prefill is compute-bound at full
chunks on our hardware, so that is not free TTFT either, even though it is free memory.

2. What 8192 actually feels like to a user

We shipped 8192 — deliberately, not by default. But the honest cost of that choice is worth
stating, because it is a product problem rather than an engineering one:

A screenshot plus a long conversation fails, and the client has no way to express why. One row
cannot advertise two context ceilings — a text ceiling of 110,080 and an image ceiling of 8,192 —
and neither the OpenAI nor the Anthropic surface has a field for it. Every client we use shows the
text number and then errors at a tenth of it.

Our answer was to ship two rows, one text and one vision, each with its own advertised ceiling.
That works but it doubles the roster and it is plainly a workaround. If image support lands with a
single advertised context, users will hit this, and the error they get should say the ceiling out
loud — ours returns prompt is too long for an image request: 9934 tokens > 8192 maximum (an image prompt must fit in one prefill chunk), which at least explains itself.

3. What the tower costs when it is actually serving

Our earlier comment gave you the static cost: the 0.836 GiB BF16 tower lands on the expert cache,
5,960 → 5,636 slots, and modelled to about −0.23 % decode. Having run it, the measured figure at the
shipping geometry is larger and — more usefully — constant:

−7.70 % decode for the vision row against the text row, mean of five contexts, no trend
decode is flat over 25.7x of context — x1.0089 from 10k to 258k

⇒ the penalty is the whole price, not the near end of a widening one. That is the number we would
have wanted before starting, because a cost that grows with context would have changed the design and
this one does not.

⛔ A warning about measuring it. Only a same-image, same-day, same-prompt arm can see that
−7.7 %. Prompt content alone moves decode ~11 % on an offload row — more than the effect under
test — because different text routes to different experts and changes the streaming pattern. We
re-measured the same geometries on a different day and read levels ~11 % apart while the ratio held
to within 0.1 points. So a banked tok/s figure travels only with its exact prompt, and any vision
benchmark that varies the prompt between arms is measuring the prompt.

4. Load behaviour, and one signature worth recognising

Cold load with the tower is 88–191 s, teardown ~20 s — the tower does not measurably change
either.

⚠️ The one thing worth recognising: during load we see pswpin climb ~209k with the GPU busy
throughout. That is the PLE table streaming from disk at ~1.77 GB/s and it is the load working
correctly. A load that is dying looks superficially similar — heavy swap — but shows 100 % of
one core with the GPU idle. We have mistaken the second for a wedged driver before. If you add any
load-progress reporting for multimodal weights, distinguishing those two states is worth more than a
percentage bar.

On the port

Still not proposing it as a PR — you have this in hand. But the gate suite behind the above (17/17
over HTTP on the port, 14/14 on the shipping configuration, covering both protocol surfaces,
malformed and remote images, the streaming pre-commit 400, and answers-from-the-picture) is the part
we would most want if I were writing this feature, and it is yours if it saves you time.

@lukascechovic

Adding a data point from a different corner of the matrix than @lukascechovic's, since their four
findings above match ours closely enough that repeating them would waste your time: theirs is
gfx1201 / ROCm / qwen4_exp, mine is RTX 4090 (sm_89) / CUDA / qwen3_5_moe —
ornith-ai/Ornith-1.5-35B-A3B-NVFP4, the 3.5 family. Two of their points bit us identically (the
one-prefill-chunk NotImplementedError at scheduler/prefill.py:174, and
FREETOKEN_LOAD_VISION=1 being a no-op outside Gemma because VISION_KEY_PREFIXES is Gemma key
naming), which is probably worth something on its own: those are not platform artifacts.

The thing I have that is not in this thread yet is a measurement of what the two protocol
surfaces do with the same image, because that is what an agent client actually hits. On main
(03c28d2), one 256x256 PNG — a yellow square on blue — and the question "What colour is the
square in this image?":

surface	result
`POST /v1/chat/completions`	400 `Unsupported content part type for text-only server: image_url`
`POST /v1/messages`	200, `input_tokens: 26`, "The square is purple/violet in colour."

input_tokens: 26 is the tell — with the image actually delivered the same request costs 92. So
the picture is dropped whole and the model answers from the text, confidently and wrongly, inside
a 200. I run Claude Code against this server and it speaks the Anthropic surface, so in practice
pasting a screenshot gets a fluent answer about something the model never saw. That is filed
separately as #348; I mention it here because it is a decision the vision work has to make anyway:
whatever the two surfaces do, they should do the same thing, and dropping a declared input is
the one outcome a caller cannot detect.

For what it is worth on the feasibility question in the original post: the 3.5-family vision path
does work on a 24 GB consumer card. The tower is 0.832 GiB of resident bf16 weights on this
checkpoint (333 visual tensors, no quant scales at all), which is why it is worth having behind a
flag rather than always-on — text-only serving never touches it, and on this card 0.8 GiB is real
money. With it enabled both surfaces answer "yellow" (input_tokens: 92); with it disabled both
return 400 with a reason.

@jason-fxz — you mentioned in #232 that you are starting vision from Qwen3.6. I have the 3.5 path
running here and can keep testing against main as it lands, on Ada and on a 24 GB budget, which
looks like a gap in what has been reported so far. If the 3.5 family is in scope for what you are
building, say the word and I will point the measurements wherever they are most useful; if it
overlaps with work already underway, I am happy to just keep testing.

One incidental trap, since it cost me a wrong conclusion for ten minutes: with max_tokens: 40
and thinking left on, both surfaces return 200 with empty content — Ornith spends the whole
budget reasoning and never reaches the answer. input_tokens was still 92, so the image had
arrived. If you see an empty 200 while testing this, check input_tokens before concluding the
image was dropped: 26 means dropped, 92 means it is the token budget.

_{Written with AI assistance; the measurements are from my own hardware and I can reproduce

them.}

feat(qwen3_5_moe): the Qwen3-VL vision tower (encoder + patch merger)

Image input for Qwen3.8-Flash-Next exists as draft PR #386 and has been running in our production server since 2026-09-04.

What it covers: the vision tower + multimodal embedder are loaded behind FREETOKEN_LOAD_VISION=1 (opt-in, since they are ~1 GiB of resident bf16 weights that text-only serving never touches), mRoPE position handling for the QSA attention layers and the indexer, /v1/chat/completions image parts on the OpenAI path, chunked prefill for image prompts, and image prompts participating in the prefix cache keyed by a per-image content hash so a repeated image is a cache hit rather than a re-encode.

Video is not covered — this is images only.

Practical notes from running it:

The tower costs about 2.6 points of expert residency on our configuration. On a model where decode is flat from 100% down to ~70% residency that is free, but on a tighter card it is a real trade, which is why it is a flag rather than always-on.
The hard part was not the tower, which transformers already provides as Qwen4ExpVisionModel; it was mRoPE for the QSA layers and the indexer, and making a chunked prefill of an image prompt scatter only the placeholders inside the current chunk.
Before that PR, FreeToken served this model text-only and the online API had no image path for any model (only the offline LLM.generate(mm_inputs=) for Gemma-4), so this touches the server layer as well as the model.

Related: #348 notes the two protocol surfaces disagree on image input (/v1/chat/completions 400s while another path accepts) — worth reading alongside if you are wiring a client.

🤖 Generated with Claude Code

https://claude.ai/code/session_0173pf9k9fSVtwbm3f898HDt

Now that #454 has landed: for anyone trying image input on more than one card, models/qwen4_exp/weight.py still raises above TP=1.

I've been serving image input on Qwen3.8-Flash-Next at TP=2 on a downstream branch since before #454 — a different and, on positional encoding, less complete implementation than the one that merged, but the TP=2 weight split works and is on the branch if it's of any use: https://github.com/lukascechovic/FreeToken/tree/rocm-gfx1201

feat(qwen3_5_moe): the Qwen3-VL vision tower (encoder + patch merger)

Image input for Qwen family has already landed in #454, closing.

Navigation Menu

Repository navigation

Please Add Multimodal Image/Video Input Support for Qwen3.8 Models #321

Why this matters

Requested feature

Why I think this should be a high-priority feature

1. Two constraints already in the tree gate what image support can be

2. `FREETOKEN_LOAD_VISION=1` is a silent no-op for every non-Gemma family

3. Enforcing a prompt ceiling at admission is necessary but not sufficient

4. One implementation detail that a reference reading gets wrong

5. The datapoint you cannot get from an NVIDIA box: the tower is nearly free

If any of it is useful

1. The prompt ceiling was answerable from source, and the answer has two traps in it

2. What 8192 actually feels like to a user

3. What the tower costs when it is actually serving

4. Load behaviour, and one signature worth recognising

On the port

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Please Add Multimodal Image/Video Input Support for Qwen3.8 Models #321

Description

Why this matters

Requested feature

Why I think this should be a high-priority feature

Activity

lukascechovic commented on Sep 2, 2026

1. Two constraints already in the tree gate what image support can be

2. FREETOKEN_LOAD_VISION=1 is a silent no-op for every non-Gemma family

3. Enforcing a prompt ceiling at admission is necessary but not sufficient

4. One implementation detail that a reference reading gets wrong

5. The datapoint you cannot get from an NVIDIA box: the tower is nearly free

If any of it is useful

lukascechovic commented on Sep 2, 2026

1. The prompt ceiling was answerable from source, and the answer has two traps in it

2. What 8192 actually feels like to a user

3. What the tower costs when it is actually serving

4. Load behaviour, and one signature worth recognising

On the port

MT-z commented on Sep 3, 2026

gdevenyi commented on Sep 5, 2026

lukascechovic commented on Sep 15, 2026

jason-fxz commented on Oct 5, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

2. `FREETOKEN_LOAD_VISION=1` is a silent no-op for every non-Gemma family