Skip to content

tutorial-qwen3-vl-32b.md

Latest commit

 

History

History
375 lines (294 loc) · 16.9 KB

tutorial-qwen3-vl-32b.md

File metadata and controls

375 lines (294 loc) · 16.9 KB
Raw

Tutorial: Deploy Qwen3-VL-32B with vLLM Neuron

This tutorial walks through deploying Qwen3-VL-32B-Instruct with vLLM Neuron. It covers environment setup, model download, an optional MXFP8 quantization step, online serving, and offline inference.

The model runs in two precisions:

  • BF16 — the default, on trn2.48xlarge or Trn3.
  • MXFP8 — micro-scaled FP8 weights for the text transformer, Trn3-only. Reduces the weight footprint and improves both throughput and latency while keeping accuracy close to BF16.
Precision Instance Steps
BF16 Trn2 or Trn3 1 → 2 → 4 (skip the MXFP8-only Step 3)
MXFP8 Trn3 only 1 → 2 → 3 → 4

BF16 is the main path throughout; each step calls out the MXFP8 differences where they apply. Step 3 (quantization) is only for MXFP8 — skip it for BF16.

Prerequisites:

  • A trn2.48xlarge instance with Neuron SDK 2.31.0 or later. See setup guide. MXFP8 requires a Trn3 instance (the native MXFP8 kernels are Trn3-only) with Neuron SDK 2.32 or later.
  • vLLM Neuron plugin 0.21.0 or above installed.
  • Python 3.10+
  • For MXFP8, LLM-Compressor to quantize the checkpoint: pip install llmcompressor compressed-tensors

Step 1: Set up your environment

Verify Neuron devices are visible:

neuron-ls
# Lists the Neuron devices and cores available on your instance.

Set environment variables before running any inference script:

# Extend timeouts for large model compilation
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1200
export NEURON_LIBTORCH_COMPILATION_TIMEOUT=1200

# Required if your home directory is on NFS
export NEURON_CC_FLAGS="--temp-dir=/tmp/neuroncc_tmp"
mkdir -p /tmp/neuroncc_tmp

Step 2: Download the model (optional)

huggingface-cli download \
    Qwen/Qwen3-VL-32B-Instruct \
    --local-dir /path/to/Qwen3-VL-32B-Instruct

Note: This step is optional. You can pass the Hugging Face model ID (e.g. Qwen/Qwen3-VL-32B-Instruct) directly to vllm serve, and the weights will be downloaded automatically on first run.

Step 3: Quantize the text model to MXFP8 (MXFP8 only)

Skip this step for BF16. Qwen3-VL is not published with MXFP8 weights, so you must quantize the BF16 checkpoint offline. This uses LLM-Compressor and compressed-tensors to apply weight-only, data-free post-training quantization (PTQ) and save a checkpoint in the compressed-tensors format. For background on the quantization workflow, see Quantize using external libraries in the quantization guide.

This is weight-only, data-free PTQ. Run this step on CPU, not on Trainium. Use a machine with enough memory to load the ~64 GB BF16 model, plus disk for both the base and quantized checkpoints. You can quantize once and reuse the checkpoint across Trn3 instances.

Only the text-model linear layers are quantized. The script builds an ignore list that keeps lm_head and every vision-encoder (visual.*) linear layer in full precision — the Neuron MXFP8 path applies to the text transformer only. This text-only scheme mirrors the published Qwen/Qwen3-VL-32B-Instruct-FP8 checkpoint, which likewise quantizes the text model and leaves the vision tower and lm_head in higher precision.

import torch
from compressed_tensors.offload import dispatch_model
from transformers import AutoTokenizer, Qwen3VLForConditionalGeneration

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

# Path (or Hugging Face ID) of the BF16 base checkpoint.
MODEL_ID = "Qwen/Qwen3-VL-32B-Instruct"
# Where to write the quantized checkpoint.
SAVE_DIR = "/path/to/Qwen3-VL-32B-Instruct-MXFP8-text-only"

# Load the model and tokenizer.
model = Qwen3VLForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

# Build the ignore list: skip lm_head and every vision-encoder linear layer so
# only the text-model linear layers are quantized to MXFP8.
ignored_list = ["lm_head"]
for name, module in model.named_modules():
    if "visual" in name and isinstance(module, torch.nn.Linear):
        ignored_list.append(name)

print(f"Ignoring {len(ignored_list)} layers from quantization")

# Quantize all remaining Linear layers to MXFP8 via weight-only PTQ.
recipe = QuantizationModifier(targets="Linear", scheme="MXFP8", ignore=ignored_list)
oneshot(model=model, recipe=recipe)

# Confirm the quantized model still generates sane text.
print("========== SAMPLE GENERATION ==============")
dispatch_model(model)
input_ids = tokenizer("Hello my name is", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))
print("===========================================")

# Save to disk in compressed-tensors format.
model.save_pretrained(SAVE_DIR)
tokenizer.save_pretrained(SAVE_DIR)
print(f"Saved quantized model to {SAVE_DIR}")

The script prints a short sample generation as a sanity check, then writes the quantized checkpoint and tokenizer to SAVE_DIR. Point the serving commands below at that directory.

Step 4: Run inference

Run the model through the online serving endpoint or the offline LLM API — choose whichever fits your deployment.

Online serving

Start a vLLM OpenAI-compatible server.

:::{note} On a Trn3 instance without EFA (Elastic Fabric Adapter) installed, the server fails to start during EFA-affinity setup — this affects both BF16 and MXFP8. To work around it, prepend NEURON_SKIP_EFA_AFFINITY=1 to the vllm serve command (and set it in the environment before the offline LLM script below). :::

BF16 (Trn2 or Trn3):

vllm serve /path/to/Qwen3-VL-32B-Instruct \
    --served-model-name Qwen3-VL-32B-Instruct \
    --max-model-len 8192 \
    --max-num-batched-tokens 8192 \
    --max-num-seqs 4 \
    --tensor-parallel-size 4 \
    --additional-config '{
        "neuron_config": {
            "quantization": "bf16",
            "num_batched_tokens_buckets": [8192],
            "num_seqs_buckets": [4],
            "on_device_sampling_config": {"all_greedy": true}
        },
        "vision_neuron_config": {
            "num_vision_tokens_buckets": [2048],
            "vision_attention_block_size": 2048
        }
    }'

MXFP8 (Trn3 only) — point at the quantized checkpoint from Step 3:

vllm serve /path/to/Qwen3-VL-32B-Instruct-MXFP8-text-only \
    --served-model-name Qwen3-VL-32B-Instruct-MXFP8 \
    --max-model-len 8192 \
    --max-num-batched-tokens 8192 \
    --max-num-seqs 4 \
    --tensor-parallel-size 4 \
    --block-size 16 \
    --tokenizer-mode slow \
    --no-enable-chunked-prefill \
    --no-enable-prefix-caching \
    --hf-overrides '{"quantization_config": {}}' \
    --additional-config '{
        "neuron_config": {
            "quantization": "mxfp8",
            "modules_to_not_convert": [],
            "num_batched_tokens_buckets": [8192],
            "num_seqs_buckets": [4],
            "on_device_sampling_config": {"all_greedy": true}
        },
        "vision_neuron_config": {
            "num_vision_tokens_buckets": [2048, 4096],
            "vision_attention_block_size": 2048
        }
    }'

The MXFP8-specific settings, compared to the BF16 command:

  • neuron_config.quantization: "mxfp8" — selects the on-device MXFP8 path. This is a Neuron-specific setting; do not pass vLLM's --quantization flag.
  • modules_to_not_convert: [] — an empty list runs the whole text transformer (attention and MLP) in MXFP8. The vision encoder stays BF16 regardless, because its weights were left unquantized in Step 3.
  • --hf-overrides '{"quantization_config": {}}' — clears the checkpoint's compressed-tensors quantization_config so the Neuron loader owns the MXFP8 path.
  • --tokenizer-mode slow, --no-enable-chunked-prefill, --no-enable-prefix-caching, --block-size 16 — pin the deployment to the validated MXFP8 configuration.

Current limitation: at tensor-parallel-size >= 32, change "modules_to_not_convert" to ["mlp"] (keeps the MLP layers in BF16); full MXFP8 ([]) is supported at tensor-parallel-size <= 16.

Once the server is up, send requests using the OpenAI Python SDK (use the --served-model-name you launched with):

import base64
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

# Text-only
response = client.chat.completions.create(
    model="Qwen3-VL-32B-Instruct",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    max_tokens=50,
)
print(response.choices[0].message.content)

# Image + text
with open("image.jpg", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="Qwen3-VL-32B-Instruct",
    messages=[{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}},
        {"type": "text", "text": "Describe this image."},
    ]}],
    max_tokens=200,
)
print(response.choices[0].message.content)

Offline inference

vLLM-Neuron compiles the model on the first run and caches the artifacts to ~/.cache/vllm/neuron/compile_cache. Subsequent runs skip recompilation and load from cache.

Configuration

The model has two components, each with its own config object passed via additional_config (see the configuration options reference):

  • neuron_config: text decoder settings (token/sequence bucket sizes, sampling, quantization).
  • vision_neuron_config: vision encoder settings (vision token buckets, attention block size, and optional vision TP/DP split).

Bucket sizes control the discrete padded shapes compiled into each NEFF. Each bucket adds compile time; start with one and add more as needed. The two components bucket along different dimensions, so their buckets are configured separately:

  • num_batched_tokens_buckets (in neuron_config): text-decoder buckets over the number of batched text tokens per forward pass. See compilation options.
  • num_vision_tokens_buckets (in vision_neuron_config): vision-encoder buckets over the number of vision patches per encoder forward pass (raw T*H*W patches from image_grid_thw, before the 2x2 spatial merge; this is the count select_vision_bucket matches against). The scheduler may batch images from multiple requests into one forward pass, so size the buckets for the total images processed together, not a single request. These scale with image count and resolution:
num_vision_tokens_buckets Approximate capacity
[2048] 1–2 images at 448×448 px
[2048, 8192] Up to ~8 images
[2048, 8192, 20480] Up to ~20 images

Run offline inference

The following script runs text-only, single-image, and multi-image inference. It shows the BF16 LLM(...) construction; for MXFP8, swap in the LLM(...) block that follows (the rest of the script is identical).

import os

os.environ["VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS"] = "1200"
os.environ["NEURON_LIBTORCH_COMPILATION_TIMEOUT"] = "1200"

from transformers import AutoProcessor
from vllm import LLM, SamplingParams
from vllm.assets.image import ImageAsset

MODEL_PATH = "/path/to/Qwen3-VL-32B-Instruct"

# BF16 (Trn2 or Trn3)
llm = LLM(
    model=MODEL_PATH,
    max_model_len=8192,
    max_num_batched_tokens=8192,
    max_num_seqs=4,
    tensor_parallel_size=4,
    additional_config={
        "neuron_config": {
            "quantization": "bf16",
            "num_batched_tokens_buckets": [8192],
            "num_seqs_buckets": [4],
            "on_device_sampling_config": {"all_greedy": True},
        },
        "vision_neuron_config": {
            "num_vision_tokens_buckets": [2048],
            "vision_attention_block_size": 2048,
        },
    },
)

processor = AutoProcessor.from_pretrained(MODEL_PATH)
sampling_params = SamplingParams(max_tokens=200, temperature=0.0)

# --- Text-only ---
outputs = llm.generate(["What is the capital of France?"], sampling_params)
print(outputs[0].outputs[0].text)

# --- Single image ---
image = ImageAsset("cherry_blossom").pil_image.resize((640, 320))
messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": "Describe this image."},
]}]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = llm.generate([{"prompt": prompt, "multi_modal_data": {"image": [image]}}], sampling_params)
print(outputs[0].outputs[0].text)

# --- Multi-image ---
images = [
    ImageAsset("stop_sign").pil_image.resize((448, 448)),
    ImageAsset("cherry_blossom").pil_image.resize((448, 448)),
]
messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "image"},
    {"type": "text", "text": "Compare these two images."},
]}]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = llm.generate([{"prompt": prompt, "multi_modal_data": {"image": images}}], sampling_params)
print(outputs[0].outputs[0].text)

For MXFP8 (Trn3 only), point MODEL_PATH at the quantized checkpoint from Step 3 and build the engine with the MXFP8 config instead — the mirror of the MXFP8 vllm serve command in Step 4:

MODEL_PATH = "/path/to/Qwen3-VL-32B-Instruct-MXFP8-text-only"

llm = LLM(
    model=MODEL_PATH,
    max_model_len=8192,
    max_num_batched_tokens=8192,
    max_num_seqs=4,
    block_size=16,
    tensor_parallel_size=4,
    tokenizer_mode="slow",
    enable_chunked_prefill=False,
    enable_prefix_caching=False,
    hf_overrides={"quantization_config": {}},
    additional_config={
        "neuron_config": {
            "quantization": "mxfp8",
            "modules_to_not_convert": [],
            "num_batched_tokens_buckets": [8192],
            "num_seqs_buckets": [4],
            "on_device_sampling_config": {"all_greedy": True},
        },
        "vision_neuron_config": {
            "num_vision_tokens_buckets": [2048, 4096],
            "vision_attention_block_size": 2048,
        },
    },
)

Vision parallelism (optional)

By default the vision encoder runs as one DP replica per NeuronCore (TP1), the recommended layout for high-throughput multi-image workloads. To shard the encoder weights across cores for a single large image instead, increase tp_size.

Set tp_size inside vision_neuron_config to change the split. DP is derived automatically as world_size / tp_size, where world_size is your tensor_parallel_size. The commands above use tensor_parallel_size=4, so:

tp_size Vision TP Vision DP Best for
1 (default) 1 4 Multi-image, high throughput
4 4 1 Single-image, low latency

The same applies at a larger tensor_parallel_size (e.g. 16), which raises the default vision DP accordingly. Example — TP1, full DP:

"vision_neuron_config": {
    "num_vision_tokens_buckets": [2048],
    "vision_attention_block_size": 2048,
    "tp_size": 1,  # DP = world_size / tp_size
}

Conclusion

You have deployed Qwen3-VL-32B-Instruct on Trainium — in BF16 on trn2.48xlarge (or Trn3), or in MXFP8 on Trn3 after self-quantizing the text model. The model supports text-only, single-image, multi-image, and video inputs via both the offline LLM API and the OpenAI-compatible online serving endpoint. To validate accuracy after enabling MXFP8, see the accuracy debugging guide. For feature support and accuracy validation results, see the model card.

Next steps

To tune this deployment for your workload — choosing vision and text sharding with roofline analysis, profiling to find the bottleneck, and enabling the multimodal features and fused kernels — see Optimizing a Vision-Language Model.