Skip to content

Performance regression from v0.1.3 to v0.2.0 on RTX4060 8GB #822

Open
Open
Performance regression from v0.1.3 to v0.2.0 on RTX4060 8GB#822
Labels
bugSomething isn't workingwindows

Description

@moro-hide
moro-hide
opened on Oct 24, 2026

Before you start

  • I have read the FAQ and my problem is not answered there.
  • I have read the Roadmap and this is not already planned there.
  • I have searched existing issues and found no duplicate.
  • I am on the latest release, or on a freshly rebuilt main when building from source.

What happened

Environment

RTX4060 8GB
i5-8400
RAM 48GB
Windows 11 + WSL2

Model
Qwen3.6-35B-A3B-NVFP4

Working version
v0.1.3

Problem version
v0.2.0

Issue

The same startup parameters and same model are much slower on v0.2.0.

v0.1.3 feels responsive.

v0.2.0 often takes much longer before the first generated token.

I also noticed:

v0.1.3
--moe-hybrid-max-fetch auto: fetching 22.7%

v0.2.0
--moe-hybrid-max-fetch auto: no ft bench bw row
--moe-locked-fetch auto: no ft bench bw row

ft bench bw completes successfully.

Logs below.
Same startup parameters were used on both versions.

Command

ft serve
--model ~/LLM-models/nvidia/Qwen3.6-35B-A3B-NVFP4
--host=0.0.0.0
--port=1919
--memory-ratio 0.87
--moe-strategy hybrid
--moe-cpu-layers auto
--max-running-requests 1
--kv-reserve-tokens 65536
--max-seq-len-override 65536

v0.1.3

freetoken version 0.1.3

[2026-10-24|16:23:42] INFO --moe-cpu-layers: layers [0, 1, 39] settled pageable
[2026-10-24|16:23:42] INFO --moe-cache-auto resolved moe_cache_size=858
[2026-10-24|16:23:42] INFO --moe-hybrid-max-fetch auto: fetching 22.7% of each decode step

[2026-10-24|16:23:45] INFO Free memory after initialization: 0.59 GiB

[2026-10-24|16:23:45] INFO Start capturing CUDA graphs with sizes: [1]

[2026-10-24|16:24:48] INFO Scheduler is idle, waiting for new reqs...
[2026-10-24|16:24:48] INFO API server is ready

Inference example:

[2026-10-24|16:26:39] INFO Prefill batch, #new-token: 8192
[2026-10-24|16:26:39] INFO Prefill batch, #new-token: 8124
[2026-10-24|16:26:41] INFO Decode batch ...

This version feels responsive and usable.

v0.2.0

freetoken version 0.2.0

[2026-10-24|17:29:59] WARNING --moe-cpu-layers: layers [0, 13, 27] settled pageable

[2026-10-24|17:30:00] WARNING --moe-hybrid-max-fetch auto: no ft bench bw row ...
[2026-10-24|17:30:00] WARNING --moe-locked-fetch auto: no ft bench bw row ...

[2026-10-24|17:30:00] INFO locked layers [0, 13, 27]: GPU slots + fetch up to ...

[2026-10-24|17:30:02] INFO Free memory after initialization: 0.51 GiB

[2026-10-24|17:30:02] INFO Start capturing CUDA graphs with sizes: [1]

[2026-10-24|17:30:05] INFO Scheduler is idle, waiting for new reqs...
[2026-10-24|17:30:05] INFO API server is ready

Inference example:

[2026-10-24|17:31:56] INFO Prefill batch, #new-token: 8192
[2026-10-24|17:31:56] INFO Prefill batch, #new-token: 6701
[2026-10-24|17:32:05] INFO Decode batch ...

Another example:

[2026-10-24|17:38:56] INFO Prefill batch, #new-token: 8192
[2026-10-24|17:38:57] INFO Prefill batch, #new-token: 6761
[2026-10-24|17:39:16] INFO Decode batch ... gen throughput (token/s): 2.42
[2026-10-24|17:39:20] INFO Decode batch ... gen throughput (token/s): 9.68

I also ran:

ft bench bw

The benchmark completed successfully and produced a profile:

~/.cache/freetoken/benchbw/*.json

However v0.2.0 still reports:

--moe-hybrid-max-fetch auto: no ft bench bw row
--moe-locked-fetch auto: no ft bench bw row

Question:

Is this expected in v0.2.0?
Has the auto MoE policy changed significantly compared to v0.1.3?

How did you install FreeToken

pip / uv wheel

FreeToken version

0.2.0,0.1.3

OS

Windows 11

OS details

26H2, WSL2 Ubuntu-26.04

GPU and driver

RTX4060 LP 8GB NVIDIA-SMI 615.78.05

CPU and system RAM

i5 8400 48GB

Checkpoint

nvidia/Qwen3.6-35B-A3B-NVFP4

Command

ft serve \
  --model ~/LLM-models/nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --host=0.0.0.0 \
  --port=1919 \
  --cors "*" \
  --memory-ratio 0.87 \
  --moe-strategy hybrid \
  --moe-cpu-layers auto \
  --max-running-requests 1 \
  --kv-reserve-tokens 65536 \
  --max-seq-len-override 65536

Full log

~$ bash ft-up.sh
[2026-10-24|16:22:49] INFO     Parsed arguments:
ServerArgs(model_path='/home/testman/LLM-models/nvidia/Qwen3.6-35B-A3B-NVFP4', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, max_running_req=1, attention_backend='auto', moe_strategy='hybrid', quant_backend=None, ple_backend='disk', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=True, kv_reserve_tokens=65536, moe_cache_policy='lru', moe_p
[2026-10-24|16:22:49|FrontendAPI] INFO     Default sampling config (source=model): temperature=1.0, top_k=20, top_p=0.95
INFO:     Started server process [61802]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:1919 (Press CTRL+C to quit)
/home/testman/FreeToken/python/freetoken/engine/engine.py:1135: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
  torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[2026-10-24|16:22:53|core|rank=0] INFO     Enabled expandable_segments (override via PYTORCH_ALLOC_CONF)
[2026-10-24|16:22:53|core|rank=0] INFO     Auto-selected attention backend: fi
[2026-10-24|16:22:53|core|rank=0] INFO     Resolved config: moe_strategy='hybrid', attention_backend='fi', cache_type='hybrid_radix', page_size=1
[2026-10-24|16:22:53|core|rank=0] INFO     Free memory before loading model: 6.92 GiB
Loading weights: 100%|███████████████████████████████████████████| 3/3 [00:08<00:00,  2.71s/it]
[2026-10-24|16:23:05|core|rank=0] INFO     MoE experts: nvfp4 via triton
[2026-10-24|16:23:05|core|rank=0] INFO     --moe-cpu-layers auto: banks 16.93 GiB > pin budget 15.67 GiB; locking 3 head+tail MoE layers for CPU decode ([0, 1, 39])
[2026-10-24|16:23:05|core|rank=0] INFO     --moe-cpu-layers split residency: disabling MoE prefill overlap (locked layers prefill via synchronous pageable copies)
[2026-10-24|16:23:05|core|rank=0] INFO     expert banks: slow path (parallel build)
Loading experts (parallel):   0%|                                  | 0.00/21.8G [00:00<?, ?B/s][2026-10-24|16:23:15|core|rank=0] WARNING  bank lock failed; leaving this and later banks pageable: [Errno 12] mlock(0.2 GiB): Cannot allocate memory (RLIMIT_MEMLOCK / `ulimit -l` caps OS-locked bytes; raise it or shrink --moe-cpu-layers)
[2026-10-24|16:23:42|core|rank=0] WARNING  --moe-cpu-layers: layers [0, 1, 39] settled pageable instead of OS-locked (lock failed); they still decode on the CPU executor but may swap under memory pressure
[2026-10-24|16:23:42|core|rank=0] INFO     --moe-cache-auto resolved moe_cache_size=858 num_pages=65556 (prefill_overlap=False)
[2026-10-24|16:23:42|core|rank=0] INFO     --moe-hybrid-max-fetch auto: fetching 22.7% of each decode step's expert misses over PCIe (benched PCIe/CPU bandwidth ratio), the rest on the CPU
[2026-10-24|16:23:42|core|rank=0] INFO     torch intra-op threads: 6 -> 1 (cores reserved for the pinned CPU MoE pool)
[2026-10-24|16:23:42|core|rank=0] INFO     CPU MoE executor ready: threads=5 (pinned to cores 0..4) isa=avx2 fmt=nvfp4 H=2048 I=512 experts=256 layers=40 top_k=8 act=silu max_tokens=1
[2026-10-24|16:23:42|core|rank=0] INFO     Multimodal enabled: QwenVLMMProcessor, encoders ['vision'] on host, serving ['image']
[2026-10-24|16:23:45|core|rank=0] INFO     Allocating 65556 tokens for KV cache, K + V = 1.25 GiB
[2026-10-24|16:23:45|core|rank=0] INFO     Free memory after initialization: 0.59 GiB
[2026-10-24|16:23:45|core|rank=0] INFO     Start capturing CUDA graphs with sizes: [1]
[2026-10-24|16:23:45|core|rank=0] INFO     Free GPU memory before capturing CUDA graphs: 0.59 GiB
Capturing graphs: bs = 1   | avail_mem = 0.60 GiB: 100%|██████| 1/1 [01:02<00:00, 62.82s/batch]
[2026-10-24|16:24:48|core|rank=0] INFO     Free GPU memory after capturing CUDA graphs: 0.60 GiB
INFO:     127.0.0.1:44652 - "POST /v1/chat/completions HTTP/1.1" 503 Service Unavailable
[2026-10-24|16:24:48|core|rank=0] INFO     Scheduler is idle, waiting for new reqs...
[2026-10-24|16:24:48|FrontendAPI] INFO     API server is ready to serve on 0.0.0.0:1919
INFO:     127.0.0.1:44652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:48722 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-24|16:25:12|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.25, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 0, #queue-req: 1, input throughput (token/s): 344.66
[2026-10-24|16:25:13|FrontendAPI] WARNING  Aborting request for user 0
INFO:     127.0.0.1:48736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-24|16:25:46|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.25, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 0, #queue-req: 1, input throughput (token/s): 390.50
[2026-10-24|16:25:57|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.37, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 0, #queue-req: 1, input throughput (token/s): 757.39
[2026-10-24|16:26:03|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.41, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 1, #queue-req: 0, input throughput (token/s): 1445.60
[2026-10-24|16:26:03|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 2309, #cached-token: 0, token usage: 0.41, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 0, input throughput (token/s): 38244.00
[2026-10-24|16:26:05|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 26926, token usage: 0.41, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 0.52, #queue-req: 0
[2026-10-24|16:26:07|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 26966, token usage: 0.41, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 19.02, #queue-req: 0
[2026-10-24|16:26:09|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 27006, token usage: 0.41, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 20.04, #queue-req: 0
[2026-10-24|16:26:11|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 27046, token usage: 0.41, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 18.26, #queue-req: 0
[2026-10-24|16:26:14|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 27086, token usage: 0.41, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 18.43, #queue-req: 0
[2026-10-24|16:26:16|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 27126, token usage: 0.41, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 18.06, #queue-req: 0
[2026-10-24|16:26:18|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 27166, token usage: 0.41, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 19.70, #queue-req: 0
INFO:     127.0.0.1:56978 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-24|16:26:18|core|rank=0] INFO     Scheduler is idle, waiting for new reqs...
INFO:     127.0.0.1:56982 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:56988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-24|16:26:39|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.25, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 1, #queue-req: 0, input throughput (token/s): 225.77
[2026-10-24|16:26:39|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8124, #cached-token: 0, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 0, input throughput (token/s): 49755.67
[2026-10-24|16:26:41|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 16349, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 1.71, #queue-req: 0
[2026-10-24|16:26:43|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 16389, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 19.60, #queue-req: 0
[2026-10-24|16:26:44|core|rank=0] INFO     Scheduler is idle, waiting for new reqs...
[2026-10-24|16:26:48|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 452, #cached-token: 0, token usage: 0.01, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 0, input throughput (token/s): 50.90
INFO:     127.0.0.1:40644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:52608 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-24|16:26:49|core|rank=0] INFO     Scheduler is idle, waiting for new reqs...
INFO:     127.0.0.1:34146 - "GET /api/v1/models HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:34146 - "GET /api/tags HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:34146 - "GET /v1/props HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:34146 - "GET /props HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:34146 - "GET /version HTTP/1.1" 404 Not Found
[2026-10-24|16:24:23] INFO     reasoning-effort profile: supported=['high', 'low', 'max', 'medium', 'minimal', 'none', 'xhigh'] default=None
INFO:     127.0.0.1:34154 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:34160 - "POST /v1/chat/completions HTTP/1.1" 503 Service Unavailable
INFO:     127.0.0.1:34160 - "POST /v1/chat/completions HTTP/1.1" 503 Service Unavailable
INFO:     127.0.0.1:34160 - "POST /v1/chat/completions HTTP/1.1" 503 Service Unavailable
INFO:     127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 503 Service Unavailable
[2026-10-24|16:25:25|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.37, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 0, #queue-req: 1, input throughput (token/s): 625.56
[2026-10-24|16:25:25|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.00, #mamba-slot: 0/8, mamba usage: 0.00, #running-req: 0, #queue-req: 1, input throughput (token/s): 38360.12
^C
[2026-10-24|16:43:24|core|rank=0] INFO     Scheduler exiting gracefully...
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [61802]
/usr/lib/python3.14/multiprocessing/resource_tracker.py:396: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown: {'/mp-1slhvi86'}
  warnings.warn(
~$ 
~$ 
~$ bash ft200-up.sh
[2026-10-24|16:55:24] INFO     Parsed arguments:
ServerArgs(model_path='/home/testman/LLM-models/nvidia/Qwen3.6-35B-A3B-NVFP4', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, hf_overrides={}, max_running_req=1, attention_backend='auto', moe_strategy='hybrid', quant_backend=None, ple_backend='disk', swa_decoder_replay='bounded', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=True, moe_cache_mem=None, moe_cache_scope='unified', moe_cache_split='equal', kv_reserve_tokens=65536, moe_cache_policy='lru', moe_prefill_overlap=True, moe_prefill_hit_d2d=False, moe_collect_stats=False, moe_cpu_threads=0, moe_cpu_layers='auto', moe_hybrid_max_fetch=-1, moe_locked_fetch=-1.0, moe_locked_max_fetch=8, cuda_graph_bs=None, cuda_graph_max_bs=None, page_size=1, memory_ratio=0.87, linear_state_cache_ratio=2.0, swa_full_tokens_ratio=0.2, swa_num_pages_override=None, distributed_timeout=60.0, use_dummy_weight=False, use_pynccl=True, max_seq_len_override=65536, num_page_override=None, num_token_override=None, mm=MultimodalConfig(disabled_encoders=frozenset(), embed_cache_device='cpu', encoder_weights='host', image_min_tokens=None, image_max_tokens=None, processor_kwargs={}), max_extend_tokens=8192, cache_type='radix', offline_mode=False, decode_log_interval=40, special_token_ckpt=False, _unique_suffix='.pid=63022', server_host='0.0.0.0', server_port=1919, num_tokenizer=0, silent_output=False, shell_mode=False, served_model_name='Qwen3.6-35B-A3B-NVFP4', tool_call_parser='qwen3_coder', reasoning_parser='qwen3', sampling_defaults='model', max_output_tokens=None, enable_cache_report=False, anthropic_inline_system='auto', allowed_media_domains='', allowed_local_media_path='', cors_origins='*', gpu=(), gpu_assigned=None)
[2026-10-24|16:55:24|FrontendAPI] INFO     Default sampling config (source=model): temperature=1.0, top_k=20, top_p=0.95
INFO:     Started server process [63022]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:1919 (Press CTRL+C to quit)
/home/testman/ft200/python/freetoken/engine/engine.py:1370: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
  torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[2026-10-24|16:55:28|core|rank=0] INFO     Enabled expandable_segments (override via PYTORCH_ALLOC_CONF)
[2026-10-24|16:55:28|core|rank=0] INFO     Auto-selected attention backend: fi
[2026-10-24|16:55:28|core|rank=0] INFO     Resolved config: moe_strategy='hybrid', attention_backend='fi', cache_type='hybrid_radix', page_size=1
[2026-10-24|16:55:28|core|rank=0] INFO     Free memory before loading model: 6.92 GiB
[2026-10-24|16:55:28|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:28|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:28|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped w4a4: the experts declare no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
[2026-10-24|16:55:32|core|rank=0] INFO     kernel triton selected; skipped b12x: the weight declares no static activation scale (W4A16 checkpoint); cublas: the weight declares no static activation scale (W4A16 checkpoint)
Loading weights: 100%|███████████████████████████████████████████| 3/3 [00:09<00:00,  3.05s/it]
[2026-10-24|16:55:42|core|rank=0] INFO     MoE experts: nvfp4 via triton
[2026-10-24|16:55:43|core|rank=0] INFO     expert banks: slow path (parallel build)
[2026-10-24|16:56:25|core|rank=0] INFO     --moe-cpu-layers auto (pin probe): pinned 40 MoE layers (16.99 GiB) in 1.1s, locked 0 []
[2026-10-24|16:56:25|core|rank=0] INFO     --moe-cache-auto resolved moe_cache_size=854 num_pages=65544 (prefill_overlap=True)
[2026-10-24|16:56:26|core|rank=0] INFO     torch intra-op threads: 6 -> 1 (cores reserved for the pinned CPU MoE pool)
[2026-10-24|16:56:26|core|rank=0] INFO     CPU MoE executor ready: threads=5 (pinned to cores 0..4) isa=avx2 fmt=nvfp4: 40 layers H=2048 I=512 experts=256 layers=40 top_k=8 act=silu max_tokens=1 spin_wait=False handshake=kernel
[2026-10-24|16:56:26|core|rank=0] WARNING  --moe-hybrid-max-fetch auto: no `ft bench bw` row for 40 MoE layers (nvfp4 2048x512) at 5 CPU threads; using a fixed fetch cap of 1 there. `ft bench bw --model <checkpoint>` benches each layer kind
[2026-10-24|16:56:26|core|rank=0] INFO     Multimodal enabled: QwenVLMMProcessor, encoders ['vision'] on host, serving ['image']
In file included from /home/testman/.local/share/uv/python/cpython-3.11.16-linux-x86_64-gnu/include/python3.11/Python.h:12,
                 from /home/testman/ft200/.venv/lib/python3.11/site-packages/triton/backends/nvidia/driver.c:9:
/home/testman/.local/share/uv/python/cpython-3.11.16-linux-x86_64-gnu/include/python3.11/pyconfig.h:1796:9: warning: ‘_POSIX_C_SOURCE’ redefined
 1796 | #define _POSIX_C_SOURCE 200809L
      |         ^~~~~~~~~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:26,
                 from /home/testman/ft200/.venv/lib/python3.11/site-packages/triton/backends/nvidia/include/cuda.h:56,
                 from /home/testman/ft200/.venv/lib/python3.11/site-packages/triton/backends/nvidia/driver.c:1:
/usr/include/features.h:319:10: note: this is the location of the previous definition
  319 | # define _POSIX_C_SOURCE        202405L
      |          ^~~~~~~~~~~~~~~
[2026-10-24|16:56:30|core|rank=0] INFO     Allocating 65544 tokens for KV cache, K + V = 1.25 GiB
[2026-10-24|16:56:30|core|rank=0] INFO     Free memory after initialization: 0.51 GiB
[2026-10-24|16:56:30|core|rank=0] INFO     Start capturing CUDA graphs with sizes: [1]
[2026-10-24|16:56:30|core|rank=0] INFO     Free GPU memory before capturing CUDA graphs: 0.64 GiB
Capturing graphs: bs = 1   | avail_mem = 0.62 GiB: 100%|██████| 1/1 [01:17<00:00, 77.83s/batch]
[2026-10-24|16:57:48|core|rank=0] INFO     Free GPU memory after capturing CUDA graphs: 0.64 GiB
[2026-10-24|16:57:49|core|rank=0] INFO     Scheduler is idle, waiting for new reqs...
[2026-10-24|16:57:49|FrontendAPI] INFO     API server is ready to serve on 0.0.0.0:1919
INFO:     127.0.0.1:40270 - "GET /api/v1/models HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:40270 - "GET /api/tags HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:40270 - "GET /v1/props HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:40270 - "GET /props HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:40280 - "GET /api/v1/models HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:40280 - "GET /api/tags HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:40270 - "GET /version HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:40280 - "GET /v1/props HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:40280 - "GET /props HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:40280 - "GET /version HTTP/1.1" 404 Not Found
[2026-10-24|16:58:26] INFO     reasoning-effort profile: supported=['high', 'low', 'max', 'medium', 'minimal', 'none', 'xhigh'] default=None
INFO:     127.0.0.1:40290 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:40302 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:40326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:40310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
/home/testman/ft200/.venv/lib/python3.11/site-packages/triton/language/core.py:2625: UserWarning: tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.
  warn("tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.")
[2026-10-24|17:02:19|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.23, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 1, #queue-req: 1, input throughput (token/s): 30.28
[2026-10-24|17:02:22|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 6663, #cached-token: 0, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 1, input throughput (token/s): 2124.69
[2026-10-24|17:02:26|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 14896, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 0.14, #queue-req: 1
INFO:     127.0.0.1:48590 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:48600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-24|17:04:42|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.23, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 1, #queue-req: 1, input throughput (token/s): 58.82
[2026-10-24|17:04:43|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 6917, #cached-token: 0, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 1, input throughput (token/s): 5123.38
[2026-10-24|17:04:46|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 15146, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 0.28, #queue-req: 1
[2026-10-24|17:04:50|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 15186, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 12.14, #queue-req: 1
/home/testman/ft200/.venv/lib/python3.11/site-packages/triton/language/core.py:2625: UserWarning: tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.
  warn("tl.make_block_ptr is deprecated. Use TensorDescriptor or tl.make_tensor_descriptor instead.")
INFO:     127.0.0.1:38910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-24|17:04:59|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 57, #cached-token: 14899, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 1, input throughput (token/s): 3.63
[2026-10-24|17:05:00|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 14970, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 3.86, #queue-req: 1
INFO:     127.0.0.1:57506 - "GET /api/v1/models HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:57506 - "GET /api/tags HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:57506 - "GET /v1/props HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:57506 - "GET /props HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:57506 - "GET /version HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:57510 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:57520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-24|17:05:15|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 1188, #cached-token: 15104, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 2, input throughput (token/s): 73.64
[2026-10-24|17:05:18|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 16327, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 2.19, #queue-req: 2
[2026-10-24|17:05:22|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 16367, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 11.21, #queue-req: 2
[2026-10-24|17:05:25|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 16407, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 11.41, #queue-req: 2
[2026-10-24|17:05:29|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 16447, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 11.04, #queue-req: 2
[2026-10-24|17:05:33|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 16487, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 10.95, #queue-req: 2
[2026-10-24|17:05:36|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 16527, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 10.55, #queue-req: 2
[2026-10-24|17:05:40|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 16567, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 11.61, #queue-req: 2
[2026-10-24|17:05:43|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 16607, token usage: 0.25, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 11.90, #queue-req: 2
[2026-10-24|17:05:47|core|rank=0] INFO     Decode batch, #running-req: 0, #token: 0, token usage: 0.00, #mamba-slot: 0/8, mamba usage: 0.00, gen throughput (token/s): 11.34, #queue-req: 2
INFO:     127.0.0.1:44090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-24|17:05:53|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 259, #cached-token: 0, token usage: 0.00, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 2, input throughput (token/s): 6.87
INFO:     127.0.0.1:56380 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:56384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-24|17:08:10|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.23, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 1, #queue-req: 2, input throughput (token/s): 59.80
[2026-10-24|17:08:11|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 6991, #cached-token: 0, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 2, input throughput (token/s): 5149.64
[2026-10-24|17:08:13|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 15210, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 0.27, #queue-req: 2
INFO:     127.0.0.1:33914 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:33938 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:33946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-24|17:08:22|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 333, #cached-token: 16647, token usage: 0.26, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 3, input throughput (token/s): 29.08
[2026-10-24|17:08:23|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 16982, token usage: 0.26, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 4.33, #queue-req: 3
[2026-10-24|17:10:36|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.23, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 1, #queue-req: 3, input throughput (token/s): 61.26
[2026-10-24|17:10:37|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 6701, #cached-token: 0, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 3, input throughput (token/s): 5110.94
INFO:     127.0.0.1:53252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-24|17:10:48|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 452, #cached-token: 0, token usage: 0.01, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 3, input throughput (token/s): 43.97
[2026-10-24|17:10:48|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 457, token usage: 0.01, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 0.27, #queue-req: 3
[2026-10-24|17:13:03|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.23, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 1, #queue-req: 2, input throughput (token/s): 60.39
[2026-10-24|17:13:05|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 6688, #cached-token: 0, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 2, input throughput (token/s): 5033.64
[2026-10-24|17:13:13|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 496, #cached-token: 0, token usage: 0.01, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 2, input throughput (token/s): 61.26
[2026-10-24|17:15:26|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.23, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 1, #queue-req: 1, input throughput (token/s): 61.40
[2026-10-24|17:15:28|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 6781, #cached-token: 0, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 1, input throughput (token/s): 5161.81
[2026-10-24|17:15:29|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 14988, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 0.14, #queue-req: 1
[2026-10-24|17:15:32|core|rank=0] INFO     Decode batch, #running-req: 0, #token: 0, token usage: 0.00, #mamba-slot: 0/8, mamba usage: 0.00, gen throughput (token/s): 12.27, #queue-req: 1
INFO:     127.0.0.1:38064 - "GET /api/v1/models HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:38064 - "GET /api/tags HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:38064 - "GET /v1/props HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:38064 - "GET /props HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:38064 - "GET /version HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:38066 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:38080 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:38086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-24|17:15:39|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 455, #cached-token: 0, token usage: 0.01, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 2, input throughput (token/s): 39.52
[2026-10-24|17:15:46|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 263, #cached-token: 0, token usage: 0.00, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 1, input throughput (token/s): 37.43
INFO:     127.0.0.1:38070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:44540 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-24|17:18:01|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.23, #mamba-slot: 3/8, mamba usage: 0.38, #running-req: 1, #queue-req: 0, input throughput (token/s): 60.64
[2026-10-24|17:18:02|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 6894, #cached-token: 0, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 0, input throughput (token/s): 5210.51
[2026-10-24|17:18:04|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 15102, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 0.26, #queue-req: 0
[2026-10-24|17:18:08|core|rank=0] INFO     Decode batch, #running-req: 1, #token: 15142, token usage: 0.23, #mamba-slot: 4/8, mamba usage: 0.50, gen throughput (token/s): 10.93, #queue-req: 0
[2026-10-24|17:18:10|core|rank=0] INFO     Scheduler is idle, waiting for new reqs...
[2026-10-24|17:18:17|core|rank=0] INFO     Prefill batch, #new-seq: 1, #new-token: 452, #cached-token: 0, token usage: 0.01, #mamba-slot: 4/8, mamba usage: 0.50, #running-req: 1, #queue-req: 0, input throughput (token/s): 30.61
INFO:     127.0.0.1:60972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-24|17:18:18|core|rank=0] INFO     Scheduler is idle, waiting for new reqs...
^C
[2026-10-24|17:23:12|core|rank=0] INFO     Scheduler exiting gracefully...
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [63022]
~$ /home/testman/.local/share/uv/python/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

~$

Anything else

No response

Activity

  1. moro-hide
    added
    bugSomething isn't working
    on Oct 24, 2026
  2. github-actions
    added
    windows
    on Oct 24, 2026

  3. jason-fxz commented on Oct 24, 2026

    @jason-fxz
    jason-fxz
    on Oct 24, 2026
    Collaborator

    Thanks for the detailed logs. The no ft bench bw row warnings come from a change in how 0.2.0 reads the bench profile:

    • 0.1.3 matched the profile by expert format only, so any nvfp4 row from a plain ft bench bw was used (that's where your fetching 22.7% came from).
    • 0.2.0 matches one row per MoE layer kind: format, expert shape, top_k and CPU thread count. A plain ft bench bw only benches one canonical shape per format (nvfp4 is benched at 3072x1536), while Qwen3.6-35B-A3B's experts are 2048x512. So no row matches, and hybrid falls back to a fixed fetch cap of 1.

    Please bench the checkpoint itself:

    ft bench bw --model ~/LLM-models/nvidia/Qwen3.6-35B-A3B-NVFP4

    It adds that model's rows to the same profile (existing rows are kept). Run it on the same machine and CPU setup you serve with, since the thread count is part of the row key. Then restart ft serve with your usual command. Both warnings should be replaced by lines like --moe-hybrid-max-fetch auto: each decode step fetches this share ....

    After that, could you post the new startup log and the timing of one long request sent with no other requests in flight? Your 0.2.0 log shows #queue-req going up to 3, so several requests were queued behind each other.

  4. jason-fxz commented on Oct 24, 2026

    @jason-fxz
    jason-fxz
    on Oct 24, 2026
    Collaborator

    One more 0.2.0 change that may matter on your machine: how layers whose host banks cannot be pinned are served.

    On WSL, CUDA can only pin about half of RAM, so with --moe-cpu-layers some MoE layers keep their experts in host memory that is OS-locked (or, like on your box where mlock fails, plain pageable) instead of CUDA-pinned. The GPU cannot DMA from that memory.

    • 0.1.3: those layers decoded only on the CPU.
    • 0.2.0 (--moe-locked-fetch, default auto): they also fetch some of their expert misses into GPU slots. A CPU thread copies each fetched expert into a pinned staging buffer, then the GPU copies it over PCIe. Every decode step of each locked layer waits for that copy thread, even when it has nothing to fetch. On the i5-8400 the CPU MoE executor already uses all 6 cores (5 workers + 1 coordinator), so the copy thread gets no core of its own and runs on cores shared with those workers. Any scheduling delay it hits lands on every token. In your run there was no bench row, so each locked layer fetched at most one expert per step. Once ft bench bw --model raises that share, the extra RAM traffic (CPU read, staging write, PCIe read) also competes with the CPU MoE kernels for memory bandwidth.

    This only applies to runs with locked layers. Your [0, 13, 27] run looks like it was started with --moe-cpu-layers 3 rather than auto; could you confirm the command for that run? In the full log you attached, auto pinned all 40 layers, so nothing was locked and this path was not active. It therefore does not explain the slowdown in that run.

    To narrow it down, here is 0.2.0 with 0.1.3's MoE layout (the same 3 CPU-only layers, prefill overlap off). This is an isolation experiment, not a recommended setting:

    ft serve \
  --model ~/LLM-models/nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --host=0.0.0.0 --port=1919 --cors "*" \
  --memory-ratio 0.87 \
  --moe-strategy hybrid \
  --moe-cpu-layers 0,1,39 \
  --moe-locked-fetch 0 \
  --disable-moe-prefill-overlap \
  --max-running-requests 1 \
  --kv-reserve-tokens 65536 \
  --max-seq-len-override 65536
    • --moe-cpu-layers 0,1,39: the 3 layers 0.1.3 picked automatically, given as an explicit list. That skips 0.2.0's pin probe, so the layout is the same on every boot.
    • --moe-locked-fetch 0: those layers decode on the CPU only, as in 0.1.3.
    • --disable-moe-prefill-overlap: 0.2.0 already turns overlap off for this layout; the flag only makes it explicit.

    The hybrid fetch share cannot be pinned to 0.1.3's 22.7%. 0.2.0 takes it from your checkpoint's own ft bench bw --model row.

    After running the bench from my previous comment, could you time three runs?

    1. your original command (--moe-cpu-layers auto)
    2. the command above
    3. the command above without --moe-locked-fetch 0

    For each, send one long prompt (about 15k tokens) with no other requests in flight. Report the time to first token as measured by the client, plus the decode gen throughput from the log. The log's prefill input throughput is not a good measure here: it divides by the time since the previous prefill log line, so idle time between requests is counted as prefill time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingwindows

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions