Description
Model load times significantly vary according to which technique is used. The goal of this issue is to investigate whether time can be saved on the existing serving-llms-on-instinct skill using the techniques below (sample model used as reference).
┌────────────┬───────────────────────────────────────────────────┬──────────────────┬──────────────┐
│ Experiment │ Loader │ Weight-load time │ vs baseline │
├────────────┼───────────────────────────────────────────────────┼──────────────────┼──────────────┤
│ E2 │ baseline (stock per-tensor pinned + non_blocking) │ 83.2 s │ 1.00× │
├────────────┼───────────────────────────────────────────────────┼──────────────────┼──────────────┤
│ E3 │ fastsafetensors + GDS (SAFETENSORS_FAST_GPU=1) │ 68.7 s │ 1.21× faster │
├────────────┼───────────────────────────────────────────────────┼──────────────────┼──────────────┤
│ E4 │ staged loader (VLLM_STAGED_MOE_LOAD=1, flush=512) │ 55.3 s │ 1.50× faster │
└────────────┴───────────────────────────────────────────────────┴──────────────────┴──────────────┘
Description
Model load times significantly vary according to which technique is used. The goal of this issue is to investigate whether time can be saved on the existing
serving-llms-on-instinctskill using the techniques below (sample model used as reference).