Dynamic Routing for vLLM #100

Activity

1. terrykong commented on Mar 31, 2025

   

   terrykong

   on Mar 31, 2025

   CollaboratorAuthor

   More actions

   Example router from dynamo: https://github.com/ai-dynamo/dynamo/blob/main/examples/llm/components/kv_router.py
2. dchichkov commented on Apr 1, 2025

   

   dchichkov

   on Apr 1, 2025

   More actions

   There's this repo - https://github.com/VectorInstitute/vector-inference

   • This repository provides an easy-to-use solution to run inference servers on Slurm-managed computing clusters using vLLM.

   Note also:
   https://docs.vllm.ai/projects/production-stack/en/latest/user_manual/router/cmd.html

   Note, slurm autoscale, with an option to backfill slurm capacity is the next obvious feature.
3. NVIDIA-NeMo

   locked and limited conversation to collaborators on Jul 30, 2025
4. snowmanwwg

   converted this issue into a discussion #800 on Jul 30, 2025