feat: Add request batching and spec tree-size options to the experimental server, fix FP8-KV and unquantized MTP draft builds - #199

feat: Add request batching and spec tree-size options to the experimental server, fix FP8-KV and unquantized MTP draft builds#199

rain-sicoreai wants to merge 4 commits into

NVIDIA:mainNVIDIA/TensorRT-Edge-LLM:mainfrom

rain-sicoreai:feat/experimental-server-batching-fp8kv-mtprain-sicoreai/TensorRT-Edge-LLM:feat/experimental-server-batching-fp8kv-mtpCopy head branch name to clipboard