Experimental OpenAI-compatible server fails to start with: ModuleNotFoundError: No module named 'tensorrt'

[FABallemand]

Describe the bug

The experimental OpenAI-compatible server currently fails to start with: ModuleNotFoundError: No module named 'tensorrt'.

Installing tensorrt 11.2.1.2 (using pip install tensorrt==11.2.1.2) does not resolve the issue. After installing tensorrt, starting the server instead fails with: FileNotFoundError: plugin library not found: build/libNvInfer_edgellm_plugin.so.

Expected behavior: The server should start successfully without requiring additional manual installation or configuration beyond the documented dependencies.

Actual behavior: The server fails to start both when TensorRT is not installed and after installing the specified TensorRT version, but with different errors.

Steps/Code to reproduce bug

Build configuration:
As described in TensorRT Edge-LLM on Jetson, I used an NVIDIA Jetson Thor as both the host and target device. I installed TensorRT Edge-LLM v0.10.1 by following the Installation Guide. The installation completed successfully without any issues.

$ python3 -m pip install pybind11==3.0.4
$ mkdir -p build
$ cd build
$ cmake .. \
    -DCMAKE_BUILD_TYPE=Release \
    -DTRT_PACKAGE_DIR=/usr \
    -DCMAKE_TOOLCHAIN_FILE=cmake/aarch64_linux_toolchain.cmake \
    -DEMBEDDED_TARGET=jetson-thor \
    -DCUDA_CTK_VERSION=13.0 \
    -DENABLE_CUTE_DSL=ALL \
    -DBUILD_PYTHON_BINDINGS=ON \
    -Dpybind11_DIR="$(python -m pybind11 --cmakedir)"
$ make -j$(nproc)

$ python3 -m pip install -e ".[server,server-tools]"

Runtime command used:

• CLI:

tensorrt-edgellm-serve Qwen/Qwen3.5-0.8B \
  --max-input-len 4096 \
  --max-kv-cache-capacity 8192 \
  --port 8000

• Python API:

from experimental.server import LLM, SamplingParams

llm = LLM(
    model="Qwen/Qwen3.5-0.8B",
    # cache_dir="/data/edgellm-cache",
    max_input_len=4096,
    max_kv_cache_capacity=8192,
)
result = llm.chat(
    [{"role": "user", "content": "Explain paged KV caches."}],
    SamplingParams(max_tokens=128, temperature=0),
)
print(result.text)

Details

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[1], line 3
      1 from experimental.server import LLM, SamplingParams
      2 
----> 3 llm = LLM(
      4     model="Qwen/Qwen3.5-0.8B",
      5     # cache_dir="/data/edgellm-cache",
      6     max_input_len=4096,

File ~/aide/TensorRT-Edge-LLM/target/TensorRT-Edge-LLM/experimental/server/runtime/engine.py:736, in LLM.__init__(self, model, cache_dir, engine_cache_max_size_gb, clear_engine_cache, max_input_len, max_batch_size, max_kv_cache_capacity, draft_top_k, draft_step, verify_tree_size, build_options, speculative_config, context_cache_config)
    733 if options.spec_type != "none":
    734     options = replace(options, tree_base=tree_base)
--> 736 prepared = prepare_model(
    737     model,
    738     cache_dir,
    739     options,
    740     max_cache_size_bytes=int(engine_cache_max_size_gb * (1 << 30)),
    741     clear_cache=clear_engine_cache,
    742 )
    743 self._cache_dir = cache_root(cache_dir)
    744 self._model_dir = prepared.model_dir

File ~/aide/TensorRT-Edge-LLM/target/TensorRT-Edge-LLM/experimental/server/runtime/engine_build.py:439, in prepare_model(model, cache_dir, options, max_cache_size_bytes, clear_cache)
    433 build_options = replace(
    434     options,
    435     plugin_path=_resolve_plugin_path(options.plugin_path),
    436 )
    437 from experimental.builder.cli import main as builder_main
--> 439 builder_main(build_options.to_argv(model_dir, staging_dir))
    440 if not _is_ready(model_dir, staging_dir, build_options):
    441     raise RuntimeError(
    442         "the checkpoint-native builder did not produce a "
    443         f"complete runtime bundle for {model!r}")

File ~/aide/TensorRT-Edge-LLM/target/TensorRT-Edge-LLM/experimental/builder/cli.py:347, in main(argv)
    342 args = parser.parse_args(argv)
    343 logging.basicConfig(
    344     level=logging.DEBUG if args.verbose else logging.INFO,
    345     format="%(asctime)s %(levelname)s %(name)s: %(message)s",
    346 )
--> 347 _build(args)

File ~/aide/TensorRT-Edge-LLM/target/TensorRT-Edge-LLM/experimental/builder/cli.py:230, in _build(args)
    229 def _build(args: argparse.Namespace) -> None:
--> 230     from .core.builder import load_plugin_library
    232     plugin_path = args.plugin_path
    233     if plugin_path is None:

File ~/aide/TensorRT-Edge-LLM/target/TensorRT-Edge-LLM/experimental/builder/core/builder.py:31
     28 from typing import Optional, Tuple
     30 import numpy as np
---> 31 import tensorrt as trt
     33 from ..ops.backend import Net
     34 from ..ops.functional.attention import KV_PAGE_SIZE

ModuleNotFoundError: No module named 'tensorrt'

• Python API with tensorrt 11.2.1.2:

from experimental.server import LLM, SamplingParams

llm = LLM(
    model="Qwen/Qwen3.5-0.8B",
    # cache_dir="/data/edgellm-cache",
    max_input_len=4096,
    max_kv_cache_capacity=8192,
)
result = llm.chat(
    [{"role": "user", "content": "Explain paged KV caches."}],
    SamplingParams(max_tokens=128, temperature=0),
)
print(result.text)

Details

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[1], line 3
      1 from experimental.server import LLM, SamplingParams
      2 
----> 3 llm = LLM(
      4     model="Qwen/Qwen3.5-0.8B",
      5     # cache_dir="/data/edgellm-cache",
      6     max_input_len=4096,

File ~/aide/TensorRT-Edge-LLM/target/TensorRT-Edge-LLM/experimental/server/runtime/engine.py:736, in LLM.__init__(self, model, cache_dir, engine_cache_max_size_gb, clear_engine_cache, max_input_len, max_batch_size, max_kv_cache_capacity, draft_top_k, draft_step, verify_tree_size, build_options, speculative_config, context_cache_config)
    733 if options.spec_type != "none":
    734     options = replace(options, tree_base=tree_base)
--> 736 prepared = prepare_model(
    737     model,
    738     cache_dir,
    739     options,
    740     max_cache_size_bytes=int(engine_cache_max_size_gb * (1 << 30)),
    741     clear_cache=clear_engine_cache,
    742 )
    743 self._cache_dir = cache_root(cache_dir)
    744 self._model_dir = prepared.model_dir

File ~/aide/TensorRT-Edge-LLM/target/TensorRT-Edge-LLM/experimental/server/runtime/engine_build.py:439, in prepare_model(model, cache_dir, options, max_cache_size_bytes, clear_cache)
    433 build_options = replace(
    434     options,
    435     plugin_path=_resolve_plugin_path(options.plugin_path),
    436 )
    437 from experimental.builder.cli import main as builder_main
--> 439 builder_main(build_options.to_argv(model_dir, staging_dir))
    440 if not _is_ready(model_dir, staging_dir, build_options):
    441     raise RuntimeError(
    442         "the checkpoint-native builder did not produce a "
    443         f"complete runtime bundle for {model!r}")

File ~/aide/TensorRT-Edge-LLM/target/TensorRT-Edge-LLM/experimental/builder/cli.py:347, in main(argv)
    342 args = parser.parse_args(argv)
    343 logging.basicConfig(
    344     level=logging.DEBUG if args.verbose else logging.INFO,
    345     format="%(asctime)s %(levelname)s %(name)s: %(message)s",
    346 )
--> 347 _build(args)

File ~/aide/TensorRT-Edge-LLM/target/TensorRT-Edge-LLM/experimental/builder/cli.py:240, in _build(args)
    238 args = _copy_args(args, plugin_path=plugin_path)
    239 bundle, components = _resolve_build_selection(args)
--> 240 plugin_handle = load_plugin_library(plugin_path)
    241 plan = _build_plan(args, bundle, components)
    242 LOGGER.info("Building %s engines for %s: %s", len(plan),
    243             bundle.root_model_type, ", ".join(label for label, *_ in plan))

File ~/aide/TensorRT-Edge-LLM/target/TensorRT-Edge-LLM/experimental/builder/core/builder.py:267, in load_plugin_library(plugin_path)
    265 path = plugin_path or "build/libNvInfer_edgellm_plugin.so"
    266 if not os.path.exists(path):
--> 267     raise FileNotFoundError(f"plugin library not found: {path}")
    268 handle = ctypes.CDLL(path, mode=ctypes.RTLD_GLOBAL)
    269 logger.info("Loaded plugin library %s", path)

FileNotFoundError: plugin library not found: build/libNvInfer_edgellm_plugin.so

System information

• Platform: Nvidia Jetson Thor
• Software release: JetPack 7.1 R38.4
• CPU architecture: aarch64
• GPU compute capability: SM110
• • Total device memory: 128 GB
• Build typ: Release
• Library versions:
  • TensorRT Edge-LLM version: 0.10.1
  • CUDA: 13.0
  • TensorRT: 10.13.3
  • C++ compiler: GCC 13.3.0
• CMake options used:
  • CMAKE_TOOLCHAIN_FILE: cmake/aarch64_linux_toolchain.cmake
  • EMBEDDED_TARGET: jetson-thor
  • TRT_PACKAGE_DIR: /usr

[sylvesterkaczmarek]

I have a focused fix ready for the plugin-discovery portion of this issue, with regression coverage for default/custom build paths and precedence. The missing TensorRT Python dependency remains separate.

Happy to open the PR once this issue is approved for code review.