TensorRT Plugin Execution Provider

The TensorRT plugin Execution Provider (EP) implements the ORT EP plugin ABI introduced in ONNX Runtime 1.23.0, enabling NVIDIA TensorRT acceleration for ONNX models. It is migrated from the in-tree TensorRT EP and exposes the same set of provider options and features.

Unlike the legacy in-tree EP, this plugin EP is built as a standalone shared library (onnxruntime_ep_tensorrt.dll / libonnxruntime_ep_tensorrt.so) and does not need to be compiled together with ONNX Runtime. It only links against the ONNX Runtime shared library (onnxruntime.dll / libonnxruntime.so).

Supported platforms: Linux and Windows (Debug / Release).

Contents

Path	Description
CMakeLists.txt	Build configuration for the plugin EP and optional unit tests.
src/	C++ source code for the plugin EP.
tests/	GTest-based unit tests (basic inference, CUDA graph, engine caching, etc.).
python/	Python package and example usage script. See python/readme.md.
csharp/	C# NuGet package and sample application. See csharp/readme.md.

Prerequisites

• ONNX Runtime ≥ 1.23.0 (headers + shared library)
• NVIDIA TensorRT (10.x or 11.x)
• CUDA Toolkit (with nvcc)
• CMake ≥ 3.25

Build Instructions

On Windows

mkdir build && cd build
cmake -S ../ -B ./ -DCMAKE_BUILD_TYPE=Release ^
  -DTENSORRT_HOME=C:/path/to/TensorRT ^
  -DORT_HOME=C:/path/to/onnxruntime ^
  -DTRT_MAJOR_VERSION=11
cmake --build ./ --config Release

On Linux

mkdir build && cd build
cmake -S ../ -B ./ -DCMAKE_BUILD_TYPE=Release \
  -DTENSORRT_HOME=/path/to/TensorRT \
  -DORT_HOME=/path/to/onnxruntime \
  -DTRT_MAJOR_VERSION=11 \
  -DCMAKE_CUDA_ARCHITECTURES=80 \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc
cmake --build ./ --config Release

Note: ORT_HOME must contain include/ and lib/ subdirectories:

ORT_HOME/
├── include/
│   ├── onnxruntime_c_api.h
│   ├── onnxruntime_ep_c_api.h
│   ├── onnxruntime_cxx_api.h
│   └── ...
└── lib/
    ├── onnxruntime.dll (or libonnxruntime.so)
    ├── onnxruntime.lib (Windows only)
    └── ...

CMake Options

Option	Default	Description
ORT_HOME	auto-download	Path to ONNX Runtime package (include + lib).
TENSORRT_HOME	(required)	Path to TensorRT installation.
TRT_MAJOR_VERSION	10	TensorRT major version (affects library names).
CMAKE_CUDA_ARCHITECTURES	86	Target CUDA architectures (e.g., 80, 86, 89, 90).
onnxruntime_ep_tensorrt_BUILD_TESTS	OFF	Build unit tests (requires GTest, fetched automatically).
onnxruntime_ep_tensorrt_OBJECT_CACHE	ON	Use sccache/ccache if available.

Windows Runtime Deployment

On Windows, the build and install steps stage the TensorRT runtime DLLs next to onnxruntime_ep_tensorrt.dll. Deploy that complete directory. The EP loads dependencies from its own directory first, so the executable can reside elsewhere and no TensorRT PATH entry is required for a complete package. If a DLL is absent, the loader falls back to the normal Windows search path. An existing DLL that fails to load reports the attempted path and Windows error.

Windows ARM64 DLA also requires nvdla_compiler.dll from the selected TensorRT SDK and cudla.dll from the matching DTK EP package. Set -DCUDLA_DLL_PATH=/path/to/NV_TENSORRT_ARM64_EP_PACKAGE/cudla.dll to stage cuDLA; CMake also checks standard sibling package layouts. Missing cuDLA produces a configuration warning, and a DLA session fails if the runtime cannot be loaded. The compiler and cuDLA are preloaded only for DLA sessions. DLA transform DLLs are not included.

Usage

The plugin EP follows the ORT EP plugin ABI workflow:

1. Register the plugin EP library with the ORT environment.
2. Discover available EP devices.
3. Append the EP to session options with provider-specific options.
4. Create an inference session and run the model.
5. Unregister the library after all sessions using it have been released.

C/C++ API

#include "onnxruntime_cxx_api.h"

Ort::InitApi();
Ort::Env env(ORT_LOGGING_LEVEL_WARNING, "MyApp");

// 1. Register the plugin EP library
env.RegisterExecutionProviderLibrary("TRTPluginEP", "path/to/onnxruntime_ep_tensorrt.dll");

// 2. Find the EP device
auto all_devices = env.GetEpDevices();
std::vector<Ort::ConstEpDevice> trt_devices;
for (const auto& d : all_devices) {
  if (std::string(d.EpName()) == "TRTPluginEP") {
    trt_devices.push_back(d);
    break;
  }
}

// 3. Create session with EP options
Ort::SessionOptions session_options;
std::unordered_map<std::string, std::string> ep_options = {
  {"trt_fp16_enable", "1"},
  {"trt_engine_cache_enable", "1"},
  {"trt_engine_cache_path", "./cache"},
};
session_options.AppendExecutionProvider_V2(env, trt_devices, ep_options);

// 4. Run inference
Ort::Session session(env, "model.onnx", session_options);
auto outputs = session.Run(Ort::RunOptions{}, input_names, inputs, num_inputs, output_names, num_outputs);

// 5. Unregister (after all sessions are released)
session = Ort::Session{nullptr};  // release session first
env.UnregisterExecutionProviderLibrary("TRTPluginEP");

Python

Install the helper package (see python/readme.md for build instructions):

import numpy as np
import onnxruntime as ort
import onnxruntime_ep_tensorrt as tensorrt_ep

# 1. Register the plugin EP library
ep_lib_path = tensorrt_ep.get_library_path()
ep_name = tensorrt_ep.get_ep_name()     # "TensorRTPluginExecutionProvider"
ort.register_execution_provider_library(ep_name, ep_lib_path)

# 2. Select an EP device
all_devices = ort.get_ep_devices()
trt_devices = [d for d in all_devices if d.ep_name == ep_name]

# 3. Create session with EP options
sess_options = ort.SessionOptions()
ep_options = {
    "trt_fp16_enable": "1",
    "trt_engine_cache_enable": "1",
}
sess_options.add_provider_for_devices(trt_devices, ep_options)

# 4. Run inference
session = ort.InferenceSession("model.onnx", sess_options=sess_options)
output = session.run([], {"input": input_data})

# 5. Unregister
del session
ort.unregister_execution_provider_library(ep_name)

C#

Install the NuGet package (see csharp/readme.md for build instructions):

using Microsoft.ML.OnnxRuntime;
using Microsoft.ML.OnnxRuntime.EP.TensorRT;

// 1. Register the plugin EP library
var env = OrtEnv.Instance();
string epLibPath = TensorRTEp.GetLibraryPath();
string epName = TensorRTEp.GetEpName();
env.RegisterExecutionProviderLibrary(epName, epLibPath);

// 2. Find the EP device
OrtEpDevice? epDevice = env.GetEpDevices()
    .FirstOrDefault(d => d.EpName == epName);

// 3. Create session with EP options
using var sessionOptions = new SessionOptions();
sessionOptions.AppendExecutionProvider(env, new[] { epDevice },
    new Dictionary<string, string> {
        { "trt_fp16_enable", "1" },
    });

// 4. Run inference
using var session = new InferenceSession("model.onnx", sessionOptions);
using var results = session.Run(runOptions, inputNames, inputValues, session.OutputNames);

// 5. Unregister
env.UnregisterExecutionProviderLibrary(epName);

Quick Test with onnxruntime_perf_test

For a quick smoke test without writing code, use the ORT perf test tool:

onnxruntime_perf_test \
  --plugin_ep_libs "TRTPluginEP|path/to/onnxruntime_ep_tensorrt.dll" \
  --plugin_eps TRTPluginEP \
  -r 1 path/to/model.onnx

DLA Device Selection

DLA requires an ONNX Runtime SDK and runtime exposing ORT API 27 or later. Set ORT_HOME to that SDK when building; the default ORT 1.25 package supports GPU execution only. Build against the SDK matching the runtime you deploy. The platform must expose an NVIDIA NPU device (ACPI vendor NVDA) through ORT hardware discovery and a single CUDA-visible GPU. ORT does not need to report that GPU as a hardware device: DLA resolves CUDA ordinal 0 directly and caches its memory info independently of the GPU device cache. Ambiguous NPU-to-GPU associations on multi-GPU systems are not supported.

Select the DLA OrtEpDevice explicitly when appending the provider:

std::vector<Ort::ConstEpDevice> dla_devices;
for (const auto& device : env.GetEpDevices()) {
  if (std::string(device.EpName()) == "TRTPluginEP" &&
      device.Device().Type() == OrtHardwareDeviceType_NPU) {
    dla_devices.push_back(device);
    break;
  }
}
// Require one selected DLA device before appending the provider.
session_options.AppendExecutionProvider_V2(env, dla_devices,
                                          {{"trt_fp16_enable", "1"}, {"trt_dla_core", "0"}});

The DLA device supplies trt_dla_enable=1 and its associated CUDA device_id as defaults. Explicit overrides must agree with the selected device. DLA uses raw cudaMallocHost allocation for NPU HOST_ACCESSIBLE tensors; no EP arena is enabled. GPU devices retain their existing raw CUDA allocator.

Models must be supported by TensorRT's DLA backend. External DLA graph transforms are not integrated. CUDA graph capture is rejected for DLA sessions. The TensorrtDlaTest.* tests cover discovery, pinned allocations, repeated inference, and invalid options. They require DLA hardware and skip when no DLA device is available.

Provider Options

When built against TensorRT 11.4 or newer, the EP uses strongly typed networks for both GPU and DLA. The legacy trt_fp16_enable, trt_int8_enable, and trt_bf16_enable options are accepted but ignored. If any is enabled, the EP logs a warning during session creation with the TensorRT build version and the disabled option names. Precision must be expressed through ONNX tensor types and explicit quantization; INT8 calibration tables and layer precision overrides are not used in these builds. This policy is selected from the SDK headers at compilation, including SDK variants with the same version number.

Provider options are passed as key-value string pairs when creating a session. These are the same options supported by the legacy in-tree TensorRT EP.

Option	Type	Default	Description
device_id	int	0	CUDA device ID.
trt_max_partition_iterations	int	1000	Maximum iterations for TensorRT graph partitioning.
trt_min_subgraph_size	int	1	Minimum number of nodes in a subgraph to be accelerated by TRT.
trt_max_workspace_size	size_t	1073741824 (1 GB)	Maximum workspace size for TensorRT engine building.
trt_fp16_enable	bool	0	Enable FP16 precision.
trt_int8_enable	bool	0	Enable INT8 precision.
trt_bf16_enable	bool	0	Enable BF16 precision.
trt_int8_calibration_table_name	string	""	Path to INT8 calibration table.
trt_int8_use_native_calibration_table	bool	0	Use native TRT calibration table format.
trt_dla_enable	bool	0	Enable DLA (Deep Learning Accelerator).
trt_dla_core	int	0	DLA core to use.
trt_engine_cache_enable	bool	0	Enable TensorRT engine caching.
trt_engine_cache_path	string	""	Directory path for cached engines.
trt_engine_cache_prefix	string	""	Filename prefix for cached engines.
trt_dump_subgraphs	bool	0	Dump subgraphs to files for debugging.
trt_force_sequential_engine_build	bool	0	Build TRT engines sequentially (for debugging).
trt_context_memory_sharing_enable	bool	0	Share context memory across TRT subgraphs.
trt_layer_norm_fp32_fallback	bool	0	Force FP32 for LayerNorm (for accuracy).
trt_timing_cache_enable	bool	0	Enable timing cache to speed up engine building.
trt_timing_cache_path	string	""	Path for the timing cache file.
trt_force_timing_cache	bool	0	Fail if timing cache is not found.
trt_detailed_build_log	bool	0	Print detailed TRT engine build log.
trt_build_heuristics_enable	bool	0	Enable builder heuristics for faster build.
trt_sparsity_enable	bool	0	Enable structured sparsity.
trt_builder_optimization_level	int	3	TRT builder optimization level (0–5).
trt_auxiliary_streams	int	-1	Number of auxiliary streams (-1 = auto).
trt_tactic_sources	string	""	Tactic sources to enable/disable.
trt_extra_plugin_lib_paths	string	""	Semicolon-separated paths to extra TRT plugin libraries.
trt_profile_min_shapes	string	""	Min shapes for optimization profiles (e.g., input:1x3x224x224).
trt_profile_max_shapes	string	""	Max shapes for optimization profiles.
trt_profile_opt_shapes	string	""	Optimal shapes for optimization profiles.
trt_cuda_graph_enable	bool	0	Enable CUDA graph capture and replay.
trt_dump_ep_context_model	bool	0	Dump EPContext model with embedded engine.
trt_ep_context_file_path	string	""	Path for the EPContext model file.
trt_ep_context_embed_mode	int	0	EPContext embedding mode.
trt_weight_stripped_engine_enable	bool	0	Enable weight-stripped engine.
trt_onnx_model_folder_path	string	""	Path to original ONNX model folder (for weight-stripped engine).
trt_engine_hw_compatible	bool	0	Build HW-compatible engine.
trt_op_types_to_exclude	string	""	Op types to exclude from TRT acceleration.

EPContext Models

GPU and DLA sessions can save compiled TensorRT engines in an EPContext model. Configure generation with the generic ORT session settings:

session_options.AddConfigEntry("ep.context_enable", "1");
session_options.AddConfigEntry("ep.context_embed_mode", "1");
session_options.AddConfigEntry("ep.context_file_path", "deploy/model_ctx.onnx");

Create the output directory before creating the session. Mode 1 embeds the engine in the model. Mode 0 writes an external engine and records a path relative to the context model, preserving any trt_engine_cache_path subdirectory. Move the model and external engine directory together. Generation requires static input shapes or explicit TensorRT shape profiles.

To reload, create a session from the saved model and select the same GPU or DLA EP device and provider options. Leave ep.context_enable disabled. When loading an external context from memory, set ep.context_file_path to its model location so the EP can resolve engine paths. Embedded contexts need no external engine cache.

Generic settings override corresponding legacy trt_dump_ep_context_model, trt_ep_context_embed_mode, and trt_ep_context_file_path options. Unspecified settings retain the legacy values; the default embedding mode remains 0.

The GpuAndDla/TensorrtContextTest.* tests cover both storage modes, relocation, memory loading, cached-engine embedding, context metadata, and foreign EP ownership. DLA tests require ORT API 27 or later and an available DLA device.

Building and Running Tests

Unit tests cover basic inference, dynamic shapes, multi-threading, engine caching, EPContext models, CUDA graph capture/replay, and TRT plugin custom ops.

Build with Tests

mkdir build && cd build
cmake -S ../ -B ./ -DCMAKE_BUILD_TYPE=Debug \
  -DTENSORRT_HOME=/path/to/TensorRT \
  -DORT_HOME=/path/to/onnxruntime \
  -DTRT_MAJOR_VERSION=11 \
  -DCMAKE_CUDA_ARCHITECTURES=80 \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc \
  -Donnxruntime_ep_tensorrt_BUILD_TESTS=ON
cmake --build ./ --config Debug

Run Tests

Set the TRT_EP_LIBRARY_PATH environment variable to point to the built plugin EP library, then run via CTest or the test binary directly:

# Via CTest
cd build
export TRT_EP_LIBRARY_PATH=$(pwd)/libonnxruntime_ep_tensorrt.so   # or onnxruntime_ep_tensorrt.dll on Windows
ctest --output-on-failure

# Or run the test binary directly
./trt_ep_tests

Test Cases

Test	Description
FunctionTest	Basic inference with a simple Add model.
TestSessionOutputs_MultipleOutputs	Verifies correct output count for multi-output models.
TestSessionOutputs_UnusedNodeOutput	Handles models with unused node outputs.
DDSOutputTest	Inference with data-dependent shapes (DDS).
MultiThreadInference	Multi-threaded inference on a single session.
MnistModelTest	End-to-end inference on the MNIST model.
EngineCacheTest	Engine caching with trt_engine_cache_enable.
EPContextNode_ForeignSourceSkipped	Skips EPContext nodes from other EPs.
EPContextNode_NoSourceAttribute_BackwardCompat	Backward compatibility with legacy EPContext nodes.
SequentialRuns	Multiple sequential runs for stability.
DynamicInputShapes	Dynamic shape support with optimization profiles.
TRTPluginsCustomOpTest	TRT plugin custom op registration.
BasicCudaGraph	CUDA graph capture, replay, and in-place input update.
WithoutCudaGraph	Baseline inference without CUDA graph.
MultipleReplays	Repeated CUDA graph replays for stability.

License

This project is licensed under the MIT License. See LICENSE for details.