#1212th of 33 in Model serving
tensorrt-llm
NVIDIA inference engine for LLMs and diffusion models with custom CUDA kernels
- Stars
- 14.8k
- License
- custom license
- Last commit
- Oct 2026
- Last release
- Apr 2026
- Language
- Python
Missed the latest GitHub check; these facts may be older.
Overview
TensorRT LLM is NVIDIA's inference library for LLMs and visual generation (diffusion) models. It combines optimized kernels, a runtime and a customizable Python framework. The README's blog list covers speculative decoding, disaggregated serving, expert parallelism, sparse attention, KV cache work and DeepSeek on Blackwell. The excerpt gives no install steps or hardware minimums.
Who it is for: Teams serving LLMs at scale on NVIDIA datacenter GPUs
Strengths
- Kernel and parallelism work for Blackwell, NVL72 and NVLink-based expert parallelism
- Documents speculative decoding, disaggregated serving, guided decoding and KV cache reuse
- Day-0 support for models such as GPT-OSS and EXAONE 4.0 is documented
- Extensible Python framework with architecture, performance and quick-start docs
Weaknesses
- Tied to NVIDIA GPUs, CUDA 13.4.1 and PyTorch 2.14.0
- Current version shown is a release candidate (1.4.0rc1)
- Detected license is NOASSERTION, while the README badge says Apache 2
- README excerpt lacks install steps, memory requirements and API port
What it needs
- GPU required
- Needs CUDA 13.4.1, PyTorch 2.14.0, Python 3.10 or 3.12
- Models: LLMs, DeepSeek, Llama, GPT-OSS, EXAONE
Also in Model serving
See all 33| Rank | Project | Score |
|---|---|---|
| 1 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 83 out of 100 |
| 2 | llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API | 81 out of 100 |
| 3 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 80 out of 100 |
| #4 | vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching | 76 out of 100 |
| #5 | colibriC inference engine that runs huge MoE models by streaming experts from disk | 73 out of 100 |