#5454th of 71 in Model serving
ctranslate2
C++ and Python inference engine for Transformer models on CPU and GPU
Documentation ↗
(opens in a new tab)Website ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 4.8k
- License
- MIT
- Last commit
- Nov 2026
- Last release
- Nov 2026
- Language
- C++
4,403 on 22 Apr 20264,751 on 11 Nov 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
CTranslate2 is a C++ runtime with Python bindings that runs converted Transformer models (NLLB, Whisper, Llama, Mistral, T5, BERT and others) using quantization, layer fusion and batch reordering. Models are converted from OpenNMT-py, OpenNMT-tf, Fairseq, Marian or Transformers into its own format. It is a library rather than a server; faster-whisper is built on it.
Who it is for: Engineers embedding fast Transformer inference in C++ or Python apps
Strengths
- Supports FP16, BF16, INT16, INT8 and AWQ INT4 quantization
- README benchmarks show lower memory than PyTorch and Marian on the same models
- One binary selects CPU backend (MKL, oneDNN, OpenBLAS, Ruy, Accelerate) at runtime
- Tensor parallelism splits large models across multiple GPUs
Weaknesses
- Models must be converted first; no direct loading of original checkpoints
- Library only; an HTTP API needs the separate ctranslate2-web-server project
- Limited to listed Transformer architectures; new models need explicit support
- Apple Metal backend is experimental and never chosen by device=auto
What it needs
- GPU optional
- Docker
- Models: Transformer base/big, NLLB, M2M-100, BART, T5
What changed
From the release notes. No flag means the note does not say so, not that the upgrade is safe.
Also in Model serving
See all 71| Rank | Project | Score |
|---|---|---|
| 1 | llama.cppC/C++ inference engine for GGUF LLMs and VLMs with an OpenAI-compatible server | 94 out of 100 |
| 2 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 92 out of 100 |
| 3 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 91 out of 100 |
| #4 | vLLMInference engine serving LLMs with PagedAttention and an OpenAI-compatible API | 88 out of 100 |
| #5 | strataRuns a 125B-parameter MoE model on one consumer GPU plus system RAM | 88 out of 100 |