#3838th of 47 in Model serving
vllm-omni
Serving engine for omni-modality, diffusion, TTS and robot-policy models, built on vLLM
- Stars
- 7.1k
- License
- Apache-2.0
- Last commit
- Oct 2026
- Last release
- Sep 2026
- Language
- Python
Missed the latest GitHub check; these facts may be older.
Overview
vLLM-Omni extends vLLM beyond text autoregressive models to Diffusion Transformers and other non-autoregressive models, handling text, image, audio, video and action inputs and outputs. It runs multi-stage pipelines with disaggregated stages and exposes an OpenAI-compatible API server, including full-duplex realtime serving with streaming audio. Releases track even-numbered upstream vLLM versions.
Who it is for: Engineers serving multimodal, diffusion or TTS models on GPU or NPU clusters
Strengths
- Covers omni, TTS, diffusion (image/video/audio) and robot-policy models in one runtime
- Reuses vLLM KV cache management; supports tensor, pipeline, data and expert parallelism
- OpenAI-compatible server with streaming outputs and full-duplex realtime audio API
- Hardware coverage listed as CUDA, ROCm, MUSA, NPU and XPU; Apache-2.0
Weaknesses
- Version and model support tied to specific vLLM releases; needs matching upstream vLLM
- README gives no RAM or VRAM requirements; check per-model recipes
- Setup and model coverage live in external docs, not the README
- Multi-stage disaggregated pipelines add operational complexity over a single-model server
What it needs
- GPU required
- Needs vLLM
- Models: Qwen3-Omni, MiniCPM-o 4.5, Cosmos3, BAGEL, Qwen3-TTS
Also in Model serving
See all 47| Rank | Project | Score |
|---|---|---|
| 1 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 83 out of 100 |
| 2 | llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API | 81 out of 100 |
| 3 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 80 out of 100 |
| #4 | strataRuns a 125B-parameter MoE model on one consumer GPU plus system RAM | 79 out of 100 |
| #5 | vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching | 77 out of 100 |