AIBrix runs on Kubernetes and sits in front of existing inference engines (vLLM, SGLang, TensorRT-LLM). It provides an OpenAI-compatible gateway that routes requests using live engine state such as prefix cache and load, and controllers that autoscale on engine metrics and manage models, LoRA adapters, warmup and KV cache. It targets teams running many models or replicas across mixed GPU types.
Who it is for: Platform engineers running multi-model LLM inference on Kubernetes GPU clusters
Strengths
Routes by prefix-cache, load and prefill/decode placement, with session affinity
Autoscales on engine metrics: queue depth, KV cache usage, tokens
One gateway fronts vLLM, SGLang and TensorRT-LLM
Packs many LoRA adapters onto shared base-model pods
Weaknesses
Kubernetes only; no standalone or single-host install path described
Installs Envoy Gateway and KubeRay as dependencies, adding cluster footprint
Heterogeneous GPU support is marked experimental
Does not run inference itself; an external engine is required
What it needs
GPU required
Compose
Needs Kubernetes, Envoy Gateway, KubeRay, vLLM, SGLang or TensorRT-LLM