DeepInfer is the inference framework DeepSeek uses for its production services, released here with offline inference only. It runs DeepSeek-V4.1-Flash with custom kernels, attention data parallelism, MoE expert parallelism with load balancing, draft-token decoding, and structured outputs. A complete serving stack still needs a KV cache system, prefill/decode disaggregation and load balancing, which are not included.
Who it is for: Engineers running DeepSeek-V4.1-Flash on SM100 or Ascend 950 hardware
Strengths
Batch-invariant inference and deterministic sampling with a fixed seq_seed (NVIDIA only)
Structured outputs constrained by regex, JSON schema or EBNF grammar
Supports both NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs
Asynchronous batch scheduling is on by default; benchmark sweep script is included
Weaknesses
Supports only one model: DeepSeek-V4.1-Flash
Offline inference only; no HTTP server or API in this release
KV cache, disaggregation and load balancing are not included
Needs Python 3.12, GCC 13+, CUDA 13 or CANN 9.2, and a source build
What it needs
GPU required
Needs CUDA 13, CANN 9.2, GCC >= 13, Python 3.12, Hugging Face model download