DeepSeek open-sources DeepInfer, but developers still need key serving components

DeepSeek released DeepInfer on October 20, 2026, opening the inference engine it says powers its large-scale production services. The Apache 2.0 release lets developers run DeepSeek-V4.1-Flash offline and inspect its optimization techniques. But the DeepInfer repository leaves out several components that DeepSeek says are necessary for a complete inference service.

In its launch post, DeepSeek described the release as a way to help the community reduce computing costs and energy use.

An engine, not a finished service

The release includes optimized computing kernels, the model runtime and sampling code, plus tools for spreading work across processors. Its supported-model list contains one entry: DeepSeek-V4.1-Flash.

Developers building a complete service still need a key-value, or KV, cache system; infrastructure that separates input processing from output generation; and load balancing across parallel copies of the model. DeepSeek lists all three as outside this release.

Drafting tokens, then choosing what to check

One central technique is DSpark, which DeepSeek describes as accelerating decoding with draft tokens—proposed pieces of output—and confidence estimates that schedule verification. Developers can enable variable-length drafting and set the maximum draft length, which defaults to five.

Another technique reduces decoder input-processing work by replaying only a bounded suffix. The default V4.1 settings retain 140 tokens, enough to cover the sliding attention window plus the maximum backtracking length.

DeepInfer also overlaps CPU batch scheduling with accelerator execution, enabled by default. The repository credits vLLM as its main original foundation, with additional inspiration from SGLang and TensorRT-LLM.

• Batch-invariant inference is designed to keep results identical when other requests in a batch change. It requires an explicit setting and is NVIDIA-only.
• Deterministic sampling targets identical outputs for identical inputs and sampling parameters. It requires batch invariance, disabled variable-length drafting and a fixed sequence seed, also on NVIDIA only.
• Structured outputs constrain generation to a regular expression, JSON schema or EBNF grammar—rules defining the response’s format.

Specific chips and large benchmark configurations

Supported hardware is limited to NVIDIA SM100 GPUs and Huawei Ascend 950 neural processing units. Builds require Linux, GCC 13 or later and Python 3.12, plus CUDA 13 for NVIDIA or CANN 9.2 for Ascend.

DeepSeek’s published benchmarks use four SM100 GPUs for prefill, or input processing, and 64 for decoding, or output generation. Each runs within one scale-up domain. The repository supplies a script for reproducing the tests.

For an initial offline run, the example downloads DeepSeek-V4.1-Flash-1017 into the Hugging Face cache. Developers can instead point it to a local copy through an environment variable.