DeepSeek Open Sources DeepInfer, the Inference Engine Behind Its V4.1 Flash Service

DeepSeek on October 20, 2026 released DeepInfer, the inference framework it says powers the production service behind its DeepSeek V4.1 Flash model, as open source code under the Apache License 2.0. The public version handles offline inference on Nvidia and Huawei Ascend chips, while the components that tie a full cluster together stay internal, according to the project's README.

An inference framework is the serving software that runs a trained model for users, batching requests and generating tokens. DeepSeek already publishes the weights of its open weight models, most recently V4.1 Flash 1017 on October 17, 2026, so DeepInfer adds the software layer DeepSeek uses to serve those weights at scale.

What did DeepSeek release on October 20?

DeepSeek released DeepInfer on October 20, 2026 as a GitHub repository under the Apache License 2.0, describing it as the high performance framework behind its large scale production services. The only supported model is DeepSeek-V4.1-Flash, and the default checkpoint is the October 17 update.

The README lists the main features. DeepInfer ships DeepSeek's own optimized kernels (FlashMLA, DeepGEMM, DeepEP and others), expert parallelism for the model's mixture of experts layers with load balancing between experts, and a speculative decoding method called DSpark. Speculative decoding drafts several tokens ahead and then checks them in one pass; DeepSeek says DSpark uses confidence estimates to decide which drafted tokens are worth checking. The framework also offers structured outputs (JSON schema, regex or grammar), deterministic sampling and batch invariant inference on Nvidia hardware, which keeps results identical regardless of which other requests share a batch.

DeepSeek says DeepInfer was "originally built primarily on" vLLM, the widely used open source serving engine, with ideas taken from SGLang and Nvidia's TensorRT-LLM. In a post on X, the company said it is sharing its production stack to help lower compute costs and energy use.

What does DeepInfer leave out?

DeepSeek's README says three parts of a complete inference service are not in this release: a KV cache system, disaggregated serving that splits stages such as prefill and decode onto separate machines, and load balancing across data parallel replicas. A KV cache stores the intermediate results of text the model has already read, so repeated prompts do not have to be processed again.

That gap sits next to the wording of DeepSeek's X post, which calls DeepInfer "now fully open source". The deployment Q&A in the same repository is more specific: the KV cache management and load balancing code is tightly coupled to DeepSeek's internal infrastructure and is "not yet fully open-sourced", though a blog post on Zhihu describes their design.

The Q&A also says the public versions of some kernels differ from internal ones. The public FlashMLA lacks one hand tuned optimization, making its decoding kernel 5% to 10% slower, while end to end performance is at most 5% apart, according to DeepSeek.

What hardware does DeepInfer need?

DeepInfer runs on Nvidia SM100 GPUs and Huawei Ascend 950 NPUs, according to the README, and DeepSeek's deployment Q&A puts the smallest efficient setup at 64 GPUs for prefill plus 64 for decode. Prefill is the stage that reads the prompt; decode generates the answer token by token.

Running on Huawei's chips matters for DeepSeek's own plans. The company is building a data center with at least 160,000 AI chips from Huawei and is working on its own inference chip, The Decoder reported on October 6, 2026. A framework with Ascend support written into its first public release fits that shift away from relying on Nvidia alone.

DeepSeek's charts, measured on 4 GPUs for prefill and 64 for decode, show decode output of about 26,000 tokens per second per GPU when each user gets about 100 tokens per second, and per user speeds above 300 tokens per second at low load. Prefill reaches roughly 90,000 input tokens per second per GPU in the largest batch setting shown. These are DeepSeek's own measurements, read from its charts.

How big is the service DeepSeek runs on it?

DeepSeek says a single cluster running DeepInfer currently serves 155 trillion tokens per day, with overall revenue of $38 per GPU hour at off peak prices. The company did not publish the size of that cluster or how many such clusters it operates.

The Q&A describes a system built around three goals: support for chips from several vendors, ordinary 8 GPU machines for prefill with 64 to 128 GPU supernodes for decode, and keeping cached context for more than 72 hours. DeepSeek says its KV cache storage holds about 100 pebibytes on SSDs, with cached data kept for 3 days for agent traffic and 7 days for chatbot traffic.

DeepSeek also pointed to deepseek-recipe, a library that translates common API request formats into DeepSeek prompts and tokens, so self hosted deployments behave like the official API, the company says. The company says inference code for future DeepSeek models will be published in DeepInfer as well.

The release came one day after the U.S. Department of Homeland Security ordered federal civilian agencies to remove DeepSeek products, in a notice filed on October 19, 2026 that also covers third party products built on DeepSeek implementations. Code that makes self hosting easier does not change that order for agency suppliers.

What we don't know yet

• Whether DeepSeek will publish the KV cache and load balancing code, and on what timeline.
• How DeepInfer performs against vLLM and SGLang serving the same model on the same hardware; DeepSeek's charts show no rival engines.
• Which future DeepSeek models DeepInfer will support first, and when.
• Whether major cloud providers will adopt DeepInfer for their hosted DeepSeek endpoints.