Chinese artificial intelligence startup Hangzhou DeepSeek Artificial Intelligence Basic Technology Research Co. Ltd. has open-sourced DeepSeek DeepInfer, making the inference framework behind its own production services available to the public. DeepInfer runs DeepSeek-V4.1-Flash, the lab’s fastest model, loading the updated 1017 build that DeepSeek released over the weekend. DeepSeek said in a deployment document that one production cluster serves 155 trillion tokens a day while generating 38 dollars per graphics processing unit every hour at off-peak rates [1].
Origins of DeepSeek DeepInfer
DeepSeek framed low serving costs as “the core problem an inference framework has to solve” when publishing the code under the Apache 2.0 license on GitHub. DeepInfer was originally built on vLLM, but DeepSeek altered the engine to fit its own cluster infrastructure. Back in April 2025, DeepSeek kept the engine private because internal cluster tools were tied to private systems. The research team lacked engineering bandwidth [1].
At that time, DeepSeek said that maintaining a large open-source project exceeded what “a small research team focused on developing better models” could manage. Instead of releasing the full engine at once, DeepSeek shared smaller components over several months, including the FlashMLA decoding library, DeepGEMM, and the DeepEP communication kernel [1]. The current GitHub release unifies those pieces, pairing them with custom TileKernels, DeepSelect, and TileLang routines. DeepSeek also packaged the engine with deepseek-recipe, a companion library published under the MIT license that converts prompt requests into token sequences matching the official DeepSeek API [3]. Matthias Bastian reported for The Decoder that deepseek-recipe helps self-hosted deployments duplicate the exact token handling of the hosted platform. On social platform X, DeepSeek said the codebase aims “to help the community lower compute costs, reduce energy use, and make AI more affordable” [2].
The GitHub repository marks a clear shift in how DeepSeek distributes its serving stack. DeepSeek DeepInfer now hosts inference logic for DeepSeek-V4.1-Flash, and DeepSeek said that code for future models will go into the same public repository. That setup gives outside developers access to the same pipeline that powers DeepSeek chat and coding tools [1].
Speculative Decoding and Attention Replay
DeepInfer speeds up token output by running a speculative decoding system known as DSpark. During each generation step, a draft model proposes five tokens while a confidence head predicts which candidates the main model will accept. Verification compute runs only on the likeliest candidates, saving time by ignoring weak proposals [1]. DeepSeek reported that this dynamic schedule cuts verified draft tokens by about 30 percent, reducing checks from 5.00 tokens down to between 3.2 and 4.4 tokens. Accepted length averaged 3.20 tokens. In production benchmarks, DSpark delivered a 4 percent to 7 percent throughput gain at 60 sequences per rank, although it ran 1.5 percent slower than fixed 5-token verification at 16 sequences per rank [3].
Prompt processing relies on an attention optimization called sliding-window attention bounded replay. During cache-hit scenarios, DeepInfer calculates solely the trailing 128 tokens of attention context instead of running an entire prompt back through the model. DeepSeek said this bounded replay nearly halves total prefill compute in decoder layers, keeping recomputation costs constant even on long prompts that approach the DeepSeek-V4.1-Flash 1-million-token context limit [1, 3].
Self-Reported Benchmarks on Blackwell Chips
All published benchmarks ran on Nvidia chips [1]. DeepSeek ran its evaluation using DeepSeek-V4.1-Flash with a context window of 131,072 tokens. DeepSeek split the workload between compute nodes, assigning four SM100 Blackwell GPUs to prompt prefill and 64 SM100 GPUs to token decoding. The evaluation tested both raw generation speed and multi-user scaling behavior [3].
In decoding tests with 64 expert parallel ranks and 75 percent token acceptance, output passed 25,000 output tokens per second per GPU at 256 requests per rank, generating nearly 100 tokens per second for each user. Eight requests passed 300 tokens per second. Prefill benchmarks on four GPUs reached roughly 60,000 non-cached tokens per second per GPU at 4,096 tokens per rank with 140-token bounded replay. That throughput approached 90,000 tokens per second when input batches grew to 65,536 tokens, running at about 730 milliseconds per step. DeepSeek’s published Pareto charts also showed that 64 expert ranks outperformed eight ranks at every tested batch volume, confirming the value of wide parallelism [3].
DeepInfer overlaps CPU batch scheduling with GPU execution so Nvidia chips avoid sitting idle between generation cycles. Duncan Riley covered the release for SiliconANGLE, founded by John Furrier, noting that two determinism features—batch-invariant inference and deterministic sampling—function only on Nvidia hardware, while structured output generation supports regular expressions, JSON schemas, and context-free grammars across all supported silicon [1, 3].
Deploying DeepInfer in Offline Clusters
Cluster operators need specific accelerator hardware to run DeepSeek DeepInfer, as the code targets only Nvidia SM100 Blackwell GPUs and Huawei Technologies Co. Ascend 950 neural processing units. Hopper architectures are not on the list [1]. Host systems need Linux, GCC 13 or later, Python 3.12, and either CUDA 13 or CANN 9.2 from Huawei Technologies Co. [3]. DeepInfer Ascend compatibility follows a Sept. 30 release in which DeepSeek adapted TileLang for Huawei chips with help from the chipmaker. In September, Bloomberg reported that DeepSeek aims to install more than 160,000 Ascend 950DT accelerators inside an upcoming computing facility in Inner Mongolia [1].
Despite its high throughput, DeepInfer does not offer an online web service out of the box. Offline inference was the only mode released. DeepInfer lacks an OpenAI-compatible HTTP server and dynamic request routing [3]. DeepSeek said its key-value cache system and load balancers stay tied to internal infrastructure, though a Zhihu post explains their design. Splitting prefill and decode across separate machines is left to operators, although DeepSeek said most of the engine logic for it is already in place. An efficient setup still needs 64 GPUs for prefill and 64 GPUs for decode [1, 3]. DeepSeek said its private FlashMLA decoding kernel is 5 percent to 10 percent faster than the public code, but estimated that the total end-to-end performance gap across real workloads reaches 5 percent or less [3].
Market Impact on Western AI Labs
DeepSeek DeepInfer creates fresh cost pressure on commercial AI labs in the United States. DeepSeek models are already known for low pricing, and sharing DeepInfer lets third-party hosting firms run them at similar efficiency. Matthias Bastian wrote in The Decoder that the engine intensifies margin challenges for Western frontier labs such as OpenAI and Anthropic [2]. Future model code will share this repository. While alternatives like vLLM and SGLang support hundreds of architectures, DeepInfer focuses its optimizations on DeepSeek-V4.1-Flash [1, 3].
The open-source release also arrived during heightened political tension between China and the United States [2]. One day before DeepInfer appeared on GitHub, American civilian agencies faced a federal directive from the Department of Homeland Security ordering teams to purge DeepSeek tools and dependent external integrations. DeepSeek moved ahead with its open release despite those bans, distributing software that allows developers worldwide to inspect its serving architecture [1]. Matthias Bastian noted that amid the AI race between China and the United States, lagging growth targets for Western firms present wider economic risks [2]. With 155 trillion tokens served daily on internal clusters, DeepSeek offers its efficiency blueprints to developers who have compatible silicon [1, 3].
- ONLINE NEWS Riley, D. (2026, October 20). DeepSeek open-sources DeepInfer, the inference framework behind its production services. SiliconANGLE. [Article Link]
- ONLINE NEWS Bastian, M. (2026, October 20). Deepseek open-sources the inference engine that powers its fastest and cheapest AI model. The Decoder. [Article Link]
- ONLINE NEWS DeepSeek Open-Sources DeepInfer, the Inference Framework Behind DeepSeek-V4.1-Flash. (2026, October 20). MarkTechPost. [Article Link]
