DeepSeek Open-Sources DeepInfer, a Large-Scale Inference Framework
On October 20, DeepSeek open-sourced DeepInfer, its internal large-scale inference framework, along with deepseek-recipe, a token codec component. Built on this framework, V4.1 Flash reaches a single-cluster throughput of 155 trillion tokens per day. The framework supports distributed inference, speculative decoding, and structured output; it runs on both NVIDIA and Huawei Ascend platforms, and KVCache can be retained for over 72 hours.
DeepSeek Open-Sources DeepInfer, a Large-Scale Inference Framework
On October 20, DeepSeek open-sourced DeepInfer, its in-house large-scale inference framework, and simultaneously open-sourced deepseek-recipe, a token codec component. Built on this framework, V4.1 Flash reaches a single-cluster throughput of 155 trillion tokens per day.
The framework supports distributed inference, speculative decoding, structured output, and other capabilities, and can run on both NVIDIA and Huawei Ascend platforms; KVCache can be retained for more than 72 hours.
Github: deepseek-ai/DeepInfer | DeepSeek: Related article
