DeepSeek open-sourced the engine and the prompt recipe behind its API, and one of the two chips it runs on is Huawei's
DeepSeek published the inference engine that serves V4.1-Flash in production and the encoding library that makes a copy behave like its API, and the same engine builds for Nvidia's SM100 or Huawei's Ascend 950.
DeepSeek's V4.1-Flash is a 763-billion-parameter model whose weights have been downloadable under an MIT licence since 17 October. 1 The software that turns those weights into a service stayed inside the company.
DeepSeek published that software this morning. DeepInfer is the inference engine that serves V4.1-Flash in DeepSeek's own production service, released under Apache 2.0. 2 deepseek-recipe is the library that turns an API request into the exact prompt and token sequence the official API uses, under MIT. 3
The announcement runs to four posts and closes on a line: open models, open kernels, open inference. DeepSeek says the inference code for future models will land in the same repository. 4
Two variables decide what a model costs you and whether your copy behaves like the original: how efficiently the engine turns electricity into tokens, and how faithfully the request is encoded into text the model reads. Both were private until this morning.
What shipped
| Component | What it is | Licence | What changes |
|---|---|---|---|
| DeepInfer | The engine behind DeepSeek's production service: kernels, parallelism, scheduling, sampling | Apache 2.0 2 | The serving path is now readable and forkable |
| deepseek-recipe | Request conversion, prompt rendering, tokenisation, output parsing | MIT 3 | A copy of the model can be fed the same input as the official API |
| V4.1-Flash weights | 763B-parameter model, 1M-token context | MIT 1 | Already free; the two rows above are the gap that remained |
One row sets your bill, the other sets whether two deployments of the same weights return the same text.
The engine is where your token bill is decided
Between an HTTP request and a GPU sits software that settles three questions. How many requests share one GPU at a time. How much memory each conversation is allowed to keep. Which low-level arithmetic routines run. 2
Those three answers set the speed a user feels and the price you pay per token. The weights set the ceiling on quality; the engine decides what you spend to reach it.
Two providers running identical weights can differ in throughput, in price, and in the text they return. A contract that names only the model leaves the two consequential variables with the vendor.
Loading content card…
Why this one is hard: 763 billion parameters and a million tokens
V4.1-Flash is a mixture-of-experts model. The published benchmark headers name the shape DeepSeek tested: 384+32 experts for prefill, 384+64 for decode, at a 131,072-token context against the model's 1M-token window. 2
Each token activates a fraction of those experts, and which experts a batch hits shifts second by second. A hot expert becomes a bottleneck while the others idle, so the engine carries an expert-parallel load balancer to even the queue. 2
Long context adds a second cost at the front of every request. Before the model answers, it must read the prompt, and a million tokens of reading costs throughput as much as memory. DeepSeek's fix is bounded replay: the decoder keeps a short suffix, 140 tokens by default, and rebuilds its sliding window from there. 2
Decoding has its own economy. DSpark drafts several tokens ahead and a confidence estimate decides how many to verify, with a default draft length of five. 2 One published run fixes the draft at three tokens and a 75% acceptance rate, which works out to 3.25 tokens cleared per verification step. 2
That curve carries the commercial argument by itself. Run the arithmetic on the endpoints and a GPU-hour buys about 9.4 million output tokens at the left end and about 93.6 million at the right. The right end comes with a user who waits three times as long for each token.
Read the two charts together and the million-token context gets a price. Filling the window is cheap per token and slow per request, which is why the context number in a model card says little about the latency your users will feel.
The switch that makes answers repeatable
The same prompt can return different text on different runs, and the cause is mechanical. Floating-point addition depends on order, the engine batches your request alongside other customers' requests, and batch size changes the order in which numbers are added. 5 Your neighbour's traffic moves your output.
DeepSeek ships two settings against it. Batch-invariant inference keeps results identical across batch compositions. Deterministic sampling with a fixed seed makes identical inputs produce identical outputs. 2
The price is documented. vLLM warns that batch invariance can reduce performance and calls the trade intentional. 6 Fixing it inside the kernels also ties determinism to kernel design, so LLM-42 proposes a verify-and-rollback scheduler that charges overhead only to the traffic that needs it. 5
Four things break when output follows traffic:
- An eval that passed last week fails for reasons outside your code.
- A regression you blame on a prompt change came from a busy afternoon.
- An audit of what the model told a customer cannot be reproduced.
- A two-vendor comparison measures their batching as much as their models.
Both settings are recorded as NVIDIA-only in this release, so the Ascend build exposes the tuning controls without the reproducibility ones. 2
The encoding decides what the model sees
Most open models ship a Jinja chat template: the small program that turns a list of messages into the prompt string the model receives, and the place where two teams running the same weights quietly diverge. The V4.1-Flash model card ships a Python reference implementation instead, and points production users at deepseek-recipe. 1 That library takes Chat Completions, Messages or Responses requests, converts them into one conversation format, renders the conversation into a V4.1 prompt, encodes it into token IDs, and parses the output back into the caller's protocol — reasoning content, tool calls, JSON-schema and regex constraints, stop sequences. 3
When a request sets no output limit, the library supplies one: 8,192 tokens outside thinking mode, 65,536 inside it, and 131,072 at maximum effort. 3 V4.0 does not take images, so the library swaps every image in a V4 request for the literal text [Unsupported Image]. 3
Its documentation also carries a security note. The image fetcher connects to any address by default, so a service that pulls URLs out of untrusted requests should install a destination filter. 3
Teams that have compared two providers on one model name have usually been comparing this layer: a different template, a different tool-call parser, a different way of counting reasoning tokens. DeepSeek's claim is that a community deployment can now be given the same behaviour as the official API. 4
The boundaries of the release
The released engine covers the middle of that diagram. The README lists what stayed behind: no KV-cache system, no prefill/decode disaggregation, and no load balancing across data-parallel replicas. 2
The release covers offline inference for a single model, and the build needs Linux, Python 3.12, GCC 13 or newer, plus CUDA 13 on Nvidia or CANN 9.2 on Ascend. 2 A team can wire this into a service. A team should not expect a turnkey endpoint.
DeepSeek published throughput-latency curves with no line for vLLM or SGLang, the two engines most teams already run. The repository says DeepInfer was originally built on vLLM. 27
The second chip carries the strategy
One environment variable selects the silicon:
DEEPINFER_ACCELERATOR, set to nvidia or ascend. 2 The supported hardware list is Nvidia's SM100 and Huawei's Ascend 950. 2The demand side explains the timing. ByteDance, Alibaba and Tencent have ordered more than three million of Nvidia's export-compliant RTX PRO 5500 cards at about $11,400 each, over $35 billion combined, and sources told Reuters the cards are mainly for inference. 8
Supply tightens it further. Nvidia has told partners it can produce no more than three million of those cards globally in 2027, one source said buyers may receive 5% to 10% of what they asked for, and orders from China have passed six million. 8
A buyer in China now has one engine that runs whichever silicon the export rules allow through. For everyone else it says where the serving layer is heading.
Amazon spent the same 24 hours on the mirror image. TorchNeuron, now in public beta and released as open source, runs PyTorch code on Trainium after a configuration change; Amazon's Colin Brace put it as knowing PyTorch meaning knowing Trainium. 9
Google is building TorchTPU for the same job, Microsoft is moving services onto its Maia chips, and more than 30 companies tested TorchNeuron in private beta, with Deepgram reporting about 60% more audio processed per dollar. 9
Nvidia's position rests partly on CUDA, and the cost of moving code is what has kept buyers in place. 9 Both camps lowered that cost in the same 24 hours.
What to do with it
| If this is your situation | Do this | Window |
|---|---|---|
| You buy inference from someone else's endpoint | Write deployment identity into the next contract: engine and version, prompt-encoding version, sampling parameters, and whether batch invariance is on | Next renewal |
| Your eval results drift between runs | Ask for deterministic sampling with a fixed seed in the eval harness, and budget for lower throughput while it is on | This sprint |
| You are costing a self-host move | Scope it as an engineering project of weeks, with KV cache, disaggregation and replica load balancing on the list | Before the business case |
| Latency and cost both matter to you | Measure your own concurrency against the decode curve before arguing about price per million tokens | This week |
The first row changes a contract. The rest change how you measure.
To test the claim rather than read it, the smallest useful pilot fits in a week. Rent an SM100 node, run the repository's offline inference example, send one fixed prompt set through DeepInfer and through DeepSeek's hosted API, and diff the outputs with deterministic sampling on, since sampling noise otherwise swamps the comparison. 12
Where the claim stops
| What today establishes | What it leaves open | Your move |
|---|---|---|
| A production engine for a frontier open model is readable and forkable | Whether it beats vLLM or SGLang on your traffic; DeepSeek's curves carry no comparison line | Ask for a bake-off on your own prompts |
| DeepSeek aligned its encoder and engine with its own API | Whether a community deployment now matches it; that is DeepSeek's claim, with no independent test yet | Run a fixed prompt set through both and diff the outputs |
| One engine runs on Nvidia and Huawei silicon | How the two compare; every published curve is on SM100 | Benchmark the hardware you would actually buy |
| A million-token window comes with published prefill costs | That long context is cheap; filling the window costs about 733 ms per 65,536-token step in DeepSeek's run | Price your real prompt lengths |
What this morning establishes: for one frontier model, the software that turns open weights into a competitive service is now public, and it runs on two vendors' silicon. What it leaves open: whether a community copy actually matches the API in practice, what the engine does against vLLM and SGLang on live traffic, and how the Huawei path performs at all. The next lab to publish its engine settles whether this is a DeepSeek habit or an industry one.
References
- 1DeepSeek-V4.1-Flash-1017
huggingface.co
- 2
- 3deepseek-recipe
github.com
- 4
- 5
- 6Batch invariance
docs.vllm.ai
- 7DeepSeek open-sources DeepInfer
aicoder.com
- 8
- 9
This story was produced automatically by a channel. One sentence is all it takes for Neodrop to keep producing for you.