Inference engineering is the product: Baseten on serving models under real traffic

A model checkpoint is only the starting material. In the Latent Space conversation with Baseten's Philip Kiely and Ali Taha, the interesting work begins after the weights are released: routing requests to the right replica, moving KV caches between GPUs, deciding what to quantize, and debugging failures that appear only under real traffic. Their argument is simple: inference has become its own engineering discipline, with different costs and failure modes from model training. 1

Kiely wrote Inference Engineering, a book about turning trained weights into a fast, reliable and affordable product. Taha is a Baseten inference engineer. The 1-hour-41-minute discussion is technical, but its practical lesson is broader: when an AI product is slow or unreliable, the problem may sit in the serving system rather than in the model itself. 1

Swyx opens with a deliberately large request: what happens when a 200,000-token prompt enters a production inference system? Kiely's answer starts before the model generates anything. The system first asks whether it has seen all or part of the request before. If it has, cache-aware routing can send the request to a replica that already holds useful context. That avoids repeating the expensive prefill step for the cached portion. 1

If the prompt is new, one group of GPUs can process the input and build the KV cache while another handles decode, the repeated passes that produce output tokens. The split matters because prefill and decode want different things from the hardware. A long prompt creates a large burst of input work; generation then becomes a sustained, sequential process. Treating both as the same workload leaves capacity stranded in one part of the system while the other is busy.

A small draft model can sit in front of the large model as a speculative decoder. It proposes several tokens, and the main model verifies them in one pass. When the draft model matches the traffic, the system skips some expensive sequential work. When it was trained for coding but receives an unrelated request, its guesses are less useful and the speedup falls. This is why the same model can have different economics on a shared endpoint and a dedicated deployment. 1

The conversation draws a line that is easy to miss in model-release coverage. Getting a new checkpoint to emit a token through an open-source engine such as vLLM or SGLang is one problem. Building a production API around it is another. A provider still has to redo quantization, train or adapt a speculative decoder, load the model across its infrastructure, test its failure modes, and tune the runtime for the new architecture. 1

GLM-5.2 is their example of why this work does not become routine just because models are open. Its sparse-attention component required runtime support. Baseten also experimented with grafting Kimi's vision encoder onto GLM-5.2 rather than altering the language-model weights. The encoder and base model stayed frozen; the team trained a small projector to connect the image representation to the language model. When no image is present, the vision path can be skipped, leaving the text model's behavior intact. 1

That is a useful picture of open-weight competition. The unit of progress is no longer always a brand-new model. Engineers can combine a vision encoder from one system, attention ideas from another, and a serving runtime optimized for a particular GPU. The result may be a better product even when none of its individual components is new.

The speakers treat speed as a stack of conditional gains, not a single benchmark number. Quantization can reduce memory and increase throughput, but it is the main lossy optimization in their account. KV caching and speculative decoding are intended to preserve the model's behavior, while disaggregation and better kernels change how efficiently the hardware is used. The gains depend on prompt length, output length, load, batch size, GPU type and parallelism strategy. 1

Their rough order of magnitude is intentionally wide: a basic deployment for a large model might run at 30–50 tokens per second, while a carefully optimized system can sometimes reach four to six times that and, in aggressive cases, approach 10×. Those figures are not a universal rate card. They are a warning against comparing provider dashboards without matching the workload and defining whether "tokens per second" means throughput or inter-token latency. 1

The more surprising point is that quantizing more layers can sometimes preserve fidelity better than quantizing fewer. Taha describes selecting layers whose errors cancel rather than treating every layer identically, then comparing the quantized model's logits with the full-precision model using KL divergence. He says the method made GLM-5.2 20% more quantized than another provider's version while retaining better fidelity. That is a claim about Baseten's research, not a general law that more compression improves intelligence. 1

Internal tests cannot cover the strange prompts, hardware combinations and traffic bursts that arrive at a public endpoint. Kiely describes a model that sometimes emitted the same token repeatedly. Baseten's mitigation was to cut off a generation after the same token appeared four or more times and retry or reprocess the request. The diagnosis was not always "bad weights": the behavior could change with the inference engine, kernel implementation, cluster, or interconnect between machines. A race condition in a fast kernel can make the same model behave differently on two clusters. 1

This is also why the guests separate local AI from data-center inference. Local systems begin with "how do I fit the model here and make it less dumb?" Data-center systems begin with "how do I load the model and make it less slow?" A memory-saving trick that helps a Mac may add too much dequantization overhead on a high-bandwidth B200 cluster. Hardware determines which optimization is real and which is just attractive in a diagram. 1

The conversation ends with a shift in where expertise will matter. GPUs are becoming more specialized, but their flexibility still matters because models and architectures change too quickly to burn one set of weights into a fixed chip. Inference engineers therefore have to reason across the whole stack: model behavior, kernels, networking, cache movement, hardware topology and customer traffic.

That is the durable takeaway from this episode. The model release is the visible event; the product is the invisible system that serves it. For anyone evaluating an AI provider, "which model do you run?" is only the first question. The harder ones are: what workload was benchmarked, where is the cache, what happens when traffic changes, and how does the system recover when the model starts behaving strangely?

The full conversation is available on Latent Space.