Modal thinks agents need a different cloud

Akshat Bubna's most useful claim is not that agents need more GPUs. It is that agents need a cloud they can operate inside: write code, run it, inspect the failure, change the environment, and try again. In the Latent Space conversation, Modal's CTO frames the shift from developer experience to "agent experience" as a practical systems problem. A human can read docs, infer missing context, and debug YAML. An agent needs a tighter loop, with infrastructure requirements close to the code and observable through tools it can actually use. 1

Modal began as a serverless runtime, not as an AI infrastructure company. Bubna says the early thesis came from a simple irritation: workflow orchestration was hard because teams had to run it on Kubernetes, and Kubernetes was built around slower-scaling web-server assumptions rather than bursty, compute-heavy work. Modal added GPUs before ChatGPT, but the bigger pattern was already visible: customers wanted specialized images, accelerators, and a way to scale up and down without managing the substrate themselves. 1

That origin matters because Bubna does not describe Modal as a general replacement for every cloud service. He narrows the job: inference, training, batch processing, and sandbox workloads for AI applications. Always-on web apps still have a place elsewhere. The interesting work is where a job needs specialized compute, bursts across regions, or a runtime that changes shape as the product changes. 1

The Modal example is strongest in custom-model inference. Bubna names audio, video, robotics, and computational biology companies that train models elsewhere but need a deployment layer that can absorb launches, regional traffic cycles, and many different model variants. His point is that the hard part is often not renting a GPU. It is scaling the right GPU in the right region, at the right time, without turning every launch into a capacity-planning exercise. 1

The phrase "agent experience" can sound like marketing until Bubna explains what changed inside Modal's own SDK team. The old developer-experience problem was reducing the surface area a human had to understand. Instead of writing and maintaining piles of YAML, the developer could express infrastructure requirements in decorators next to the code that would run. 1

Agents make the same design choice more urgent. Asking an agent to read hundreds of Kubernetes files, infer deployment intent, edit untyped configuration, and debug the result is a poor use of model context. Asking it to change a few typed, co-located declarations and immediately observe the result is closer to the loop agents are good at. Bubna says Modal has pushed more investigation into the CLI because agents can use it, while humans still need dashboards to make judgment calls. 1

That is a useful distinction for teams building with coding agents. The question is not whether humans stop reading code. The question is whether the system exposes enough state for a human or an agent to understand what happened after the code ran. In this framing, observability becomes a product surface, not an afterthought.

The episode's most concrete agent-infrastructure thread is sandboxes. Modal built sandboxing in 2023, before the current wave of coding-agent products made the need obvious. Bubna describes the agent loop as a place to write code, run it, inspect output, change the environment, debug failures, and repeat. 1

That loop changes scale in reinforcement-learning settings. Bubna says RL rollouts can require 100,000 sandboxes. That number is a reminder that "agent runtime" is not only a developer workstation metaphor. At training and evaluation scale, the sandbox becomes a distributed systems primitive: many isolated environments, started and stopped quickly, with enough control to make failures meaningful rather than random. 1

Modal is also adding the less glamorous pieces that make sandboxes usable in production. Bubna talks about sidecars, outbound network control, credential injection, private IPv6 addressing between containers, and RDMA-backed multi-node training. These are not features a demo agent needs on day one. They are features a company asks for once the agent is touching files, networks, credentials, customer data, or GPUs. 1

Modal is also moving closer to the model-serving layer. Bubna points to GPU snapshotting, which captures GPU state such as a compiled model so the next cold start can be faster. He also describes DeFlash, Modal's block-based speculative decoding work, and Auto Endpoints, which package optimized inference while still letting users inspect or eject into full Modal code. 1

Speculative decoding is worth spelling out because it shows why inference is not just an API wrapper. A smaller draft model predicts tokens ahead of the larger model; the larger model verifies them in a batch. If enough draft tokens are accepted, throughput improves because the system uses compute more efficiently than one-token-at-a-time decoding. Bubna argues that improving accept length can produce multiplicative speedups, while kernel-level improvements may only add a few percentage points. 1

The broader claim is that production-grade inference includes tail latency, delivery guarantees, autoscaling, and cost control. A team can run vLLM or SGLang on rented GPUs. Modal's pitch is that the surrounding machinery, especially elastic scale-to-zero and burst handling, is where many companies lose time. 1

One of the more revealing parts of the conversation is Modal's "supercloud" stance. Bubna says Modal runs across 17 cloud providers rather than owning data centers. That lets the company stay capital-light, but it also forces Modal to build a reliability layer over providers whose hardware and operational quality vary. 1

This is where AI infrastructure starts to look less like conventional SaaS and more like an operations business. Modal has a compute-strategy function that thinks about one-year versus three-year reservations, regional supply, GPU fungibility, and future capacity bets. Bubna also says Modal is building batch tiers for customers that can wait up to 24 hours if that lowers price. 1

For builders, that means the infrastructure provider is no longer just a neutral execution venue. It is making supply-chain judgments on your behalf. If those judgments are good, you get elasticity and lower cost. If they are wrong, your application feels the shortage.

Bubna is friendly toward managed agents from foundation labs, but he draws the line at production control. A managed agent may be the easiest place to start. Once a company needs file persistence, snapshot-and-restore, network control, GPUs, and strict sandbox boundaries, Bubna argues it needs a specialized sandbox provider. 1

He is also skeptical of letting an LLM mediate sandbox-level permissions by itself. His view is simple: hard guardrails still matter because the agent may otherwise exfiltrate data or cross a boundary it should not cross. Softer, model-mediated permissions can sit above that layer, but the substrate still needs enforceable limits. 1

That is the episode's practical takeaway. If your AI product is one hosted model call, Modal may be too much machinery. If your product needs agents that execute code, operate on files, run jobs, inspect logs, use GPUs, and stay inside hard boundaries, the infrastructure is no longer background plumbing. It is part of the product's behavior.