NVIDIA ships a fast model and router for long-running agents

NVIDIA’s August 11 release pairs a lightweight open model for repetitive agent work with a router that can choose among different models. The useful idea is not another all-purpose chatbot: it is a faster execution layer for the many tool calls, checks, and handoffs inside an agent.

NVIDIA introduced Nemotron 3.5 Lightning, an open 30-billion-parameter mixture-of-experts model with 3 billion active parameters. Its hybrid architecture combines Mamba-2, mixture-of-experts layers, and attention, and the release includes support for speculative decoding. NVIDIA positions it for high-volume, low-latency steps in long-running or always-on agents: tool calls, validation, delegation to subagents, and other specialized tasks. 1

The companion release is NeMo Switchyard, a routing library. It is meant to let an application send each request to a suitable model, including open, proprietary, frontier, or NVIDIA models, without rewriting the application around each choice. That makes the launch a two-part systems proposition: Lightning can handle the frequent smaller actions, while a router decides when a larger or more capable model is worth the extra cost or latency. 2

Agent systems spend much of their time outside the headline reasoning step. They read files, call tools, validate outputs, retry failures, and pass focused jobs to other agents. A model that is cheaper or faster at those steps can change the economics of the whole workflow. A router also creates a practical escape hatch: teams can test a local or open model for routine work while retaining a hosted frontier model for difficult cases.

There is an important qualifier. NVIDIA says Lightning can reach up to four times the output speed of similar-sized models. That is a vendor claim, not an independent benchmark, so the number should be treated as a starting hypothesis for testing rather than a deployment guarantee.

The official model card lists 30 billion total parameters, 3 billion active parameters, and a context length of up to 1 million tokens. It notes that a single H100 deployment uses 256,000 tokens, and lists one 80-gigabyte H100 or A100 as a single-GPU deployment target. BF16 is intended for customization and research; an NVFP4 checkpoint targets optimized inference. The model supports English, code, several European languages, and Japanese, and is released under NVIDIA’s OpenMDW License Agreement v1.1, not Apache 2.0. 3

For developers, the sensible first test is to put Lightning behind Switchyard as a worker model. Measure end-to-end latency, tool-call errors, retries, and cost on your own agent traces. Then check whether the license and hardware envelope fit your deployment before treating the release as a general replacement for a frontier model.