Stop paying your LangGraph bill on every request

Notion lost roughly five percentage points of gross margin to AI inference costs in 2025. 1 SaaS companies broadly shed five to ten points of gross margin to AI compute over the same period. 1 AWS quietly raised GPU prices 15%. A single developer on a $200/month plan generated $5,000 in Claude Code inference charges in one sprint. 1

There is a structural reason for this, and it has nothing to do with token prices. Jaroslaw Wasowski, writing in May 2026 about LangGraph and its peers, named it cleanly: "every LLM call within the orchestrator loop of these frameworks acts like an interpreter — you pay repeatedly (even ten times or more) for the execution of the same standard procedure." 1 Your customer support flow, claims processing pipeline, or onboarding sequence has the same logic on Tuesday as it did on Monday. You are still paying frontier model rates to rediscover that logic from scratch on every single conversation.

A May 21 arXiv paper from the University of Melbourne argues this is solvable — and the economics are striking enough that Latent.Space called it "a serious economic idea." [cite:2|[AINews] All Model Labs are now Agent Labs|[https://www.latent.space/p/ainews-all-model-labs-are-now-agent]] Google has already launched an early-access managed service for it.

Simon Dennis, Rivaan Patil, Kevin Shabahang, and Hao Guo (University of Melbourne's i14 group) demonstrated a four-step process for eliminating the runtime orchestrator entirely. 2

They call the result a "subterranean agent": the orchestrator exists only during training, then disappears. At runtime, the user talks to the fine-tuned model — no tool-calling overhead, no multi-step API chaining, no interpreter loop. Dennis et al. summarized the principle: "Persistent structure belongs in the weights, transient state belongs in the prompt." 2

They validated the approach across three domains, each with n=200 evaluation runs judged by Claude Sonnet 4.5 (verified independently by GPT-4.1):

Two patterns stand out. First, the cost advantage scales with workflow complexity — the 55-node insurance workflow yields 3.6× better economics than the 14-node travel workflow. Second, the compiled model actually outperformed the same model running explicit orchestration on all five quality dimensions (task success, information accuracy, consistency, graceful handling, naturalness), with p<0.001 on four of them. The compiled 8B model matched a frontier model running in a LangGraph orchestrator — a model roughly 70× larger. 2

One important constraint: LoRA fine-tuning (at ranks 16 through 128) failed to replicate these results. LoRA, a parameter-efficient method that updates only a small fraction of a model's weights, is standard for most fine-tuning tasks — but full parameter updates are required here. Dennis et al. explain why: procedural knowledge — the kind that governs "what step comes after which condition" — is not low-rank. 2 The one-time training cost runs $50–80 on a standard cloud GPU, and the payback threshold is approximately 500 conversations.

Full compilation isn't the only option. Stanford researchers Vishnu Sarukkai and colleagues proposed a lighter path they call inference-time distillation: use a few teacher-model demonstrations to guide a cheaper student model, falling back to the teacher only when the student is uncertain. 3 No fine-tuning, no training pipeline. On AppWorld (a benchmark for multi-step tool-use agents), this approach delivered 3.5× cost reduction while recovering 79% of the teacher model's accuracy — deployable the same day you decide to try it.

For teams with rigid, measurable success criteria, a third option: skill distillation. Binyan Xu and colleagues at CUHK and Tencent's LIGHTSPEED lab found that the payoff of distillation depends heavily on how you measure success, not what the task is. Workflows with "rigid" metrics — ones where the scoring landscape penalizes any deviation from the correct path — benefit most from distillation, yielding up to +28 percentage points of task improvement at 1.4–8× lower cost. 4 Workflows with fuzzy metrics can show negative returns from the same technique.

Google is signaling where this is heading commercially. Google Cloud launched Gemini Distillation Service in early access this week, distilling Gemini 3.1 Pro reasoning into Gemini 2.5 Flash using the teacher model's internal reasoning traces — not just its final outputs. 5 It requires a minimum of 1,000 examples and currently runs in Early Access via Google's sales team.

The $50–80 training cost amortizes at roughly 500 conversations. 2 If your agent feature handles more than 500 daily conversations and the workflow logic stays stable for weeks at a time, full compilation likely has a one-week payback. Below 500 daily conversations, inference-time distillation (no training cost, deploy immediately, 2.5–3.5× savings) is the better starting point.

Customer support, claims processing, and structured onboarding flows compile cleanly — their decision logic is deterministic enough to encode as a flowchart. Creative tasks, open-ended research agents, and workflows that require improvising tool selection on the fly are poor candidates. The test: could a human analyst draw a flowchart of the workflow before the conversation starts? If yes, compilation is viable.

One operational note on recompilation: if the workflow logic changes, you recompile. On production-grade hardware (eight H200 GPUs), that takes 30–50 minutes — comparable to a CI/CD build cycle, not a product relaunch. 2 Dennis et al. are explicit that the flexibility gap "is a deployment cycle rather than a paradigm shift."

This research cluster — eight independent papers from April through May 2026 converging on agent workflow compression — is running ahead of the platform tooling. [cite:2|[AINews] All Model Labs are now Agent Labs|[https://www.latent.space/p/ainews-all-model-labs-are-now-agent]] Google's managed service is in Early Access; OpenAI and Anthropic have not announced equivalent services. The window where you need to build this yourself is likely measured in months, not years. But the window where this changes your cost structure is open now.