Your frozen model has headroom you haven't used

The default response to a performance ceiling is to swap in a larger model or pay for a more capable API. That reflex is getting expensive — and, for a growing class of deployment scenarios, it's also the wrong move. Two papers posted to arXiv on May 22, 2026 offer a cleaner alternative: get more out of the model you already have.

Jensen Huang called it at NVIDIA GTC this March: AI inference compute demand has grown roughly 10,000× in two years. [cite:1|[AINews] The Inference Inflection|[https://www.latent.space/p/ainews-the-inference-inflection]] "In very large part, we now need to be an AI inference company," Sam Altman said at roughly the same time. [cite:1|[AINews] The Inference Inflection|[https://www.latent.space/p/ainews-the-inference-inflection]] Inference is where the budget goes.

But the deeper issue is that brute scaling has a ceiling that practitioners rarely model explicitly. A team at ByteDance Seed, UC Berkeley, and the University of Virginia — accepted at ICML 2026 — formalized what practitioners have been observing empirically: scaling model size or training data without preserving a sufficient signal-to-noise ratio (SNR) causes performance to degrade in a U-shaped curve rather than improve monotonically. 1 Their framework — the Shannon Scaling Law — models LLM training as information transmission over a noisy channel. The key finding: when you apply quantization, supervised fine-tuning, or any real-world perturbation, the model noise exponent (γ) exceeds the bandwidth exponent (α). Scaling model size amplifies noise faster than it expands capacity. 2

The law was fitted on Pythia models up to 6.9B parameters and predicted unseen 12B model behavior at up to 307B training tokens with a pooled R²=0.847. 1 Standard monotonic scaling laws (OpenAI, Chinchilla) collapsed to near-zero or negative R² on the same task. The implication: for deployed models operating under quantization or fine-tuning pressure, there is an optimal point on both the model size and data axes. Bigger is not reliably better.

A separate paper from five researchers at UT Austin offers a practical technique that takes advantage of the capacity already baked into frozen checkpoints. 3

The intuition is precise. Each transformer block can be understood as a single forward Euler step integrating an underlying ODE — a coarse approximation with step size h=1. Looping the same block K times replaces that coarse step with finer sub-steps (Runge-Kutta style, h=1/K), refining the approximation toward the same target without changing any weights. 4 The mid-stack layers that researchers typically prune away as redundant are, by the same logic, safe to re-apply — their redundancy is what makes looping stable.

The empirical results across 7 model families and 45 (model, benchmark) evaluation cells: 3

87% of evaluated cells are non-negative under a single out-of-the-box recipe with no per-cell hyperparameter tuning. 3 There are two iteration modes: block-mode for dense models (loops the entire window K times) and layer-mode for mixture-of-experts (MoE) models, which avoids routing instability by iterating each layer independently. The optimal loop window sits at roughly 45-60% of total network depth across architectures.

Step 1: Start with bypass mode. In bypass mode, the loop only runs during the prefill (prompt-processing) phase, not during token generation. Wall-clock overhead is approximately 0%. 4 This is the right first test: run your existing evaluation suite against bypass-mode looping on your current deployed checkpoint. If you see gains at zero latency cost, you have a free improvement.

Step 2: Match iteration mode to model architecture. If you're running a dense model (Qwen3, Llama-3), use block-mode. If you're running a mixture-of-experts model (Qwen3-30B-A3B, DeepSeek-V2-Lite), use layer-mode to prevent expert routing from destabilizing across loops. 3 The depth fraction rule — loop window between 45% and 60% of total layers — holds across both families.

Step 3: Use the Shannon framework to decide when to route up. The U-shaped degradation finding gives you a principled decision rule: if your model is already quantized (INT4/INT8/FP8) and undergoing domain fine-tuning, adding more parameters may backfire rather than help. 1 In that regime, looping your current checkpoint is more likely to recover performance than switching to a larger model — because the larger model faces the same SNR constraint at higher cost. AI gateway routing (smart routing: simple requests to smaller models, complex requests to frontier models) remains valid, but the model selection decision should now account for SNR headroom, not just benchmark rank.

Neither paper has publicly released code yet — the looped transformer wrapper is the highest-priority candidate given it's a pure inference technique with no training required. Watch the UT Austin group's GitHub and Hugging Face for a release.

MELT (Qualcomm AI Research) — Qualcomm's Memory-Efficient Looped Transformer solves the KV cache growth problem that would otherwise make deep looping impractical: it maintains a single shared KV cache per layer across all loops, updated via learnable gating. 5 Models fine-tuned on MELT match standard looped transformer performance at constant memory cost — decoupling reasoning depth from memory consumption.

LT2 (Rice University) — Linear-Time Looped Transformers replace quadratic softmax attention with subquadratic variants inside the loop, making looping computationally viable at longer context lengths. 6 Their Ouro-hybrid-1.4B model, trained on roughly 1 billion tokens, is competitive with industry-level 4B models — roughly a 3× parameter efficiency gain.

Claude Mythos speculation — Community developer Kye Gomez has published an open-source PyTorch reconstruction of what he theorizes is Claude Mythos's architecture: a recurrent-depth transformer with a looped block running T iterations between a Prelude and a Coda. 7 Anthropic has not confirmed this. It is worth tracking because if true, it means a frontier lab has already bet production inference on looped recurrence — which would change the calculus for teams evaluating this direction.