The same model weights score 26% on one serving route and 62% on another
A controlled enterprise audit reveals that identical model weights swing from 26% to 62% across different serving routes, demonstrating why PMs must evaluate deployed endpoints rather than model names.
When enterprise product teams select a foundation model for an agentic workflow, procurement conversations usually center on public leaderboard rankings. Teams compare benchmark scores for models such as GPT-5.6, Claude Fable, or open-weight releases like Qwen3.8, pick the highest-ranked checkpoint within their budget, and route traffic through a commercial inference provider. That workflow conceals a structural blind spot: public benchmarks evaluate naked model names, whereas production software calls specific serving routes. When an inference endpoint enforces a hidden token ceiling, restricts image payloads, or swaps the tool-call parser, the model's observable capability changes drastically.
An empirical investigation released on arXiv, titled IB2 1, exposes the magnitude of this distortion. Auditing 18 major enterprise benchmarks, the authors found that every single suite scored advertised model identifiers rather than deployed systems. When the research team evaluated identical model weights across two commercial serving routes on 128 locked enterprise tasks, DeepInfra FP8 scored 26.12 while CoreWeave FP8 scored 62.25. The 36-point gap had nothing to do with reasoning quality; it stemmed entirely from route-level payload limits and truncation thresholds. For product managers building autonomous agents, this finding establishes an urgent operational reality: you do not deploy a model checkpoint; you deploy a served endpoint.
What changed: Evaluating the serving route, not the model checkpoint
The core insight of the IB2 framework is that a production AI system consists of eight interdependent variables rather than an isolated set of neural network weights 1. The authors formalize any deployed system under test as a tuple:
S = <m, r, p, e, Ω, T, C, H>In this structure, m represents the underlying model weights, r is the physical serving route and cloud hosting provider, p is the numerical precision (such as FP8, BF16, or INT4), e is the allocated reasoning effort, Ω is the operational payload envelope, T is the tool implementation catalog, C is the strict output contract, and H is the client-side evaluation harness. Changing any single variable produces a distinct production system with its own failure modes.
To turn this measurement error into a verifiable engineering protocol, IB2 introduces three structural components 1:
- Gold-blind capability-binding preflight: Before a single evaluation task reaches the serving endpoint, an automated probe suite verifies whether the route satisfies the technical envelope. The preflight checks ten concrete predicates, including whether the endpoint accepts 13 byte-identical images in a single call, parses a 25-tool catalog, emits strict JSON schemas, executes SQL queries up to 65,536 bytes, and processes completions exceeding 32,768 observed tokens. If an endpoint fails any predicate, the evaluation halts immediately, marking the route as incompatible with that workload.
- Reliability-inclusive scoring: Traditional benchmark scripts frequently discard HTTP 500 errors, rate-limit drops, and schema rejections from the denominator, reporting accuracy only over completed responses. IB2 keeps all failed responses in the denominator. When a route drops a request or truncates mid-generation, that failure directly depresses the final score.
- Score-blind adjudication: Any required network retries or quota resets are handled by a supervisory controller without inspecting output correctness, preventing optimistic retry bias.
The open-source specification and evaluation schemas are published under the IBIB project repository 3.
The evidence: How serving constraints distort observable intelligence
The empirical results from IB2 demonstrate how severely infrastructure boundaries warp benchmark scores 1.
When the researchers evaluated identical open-weight model files on 128 locked enterprise tasks spanning documents, multi-tab financial spreadsheets, and SQL databases, the scores diverged wildly based on the hosting provider:
- DeepInfra FP8 registered a composite score of 26.12. Post-hoc analysis showed that DeepInfra enforced a hard route ceiling of 4 images per API request. Because the evaluation contract required processing document packets containing 13 images, the endpoint rejected entire test suites outright.
- CoreWeave FP8 achieved a composite score of 62.25 on the exact same model weights. CoreWeave successfully ingested the images, yet it hit an unexpected route limit at exactly 32,768 completion tokens, truncating long-form financial syntheses.
- Provider P, a third hosting route evaluated under prospective preflight binding, cleared all ten predicates and completed the suite without payload rejections.
Serving-arm configuration also altered precision significantly. Evaluating the same underlying model revision across different serving backends shifted task accuracy from 77.38 to 82.54 (paired interval [0.11, 10.60]). Variations in harness generation, guided-decoding backends, and tool-call parsers produced a 5.16 percentage point swing on identical frozen weights.
Furthermore, IB2 proved that discarding route failures reshuffles benchmark leaderboards. When the authors recalculated scores by removing failed and refused API calls from the denominator, the rank ordering of eleven evaluated commercial systems inverted. A provider with higher intrinsic reasoning capability dropped several places because its serving endpoint aggressively rate-limited complex calls, while a less capable model with a permissive API gateway rose to the top.
+-----------------------------------------------------------------------------+
| IB2 BENCHMARK RUNS ON IDENTICAL MODEL WEIGHTS |
+----------------------+---------------+--------------------------------------+
| Serving Route | IB2-7 Score | Primary Failure Mode Observed |
+----------------------+---------------+--------------------------------------+
| DeepInfra FP8 | 26.12 | 4-image route limit rejected requests|
| CoreWeave FP8 | 62.25 | Truncated output at 32,768 tokens |
| Provider P (Bound) | Prospectively | Passed full 10-predicate gate |
+----------------------+---------------+--------------------------------------+These dynamics are visible across frontier evaluation suites. In AutoResearchExam 4, an open-ended 24-hour research benchmark released by Bespoke Labs and UC Berkeley researchers, agents tackled 29 machine learning research challenges inside isolated execution containers. Benchmark founder Alex Dimakis reported 5 that GPT-6 Astra established an early lead and maintained it through hour 19, after which Claude Fable 5.1 caught up and achieved the top score at hour 24 (reaching an AUARC of 0.602 versus Astra's 0.600). As AI researcher Elvis Saravia highlighted 6, agent performance shifts continuously across execution horizons, and models tend to overfit to local validation feedback unless evaluated against sealed hidden tests.
Production reality check: Moving to dual-loop quality gates
Engineering teams operating mission-critical agent fleets have already abandoned static model benchmarks in favor of route-bound, continuous evaluation.
A prominent production blueprint comes from fast-delivery enterprise Zepto, detailed in an engineering case study on Databricks 7. Zepto deployed an evaluation-first agent architecture that handles over 80% of customer support tickets autonomously, cutting support costs by 65% with a payback period under one month.
Zepto's deployment enforces four operational pillars containing 11 quantitative quality gates:
- Intent Classification: Evaluates exact-match accuracy and per-class F1 scores.
- Response Quality: Measures semantic correctness, enterprise policy adherence, and brand tone.
- Tool Calling: Assesses function selection accuracy, argument validation, and execution latency.
- Performance & Cost: Tracks P50 and P95 latency alongside per-interaction spend.
Before any agent update ships, the system validates the candidate route against a curated golden dataset of 5,247 human-labeled conversations. In production, automated telemetry continuously samples 18% to 20% of live traffic, catching edge cases within 4 to 6 minutes and triggering automated rollbacks whenever intent accuracy drops or latency breaches target thresholds.
This methodology mirrors architectural guidance from Meta Superintelligence Labs 8, where infrastructure engineers stress that offline benchmarks fail because agent execution is inherently non-deterministic. Production teams must measure cumulative trajectory success across multiple attempts (
pass^k) rather than relying on isolated single-turn accuracy.How to implement now: The 4-week PM pilot
Product managers can eliminate model-identifier ambiguity by structuring a route-qualification pipeline within their existing development workflow.
The deployment unit
Organize your serving architecture around six modular verification components:
| Component | Responsibility | Permissions & Constraints | Acceptance Gate |
|---|---|---|---|
| Capability Preflight Gate | Sends synthetic edge-case payloads to test physical endpoint limits | Read-only synthetic test runner | Route passes 100% of payload predicates (tokens, images, JSON) |
| Route Pinner | Locks the exact provider, model revision hash, and quantization level | Configuration management service | Explicit endpoint URI and revision tag registered |
| Payload Normalizer | Formats tool definitions, system prompts, and media inputs to route specs | Middleware proxy layer | 0 schema validation errors on request dispatch |
| Evaluation Harness | Executes task bank against the pinned route with timeout bounds | Isolated container execution | Reliability-inclusive scoring treats timeouts as failures |
| Dual-Loop Scorer | Runs golden test bank offline; samples 20% of live production traffic | Delta tables and observability pipeline | Clears all 11 quantitative quality gates |
| Circuit Breaker | Automatically routes traffic back to fallback provider on threshold breach | API gateway router | Switches endpoints within 30 seconds of persistent failure |
4-week pilot rollout plan
- Week 1: Audit serving routes and catalog constraints. Map every external LLM endpoint used in your product. Document physical provider boundaries: maximum image attachments, context length limits, completion token ceilings, supported quantization formats, and strict JSON enforcement. Benchmark your current production agents across 50 representative historical tasks, recording baseline completion rates and unhandled route errors.
- Week 2: Construct capability preflight and golden test suites. Build an automated preflight script that verifies whether candidate routes handle your application's actual payload demands before sending user traffic. Assemble a golden evaluation dataset containing at least 200 diverse domain queries, including 30% complex edge cases and 10% known historical failure modes.
- Week 3: Run prospective route binding in shadow mode. Route production traffic in parallel to your primary endpoint and a candidate secondary route. Run both endpoints through the reliability-inclusive evaluation harness. Measure the delta in completion accuracy, token latency, and error frequency without exposing end users to experimental responses.
- Week 4: Enforce deployment quality gates and circuit breakers. Wire your continuous integration pipeline to block deployments whenever a candidate endpoint fails the preflight gate or registers a drop in golden-dataset accuracy. Implement automated traffic shifting that diverts user requests to an established fallback route if production error rates exceed acceptable thresholds.
Pilot measurement scorecard
Track four operational metrics to govern provider procurement and deployment:
| Metric | Target Threshold | Minimum Viable Gate | Rollback Trigger |
|---|---|---|---|
| Preflight Predicate Pass Rate | 100% of required capabilities | 100% (zero tolerance for payload rejections) | < 100% (reject route immediately) |
| Reliability-Inclusive Completion | ≥ 85% on golden benchmark | ≥ 75% | < 70% (lower than current baseline) |
| Endpoint P95 Latency | ≤ 2,500 ms per interaction | ≤ 4,000 ms | > 5,000 ms across a 15-minute window |
| Cost per Successful Task | Within target unit economics | ≤ 1.25x baseline budget | > 1.50x budget without quality gain |
Urgency and action window
- Immediate priority (current quarter): Teams deploying multi-modal agents, financial document processors, autonomous coding assistants, or customer-facing support agents that execute multi-step workflows. When agents call tools and handle rich payloads, hidden endpoint constraints cause silent failures that destroy user trust.
- Monitor and defer: Simple conversational wrappers, static single-prompt summarizers, or internal prototypes operating well below token and payload ceilings. In low-complexity contexts, standard provider defaults generally suffice, allowing teams to defer rigorous route-binding infrastructure until workflows expand.
References
- 1
- 2Demystifying evals for AI agents
anthropic.com
- 3
- 4AutoResearchExam: Benchmarking Open-Ended Long-Horizon AI Agents
benchmarks.bespokelabs.ai
- 5
- 6
- 7
- 8
This story was produced automatically by a channel. One sentence is all it takes for Neodrop to keep producing for you.