Train agents in a world you never built
Apple's ESAT shows how API specifications, an LLM simulator, and a strict judge can create useful agent-training trajectories before a team builds a full executable environment.
The product implication
Most teams building API-calling agents still treat a working environment as a prerequisite for training: implement the APIs, populate a believable database, run the agent, and collect the trajectory. Apple's ESAT paper asks whether that order is backwards. Give the trainer API specifications instead, let an LLM simulate a stateful digital world, and use a judge to discard bad trajectories.
That changes the first product question from "Which sandbox should we build?" to "Which behavior can our API contract teach?" It does not make a simulator safe for production actions. It makes it possible to train and compare agent behavior before a team has built a full replica of every downstream system.
ESAT, short for Environment-free Synthetic Data Generation for API-Calling Agents, is a preprint submitted to arXiv on July 18 and revised on July 21. The paper is associated with Apple on its paper page and reports results on AppWorld and OfficeBench, two benchmarks built around multi-step API use. 12
What changed
The environment becomes a training-time model
Traditional trajectory collection needs executable tools and realistic backend state. That is expensive when the target is a new SaaS product, an internal system, or a cross-application workflow that does not yet have a stable test environment.
ESAT replaces the missing environment with an LLM simulator. The simulator receives the API specification, the task, the current call, and the interaction history. It returns a response that must remain consistent with the virtual state established by earlier calls. If the agent creates a user in step one, later calls should be able to find that user. The paper also adds programmatic parameter checks, schema checks, semantic consistency checks, retries, and a trajectory-level judge. 1
The important distinction is that the simulator is a data-generation instrument, not the product runtime. A PM should think of it as a rehearsal stage for tool behavior. Real customer data, credentials, approvals, and side effects remain outside that stage.
Three stages turn an API contract into supervision
- Task synthesis. A model generates tasks from the available API specifications. ESAT varies difficulty, read versus write actions, mixed workflows, app count, and API count, then samples less-used apps and APIs more often to improve coverage.
- Trajectory synthesis. A teacher agent tries to solve each task. A separate simulator produces API responses while tracking the task and prior history. The target is a multi-step trajectory, not a single tool-call example.
- Trajectory filtering. An LLM judge checks whether the trajectory is coherent, complete, and semantically correct. ESAT uses repeated judging and majority voting, then keeps only the accepted examples for supervised fine-tuning.
In product terms, the API specification is the contract, the simulator is a temporary state machine expressed in language, and the judge is the data gate. That is a much more useful mental model than "ask a model to make synthetic traces."
The headline result is a smaller build dependency
On AppWorld, the paper generated about 6,000 trajectories from 52 synthetic apps and 1,017 APIs without using AppWorld's executable environment. The resulting ESAT-S52 data improved the evaluated models by 8.4 to 47.0 percentage points across the reported settings. The authors also generated about 9,000 trajectories from a subset of AppWorld's APIs and compared them with trajectories collected from the real AppWorld training environment. 1
A concrete comparison shows why the result is interesting. On AppWorld's normal and challenge splits, Qwen3.5-27B reached 84.2% and 79.0% after training on ESAT data, versus 83.5% and 63.7% after training on the real-environment AWT data used in the paper. Those are benchmark scores under the paper's training and evaluation setup, not evidence that a language-model simulator is more reliable than a real backend in general. 1
OfficeBench points in the same direction for smaller models. Its Qwen3.5-2B result rises from 9.1% to 69.6% on two-app tasks and from 0.7% to 40.9% on three-app tasks after ESAT-OB fine-tuning. The comparison is zero-shot versus fine-tuned performance, so it should be read as evidence that targeted supervision can matter a great deal, not as a claim that a 2B model is generally stronger than a frontier model. 1
Why PMs should care
The bottleneck moves to specifications and evaluation
If API specs can produce useful training data, a team can start with a contract and a holdout evaluator instead of waiting for a complete clone of every tool environment. That is relevant for internal operations agents, support workflows, finance assistants, and any product that needs to coordinate several APIs before the underlying integration layer is mature.
The new work is not free. Someone still has to define valid schemas, state transitions, permissions, error behavior, and task coverage. In fact, ESAT makes those artifacts more valuable: a vague API description gives the simulator room to invent states, while a precise contract gives both the simulator and the production agent fewer ways to drift.
The operational numbers show where a real pipeline needs guardrails. In the AppWorld generation run, 45.38% of failed attempts were simulation failures, 41.78% were rejected by the judge, and 10.72% came from the agent using an unexpected app. On a separate check of 720 AppWorld trajectories, the judge's precision against an execution-based verifier was 95.2%. That supports using the judge as a filter, but it does not eliminate the need for real-environment validation. 1
The simulator also has a length boundary. The paper reports a roughly 2–3% simulation failure rate for responses under 500 tokens and about 23% for responses above 2,000 tokens. Multi-app tasks are harder to synthesize than single-app tasks, and the paper reports an uneven distribution of generated tasks across those categories. A workflow that looks good in a short demo may still be a poor training target for a long, cross-system process. 1
The surrounding engineering signal is about throughput
ESAT is a research result, not a production training framework. An adjacent Google Developers Blog post about Tunix makes the infrastructure problem explicit: multi-turn agent training wastes accelerator time when rollouts wait on tools or environments, so Tunix uses asynchronous rollouts, a decoupled producer-consumer pipeline, composable agent and environment abstractions, and continuous RL-specific telemetry. That post does not validate ESAT, but it points to the implementation work a PM should budget for once synthetic trajectories become a real training loop: queueing, concurrency, traceability, and cost measurement. 3
The X signal is attention, not independent evidence. AK's July 21 post describing the paper had 17,269 views, 66 likes, and 10 reposts when retrieved. It shows that the environment-free framing is legible to the AI research audience; it says nothing about whether ESAT transfers to a company's own APIs. 4
For a quick orientation, Vinh Nguyen's 6-minute-38-second explainer walks through the task, trajectory, and filtering stages. Its description identifies the video as NotebookLM-generated, and it had 50 views when retrieved. Use it to understand the mechanism, not as a second evaluation of the paper. 5
Loading content card…
How to implement now
- Pick one bounded API domain. Start with a read-heavy workflow or a reversible internal operation. Write machine-readable schemas for inputs, outputs, permissions, state changes, and error cases. Do not start with a general-purpose assistant.
- Build a coverage matrix before generating data. Vary read, write, and mixed tasks; one-app and multi-app paths; short and long responses; missing arguments; retries; and permission failures. Up-sample APIs that rarely appear. Treat coverage as a product requirement, not a byproduct of prompting.
- Separate the simulator from the production control plane. Let the simulator create rehearsal data. Run schema validation, state-consistency checks, and a judge before a trajectory enters training. Log simulator failures separately from judge rejections so the team can tell whether the contract, the simulator, or the teacher agent is at fault.
- Train a small adapter and test on a real holdout. Compare zero-shot, synthetic-only, real-trajectory-only, and mixed-data variants with the same model and action budget. Measure task completion, valid tool-call rate, unexpected-tool rate, steps to completion, and the gap between simulator outcomes and execution-based verification. The holdout must exercise real state transitions; otherwise the team is only measuring fluency inside its own imagined world.
- Roll out in shadow mode. Keep the simulator confined to training and offline evaluation. In production, require real authentication, permission checks, approval gates, idempotency, and rollback for side effects. Let the trained agent recommend actions before it can execute them.
The PM decision is therefore straightforward: if building a faithful environment is blocking a valuable agent pilot, ESAT offers a way to move the training milestone earlier. But the acceptance gate should be a real-environment holdout and a measurable simulator-to-real gap. The simulator can help you learn what an agent should do. Only the real system can tell you whether the action is allowed, current, and safe.
References
- 1
- 2
- 3Scaling Agentic RL: High-Throughput Agentic Training with Tunix
developers.googleblog.com
- 4AK's paper post on X
x.com
- 5
This story was produced automatically by a channel. One sentence is all it takes for Neodrop to keep producing for you.