Stop before the score.

EarlyEval changes agent evaluation by predicting an agent’s final success or failure from a partial trajectory. The method combines behavioral, textual, and reference-solution features with two calibrated LightGBM classifiers, then stops a run when confidence crosses a chosen threshold. 1

Why does that matter for a product team? Agent evaluation can spend steps and tokens after the outcome has become predictable. Across SWE-bench Verified, TerminalBench, and Toolathlon, the paper’s recommended operating points cut 13–26% of steps, up to 44.1% of input tokens, and up to 29.4% of output tokens, with 89–97% prediction accuracy. 2

The reversible first build is a shadow evaluator: collect completed, outcome-labeled trajectories; train leave-one-agent-out predictors; and run the early-stop decision beside full execution. Compare cost, resolve rate, and ranking stability before changing the evaluation path. The paper reports resolve-rate shifts of about 1–2 percentage points, so full execution remains the canonical source for benchmark scores. EarlyEval is best used as a cheaper signal for iterative testing. 2