Beyond Final Scores changes how product teams should test long-horizon AI agents: not just by the final score, but by how they frame a solution, execute it, recover from regressions, reuse experience, and respond to the harness around them. The paper evaluates 7 frontier models on 36 long-horizon tasks with rule-based views of Solution Framing, Execution, and Feedback Control, plus controlled experience-reuse and harness comparisons. Its conclusion is blunt: current agents look more like engineering optimizers than autonomous researchers. 1
That matters because one final score can hide different product risks. In the paper's process analysis, Gemini captured 83.7% of its eventual peak early, leaving only 16.5% of later headroom; another model may spend more of its run finding and repairing a path. Experience transfer also cut both ways: DeepSeek-V4-Pro gained 0.093 on inter-task avg@3, while Gemini-3.1-Pro fell 0.017. Among 252 best-of-three solutions, manual review retained only 3 as genuinely novel, versus 16 evaluation-specific shortcuts. These are controlled paper results, not deployment guarantees. 1
First build: add a shadow evaluation layer around your current agent. Log each evaluated commit with verifier score, diff, build errors, wall time, model, and harness. Compute three process views: framing, delivery, and retention/recovery. At a branch point, compare the next run with and without extracted lessons in isolated workspaces. Then hold task, model, budget, and environment fixed while swapping the native and shared scaffolds. The paper is an evaluation framework, not a model checkpoint; its Hugging Face paper entry links the public project page. Keep the current production path until your own task-success, reliability, cost, and regression gate passes. 2
评论