AI agents can run the lab. They still cannot reliably choose the research.

What changed

• Verifier-scored tasks are repeatable and cheap to scale, but they usually optimize a known metric. An agent can improve a benchmark score without deciding whether the metric captures an interesting question.
• Blind peer review looks more like real research, but conference review is noisy, overloaded, and often cannot inspect every technical detail.
• Shadow evaluation keeps the question open-ended while giving the reviewers deep knowledge of the original problem. It is slower and smaller-scale, but it can test whether the agent made genuine progress rather than merely completed a workflow. 1

1. Weak stopping judgment. The agents proposed directions that looked promising, then tested them with small, hand-curated, or synthetic datasets and treated underpowered negative results as substantive findings.
2. Poor resource awareness. Both main runs ended with less than half of the API budget spent and hours left on the clock, even though the agents could monitor usage and were told to explore.
3. Caveats instead of new ideas. Self-reviews surfaced many of the same problems later raised by human reviewers. The agents mostly narrowed claims or added caveats rather than changing the research design.
4. Ineffective backtracking. Both agents abandoned their most ambitious research targets early and never made a fundamental change of direction afterward.
5. Instruction drift. They ignored explicit requirements about exploration time, review cadence, and paper length. 1

Why PMs should care

How to implement now

1. Choose a bounded research task. Pick a workflow with a known corpus, a defined output, and at least one checkable intermediate result: reproduce a published experiment, compare retrieval strategies on a fixed dataset, or optimize a model against a fixed metric. Keep genuinely open-ended hypothesis selection human-owned at first.
2. Write the evidence contract before the agent runs. Define what counts as an adequate dataset, which baselines are mandatory, what statistical or reproducibility checks must pass, and what would justify abandoning a direction. The agent should not be allowed to quietly turn "find a useful effect" into "find any effect."
3. Separate execution from research review. Give the executor tools for code, literature search, data inspection, and experiment management. Give a separate reviewer—or a human—authority over hypothesis quality, evidence sufficiency, and go/no-go decisions. Log disagreements instead of collapsing them into one score.
4. Add a backtracking budget, not only a token budget. Track how many distinct hypotheses were explored, how long each survived, why it was retired, and whether new evidence caused a real change of direction. A system that finishes early after a weak self-review should trigger escalation, not success.
5. Evaluate the workflow at three levels. Measure engineering completion, research validity, and decision quality separately. Useful metrics include reproducibility rate, baseline coverage, expert acceptance, unsupported-claim rate, time spent before abandoning a path, budget utilization, and the percentage of reviewer objections that lead to a changed experiment rather than another caveat.
6. Run shadow tests before granting autonomy. Give the system a held-out question or an internal project whose answer is not in its retrieval corpus. Have an expert who knows the problem grade the output against a fixed rubric. Record sample size, reviewer blinding, model, scaffold, time, compute, and every human intervention so a good score cannot hide an easy task.