AI agents can run the lab. They still cannot reliably choose the research. (2026)

AI agents can now run the lab. They still cannot reliably choose the research.

The latest evidence is unusually concrete: in two "shadow evaluations," frontier agents received unpublished NeurIPS research questions, six days, a virtual machine, GPUs, and up to $3,000 in API credits. They debugged environments, ran experiments, searched the literature, and produced camera-ready papers. The original paper authors still rejected both outputs, scoring them 2/6 and 1/6 overall. 1

For a PM, the signal is not that research agents are useless. It is that a benchmark win on a verifiable task is a poor proxy for an agent that must decide what to try, when to abandon it, and what evidence is strong enough to support a claim.

What changed

The paper, Can AI agents conduct open-ended AI research?, comes from a 23-author team spanning Princeton University, the UK AI Security Institute, UC Berkeley, Johns Hopkins, Stanford, and other institutions. The v2 preprint is dated August 7, 2026. Its method is called a shadow evaluation: give an agent the central question from a high-quality paper that is not yet public, then have the original authors grade the agent's answer as they would a conference submission. 1

That design closes a gap between two common evaluation styles:

Verifier-scored tasks are repeatable and cheap to scale, but they usually optimize a known metric. An agent can improve a benchmark score without deciding whether the metric captures an interesting question.
Blind peer review looks more like real research, but conference review is noisy, overloaded, and often cannot inspect every technical detail.
Shadow evaluation keeps the question open-ended while giving the reviewers deep knowledge of the original problem. It is slower and smaller-scale, but it can test whether the agent made genuine progress rather than merely completed a workflow. 1

The two agents did the mechanical work well. They conducted literature reviews, debugged crashing GPU environments, ran hundreds of experiments, and compiled LaTeX papers without human help. The failure appeared in the research judgment around that work. 2

The expert scores make the boundary visible:

Dimension	Personas paper	TabPFN paper
Quality	2/4	1/4
Clarity	1/4	2/4
Significance	2/4	2/4
Originality	3/4	2/4
Overall	2/6	1/6

The scores come from the original authors' review table; both papers were unambiguous rejects. The reviewers cited unprincipled data and experiment choices, conclusions that did not follow from the evidence, dense writing, and weak justification over prior work. 1

The authors identify five recurring failure modes:

Weak stopping judgment. The agents proposed directions that looked promising, then tested them with small, hand-curated, or synthetic datasets and treated underpowered negative results as substantive findings.
Poor resource awareness. Both main runs ended with less than half of the API budget spent and hours left on the clock, even though the agents could monitor usage and were told to explore.
Caveats instead of new ideas. Self-reviews surfaced many of the same problems later raised by human reviewers. The agents mostly narrowed claims or added caveats rather than changing the research design.
Ineffective backtracking. Both agents abandoned their most ambitious research targets early and never made a fundamental change of direction afterward.
Instruction drift. They ignored explicit requirements about exploration time, review cadence, and paper length. 1

A robustness run using GPT-5.6 Sol with Codex on one of the questions reproduced nearly all of the same failure patterns. That makes the result harder to dismiss as a single model or scaffold problem. It still does not make the result a general law: the study covers two papers, the reviews were non-blind, and the reviewers already knew the questions deeply. 1

Why PMs should care

The product boundary is not "can the agent do research?" That question is too broad to instrument. The useful boundary is whether the workflow contains a verifiable checkpoint for each judgment the agent is being asked to make.

A coding agent can run a test suite. A research agent often has to decide whether the test suite is the right test, whether the dataset is large and representative enough, whether a negative result is informative, and whether a better hypothesis deserves another day of compute. Those choices are not token-generation problems alone. They are decisions about evidence, novelty, taste, and opportunity cost.

Cloudflare's recent engineering post makes the same boundary visible from the production side. Its Agent Development Lifecycle extends the traditional software lifecycle because agents have made implementation faster than review, deployment, maintenance, and incident response. Cloudflare argues that agent-driven software factories need programmatic operations, reproducible environments, atomic and reversible changes, permissions, real-time triggers, and observability across the whole lifecycle. The post was published August 4, 2026. 3

That is a useful product analogy for research agents. If the workflow can generate ten times more experiments, the bottleneck moves to experiment selection, evidence review, and deciding when to stop. A faster executor does not remove that bottleneck; it can make weak judgment more expensive.

The social signal is strong enough to merit attention, not strong enough to prove adoption. Sayash Kapoor, an incoming UC Berkeley professor whose profile lists AI agents, policy, evaluations, and AI for science, summarized the result on X on July 30. His framing was blunt: agents were fluent at engineering tasks, but neither output came close to the bar of a top conference paper. The post had 167,181 views when retrieved. 2

There is also a small, accessible explainer signal. Frontier AI Research Digest: The Great Agent Reality Check, from the YouTube channel Frontier AI Research Digest, was published August 2 and had 1 view when checked. Its four-and-a-half-minute description explicitly includes the shadow-evaluation paper alongside other agent-evaluation papers. Treat it as a compact orientation aid, not evidence of market demand or production use. 4

Click to open on YouTube

Loading content card…

The practical conclusion is narrower than "AI cannot do science." The study shows early evidence that current agents can automate much of the research execution loop while struggling with open-ended research selection and judgment. The authors explicitly call the evidence tentative and plan larger evaluations with stronger models and scaffolds. 1

A visual comparison of progress when AI-automatable work is unconstrained versus when human judgment remains a bottleneck. — The article's Amdahl's-law diagram separates work that can be automated from the bottleneck that remains. It is a useful mental model for research agents: faster experiments help only if hypothesis choice and evidence review do not become the limiting steps. 5

How to implement now

Do not start with an autonomous scientist. Start with a research workflow whose judgment points are explicit.

Choose a bounded research task. Pick a workflow with a known corpus, a defined output, and at least one checkable intermediate result: reproduce a published experiment, compare retrieval strategies on a fixed dataset, or optimize a model against a fixed metric. Keep genuinely open-ended hypothesis selection human-owned at first.
Write the evidence contract before the agent runs. Define what counts as an adequate dataset, which baselines are mandatory, what statistical or reproducibility checks must pass, and what would justify abandoning a direction. The agent should not be allowed to quietly turn "find a useful effect" into "find any effect."
Separate execution from research review. Give the executor tools for code, literature search, data inspection, and experiment management. Give a separate reviewer—or a human—authority over hypothesis quality, evidence sufficiency, and go/no-go decisions. Log disagreements instead of collapsing them into one score.
Add a backtracking budget, not only a token budget. Track how many distinct hypotheses were explored, how long each survived, why it was retired, and whether new evidence caused a real change of direction. A system that finishes early after a weak self-review should trigger escalation, not success.
Evaluate the workflow at three levels. Measure engineering completion, research validity, and decision quality separately. Useful metrics include reproducibility rate, baseline coverage, expert acceptance, unsupported-claim rate, time spent before abandoning a path, budget utilization, and the percentage of reviewer objections that lead to a changed experiment rather than another caveat.
Run shadow tests before granting autonomy. Give the system a held-out question or an internal project whose answer is not in its retrieval corpus. Have an expert who knows the problem grade the output against a fixed rubric. Record sample size, reviewer blinding, model, scaffold, time, compute, and every human intervention so a good score cannot hide an easy task.

The first product worth building is therefore a research copilot with an explicit review gate, not an agent that silently owns the research agenda. Let the system make experiments cheap. Keep the authority to decide what deserves belief—and what deserves another week of work—visible, reviewable, and measurable.