A coding agent can reproduce a paper. That is not the same as reproducing the science.
A new scientific-ML replication study shows how coding agents can produce auditable evidence, while its own edge cases explain why completion must remain separate from scientific fidelity.
The claim is only as strong as the workspace behind it
A coding agent can now do more than write a plausible reproduction script. A new paper asks whether it can rebuild scientific machine-learning experiments from the paper materials, keep a record of what it did, and refuse to call the job complete until the evidence is checked.
The paper, revised on July 10, introduces Paper-replication, a workflow that turns each selected claim into a tracked target. The useful idea is not that an agent can run code. It is that the agent's final answer is no longer the completion record: the persistent workspace is. 1
That distinction matters for any data-science team using coding agents around research, model evaluation, or production analysis. A green status can mean "the recorded checks passed." It should not silently become "the result is scientifically true."
What the workflow changes
A conventional agent task often ends with a notebook, a chart, and a confident paragraph. Paper-replication adds a contract around those artifacts:
- Targets: claims are broken into finite units, with the source location, method components, expected output, and acceptance rule recorded.
- Evidence bundles: each target links the generated result to an execution record, configuration, seed, paper basis, comparison with the source claim, and its place in the final report.
- Validation gates: a target is marked
MATCHEDonly when the required evidence exists and the external checks pass. A copied figure or a plausible-looking output is not enough. - Completion state: the workspace must have a complete specification, no active targets, report coverage for every matched target, and a final PDF. Completion is a property of the recorded workspace, not the agent's last message. 1
This is a workflow-control move. It makes provenance and acceptance criteria part of the task rather than cleanup performed after the agent has already produced an answer.
What was actually tested
The authors evaluated the workflow on four scientific machine-learning papers: physics-informed information field theory, two physics-informed neural-network studies, and SINDy. They ran three independent replications per paper, for 12 runs in total. The evaluation used Codex with GPT-5.4 at the Extra High reasoning setting, and the authors did not allow the papers' original code. 1
The headline result is strong within that test:
- All 12 workspaces passed the completion gate.
- All 158 recorded targets reached
MATCHEDstatus with report coverage. - The runs still differed in how they decomposed papers, how long they took, how many executions were superseded, and how they chose acceptance rules. 2
The decomposition variance is not a footnote. The three PIFT runs produced 8, 8, and 25 targets; the PINN-II runs produced 9, 9, and 15. If one run treats a claim as a single target and another splits it into several checks, a simple pass count is not a stable measure of work or coverage. 1
The reported elapsed replication time also varied by paper. Posterior median times ranged from 1.9 hours for SINDy to 6.9 hours for PINN-II, with the paper reporting wider uncertainty intervals around those medians. That makes the workflow more credible as an engineering study, but it also shows why teams need cost and time records alongside correctness claims. 1
Why 12 out of 12 is not the conclusion
The paper includes a more important result than its completion rate. Across 39 scalar anchor-run observations, 37 fell inside fixed, paper-anchored thresholds. Two did not: one Schrödinger run had a relative L2 error of 4.8% against a 1% threshold, and one Navier-Stokes coefficient result had a 16.4% error against a 10% threshold. Both targets were nevertheless marked
MATCHED under their workspace-specific acceptance rules. 1That is not necessarily a contradiction. A paper claim may be structural, distributional, visual, or numeric, and not every claim has a single universal tolerance. But it creates two separate questions that a dashboard should expose:
- Did the agent complete the target as the workspace defined it?
- How closely did the result meet an independently fixed comparison standard?
The first question is about process completion. The second is about fidelity. Combining them into one green badge would hide the exact information a reviewer needs.
This is the practical lesson for model and data teams: define the comparison rule before the agent runs, and report both the local acceptance rule and the external anchor threshold when they differ. Otherwise, an agent can satisfy a reasonable workflow contract while still producing a result that deserves human review.
The replication record is the product
For teams adopting this pattern, the most reusable part is not the particular skill file. It is the record structure. A useful replication workspace should make these questions answerable without replaying a chat transcript:
- Which claim was being tested, and where does it appear in the source paper?
- Which data, code path, configuration, preprocessing steps, and random seeds were used?
- What ran, what failed, and which execution was superseded?
- Which acceptance rule was applied, and who or what set its threshold?
- Where is the generated evidence in the report?
- What remains an assumption because the paper did not specify it?
That structure supports a compact two-gate review:
| Gate | What it establishes | What it does not establish |
|---|---|---|
| Workspace completion | The recorded targets have evidence, validation results, and report coverage | That every aspect of the paper was reproduced |
| Independent fidelity review | Results meet pre-agreed comparison criteria and withstand scrutiny | That the implementation is the only valid interpretation of the paper |
The distinction is especially important for machine-learning papers, where missing seeds, optimizer state, preprocessing details, sampling choices, or plotting conventions can make exact numerical identity unrealistic. A good record does not pretend those gaps are absent. It names the assumption, measures its effect where possible, and keeps the uncertainty attached to the result.
What the paper does not prove
The study is a case study of one workflow, one model setup, four papers, and 12 runs. Its complete workspaces show that the process can be executed under the tested conditions. They do not establish that every scientific-ML paper can be replicated from paper materials, or that the chosen implementation is uniquely implied by the source.
The authors also note that hash checks can catch direct reuse of paper assets without ruling out every transformed or adversarial reuse. And because completion is relative to the recorded target set, an omitted target can produce a clean workspace without covering an omitted claim. 1
Those limits point to the next evaluation questions. Can another team reproduce the same workspace with a different model? Do independent reviewers agree on the target decomposition and acceptance rules? How much does the result change when the agent is denied a particular tool, given a smaller compute budget, or asked to work from a paper with incomplete artifacts? Can provenance checks detect subtle reuse rather than only copied files?
Until those questions are answered, Paper-replication is best understood as a control system for agent-assisted research, not an automated peer reviewer.
A practical adoption pattern
A data-science team can borrow the workflow without adopting its exact implementation:
- Inventory claims before coding. Separate numeric, distributional, structural, and visual claims. Write the acceptance rule beside each one.
- Prohibit silent substitutions. If the original data, seed, library version, or method detail is missing, record the substitute and its reason.
- Persist every run. Store commands, configurations, outputs, failures, timestamps, and hashes in a machine-readable ledger.
- Review two scores. Show workspace completion and independent fidelity side by side. Never let the first stand in for the second.
- Publish the gaps. A replication report should make unresolved assumptions and failed anchors easy to find.
The broader shift is from asking an agent, "Did you reproduce the paper?" to asking, "Show me the target inventory, the evidence chain, the acceptance rules, and the failed checks." That is a much better question for science, and for any production workflow where a plausible answer is cheaper than a trustworthy one.
Read the primary source
Follow the author
This story was produced automatically by a channel. One sentence is all it takes for Neodrop to keep producing for you.