According to the paper, none of the 135 submissions from nine current AI agents met all requirements. The authors see the benchmark as a testing ground to systematically improve AI agents for research, not to question whether AI belongs in science at all.
A total of 114 submissions already fail the integrity check: most claim work that was never done or cite numbers that contradict their own logs. The agents readily produced texts that look like follow-up studies, the authors write, but none met the experts’ standards.
Experts set the bar
Each agent receives a scientific paper along with its code and data, plus a research direction. Some of these papers have already been published, while others have only been submitted. From there, the agent is supposed to carry out a follow-up study and write it up as a paper. The 15 tasks come from ten researchers across quantum computing, plasma physics, chemistry, materials science, and scientific machine learning. The directions were proposed by the original papers’ authors themselves, and to their knowledge nothing has been published on these topics yet.
Open research poses a basic problem: there’s no answer key. AI reviewers, which have often been used for this purpose, are considered unreliable. So the experts defined the minimum conditions a successful follow-up study has to meet.
The evaluation checks whether a submission is honest, whether it’s complete, and whether it clears the experts’ bar. GPT-6 Astra handles the scoring, and each submission is judged four times. The agents tested were Claude Opus 4.8, Claude Opus 5, Claude Fable 5, Claude Opus 5.5, GPT-5.6 Sol, GPT-6 Astra, Deepseek V4 Pro, Kimi K3, and GLM-5.3. They had access to powerful machines, internet connectivity, no time limit, and up to five restarts to improve their work. In total, they ran for nearly 2,000 hours.
Only three agents produce honest work at all
Just three agents managed submissions without integrity violations: GPT-6 Astra on 11 of 15 tasks, GPT-5.6 Sol on six, and Claude Opus 5.5 on four. Every submission from the remaining six agents contained at least one violation.
The most common problems are mismatches between paper and code, contradictory numbers, and false claims about experiments that were supposedly carried out. Misreported baseline values pile up among the open models Deepseek V4 Pro, Kimi K3, and GLM-5.3. Two submissions with hardcoded rather than computed results came from Kimi K3 and GLM-5.3.
Only one submission is complete, and it came from GPT-6 Astra. Nearly all others leave out required documents, such as a pre-registered experiment plan, and no agent clears the experts’ bar. Under a more lenient scoring that accepts 90 percent of the rubric items, GPT-6 Astra comes closest, followed by Claude Opus 5.5 and GPT-5.6 Sol. The open models perform considerably worse overall.
Verification script confirms itself
The researchers analyzed 64 work traces and found three typical failure modes. The most common, appearing in 58 cases, is that agents check their own work too leniently. In 34 cases, agents spot a problem but document it instead of fixing it. In 21 cases, they unilaterally shrink the scope of the task.
One example: Claude Fable 5 forgot to launch four planned runs when starting its experiments. Its verification script adopted that same incomplete list and reported “41/41” runs completed. The agent trusted that result until the very end. In a different task, GPT-5.6 Sol noticed that its method missed the targets but wrote the paper anyway instead of starting over. DeepSeek V4 Pro dropped experiments because they were supposedly too slow for the “remaining time budget,” even though there was no time limit.
There are counterexamples, too: in a quantum task, GPT-5.6 Sol caught a fundamental error, stopped the running experiments, and recomputed everything from scratch.
More compute time doesn’t help
Longer runtimes didn’t lead to better results. DeepSeek V4 Pro and GLM-5.3 ran the longest at median times of 19.5 and 13.9 hours but scored poorly. GPT-6 Astra got by with a median of just 1.3 hours and delivered the best results.
Because the strongest model also served as the evaluator, the researchers ran a cross-check: the original papers’ authors judged anonymized AI papers themselves. On whether a paper matched the given research direction, the automated system agreed with the experts in every case.
Concerning scientific quality, the judgments aligned in only about two-thirds of cases. One expert criticized, for instance, that the papers didn’t tell a story. Judgments like these remain the job of human reviewers, the authors say.
The study’s code and data are publicly available, and the project page hosts a leaderboard of the tested models.
AI agents can handle scientific tasks with growing independence, but a substantial gap remains between running experiments and doing independent research. A study from August already showed how much AI agents struggle with unpublished research questions. Epoch AI and Anthropic recently found that agents tend to present their results too favorably and don’t sufficiently question failed approaches.
The picture looks much better for well-defined tasks: GPT-6 Astra solved 49 out of 50 unpublished problems in theoretical physics, though those solutions could be verified unambiguously. An analysis of more than 700 tasks from AI development also showed that while agents take on more and more practical work, humans still make the key research decisions.
