Physicist pauses new PhD intake as AI solves 49 of 50 benchmark problems
In newly published interviews, researchers describe faster calculations and growing unease about how students will develop the judgment those calculations once taught.
Loading page…
In newly published interviews, researchers describe faster calculations and growing unease about how students will develop the judgment those calculations once taught.
Listen to this story
TheoPhysEval’s result shows that tool-equipped AI agents can now handle many demanding, checkable theoretical-physics calculations, but it does not establish that they can independently make major discoveries. The shift is already affecting research and training: benchmark co-creator Suvrat Raju is pausing new PhD admissions while considering students’ prospects, as physicists debate how trainees will develop the judgment and intuition once built through calculation work.
GPT-6-Astra, running in Codex with web search and calculation tools, solved 49 of 50 problems at an average reported cost below $3 per problem.
The test’s answer key was corrected after Astra identified subtle errors in human solutions; the benchmark authors said they would not build a second version.
Andreas Karch uses agents to check old papers, while Daniel Harlow says deciding what to work on—not calculating—is the main bottleneck in his research.
AI can finish calculations that once helped young physicists prove themselves. Whether it can replace the training those calculations provided is another matter. In interviews published November 17 by Understanding AI, benchmark co-creator Suvrat Raju said he has stopped taking new PhD students while he assesses their prospects, following AI’s near-perfect performance on his team’s research-level physics test.
Dan Kagan-Kans’s report draws on conversations with Raju and six other experts about AI’s growing role in theoretical physics. Their experiences expose a divide: models are handling substantial calculations, but choosing worthwhile questions—and preparing people to choose them—remains a different challenge.
The benchmark, TheoPhysEval, is not new: its paper was first submitted October 9. It contains 50 unpublished theoretical-physics problems with checkable answers. A world expert would typically need about a day or less for one problem; a researcher in a neighboring specialty could take considerably longer.
The authors report that GPT-6-Astra, running in Codex, answered 49 correctly at an average cost below $3 per problem. It worked without human hints, with tools that could search the literature and run numerical or symbolic calculations. That is a result for a tool-equipped agent, not an unaided chatbot.
There was also a revealing grading reversal. Kagan-Kans reports that, when shown the human-written solutions to apparent misses, Astra identified subtle errors in those solutions. Correcting the answer key brought its score to 49 of 50. Raju said he would not build a second version: contributors had already supplied the hardest well-defined calculations they could.
Unpublished questions do not mean the agents lacked relevant learned material. The paper says removing web search and background papers changed accuracy only slightly on average, suggesting significant memorization of the literature. The finding concerns concrete research calculations, not proof that an agent can independently deliver a major physics discovery.
For some interviewees, the change is already practical. University of Texas at Austin physicist Andreas Karch said he runs agents around the clock and sends old papers through Astra to look for errors. OpenAI supplies his high-tier ChatGPT subscription free. Harvard’s Matthew Schwartz described running 20 problem-solving threads during an October interview, with 10 to 20 agents per thread and tokens supplied by Anthropic.
MIT’s Daniel Harlow offered a different account. In his area, most work goes into deciding what to do; by the calculation stage, he considers himself roughly 95% finished. He uses language models to learn about other areas and find ideas to borrow, rather than treating them as a wholesale change to his research.
Reading the resulting work presents another bottleneck. Understanding AI reports that January–October submissions to arXiv’s high-energy theory category rose about 30% from the same period last year. Schwartz thought many papers had solid science despite poor writing, but the report describes researchers struggling to assess the expanding volume.
If you do enough of this mechanical grinding through problems, at some point, you pick up the concepts behind it.
Andreas Karch, speaking to Understanding AI
That leaves an apprenticeship problem, not just a productivity gain. Raju described calculations as a way to judge students’ talent. Karch questioned how students could skip that work without losing the intuition it develops. He also described greater pressure to guard ideas, fearing someone else could quickly hand them to an agent.
For Raju, uncertainty about the next generation’s prospects has already changed a concrete decision: he is holding off on accepting new doctoral students.
Loading discussion...
Join the conversation
Explain where useful training ends and unnecessary repetition begins.
Be the first to share a perspective or an experience.
Reader comments
Newest comments first. Replies stay oldest first.