Physics has long been considered a harder target for artificial intelligence than mathematics. The problems are messier, the definitions fuzzier, and the phenomena being studied often resist the kind of clean formalization that has allowed AI to thrive in pure math. Yet researchers at Yale University are now questioning whether that reputation is deserved, and the answer has implications for how science itself gets done.
The Benchmark Problem Nobody Talked About
The standard narrative goes like this: AI models score poorly on rigorous physics tests, therefore AI is not ready for serious physics research. Scores as low as 30 percent on six different benchmarks designed to assess AI performance on difficult physics problems seemed to confirm that view.
John Sous at Yale University decided to look more carefully at those benchmarks. What his team found was not a failure of AI, but a failure of the tests themselves. The problems contained errors and ambiguous definitions. Once those were corrected, scores on the same tests climbed to as high as 94 percent.
This is what most coverage of AI in science misses. The question is not just “how well does AI perform?” but “what exactly is it being asked to do?” A poorly constructed question produces a misleading answer. The benchmark scores that shaped the field’s skepticism about AI in physics were, at least in part, measuring the quality of the tests rather than the capability of the models.
Sous draws a clear line from this finding to a broader principle: anything that is well defined, even if it is very hard, is territory where AI can perform at or above the level of the best human experts. That claim is backed by his team’s data, not by speculation.
What Happens When AI Controls the Laser
The more immediate story is not about benchmarks. It is about what is already happening in working laboratories.
Logan Wright, also at Yale University, now teaches a course on using AI in physics research. The syllabus changes constantly to keep pace with developments. His approach to integrating AI into experimental work rests on a specific condition: the inputs and outputs of an experiment must be controllable through code. When that condition is met, an AI system can take over the controls, explore the problem space, and generate meaningful results without human intervention at every step.
A concrete example from Wright’s own laboratory involves studying how different shapes of laser pulses affect a rubidium atom. The researchers use Anthropic’s AI Claude to control the laser pulses and collect data on the effects. Claude can form a prediction about what a given pulse will produce, then test that prediction directly. Experiments of this kind can run overnight and complete work that would otherwise take a human physicist a week. Researchers can even submit ideas remotely through a phone app and review results when they return to the lab.
This is not AI replacing physicists. It is AI handling the volume and repetition that physicists cannot. The human contribution shifts toward designing the experimental setup, interpreting what the results mean, and deciding which questions are worth asking in the first place.
Wright is candid about the limits. Not all of the universe can be explored this way. The approach works within constrained domains where everything can be reduced to code, and it will not generalize easily to the open-ended, observation-driven work that defines the frontier of physics.
The Harder Question Behind the Results
Here is where the conversation becomes genuinely difficult. Wright raises the possibility that AI may soon design an experiment that humans do not fully understand, ask humans to build it, and produce exciting new results. That scenario is not science fiction. It is a near-term extrapolation of what is already happening.
But it surfaces a question that the physics community has not resolved: does a result count as science if no one understands how it was reached? The same debate is already underway in mathematics, where AI-generated proofs have raised concerns that the field could be overwhelmed by outputs it cannot meaningfully evaluate.
Sous points to formalisation as the key tool that has allowed AI to make such rapid progress in mathematics. Formalisation converts a mathematical idea into computer code that can be checked systematically. If a theorem survives that process, it is considered correct beyond reasonable doubt. Physics has begun to use similar techniques, but the field is at an early stage compared to mathematics, and many physics problems resist the kind of precise definition that formalisation requires.
The creative, interpretive work of physics, identifying a confusing phenomenon, building a model that captures it, testing that model against reality, remains genuinely open. Sous describes it as an artistic process. Whether AI can do that end-to-end, defining the problem, creating the model, exploring it, and testing it, is a question that current evidence does not yet answer.
In Short
AI’s apparent weakness in physics was partly a measurement problem, not a capability problem. Corrected benchmarks tell a different story. In constrained experimental settings, AI is already accelerating research in meaningful ways, handling overnight what would take humans a week. The deeper challenge is not computational. It is conceptual: physics at its frontier is about making sense of things we do not yet understand, and that remains territory where the role of AI is genuinely unclear.
Based on reporting from New Scientist.