A defining reminiscence from my senior 12 months of highschool was a nine-hour math examination with simply six questions. Six of the highest scorers gained slots on the U.S. crew for the Worldwide Math Olympiad (IMO), the world’s longest operating math competitors for highschool college students. I didn’t make the reduce, however turned a tenured arithmetic professor anyway.
This 12 months’s olympiad, held final month on Australia’s Sunshine Coast, had an uncommon sideshow. Whereas 110 college students from around the globe went to work on complicated math issues utilizing pen and paper, a number of AI corporations quietly examined new fashions in growth on a computerized approximation of the examination. Proper after the closing ceremonies, OpenAI and later Google DeepMind introduced that their fashions earned (unofficial) gold medals for fixing 5 of the six issues. Researchers like Sébastien Bubeck of OpenAI celebrated these fashions’ successes as a “moon touchdown second” by trade.
However are they? Is AI going to interchange skilled mathematicians? I’m nonetheless ready for the proof.
On supporting science journalism
In case you’re having fun with this text, think about supporting our award-winning journalism by subscribing. By buying a subscription you’re serving to to make sure the way forward for impactful tales in regards to the discoveries and concepts shaping our world at this time.
The hype round this 12 months’s AI outcomes is simple to grasp as a result of the olympiad is difficult. To wit, in my senior 12 months of highschool, I put aside calculus and linear algebra to concentrate on olympiad-style issues, which had been extra of a problem. Plus the cutting-edge fashions nonetheless in growth did so a lot better on the examination than the business fashions already on the market. In a parallel contest administered by MathArena.ai, Gemini 2.5 professional, Grok 4, o3 excessive, o4-mini excessive and DeepSeek R1 all failed to provide a single utterly appropriate answer. It exhibits that AI fashions are getting smarter, their reasoning capabilities enhancing quite dramatically.
But I’m nonetheless not frightened.
The newest fashions simply received a great grade on a single check—as did most of the college students—and a head-to-head comparability isn’t fully honest. The fashions typically make use of a “best-of-n” technique, producing a number of options after which grading themselves to pick out the strongest. That is akin to having a number of college students work independently, then get collectively to choose the very best answer and submit solely that one. If the human contestants had been allowed this feature, their scores would possible enhance too.
Different mathematicians are equally cautioning towards the hype. IMO gold medalist Terence Tao (at the moment a mathematician on the College of California, Los Angeles) famous on Mastodon that what AI can do will depend on what the testing methodology is. IMO president Gregor Dolinar mentioned that the group “can’t validate the strategies [used by the AI models], together with the quantity of compute used or whether or not there was any human involvement, or whether or not the outcomes will be reproduced.”
In addition to, IMO examination questions don’t examine to the sorts of questions skilled mathematicians attempt to reply, the place it might take 9 years, quite than 9 hours, to resolve an issue on the frontier of mathematical analysis. As Kevin Buzzard, a arithmetic professor at Imperial School London, mentioned in a web based discussion board, “Once I arrived in Cambridge UK as an undergraduate clutching my IMO gold medal I used to be in no place to assist any of the analysis mathematicians there.”
As of late, mathematical analysis can take a couple of lifespan to accumulate the proper experience. Like lots of my colleagues, I’ve been tempted to strive “vibe proving”—having a math chat with an LLM as one would with a colleague, asking “Is it true that…” adopted by a technical mathematical conjecture. The chatbot typically then provides a clearly articulated argument that, in my expertise, tends to be appropriate in terms of normal subjects however subtly flawed on the innovative. For instance, each mannequin I’ve requested has made the identical delicate mistake in assuming that the speculation of idempotents behaves the identical for weak infinite-dimensional classes because it does for peculiar ones, one thing that human specialists (belief me on this) in my subject know to be false.
I’ll by no means belief an LLM—which at its core is simply predicting what textual content will come subsequent in a string of phrases, primarily based on what’s in its dataset—to supply a mathematical proof that I can’t confirm myself.
The excellent news is, we do have an automatic mechanism for figuring out whether or not proofs will be trusted. Comparatively latest instruments referred to as “proof assistants” are software program packages (they don’t use AI) designed to test whether or not a logical argument proves the acknowledged declare. They’re more and more attracting consideration from mathematicians like Tao, Buzzard and myself who need extra assurance that our personal proofs are appropriate. And so they supply the potential to assist democratize arithmetic and even enhance AI security.
Suppose I acquired a letter, in unfamiliar handwriting, from Erode, a metropolis in Tamil Nadu, India, purporting to include a mathematical proof. Perhaps its concepts are good, or perhaps they’re nonsensical. I’d must spend hours rigorously learning each line, ensuring the argument flowed step-by-step, earlier than I’d have the ability to decide whether or not the conclusions are true or false.
But when the mathematical textual content had been written in an acceptable laptop syntax as an alternative of pure language, a proof assistant might test the logic for me. A human mathematician, reminiscent of I, would then solely want to grasp the which means of the technical phrases within the theorem assertion. Within the case of Srinivasa Ramanujan, a generational mathematical genius who did hail from Erode, an knowledgeable did take the time to rigorously decipher his letter. In 1913 Ramanujan wrote to the British mathematician G. H. Hardy together with his concepts. Fortunately, Hardy acknowledged Ramanujan’s brilliance and invited him to Cambridge to collaborate, launching the profession of one of many all-time mathematical “greats.”
What’s fascinating is that a number of the AI IMO contestants submitted their solutions within the language of the Lean laptop proof assistant in order that the pc program might routinely test for errors of their reasoning. A start-up referred to as Harmonic posted formal proofs generated by their mannequin for 5 of the six issues, and ByteDance achieved a silver-medal stage efficiency by fixing 4 of the six issues. However the questions needed to be written to accommodate the fashions’ language limitations, they usually nonetheless wanted days to determine it out.
Nonetheless, formal proofs are uniquely reliable. Whereas so-called “reasoning” fashions are prompted to interrupt issues down into items and clarify their “pondering” step-by-step, the output is as prone to produce an argument that sounds logical however isn’t, as to represent a real proof. Against this, a proof assistant won’t settle for a proof until it’s totally exact and totally rigorous, justifying each step in its chain-of-thought. In some circumstances, a hand-waving or approximate answer is sweet sufficient, however when mathematical accuracy issues, we should always demand that AI-generated proofs are formally verifiable.
Not each utility of generative AI is so black and white, the place people with the proper experience can decide whether or not the outcomes are appropriate or incorrect. In life, there may be a whole lot of uncertainty and it’s simple to make errors. As I realized in highschool, among the finest issues about math is the truth that you’ll be able to show definitively that some concepts are flawed. So I’m joyful to have an AI attempt to resolve my private math issues, however provided that the outcomes are formally verifiable. And we aren’t fairly there, but.
That is an opinion and evaluation article, and the views expressed by the writer or authors will not be essentially these of Scientific American.
It’s Time to Stand Up for Science
Earlier than you shut the web page, we have to ask in your help. Scientific American has served as an advocate for science and trade for 180 years, and we predict proper now could be probably the most vital second in that two-century historical past.
We’re not asking for charity. In case you turn out to be a Digital, Print or Limitless subscriber to Scientific American, you’ll be able to assist be sure that our protection is centered on significant analysis and discovery; that we have now the sources to report on the choices that threaten labs throughout the U.S.; and that we help each future and dealing scientists at a time when the worth of science itself typically goes unrecognized. Click on right here to subscribe.
