A couple of months earlier than the 2025 Worldwide Mathematical Olympiad (IMO) in July, a three-person staff at OpenAI made an extended guess that they may use the competitors’s brutally robust issues to coach a man-made intelligence mannequin to suppose by itself for hours in order that it was able to writing math proofs. Their aim wasn’t merely to create an AI that might do advanced math however one that might consider ambiguity and nuance—expertise AIs will want if they’re to sometime tackle many difficult real-world duties. In actual fact, these are exactly the abilities required to create synthetic basic intelligence, or AGI: human-level understanding and reasoning.
The IMO, held this 12 months on Australia’s Sunshine Coast, is the world’s premier math competitors for top schoolers, bringing collectively prime contenders from greater than 100 nations. All are given the identical six issues—three per day, every price seven factors—to resolve over two days. However these issues are nothing like what you in all probability bear in mind from highschool. Moderately than a short numeric reply, every calls for sustained reasoning and creativity within the type of a pages-long written proof. These logical, step-by-step arguments must span many fields of arithmetic—precisely the type of issues that, till simply this 12 months, AI methods failed at spectacularly.
The OpenAI staff of researchers and engineers—Alex Wei, Sheryl Hsu and Noam Brown—used a general-purpose reasoning mannequin: an AI designed to “suppose” by difficult issues by breaking them into steps, checking its personal work and adapting its strategy because it goes. Although AI methods couldn’t formally compete as contributors, the notoriously robust take a look at served as an indication of what they’ll do, and the AIs tackled this 12 months’s questions in the identical take a look at format and with the identical constraints as human contributors. Upon receiving the questions, the staff’s experimental system labored for 2 4.5‑hour periods (simply as the coed contestants did), with out instruments or the Web—it had completely no exterior help from instruments resembling search engines like google and yahoo or software program designed for math. The proofs it produced had been graded by three former IMO medalists and posted on-line. The AI accomplished 5 of the six issues accurately, receiving 35 out of 42 factors—the minimal required for an IMO gold medal. (Google’s DeepMind AI system additionally achieved that rating this 12 months.) Out of 630 rivals, solely 26 college students, or 4 p.c, outperformed the AI; 5 college students achieved excellent 42s. Given {that a} 12 months in the past language-based AI methods like OpenAI’s struggled to do elementary math, the outcomes had been a dramatic leap in efficiency.
On supporting science journalism
Should you’re having fun with this text, think about supporting our award-winning journalism by subscribing. By buying a subscription you’re serving to to make sure the way forward for impactful tales in regards to the discoveries and concepts shaping our world in the present day.
Within the following dialog, Scientific American spoke with two members of the OpenAI staff, Alex Wei and Sheryl Hsu, to debate how they performed their work, why the mannequin’s lack of response to the sixth query was truly a significant step towards addressing AI’s “hallucination” downside and the way creating a system able to writing advanced proofs might assist result in synthetic basic intelligence.
[An edited transcript of the interview follows.]
What led you to instantly start getting ready an AI mannequin for the IMO just some months earlier than the competitors? What was the spark?
WEI: I had been interested by math proofs for fairly some time. I’m on a staff at OpenAI known as MathGen. We had simply seen the outcomes progress quite a bit. We felt like we had a shot to get a mannequin that might do rather well on the IMO, and we needed to make a mad sprint to get there.
HSU: I used to do math competitions. [Wei] used to do math competitions—he was quite a bit higher than me. The IMO is unquestionably well-known inside the [AI research] neighborhood, together with amongst researchers at OpenAI. So it was actually inspiring to push particularly for that.
Are you able to discuss your determination to work with a basic‑goal AI system quite than a system that was particularly designed to reply math issues?
WEI: The philosophy is that we need to construct basic‑goal AI and develop strategies that don’t simply work for math. Math is an excellent proving floor for AI as a result of it’s pretty goal: when you have a proof, it’s simpler to get consensus on whether or not it’s appropriate. That’s more durable for, say, poetry—you’ll have extra disagreement amongst readers. And IMO issues are very onerous, so we needed to sort out onerous issues with basic‑goal strategies within the hope that they’ll additionally apply to domains past math.
HSU: I’d additionally say the aim at OpenAI is to construct AGI—it’s not essentially to put in writing papers or win competitions. It was necessary that all the pieces we did for this undertaking even be helpful for the larger aim of constructing AGI and higher fashions that customers can truly use.
In what methods might a reasoning mannequin profitable a gold within the IMO assist result in AGI?
WEI: One perspective is to suppose by way of how lengthy duties take. A 12 months in the past, ChatGPT might solely do very primary math issues. Two years in the past—and even a 12 months and a half in the past—we had been usually interested by grade‑college math issues you’d discover on fifth‑grade homework. For somebody actually good at math, these take a second or two to learn and resolve. Then we began evaluating utilizing AIME [the American Invitational Mathematics Examination, a 15-question high school math contest]. That takes round 10 minutes per downside, with about three hours for 15 issues. The IMO is 4 and a half hours for simply three issues—that’s 90 minutes per downside. ChatGPT began off being good for fast questions. Now it’s higher at longer‑working duties, resembling “Are you able to edit this paragraph for me?” As AI improves, you may develop the time horizon of duties, and you may see that development clearly in math.
HSU: One other side is that reasoning fashions had been beforehand excellent at duties which are straightforward to confirm. Should you’re fixing a non‑proof‑primarily based math downside, there’s one numerically appropriate reply. It’s straightforward to test. However in the actual world—and within the duties folks truly need assist with—it’s extra advanced. There’s nuance: possibly it’s largely appropriate however has some errors; possibly it’s appropriate however might be stylized higher. Proof‑primarily based math isn’t trivial to judge. If we take into consideration AGI, these duties received’t be straightforward to guage as appropriate or not; they’ll be extra loosely specified and more durable general.
What was the method for coaching the mannequin?
WEI: Basically, reinforcement studying trains a mannequin by rewarding good habits and penalizing unhealthy habits. Should you repeatedly reinforce good habits and discourage unhealthy habits, the mannequin turns into extra prone to exhibit the nice habits.
HSU: Towards the top, we additionally scaled up take a look at‑time compute [how long the AI model was able to “think” before answering]. Beforehand, for a human, issues of this kind may be a couple of minutes; now we had been scaling to hours. That further considering time gave shocking good points. There was a second once we ran evaluations on our inside take a look at set that took a very long time due to the elevated take a look at‑time compute. Once we lastly appeared on the outcomes—and Alex graded them—seeing the progress made me suppose gold may be inside attain. That was fairly thrilling.
On the IMO take a look at, the mannequin you developed bought 5 out of six solutions appropriate. However with the sixth query, the mannequin didn’t attempt to present a solution. Are you able to inform me extra in regards to the significance of this response?
WEI: The mannequin realizing what it doesn’t know was one of many early indicators of [progress] we noticed. At the moment if you happen to use ChatGPT, you’ll typically see “hallucinations”—fashions don’t reliably know once they don’t know. That functionality isn’t particular to math. I’d adore it if, for on a regular basis questions, the mannequin might truthfully say when it doesn’t know as a substitute of giving a solution I have to confirm independently.
What sort of affect might your work on this mannequin have on future fashions?
HSU: Every thing we did for this undertaking is pretty basic‑goal—having the ability to grade outputs that aren’t single solutions and to work on onerous issues for a very long time whereas making regular progress. These contributed quite a bit to the success right here, and now we and others at OpenAI are making use of them past math. It’s not in GPT‑5, however in future fashions, we’re excited to combine these capabilities.
WEI: Should you have a look at the options we publicly posted for the IMO issues, some are very lengthy—5 to 10 pages. This mannequin can generate lengthy outputs which are constant and coherent, with out errors. Many present state‑of‑the‑artwork fashions can’t produce a very coherent 5‑web page report. I’m excited that this care and precision will assist in many different domains.
