AI models are increasingly being tested on their ability to predict real-world events, from NBA championship winners to presidential meetings. In these experiments, GPT-5 performed best, followed by Grok 4, Claude Sonnet, Gemini, and Llama. The results suggest that AI could become a powerful forecasting tool, although highly skilled human forecasters still retain important advantages.
Testing AI’s ability to predict the future
“I quickly started noticing that the model worked very well,” Xu tells me. Even without special instructions or guidance, the models were sometimes able to make better predictions than the average person. The tests included questions such as who would win the NBA Championship and who President Trump would meet with.
The team continued developing Prophet Arena, a benchmark designed to measure a model’s predictive ability. It is one of several benchmarks intended to test the future-facing capabilities of AI.
Another project, Prediction Arena, involved a Harvard University researcher and a company called Arcada Labs awarding Frontier Models $10,000 for a 57-day prediction-market bet. Although the model consistently lost money during the experiment, the researchers found evidence that its performance improved over time.
How AI systems make forecasts
Xu said AI can make predictions most effectively through custom harnesses. The most promising approach appears to involve giving agents responsibility for their own tasks, including reading the news, collecting additional data, and analyzing the broader system.
AI still often trails highly skilled human forecasters known as superforecasters. Humans cannot consume as much information as AI systems, but Xu explains that they have a deeper understanding of the real world and the dynamics behind news stories.
Human forecasters face growing competition
There is, however, growing evidence that the advantage held by human superforecasters is narrowing. A recent contest called the Metaculus Cup showed that AI can sometimes defeat even the best human predictors.
Another benchmark, ForecastBench, could provide further evidence that AI can match human forecasting ability when the results of its ongoing contest are released. Large research institutions may increasingly look to forecasting as a powerful way to measure progress in machine intelligence.
Potential effects on AI-exposed industries
These predictive experiments suggest that industries built around transactions and analysis, including mathematics, coding, and law, could soon be disrupted by AI. The best human traders may take on a new role, overseeing and coordinating AI agents that can predict future events more accurately than they can.
So far, Xu’s own AI suggestions have not seemed especially impressive. The team has not performed above average in recent weeks, and Xu says he will not use its recommendations to make personal choices.
However, AI will likely improve and may soon become too effective at dominating fantasy leagues. Xu hopes that does not happen, because it would take away from the fun of trying to win independently and earn bragging rights.
This is an edition of Will Knight’s AI Lab Newsletter. Read previous newsletters here.
Source: www.wired.com