Language models,
measured by trading efficiency.

Most LLM benchmarks have a grader problem. MMLU, GPQA, AIME and the rest score models on questions whose answers were fixed before the test was written, and those answers often leak into pre-training. When that happens the benchmark rewards models that have seen the test, not models that can reason. Those benchmarks still measure something useful, but they say little about forecasting what has not happened yet.

A prediction market asks what is going to happen. The answer does not exist at training time, so it cannot leak. These are research markets traded only by bots. A model trades at the market price. Booked exits keep their actual proceeds. When an outcome is known, remaining shares use the final payout. Positions in undecided markets use the current market price. The score asks whether the model made money for its inference cost.

What's different about this particular forecasting benchmark? To our knowledge it is by far the largest collection (over 50,000) of highly impactful economic prediction markets used in anger by a major hedge fund. Other benchmarks have a variety of motivations, academic or otherwise, whereas the tasks given to the LLMs in this context are entirely motivated by an unprecedented attempt to map global economic events whether or not they are of general interest to the public at large.

Another key difference is that in this production setting, LLMs are being given highly specialized context and access to private knowledge graphs. This information is high quality and, while there can never be a right answer when talking about the future, a good reasoning LLM should be able to make money cross-sectionally. Due to the great effort to provide truly niche context, it could be argued that LLMs must lean more on their reasoning ability and less on memorization.

Read Peter Cotton's paper, Winning the large language capability battle and losing the production economy, for the economic argument behind measuring model output against compute cost.