AI labs such as OpenAI assert that their reasoning models, capable of step-by-step problem-solving, outperform non-reasoning counterparts in specialized areas like physics. However, while these claims often hold true, the benchmarking process for reasoning models is significantly more expensive, complicating independent verification. According to Artificial Analysis, evaluating OpenAI's o1 reasoning model across seven benchmarks costs $2,767.05, with other models showing similar price tags. Overall, Artificial Analysis has spent approximately $5,200 on reasoning models compared to $2,400 for non-reasoning ones.
Reasoning models are costly due to their extensive token generation during evaluations. Tokens represent text fragments, and modern benchmarks demand multi-step tasks, increasing token output. Prices per million tokens have risen over time, with some models now costing hundreds of dollars per million tokens. While access to certain models is subsidized by AI labs, this raises concerns about impartiality in testing results.
Economic Challenges in Benchmarking Reasoning Models
Benchmarking reasoning models presents a financial hurdle due to the substantial cost involved. For example, evaluating OpenAI's o1 model requires nearly three thousand dollars, reflecting the complexity and computational demands of these systems. The disparity between reasoning and non-reasoning models is evident, with the former demanding far greater investment. This economic barrier affects independent verification efforts, raising questions about the transparency of performance claims made by AI labs.
Artificial Analysis highlights the significant budget required for evaluating reasoning models. Their expenditure demonstrates that the cost is not just an anomaly but a consistent trend across different models. For instance, Claude 3.7 Sonnet's evaluation costs almost fifteen hundred dollars, while simpler models like OpenAI's o1-mini are relatively cheaper at around one hundred forty dollars. Despite variations, the average cost remains high. Experts suggest that the rise in token generation contributes significantly to these expenses. Modern benchmarks require models to perform complex, multi-step tasks, leading to increased token usage. Consequently, companies charge by the token, exacerbating the financial burden. As more advanced models emerge, the cost per token continues to escalate, further complicating the affordability of comprehensive benchmarking.
Implications of Subsidized Access and Transparency Concerns
Subsidized access to reasoning models from AI labs introduces potential biases into the evaluation process. While organizations like Artificial Analysis receive support for benchmarking activities, this arrangement may compromise the integrity of results. Critics argue that if evaluations cannot be independently replicated, the scientific validity of these assessments comes into question. The involvement of AI labs, even without evidence of manipulation, casts doubt on the reliability of reported scores.
George Cameron of Artificial Analysis acknowledges the rising costs associated with evaluating reasoning models and anticipates further increases as new models are released. However, the issue of subsidized access remains contentious. Some experts believe that allowing AI labs to provide free or discounted model access for testing purposes undermines the credibility of benchmarking outcomes. Ross Taylor from General Reasoning emphasizes the growing difficulty in reproducing results due to the high computational requirements of modern benchmarks. He points out that labs might report performance metrics based on substantial compute resources, making it impossible for academics with limited budgets to verify these findings. Jean-Stanislas Denain from Epoch AI explains that although the cost per token has risen for top-tier models, overall progress in AI efficiency means reaching specific performance levels is less expensive than before. Nonetheless, evaluating state-of-the-art models still incurs considerable expenses. The debate centers on whether current practices align with scientific principles of reproducibility and transparency, prompting calls for more open and accessible benchmarking methodologies.
