Skip to content

Benchmark

JudgeBench: Math Score

Math benchmarks measure how well models solve maths problems. Scores are percentages. Higher is better.

Models measured
13
Scores recorded
23
Sources
1
IQ weight
1.33

Score distribution

62.8%
Mean
64.3%
Median
16.7%
Std dev
41.2%
10th percentile
85.7%
90th percentile
32.4%
Min
32.4%87.5%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.33 for chat models, measured over 13 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1o3 MiniGPT-4o Dataset, Prompted Judge, Arena-Hard (o3-mini-2025-01-31 (high)), Math Score· JudgeBench data docs87.5%
2O1 PreviewGPT-4o Dataset, Prompted Judge, Arena-Hard (o1-preview-2024-09-12), Math Score· JudgeBench data docs85.7%
3o3 MiniGPT-4o Dataset, Prompted Judge, Arena-Hard (o3-mini-2025-01-31 (medium)), Math Score· JudgeBench data docs85.7%
4o3 MiniGPT-4o Dataset, Prompted Judge, Arena-Hard (o3-mini-2025-01-31 (low)), Math Score· JudgeBench data docs83.9%
5O1 MiniGPT-4o Dataset, Prompted Judge, Arena-Hard (o1-mini-2024-09-12), Math Score· JudgeBench data docs82.1%
6DeepSeek R1GPT-4o Dataset, Prompted Judge, Arena-Hard (DeepSeek-R1-250120), Math Score· JudgeBench data docs80.4%
7GPT 4oGPT-4o Dataset, Prompted Judge, Arena-Hard (gpt-4o-2024-05-13), Math Score· JudgeBench data docs75%
8GPT-4o miniGPT-4o Dataset, Prompted Judge, Arena-Hard (gpt-4o-mini-2024-07-18), Math Score· JudgeBench data docs69.6%
9Llama 3.1 405B InstructGPT-4o Dataset, Prompted Judge, Arena-Hard (Llama-3.1-405B-Instruct), Math Score· JudgeBench data docs69.6%
10Claude 3.5 SonnetGPT-4o Dataset, Prompted Judge, Arena-Hard (claude-3-5-sonnet-20240620), Math Score· JudgeBench data docs66.1%
11GPT 4oGPT-4o Dataset, Prompted Judge, Vanilla (gpt-4o-2024-05-13), Math Score· JudgeBench data docs66.1%
12gemini-1.5-pro-001GPT-4o Dataset, Prompted Judge, Arena-Hard (gemini-1.5-pro-001), Math Score· JudgeBench data docs64.3%
13GPT 4oClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (gpt-4o-2024-05-13), Math Score· JudgeBench data docs61.8%
14Llama 3.1 70B InstructGPT-4o Dataset, Prompted Judge, Arena-Hard (Llama-3.1-70B-Instruct), Math Score· JudgeBench data docs60.7%
15Claude 3.5 SonnetClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (claude-3-5-sonnet-20240620), Math Score· JudgeBench data docs55.9%
16gemini-1.5-pro-001GPT-4o Dataset, Prompted Judge, VertexAI Evaluation (gemini-1.5-pro-001), Math Score· JudgeBench data docs53.6%
17gemini-1.5-flash-001GPT-4o Dataset, Prompted Judge, Arena-Hard (gemini-1.5-flash-001), Math Score· JudgeBench data docs50%
18Llama 3.1 8B InstructClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (Llama-3.1-8B-Instruct), Math Score· JudgeBench data docs50%
19GPT 4oGPT-4o Dataset, Multi-Agent Judge, ChatEval (gpt-4o-2024-05-13), Math Score· JudgeBench data docs44.6%
20Llama 3.1 8B InstructGPT-4o Dataset, Prompted Judge, Arena-Hard (Llama-3.1-8B-Instruct), Math Score· JudgeBench data docs44.6%
21Llama 3.1 70B InstructClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (Llama-3.1-70B-Instruct), Math Score· JudgeBench data docs41.2%
22Claude 3 HaikuGPT-4o Dataset, Prompted Judge, Arena-Hard (claude-3-haiku-20240307), Math Score· JudgeBench data docs33.9%
23Claude 3 HaikuClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (claude-3-haiku-20240307), Math Score· JudgeBench data docs32.4%

Data credits: JudgeBench, Sijun Tan et al., ScalerLab