Skip to content

Benchmark

JudgeBench: Reasoning Score

Scores are percentages. Higher is better.

Models measured
13
Scores recorded
23
Sources
1
IQ weight
1.71

Score distribution

54%
Mean
49%
Median
17.1%
Std dev
34.7%
10th percentile
82.7%
90th percentile
29.4%
Min
29.4%89.8%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.71 for chat models, measured over 13 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1o3 MiniGPT-4o Dataset, Prompted Judge, Arena-Hard (o3-mini-2025-01-31 (high)), Reasoning Score· JudgeBench data docs89.8%
2o3 MiniGPT-4o Dataset, Prompted Judge, Arena-Hard (o3-mini-2025-01-31 (medium)), Reasoning Score· JudgeBench data docs86.7%
3DeepSeek R1GPT-4o Dataset, Prompted Judge, Arena-Hard (DeepSeek-R1-250120), Reasoning Score· JudgeBench data docs82.7%
4O1 PreviewGPT-4o Dataset, Prompted Judge, Arena-Hard (o1-preview-2024-09-12), Reasoning Score· JudgeBench data docs79.6%
5o3 MiniGPT-4o Dataset, Prompted Judge, Arena-Hard (o3-mini-2025-01-31 (low)), Reasoning Score· JudgeBench data docs69.4%
6Claude 3.5 SonnetGPT-4o Dataset, Prompted Judge, Arena-Hard (claude-3-5-sonnet-20240620), Reasoning Score· JudgeBench data docs66.3%
7O1 MiniGPT-4o Dataset, Prompted Judge, Arena-Hard (o1-mini-2024-09-12), Reasoning Score· JudgeBench data docs62.2%
8GPT 4oGPT-4o Dataset, Prompted Judge, Arena-Hard (gpt-4o-2024-05-13), Reasoning Score· JudgeBench data docs54.1%
9Llama 3.1 405B InstructGPT-4o Dataset, Prompted Judge, Arena-Hard (Llama-3.1-405B-Instruct), Reasoning Score· JudgeBench data docs54.1%
10Claude 3.5 SonnetClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (claude-3-5-sonnet-20240620), Reasoning Score· JudgeBench data docs52.9%
11GPT 4oClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (gpt-4o-2024-05-13), Reasoning Score· JudgeBench data docs51%
12Llama 3.1 70B InstructGPT-4o Dataset, Prompted Judge, Arena-Hard (Llama-3.1-70B-Instruct), Reasoning Score· JudgeBench data docs49%
13GPT 4oGPT-4o Dataset, Prompted Judge, Vanilla (gpt-4o-2024-05-13), Reasoning Score· JudgeBench data docs48%
14Llama 3.1 8B InstructGPT-4o Dataset, Prompted Judge, Arena-Hard (Llama-3.1-8B-Instruct), Reasoning Score· JudgeBench data docs45.9%
15gemini-1.5-pro-001GPT-4o Dataset, Prompted Judge, VertexAI Evaluation (gemini-1.5-pro-001), Reasoning Score· JudgeBench data docs44.9%
16GPT-4o miniGPT-4o Dataset, Prompted Judge, Arena-Hard (gpt-4o-mini-2024-07-18), Reasoning Score· JudgeBench data docs43.9%
17Llama 3.1 70B InstructClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (Llama-3.1-70B-Instruct), Reasoning Score· JudgeBench data docs43.1%
18Llama 3.1 8B InstructClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (Llama-3.1-8B-Instruct), Reasoning Score· JudgeBench data docs43.1%
19gemini-1.5-pro-001GPT-4o Dataset, Prompted Judge, Arena-Hard (gemini-1.5-pro-001), Reasoning Score· JudgeBench data docs42.9%
20gemini-1.5-flash-001GPT-4o Dataset, Prompted Judge, Arena-Hard (gemini-1.5-flash-001), Reasoning Score· JudgeBench data docs36.7%
21Claude 3 HaikuGPT-4o Dataset, Prompted Judge, Arena-Hard (claude-3-haiku-20240307), Reasoning Score· JudgeBench data docs34.7%
22GPT 4oGPT-4o Dataset, Multi-Agent Judge, ChatEval (gpt-4o-2024-05-13), Reasoning Score· JudgeBench data docs31.6%
23Claude 3 HaikuClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (claude-3-haiku-20240307), Reasoning Score· JudgeBench data docs29.4%

Data credits: JudgeBench, Sijun Tan et al., ScalerLab