Skip to content

Benchmark

JudgeBench: Overall Score

Scores are percentages. Higher is better.

Models measured
13
Scores recorded
23
Sources
1
IQ weight
1.58

Score distribution

53.2%
Mean
50.9%
Median
14.6%
Std dev
34%
10th percentile
75.4%
90th percentile
32.2%
Min
32.2%80.9%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.58 for chat models, measured over 13 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1o3 MiniGPT-4o Dataset, Prompted Judge, Arena-Hard (o3-mini-2025-01-31 (high)), Overall Score· JudgeBench data docs80.9%
2o3 MiniGPT-4o Dataset, Prompted Judge, Arena-Hard (o3-mini-2025-01-31 (medium)), Overall Score· JudgeBench data docs76.6%
3O1 PreviewGPT-4o Dataset, Prompted Judge, Arena-Hard (o1-preview-2024-09-12), Overall Score· JudgeBench data docs75.4%
4DeepSeek R1GPT-4o Dataset, Prompted Judge, Arena-Hard (DeepSeek-R1-250120), Overall Score· JudgeBench data docs73.1%
5o3 MiniGPT-4o Dataset, Prompted Judge, Arena-Hard (o3-mini-2025-01-31 (low)), Overall Score· JudgeBench data docs70.6%
6O1 MiniGPT-4o Dataset, Prompted Judge, Arena-Hard (o1-mini-2024-09-12), Overall Score· JudgeBench data docs65.7%
7Claude 3.5 SonnetGPT-4o Dataset, Prompted Judge, Arena-Hard (claude-3-5-sonnet-20240620), Overall Score· JudgeBench data docs64.3%
8Llama 3.1 405B InstructGPT-4o Dataset, Prompted Judge, Arena-Hard (Llama-3.1-405B-Instruct), Overall Score· JudgeBench data docs56.9%
9GPT 4oGPT-4o Dataset, Prompted Judge, Arena-Hard (gpt-4o-2024-05-13), Overall Score· JudgeBench data docs56.6%
10Llama 3.1 70B InstructGPT-4o Dataset, Prompted Judge, Arena-Hard (Llama-3.1-70B-Instruct), Overall Score· JudgeBench data docs52.3%
11GPT 4oClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (gpt-4o-2024-05-13), Overall Score· JudgeBench data docs51.9%
12GPT 4oGPT-4o Dataset, Prompted Judge, Vanilla (gpt-4o-2024-05-13), Overall Score· JudgeBench data docs50.9%
13GPT-4o miniGPT-4o Dataset, Prompted Judge, Arena-Hard (gpt-4o-mini-2024-07-18), Overall Score· JudgeBench data docs50%
14gemini-1.5-pro-001GPT-4o Dataset, Prompted Judge, Arena-Hard (gemini-1.5-pro-001), Overall Score· JudgeBench data docs47.1%
15Llama 3.1 70B InstructClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (Llama-3.1-70B-Instruct), Overall Score· JudgeBench data docs45.2%
16Claude 3.5 SonnetClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (claude-3-5-sonnet-20240620), Overall Score· JudgeBench data docs44.8%
17gemini-1.5-pro-001GPT-4o Dataset, Prompted Judge, VertexAI Evaluation (gemini-1.5-pro-001), Overall Score· JudgeBench data docs44.6%
18Llama 3.1 8B InstructGPT-4o Dataset, Prompted Judge, Arena-Hard (Llama-3.1-8B-Instruct), Overall Score· JudgeBench data docs40.9%
19gemini-1.5-flash-001GPT-4o Dataset, Prompted Judge, Arena-Hard (gemini-1.5-flash-001), Overall Score· JudgeBench data docs39.7%
20Llama 3.1 8B InstructClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (Llama-3.1-8B-Instruct), Overall Score· JudgeBench data docs36.7%
21GPT 4oGPT-4o Dataset, Multi-Agent Judge, ChatEval (gpt-4o-2024-05-13), Overall Score· JudgeBench data docs34%
22Claude 3 HaikuGPT-4o Dataset, Prompted Judge, Arena-Hard (claude-3-haiku-20240307), Overall Score· JudgeBench data docs33.1%
23Claude 3 HaikuClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (claude-3-haiku-20240307), Overall Score· JudgeBench data docs32.2%

Data credits: JudgeBench, Sijun Tan et al., ScalerLab