Skip to content

Benchmark

JudgeBench: Knowledge Score

Scores are percentages. Higher is better.

Models measured
13
Scores recorded
23
Sources
1
IQ weight
1.52

Score distribution

49.9%
Mean
50.6%
Median
10.4%
Std dev
35.1%
10th percentile
63%
90th percentile
32.5%
Min
32.5%67.5%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.52 for chat models, measured over 13 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1o3 MiniGPT-4o Dataset, Prompted Judge, Arena-Hard (o3-mini-2025-01-31 (high)), Knowledge Score· JudgeBench data docs67.5%
2O1 PreviewGPT-4o Dataset, Prompted Judge, Arena-Hard (o1-preview-2024-09-12), Knowledge Score· JudgeBench data docs66.2%
3o3 MiniGPT-4o Dataset, Prompted Judge, Arena-Hard (o3-mini-2025-01-31 (low)), Knowledge Score· JudgeBench data docs63%
4Claude 3.5 SonnetGPT-4o Dataset, Prompted Judge, Arena-Hard (claude-3-5-sonnet-20240620), Knowledge Score· JudgeBench data docs62.3%
5o3 MiniGPT-4o Dataset, Prompted Judge, Arena-Hard (o3-mini-2025-01-31 (medium)), Knowledge Score· JudgeBench data docs62.3%
6DeepSeek R1GPT-4o Dataset, Prompted Judge, Arena-Hard (DeepSeek-R1-250120), Knowledge Score· JudgeBench data docs59.1%
7O1 MiniGPT-4o Dataset, Prompted Judge, Arena-Hard (o1-mini-2024-09-12), Knowledge Score· JudgeBench data docs58.4%
8Llama 3.1 405B InstructGPT-4o Dataset, Prompted Judge, Arena-Hard (Llama-3.1-405B-Instruct), Knowledge Score· JudgeBench data docs55.8%
9GPT 4oClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (gpt-4o-2024-05-13), Knowledge Score· JudgeBench data docs52.6%
10Llama 3.1 70B InstructGPT-4o Dataset, Prompted Judge, Arena-Hard (Llama-3.1-70B-Instruct), Knowledge Score· JudgeBench data docs51.3%
11GPT 4oGPT-4o Dataset, Prompted Judge, Arena-Hard (gpt-4o-2024-05-13), Knowledge Score· JudgeBench data docs50.6%
12Llama 3.1 70B InstructClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (Llama-3.1-70B-Instruct), Knowledge Score· JudgeBench data docs50.6%
13gemini-1.5-pro-001GPT-4o Dataset, Prompted Judge, Arena-Hard (gemini-1.5-pro-001), Knowledge Score· JudgeBench data docs49.4%
14GPT-4o miniGPT-4o Dataset, Prompted Judge, Arena-Hard (gpt-4o-mini-2024-07-18), Knowledge Score· JudgeBench data docs48.1%
15gemini-1.5-pro-001GPT-4o Dataset, Prompted Judge, VertexAI Evaluation (gemini-1.5-pro-001), Knowledge Score· JudgeBench data docs45.5%
16GPT 4oGPT-4o Dataset, Prompted Judge, Vanilla (gpt-4o-2024-05-13), Knowledge Score· JudgeBench data docs44.2%
17gemini-1.5-flash-001GPT-4o Dataset, Prompted Judge, Arena-Hard (gemini-1.5-flash-001), Knowledge Score· JudgeBench data docs42.9%
18Claude 3.5 SonnetClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (claude-3-5-sonnet-20240620), Knowledge Score· JudgeBench data docs42.2%
19Llama 3.1 8B InstructGPT-4o Dataset, Prompted Judge, Arena-Hard (Llama-3.1-8B-Instruct), Knowledge Score· JudgeBench data docs38.3%
20Claude 3 HaikuClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (claude-3-haiku-20240307), Knowledge Score· JudgeBench data docs37.7%
21Claude 3 HaikuGPT-4o Dataset, Prompted Judge, Arena-Hard (claude-3-haiku-20240307), Knowledge Score· JudgeBench data docs35.1%
22Llama 3.1 8B InstructClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (Llama-3.1-8B-Instruct), Knowledge Score· JudgeBench data docs33.1%
23GPT 4oGPT-4o Dataset, Multi-Agent Judge, ChatEval (gpt-4o-2024-05-13), Knowledge Score· JudgeBench data docs32.5%

Data credits: JudgeBench, Sijun Tan et al., ScalerLab