Skip to content

Benchmark

JudgeBench: Coding Score

Coding benchmarks compare models on programming tasks. Scores are percentages. Higher is better.

Models measured
13
Scores recorded
23
Sources
1
IQ weight
1.14

Score distribution

50.6%
Mean
45.2%
Median
26.6%
Std dev
21.4%
10th percentile
92.9%
90th percentile
9.7%
Min
9.7%100%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.14 for chat models, measured over 13 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1o3 MiniGPT-4o Dataset, Prompted Judge, Arena-Hard (o3-mini-2025-01-31 (high)), Coding Score· JudgeBench data docs100%
2DeepSeek R1GPT-4o Dataset, Prompted Judge, Arena-Hard (DeepSeek-R1-250120), Coding Score· JudgeBench data docs92.9%
3o3 MiniGPT-4o Dataset, Prompted Judge, Arena-Hard (o3-mini-2025-01-31 (medium)), Coding Score· JudgeBench data docs92.9%
4O1 PreviewGPT-4o Dataset, Prompted Judge, Arena-Hard (o1-preview-2024-09-12), Coding Score· JudgeBench data docs85.7%
5o3 MiniGPT-4o Dataset, Prompted Judge, Arena-Hard (o3-mini-2025-01-31 (low)), Coding Score· JudgeBench data docs83.3%
6O1 MiniGPT-4o Dataset, Prompted Judge, Arena-Hard (o1-mini-2024-09-12), Coding Score· JudgeBench data docs78.6%
7Claude 3.5 SonnetGPT-4o Dataset, Prompted Judge, Arena-Hard (claude-3-5-sonnet-20240620), Coding Score· JudgeBench data docs64.3%
8GPT 4oGPT-4o Dataset, Prompted Judge, Vanilla (gpt-4o-2024-05-13), Coding Score· JudgeBench data docs61.9%
9GPT 4oGPT-4o Dataset, Prompted Judge, Arena-Hard (gpt-4o-2024-05-13), Coding Score· JudgeBench data docs59.5%
10Llama 3.1 70B InstructGPT-4o Dataset, Prompted Judge, Arena-Hard (Llama-3.1-70B-Instruct), Coding Score· JudgeBench data docs52.4%
11Llama 3.1 405B InstructGPT-4o Dataset, Prompted Judge, Arena-Hard (Llama-3.1-405B-Instruct), Coding Score· JudgeBench data docs50%
12GPT-4o miniGPT-4o Dataset, Prompted Judge, Arena-Hard (gpt-4o-mini-2024-07-18), Coding Score· JudgeBench data docs45.2%
13GPT 4oClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (gpt-4o-2024-05-13), Coding Score· JudgeBench data docs38.7%
14Llama 3.1 8B InstructGPT-4o Dataset, Prompted Judge, Arena-Hard (Llama-3.1-8B-Instruct), Coding Score· JudgeBench data docs33.3%
15Claude 3.5 SonnetClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (claude-3-5-sonnet-20240620), Coding Score· JudgeBench data docs32.3%
16GPT 4oGPT-4o Dataset, Multi-Agent Judge, ChatEval (gpt-4o-2024-05-13), Coding Score· JudgeBench data docs31%
17Llama 3.1 8B InstructClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (Llama-3.1-8B-Instruct), Coding Score· JudgeBench data docs29%
18gemini-1.5-pro-001GPT-4o Dataset, Prompted Judge, VertexAI Evaluation (gemini-1.5-pro-001), Coding Score· JudgeBench data docs28.6%
19gemini-1.5-pro-001GPT-4o Dataset, Prompted Judge, Arena-Hard (gemini-1.5-pro-001), Coding Score· JudgeBench data docs26.2%
20Llama 3.1 70B InstructClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (Llama-3.1-70B-Instruct), Coding Score· JudgeBench data docs25.8%
21Claude 3 HaikuGPT-4o Dataset, Prompted Judge, Arena-Hard (claude-3-haiku-20240307), Coding Score· JudgeBench data docs21.4%
22gemini-1.5-flash-001GPT-4o Dataset, Prompted Judge, Arena-Hard (gemini-1.5-flash-001), Coding Score· JudgeBench data docs21.4%
23Claude 3 HaikuClaude-3.5-Sonnet Dataset, Prompted Judge, Arena-Hard (claude-3-haiku-20240307), Coding Score· JudgeBench data docs9.7%

Data credits: JudgeBench, Sijun Tan et al., ScalerLab