Skip to content

Benchmark

BigCodeBench-Hard Complete

Coding benchmarks compare models on programming tasks. Scores are percentages.

Models measured
94
Scores recorded
95
Sources
1
IQ weight
1.40

Score distribution

21.6%
Mean
20.9%
Median
10.5%
Std dev
6.1%
10th percentile
33.8%
90th percentile
1.4%
Min
1.4%40.5%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.40 for chat models, measured over 94 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1DeepSeek R1· BigCodeBench Leaderboard data40.5%
2DeepSeek V3· BigCodeBench Leaderboard data40.5%
3Grok 3beta· BigCodeBench Leaderboard data39.2%
4Athene V2 Chat· BigCodeBench Leaderboard data37.2%
5Gemini 2.5 Pro· BigCodeBench Leaderboard data36.5%
6gemini-2.0-pro-exp-02-05· BigCodeBench Leaderboard data36.5%
7Qwen2.5-Max· BigCodeBench Leaderboard data35.8%
8GPT-4 Turbo· BigCodeBench Leaderboard data35.1%
9Claude Haiku 3.5· BigCodeBench Leaderboard data34.5%
10Claude Sonnet 3.7· BigCodeBench Leaderboard data33.8%
11Gemini 2.0 Flash· BigCodeBench Leaderboard data33.8%
12Gemini 2.0 Flash Experimental· BigCodeBench Leaderboard data33.8%
13GPT 4.1· BigCodeBench Leaderboard data33.8%
14Qwen 2.5 Coder 32B Instruct· BigCodeBench Leaderboard data33.8%
15ChatGPT-4o Latest· BigCodeBench Leaderboard data33.1%
16Claude 3.5 Sonnet· BigCodeBench Leaderboard data33.1%
17gemini-1.5-pro-002· BigCodeBench Leaderboard data32.4%
18gemini-exp· BigCodeBench Leaderboard data32.4%
19GPT-4.1 mini· BigCodeBench Leaderboard data31.8%
20GPT-4.1 nano· BigCodeBench Leaderboard data31.8%
21Gemini 2.0 Flash-Lite Preview· BigCodeBench Leaderboard data31.1%
22Gemini 2.0 Flash Experimentalthinking· BigCodeBench Leaderboard data30.4%
23Llama 3.1 405B Instruct· BigCodeBench Leaderboard data30.4%
24Phi 4· BigCodeBench Leaderboard data30.4%
25Claude 3 Opus· BigCodeBench Leaderboard data29.7%
26DeepSeek Coder V2 Instruct· BigCodeBench Leaderboard data29.7%
27Grok Beta· BigCodeBench Leaderboard data29.7%
28Mistral-Large-Instruct-2407· BigCodeBench Leaderboard data29.7%
29qwen2.5-coder-14b-instruct· BigCodeBench Leaderboard data29.7%
30DeepSeek R1 Distill Qwen 32B· BigCodeBench Leaderboard data29.1%
31DeepSeek V2.5· BigCodeBench Leaderboard data29.1%
32GPT 4o· BigCodeBench Leaderboard data29.1%
33Llama 4 Maverick· BigCodeBench Leaderboard data29.1%
34Qwen2.5 72B Instruct· BigCodeBench Leaderboard data29.1%
35Llama 3.1 Nemotron 70B Instruct· BigCodeBench Leaderboard data28.4%
36Llama 3.3 70B Instruct· BigCodeBench Leaderboard data28.4%
37QwQ 32B Preview· BigCodeBench Leaderboard data28.4%
38Llama 3.1 70B Instruct· BigCodeBench Leaderboard data27.7%
39Gemma 3 27B IT· BigCodeBench Leaderboard data27%
40GPT-4o mini· BigCodeBench Leaderboard data27%
41Llama 3 70B Instruct· BigCodeBench Leaderboard data27%
42Qwen2.5 32B Instruct· BigCodeBench Leaderboard data27%
43Claude 3 Sonnet· BigCodeBench Leaderboard data26.4%
44Athene 70B· BigCodeBench Leaderboard data24.3%
45GPT-3.5 Turbo· BigCodeBench Leaderboard data21.6%
46Mistral Nemo 12B Instruct· BigCodeBench Leaderboard data21.6%
47DeepSeek Coder 33B Instruct· BigCodeBench Leaderboard data20.9%
48DeepSeek R1 Distill Qwen 14B· BigCodeBench Leaderboard data20.9%
49Mixtral 8x22B Instruct· BigCodeBench Leaderboard data20.9%
50Qwen 2.5 14B Instruct· BigCodeBench Leaderboard data20.9%

Showing 1–50 of 95 results.