Skip to content

Benchmark

BigCodeBench-Hard Instruct

Coding benchmarks compare models on programming tasks. Scores are percentages.

Models measured
89
Scores recorded
90
Sources
1
IQ weight
1.46

Score distribution

18%
Mean
20.9%
Median
8.7%
Std dev
4.7%
10th percentile
28.4%
90th percentile
0%
Min
0%31.8%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.46 for chat models, measured over 89 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude Sonnet 3.7· BigCodeBench Leaderboard data31.8%
2GPT 4.1· BigCodeBench Leaderboard data31.8%
3GPT-4.1 mini· BigCodeBench Leaderboard data31.8%
4DeepSeek R1· BigCodeBench Leaderboard data29.7%
5Gemini 2.5 Pro· BigCodeBench Leaderboard data29.7%
6GPT-4 Turbo· BigCodeBench Leaderboard data29.1%
7Qwen2.5-Max· BigCodeBench Leaderboard data29.1%
8DeepSeek V3· BigCodeBench Leaderboard data28.4%
9GPT-4.1 nano· BigCodeBench Leaderboard data28.4%
10Llama 3.3 70B Instruct· BigCodeBench Leaderboard data28.4%
11Llama 4 Maverick· BigCodeBench Leaderboard data27.7%
12Qwen 2.5 Coder 32B Instruct· BigCodeBench Leaderboard data27.7%
13Athene V2 Chat· BigCodeBench Leaderboard data27%
14gemini-2.0-pro-exp-02-05· BigCodeBench Leaderboard data27%
15Grok 3beta· BigCodeBench Leaderboard data27%
16gemini-exp· BigCodeBench Leaderboard data26.4%
17ChatGPT-4o Latest· BigCodeBench Leaderboard data25.7%
18Claude 3.5 Sonnet· BigCodeBench Leaderboard data25.7%
19Claude Haiku 3.5· BigCodeBench Leaderboard data25.7%
20Gemma 3 27B IT· BigCodeBench Leaderboard data25%
21GPT 4o· BigCodeBench Leaderboard data25%
22QwQ 32B Preview· BigCodeBench Leaderboard data25%
23DeepSeek Coder V2 Instruct· BigCodeBench Leaderboard data24.3%
24Gemini 2.0 Flash Experimentalthinking· BigCodeBench Leaderboard data24.3%
25Phi 4· BigCodeBench Leaderboard data24.3%
26DeepSeek R1 Distill Qwen 32B· BigCodeBench Leaderboard data23.6%
27Gemini 2.0 Flash· BigCodeBench Leaderboard data23.6%
28GPT-4o mini· BigCodeBench Leaderboard data23.6%
29Grok Beta· BigCodeBench Leaderboard data23.6%
30qwen2.5-coder-14b-instruct· BigCodeBench Leaderboard data23.6%
31DeepSeek V2.5· BigCodeBench Leaderboard data23%
32Gemini 2.0 Flash Experimental· BigCodeBench Leaderboard data23%
33Llama 3.1 70B Instruct· BigCodeBench Leaderboard data23%
34Claude 3 Opus· BigCodeBench Leaderboard data22.3%
35Llama 3 70B Instruct· BigCodeBench Leaderboard data22.3%
36Llama 3.1 405B Instruct· BigCodeBench Leaderboard data22.3%
37Mistral-Large-Instruct-2407· BigCodeBench Leaderboard data22.3%
38Qwen2.5 32B Instruct· BigCodeBench Leaderboard data22.3%
39DeepSeek V2 Chat· BigCodeBench Leaderboard data21.6%
40Mistral Small 24B Instruct2501· BigCodeBench Leaderboard data21.6%
41Qwen2.5 72B Instruct· BigCodeBench Leaderboard data21.6%
42DeepSeek R1 Distill Qwen 14B· BigCodeBench Leaderboard data20.9%
43gemini-1.5-pro-002· BigCodeBench Leaderboard data20.9%
44Llama 3.1 Nemotron 70B Instruct· BigCodeBench Leaderboard data20.9%
45Qwen 2.5 14B Instruct· BigCodeBench Leaderboard data20.9%
46Claude 3 Sonnet· BigCodeBench Leaderboard data20.3%
47DeepSeek R1 Distill Llama 70b· BigCodeBench Leaderboard data20.3%
48Qwen2.5 Coder 7B Instruct· BigCodeBench Leaderboard data20.3%
49Gemini 2.0 Flash-Lite Preview· BigCodeBench Leaderboard data19.6%
50Mixtral 8x22B Instruct· BigCodeBench Leaderboard data18.9%

Showing 1–50 of 90 results.