Skip to content

Benchmark

BigCodeBench Instruct

Coding benchmarks compare models on programming tasks. Scores are percentages.

Models measured
71
Scores recorded
71
Sources
1
IQ weight
1.82

Score distribution

36.9%
Mean
38.7%
Median
10.8%
Std dev
20.5%
10th percentile
48.2%
90th percentile
7%
Min
7%51.1%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.82 for chat models, measured over 71 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1GPT 4o· BigCodeBench Leaderboard data51.1%
2DeepSeek V3· BigCodeBench Leaderboard data50%
3Llama 4 Maverick· BigCodeBench Leaderboard data49.7%
4gemini-exp· BigCodeBench Leaderboard data49.2%
5Qwen 2.5 Coder 32B Instruct· BigCodeBench Leaderboard data49%
6GPT-4.1 mini· BigCodeBench Leaderboard data48.9%
7DeepSeek V2.5· BigCodeBench Leaderboard data48.6%
8DeepSeek Coder V2 Instruct· BigCodeBench Leaderboard data48.2%
9GPT-4 Turbo· BigCodeBench Leaderboard data48.2%
10qwen2.5-coder-14b-instruct· BigCodeBench Leaderboard data48.2%
11Athene V2 Chat· BigCodeBench Leaderboard data47.2%
12Llama 3.3 70B Instruct· BigCodeBench Leaderboard data46.9%
13Claude 3.5 Sonnet· BigCodeBench Leaderboard data46.8%
14Claude Haiku 3.5· BigCodeBench Leaderboard data46.1%
15GPT-4o mini· BigCodeBench Leaderboard data46.1%
16Llama 3.1 70B Instruct· BigCodeBench Leaderboard data46.1%
17GPT 4· BigCodeBench Leaderboard data46%
18Gemini 2.0 Flash Experimental· BigCodeBench Leaderboard data45.9%
19Qwen2.5 72B Instruct· BigCodeBench Leaderboard data45.8%
20Claude 3 Opus· BigCodeBench Leaderboard data45.5%
21Phi 4· BigCodeBench Leaderboard data45.5%
22Mistral Small 24B Instruct2501· BigCodeBench Leaderboard data45.3%
23Qwen2.5 32B Instruct· BigCodeBench Leaderboard data45%
24QwQ 32B Preview· BigCodeBench Leaderboard data44.6%
25DeepSeek R1 Distill Qwen 32B· BigCodeBench Leaderboard data43.9%
26Llama 3 70B Instruct· BigCodeBench Leaderboard data43.6%
27Claude 3 Sonnet· BigCodeBench Leaderboard data42.7%
28DeepSeek Coder 33B Instruct· BigCodeBench Leaderboard data42%
29Code Llama 70B Instruct· BigCodeBench Leaderboard data40.7%
30Mixtral 8x22B Instruct· BigCodeBench Leaderboard data40.6%
31DeepSeek V2 Chat· BigCodeBench Leaderboard data40.4%
32Qwen2.5 Coder 7B Instruct· BigCodeBench Leaderboard data40.4%
33Qwen 2.5 14B Instruct· BigCodeBench Leaderboard data39.8%
34Claude 3 Haiku· BigCodeBench Leaderboard data39.4%
35GPT-3.5 Turbo· BigCodeBench Leaderboard data39.1%
36Llama 3.1 Nemotron 70B Instruct· BigCodeBench Leaderboard data38.7%
37DeepSeek R1 Distill Qwen 14B· BigCodeBench Leaderboard data38.1%
38yi-large· BigCodeBench Leaderboard data37.7%
39Phi-3 Medium 128K Instruct· BigCodeBench Leaderboard data37.6%
40Qwen2.5 7B Instruct· BigCodeBench Leaderboard data37.6%
41C4AI Command R 08 2024· BigCodeBench Leaderboard data37.1%
42Athene 70B· BigCodeBench Leaderboard data36.8%
43deepseek-coder-v2-lite-instruct· BigCodeBench Leaderboard data36.8%
44codestral-mamba· BigCodeBench Leaderboard data35.9%
45deepseek-coder-6.7b-instruct· BigCodeBench Leaderboard data35.5%
46DeepSeek R1 Distill Llama 70b· BigCodeBench Leaderboard data35.3%
47Qwen1.5-110B-Chat· BigCodeBench Leaderboard data35%
48Yi 1.5 34B Chat· BigCodeBench Leaderboard data33.9%
49Command R+· BigCodeBench Leaderboard data33.8%
50Qwen1.5-72B-Chat· BigCodeBench Leaderboard data33.2%

Showing 1–50 of 71 results.