Skip to content

Benchmark

BigCodeBench-Hard Instruct

Coding benchmarks compare models on programming tasks. Scores are percentages.

Models measured
89
Scores recorded
90
Sources
1
IQ weight
1.46

Score distribution

18%
Mean
20.9%
Median
8.7%
Std dev
4.7%
10th percentile
28.4%
90th percentile
0%
Min
0%31.8%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.46 for chat models, measured over 89 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
51GPT-3.5 Turbo· BigCodeBench Leaderboard data18.2%
52DeepSeek Coder 33B Instruct· BigCodeBench Leaderboard data17.6%
53Claude 3 Haiku· BigCodeBench Leaderboard data16.9%
54Llama 4 Scout· BigCodeBench Leaderboard data16.9%
55deepseek-coder-v2-lite-instruct· BigCodeBench Leaderboard data16.2%
56Athene 70B· BigCodeBench Leaderboard data15.5%
57yi-large· BigCodeBench Leaderboard data15.5%
58GPT 4· BigCodeBench Leaderboard data14.9%
59C4AI Command R 08 2024· BigCodeBench Leaderboard data13.5%
60Command R+· BigCodeBench Leaderboard data13.5%
61Llama 3.1 8B Instruct· BigCodeBench Leaderboard data13.5%
62codestral-mamba· BigCodeBench Leaderboard data12.8%
63Mistral Small· BigCodeBench Leaderboard data12.8%
64Qwen1.5-110B-Chat· BigCodeBench Leaderboard data12.2%
65Qwen1.5-72B-Chat· BigCodeBench Leaderboard data12.2%
66Qwen2.5 7B Instruct· BigCodeBench Leaderboard data12.2%
67Code Llama 70B Instruct· BigCodeBench Leaderboard data11.5%
68Mistral Nemo 12B Instruct· BigCodeBench Leaderboard data11.5%
69Phi-3 Medium 128K Instruct· BigCodeBench Leaderboard data10.8%
70deepseek-coder-6.7b-instruct· BigCodeBench Leaderboard data10.1%
71Qwen2 7B Instruct· BigCodeBench Leaderboard data10.1%
72code-llama-13b-instruct· BigCodeBench Leaderboard data9.5%
73Qwen1.5-32B-Chat· BigCodeBench Leaderboard data9.5%
74Code Llama 34B Instruct· BigCodeBench Leaderboard data8.8%
75Yi 1.5 34B Chat· BigCodeBench Leaderboard data8.8%
76Granite 3.0 8B Instruct· BigCodeBench Leaderboard data7.4%
77Phi-3 Mini 128K Instruct· BigCodeBench Leaderboard data7.4%
78Llama 3 8B Instruct· BigCodeBench Leaderboard data6.8%
79Llama 3.2 3B· BigCodeBench Leaderboard data5.4%
80Qwen2.5 Coder 1.5b Instruct· BigCodeBench Leaderboard data5.4%
81Mistral (7B) Instruct v0.3· BigCodeBench Leaderboard data4.7%
82Mistral Large2402· BigCodeBench Leaderboard data4.1%
83code-llama-7b-instruct· BigCodeBench Leaderboard data3.4%
84DeepSeek R1 Distill Qwen 7B· BigCodeBench Leaderboard data3.4%
85DeepSeek R1 Distill Llama 8B· BigCodeBench Leaderboard data2%
86Granite 3.0 2B Instruct· BigCodeBench Leaderboard data2%
87Qwen2.5 1.5B Instruct· BigCodeBench Leaderboard data1.4%
88Llama 3.2 1B Instruct· BigCodeBench Leaderboard data0.7%
89Qwen2.5 0.5b Instruct· BigCodeBench Leaderboard data0.7%
90DeepSeek R1 Distill Qwen 1.5B· BigCodeBench Leaderboard data0%

Showing 51–90 of 90 results.