Skip to content

Benchmark

BigCodeBench-Hard Complete

Coding benchmarks compare models on programming tasks. Scores are percentages.

Models measured
94
Scores recorded
95
Sources
1
IQ weight
1.40

Score distribution

21.6%
Mean
20.9%
Median
10.5%
Std dev
6.1%
10th percentile
33.8%
90th percentile
1.4%
Min
1.4%40.5%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.40 for chat models, measured over 94 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
51DeepSeek R1 Distill Llama 70b· BigCodeBench Leaderboard data20.3%
52GPT 4· BigCodeBench Leaderboard data20.3%
53Qwen2.5 Coder 7B Instruct· BigCodeBench Leaderboard data20.3%
54Claude 3 Haiku· BigCodeBench Leaderboard data19.6%
55Mistral Small 24B Instruct2501· BigCodeBench Leaderboard data19.6%
56Qwen1.5-110B-Chat· BigCodeBench Leaderboard data19.6%
57yi-large· BigCodeBench Leaderboard data19.6%
58deepseek-coder-v2-lite-instruct· BigCodeBench Leaderboard data18.2%
59Llama 4 Scout· BigCodeBench Leaderboard data16.9%
60Qwen2.5 7B Instruct· BigCodeBench Leaderboard data16.2%
61Code Llama 70B Instruct· BigCodeBench Leaderboard data15.5%
62DeepSeek V2 Chat· BigCodeBench Leaderboard data15.5%
63deepseek-coder-6.7b-instruct· BigCodeBench Leaderboard data15.5%
64Phi-3 Medium 128K Instruct· BigCodeBench Leaderboard data15.5%
65codestral-mamba· BigCodeBench Leaderboard data14.9%
66Mistral Small· BigCodeBench Leaderboard data14.9%
67Command R+· BigCodeBench Leaderboard data14.2%
68Phi-3 Mini 128K Instruct· BigCodeBench Leaderboard data14.2%
69C4AI Command R 08 2024· BigCodeBench Leaderboard data13.5%
70Llama 3.1 8B Instruct· BigCodeBench Leaderboard data13.5%
71Qwen2 7B Instruct· BigCodeBench Leaderboard data13.5%
72Mistral Large2402· BigCodeBench Leaderboard data12.8%
73Qwen1.5-32B-Chat· BigCodeBench Leaderboard data12.8%
74Qwen1.5-72B-Chat· BigCodeBench Leaderboard data12.2%
75Starcoder2 15b· BigCodeBench Leaderboard data12.2%
76deepseek-coder-v2-lite-base· BigCodeBench Leaderboard data11.5%
77Code Llama 34B Instruct· BigCodeBench Leaderboard data10.8%
78codegemma-7b· BigCodeBench Leaderboard data10.1%
79Granite 3.0 8B Instruct· BigCodeBench Leaderboard data9.5%
80Yi 1.5 34B Chat· BigCodeBench Leaderboard data9.5%
81starcoder2-7b· BigCodeBench Leaderboard data8.8%
82Llama 3 8B Instruct· BigCodeBench Leaderboard data8.1%
83codegemma-2b· BigCodeBench Leaderboard data7.4%
84code-llama-13b-instruct· BigCodeBench Leaderboard data6.8%
85DeepSeek R1 Distill Qwen 7B· BigCodeBench Leaderboard data6.1%
86Mistral (7B) Instruct v0.3· BigCodeBench Leaderboard data6.1%
87Qwen2.5 Coder 1.5b Instruct· BigCodeBench Leaderboard data6.1%
88Llama 3.2 3B· BigCodeBench Leaderboard data5.4%
89mistral-7b-v0.3· BigCodeBench Leaderboard data4.7%
90Qwen2.5 1.5B Instruct· BigCodeBench Leaderboard data4.7%
91starcoder2-3b· BigCodeBench Leaderboard data4.7%
92code-llama-7b-instruct· BigCodeBench Leaderboard data4.1%
93DeepSeek R1 Distill Llama 8B· BigCodeBench Leaderboard data3.4%
94Qwen2.5 0.5b Instruct· BigCodeBench Leaderboard data2%
95Llama 3.2 1B Instruct· BigCodeBench Leaderboard data1.4%

Showing 51–95 of 95 results.