Skip to content

Benchmark

BigCodeBench Complete

Coding benchmarks compare models on programming tasks. Scores are percentages.

Models measured
77
Scores recorded
77
Sources
1
IQ weight
1.92

Score distribution

44.3%
Mean
47.6%
Median
13.1%
Std dev
25.7%
10th percentile
58.2%
90th percentile
7.9%
Min
7.9%62.2%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.92 for chat models, measured over 77 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
51Command R+· BigCodeBench Leaderboard data41.9%
52Mistral Small· BigCodeBench Leaderboard data41.3%
53Phi-3 Mini 128K Instruct· BigCodeBench Leaderboard data40.6%
54Llama 3.1 8B Instruct· BigCodeBench Leaderboard data40.5%
55Qwen1.5-72B-Chat· BigCodeBench Leaderboard data40.3%
56Starcoder2 15b· BigCodeBench Leaderboard data38.4%
57codegemma-7b· BigCodeBench Leaderboard data38.3%
58Mistral Large2402· BigCodeBench Leaderboard data38.3%
59Llama 3 8B Instruct· BigCodeBench Leaderboard data36.9%
60Code Llama 34B Instruct· BigCodeBench Leaderboard data35.6%
61Granite 3.0 8B Instruct· BigCodeBench Leaderboard data35.4%
62Qwen2.5 Coder 1.5b Instruct· BigCodeBench Leaderboard data32.7%
63code-llama-13b-instruct· BigCodeBench Leaderboard data31.7%
64deepseek-coder-v2-lite-base· BigCodeBench Leaderboard data30.6%
65Llama 3.2 3B· BigCodeBench Leaderboard data28.3%
66starcoder2-7b· BigCodeBench Leaderboard data27.7%
67Mistral (7B) Instruct v0.3· BigCodeBench Leaderboard data27.3%
68Qwen2.5 1.5B Instruct· BigCodeBench Leaderboard data27.1%
69code-llama-7b-instruct· BigCodeBench Leaderboard data25.7%
70DeepSeek R1 Distill Qwen 7B· BigCodeBench Leaderboard data24.9%
71codegemma-2b· BigCodeBench Leaderboard data23.9%
72mistral-7b-v0.3· BigCodeBench Leaderboard data23.5%
73starcoder2-3b· BigCodeBench Leaderboard data21.4%
74DeepSeek R1 Distill Llama 8B· BigCodeBench Leaderboard data15.3%
75Llama 3.2 1B Instruct· BigCodeBench Leaderboard data11.3%
76Qwen2.5 0.5b Instruct· BigCodeBench Leaderboard data11.3%
77DeepSeek R1 Distill Qwen 1.5B· BigCodeBench Leaderboard data7.9%

Showing 51–77 of 77 results.