Skip to content

Benchmark

MBPP+

Coding benchmarks compare models on programming tasks. Scores are percentages. Higher is better.

Models and systems measured
49
Scores recorded
49
Sources
3
IQ weight
1.45

Score distribution

60.6%
Mean
63.5%
Median
14.2%
Std dev
40.2%
10th percentile
75.1%
90th percentile
23.3%
Min
23.3%86%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.45 for chat models, measured over 48 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model or systemScore
1Llama 3.1 70B Instructbase split · EvalPlus· Reported by Meta· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.86%
2O1 Preview· EvalPlus Leaderboard data80.2%
3O1 Mini· EvalPlus Leaderboard data78.8%
4Mistral Small 3.2 24B Instruct2506· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim78.3%
5Qwen 2.5 Coder 32B Instruct· EvalPlus Leaderboard data77%
6DeepSeek Coder V2 Instruct· EvalPlus Leaderboard data75.1%
7gemini-1.5-pro-002· EvalPlus Leaderboard data74.6%
8DeepSeek V2.5· EvalPlus Leaderboard data74.1%
9MiMo V2.5 Pro· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim74.1%
10Claude 3 Opus· EvalPlus Leaderboard data73.3%
11GPT-4 Turbo· EvalPlus Leaderboard data73.3%
12DeepSeek V3· EvalPlus Leaderboard data73%
13GPT 4o· EvalPlus Leaderboard data72.2%
14DeepSeek Coder 33B Instruct· EvalPlus Leaderboard data70.1%
15GPT-3.5 Turbo· EvalPlus Leaderboard data69.7%
16Claude 3 Sonnet· EvalPlus Leaderboard data69.3%
17Llama 3 70B Instruct· EvalPlus Leaderboard data69%
18Claude 3 Haiku· EvalPlus Leaderboard data68.8%
19gemini-1.5-flash-002· EvalPlus Leaderboard data67.5%
20Qwen2.5 32B Instruct· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim67.2%
21deepseek-coder-6.7b-instruct· EvalPlus Leaderboard data65.6%
22Grok Beta· EvalPlus Leaderboard data65.6%
23Mixtral 8x22B Instruct v0.1· EvalPlus Leaderboard data64.3%
24Command R+· EvalPlus Leaderboard data63.5%
25Qwen 2.5 14B Instruct· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim63.2%
26deepseek-coder-7b-instruct-v1p5· EvalPlus Leaderboard data62.2%
27Qwen1.5-72B-Chat· EvalPlus Leaderboard data61.6%
28Mistral Large· EvalPlus Leaderboard data59.5%
29dolphin-2p6-mixtral-8x7b· EvalPlus Leaderboard data59%
30code-llama-34b· EvalPlus Leaderboard data56.3%
31DBRX-instruct· EvalPlus Leaderboard data55.8%
32Llama 3.1 8B Instruct· EvalPlus Leaderboard data55.6%
33Llama 3 8B Instruct· EvalPlus Leaderboard data54.8%
34Phi-3 Mini 4K Instruct· EvalPlus Leaderboard data54.2%
35code-llama-13b· EvalPlus Leaderboard data52.6%
36codegemma-7b· EvalPlus Leaderboard data52.4%
37Mixtral 8x7B Instruct v0.1· EvalPlus Leaderboard data49.7%
38codellama-7b· EvalPlus Leaderboard data46.8%
39codegemma-2b· EvalPlus Leaderboard data46.6%
40Gemma 1.1 7B IT· EvalPlus Leaderboard data45%
41gemma-7b· EvalPlus Leaderboard data43.4%
42mistral-7b· EvalPlus Leaderboard data42.1%
43ERNIE 4.5· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim40.2%
44Mistral 7B Instruct v0.2· EvalPlus Leaderboard data37%
45Gemma 7B IT· EvalPlus Leaderboard data36.8%
46SOLAR 10.7B Instruct v1.0· EvalPlus Leaderboard data36.2%
47gemma-2b· EvalPlus Leaderboard data34.1%
48Gemma 1.1 2B IT· EvalPlus Leaderboard data23.3%

Not ranked

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
MiMo V2 Flash Baseofficial Base table · 3-shot· Reported by Xiaomi MiMo· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.71.4%

Data credits: EvalPlus Leaderboard and EvalPerf (evalplus.github.io), Apache-2.0; EvalPerf win-rate fractions converted to percentages (licence) · Data by LLM Stats · BenchBoard by SYuan03 (github.com/SYuan03/benchboard). MIT covers its code and data organization; no results licence is inferred. Numeric relays retain the original stating-source links.