Skip to content

Benchmark

MBPP

Coding benchmarks compare models on programming tasks. Scores are percentages.

Models measured
65
Scores recorded
65
Sources
2
IQ weight
1.60

Score distribution

72.3%
Mean
74.4%
Median
15.1%
Std dev
52.4%
10th percentile
89.7%
90th percentile
29.8%
Min
29.8%95.5%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.60 for chat models, measured over 64 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1O1 Preview· EvalPlus Leaderboard data95.5%
2O1 Mini· EvalPlus Leaderboard data93.1%
3Sarvam-30B· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim92.7%
4Llama 3.3 Nemotron Super 49B v1· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim91.3%
5Qwen 2.5 Coder 32B Instruct· EvalPlus Leaderboard data
Values from 2 sources
  • EvalPlus Leaderboard data · 90.5% · EvalPlus Leaderboard and EvalPerf (evalplus.github.io), Apache-2.0; EvalPerf win-rate fractions converted to percentages
  • LLM Stats data · 90.2% · Data by LLM Stats
90.5%
6gemini-1.5-pro-002· EvalPlus Leaderboard data89.7%
7Claude 3 Opus· EvalPlus Leaderboard data89.4%
8DeepSeek Coder V2 Instruct· EvalPlus Leaderboard data89.4%
9Qwen2.5 72B Instruct· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim88.2%
10DeepSeek V2.5· EvalPlus Leaderboard data87.6%
11DeepSeek V3· EvalPlus Leaderboard data87.6%
12GPT 4o· EvalPlus Leaderboard data87.6%
13Grok Beta· EvalPlus Leaderboard data86%
14GPT-4 Turbo· EvalPlus Leaderboard data85.7%
15gemini-1.5-flash-002· EvalPlus Leaderboard data84.7%
16Qwen2.5 32B Instruct· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim84%
17Claude 3 Sonnet· EvalPlus Leaderboard data83.6%
18Qwen2.5 Coder 7B Instruct· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim83.5%
19GPT-3.5 Turbo· EvalPlus Leaderboard data82.5%
20Llama 3 70B Instruct· EvalPlus Leaderboard data82.3%
21Qwen 2.5 14B Instruct· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim82%
22Qwen 3 235B A22B· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim81.4%
23phi-3.5-moe-instruct· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim80.8%
24DeepSeek Coder 33B Instruct· EvalPlus Leaderboard data80.4%
25Claude 3 Haiku· EvalPlus Leaderboard data80.2%
26Qwen2 72B Instruct· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim80.2%
27Qwen2.5 7B Instruct· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim79.2%
28Llama 4 Maverick· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim77.6%
29deepseek-coder-7b-instruct-v1p5· EvalPlus Leaderboard data75.2%
30deepseek-coder-6.7b-instruct· EvalPlus Leaderboard data74.9%
31Mistral Small 3.1 24b Instruct 2503· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim74.7%
32Gemma 3 27B IT· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim74.4%
33Command R+· EvalPlus Leaderboard data74.3%
34Mixtral 8x22B Instruct v0.1· EvalPlus Leaderboard data73.8%
35Qwen2.5-Omni 7B· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim73.2%
36Gemma 3 12B IT· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim73%
37Mistral Large· EvalPlus Leaderboard data72.8%
38Qwen1.5-72B-Chat· EvalPlus Leaderboard data72.5%
39dolphin-2p6-mixtral-8x7b· EvalPlus Leaderboard data70.6%
40code-llama-34b· EvalPlus Leaderboard data69.3%
41Llama 3.1 8B Instruct· EvalPlus Leaderboard data68.3%
42Llama 4 Scout· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim67.8%
43DBRX-instruct· EvalPlus Leaderboard data67.2%
44Qwen2 7B Instruct· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim67.2%
45Phi-3 Mini 4K Instruct· EvalPlus Leaderboard data65.9%
46codegemma-7b· EvalPlus Leaderboard data65.1%
47Llama 3 8B Instruct· EvalPlus Leaderboard data64.6%
48Gemma 3n E4B IT· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim63.6%
49code-llama-13b· EvalPlus Leaderboard data63.5%
50Gemma 3 4B IT· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim63.2%

Showing 1–50 of 65 results.

Data credits: EvalPlus Leaderboard and EvalPerf (evalplus.github.io), Apache-2.0; EvalPerf win-rate fractions converted to percentages (licence) · Data by LLM Stats