Skip to content

Benchmark

HumanEval+

Coding benchmarks compare models on programming tasks. Scores are percentages. Higher is better.

Models and systems measured
64
Scores recorded
64
Sources
3
IQ weight
1.15

Score distribution

57.8%
Mean
62.2%
Median
23.2%
Std dev
23.8%
10th percentile
87.2%
90th percentile
11.6%
Min
11.6%92.9%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.15 for chat models, measured over 63 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model or systemScore
1Mistral Small 3.2 24B Instruct2506· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim92.9%
2Phi-4-reasoning· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim92.9%
3Phi 4 Reasoning Plus· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim92.3%
4O1 Mini· EvalPlus Leaderboard data89%
5O1 Preview· EvalPlus Leaderboard data89%
6GPT 4o· EvalPlus Leaderboard data87.2%
7Qwen 2.5 Coder 32B Instruct· EvalPlus Leaderboard data87.2%
8DeepSeek V3· EvalPlus Leaderboard data86.6%
9DeepSeek V2.5· EvalPlus Leaderboard data83.5%
10Phi 4· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim82.8%
11DeepSeek Coder V2 Instruct· EvalPlus Leaderboard data82.3%
12GPT-4 Turbo· EvalPlus Leaderboard data81.7%
13Grok Beta· EvalPlus Leaderboard data80.5%
14gemini-1.5-pro-002· EvalPlus Leaderboard data79.3%
15GPT 4· EvalPlus Leaderboard data79.3%
16Claude 3 Opus· EvalPlus Leaderboard data77.4%
17gemini-1.5-flash-002· EvalPlus Leaderboard data75.6%
18MiMo V2.5 Pro· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim75.6%
19DeepSeek Coder 33B Instruct· EvalPlus Leaderboard data75%
20Llama 3 70B Instruct· EvalPlus Leaderboard data72%
21Mixtral 8x22B Instruct v0.1· EvalPlus Leaderboard data72%
22deepseek-coder-6.7b-instruct· EvalPlus Leaderboard data71.3%
23deepseek-coder-7b-instruct-v1p5· EvalPlus Leaderboard data71.3%
24GPT-3.5 Turbo· EvalPlus Leaderboard data70.7%
25DBRX-instruct· EvalPlus Leaderboard data70.1%
26Claude 3 Haiku· EvalPlus Leaderboard data68.9%
27GPT-3.5· EvalPlus Leaderboard data66.5%
28Code Llama 70B Instruct· EvalPlus Leaderboard data65.9%
29Claude 3 Sonnet· EvalPlus Leaderboard data64%
30Llama 3.1 8B Instruct· EvalPlus Leaderboard data62.8%
31Mistral Large· EvalPlus Leaderboard data62.2%
32Claude 2· EvalPlus Leaderboard data61.6%
33gemini-pro-1.5· EvalPlus Leaderboard data61%
34Phi-3 Mini 4K Instruct· EvalPlus Leaderboard data59.1%
35Qwen1.5-72B-Chat· EvalPlus Leaderboard data59.1%
36dolphin-2p6-mixtral-8x7b· EvalPlus Leaderboard data57.3%
37Command R+· EvalPlus Leaderboard data56.7%
38Llama 3 8B Instruct· EvalPlus Leaderboard data56.7%
39Qwen2.5 32B Instruct· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim52.4%
40Qwen 2.5 14B Instruct· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim51.2%
41Claude Instant 1· EvalPlus Leaderboard data50.6%
42code-llama-70b· EvalPlus Leaderboard data50.6%
43code-llama-34b· EvalPlus Leaderboard data43.9%
44codegemma-7b· EvalPlus Leaderboard data41.5%
45Mixtral 8x7B Instruct v0.1· EvalPlus Leaderboard data39.6%
46code-llama-13b· EvalPlus Leaderboard data38.4%
47Starcoder2 15b· EvalPlus Leaderboard data37.8%
48SOLAR 10.7B Instruct v1.0· EvalPlus Leaderboard data37.2%
49Mistral 7B Instruct v0.2· EvalPlus Leaderboard data36%
50codellama-7b· EvalPlus Leaderboard data35.4%

Showing 1–50 of 63 results.

Not ranked

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
MiMo V2 Flash Baseofficial Base table · 1-shot· Reported by Xiaomi MiMo· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.70.7%

Data credits: EvalPlus Leaderboard and EvalPerf (evalplus.github.io), Apache-2.0; EvalPerf win-rate fractions converted to percentages (licence) · Data by LLM Stats · BenchBoard by SYuan03 (github.com/SYuan03/benchboard). MIT covers its code and data organization; no results licence is inferred. Numeric relays retain the original stating-source links.