Skip to content

Benchmark

HumanEval

Coding benchmarks compare models on programming tasks. Scores are percentages. Higher is better.

Models and systems measured
98
Scores recorded
99
Sources
3
IQ weight
1.81

Score distribution

71.5%
Mean
78.7%
Median
21.1%
Std dev
37.8%
10th percentile
92%
90th percentile
11.6%
Min
11.6%96.3%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.81 for chat models, measured over 96 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model or systemScore
51GPT-3.5 Turbo· EvalPlus Leaderboard data
Values from 2 sources
  • EvalPlus Leaderboard data · 76.8% · EvalPlus Leaderboard and EvalPerf (evalplus.github.io), Apache-2.0; EvalPerf win-rate fractions converted to percentages
  • LLM Stats data · 68% · Data by LLM Stats
76.8%
52Mixtral 8x22B Instruct v0.1· EvalPlus Leaderboard data76.2%
53deepseek-coder-7b-instruct-v1p5· EvalPlus Leaderboard data75.6%
54DBRX-instruct· EvalPlus Leaderboard data75%
55Gemma 3n E4B IT· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim75%
56deepseek-coder-6.7b-instruct· EvalPlus Leaderboard data74.4%
57Gemini 1.5 Flash· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim74.3%
58GPT-3.5· EvalPlus Leaderboard data73.2%
59Claude 3 Sonnet· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim
Values from 2 sources
  • LLM Stats data · 73% · Data by LLM Stats
  • EvalPlus Leaderboard data · 70.7% · EvalPlus Leaderboard and EvalPerf (evalplus.github.io), Apache-2.0; EvalPerf win-rate fractions converted to percentages
73%
60Llama 3.1 8B Instruct· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim
Values from 2 sources
  • LLM Stats data · 72.6% · Data by LLM Stats
  • EvalPlus Leaderboard data · 69.5% · EvalPlus Leaderboard and EvalPerf (evalplus.github.io), Apache-2.0; EvalPerf win-rate fractions converted to percentages
72.6%
61Code Llama 70B Instruct· EvalPlus Leaderboard data72%
62Pixtral 12B2409· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim72%
63Gemma 3 4B IT· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim71.3%
64phi-3.5-moe-instruct· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim70.7%
65Claude 2· EvalPlus Leaderboard data69.5%
66Mistral Large· EvalPlus Leaderboard data69.5%
67gemini-pro-1.5· EvalPlus Leaderboard data68.3%
68Qwen1.5-72B-Chat· EvalPlus Leaderboard data68.3%
69Gemma 3n E2b It· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim66.5%
70Phi-3 Mini 4K Instruct· EvalPlus Leaderboard data64.6%
71Command R+· EvalPlus Leaderboard data64%
72dolphin-2p6-mixtral-8x7b· EvalPlus Leaderboard data64%
73Llama 3 8B Instruct· EvalPlus Leaderboard data61.6%
74Claude Instant 1· EvalPlus Leaderboard data57.3%
75code-llama-70b· EvalPlus Leaderboard data55.5%
76code-llama-34b· EvalPlus Leaderboard data51.8%
77Gemma 2 27B IT· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim51.8%
78Starcoder2 15b· EvalPlus Leaderboard data46.3%
79Mixtral 8x7B Instruct v0.1· EvalPlus Leaderboard data45.1%
80codegemma-7b· EvalPlus Leaderboard data44.5%
81SOLAR 10.7B Instruct v1.0· EvalPlus Leaderboard data43.3%
82code-llama-13b· EvalPlus Leaderboard data42.7%
83Gemma 1.1 7B IT· EvalPlus Leaderboard data42.7%
84Mistral 7B Instruct v0.2· EvalPlus Leaderboard data42.1%
85Gemma 3 1b it· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim41.5%
86Gemma 2 9B IT· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim40.2%
87codellama-7b· EvalPlus Leaderboard data37.8%
88gemma-7b· EvalPlus Leaderboard data35.4%
89starcoder2-7b· EvalPlus Leaderboard data35.4%
90starcoder2-3b· EvalPlus Leaderboard data31.7%
91Gemma 7B IT· EvalPlus Leaderboard data28.7%
92mistral-7b· EvalPlus Leaderboard data28.7%
93codegemma-2b· EvalPlus Leaderboard data26.8%
94gemma-2b· EvalPlus Leaderboard data25%
95Gemma 1.1 2B IT· EvalPlus Leaderboard data22.6%
96Gemma 2B IT· EvalPlus Leaderboard data17.7%
97Vicuna 13B· EvalPlus Leaderboard data17.1%
98Vicuna 7B· EvalPlus Leaderboard data11.6%

Showing 51–98 of 98 results.

Not ranked

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
DeepSeek V4.1 Flash Basepass@1 · 0-shot· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.79.4%

Data credits: EvalPlus Leaderboard and EvalPerf (evalplus.github.io), Apache-2.0; EvalPerf win-rate fractions converted to percentages (licence) · Data by LLM Stats · BenchBoard by SYuan03 (github.com/SYuan03/benchboard). MIT covers its code and data organization; no results licence is inferred. Numeric relays retain the original stating-source links.