Benchmark
M-IFEval Average (ES/FR/JA)
Instruction following benchmarks measure how closely models follow instructions. Scores are percentages. Higher is better.
- Models measured
- 12
- Scores recorded
- 12
- Sources
- 1
- IQ weight
- 1.22
Score distribution
- 66.8%
- Mean
- 74.4%
- Median
- 16.6%
- Std dev
- 45.8%
- 10th percentile
- 83.3%
- 90th percentile
- 30.1%
- Min
This benchmark has a weight of 1.22 for chat models, measured over 12 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.
Leaderboard
| # | Model | Score |
|---|---|---|
| 1 | o1· M-IFEval Leaderboard data docs | 86.6% |
| 2 | O1 Mini· M-IFEval Leaderboard data docs | 83.3% |
| 3 | GPT 4o· M-IFEval Leaderboard data docs | 82.7% |
| 4 | GPT-4o mini· M-IFEval Leaderboard data docs | 78.9% |
| 5 | Qwen2.5 32B Instruct4-bit· M-IFEval Leaderboard data docs | 76.7% |
| 6 | Qwen 2.5 14B Instruct4-bit· M-IFEval Leaderboard data docs | 74.4% |
| 7 | Qwen2.5 7B Instruct4-bit· M-IFEval Leaderboard data docs | 68.6% |
| 8 | Llama 3.1 8B Instruct· M-IFEval Leaderboard data docs | 63.8% |
| 9 | Qwen2.5 3B Instruct4-bit· M-IFEval Leaderboard data docs | 59.8% |
| 10 | Mistral 7B Instruct· M-IFEval Leaderboard data docs | 51.2% |
| 11 | Qwen2.5 1.5B Instruct4-bit· M-IFEval Leaderboard data docs | 45.8% |
| 12 | Qwen2.5 0.5b Instruct4-bit· M-IFEval Leaderboard data docs | 30.1% |
Data credits: M-IFEval: Dussolle, Cardeña Díaz, Sato and Devine (Lightblue), arXiv:2502.04688. README leaderboard, repository licensed Apache-2.0. (licence)