Benchmark
Arena-Hard-v2.0-Preview
People compare model answers and choose which they prefer. Scores are percentages. Higher is better.
- Models measured
- 21
- Scores recorded
- 69
- Sources
- 1
- IQ weight
- 1.81
Score distribution
- 45.2%
- Mean
- 48%
- Median
- 23.4%
- Std dev
- 12.6%
- 10th percentile
- 77%
- 90th percentile
- 8%
- Min
This benchmark has a weight of 1.81 for chat models, measured over 21 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.
Leaderboard
| # | Model | Score |
|---|---|---|
| 1 | OpenAI o3Creative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data | 88.8% |
| 2 | OpenAI o3Hard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data | 87% |
| 3 | OpenAI o3Hard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data | 85.9% |
| 4 | Gemini 2.5 FlashCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data | 83.9% |
| 5 | o4 MiniHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data | 81.7% |
| 6 | o4 MiniHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data | 79.1% |
| 7 | o4 MiniHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data | 78% |
| 8 | DeepSeek R1Creative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data | 77% |
| 9 | o4 MiniHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data | 74.6% |
| 10 | Qwen 3 235B A22BCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data | 73.5% |
| 11 | Gemma 3 27B ITCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data | 69.9% |
| 12 | Gemini 2.5 FlashHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data | 68.6% |
| 13 | o3 MiniHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data | 66.1% |
| 14 | o3 MiniHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data | 64.8% |
| 15 | Claude Sonnet 3.7Creative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data | 63.9% |
| 16 | GPT 4.1Creative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data | 61.5% |
| 17 | o1Hard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data | 61% |
| 18 | QwQ 32BCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data | 60.9% |
| 19 | o1Creative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data | 59.9% |
| 20 | Claude Sonnet 3.7Hard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data | 59.8% |
| 21 | o1Hard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data | 58.7% |
| 22 | o4 MiniCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data | 58.7% |
| 23 | Qwen 3 235B A22BHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data | 58.4% |
| 24 | GPT 4.1Hard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data | 58.3% |
| 25 | DeepSeek R1Hard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data | 58% |
| 26 | o1Creative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data | 56.6% |
| 27 | o1Hard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data | 55.9% |
| 28 | o4 MiniCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data | 55.6% |
| 29 | Qwen3 32BCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data | 53.3% |
| 30 | GPT-4.5 PreviewCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data | 51.4% |
| 31 | o1Hard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data | 50.2% |
| 32 | GPT 4.1Hard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data | 50% |
| 33 | GPT-4.5 PreviewHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data | 50% |
| 34 | GPT-4.1 miniHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data | 48.6% |
| 35 | DeepSeek R1Hard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data | 48% |
| 36 | Claude Sonnet 3.7Hard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data | 47% |
| 37 | GPT-4.1 miniHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data | 46.9% |
| 38 | Qwen 3 235B A22BHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data | 46.7% |
| 39 | Gemini 2.5 FlashHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data | 45.1% |
| 40 | Qwen3 32BHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data | 44.5% |
| 41 | QwQ 32BHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data | 43.5% |
| 42 | GPT-4.5 PreviewHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data | 43% |
| 43 | o3 MiniCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data | 43% |
| 44 | QwQ 32BHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data | 36.1% |
| 45 | Qwen3 32BHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data | 35.8% |
| 46 | Qwen3 30B A3BCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data | 34.9% |
| 47 | Qwen3 30B A3BHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data | 33.9% |
| 48 | Claude 3.5 SonnetHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data | 33% |
| 49 | Qwen3 30B A3BHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data | 28.7% |
| 50 | GPT-4.1 miniCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data | 28.2% |
Showing 1–50 of 69 results.
Data credits: Arena-Hard-Auto, LMArena, Apache-2.0 (licence)