Benchmark
Arena-Hard-v2.0-Preview
People compare model answers and choose which they prefer. Scores are percentages. Higher is better.
- Models measured
- 21
- Scores recorded
- 69
- Sources
- 1
- IQ weight
- 1.81
Score distribution
- 45.2%
- Mean
- 48%
- Median
- 23.4%
- Std dev
- 12.6%
- 10th percentile
- 77%
- 90th percentile
- 8%
- Min
This benchmark has a weight of 1.81 for chat models, measured over 21 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.
Leaderboard
| # | Model | Score |
|---|---|---|
| 51 | Claude 3.5 SonnetHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data | 25.8% |
| 52 | Claude 3.5 SonnetCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data | 24.2% |
| 53 | Athene V2 ChatCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data | 18.1% |
| 54 | Llama 4 MaverickHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data | 17.2% |
| 55 | Athene V2 ChatHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data | 16.4% |
| 56 | GPT-4.1 nanoHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data | 15.4% |
| 57 | Gemma 3 27B ITHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data | 15% |
| 58 | Qwen3 4BHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data | 15% |
| 59 | GPT-4.1 nanoHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data | 13.7% |
| 60 | Qwen3 4BCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data | 13.2% |
| 61 | Athene V2 ChatHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data | 12.6% |
| 62 | Qwen3 4BHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data | 12.6% |
| 63 | Llama 4 MaverickHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data | 12% |
| 64 | GPT-4.1 nanoCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data | 10.7% |
| 65 | Llama 4 MaverickCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data | 10.5% |
| 66 | Qwen2.5 72B InstructCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data | 10.2% |
| 67 | Qwen2.5 72B InstructHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data | 10.1% |
| 68 | Gemma 3 27B ITHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data | 9.7% |
| 69 | Qwen2.5 72B InstructHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data | 8% |
Showing 51–69 of 69 results.
Data credits: Arena-Hard-Auto, LMArena, Apache-2.0 (licence)