Skip to content

Benchmark

Arena-Hard-v2.0-Preview

People compare model answers and choose which they prefer. Scores are percentages. Higher is better.

Models measured
21
Scores recorded
69
Sources
1
IQ weight
1.81

Score distribution

45.2%
Mean
48%
Median
23.4%
Std dev
12.6%
10th percentile
77%
90th percentile
8%
Min
8%88.8%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.81 for chat models, measured over 21 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
51Claude 3.5 SonnetHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data25.8%
52Claude 3.5 SonnetCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data24.2%
53Athene V2 ChatCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data18.1%
54Llama 4 MaverickHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data17.2%
55Athene V2 ChatHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data16.4%
56GPT-4.1 nanoHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data15.4%
57Gemma 3 27B ITHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data15%
58Qwen3 4BHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data15%
59GPT-4.1 nanoHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data13.7%
60Qwen3 4BCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data13.2%
61Athene V2 ChatHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data12.6%
62Qwen3 4BHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data12.6%
63Llama 4 MaverickHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data12%
64GPT-4.1 nanoCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data10.7%
65Llama 4 MaverickCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data10.5%
66Qwen2.5 72B InstructCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data10.2%
67Qwen2.5 72B InstructHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data10.1%
68Gemma 3 27B ITHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data9.7%
69Qwen2.5 72B InstructHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data8%

Showing 51–69 of 69 results.

Data credits: Arena-Hard-Auto, LMArena, Apache-2.0 (licence)