Skip to content

Benchmark

Arena-Hard-v2.0-Preview

People compare model answers and choose which they prefer. Scores are percentages. Higher is better.

Models measured
21
Scores recorded
69
Sources
1
IQ weight
1.81

Score distribution

45.2%
Mean
48%
Median
23.4%
Std dev
12.6%
10th percentile
77%
90th percentile
8%
Min
8%88.8%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.81 for chat models, measured over 21 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1OpenAI o3Creative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data88.8%
2OpenAI o3Hard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data87%
3OpenAI o3Hard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data85.9%
4Gemini 2.5 FlashCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data83.9%
5o4 MiniHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data81.7%
6o4 MiniHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data79.1%
7o4 MiniHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data78%
8DeepSeek R1Creative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data77%
9o4 MiniHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data74.6%
10Qwen 3 235B A22BCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data73.5%
11Gemma 3 27B ITCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data69.9%
12Gemini 2.5 FlashHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data68.6%
13o3 MiniHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data66.1%
14o3 MiniHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data64.8%
15Claude Sonnet 3.7Creative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data63.9%
16GPT 4.1Creative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data61.5%
17o1Hard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data61%
18QwQ 32BCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data60.9%
19o1Creative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data59.9%
20Claude Sonnet 3.7Hard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data59.8%
21o1Hard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data58.7%
22o4 MiniCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data58.7%
23Qwen 3 235B A22BHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data58.4%
24GPT 4.1Hard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data58.3%
25DeepSeek R1Hard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data58%
26o1Creative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data56.6%
27o1Hard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data55.9%
28o4 MiniCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data55.6%
29Qwen3 32BCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data53.3%
30GPT-4.5 PreviewCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data51.4%
31o1Hard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data50.2%
32GPT 4.1Hard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data50%
33GPT-4.5 PreviewHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data50%
34GPT-4.1 miniHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data48.6%
35DeepSeek R1Hard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data48%
36Claude Sonnet 3.7Hard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data47%
37GPT-4.1 miniHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data46.9%
38Qwen 3 235B A22BHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data46.7%
39Gemini 2.5 FlashHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data45.1%
40Qwen3 32BHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data44.5%
41QwQ 32BHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data43.5%
42GPT-4.5 PreviewHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data43%
43o3 MiniCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data43%
44QwQ 32BHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data36.1%
45Qwen3 32BHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data35.8%
46Qwen3 30B A3BCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data34.9%
47Qwen3 30B A3BHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data33.9%
48Claude 3.5 SonnetHard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)· Arena-Hard-Auto data33%
49Qwen3 30B A3BHard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)· Arena-Hard-Auto data28.7%
50GPT-4.1 miniCreative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)· Arena-Hard-Auto data28.2%

Showing 1–50 of 69 results.

Data credits: Arena-Hard-Auto, LMArena, Apache-2.0 (licence)