Skip to content

Benchmark

Arena ELO - Industry Mathematical

People compare model answers and choose which they prefer. Higher scores are better.

Models measured
348
Scores recorded
402
Sources
1
IQ weight
1.25

Score distribution

1,321.4
Mean
1,367.7
Median
154.1
Std dev
1,094.5
10th percentile
1,481.4
90th percentile
833.6
Min
833.61,566.4
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.25 for chat models, measured over 348 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
301Jamba 1.5 Large· LMArena data1,227.7
302Gemini 1.5 Flash-8B001· LMArena data1,225.9
303Gemma 2 27B IT· LMArena data1,221.6
304QwQ 32B Preview· LMArena data1,221.6
305Nemotron-4 340B Instruct· LMArena data1,216
306Llama 3 70B Instruct· LMArena data1,213.7
307Claude 3 Sonnet20240229· LMArena data1,213.2
308Nova Micro 1.0· LMArena data1,212.4
309Aya Expanse 32B· LMArena data1,208.2
310Reka Flash20240904· LMArena data1,202.2
311GPT 40613· LMArena data1,201.2
312Command R+08-2024· LMArena data1,198.1
313Gemma 2 9B IT SimPO· LMArena data1,196.1
314OLMo 2 32B Instruct0325· LMArena data1,195.6
315Gemma 2 9B IT· LMArena data1,193.4
316Llama 3.1 Tulu 3 8B· LMArena data1,192.8
317Qwen1.5-110B-Chat· LMArena data1,190.3
318Mistral Large2402· LMArena data1,190
319Claude 3 Haiku20240307· LMArena data1,187.5
320Granite 3.1 2B Instruct· LMArena data1,186.3
321Yi 1.5 34B Chat· LMArena data1,185.6
322Ministral 8B2410· LMArena data1,184.2
323InternLM2.5 20B Chat· LMArena data1,181.1
324Granite 3.1 8B Instruct· LMArena data1,180.6
325Llama 3.1 8B Instruct· LMArena data1,180.4
326Mixtral 8x22B Instruct v0.1· LMArena data1,179.2
327Mistral Medium· LMArena data1,178.1
328Aya Expanse 8B· LMArena data1,176.6
329Phi-3 Medium 4K Instruct· LMArena data1,176
330Qwen1.5-72B-Chat· LMArena data1,172
331Reka Flash 21BOnline, 20240226· LMArena data1,166
332Command R+· LMArena data1,165.3
333Qwen1.5-32B-Chat· LMArena data1,162.8
334Jamba 1.5 Mini· LMArena data1,161.2
335Reka Flash 21B20240226· LMArena data1,158.8
336Llama 3 8B Instruct· LMArena data1,153.5
337Phi-3 Small 8K Instruct· LMArena data1,153.2
338Granite 3.0 8B Instruct· LMArena data1,152.4
339Phi-3 Mini 4K InstructJune 2024· LMArena data1,147.2
340Gemma 2 2B IT· LMArena data1,145.8
341Zephyr ORPO 141B A35B v0.1· LMArena data1,144.8
342Mixtral 8x7B Instruct v0.1· LMArena data1,143.1
343Gemini Pro· LMArena data1,136.2
344Qwen1.5-14B-Chat· LMArena data1,135.8
345DBRX Instruct Preview· LMArena data1,131.2
346Granite 3.0 2B Instruct· LMArena data1,130.7
347Starling LM 7B Beta· LMArena data1,129.5
348Llama 3.2 3B· LMArena data1,128.6
349GPT-3.5 Turbo1106· LMArena data1,128.5
350GPT-3.5 Turbo0125· LMArena data1,127

Showing 301–350 of 402 results.

Data credits: LMArena, CC BY 4.0 (licence)