Skip to content

Benchmark

Arena ELO - Math

Math benchmarks measure how well models solve maths problems. Higher scores are better.

Models measured
350
Scores recorded
405
Sources
1
IQ weight
1.24

Score distribution

1,316
Mean
1,367.4
Median
152.6
Std dev
1,089
10th percentile
1,474.8
90th percentile
837.8
Min
837.81,547.6
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.24 for chat models, measured over 350 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
301Jamba 1.5 Large· LMArena data1,221
302GLM 40520· LMArena data1,218
303Llama 3 70B Instruct· LMArena data1,217.9
304GPT 40613· LMArena data1,217
305Nemotron-4 340B Instruct· LMArena data1,216.1
306QwQ 32B Preview· LMArena data1,213.5
307Claude 3 Sonnet20240229· LMArena data1,213.4
308Gemma 2 27B IT· LMArena data1,212
309OLMo 2 32B Instruct0325· LMArena data1,207.6
310Gemini 1.5 Flash-8B001· LMArena data1,206.7
311Nova Micro 1.0· LMArena data1,206.1
312Mistral Large2402· LMArena data1,199.8
313Aya Expanse 32B· LMArena data1,199.8
314Reka Flash20240904· LMArena data1,195.3
315Llama 3.1 Tulu 3 8B· LMArena data1,194.8
316Ministral 8B2410· LMArena data1,188.2
317Claude 3 Haiku20240307· LMArena data1,188.1
318Command R+08-2024· LMArena data1,187.8
319Qwen1.5-110B-Chat· LMArena data1,185.4
320Mixtral 8x22B Instruct v0.1· LMArena data1,184.3
321Gemma 2 9B IT· LMArena data1,183.2
322Yi 1.5 34B Chat· LMArena data1,182
323Mistral Medium· LMArena data1,179.8
324InternLM2.5 20B Chat· LMArena data1,179.8
325Llama 3.1 8B Instruct· LMArena data1,179.2
326Phi-3 Medium 4K Instruct· LMArena data1,173.4
327Gemma 2 9B IT SimPO· LMArena data1,172.7
328Aya Expanse 8B· LMArena data1,168.1
329Reka Flash 21BOnline, 20240226· LMArena data1,168
330Command R+· LMArena data1,164.4
331Qwen1.5-72B-Chat· LMArena data1,163.7
332Jamba 1.5 Mini· LMArena data1,160.2
333Granite 3.1 2B Instruct· LMArena data1,159.3
334Reka Flash 21B20240226· LMArena data1,155.6
335Qwen1.5-32B-Chat· LMArena data1,155.1
336Phi-3 Mini 4K InstructJune 2024· LMArena data1,152
337Granite 3.1 8B Instruct· LMArena data1,151.6
338Llama 3 8B Instruct· LMArena data1,151.2
339Phi-3 Small 8K Instruct· LMArena data1,150.8
340Zephyr ORPO 141B A35B v0.1· LMArena data1,147.8
341Mixtral 8x7B Instruct v0.1· LMArena data1,147.1
342DBRX Instruct Preview· LMArena data1,145.1
343Granite 3.0 8B Instruct· LMArena data1,143.4
344GPT-3.5 Turbo0125· LMArena data1,141.8
345GPT-3.5 Turbo1106· LMArena data1,141.3
346Gemma 2 2B IT· LMArena data1,134.6
347Gemini ProDev API· LMArena data1,132.1
348Gemini Pro· LMArena data1,128.9
349Llama 3.2 3B· LMArena data1,125.9
350Qwen1.5-14B-Chat· LMArena data1,125.4

Showing 301–350 of 405 results.

Data credits: LMArena, CC BY 4.0 (licence)