Skip to content

Benchmark

Arena ELO - Coding

Coding benchmarks compare models on programming tasks. Higher scores are better.

Models measured
361
Scores recorded
418
Sources
1
IQ weight
1.28

Score distribution

1,325
Mean
1,374.5
Median
162.1
Std dev
1,081.6
10th percentile
1,488
90th percentile
682.8
Min
682.81,564.3
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.28 for chat models, measured over 361 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
351Starling LM 7B Beta· LMArena data1,142.6
352Qwen1.5-14B-Chat· LMArena data1,137.7
353GPT-3.5 Turbo0125· LMArena data1,136.3
354DBRX Instruct Preview· LMArena data1,131.7
355Phi-3 Medium 4K Instruct· LMArena data1,130
356Zephyr ORPO 141B A35B v0.1· LMArena data1,129.1
357Mixtral 8x7B Instruct v0.1· LMArena data1,126.3
358Tulu 2 DPO 70B· LMArena data1,116.1
359GPT-3.5 Turbo1106· LMArena data1,115.7
360OpenChat 3.50106· LMArena data1,112.8
361Granite 3.0 8B Instruct· LMArena data1,112.4
362Gemma 2 2B IT· LMArena data1,112.2
363Yi 34B Chat· LMArena data1,111.7
364Gemini Pro· LMArena data1,107.8
365Qwen1.5-7B-Chat· LMArena data1,107.2
366Gemini ProDev API· LMArena data1,103.9
367Phi-3 Small 8K Instruct· LMArena data1,101.3
368Llama 3.2 3B· LMArena data1,097.6
369Starling LM 7B Alpha· LMArena data1,097.4
370DeepSeek LLM 67B Chat· LMArena data1,095.5
371Phi-3 Mini 4K InstructJune 2024· LMArena data1,093
372Phi-3 Mini 4K Instruct· LMArena data1,092.5
373Granite 3.0 2B Instruct· LMArena data1,090.3
374Snowflake Arctic Instruct· LMArena data1,089.3
375Gemma 1.1 7B IT· LMArena data1,083.7
376Mistral 7B Instruct v0.2· LMArena data1,081.6
377WizardLM 70B· LMArena data1,081.1
378Nous Hermes 2 Mixtral 8x7B DPO· LMArena data1,079.6
379Llama 2 70B Chat· LMArena data1,078.7
380Vicuna 33B· LMArena data1,078.2
381OpenChat 3.5· LMArena data1,074.1
382Qwen-14B-Chat· LMArena data1,071.5
383Llama 3.2 1B Instruct· LMArena data1,070.4
384SOLAR 10.7B Instruct v1.0· LMArena data1,066.7
385OpenHermes 2.5 Mistral 7B· LMArena data1,064.3
386Llama 2 13B Chat· LMArena data1,062.2
387Zephyr 7B Alpha· LMArena data1,052.7
388Gemma 7B IT· LMArena data1,048.1
389SmolLM2 1.7B Instruct· LMArena data1,047.8
390Code Llama 34B Instruct· LMArena data1,045.7
391Zephyr 7B Beta· LMArena data1,044.5
392Vicuna 13B· LMArena data1,040.6
393Phi-3 Mini 128K Instruct· LMArena data1,039.4
394MPT 30B Chat· LMArena data1,039.2
395WizardLM 13B· LMArena data1,034.8
396Gemma 1.1 2B IT· LMArena data1,034.1
397Llama 2 70B SteerLM Chat· LMArena data1,024.7
398Mistral 7B Instruct· LMArena data1,018.4
399OLMo 7B Instruct· LMArena data1,015.6
400Vicuna 7B· LMArena data1,009.9

Showing 351–400 of 418 results.

Data credits: LMArena, CC BY 4.0 (licence)