Skip to content

Benchmark

WildBench: mixture_of_rewards.Coding & Debugging.K=-1

Coding benchmarks compare models on programming tasks. Higher scores are better.

Models measured
30
Scores recorded
30
Sources
1
IQ weight
1.46

Score distribution

-4.8
Mean
0.621
Median
32.7
Std dev
-32.1
10th percentile
35.3
90th percentile
-79
Min
-7948.7
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.46 for chat models, measured over 30 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Gemini 1.5 Proallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data48.7
2Claude 3.5 Sonnetallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data47.2
3GPT 4oallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data41.7
4Gemini 1.5 Flashallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data35.3
5GPT-4-0125-previewallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data31.1
6Claude 3 Opusallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data28.3
7Nemotron-4 340B Instructallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data25.7
8Llama 3 70B Instructallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data20.7
9Claude 3 Sonnetallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data15.9
10Reka Coreallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data15.7
11yi-largeallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data13.7
12DeepSeek Coder V2 Instructallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data12
13Yi 1.5 34B Chatallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data7.4
14GLM-4-9b-chatallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data0.669
15Qwen2 72B Instructallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data0.621
16Mistral Largeallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf, 2402· WildBench direct data-13.8
17Reka Flashallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data-14.4
18Command R Plusallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data-15.7
19Starling LM 7B Betaallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data-18.8
20Mixtral 8x7B Instruct v0.1allenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data-22.1
21Llama 3 8B Instructallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data-22.7
22Nous Hermes 2 Mixtral 8x7B DPOallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data-23.4
23Tulu 2 DPO 70Ballenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data-29.3
24GPT-3.5 Turboallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data-30.3
25Reka Edgeallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data-30.8
26Mistral 7B Instruct v0.2allenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data-31.1
27Command R+allenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data-32.1
28Gemma 7B ITallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data-58.3
29Llama-2-7b-chat-hfallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data-68
30Gemma 2B ITallenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf· WildBench direct data-79

Data credits: WildBench, Allen Institute for AI