Skip to content

Benchmark

WildBench: haiku_reward.Coding & Debugging.K=-1

Coding benchmarks compare models on programming tasks. Higher scores are better.

Models measured
32
Scores recorded
32
Sources
1
IQ weight
1.36

Score distribution

-2.5
Mean
2.9
Median
39.2
Std dev
-51.4
10th percentile
48.4
90th percentile
-81.1
Min
-81.158.9
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.36 for chat models, measured over 32 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Gemini 1.5 Proallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data58.9
2Claude 3.5 Sonnetallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data56
3GPT 4oallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data54.6
4GPT-4 Turboallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data48.4
5Claude 3 Opusallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data44.8
6Gemini 1.5 Flashallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data44.6
7Nemotron-4 340B Instructallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data37.8
8GPT-4-0125-previewallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data35.9
9Llama 3 70B Instructallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data27.1
10Claude 3 Sonnetallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data23.7
11Reka Coreallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data23.5
12yi-largeallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data21.3
13DeepSeek Coder V2 Instructallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data16.5
14Yi 1.5 34B Chatallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data11
15Qwen2 72B Instructallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data4.5
16GLM-4-9b-chatallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data2.9
17Command R Plusallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data-14.6
18Mistral Largeallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307, 2402· WildBench direct data-15.2
19Reka Flashallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data-18.8
20Starling LM 7B Betaallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data-20.5
21Nous Hermes 2 Mixtral 8x7B DPOallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data-21.6
22Mixtral 8x7B Instruct v0.1allenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data-25.8
23Llama 3 8B Instructallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data-27.5
24GPT-3.5 Turboallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data-33.8
25Reka Edgeallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data-34.7
26Tulu 2 DPO 70Ballenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data-35
27Command R+allenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data-37.9
28Mistral 7B Instruct v0.2allenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data-38.2
29Llama-2-70b-chat-hfallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data-51.4
30Gemma 7B ITallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data-64.7
31Llama-2-7b-chat-hfallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data-71.3
32Gemma 2B ITallenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307· WildBench direct data-81.1

Data credits: WildBench, Allen Institute for AI