Skip to content

Benchmark

WinoGrande

Scores are percentages. Higher is better.

Models and systems measured
53
Scores recorded
53
Sources
3
IQ weight
0.14

Score distribution

75.5%
Mean
77.7%
Median
10%
Std dev
57.1%
10th percentile
85.6%
90th percentile
51.3%
Min
51.3%89.2%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.14 for chat models, measured over 52 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model or systemScore
1Llama 3.1 405B· Epoch AI Benchmarking Hub data89.2%
2Claude 3 Opus· Epoch AI Benchmarking Hub data88.5%
3GPT 4· Epoch AI Benchmarking Hub data
Values from 2 sources
87.5%
4GPT 4 32k· Epoch AI Benchmarking Hub data87.5%
5MiMo V2.5 Pro· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim85.6%
6Command R Plus· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim85.4%
7DeepSeek V3· Epoch AI Benchmarking Hub data85.2%
8Qwen2 72B Instruct· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim85.1%
9Llama 3.1 Nemotron 70B Instruct· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim84.5%
10deepseek-coder-v2-base· Epoch AI Benchmarking Hub data83.7%
11Gemma 2 27B IT· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim83.7%
12Meta Llama 3 70b· Epoch AI Benchmarking Hub data83.5%
13Qwen2.5 72B· Epoch AI Benchmarking Hub data82.3%
14Llama 3.1 405B Instruct· Epoch AI Benchmarking Hub data82.2%
15Qwen2.5 32B Instruct· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim82%
16GPT-3.5 Turbo· Epoch AI Benchmarking Hub data81.6%
17Text Davinci 002· Epoch AI Benchmarking Hub data81.6%
18Phi-3 Medium 128K Instruct· Epoch AI Benchmarking Hub data81.5%
19Phi-3 Small 8K Instruct· Epoch AI Benchmarking Hub data81.5%
20phi-3.5-moe-instruct· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim81.3%
21Qwen 2.5 Coder 32B Instruct· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim80.8%
22Qwen 2.5 Coder 32B· Epoch AI Benchmarking Hub data80.8%
23Gemma 2 9B IT· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim80.6%
24gemma-7b· Epoch AI Benchmarking Hub data79%
25text-davinci-001· Epoch AI Benchmarking Hub data77.7%
26Mixtral 8X7b V0.1· Epoch AI Benchmarking Hub data77.2%
27Mistral Nemo· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim76.8%
28qwen2p5-coder-14b· Epoch AI Benchmarking Hub data76.8%
29Meta Llama 3 8b· Epoch AI Benchmarking Hub data75.7%
30Mistral 7B v0.1· Epoch AI Benchmarking Hub data75.3%
31Claude 3 Sonnet· Epoch AI Benchmarking Hub data75.1%
32Claude 3 Haiku· Epoch AI Benchmarking Hub data74.2%
33LLaMA 13B· Epoch AI Benchmarking Hub data73%
34deepseek-coder-v2-lite-base· Epoch AI Benchmarking Hub data72.9%
35Qwen2.5-Coder-7B· Epoch AI Benchmarking Hub data72.9%
36Qwen2.5 Coder 7B Instruct· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim72.9%
37Llama 2 13b· Epoch AI Benchmarking Hub data72.8%
38Yi-6B· Epoch AI Benchmarking Hub data71.3%
39Phi-3 Mini 4K Instruct· Epoch AI Benchmarking Hub data70.8%
40Llama 2 7b· Epoch AI Benchmarking Hub data69.2%
41qwen2p5-coder-3b· Epoch AI Benchmarking Hub data67.4%
42Phi 4 Mini· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim67%
43gemma-2b· Epoch AI Benchmarking Hub data65.4%
44Llama 3 8B Instruct· Epoch AI Benchmarking Hub data65%
45Starcoder2 15b· Epoch AI Benchmarking Hub data64.3%
46qwen2p5-coder-1p5b· Epoch AI Benchmarking Hub data60.7%
47starcoder2-3b· Epoch AI Benchmarking Hub data57.1%
48starcoder2-7b· Epoch AI Benchmarking Hub data57.1%
49qwen2p5-coder-0p5b· Epoch AI Benchmarking Hub data54.8%
50phi-2· Epoch AI Benchmarking Hub data54.7%

Showing 1–50 of 52 results.

Not ranked

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
MiMo V2 Flash Baseofficial Base table · 5-shot· Reported by Xiaomi MiMo· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.83.8%

Data credits: Epoch AI Benchmarking Hub, CC BY 4.0 (licence) · Data by LLM Stats · BenchBoard by SYuan03 (github.com/SYuan03/benchboard). MIT covers its code and data organization; no results licence is inferred. Numeric relays retain the original stating-source links.