Skip to content

Benchmark

LLM Chess Wrong Moves per 1000 Moves

Lower scores are better.

Models measured
149
Scores recorded
206
Sources
1
IQ weight
1.97

Score distribution

43.7
Mean
9.1
Median
76.9
Std dev
0.483
10th percentile
141
90th percentile
0
Min
0458.3
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.97 for chat models, measured over 149 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
101DeepSeek V3.2none· LLM Chess Leaderboard data docs7.5
102DeepSeek R1default· LLM Chess Leaderboard data docs8.2
103Gemini 2.0 Flashnone· LLM Chess Leaderboard data docs8.9
104Claude Opus 4.1none· LLM Chess Leaderboard data docs9.1
105Llama 3 70B Instructnone· LLM Chess Leaderboard data docs9.2
106o3 Minimedium· LLM Chess Leaderboard data docs9.3
107GPT-5 nanohigh· LLM Chess Leaderboard data docs9.3
108o4 Minilow@PGN· LLM Chess Leaderboard data docs10
109GPT-5 minimedium· LLM Chess Leaderboard data docs10.3
110Claude Sonnet 4none· LLM Chess Leaderboard data docs10.7
111Claude Sonnet 5none· LLM Chess Leaderboard data docs10.8
112Claude Sonnet 3.7none· LLM Chess Leaderboard data docs10.9
113o3 Minilow· LLM Chess Leaderboard data docs10.9
114Claude Opus 4none· LLM Chess Leaderboard data docs11.5
115o3 Minihigh· LLM Chess Leaderboard data docs11.8
116Claude Haiku 5.5high· LLM Chess Leaderboard data docs11.9
117GPT-5.6 Lunamedium· LLM Chess Leaderboard data docs12
118GPT-5.2 Chatnone· LLM Chess Leaderboard data docs14.2
119Kimi K2.5default· LLM Chess Leaderboard data docs14.2
120gemini-1.5-flash-001none· LLM Chess Leaderboard data docs14.3
121Gemma 2 9B ITnone· LLM Chess Leaderboard data docs14.9
122GLM 5.3high· LLM Chess Leaderboard data docs15.8
123Agents A1default@q4_k_m· LLM Chess Leaderboard data docs15.8
124Gemma 2 27B ITnone@q6_k_l· LLM Chess Leaderboard data docs15.9
125GPT-5 nanomedium· LLM Chess Leaderboard data docs16.2
126Grok 2none· LLM Chess Leaderboard data docs16.6
127Qwen3 235B A22B Instruct 2507none· LLM Chess Leaderboard data docs16.7
128GPT-4o mininone· LLM Chess Leaderboard data docs16.9
129Gemma 3 12B ITnone@iq4_xs· LLM Chess Leaderboard data docs17.5
130GPT-5 Chatnone· LLM Chess Leaderboard data docs18
131Claude Sonnet 4.5none· LLM Chess Leaderboard data docs18
132GLM 5.3 Flashhigh· LLM Chess Leaderboard data docs19.2
133Claude Haiku 3.5none· LLM Chess Leaderboard data docs19.3
134Claude Haiku 4.5none· LLM Chess Leaderboard data docs20.4
135Qwen3.6 27Bdefault@q4_k_s· LLM Chess Leaderboard data docs20.4
136Grok 3 Minihigh, beta· LLM Chess Leaderboard data docs22.2
137GPT OSS 20bmedium· LLM Chess Leaderboard data docs24.8
138GPT-5.4 minilow· LLM Chess Leaderboard data docs30.5
139Qwen Plusnone· LLM Chess Leaderboard data docs30.8
140Gemini 2.0 Flash Litenone· LLM Chess Leaderboard data docs30.9
141Gemini 2.5 Pro Preview 05-06default· LLM Chess Leaderboard data docs31.2
142Llama 2 7B Chatnone· LLM Chess Leaderboard data docs31.3
143Gemini 2.0 Flash Experimentalnone· LLM Chess Leaderboard data docs31.4
144Qwen3.5 35B A3Bdefault@q4_k_m· LLM Chess Leaderboard data docs31.8
145Gemma 3 12B ITnone@q8_0· LLM Chess Leaderboard data docs32.1
146GPT OSS 20bhigh· LLM Chess Leaderboard data docs33.1
147Gemini 2.5 Prodefault· LLM Chess Leaderboard data docs33.4
148Grok 4.20none· LLM Chess Leaderboard data docs34.5
149Llama 3.3 70B Instructnone· LLM Chess Leaderboard data docs34.7
150Qwen2.5 72B Instructnone· LLM Chess Leaderboard data docs36.9

Showing 101–150 of 206 results.

Data credits: LLM Chess by Maxim Saplin, Apache-2.0 (Copyright [2024] [Maxim Saplin]) (licence)