Skip to content

Benchmark

LLM Chess Wrong Moves per 1000 Moves

Lower scores are better.

Models measured
149
Scores recorded
206
Sources
1
IQ weight
1.97

Score distribution

43.7
Mean
9.1
Median
76.9
Std dev
0.483
10th percentile
141
90th percentile
0
Min
0458.3
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.97 for chat models, measured over 149 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
151GPT OSS 20blow· LLM Chess Leaderboard data docs37.6
152Qwen3.5 2Bnone@bf16· LLM Chess Leaderboard data docs40.5
153Qwen3 30B A3Bdefault@q4_k_m· LLM Chess Leaderboard data docs42.1
154GPT-5 nanolow· LLM Chess Leaderboard data docs43.2
155Gemini 2.0 Flash-Lite Previewnone· LLM Chess Leaderboard data docs46.5
156Gemini 2.5 Pro Preview 03-25default· LLM Chess Leaderboard data docs48.6
157Qwen3.5 9Bnone@q8_0· LLM Chess Leaderboard data docs48.7
158Grok 4 Fastnone· LLM Chess Leaderboard data docs53.6
159Mistral Nemo 12B Instructnone· LLM Chess Leaderboard data docs54.5
160Claude 3 Sonnetnone· LLM Chess Leaderboard data docs60.9
161Grok 4.1 Fastnone· LLM Chess Leaderboard data docs62.1
162Qwen 3.8 27Bdefault@q4_k_m· LLM Chess Leaderboard data docs63.2
163Qwen3 4Bdefault@q8· LLM Chess Leaderboard data docs63.4
164DeepSeek R1 Distill Qwen 32Bdefault@q4_k_m|isol_temp06· LLM Chess Leaderboard data docs63.6
165Grok 3none, beta· LLM Chess Leaderboard data docs70.1
166Qwen3.5 9Bdefault@bf16· LLM Chess Leaderboard data docs70.7
167GPT OSS 120Bmedium· LLM Chess Leaderboard data docs71
168QwQ 32B Previewdefault@q4_k_m· LLM Chess Leaderboard data docs71.1
169Llama3 8B 8192none· LLM Chess Leaderboard data docs73.2
170Qwen3.5 4Bdefault@bf16· LLM Chess Leaderboard data docs76.8
171Llama 3.1 8Bnone· LLM Chess Leaderboard data docs77.2
172GPT OSS 120Blow· LLM Chess Leaderboard data docs77.9
173GPT 5.4low· LLM Chess Leaderboard data docs84.1
174GPT 5.4 Nanohigh· LLM Chess Leaderboard data docs88.5
175GPT-4.1 nanonone· LLM Chess Leaderboard data docs91.1
176Granite 3.1 8B Instructnone· LLM Chess Leaderboard data docs95.2
177GLM 5default· LLM Chess Leaderboard data docs96.7
178GPT OSS 120Bhigh· LLM Chess Leaderboard data docs97.5
179DeepSeek R1 Distill Qwen 14Bdefault@q8_0· LLM Chess Leaderboard data docs102.6
180DeepSeek V3none· LLM Chess Leaderboard data docs107.4
181QwQ 32Bdefault· LLM Chess Leaderboard data docs113.3
182Kimi K2 Instructnone· LLM Chess Leaderboard data docs117
183MiniMax M2none· LLM Chess Leaderboard data docs126.1
184GPT-5.4 minihigh· LLM Chess Leaderboard data docs127
185Qwen Turbonone· LLM Chess Leaderboard data docs141
186Mistral Small 24B Instructnone@q4_k_m· LLM Chess Leaderboard data docs144
187Qwen 2.5 14B Instructnone@q8_0· LLM Chess Leaderboard data docs148.2
188Gemini 2.5 Flashdefault· LLM Chess Leaderboard data docs150.5
189Llama 3.1 Tulu 3 8Bnone@q8_0· LLM Chess Leaderboard data docs176.9
190Claude 2.1none· LLM Chess Leaderboard data docs178.4
191MiniMax M2.1none· LLM Chess Leaderboard data docs197.1
192Phi 4none· LLM Chess Leaderboard data docs198.3
193Gemma 3 4B ITnone@iq4_qs@PGN· LLM Chess Leaderboard data docs208.7
194Magistral Small 2509none· LLM Chess Leaderboard data docs211.8
195Claude 2none· LLM Chess Leaderboard data docs226.7
196Nova Pronone· LLM Chess Leaderboard data docs226.8
197Nova Litenone· LLM Chess Leaderboard data docs237.1
198Gemma 3 4B ITnone@iq4_qs· LLM Chess Leaderboard data docs240.5
199Qwen3 14Bdefault@iq4_xs· LLM Chess Leaderboard data docs245.2
200Gemma 3 4B ITnone@bf16· LLM Chess Leaderboard data docs274.3

Showing 151–200 of 206 results.

Data credits: LLM Chess by Maxim Saplin, Apache-2.0 (Copyright [2024] [Maxim Saplin]) (licence)