Skip to content

Benchmark

Estonian Native Benchmark word meaning accuracy (instruction-tuned models)

Multilingual benchmarks compare model performance across languages. Higher scores are better.

Models and systems measured
26
Scores recorded
26
Sources
1
IQ weight
No weight published

Score distribution

0.355
Mean
0.338
Median
0.142
Std dev
0.177
10th percentile
0.555
90th percentile
0.09
Min
0.090.631
The band covers the middle 80% of scores. The line marks the median.

Results

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
Claude Sonnet 3.7instruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.57
Claude Sonnet 4instruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.45
Deepseek V3 (671B-A37B)instruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.452
EuroLLM-9B-Instructinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.338
Gemini 2.0 Flashinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.54
Gemini 2.5 Proinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.631
Gemma-2-27B-Itinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.266
Gemma-2-9B-Itinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.178
Gemma-3 27B Instructinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.356
Gemma-3-12b-Itinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.289
GPT-4oinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.555
GPT-4o Miniinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.331
GPT-5instruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.529
GPT-OSS-120Binstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.29
Kimi-K2-0905 (1000B-A32B)instruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.435
Llama-3.1 405B Instructinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.41
Llama-3.1 70B Instructinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.306
Llama-3.1 8B EstLLM-0825instruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.327
Llama-3.1 8B Instructinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.132
Llama-4 Maverick (400B-A17B)instruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.453
Llama-4 Scout (109B-A17B)instruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.407
Mistral Nemo Instruct (12B)instruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.177
Qwen2-72B-Instructinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.152
Qwen3-235B-A22B-2507instruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.324
Qwen3-4B-Instruct-2507instruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.09
Qwen3-Next-80B-A3B-Instructinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Words· Estonian Native Benchmark (arXiv:2510.21193v2) data0.246

Data credits: Lillepalu and Alumäe, Estonian Native Large Language Model Benchmark, arXiv:2510.21193v2. Cited numeric facts only. arXiv non-exclusive distribution is not a results reuse grant. Emitted and committed payloads exclude tasks, examples and generations. Human preference models were available in March 2025; judge evaluation was in March 2025.