Skip to content

Benchmark

Estonian Native Benchmark speaker name extraction F1 (instruction-tuned models)

Multilingual benchmarks compare model performance across languages. Higher scores are better.

Models and systems measured
24
Scores recorded
24
Sources
1
IQ weight
No weight published

Score distribution

0.572
Mean
0.656
Median
0.239
Std dev
0.038
10th percentile
0.785
90th percentile
0
Min
00.82
The band covers the middle 80% of scores. The line marks the median.

Results

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
Claude Sonnet 3.7instruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0.785
Claude Sonnet 4instruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0.776
Deepseek V3 (671B-A37B)instruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0.668
EuroLLM-9B-Instructinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0.038
Gemini 2.0 Flashinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0.748
Gemini 2.5 Proinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0.82
Gemma-3 27B Instructinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0.471
Gemma-3-12b-Itinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0
GPT-4oinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0.724
GPT-4o Miniinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0.651
GPT-5instruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0.806
GPT-OSS-120Binstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0
Kimi-K2-0905 (1000B-A32B)instruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0.783
Llama-3.1 405B Instructinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0.702
Llama-3.1 70B Instructinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0.674
Llama-3.1 8B EstLLM-0825instruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0.452
Llama-3.1 8B Instructinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0.505
Llama-4 Maverick (400B-A17B)instruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0.656
Llama-4 Scout (109B-A17B)instruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0.622
Mistral Nemo Instruct (12B)instruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0.41
Qwen2-72B-Instructinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0.591
Qwen3-235B-A22B-2507instruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0.713
Qwen3-4B-Instruct-2507instruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0.54
Qwen3-Next-80B-A3B-Instructinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: Speakers· Estonian Native Benchmark (arXiv:2510.21193v2) data0.585

Data credits: Lillepalu and Alumäe, Estonian Native Large Language Model Benchmark, arXiv:2510.21193v2. Cited numeric facts only. arXiv non-exclusive distribution is not a results reuse grant. Emitted and committed payloads exclude tasks, examples and generations. Human preference models were available in March 2025; judge evaluation was in March 2025.