Skip to content

Benchmark

Estonian Native Benchmark news summarization ROUGE-L (instruction-tuned models)

Multilingual benchmarks compare model performance across languages. Higher scores are better.

Models and systems measured
26
Scores recorded
26
Sources
1
IQ weight
No weight published

Score distribution

0.118
Mean
0.124
Median
0.022
Std dev
0.105
10th percentile
0.138
90th percentile
0.034
Min
0.0340.152
The band covers the middle 80% of scores. The line marks the median.

Results

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
Claude Sonnet 3.7instruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.135
Claude Sonnet 4instruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.131
Deepseek V3 (671B-A37B)instruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.113
EuroLLM-9B-Instructinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.104
Gemini 2.0 Flashinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.124
Gemini 2.5 Proinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.138
Gemma-2-27B-Itinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.128
Gemma-2-9B-Itinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.128
Gemma-3 27B Instructinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.118
Gemma-3-12b-Itinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.119
GPT-4oinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.127
GPT-4o Miniinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.115
GPT-5instruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.114
GPT-OSS-120Binstruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.074
Kimi-K2-0905 (1000B-A32B)instruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.14
Llama-3.1 405B Instructinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.129
Llama-3.1 70B Instructinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.129
Llama-3.1 8B EstLLM-0825instruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.152
Llama-3.1 8B Instructinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.119
Llama-4 Maverick (400B-A17B)instruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.131
Llama-4 Scout (109B-A17B)instruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.125
Mistral Nemo Instruct (12B)instruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.034
Qwen2-72B-Instructinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.112
Qwen3-235B-A22B-2507instruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.108
Qwen3-4B-Instruct-2507instruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.105
Qwen3-Next-80B-A3B-Instructinstruction-tuned modelstable: 9, setting: instruction-tuned models, task: News· Estonian Native Benchmark (arXiv:2510.21193v2) data0.109

Data credits: Lillepalu and Alumäe, Estonian Native Large Language Model Benchmark, arXiv:2510.21193v2. Cited numeric facts only. arXiv non-exclusive distribution is not a results reuse grant. Emitted and committed payloads exclude tasks, examples and generations. Human preference models were available in March 2025; judge evaluation was in March 2025.