Skip to content

Benchmark

Estonian Native Benchmark open-ended Claude 3.7 Sonnet judge win rate

Multilingual benchmarks compare model performance across languages. Higher scores are better.

Models and systems measured
12
Scores recorded
12
Sources
1
IQ weight
No weight published

Score distribution

0.517
Mean
0.62
Median
0.212
Std dev
0.214
10th percentile
0.77
90th percentile
0.205
Min
0.2050.772
The band covers the middle 80% of scores. The line marks the median.

Results

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
Claude 3.7 Sonnetinstruction-tuned modelstable: 12, evaluation: pairwise-open-ended· Estonian Native Benchmark (arXiv:2510.21193v2) data0.759
Deepseek V3instruction-tuned modelstable: 12, evaluation: pairwise-open-ended· Estonian Native Benchmark (arXiv:2510.21193v2) data0.772
Gemini 2.0 Flash 001instruction-tuned modelstable: 12, evaluation: pairwise-open-ended· Estonian Native Benchmark (arXiv:2510.21193v2) data0.77
Gemma 2 27B Itinstruction-tuned modelstable: 12, evaluation: pairwise-open-ended· Estonian Native Benchmark (arXiv:2510.21193v2) data0.654
Gemma 2 9B Itinstruction-tuned modelstable: 12, evaluation: pairwise-open-ended· Estonian Native Benchmark (arXiv:2510.21193v2) data0.214
GPT-4oinstruction-tuned modelstable: 12, evaluation: pairwise-open-ended· Estonian Native Benchmark (arXiv:2510.21193v2) data0.677
GPT-4o Miniinstruction-tuned modelstable: 12, evaluation: pairwise-open-ended· Estonian Native Benchmark (arXiv:2510.21193v2) data0.62
Llama 3.1 405B Instructinstruction-tuned modelstable: 12, evaluation: pairwise-open-ended· Estonian Native Benchmark (arXiv:2510.21193v2) data0.428
Llama 3.1 70B Instructinstruction-tuned modelstable: 12, evaluation: pairwise-open-ended· Estonian Native Benchmark (arXiv:2510.21193v2) data0.473
Llama 3.1 8B Instructinstruction-tuned modelstable: 12, evaluation: pairwise-open-ended· Estonian Native Benchmark (arXiv:2510.21193v2) data0.205
Mistral Nemo Instructinstruction-tuned modelstable: 12, evaluation: pairwise-open-ended· Estonian Native Benchmark (arXiv:2510.21193v2) data0.226
Qwen2 72B Instructinstruction-tuned modelstable: 12, evaluation: pairwise-open-ended· Estonian Native Benchmark (arXiv:2510.21193v2) data0.403

Data credits: Lillepalu and Alumäe, Estonian Native Large Language Model Benchmark, arXiv:2510.21193v2. Cited numeric facts only. arXiv non-exclusive distribution is not a results reuse grant. Emitted and committed payloads exclude tasks, examples and generations. Human preference models were available in March 2025; judge evaluation was in March 2025.