Skip to content

Benchmark

Macedonian LM Evaluation Harness WinoGrande accuracy

Multilingual benchmarks compare model performance across languages. Higher scores are better.

Models and systems measured
12
Scores recorded
12
Sources
1
IQ weight
No weight published

Score distribution

0.582
Mean
0.606
Median
0.042
Std dev
0.519
10th percentile
0.63
90th percentile
0.509
Min
0.5090.63
The band covers the middle 80% of scores. The line marks the median.

Results

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
domestic-yak-basetable: 4, task: WinoGrande, stage: domestic-yak-base· Macedonian Foundation Model (arXiv:2506.09560v1) data0.63
domestic-yak-instructtable: 4, task: WinoGrande, stage: domestic-yak-instruct· Macedonian Foundation Model (arXiv:2506.09560v1) data0.63
domestic-yak†8Btable: 3, task: WinoG· Macedonian Foundation Model (arXiv:2506.09560v1) data0.627
Llama 3.18Btable: 3, task: WinoG· Macedonian Foundation Model (arXiv:2506.09560v1) data0.568
Llama 3.1Llama-3.1-8B-Instructtable: 4, task: WinoGrande, stage: Llama 3.1· Macedonian Foundation Model (arXiv:2506.09560v1) data0.57
Llama 3.21Btable: 3, task: WinoG· Macedonian Foundation Model (arXiv:2506.09560v1) data0.509
Llama 3.370Btable: 3, task: WinoG· Macedonian Foundation Model (arXiv:2506.09560v1) data0.609
Mistral7Btable: 3, task: WinoG· Macedonian Foundation Model (arXiv:2506.09560v1) data0.561
Mistral Nemo12Btable: 3, task: WinoG· Macedonian Foundation Model (arXiv:2506.09560v1) data0.606
MKLLM†7Btable: 3, task: WinoG· Macedonian Foundation Model (arXiv:2506.09560v1) data0.615
Phi-3.5-mini3.8Btable: 3, task: WinoG· Macedonian Foundation Model (arXiv:2506.09560v1) data0.519
Qwen2.57Btable: 3, task: WinoG· Macedonian Foundation Model (arXiv:2506.09560v1) data0.535

Data credits: Stefan Krsteski, Matea Tashkovska, Borjan Sazdov, Hristijan Gjoreski and Branislav Gerazov. Towards Open Foundation Language Model and Corpus for Macedonian, arXiv:2506.09560v1. CC BY 4.0 article numeric/protocol extract; restructured, printed values unchanged. No task, corpus, weight or raw survey rights asserted. (licence)