Skip to content

Benchmark

Irish-BLiMP per-category score standard deviation

Multilingual benchmarks compare model performance across languages.

Models and systems measured
10
Scores recorded
10
Sources
1
IQ weight
No weight published

Score distribution

7.6
Mean
7.7
Median
1.9
Std dev
5.5
10th percentile
8.6
90th percentile
5.4
Min
5.412.2
The band covers the middle 80% of scores. The line marks the median.

Results

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
claude-sonnet-4.5table: 2· Irish-BLiMP (arXiv:2510.20957v1) data12.2
gemini-2.5-flash-litetable: 2· Irish-BLiMP (arXiv:2510.20957v1) data8.6
gemma-3-27b-ittable: 2· Irish-BLiMP (arXiv:2510.20957v1) data8.2
gpt-5table: 2· Irish-BLiMP (arXiv:2510.20957v1) data8.4
gpt-oss-120btable: 2· Irish-BLiMP (arXiv:2510.20957v1) data6.5
gpt-oss-20btable: 2· Irish-BLiMP (arXiv:2510.20957v1) data5.5
Llama-2-13btable: 2· Irish-BLiMP (arXiv:2510.20957v1) data7.7
Llama-4-Scout-17B-16E-Instr.table: 2· Irish-BLiMP (arXiv:2510.20957v1) data6
Mistral-Small-3.2-24B-Instr.table: 2· Irish-BLiMP (arXiv:2510.20957v1) data5.4
UCCIXtable: 2· Irish-BLiMP (arXiv:2510.20957v1) data7.6

Data credits: Josh McGiff, Khanh-Tung Tran, William Mulcahy, Dáibhidh Ó Luinín, Jake Dalzell, Róisín Ní Bhroin, Adam Burke, Barry O'Sullivan, Hoang D. Nguyen and Nikola S. Nikolov. Irish-BLiMP, arXiv:2510.20957v1, submitted 2025-10-23. CC BY 4.0 article numeric/protocol extract; restructured, printed values unchanged. No task, prompt or model rights asserted. (licence)