Skip to content

Benchmark

FinBen EDTSUM BertScore

Higher scores are better.

Models measured
9
Scores recorded
9
Sources
1
IQ weight
0.75

Score distribution

0.654
Mean
0.663
Median
0.037
Std dev
0.61
10th percentile
0.68
90th percentile
0.596
Min
0.5960.72
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.75 for chat models, measured over 9 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1gemini-1.0-propublished-demo-leaderboard, EDTSUM, BertScore· TheFinAI public numeric results0.72
2Llama 2 70bpublished-demo-leaderboard, EDTSUM, BertScore· TheFinAI public numeric results0.68
3GPT-4 Turbopublished-demo-leaderboard, EDTSUM, BertScore· TheFinAI public numeric results0.67
4Qwen2 7B Instructpublished-demo-leaderboard, EDTSUM, BertScore· TheFinAI public numeric results0.67
5DeepSeek V3published-demo-leaderboard, EDTSUM, BertScore· TheFinAI public numeric results0.663
6GPT-3.5 Turbopublished-demo-leaderboard, EDTSUM, BertScore· TheFinAI public numeric results0.66
7Llama-2-7b-chat-hfpublished-demo-leaderboard, EDTSUM, BertScore· TheFinAI public numeric results0.62
8Mixtral 8X7b V0.1published-demo-leaderboard, EDTSUM, BertScore· TheFinAI public numeric results0.61
9DeepSeek R1 Distill Qwen 7Bpublished-demo-leaderboard, EDTSUM, BertScore· TheFinAI public numeric results0.596

Data credits: The Fin AI. Published FinBen result summaries and Open Greek Financial LLM Leaderboard aggregates. Results licence not separately stated; fractional FinBen success metrics converted to percentages.