Skip to content

Benchmark

FinBen EDTSUM Rouge-1

Scores are percentages. Higher is better.

Models measured
12
Scores recorded
12
Sources
1
IQ weight
0.44

Score distribution

20.2%
Mean
20%
Median
7.1%
Std dev
12%
10th percentile
25%
90th percentile
8.9%
Min
8.9%39%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.44 for chat models, measured over 12 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1gemini-1.0-propublished-demo-leaderboard, EDTSUM, Rouge-1· TheFinAI public numeric results39%
2Llama 2 70bpublished-demo-leaderboard, EDTSUM, Rouge-1· TheFinAI public numeric results25%
3Qwen 2 (72B)published-demo-leaderboard, EDTSUM, Rouge-1· TheFinAI public numeric results22%
4Qwen2 7B Instructpublished-demo-leaderboard, EDTSUM, Rouge-1· TheFinAI public numeric results22%
5DeepSeek V3published-demo-leaderboard, EDTSUM, Rouge-1· TheFinAI public numeric results21.7%
6GPT-4 Turbopublished-demo-leaderboard, EDTSUM, Rouge-1· TheFinAI public numeric results20%
7Llama 3.1 8Bpublished-demo-leaderboard, EDTSUM, Rouge-1· TheFinAI public numeric results20%
8Llama 3.1 70Bpublished-demo-leaderboard, EDTSUM, Rouge-1· TheFinAI public numeric results18%
9GPT-3.5 Turbopublished-demo-leaderboard, EDTSUM, Rouge-1· TheFinAI public numeric results17%
10Llama-2-7b-chat-hfpublished-demo-leaderboard, EDTSUM, Rouge-1· TheFinAI public numeric results17%
11Mixtral 8X7b V0.1published-demo-leaderboard, EDTSUM, Rouge-1· TheFinAI public numeric results12%
12DeepSeek R1 Distill Qwen 7Bpublished-demo-leaderboard, EDTSUM, Rouge-1· TheFinAI public numeric results8.9%

Data credits: The Fin AI. Published FinBen result summaries and Open Greek Financial LLM Leaderboard aggregates. Results licence not separately stated; fractional FinBen success metrics converted to percentages.