Skip to content

Anthropic · Chat

Claude 3.5 Sonnet

Anthropic 🇺🇸ChatClaude familyOne of its sources says the weights aren't publicly available.The maker lists this model as deprecated (13 Nov 2026).

Newer version: Claude Sonnet 5.5. We compared release dates and version names in our catalogue.

llminfo IQ
119.3
Maker's price
$6.00 blended / 1M
Max context
200K
Listings
34

Evidence and sources

llminfo IQ 119.3, ranked among chat models. From 546 benchmarks. Plenty of data behind it.How llminfo IQ is computedCoding IQ 89.4, 95% interval 81–98. From 12 benchmark families across 12 sources. Plenty of data behind it. Uses the model's best published results. It has its own scale, separate from llminfo IQ.How Coding IQ is computedPoor value (27.0). Costs more per task than its benchmark results predict. From 18 benchmarks with measured task costs. Plenty of data behind it.How Value for money worksReleased 20 Jun 2024Maker HQ: US (according to Anthropic's own website)
Sources: LMArena data, LiteLLM data, Models.dev data, llminfo researchPrimary data from LMArena from LiteLLM: deprecation date from Models.dev: weights availability from llminfo research: release date

Advantages and trade-offs

Compared with 10 chat models, picked mainly by popularity and IQ

Trade-offs

Deprecated by its maker

The maker has already deprecated this model (13 Nov 2026).

No open weights

The weights aren't public, so you can't run it yourself.

We only make a claim when there's enough data on models of the same type. Updated 22 Nov, 06:07 UTC.Prices come from the model's maker where they sell it, otherwise their official partner, otherwise the market median. This comparison uses 9 maker's prices; 1 market median price.

Input & output

as stated by the model's providers
Text
Images
Files
Text

Capabilities

Supported operations and provider-stated features

VisionFunction callingPrompt cachingPDF inputComputer useBatch API

Listings

 Anthropicclaude-3-5-sonnetSource: GenAI Prices data200K$3.00cache read $0.30$15.00cache write $3.75
 Poeanthropic/claude-sonnet-3.5Source: Models.dev data189K$2.60cache read $0.26$13.00cache write $3.20
 Amazon Web ServicesRegions: regionalregional.anthropic.claude-3-5-sonnet-20240620-v1:0 · 20240620Source: GenAI Prices dataNot published by the provider$3.00cache read $0.30$15.00cache write $3.75
 Googleclaude-3-5-sonnetSources: GenAI Prices data, LiteLLM data200K$3.00cache read $0.30$15.00cache write $3.75
 302aiclaude-3-5-sonnet-20240620 · 20240620Source: 302.ai data200K$3.00cache read $0.30$15.00cache write $3.75
 Flowdotclaude-3-5-sonnetSource: FlowDot data200K$3.00$15.00
 OpenRouterclaude-3.5-sonnetSource: GenAI Prices dataNot published by the provider$3.00cache read $0.30$15.00cache write $3.75
 Vercelanthropic/claude-3-5-sonnetSource: Helicone Costs dataNot published by the provider$3.00$15.00
 Gradient AIgradient_ai/anthropic-claude-3.5-sonnetSource: LiteLLM data200K$3.00$15.00
 Snowflakesnowflake/claude-3-5-sonnetSource: LiteLLM data200K$3.00cache read $0.30$15.00
Show all 32 listings

Showing 10 of 32: the maker's own listing first, then the cheapest per 1M input tokens from each provider.

Also sold by Replicate, Vercel AI Gateway.

2 listings without a price are hidden.

Benchmarks

Named effort levels run from lowest to highest. Select a result to see its source details.

Safety

HELM66 related benchmarks · 125 results
Show 114 more resultsHide 114 results
AIR-Bench: AIRBench 2024 - #13: Harassment
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
AIR-Bench: AIRBench 2024 - #18: Erotic
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
AIR-Bench: AIRBench 2024 - #20: Monetized
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
AIR-Bench: AIRBench 2024 - #32: Fraud
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
AIR-Bench: AIRBench 2024 - Child Harm
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
AIR-Bench: AIRBench 2024 - Deception
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
AIR-Bench: AIRBench 2024 - Defamation
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
AIR-Bench: AIRBench 2024 - Economic Harm
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
AIR-Bench: AIRBench 2024 - Hate/Toxicity
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
AIR-Bench: AIRBench 2024 - Manipulation
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
AIR-Bench: AIRBench 2024 - Political Usage
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
AIR-Bench: AIRBench 2024 - Privacy
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
AIR-Bench: AIRBench 2024 - Refusal Rate
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
AIR-Bench: AIRBench 2024 - Security Risks
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
AIR-Bench: AIRBench 2024 - Self-harm
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
AIR-Bench: AIRBench 2024 - Sexual Content
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated

Knowledge

HELM60 related benchmarks · 116 results
Show 108 more resultsHide 108 results
MMLU: Astronomy - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Business Ethics - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Clinical Knowledge - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: College Biology - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: College Chemistry - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: College Computer Science - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: College Mathematics - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: College Medicine - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: College Physics - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Computer Security - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Conceptual Physics - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Econometrics - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Electrical Engineering - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Elementary Mathematics - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: High School Biology - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: High School Chemistry - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: High School European History - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: High School Geography - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: High School Macroeconomics - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: High School Mathematics - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: High School Microeconomics - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: High School Physics - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: High School Psychology - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: High School Statistics - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: High School US History - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: High School World History - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Human Aging - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Human Sexuality - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: International Law - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Jurisprudence - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Logical Fallacies - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Machine Learning - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Management - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Marketing - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Medical Genetics - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Miscellaneous - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: MMLU All Subjects - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Moral Disputes - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Moral Scenarios - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Nutrition - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Philosophy - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Prehistory - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Professional Accounting - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Professional Law - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Professional Medicine - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Professional Psychology - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Public Relations - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Security Studies - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Sociology - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Us Foreign Policy - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: Virology - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLU: World Religions - EM
Source: Stanford HELM specialized suites data docs
Setting not stated
Setting not stated
MMLUSource: LLM Stats data · Reported by the model's developer. The source relaying it did not verify it. claim
MMLU-ProSource: LLM Stats data · Reported by the model's developer. The source relaying it did not verify it. claim

Human preference

Arena ELO34 related benchmarks · 67 results
Arena ELO
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Arena-Hard-v2.0-Preview
Source: Arena-Hard-Auto data
Creative Writing, Ensemble GPT-4.1 and Gemini 2.5 as Judges (Best Configuration for Creative Writing)
Hard Prompt, Style Control, and Gemini-2.5 as Judge (Official Configuration)
Hard Prompt, Style Control, and GPT-4.1 as Judge (If prefer OpenAI API)
Chinese
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Creative Writing
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
English
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Exclude Ties
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Show 54 more resultsHide 54 results
Expert
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
French
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
German
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Hard Prompts
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Hard Prompts English
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Industry Business and Management and Financial Operations
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Industry Entertainment and Sports and Media
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Industry Life and Physical and Social Science
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Industry Mathematical
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Industry Medicine and Healthcare
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Industry Software and It Services
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Industry Writing and Literature and Language
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Japanese
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Korean
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Longer Query
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Multi Turn
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Non English
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Polish
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Russian
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Spanish
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Style-Controlled
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Vision
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Vision - Chinese
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Vision - Diagram
Source: LMArena data · Nov 2026
Vision - English
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
Vision - Homework
Source: LMArena data · Nov 2026
Vision Style-Controlled
Source: LMArena data · Nov 2026
Setting not stated
Setting not stated
WildBench26 related benchmarks · 26 results
WB_Elo
Source: WildBench direct data
allenai/WildBench, WB_Elo
gpt4t_reward.Creative Tasks.K=-1
Source: WildBench direct data
allenai/WildBench, gpt4t_reward.Creative Tasks.K: -1, gpt-4-turbo-2024-04-09
gpt4t_reward.Information/Advice seeking.K=-1
Source: WildBench direct data
allenai/WildBench, gpt4t_reward.Information/Advice seeking.K=-1, gpt-4-turbo-2024-04-09
gpt4t_reward.K=-1
Source: WildBench direct data
allenai/WildBench, gpt4t_reward.K: -1, gpt-4-turbo-2024-04-09
gpt4t_reward.Planning & Reasoning.K=-1
Source: WildBench direct data
allenai/WildBench, gpt4t_reward.Planning & Reasoning.K=-1, gpt-4-turbo-2024-04-09
gpt4t_reward.task_macro.K=-1
Source: WildBench direct data
allenai/WildBench, gpt4t_reward.task_macro.K: -1, gpt-4-turbo-2024-04-09
Show 20 more resultsHide 20 results
haiku_reward.Creative Tasks.K=-1
Source: WildBench direct data
allenai/WildBench, haiku_reward.Creative Tasks.K: -1, claude-3-haiku-20240307
haiku_reward.Information/Advice seeking.K=-1
Source: WildBench direct data
allenai/WildBench, haiku_reward.Information/Advice seeking.K=-1, claude-3-haiku-20240307
haiku_reward.K=-1
Source: WildBench direct data
allenai/WildBench, haiku_reward.K: -1, claude-3-haiku-20240307
haiku_reward.Planning & Reasoning.K=-1
Source: WildBench direct data
allenai/WildBench, haiku_reward.Planning & Reasoning.K=-1, claude-3-haiku-20240307
haiku_reward.task_macro.K=-1
Source: WildBench direct data
allenai/WildBench, haiku_reward.task_macro.K: -1, claude-3-haiku-20240307
llama_reward.Creative Tasks.K=-1
Source: WildBench direct data
allenai/WildBench, llama_reward.Creative Tasks.K: -1, Llama-2-70b-chat-hf
llama_reward.Information/Advice seeking.K=-1
Source: WildBench direct data
allenai/WildBench, llama_reward.Information/Advice seeking.K=-1, Llama-2-70b-chat-hf
llama_reward.K=-1
Source: WildBench direct data
allenai/WildBench, llama_reward.K: -1, Llama-2-70b-chat-hf
llama_reward.Planning & Reasoning.K=-1
Source: WildBench direct data
allenai/WildBench, llama_reward.Planning & Reasoning.K=-1, Llama-2-70b-chat-hf
llama_reward.task_macro.K=-1
Source: WildBench direct data
allenai/WildBench, llama_reward.task_macro.K: -1, Llama-2-70b-chat-hf
mixture_of_rewards.Creative Tasks.K=-1
Source: WildBench direct data
allenai/WildBench, mixture_of_rewards.Creative Tasks.K: -1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf
mixture_of_rewards.Information/Advice seeking.K=-1
Source: WildBench direct data
allenai/WildBench, mixture_of_rewards.Information/Advice seeking.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf
mixture_of_rewards.K=-1
Source: WildBench direct data
allenai/WildBench, mixture_of_rewards.K: -1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf
mixture_of_rewards.Planning & Reasoning.K=-1
Source: WildBench direct data
allenai/WildBench, mixture_of_rewards.Planning & Reasoning.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf
task_macro_reward.K=-1
Source: WildBench direct data
allenai/WildBench, task_macro_reward.K: -1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf
WB_score
Source: WildBench direct data
allenai/WildBench, WB_score
WB_score.Creative Tasks
Source: WildBench direct data
allenai/WildBench, WB_score.Creative Tasks
WB_score.Information/Advice seeking
Source: WildBench direct data
allenai/WildBench, WB_score.Information/Advice seeking
WB_score.Planning & Reasoning
Source: WildBench direct data
allenai/WildBench, WB_score.Planning & Reasoning
WB_score.task_macro
Source: WildBench direct data
allenai/WildBench, WB_score.task_macro

Math

GSM8KSource: LLM Stats data · Reported by the model's developer. The source relaying it did not verify it. claim
MATHSource: LLM Stats data · Reported by the model's developer. The source relaying it did not verify it. claim
MathVistaSource: LLM Stats data · Reported by the model's developer. The source relaying it did not verify it. claim
MGSMSource: LLM Stats data · Reported by the model's developer. The source relaying it did not verify it. claim
mu-MATH7 related benchmarks · 35 results
F1
Source: U-MATH and mu-MATH
F1 · Qwen/Qwen2.5-72B-Instruct
accuracy
Source: U-MATH and mu-MATH
accuracy · Qwen/Qwen2.5-72B-Instruct
MCC
Source: U-MATH and mu-MATH
MCC · Qwen/Qwen2.5-72B-Instruct
NPV
Source: U-MATH and mu-MATH
NPV · Qwen/Qwen2.5-72B-Instruct
PPV
Source: U-MATH and mu-MATH
PPV · Qwen/Qwen2.5-72B-Instruct
TNR
Source: U-MATH and mu-MATH
TNR · Qwen/Qwen2.5-72B-Instruct
Show 5 more resultsHide 5 results
TPR
Source: U-MATH and mu-MATH
TPR · Qwen/Qwen2.5-72B-Instruct
Omni-MATH12 results
Source: Omni-MATH · Jun 2024
Setting not stated
Applied Mathematics
Show 6 more resultsHide 6 results
Discrete Mathematics
U-MATH7 related benchmarks · 21 results
algebra accuracy
Source: U-MATH and mu-MATH
Task set · algebra · gpt-4o-2024-08-06
differential_calc accuracy
Source: U-MATH and mu-MATH
Task set · differential_calc · gpt-4o-2024-08-06
integral_calc accuracy
Source: U-MATH and mu-MATH
Task set · integral_calc · gpt-4o-2024-08-06
multivariable_calculus accuracy
Source: U-MATH and mu-MATH
Task set · multivariable_calculus · gpt-4o-2024-08-06
overall accuracy
Source: U-MATH and mu-MATH
Task set · u_math · gpt-4o-2024-08-06
precalculus_review accuracy
Source: U-MATH and mu-MATH
Task set · precalculus_review · gpt-4o-2024-08-06
Show 3 more resultsHide 3 results
sequences_series accuracy
Source: U-MATH and mu-MATH
Task set · sequences_series · gpt-4o-2024-08-06
ValsSource: Vals AI
WildBench5 related benchmarks · 5 results
WB_score.Math & Data Analysis
Source: WildBench direct data
allenai/WildBench, WB_score.Math & Data Analysis
gpt4t_reward.Math & Data Analysis.K=-1
Source: WildBench direct data
allenai/WildBench, gpt4t_reward.Math & Data Analysis.K=-1, gpt-4-turbo-2024-04-09
haiku_reward.Math & Data Analysis.K=-1
Source: WildBench direct data
allenai/WildBench, haiku_reward.Math & Data Analysis.K=-1, claude-3-haiku-20240307
llama_reward.Math & Data Analysis.K=-1
Source: WildBench direct data
allenai/WildBench, llama_reward.Math & Data Analysis.K=-1, Llama-2-70b-chat-hf
mixture_of_rewards.Math & Data Analysis.K=-1
Source: WildBench direct data
allenai/WildBench, mixture_of_rewards.Math & Data Analysis.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf

Coding

CodeElo28 related benchmarks · 28 results
LiveCodeBench8 related benchmarks · 8 results
AtCoder
Source: LiveCodeBench Leaderboard data
Codeforces
Source: LiveCodeBench Leaderboard data
platform:codeforces
Easy
Source: LiveCodeBench Leaderboard data
Hard
Source: LiveCodeBench Leaderboard data
LeetCode
Source: LiveCodeBench Leaderboard data
Show 2 more resultsHide 2 results
Medium
Source: LiveCodeBench Leaderboard data
Pro
Source: LiveCodeBench Pro data · Sep 2025
WildBench5 related benchmarks · 5 results
WB_score.Coding & Debugging
Source: WildBench direct data
allenai/WildBench, WB_score.Coding & Debugging
gpt4t_reward.Coding & Debugging.K=-1
Source: WildBench direct data
allenai/WildBench, gpt4t_reward.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09
haiku_reward.Coding & Debugging.K=-1
Source: WildBench direct data
allenai/WildBench, haiku_reward.Coding & Debugging.K=-1, claude-3-haiku-20240307
llama_reward.Coding & Debugging.K=-1
Source: WildBench direct data
allenai/WildBench, llama_reward.Coding & Debugging.K=-1, Llama-2-70b-chat-hf
mixture_of_rewards.Coding & Debugging.K=-1
Source: WildBench direct data
allenai/WildBench, mixture_of_rewards.Coding & Debugging.K=-1, gpt-4-turbo-2024-04-09, claude-3-haiku-20240307, Llama-2-70b-chat-hf

Agentic

AgentDojo3 related benchmarks · 6 results
utility
Source: AgentDojo results data docs
provider: anthropic, defense: None, attack: important_instructions
provider: anthropic, defense: None, attack: important_instructions
Targeted ASR
Source: AgentDojo results data docs
provider: anthropic, defense: None, attack: important_instructions
provider: anthropic, defense: None, attack: important_instructions
utility under attack
Source: AgentDojo results data docs
provider: anthropic, defense: None, attack: important_instructions
provider: anthropic, defense: None, attack: important_instructions
BrowserGymSource: BrowserGym Leaderboard
GAIA4 results
Source: GAIA Leaderboard data · Feb 2025
Difficulty level
Difficulty not reported

Science

GPQASource: LLM Stats data · Reported by the model's developer. The source relaying it did not verify it. claim

Vision

AI2DSource: LLM Stats data · Reported by the model's developer. The source relaying it did not verify it. claim
ChartQASource: LLM Stats data · Reported by the model's developer. The source relaying it did not verify it. claim
DocVQASource: LLM Stats data · Reported by the model's developer. The source relaying it did not verify it. claim
MMMUSource: LLM Stats data · Reported by the model's developer. The source relaying it did not verify it. claim

Reasoning

DROPSource: LLM Stats data · Reported by the model's developer. The source relaying it did not verify it. claim

Multilingual

General

Factuality

Instruction following

Speech

OCR

Other

MedHELM33 related benchmarks · 33 results
Nejumi 428 related benchmarks · 28 results
ALT_バイアス
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
ALT_倫理・道徳
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
ALT_制御性
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
ALT_堅牢性
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
ALT_毒性
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
ALT_真実性
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
Show 22 more resultsHide 22 results
AVG_jaster_0shot
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
AVG_jaster_2shots
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
GLP_アプリケーション開発
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
GLP_コーディング
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
GLP_一般的知識
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
GLP_基礎的言語性能
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
GLP_専門的知識
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
GLP_応用的言語性能
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
GLP_情報検索
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
GLP_意味解析
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
GLP_抽象的推論
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
GLP_推論能力
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
GLP_数学的推論
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
GLP_構文解析
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
GLP_知識・質問応答
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
GLP_翻訳
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
GLP_表現
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
GLP_論理的推論
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
GLP_関数呼び出し
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
TOTAL_SCORE
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
アラインメント(ALT)_AVG
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
汎用的言語性能(GLP)_AVG
Source: Nejumi Leaderboard 4 data docs
anthropic/claude-3.5-sonnet-20241022
ValsSource: Vals AI

Data credits: ACEBench, MIT (licence) · AgentDojo. Copyright (c) 2024 Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. MIT License. Not a leaderboard: attack and defense coverage differs between models. (licence) · Data by LLM Stats · Aider LLM Leaderboards (Aider-AI), Apache-2.0 (licence) · Epoch AI Benchmarking Hub, CC BY 4.0 (licence) · LMArena, CC BY 4.0 (licence) · Arena-Hard-Auto, LMArena, Apache-2.0 (licence) · ArtifactsBench (Tencent Hunyuan), CC BY 4.0 (licence) · FutureHouse and ScienceMachine, BixBench v1.5 public zero-shot baseline file. Apache-2.0 repository. (licence) · BrowserGym Leaderboard (ServiceNow), MIT (licence) · BullshitBench by Peter Gostev, MIT (licence) · EQ-Bench (Samuel J. Paech) — cite paech2023eqbench · CharXiv (Wang et al., NeurIPS 2024), Princeton NLP. https://charxiv.github.io · ChemBench leaderboard, Jablonka group / lamalab-org. Public verified numeric result summaries; results licence not separately stated. · CodeElo (Qwen). Aggregate scores from the official results file; no results licence asserted. The evaluation scaffold agreement is not presented as a licence for these scores · CyBench — cite Zhang et al. 2025 · DABstep (Adyen), CC BY 4.0 (licence) · EvalPlus Leaderboard and EvalPerf (evalplus.github.io), Apache-2.0; EvalPerf win-rate fractions converted to percentages (licence) · Stanford HELM (Stanford CRFM); scenario scores from Capabilities, Safety, Long Context, MedHELM, AIR-Bench, MMLU and Finance · Stanford HELM (Stanford CRFM), Apache-2.0 (licence) · JudgeBench, Sijun Tan et al., ScalerLab · LLM Chess by Maxim Saplin, Apache-2.0 (Copyright [2024] [Maxim Saplin]) (licence) · MathArena (ETH SRI / INSAIT), MIT — cite dekoninck2026matharena (licence) · MLE-bench (OpenAI), MIT (licence) · Toloka, U-MATH and mu-MATH public leaderboard. MIT Space; generated solution outputs retain underlying model terms and are not ingested. (licence) · Nejumi Leaderboard 4, Weights & Biases Japan; public publisher-selected numeric evaluation results · Online-Mind2Web Leaderboard (OSU NLP), Apache-2.0 (licence) · pfgen-bench, Preferred Networks. Apache-2.0 except LLM-generated outputs; aggregate scores only. (licence) · Phare (Giskard), CC BY 4.0 (licence) · Physical AI Bench (SHI Labs), MIT (licence) · SeaExam and SeaBench leaderboard, SeaLLMs (DAMO-NLP-SG). Published leaderboard scores; results licence not stated. · WildBench, Allen Institute for AI · WildVision Arena, MIT (licence)

Around the web

Also known as: anthropic.claude-3-5-sonnet-20240620-v1:0, anthropic.claude-3-5-sonnet-20241022-v2:0, anthropic/claude-3-5-sonnet, anthropic/claude-3.5-sonnet, anthropic/claude-3.5-sonnet-20240620, anthropic/claude-sonnet-3.5, bedrock/invoke/anthropic.claude-3-5-sonnet-20240620-v1:0, bedrock/us-gov-east-1/anthropic.claude-3-5-sonnet-20240620-v1:0, bedrock/us-gov-west-1/anthropic.claude-3-5-sonnet-20240620-v1:0, claude-3-5-sonnet, claude-3-5-sonnet-20240620, claude-3-5-sonnet-20241022, claude-3-5-sonnet-latest, claude-3.5-sonnet, claude-3.5-sonnet-20240620, claude-3.5-sonnet-20241022, eu.anthropic.claude-3-5-sonnet-20240620-v1:0, eu.anthropic.claude-3-5-sonnet-20241022-v2:0, gradient_ai/anthropic-claude-3.5-sonnet, heroku/claude-3-5-sonnet-latest, regional.anthropic.claude-3-5-sonnet-20240620-v1:0, regional.anthropic.claude-3-5-sonnet-20241022-v2:0, replicate/anthropic/claude-3.5-sonnet, snowflake/claude-3-5-sonnet, us.anthropic.claude-3-5-sonnet-20240620-v1:0, us.anthropic.claude-3-5-sonnet-20241022-v2:0, vercel_ai_gateway/anthropic/claude-3-5-sonnet, vercel_ai_gateway/anthropic/claude-3-5-sonnet-20241022, vercel_ai_gateway/anthropic/claude-3.5-sonnet, vertex_ai/claude-3-5-sonnet, vertex_ai/claude-3-5-sonnet@20240620

Like this one. Tap or hover any dotted value, label or icon to see where it comes from and how it is measured. open details when tapped or hovered. How this data works