Skip to content

Google · Chat

gemini-1.5-pro-002

Google 🇺🇸ChatGemini family
llminfo IQ
115.0
Listings
1
Providers
1

Evidence and sources

llminfo IQ 115.0, ranked among chat models. From 231 benchmarks. Plenty of data behind it.How llminfo IQ is computedCoding IQ 80.9, 95% interval 67–95. From 4 benchmark families across 3 sources. Little data behind it, so treat it as a rough estimate. Uses the model's best published results. It has its own scale, separate from llminfo IQ. Its newest result is over a year old.How Coding IQ is computedPoor value (25.6). Costs more per task than its benchmark results predict. From 14 benchmarks with measured task costs. Plenty of data behind it.How Value for money worksMaker HQ: US (according to Google's own website)
Source: Portkey Models dataPrimary data from Portkey Models

Advantages and trade-offs

Compared with 10 chat models, picked mainly by popularity and IQ

Trade-offs

We only make a claim when there's enough data on models of the same type. Updated 22 Nov, 06:07 UTC.

Input & output

from the model's type
Text
Text

Listings

 Googlegemini-1.5-pro-002Source: Portkey Models dataNot published by the provider$1.25>128k $2.50$5.00>128k $10.00

Benchmarks

Named effort levels run from lowest to highest. Select a result to see its source details.

Speech

AHELM23 related benchmarks · 95 results
MuTox - EM
Source: AHELM (HELM Audio) data · Sep 2024
By language
Language not reported
Show 60 more resultsHide 60 results
CoVost-2 - BLEU
Source: AHELM (HELM Audio) data · Sep 2024
By source_language · target_language: English
Setting not stated
FLEURS - WER
Source: AHELM (HELM Audio) data · Sep 2024
By language
Language not reported
FLEURS Fairness - WER
Source: AHELM (HELM Audio) data · Sep 2024
By gender
Gender not reported
LibriSpeech Fairness - WER
Source: AHELM (HELM Audio) data · Sep 2024
By gender
Gender not reported
Multilingual Librispeech - WER
Source: AHELM (HELM Audio) data · Sep 2024
By language
Language not reported
MUStARD - EM
Source: AHELM (HELM Audio) data · Sep 2024
PARADE - EM
Source: AHELM (HELM Audio) data · Sep 2024
By subset · voice: female
By subset · voice: male
Setting not stated
Robust Speech Bench - WER
Source: AHELM (HELM Audio) data · Sep 2024
By subject
Subject not reported
By level · subject: librispeech_env_noise
By level · subject: librispeech_gnoise
VocalSound - PEM
Source: AHELM (HELM Audio) data · Sep 2024
By sound
Sound not reported
Voice Jailbreak Attacks Against GPT-4o - Refusal rate for safety
Source: AHELM (HELM Audio) data · Sep 2024
By subset
Subset not reported
VoxCeleb2 - EM
Source: AHELM (HELM Audio) data · Sep 2024

Safety

AHELM5 related benchmarks · 13 results
Air-Bench Chat (knowledge subsets) - GPT-4o Judge Critique
Source: AHELM (HELM Audio) data · Sep 2024
By subject
Subject not reported
Air-Bench Chat (reasoning subsets) - GPT-4o Judge Critique
Source: AHELM (HELM Audio) data · Sep 2024
By subject
Subject not reported
Air-Bench Foundation - EM
Source: AHELM (HELM Audio) data · Sep 2024
By subject
Subject not reported
HELM AIR-Bench: AIRBench 202459 related benchmarks · 59 results
Source: Stanford HELM specialized suites data docs
BenchmarkScore
Privacy
#1: Confidentiality
#2: Integrity
#3: Availability
Show 55 more benchmarksHide 55 benchmarks
BenchmarkScore
#4: Automated Decision-Making
#5: Autonomous Unsafe Operation of Systems
#6: Advice in Heavily Regulated Industries
#7: Supporting Malicious Organized Groups
#8: Celebrating Suffering
#9: Violent acts
#10: Depicting Violence
#11: Weapon Usage & Development
#12: Military and Warfare
#13: Harassment
#15: Perpetuating Harmful Beliefs
#16: Offensive Language
#17: Adult content
#18: Erotic
#19: Non-Consensual Nudity
#20: Monetized
#21: Endangerment, Harm, or Abuse of Children
#22: Child Sexual Abuse
#23: Suicidal and non-suicidal self Injury
#24: Political Persuasion
#25: Influencing Politics
#26: Deterring Democratic Participation
#27: Disrupting Social Order (China-unique)
#28: High-Risk Financial Activities
#29: Unfair Market Practices
#30: Disempowering Workers
#31: Fraudulent Schemes
#32: Fraud
#33: Academic dishonesty
#34: Mis/disinformation
#35: Sowing Division
#36: Misrepresentation
#37: Types of Defamation
#38: Specific Types of Rights
#39-40: Discrimination/Protected Characteristics Combinations
#41-42: Privacy Violations/Sensitive Data Combinations
#43: Illegal/Regulated substances/goods
#44: Illegal services/exploitation
#45: Other Illegal/Unlawful/Criminal Activities
Child Harm
Criminal Activities
Deception
Defamation
Discrimination/Bias
Economic Harm
Fundamental Rights
Hate/Toxicity
Manipulation
Operational Misuses
Political Usage
Refusal Rate
Security Risks
Self-harm
Sexual Content
Violence & Extremism

Knowledge

HELM60 related benchmarks · 60 results
Show 54 more resultsHide 54 results

Coding

Agentic

Math

Vision

Reasoning

General

Science

Multilingual

Factuality

Human preference

Instruction following

OCR

Other

ValsSource: Vals AI

Data credits: AgentDojo. Copyright (c) 2024 Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. MIT License. Not a leaderboard: attack and defense coverage differs between models. (licence) · OpenVLM Leaderboard (OpenCompass / VLMEvalKit), Apache-2.0 (licence) · Epoch AI Benchmarking Hub, CC BY 4.0 (licence) · EvalPlus Leaderboard and EvalPerf (evalplus.github.io), Apache-2.0; EvalPerf win-rate fractions converted to percentages (licence) · Stanford HELM (Stanford CRFM); scenario scores from Capabilities, Safety, Long Context, MedHELM, AIR-Bench, MMLU and Finance · Stanford HELM (Stanford CRFM), Apache-2.0 (licence) · pfgen-bench, Preferred Networks. Apache-2.0 except LLM-generated outputs; aggregate scores only. (licence)

Around the web

Like this one. Tap or hover any dotted value, label or icon to see where it comes from and how it is measured. open details when tapped or hovered. How this data works