Skip to content

Google · Chat

gemini-1.5-pro-001

Google 🇺🇸ChatGemini family
llminfo IQ
103.7
Listings
1
Providers
1

Evidence and sources

llminfo IQ 103.7, ranked among chat models. From 165 benchmarks. Plenty of data behind it.How llminfo IQ is computedMaker HQ: US (according to Google's own website)
Source: Portkey Models dataPrimary data from Portkey Models

Input & output

from the model's type
Text
Text

Listings

 Googlegemini-1.5-pro-001Source: Portkey Models dataNot published by the provider$1.25>128k $2.50$5.00>128k $10.00

Benchmarks

Named effort levels run from lowest to highest. Select a result to see its source details.

Speech

AHELM23 related benchmarks · 95 results
MuTox - EM
Source: AHELM (HELM Audio) data · May 2024
By language
Language not reported
Show 60 more resultsHide 60 results
CoVost-2 - BLEU
Source: AHELM (HELM Audio) data · May 2024
By source_language · target_language: English
Setting not stated
FLEURS - WER
Source: AHELM (HELM Audio) data · May 2024
By language
Language not reported
FLEURS Fairness - WER
Source: AHELM (HELM Audio) data · May 2024
By gender
Gender not reported
LibriSpeech Fairness - WER
Source: AHELM (HELM Audio) data · May 2024
By gender
Gender not reported
Multilingual Librispeech - WER
Source: AHELM (HELM Audio) data · May 2024
By language
Language not reported
MUStARD - EM
Source: AHELM (HELM Audio) data · May 2024
PARADE - EM
Source: AHELM (HELM Audio) data · May 2024
By subset · voice: female
By subset · voice: male
Setting not stated
Robust Speech Bench - WER
Source: AHELM (HELM Audio) data · May 2024
By subject
Subject not reported
By level · subject: librispeech_env_noise
By level · subject: librispeech_gnoise
VocalSound - PEM
Source: AHELM (HELM Audio) data · May 2024
By sound
Sound not reported
Voice Jailbreak Attacks Against GPT-4o - Refusal rate for safety
Source: AHELM (HELM Audio) data · May 2024
By subset
Subset not reported
VoxCeleb2 - EM
Source: AHELM (HELM Audio) data · May 2024

Knowledge

HELM59 related benchmarks · 59 results
Show 53 more resultsHide 53 results

Safety

AHELM5 related benchmarks · 13 results
Air-Bench Chat (knowledge subsets) - GPT-4o Judge Critique
Source: AHELM (HELM Audio) data · May 2024
By subject
Subject not reported
Air-Bench Chat (reasoning subsets) - GPT-4o Judge Critique
Source: AHELM (HELM Audio) data · May 2024
By subject
Subject not reported
Air-Bench Foundation - EM
Source: AHELM (HELM Audio) data · May 2024
By subject
Subject not reported

General

Reasoning

Source: JudgeBench data docs
GPT-4o Dataset, Prompted Judge, Arena-Hard (gemini-1.5-pro-001), Reasoning Score
GPT-4o Dataset, Prompted Judge, VertexAI Evaluation (gemini-1.5-pro-001), Reasoning Score

Math

Source: JudgeBench data docs
GPT-4o Dataset, Prompted Judge, Arena-Hard (gemini-1.5-pro-001), Math Score
GPT-4o Dataset, Prompted Judge, VertexAI Evaluation (gemini-1.5-pro-001), Math Score

Science

Coding

Source: JudgeBench data docs
GPT-4o Dataset, Prompted Judge, Arena-Hard (gemini-1.5-pro-001), Coding Score
GPT-4o Dataset, Prompted Judge, VertexAI Evaluation (gemini-1.5-pro-001), Coding Score

Vision

Agentic

Multilingual

Factuality

Other

Source: JudgeBench data docs
GPT-4o Dataset, Prompted Judge, Arena-Hard (gemini-1.5-pro-001), Knowledge Score
GPT-4o Dataset, Prompted Judge, VertexAI Evaluation (gemini-1.5-pro-001), Knowledge Score
Source: JudgeBench data docs
GPT-4o Dataset, Prompted Judge, Arena-Hard (gemini-1.5-pro-001), Overall Score
GPT-4o Dataset, Prompted Judge, VertexAI Evaluation (gemini-1.5-pro-001), Overall Score
MedHELM33 related benchmarks · 33 results

Data credits: AgentDojo. Copyright (c) 2024 Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. MIT License. Not a leaderboard: attack and defense coverage differs between models. (licence) · Epoch AI Benchmarking Hub, CC BY 4.0 (licence) · Stanford HELM (Stanford CRFM), Apache-2.0 (licence) · Stanford HELM (Stanford CRFM); scenario scores from Capabilities, Safety, Long Context, MedHELM, AIR-Bench, MMLU and Finance · JudgeBench, Sijun Tan et al., ScalerLab

Around the web

  • Gemini 1.5 Flash-8B is now production ready - Google Developers Blog
    Google for Developers Community/Events Learn Blog YouTube Search Google for Developers Community/Events Learn Blog YouTube …Google for Developers Community/Events Learn Blog YouTube Search Google for Developers Community/Events Learn Blog YouTube Gemini Gemini 1.5 Flash-8B is now production ready OCT. 3, 2024 Logan Kilpat…

    developers.googleblog.com

  • Gemini 1.5: Google’s next-gen AI model is almost ready | The Verge
    part of the overall model when you send in a query, rather than processing the …part of the overall model when you send in a query, rather than processing the whole thing the whole time. ( Here’s a good explainer on the subject .) That approach should make the model both faster …

    theverge.com

  • Gemini 1.5 Flash price drop with tuning rollout complete, and more - Google Developers Blog
    Learn Blog YouTube Search Google for Developers Community/Events Learn Blog YouTube Gemini Gemini 1.5 Flash …Learn Blog YouTube Search Google for Developers Community/Events Learn Blog YouTube Gemini Gemini 1.5 Flash price drop with tuning rollout complete, and more AUG. 8, 2024 Logan Kilpatrick Group Produ…

    developers.googleblog.com

  • Gemini 1.5 Pro updates, 1.5 Flash debut and 2 new Gemma models
    Give 1.5 Pro and 1.5 Flash a try in Google AI Studio today. Try Gemini …Give 1.5 Pro and 1.5 Flash a try in Google AI Studio today. Try Gemini 1.5 Grab an API key in Google AI Studio , and get started with the Gemini API Cookbook . Last December, we released Gemini 1.0 a…

    blog.google

  • Gemini 1.5: Unlocking multimodal understanding across ...
    associated with the development of advanced models, conducted by the Responsible Development and Innovation team …associated with the development of advanced models, conducted by the Responsible Development and Innovation team and reviewed by our Google DeepMind Responsibility and Safety Council in order to upho…

    storage.googleapis.com

  • Google announces Gemini 1.5 with greatly expanded context window
    Avatar for Abner Li Abner Li · Feb 15 2024 - 7:59 am PT 13 …Avatar for Abner Li Abner Li · Feb 15 2024 - 7:59 am PT 13 Comments Following the 1.0 launch in December, Google today announced Gemini 1.5 as its next-generation model with “dramatically enhanced pe…

    9to5google.com

  • Gemini 1.5 flash retirement date - Gemini API - Google AI Developers Forum
    Google AI for Developers Gemini models · Gemini API · Google AI for Developers Learn …Google AI for Developers Gemini models · Gemini API · Google AI for Developers Learn about Google's most advanced AI models including Gemini 2.5 Pro Edit: This was prompted by the announcement that G…

    discuss.ai.google.dev

  • Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context - Drexel University
    Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of …Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5 Pro's long-context ability, we find continued improvement in next-token prediction …

    researchdiscovery.drexel.edu

  • Has Anyone Gained Access to Gemini 1.5 Pro API? (Re: Gemini 1.5 Pro API's Multimodal Features) - Gemini API - Google AI Developers Forum
    Section: > Has Anyone Gained Access to Gemini 1.5 Pro API? (Re Mar 22, 2024 …Section: > Has Anyone Gained Access to Gemini 1.5 Pro API? (Re Mar 22, 2024 — Technical Report for Gemini 1.5 Pro, it's a multimodal model, 's a single model that handles text, code, images, audio, a…

    discuss.ai.google.dev

  • Gemini 1.5: Unlocking multimodal understanding across ...
    github.com/ google/jax . Thorsten Brants, Ashok Popat, Peng Xu, Franz Och, and Jeffrey Dean. Large …github.com/ google/jax . Thorsten Brants, Ashok Popat, Peng Xu, Franz Och, and Jeffrey Dean. Large language models in machine translation. pages 858–867, 01 2007. Gemini 1.5: Unlocking multimodal und…

    kapler.cz

Like this one. Tap or hover any dotted value, label or icon to see where it comes from and how it is measured. open details when tapped or hovered. How this data works