Gemini

Reported on 5 benchmarks across 3 tasks · 2 papers · 2 SOTA

Note: results are matched by exact model name. Different papers may use the same name for different model variants.

Reasoning4 results

Video Question AnsweringonVNBench
Accuracy· 2024-03-08
66.7
best: 77.88 (BIMBA-LLaVA-Qwen2-7B)
SOTA
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context arXiv:2403.05530
Visual ReasoningonWinoground
Group Score· 2024-01-05
25
best: 58.75 (GPT-4V (CoT, pick b/w two options))
CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs arXiv:2401.02582
Visual ReasoningonWinoground
Image Score· 2024-01-05
26
best: 68.75 (GPT-4V (CoT, pick b/w two options))
CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs arXiv:2401.02582
Visual ReasoningonWinoground
Text Score· 2024-01-05
30.75
best: 75.5 (GPT-4o + CA)
CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs arXiv:2401.02582

Natural Language Processing1 result

Question AnsweringonVNBench
Accuracy· 2024-03-08
66.7
best: 77.88 (BIMBA-LLaVA-Qwen2-7B)
SOTA
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context arXiv:2403.05530