TasksSotADatasetsPapersMethodsSubmitAbout
Papers With Code 2

A community resource for machine learning research: papers, code, benchmarks, and state-of-the-art results.

Explore

Notable BenchmarksAll SotADatasetsPapersMethods

Community

Submit ResultsAbout

Data sourced from the PWC Archive (CC-BY-SA 4.0). Built by the community, for the community.

Models/VideoChat2

VideoChat2

Reported on 59 benchmarks across 8 tasks · 2 papers · 13 SOTA

Note: results are matched by exact model name. Different papers may use the same name for different model variants.

Reasoning36 results

  • Video Question AnsweringonTVBench
    Average Accuracy· 2023-11-28
    35
    best: 63.6 (Seed1.5-VL thinking)
    SOTA
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Video Question AnsweringonMVBench
    Avg.· 2023-11-28
    51.9
    best: 69.3 (LinVT-Qwen2-VL (7B))
    SOTA
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Video Question AnsweringonSTAR Benchmark
    Accuracy· 2023-11-28
    59
    SOTA
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Generative Visual Question AnsweringonVideoInstruct
    Temporal Understanding· 2023-11-28
    2.66
    best: 3.23 (VLM-RLAIF)
    SOTA
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Video-based Generative Performance BenchmarkingonVideoInstruct
    Temporal Understanding· 2023-11-28
    2.66
    best: 3.23 (VLM-RLAIF)
    SOTA
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Video Question AnsweringonVNBench
    Accuracy· 2023-05-10
    12.4
    best: 77.88 (BIMBA-LLaVA-Qwen2-7B)
    SOTA
    VideoChat: Chat-Centric Video UnderstandingarXiv:2305.06355
  • Video Question AnsweringonActivityNet-QA
    Accuracy· 2023-11-28
    49.1
    best: 61.6 (Tarsier (34B))
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Video Question AnsweringonActivityNet-QA
    Confidence score· 2023-11-28
    3.3
    best: 2.2 (Video Chat)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Video Question AnsweringonNExT-QA
    Accuracy· 2023-11-28
    68.6
    best: 85.5 (LinVT-Qwen2-VL (7B))
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Video Question AnsweringonNExT-QA
    Accuracy· 2023-11-28
    61.7
    best: 85.5 (LinVT-Qwen2-VL (7B))
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Video Question AnsweringonMSVD-QA
    Accuracy· 2023-11-28
    70
    best: 80.3 (Tarsier (34B))
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Video Question AnsweringonMSVD-QA
    Confidence Score· 2023-11-28
    3.9
    best: 2.5 (Video LLaMA-7B)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Video Question AnsweringonMSRVTT-QA
    Accuracy· 2023-11-28
    54.1
    best: 72.4 (Flash-VStream)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Video Question AnsweringonMSRVTT-QA
    Confidence Score· 2023-11-28
    3.3
    best: 1.8 (Video LLaMA-7B)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Video Question AnsweringonActivityNet-QA
    Accuracy· 2023-11-28
    49.1
    best: 61.6 (Tarsier (34B))
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Video Question AnsweringonActivityNet-QA
    Confidence Score· 2023-11-28
    3.3
    best: 1.1 (Video LLaMA)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Generative Visual Question AnsweringonVideoInstruct
    Consistency· 2023-11-28
    2.81
    best: 3.81 (PPLLaVA-7B-dpo)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Generative Visual Question AnsweringonVideoInstruct
    Contextual Understanding· 2023-11-28
    3.51
    best: 4.21 (PPLLaVA-7B-dpo)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Generative Visual Question AnsweringonVideoInstruct
    Correctness of Information· 2023-11-28
    3.02
    best: 3.85 (PPLLaVA-7B-dpo)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Generative Visual Question AnsweringonVideoInstruct
    Detail Orientation· 2023-11-28
    2.88
    best: 3.56 (PPLLaVA-7B-dpo)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Generative Visual Question AnsweringonVideoInstruct
    mean· 2023-11-28
    2.98
    best: 3.73 (PPLLaVA-7B-dpo)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Generative Visual Question AnsweringonVideoInstruct
    gpt-score· 2023-11-28
    3.51
    best: 4.21 (PPLLaVA-7B)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Generative Visual Question AnsweringonVideoInstruct
    gpt-score· 2023-11-28
    3.02
    best: 4.21 (PPLLaVA-7B)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Generative Visual Question AnsweringonVideoInstruct
    gpt-score· 2023-11-28
    2.88
    best: 4.21 (PPLLaVA-7B)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Generative Visual Question AnsweringonVideoInstruct
    gpt-score· 2023-11-28
    2.66
    best: 4.21 (PPLLaVA-7B)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Generative Visual Question AnsweringonVideoInstruct
    gpt-score· 2023-11-28
    2.81
    best: 4.21 (PPLLaVA-7B)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Video-based Generative Performance BenchmarkingonVideoInstruct
    Consistency· 2023-11-28
    2.81
    best: 3.81 (PPLLaVA-7B-dpo)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Video-based Generative Performance BenchmarkingonVideoInstruct
    Contextual Understanding· 2023-11-28
    3.51
    best: 4.21 (PPLLaVA-7B-dpo)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Video-based Generative Performance BenchmarkingonVideoInstruct
    Correctness of Information· 2023-11-28
    3.02
    best: 3.85 (PPLLaVA-7B-dpo)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Video-based Generative Performance BenchmarkingonVideoInstruct
    Detail Orientation· 2023-11-28
    2.88
    best: 3.56 (PPLLaVA-7B-dpo)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Video-based Generative Performance BenchmarkingonVideoInstruct
    mean· 2023-11-28
    2.98
    best: 3.73 (PPLLaVA-7B-dpo)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Video-based Generative Performance BenchmarkingonVideoInstruct
    gpt-score· 2023-11-28
    3.51
    best: 4.21 (PPLLaVA-7B)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Video-based Generative Performance BenchmarkingonVideoInstruct
    gpt-score· 2023-11-28
    3.02
    best: 4.21 (PPLLaVA-7B)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Video-based Generative Performance BenchmarkingonVideoInstruct
    gpt-score· 2023-11-28
    2.88
    best: 4.21 (PPLLaVA-7B)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Video-based Generative Performance BenchmarkingonVideoInstruct
    gpt-score· 2023-11-28
    2.66
    best: 4.21 (PPLLaVA-7B)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Video-based Generative Performance BenchmarkingonVideoInstruct
    gpt-score· 2023-11-28
    2.81
    best: 4.21 (PPLLaVA-7B)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005

Natural Language Processing26 results

  • Question AnsweringonSTAR Benchmark
    Accuracy· 2023-11-28
    59
    SOTA
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Question AnsweringonActivityNet-QA
    Accuracy· 2023-11-28
    49.1
    best: 61.6 (Tarsier (34B))
    SOTA
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Visual Question Answering (VQA)onVideoInstruct
    Temporal Understanding· 2023-11-28
    2.66
    best: 3.23 (VLM-RLAIF)
    SOTA
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Question AnsweringonVNBench
    Accuracy· 2023-05-10
    12.4
    best: 77.88 (BIMBA-LLaVA-Qwen2-7B)
    SOTA
    VideoChat: Chat-Centric Video UnderstandingarXiv:2305.06355
  • Question AnsweringonNExT-QA
    Accuracy· 2023-11-28
    61.7
    best: 79.6 (VideoMultiAgent (GPT-4o))
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Question AnsweringonMSVD-QA
    Accuracy· 2023-11-28
    70
    best: 80.3 (Tarsier (34B))
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Question AnsweringonMSVD-QA
    Confidence Score· 2023-11-28
    3.9
    best: 2.5 (Video LLaMA-7B)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Question AnsweringonMSRVTT-QA
    Accuracy· 2023-11-28
    54.1
    best: 72.4 (Flash-VStream)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Question AnsweringonMSRVTT-QA
    Confidence Score· 2023-11-28
    3.3
    best: 1.8 (Video LLaMA-7B)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Question AnsweringonActivityNet-QA
    Confidence Score· 2023-11-28
    3.3
    best: 1.1 (Video LLaMA)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Visual Question Answering (VQA)onSQA3D
    Exact Match· 2023-11-28
    37.3
    best: 60.1 (LLaVA-3D)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Visual Question Answering (VQA)onScanQA Test w/ objects
    BLEU-4· 2023-11-28
    9.6
    best: 24.06 (BridgeQA)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Visual Question Answering (VQA)onScanQA Test w/ objects
    CIDEr· 2023-11-28
    49.2
    best: 103.1 (LLaVA-3D)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Visual Question Answering (VQA)onScanQA Test w/ objects
    Exact Match· 2023-11-28
    19.2
    best: 31.29 (BridgeQA)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Visual Question Answering (VQA)onScanQA Test w/ objects
    METEOR· 2023-11-28
    9.5
    best: 20.8 (LLaVA-3D)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Visual Question Answering (VQA)onScanQA Test w/ objects
    ROUGE· 2023-11-28
    28.2
    best: 49.6 (LLaVA-3D)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Visual Question Answering (VQA)onVideoInstruct
    Consistency· 2023-11-28
    2.81
    best: 3.81 (PPLLaVA-7B-dpo)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Visual Question Answering (VQA)onVideoInstruct
    Contextual Understanding· 2023-11-28
    3.51
    best: 4.21 (PPLLaVA-7B-dpo)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Visual Question Answering (VQA)onVideoInstruct
    Correctness of Information· 2023-11-28
    3.02
    best: 3.85 (PPLLaVA-7B-dpo)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Visual Question Answering (VQA)onVideoInstruct
    Detail Orientation· 2023-11-28
    2.88
    best: 3.56 (PPLLaVA-7B-dpo)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Visual Question Answering (VQA)onVideoInstruct
    mean· 2023-11-28
    2.98
    best: 3.73 (PPLLaVA-7B-dpo)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Visual Question Answering (VQA)onVideoInstruct
    gpt-score· 2023-11-28
    3.51
    best: 4.21 (PPLLaVA-7B)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Visual Question Answering (VQA)onVideoInstruct
    gpt-score· 2023-11-28
    3.02
    best: 4.21 (PPLLaVA-7B)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Visual Question Answering (VQA)onVideoInstruct
    gpt-score· 2023-11-28
    2.88
    best: 4.21 (PPLLaVA-7B)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Visual Question Answering (VQA)onVideoInstruct
    gpt-score· 2023-11-28
    2.66
    best: 4.21 (PPLLaVA-7B)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • Visual Question Answering (VQA)onVideoInstruct
    gpt-score· 2023-11-28
    2.81
    best: 4.21 (PPLLaVA-7B)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005

Computer Vision9 results

  • VCGBench-DiverseonVideoInstruct
    Spatial Understanding· 2023-11-28
    2.43
    best: 2.8 (VideoGPT+)
    SOTA
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • VCGBench-DiverseonVideoInstruct
    Temporal Understanding· 2023-11-28
    1.66
    best: 1.78 (VideoGPT+)
    SOTA
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • VCGBench-DiverseonVideoInstruct
    Consistency· 2023-11-28
    2.27
    best: 2.59 (VideoGPT+)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • VCGBench-DiverseonVideoInstruct
    Contextual Understanding· 2023-11-28
    2.51
    best: 2.81 (VideoGPT+)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • VCGBench-DiverseonVideoInstruct
    Correctness of Information· 2023-11-28
    2.13
    best: 2.46 (VideoGPT+)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • VCGBench-DiverseonVideoInstruct
    Dense Captioning· 2023-11-28
    1.26
    best: 1.38 (VideoGPT+)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • VCGBench-DiverseonVideoInstruct
    Detail Orientation· 2023-11-28
    2.42
    best: 2.73 (VideoGPT+)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • VCGBench-DiverseonVideoInstruct
    Reasoning· 2023-11-28
    3.13
    best: 3.63 (VideoGPT+)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005
  • VCGBench-DiverseonVideoInstruct
    mean· 2023-11-28
    2.2
    best: 2.47 (VideoGPT+)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005

Methodology1 result

  • Zero-Shot LearningonTVQA
    Accuracy· 2023-11-28
    40.6
    SOTA
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005

Other1 result

  • Video-based Generative Performance Benchmarking (Correctness of Information)onVideoInstruct
    gpt-score· 2023-11-28
    3.02
    best: 3.85 (PPLLaVA-7B)
    MVBench: A Comprehensive Multi-modal Video Understanding BenchmarkarXiv:2311.17005