TasksSotADatasetsPapersMethodsSubmitAbout
Papers With Code 2

A community resource for machine learning research: papers, code, benchmarks, and state-of-the-art results.

Explore

Notable BenchmarksAll SotADatasetsPapersMethods

Community

Submit ResultsAbout

Data sourced from the PWC Archive (CC-BY-SA 4.0). Built by the community, for the community.

Models/VAST

VAST

Reported on 84 benchmarks across 13 tasks · 1 paper · 53 SOTA

Note: results are matched by exact model name. Different papers may use the same name for different model variants.

Computer Vision45 results

  • VideoonVATEX
    text-to-video R@1· uses extra data· 2023-05-29
    83
    best: 87.7 (GRAM)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • VideoonVATEX
    text-to-video R@10· uses extra data· 2023-05-29
    99.2
    best: 100 (GRAM)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • VideoonVATEX
    text-to-video R@5· uses extra data· 2023-05-29
    98.2
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • VideoonActivityNet
    text-to-video R@1· uses extra data· 2023-05-29
    70.5
    best: 74.1 (InternVideo2-6B)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • VideoonActivityNet
    text-to-video R@10· uses extra data· 2023-05-29
    95.5
    best: 96.1 (GRAM)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • VideoonActivityNet
    text-to-video R@5· uses extra data· 2023-05-29
    90.9
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • VideoonYouCook2
    text-to-video R@1· uses extra data· 2023-05-29
    50.4
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • VideoonYouCook2
    text-to-video R@10· uses extra data· 2023-05-29
    80.8
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • VideoonYouCook2
    text-to-video R@5· uses extra data· 2023-05-29
    74.3
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • VideoonDiDeMo
    text-to-video R@1· uses extra data· 2023-05-29
    72
    best: 74.2 (InternVideo2-6B)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • VideoonMSR-VTT
    text-to-video R@1· uses extra data· 2023-05-29
    63.9
    best: 64 (GRAM)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • VideoonMSR-VTT
    text-to-video R@5· uses extra data· 2023-05-29
    84.3
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video CaptioningonVATEX
    CIDEr· uses extra data· 2023-05-29
    99.5
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video CaptioningonTVC
    BLEU-4· uses extra data· 2023-05-29
    19.9
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video CaptioningonTVC
    CIDEr· uses extra data· 2023-05-29
    74.1
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video CaptioningonYouCook2
    BLEU-4· uses extra data· 2023-05-29
    18.2
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video RetrievalonVATEX
    text-to-video R@1· uses extra data· 2023-05-29
    83
    best: 87.7 (GRAM)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video RetrievalonVATEX
    text-to-video R@10· uses extra data· 2023-05-29
    99.2
    best: 100 (GRAM)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video RetrievalonVATEX
    text-to-video R@5· uses extra data· 2023-05-29
    98.2
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video RetrievalonActivityNet
    text-to-video R@1· uses extra data· 2023-05-29
    70.5
    best: 74.1 (InternVideo2-6B)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video RetrievalonActivityNet
    text-to-video R@10· uses extra data· 2023-05-29
    95.5
    best: 96.1 (GRAM)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video RetrievalonActivityNet
    text-to-video R@5· uses extra data· 2023-05-29
    90.9
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video RetrievalonYouCook2
    text-to-video R@1· uses extra data· 2023-05-29
    50.4
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video RetrievalonYouCook2
    text-to-video R@10· uses extra data· 2023-05-29
    80.8
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video RetrievalonYouCook2
    text-to-video R@5· uses extra data· 2023-05-29
    74.3
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video RetrievalonDiDeMo
    text-to-video R@1· uses extra data· 2023-05-29
    72
    best: 74.2 (InternVideo2-6B)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video RetrievalonMSR-VTT
    text-to-video R@1· uses extra data· 2023-05-29
    63.9
    best: 64 (GRAM)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video RetrievalonMSR-VTT
    text-to-video R@5· uses extra data· 2023-05-29
    84.3
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Zero-Shot Video RetrievalonMSR-VTT
    text-to-video R@1· uses extra data· 2023-05-29
    49.3
    best: 55.9 (InternVideo2-6B)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Zero-Shot Video RetrievalonDiDeMo
    text-to-video R@1· uses extra data· 2023-05-29
    55.5
    best: 57.9 (InternVideo2-6B)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Zero-Shot Video RetrievalonDiDeMo
    text-to-video R@10· uses extra data· 2023-05-29
    79.6
    best: 85.1 (InternVideo2-1B)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Zero-Shot Video RetrievalonDiDeMo
    text-to-video R@5· uses extra data· 2023-05-29
    74.3
    best: 80 (InternVideo2-6B)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Audio-visual Question AnsweringonMUSIC-AVQA
    Acc· uses extra data· 2023-05-29
    80.7
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • VideoonDiDeMo
    text-to-video R@10· uses extra data· 2023-05-29
    91.4
    best: 94.2 (vid-TLDR (UMT-L))
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • VideoonDiDeMo
    text-to-video R@5· uses extra data· 2023-05-29
    89
    best: 91.2 (vid-TLDR (UMT-L))
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • VideoonMSR-VTT
    text-to-video R@10· uses extra data· 2023-05-29
    89.6
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video CaptioningonMSR-VTT
    BLEU-4· uses extra data· 2023-05-29
    56.7
    best: 57.8 (mPLUG-2)
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video CaptioningonMSR-VTT
    CIDEr· uses extra data· 2023-05-29
    78
    best: 80 (mPLUG-2)
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video CaptioningonVATEX
    BLEU-4· uses extra data· 2023-05-29
    45
    best: 45.6 (VALOR)
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video CaptioningonYouCook2
    CIDEr· uses extra data· 2023-05-29
    1.99
    best: 116.4 (HowToCaption)
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video RetrievalonDiDeMo
    text-to-video R@10· uses extra data· 2023-05-29
    91.4
    best: 94.2 (vid-TLDR (UMT-L))
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video RetrievalonDiDeMo
    text-to-video R@5· uses extra data· 2023-05-29
    89
    best: 91.2 (vid-TLDR (UMT-L))
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video RetrievalonMSR-VTT
    text-to-video R@10· uses extra data· 2023-05-29
    89.6
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Zero-Shot Video RetrievalonMSR-VTT
    text-to-video R@10· uses extra data· 2023-05-29
    73.9
    best: 85.1 (InternVideo2-6B)
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Zero-Shot Video RetrievalonMSR-VTT
    text-to-video R@5· uses extra data· 2023-05-29
    68.3
    best: 78.3 (InternVideo2-6B)
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500

Audio14 results

  • Audio captioningonClotho
    BLEU-4· uses extra data· 2023-05-29
    19
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Audio captioningonClotho
    CIDEr· uses extra data· 2023-05-29
    0.519
    best: 14 (ZerAuCap)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Audio captioningonClotho
    METEOR· uses extra data· 2023-05-29
    19.3
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Audio captioningonClotho
    ROUGE-L· uses extra data· 2023-05-29
    40.8
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Audio captioningonAudioCaps
    BLEU-4· uses extra data· 2023-05-29
    0.295
    best: 14.3 (Audio Flamingo)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Audio captioningonAudioCaps
    CIDEr· uses extra data· 2023-05-29
    0.781
    best: 50.2 (Audio Flamingo)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Audio captioningonAudioCaps
    METEOR· uses extra data· 2023-05-29
    0.247
    best: 20.5 (Audio Flamingo)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Audio captioningonAudioCaps
    ROUGE-L· uses extra data· 2023-05-29
    0.509
    best: 40.8 (Audio Flamingo)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Text to Audio RetrievalonAudioCaps
    R@1· uses extra data· 2023-05-29
    52
    best: 55.2 (InternVideo2-6B)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Text to Audio RetrievalonClotho
    R@1· uses extra data· 2023-05-29
    26.9
    best: 27.69 (PaSST-RoBERTa & Estimated Audio–Caption Correspondences)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Text to Audio RetrievalonClotho
    R@10· uses extra data· 2023-05-29
    66.1
    best: 70.39 (PaSST-RoBERTa & Estimated Audio–Caption Correspondences)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Text to Audio RetrievalonClotho
    R@5· uses extra data· 2023-05-29
    53.2
    best: 57.03 (PaSST-RoBERTa & Estimated Audio–Caption Correspondences)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Text to Audio RetrievalonAudioCaps
    R@10· uses extra data· 2023-05-29
    82.9
    best: 88.4 (ONE-PEACE)
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Text to Audio RetrievalonAudioCaps
    R@5· uses extra data· 2023-05-29
    76.8
    best: 77.5 (ONE-PEACE)
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500

Miscellaneous13 results

  • Image Retrieval with Multi-Modal QueryonCOCO 2014
    Text-to-image R@1· uses extra data· 2023-05-29
    68
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Image Retrieval with Multi-Modal QueryonCOCO 2014
    Text-to-image R@10· uses extra data· 2023-05-29
    92.8
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Cross-Modal Information RetrievalonCOCO 2014
    Text-to-image R@1· uses extra data· 2023-05-29
    68
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Cross-Modal Information RetrievalonCOCO 2014
    Text-to-image R@10· uses extra data· 2023-05-29
    92.8
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Image Retrieval with Multi-Modal QueryonFlickr30k
    Text-to-image R@1· uses extra data· 2023-05-29
    91
    best: 93.3 (ERNIE-ViL 2.0)
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Image Retrieval with Multi-Modal QueryonFlickr30k
    Text-to-image R@10· uses extra data· 2023-05-29
    99.5
    best: 99.8 (ERNIE-ViL 2.0)
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Image Retrieval with Multi-Modal QueryonFlickr30k
    Text-to-image R@5· uses extra data· 2023-05-29
    98.5
    best: 99.5 (M2-Encoder)
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Image Retrieval with Multi-Modal QueryonCOCO 2014
    Text-to-image R@5· uses extra data· 2023-05-29
    87.7
    best: 92.8 (BEiT-3)
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Image Retrieval with Multi-Modal QueryonFlickr30k
    Text-to-image R@1· uses extra data· 2023-05-29
    90.4
    best: 93.3 (ERNIE-ViL 2.0)
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Cross-Modal Information RetrievalonFlickr30k
    Text-to-image R@1· uses extra data· 2023-05-29
    91
    best: 93.3 (ERNIE-ViL 2.0)
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Cross-Modal Information RetrievalonFlickr30k
    Text-to-image R@10· uses extra data· 2023-05-29
    99.5
    best: 99.8 (ERNIE-ViL 2.0)
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Cross-Modal Information RetrievalonFlickr30k
    Text-to-image R@5· uses extra data· 2023-05-29
    98.5
    best: 99.4 (ERNIE-ViL 2.0)
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Cross-Modal Information RetrievalonCOCO 2014
    Text-to-image R@5· uses extra data· 2023-05-29
    87.7
    best: 92.8 (BEiT-3)
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500

Natural Language Processing11 results

  • Image CaptioningonCOCO Captions
    SPICE· uses extra data· 2023-05-29
    27
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Cross-Modal RetrievalonCOCO 2014
    Text-to-image R@1· uses extra data· 2023-05-29
    68
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Cross-Modal RetrievalonCOCO 2014
    Text-to-image R@10· uses extra data· 2023-05-29
    92.8
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Visual Question Answering (VQA)onMSVD-QA
    Accuracy· uses extra data· 2023-05-29
    0.6
    best: 0.61 (VLAB)
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Image CaptioningonCOCO Captions
    CIDER· uses extra data· 2023-05-29
    149
    best: 155.1 (mPLUG)
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Cross-Modal RetrievalonFlickr30k
    Text-to-image R@1· uses extra data· 2023-05-29
    91
    best: 93.3 (ERNIE-ViL 2.0)
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Cross-Modal RetrievalonFlickr30k
    Text-to-image R@10· uses extra data· 2023-05-29
    99.5
    best: 99.8 (ERNIE-ViL 2.0)
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Cross-Modal RetrievalonFlickr30k
    Text-to-image R@5· uses extra data· 2023-05-29
    98.5
    best: 99.4 (ERNIE-ViL 2.0)
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Cross-Modal RetrievalonCOCO 2014
    Text-to-image R@5· uses extra data· 2023-05-29
    87.7
    best: 92.8 (BEiT-3)
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Visual Question Answering (VQA)onVQA v2 test-dev
    Accuracy· uses extra data
    80.23
    best: 84.3 (PaLI)
  • Visual Question Answering (VQA)onVQA v2 test-std
    overall· uses extra data
    80.19
    best: 84.03 (BEiT-3)

Reasoning2 results

  • Video Question AnsweringonMSRVTT-QA
    Accuracy· uses extra data· 2023-05-29
    50.1
    best: 72.4 (Flash-VStream)
    SOTA
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500
  • Video Question AnsweringonActivityNet-QA
    Accuracy· uses extra data· 2023-05-29
    50.4
    best: 61.6 (Tarsier (34B))
    VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetarXiv:2305.18500