TasksSotADatasetsPapersMethodsSubmitAbout
Papers With Code 2

A community resource for machine learning research: papers, code, benchmarks, and state-of-the-art results.

Explore

Notable BenchmarksAll SotADatasetsPapersMethods

Community

Submit ResultsAbout

Data sourced from the PWC Archive (CC-BY-SA 4.0). Built by the community, for the community.

Models/mPLUG-2

mPLUG-2

Reported on 49 benchmarks across 8 tasks · 1 paper · 21 SOTA

Note: results are matched by exact model name. Different papers may use the same name for different model variants.

Computer Vision44 results

  • Video CaptioningonMSR-VTT
    BLEU-4· 2023-02-01
    57.8
    SOTA
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Video CaptioningonMSR-VTT
    CIDEr· 2023-02-01
    80
    SOTA
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Video CaptioningonMSR-VTT
    ROUGE-L· 2023-02-01
    70.1
    SOTA
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Video CaptioningonMSVD
    CIDEr· 2023-02-01
    165.8
    best: 195.6 (MaMMUT)
    SOTA
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Video CaptioningonMSVD
    METEOR· 2023-02-01
    48.4
    best: 51.2 (VLAB)
    SOTA
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Video CaptioningonMSVD
    ROUGE-L· 2023-02-01
    85.3
    best: 87.9 (VLAB)
    SOTA
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Visual GroundingonRefCOCO+ test B
    Accuracy (%)· 2023-02-01
    86.05
    best: 92 (Florence-2-large-ft)
    SOTA
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Visual GroundingonRefCOCO+ val
    Accuracy (%)· 2023-02-01
    90.33
    best: 93.4 (Florence-2-large-ft)
    SOTA
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Visual GroundingonRefCOCO+ testA
    Accuracy (%)· 2023-02-01
    92.8
    best: 95.3 (Florence-2-large-ft)
    SOTA
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Zero-Shot Video RetrievalonMSR-VTT
    text-to-video R@1· 2023-02-01
    47.1
    best: 55.9 (InternVideo2-6B)
    SOTA
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Zero-Shot Video RetrievalonMSR-VTT
    text-to-video R@10· 2023-02-01
    79
    best: 85.1 (InternVideo2-6B)
    SOTA
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Zero-Shot Video RetrievalonMSR-VTT
    text-to-video R@5· 2023-02-01
    69.7
    best: 78.3 (InternVideo2-6B)
    SOTA
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Zero-Shot Video RetrievalonDiDeMo
    text-to-video R@1· 2023-02-01
    45.7
    best: 57.9 (InternVideo2-6B)
    SOTA
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Zero-Shot Video RetrievalonDiDeMo
    text-to-video R@10· 2023-02-01
    79.2
    best: 85.1 (InternVideo2-1B)
    SOTA
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Zero-Shot Video RetrievalonDiDeMo
    text-to-video R@5· 2023-02-01
    71.1
    best: 80 (InternVideo2-6B)
    SOTA
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Zero-Shot Video RetrievalonLSMDC
    text-to-video R@1· 2023-02-01
    24.1
    best: 33.8 (InternVideo2-6B)
    SOTA
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Zero-Shot Video RetrievalonLSMDC
    text-to-video R@10· 2023-02-01
    52
    best: 62.2 (InternVideo2-6B)
    SOTA
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Zero-Shot Video RetrievalonLSMDC
    text-to-video R@5· 2023-02-01
    43.8
    best: 55.9 (InternVideo2-6B)
    SOTA
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • VideoonMSR-VTT-1kA
    text-to-video R@1· 2023-02-01
    53.1
    best: 62.9 (HunYuan_tvr (huge))
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • VideoonMSR-VTT-1kA
    text-to-video R@10· 2023-02-01
    84.7
    best: 90.8 (HunYuan_tvr (huge))
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • VideoonMSR-VTT-1kA
    text-to-video R@5· 2023-02-01
    77.6
    best: 84.5 (HunYuan_tvr (huge))
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • VideoonDiDeMo
    text-to-video R@1· uses extra data· 2023-02-01
    56.4
    best: 74.2 (InternVideo2-6B)
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • VideoonDiDeMo
    text-to-video R@10· uses extra data· 2023-02-01
    85.2
    best: 94.2 (vid-TLDR (UMT-L))
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • VideoonDiDeMo
    text-to-video R@5· uses extra data· 2023-02-01
    79.1
    best: 91.2 (vid-TLDR (UMT-L))
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • VideoonLSMDC
    text-to-video R@1· uses extra data· 2023-02-01
    34.4
    best: 46.4 (InternVideo2-6B)
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • VideoonLSMDC
    text-to-video R@10· uses extra data· 2023-02-01
    65.1
    best: 92.8 (HunYuan_tvr (huge))
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • VideoonLSMDC
    text-to-video R@5· uses extra data· 2023-02-01
    55.2
    best: 80.1 (HunYuan_tvr (huge))
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • VideoonKinetics-700
    Top-1 Accuracy· uses extra data· 2023-02-01
    80.4
    best: 85.9 (InternVideo2-6B)
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • VideoonKinetics-700
    Top-5 Accuracy· uses extra data· 2023-02-01
    94.9
    best: 96.7 (UMT-L (ViT-L/16))
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • VideoonKinetics-400
    Acc@1· 2023-02-01
    87.1
    best: 93.6 (OmniVec2)
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • VideoonKinetics-400
    Acc@5· 2023-02-01
    97.7
    best: 98.9 (TubeViT-H (ImageNet-1k))
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • VideoonKinetics-600
    Top-1 Accuracy· uses extra data· 2023-02-01
    89.8
    best: 91.9 (InternVideo2-6B)
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • VideoonKinetics-600
    Top-5 Accuracy· uses extra data· 2023-02-01
    98.3
    best: 98.9 (TubeVit-H)
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Video CaptioningonMSR-VTT
    METEOR· 2023-02-01
    34.9
    best: 38.7 (MV-GPT)
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Video CaptioningonMSVD
    BLEU-4· 2023-02-01
    70.5
    best: 80.7 (VALOR)
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Video RetrievalonMSR-VTT-1kA
    text-to-video R@1· 2023-02-01
    53.1
    best: 62.9 (HunYuan_tvr (huge))
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Video RetrievalonMSR-VTT-1kA
    text-to-video R@10· 2023-02-01
    84.7
    best: 90.8 (HunYuan_tvr (huge))
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Video RetrievalonMSR-VTT-1kA
    text-to-video R@5· 2023-02-01
    77.6
    best: 84.5 (HunYuan_tvr (huge))
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Video RetrievalonDiDeMo
    text-to-video R@1· uses extra data· 2023-02-01
    56.4
    best: 74.2 (InternVideo2-6B)
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Video RetrievalonDiDeMo
    text-to-video R@10· uses extra data· 2023-02-01
    85.2
    best: 94.2 (vid-TLDR (UMT-L))
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Video RetrievalonDiDeMo
    text-to-video R@5· uses extra data· 2023-02-01
    79.1
    best: 91.2 (vid-TLDR (UMT-L))
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Video RetrievalonLSMDC
    text-to-video R@1· uses extra data· 2023-02-01
    34.4
    best: 46.4 (InternVideo2-6B)
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Video RetrievalonLSMDC
    text-to-video R@10· uses extra data· 2023-02-01
    65.1
    best: 92.8 (HunYuan_tvr (huge))
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Video RetrievalonLSMDC
    text-to-video R@5· uses extra data· 2023-02-01
    55.2
    best: 80.1 (HunYuan_tvr (huge))
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402

Natural Language Processing4 results

  • Visual Question Answering (VQA)onMSRVTT-QA
    Accuracy· uses extra data· 2023-02-01
    0.48
    best: 0.496 (VLAB)
    SOTA
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Visual Question Answering (VQA)onMSVD-QA
    Accuracy· uses extra data· 2023-02-01
    0.581
    best: 0.61 (VLAB)
    SOTA
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Visual Question Answering (VQA)onVQA v2 test-dev
    Accuracy· 2023-02-01
    81.11
    best: 84.3 (PaLI)
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402
  • Visual Question AnsweringonVQA v2 test-dev
    Accuracy· 2023-02-01
    81.11
    best: 82.3 (BLIP-2 ViT-G OPT 6.7B (fine-tuned))
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402

Reasoning1 result

  • Video Question AnsweringonMSRVTT-QA
    Accuracy· 2023-02-01
    48
    best: 72.4 (Flash-VStream)
    SOTA
    mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoarXiv:2302.00402