Tasks SotA Datasets Papers Methods Submit About

Papers With Code 2

A community resource for machine learning research: papers, code, benchmarks, and state-of-the-art results.

Explore

Notable Benchmarks All SotA Datasets Papers Methods

Community

Submit Results About

Data sourced from the PWC Archive (CC-BY-SA 4.0). Built by the community, for the community.

Models/vid-TLDR (UMT-L)

vid-TLDR (UMT-L)

Reported on 98 benchmarks across 4 tasks · 1 paper · 24 SOTA

Note: results are matched by exact model name. Different papers may use the same name for different model variants.

Computer Vision96 results

VideoonSSv2-label retrieval
text-to-video R@5· uses extra data· 2024-03-20
93.3
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonDiDeMo
text-to-video R@1· uses extra data· 2024-03-20
72.3
best: 74.2 (InternVideo2-6B)
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonDiDeMo
text-to-video R@10· uses extra data· 2024-03-20
94.2
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonDiDeMo
text-to-video R@5· uses extra data· 2024-03-20
91.2
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonDiDeMo
video-to-text R@1· uses extra data· 2024-03-20
68.5
best: 71.9 (InternVideo2-6B)
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonDiDeMo
video-to-text R@10· uses extra data· 2024-03-20
93.8
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonDiDeMo
video-to-text R@5· uses extra data· 2024-03-20
89.8
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonLSMDC
text-to-video R@1· uses extra data· 2024-03-20
43.1
best: 46.4 (InternVideo2-6B)
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonMSVD
video-to-text R@1· uses extra data· 2024-03-20
82.7
best: 85.2 (InternVideo2-6B)
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonSSv2-label retrieval
text-to-video R@5· uses extra data· 2024-03-20
93.3
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonDiDeMo
text-to-video R@1· uses extra data· 2024-03-20
72.3
best: 74.2 (InternVideo2-6B)
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonDiDeMo
text-to-video R@10· uses extra data· 2024-03-20
94.2
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonDiDeMo
text-to-video R@5· uses extra data· 2024-03-20
91.2
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonDiDeMo
video-to-text R@1· uses extra data· 2024-03-20
68.5
best: 71.9 (InternVideo2-6B)
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonDiDeMo
video-to-text R@10· uses extra data· 2024-03-20
93.8
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonDiDeMo
video-to-text R@5· uses extra data· 2024-03-20
89.8
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonLSMDC
text-to-video R@1· uses extra data· 2024-03-20
43.1
best: 46.4 (InternVideo2-6B)
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonMSVD
video-to-text R@1· uses extra data· 2024-03-20
82.7
best: 85.2 (InternVideo2-6B)
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonMSVD
video-to-text R@1· uses extra data· 2024-03-20
75.7
best: 83.3 (InternVideo2-1B)
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonDiDeMo
text-to-video R@10· uses extra data· 2024-03-20
81
best: 85.1 (InternVideo2-1B)
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonDiDeMo
video-to-text R@1· uses extra data· 2024-03-20
52
best: 57.1 (InternVideo2-6B)
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonDiDeMo
video-to-text R@10· uses extra data· 2024-03-20
83.8
best: 85 (InternVideo2-6B)
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonDiDeMo
video-to-text R@5· uses extra data· 2024-03-20
75.9
best: 79.9 (InternVideo2-6B)
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonActivityNet
video-to-text R@1· uses extra data· 2024-03-20
41.2
best: 56.5 (InternVideo2-6B)
SOTA
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonSSv2-template retrieval
text-to-video R@1· uses extra data· 2024-03-20
90.2
best: 90.8 (UMT-L (ViT-L/16))
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonSSv2-template retrieval
text-to-video R@10· uses extra data· 2024-03-20
100
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonSSv2-template retrieval
text-to-video R@5· uses extra data· 2024-03-20
100
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonActivityNet
text-to-video R@1· uses extra data· 2024-03-20
66.7
best: 74.1 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonActivityNet
text-to-video R@10· uses extra data· 2024-03-20
94.4
best: 96.1 (GRAM)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonActivityNet
text-to-video R@5· uses extra data· 2024-03-20
88.6
best: 90.9 (VAST)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonActivityNet
video-to-text R@1· uses extra data· 2024-03-20
63.9
best: 69.7 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonActivityNet
video-to-text R@10· uses extra data· 2024-03-20
94.5
best: 95.4 (GRAM)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonActivityNet
video-to-text R@5· uses extra data· 2024-03-20
88.7
best: 89.1 (UMT-L (ViT-L/16))
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonSSv2-label retrieval
text-to-video R@1· uses extra data· 2024-03-20
73.1
best: 73.3 (UMT-L (ViT-L/16))
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonSSv2-label retrieval
text-to-video R@10· uses extra data· 2024-03-20
96.6
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonMSR-VTT
text-to-video R@1· uses extra data· 2024-03-20
58.1
best: 64 (GRAM)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonMSR-VTT
text-to-video R@10· uses extra data· 2024-03-20
81.6
best: 89.6 (VAST)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonMSR-VTT
text-to-video R@5· uses extra data· 2024-03-20
81
best: 84.3 (VAST)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonMSR-VTT
video-to-text R@1· uses extra data· 2024-03-20
58.7
best: 64.8 (GRAM)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonMSR-VTT
video-to-text R@10· uses extra data· 2024-03-20
86.9
best: 92.8 (CAMoE)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonMSR-VTT
video-to-text R@5· uses extra data· 2024-03-20
81.6
best: 86.2 (CAMoE)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonLSMDC
text-to-video R@10· uses extra data· 2024-03-20
71.4
best: 92.8 (HunYuan_tvr (huge))
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonLSMDC
text-to-video R@5· uses extra data· 2024-03-20
64.5
best: 80.1 (HunYuan_tvr (huge))
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonLSMDC
video-to-text R@1· uses extra data· 2024-03-20
40.7
best: 46.7 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonLSMDC
video-to-text R@10· uses extra data· 2024-03-20
63.6
best: 91.8 (HunYuan_tvr (huge))
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonLSMDC
video-to-text R@5· uses extra data· 2024-03-20
70.2
best: 71.8 (HunYuan_tvr (huge))
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonMSVD
text-to-video R@1· uses extra data· 2024-03-20
57.9
best: 61.4 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonMSVD
text-to-video R@10· uses extra data· 2024-03-20
89.4
best: 90.3 (HunYuan_tvr (huge))
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonMSVD
text-to-video R@5· uses extra data· 2024-03-20
83.8
best: 87.6 (CAMoE)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonMSVD
video-to-text R@10· uses extra data· 2024-03-20
96.3
best: 97.1 (PAU)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
VideoonMSVD
video-to-text R@5· uses extra data· 2024-03-20
94.5
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonSSv2-template retrieval
text-to-video R@1· uses extra data· 2024-03-20
90.2
best: 90.8 (UMT-L (ViT-L/16))
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonSSv2-template retrieval
text-to-video R@10· uses extra data· 2024-03-20
100
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonSSv2-template retrieval
text-to-video R@5· uses extra data· 2024-03-20
100
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonActivityNet
text-to-video R@1· uses extra data· 2024-03-20
66.7
best: 74.1 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonActivityNet
text-to-video R@10· uses extra data· 2024-03-20
94.4
best: 96.1 (GRAM)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonActivityNet
text-to-video R@5· uses extra data· 2024-03-20
88.6
best: 90.9 (VAST)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonActivityNet
video-to-text R@1· uses extra data· 2024-03-20
63.9
best: 69.7 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonActivityNet
video-to-text R@10· uses extra data· 2024-03-20
94.5
best: 95.4 (GRAM)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonActivityNet
video-to-text R@5· uses extra data· 2024-03-20
88.7
best: 89.1 (UMT-L (ViT-L/16))
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonSSv2-label retrieval
text-to-video R@1· uses extra data· 2024-03-20
73.1
best: 73.3 (UMT-L (ViT-L/16))
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonSSv2-label retrieval
text-to-video R@10· uses extra data· 2024-03-20
96.6
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonMSR-VTT
text-to-video R@1· uses extra data· 2024-03-20
58.1
best: 64 (GRAM)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonMSR-VTT
text-to-video R@10· uses extra data· 2024-03-20
81.6
best: 89.6 (VAST)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonMSR-VTT
text-to-video R@5· uses extra data· 2024-03-20
81
best: 84.3 (VAST)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonMSR-VTT
video-to-text R@1· uses extra data· 2024-03-20
58.7
best: 64.8 (GRAM)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonMSR-VTT
video-to-text R@10· uses extra data· 2024-03-20
86.9
best: 92.8 (CAMoE)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonMSR-VTT
video-to-text R@5· uses extra data· 2024-03-20
81.6
best: 86.2 (CAMoE)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonLSMDC
text-to-video R@10· uses extra data· 2024-03-20
71.4
best: 92.8 (HunYuan_tvr (huge))
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonLSMDC
text-to-video R@5· uses extra data· 2024-03-20
64.5
best: 80.1 (HunYuan_tvr (huge))
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonLSMDC
video-to-text R@1· uses extra data· 2024-03-20
40.7
best: 46.7 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonLSMDC
video-to-text R@10· uses extra data· 2024-03-20
63.6
best: 91.8 (HunYuan_tvr (huge))
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonLSMDC
video-to-text R@5· uses extra data· 2024-03-20
70.2
best: 71.8 (HunYuan_tvr (huge))
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonMSVD
text-to-video R@1· uses extra data· 2024-03-20
57.9
best: 61.4 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonMSVD
text-to-video R@10· uses extra data· 2024-03-20
89.4
best: 90.3 (HunYuan_tvr (huge))
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonMSVD
text-to-video R@5· uses extra data· 2024-03-20
83.8
best: 87.6 (CAMoE)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonMSVD
video-to-text R@10· uses extra data· 2024-03-20
96.3
best: 97.1 (PAU)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Video RetrievalonMSVD
video-to-text R@5· uses extra data· 2024-03-20
94.5
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonMSR-VTT
text-to-video R@1· uses extra data· 2024-03-20
42.1
best: 55.9 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonMSR-VTT
text-to-video R@10· uses extra data· 2024-03-20
72.4
best: 85.1 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonMSR-VTT
text-to-video R@5· uses extra data· 2024-03-20
63.9
best: 78.3 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonMSR-VTT
video-to-text R@1· uses extra data· 2024-03-20
37.7
best: 53.7 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonMSR-VTT
video-to-text R@10· uses extra data· 2024-03-20
69.4
best: 84.1 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonMSR-VTT
video-to-text R@5· uses extra data· 2024-03-20
59.8
best: 77.5 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonMSVD
text-to-video R@1· uses extra data· 2024-03-20
50
best: 59.3 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonMSVD
text-to-video R@10· uses extra data· 2024-03-20
85.5
best: 89.6 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonMSVD
text-to-video R@5· uses extra data· 2024-03-20
77.6
best: 84.4 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonMSVD
video-to-text R@10· uses extra data· 2024-03-20
95.1
best: 97.9 (LanguageBind(ViT-L/14))
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonMSVD
video-to-text R@5· uses extra data· 2024-03-20
90
best: 94.3 (InternVideo2-1B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonDiDeMo
text-to-video R@1· uses extra data· 2024-03-20
52
best: 57.9 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonDiDeMo
text-to-video R@5· uses extra data· 2024-03-20
74
best: 80 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonActivityNet
text-to-video R@1· uses extra data· 2024-03-20
42.8
best: 63.2 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonActivityNet
text-to-video R@10· uses extra data· 2024-03-20
79.6
best: 92.5 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonActivityNet
text-to-video R@5· uses extra data· 2024-03-20
69.4
best: 85.6 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonActivityNet
video-to-text R@10· uses extra data· 2024-03-20
79.1
best: 90.3 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Zero-Shot Video RetrievalonActivityNet
video-to-text R@5· uses extra data· 2024-03-20
68.2
best: 82.8 (InternVideo2-6B)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347

Natural Language Processing2 results

Visual Question Answering (VQA)onMSRVTT-QA
Accuracy· uses extra data· 2024-03-20
0.47
best: 0.496 (VLAB)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347
Visual Question Answering (VQA)onMSVD-QA
Accuracy· uses extra data· 2024-03-20
0.549
best: 0.61 (VLAB)
vid-TLDR: Training Free Token merging for Light-weight Video Transformer arXiv:2403.13347