Cross-Modal Information Retrieval on COCO 2014

Metric: Text-to-image R@5 (higher is better)

LeaderboardDataset

Loading chart...

Results

Hide extra data

Sort:

#	Model↕	Text-to-image R@5▼	Extra Data	Paper	Date↕	Code
1	BEiT-3	92.8	Yes	Image as a Foreign Language: BEiT Pretraining fo...	2022-08-22	Code
2	VAST	87.7	Yes	VAST: A Vision-Audio-Subtitle-Text Omni-Modality...	2023-05-29	Code
3	X2-VLM (large)	87.5	Yes	X$^2$-VLM: All-In-One Pre-trained Model For Visi...	2022-11-22	Code
4	PTP-BLIP (14M)	87.4	Yes	Position-guided Text Prompt for Vision-Language ...	2022-12-19	Code
5	XFM (base)	87.2	Yes	Toward Building General Foundation Models for La...	2023-01-12	Code
6	X2-VLM (base)	87.1	Yes	X$^2$-VLM: All-In-One Pre-trained Model For Visi...	2022-11-22	Code
7	OmniVL (14M)	86.1	Yes	OmniVL:One Foundation Model for Image-Language a...	2022-09-15	-
8	VSE-Gradient	86	Yes	Dissecting Deep Metric Learning Losses for Image...	2022-10-21	Code
9	DSMD	85.9	No	Dynamic Self-adaptive Multiscale Distillation fr...	2024-04-16	Code
10	X-VLM (base)	85.8	Yes	Multi-Grained Vision Language Pre-Training: Alig...	2021-11-16	Code
11	Florence	85.7	Yes	Florence: A New Foundation Model for Computer Vi...	2021-11-22	Code
12	VK-OOD	84.8	Yes	-	-	Code
13	Aurora (ours, r=128)	84.8	Yes	-	-	-
14	VALOR	84.4	Yes	VALOR: Vision-Audio-Language Omni-Perception Pre...	2023-04-17	Code
15	ALBEF	84.3	Yes	Align before Fuse: Vision and Language Represent...	2021-07-16	Code
16	ERNIE-ViL 2.0	83.4	Yes	ERNIE-ViL 2.0: Multi-view Contrastive Learning f...	2022-09-30	Code
17	ALIGN	83.3	Yes	Scaling Up Visual and Vision-Language Representa...	2021-02-11	Code
18	TCL	83.2	Yes	Learning to Generate Text-grounded Mask for Open...	2022-12-01	Code
19	Oscar	82.8	Yes	Oscar: Object-Semantics Aligned Pre-training for...	2020-04-13	Code
20	METER	82.66	Yes	An Empirical Study of Training End-to-End Vision...	2021-11-03	Code
21	ViSTA	79.6	Yes	ViSTA: Vision and Scene Text Aggregation for Cro...	2022-03-31	-
22	3SHNet	79.3	No	3SHNet: Boosting Image-Sentence Retrieval via Vi...	2024-04-26	Code
23	ALADIN	79.2	Yes	ALADIN: Distilling Fine-grained Alignment Scores...	2022-07-29	Code
24	RCAR	73.2	No	Plug-and-Play Regulators for Image-Text Matching	2023-03-23	Code
25	ViLT-B/32	72.9	Yes	ViLT: Vision-and-Language Transformer Without Co...	2021-02-05	Code
26	VisualSparta	72.8	Yes	VisualSparta: An Embarrassingly Simple Approach ...	2021-01-01	Code
27	LILE	72.1	No	LILE: Look In-Depth before Looking Elsewhere -- ...	2022-03-02	-
28	SGRAF	70.7	No	Similarity Reasoning and Filtration for Image-Te...	2021-01-05	Code
29	VSRN	70.6	No	Visual Semantic Reasoning for Image-Text Matching	2019-09-06	Code
30	SCAN	69.3	No	Stacked Cross Attention for Image-Text Matching	2018-03-21	Code
31	IMRAM	69.1	No	IMRAM: Iterative Matching with Recurrent Attenti...	2020-03-08	Code
32	PVSE	63	No	Polysemous Visual-Semantic Embedding for Cross-M...	2019-06-11	Code
33	SCO (ResNet)	62.9	No	Learning Semantic Concepts and Order for Image a...	2017-12-06	-
34	Dual-Path (ResNet)	53.4	No	Deep Visual-Semantic Alignments for Generating I...	2014-12-07	Code

#1BEiT-3SOTA
92.8
Text-to-image R@5· Extra Data· 2022-08-22
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks Code
#2VAST
87.7
Text-to-image R@5· Extra Data· 2023-05-29
VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset Code
#3X2-VLM (large)
87.5
Text-to-image R@5· Extra Data· 2022-11-22
X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks Code
#4PTP-BLIP (14M)
87.4
Text-to-image R@5· Extra Data· 2022-12-19
Position-guided Text Prompt for Vision-Language Pre-training Code
#5XFM (base)
87.2
Text-to-image R@5· Extra Data· 2023-01-12
Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Tasks Code
#6X2-VLM (base)
87.1
Text-to-image R@5· Extra Data· 2022-11-22
X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks Code
#7OmniVL (14M)
86.1
Text-to-image R@5· Extra Data· 2022-09-15
OmniVL:One Foundation Model for Image-Language and Video-Language Tasks
#8VSE-Gradient
86
Text-to-image R@5· Extra Data· 2022-10-21
Dissecting Deep Metric Learning Losses for Image-Text Retrieval Code
#9DSMD
85.9
Text-to-image R@5· 2024-04-16
Dynamic Self-adaptive Multiscale Distillation from Pre-trained Multimodal Large Model for Efficient Cross-modal Representation Learning Code
#10X-VLM (base)SOTA
85.8
Text-to-image R@5· Extra Data· 2021-11-16
Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts Code
#11Florence
85.7
Text-to-image R@5· Extra Data· 2021-11-22
Florence: A New Foundation Model for Computer Vision Code
#12VK-OOD
84.8
Text-to-image R@5· Extra Data
No paperCode
#13Aurora (ours, r=128)
84.8
Text-to-image R@5· Extra Data
No paper
#14VALOR
84.4
Text-to-image R@5· Extra Data· 2023-04-17
VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset Code
#15ALBEFSOTA
84.3
Text-to-image R@5· Extra Data· 2021-07-16
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation Code
#16ERNIE-ViL 2.0
83.4
Text-to-image R@5· Extra Data· 2022-09-30
ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training Code
#17ALIGNSOTA
83.3
Text-to-image R@5· Extra Data· 2021-02-11
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Code
#18TCL
83.2
Text-to-image R@5· Extra Data· 2022-12-01
Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairs Code
#19OscarSOTA
82.8
Text-to-image R@5· Extra Data· 2020-04-13
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks Code
#20METER
82.66
Text-to-image R@5· Extra Data· 2021-11-03
An Empirical Study of Training End-to-End Vision-and-Language Transformers Code
#21ViSTA
79.6
Text-to-image R@5· Extra Data· 2022-03-31
ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrieval
#223SHNet
79.3
Text-to-image R@5· 2024-04-26
3SHNet: Boosting Image-Sentence Retrieval via Visual Semantic-Spatial Self-Highlighting Code
#23ALADIN
79.2
Text-to-image R@5· Extra Data· 2022-07-29
ALADIN: Distilling Fine-grained Alignment Scores for Efficient Image-Text Matching and Retrieval Code
#24RCAR
73.2
Text-to-image R@5· 2023-03-23
Plug-and-Play Regulators for Image-Text Matching Code
#25ViLT-B/32
72.9
Text-to-image R@5· Extra Data· 2021-02-05
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision Code
#26VisualSparta
72.8
Text-to-image R@5· Extra Data· 2021-01-01
VisualSparta: An Embarrassingly Simple Approach to Large-scale Text-to-Image Search with Weighted Bag-of-words Code
#27LILE
72.1
Text-to-image R@5· 2022-03-02
LILE: Look In-Depth before Looking Elsewhere -- A Dual Attention Network using Transformers for Cross-Modal Information Retrieval in Histopathology Archives
#28SGRAF
70.7
Text-to-image R@5· 2021-01-05
Similarity Reasoning and Filtration for Image-Text Matching Code
#29VSRNSOTA
70.6
Text-to-image R@5· 2019-09-06
Visual Semantic Reasoning for Image-Text Matching Code
#30SCANSOTA
69.3
Text-to-image R@5· 2018-03-21
Stacked Cross Attention for Image-Text Matching Code
#31IMRAM
69.1
Text-to-image R@5· 2020-03-08
IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrieval Code
#32PVSE
63
Text-to-image R@5· 2019-06-11
Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval Code
#33SCO (ResNet)SOTA
62.9
Text-to-image R@5· 2017-12-06
Learning Semantic Concepts and Order for Image and Sentence Matching
#34Dual-Path (ResNet)SOTA
53.4
Text-to-image R@5· 2014-12-07
Deep Visual-Semantic Alignments for Generating Image Descriptions Code