Image Retrieval with Multi-Modal Query on Flickr30k

Metric: Text-to-image R@5 (higher is better)

LeaderboardDataset

Loading chart...

Results

Hide extra data

Sort:

#	Model↕	Text-to-image R@5▼	Extra Data	Paper	Date↕	Code
1	M2-Encoder	99.5	Yes	M2-Encoder: Advancing Bilingual Image-Text Under...	2024-01-29	Code
2	ERNIE-ViL 2.0	99.4	Yes	ERNIE-ViL 2.0: Multi-view Contrastive Learning f...	2022-09-30	Code
3	BEiT-3	98.7	Yes	Image as a Foreign Language: BEiT Pretraining fo...	2022-08-22	Code
4	X2-VLM (large)	98.6	Yes	X$^2$-VLM: All-In-One Pre-trained Model For Visi...	2022-11-22	Code
5	VAST	98.5	Yes	VAST: A Vision-Audio-Subtitle-Text Omni-Modality...	2023-05-29	Code
6	X2-VLM (base)	98.2	Yes	X$^2$-VLM: All-In-One Pre-trained Model For Visi...	2022-11-22	Code
7	OmniVL (14M)	97.8	Yes	OmniVL:One Foundation Model for Image-Language a...	2022-09-15	-
8	Aurora (ours, r=128)	97.6	Yes	-	-	-
9	VSE-Gradient	97.4	Yes	Dissecting Deep Metric Learning Losses for Image...	2022-10-21	Code
10	ALIGN	97.4	Yes	Scaling Up Visual and Vision-Language Representa...	2021-02-11	Code
11	X-VLM (base)	97.3	Yes	Multi-Grained Vision Language Pre-Training: Alig...	2021-11-16	Code
12	InternVL-G	97	No	InternVL: Scaling up Vision Foundation Models an...	2023-12-21	Code
13	RO-ViT	96.1	No	Region-Aware Pretraining for Open-Vocabulary Obj...	2023-05-11	Code
14	InternVL-C	96	No	InternVL: Scaling up Vision Foundation Models an...	2023-12-21	Code
15	CoCa	95.7	No	CoCa: Contrastive Captioners are Image-Text Foun...	2022-05-04	Code
16	BEiT-3	95.6	No	Image as a Foreign Language: BEiT Pretraining fo...	2022-08-22	Code
17	COSMOS ViT-B/16	95.3	No	COSMOS: Cross-Modality Self-Distillation for Vis...	2024-12-02	Code
18	Flamingo	95.3	No	Flamingo: a Visual Language Model for Few-Shot L...	2022-04-29	Code
19	VK-OOD	94.3	No	-	-	Code
20	ViSTA	94.2	Yes	ViSTA: Vision and Scene Text Aggregation for Cro...	2022-03-31	-
21	OpenCLIP VIT-H/14	94.1	No	Reproducible scaling laws for contrastive langua...	2022-12-14	Code
22	ERNIE-ViL 2.0	93.8	No	ERNIE-ViL 2.0: Multi-view Contrastive Learning f...	2022-09-30	Code
23	ALIGN	93.8	No	Scaling Up Visual and Vision-Language Representa...	2021-02-11	Code
24	ALBEF	93.7	No	Align before Fuse: Vision and Language Represent...	2021-07-16	Code
25	Florence	93.6	No	Florence: A New Foundation Model for Computer Vi...	2021-11-22	Code
26	IAIS	93.3	Yes	Learning Relation Alignment for Calibrated Cross...	2021-05-28	Code
27	COSMOS ViT-B/32	92.8	No	COSMOS: Cross-Modality Self-Distillation for Vis...	2024-12-02	Code
28	AltCLIP	91.6	No	AltCLIP: Altering the Language Encoder in CLIP f...	2022-11-12	Code
29	3SHNet	91	No	3SHNet: Boosting Image-Sentence Retrieval via Vi...	2024-04-26	Code
30	PTP-BLIP (14M)	91	No	Position-guided Text Prompt for Vision-Language ...	2022-12-19	Code
31	DSMD	90.8	No	Dynamic Self-adaptive Multiscale Distillation fr...	2024-04-16	Code
32	CLIP	90.6	No	Learning Transferable Visual Models From Natural...	2021-02-26	Code
33	ViLT-B/32	88.7	Yes	ViLT: Vision-and-Language Transformer Without Co...	2021-02-05	Code
34	UNITER	88.4	No	UNITER: UNiversal Image-TExt Representation Lear...	2019-09-25	Code
35	RCAR	85.8	No	Plug-and-Play Regulators for Image-Text Matching	2023-03-23	Code
36	SGRAF	83	No	Similarity Reasoning and Filtration for Image-Te...	2021-01-05	Code
37	ViLT-B/32	82.5	No	ViLT: Vision-and-Language Transformer Without Co...	2021-02-05	Code
38	GSMN	82.3	No	Graph Structured Network for Image-Text Matching	2020-04-01	Code
39	Pearl	81.3	No	-	-	-
40	ImageBERT	79.6	No	ImageBERT: Cross-modal Pre-training with Large-s...	2020-01-22	-
41	IMRAM	79.4	No	IMRAM: Iterative Matching with Recurrent Attenti...	2020-03-08	Code
42	SCAN	77.7	No	Stacked Cross Attention for Image-Text Matching	2018-03-21	Code
43	SCO (ResNet)	70.5	No	Learning Semantic Concepts and Order for Image a...	2017-12-06	-
44	VSE++ (ResNet)	70.1	No	VSE++: Improving Visual-Semantic Embeddings with...	2017-07-18	Code
45	Dual-Path (ResNet)	69.2	No	Dual-Path Convolutional Image-Text Embeddings wi...	2017-11-15	Code
46	CMPL (ResNet)	65.7	No	-	-	Code

#1M2-EncoderSOTA
99.5
Text-to-image R@5· Extra Data· 2024-01-29
M2-Encoder: Advancing Bilingual Image-Text Understanding by Large-scale Efficient Pretraining Code
#2ERNIE-ViL 2.0SOTA
99.4
Text-to-image R@5· Extra Data· 2022-09-30
ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training Code
#3BEiT-3SOTA
98.7
Text-to-image R@5· Extra Data· 2022-08-22
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks Code
#4X2-VLM (large)
98.6
Text-to-image R@5· Extra Data· 2022-11-22
X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks Code
#5VAST
98.5
Text-to-image R@5· Extra Data· 2023-05-29
VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset Code
#6X2-VLM (base)
98.2
Text-to-image R@5· Extra Data· 2022-11-22
X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks Code
#7OmniVL (14M)
97.8
Text-to-image R@5· Extra Data· 2022-09-15
OmniVL:One Foundation Model for Image-Language and Video-Language Tasks
#8Aurora (ours, r=128)
97.6
Text-to-image R@5· Extra Data
No paper
#9VSE-Gradient
97.4
Text-to-image R@5· Extra Data· 2022-10-21
Dissecting Deep Metric Learning Losses for Image-Text Retrieval Code
#10ALIGNSOTA
97.4
Text-to-image R@5· Extra Data· 2021-02-11
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Code
#11X-VLM (base)
97.3
Text-to-image R@5· Extra Data· 2021-11-16
Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts Code
#12InternVL-G
97
Text-to-image R@5· 2023-12-21
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks Code
#13RO-ViT
96.1
Text-to-image R@5· 2023-05-11
Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers Code
#14InternVL-C
96
Text-to-image R@5· 2023-12-21
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks Code
#15CoCa
95.7
Text-to-image R@5· 2022-05-04
CoCa: Contrastive Captioners are Image-Text Foundation Models Code
#16BEiT-3
95.6
Text-to-image R@5· 2022-08-22
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks Code
#17COSMOS ViT-B/16
95.3
Text-to-image R@5· 2024-12-02
COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training Code
#18Flamingo
95.3
Text-to-image R@5· 2022-04-29
Flamingo: a Visual Language Model for Few-Shot Learning Code
#19VK-OOD
94.3
Text-to-image R@5
No paperCode
#20ViSTA
94.2
Text-to-image R@5· Extra Data· 2022-03-31
ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrieval
#21OpenCLIP VIT-H/14
94.1
Text-to-image R@5· 2022-12-14
Reproducible scaling laws for contrastive language-image learning Code
#22ERNIE-ViL 2.0
93.8
Text-to-image R@5· 2022-09-30
ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training Code
#23ALIGN
93.8
Text-to-image R@5· 2021-02-11
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Code
#24ALBEF
93.7
Text-to-image R@5· 2021-07-16
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation Code
#25Florence
93.6
Text-to-image R@5· 2021-11-22
Florence: A New Foundation Model for Computer Vision Code
#26IAIS
93.3
Text-to-image R@5· Extra Data· 2021-05-28
Learning Relation Alignment for Calibrated Cross-modal Retrieval Code
#27COSMOS ViT-B/32
92.8
Text-to-image R@5· 2024-12-02
COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training Code
#28AltCLIP
91.6
Text-to-image R@5· 2022-11-12
AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities Code
#293SHNet
91
Text-to-image R@5· 2024-04-26
3SHNet: Boosting Image-Sentence Retrieval via Visual Semantic-Spatial Self-Highlighting Code
#30PTP-BLIP (14M)
91
Text-to-image R@5· 2022-12-19
Position-guided Text Prompt for Vision-Language Pre-training Code
#31DSMD
90.8
Text-to-image R@5· 2024-04-16
Dynamic Self-adaptive Multiscale Distillation from Pre-trained Multimodal Large Model for Efficient Cross-modal Representation Learning Code
#32CLIP
90.6
Text-to-image R@5· 2021-02-26
Learning Transferable Visual Models From Natural Language Supervision Code
#33ViLT-B/32SOTA
88.7
Text-to-image R@5· Extra Data· 2021-02-05
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision Code
#34UNITERSOTA
88.4
Text-to-image R@5· 2019-09-25
UNITER: UNiversal Image-TExt Representation Learning Code
#35RCAR
85.8
Text-to-image R@5· 2023-03-23
Plug-and-Play Regulators for Image-Text Matching Code
#36SGRAF
83
Text-to-image R@5· 2021-01-05
Similarity Reasoning and Filtration for Image-Text Matching Code
#37ViLT-B/32
82.5
Text-to-image R@5· 2021-02-05
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision Code
#38GSMN
82.3
Text-to-image R@5· 2020-04-01
Graph Structured Network for Image-Text Matching Code
#39Pearl
81.3
Text-to-image R@5
No paper
#40ImageBERT
79.6
Text-to-image R@5· 2020-01-22
ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
#41IMRAM
79.4
Text-to-image R@5· 2020-03-08
IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrieval Code
#42SCANSOTA
77.7
Text-to-image R@5· 2018-03-21
Stacked Cross Attention for Image-Text Matching Code
#43SCO (ResNet)SOTA
70.5
Text-to-image R@5· 2017-12-06
Learning Semantic Concepts and Order for Image and Sentence Matching
#44VSE++ (ResNet)SOTA
70.1
Text-to-image R@5· 2017-07-18
VSE++: Improving Visual-Semantic Embeddings with Hard Negatives Code
#45Dual-Path (ResNet)
69.2
Text-to-image R@5· 2017-11-15
Dual-Path Convolutional Image-Text Embeddings with Instance Loss Code
#46CMPL (ResNet)
65.7
Text-to-image R@5
No paperCode