Image Retrieval with Multi-Modal Query on Flickr30k

Metric: Image-to-text R@10 (higher is better)

LeaderboardDataset

Loading chart...

Results

Hide extra data

Sort:

#	Model↕	Image-to-text R@10▼	Extra Data	Paper	Date↕	Code
1	X2-VLM (large)	100	Yes	X$^2$-VLM: All-In-One Pre-trained Model For Visi...	2022-11-22	Code
2	X2-VLM (base)	100	Yes	X$^2$-VLM: All-In-One Pre-trained Model For Visi...	2022-11-22	Code
3	BEiT-3	100	Yes	Image as a Foreign Language: BEiT Pretraining fo...	2022-08-22	Code
4	OmniVL (14M)	100	Yes	OmniVL:One Foundation Model for Image-Language a...	2022-09-15	-
5	ERNIE-ViL 2.0	100	Yes	ERNIE-ViL 2.0: Multi-view Contrastive Learning f...	2022-09-30	Code
6	Aurora (ours, r=128)	100	Yes	-	-	-
7	X-VLM (base)	100	Yes	Multi-Grained Vision Language Pre-Training: Alig...	2021-11-16	Code
8	VSE-Gradient	100	Yes	Dissecting Deep Metric Learning Losses for Image...	2022-10-21	Code
9	ALIGN	100	Yes	Scaling Up Visual and Vision-Language Representa...	2021-02-11	Code
10	BEiT-3	100	No	Image as a Foreign Language: BEiT Pretraining fo...	2022-08-22	Code
11	InternVL-G	99.9	No	InternVL: Scaling up Vision Foundation Models an...	2023-12-21	Code
12	InternVL-C	99.9	No	InternVL: Scaling up Vision Foundation Models an...	2023-12-21	Code
13	COSMOS ViT-B/16	99.9	No	COSMOS: Cross-Modality Self-Distillation for Vis...	2024-12-02	Code
14	CoCa	99.9	No	CoCa: Contrastive Captioners are Image-Text Foun...	2022-05-04	Code
15	ERNIE-ViL 2.0	99.8	No	ERNIE-ViL 2.0: Multi-view Contrastive Learning f...	2022-09-30	Code
16	VK-OOD	99.8	No	-	-	Code
17	RO-ViT	99.7	No	Region-Aware Pretraining for Open-Vocabulary Obj...	2023-05-11	Code
18	ALBEF	99.7	No	Align before Fuse: Vision and Language Represent...	2021-07-16	Code
19	Flamingo	99.7	No	Flamingo: a Visual Language Model for Few-Shot L...	2022-04-29	Code
20	ALIGN	99.7	No	Scaling Up Visual and Vision-Language Representa...	2021-02-11	Code
21	ViSTA	99.6	Yes	ViSTA: Vision and Scene Text Aggregation for Cro...	2022-03-31	-
22	M2-Encoder	99.6	Yes	M2-Encoder: Advancing Bilingual Image-Text Under...	2024-01-29	Code
23	IAIS	99.4	Yes	Learning Relation Alignment for Calibrated Cross...	2021-05-28	Code
24	CLIP	99.4	No	Learning Transferable Visual Models From Natural...	2021-02-26	Code
25	COSMOS ViT-B/32	99.3	No	COSMOS: Cross-Modality Self-Distillation for Vis...	2024-12-02	Code
26	PTP-BLIP (14M)	99.3	No	Position-guided Text Prompt for Vision-Language ...	2022-12-19	Code
27	3SHNet	99.2	No	3SHNet: Boosting Image-Sentence Retrieval via Vi...	2024-04-26	Code
28	AltCLIP	99.1	No	AltCLIP: Altering the Language Encoder in CLIP f...	2022-11-12	Code
29	ViLT-B/32	98.6	Yes	ViLT: Vision-and-Language Transformer Without Co...	2021-02-05	Code
30	RCAR	98.4	No	Plug-and-Play Regulators for Image-Text Matching	2023-03-23	Code
31	UNITER	98	No	HADA: A Graph-based Amalgamation Framework in Im...	2023-01-11	Code
32	DSMD	97.7	No	Dynamic Self-adaptive Multiscale Distillation fr...	2024-04-16	Code
33	SGRAF	97.4	No	Similarity Reasoning and Filtration for Image-Te...	2021-01-05	Code
34	GSMN	97.3	No	A Deep Local and Global Scene-Graph Matching for...	2021-06-04	Code
35	Pearl	97.3	No	-	-	-
36	IMRAM	96.6	No	IMRAM: Iterative Matching with Recurrent Attenti...	2020-03-08	Code
37	ViLT-B/32	96.5	No	ViLT: Vision-and-Language Transformer Without Co...	2021-02-05	Code
38	SCAN	95.8	No	Stacked Cross Attention for Image-Text Matching	2018-03-21	Code
39	ImageBERT	94	No	ImageBERT: Cross-modal Pre-training with Large-s...	2020-01-22	-
40	Dual-Path (ResNet)	89.5	No	Dual-Path Convolutional Image-Text Embeddings wi...	2017-11-15	Code
41	SCO (ResNet)	89.3	No	Learning Semantic Concepts and Order for Image a...	2017-12-06	-
42	VSE++ (ResNet)	87.2	No	VSE++: Improving Visual-Semantic Embeddings with...	2017-07-18	Code
43	CMPL (ResNet)	86.1	No	-	-	Code

#1X2-VLM (large)
100
Image-to-text R@10· Extra Data· 2022-11-22
X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks Code
#2X2-VLM (base)
100
Image-to-text R@10· Extra Data· 2022-11-22
X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks Code
#3BEiT-3
100
Image-to-text R@10· Extra Data· 2022-08-22
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks Code
#4OmniVL (14M)
100
Image-to-text R@10· Extra Data· 2022-09-15
OmniVL:One Foundation Model for Image-Language and Video-Language Tasks
#5ERNIE-ViL 2.0
100
Image-to-text R@10· Extra Data· 2022-09-30
ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training Code
#6Aurora (ours, r=128)
100
Image-to-text R@10· Extra Data
No paper
#7X-VLM (base)
100
Image-to-text R@10· Extra Data· 2021-11-16
Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts Code
#8VSE-Gradient
100
Image-to-text R@10· Extra Data· 2022-10-21
Dissecting Deep Metric Learning Losses for Image-Text Retrieval Code
#9ALIGNSOTA
100
Image-to-text R@10· Extra Data· 2021-02-11
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Code
#10BEiT-3
100
Image-to-text R@10· 2022-08-22
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks Code
#11InternVL-G
99.9
Image-to-text R@10· 2023-12-21
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks Code
#12InternVL-C
99.9
Image-to-text R@10· 2023-12-21
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks Code
#13COSMOS ViT-B/16
99.9
Image-to-text R@10· 2024-12-02
COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training Code
#14CoCa
99.9
Image-to-text R@10· 2022-05-04
CoCa: Contrastive Captioners are Image-Text Foundation Models Code
#15ERNIE-ViL 2.0
99.8
Image-to-text R@10· 2022-09-30
ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training Code
#16VK-OOD
99.8
Image-to-text R@10
No paperCode
#17RO-ViT
99.7
Image-to-text R@10· 2023-05-11
Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers Code
#18ALBEF
99.7
Image-to-text R@10· 2021-07-16
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation Code
#19Flamingo
99.7
Image-to-text R@10· 2022-04-29
Flamingo: a Visual Language Model for Few-Shot Learning Code
#20ALIGN
99.7
Image-to-text R@10· 2021-02-11
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Code
#21ViSTA
99.6
Image-to-text R@10· Extra Data· 2022-03-31
ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrieval
#22M2-Encoder
99.6
Image-to-text R@10· Extra Data· 2024-01-29
M2-Encoder: Advancing Bilingual Image-Text Understanding by Large-scale Efficient Pretraining Code
#23IAIS
99.4
Image-to-text R@10· Extra Data· 2021-05-28
Learning Relation Alignment for Calibrated Cross-modal Retrieval Code
#24CLIP
99.4
Image-to-text R@10· 2021-02-26
Learning Transferable Visual Models From Natural Language Supervision Code
#25COSMOS ViT-B/32
99.3
Image-to-text R@10· 2024-12-02
COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training Code
#26PTP-BLIP (14M)
99.3
Image-to-text R@10· 2022-12-19
Position-guided Text Prompt for Vision-Language Pre-training Code
#273SHNet
99.2
Image-to-text R@10· 2024-04-26
3SHNet: Boosting Image-Sentence Retrieval via Visual Semantic-Spatial Self-Highlighting Code
#28AltCLIP
99.1
Image-to-text R@10· 2022-11-12
AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities Code
#29ViLT-B/32SOTA
98.6
Image-to-text R@10· Extra Data· 2021-02-05
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision Code
#30RCAR
98.4
Image-to-text R@10· 2023-03-23
Plug-and-Play Regulators for Image-Text Matching Code
#31UNITER
98
Image-to-text R@10· 2023-01-11
HADA: A Graph-based Amalgamation Framework in Image-text Retrieval Code
#32DSMD
97.7
Image-to-text R@10· 2024-04-16
Dynamic Self-adaptive Multiscale Distillation from Pre-trained Multimodal Large Model for Efficient Cross-modal Representation Learning Code
#33SGRAFSOTA
97.4
Image-to-text R@10· 2021-01-05
Similarity Reasoning and Filtration for Image-Text Matching Code
#34GSMN
97.3
Image-to-text R@10· 2021-06-04
A Deep Local and Global Scene-Graph Matching for Image-Text Retrieval Code
#35Pearl
97.3
Image-to-text R@10
No paper
#36IMRAMSOTA
96.6
Image-to-text R@10· 2020-03-08
IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrieval Code
#37ViLT-B/32
96.5
Image-to-text R@10· 2021-02-05
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision Code
#38SCANSOTA
95.8
Image-to-text R@10· 2018-03-21
Stacked Cross Attention for Image-Text Matching Code
#39ImageBERT
94
Image-to-text R@10· 2020-01-22
ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
#40Dual-Path (ResNet)SOTA
89.5
Image-to-text R@10· 2017-11-15
Dual-Path Convolutional Image-Text Embeddings with Instance Loss Code
#41SCO (ResNet)
89.3
Image-to-text R@10· 2017-12-06
Learning Semantic Concepts and Order for Image and Sentence Matching
#42VSE++ (ResNet)SOTA
87.2
Image-to-text R@10· 2017-07-18
VSE++: Improving Visual-Semantic Embeddings with Hard Negatives Code
#43CMPL (ResNet)
86.1
Image-to-text R@10
No paperCode