Image Retrieval with Multi-Modal Query on COCO 2014

Metric: Image-to-text R@10 (higher is better)

LeaderboardDataset

Loading chart...

Results

Hide extra data

Sort:

#	Model↕	Image-to-text R@10▼	Extra Data	Paper	Date↕	Code
1	X2-VLM (large)	98.5	Yes	X$^2$-VLM: All-In-One Pre-trained Model For Visi...	2022-11-22	Code
2	X2-VLM (base)	98.5	Yes	X$^2$-VLM: All-In-One Pre-trained Model For Visi...	2022-11-22	Code
3	XFM (base)	98.4	Yes	Toward Building General Foundation Models for La...	2023-01-12	Code
4	BEiT-3	98.3	Yes	Image as a Foreign Language: BEiT Pretraining fo...	2022-08-22	Code
5	X-VLM (base)	98.2	Yes	Multi-Grained Vision Language Pre-Training: Alig...	2021-11-16	Code
6	OmniVL (14M)	98.1	Yes	OmniVL:One Foundation Model for Image-Language a...	2022-09-15	-
7	PTP-BLIP (14M)	97.9	Yes	Position-guided Text Prompt for Vision-Language ...	2022-12-19	Code
8	VSE-Gradient	97.9	Yes	Dissecting Deep Metric Learning Losses for Image...	2022-10-21	Code
9	Aurora (ours, r=128)	97.8	Yes	-	-	-
10	ALBEF	97.2	Yes	Align before Fuse: Vision and Language Represent...	2021-07-16	Code
11	ERNIE-ViL 2.0	97.1	Yes	ERNIE-ViL 2.0: Multi-view Contrastive Learning f...	2022-09-30	Code
12	ALIGN	96.9	Yes	Scaling Up Visual and Vision-Language Representa...	2021-02-11	Code
13	METER	96.82	Yes	An Empirical Study of Training End-to-End Vision...	2021-11-03	Code
14	VK-OOD	96.8	Yes	-	-	Code
15	TCL	96.7	Yes	Vision-Language Pre-Training with Triple Contras...	2022-02-21	Code
16	M2-Encoder	96.3	No	M2-Encoder: Advancing Bilingual Image-Text Under...	2024-01-29	Code
17	Oscar	96	Yes	Oscar: Object-Semantics Aligned Pre-training for...	2020-04-13	Code
18	ViSTA	95.4	Yes	ViSTA: Vision and Scene Text Aggregation for Cro...	2022-03-31	-
19	3SHNet	95.4	No	3SHNet: Boosting Image-Sentence Retrieval via Vi...	2024-04-26	Code
20	TCL	95.4	No	Vision-Language Pre-Training with Triple Contras...	2022-02-21	Code
21	InternVL-G	95.2	No	InternVL: Scaling up Vision Foundation Models an...	2023-12-21	Code
22	PTP-BLIP	94.7	No	Position-guided Text Prompt for Vision-Language ...	2022-12-19	Code
23	ALBEF	94.7	No	Align before Fuse: Vision and Language Represent...	2021-07-16	Code
24	ALADIN	94.5	Yes	ALADIN: Distilling Fine-grained Alignment Scores...	2022-07-29	Code
25	MaMMUT (ours)	93.7	No	MaMMUT: A Simple Architecture for Joint Learning...	2023-03-29	Code
26	InternVL-C	93.5	No	InternVL: Scaling up Vision Foundation Models an...	2023-12-21	Code
27	Flamingo	92.9	No	Flamingo: a Visual Language Model for Few-Shot L...	2022-04-29	Code
28	ViLT-B/32	92.7	Yes	ViLT: Vision-and-Language Transformer Without Co...	2021-02-05	Code
29	RCAR	92.6	No	Plug-and-Play Regulators for Image-Text Matching	2023-03-23	Code
30	COSMOS ViT-B/16	92.5	No	COSMOS: Cross-Modality Self-Distillation for Vis...	2024-12-02	Code
31	RO-ViT	92.2	No	Region-Aware Pretraining for Open-Vocabulary Obj...	2023-05-11	Code
32	COSMOS ViT-B/32	92	No	COSMOS: Cross-Modality Self-Distillation for Vis...	2024-12-02	Code
33	CoCa	91.8	No	CoCa: Contrastive Captioners are Image-Text Foun...	2022-05-04	Code
34	SGRAF	91.6	No	Similarity Reasoning and Filtration for Image-Te...	2021-01-05	Code
35	ERNIE-ViL 2.0	91.4	No	ERNIE-ViL 2.0: Multi-view Contrastive Learning f...	2022-09-30	Code
36	LILE	91	No	LILE: Look In-Depth before Looking Elsewhere -- ...	2022-03-02	-
37	IMRAM	91	No	IMRAM: Iterative Matching with Recurrent Attenti...	2020-03-08	Code
38	SCAN	90	No	Stacked Cross Attention for Image-Text Matching	2018-03-21	Code
39	ALIGN	89.7	No	Scaling Up Visual and Vision-Language Representa...	2021-02-11	Code
40	ViLT-B/32	89.6	No	ViLT: Vision-and-Language Transformer Without Co...	2021-02-05	Code
41	VSRN	89.4	No	Visual Semantic Reasoning for Image-Text Matching	2019-09-06	Code
42	CLIP	88.1	No	Learning Transferable Visual Models From Natural...	2021-02-26	Code
43	DSMD	84.5	No	Dynamic Self-adaptive Multiscale Distillation fr...	2024-04-16	Code
44	PVSE	84.5	No	Polysemous Visual-Semantic Embedding for Cross-M...	2019-06-11	Code
45	SCO (ResNet)	83	No	Learning Semantic Concepts and Order for Image a...	2017-12-06	-
46	Dual-Path (ResNet)	81.1	No	Deep Visual-Semantic Alignments for Generating I...	2014-12-07	Code
47	ImageBERT	80.4	No	ImageBERT: Cross-modal Pre-training with Large-s...	2020-01-22	-
48	dfdf	0	No	-	-	-

#1X2-VLM (large)SOTA
98.5
Image-to-text R@10· Extra Data· 2022-11-22
X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks Code
#2X2-VLM (base)
98.5
Image-to-text R@10· Extra Data· 2022-11-22
X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks Code
#3XFM (base)
98.4
Image-to-text R@10· Extra Data· 2023-01-12
Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Tasks Code
#4BEiT-3SOTA
98.3
Image-to-text R@10· Extra Data· 2022-08-22
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks Code
#5X-VLM (base)SOTA
98.2
Image-to-text R@10· Extra Data· 2021-11-16
Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts Code
#6OmniVL (14M)
98.1
Image-to-text R@10· Extra Data· 2022-09-15
OmniVL:One Foundation Model for Image-Language and Video-Language Tasks
#7PTP-BLIP (14M)
97.9
Image-to-text R@10· Extra Data· 2022-12-19
Position-guided Text Prompt for Vision-Language Pre-training Code
#8VSE-Gradient
97.9
Image-to-text R@10· Extra Data· 2022-10-21
Dissecting Deep Metric Learning Losses for Image-Text Retrieval Code
#9Aurora (ours, r=128)
97.8
Image-to-text R@10· Extra Data
No paper
#10ALBEFSOTA
97.2
Image-to-text R@10· Extra Data· 2021-07-16
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation Code
#11ERNIE-ViL 2.0
97.1
Image-to-text R@10· Extra Data· 2022-09-30
ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training Code
#12ALIGNSOTA
96.9
Image-to-text R@10· Extra Data· 2021-02-11
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Code
#13METER
96.82
Image-to-text R@10· Extra Data· 2021-11-03
An Empirical Study of Training End-to-End Vision-and-Language Transformers Code
#14VK-OOD
96.8
Image-to-text R@10· Extra Data
No paperCode
#15TCL
96.7
Image-to-text R@10· Extra Data· 2022-02-21
Vision-Language Pre-Training with Triple Contrastive Learning Code
#16M2-Encoder
96.3
Image-to-text R@10· 2024-01-29
M2-Encoder: Advancing Bilingual Image-Text Understanding by Large-scale Efficient Pretraining Code
#17OscarSOTA
96
Image-to-text R@10· Extra Data· 2020-04-13
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks Code
#18ViSTA
95.4
Image-to-text R@10· Extra Data· 2022-03-31
ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrieval
#193SHNet
95.4
Image-to-text R@10· 2024-04-26
3SHNet: Boosting Image-Sentence Retrieval via Visual Semantic-Spatial Self-Highlighting Code
#20TCL
95.4
Image-to-text R@10· 2022-02-21
Vision-Language Pre-Training with Triple Contrastive Learning Code
#21InternVL-G
95.2
Image-to-text R@10· 2023-12-21
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks Code
#22PTP-BLIP
94.7
Image-to-text R@10· 2022-12-19
Position-guided Text Prompt for Vision-Language Pre-training Code
#23ALBEF
94.7
Image-to-text R@10· 2021-07-16
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation Code
#24ALADIN
94.5
Image-to-text R@10· Extra Data· 2022-07-29
ALADIN: Distilling Fine-grained Alignment Scores for Efficient Image-Text Matching and Retrieval Code
#25MaMMUT (ours)
93.7
Image-to-text R@10· 2023-03-29
MaMMUT: A Simple Architecture for Joint Learning for MultiModal Tasks Code
#26InternVL-C
93.5
Image-to-text R@10· 2023-12-21
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks Code
#27Flamingo
92.9
Image-to-text R@10· 2022-04-29
Flamingo: a Visual Language Model for Few-Shot Learning Code
#28ViLT-B/32
92.7
Image-to-text R@10· Extra Data· 2021-02-05
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision Code
#29RCAR
92.6
Image-to-text R@10· 2023-03-23
Plug-and-Play Regulators for Image-Text Matching Code
#30COSMOS ViT-B/16
92.5
Image-to-text R@10· 2024-12-02
COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training Code
#31RO-ViT
92.2
Image-to-text R@10· 2023-05-11
Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers Code
#32COSMOS ViT-B/32
92
Image-to-text R@10· 2024-12-02
COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training Code
#33CoCa
91.8
Image-to-text R@10· 2022-05-04
CoCa: Contrastive Captioners are Image-Text Foundation Models Code
#34SGRAF
91.6
Image-to-text R@10· 2021-01-05
Similarity Reasoning and Filtration for Image-Text Matching Code
#35ERNIE-ViL 2.0
91.4
Image-to-text R@10· 2022-09-30
ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training Code
#36LILE
91
Image-to-text R@10· 2022-03-02
LILE: Look In-Depth before Looking Elsewhere -- A Dual Attention Network using Transformers for Cross-Modal Information Retrieval in Histopathology Archives
#37IMRAMSOTA
91
Image-to-text R@10· 2020-03-08
IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrieval Code
#38SCANSOTA
90
Image-to-text R@10· 2018-03-21
Stacked Cross Attention for Image-Text Matching Code
#39ALIGN
89.7
Image-to-text R@10· 2021-02-11
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Code
#40ViLT-B/32
89.6
Image-to-text R@10· 2021-02-05
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision Code
#41VSRN
89.4
Image-to-text R@10· 2019-09-06
Visual Semantic Reasoning for Image-Text Matching Code
#42CLIP
88.1
Image-to-text R@10· 2021-02-26
Learning Transferable Visual Models From Natural Language Supervision Code
#43DSMD
84.5
Image-to-text R@10· 2024-04-16
Dynamic Self-adaptive Multiscale Distillation from Pre-trained Multimodal Large Model for Efficient Cross-modal Representation Learning Code
#44PVSE
84.5
Image-to-text R@10· 2019-06-11
Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval Code
#45SCO (ResNet)SOTA
83
Image-to-text R@10· 2017-12-06
Learning Semantic Concepts and Order for Image and Sentence Matching
#46Dual-Path (ResNet)SOTA
81.1
Image-to-text R@10· 2014-12-07
Deep Visual-Semantic Alignments for Generating Image Descriptions Code
#47ImageBERT
80.4
Image-to-text R@10· 2020-01-22
ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
#48dfdf
0
Image-to-text R@10
No paper