Image Retrieval with Multi-Modal Query on COCO 2014

Metric: Text-to-image R@1 (higher is better)

LeaderboardDataset

Loading chart...

Results

Hide extra data

Sort:

#	Model↕	Text-to-image R@1▼	Extra Data	Paper	Date↕	Code
1	VAST	68	Yes	VAST: A Vision-Audio-Subtitle-Text Omni-Modality...	2023-05-29	Code
2	X2-VLM (large)	67.7	Yes	X$^2$-VLM: All-In-One Pre-trained Model For Visi...	2022-11-22	Code
3	BEiT-3	67.2	Yes	Image as a Foreign Language: BEiT Pretraining fo...	2022-08-22	Code
4	XFM (base)	67	Yes	Toward Building General Foundation Models for La...	2023-01-12	Code
5	X2-VLM (base)	66.2	Yes	X$^2$-VLM: All-In-One Pre-trained Model For Visi...	2022-11-22	Code
6	PTP-BLIP (14M)	64.9	Yes	Position-guided Text Prompt for Vision-Language ...	2022-12-19	Code
7	OmniVL (14M)	64.8	Yes	OmniVL:One Foundation Model for Image-Language a...	2022-09-15	-
8	VSE-Gradient	63.6	Yes	Dissecting Deep Metric Learning Losses for Image...	2022-10-21	Code
9	X-VLM (base)	63.4	Yes	Multi-Grained Vision Language Pre-Training: Alig...	2021-11-16	Code
10	Florence	63.2	Yes	Florence: A New Foundation Model for Computer Vi...	2021-11-22	Code
11	VK-OOD	62.9	Yes	-	-	Code
12	Aurora (ours, r=128)	62.8	Yes	-	-	-
13	DSMD	62.1	No	Dynamic Self-adaptive Multiscale Distillation fr...	2024-04-16	Code
14	VALOR	61.4	Yes	VALOR: Vision-Audio-Language Omni-Perception Pre...	2023-04-17	Code
15	ALBEF	60.7	Yes	Align before Fuse: Vision and Language Represent...	2021-07-16	Code
16	ALIGN	59.9	Yes	Scaling Up Visual and Vision-Language Representa...	2021-02-11	Code
17	ERNIE-ViL 2.0	59.5	Yes	ERNIE-ViL 2.0: Multi-view Contrastive Learning f...	2022-09-30	Code
18	TCL	59	Yes	Vision-Language Pre-Training with Triple Contras...	2022-02-21	Code
19	InternVL-G	58.6	No	InternVL: Scaling up Vision Foundation Models an...	2023-12-21	Code
20	Oscar	57.5	Yes	Oscar: Object-Semantics Aligned Pre-training for...	2020-04-13	Code
21	METER	57.08	Yes	An Empirical Study of Training End-to-End Vision...	2021-11-03	Code
22	M2-Encoder	56.5	No	M2-Encoder: Advancing Bilingual Image-Text Under...	2024-01-29	Code
23	InternVL-C	54.1	No	InternVL: Scaling up Vision Foundation Models an...	2023-12-21	Code
24	TCL	53.5	No	Vision-Language Pre-Training with Triple Contras...	2022-02-21	Code
25	ViSTA	52.6	Yes	ViSTA: Vision and Scene Text Aggregation for Cro...	2022-03-31	-
26	COSMOS ViT-B/16	52.5	No	COSMOS: Cross-Modality Self-Distillation for Vis...	2024-12-02	Code
27	RO-ViT	51.8	No	Region-Aware Pretraining for Open-Vocabulary Obj...	2023-05-11	Code
28	ALADIN	51.3	Yes	ALADIN: Distilling Fine-grained Alignment Scores...	2022-07-29	Code
29	CoCa	51.2	No	CoCa: Contrastive Captioners are Image-Text Foun...	2022-05-04	Code
30	3SHNet	50.3	No	3SHNet: Boosting Image-Sentence Retrieval via Vi...	2024-04-26	Code
31	ALBEF	50.1	No	Align before Fuse: Vision and Language Represent...	2021-07-16	Code
32	PTP-BLIP	49.5	No	Position-guided Text Prompt for Vision-Language ...	2022-12-19	Code
33	COSMOS ViT-B/32	48.4	No	COSMOS: Cross-Modality Self-Distillation for Vis...	2024-12-02	Code
34	Flamingo	48	No	Flamingo: a Visual Language Model for Few-Shot L...	2022-04-29	Code
35	Florence	47.2	No	Florence: A New Foundation Model for Computer Vi...	2021-11-22	Code
36	ERNIE-ViL 2.0	46	No	ERNIE-ViL 2.0: Multi-view Contrastive Learning f...	2022-09-30	Code
37	ALIGN	45.6	No	Scaling Up Visual and Vision-Language Representa...	2021-02-11	Code
38	VisualSparta	44.4	Yes	VisualSparta: An Embarrassingly Simple Approach ...	2021-01-01	Code
39	RCAR	44.3	No	Plug-and-Play Regulators for Image-Text Matching	2023-03-23	Code
40	NAPReg	43	No	-	-	Code
41	ViLT-B/32	42.7	Yes	ViLT: Vision-and-Language Transformer Without Co...	2021-02-05	Code
42	SGRAF	41.9	No	Similarity Reasoning and Filtration for Image-Te...	2021-01-05	Code
43	LILE	41.5	No	LILE: Look In-Depth before Looking Elsewhere -- ...	2022-03-02	-
44	VSRN	40.5	No	Visual Semantic Reasoning for Image-Text Matching	2019-09-06	Code
45	ViLT-B/32	40.4	No	ViLT: Vision-and-Language Transformer Without Co...	2021-02-05	Code
46	IMRAM	39.7	No	IMRAM: Iterative Matching with Recurrent Attenti...	2020-03-08	Code
47	SCAN	38.6	No	Stacked Cross Attention for Image-Text Matching	2018-03-21	Code
48	CLIP	37.8	No	Learning Transferable Visual Models From Natural...	2021-02-26	Code
49	SCO (ResNet)	33.1	No	Learning Semantic Concepts and Order for Image a...	2017-12-06	-
50	PVSE	32.4	No	Polysemous Visual-Semantic Embedding for Cross-M...	2019-06-11	Code
51	ImageBERT	32.3	No	ImageBERT: Cross-modal Pre-training with Large-s...	2020-01-22	-
52	Dual-Path (ResNet)	25.3	No	Deep Visual-Semantic Alignments for Generating I...	2014-12-07	Code
53	dfdf	0	No	-	-	-

#1VASTSOTA
68
Text-to-image R@1· Extra Data· 2023-05-29
VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset Code
#2X2-VLM (large)SOTA
67.7
Text-to-image R@1· Extra Data· 2022-11-22
X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks Code
#3BEiT-3SOTA
67.2
Text-to-image R@1· Extra Data· 2022-08-22
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks Code
#4XFM (base)
67
Text-to-image R@1· Extra Data· 2023-01-12
Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Tasks Code
#5X2-VLM (base)
66.2
Text-to-image R@1· Extra Data· 2022-11-22
X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks Code
#6PTP-BLIP (14M)
64.9
Text-to-image R@1· Extra Data· 2022-12-19
Position-guided Text Prompt for Vision-Language Pre-training Code
#7OmniVL (14M)
64.8
Text-to-image R@1· Extra Data· 2022-09-15
OmniVL:One Foundation Model for Image-Language and Video-Language Tasks
#8VSE-Gradient
63.6
Text-to-image R@1· Extra Data· 2022-10-21
Dissecting Deep Metric Learning Losses for Image-Text Retrieval Code
#9X-VLM (base)SOTA
63.4
Text-to-image R@1· Extra Data· 2021-11-16
Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts Code
#10Florence
63.2
Text-to-image R@1· Extra Data· 2021-11-22
Florence: A New Foundation Model for Computer Vision Code
#11VK-OOD
62.9
Text-to-image R@1· Extra Data
No paperCode
#12Aurora (ours, r=128)
62.8
Text-to-image R@1· Extra Data
No paper
#13DSMD
62.1
Text-to-image R@1· 2024-04-16
Dynamic Self-adaptive Multiscale Distillation from Pre-trained Multimodal Large Model for Efficient Cross-modal Representation Learning Code
#14VALOR
61.4
Text-to-image R@1· Extra Data· 2023-04-17
VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset Code
#15ALBEFSOTA
60.7
Text-to-image R@1· Extra Data· 2021-07-16
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation Code
#16ALIGNSOTA
59.9
Text-to-image R@1· Extra Data· 2021-02-11
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Code
#17ERNIE-ViL 2.0
59.5
Text-to-image R@1· Extra Data· 2022-09-30
ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training Code
#18TCL
59
Text-to-image R@1· Extra Data· 2022-02-21
Vision-Language Pre-Training with Triple Contrastive Learning Code
#19InternVL-G
58.6
Text-to-image R@1· 2023-12-21
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks Code
#20OscarSOTA
57.5
Text-to-image R@1· Extra Data· 2020-04-13
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks Code
#21METER
57.08
Text-to-image R@1· Extra Data· 2021-11-03
An Empirical Study of Training End-to-End Vision-and-Language Transformers Code
#22M2-Encoder
56.5
Text-to-image R@1· 2024-01-29
M2-Encoder: Advancing Bilingual Image-Text Understanding by Large-scale Efficient Pretraining Code
#23InternVL-C
54.1
Text-to-image R@1· 2023-12-21
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks Code
#24TCL
53.5
Text-to-image R@1· 2022-02-21
Vision-Language Pre-Training with Triple Contrastive Learning Code
#25ViSTA
52.6
Text-to-image R@1· Extra Data· 2022-03-31
ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrieval
#26COSMOS ViT-B/16
52.5
Text-to-image R@1· 2024-12-02
COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training Code
#27RO-ViT
51.8
Text-to-image R@1· 2023-05-11
Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers Code
#28ALADIN
51.3
Text-to-image R@1· Extra Data· 2022-07-29
ALADIN: Distilling Fine-grained Alignment Scores for Efficient Image-Text Matching and Retrieval Code
#29CoCa
51.2
Text-to-image R@1· 2022-05-04
CoCa: Contrastive Captioners are Image-Text Foundation Models Code
#303SHNet
50.3
Text-to-image R@1· 2024-04-26
3SHNet: Boosting Image-Sentence Retrieval via Visual Semantic-Spatial Self-Highlighting Code
#31ALBEF
50.1
Text-to-image R@1· 2021-07-16
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation Code
#32PTP-BLIP
49.5
Text-to-image R@1· 2022-12-19
Position-guided Text Prompt for Vision-Language Pre-training Code
#33COSMOS ViT-B/32
48.4
Text-to-image R@1· 2024-12-02
COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training Code
#34Flamingo
48
Text-to-image R@1· 2022-04-29
Flamingo: a Visual Language Model for Few-Shot Learning Code
#35Florence
47.2
Text-to-image R@1· 2021-11-22
Florence: A New Foundation Model for Computer Vision Code
#36ERNIE-ViL 2.0
46
Text-to-image R@1· 2022-09-30
ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training Code
#37ALIGN
45.6
Text-to-image R@1· 2021-02-11
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Code
#38VisualSparta
44.4
Text-to-image R@1· Extra Data· 2021-01-01
VisualSparta: An Embarrassingly Simple Approach to Large-scale Text-to-Image Search with Weighted Bag-of-words Code
#39RCAR
44.3
Text-to-image R@1· 2023-03-23
Plug-and-Play Regulators for Image-Text Matching Code
#40NAPReg
43
Text-to-image R@1
No paperCode
#41ViLT-B/32
42.7
Text-to-image R@1· Extra Data· 2021-02-05
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision Code
#42SGRAF
41.9
Text-to-image R@1· 2021-01-05
Similarity Reasoning and Filtration for Image-Text Matching Code
#43LILE
41.5
Text-to-image R@1· 2022-03-02
LILE: Look In-Depth before Looking Elsewhere -- A Dual Attention Network using Transformers for Cross-Modal Information Retrieval in Histopathology Archives
#44VSRNSOTA
40.5
Text-to-image R@1· 2019-09-06
Visual Semantic Reasoning for Image-Text Matching Code
#45ViLT-B/32
40.4
Text-to-image R@1· 2021-02-05
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision Code
#46IMRAM
39.7
Text-to-image R@1· 2020-03-08
IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrieval Code
#47SCANSOTA
38.6
Text-to-image R@1· 2018-03-21
Stacked Cross Attention for Image-Text Matching Code
#48CLIP
37.8
Text-to-image R@1· 2021-02-26
Learning Transferable Visual Models From Natural Language Supervision Code
#49SCO (ResNet)SOTA
33.1
Text-to-image R@1· 2017-12-06
Learning Semantic Concepts and Order for Image and Sentence Matching
#50PVSE
32.4
Text-to-image R@1· 2019-06-11
Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval Code
#51ImageBERT
32.3
Text-to-image R@1· 2020-01-22
ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
#52Dual-Path (ResNet)SOTA
25.3
Text-to-image R@1· 2014-12-07
Deep Visual-Semantic Alignments for Generating Image Descriptions Code
#53dfdf
0
Text-to-image R@1
No paper