UpDown

Reported on 65 benchmarks across 2 tasks · 1 paper

Note: results are matched by exact model name. Different papers may use the same name for different model variants.

Natural Language Processing65 results

Visual Question Answering (VQA)onVQA-CE
Accuracy (Counterexamples)· 2021-04-07
33.91
best: 34.41 (RandImg)
Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answering arXiv:2104.03149
Image Captioningonnocaps near-domain
B1
75.25
best: 88.9 (GIT2, Single Model)
Image Captioningonnocaps near-domain
B2
56.93
best: 75.86 (GIT2, Single Model)
Image Captioningonnocaps near-domain
B3
36.91
best: 58.99 (PaLI)
Image Captioningonnocaps near-domain
B4
20.49
best: 39.98 (PaLI)
Image Captioningonnocaps near-domain
CIDEr
56.85
best: 125.51 (GIT2, Single Model)
Image Captioningonnocaps near-domain
METEOR
23.6
best: 33.47 (PaLI)
Image Captioningonnocaps near-domain
ROUGE-L
51.84
best: 63.99 (PaLI)
Image Captioningonnocaps near-domain
SPICE
10.33
best: 16.11 (GIT2, Single Model)
Image Captioningonnocaps entire
B1
74
best: 88.1 (GIT, Single Model)
Image Captioningonnocaps entire
B2
55.11
best: 74.81 (GIT, Single Model)
Image Captioningonnocaps entire
B3
35.23
best: 57.68 (GIT, Single Model)
Image Captioningonnocaps entire
B4
19.16
best: 37.71 (CoCa - Google Brain)
Image Captioningonnocaps entire
CIDEr
54.25
best: 126.8 (Lyrics)
Image Captioningonnocaps entire
METEOR
22.96
best: 32.5 (GIT, Single Model)
Image Captioningonnocaps entire
ROUGE-L
50.92
best: 63.12 (GIT, Single Model)
Image Captioningonnocaps entire
SPICE
10.14
best: 15.94 (GIT, Single Model)
Image Captioningonnocaps out-of-domain
B1
66.54
best: 86.28 (PaLI)
Image Captioningonnocaps out-of-domain
B2
44.28
best: 71.28 (GIT, Single Model)
Image Captioningonnocaps out-of-domain
B3
24.23
best: 52.66 (GIT, Single Model)
Image Captioningonnocaps out-of-domain
B4
10.17
best: 32 (PaLI)
Image Captioningonnocaps out-of-domain
CIDEr
30.09
best: 126.67 (PaLI)
Image Captioningonnocaps out-of-domain
METEOR
18.29
best: 30.99 (PaLI)
Image Captioningonnocaps out-of-domain
ROUGE-L
44.84
best: 61.35 (PaLI)
Image Captioningonnocaps out-of-domain
SPICE
8.08
best: 15.7 (GIT, Single Model)
Image Captioningonnocaps-XD in-domain
B1
77.68
best: 88.86 (GIT2)
Image Captioningonnocaps-XD in-domain
B2
60.34
best: 76.1 (GIT)
Image Captioningonnocaps-XD in-domain
B3
41.5
best: 60.53 (GIT)
Image Captioningonnocaps-XD in-domain
B4
24.57
best: 41.65 (GIT)
Image Captioningonnocaps-XD in-domain
CIDEr
74.27
best: 124.18 (GIT2)
Image Captioningonnocaps-XD in-domain
METEOR
26.04
best: 33.83 (GIT2)
Image Captioningonnocaps-XD in-domain
ROUGE-L
54.42
best: 64.02 (GIT)
Image Captioningonnocaps-XD in-domain
SPICE
11.47
best: 16.36 (GIT2)
Image Captioningonnocaps in-domain
B1
77.68
best: 88.86 (GIT2, Single Model)
Image Captioningonnocaps in-domain
B2
60.34
best: 76.1 (GIT, Single Model)
Image Captioningonnocaps in-domain
B3
41.5
best: 60.53 (GIT, Single Model)
Image Captioningonnocaps in-domain
B4
24.57
best: 41.65 (GIT, Single Model)
Image Captioningonnocaps in-domain
CIDEr
74.27
best: 149.1 (PaLI)
Image Captioningonnocaps in-domain
METEOR
26.04
best: 34.22 (PaLI)
Image Captioningonnocaps in-domain
ROUGE-L
54.42
best: 64.39 (PaLI)
Image Captioningonnocaps in-domain
SPICE
11.47
best: 16.36 (GIT2, Single Model)
Image Captioningonnocaps-XD near-domain
B1
75.25
best: 88.9 (GIT2)
Image Captioningonnocaps-XD near-domain
B2
56.93
best: 75.86 (GIT2)
Image Captioningonnocaps-XD near-domain
B3
36.91
best: 58.9 (GIT2)
Image Captioningonnocaps-XD near-domain
B4
20.49
best: 38.95 (GIT2)
Image Captioningonnocaps-XD near-domain
CIDEr
56.85
best: 125.51 (GIT2)
Image Captioningonnocaps-XD near-domain
METEOR
23.6
best: 32.95 (GIT2)
Image Captioningonnocaps-XD near-domain
ROUGE-L
51.84
best: 63.66 (GIT2)
Image Captioningonnocaps-XD near-domain
SPICE
10.33
best: 16.11 (GIT2)
Image Captioningonnocaps-XD entire
B1
74
best: 88.43 (GIT2)
Image Captioningonnocaps-XD entire
B2
55.11
best: 75.02 (GIT2)
Image Captioningonnocaps-XD entire
B3
35.23
best: 57.87 (GIT2)
Image Captioningonnocaps-XD entire
B4
19.16
best: 37.65 (GIT2)
Image Captioningonnocaps-XD entire
CIDEr
54.25
best: 124.77 (GIT2)
Image Captioningonnocaps-XD entire
METEOR
22.96
best: 32.56 (GIT2)
Image Captioningonnocaps-XD entire
ROUGE-L
50.92
best: 63.19 (GIT2)
Image Captioningonnocaps-XD entire
SPICE
10.14
best: 16.06 (GIT2)
Image Captioningonnocaps-XD out-of-domain
B1
66.54
best: 86.28 (GIT2)
Image Captioningonnocaps-XD out-of-domain
B2
44.28
best: 71.28 (GIT)
Image Captioningonnocaps-XD out-of-domain
B3
24.23
best: 52.66 (GIT)
Image Captioningonnocaps-XD out-of-domain
B4
10.17
best: 30.15 (GIT2)
Image Captioningonnocaps-XD out-of-domain
CIDEr
30.09
best: 122.27 (GIT2)
Image Captioningonnocaps-XD out-of-domain
METEOR
18.29
best: 30.45 (GIT)
Image Captioningonnocaps-XD out-of-domain
ROUGE-L
44.84
best: 60.96 (GIT)
Image Captioningonnocaps-XD out-of-domain
SPICE
8.08
best: 15.7 (GIT)