Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick
This paper shows that masked autoencoders (MAE) are scalable self-supervised learners for computer vision. Our MAE approach is simple: we mask random patches of the input image and reconstruct the missing pixels. It is based on two core designs. First, we develop an asymmetric encoder-decoder architecture, with an encoder that operates only on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask tokens. Second, we find that masking a high proportion of the input image, e.g., 75%, yields a nontrivial and meaningful self-supervisory task. Coupling these two designs enables us to train large models efficiently and effectively: we accelerate training (by 3x or more) and improve accuracy. Our scalable approach allows for learning high-capacity models that generalize well: e.g., a vanilla ViT-Huge model achieves the best accuracy (87.8%) among methods that use only ImageNet-1K data. Transfer performance in downstream tasks outperforms supervised pre-training and shows promising scaling behavior.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Domain Adaptation | ImageNet-R | Top-1 Error Rate | 33.5 | MAE (ViT-H, 448) |
| Domain Adaptation | ImageNet-A | Top-1 accuracy % | 76.7 | MAE (ViT-H, 448) |
| Domain Adaptation | ImageNet-C | mean Corruption Error (mCE) | 33.8 | MAE (ViT-H) |
| Domain Adaptation | ImageNet-Sketch | Top-1 accuracy | 50.9 | MAE (ViT-H, 448) |
| Semantic Segmentation | ImageNet-S | mIoU (test) | 61.2 | MAE (ViT-B/16, 224x224, SSL+FT, mmseg) |
| Semantic Segmentation | ImageNet-S | mIoU (val) | 61.6 | MAE (ViT-B/16, 224x224, SSL+FT, mmseg) |
| Semantic Segmentation | ImageNet-S | mIoU (test) | 60.2 | MAE (ViT-B/16, 224x224, SSL+FT) |
| Semantic Segmentation | ImageNet-S | mIoU (val) | 61 | MAE (ViT-B/16, 224x224, SSL+FT) |
| Semantic Segmentation | ImageNet-S | mIoU (test) | 40.3 | MAE (ViT-B/16, 224x224, SSL, mmseg) |
| Semantic Segmentation | ImageNet-S | mIoU (val) | 40 | MAE (ViT-B/16, 224x224, SSL, mmseg) |
| Semantic Segmentation | ImageNet-S | mIoU (test) | 37 | MAE (ViT-B/16, 224x224, SSL) |
| Semantic Segmentation | ImageNet-S | mIoU (val) | 38.3 | MAE (ViT-B/16, 224x224, SSL) |
| Semantic Segmentation | ADE20K | Validation mIoU | 53.6 | MAE (ViT-L, UperNet) |
| Semantic Segmentation | ADE20K | Validation mIoU | 48.1 | MAE (ViT-B, UperNet) |
| Object Detection | COCO minival | box AP | 53.3 | MAE (ViT-L, Mask R-CNN) |
| Object Detection | COCO minival | box AP | 50.3 | MAE (ViT-B, Mask R-CNN) |
| Image Classification | iNaturalist | Top 1 Accuracy | 83.4 | MAE (ViT-H, 448) |
| Image Classification | Places205 | Top 1 Accuracy | 66.8 | MAE (ViT-H, 448) |
| Image Classification | OmniBenchmark | Average Top-1 Accuracy | 30.6 | MAE |
| Image Classification | Places365-Standard | Top 1 Accuracy | 60.3 | MAE (ViT-H, 448) |
| Image Classification | iNaturalist 2019 | Top-1 Accuracy | 88.3 | MAE (ViT-H, 448) |
| 3D | COCO minival | box AP | 53.3 | MAE (ViT-L, Mask R-CNN) |
| 3D | COCO minival | box AP | 50.3 | MAE (ViT-B, Mask R-CNN) |
| 2D Classification | COCO minival | box AP | 53.3 | MAE (ViT-L, Mask R-CNN) |
| 2D Classification | COCO minival | box AP | 50.3 | MAE (ViT-B, Mask R-CNN) |
| 2D Object Detection | COCO minival | box AP | 53.3 | MAE (ViT-L, Mask R-CNN) |
| 2D Object Detection | COCO minival | box AP | 50.3 | MAE (ViT-B, Mask R-CNN) |
| Domain Generalization | ImageNet-R | Top-1 Error Rate | 33.5 | MAE (ViT-H, 448) |
| Domain Generalization | ImageNet-A | Top-1 accuracy % | 76.7 | MAE (ViT-H, 448) |
| Domain Generalization | ImageNet-C | mean Corruption Error (mCE) | 33.8 | MAE (ViT-H) |
| Domain Generalization | ImageNet-Sketch | Top-1 accuracy | 50.9 | MAE (ViT-H, 448) |
| 10-shot image generation | ImageNet-S | mIoU (test) | 61.2 | MAE (ViT-B/16, 224x224, SSL+FT, mmseg) |
| 10-shot image generation | ImageNet-S | mIoU (val) | 61.6 | MAE (ViT-B/16, 224x224, SSL+FT, mmseg) |
| 10-shot image generation | ImageNet-S | mIoU (test) | 60.2 | MAE (ViT-B/16, 224x224, SSL+FT) |
| 10-shot image generation | ImageNet-S | mIoU (val) | 61 | MAE (ViT-B/16, 224x224, SSL+FT) |
| 10-shot image generation | ImageNet-S | mIoU (test) | 40.3 | MAE (ViT-B/16, 224x224, SSL, mmseg) |
| 10-shot image generation | ImageNet-S | mIoU (val) | 40 | MAE (ViT-B/16, 224x224, SSL, mmseg) |
| 10-shot image generation | ImageNet-S | mIoU (test) | 37 | MAE (ViT-B/16, 224x224, SSL) |
| 10-shot image generation | ImageNet-S | mIoU (val) | 38.3 | MAE (ViT-B/16, 224x224, SSL) |
| 10-shot image generation | ADE20K | Validation mIoU | 53.6 | MAE (ViT-L, UperNet) |
| 10-shot image generation | ADE20K | Validation mIoU | 48.1 | MAE (ViT-B, UperNet) |
| 16k | COCO minival | box AP | 53.3 | MAE (ViT-L, Mask R-CNN) |
| 16k | COCO minival | box AP | 50.3 | MAE (ViT-B, Mask R-CNN) |