Kumar Ayush, Burak Uzkent, Chenlin Meng, Kumar Tanmay, Marshall Burke, David Lobell, Stefano Ermon
Contrastive learning methods have significantly narrowed the gap between supervised and unsupervised learning on computer vision tasks. In this paper, we explore their application to geo-located datasets, e.g. remote sensing, where unlabeled data is often abundant but labeled data is scarce. We first show that due to their different characteristics, a non-trivial gap persists between contrastive and supervised learning on standard benchmarks. To close the gap, we propose novel training methods that exploit the spatio-temporal structure of remote sensing data. We leverage spatially aligned images over time to construct temporal positive pairs in contrastive learning and geo-location to design pre-text tasks. Our experiments show that our proposed method closes the gap between contrastive and supervised learning on image classification, object detection and semantic segmentation for remote sensing. Moreover, we demonstrate that the proposed method can also be applied to geo-tagged ImageNet images, improving downstream performance on various tasks. Project Webpage can be found at this link geography-aware-ssl.github.io.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Semantic Segmentation | SpaceNet 1 | Mean IoU | 78.48 | PSANet w/ ResNet50 - FMoW self-supervised pre-training w/ MoCo-V2 + Temporal Positives |
| Semantic Segmentation | SpaceNet 1 | Mean IoU | 78.05 | PSANet w/ ResNet50 backbone - FMoW self-supervised pre-training w/ MoCo-V2 |
| Semantic Segmentation | SpaceNet 1 | Mean IoU | 75.57 | PSANet w/ ResNet50 backbone - FMoW pretrained |
| Semantic Segmentation | SpaceNet 1 | Mean IoU | 75.23 | PSANet w/ ResNet50 backbone - ImageNet pretrained |
| Semantic Segmentation | SpaceNet 1 | Mean IoU | 74.93 | PSANet w/ ResNet50 backbone |
| 10-shot image generation | SpaceNet 1 | Mean IoU | 78.48 | PSANet w/ ResNet50 - FMoW self-supervised pre-training w/ MoCo-V2 + Temporal Positives |
| 10-shot image generation | SpaceNet 1 | Mean IoU | 78.05 | PSANet w/ ResNet50 backbone - FMoW self-supervised pre-training w/ MoCo-V2 |
| 10-shot image generation | SpaceNet 1 | Mean IoU | 75.57 | PSANet w/ ResNet50 backbone - FMoW pretrained |
| 10-shot image generation | SpaceNet 1 | Mean IoU | 75.23 | PSANet w/ ResNet50 backbone - ImageNet pretrained |
| 10-shot image generation | SpaceNet 1 | Mean IoU | 74.93 | PSANet w/ ResNet50 backbone |