Juhong Min, Dahyun Kang, Minsu Cho
Few-shot semantic segmentation aims at learning to segment a target object from a query image using only a few annotated support images of the target class. This challenging task requires to understand diverse levels of visual cues and analyze fine-grained correspondence relations between the query and the support images. To address the problem, we propose Hypercorrelation Squeeze Networks (HSNet) that leverages multi-level feature correlation and efficient 4D convolutions. It extracts diverse features from different levels of intermediate convolutional layers and constructs a collection of 4D correlation tensors, i.e., hypercorrelations. Using efficient center-pivot 4D convolutions in a pyramidal architecture, the method gradually squeezes high-level semantic and low-level geometric cues of the hypercorrelation into precise segmentation masks in coarse-to-fine manner. The significant performance improvements on standard few-shot segmentation benchmarks of PASCAL-5i, COCO-20i, and FSS-1000 verify the efficacy of the proposed method.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Few-Shot Learning | FSS-1000 (5-shot) | Mean IoU | 88.5 | HSNet (ResNet-101) |
| Few-Shot Learning | FSS-1000 (5-shot) | Mean IoU | 87.8 | HSNet (ResNet-50) |
| Few-Shot Learning | FSS-1000 (5-shot) | Mean IoU | 85.8 | HSNet (VGG-16) |
| Few-Shot Learning | COCO-20i (5-shot) | FB-IoU | 72.4 | HSNet (ResNet-101) |
| Few-Shot Learning | COCO-20i (5-shot) | Mean IoU | 49.5 | HSNet (ResNet-101) |
| Few-Shot Learning | COCO-20i (5-shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-101) |
| Few-Shot Learning | COCO-20i (5-shot) | FB-IoU | 70.7 | HSNet (ResNet-50) |
| Few-Shot Learning | COCO-20i (5-shot) | Mean IoU | 46.9 | HSNet (ResNet-50) |
| Few-Shot Learning | COCO-20i (5-shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-50) |
| Few-Shot Learning | FSS-1000 (1-shot) | Mean IoU | 86.5 | HSNet (ResNet-101) |
| Few-Shot Learning | FSS-1000 (1-shot) | Mean IoU | 85.5 | HSNet (ResNet-50) |
| Few-Shot Learning | FSS-1000 (1-shot) | Mean IoU | 82.3 | HSNet (VGG-16) |
| Few-Shot Learning | PASCAL-5i (1-Shot) | FB-IoU | 77.6 | HSNet (ResNet-101) |
| Few-Shot Learning | PASCAL-5i (1-Shot) | Mean IoU | 66.2 | HSNet (ResNet-101) |
| Few-Shot Learning | PASCAL-5i (1-Shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-101) |
| Few-Shot Learning | PASCAL-5i (1-Shot) | FB-IoU | 76.7 | HSNet (ResNet-50) |
| Few-Shot Learning | PASCAL-5i (1-Shot) | Mean IoU | 64 | HSNet (ResNet-50) |
| Few-Shot Learning | PASCAL-5i (1-Shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-50) |
| Few-Shot Learning | PASCAL-5i (1-Shot) | FB-IoU | 73.4 | HSNet (VGG-16) |
| Few-Shot Learning | PASCAL-5i (1-Shot) | Mean IoU | 59.7 | HSNet (VGG-16) |
| Few-Shot Learning | COCO-20i (1-shot) | FB-IoU | 69.1 | HSNet (ResNet-101) |
| Few-Shot Learning | COCO-20i (1-shot) | Mean IoU | 41.2 | HSNet (ResNet-101) |
| Few-Shot Learning | COCO-20i (1-shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-101) |
| Few-Shot Learning | COCO-20i (1-shot) | FB-IoU | 68.2 | HSNet (ResNet-50) |
| Few-Shot Learning | COCO-20i (1-shot) | Mean IoU | 39.2 | HSNet (ResNet-50) |
| Few-Shot Learning | COCO-20i (1-shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-50) |
| Few-Shot Learning | PASCAL-5i (5-Shot) | FB-IoU | 80.6 | HSNet (ResNet-101) |
| Few-Shot Learning | PASCAL-5i (5-Shot) | Mean IoU | 70.4 | HSNet (ResNet-101) |
| Few-Shot Learning | PASCAL-5i (5-Shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-101) |
| Few-Shot Learning | PASCAL-5i (5-Shot) | FB-IoU | 80.6 | HSNet (ResNet-50) |
| Few-Shot Learning | PASCAL-5i (5-Shot) | Mean IoU | 69.5 | HSNet (ResNet-50) |
| Few-Shot Learning | PASCAL-5i (5-Shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-50) |
| Few-Shot Learning | PASCAL-5i (5-Shot) | FB-IoU | 76.6 | HSNet (VGG-16) |
| Few-Shot Learning | PASCAL-5i (5-Shot) | Mean IoU | 64.1 | HSNet (VGG-16) |
| Few-Shot Semantic Segmentation | FSS-1000 (5-shot) | Mean IoU | 88.5 | HSNet (ResNet-101) |
| Few-Shot Semantic Segmentation | FSS-1000 (5-shot) | Mean IoU | 87.8 | HSNet (ResNet-50) |
| Few-Shot Semantic Segmentation | FSS-1000 (5-shot) | Mean IoU | 85.8 | HSNet (VGG-16) |
| Few-Shot Semantic Segmentation | COCO-20i (5-shot) | FB-IoU | 72.4 | HSNet (ResNet-101) |
| Few-Shot Semantic Segmentation | COCO-20i (5-shot) | Mean IoU | 49.5 | HSNet (ResNet-101) |
| Few-Shot Semantic Segmentation | COCO-20i (5-shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-101) |
| Few-Shot Semantic Segmentation | COCO-20i (5-shot) | FB-IoU | 70.7 | HSNet (ResNet-50) |
| Few-Shot Semantic Segmentation | COCO-20i (5-shot) | Mean IoU | 46.9 | HSNet (ResNet-50) |
| Few-Shot Semantic Segmentation | COCO-20i (5-shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-50) |
| Few-Shot Semantic Segmentation | FSS-1000 (1-shot) | Mean IoU | 86.5 | HSNet (ResNet-101) |
| Few-Shot Semantic Segmentation | FSS-1000 (1-shot) | Mean IoU | 85.5 | HSNet (ResNet-50) |
| Few-Shot Semantic Segmentation | FSS-1000 (1-shot) | Mean IoU | 82.3 | HSNet (VGG-16) |
| Few-Shot Semantic Segmentation | PASCAL-5i (1-Shot) | FB-IoU | 77.6 | HSNet (ResNet-101) |
| Few-Shot Semantic Segmentation | PASCAL-5i (1-Shot) | Mean IoU | 66.2 | HSNet (ResNet-101) |
| Few-Shot Semantic Segmentation | PASCAL-5i (1-Shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-101) |
| Few-Shot Semantic Segmentation | PASCAL-5i (1-Shot) | FB-IoU | 76.7 | HSNet (ResNet-50) |
| Few-Shot Semantic Segmentation | PASCAL-5i (1-Shot) | Mean IoU | 64 | HSNet (ResNet-50) |
| Few-Shot Semantic Segmentation | PASCAL-5i (1-Shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-50) |
| Few-Shot Semantic Segmentation | PASCAL-5i (1-Shot) | FB-IoU | 73.4 | HSNet (VGG-16) |
| Few-Shot Semantic Segmentation | PASCAL-5i (1-Shot) | Mean IoU | 59.7 | HSNet (VGG-16) |
| Few-Shot Semantic Segmentation | COCO-20i (1-shot) | FB-IoU | 69.1 | HSNet (ResNet-101) |
| Few-Shot Semantic Segmentation | COCO-20i (1-shot) | Mean IoU | 41.2 | HSNet (ResNet-101) |
| Few-Shot Semantic Segmentation | COCO-20i (1-shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-101) |
| Few-Shot Semantic Segmentation | COCO-20i (1-shot) | FB-IoU | 68.2 | HSNet (ResNet-50) |
| Few-Shot Semantic Segmentation | COCO-20i (1-shot) | Mean IoU | 39.2 | HSNet (ResNet-50) |
| Few-Shot Semantic Segmentation | COCO-20i (1-shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-50) |
| Few-Shot Semantic Segmentation | PASCAL-5i (5-Shot) | FB-IoU | 80.6 | HSNet (ResNet-101) |
| Few-Shot Semantic Segmentation | PASCAL-5i (5-Shot) | Mean IoU | 70.4 | HSNet (ResNet-101) |
| Few-Shot Semantic Segmentation | PASCAL-5i (5-Shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-101) |
| Few-Shot Semantic Segmentation | PASCAL-5i (5-Shot) | FB-IoU | 80.6 | HSNet (ResNet-50) |
| Few-Shot Semantic Segmentation | PASCAL-5i (5-Shot) | Mean IoU | 69.5 | HSNet (ResNet-50) |
| Few-Shot Semantic Segmentation | PASCAL-5i (5-Shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-50) |
| Few-Shot Semantic Segmentation | PASCAL-5i (5-Shot) | FB-IoU | 76.6 | HSNet (VGG-16) |
| Few-Shot Semantic Segmentation | PASCAL-5i (5-Shot) | Mean IoU | 64.1 | HSNet (VGG-16) |
| Meta-Learning | FSS-1000 (5-shot) | Mean IoU | 88.5 | HSNet (ResNet-101) |
| Meta-Learning | FSS-1000 (5-shot) | Mean IoU | 87.8 | HSNet (ResNet-50) |
| Meta-Learning | FSS-1000 (5-shot) | Mean IoU | 85.8 | HSNet (VGG-16) |
| Meta-Learning | COCO-20i (5-shot) | FB-IoU | 72.4 | HSNet (ResNet-101) |
| Meta-Learning | COCO-20i (5-shot) | Mean IoU | 49.5 | HSNet (ResNet-101) |
| Meta-Learning | COCO-20i (5-shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-101) |
| Meta-Learning | COCO-20i (5-shot) | FB-IoU | 70.7 | HSNet (ResNet-50) |
| Meta-Learning | COCO-20i (5-shot) | Mean IoU | 46.9 | HSNet (ResNet-50) |
| Meta-Learning | COCO-20i (5-shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-50) |
| Meta-Learning | FSS-1000 (1-shot) | Mean IoU | 86.5 | HSNet (ResNet-101) |
| Meta-Learning | FSS-1000 (1-shot) | Mean IoU | 85.5 | HSNet (ResNet-50) |
| Meta-Learning | FSS-1000 (1-shot) | Mean IoU | 82.3 | HSNet (VGG-16) |
| Meta-Learning | PASCAL-5i (1-Shot) | FB-IoU | 77.6 | HSNet (ResNet-101) |
| Meta-Learning | PASCAL-5i (1-Shot) | Mean IoU | 66.2 | HSNet (ResNet-101) |
| Meta-Learning | PASCAL-5i (1-Shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-101) |
| Meta-Learning | PASCAL-5i (1-Shot) | FB-IoU | 76.7 | HSNet (ResNet-50) |
| Meta-Learning | PASCAL-5i (1-Shot) | Mean IoU | 64 | HSNet (ResNet-50) |
| Meta-Learning | PASCAL-5i (1-Shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-50) |
| Meta-Learning | PASCAL-5i (1-Shot) | FB-IoU | 73.4 | HSNet (VGG-16) |
| Meta-Learning | PASCAL-5i (1-Shot) | Mean IoU | 59.7 | HSNet (VGG-16) |
| Meta-Learning | COCO-20i (1-shot) | FB-IoU | 69.1 | HSNet (ResNet-101) |
| Meta-Learning | COCO-20i (1-shot) | Mean IoU | 41.2 | HSNet (ResNet-101) |
| Meta-Learning | COCO-20i (1-shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-101) |
| Meta-Learning | COCO-20i (1-shot) | FB-IoU | 68.2 | HSNet (ResNet-50) |
| Meta-Learning | COCO-20i (1-shot) | Mean IoU | 39.2 | HSNet (ResNet-50) |
| Meta-Learning | COCO-20i (1-shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-50) |
| Meta-Learning | PASCAL-5i (5-Shot) | FB-IoU | 80.6 | HSNet (ResNet-101) |
| Meta-Learning | PASCAL-5i (5-Shot) | Mean IoU | 70.4 | HSNet (ResNet-101) |
| Meta-Learning | PASCAL-5i (5-Shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-101) |
| Meta-Learning | PASCAL-5i (5-Shot) | FB-IoU | 80.6 | HSNet (ResNet-50) |
| Meta-Learning | PASCAL-5i (5-Shot) | Mean IoU | 69.5 | HSNet (ResNet-50) |
| Meta-Learning | PASCAL-5i (5-Shot) | learnable parameters (million) | 2.5 | HSNet (ResNet-50) |
| Meta-Learning | PASCAL-5i (5-Shot) | FB-IoU | 76.6 | HSNet (VGG-16) |
| Meta-Learning | PASCAL-5i (5-Shot) | Mean IoU | 64.1 | HSNet (VGG-16) |