Ehtesham Iqbal, Sirojbek Safarov, Seongdeok Bang
Few-shot segmentation aims to segment unseen-class objects given only a handful of densely labeled samples. Prototype learning, where the support feature yields a singleor several prototypes by averaging global and local object information, has been widely used in FSS. However, utilizing only prototype vectors may be insufficient to represent the features for all training data. To extract abundant features and make more precise predictions, we propose a Multi-Similarity and Attention Network (MSANet) including two novel modules, a multi-similarity module and an attention module. The multi-similarity module exploits multiple feature-maps of support images and query images to estimate accurate semantic relationships. The attention module instructs the network to concentrate on class-relevant information. The network is tested on standard FSS datasets, PASCAL-5i 1-shot, PASCAL-5i 5-shot, COCO-20i 1-shot, and COCO-20i 5-shot. The MSANet with the backbone of ResNet-101 achieves the state-of-the-art performance for all 4-benchmark datasets with mean intersection over union (mIoU) of 69.13%, 73.99%, 51.09%, 56.80%, respectively. Code is available at https://github.com/AIVResearch/MSANet
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Few-Shot Learning | COCO-20i (5-shot) | FB-IoU | 56.8 | MSANet (ResNet-101) |
| Few-Shot Learning | COCO-20i (5-shot) | Mean IoU | 56.3 | MSANet (ResNet-101) |
| Few-Shot Learning | COCO-20i (5-shot) | FB-IoU | 53.67 | MSANet (ResNet-50) |
| Few-Shot Learning | COCO-20i (5-shot) | Mean IoU | 50.47 | MSANet (ResNet-50) |
| Few-Shot Learning | PASCAL-5i (1-Shot) | FB-IoU | 80.38 | MSANet (ResNet-101) |
| Few-Shot Learning | PASCAL-5i (1-Shot) | Mean IoU | 69.13 | MSANet (ResNet-101) |
| Few-Shot Learning | PASCAL-5i (1-Shot) | FB-IoU | 80.44 | MSANet (ResNet-50) |
| Few-Shot Learning | PASCAL-5i (1-Shot) | Mean IoU | 68.52 | MSANet (ResNet-50) |
| Few-Shot Learning | PASCAL-5i (1-Shot) | FB-IoU | 78.01 | MSANet (VGG-16) |
| Few-Shot Learning | PASCAL-5i (1-Shot) | Mean IoU | 65.76 | MSANet (VGG-16) |
| Few-Shot Learning | COCO-20i (1-shot) | FB-IoU | 51.09 | MSANet (ResNet-101) |
| Few-Shot Learning | COCO-20i (1-shot) | Mean IoU | 50.45 | MSANet (ResNet-101) |
| Few-Shot Learning | COCO-20i (1-shot) | FB-IoU | 48.03 | MSANet (ResNet-50) |
| Few-Shot Learning | COCO-20i (1-shot) | Mean IoU | 46.44 | MSANet (ResNet-50) |
| Few-Shot Learning | PASCAL-5i (5-Shot) | FB-IoU | 84.3 | MSANet (ResNet-101) |
| Few-Shot Learning | PASCAL-5i (5-Shot) | Mean IoU | 73.99 | MSANet (ResNet-101) |
| Few-Shot Learning | PASCAL-5i (5-Shot) | FB-IoU | 83.23 | MSANet (ResNet-50) |
| Few-Shot Learning | PASCAL-5i (5-Shot) | Mean IoU | 72.6 | MSANet (ResNet-50) |
| Few-Shot Learning | PASCAL-5i (5-Shot) | FB-IoU | 80.5 | MSANet (VGG-16) |
| Few-Shot Learning | PASCAL-5i (5-Shot) | Mean IoU | 70.4 | MSANet (VGG-16) |
| Few-Shot Semantic Segmentation | COCO-20i (5-shot) | FB-IoU | 56.8 | MSANet (ResNet-101) |
| Few-Shot Semantic Segmentation | COCO-20i (5-shot) | Mean IoU | 56.3 | MSANet (ResNet-101) |
| Few-Shot Semantic Segmentation | COCO-20i (5-shot) | FB-IoU | 53.67 | MSANet (ResNet-50) |
| Few-Shot Semantic Segmentation | COCO-20i (5-shot) | Mean IoU | 50.47 | MSANet (ResNet-50) |
| Few-Shot Semantic Segmentation | PASCAL-5i (1-Shot) | FB-IoU | 80.38 | MSANet (ResNet-101) |
| Few-Shot Semantic Segmentation | PASCAL-5i (1-Shot) | Mean IoU | 69.13 | MSANet (ResNet-101) |
| Few-Shot Semantic Segmentation | PASCAL-5i (1-Shot) | FB-IoU | 80.44 | MSANet (ResNet-50) |
| Few-Shot Semantic Segmentation | PASCAL-5i (1-Shot) | Mean IoU | 68.52 | MSANet (ResNet-50) |
| Few-Shot Semantic Segmentation | PASCAL-5i (1-Shot) | FB-IoU | 78.01 | MSANet (VGG-16) |
| Few-Shot Semantic Segmentation | PASCAL-5i (1-Shot) | Mean IoU | 65.76 | MSANet (VGG-16) |
| Few-Shot Semantic Segmentation | COCO-20i (1-shot) | FB-IoU | 51.09 | MSANet (ResNet-101) |
| Few-Shot Semantic Segmentation | COCO-20i (1-shot) | Mean IoU | 50.45 | MSANet (ResNet-101) |
| Few-Shot Semantic Segmentation | COCO-20i (1-shot) | FB-IoU | 48.03 | MSANet (ResNet-50) |
| Few-Shot Semantic Segmentation | COCO-20i (1-shot) | Mean IoU | 46.44 | MSANet (ResNet-50) |
| Few-Shot Semantic Segmentation | PASCAL-5i (5-Shot) | FB-IoU | 84.3 | MSANet (ResNet-101) |
| Few-Shot Semantic Segmentation | PASCAL-5i (5-Shot) | Mean IoU | 73.99 | MSANet (ResNet-101) |
| Few-Shot Semantic Segmentation | PASCAL-5i (5-Shot) | FB-IoU | 83.23 | MSANet (ResNet-50) |
| Few-Shot Semantic Segmentation | PASCAL-5i (5-Shot) | Mean IoU | 72.6 | MSANet (ResNet-50) |
| Few-Shot Semantic Segmentation | PASCAL-5i (5-Shot) | FB-IoU | 80.5 | MSANet (VGG-16) |
| Few-Shot Semantic Segmentation | PASCAL-5i (5-Shot) | Mean IoU | 70.4 | MSANet (VGG-16) |
| Meta-Learning | COCO-20i (5-shot) | FB-IoU | 56.8 | MSANet (ResNet-101) |
| Meta-Learning | COCO-20i (5-shot) | Mean IoU | 56.3 | MSANet (ResNet-101) |
| Meta-Learning | COCO-20i (5-shot) | FB-IoU | 53.67 | MSANet (ResNet-50) |
| Meta-Learning | COCO-20i (5-shot) | Mean IoU | 50.47 | MSANet (ResNet-50) |
| Meta-Learning | PASCAL-5i (1-Shot) | FB-IoU | 80.38 | MSANet (ResNet-101) |
| Meta-Learning | PASCAL-5i (1-Shot) | Mean IoU | 69.13 | MSANet (ResNet-101) |
| Meta-Learning | PASCAL-5i (1-Shot) | FB-IoU | 80.44 | MSANet (ResNet-50) |
| Meta-Learning | PASCAL-5i (1-Shot) | Mean IoU | 68.52 | MSANet (ResNet-50) |
| Meta-Learning | PASCAL-5i (1-Shot) | FB-IoU | 78.01 | MSANet (VGG-16) |
| Meta-Learning | PASCAL-5i (1-Shot) | Mean IoU | 65.76 | MSANet (VGG-16) |
| Meta-Learning | COCO-20i (1-shot) | FB-IoU | 51.09 | MSANet (ResNet-101) |
| Meta-Learning | COCO-20i (1-shot) | Mean IoU | 50.45 | MSANet (ResNet-101) |
| Meta-Learning | COCO-20i (1-shot) | FB-IoU | 48.03 | MSANet (ResNet-50) |
| Meta-Learning | COCO-20i (1-shot) | Mean IoU | 46.44 | MSANet (ResNet-50) |
| Meta-Learning | PASCAL-5i (5-Shot) | FB-IoU | 84.3 | MSANet (ResNet-101) |
| Meta-Learning | PASCAL-5i (5-Shot) | Mean IoU | 73.99 | MSANet (ResNet-101) |
| Meta-Learning | PASCAL-5i (5-Shot) | FB-IoU | 83.23 | MSANet (ResNet-50) |
| Meta-Learning | PASCAL-5i (5-Shot) | Mean IoU | 72.6 | MSANet (ResNet-50) |
| Meta-Learning | PASCAL-5i (5-Shot) | FB-IoU | 80.5 | MSANet (VGG-16) |
| Meta-Learning | PASCAL-5i (5-Shot) | Mean IoU | 70.4 | MSANet (VGG-16) |