Qianxiong Xu, Guosheng Lin, Chen Change Loy, Cheng Long, Ziyue Li, Rui Zhao
Recent advancements in few-shot segmentation (FSS) have exploited pixel-by-pixel matching between query and support features, typically based on cross attention, which selectively activate query foreground (FG) features that correspond to the same-class support FG features. However, due to the large receptive fields in deep layers of the backbone, the extracted query and support FG features are inevitably mingled with background (BG) features, impeding the FG-FG matching in cross attention. Hence, the query FG features are fused with less support FG features, i.e., the support information is not well utilized. This paper presents a novel plug-in termed ambiguity elimination network (AENet), which can be plugged into any existing cross attention-based FSS methods. The main idea is to mine discriminative query FG regions to rectify the ambiguous FG features, increasing the proportion of FG information, so as to suppress the negative impacts of the doped BG features. In this way, the FG-FG matching is naturally enhanced. We plug AENet into three baselines CyCTR, SCCAN and HDMNet for evaluation, and their scores are improved by large margins, e.g., the 1-shot performance of SCCAN can be improved by 3.0%+ on both PASCAL-5$^i$ and COCO-20$^i$. The code is available at https://github.com/Sam1224/AENet.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Few-Shot Learning | COCO-20i (5-shot) | FB-IoU | 78.5 | AENet (ResNet-50) |
| Few-Shot Learning | COCO-20i (5-shot) | Mean IoU | 57.1 | AENet (ResNet-50) |
| Few-Shot Learning | COCO-20i (5-shot) | FB-IoU | 74.3 | AENet (VGG-16) |
| Few-Shot Learning | COCO-20i (5-shot) | Mean IoU | 52.8 | AENet (VGG-16) |
| Few-Shot Learning | PASCAL-5i (1-Shot) | FB-IoU | 81.2 | AENet (ResNet-50) |
| Few-Shot Learning | PASCAL-5i (1-Shot) | Mean IoU | 70.3 | AENet (ResNet-50) |
| Few-Shot Learning | PASCAL-5i (1-Shot) | FB-IoU | 79 | AENet (VGG-16) |
| Few-Shot Learning | PASCAL-5i (1-Shot) | Mean IoU | 66.6 | AENet (VGG-16) |
| Few-Shot Learning | COCO-20i (1-shot) | FB-IoU | 74.4 | AENet (ResNet-50) |
| Few-Shot Learning | COCO-20i (1-shot) | Mean IoU | 51.3 | AENet (ResNet-50) |
| Few-Shot Learning | COCO-20i (1-shot) | FB-IoU | 71.8 | AENet (VGG-16) |
| Few-Shot Learning | COCO-20i (1-shot) | Mean IoU | 46.4 | AENet (VGG-16) |
| Few-Shot Learning | PASCAL-5i (5-Shot) | FB-IoU | 84.5 | AENet (ResNet-50) |
| Few-Shot Learning | PASCAL-5i (5-Shot) | Mean IoU | 74.2 | AENet (ResNet-50) |
| Few-Shot Learning | PASCAL-5i (5-Shot) | FB-IoU | 81.8 | AENet (VGG-16) |
| Few-Shot Learning | PASCAL-5i (5-Shot) | Mean IoU | 70.6 | AENet (VGG-16) |
| Few-Shot Semantic Segmentation | COCO-20i (5-shot) | FB-IoU | 78.5 | AENet (ResNet-50) |
| Few-Shot Semantic Segmentation | COCO-20i (5-shot) | Mean IoU | 57.1 | AENet (ResNet-50) |
| Few-Shot Semantic Segmentation | COCO-20i (5-shot) | FB-IoU | 74.3 | AENet (VGG-16) |
| Few-Shot Semantic Segmentation | COCO-20i (5-shot) | Mean IoU | 52.8 | AENet (VGG-16) |
| Few-Shot Semantic Segmentation | PASCAL-5i (1-Shot) | FB-IoU | 81.2 | AENet (ResNet-50) |
| Few-Shot Semantic Segmentation | PASCAL-5i (1-Shot) | Mean IoU | 70.3 | AENet (ResNet-50) |
| Few-Shot Semantic Segmentation | PASCAL-5i (1-Shot) | FB-IoU | 79 | AENet (VGG-16) |
| Few-Shot Semantic Segmentation | PASCAL-5i (1-Shot) | Mean IoU | 66.6 | AENet (VGG-16) |
| Few-Shot Semantic Segmentation | COCO-20i (1-shot) | FB-IoU | 74.4 | AENet (ResNet-50) |
| Few-Shot Semantic Segmentation | COCO-20i (1-shot) | Mean IoU | 51.3 | AENet (ResNet-50) |
| Few-Shot Semantic Segmentation | COCO-20i (1-shot) | FB-IoU | 71.8 | AENet (VGG-16) |
| Few-Shot Semantic Segmentation | COCO-20i (1-shot) | Mean IoU | 46.4 | AENet (VGG-16) |
| Few-Shot Semantic Segmentation | PASCAL-5i (5-Shot) | FB-IoU | 84.5 | AENet (ResNet-50) |
| Few-Shot Semantic Segmentation | PASCAL-5i (5-Shot) | Mean IoU | 74.2 | AENet (ResNet-50) |
| Few-Shot Semantic Segmentation | PASCAL-5i (5-Shot) | FB-IoU | 81.8 | AENet (VGG-16) |
| Few-Shot Semantic Segmentation | PASCAL-5i (5-Shot) | Mean IoU | 70.6 | AENet (VGG-16) |
| Meta-Learning | COCO-20i (5-shot) | FB-IoU | 78.5 | AENet (ResNet-50) |
| Meta-Learning | COCO-20i (5-shot) | Mean IoU | 57.1 | AENet (ResNet-50) |
| Meta-Learning | COCO-20i (5-shot) | FB-IoU | 74.3 | AENet (VGG-16) |
| Meta-Learning | COCO-20i (5-shot) | Mean IoU | 52.8 | AENet (VGG-16) |
| Meta-Learning | PASCAL-5i (1-Shot) | FB-IoU | 81.2 | AENet (ResNet-50) |
| Meta-Learning | PASCAL-5i (1-Shot) | Mean IoU | 70.3 | AENet (ResNet-50) |
| Meta-Learning | PASCAL-5i (1-Shot) | FB-IoU | 79 | AENet (VGG-16) |
| Meta-Learning | PASCAL-5i (1-Shot) | Mean IoU | 66.6 | AENet (VGG-16) |
| Meta-Learning | COCO-20i (1-shot) | FB-IoU | 74.4 | AENet (ResNet-50) |
| Meta-Learning | COCO-20i (1-shot) | Mean IoU | 51.3 | AENet (ResNet-50) |
| Meta-Learning | COCO-20i (1-shot) | FB-IoU | 71.8 | AENet (VGG-16) |
| Meta-Learning | COCO-20i (1-shot) | Mean IoU | 46.4 | AENet (VGG-16) |
| Meta-Learning | PASCAL-5i (5-Shot) | FB-IoU | 84.5 | AENet (ResNet-50) |
| Meta-Learning | PASCAL-5i (5-Shot) | Mean IoU | 74.2 | AENet (ResNet-50) |
| Meta-Learning | PASCAL-5i (5-Shot) | FB-IoU | 81.8 | AENet (VGG-16) |
| Meta-Learning | PASCAL-5i (5-Shot) | Mean IoU | 70.6 | AENet (VGG-16) |