Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi, Yang Gao
Few-shot segmentation (FSS) aims to segment unseen classes given only a few annotated samples. Existing methods suffer the problem of feature undermining, i.e. potential novel classes are treated as background during training phase. Our method aims to alleviate this problem and enhance the feature embedding on latent novel classes. In our work, we propose a novel joint-training framework. Based on conventional episodic training on support-query pairs, we add an additional mining branch that exploits latent novel classes via transferable sub-clusters, and a new rectification technique on both background and foreground categories to enforce more stable prototypes. Over and above that, our transferable sub-cluster has the ability to leverage extra unlabeled data for further feature enhancement. Extensive experiments on two FSS benchmarks demonstrate that our method outperforms previous state-of-the-art by a large margin of 3.7% mIOU on PASCAL-5i and 7.0% mIOU on COCO-20i at the cost of 74% fewer parameters and 2.5x faster inference speed. The source code is available at https://github.com/LiheYoung/MiningFSS.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Few-Shot Learning | COCO-20i (5-shot) | Mean IoU | 45.1 | MLC (ResNet-101) |
| Few-Shot Learning | COCO-20i (5-shot) | Mean IoU | 41.4 | MLC (ResNet-50) |
| Few-Shot Learning | PASCAL-5i (1-Shot) | Mean IoU | 63.8 | MLC (ResNet-101) |
| Few-Shot Learning | PASCAL-5i (1-Shot) | Mean IoU | 63.6 | MLC (ResNet-50) |
| Few-Shot Learning | COCO-20i (1-shot) | Mean IoU | 37.5 | MLC (ResNet-101) |
| Few-Shot Learning | COCO-20i (1-shot) | Mean IoU | 35.1 | MLC (ResNet-50) |
| Few-Shot Learning | PASCAL-5i (5-Shot) | Mean IoU | 69.3 | MLC (ResNet-101) |
| Few-Shot Learning | PASCAL-5i (5-Shot) | Mean IoU | 66.8 | MLC (ResNet-50) |
| Few-Shot Semantic Segmentation | COCO-20i (5-shot) | Mean IoU | 45.1 | MLC (ResNet-101) |
| Few-Shot Semantic Segmentation | COCO-20i (5-shot) | Mean IoU | 41.4 | MLC (ResNet-50) |
| Few-Shot Semantic Segmentation | PASCAL-5i (1-Shot) | Mean IoU | 63.8 | MLC (ResNet-101) |
| Few-Shot Semantic Segmentation | PASCAL-5i (1-Shot) | Mean IoU | 63.6 | MLC (ResNet-50) |
| Few-Shot Semantic Segmentation | COCO-20i (1-shot) | Mean IoU | 37.5 | MLC (ResNet-101) |
| Few-Shot Semantic Segmentation | COCO-20i (1-shot) | Mean IoU | 35.1 | MLC (ResNet-50) |
| Few-Shot Semantic Segmentation | PASCAL-5i (5-Shot) | Mean IoU | 69.3 | MLC (ResNet-101) |
| Few-Shot Semantic Segmentation | PASCAL-5i (5-Shot) | Mean IoU | 66.8 | MLC (ResNet-50) |
| Meta-Learning | COCO-20i (5-shot) | Mean IoU | 45.1 | MLC (ResNet-101) |
| Meta-Learning | COCO-20i (5-shot) | Mean IoU | 41.4 | MLC (ResNet-50) |
| Meta-Learning | PASCAL-5i (1-Shot) | Mean IoU | 63.8 | MLC (ResNet-101) |
| Meta-Learning | PASCAL-5i (1-Shot) | Mean IoU | 63.6 | MLC (ResNet-50) |
| Meta-Learning | COCO-20i (1-shot) | Mean IoU | 37.5 | MLC (ResNet-101) |
| Meta-Learning | COCO-20i (1-shot) | Mean IoU | 35.1 | MLC (ResNet-50) |
| Meta-Learning | PASCAL-5i (5-Shot) | Mean IoU | 69.3 | MLC (ResNet-101) |
| Meta-Learning | PASCAL-5i (5-Shot) | Mean IoU | 66.8 | MLC (ResNet-50) |