Konstantin Sofiiuk, Olga Barinova, Anton Konushin
We present Adaptive Instance Selection network architecture for class-agnostic instance segmentation. Given an input image and a point $(x, y)$, it generates a mask for the object located at $(x, y)$. The network adapts to the input point with a help of AdaIN layers, thus producing different masks for different objects on the same image. AdaptIS generates pixel-accurate object masks, therefore it accurately segments objects of complex shape or severely occluded ones. AdaptIS can be easily combined with standard semantic segmentation pipeline to perform panoptic segmentation. To illustrate the idea, we perform experiments on a challenging toy problem with difficult occlusions. Then we extensively evaluate the method on panoptic segmentation benchmarks. We obtain state-of-the-art results on Cityscapes and Mapillary even without pretraining on COCO, and show competitive results on a challenging COCO dataset. The source code of the method and the trained models are available at https://github.com/saic-vul/adaptis.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Semantic Segmentation | Cityscapes val | AP | 36.3 | AdaptIS (ResNeXt-101) |
| Semantic Segmentation | Cityscapes val | PQ | 62 | AdaptIS (ResNeXt-101) |
| Semantic Segmentation | Cityscapes val | PQst | 64.4 | AdaptIS (ResNeXt-101) |
| Semantic Segmentation | Cityscapes val | PQth | 58.7 | AdaptIS (ResNeXt-101) |
| Semantic Segmentation | Cityscapes val | mIoU | 79.2 | AdaptIS (ResNeXt-101) |
| Semantic Segmentation | Cityscapes val | AP | 33.9 | AdaptIS (ResNet-101) |
| Semantic Segmentation | Cityscapes val | PQ | 60.6 | AdaptIS (ResNet-101) |
| Semantic Segmentation | Cityscapes val | PQst | 62.9 | AdaptIS (ResNet-101) |
| Semantic Segmentation | Cityscapes val | PQth | 57.5 | AdaptIS (ResNet-101) |
| Semantic Segmentation | Cityscapes val | mIoU | 77.2 | AdaptIS (ResNet-101) |
| Semantic Segmentation | Cityscapes val | AP | 32.3 | AdaptIS (ResNet-50) |
| Semantic Segmentation | Cityscapes val | PQ | 59 | AdaptIS (ResNet-50) |
| Semantic Segmentation | Cityscapes val | PQst | 61.3 | AdaptIS (ResNet-50) |
| Semantic Segmentation | Cityscapes val | PQth | 55.8 | AdaptIS (ResNet-50) |
| Semantic Segmentation | Cityscapes val | mIoU | 75.3 | AdaptIS (ResNet-50) |
| Semantic Segmentation | Mapillary val | PQ | 40.3 | AdaptIS (ResNeXt-101) |
| Semantic Segmentation | Mapillary val | mIoU | 56.8 | AdaptIS (ResNeXt-101) |
| Semantic Segmentation | COCO test-dev | PQ | 42.8 | AdaptIS (ResNeXt-101) |
| Semantic Segmentation | COCO test-dev | PQst | 31.8 | AdaptIS (ResNeXt-101) |
| Semantic Segmentation | COCO test-dev | PQth | 50.1 | AdaptIS (ResNeXt-101) |
| 10-shot image generation | Cityscapes val | AP | 36.3 | AdaptIS (ResNeXt-101) |
| 10-shot image generation | Cityscapes val | PQ | 62 | AdaptIS (ResNeXt-101) |
| 10-shot image generation | Cityscapes val | PQst | 64.4 | AdaptIS (ResNeXt-101) |
| 10-shot image generation | Cityscapes val | PQth | 58.7 | AdaptIS (ResNeXt-101) |
| 10-shot image generation | Cityscapes val | mIoU | 79.2 | AdaptIS (ResNeXt-101) |
| 10-shot image generation | Cityscapes val | AP | 33.9 | AdaptIS (ResNet-101) |
| 10-shot image generation | Cityscapes val | PQ | 60.6 | AdaptIS (ResNet-101) |
| 10-shot image generation | Cityscapes val | PQst | 62.9 | AdaptIS (ResNet-101) |
| 10-shot image generation | Cityscapes val | PQth | 57.5 | AdaptIS (ResNet-101) |
| 10-shot image generation | Cityscapes val | mIoU | 77.2 | AdaptIS (ResNet-101) |
| 10-shot image generation | Cityscapes val | AP | 32.3 | AdaptIS (ResNet-50) |
| 10-shot image generation | Cityscapes val | PQ | 59 | AdaptIS (ResNet-50) |
| 10-shot image generation | Cityscapes val | PQst | 61.3 | AdaptIS (ResNet-50) |
| 10-shot image generation | Cityscapes val | PQth | 55.8 | AdaptIS (ResNet-50) |
| 10-shot image generation | Cityscapes val | mIoU | 75.3 | AdaptIS (ResNet-50) |
| 10-shot image generation | Mapillary val | PQ | 40.3 | AdaptIS (ResNeXt-101) |
| 10-shot image generation | Mapillary val | mIoU | 56.8 | AdaptIS (ResNeXt-101) |
| 10-shot image generation | COCO test-dev | PQ | 42.8 | AdaptIS (ResNeXt-101) |
| 10-shot image generation | COCO test-dev | PQst | 31.8 | AdaptIS (ResNeXt-101) |
| 10-shot image generation | COCO test-dev | PQth | 50.1 | AdaptIS (ResNeXt-101) |
| Panoptic Segmentation | Cityscapes val | AP | 36.3 | AdaptIS (ResNeXt-101) |
| Panoptic Segmentation | Cityscapes val | PQ | 62 | AdaptIS (ResNeXt-101) |
| Panoptic Segmentation | Cityscapes val | PQst | 64.4 | AdaptIS (ResNeXt-101) |
| Panoptic Segmentation | Cityscapes val | PQth | 58.7 | AdaptIS (ResNeXt-101) |
| Panoptic Segmentation | Cityscapes val | mIoU | 79.2 | AdaptIS (ResNeXt-101) |
| Panoptic Segmentation | Cityscapes val | AP | 33.9 | AdaptIS (ResNet-101) |
| Panoptic Segmentation | Cityscapes val | PQ | 60.6 | AdaptIS (ResNet-101) |
| Panoptic Segmentation | Cityscapes val | PQst | 62.9 | AdaptIS (ResNet-101) |
| Panoptic Segmentation | Cityscapes val | PQth | 57.5 | AdaptIS (ResNet-101) |
| Panoptic Segmentation | Cityscapes val | mIoU | 77.2 | AdaptIS (ResNet-101) |
| Panoptic Segmentation | Cityscapes val | AP | 32.3 | AdaptIS (ResNet-50) |
| Panoptic Segmentation | Cityscapes val | PQ | 59 | AdaptIS (ResNet-50) |
| Panoptic Segmentation | Cityscapes val | PQst | 61.3 | AdaptIS (ResNet-50) |
| Panoptic Segmentation | Cityscapes val | PQth | 55.8 | AdaptIS (ResNet-50) |
| Panoptic Segmentation | Cityscapes val | mIoU | 75.3 | AdaptIS (ResNet-50) |
| Panoptic Segmentation | Mapillary val | PQ | 40.3 | AdaptIS (ResNeXt-101) |
| Panoptic Segmentation | Mapillary val | mIoU | 56.8 | AdaptIS (ResNeXt-101) |
| Panoptic Segmentation | COCO test-dev | PQ | 42.8 | AdaptIS (ResNeXt-101) |
| Panoptic Segmentation | COCO test-dev | PQst | 31.8 | AdaptIS (ResNeXt-101) |
| Panoptic Segmentation | COCO test-dev | PQth | 50.1 | AdaptIS (ResNeXt-101) |