Qizhu Li, Anurag Arnab, Philip H. S. Torr
We present a weakly supervised model that jointly performs both semantic- and instance-segmentation -- a particularly relevant problem given the substantial cost of obtaining pixel-perfect annotation for these tasks. In contrast to many popular instance segmentation approaches based on object detectors, our method does not predict any overlapping instances. Moreover, we are able to segment both "thing" and "stuff" classes, and thus explain all the pixels in the image. "Thing" classes are weakly-supervised with bounding boxes, and "stuff" with image-level tags. We obtain state-of-the-art results on Pascal VOC, for both full and weak supervision (which achieves about 95% of fully-supervised performance). Furthermore, we present the first weakly-supervised results on Cityscapes for both semantic- and instance-segmentation. Finally, we use our weakly supervised framework to analyse the relationship between annotation quality and predictive performance, which is of interest to dataset creators.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Semantic Segmentation | Cityscapes val | AP | 28.6 | Dynamically Instantiated Network (ResNet-101) |
| Semantic Segmentation | Cityscapes val | PQ | 53.8 | Dynamically Instantiated Network (ResNet-101) |
| Semantic Segmentation | Cityscapes val | PQst | 62.1 | Dynamically Instantiated Network (ResNet-101) |
| Semantic Segmentation | Cityscapes val | PQth | 42.5 | Dynamically Instantiated Network (ResNet-101) |
| Semantic Segmentation | Cityscapes val | mIoU | 79.8 | Dynamically Instantiated Network (ResNet-101) |
| 10-shot image generation | Cityscapes val | AP | 28.6 | Dynamically Instantiated Network (ResNet-101) |
| 10-shot image generation | Cityscapes val | PQ | 53.8 | Dynamically Instantiated Network (ResNet-101) |
| 10-shot image generation | Cityscapes val | PQst | 62.1 | Dynamically Instantiated Network (ResNet-101) |
| 10-shot image generation | Cityscapes val | PQth | 42.5 | Dynamically Instantiated Network (ResNet-101) |
| 10-shot image generation | Cityscapes val | mIoU | 79.8 | Dynamically Instantiated Network (ResNet-101) |
| Panoptic Segmentation | Cityscapes val | AP | 28.6 | Dynamically Instantiated Network (ResNet-101) |
| Panoptic Segmentation | Cityscapes val | PQ | 53.8 | Dynamically Instantiated Network (ResNet-101) |
| Panoptic Segmentation | Cityscapes val | PQst | 62.1 | Dynamically Instantiated Network (ResNet-101) |
| Panoptic Segmentation | Cityscapes val | PQth | 42.5 | Dynamically Instantiated Network (ResNet-101) |
| Panoptic Segmentation | Cityscapes val | mIoU | 79.8 | Dynamically Instantiated Network (ResNet-101) |