Daan de Geus, Gijs Dubbelman
Unified panoptic segmentation methods are achieving state-of-the-art results on several datasets. To achieve these results on high-resolution datasets, these methods apply crop-based training. In this work, we find that, although crop-based training is advantageous in general, it also has a harmful side-effect. Specifically, it limits the ability of unified networks to discriminate between large object instances, causing them to make predictions that are confused between multiple instances. To solve this, we propose Intra-Batch Supervision (IBS), which improves a network's ability to discriminate between instances by introducing additional supervision using multiple images from the same batch. We show that, with our IBS, we successfully address the confusion problem and consistently improve the performance of unified networks. For the high-resolution Cityscapes and Mapillary Vistas datasets, we achieve improvements of up to +2.5 on the Panoptic Quality for thing classes, and even more considerable gains of up to +5.8 on both the pixel accuracy and pixel precision, which we identify as better metrics to capture the confusion problem.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Semantic Segmentation | Cityscapes val | PQ | 62.4 | Mask2Former + Intra-Batch Supervision (ResNet-50) |
| Semantic Segmentation | Cityscapes val | PQst | 67.3 | Mask2Former + Intra-Batch Supervision (ResNet-50) |
| Semantic Segmentation | Cityscapes val | PQth | 54.7 | Mask2Former + Intra-Batch Supervision (ResNet-50) |
| Semantic Segmentation | Mapillary val | PQ | 42.2 | Mask2Former + Intra-Batch Supervision (ResNet-50) |
| Semantic Segmentation | Mapillary val | PQst | 52 | Mask2Former + Intra-Batch Supervision (ResNet-50) |
| Semantic Segmentation | Mapillary val | PQth | 34.9 | Mask2Former + Intra-Batch Supervision (ResNet-50) |
| 10-shot image generation | Cityscapes val | PQ | 62.4 | Mask2Former + Intra-Batch Supervision (ResNet-50) |
| 10-shot image generation | Cityscapes val | PQst | 67.3 | Mask2Former + Intra-Batch Supervision (ResNet-50) |
| 10-shot image generation | Cityscapes val | PQth | 54.7 | Mask2Former + Intra-Batch Supervision (ResNet-50) |
| 10-shot image generation | Mapillary val | PQ | 42.2 | Mask2Former + Intra-Batch Supervision (ResNet-50) |
| 10-shot image generation | Mapillary val | PQst | 52 | Mask2Former + Intra-Batch Supervision (ResNet-50) |
| 10-shot image generation | Mapillary val | PQth | 34.9 | Mask2Former + Intra-Batch Supervision (ResNet-50) |
| Panoptic Segmentation | Cityscapes val | PQ | 62.4 | Mask2Former + Intra-Batch Supervision (ResNet-50) |
| Panoptic Segmentation | Cityscapes val | PQst | 67.3 | Mask2Former + Intra-Batch Supervision (ResNet-50) |
| Panoptic Segmentation | Cityscapes val | PQth | 54.7 | Mask2Former + Intra-Batch Supervision (ResNet-50) |
| Panoptic Segmentation | Mapillary val | PQ | 42.2 | Mask2Former + Intra-Batch Supervision (ResNet-50) |
| Panoptic Segmentation | Mapillary val | PQst | 52 | Mask2Former + Intra-Batch Supervision (ResNet-50) |
| Panoptic Segmentation | Mapillary val | PQth | 34.9 | Mask2Former + Intra-Batch Supervision (ResNet-50) |