Lorenzo Porzi, Samuel Rota Bulò, Aleksander Colovic, Peter Kontschieder
In this work we introduce a novel, CNN-based architecture that can be trained end-to-end to deliver seamless scene segmentation results. Our goal is to predict consistent semantic segmentation and detection results by means of a panoptic output format, going beyond the simple combination of independently trained segmentation and detection models. The proposed architecture takes advantage of a novel segmentation head that seamlessly integrates multi-scale features generated by a Feature Pyramid Network with contextual information conveyed by a light-weight DeepLab-like module. As additional contribution we review the panoptic metric and propose an alternative that overcomes its limitations when evaluating non-instance categories. Our proposed network architecture yields state-of-the-art results on three challenging street-level datasets, i.e. Cityscapes, Indian Driving Dataset and Mapillary Vistas.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Semantic Segmentation | KITTI Panoptic Segmentation | PQ | 42.2 | Seamless |
| Semantic Segmentation | Indian Driving Dataset | PQ | 48.5 | Seamless |
| 10-shot image generation | KITTI Panoptic Segmentation | PQ | 42.2 | Seamless |
| 10-shot image generation | Indian Driving Dataset | PQ | 48.5 | Seamless |
| Panoptic Segmentation | KITTI Panoptic Segmentation | PQ | 42.2 | Seamless |
| Panoptic Segmentation | Indian Driving Dataset | PQ | 48.5 | Seamless |