Weiyue Wang, Ronald Yu, Qiangui Huang, Ulrich Neumann
We introduce Similarity Group Proposal Network (SGPN), a simple and intuitive deep learning framework for 3D object instance segmentation on point clouds. SGPN uses a single network to predict point grouping proposals and a corresponding semantic class for each proposal, from which we can directly extract instance segmentation results. Important to the effectiveness of SGPN is its novel representation of 3D instance segmentation results in the form of a similarity matrix that indicates the similarity between each pair of points in embedded feature space, thus producing an accurate grouping proposal for each point. To the best of our knowledge, SGPN is the first framework to learn 3D instance-aware semantic segmentation on point clouds. Experimental results on various 3D scenes show the effectiveness of our method on 3D instance segmentation, and we also evaluate the capability of SGPN to improve 3D object detection and semantic segmentation results. We also demonstrate its flexibility by seamlessly incorporating 2D CNN features into the framework to boost performance.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Semantic Segmentation | ShapeNet-Part | Instance Average IoU | 85.8 | SGPN |
| Object Detection | NYU Depth v2 | MAP | 41.3 | SGPN-CNN |
| Object Detection | ScanNetV2 | mAP@0.25 | 20.7 | SGPN |
| 3D | NYU Depth v2 | MAP | 41.3 | SGPN-CNN |
| 3D | ScanNetV2 | mAP@0.25 | 20.7 | SGPN |
| Instance Segmentation | NYU Depth v2 | mAP@0.5 | 30.5 | SGPN-CNN |
| Instance Segmentation | ScanNetV1 | mAP@0.25 | 35.1 | SGPN |
| Instance Segmentation | ScanNetV2 | mAP@0.50 | 14.3 | SGPN |
| 3D Object Detection | NYU Depth v2 | MAP | 41.3 | SGPN-CNN |
| 3D Object Detection | ScanNetV2 | mAP@0.25 | 20.7 | SGPN |
| 2D Classification | NYU Depth v2 | MAP | 41.3 | SGPN-CNN |
| 2D Classification | ScanNetV2 | mAP@0.25 | 20.7 | SGPN |
| 2D Object Detection | NYU Depth v2 | MAP | 41.3 | SGPN-CNN |
| 2D Object Detection | ScanNetV2 | mAP@0.25 | 20.7 | SGPN |
| 10-shot image generation | ShapeNet-Part | Instance Average IoU | 85.8 | SGPN |
| 16k | NYU Depth v2 | MAP | 41.3 | SGPN-CNN |
| 16k | ScanNetV2 | mAP@0.25 | 20.7 | SGPN |