Shun-Cheng Wu, Johanna Wald, Keisuke Tateno, Nassir Navab, Federico Tombari
Scene graphs are a compact and explicit representation successfully used in a variety of 2D scene understanding tasks. This work proposes a method to incrementally build up semantic scene graphs from a 3D environment given a sequence of RGB-D frames. To this end, we aggregate PointNet features from primitive scene components by means of a graph neural network. We also propose a novel attention mechanism well suited for partial and missing graph data present in such an incremental reconstruction scenario. Although our proposed method is designed to run on submaps of the scene, we show it also transfers to entire 3D scenes. Experiments show that our approach outperforms 3D scene graph prediction methods by a large margin and its accuracy is on par with other 3D semantic and panoptic segmentation methods while running at 35 Hz.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Scene Parsing | 3R-Scan | Top-5 Accuracy | 0.87 | SceneGraphFusion |
| Scene Parsing | 3R-Scan | Top-5 Accuracy | 0.66 | 3DSSG [Wald2020_3dssg] |
| Semantic Segmentation | ScanNet | PQ | 31.5 | SceneGraphFusion |
| Semantic Segmentation | ScanNet | PQ_st | 43.4 | SceneGraphFusion |
| Semantic Segmentation | ScanNet | PQ_th | 30.2 | SceneGraphFusion |
| Semantic Segmentation | ScanNetV2 | PQ | 31.5 | SceneGraphFusion (NN mapping) |
| Semantic Segmentation | ScanNetV2 | Params (M) | 2.9 | SceneGraphFusion (NN mapping) |
| Semantic Segmentation | ScanNetV2 | RQ | 42.2 | SceneGraphFusion (NN mapping) |
| Semantic Segmentation | ScanNetV2 | SQ | 72.9 | SceneGraphFusion (NN mapping) |
| 3D | 3R-Scan | Top-10 Accuracy | 0.8 | SceneGraphFusion |
| 3D | 3R-Scan | Top-5 Accuracy | 0.7 | SceneGraphFusion |
| 3D | 3R-Scan | Top-10 Accuracy | 0.78 | 3DSSG [Wald2020_3dssg] |
| 3D | 3R-Scan | Top-5 Accuracy | 0.68 | 3DSSG [Wald2020_3dssg] |
| Shape Representation Of 3D Point Clouds | 3R-Scan | Top-10 Accuracy | 0.8 | SceneGraphFusion |
| Shape Representation Of 3D Point Clouds | 3R-Scan | Top-5 Accuracy | 0.7 | SceneGraphFusion |
| Shape Representation Of 3D Point Clouds | 3R-Scan | Top-10 Accuracy | 0.78 | 3DSSG [Wald2020_3dssg] |
| Shape Representation Of 3D Point Clouds | 3R-Scan | Top-5 Accuracy | 0.68 | 3DSSG [Wald2020_3dssg] |
| 3D Object Classification | 3R-Scan | Top-10 Accuracy | 0.8 | SceneGraphFusion |
| 3D Object Classification | 3R-Scan | Top-5 Accuracy | 0.7 | SceneGraphFusion |
| 3D Object Classification | 3R-Scan | Top-10 Accuracy | 0.78 | 3DSSG [Wald2020_3dssg] |
| 3D Object Classification | 3R-Scan | Top-5 Accuracy | 0.68 | 3DSSG [Wald2020_3dssg] |
| 3D Point Cloud Classification | 3R-Scan | Top-10 Accuracy | 0.8 | SceneGraphFusion |
| 3D Point Cloud Classification | 3R-Scan | Top-5 Accuracy | 0.7 | SceneGraphFusion |
| 3D Point Cloud Classification | 3R-Scan | Top-10 Accuracy | 0.78 | 3DSSG [Wald2020_3dssg] |
| 3D Point Cloud Classification | 3R-Scan | Top-5 Accuracy | 0.68 | 3DSSG [Wald2020_3dssg] |
| 3D Classification | 3R-Scan | Top-10 Accuracy | 0.8 | SceneGraphFusion |
| 3D Classification | 3R-Scan | Top-5 Accuracy | 0.7 | SceneGraphFusion |
| 3D Classification | 3R-Scan | Top-10 Accuracy | 0.78 | 3DSSG [Wald2020_3dssg] |
| 3D Classification | 3R-Scan | Top-5 Accuracy | 0.68 | 3DSSG [Wald2020_3dssg] |
| 2D Semantic Segmentation | 3R-Scan | Top-5 Accuracy | 0.87 | SceneGraphFusion |
| 2D Semantic Segmentation | 3R-Scan | Top-5 Accuracy | 0.66 | 3DSSG [Wald2020_3dssg] |
| Scene Graph Generation | 3R-Scan | Top-5 Accuracy | 0.87 | SceneGraphFusion |
| Scene Graph Generation | 3R-Scan | Top-5 Accuracy | 0.66 | 3DSSG [Wald2020_3dssg] |
| 10-shot image generation | ScanNet | PQ | 31.5 | SceneGraphFusion |
| 10-shot image generation | ScanNet | PQ_st | 43.4 | SceneGraphFusion |
| 10-shot image generation | ScanNet | PQ_th | 30.2 | SceneGraphFusion |
| 10-shot image generation | ScanNetV2 | PQ | 31.5 | SceneGraphFusion (NN mapping) |
| 10-shot image generation | ScanNetV2 | Params (M) | 2.9 | SceneGraphFusion (NN mapping) |
| 10-shot image generation | ScanNetV2 | RQ | 42.2 | SceneGraphFusion (NN mapping) |
| 10-shot image generation | ScanNetV2 | SQ | 72.9 | SceneGraphFusion (NN mapping) |
| Panoptic Segmentation | ScanNet | PQ | 31.5 | SceneGraphFusion |
| Panoptic Segmentation | ScanNet | PQ_st | 43.4 | SceneGraphFusion |
| Panoptic Segmentation | ScanNet | PQ_th | 30.2 | SceneGraphFusion |
| Panoptic Segmentation | ScanNetV2 | PQ | 31.5 | SceneGraphFusion (NN mapping) |
| Panoptic Segmentation | ScanNetV2 | Params (M) | 2.9 | SceneGraphFusion (NN mapping) |
| Panoptic Segmentation | ScanNetV2 | RQ | 42.2 | SceneGraphFusion (NN mapping) |
| Panoptic Segmentation | ScanNetV2 | SQ | 72.9 | SceneGraphFusion (NN mapping) |
| 3D Point Cloud Reconstruction | 3R-Scan | Top-10 Accuracy | 0.8 | SceneGraphFusion |
| 3D Point Cloud Reconstruction | 3R-Scan | Top-5 Accuracy | 0.7 | SceneGraphFusion |
| 3D Point Cloud Reconstruction | 3R-Scan | Top-10 Accuracy | 0.78 | 3DSSG [Wald2020_3dssg] |
| 3D Point Cloud Reconstruction | 3R-Scan | Top-5 Accuracy | 0.68 | 3DSSG [Wald2020_3dssg] |