Yinyu Nie, Xiaoguang Han, Shihui Guo, Yujian Zheng, Jian Chang, Jian Jun Zhang
Semantic reconstruction of indoor scenes refers to both scene understanding and object reconstruction. Existing works either address one part of this problem or focus on independent objects. In this paper, we bridge the gap between understanding and reconstruction, and propose an end-to-end solution to jointly reconstruct room layout, object bounding boxes and meshes from a single image. Instead of separately resolving scene understanding and object reconstruction, our method builds upon a holistic scene context and proposes a coarse-to-fine hierarchy with three components: 1. room layout with camera pose; 2. 3D object bounding boxes; 3. object meshes. We argue that understanding the context of each component can assist the task of parsing the others, which enables joint understanding and reconstruction. The experiments on the SUN RGB-D and Pix3D datasets demonstrate that our method consistently outperforms existing methods in indoor layout estimation, 3D object detection and mesh reconstruction.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Object Detection | SUN RGB-D | AP@0.15 (10 / NYU-37) | 26.38 | Total3D joint |
| Object Detection | SUN RGB-D | AP@0.15 (NYU-37) | 14.28 | Total3D joint |
| Object Detection | SUN RGB-D | AP@0.15 (10 / NYU-37) | 23.32 | Total3D w/o. joint |
| Object Detection | SUN RGB-D | AP@0.15 (NYU-37) | 13.25 | Total3D w/o. joint |
| 3D | SUN RGB-D | AP@0.15 (10 / NYU-37) | 26.38 | Total3D joint |
| 3D | SUN RGB-D | AP@0.15 (NYU-37) | 14.28 | Total3D joint |
| 3D | SUN RGB-D | AP@0.15 (10 / NYU-37) | 23.32 | Total3D w/o. joint |
| 3D | SUN RGB-D | AP@0.15 (NYU-37) | 13.25 | Total3D w/o. joint |
| 3D | Pix3D | CD | 0.0836 | MGN |
| 3D Object Detection | SUN RGB-D | AP@0.15 (10 / NYU-37) | 26.38 | Total3D joint |
| 3D Object Detection | SUN RGB-D | AP@0.15 (NYU-37) | 14.28 | Total3D joint |
| 3D Object Detection | SUN RGB-D | AP@0.15 (10 / NYU-37) | 23.32 | Total3D w/o. joint |
| 3D Object Detection | SUN RGB-D | AP@0.15 (NYU-37) | 13.25 | Total3D w/o. joint |
| 3D Shape Reconstruction | Pix3D | CD | 0.0836 | MGN |
| 2D Classification | SUN RGB-D | AP@0.15 (10 / NYU-37) | 26.38 | Total3D joint |
| 2D Classification | SUN RGB-D | AP@0.15 (NYU-37) | 14.28 | Total3D joint |
| 2D Classification | SUN RGB-D | AP@0.15 (10 / NYU-37) | 23.32 | Total3D w/o. joint |
| 2D Classification | SUN RGB-D | AP@0.15 (NYU-37) | 13.25 | Total3D w/o. joint |
| 2D Object Detection | SUN RGB-D | AP@0.15 (10 / NYU-37) | 26.38 | Total3D joint |
| 2D Object Detection | SUN RGB-D | AP@0.15 (NYU-37) | 14.28 | Total3D joint |
| 2D Object Detection | SUN RGB-D | AP@0.15 (10 / NYU-37) | 23.32 | Total3D w/o. joint |
| 2D Object Detection | SUN RGB-D | AP@0.15 (NYU-37) | 13.25 | Total3D w/o. joint |
| 16k | SUN RGB-D | AP@0.15 (10 / NYU-37) | 26.38 | Total3D joint |
| 16k | SUN RGB-D | AP@0.15 (NYU-37) | 14.28 | Total3D joint |
| 16k | SUN RGB-D | AP@0.15 (10 / NYU-37) | 23.32 | Total3D w/o. joint |
| 16k | SUN RGB-D | AP@0.15 (NYU-37) | 13.25 | Total3D w/o. joint |
| Room Layout Estimation | SUN RGB-D | Camera Pitch | 3.15 | Total3D joint |
| Room Layout Estimation | SUN RGB-D | Camera Roll | 2.09 | Total3D joint |
| Room Layout Estimation | SUN RGB-D | IoU | 59.2 | Total3D joint |
| Room Layout Estimation | SUN RGB-D | Camera Pitch | 3.68 | Total w/o. joint |
| Room Layout Estimation | SUN RGB-D | Camera Roll | 2.59 | Total w/o. joint |
| Room Layout Estimation | SUN RGB-D | IoU | 57.6 | Total w/o. joint |