Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, Tian Xia
This paper aims at high-accuracy 3D object detection in autonomous driving scenario. We propose Multi-View 3D networks (MV3D), a sensory-fusion framework that takes both LIDAR point cloud and RGB images as input and predicts oriented 3D bounding boxes. We encode the sparse 3D point cloud with a compact multi-view representation. The network is composed of two subnetworks: one for 3D object proposal generation and another for multi-view feature fusion. The proposal network generates 3D candidate boxes efficiently from the bird's eye view representation of 3D point cloud. We design a deep fusion scheme to combine region-wise features from multiple views and enable interactions between intermediate layers of different paths. Experiments on the challenging KITTI benchmark show that our approach outperforms the state-of-the-art by around 25% and 30% AP on the tasks of 3D localization and 3D detection. In addition, for 2D detection, our approach obtains 10.3% higher AP than the state-of-the-art on the hard data among the LIDAR-based methods.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Object Detection | KITTI Cars Hard val | AP | 56.56 | MV3D |
| Object Detection | KITTI Cars Moderate val | AP | 62.68 | MV3D |
| Object Detection | KITTI Cars Easy val | AP | 71.29 | MV3D |
| Object Detection | KITTI Cars Easy val | AP | 71.19 | MV3D (LiDAR) |
| 3D | KITTI Cars Hard val | AP | 56.56 | MV3D |
| 3D | KITTI Cars Moderate val | AP | 62.68 | MV3D |
| 3D | KITTI Cars Easy val | AP | 71.29 | MV3D |
| 3D | KITTI Cars Easy val | AP | 71.19 | MV3D (LiDAR) |
| Birds Eye View Object Detection | KITTI Cars Hard val | AP | 76.33 | MV (BV+FV) |
| Birds Eye View Object Detection | KITTI Cars Moderate val | AP | 77.32 | MV (BV+FV) |
| Birds Eye View Object Detection | KITTI Cars Easy val | AP | 86.18 | MV (BV+FV) |
| 3D Object Detection | KITTI Cars Hard val | AP | 56.56 | MV3D |
| 3D Object Detection | KITTI Cars Moderate val | AP | 62.68 | MV3D |
| 3D Object Detection | KITTI Cars Easy val | AP | 71.29 | MV3D |
| 3D Object Detection | KITTI Cars Easy val | AP | 71.19 | MV3D (LiDAR) |
| 2D Classification | KITTI Cars Hard val | AP | 56.56 | MV3D |
| 2D Classification | KITTI Cars Moderate val | AP | 62.68 | MV3D |
| 2D Classification | KITTI Cars Easy val | AP | 71.29 | MV3D |
| 2D Classification | KITTI Cars Easy val | AP | 71.19 | MV3D (LiDAR) |
| 2D Object Detection | KITTI Cars Hard val | AP | 56.56 | MV3D |
| 2D Object Detection | KITTI Cars Moderate val | AP | 62.68 | MV3D |
| 2D Object Detection | KITTI Cars Easy val | AP | 71.29 | MV3D |
| 2D Object Detection | KITTI Cars Easy val | AP | 71.19 | MV3D (LiDAR) |
| 16k | KITTI Cars Hard val | AP | 56.56 | MV3D |
| 16k | KITTI Cars Moderate val | AP | 62.68 | MV3D |
| 16k | KITTI Cars Easy val | AP | 71.29 | MV3D |
| 16k | KITTI Cars Easy val | AP | 71.19 | MV3D (LiDAR) |