Shubham Shrivastava, Punarjay Chakravarty
We introduce a method for 3D object detection using a single monocular image. Starting from a synthetic dataset, we pre-train an RGB-to-Depth Auto-Encoder (AE). The embedding learnt from this AE is then used to train a 3D Object Detector (3DOD) CNN which is used to regress the parameters of 3D object poses after the encoder from the AE generates a latent embedding from the RGB image. We show that we can pre-train the AE using paired RGB and depth images from simulation data once and subsequently only train the 3DOD network using real data, comprising of RGB images and 3D object pose labels (without the requirement of dense depth). Our 3DOD network utilizes a particular `cubification' of 3D space around the camera, where each cuboid is tasked with predicting N object poses, along with their class and confidence values. The AE pre-training and this method of dividing the 3D space around the camera into cuboids give our method its name - CubifAE-3D. We demonstrate results for monocular 3D object detection in the Autonomous Vehicle (AV) use-case with the Virtual KITTI 2 and the KITTI datasets.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Object Detection | KITTI Cars Moderate | AP Medium | 7.94 | CubifAE-3D |
| Object Detection | KITTI Pedestrian Hard | AP Hard | 4.82 | CubifAE-3D |
| Object Detection | KITTI Pedestrians Moderate val | AP Medium | 5.43 | CubifAE-3D |
| Object Detection | KITTI Cars Hard | AP Hard | 6.42 | CubifAE-3D |
| Object Detection | Virtual KITTI 2 | mAP@0.3 | 86.6 | CubifAE-3D |
| Object Detection | Virtual KITTI 2 | mAP@0.5 | 66.7 | CubifAE-3D |
| 3D | KITTI Cars Moderate | AP Medium | 7.94 | CubifAE-3D |
| 3D | KITTI Pedestrian Hard | AP Hard | 4.82 | CubifAE-3D |
| 3D | KITTI Pedestrians Moderate val | AP Medium | 5.43 | CubifAE-3D |
| 3D | KITTI Cars Hard | AP Hard | 6.42 | CubifAE-3D |
| 3D | Virtual KITTI 2 | mAP@0.3 | 86.6 | CubifAE-3D |
| 3D | Virtual KITTI 2 | mAP@0.5 | 66.7 | CubifAE-3D |
| 3D Object Detection | KITTI Cars Moderate | AP Medium | 7.94 | CubifAE-3D |
| 3D Object Detection | KITTI Pedestrian Hard | AP Hard | 4.82 | CubifAE-3D |
| 3D Object Detection | KITTI Pedestrians Moderate val | AP Medium | 5.43 | CubifAE-3D |
| 3D Object Detection | KITTI Cars Hard | AP Hard | 6.42 | CubifAE-3D |
| 3D Object Detection | Virtual KITTI 2 | mAP@0.3 | 86.6 | CubifAE-3D |
| 3D Object Detection | Virtual KITTI 2 | mAP@0.5 | 66.7 | CubifAE-3D |
| 2D Classification | KITTI Cars Moderate | AP Medium | 7.94 | CubifAE-3D |
| 2D Classification | KITTI Pedestrian Hard | AP Hard | 4.82 | CubifAE-3D |
| 2D Classification | KITTI Pedestrians Moderate val | AP Medium | 5.43 | CubifAE-3D |
| 2D Classification | KITTI Cars Hard | AP Hard | 6.42 | CubifAE-3D |
| 2D Classification | Virtual KITTI 2 | mAP@0.3 | 86.6 | CubifAE-3D |
| 2D Classification | Virtual KITTI 2 | mAP@0.5 | 66.7 | CubifAE-3D |
| 2D Object Detection | KITTI Cars Moderate | AP Medium | 7.94 | CubifAE-3D |
| 2D Object Detection | KITTI Pedestrian Hard | AP Hard | 4.82 | CubifAE-3D |
| 2D Object Detection | KITTI Pedestrians Moderate val | AP Medium | 5.43 | CubifAE-3D |
| 2D Object Detection | KITTI Cars Hard | AP Hard | 6.42 | CubifAE-3D |
| 2D Object Detection | Virtual KITTI 2 | mAP@0.3 | 86.6 | CubifAE-3D |
| 2D Object Detection | Virtual KITTI 2 | mAP@0.5 | 66.7 | CubifAE-3D |
| 16k | KITTI Cars Moderate | AP Medium | 7.94 | CubifAE-3D |
| 16k | KITTI Pedestrian Hard | AP Hard | 4.82 | CubifAE-3D |
| 16k | KITTI Pedestrians Moderate val | AP Medium | 5.43 | CubifAE-3D |
| 16k | KITTI Cars Hard | AP Hard | 6.42 | CubifAE-3D |
| 16k | Virtual KITTI 2 | mAP@0.3 | 86.6 | CubifAE-3D |
| 16k | Virtual KITTI 2 | mAP@0.5 | 66.7 | CubifAE-3D |