Jiangmiao Pang, Kai Chen, Jianping Shi, Huajun Feng, Wanli Ouyang, Dahua Lin
Compared with model architectures, the training process, which is also crucial to the success of detectors, has received relatively less attention in object detection. In this work, we carefully revisit the standard training practice of detectors, and find that the detection performance is often limited by the imbalance during the training process, which generally consists in three levels - sample level, feature level, and objective level. To mitigate the adverse effects caused thereby, we propose Libra R-CNN, a simple but effective framework towards balanced learning for object detection. It integrates three novel components: IoU-balanced sampling, balanced feature pyramid, and balanced L1 loss, respectively for reducing the imbalance at sample, feature, and objective level. Benefitted from the overall balanced design, Libra R-CNN significantly improves the detection performance. Without bells and whistles, it achieves 2.5 points and 2.0 points higher Average Precision (AP) than FPN Faster R-CNN and RetinaNet respectively on MSCOCO.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Object Detection | COCO test-dev | AP50 | 64 | Libra R-CNN (ResNeXt-101-FPN) |
| Object Detection | COCO test-dev | AP75 | 47 | Libra R-CNN (ResNeXt-101-FPN) |
| Object Detection | COCO test-dev | APL | 54.6 | Libra R-CNN (ResNeXt-101-FPN) |
| Object Detection | COCO test-dev | APM | 45.6 | Libra R-CNN (ResNeXt-101-FPN) |
| Object Detection | COCO test-dev | APS | 25.3 | Libra R-CNN (ResNeXt-101-FPN) |
| Object Detection | COCO test-dev | box mAP | 43 | Libra R-CNN (ResNeXt-101-FPN) |
| Object Detection | COCO minival | AP50 | 59.3 | Libra R-CNN (ResNet-50 FPN) |
| Object Detection | COCO minival | AP75 | 42 | Libra R-CNN (ResNet-50 FPN) |
| Object Detection | COCO minival | APL | 50.5 | Libra R-CNN (ResNet-50 FPN) |
| Object Detection | COCO minival | APM | 42.1 | Libra R-CNN (ResNet-50 FPN) |
| Object Detection | COCO minival | APS | 22.9 | Libra R-CNN (ResNet-50 FPN) |
| Object Detection | COCO minival | box AP | 38.5 | Libra R-CNN (ResNet-50 FPN) |
| 3D | COCO test-dev | AP50 | 64 | Libra R-CNN (ResNeXt-101-FPN) |
| 3D | COCO test-dev | AP75 | 47 | Libra R-CNN (ResNeXt-101-FPN) |
| 3D | COCO test-dev | APL | 54.6 | Libra R-CNN (ResNeXt-101-FPN) |
| 3D | COCO test-dev | APM | 45.6 | Libra R-CNN (ResNeXt-101-FPN) |
| 3D | COCO test-dev | APS | 25.3 | Libra R-CNN (ResNeXt-101-FPN) |
| 3D | COCO test-dev | box mAP | 43 | Libra R-CNN (ResNeXt-101-FPN) |
| 3D | COCO minival | AP50 | 59.3 | Libra R-CNN (ResNet-50 FPN) |
| 3D | COCO minival | AP75 | 42 | Libra R-CNN (ResNet-50 FPN) |
| 3D | COCO minival | APL | 50.5 | Libra R-CNN (ResNet-50 FPN) |
| 3D | COCO minival | APM | 42.1 | Libra R-CNN (ResNet-50 FPN) |
| 3D | COCO minival | APS | 22.9 | Libra R-CNN (ResNet-50 FPN) |
| 3D | COCO minival | box AP | 38.5 | Libra R-CNN (ResNet-50 FPN) |
| 2D Classification | COCO test-dev | AP50 | 64 | Libra R-CNN (ResNeXt-101-FPN) |
| 2D Classification | COCO test-dev | AP75 | 47 | Libra R-CNN (ResNeXt-101-FPN) |
| 2D Classification | COCO test-dev | APL | 54.6 | Libra R-CNN (ResNeXt-101-FPN) |
| 2D Classification | COCO test-dev | APM | 45.6 | Libra R-CNN (ResNeXt-101-FPN) |
| 2D Classification | COCO test-dev | APS | 25.3 | Libra R-CNN (ResNeXt-101-FPN) |
| 2D Classification | COCO test-dev | box mAP | 43 | Libra R-CNN (ResNeXt-101-FPN) |
| 2D Classification | COCO minival | AP50 | 59.3 | Libra R-CNN (ResNet-50 FPN) |
| 2D Classification | COCO minival | AP75 | 42 | Libra R-CNN (ResNet-50 FPN) |
| 2D Classification | COCO minival | APL | 50.5 | Libra R-CNN (ResNet-50 FPN) |
| 2D Classification | COCO minival | APM | 42.1 | Libra R-CNN (ResNet-50 FPN) |
| 2D Classification | COCO minival | APS | 22.9 | Libra R-CNN (ResNet-50 FPN) |
| 2D Classification | COCO minival | box AP | 38.5 | Libra R-CNN (ResNet-50 FPN) |
| 2D Object Detection | COCO test-dev | AP50 | 64 | Libra R-CNN (ResNeXt-101-FPN) |
| 2D Object Detection | COCO test-dev | AP75 | 47 | Libra R-CNN (ResNeXt-101-FPN) |
| 2D Object Detection | COCO test-dev | APL | 54.6 | Libra R-CNN (ResNeXt-101-FPN) |
| 2D Object Detection | COCO test-dev | APM | 45.6 | Libra R-CNN (ResNeXt-101-FPN) |
| 2D Object Detection | COCO test-dev | APS | 25.3 | Libra R-CNN (ResNeXt-101-FPN) |
| 2D Object Detection | COCO test-dev | box mAP | 43 | Libra R-CNN (ResNeXt-101-FPN) |
| 2D Object Detection | COCO minival | AP50 | 59.3 | Libra R-CNN (ResNet-50 FPN) |
| 2D Object Detection | COCO minival | AP75 | 42 | Libra R-CNN (ResNet-50 FPN) |
| 2D Object Detection | COCO minival | APL | 50.5 | Libra R-CNN (ResNet-50 FPN) |
| 2D Object Detection | COCO minival | APM | 42.1 | Libra R-CNN (ResNet-50 FPN) |
| 2D Object Detection | COCO minival | APS | 22.9 | Libra R-CNN (ResNet-50 FPN) |
| 2D Object Detection | COCO minival | box AP | 38.5 | Libra R-CNN (ResNet-50 FPN) |
| 16k | COCO test-dev | AP50 | 64 | Libra R-CNN (ResNeXt-101-FPN) |
| 16k | COCO test-dev | AP75 | 47 | Libra R-CNN (ResNeXt-101-FPN) |
| 16k | COCO test-dev | APL | 54.6 | Libra R-CNN (ResNeXt-101-FPN) |
| 16k | COCO test-dev | APM | 45.6 | Libra R-CNN (ResNeXt-101-FPN) |
| 16k | COCO test-dev | APS | 25.3 | Libra R-CNN (ResNeXt-101-FPN) |
| 16k | COCO test-dev | box mAP | 43 | Libra R-CNN (ResNeXt-101-FPN) |
| 16k | COCO minival | AP50 | 59.3 | Libra R-CNN (ResNet-50 FPN) |
| 16k | COCO minival | AP75 | 42 | Libra R-CNN (ResNet-50 FPN) |
| 16k | COCO minival | APL | 50.5 | Libra R-CNN (ResNet-50 FPN) |
| 16k | COCO minival | APM | 42.1 | Libra R-CNN (ResNet-50 FPN) |
| 16k | COCO minival | APS | 22.9 | Libra R-CNN (ResNet-50 FPN) |
| 16k | COCO minival | box AP | 38.5 | Libra R-CNN (ResNet-50 FPN) |