Xiang Wang, Zhiwu Qing, Ziyuan Huang, Yutong Feng, Shiwei Zhang, Jianwen Jiang, Mingqian Tang, Changxin Gao, Nong Sang
This technical report presents our solution for temporal action detection task in AcitivityNet Challenge 2021. The purpose of this task is to locate and identify actions of interest in long untrimmed videos. The crucial challenge of the task comes from that the temporal duration of action varies dramatically, and the target actions are typically embedded in a background of irrelevant activities. Our solution builds on BMN, and mainly contains three steps: 1) action classification and feature encoding by Slowfast, CSN and ViViT; 2) proposal generation. We improve BMN by embedding the proposed Proposal Relation Network (PRN), by which we can generate proposals of high quality; 3) action detection. We calculate the detection results by assigning the proposals with corresponding classification results. Finally, we ensemble the results under different settings and achieve 44.7% on the test set, which improves the champion result in ActivityNet 2020 by 1.9% in terms of average mAP.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Video | ActivityNet-1.3 | mAP | 42 | PRN+BMN (ensemble) |
| Video | ActivityNet-1.3 | mAP IOU@0.5 | 59.7 | PRN+BMN (ensemble) |
| Video | ActivityNet-1.3 | mAP | 39.4 | PRN (CSN) |
| Video | ActivityNet-1.3 | mAP IOU@0.5 | 57.9 | PRN (CSN) |
| Video | ActivityNet-1.3 | mAP | 37.5 | PRN (ViViT) |
| Video | ActivityNet-1.3 | mAP IOU@0.5 | 55.5 | PRN (ViViT) |
| Temporal Action Localization | ActivityNet-1.3 | mAP | 42 | PRN+BMN (ensemble) |
| Temporal Action Localization | ActivityNet-1.3 | mAP IOU@0.5 | 59.7 | PRN+BMN (ensemble) |
| Temporal Action Localization | ActivityNet-1.3 | mAP | 39.4 | PRN (CSN) |
| Temporal Action Localization | ActivityNet-1.3 | mAP IOU@0.5 | 57.9 | PRN (CSN) |
| Temporal Action Localization | ActivityNet-1.3 | mAP | 37.5 | PRN (ViViT) |
| Temporal Action Localization | ActivityNet-1.3 | mAP IOU@0.5 | 55.5 | PRN (ViViT) |
| Zero-Shot Learning | ActivityNet-1.3 | mAP | 42 | PRN+BMN (ensemble) |
| Zero-Shot Learning | ActivityNet-1.3 | mAP IOU@0.5 | 59.7 | PRN+BMN (ensemble) |
| Zero-Shot Learning | ActivityNet-1.3 | mAP | 39.4 | PRN (CSN) |
| Zero-Shot Learning | ActivityNet-1.3 | mAP IOU@0.5 | 57.9 | PRN (CSN) |
| Zero-Shot Learning | ActivityNet-1.3 | mAP | 37.5 | PRN (ViViT) |
| Zero-Shot Learning | ActivityNet-1.3 | mAP IOU@0.5 | 55.5 | PRN (ViViT) |
| Action Localization | ActivityNet-1.3 | mAP | 42 | PRN+BMN (ensemble) |
| Action Localization | ActivityNet-1.3 | mAP IOU@0.5 | 59.7 | PRN+BMN (ensemble) |
| Action Localization | ActivityNet-1.3 | mAP | 39.4 | PRN (CSN) |
| Action Localization | ActivityNet-1.3 | mAP IOU@0.5 | 57.9 | PRN (CSN) |
| Action Localization | ActivityNet-1.3 | mAP | 37.5 | PRN (ViViT) |
| Action Localization | ActivityNet-1.3 | mAP IOU@0.5 | 55.5 | PRN (ViViT) |