Jiyang Gao, Zhenheng Yang, Chen Sun, Kan Chen, Ram Nevatia
Temporal Action Proposal (TAP) generation is an important problem, as fast and accurate extraction of semantically important (e.g. human actions) segments from untrimmed videos is an important step for large-scale video analysis. We propose a novel Temporal Unit Regression Network (TURN) model. There are two salient aspects of TURN: (1) TURN jointly predicts action proposals and refines the temporal boundaries by temporal coordinate regression; (2) Fast computation is enabled by unit feature reuse: a long untrimmed video is decomposed into video units, which are reused as basic building blocks of temporal proposals. TURN outperforms the state-of-the-art methods under average recall (AR) by a large margin on THUMOS-14 and ActivityNet datasets, and runs at over 880 frames per second (FPS) on a TITAN X GPU. We further apply TURN as a proposal generation stage for existing temporal action localization pipelines, it outperforms state-of-the-art performance on THUMOS-14 and ActivityNet.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Video | THUMOS’14 | mAP IOU@0.1 | 54 | TURN-FL-16 + S-CNN |
| Video | THUMOS’14 | mAP IOU@0.2 | 50.9 | TURN-FL-16 + S-CNN |
| Video | THUMOS’14 | mAP IOU@0.3 | 44.1 | TURN-FL-16 + S-CNN |
| Video | THUMOS’14 | mAP IOU@0.4 | 34.9 | TURN-FL-16 + S-CNN |
| Video | THUMOS’14 | mAP IOU@0.5 | 25.6 | TURN-FL-16 + S-CNN |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.1 | 54 | TURN-FL-16 + S-CNN |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.2 | 50.9 | TURN-FL-16 + S-CNN |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.3 | 44.1 | TURN-FL-16 + S-CNN |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.4 | 34.9 | TURN-FL-16 + S-CNN |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.5 | 25.6 | TURN-FL-16 + S-CNN |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.1 | 54 | TURN-FL-16 + S-CNN |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.2 | 50.9 | TURN-FL-16 + S-CNN |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.3 | 44.1 | TURN-FL-16 + S-CNN |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.4 | 34.9 | TURN-FL-16 + S-CNN |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.5 | 25.6 | TURN-FL-16 + S-CNN |
| Activity Recognition | THUMOS’14 | mAP@0.3 | 46.3 | TURN |
| Activity Recognition | THUMOS’14 | mAP@0.4 | 35.3 | TURN |
| Activity Recognition | THUMOS’14 | mAP@0.5 | 24.5 | TURN |
| Action Localization | THUMOS’14 | mAP IOU@0.1 | 54 | TURN-FL-16 + S-CNN |
| Action Localization | THUMOS’14 | mAP IOU@0.2 | 50.9 | TURN-FL-16 + S-CNN |
| Action Localization | THUMOS’14 | mAP IOU@0.3 | 44.1 | TURN-FL-16 + S-CNN |
| Action Localization | THUMOS’14 | mAP IOU@0.4 | 34.9 | TURN-FL-16 + S-CNN |
| Action Localization | THUMOS’14 | mAP IOU@0.5 | 25.6 | TURN-FL-16 + S-CNN |
| Action Recognition | THUMOS’14 | mAP@0.3 | 46.3 | TURN |
| Action Recognition | THUMOS’14 | mAP@0.4 | 35.3 | TURN |
| Action Recognition | THUMOS’14 | mAP@0.5 | 24.5 | TURN |