Guo Chen, Yin-Dong Zheng, LiMin Wang, Tong Lu
Temporal action detection aims to locate the boundaries of action in the video. The current method based on boundary matching enumerates and calculates all possible boundary matchings to generate proposals. However, these methods neglect the long-range context aggregation in boundary prediction. At the same time, due to the similar semantics of adjacent matchings, local semantic aggregation of densely-generated matchings cannot improve semantic richness and discrimination. In this paper, we propose the end-to-end proposal generation method named Dual Context Aggregation Network (DCAN) to aggregate context on two levels, namely, boundary level and proposal level, for generating high-quality action proposals, thereby improving the performance of temporal action detection. Specifically, we design the Multi-Path Temporal Context Aggregation (MTCA) to achieve smooth context aggregation on boundary level and precise evaluation of boundaries. For matching evaluation, Coarse-to-fine Matching (CFM) is designed to aggregate context on the proposal level and refine the matching map from coarse to fine. We conduct extensive experiments on ActivityNet v1.3 and THUMOS-14. DCAN obtains an average mAP of 35.39% on ActivityNet v1.3 and reaches mAP 54.14% at IoU@0.5 on THUMOS-14, which demonstrates DCAN can generate high-quality proposals and achieve state-of-the-art performance. We release the code at https://github.com/cg1177/DCAN.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Video | ActivityNet-1.3 | mAP | 35.39 | DCAN (TSN features) |
| Video | ActivityNet-1.3 | mAP IOU@0.5 | 51.78 | DCAN (TSN features) |
| Video | ActivityNet-1.3 | mAP IOU@0.75 | 35.98 | DCAN (TSN features) |
| Video | ActivityNet-1.3 | mAP IOU@0.95 | 9.45 | DCAN (TSN features) |
| Video | THUMOS’14 | Avg mAP (0.3:0.7) | 52.3 | DCAN (TSN features) |
| Video | THUMOS’14 | mAP IOU@0.3 | 68.2 | DCAN (TSN features) |
| Video | THUMOS’14 | mAP IOU@0.4 | 62.7 | DCAN (TSN features) |
| Video | THUMOS’14 | mAP IOU@0.5 | 54.1 | DCAN (TSN features) |
| Video | THUMOS’14 | mAP IOU@0.6 | 43.9 | DCAN (TSN features) |
| Video | THUMOS’14 | mAP IOU@0.7 | 32.6 | DCAN (TSN features) |
| Temporal Action Localization | ActivityNet-1.3 | mAP | 35.39 | DCAN (TSN features) |
| Temporal Action Localization | ActivityNet-1.3 | mAP IOU@0.5 | 51.78 | DCAN (TSN features) |
| Temporal Action Localization | ActivityNet-1.3 | mAP IOU@0.75 | 35.98 | DCAN (TSN features) |
| Temporal Action Localization | ActivityNet-1.3 | mAP IOU@0.95 | 9.45 | DCAN (TSN features) |
| Temporal Action Localization | THUMOS’14 | Avg mAP (0.3:0.7) | 52.3 | DCAN (TSN features) |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.3 | 68.2 | DCAN (TSN features) |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.4 | 62.7 | DCAN (TSN features) |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.5 | 54.1 | DCAN (TSN features) |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.6 | 43.9 | DCAN (TSN features) |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.7 | 32.6 | DCAN (TSN features) |
| Zero-Shot Learning | ActivityNet-1.3 | mAP | 35.39 | DCAN (TSN features) |
| Zero-Shot Learning | ActivityNet-1.3 | mAP IOU@0.5 | 51.78 | DCAN (TSN features) |
| Zero-Shot Learning | ActivityNet-1.3 | mAP IOU@0.75 | 35.98 | DCAN (TSN features) |
| Zero-Shot Learning | ActivityNet-1.3 | mAP IOU@0.95 | 9.45 | DCAN (TSN features) |
| Zero-Shot Learning | THUMOS’14 | Avg mAP (0.3:0.7) | 52.3 | DCAN (TSN features) |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.3 | 68.2 | DCAN (TSN features) |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.4 | 62.7 | DCAN (TSN features) |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.5 | 54.1 | DCAN (TSN features) |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.6 | 43.9 | DCAN (TSN features) |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.7 | 32.6 | DCAN (TSN features) |
| Action Localization | ActivityNet-1.3 | mAP | 35.39 | DCAN (TSN features) |
| Action Localization | ActivityNet-1.3 | mAP IOU@0.5 | 51.78 | DCAN (TSN features) |
| Action Localization | ActivityNet-1.3 | mAP IOU@0.75 | 35.98 | DCAN (TSN features) |
| Action Localization | ActivityNet-1.3 | mAP IOU@0.95 | 9.45 | DCAN (TSN features) |
| Action Localization | THUMOS’14 | Avg mAP (0.3:0.7) | 52.3 | DCAN (TSN features) |
| Action Localization | THUMOS’14 | mAP IOU@0.3 | 68.2 | DCAN (TSN features) |
| Action Localization | THUMOS’14 | mAP IOU@0.4 | 62.7 | DCAN (TSN features) |
| Action Localization | THUMOS’14 | mAP IOU@0.5 | 54.1 | DCAN (TSN features) |
| Action Localization | THUMOS’14 | mAP IOU@0.6 | 43.9 | DCAN (TSN features) |
| Action Localization | THUMOS’14 | mAP IOU@0.7 | 32.6 | DCAN (TSN features) |