Fa-Ting Hong, Jia-Chang Feng, Dan Xu, Ying Shan, Wei-Shi Zheng
Weakly supervised temporal action localization (WS-TAL) is a challenging task that aims to localize action instances in the given video with video-level categorical supervision. Both appearance and motion features are used in previous works, while they do not utilize them in a proper way but apply simple concatenation or score-level fusion. In this work, we argue that the features extracted from the pretrained extractor, e.g., I3D, are not the WS-TALtask-specific features, thus the feature re-calibration is needed for reducing the task-irrelevant information redundancy. Therefore, we propose a cross-modal consensus network (CO2-Net) to tackle this problem. In CO2-Net, we mainly introduce two identical proposed cross-modal consensus modules (CCM) that design a cross-modal attention mechanism to filter out the task-irrelevant information redundancy using the global information from the main modality and the cross-modal local information of the auxiliary modality. Moreover, we treat the attention weights derived from each CCMas the pseudo targets of the attention weights derived from another CCM to maintain the consistency between the predictions derived from two CCMs, forming a mutual learning manner. Finally, we conduct extensive experiments on two common used temporal action localization datasets, THUMOS14 and ActivityNet1.2, to verify our method and achieve the state-of-the-art results. The experimental results show that our proposed cross-modal consensus module can produce more representative features for temporal action localization.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Video | THUMOS’14 | mAP IOU@0.1 | 70.1 | CO2-Net |
| Video | THUMOS’14 | mAP IOU@0.2 | 63.6 | CO2-Net |
| Video | THUMOS’14 | mAP IOU@0.3 | 54.5 | CO2-Net |
| Video | THUMOS’14 | mAP IOU@0.4 | 45.7 | CO2-Net |
| Video | THUMOS’14 | mAP IOU@0.5 | 38.3 | CO2-Net |
| Video | THUMOS’14 | mAP IOU@0.6 | 26.4 | CO2-Net |
| Video | THUMOS’14 | mAP IOU@0.7 | 13.4 | CO2-Net |
| Video | THUMOS’14 | mAP IOU@0.8 | 6.9 | CO2-Net |
| Video | THUMOS’14 | mAP IOU@0.9 | 2 | CO2-Net |
| Video | THUMOS’14 | mAP@AVG(0.1:0.9) | 35.7 | CO2-Net |
| Video | THUMOS’14 | mAP IOU@0.5 | 38.3 | CO2-Net |
| Video | THUMOS’14 | mAP@AVG(0.1:0.9) | 35.7 | CO2-Net |
| Video | THUMOS 2014 | mAP@0.1:0.5 | 54.4 | CO2-Net |
| Video | THUMOS 2014 | mAP@0.1:0.7 | 44.6 | CO2-Net |
| Video | THUMOS 2014 | mAP@0.5 | 38.3 | CO2-Net |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.1 | 70.1 | CO2-Net |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.2 | 63.6 | CO2-Net |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.3 | 54.5 | CO2-Net |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.4 | 45.7 | CO2-Net |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.5 | 38.3 | CO2-Net |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.6 | 26.4 | CO2-Net |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.7 | 13.4 | CO2-Net |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.8 | 6.9 | CO2-Net |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.9 | 2 | CO2-Net |
| Temporal Action Localization | THUMOS’14 | mAP@AVG(0.1:0.9) | 35.7 | CO2-Net |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.5 | 38.3 | CO2-Net |
| Temporal Action Localization | THUMOS’14 | mAP@AVG(0.1:0.9) | 35.7 | CO2-Net |
| Temporal Action Localization | THUMOS 2014 | mAP@0.1:0.5 | 54.4 | CO2-Net |
| Temporal Action Localization | THUMOS 2014 | mAP@0.1:0.7 | 44.6 | CO2-Net |
| Temporal Action Localization | THUMOS 2014 | mAP@0.5 | 38.3 | CO2-Net |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.1 | 70.1 | CO2-Net |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.2 | 63.6 | CO2-Net |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.3 | 54.5 | CO2-Net |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.4 | 45.7 | CO2-Net |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.5 | 38.3 | CO2-Net |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.6 | 26.4 | CO2-Net |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.7 | 13.4 | CO2-Net |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.8 | 6.9 | CO2-Net |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.9 | 2 | CO2-Net |
| Zero-Shot Learning | THUMOS’14 | mAP@AVG(0.1:0.9) | 35.7 | CO2-Net |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.5 | 38.3 | CO2-Net |
| Zero-Shot Learning | THUMOS’14 | mAP@AVG(0.1:0.9) | 35.7 | CO2-Net |
| Zero-Shot Learning | THUMOS 2014 | mAP@0.1:0.5 | 54.4 | CO2-Net |
| Zero-Shot Learning | THUMOS 2014 | mAP@0.1:0.7 | 44.6 | CO2-Net |
| Zero-Shot Learning | THUMOS 2014 | mAP@0.5 | 38.3 | CO2-Net |
| Action Localization | THUMOS’14 | mAP IOU@0.1 | 70.1 | CO2-Net |
| Action Localization | THUMOS’14 | mAP IOU@0.2 | 63.6 | CO2-Net |
| Action Localization | THUMOS’14 | mAP IOU@0.3 | 54.5 | CO2-Net |
| Action Localization | THUMOS’14 | mAP IOU@0.4 | 45.7 | CO2-Net |
| Action Localization | THUMOS’14 | mAP IOU@0.5 | 38.3 | CO2-Net |
| Action Localization | THUMOS’14 | mAP IOU@0.6 | 26.4 | CO2-Net |
| Action Localization | THUMOS’14 | mAP IOU@0.7 | 13.4 | CO2-Net |
| Action Localization | THUMOS’14 | mAP IOU@0.8 | 6.9 | CO2-Net |
| Action Localization | THUMOS’14 | mAP IOU@0.9 | 2 | CO2-Net |
| Action Localization | THUMOS’14 | mAP@AVG(0.1:0.9) | 35.7 | CO2-Net |
| Action Localization | THUMOS’14 | mAP IOU@0.5 | 38.3 | CO2-Net |
| Action Localization | THUMOS’14 | mAP@AVG(0.1:0.9) | 35.7 | CO2-Net |
| Action Localization | THUMOS 2014 | mAP@0.1:0.5 | 54.4 | CO2-Net |
| Action Localization | THUMOS 2014 | mAP@0.1:0.7 | 44.6 | CO2-Net |
| Action Localization | THUMOS 2014 | mAP@0.5 | 38.3 | CO2-Net |
| Weakly Supervised Action Localization | THUMOS 2014 | mAP@0.1:0.5 | 54.4 | CO2-Net |
| Weakly Supervised Action Localization | THUMOS 2014 | mAP@0.1:0.7 | 44.6 | CO2-Net |
| Weakly Supervised Action Localization | THUMOS 2014 | mAP@0.5 | 38.3 | CO2-Net |
| Weakly-supervised Temporal Action Localization | THUMOS’14 | mAP IOU@0.1 | 70.1 | CO2-Net |
| Weakly-supervised Temporal Action Localization | THUMOS’14 | mAP IOU@0.2 | 63.6 | CO2-Net |
| Weakly-supervised Temporal Action Localization | THUMOS’14 | mAP IOU@0.3 | 54.5 | CO2-Net |
| Weakly-supervised Temporal Action Localization | THUMOS’14 | mAP IOU@0.4 | 45.7 | CO2-Net |
| Weakly-supervised Temporal Action Localization | THUMOS’14 | mAP IOU@0.5 | 38.3 | CO2-Net |
| Weakly-supervised Temporal Action Localization | THUMOS’14 | mAP IOU@0.6 | 26.4 | CO2-Net |
| Weakly-supervised Temporal Action Localization | THUMOS’14 | mAP IOU@0.7 | 13.4 | CO2-Net |
| Weakly-supervised Temporal Action Localization | THUMOS’14 | mAP IOU@0.8 | 6.9 | CO2-Net |
| Weakly-supervised Temporal Action Localization | THUMOS’14 | mAP IOU@0.9 | 2 | CO2-Net |
| Weakly-supervised Temporal Action Localization | THUMOS’14 | mAP@AVG(0.1:0.9) | 35.7 | CO2-Net |
| Weakly-supervised Temporal Action Localization | THUMOS’14 | mAP IOU@0.5 | 38.3 | CO2-Net |
| Weakly-supervised Temporal Action Localization | THUMOS’14 | mAP@AVG(0.1:0.9) | 35.7 | CO2-Net |