Fan Ma, Linchao Zhu, Yi Yang, Shengxin Zha, Gourab Kundu, Matt Feiszli, Zheng Shou
In this paper, we study an intermediate form of supervision, i.e., single-frame supervision, for temporal action localization (TAL). To obtain the single-frame supervision, the annotators are asked to identify only a single frame within the temporal window of an action. This can significantly reduce the labor cost of obtaining full supervision which requires annotating the action boundary. Compared to the weak supervision that only annotates the video-level label, the single-frame supervision introduces extra temporal action signals while maintaining low annotation overhead. To make full use of such single-frame supervision, we propose a unified system called SF-Net. First, we propose to predict an actionness score for each video frame. Along with a typical category score, the actionness score can provide comprehensive information about the occurrence of a potential action and aid the temporal boundary refinement during inference. Second, we mine pseudo action and background frames based on the single-frame annotations. We identify pseudo action frames by adaptively expanding each annotated single frame to its nearby, contextual frames and we mine pseudo background frames from all the unannotated frames across multiple videos. Together with the ground-truth labeled frames, these pseudo-labeled frames are further used for training the classifier. In extensive experiments on THUMOS14, GTEA, and BEOID, SF-Net significantly improves upon state-of-the-art weakly-supervised methods in terms of both segment localization and single-frame localization. Notably, SF-Net achieves comparable results to its fully-supervised counterpart which requires much more resource intensive annotations. The code is available at https://github.com/Flowerfan/SF-Net.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Video | GTEA | mAP@0.1:0.7 | 31 | SF-Net |
| Video | GTEA | mAP@0.5 | 19.3 | SF-Net |
| Video | BEOID | mAP@0.1:0.7 | 30.1 | SF-Net |
| Video | BEOID | mAP@0.5 | 16.7 | SF-Net |
| Video | THUMOS 2014 | mAP@0.1:0.5 | 51.2 | SF-Net |
| Video | THUMOS 2014 | mAP@0.1:0.7 | 41.2 | SF-Net |
| Video | THUMOS 2014 | mAP@0.5 | 30.5 | SF-Net |
| Video | ActivityNet-1.2 | Mean mAP | 22.8 | SF-Net |
| Video | ActivityNet-1.2 | mAP@0.5 | 37.8 | SF-Net |
| Temporal Action Localization | GTEA | mAP@0.1:0.7 | 31 | SF-Net |
| Temporal Action Localization | GTEA | mAP@0.5 | 19.3 | SF-Net |
| Temporal Action Localization | BEOID | mAP@0.1:0.7 | 30.1 | SF-Net |
| Temporal Action Localization | BEOID | mAP@0.5 | 16.7 | SF-Net |
| Temporal Action Localization | THUMOS 2014 | mAP@0.1:0.5 | 51.2 | SF-Net |
| Temporal Action Localization | THUMOS 2014 | mAP@0.1:0.7 | 41.2 | SF-Net |
| Temporal Action Localization | THUMOS 2014 | mAP@0.5 | 30.5 | SF-Net |
| Temporal Action Localization | ActivityNet-1.2 | Mean mAP | 22.8 | SF-Net |
| Temporal Action Localization | ActivityNet-1.2 | mAP@0.5 | 37.8 | SF-Net |
| Zero-Shot Learning | GTEA | mAP@0.1:0.7 | 31 | SF-Net |
| Zero-Shot Learning | GTEA | mAP@0.5 | 19.3 | SF-Net |
| Zero-Shot Learning | BEOID | mAP@0.1:0.7 | 30.1 | SF-Net |
| Zero-Shot Learning | BEOID | mAP@0.5 | 16.7 | SF-Net |
| Zero-Shot Learning | THUMOS 2014 | mAP@0.1:0.5 | 51.2 | SF-Net |
| Zero-Shot Learning | THUMOS 2014 | mAP@0.1:0.7 | 41.2 | SF-Net |
| Zero-Shot Learning | THUMOS 2014 | mAP@0.5 | 30.5 | SF-Net |
| Zero-Shot Learning | ActivityNet-1.2 | Mean mAP | 22.8 | SF-Net |
| Zero-Shot Learning | ActivityNet-1.2 | mAP@0.5 | 37.8 | SF-Net |
| Action Localization | GTEA | mAP@0.1:0.7 | 31 | SF-Net |
| Action Localization | GTEA | mAP@0.5 | 19.3 | SF-Net |
| Action Localization | BEOID | mAP@0.1:0.7 | 30.1 | SF-Net |
| Action Localization | BEOID | mAP@0.5 | 16.7 | SF-Net |
| Action Localization | THUMOS 2014 | mAP@0.1:0.5 | 51.2 | SF-Net |
| Action Localization | THUMOS 2014 | mAP@0.1:0.7 | 41.2 | SF-Net |
| Action Localization | THUMOS 2014 | mAP@0.5 | 30.5 | SF-Net |
| Action Localization | ActivityNet-1.2 | Mean mAP | 22.8 | SF-Net |
| Action Localization | ActivityNet-1.2 | mAP@0.5 | 37.8 | SF-Net |
| Weakly Supervised Action Localization | GTEA | mAP@0.1:0.7 | 31 | SF-Net |
| Weakly Supervised Action Localization | GTEA | mAP@0.5 | 19.3 | SF-Net |
| Weakly Supervised Action Localization | BEOID | mAP@0.1:0.7 | 30.1 | SF-Net |
| Weakly Supervised Action Localization | BEOID | mAP@0.5 | 16.7 | SF-Net |
| Weakly Supervised Action Localization | THUMOS 2014 | mAP@0.1:0.5 | 51.2 | SF-Net |
| Weakly Supervised Action Localization | THUMOS 2014 | mAP@0.1:0.7 | 41.2 | SF-Net |
| Weakly Supervised Action Localization | THUMOS 2014 | mAP@0.5 | 30.5 | SF-Net |
| Weakly Supervised Action Localization | ActivityNet-1.2 | Mean mAP | 22.8 | SF-Net |
| Weakly Supervised Action Localization | ActivityNet-1.2 | mAP@0.5 | 37.8 | SF-Net |