Xiaolong Liu, Yao Hu, Song Bai, Fei Ding, Xiang Bai, Philip H. S. Torr
Current developments in temporal event or action localization usually target actions captured by a single camera. However, extensive events or actions in the wild may be captured as a sequence of shots by multiple cameras at different positions. In this paper, we propose a new and challenging task called multi-shot temporal event localization, and accordingly, collect a large scale dataset called MUlti-Shot EventS (MUSES). MUSES has 31,477 event instances for a total of 716 video hours. The core nature of MUSES is the frequent shot cuts, for an average of 19 shots per instance and 176 shots per video, which induces large intrainstance variations. Our comprehensive evaluations show that the state-of-the-art method in temporal action localization only achieves an mAP of 13.1% at IoU=0.5. As a minor contribution, we present a simple baseline approach for handling the intra-instance variations, which reports an mAP of 18.9% on MUSES and 56.9% on THUMOS14 at IoU=0.5. To facilitate research in this direction, we release the dataset and the project code at https://songbai.site/muses/ .
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Video | THUMOS’14 | Avg mAP (0.3:0.7) | 53.4 | MUSES |
| Video | THUMOS’14 | mAP IOU@0.3 | 68.9 | MUSES |
| Video | THUMOS’14 | mAP IOU@0.4 | 64 | MUSES |
| Video | THUMOS’14 | mAP IOU@0.5 | 56.9 | MUSES |
| Video | THUMOS’14 | mAP IOU@0.6 | 46.3 | MUSES |
| Video | THUMOS’14 | mAP IOU@0.7 | 31 | MUSES |
| Video | MUSES | mAP | 18.6 | MUSES |
| Video | MUSES | mAP@0.3 | 25.9 | MUSES |
| Video | MUSES | mAP@0.4 | 22.6 | MUSES |
| Video | MUSES | mAP@0.5 | 18.9 | MUSES |
| Video | MUSES | mAP@0.6 | 15 | MUSES |
| Video | MUSES | mAP@0.7 | 10.6 | MUSES |
| Temporal Action Localization | THUMOS’14 | Avg mAP (0.3:0.7) | 53.4 | MUSES |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.3 | 68.9 | MUSES |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.4 | 64 | MUSES |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.5 | 56.9 | MUSES |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.6 | 46.3 | MUSES |
| Temporal Action Localization | THUMOS’14 | mAP IOU@0.7 | 31 | MUSES |
| Temporal Action Localization | MUSES | mAP | 18.6 | MUSES |
| Temporal Action Localization | MUSES | mAP@0.3 | 25.9 | MUSES |
| Temporal Action Localization | MUSES | mAP@0.4 | 22.6 | MUSES |
| Temporal Action Localization | MUSES | mAP@0.5 | 18.9 | MUSES |
| Temporal Action Localization | MUSES | mAP@0.6 | 15 | MUSES |
| Temporal Action Localization | MUSES | mAP@0.7 | 10.6 | MUSES |
| Zero-Shot Learning | THUMOS’14 | Avg mAP (0.3:0.7) | 53.4 | MUSES |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.3 | 68.9 | MUSES |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.4 | 64 | MUSES |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.5 | 56.9 | MUSES |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.6 | 46.3 | MUSES |
| Zero-Shot Learning | THUMOS’14 | mAP IOU@0.7 | 31 | MUSES |
| Zero-Shot Learning | MUSES | mAP | 18.6 | MUSES |
| Zero-Shot Learning | MUSES | mAP@0.3 | 25.9 | MUSES |
| Zero-Shot Learning | MUSES | mAP@0.4 | 22.6 | MUSES |
| Zero-Shot Learning | MUSES | mAP@0.5 | 18.9 | MUSES |
| Zero-Shot Learning | MUSES | mAP@0.6 | 15 | MUSES |
| Zero-Shot Learning | MUSES | mAP@0.7 | 10.6 | MUSES |
| Action Localization | THUMOS’14 | Avg mAP (0.3:0.7) | 53.4 | MUSES |
| Action Localization | THUMOS’14 | mAP IOU@0.3 | 68.9 | MUSES |
| Action Localization | THUMOS’14 | mAP IOU@0.4 | 64 | MUSES |
| Action Localization | THUMOS’14 | mAP IOU@0.5 | 56.9 | MUSES |
| Action Localization | THUMOS’14 | mAP IOU@0.6 | 46.3 | MUSES |
| Action Localization | THUMOS’14 | mAP IOU@0.7 | 31 | MUSES |
| Action Localization | MUSES | mAP | 18.6 | MUSES |
| Action Localization | MUSES | mAP@0.3 | 25.9 | MUSES |
| Action Localization | MUSES | mAP@0.4 | 22.6 | MUSES |
| Action Localization | MUSES | mAP@0.5 | 18.9 | MUSES |
| Action Localization | MUSES | mAP@0.6 | 15 | MUSES |
| Action Localization | MUSES | mAP@0.7 | 10.6 | MUSES |