Chao Li, Qiaoyong Zhong, Di Xie, ShiLiang Pu
Current state-of-the-art approaches to skeleton-based action recognition are mostly based on recurrent neural networks (RNN). In this paper, we propose a novel convolutional neural networks (CNN) based framework for both action classification and detection. Raw skeleton coordinates as well as skeleton motion are fed directly into CNN for label prediction. A novel skeleton transformer module is designed to rearrange and select important skeleton joints automatically. With a simple 7-layer network, we obtain 89.3% accuracy on validation set of the NTU RGB+D dataset. For action detection in untrimmed videos, we develop a window proposal network to extract temporal segment proposals, which are further classified within the same network. On the recent PKU-MMD dataset, we achieve 93.7% mAP, surpassing the baseline by a large margin.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Video | PKU-MMD | mAP@0.50 (CS) | 90.4 | Li et al. [[Li et al.2017b]] |
| Video | PKU-MMD | mAP@0.50 (CV) | 93.7 | Li et al. [[Li et al.2017b]] |
| Video | NTU RGB+D | Accuracy (CS) | 83.2 | CNN+Motion+Trans |
| Video | NTU RGB+D | Accuracy (CV) | 89.3 | CNN+Motion+Trans |
| Temporal Action Localization | PKU-MMD | mAP@0.50 (CS) | 90.4 | Li et al. [[Li et al.2017b]] |
| Temporal Action Localization | PKU-MMD | mAP@0.50 (CV) | 93.7 | Li et al. [[Li et al.2017b]] |
| Temporal Action Localization | NTU RGB+D | Accuracy (CS) | 83.2 | CNN+Motion+Trans |
| Temporal Action Localization | NTU RGB+D | Accuracy (CV) | 89.3 | CNN+Motion+Trans |
| Zero-Shot Learning | PKU-MMD | mAP@0.50 (CS) | 90.4 | Li et al. [[Li et al.2017b]] |
| Zero-Shot Learning | PKU-MMD | mAP@0.50 (CV) | 93.7 | Li et al. [[Li et al.2017b]] |
| Zero-Shot Learning | NTU RGB+D | Accuracy (CS) | 83.2 | CNN+Motion+Trans |
| Zero-Shot Learning | NTU RGB+D | Accuracy (CV) | 89.3 | CNN+Motion+Trans |
| Activity Recognition | PKU-MMD | mAP@0.50 (CS) | 90.4 | Li et al. [[Li et al.2017b]] |
| Activity Recognition | PKU-MMD | mAP@0.50 (CV) | 93.7 | Li et al. [[Li et al.2017b]] |
| Activity Recognition | NTU RGB+D | Accuracy (CS) | 83.2 | CNN+Motion+Trans |
| Activity Recognition | NTU RGB+D | Accuracy (CV) | 89.3 | CNN+Motion+Trans |
| Action Localization | PKU-MMD | mAP@0.50 (CS) | 90.4 | Li et al. [[Li et al.2017b]] |
| Action Localization | PKU-MMD | mAP@0.50 (CV) | 93.7 | Li et al. [[Li et al.2017b]] |
| Action Localization | NTU RGB+D | Accuracy (CS) | 83.2 | CNN+Motion+Trans |
| Action Localization | NTU RGB+D | Accuracy (CV) | 89.3 | CNN+Motion+Trans |
| Action Detection | PKU-MMD | mAP@0.50 (CS) | 90.4 | Li et al. [[Li et al.2017b]] |
| Action Detection | PKU-MMD | mAP@0.50 (CV) | 93.7 | Li et al. [[Li et al.2017b]] |
| Action Detection | NTU RGB+D | Accuracy (CS) | 83.2 | CNN+Motion+Trans |
| Action Detection | NTU RGB+D | Accuracy (CV) | 89.3 | CNN+Motion+Trans |
| 3D Action Recognition | PKU-MMD | mAP@0.50 (CS) | 90.4 | Li et al. [[Li et al.2017b]] |
| 3D Action Recognition | PKU-MMD | mAP@0.50 (CV) | 93.7 | Li et al. [[Li et al.2017b]] |
| 3D Action Recognition | NTU RGB+D | Accuracy (CS) | 83.2 | CNN+Motion+Trans |
| 3D Action Recognition | NTU RGB+D | Accuracy (CV) | 89.3 | CNN+Motion+Trans |
| Action Recognition | PKU-MMD | mAP@0.50 (CS) | 90.4 | Li et al. [[Li et al.2017b]] |
| Action Recognition | PKU-MMD | mAP@0.50 (CV) | 93.7 | Li et al. [[Li et al.2017b]] |
| Action Recognition | NTU RGB+D | Accuracy (CS) | 83.2 | CNN+Motion+Trans |
| Action Recognition | NTU RGB+D | Accuracy (CV) | 89.3 | CNN+Motion+Trans |