Jun Liu, Amir Shahroudy, Dong Xu, Gang Wang
3D action recognition - analysis of human actions based on 3D skeleton data - becomes popular recently due to its succinctness, robustness, and view-invariant representation. Recent attempts on this problem suggested to develop RNN-based learning methods to model the contextual dependency in the temporal domain. In this paper, we extend this idea to spatio-temporal domains to analyze the hidden sources of action-related information within the input data over both domains concurrently. Inspired by the graphical structure of the human skeleton, we further propose a more powerful tree-structure based traversal method. To handle the noise and occlusion in 3D skeleton data, we introduce new gating mechanism within LSTM to learn the reliability of the sequential input data and accordingly adjust its effect on updating the long-term context information stored in the memory cell. Our method achieves state-of-the-art performance on 4 challenging benchmark datasets for 3D human action analysis.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Video | NTU RGB+D | Accuracy (CS) | 69.2 | Spatio-Temporal LSTM |
| Video | NTU RGB+D | Accuracy (CV) | 77.7 | Spatio-Temporal LSTM |
| Video | NTU RGB+D | Accuracy (CS) | 61.7 | ST-LSTM |
| Video | NTU RGB+D | Accuracy (CV) | 75.5 | ST-LSTM |
| Temporal Action Localization | NTU RGB+D | Accuracy (CS) | 69.2 | Spatio-Temporal LSTM |
| Temporal Action Localization | NTU RGB+D | Accuracy (CV) | 77.7 | Spatio-Temporal LSTM |
| Temporal Action Localization | NTU RGB+D | Accuracy (CS) | 61.7 | ST-LSTM |
| Temporal Action Localization | NTU RGB+D | Accuracy (CV) | 75.5 | ST-LSTM |
| Zero-Shot Learning | NTU RGB+D | Accuracy (CS) | 69.2 | Spatio-Temporal LSTM |
| Zero-Shot Learning | NTU RGB+D | Accuracy (CV) | 77.7 | Spatio-Temporal LSTM |
| Zero-Shot Learning | NTU RGB+D | Accuracy (CS) | 61.7 | ST-LSTM |
| Zero-Shot Learning | NTU RGB+D | Accuracy (CV) | 75.5 | ST-LSTM |
| Activity Recognition | NTU RGB+D | Accuracy (CS) | 69.2 | Spatio-Temporal LSTM |
| Activity Recognition | NTU RGB+D | Accuracy (CV) | 77.7 | Spatio-Temporal LSTM |
| Activity Recognition | NTU RGB+D | Accuracy (CS) | 61.7 | ST-LSTM |
| Activity Recognition | NTU RGB+D | Accuracy (CV) | 75.5 | ST-LSTM |
| Action Localization | NTU RGB+D | Accuracy (CS) | 69.2 | Spatio-Temporal LSTM |
| Action Localization | NTU RGB+D | Accuracy (CV) | 77.7 | Spatio-Temporal LSTM |
| Action Localization | NTU RGB+D | Accuracy (CS) | 61.7 | ST-LSTM |
| Action Localization | NTU RGB+D | Accuracy (CV) | 75.5 | ST-LSTM |
| Action Detection | NTU RGB+D | Accuracy (CS) | 69.2 | Spatio-Temporal LSTM |
| Action Detection | NTU RGB+D | Accuracy (CV) | 77.7 | Spatio-Temporal LSTM |
| Action Detection | NTU RGB+D | Accuracy (CS) | 61.7 | ST-LSTM |
| Action Detection | NTU RGB+D | Accuracy (CV) | 75.5 | ST-LSTM |
| 3D Action Recognition | NTU RGB+D | Accuracy (CS) | 69.2 | Spatio-Temporal LSTM |
| 3D Action Recognition | NTU RGB+D | Accuracy (CV) | 77.7 | Spatio-Temporal LSTM |
| 3D Action Recognition | NTU RGB+D | Accuracy (CS) | 61.7 | ST-LSTM |
| 3D Action Recognition | NTU RGB+D | Accuracy (CV) | 75.5 | ST-LSTM |
| Action Recognition | NTU RGB+D | Accuracy (CS) | 69.2 | Spatio-Temporal LSTM |
| Action Recognition | NTU RGB+D | Accuracy (CV) | 77.7 | Spatio-Temporal LSTM |
| Action Recognition | NTU RGB+D | Accuracy (CS) | 61.7 | ST-LSTM |
| Action Recognition | NTU RGB+D | Accuracy (CV) | 75.5 | ST-LSTM |