Yuheng Yang, Haipeng Chen, Zhenguang Liu, Yingda Lyu, Beibei Zhang, Shuang Wu, Zhibo Wang, Kui Ren
Action recognition has long been a fundamental and intriguing problem in artificial intelligence. The task is challenging due to the high dimensionality nature of an action, as well as the subtle motion details to be considered. Current state-of-the-art approaches typically learn from articulated motion sequences in the straightforward 3D Euclidean space. However, the vanilla Euclidean space is not efficient for modeling important motion characteristics such as the joint-wise angular acceleration, which reveals the driving force behind the motion. Moreover, current methods typically attend to each channel equally and lack theoretical constrains on extracting task-relevant features from the input. In this paper, we seek to tackle these challenges from three aspects: (1) We propose to incorporate an acceleration representation, explicitly modeling the higher-order variations in motion. (2) We introduce a novel Stream-GCN network equipped with multi-stream components and channel attention, where different representations (i.e., streams) supplement each other towards a more precise action recognition while attention capitalizes on those important channels. (3) We explore feature-level supervision for maximizing the extraction of task-relevant information and formulate this into a mutual information loss. Empirically, our approach sets the new state-of-the-art performance on three benchmark datasets, NTU RGB+D, NTU RGB+D 120, and NW-UCLA. Our code is anonymously released at https://github.com/ActionR-Group/Stream-GCN, hoping to inspire the community.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Video | NTU RGB+D 120 | Accuracy (Cross-Setup) | 91 | Stream-GCN |
| Video | NTU RGB+D 120 | Accuracy (Cross-Subject) | 89.7 | Stream-GCN |
| Video | NTU RGB+D | Accuracy (CS) | 92.9 | Stream-GCN |
| Video | NTU RGB+D | Accuracy (CV) | 96.9 | Stream-GCN |
| Temporal Action Localization | NTU RGB+D 120 | Accuracy (Cross-Setup) | 91 | Stream-GCN |
| Temporal Action Localization | NTU RGB+D 120 | Accuracy (Cross-Subject) | 89.7 | Stream-GCN |
| Temporal Action Localization | NTU RGB+D | Accuracy (CS) | 92.9 | Stream-GCN |
| Temporal Action Localization | NTU RGB+D | Accuracy (CV) | 96.9 | Stream-GCN |
| Zero-Shot Learning | NTU RGB+D 120 | Accuracy (Cross-Setup) | 91 | Stream-GCN |
| Zero-Shot Learning | NTU RGB+D 120 | Accuracy (Cross-Subject) | 89.7 | Stream-GCN |
| Zero-Shot Learning | NTU RGB+D | Accuracy (CS) | 92.9 | Stream-GCN |
| Zero-Shot Learning | NTU RGB+D | Accuracy (CV) | 96.9 | Stream-GCN |
| Activity Recognition | NTU RGB+D 120 | Accuracy (Cross-Setup) | 91 | Stream-GCN |
| Activity Recognition | NTU RGB+D 120 | Accuracy (Cross-Subject) | 89.7 | Stream-GCN |
| Activity Recognition | NTU RGB+D | Accuracy (CS) | 92.9 | Stream-GCN |
| Activity Recognition | NTU RGB+D | Accuracy (CV) | 96.9 | Stream-GCN |
| Action Localization | NTU RGB+D 120 | Accuracy (Cross-Setup) | 91 | Stream-GCN |
| Action Localization | NTU RGB+D 120 | Accuracy (Cross-Subject) | 89.7 | Stream-GCN |
| Action Localization | NTU RGB+D | Accuracy (CS) | 92.9 | Stream-GCN |
| Action Localization | NTU RGB+D | Accuracy (CV) | 96.9 | Stream-GCN |
| Action Detection | NTU RGB+D 120 | Accuracy (Cross-Setup) | 91 | Stream-GCN |
| Action Detection | NTU RGB+D 120 | Accuracy (Cross-Subject) | 89.7 | Stream-GCN |
| Action Detection | NTU RGB+D | Accuracy (CS) | 92.9 | Stream-GCN |
| Action Detection | NTU RGB+D | Accuracy (CV) | 96.9 | Stream-GCN |
| 3D Action Recognition | NTU RGB+D 120 | Accuracy (Cross-Setup) | 91 | Stream-GCN |
| 3D Action Recognition | NTU RGB+D 120 | Accuracy (Cross-Subject) | 89.7 | Stream-GCN |
| 3D Action Recognition | NTU RGB+D | Accuracy (CS) | 92.9 | Stream-GCN |
| 3D Action Recognition | NTU RGB+D | Accuracy (CV) | 96.9 | Stream-GCN |
| Action Recognition | NTU RGB+D 120 | Accuracy (Cross-Setup) | 91 | Stream-GCN |
| Action Recognition | NTU RGB+D 120 | Accuracy (Cross-Subject) | 89.7 | Stream-GCN |
| Action Recognition | NTU RGB+D | Accuracy (CS) | 92.9 | Stream-GCN |
| Action Recognition | NTU RGB+D | Accuracy (CV) | 96.9 | Stream-GCN |