Haodong Duan, Jiaqi Wang, Kai Chen, Dahua Lin
We present PYSKL: an open-source toolbox for skeleton-based action recognition based on PyTorch. The toolbox supports a wide variety of skeleton action recognition algorithms, including approaches based on GCN and CNN. In contrast to existing open-source skeleton action recognition projects that include only one or two algorithms, PYSKL implements six different algorithms under a unified framework with both the latest and original good practices to ease the comparison of efficacy and efficiency. We also provide an original GCN-based skeleton action recognition model named ST-GCN++, which achieves competitive recognition performance without any complicated attention schemes, serving as a strong baseline. Meanwhile, PYSKL supports the training and testing of nine skeleton-based action recognition benchmarks and achieves state-of-the-art recognition performance on eight of them. To facilitate future research on skeleton action recognition, we also provide a large number of trained models and detailed benchmark results to give some insights. PYSKL is released at https://github.com/kennymckormick/pyskl and is actively maintained. We will update this report when we add new features or benchmarks. The current version corresponds to PYSKL v0.2.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Video | NTU RGB+D 120 | Accuracy (Cross-Setup) | 90.8 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Video | NTU RGB+D 120 | Accuracy (Cross-Subject) | 88.6 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Video | NTU RGB+D 120 | Ensembled Modalities | 4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Video | NTU RGB+D | Accuracy (CS) | 92.6 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Video | NTU RGB+D | Accuracy (CV) | 97.4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Video | NTU RGB+D | Ensembled Modalities | 4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Video | NTU RGB+D | Accuracy (CS) | 91.4 | ST-GCN [PYSKL, 2D Skeleton] |
| Video | NTU RGB+D | Accuracy (CV) | 98.3 | ST-GCN [PYSKL, 2D Skeleton] |
| Video | NTU RGB+D | Ensembled Modalities | 4 | ST-GCN [PYSKL, 2D Skeleton] |
| Temporal Action Localization | NTU RGB+D 120 | Accuracy (Cross-Setup) | 90.8 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Temporal Action Localization | NTU RGB+D 120 | Accuracy (Cross-Subject) | 88.6 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Temporal Action Localization | NTU RGB+D 120 | Ensembled Modalities | 4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Temporal Action Localization | NTU RGB+D | Accuracy (CS) | 92.6 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Temporal Action Localization | NTU RGB+D | Accuracy (CV) | 97.4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Temporal Action Localization | NTU RGB+D | Ensembled Modalities | 4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Temporal Action Localization | NTU RGB+D | Accuracy (CS) | 91.4 | ST-GCN [PYSKL, 2D Skeleton] |
| Temporal Action Localization | NTU RGB+D | Accuracy (CV) | 98.3 | ST-GCN [PYSKL, 2D Skeleton] |
| Temporal Action Localization | NTU RGB+D | Ensembled Modalities | 4 | ST-GCN [PYSKL, 2D Skeleton] |
| Zero-Shot Learning | NTU RGB+D 120 | Accuracy (Cross-Setup) | 90.8 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Zero-Shot Learning | NTU RGB+D 120 | Accuracy (Cross-Subject) | 88.6 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Zero-Shot Learning | NTU RGB+D 120 | Ensembled Modalities | 4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Zero-Shot Learning | NTU RGB+D | Accuracy (CS) | 92.6 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Zero-Shot Learning | NTU RGB+D | Accuracy (CV) | 97.4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Zero-Shot Learning | NTU RGB+D | Ensembled Modalities | 4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Zero-Shot Learning | NTU RGB+D | Accuracy (CS) | 91.4 | ST-GCN [PYSKL, 2D Skeleton] |
| Zero-Shot Learning | NTU RGB+D | Accuracy (CV) | 98.3 | ST-GCN [PYSKL, 2D Skeleton] |
| Zero-Shot Learning | NTU RGB+D | Ensembled Modalities | 4 | ST-GCN [PYSKL, 2D Skeleton] |
| Activity Recognition | NTU RGB+D 120 | Accuracy (Cross-Setup) | 90.8 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Activity Recognition | NTU RGB+D 120 | Accuracy (Cross-Subject) | 88.6 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Activity Recognition | NTU RGB+D 120 | Ensembled Modalities | 4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Activity Recognition | NTU RGB+D | Accuracy (CS) | 92.6 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Activity Recognition | NTU RGB+D | Accuracy (CV) | 97.4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Activity Recognition | NTU RGB+D | Ensembled Modalities | 4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Activity Recognition | NTU RGB+D | Accuracy (CS) | 91.4 | ST-GCN [PYSKL, 2D Skeleton] |
| Activity Recognition | NTU RGB+D | Accuracy (CV) | 98.3 | ST-GCN [PYSKL, 2D Skeleton] |
| Activity Recognition | NTU RGB+D | Ensembled Modalities | 4 | ST-GCN [PYSKL, 2D Skeleton] |
| Action Localization | NTU RGB+D 120 | Accuracy (Cross-Setup) | 90.8 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Action Localization | NTU RGB+D 120 | Accuracy (Cross-Subject) | 88.6 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Action Localization | NTU RGB+D 120 | Ensembled Modalities | 4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Action Localization | NTU RGB+D | Accuracy (CS) | 92.6 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Action Localization | NTU RGB+D | Accuracy (CV) | 97.4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Action Localization | NTU RGB+D | Ensembled Modalities | 4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Action Localization | NTU RGB+D | Accuracy (CS) | 91.4 | ST-GCN [PYSKL, 2D Skeleton] |
| Action Localization | NTU RGB+D | Accuracy (CV) | 98.3 | ST-GCN [PYSKL, 2D Skeleton] |
| Action Localization | NTU RGB+D | Ensembled Modalities | 4 | ST-GCN [PYSKL, 2D Skeleton] |
| Action Detection | NTU RGB+D 120 | Accuracy (Cross-Setup) | 90.8 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Action Detection | NTU RGB+D 120 | Accuracy (Cross-Subject) | 88.6 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Action Detection | NTU RGB+D 120 | Ensembled Modalities | 4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Action Detection | NTU RGB+D | Accuracy (CS) | 92.6 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Action Detection | NTU RGB+D | Accuracy (CV) | 97.4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Action Detection | NTU RGB+D | Ensembled Modalities | 4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Action Detection | NTU RGB+D | Accuracy (CS) | 91.4 | ST-GCN [PYSKL, 2D Skeleton] |
| Action Detection | NTU RGB+D | Accuracy (CV) | 98.3 | ST-GCN [PYSKL, 2D Skeleton] |
| Action Detection | NTU RGB+D | Ensembled Modalities | 4 | ST-GCN [PYSKL, 2D Skeleton] |
| 3D Action Recognition | NTU RGB+D 120 | Accuracy (Cross-Setup) | 90.8 | ST-GCN++ [PYSKL, 3D Skeleton] |
| 3D Action Recognition | NTU RGB+D 120 | Accuracy (Cross-Subject) | 88.6 | ST-GCN++ [PYSKL, 3D Skeleton] |
| 3D Action Recognition | NTU RGB+D 120 | Ensembled Modalities | 4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| 3D Action Recognition | NTU RGB+D | Accuracy (CS) | 92.6 | ST-GCN++ [PYSKL, 3D Skeleton] |
| 3D Action Recognition | NTU RGB+D | Accuracy (CV) | 97.4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| 3D Action Recognition | NTU RGB+D | Ensembled Modalities | 4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| 3D Action Recognition | NTU RGB+D | Accuracy (CS) | 91.4 | ST-GCN [PYSKL, 2D Skeleton] |
| 3D Action Recognition | NTU RGB+D | Accuracy (CV) | 98.3 | ST-GCN [PYSKL, 2D Skeleton] |
| 3D Action Recognition | NTU RGB+D | Ensembled Modalities | 4 | ST-GCN [PYSKL, 2D Skeleton] |
| Action Recognition | NTU RGB+D 120 | Accuracy (Cross-Setup) | 90.8 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Action Recognition | NTU RGB+D 120 | Accuracy (Cross-Subject) | 88.6 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Action Recognition | NTU RGB+D 120 | Ensembled Modalities | 4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Action Recognition | NTU RGB+D | Accuracy (CS) | 92.6 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Action Recognition | NTU RGB+D | Accuracy (CV) | 97.4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Action Recognition | NTU RGB+D | Ensembled Modalities | 4 | ST-GCN++ [PYSKL, 3D Skeleton] |
| Action Recognition | NTU RGB+D | Accuracy (CS) | 91.4 | ST-GCN [PYSKL, 2D Skeleton] |
| Action Recognition | NTU RGB+D | Accuracy (CV) | 98.3 | ST-GCN [PYSKL, 2D Skeleton] |
| Action Recognition | NTU RGB+D | Ensembled Modalities | 4 | ST-GCN [PYSKL, 2D Skeleton] |