Jie Hong, Zeeshan Hayder, Junlin Han, Pengfei Fang, Mehrtash Harandi, Lars Petersson
Audio-visual zero-shot learning aims to classify samples consisting of a pair of corresponding audio and video sequences from classes that are not present during training. An analysis of the audio-visual data reveals a large degree of hyperbolicity, indicating the potential benefit of using a hyperbolic transformation to achieve curvature-aware geometric learning, with the aim of exploring more complex hierarchical data structures for this task. The proposed approach employs a novel loss function that incorporates cross-modality alignment between video and audio features in the hyperbolic space. Additionally, we explore the use of multiple adaptive curvatures for hyperbolic projections. The experimental results on this very challenging task demonstrate that our proposed hyperbolic approach for zero-shot learning outperforms the SOTA method on three datasets: VGGSound-GZSL, UCF-GZSL, and ActivityNet-GZSL achieving a harmonic mean (HM) improvement of around 3.0%, 7.0%, and 5.3%, respectively.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Zero-Shot Learning | ActivityNet-GZSL(main) | HM | 12.65 | Hyper-multiple |
| Zero-Shot Learning | ActivityNet-GZSL(main) | ZSL | 9.5 | Hyper-multiple |
| Zero-Shot Learning | VGGSound-GZSL (cls) | HM | 8.67 | Hyper-multiple |
| Zero-Shot Learning | VGGSound-GZSL (cls) | ZSL | 7.31 | Hyper-multiple |
| Zero-Shot Learning | ActivityNet-GZSL (cls) | HM | 15.25 | Hyper-multiple |
| Zero-Shot Learning | ActivityNet-GZSL (cls) | ZSL | 10.39 | Hyper-multiple |
| Zero-Shot Learning | VGGSound-GZSL(main) | HM | 9.32 | Hyper-multiple |
| Zero-Shot Learning | VGGSound-GZSL(main) | ZSL | 7.97 | Hyper-multiple |
| Zero-Shot Learning | UCF-GZSL (cls) | HM | 48.3 | Hyper-multiple |
| Zero-Shot Learning | UCF-GZSL (cls) | ZSL | 52.11 | Hyper-multiple |
| Zero-Shot Learning | UCF-GZSL(main) | HM | 29.32 | Hyper-multiple |
| Zero-Shot Learning | UCF-GZSL(main) | ZSL | 22.24 | Hyper-multiple |