Guo Chen, Sen Xing, Zhe Chen, Yi Wang, Kunchang Li, Yizhuo Li, Yi Liu, Jiahao Wang, Yin-Dong Zheng, Bingkun Huang, Zhiyu Zhao, Junting Pan, Yifei HUANG, Zun Wang, Jiashuo Yu, Yinan He, Hongjie Zhang, Tong Lu, Yali Wang, LiMin Wang, Yu Qiao
In this report, we present our champion solutions to five tracks at Ego4D challenge. We leverage our developed InternVideo, a video foundation model, for five Ego4D tasks, including Moment Queries, Natural Language Queries, Future Hand Prediction, State Change Object Detection, and Short-term Object Interaction Anticipation. InternVideo-Ego4D is an effective paradigm to adapt the strong foundation model to the downstream ego-centric video understanding tasks with simple head designs. In these five tasks, the performance of InternVideo-Ego4D comprehensively surpasses the baseline methods and the champions of CVPR2022, demonstrating the powerful representation ability of InternVideo as a video foundation model. Our code will be released at https://github.com/OpenGVLab/ego4d-eccv2022-solutions
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| State Change Object Detection | Ego4D | AP | 37.19 | InternVideo |
| State Change Object Detection | Ego4D | AP50 | 55.97 | InternVideo |
| State Change Object Detection | Ego4D | AP75 | 38.44 | InternVideo |
| Short-term Object Interaction Anticipation | Ego4D | Noun (Top5 mAP) | 24.6 | InternVideo |
| Short-term Object Interaction Anticipation | Ego4D | Noun+TTC (Top5 mAP) | 7.64 | InternVideo |
| Short-term Object Interaction Anticipation | Ego4D | Noun+Verb(Top5 mAP) | 9.18 | InternVideo |
| Short-term Object Interaction Anticipation | Ego4D | Overall (Top5 mAP) | 3.4 | InternVideo |
| Future Hand Prediction | Ego4D | C.Disp(Left) | 53.33 | InternVideo |
| Future Hand Prediction | Ego4D | C.Disp(Right) | 53.37 | InternVideo |
| Future Hand Prediction | Ego4D | Disp(Total) | 196.8 | InternVideo |
| Future Hand Prediction | Ego4D | M.Disp(Left) | 43.25 | InternVideo |
| Future Hand Prediction | Ego4D | M.Disp(Right) | 46.25 | InternVideo |
| Natural Language Queries | Ego4D | R@1 IoU=0.3 | 16.45 | InternVideo |
| Natural Language Queries | Ego4D | R@1 IoU=0.5 | 10.06 | InternVideo |
| Natural Language Queries | Ego4D | R@1 Mean(0.3 and 0.5) | 13.26 | InternVideo |
| Natural Language Queries | Ego4D | R@5 IoU=0.3 | 22.95 | InternVideo |
| Natural Language Queries | Ego4D | R@5 IoU=0.5 | 16.1 | InternVideo |