• The Sims4Action Dataset: a videogame-based dataset for Synthetic→Real domain adaptation for human activity recognition.

• Goal : Exploring the concept of constructing training examples for Activities of Daily Living (ADL) recognition by playing life simulation video games.

• ** Sims4Action dataset is created with the commercial game THE SIMS 4** by executing actions-of-interest within the game in a "top-down" manner. It features ten hours of video material of eight diverse characters and multiple environments. Ten actions are selected to have a direct correspondence to categories covered in the real-life dataset Toyota Smarthome [2] to enable the research of Synthetic→Real transfer in action recognition.

• Two benchmarks :* Gaming→Gaming* (training and evaluation on Sims4Action) and Gaming→Real (training on Sims4Action, evaluation on the real Toyota Smarthome data [2]).

• Main challenge: Gaming→Real domain adaptation
  While ADL recognition on gaming data is interesting from a theoretical perspective, the key challenge arises from transferring knowledge learned from simulated data to real-world applications. Sims4Action specifically provides a benchmark for this scenario since it describes a Gaming→Real challenge, which evaluates models on real videos derived from the existing Toyota Smarthome dataset .

References

[1] Let's Play for Action: Recognizing Activities of Daily Living by Learning from Life Simulation Video Games. Alina Roitberg*, David Schneider*, Aulia Djamal, Constantin Seibold, Simon Reiß, Rainer Stiefelhagen, In International Conference on Intelligent Robots and Systems (IROS), 2021 (* denotes equal contribution.)

[2] Toyota smarthome: Real-world activities of daily living. Srijan Das, Rui Dai, Michal Koperski, Luca Minciullo, Lorenzo Garattoni, Francois Bremond, Gianpiero Francesca, In International Conference on Computer Vision (ICCV), 2019.