Automatic Cybersickness Detection by Deep Learning of Augmented Physiological Data from Off-the-Shelf Consumer-Grade Sensors Provisionally Accepted

Murat Yalcin^1* Andreas Halbig¹ Martin Fischbach¹ Marc E. Latoschik¹

• ¹Julius Maximilian University of Würzburg, Germany

Cybersickness still is a prominent risk factor potentially affecting the usability of Virtual Reality applications. Automated real-time detection of cybersickness promises to support a better general understanding of the phenomena as well as to avoid and counter-act its occurrence. It could be used to facilitate application optimization, .i.e., to systematically link potential causes (technical development and conceptual design decisions) to cybersickness in closed-loop user-centered development cycles. In addition, it could be used to monitor, warn, and hence safeguard users against any onset of cybersickness during a virtual reality exposure, especially in healthcare applications. This article presents a novel real-time-capable cybersickness detection method by deep learning of augmented physiological data. In contrast to related preliminary work, we are exploring a unique combination of mid-immersion ground truth elicitation, an unobtrusive wireless setup and moderate training performance requirements. We developed a proof-ofconcept prototype to compare (combinations of) Convolutional Neural Networks, Long-Short Term Memory, as well as Support Vector Machines with respect to detection performance.We demonstrate that the use of a conditional Generative Adversarial Network based data augmentation technique increases detection performance significantly, and showcase the feasibility of real-time cybersickness detection in a genuine application example. Finally, a comprehensive performance analysis demonstrates that a 4-layered bidirectional Long Short Term Memory network with the developed data augmentation delivers superior performance (91.1% f1-score) for real-time cybersickness detection. To encourage replicability and reuse in future cybersickness studies, we will release the code and the dataset as publicly available.