AUTHOR=Yalcin Murat , Halbig Andreas , Fischbach Martin , Latoschik Marc Erich 

TITLE=Automatic cybersickness detection by deep learning of augmented physiological data from off-the-shelf consumer-grade sensors

JOURNAL=Frontiers in Virtual Reality

VOLUME=Volume 5 - 2024

YEAR=2024

URL=https://www.frontiersin.org/journals/virtual-reality/articles/10.3389/frvir.2024.1364207

DOI=10.3389/frvir.2024.1364207

ISSN=2673-4192

ABSTRACT=Cybersickness still is a prominent risk factor potentially affecting the usability of Virtual Reality applications. Automated real-time detection of cybersickness promises to support a better general understanding of the phenomena as well as to avoid and counter-act its occurrence. It could be used to facilitate application optimization, .i.e., to systematically link potential causes (technical development and conceptual design decisions) to cybersickness in closed-loop user-centered development cycles. In addition, it could be used to monitor, warn, and hence safeguard users against any onset of cybersickness during a virtual reality exposure, especially in healthcare applications. This article presents a novel real-time-capable cybersickness detection method by deep learning of augmented physiological data. In contrast to related preliminary work, we are exploring a unique combination of mid-immersion ground truth elicitation, an unobtrusive wireless setup and moderate training performance requirements. We developed a proof-ofconcept prototype to compare (combinations of) Convolutional Neural Networks, Long-Short Term Memory, as well as Support Vector Machines with respect to detection performance.We demonstrate that the use of a conditional Generative Adversarial Network based data augmentation technique increases detection performance significantly, and showcase the feasibility of real-time cybersickness detection in a genuine application example. Finally, a comprehensive performance analysis demonstrates that a 4-layered bidirectional Long Short Term Memory network with the developed data augmentation delivers superior performance (91.1% f1-score) for real-time cybersickness detection. To encourage replicability and reuse in future cybersickness studies, we will release the code and the dataset as publicly available.