Machine-learning detection of stress severity expressed on a continuous scale using acoustic, verbal, visual, and physiological data: Lessons learned

Marketa Ciharova^1*Khadicha Amarti¹Ward Van Breda^1,2Martin Gevonden¹Sina Ghassemi¹Annet Kleiboer¹Christiaan H Vinkers^3,4,5Milou S C Sep^3,4,5Sophia Trofimova¹Alexander C Cooper¹Xianhua Peng^1,6Mieke Schulte^1,7Eirini Karyotaki^1,8,9Pim Cuijpers^1,10,9Heleen Riper^1,3

• ¹VU Amsterdam, Amsterdam, Netherlands
• ²Sentimentics, Dordrecht, Netherlands
• ³Amsterdam University Medical Center, Amsterdam, Netherlands
• ⁴Amsterdam Neuroscience Research Institute, Amsterdam, Netherlands
• ⁵GGZ inGeest, Amsterdam, Netherlands
• ⁶Tilburg University, Tilburg, Netherlands
• ⁷Erasmus University Rotterdam, Rotterdam, Netherlands
• ⁸University of Amsterdam, Amsterdam, Netherlands
• ⁹WHO Collaborating Center for Research and Dissemination of Psychological Interventions, Amsterdam, Netherlands
• ¹⁰Babeș-Bolyai University, Cluj-Napoca, Cluj, Romania

Early detection of elevated acute stress is necessary if we aim to reduce consequences associated with prolonged or recurrent stress exposure. Stress monitoring may be supported by valid and reliable machine-learning algorithms. However, investigation of algorithms detecting stress severity on a continuous scale is missing. Use of multimodal data might contribute to such detection. We aimed to detect stress using multimodal data and identify challenges in such a study. We assessed performance of a machine-learning algorithm trained on multimodal data, namely visual, acoustic, verbal, and physiological features, in its ability to detect stress severity following a partially automated online version of the Trier Social Stress Test. College students (n = 42; M age = 20.79, 69% female) completed a self-reported stress visual analogue scale at five time-points: After the resting period (P1), during the three stress-inducing tasks (i.e., preparation for a presentation, presentation, and arithmetic task, P2-4) and after a recovery (P5). We recorded the participants' voice and facial expressions by a video camera and measured cardiovascular and electrodermal physiology by an ambulatory monitoring system. We evaluated the performance of the algorithm in detection of stress severity using combinations of visual, acoustic, verbal, and physiological data collected at each of the periods of the experiment (P1-5). We found a weak association between the detected and observed scores (r 2 = .154; p = .021). In post-hoc analysis, we classified participants into categories of stressed and non-stressed individuals.When applying all available features (i.e., visual, acoustic, verbal, and physiological), or a combination of visual, acoustic and verbal features, performance ranged from acceptable to good, but only for the presentation task (accuracy up to .71, F1-score up to .73).The complexity of input features needed for machine-learning detection of stress severity based on multimodal data requires large sample sizes with wide variability of stress reactions and inputs among participants. These are difficult to recruit for laboratory setting, due to high time and effort demands on the side of both researcher and participant. Resources needed may be decreased using automatization of experimental procedures, which may lead to additional technological challenges, potentially causing other recruitment setbacks.