Estimating the valence and arousal of dyadic conversations using autonomic nervous system responses and regression algorithms

Iman Chatterjee^1*Maja Goršič²Robert A Kaya³Joshua Clapp³Vesna Dominika Novak^1*

• ¹Department of Electrical and Computer Engineering, University of Cincinnati, Cincinnati, United States
• ²Department of Biomedical Engineering, Marquette University, Milwaukee, United States
• ³Department of Psychology, University of Wyoming, Laramie, United States

Autonomic nervous system responses provide valuable information about interactions between pairs or groups of people but have primarily been studied using group-level statistical analysis, with a few studies attempting single-trial classification. As an alternative to classification, our study uses regression algorithms to estimate the valence and arousal of specific conversation intervals from dyads' autonomic nervous system responses. Forty-one dyads took part in 20-minute conversations following several different prompts. The conversations were divided into ten 2-minute intervals, with participants self-reporting perceived conversation valence and arousal after each 2-minute interval. Observers watched videos of the conversations and separately also rated valence and arousal. Four autonomic nervous system responses (electrocardiogram, electrodermal activity, respiration, skin temperature) were recorded, and both individual and synchrony features were extracted for each 2-minute interval. These extracted features were used with feature selection and a multilinear perceptron to estimate self-reported and observer-reported valence and arousal of each interval in both a dyad-specific (based on data from same dyad) and dyad-nonspecific (based on data from other dyads) manner. Both dyad-specific and dyad-nonspecific regression using the multilinear perceptron resulted in lower root-mean-square errors than a simple median-based estimator and two other regression methods (linear regression and support vector machines), suggesting that physiological measurements can be used to characterize dyadic conversations on the level of individual dyads and conversation intervals. In the long term, such regression algorithms could potentially be used in applications such as education and mental health counseling.