AUTHOR=Huang Yating , Zhai Dengyue , Song Jingze , Rao Xuanheng , Sun Xiao , Tang Jin 

TITLE=Mental states and personality based on real-time physical activity and facial expression recognition

JOURNAL=Frontiers in Psychiatry

VOLUME=Volume 13 - 2022

YEAR=2023

URL=https://www.frontiersin.org/journals/psychiatry/articles/10.3389/fpsyt.2022.1019043

DOI=10.3389/fpsyt.2022.1019043

ISSN=1664-0640

ABSTRACT=To explore a quick and non-invasive way to measure individual psychological states, this study developed interview-based scales and multi-modal information was collected from 172 participants. The Interview Psychological Symptom Inventory (IPSI) we developed eventually retained 53 items with 9 main factors, all of them performed well in terms of reliability and validity. Used optimized convolutional neural networks and original detection algorithms for the recognition of individuals’ facial expressions and physical activity based on Russell’s circumplex model and Five Factor Model. It was found that there was a significant correlation between the developed scale and participants’ scores on each factor in the SCl-90 and BFI-2, Cronbach’s α = [−0.257, 0.632]. Between the multi-modal data, the arousal of facial expressions was significantly correlated with interval of validity (p < 0.01), valence was significantly correlated with IPSI and SCL-90, physical activity was significantly correlated with gender, age and factors of the scales. Our research demonstrates that mental health can be monitored and assessed remotely by collecting and analyzing multimodal data from individuals captured by digital tools.