Mahsa Sheikh
M. Qassem
Panicos A. Kyriacou- Research Centre for Biomedical Engineering, School of Mathematics, Computer Science & Engineering, City, University of London, London, United Kingdom
Collecting and analyzing data from sensors embedded in the context of daily life has been widely employed for the monitoring of mental health. Variations in parameters such as movement, sleep duration, heart rate, electrocardiogram, skin temperature, etc., are often associated with psychiatric disorders. Namely, accelerometer data, microphone, and call logs can be utilized to identify voice features and social activities indicative of depressive symptoms, and physiological factors such as heart rate and skin conductance can be used to detect stress and anxiety disorders. Therefore, a wide range of devices comprising a variety of sensors have been developed to capture these physiological and behavioral data and translate them into phenotypes and states related to mental health. Such systems aim to identify behaviors that are the consequence of an underlying physiological alteration, and hence, the raw sensor data are captured and converted into features that are used to define behavioral markers, often through machine learning. However, due to the complexity of passive data, these relationships are not simple and need to be well-established. Furthermore, parameters such as intrapersonal and interpersonal differences need to be considered when interpreting the data. Altogether, combining practical mobile and wearable systems with the right data analysis algorithms can provide a useful tool for the monitoring and management of mental disorders. The current review aims to comprehensively present and critically discuss all available smartphone-based, wearable, and environmental sensors for detecting such parameters in relation to the treatment and/or management of the most common mental health conditions.
Introduction
Mental illness is a health condition that alters a person's thoughts, feelings, and/or behaviors and causes the person distress and difficulty in functioning. The aggregate lifetime prevalence of common mental disorders across 39 countries has been estimated at around 30% (1). Mental health disorders are a major contributor to global disease burden due to their high prevalence, impairment of critical brain functions, and clinical course that is either chronic or remitting and relapsing. Some of the general symptoms and warning signs for mental disorders include marked personality change, inability to cope with problems and daily activities, strange and grandiose ideas, extreme mood swings, excessive anxieties, violent behavior, and thinking or talking about suicide or harming oneself (2).
Conventionally, mental disorders are diagnosed by self-report screening questionnaires or based on the Diagnostic and Statistical Manual of Mental Disorders V. Therefore, clinical diagnosis of mental disorders is achieved through the patient's subjective description of the symptoms, interviews and psychological questionnaires, and the physician's own expertise. However, the conventional assessment of psychiatric disorders based on a patient's or informant's recall is subject to inherent biases, and unreliability as the key feature in mental disorders is the variation of mood over time. In addition, mental health disorders are often chronic and relapsing in nature, and thus, long-term treatment management and assessment are essential for patient symptom reduction and recovery, but this is difficult to achieve with traditional methods, which rely on retrospective reports that are subject to recall bias. The lack of accurate methods for characterizing behavior and the need for regular monitoring of mental health conditions poses several physical and economical challenges, which, in recent years, has prompted great interest in digital phenotyping of mental health disorders through monitoring of physiological and behavioral parameters that can be translated into biomarkers of conditions, such as depression, anxiety disorders, bipolar disorder, and schizophrenia.
Physiological measures, such as heart rate variability (HRV), skin temperature (ST), electromyography (EMG), blood volume pulse (BVP), blood pressure, and cortisol levels as well as behavioral data, including sleep duration, social activities, and voice features, are among the many factors influenced by many mental health disorders. Consequently, capturing physiological and behavioral data has been widely implemented for the monitoring and general management of mental health.
In general, context sensing, personal sensing, mobile sensing, and digital phenotyping are all terms referring to identifying behaviors that relate to physical and mental health (2). Personal sensing involves collecting and analyzing data from sensors embedded in the context of daily life to detect and measure physical properties. To translate the raw sensor data into markers of behaviors and states related to mental health, the captured data is converted into features that are used to define behavioral markers through machine learning (3). Smartphones and connected devices, such as smartwatches, can be used to monitor behavior through passive detection of location, acceleration, social activities, and voice features.
Passive collection of physiological and behavioral data has been implemented for the diagnosis of various mental health disorders. This includes a wide range of studies that have investigated smartphone-based, wearable, and environmental sensors for digital phenotyping and monitoring of mental health. Until recent years, direct measurement of physiological data (e.g., HR, sleep quality, skin conductance) has been impractical as such devices have traditionally been bulky, large, and expensive (4). However, the development of various wearable devices, such as wristbands, smartwatches, and fitness tracking devices, and the common use of smartphones have facilitated the collection of biologically significant data. As a result, new technologies are rapidly being developed in fostering mental health and will allow for collection and analysis of data derived from self-reports, monitoring of behavioral patterns, and physiological sensing. More importantly, patients with serious mental illnesses have mostly shown good adherence and interest in using such devices to collect physiological and behavioral data (4), which would provide a vital complement to clinical visits.
The enthusiastic consumer and patient use of mobile devices, the availability of high-speed wireless internet, and a high prevalence of connected mobile devices with operating systems have culminated in patient and provider interest in monitoring mental symptoms and capturing behavioral and physiological data. This paper provides a review on recent developments in digital phenotyping based on physiological and behavioral monitoring of symptoms related to mental health and discusses the impact of such technologies in mental health care as, despite the expanded interest in mobile and connected technology in the arena of mental health, very few e-tools have been well-studied and validated for their impact when implemented on a broad scale (4).
Methods
Searches of the literature were conducted in Web of Science, PubMed, and Ovid [including Journals from Ovid, CityLibrary Journals@Ovid, AMED (Allied and Complementary Medicine), Embase, Global health, and Ovid MEDLINE]. Keywords used in this search included “sensors,” “mental health monitoring,” “personal sensing,” “mental disorders,” “physiological and behavioral monitoring,” and “digital phenotyping.” Database searches yielded 851 results of which 21 were review papers. The references of relevant review papers were scanned to identify applicable studies. From the combinations of the keywords and 36 relevant articles found in review references, 866 articles were identified. Studies investigating physiological and behavioral monitoring in any condition other than mental disorders were excluded. Studies in which no sensing device was employed for monitoring physiological and behavioral parameters were also excluded. From careful analysis of titles and abstracts, 139 articles were identified, 73 of which met the inclusion criteria and were included for analysis (Figure 1).
Figure 1. Diagram of the methodology used for the literature review process.
The literature search yielded 73 studies, 55 of which investigated wearable, mobile phone, and ambient sensors, summarized in Tables 1–3. The remaining 18 papers focused on identifying behavioral and physiological changes linked to mental health and used existing monitoring systems without considering and investigating a specific sensing mode. These studies focused on identifying mental health–related physiological and behavioral changes, some of which are specifically discussed in section Mental Health–Related Physiological and Behavioral Changes. The years of publications analyzing different sensing devices for mental health monitoring ranged from 2008 to 2020 with the years 2015 to 2020 producing the majority of publications (Figure 2). Wearables and sensor-enabled smartphone applications constituted the majority of studies with 24 and 22 publications, respectively. A total number of nine studies were found to investigate ambient sensors in mental health monitoring.
Table 1. Wearable physiological and emotional monitoring research prototypes.
Figure 2. Number of publications investigating monitoring systems in mental health from 2008 to 2020.
In sorting the included studies by application types, it is demonstrated that the majority of publications considered in this review investigated the use of monitoring systems for stress along with anxiety disorders and depression with just 2% difference between the two. Bipolar disorder monitoring, accounting for 15% of the studies, is the next well-studied condition for employment of behavioral and physiological monitoring systems. Schizophrenia and posttraumatic stress disorder (PTSD) monitoring comprise the smallest number of publications (Figure 3A). The application of different devices, including wearable, mobile phone, and ambient sensors, was also analyzed for each mental health condition. In the management of stress and anxiety disorders, wearable sensors were the most employed monitoring systems, mostly sensing physiological parameters, such as HR, electrodermal activity, and ST to evaluate anxiety levels. Whereas, for depression and bipolar disorder most of the studies used smartphone sensors as the monitoring system. In the monitoring of depression, the majority of studies focused on behavioral parameters, including physical activity, sleep duration, and conversation frequency to identify depressed mood. Furthermore, to assess the pathological mood states involved in bipolar disorder, physiological signals, such as HR and respiration activity, as well as behavioral parameters, such as mobility, social activity, sleep patterns, and voice features, were monitored. Although ambient sensors are generally the least investigated devices for mental health monitoring, they have been widely considered for detecting cognitive impairment (Figure 3B).
Figure 3. (A) Domain of the reviewed papers. (B) Investigated sensing devices for different mental disorders.
Mental Health–Related Physiological and Behavioral Changes
A wide range of studies have demonstrated the association between physiological and behavioral patterns and different mental disorders. For example, it has been suggested that demonstrations of more phone activity from a combination of increased GPS position changes, erratic accelerometer movements, and increased social activity may be indicative of the manic phase in bipolar patients (55). Moreover, analysis of voice data through machine learning and natural language processing and detection of the pitch, tempo, and loudness of voice might serve as a potential marker of many psychiatric illnesses, such as depression, anxiety, and even suicidality (56). Portable noise sensors and GPS trackers have been used to identify links between mental health and personal noise exposure with high exposure to noise at the entire day level associated with worse mental health (57). Moreover, HRV, respiratory sinus arrythmia (RSA), electrodermal activity (EDA), ST, EMG, BVP, blood pressure, and cortisol levels are among the physiological markers that react to emotional experiences and can be used to detect stress and other emotional states (58). Wearables can also measure variables such as skin conductance and HR with greater asymmetries in skin conductance amplitude on the two sides of the body identified as an indicator of emotional arousal (3). Therefore, sensor data can be used to monitor human emotion states. However, the signals identified as physiological responses during various emotional states should be stable and reliable (59). Data from accelerometer-based wearable devices have also been used to detect the association between physical activity and depression (60). Although decreased levels of mobility and social communications were found to be indicative of higher depressive symptoms, factors such as light intensity and smartphone screen usage were unlikely to be predictive of depressed mood (61). Also, physiological sensing, such as measurement of HRV data as a marker of decompensation, has shown promising results in monitoring schizophrenia. Furthermore, Depp et al. suggests that, in patients with schizophrenia, less GPS mobility is associated with greater negative symptom severity and motivational deficit (62). However, it is important to note that these relationships are not simple, and due to their complexities, passive data may still have limited clinical utility (63). Moreover, the validity of the data collected with consumer technologies and how the information collected on these platforms correlates with the disorders should be further investigated (4).
General Workflow of Mental Health Monitoring Systems
The internet of things–based layers involved in the structure of mental health monitoring systems is demonstrated in Figure 4. The general system collects health information from different sensing devices, including wearables, mobile phone sensors, and ambient sensors. These devices are capable of recording various physiological and behavioral data, including HR, lung function, sleep duration, etc. The sensing layer is typically followed by network and analysis layers, which involve employing the appropriate data acquisition and analysis techniques to eventually translate the raw sensor to achieve digital phenotyping in the application layer. The network layer comprising wired or wireless networks (e.g., Bluetooth, Wi-Fi) performs the collection and storage of data. Thereafter, the raw data is processed to extract features in the analysis layer, which is discussed in detail below. The activity classification step eventually categorizes the extracted features into different conditions in the application layer.
Figure 4. Overview of the workflow of mental health monitoring systems.
Sensing Layer
An important aspect in the development of mental health monitoring systems is the sensing layer, whose role is to collect and transmit physiological, behavioral, and/or environmental parameters, thereby detecting changes in these in a continuous manner. To achieve the measurement of attributes related to individuals, various sensors, such as inertial sensors (e.g., accelerometer, gyroscope, or barometric pressure sensors) or physiological sensors (e.g., spirometer, ST, and blood pressure sensors) can be used. Moreover, a number of on-object sensors, such as environmental sensors, can be used for measuring indoor environmental conditions, including humidity and temperature.
The studies investigating wearable and mobile monitoring systems require constant interactions between fields such as medicine and computer science. Therefore, prerequisites are an important phase in these studies, some of which include the need for user consent on data collection; ethical approvals; and IT infrastructure, software, and hardware requirements (64). Furthermore, the type and number of sensors are often determined on the basis of application, all while considering their impact on factors such as battery consumption, obtrusiveness, and privacy (65). Sensor sampling rates, defined as the frequency at which the data is collected, should also be determined. Although high sampling rates provide more fine-grained pattern details, its impact on the device battery life is an important consideration (65). The studies are often performed as longitudinal and quantitative to provide the data required for building automatic predictive models and to measure a certain state over a period of time. Clinicians, health researchers, and patients should also couple mental health monitoring systems with frequent standard self-report scales, such as the Hamilton Depression Scale and the Young Mania Rating Scale, to evaluate their correlations. Therefore, the wearables and sensor-enabled smartphone applications are combined with self-reports to compare the sensor-derived measures with the reported conditions and determine the associations between the sensor-based auto-report and the self-report.
Network Layer
The network layer is responsible for connecting all the devices in the sensory layer and allows the data to be collected, stored, transmitted, shared, and aggregated. To protect the participants' privacy, the data is anonymized, encrypted, transmitted via secure connections, and finally stored in secure servers. The latter can be performed in various ways; e.g., data acquired directly from smartphones can be stored in the internal device memory. Additional methods exist for data collected using wearables, such as watches and wristbands, with which data may be sent to a smartphone via Bluetooth or stored in secure storage platforms managing sensitive data (66) or to a remote server or a computer using wired, wireless, or internet connection. Various networks, including Blutooth, Wi-Fi, Zigbee, and cellular, can be adopted in the network layer.
Analysis Layer
Data Labeling
Machine learning algorithms rely on training data to find patterns and generate predictive models. Therefore, the data labeling phase, including tagging the sensor data with their corresponding ground truth state, is important in training the final predictive models. The data labeling phase can happen on-site, at the hospital/clinic, or off-site while the participants perform their daily routines at home/work/etc. Data tagging can be achieved through periodic specialist/doctor assessments on-site or by phone (66, 67), using self-reports often presented by a mobile application in periodic intervals in which the participant is in charge of the labeling process (68, 69) or according to some event, such as labeling stress during an exam situation (70).
Data Analysis and Preprocessing
After collecting the data, exploratory data analysis and preprocessing are used to better understand and visualize the data and detect outliers. Preprocessing includes applying filters and transformations, such as scaling, quantization, binarization, and so on, to the raw data to reduce noise and remove outliers. Dimensionality reduction techniques, including principle component analysis (71) and multidimensional scaling (72), are among the most common preprocessing techniques. Feature extraction is then used to build feature vectors from the raw sensor data. Feature vectors are required for machine learning algorithms as they are representations of the original data. Arithmetic mean, standard deviation, min, max, skewness, kurtosis, root mean square, power spectrum density, energy, and correlation coefficient are some of the common extracted features for mental states detection (68, 73, 74).
Machine Learning Model Training
Different types of machine learning algorithms, including supervised, unsupervised, semi-supervised, transfer, and reinforcement learning, are used to find particular patterns and relationships from the vast amount of collected data and finally yield the appropriate predictive model(s). It should be noted that algorithms are often combined to get the final predictive models. However, there is a distinction between certain training schemes, such as user-dependent and user-independent (general) models. User-dependent models, trained with data from the specific user under consideration and capturing the specific behavior of each user, yield better results but require a lot of data training. User-independent models, trained with data from all other users excluding the target user, do not require any data for the target user but might not perform well for atypical users. Although Lu et al. suggest that user-dependent models perform much better for stress detection (75), some studies propose that hybrid models have the best of both user-dependent and -independent models (68, 76).
As mentioned previously, physiological and/or behavioral monitoring systems are often coupled with data analysis platforms whose role is to convert raw sensor data into measures of mental disorders. Accordingly, different machine learning software tools/libraries are used for evaluating and training personalized or general models. These include the Weka software and the scikit-learn library used for bipolar detection (77, 78) and anxiety recognition (79), respectively. Furthermore, InSTIL (Intelligent Sensing to Inform and Learn) is a software platform designed for digital phenotyping. The platform provides acquisition of sensor data from consumer smartphones and is being used by studies that seek to collect passive and active sensor signals (80). Another developed platform is the Remote Assessment of Disease and Relapse (RADAR)-base platform, which can be integrated with remote monitoring initiatives and caters to large-scale data collection (81). These platforms provide general remote data collection at scale, management of studies, and real-time visualizations and ensure user privacy and security. However, other tools, such as Matlab machine learning toolboxes, can be also used based on different application types.
Application Layer
Remote mental health monitoring is of great significance to individuals with mental disorders as well as their caregivers and physicians. Physiological and behavioral patterns can reflect mental states of the patients, and thus, recording such data provides physicians and caregivers with a useful method for accurate intervention and diagnosis. For example, data from accelerometers, GPS, and mobile phone sensors can be used to detect physiological patterns associated with depression or depressed mood states involved in bipolar disorder. Furthermore, HR and EDA sensors can be employed to monitor physiological markers that react to emotional experiences and allow emotion recognition and the detection of stress levels. Sleep, mobility patterns, and conversation frequency and patterns are among the parameters used for monitoring schizophrenia. Last, microphone, visible light, accelerometer, temperature, and digital compass sensors are often utilized to provide cognitive assistance.
Mental health monitoring systems can be also used to facilitate monitoring of symptoms in home and hospital environments. Using these systems instead of the conventional questionnaires or manual tests delivers particular information for physicians and caregivers and, thus, potentially assists self-management of well-being, reduces health care costs, and avoids undesirable consequences in a personalized manner. Furthermore, employing these systems to provide ambient assisted living, particularly for the elderly, provides an emergency system that is essential for monitoring and detecting abnormal physiological and behavioral states.
Applications of Physiological and Behavioral Monitoring Systems
A wide range of devices aimed at physiological monitoring are presented, including wristbands and watches, chest straps, vests, garments and shirts, patches and sleeves (82–87). A list of wearable physiological and emotional monitoring systems can be found in Table 1. Various mental health studies on personal sensing have also utilized mobile phone sensors as they are widely owned and contain a large number of embedded sensors able to detect behavioral markers, such as sleep, social context, mood, and stress (Table 2). The use of smartphone sensors has been investigated for detecting the presence and severity of several mental disorders, including depression, bipolar disorder, and schizophrenia (29, 31, 88–90). Additionally, environmental or ambient sensors are generally installed in a single room, a house, or an entire building to understand the context of that environment by gathering information from sensors and providing assistance to the inhabitants. Ambient sensors have been investigated in some mental health monitoring studies, especially for detecting cognitive impairment (Table 3). The sections below discuss in detail the various wearable, ambient, and smartphone-based systems that have been investigated in recent years for treatment and management of common mental health disorders.
Table 2. Smartphone-based physiological and emotional monitoring research prototypes.
Table 3. Ambient mental health monitoring research prototypes.
Emotion and Autonomic Activity Recognition
Several wearable devices have been developed to monitor emotional information. The scarf-based monitor developed by Guo et al. uses an HR sensor and an EDA sensor to detect emotional states. The scarf also emits an odor and changes color to enhance the mood in response to negative emotions (Figure 5) (6). Similarly, a smart glove developed by Hui et al. utilizes a photoplethysmography (PPG) sensor for HR, EDA, ST, and EMG sensors to recognize different emotions, such as happiness, anger, fear, and disgust (7). To achieve emotion recognition using sensor data, a study by Jang et al. investigates the consistency on changes of bio-signals as physiological responses induced by six basic emotions—happiness, sadness, anger, fear, disgust, and surprise—using 60 different emotional stimuli. HR, skin conductance level (SCL), mean of ST (meanSKT), and mean of PPG (meanPPG) were measured before and during the presentation of the stimuli. The results suggest that biosensors are useful tools for emotion recognition as the physiological responses by six emotions were consistent over time; in particular, physiological features, such as SCL, HR, and PPG, are found to be very reliable (59). Moreover, a smart glass containing dye-sensitized photovoltaic cells, in which the optical sensors are positioned at the lateral side of the eye, has been designed for fatigue assessment based on eye blinks. The glass measures frequency, duration, and velocity of eye blinks as fatigue indices, which can potentially benefit the maintenance of physical and mental health (20). Also, a novel measurement technique using wearable eye tracking for mental health monitoring has been proposed based on the link between eye movements and cognition (14). A number of wearable monitoring systems have been studied to assess cognitive health in older adults. Using the data from a chest-worn sensor, it has been suggested that physical activity, including sedentary and light activities, percentage of walk and step count, and total sleep time and time in bed, are significant metrics for identifying cognitive frailty in older adults (23). Furthermore, a link between greater gait symmetry and better mental health has been identified as well by employing a high-speed 3-D motion sensing system to record gait mechanics in older adults (92). Another system is the mobile sensing platform, which is a small, wearable device designed for automatic activity recognition and cognitive assistance using on-body sensors, including microphone, visible light, pho-transistor, three-axis digital accelerometer, digital barometer, temperature, and digital compass (12).
Figure 5. Concept of a smart scarf designed for emotion recognition (91).
Among studies employing ambient sensors, Alam et al. objectified different home appliances and sensors as a web of objects to create a smart home environment and combined it with lightweight biosensors and web-based psychiatric screening scales to assess patients' psychiatric symptoms (49). Additionally, with agitation being a symptom of many mental illnesses, including dementia, Mielke et al. attempted to detect psychomotor agitation patterns via installation of sensors in the apartment and building a smart home environment (51). The study proposes that the movement sequences identified as conspicuous or critical can be indicative of psychomotor agitation or possible mood and behavioral changes (51). Particularly, ambient sensors have been used in a couple of studies to achieve monitoring of cognitive health in older adults. In a study by Kasteren et al. state change sensors were located in doors, cupboards, refrigerator, etc., to provide an activity monitoring system and assist in caregiving for the elderly in their homes (52, 93). The Smarter Safer Homes project (53) and SmartFABER (54) sought to detect cognitive decline and abnormal behaviors in the elderly by creating sensor-based in-home monitoring systems that acquire data about the interaction of the senior with the home environment. Early detection of cognitive impairment has also been investigated in (50), by installing an unobtrusive activity assessment system containing motion sensors and contact sensors to monitor the activity of older adults in their homes. Moreover, passive infrared motion sensors have been employed to monitor the daily activities of elderly persons and achieve long-term depression monitoring by creating smart homes (Figure 6) (47). Last, a smart home test bed was designed at Washington State University represented by an apartment that was instrumented with motion sensors on the ceiling and sensors on cabinets and doors as well as sensors on selected kitchen items. The task quality of smart home activities were quantified to assess the cognitive health of the individuals (48).
Figure 6. Sample configuration in an elderly house for long-term monitoring of depression. [Modified from Kim et al., IEEE Sensors, 2017 (47)].
Stress and Anxiety Disorders
Anxiety disorders are characterized by unrealistic and excessive worry accompanied by symptoms such as extreme vigilance, motor tension, restlessness, muscle tension, and disturbed sleep (94). Various devices have been developed for emotion recognition and stress management applications. As such, the MyFeel wristband (8) developed by Sentio Solutions, Inc., and Reveal (9), collect HR, EDA, and ST to evaluate anxiety levels by applying data analytics and cognitive-behavioral techniques. Thync (10) developed by Thync Global, Inc., employs electrical nerve stimulation (TENS) and transcranial direct current stimulation (tDCS) as neuro-stimulation technologies to increase energy and lower stress. TENS delivers small electrical impulses that reach the brain through the spinal cord and may help alleviate stress levels by relaxing the muscles (95). tDCS has been found to prevent chronic stress in a preliminary animal study by modulating the neural activity through the application of a small current (96). A wireless sensor suite called AutoSense (11) also provides information relating to general stress with 90% accuracy by collecting and processing cardiovascular, respiratory, and thermoregularity measurements. AutoSense integrates six sensors in a small device and includes wireless transmission to a mobile phone in real time, which provides monitoring of physiological responses to real-life stressors and various behaviors that may be related to stress, such as drinking, smoking, physical activity, movement patterns, conversations, etc., (Figure 7) (11). Furthermore, in a study by Can et al., HR activity, skin conductance, and accelerometer signals have been captured by smart wearable devices to detect and discriminate stress during situations such as contests, higher cognitive load lectures, and relaxed time activities using machine learning methods. The study used Samsung Smartwatches and Empatica E4 wristbands for data acquisition with higher accuracy and data quality obtained with the later class of devices (18). The combination of different physiological parameters seems to provide the highest accuracy for emotion classification. For example, Can et al. suggest that a higher classification accuracy was obtained when heart activity is combined with EDA than when these modalities were used separately (18).
Figure 7. AutoSense sensor suite including ECG leads and RIP connectors plug in to the unit on the side. The red clip in the wearable packaging is used to attach the unit to the (blue) RIP band that goes around the chest [Modified from Ertin et al., SenSys, 2011(11)].
In relation to stress detection, a portable stress detection system based on the RABio w8 (real-time acquisition of biosignals, wireless, eight channels) system has been designed, which is combined with software that comprises an application programming interface and a graphical user interface. By inducing different levels of stress, the system is demonstrated to be efficient in classifying three levels of stress (stress, relaxed, and neutral). The team is also working on a more wearable version of this system as a cap embedded with all the electronics (21). Furthermore, Jiang et al. develop a long-term wearable well-being sensing watch that continuously and unobtrusively detects details of behaviors that might be related to the onset of anxiety and autism (5). The portable wearable device consists of audio feature calculation without preserving the raw data and a variety of digital sensors for collecting multimodal data from the environment as well as physiological signals and behavioral activity. It has been demonstrated that the social audio features captured by the device are highly correlated with questionnaire scores for monitoring individuals with high to low autism traits (5). An undergoing study is also aiming to develop digital phenotyping of patients with alcohol use disorder and anxiety symptoms using data collected from a smartphone and a wearable sensor. In this study, factors including sleep cycle, HR, movement patterns, and sociability are sent and saved to the HumanITcare app (35).
Utilizing the sensors embedded in smart mobile phones, a new generation of well-being applications have been designed to automatically monitor multiple aspects of physical and mental health, for example, changes in the speech production process, which happens during stress. Therefore, microphones, embedded in mobile phones and carried ubiquitously by people, provide the opportunity for the non-invasive and continuous detection of voice-based stress (75). StressSense (75) and BeWell+ (27) are sensor-enabled smartphone apps designed to monitor user behavior. StressSense is designed to identify stress from the human voice using microphones embedded in mobile phones. BeWell+ continuously monitors user behavior, namely sleep, physical activity, and social interaction, and promotes improved behavioral patterns via feedback rendered as notifications, and this has been shown to successfully convey information and increase awareness (31). Ultimately, in the study by Boukhechba et al. it is suggested that mobility features, such as location entropy, are negatively associated with social anxiety with socially anxious students avoiding public areas and engagement in leisure activities. These findings were demonstrated by passively generating GPS data from an app installed on participants' personal mobile phones. Last, Ramon-Fernandez et al. propose an environmental stressor monitoring system to identify stressful environments based on parameters such as level of noise in the room, flow of people moving through it, temperature, luminosity, and air quality (46).
Depression
Depression is one of the most common mood disorders, and it is characterized by the absence of a positive affect; loss of interest in activities and experiences; low mood; and a range of associated emotional, cognitive, physical, and behavioral symptoms (94). Interpretation of data from smartphone-based geolocation sensors has identified the association between digital markers and mental illness concepts, especially mood (97). These geolocation-derived digital markers include number of locations visited, distance traveled, and time spent at a specific location. GPS embedded in participants' smartphone, GSM cellular network, and Wi-Fi are the common tools for measuring geolocation (97). Conversation frequency and duration, sleep disruption, social withdrawal and avoidance, mobility and GPS features are among the factors related to depression that can be detected using smartphone sensors (3). The evidence-based behavior (eB2) app has been designed to identify changes in the mobility patterns of patients with depression based on the smartphone's native sensors and advanced machine learning and signal processing techniques. The app captures inertial sensors, physical activity, phone calls and message logs, app usage, nearby Bluetooth and Wi-Fi connections, and location. The study has proposed that some specific mobility pattern changes can be indicators of relapses or clinical changes; however, that might not always be the case (30). Similarly, in a study by Ben-Zeev et al. it is suggested that increased geospatial activity is associated with better depression scores. Nevertheless, the same study identified that geospatial activity and sleep duration are inversely associated with daily stress, and no correlations were identified between sleep duration, geospatial activity, or speech duration and loneliness (33). Furthermore, a momentary assessment study aims to predict future depressed mood within hour-to-hour time windows. Utilizing ideographically weighted machine learning models and passive mobile phone sensor data, the study shows the significant correlation between observed and predicted hourly mood (34).
A study by Jin et al. proposes an attention-based deep learning architecture combined with wearable sensing devices, which can effectively classify mental states by analyzing behavior signals and speech under different emotions (17). The sensing system of the wearable device is used for analyzing the degree of anxiety and depression and consists of 6-D acceleration and angular sensors, a temperature and humidity sensor, MEMS microphones, and an audio code unit (17). Also, a study by McGinnis et al. sought to identify anxiety and depression in children under the age of eight by developing objective measures. To recognize children with internalizing disorders, the study utilizes an instrumented fear induction task and captures the six degree–of-freedom movement of a child using data from a belt-worn inertial measurement unit. The results suggest that the collected motion data are sensitive to behaviors that are representative of child psychopathology (24). Last, a fingertip PPG sensor has been developed by Dagdanpurev et al. as a major depressive disorder (MDD) screening system (19). The system employs autonomic nerve transient responses induced by a mental task and logistic regression analysis to identify MDD patients from healthy subjects. The self-monitoring system achieved 83% sensitivity and 93% specificity in MDD screening determined by ECG-derived HRV (20).
Bipolar Disorder
Bipolar disorder (BD), characterized by recurrent episodes of depressed and manic mood states, is a leading cause of disability worldwide and is associated with significant functional impairment (98). In the management of BD, PSYCHE, a personalized wearable monitoring system was trialed on 10 bipolar patients exhibiting depression, hypomania, mixed state, and euthymia symptoms to assess the pathological mood states by recording physiological signals (Figure 8). The pathological mood states were assessed using a data mining strategy by recording the physiological signals through information collected from the autonomic nervous system. PSYCHE employs a commonly used measure of entropy to analyze more than 400 h of cardiovascular dynamics and characterize the symptoms. This helped to improve the diagnosis and management of psychiatric disorders (15). The system consists of a comfortable t-shirt with embedded sensors, such as textile electrodes, to monitor electrocardiogram (ECG) for HRV series, piezoresistive sensors for respiration activity, and tri-axial accelerometers for activity recognition, and provides a smartphone-based interactive platform and data visualization to the patient and physician, respectively. The t-shirt with the above-described sensors is coupled with embedded electronics to acquire and store the data and an internal tri-axial accelerometer to monitor movement activity (15). Once the electronic device is detected by a mobile application through Bluetooth, the physiological data streaming is initiated. Several algorithms are then used to process the physiological data collected and correlate it with the mental health status of the patient. The data is then uploaded automatically by patients who are connected at home to produce predictive results that allow the physician to optimize the patient's treatment. Therefore, the textile-integrated platform together with the smartphone framework offered long-term acquisition of HRV data and helped discriminate different pathological mood states and investigated the response to treatment on BD patients (15). It has been demonstrated that data coming from the PSYCHE platform (long-term HRV series) could be considered as a viable biomarker for discriminating of bipolar patients and their response to treatment (15).
Figure 8. Overview of the PSYCHE wearable monitoring system. The system involves a T-shirt with embedded textile sensors, mobile application for physiological signal acquisition from the wearable platforms, and feedback to patient and clinician [Modified from Lanata et al., IEEE Journal of Biomedical and Health Informatics, 2015 (15)].
In a study by Prociow et al. wearable sensing techniques were coupled with environmental sensors to observe the areas of life influenced by BD and match the collected information with bipolar symptoms (16). The system included a wearable light sensor and a belt-worn accelerometer, which was used to detect restless behaviors, such as psychomotor agitation and increased or decreased physical activity to identify mood states. Moreover, the environmental system included bed sensors and light detectors installed in the patients' home to monitor altered sleep patterns, such as insomnia, hypersomnia, and self-deprivation of sleep, which are important diagnostic indictors that a manic or depressive episode is occurring. Basic processing performed on the behavioral data yielded from these sensors provided information about early effects of a bipolar episode on activity patterns. The study was performed on four healthy subjects and one participant with BD. The recruited bipolar patient remained euthymic (asymptomatic) throughout the monitoring period (16).
Combining self-reports and passive sensor data, several studies have sought to monitor mood fluctuations involved in bipolar disorder. MoodRhythm is a smartphone application that combines self-report and passive sensing via the use of smartphone sensors for the long-term monitoring of bipolar disorder. The application incorporates existing self-report strategies from interpersonal and social rhythm therapy and combines them with inputs from smartphone sensors (28). Additionally, the MONARCA system was developed to investigate the monitoring, treatment and prediction of bipolar disorder episodes (Figure 9) (67, 78, 100–102). The MONARCA project is a smartphone-based behavior monitoring technology that leverages a variety of phone sensors to detect changes in mental states. The system also includes a feedback loop between patients and clinicians through a web portal that provides detailed historical overviews of a patient's data to generate measures of illness activity (29, 55). The system investigates self-monitored (subjective) or sensor-based automatically generated (objective) behavioral data to monitor mood states in patients with bipolar disorder. The automatically generated behavioral data is collected by smartphones supporting different types of sensors. Some of the sensor-based data on measures of illness activity include phone usage, social activity measured as the number and length of incoming and outgoing phone calls and text messages, physical activity measured through a step counter, mobility based on location estimation, and speech activity collected by extraction of different voice features without recording the actual conversation. The study was performed for a 9-month trial period to investigate the correlations between subjective and objective behavioral data and the severity of depressive and manic symptoms. Altogether, the MONARCA system has been shown to recognize the clinical states of bipolar patients (depression/mania) with 72–81% accuracy based on location features. The system also shows high acceptance within the patients and health care providers and is proved to be effective in reducing the level of bipolar symptoms (29). Among other studies targeting bipolar patients, Social Information Monitoring for Patients with Bipolar Affective Disorder (SIMBA) is a smartphone monitoring app tracking daily mood, physical activity, and social communication in bipolar patients (32). Using the data collected from the SIMBA app, it has been suggested that the distance traveled by the bipolar patient is negatively associated with manic states (32). However, this is contrary to the findings of a study by Faurholt-Jepsen et al. (67) suggesting that the number of changes in communication are positively associated with manic states.
Figure 9. The MONARCA application user interface [Modified from Bardram et al., 2013 (99)].
PTSD
PTSD follows significant trauma and is characterized by emotional numbness; intrusive reliving of the traumatic episode; disturbed sleep, including nightmares; and hyperarousal, such as exaggerated startle responses (94). A couple of wearable monitoring systems have been developed for individuals with PTSD. Namely, a wrist-worn actigraphy sensor has been investigated for the monitoring of activity, sleep, and circadian patterns in PTSD, which involves long-lasting symptoms, such as avoidance behaviors and sleep disturbance. The proposed sleep detection algorithm has been demonstrated to match the participants' sleep diaries. Also, determined from the greater nocturnal activity and awakenings, participants with PTSD were found to exhibit more fragmented sleep patterns compared with non-traumatized control groups (22). Furthermore, a wearable sensor system has been designed and implemented consisting of a neoprene band worn on the ankle as well as an optional custom-designed ECG heart monitor worn on the chest, both containing a Bluetooth radio that enables communication with the patient's mobile phone. The sensor bands contain circuitry for measuring EDA, three-axis motion, temperature, and ECG. The system has been studied for PTSD and drug addiction and works by presenting therapeutic and empathetic messages to the patient in the tradition of cognitive behavioral therapy when a specific arousal event is detected (13).
Schizophrenia
Schizophrenia is a neuropsychiatric syndrome exhibiting psychotic symptoms, such as hallucinations and delusions; negative symptoms, including loss of motivation and blunted affect; and last, cognitive symptoms, including impairments in attention, working memory, verbal fluency, and various aspects of social cognition (94). Continuous remote monitoring and identification of subjective and objective indicators of psychotic relapse is known to improve the management of schizophrenia-spectrum disorders. Accordingly, the CrossCheck project has identified statistically significant associations between passive smartphone sensor data and self-reported indicators of mental health in schizophrenia (31, 90). The study suggests that lower rates of physical activity are linked to negative mental health. Also, it is demonstrated that patients showing lower sociability indicated particularly with fewer conversations during the morning and afternoon periods are likely to exhibit negative feelings. Nevertheless, it has been indicated that an increased number of phone calls and SMS messages can also be associated with negative dimensions of mental health as some individuals prefer to use the phone instead of face-to-face communication during negative mental states. Among other findings from monitoring of individuals with schizophrenia-spectrum disorders is that visiting fewer new places and going to bed later is associated with negative feelings, and getting up earlier is suggested to be linked with positive mood (103, 104).
Discussion
The current review presents a comprehensive analysis of developed sensor-based strategies in the management of mental disorders that seek to monitor physiological and behavioral data as indirect measures of health states. The literature review identifies studies investigating monitoring systems for the management of different mental disorders, including stress and anxiety disorders, depression, BD, cognitive impairment, PTSD, and schizophrenia. Principally, mental health monitoring systems collect and interpret physiological and behavioral data to relate them to the symptoms of mental conditions and achieve digital phenotyping. Thereby, a number of environmental, wearable, and mobile phone sensors have been employed to monitor parameters such as HRV, ST, EMG, BVP, blood pressure, etc. In each study, the type and variety of sensors were determined based on the application of the monitoring system as well as the behavioral and physiological parameters that were expected to be altered as a consequence of a specific disorder. The predicted mood states were often compared with the self-reports coupled with the monitoring systems. The majority of the studies have also suggested specific correlations between the collected behavioral and physiological parameters and the studied mental disorder. However, it should be noted that most of the developed technologies have not been thoroughly validated for their usefulness in clinical applications. Furthermore, the majority of monitoring systems target the general public and healthy individuals for health and fitness applications and are not certified as medical devices, and their impact on emotional states needs to be assessed (58). Also, these studies are subject to small sample sizes and short follow-up times. Therefore, there remain many technical and practical issues to overcome in digital approaches to mental disorders that include maintaining compliance over time, constraints on battery, manufacturing, durability, integration, constraints with analysis and interpretation of information, data accuracy, ethics/privacy issues, acceptability, and engagement issues (98). The mental health monitoring systems employing user-independent (general) models might also need to consider diversities such as age and ethnicity when introducing a predictive model. Moreover, despite the rapid development of sensing technology for monitoring physiological and behavioral data, challenges remain for the effective use of this abundant data. These challenges include processing the raw sensor data and managing the noise and artifacts, considering intrapersonal and interpersonal differences when interpreting the data, cost-effectiveness, and practicality of the device, providing privacy-protecting strategies, designing an accessible data storage and monitoring platform, and providing personalized coaching strategies. The only way to overcome these challenges is by interdisciplinary teams sharing expertise and methods and involvement of end users.
Although body-worn sensors that recognize human activities have the advantage of being at the user's hand, deploying these systems imposes some constraints, including protecting the user's privacy, being lightweight, offering limited computing power, handling the real world's noisy data and complexities, and comprising machine-learning algorithms that are trainable without requiring extensive human supervision. For instance, to avoid privacy issues, speech information should be automatically evaluated without preserving raw audio data (5). Another constraint is that, among wearable systems, the participant acceptability varies between different device forms, which is often related to the appearance and comfort of the wearing sensors. Accordingly, in a study by Huberty et al., it is demonstrated that wrist and upper arm sensors are more preferred than a sensor worn on the non-dominant hip in a sample of middle-aged women (105). Moreover, the results from a patient trial address the issue of non-adhering due to the discomfort of carrying extra devices, forgetfulness, and lack of familiarity with personal technology (16). Therefore, wearing and carrying extra devices on a day-to-day basis has been found to be impractical, and pursuing the environmental sensing via only a mobile phone seems to be ideal. However, it should be noted that wearable sensor features including skin conductance and temperature were found to be more accurate (78.3% accuracy) in identifying stress and poor mental health than mobile phone features (73.5% accuracy) such as phone usage and mobility patterns (106). Another shortcoming of mobile phone apps monitoring behavioral markers is their noticeable effect on the battery life, which can be a problem for users (61). Altogether, incorporating sensors monitoring physiological and behavioral parameters into a single, easy-to-use device has potential for the development of a practical mental health monitoring system. Furthermore, there is a crucial need for transparency and collaborative partnership between providers and patients because trust in the organization that collects the data and the purpose of data collection are the main factors for acceptability of the sensing systems. Additionally, ambient sensors can be expensive and limited to a particular physical space, their proper functioning requires calibration, and they might also raise privacy concerns. Nevertheless, these sensors have the advantage of not requiring direct contact with the user, hence, being free of wearing or carrying devices. Last, there is crucial need for patients, psychiatrists, and psychologists to adopt such means into their treatment and general practice (16).
When considering physiological and behavioral data as measures of mental health, it is important to consider that sensors do not sense the mental state itself, but a behavior that is the consequence of an underlying physiological alteration. Furthermore, relying solely on physiological sensor data might not be optimal as body changes may occur from other factors, such as food or medication intake and physical exercise. Thus, the mental health monitoring system should also consider the individual characteristics of each user. Another important aspect that should be considered is that the nature of the collected data from these sensors differs from the usual information that is available to and discussed with health care professionals. Patients have been found to express more willingness to consent to passive data retrieval from less personal sources, such as mobile phone screen time, whereas respondents were found to be at least willing to consent to more personal data, such as communication and location data, as well as audio recording and analysis (107). It has been noted that participants' comfort with sharing data depends on the data type and the recipient. Although some individuals were shown to be comfortable with sharing their health information, including sleep, mood, and physical activity with their doctors, they were less comfortable sharing personal data, such as communication logs, location, and social activity, especially with the electronic health record systems and their family (108).
Although there have been great advances in the automatic monitoring of mental health, there remain challenges that can pose several research opportunities. One of the key steps in the development of mental health monitoring systems is data labeling, which is used for finding associations between the sensor data and the corresponding true mental state at that time span. Therefore, machine learning models should be further explored to evaluate their potential for mental health monitoring. Another aspect that needs to be considered when monitoring physiological and behavioral patterns is the inter-user differences. As aforementioned, user-dependent models are suggested to perform better than user-independent (general) models, and hybrid models can be used to combine the strengths of both models (68, 75). Therefore, the potential of this type of hybrid model should be further explored. Additionally, intra-user differences that refer to the variability of physiological and behavioral patterns for the same user should also be addressed. Last, the mental health monitoring systems need to be clinically validated and should be integrated with other systems, including user databases, administrative tools for caregivers and physicians, and support systems to achieve optimum communication platforms. The application of multimodal sensing technologies along with the appropriate machine learning methods is of significance for achieving monitoring of mental health.
Conclusion
In conclusion, facile physiological and behavioral sensing systems have the potential to directly enhance the management and monitoring of mental health. Use of these devices can increase patients' self-awareness, which can positively affect the management of their condition. This consciousness about one's mental state can prevent the worsening and further adverse effects of many mental disorders. Therefore, technology needs to be established in mental health as it has in other fields of medicine to provide practical and point-of-care devices for those who are affected. Development of monitoring devices will aid patients in the management of their condition and immensely increase their quality of life.
Author's Note
The review paper has been written by MS under the supervision of MQ and PK. MQ and PK have reviewed the paper manuscript and made comments and statements. The authors of the review have complementary expertise, ensuring a balanced viewpoint. PK is a leader in the field of medical devices, developing innovative applications for the technology in a range of settings. MQ is a lecturer in biomedical engineering with expertise in optical spectroscopy and developing minimally-invasive monitoring devices. MS research focuses on non-invasive lithium monitoring in the management of bipolar disorder. Together, the authors provide a broad perspective on the topic, ensuring that the review is of interest to a wide audience.
Author Contributions
MS and MQ: conceptualization and methodology. MQ and PK: validation, investigation, and supervision. MS: writing-original draft preparation, writing-review, and editing. All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication, have read and agreed to the published version of the manuscript.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
References
1. Steel Z, Marnane C, Iranpour C, Chey T, Jackson JW, and Patel V. The global prevalence of common mental disorders: a systematic review and meta-analysis 1980–2013. Int. J. Epidemiol. (2014) 43:18. doi: 10.1093/ije/dyu038
2. Mohr DC, Shilton K, and Hotopf M. Digital phenotyping, behavioral sensing, or personal sensing: names and transparency in the digital age. Npj Digit Med. (2020) 3:45. doi: 10.1038/s41746-020-0251-5
3. Mohr DC, Zhang M, and Schueller SM. Personal sensing: understanding mental health using ubiquitous sensors and machine learning. Annu Rev Clin Psychol. (2017) 13:23–47. doi: 10.1146/annurev-clinpsy-032816-044949
4. Roberts LW, Chan S, and Torous J. New tests, new tools: mobile and connected technologies in advancing psychiatric diagnosis. Npj Digit Med. (2018) 1:20176. doi: 10.1038/s41746-017-0006-0
5. Jiang L, Gao B, Gu J, Chen Y, Gao Z, and Ma X. Wearable long-term social sensing for mental wellbeing. IEEE Sens J. (2019) 199:8532–42. doi: 10.1109/JSEN.2018.2877427
6. Guo C, Chen YV, Qian ZC, Ma Y, Dinh H, and Anasingaraju S. Designing a smart scarf to influence group members' emotions in ambience: design process and user experience. In: Antona M, Stephanidis C, editors. International Conference on Universal Access in Human-Computer Interaction. Interaction Techniques and Environments. Vol. 9738. Cham: Springer. doi: 10.1007/978-3-319-40244-4_38
7. Hui TKL, and Sherratt RS. Coverage of emotion recognition for common wearable biosensors. Biosensors. (2018) 8:30. doi: 10.3390/bios8020030
8. Sentio Solutions Inc. MyFeel'. Available online at: https://www.myfeel.co/science (accessed March 24, 2021).
9. Awake Labs. Reveal'. Available online at: https://awakelabs.com (accessed March 24, 2021).
10. Thync Global Inc. Thync'. Available online at: https://www.thync.com (accessed March 24, 2021).
11. Ertin E, Stohs N, Kumar S A, Raij al'Absi M, and Shah S. AutoSense: unobtrusively wearable sensor suite for inferring the onset, causality, and consequences of stress in the field. In: Proceedings of the 9th ACM Conference on Embedded Networked Sensor Systems - SenSys '11. Seattle, WA. (2011). p. 274.
12. Choudhury T, Borriello G, Consolvo S, Haehnel D, Harrison B, and Hemingway B. The mobile sensing platform: an embedded activity recognition system. IEEE Pervasive Comput. (2008) 7:32–41. doi: 10.1109/MPRV.2008.39
13. Fletcher RR, Tam S, Omojola O, Redemske R, and Kwan J. Wearable sensor platform and mobile application for use in cognitive behavioral therapy for drug addiction and PTSD. Conf Proc IEEE Eng Med Biol Soc. (2011) 2011:1802–5. doi: 10.1109/IEMBS.2011.6090513
14. Vidal M, Turner J, Bulling A, and Gellersen H. Wearable eye tracking for mental health monitoring. Comput Commun. (2012) 351:1306–11. doi: 10.1016/j.comcom.2011.11.002
15. Lanata Valenza G, Nardelli M, Gentili C, and Scilingo EP. Complexity index from a personalized wearable monitoring system for assessing remission in mental health. IEEE J Biomed Health Inform. (2015) 19:132–9. doi: 10.1109/JBHI.2014.2360711
16. Prociow P, Wac K, and Crowe J. Mobile psychiatry: towards improving the care for bipolar disorder. Int J Ment Health Syst. (2012) 6:5. doi: 10.1186/1752-4458-6-5
17. Jin J, Gao B, Yang S, Zhao B, Luo L, and Woo WL. Attention-block deep learning based features fusion in wearable social sensor for mental wellbeing evaluations. IEEE Access. (2020) 8:89258–68. doi: 10.1109/ACCESS.2020.2994124
18. Can YS, Chalabianloo N, Ekiz D, and Ersoy C. Continuous stress detection using wearable sensors in real life: algorithmic programming contest case study. Sensors. (2019) 19:1849. doi: 10.3390/s19081849
19. Dagdanpurev S, Sun G, Shinba T, Kobayashi M, Kariya N, and Choimaa L. Development and clinical application of a novel autonomic transient response-based screening system for major depressive disorder using a fingertip photoplethysmographic sensor. Front Bioeng Biotechnol. (2018) 6:64. doi: 10.3389/fbioe.2018.00064
20. Horiuchi R, Ogasawara T, and Miki N. Fatigue assessment by blink detected with attachable optical sensors of dye-sensitized photovoltaic cells. Micromachines. (2018) 9:310. doi: 10.3390/mi9060310
21. Minguillon J, Perez E, Lopez-Gordo M, and Pelayo FM. Sanchez-carrion. Portable system for real-time detection of stress level. Sensors. (2018) 18:2504. doi: 10.3390/s18082504
22. Tsanas A, Woodward E, and Ehlers A. Objective characterization of activity, sleep, and circadian rhythm patterns using a wrist-worn actigraphy sensor: insights into posttraumatic stress disorder. JMIR MHealth UHealth. (2020) 8:e14306. doi: 10.2196/14306
23. Razjouyan J, Najafi B, Horstman M, Sharafkhaneh A, Amirmazaheri M, and Zhou H. Toward using wearables to remotely monitor cognitive frailty in community-living older adults: an observational study. Sensors. (2020) 20:2218. doi: 10.3390/s20082218
24. McGinnis EW, McGinnis RS, Hruschak J, Bilek E, Ip K, and Morlen D. Wearable sensors detect childhood internalizing disorders during mood induction task. PLoS ONE. (2018) 13:e0195598. doi: 10.1371/journal.pone.0195598
25. Correia B, Dias N, Costa P, and Pêgo JM. Validation of a wireless bluetooth photoplethysmography sensor used on the earlobe for monitoring heart rate variability features during a stress-inducing mental task in healthy individuals. Sensors. (2020) 204:3905. doi: 10.3390/s20143905
26. Mohino-Herranz Gil-Pita R, Ferreira J, Rosa-Zurera M, and Seoane F. Assessment of mental, emotional and physical stress through analysis of physiological signals using smartphones. Sensors. (2015) 150:25607–27. doi: 10.3390/s151025607
27. Lin M, Campbell AT, Choudhury T, Lane ND, Mohammod M, and Yang X. BeWell+: multi-dimensional wellbeing monitoring with community-guided user feedback and energy optimization. In: Proceedings of the Conference on Wireless Health - WH '12. San Diego, CA. (2012). p. 1–8.
28. Matthews M, Abdullah S, Murnane E, Voida S, Choudhury T, and Gay G. Development and evaluation of a smartphone-based measure of social rhythms for bipolar disorder. Assessment. (2016) 23:472–83. doi: 10.1177/1073191116656794
29. Faurholt-Jepsen M, Vinberg M, Frost M, Christensen EM, Bardram J, and Kessing LV. Daily electronic monitoring of subjective and objective measures of illness activity in bipolar disorder using smartphones– the MONARCA II trial protocol: a randomized controlled single-blind parallel-group trial. BMC Psychiatry. (2014) 14:309. doi: 10.1186/s12888-014-0309-5
30. Berrouiguet S, Ramírez D, Barrigón ML, Moreno-Muñoz P, Camacho RC, and Baca-García E. Combining continuous smartphone native sensors data capture and unsupervised data mining techniques for behavioral changes detection: a case series of the Evidence-Based Behavior (eB2) study. JMIR Mhealth Uhealth. 21:e16399. doi: 10.2196/16399
31. Wang R, Aung MSH, Abdullah S, Brian R, Campbell AT, and Choudhury T. CrossCheck: toward passive sensing and detection of mental health changes in people with schizophrenia. In: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, Heidelberg. (2016). p. 886–897.
32. Beiwinkel T, Kindermann S, Maier A, Kerl C, Moock J, and Barbian G. Using smartphones to monitor bipolar disorder symptoms: a pilot study. JMIR Ment Health. (2016) 3:e2. doi: 10.2196/mental.4560
33. Ben-Zeev D, Scherer EA, Wang R, Xie H, and Campbell AT. Next-generation psychiatric assessment: using smartphone sensors to monitor behavior mental health. Psychiatr Rehabil J. (2015) 38:218–26. doi: 10.1037/prj0000130
34. Jacobson NC, and Chung YJ. Passive sensing of prediction of moment-to-moment depressed mood among undergraduates with clinical levels of depression sample using smartphones. Sensors. (2020) 202:3572. doi: 10.3390/s20123572
35. Pastor N, Khalilian E, Caballeria E, Morrison D, Sanchez Luque U, and Matrai S. Remote Monitoring Telemedicine (REMOTE) platform for patients with anxiety symptoms and alcohol use disorder: protocol for a case-control study. JMIR Res Protoc. (2020) 9:e16964. doi: 10.2196/16964
36. Schueller SM, Begale M, Penedo FJ, and Mohr DC. Purple: a modular system for developing deploying behavioral intervention technologies. J Med Internet Res. (2014) 16:e181. doi: 10.2196/jmir.3376
37. Burns MN, Begale M, Duffecy J, Gergle D, Karr CJ, and Giangrande E. Harnessing context sensing to develop a mobile intervention for depression. J Med Internet Res. (2011) 13:e55. doi: 10.2196/jmir.1838
38. Gideon Provost EM, and McInnis M. Mood state prediction from speech of varying acoustic quality for individuals with bipolar disorder. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Shanghai. (2016) p. 2359–2363.
39. Hidalgo-Mazzei D, Mateu A, Reinares M, Undurraga J C, del Bonnín M, and Sánchez-Moreno J. Self-monitoring and psychoeducation in bipolar patients with a smart-phone application (SIMPLe) project: design, development and studies protocols. BMC Psychiatry. (2015) 15:52. doi: 10.1186/s12888-015-0437-6
40. Asare O, Visuri A, and Ferriera DST. Towards early detection of depression through smartphone sensing. In: Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers - UbiComp/ISWC '19. London. (2019). p. 1158–1161.
41. Cao Truong AL, Banu S, Shah AA, Sabharwal A, and Moukaddam N. Tracking and predicting depressive symptoms of adolescents using smartphone-based self-reports, parental evaluations, and passive phone sensor data: development and usability study. JMIR Ment Health. (2020) 7:e14045. doi: 10.2196/14045
42. Dargél Mosconi E, Masson M, Plaze M, Taieb F, and Von Platen C. Toi Même, a mobile health platform for measuring bipolar illness activity: protocol for a feasibility study. JMIR Res Protoc. (2020) 9:e18818. doi: 10.2196/18818
43. DeMasi O, Feygin S, Dembo A, Aguilera A, and Recht B. Well-being tracking via smartphone-measured activity and sleep: cohort study. JMIR MHealth UHealth. (2017) 50:e137. doi: 10.2196/mhealth.7820
44. Wahle F, Kowatsch T, Fleisch E, Rufer M, and Weidt S. Mobile sensing and support for people with depression: a pilot trial in the wild. JMIR MHealth UHealth. (2016) 4:e111. doi: 10.2196/mhealth.5960
45. Daus H, Bloecher T, Egeler R, De Klerk R, Stork W, and Backenstrass M. Development of an emotion-sensitive mHealth approach for mood-state recognition in bipolar disorder. JMIR Ment Health. (2020) 7:e14267. doi: 10.2196/14267
46. de Ramón-Fernández, Ruiz-Fernández D, and Marcos-Jorquera D V. Gilart-Iglesias. A distributed model for stressors monitoring based on environmental smart sensors. Sensors. (2018) 18:1935. doi: 10.3390/s18061935
47. Kim JY, Liu N, Tan HX, and Chu CH. Unobtrusive monitoring to detect depression for elderly with chronic illnesses. IEEE Sens J. (2017) 177:5694–704. doi: 10.1109/JSEN.2017.2729594
48. Dawadi PN, Cook DJ, Schmitter-Edgecombe M, and Parsey C. Automated assessment of cognitive health using smart home technologies. Technol Health Care. (2013) 21:323–43. doi: 10.3233/THC-130734
49. Alam M, Abedin S, Al Ameen M, and Hong C. Web of objects based ambient assisted living framework for emergency psychiatric state prediction. Sensors. (2016) 16:1431. doi: 10.3390/s16091431
50. Hayes TL, Abendroth F, Adami A, Pavel M, and Zitzelberger TA. Unobtrusive assessment of activity patterns associated with mild cognitive impairment. (2009). doi: 10.4017/gt.2010.09.02.109.00
51. Mielkea C, and Antonsa R. Detection of psychomotor agitation pattern from motion sensor data in a living environment of a patient with dementia. Digit Pers Health Med. (2020) 270:746–50. doi: 10.3233/SHTI200260
52. van Kasteren TLM, Englebienne G, and Kröse BJA. An activity monitoring system for elderly care using generative and discriminative models. Pers Ubiquitous Comput. (2010) 14:489–98. doi: 10.1007/s00779-009-0277-9
53. Bradford D. Detecting degeneration: monitoring cognitive health in independent elders. Nnu Int Conf IEEE Eng Med Biol Soc. (2013) 2013:029–032. doi: 10.1109/EMBC.2013.6611176
54. Ribonia D, and Bettini C. SmartFABER: recognizing fine-grained abnormal behaviors for early detection of mild cognitive impairment. Artif Intell Med. (2016) 67:57–74. doi: 10.1016/j.artmed.2015.12.001
55. Frost M, Marcu G, Hansen R, Szaántó K, and Bardram J. The MONARCA self-assessment system: persuasive personal monitoring for bipolar patients. In: Presented at the 5th International ICST Conference on Pervasive Computing Technologies for Healthcare, Dublin: Republic of Ireland. (2011). doi: 10.4108/icst.pervasivehealth.2011.246050
56. Cummins N, Scherer S, Krajewski J, Schnieder S, Epps J, and Quatieri TF. A review of depression and suicide risk assessment using speech analysis. Speech Commun. (2015) 71:10–49. doi: 10.1016/j.specom.2015.03.004
57. Ma J, Li C, Kwan MP, Kou L, and Chai Y. Assessing personal noise exposure and its relationship with mental health in Beijing based on individuals' space-time behavior. Environ Int. (2020) 139:105737. doi: 10.1016/j.envint.2020.105737
58. Taj-Eldin M, Ryan C, O'Flynn B, and Galvin P. A review of wearable solutions for physiological and emotional monitoring for use by people with autism spectrum disorder and their caregivers. Sensors. (2018) 182:4271. doi: 10.3390/s18124271
59. Jang EH, Kim AY, Kim SH, Yu HY, and Sohn JH. Internal consistency of physiological responses during exposure to emotional stimuli using biosensors'. In: Proceedings of the 6th International Joint Conference on Pervasive and Embedded Computing and Communication Systems. Lisbon. (2016). p. 110–115.
60. Vallance JK. Associations of objectively-assessed physical activity and sedentary time with depression: NHANES (2005–2006). Prev Med. (2011) 53:284–8. doi: 10.1016/j.ypmed.2011.07.013
61. Boonstra TW, Nicholas J, Wong QJ, Shaw F, Townsend S, and Christensen H. Using mobile phone sensor technology for mental health research: integrated analysis to identify hidden challenges potential solutions. J Med Internet Res. (2018) 20:e10131. doi: 10.2196/10131
62. Depp CA, Bashem J, Moore RC, Holden JL, Mikhael T, and Swendsen J. GPS mobility as a digital biomarker of negative symptoms in schizophrenia: a case control study. Npj Digit Med. (2019) 2:108. doi: 10.1038/s41746-019-0182-1
63. Bedi G, Carrillo F, Cecchi GA, Slezak DF, Sigman M, and Mota NB. Automated analysis of free speech predicts psychosis onset in high-risk youths. Npj Schizophr. (2015) 1:15030. doi: 10.1038/npjschz.2015.30
64. Riegler M, Gurrin C, Johansen D, Johansen H, Halvorsen P, and Lux M. Multimedia and medicine: teammates for better disease detection and survival. In: Proceedings of the 2016 ACM on Multimedia Conference - MM '16. Amsterdam. (2016). p. 968–977.
65. Harari GM, Lane ND, Wang R, Crosier BS, Campbell AT, and Gosling SD. Using smartphones to collect behavioral data in psychological science: opportunities, practical considerations, and challenges. Perspect Psychol Sci. (2016) 11:838–54. doi: 10.1177/1745691616650285
66. University of Oslo. Services for sensitive data (TSD). (2018). Available online at: http://www.uio.no/english/services/it/research/sensitive-data/
67. Faurholt-Jepsen M, Vinberg M, Frost M, Debel S, Margrethe Christensen E, and Bardram JE. Behavioral activities collected through smartphones the association with illness activity in bipolar disorder: smartphone data in bipolar disorder. Int J Methods Psychiatr Res. (2016) 25:309–23. doi: 10.1002/mpr.1502
68. Garcia-Ceja E, Osmani V, and Mayora O. Automatic stress detection in working environments from smartphones' accelerometer data: a first step. IEEE J Biomed Health Inform. (2016) 20:1053–60. doi: 10.1109/JBHI.2015.2446195
69. Zenonos Khan A, Kalogridis G, Vatsikas S, Lewis T, and Sooriyabandara M. HealthyOffice: mood recognition at work using smartphones and wearable sensors. In: 2016 IEEE Annual Conference on Pervasive Computing and Communications Workshops. Sydney, NSW: PerCom (2016). p. 1–6. doi: 10.1109/PERCOMW.2016.7457166
70. Bauer G, and Lukowicz P. Can smartphones detect stress-related changes in the behaviour of individuals? In: 2012 IEEE Annual Conference on Pervasive Computing and Communications Workshops. Lugano: PerCom (2012). p. 423–426. doi: 10.1109/PerComW.2012.6197525
71. Karl Pearson FRS LIII. On lines planes of closest fit to systems of points in space. Lond Edinb Dublin Philos Mag J Sci. (2010) 21:559–72. doi: 10.1080/14786440109462720
72. GOWER JC. Some distance properties of latent root and vector methods used in multivariate analysis. Biometrika. (1966) 53:325–38. doi: 10.1093/biomet/53.3-4.325
73. Maxhuni Muñoz-Meléndez A, Osmani V, Perez H, Mayora O, and Morales EF. Classification of bipolar disorder episodes based on analysis of voice and motor activity of patients. Pervasive Mob Comput. (2016) 31:50–66. doi: 10.1016/j.pmcj.2016.01.008
74. Garcia-Ceja E, Riegler M, Jakobsen P, Tørresen J, Nordgreen T, and Oedegaard KJ. Depresjon: a motor activity database of depression episodes in unipolar and bipolar patients. In: Proceedings of the 9th ACM Multimedia Systems Conference, Amsterdam. (2018). p. 472–477.
75. Lu H, Frauendorfer D, Rabbi M, Mast MS, Chittaranjan GT, and Andrew T. Campbell. StressSense: detecting stress in unconstrained acoustic environments using smartphones. In: Proceedings of the 2012 ACM Conference on Ubiquitous Computing - UbiComp '12. Pittsburgh, PA (2012) 351.
76. Xu Q, Nwe TL, and Guan C. Cluster-based analysis for personalized stress evaluation using physiological signals. IEEE J Biomed Health Inform. (2015) 18:275–81. doi: 10.1109/JBHI.2014.2311044
77. Grünerbl A, Muaremi A, Osmani V, Bahle G, Ohler S, and Tröster G. Smartphone-based recognition of states state changes in bipolar disorder patients. IEEE J Biomed Health Inform. (2015) 19:140–8. doi: 10.1109/JBHI.2014.2343154
78. Gruenerbl Osmani V, Bahle G, Carrasco JC, Oehler S, and Mayora O. Using smart phone mobility traces for the diagnosis of depressive and manic episodes in bipolar patients. In: Proceedings of the 5th Augmented Human International Conference on - AH '14. Kobe. (2014). p. 1–8.
79. Miranda D, Favela J, and Arnrich B. Detecting anxiety states when caring for people with dementia. Methods Inf Med. (2017) 56:55–62. doi: 10.3414/ME15-02-0012
80. Barnett S, Huckvale K, Christensen H, Venkatesh S, Mouzakis K, and Vasa R. Intelligent Sensing to Inform Learn (InSTIL): a scalable governance-aware platform for universal, smartphone-based digital phenotyping for research clinical applications. J Med Internet Res. (2019) 211:e16399. doi: 10.2196/preprints.16399
81. Ranjan Y, Rashid Z, Stewart C, Conde P, Begale M, and Verbeeck D. RADAR-base: open source mobile health platform for collecting, monitoring, and analyzing data using sensors, wearables, mobile devices. JMIR MHealth UHealth. (2019) 7:e11734. doi: 10.2196/11734
82. ‘Wahoo Fitness'. Available online at: https://eu.wahoofitness.com/devices/heart-rate-monitors/wahoo-tickr-x-heart-rate-strap (accessed March 24, 2021).
83. ‘Suunto’. Available online at: https://www.suunto.com/Products/sports-watches/suunto-9/suunto-9-baro-black/ (accessed March 24, 2021).
84. ‘Healbe Inc.’ Available online at: https://healbe.com/us/gobe2/ (accessed March 24, 2021).
85. ‘VialConnect.’ Available online at: https://vitalconnect.com/solutions/vitalpatch/ (accessed March 24, 2021).
86. ‘Komodo, Technologies,.’ Available online at: http://komodotec.com/product/aio-sleeve/ (accessed March 24, 2021).
87. ‘BrainPower LLC.’ Available online at: http://www.brain-power.com/ (accessed March 24, 2021).
88. Saeb S, Lattie EG, Kording KP, and Mohr DC. Mobile phone detection of semantic location and its relationship to depression and anxiety. JMIR MHealth UHealth. (2017) 5:e112. doi: 10.2196/mhealth.7297
89. Saeb S, Zhang M, Karr CJ, Schueller SM, Corden ME, and Kording KP. Mobile phone sensor correlates of depressive symptom severity in daily-life behavior: an exploratory study. J Med Internet Res. (2015) 17:e175. doi: 10.2196/jmir.4273
90. Ben-Zeev D, Wang R, Abdullah S, Brian R, Scherer EA, and Mistler LA. Mobile behavioral sensing for outpatients and inpatients with schizophrenia. Psychiatr Serv. (2016) 67:558–61. doi: 10.1176/appi.ps.201500130
91. Guo C, Chen YV, Qian ZC, Ma Y, Dinh H, and Anasingaraju S. Designing a smart scarf to influence group members' emotions in ambience: design process and user experience. In: Antona M, Stephanidis C, editors. Universal Access in Human-Computer Interaction. Interaction Techniques and Environments. Vol. 9738. Cham: Springer (2016). p. 392–402.
92. Nagano H, Sarashina E, Sparrow W, Mizukami K, and Begg R. General mental health is associated with gait asymmetry. Sensors. (2019) 192:4908. doi: 10.3390/s19224908
93. van Kasteren T, Noulas A, Englebienne G, and Kröse B. Accurate activity recognition in a home setting. Assoc Comput. (2008). 5:e56. doi: 10.1145/1409635.1409637
94. National Collaborating Centre for Mental Health (Great Britain) National Institute for Health and Clinical Excellence (Great Britain) British Psychological Society and Royal College of Psychiatrists Common mental health disorders: identification and pathways to care. Liecester; London: British Psychological Society; Royal College of Psychiatrists (2011).
95. Yoshimoto S, and Babygirija R. Anti-stress effects of Transcutaneous Electrical Nerve Stimulation (TENS) on colonic motility in rats. Dig Dis Sci. (2012) 57:1213–21. doi: 10.1007/s10620-012-2040-8
96. Fregni F. Transcranial direct current stimulation (tDCS) prevents chronic stress-induced hyperalgesia in rats. Brain Stimulat. (2018) 11:299–301. doi: 10.1016/j.brs.2017.11.009
97. Fraccaro P, Beukenhorst A, Sperrin M, Harper S, Palmier-Claus J, and Lewis S. Digital biomarkers from geolocation data in bipolar disorder schizophrenia: a systematic review. J Am Med Inform Assoc. (2019) 261:1412–20. doi: 10.1093/jamia/ocz043
98. Harrison PJ, Geddes JR, and Tunbridge EM. The emerging neurobiology of bipolar disorder. Trends Neurosci. (2018) 41:18–30. doi: 10.1016/j.tins.2017.10.006
99. Bardram E, Frost M, Szántó K, Faurholt-Jepsen M, Vinberg M, and Kessing LV. Designing mobile health technology for bipolar disorder: a field trial of the monarca system. In: Proceedings of the Institute of Electrical and Electronics Engineers (IEEE) International Conference on Acoustics, Speech, and Signal Processing. ICASSP, Paris (2013). p. 2627.
100. Frost M, Doryab A, Faurholt-Jepsen M, Kessing LV, and Bardram JE. Supporting disease insight through data analysis: refinements of the monarca self-assessment system. in Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing - UbiComp '13. Zurich. (2013) 133.
101. Faurholt-Jepsen M, Frost M, Vinberg M, Christensen EM, Bardram JE, and Kessing LV. Smartphone data as objective measures of bipolar disorder symptoms. Psychiatry Res. (2014) 217:124–7. doi: 10.1016/j.psychres.2014.03.009
102. Fitzgerald J, and Fenniri H. Cutting edge methods for non-invasive disease diagnosis using e-tongue and e-nose devices. Biosensors. (2017) 7:59. doi: 10.3390/bios7040059
103. Ben-Zeev D, Scherer EA, Brian RM, Mistler LA, Campbell AT, and Wang R. Use of multimodal technology to identify digital correlates of violence among inpatients with serious mental illness: a pilot study. Psychiatr Serv. (2017) 680:1088–92. doi: 10.1176/appi.ps.201700077
104. Ben-Zeev D, Brian R, Wang R, Wang W, Campbell AT, Aung MSH, et al. CrossCheck: Integrating self-report, behavioral sensing, and smartphone use to identify digital indicators of psychotic relapse. Psychiatr Rehabil J. (2017) 40:266–75. doi: 10.1037/prj0000243
105. Huberty J, Ehlers DK, Kurka J, Ainsworth B, and Buman M. Feasibility of three wearable sensors for 24 hour monitoring in middle-aged women. BMC Womens Health. (2015) 15:55. doi: 10.1186/s12905-015-0212-3
106. Sano Taylor S, McHill AW, Phillips AJ, Barger LK, and Klerman E. Identifying objective physiological markers modifiable behaviors for self-reported stress mental health status using wearable sensors mobile phones: observational study. J Med Internet Res. (2018) 20:e210. doi: 10.2196/jmir.9410
107. Di Matteo D, Fine A, Fotinos K, Rose J, and Katzman M. Patient Willingness to Consent to Mobile Phone Data Collection for Mental Health Apps: Structured Questionnaire. JMIR Ment. Health. 5:e56. (2018). doi: 10.2196/mental.9539
108. Nicholas J, Shilton K, Schueller SM, Gray EL, Kwasny MJ, and Mohr DC. The role of data type and recipient in individuals' perspectives on sharing passively collected smartphone data for mental health: cross-sectional questionnaire study. JMIR MHealth UHealth. (2019) 7:e12578. doi: 10.2196/12578
Keywords: mental health monitoring, wearables, personal sensing, physiological and behavioral monitoring, digital phenotyping
Citation: Sheikh M, Qassem M and Kyriacou PA (2021) Wearable, Environmental, and Smartphone-Based Passive Sensing for Mental Health Monitoring. Front. Digit. Health 3:662811. doi: 10.3389/fdgth.2021.662811
Received: 01 February 2021; Accepted: 02 March 2021;
Published: 07 April 2021.
Edited by:
Pradeep Nair, Central University of Himachal Pradesh, IndiaReviewed by:
Carlo Massaroni, Campus Bio-Medico University, ItalyIan Cleland, Ulster University, United Kingdom
Copyright © 2021 Sheikh, Qassem and Kyriacou. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Panicos A. Kyriacou, p.kyriacou@city.ac.uk