Non-verbal behaviours as predictors for treatment response in patients with depression or schizophrenia: a systematic review

Introduction: Non-verbal behaviours (NVBs) reveal information about mood and emotions, potentially providing an objective means to measure and monitor early treatment responses. Previous research has examined NVB changes during treatment in patients with depression and psychosis but a systematic evaluation of the evidence is lacking. Furthermore, this review could inform the fast moving field of digital and precision psychiatry due to the use of AI-based technology that could transform the potential of NVBs as reliable biomarkers for treatment response.

Methods: Medline, Embase, PsycINFO and CINAHL were searched in June 2024. Included papers studied adults diagnosed with depression or schizophrenia and measured NVBs at least twice during separate clinical interviews. Outcomes of interest were changes in clinical symptoms and NVBs following treatment initiation. Treatment strategies included hospitalisation, pharmacological, psychological, neuromodulatory, other non-standardised interventions, or a combination of approaches. Two reviewers independently extracted data and assessed risk of bias. The protocol for the review was registered on PROSPERO.

Results: 20 papers were identified; 15 on depression, two on schizophrenia and three evaluating both conditions separately. Methodological variations across studies made comparisons challenging. NVBs consistently associated with improvement in depression symptoms included: increased smiling, facial expressivity, and amplified head and body movements. Results across studies were more consistent when considering general categories of behaviour, versus discrete facial behaviours. No commonalities were observed in NVB changes over time for patients with schizophrenia.

Discussion: The existing evidence is presently insufficient to establish distinct behavioural profiles for clinical improvement depression or schizophrenia. Despite implicit challenges, there is considerable future scope in the evaluation of NVBs as predictors of clinical outcomes or change.

Systematic Review Registration: https://www.crd.york.ac.uk/PROSPERO, identifier CRD42022368599.

Introduction

Non-verbal behaviour (NVB) is defined as any movement or position (including distance and proximity) of the face and/or body (1). Typical NVBs include facial expressions, gestures, eye contact and posture. NVBs are a rich source of information less influenced by conscious control that provides additional insight into a person’s internal state compared to information conveyed through speech alone (2).

The study of NVB in psychiatric conditions began in the 1960s (3). However, ethological methods applied to the diagnosis and treatment of mental illness have been only used occasionally in the years since, and the potential of NVBs as a source of clinical information remains largely underutilised.

The current approach to diagnosing psychiatric illness and assessing treatment response often relies on unstructured clinical consultations or the use of rating scales and questionnaires. Although some commonly used rating scales can assess observable physical behaviours, there is potential for integrating automated methods or more objective means of analysis to enhance the assessment process. For example, the Bush-Francis Catatonia Rating Scale is a valuable tool for the diagnosis and monitoring of catatonia and recent research highlights the utility of measuring catatonia motor behaviour with objective instrumentation, suggesting a promising avenue for future advancements in the field (4).

Generally, there appears to be a pressing need for objective measures that can accurately determine clinical changes in response to treatment, addressing limitations and enhancing the role of traditional rating scales (5, 6).

The predictive value and importance of closely monitoring treatment responses in psychiatric illnesses, particularly in the early stages, have been highlighted over recent years. A meta-analysis found that early improvement in depression treatment was linked to better outcomes in 15 out of 16 studies (7). In addition, a review of four randomised controlled trials involving 1–627 patients with schizophrenia found that 53% of patients stop treatment early, with poor symptomatic response being the primary reason (8). NVBs are an example of potential behavioural biomarkers (i.e. objective indicators derived from behaviour) that can be quantified to measure treatment response that may be superior to the use of rating scales, providing earlier indications of therapeutic response and leading to improved patient outcomes (9, 10).

The utility of NVBs in psychiatry is derived from the bidirectional relationship between internal emotions and outwards expressions conveyed through facial expressions or movement. Sensory feedback from facial and bodily movements can contribute to internal emotional experiences (2, 11, 12), and the theory of embodied emotion describes how emotional expression, perception and processing are related to internal emotion and arousal (13). A more granular understanding of the emotions and symptoms underlying movement and behaviour could give further clinical relevance to the use of behavioural metrics for measuring treatment response, rather than just quantifying the change in NVB in isolation.

Reduced reactivity and aversive behaviours are consistent with evolutionary theories of depression, such as the social withdrawal hypothesis, and highlight the manifestation of psychomotor retardation (14–16). Thus, a reduction in psychomotor retardation, withdrawal behaviours and a subsequent increase in movement behaviours could be linked to reduced symptom severity and clinical improvement in patients with depression.

The relationship between movement and emotion in schizophrenia is more challenging to discern due to the potential confounding effect of antipsychotic medications on movement. However, research has shown that greater levels of pro-social behaviours are conducive to reduced symptom severity (17).

Additionally, the relationship between NVB and clinical outcomes, particularly in the therapeutic setting, could be mediated through the synergy of NVBs and the therapeutic alliance between patient and clinician. Research has found that patients with schizophrenia who exhibited more social behaviours towards their psychiatrist reported greater satisfaction with communication and a better therapeutic relationship, contributing to more favourable outcomes (17). In patients with depression, greater behavioural synchrony between patients and interviewers has been associated with reduced severity of depression over the course of the illness (18).

NVBs can be quantified using manual or automatic coding methods. Manual methods employ trained raters to watch videotaped subjects frame by frame and record each behaviour that occurs, using frameworks such as the Facial Action Coding System (FACS) developed by Ekman and Friesen (19). FACS objectively codes facial behaviours based on facial anatomy and divides them into Action Units (AUs which represent small visually discriminable changes in facial movements, making it an accurate and powerful measure (20, 21). Automated methods are being increasingly employed to analyse NVBs and increase efficiencies in the labour-intensive coding process. One study comparing manual and automated coding of nonverbal synchrony found that manual coding is comparable to automated methods in terms of accuracy, yet the process is greatly more efficient. The authors illustrated that the manual coding of 110 subjects, including time taken to develop the coding system, train and establish reliability among coders took approximately 12 months, whereas automated coding of the same participants took 2 weeks (22). Moreover, using AI-based and automated methods provide a more objective lens for interpreting findings that are not influenced by the assessors’ emotions or other variable factors. AI models support feature extraction from complex facial data, reducing the need for manual feature extraction and analysis (23). An application of this is the use of convolutional neural networks and machine learning models to study the early detection of depression, for example through comparing data from depressed participants and healthy controls to automatically estimate the likelihood of depression based on differences in facial features (24).

The potential of NVB as an objective biomarker for disorder progression has been highlighted in recent technology-enabled studies which have demonstrated new methods of objectively measuring symptoms. For example, using remote measurements of facial expressions, movement, voice and speech, sleep, and general activity to monitor symptoms, responses to treatment and risk of relapse in psychiatric populations (25, 26).

The field of technology that can analyse behaviour is expanding rapidly, yet before more advanced ways of monitoring responses to treatment can become widely used, it is beneficial to understand the existing knowledge base in relation to the potential use of NVBs as markers of response to treatment in serious mental illness. To date, there has been no systematic evaluation of the literature on how NVBs change in response to treatment and only a few narrative reviews on the wider topic of NVBs are presently available (2, 9, 17, 27, 28). This review aims to systematically evaluate the existing literature, and compare and synthesise findings across studies to identify NVBs that could be used to assess improvement or deterioration in response to treatment. The review will also assess the quality and methodologies of existing studies and make suggestions for future research.

Methods

This review was guided and reported in accordance with PRISMA guidelines (29). The review protocol is registered in PROSPERO and is available online (reg: CRD42022368599).

Eligibility criteria

The inclusion and exclusion criteria are presented in Table 1.

Table 1

Table 1. The eligibility criteria used for paper selection.

Population

The populations of interest were adult inpatients or outpatients with a diagnosis of a serious mental illness namely depression or schizophrenia. Other psychiatric disorders such as anxiety disorders, eating disorders, obsessive compulsive disorder, personality disorders were not in the scope of this review. The literature on NVBs in these conditions is limited, and treatment strategies are more varied and heterogeneous, which would not enable results to be synthesised to illustrate a profile of NVBs in these patients.

Intervention

As the literature in this area is limited, limiting the search to specific interventions could have yielded too few studies for a meaningful synthesis of the evidence. Therefore, only studies where patients did not receive any treatment or intervention were excluded.

Comparator

Included studies compared NVBs at multiple time points (at least twice) throughout the course of treatment to study how NVB changes alongside clinical improvement, stabilisation or deterioration. Only studies assessing NVB during a clinical interview were selected as this is the most standardised means for measuring NVB across studies, as opposed to image viewing tasks or naturalistic observation which display a greater degree of variation across studies. The search aimed to identify cohort studies where the same group of patients were observed over time to study how NVBs change over time rather than identifying how NVBs differ in patients versus healthy controls.

Any study that analysed changes in NVB and clinical change over time was included. The relationship could be assessed using any methodology or statistical test - for example, simple group comparisons to assess treatment-induced changes, or more advanced analyses to explore if NVBs could predict clinical change. Due to the already small evidence base, studies were not excluded based on analysis approach.

Outcome

The outcome measures studied were clinical improvement or deterioration over the duration of the study, measured using any method such as clinical rating scales or clinical interviews (30). Changes in NVBs could also be assessed using method. No exclusions were placed on the approach used to measure outcomes, again to avoid narrowing the scope too extensively. Whilst NVBs can also encompass speaking behaviours such as speech rate, pitch, tone, volume, rhythm, as well as physical movements, these were not in the scope of this particular review.

Information sources

The databases searched were Medline, Embase, PsycINFO and CINAHL. The search was conducted in May 2022 and updated in June 2024 to include new research prior to publication. Reference lists of included studies were screened to identify additional papers.

Search strategy

The final search strategy is presented in the Supplementary Materials. No limits were placed on the date or location of publication. An English language filter was applied to the search. The search strategy was created through an iterative process where HG, EV, SP and AK all inputted.

Selection process

Results from the search were imported into Rayyan for screening (31). HG and EV independently screened the title and abstract of all papers. The interrater reliability at this stage was 0.7 (Cohen’s Kappa). Once all abstracts were screened, HG and EV undertook full-text screening of all remaining papers. HG and EV discussed disagreements in the records that were included or excluded, and in cases where a decision could not be made, SP and AK were consulted.

Data extraction

Key outcomes data collected were author, year of publication, country, study design and setting, assessment tool for NVB and clinical change, statistical tests and method of aggregation, type and length of intervention and outcomes and main results. Data from each study were extracted into an Excel spreadsheet. Three studies were subject to an initial data extraction by HG and EV and compared to confirm that the result was satisfactorily similar. The interrater agreement for the data extraction process was 83%. HG and EV discussed any discrepancies before proceeding with the data extraction for the remaining studies independently. The full data extraction form is presented in the Supplementary Materials.

Risk of bias assessment

The CASP cohort study checklist was used to assess the risk of bias in each included paper (32). Three studies were subject to an initial quality assessment pilot by HG and EV and were compared to confirm that the result was similar. HG and EV extracted data from the remaining papers independently. Disagreements were to be resolved by separate discussions with SP and AK.

Data synthesis

Due to the heterogeneity of methods used to measure and record NVB, formal synthesis and meta-analysis of results were not possible. Results were subject to narrative synthesis by summarising the main outcomes of each paper and determining which changes in NVB were significantly associated with clinical improvement or deterioration. We compared the behavioural profiles for depression and schizophrenia, and whether these differed across intervention types. We also explored the overall quality and strengths and weaknesses of the evidence and highlighted key knowledge gaps and avenues for future research.

Results

Study properties

We identified 9–743 publications from the searching stage and screened 7–493 titles and abstracts after removal of duplicates. Following abstract screening, 75 studies underwent full text screening, and 20 studies met the final eligibility criteria for inclusion (Figure 1). Table 2 summarises the properties of included studies. 15 studies observed patients with depression, two studies observed patients with schizophrenia and three studies observed separate groups of patients with depression and patients with schizophrenia within the same study.

Figure 1

Figure 1. PRISMA flow diagram (33).

Table 2

Table 2. Properties of all included studies.

Study sample sizes ranged from 12–164 participants, with the average being 43 participants. Interventions included pharmacological intervention, psychological intervention, neuromodulatory intervention, and a variety of non-standardised interventions including hospitalisation, pharmacological and psychological interventions. Interventions ranged in length from 2–32 weeks, with the average length being 12 weeks. Most studies (14/20) assessed patient outcomes using The Hamilton Rating Scale for Depression (HRSD).

Regarding the analysis of NVBs, 12/20 studies used manual analysis, 6/20 used automated analysis, and 2 studies compared both automated and manual methods. NVB was assessed most frequently during a videotaped clinical interview based on the administration of the HRSD. Other assessment settings included videotaped psychotherapy sessions or less structured mental state examinations.

Most studies (15/20) used group comparison tests or correlations to compare NVBs before and after treatment, showing treatment induced changes in NVBs. Five studies used multilevel modelling or regression approaches, and two studies extended the analysis to make predictive inferences about how NVBs may predict clinical change. Fiquer et al. (2013) (38) found that adaptive gestures (self-touching and object touching) at baseline predicted higher negative affect scores after treatment (p=0.04). Fisher et al. (41) found that an increase in emotional valence during the therapy session predicted later reductions in depression symptom scores (p=0.018).

Heterogeneity across studies made synthesis challenging, mostly due to the wide variety of behavioural categorisation and measurement approaches used. For example, some studies focused on measuring specific facial AUs and discrete movements, whereas others grouped behaviours into broader categories and ascribed emotional states (such as happiness, anger and restlessness) to patterns of NVBs behaviours. This led us to focus our synthesis on broader categories of behaviour. A wide variety of methods to measure NVBs and lack of standardised procedure also impaired comparisons. Some studies measured NVB continuously throughout the clinical interview, others for just a short portion of the interview, and the features of NVB that were recorded varied across studies (e.g. duration, frequency and intensity). It was also challenging to determine if automated methods led to more consistent results due to the still broad range of approaches used to measure and categorise behaviour among this subset of studies. Among the studies employing automated coding methods, NVBs measured included nonverbal synchrony between the patient and interviewer (34, 49), emotional valence (41), changes in AUs (16, 43, 50), changes in behaviours associated with specific emotions (44), behavioural depression degree (45). This emphasises the need for more standardised measurement approaches to strengthen conclusions.

Overview of behavioural changes

Definitions of key NVBs discussed in this section are presented in Table 3.

Table 3

Table 3. Behaviour definitions.

Facial behaviours

The synthesis of results firstly focused on the analysis of behaviours relating to smaller, discrete parts of the face (e.g., eye and eyebrow movements, mouth movements) and the relationship between discrete behaviours and responses to treatment. Results were highly variable, and the only consistently occurring facial behaviour associated with symptomatic improvement in patients with depression was increased mouth movements/smiling, observed in 4/18 studies (16, 30, 43, 47). Only two studies assessed facial behaviours in patients with schizophrenia, therefore a robust conclusion cannot be drawn, although both studies both observed a significant reduction in facial activity and expressiveness over time (42, 50).

General NVBs

Due to the lack of consistency across specific facial behaviours, we focused our analysis on more general behaviours such as body movements, gesturing, emotional expressivity, and emotional valence. Studies seemed to shift from more specific methods of analysing NVB in older studies (e.g., using FACS or ethograms comprised of specific muscle movements) to a more global and generalised approach of assessing the nature of NVB changes in studies published from 2021 onwards. A more generalised approach may enable inter-individual differences in specific facial movements and differences in methods of measuring NVB to be accounted for. 16/20 studies conducted a more global assessment of behaviours (as opposed to assessing facial AUs only or blinking/eye movements), 14 studies included patients with depression in the sample, and 2 included both patients with depression and patients with schizophrenia in the sample.

Depression

Generally, as clinical improvement occurred over time, patients with depression showed increasing levels of social and affiliative behaviours, including head movements (5/14 studies) (16, 35–37, 47), gesturing (2/14 studies) (37, 47), body contact/touching (35–37) (3/14 studies), facial expressivity and positive emotional valence (6/14 studies) (38, 41, 44, 45, 47, 51), movement (including restlessness and fidgeting) (3/14 studies) (35, 40, 47). Such behaviours portray an image of the patient transitioning from behaviours associated with social withdrawal and diminished emotions to more active and engaged as treatment progressed. Although increased levels of restlessness and fidgeting could also be interpreted as occurring due to discomfort. Hence, despite the differing definitions of body movement and restlessness, many studies did not differentiate between the two behaviours nor acknowledge that restlessness and fidgeting could be associated with negative valence.

Conflicting results were reported by Bouhuys and Albersnagel (1992) (35) and Fiquer et al. (2017) (39) who found that patients who did not improve showed more speaking-effort behaviours which contrasts with the theory of pro-social behaviours being associated with improvement. In addition, Gaebel and Wölwer (42) (2004) reported that as symptoms improved, facial expressivity and emotional repertoire reduced.

Schizophrenia

No consistently occurring behaviours were observed across the two studies which assessed more general behaviours in patients with schizophrenia (30, 42).

Effect of intervention type on general NVBs

7 of the 16 studies that measured generalised behaviours assessed the association between NVB and pharmacological intervention (16, 35–37, 39, 42, 51). This subset of studies corroborated with some of the overall findings which showed an increase in body touching (35), gesturing (51), nodding and head movements (16, 35–37), and a general increase in expressivity and AUs (16, 51) as patients received pharmacological intervention. One conflicting result in this subset of studies was that of facial expressivity, Ulrich and Harms (1985) found increased facial expressivity as symptoms improved whereas Gaebel and Wölwer (2004) showed reduced facial expressivity after 3 and 4 weeks of antidepressant treatment respectively.

3 studies measured the effect of neuromodulatory treatments (2 deep brain stimulation (44, 47), 1 transcranial direct current stimulation (38)) and found that patients showed increased nodding and eye contact after 2 weeks of transcranial direct current stimulation and increased facial expressivity (38), more reactive, engaging and fidgeting behaviours (including head movements, laughing, smiling, general mouth movements, raised eyebrows, head touching and illustrative gestures), and fewer behaviours associated with disengagement after 6–8 months of deep brain stimulation (44, 47).

3 studies observed NVB changes whilst patients underwent courses of psychotherapy, and found increased positive emotional valence (41), increased expression entropy (45), and increased nonverbal synchrony (49) as treatment progressed.

Effects of medication on NVB

In 12/20 studies, patients commenced medication or treatment at the same time of the baseline interview (30, 35–39, 41, 44–48). 2/20 studies included patients who were treatment naïve or medicated at the baseline interview (42, 50), and the remaining 6/20 studies did not specify if patients were already being treated prior to or commencing treatment at the time of the baseline interview (16, 34, 40, 43, 49, 51).

5 studies acknowledged and discussed the impact of medication effects on the study results. 1/5 studied the effect of medication on NVB in patients with depression, 1 included a sample of both patients with depression and patients with schizophrenia, and 3 explored the effect of medication on NVB in patients with schizophrenia (35, 42, 46, 48, 50).

In studies observing NVBs in patients with depression, Bouhuys et al. (1992) (35) mentioned that the pattern of behaviours at baseline did not significantly differ between medicated and unmedicated groups, but acknowledged that treatment conditions in the study were not controlled for which may have impacted findings.

Mackintosh et al. (1983) (46) studied a combined sample of patients with schizophrenia and patients with depression and found that there were no significant differences in the blink rate when patients were compared based on medication (patients were receiving one of three types of treatment: minor tranquilisers, tricyclic antidepressants, or major tranquilisers combined with antidepressants).

The studies exploring medication effects in schizophrenia described the impact on results in more depth. Meya and Renfordt (1986) (48) compared the effect of haloperidol and perazine in previously unmedicated patients with schizophrenia, and found that patients treated with haloperidol showed a tendency towards increased eye contact with the interviewer during the first 14 days of treatment compared to patients treated with perazine. The authors stated that more research is required to validate this finding and explore why this may have occurred.

Gaebel and Wölwer (2004) (42) conducted exploratory analysis of differential treatment effects of haloperidol and perazine in patients with schizophrenia, and found a significant medication effect regarding the facial repertoire and facial activity in the acute phase of schizophrenia, stating that an attenuation of facial expressivity between baseline and T1 was more pronounced in patients treated with haloperidol than perazine, and more pronounced in patients treated with perazine versus unmedicated control participants. When comparing unmedicated versus medicated patients at baseline, there was no significant difference in facial expressivity. The authors discussed that attenuated facial activity seen in the study is unlikely to be due to drug treatment and more likely to be explained by habituation to the interview, stating that the effect size was too small to be of clinical significance. Gaebel and Wölwer’s (2004) study did not align with the results observed by Meya and Renfordt (1986), signifying that the question of the influence of haloperidol and perazaine on alterations in NVB remains unanswered.

Further, Schneider et al. (1992) (50), included one group of patients (group 1) with schizophrenia who were unmedicated at the start of the study, and group 2 who commenced the study whilst continuing to take neuroleptic medication. The study found a significant reduction of facial action over time in group 1, and no significant change in group 2, despite there being no significant difference at baseline between groups in psychopathology scores. The authors also stated that more research is required to determine if reduced activity induced by neuroleptics is due to both motor and emotional sedation. The lack of consistent conclusions across all three studies represents a significant gap in the literature on the effect of medication on NVB in schizophrenia and the subsequent relationship to symptom scores and patient outcomes.

Risk of bias assessment

Results from the risk of bias assessment using the CASP Cohort Study checklist (32) are presented in Table 4. Regarding areas of concern, just 9/20 studies clearly identified all confounding factors, 8/20 studies accurately measured the exposure (e.g. interview condition) and outcomes (clinical outcome and NVBs) to minimise bias, and only 5/20 studies had results that could be applied to the local population due to small sample sizes.

Table 4

Table 4. Results from risk of bias assessment conducted using the CASP cohort study checklist.

Strengths of the evidence base included that all studies addressed a clearly focused issue, 15/20 studies had a follow up of subjects that was sufficiently complete, and 13/20 studies followed up subjects for long enough.

Discussion

To our knowledge, this is the first systematic review on how NVBs change over time as a response to treatment in patients with depression and schizophrenia. Methodological differences and a lack of studies including patients with schizophrenia made comparisons across studies and drawing conclusions challenging. Specific behavioural differences associated with improvement in depression symptoms that were found across multiple studies was increased mouth movements/smiling. No commonalities in NVB changes over time were observed for patients with schizophrenia. The degree of consistency across the literature was larger when our synthesis looked at broader behaviours associated with movement and expressivity and saw that such behaviours increased as clinical improvement occurred in patients with depression. In the majority of studies, patients commenced medication or treatment at the same time as they commenced involvement in the study, so these findings are perhaps most applicable to the earlier stages of treatment initiation following hospital admission or presentation to services, rather than monitoring long-term treatment responses in clinically stable patients.

Consistencies across intervention types suggests that broad categories of behaviour or movement and expressivity may also be less influenced by the means of analysis and by intervention – which better reflects the reality of how future remote-monitoring technologies could be used in clinical practice due to patient variation. In addition, the benefits of this approach are that it may be easier to validate such findings in a larger scale study using generalised automated analysis of behaviour rather than manual coding of specific AUs. A recent review of automated depression recognition highlighted that analysis of body expressions for the automated identification of depression is an important research avenue that should be explored in addition to research on facial expression analysis to identify depression (23).

Evidence limitations

Sources of bias were largely related to the study design. Many studies did not consider confounding factors – e.g., if raters were blinded, if the same interviewer was used throughout, if the interviewer was familiar to the patient, which medications and dose were prescribed, and other patient demographics or comorbidities which may influence behaviours. The impact of cultural differences could not be extensively discussed due to limited reporting of patient demographics and limited acknowledgement of the impact of cultural factors within the included studies.

We also noted how the use of a variety of outcome measures when evaluating clinical improvement may account for the differences between studies. For example, across studies that used the HRSD to measure clinical outcomes we observed discrepancies regarding thresholds and questionnaire versions.

More is also to be understood about how NVBs change throughout the interview process. For example, reductions in NVB may occur due to habituation to the interview process (42). Several studies mitigated this by only recording NVBs during the first part of the interview (36, 38, 47). Still, studies that used this method differed in the length of the initial segment of the interview that was coded, ranging from 5–20 minutes.

Participants will also have increased familiarity with the interviewer throughout subsequent sessions which may impact NVB and introduces variation and uncertainty whereby patients within the same study with differing levels of familiarity with the interviewer are compared. Bhatia et al. (2019) acknowledged that one potential reason for not observing increased head movement synchrony between the patient and interviewer as clinical improvement occurred could be due to a lack of consistency in whether or not patients saw the same therapist during the 21 week trial – some patients had the same therapist, others may have changed therapist up to 4 times. This phenomenon has been recorded in the wider literature whereby the interviewer’s NVBs and nonverbal synchrony may influence patient improvement (18, 53).

Such sources of bias are implicit to using a clinical interview as the environment in which NVBs are measured. The benefits of assessing behaviours during a clinical interview are that it enables the observation of a larger range of behaviours that would occur during a social interaction which may not occur during an image viewing task or another means of assessing behaviour, however it is important to note that results from such studies may be influenced by the interpersonal relationship between the patient and the interviewer.

Areas of potential bias, such as expectation bias from the NVB raters if they are aware of the patient’s clinical status, can be overcome with the increasing use of automated means of NVB analysis. The use of automated analysis in future studies may also help to study the potential habituation to the interview by coding behaviours occurring across the entire interview, as it can be inferred that a reason for only analysing segments of interviews is due to the time burden associated with manual coding.

Depression

General increases in movement and expressivity seen over time in patients with depression is in line with the Social Withdrawal hypothesis of depression, which suggests that patients with high levels of depression show less social or affiliative behaviours to maintain or increase interpersonal distance (16); a response that may have biological and evolutionary roots to protect the self from failure and/or harm (54–56). These observations could translate into a biomarker for assessing treatment response in depression.

There were studies outside the scope of this review which supported these findings. Two studies of NVBs in remitted patients with depression revealed that patients who went on to relapse displayed less active listening behaviours during the discharge interview compared to patients who remitted, and patients who showed less involvement behaviours at baseline had less favourable outcomes (57, 58).

Results which conflicted with this hypothesis were reported by Bouhuys and Albersnagel (1992) (35) and Fiquer et al. (2017) (39) who saw more speaking effort and restlessness in patients who did not improve. A potential explanation was that when patients with high speaking effort evoke excessive social support, they may be more prone to interpersonal distress and depression maintenance and persistence (39). Gaebel and Wölwer (42) (2004) found that patients had reduced expressivity as symptoms improved and that facial behaviours stabilised over a 3 month period. The authors stated that the reason for NVBs not changing as clinical improvement occurs could be due to habituation to the interview setting or a mismatch between the experience of emotion and displaying the emotion via facial expressions. Another explanation is offered by Fisher et al. (41) (2023) who found that emotional expressions follow a U shaped curve whereby positive emotions reduce as patients face challenging emotions in the middle of a course of therapy, but that the level of positive emotions at the end of treatment is higher than at the beginning.

One study outside of the scope of this review compared NVBs between patients with depression and a healthy control group. At baseline, the patient group showed higher levels of negative NVB (frowning, crying, head and lips down), and lower levels of positive NVB (eye contact and smiling). However, after eight weeks of treatment the patients with depression had no significant changes in their NVBs regardless of clinical improvement. The authors suggested that NVB may be independent of depression severity and may rather be a sign of depression (59).

We identified several noteworthy papers that investigated how NVB can be used to predict clinical change prospectively or retrospectively but could not be included in our review as NVB was only measured at one time point (57, 58, 60–62). Two papers studied behavioural predictors of amitriptyline response in depressed inpatients (60, 61). One showed that patients who did not respond to medication showed more body-focused movements, postural changes and reduced smiling at the baseline interview (61). Another found that non-responders demonstrated more assertive and affiliative behaviours (defined as a behaviour with an attractive function to draw the attention of the interviewer to the individual, and that express communicative efforts for social interaction) and had more engagement with the interviewer, compared to no significant behaviours associated with treatment responders (60). These findings from the wider literature contrast with results from this review in which clinical improvement in depression is associated with affiliative and interactive NVBs. This further exemplifies the need for larger scale studies and longer term follow ups that span the entire course of treatment.

Schizophrenia

There were no consistently occurring behaviours identified across studies in patients with schizophrenia. Results were inconclusive from so few studies thus it cannot be determined if NVBs are a useful metric to measure treatment response in schizophrenia. Further research is needed to discern the impact of medication on NVBs and whether behaviour changes are a result of medication side effects or symptom improvement. There is a need to identify patients at risk of deterioration, and conversely, early identification of patients who may be improving, to guide treatment strategy. Treatment resistance is linked with a higher risk of clinical deterioration, hospitalisation, poor quality of life and impaired real-world functioning, therefore the early identification of patients who fail to respond to initial interventions can improve the treatment approach at an earlier phase and contribute to improving outcomes.

The paucity of research in this area is exemplified by the fact that no studies since 2004 have been conducted to further explore the role of NVBs in monitoring treatment response over time in schizophrenia. Specifically, the lack of studies on clinical deterioration could allude to potential reporting or publication bias, in that many of the NVB studies assessed in this review were conducted in parallel with an interventional clinical trial, and only studies which saw clinical improvement in patients (rather than deterioration) may have ultimately sought publication. In addition, it may be more challenging to recruit patients into long-term studies who could be more acutely unwell or at risk of deterioration.

NVBs could be a potential avenue on the search for reliable signals that allude to early treatment response, however the literature, in its current format, does not support this use.

There are two important factors which may have a strong influence over NVBs in patients with schizophrenia: the impact of negative symptoms and the effects of medication. Patients with higher levels of negative symptoms typically show reduced levels of NVB expression and a greater avoidance of social contact (63, 64). Avolition and blunted affect can impede social perception in patients with schizophrenia and ultimately reduce NVB expression (65). Furthermore, the presence of negative symptoms may be associated with a higher rate of spontaneous movement disorders in patients prior to any exposure to antipsychotic medication which may further affect NVB production (66). Movement disorders such as spontaneous dyskinesia or parkinsonism may therefore be related to the pathophysiology of schizophrenia in a proportion of patients rather than a side effect of antipsychotics, though medication may also alter the expression of NVBs (67, 68).

Known side effects of antipsychotic medication include reduced ability to move the eyes, facial activity, blink rate, and emotional bluntness (46, 50, 69, 70). Two studies included in the review discussed the potential impact of antipsychotic medication on facial expressions and reported conflicting results. One concluded that attenuated facial expressivity in patients with schizophrenia over time is unlikely to be due to medication effects, is unlikely to be clinically significant and is more likely to be attributable to habituation to the interview process (42). Conversely, Schneider et al. concluded that the reduction in facial expression is likely to be due to medication effects. Patients who were already medicated at baseline showed no change in their facial expression over time. Patients who were unmedicated at baseline but commenced medication during the study showed reduced facial expression over time, despite no significant difference in the psychopathology scores between groups. This finding raises questions about whether the observed reduction in movement is a result of high levels of agitation at baseline or due to side effects of antipsychotic medication and EPSEs (50). It is important to note the small sample sizes in either study (n=16 and n=34), exemplifying the importance of further research. A potential use of the further study of NVB in schizophrenia could be in trials comparing different antipsychotic regimes to determine the level of impact of the medication on NVBs and emotional blunting, and how alleviating this could support the expression of prosocial behaviours and social functioning, and facilitate recovery (17).

The study design may also influence the lack of consistent findings across patients with schizophrenia. Patients with different subtypes were included together in the analysis but considering different manifestations of schizophrenia and experiences of positive and negative symptoms, this may have not been appropriate. Compared to depression, the field of schizophrenia is far less researched and despite the implicit challenges outlined above, there is considerable scope in the evaluation of NVBs as predictors of clinical outcomes or change. This could be demonstrated in a large-scale, long-term study considering confounding factors (medication type and dosage, medical history, positive and negative symptoms) that have not been sufficiently accounted for in the existing literature to confirm or dispute historic findings. This is especially important considering that the last papers exploring NVBs in patients with schizophrenia were published 20 years ago so do not reflect recent treatment advancements.

Strengths and limitations

To our knowledge, this is the first systematic review to examine NVB indicators of treatment response in adults with SMI. We performed a robust systematic search in line with PRISMA guidelines. Limitations of the review relate to the limited evidence and lack of large-scale studies, meaning the results must be interpreted with caution when generalising to the wider patient population. Additionally, requiring studies to record NVB at multiple time points or only during a clinical interview may have excluded other relevant papers that provide predictive insight into how NVBs relate to clinical improvement or deterioration.

Due to the heterogeneity of methods used to measure NVB’s, specific definitions used to define behaviours or groups of behaviours may vary across studies whereby the same behaviour is measured differently, which may be a limitation in our interpretation and analysis of behavioural results.

Clinical practice considerations

Technologies that can automatically analyse facial expressions and emotions from video provide opportunities to incorporate real-time NVB analysis into routine video consultations or smartphone-based monitoring software (71), particularly given the increase in remotely delivered mental health care and the growing market for mental health apps (72). This integration of NVB analysis into routine mental health assessment and treatment processes could be beneficial to its broader adoption and consistent use, given that engagement with mental health apps is typically low and attrition is high, particularly if such technologies do not integrate seamlessly into mental health services (73). Having a behavioural biomarker that can be assessed in real time (such as NVBs being monitored during a routine consultation) may be preferable to the need for additional apps and technologies that collect data outside of clinical consultations (72). This may also reduce the burden of data collection and monitoring for both patients and healthcare professionals. This review concluded that there are several behavioural changes that could indicate clinical improvement in depression (head movements, quantity of movements, smiling). These are fairly simple, objective and easy-to-recognise behavioural markers that if validated further, could reasonably be tracked and monitored in real time during appointments.

Telemedicine and remote patient monitoring have led to statistically significant improvements in health outcomes, healthcare costs, patient satisfaction and accessibility across many disease areas (74). There is not yet a large evidence base on outcomes from the use of novel mental health technologies (75), and much more research is needed to address unknowns regarding the longer-term consequences and potential unintended consequences of digital monitoring interventions (72). Yet the fact that innovative monitoring technologies have been successfully integrated into care pathways in other chronic disease areas shows that enhancing mental health services with advanced digital solutions and novel behavioural biomarkers is a realistic possibility (73).

Future recommendations

It is important to emphasise the potential challenges of assessing NVBs in psychiatric disorders. Firstly, the NVBs associated with a particular mental illness may vary across individuals within the same group, making it difficult to develop standardised assessments¹⁸. Additionally, the presence of mental, physical, and/or substance use comorbidities can further complicate the assessment of NVB (76–79). Similarly, treatment can affect the expression of NVBs, often not allowing to separate the effects of the illness from medication side effects. Therefore, future research should consider controlling for comorbidities, effects of medication and individual differences.

To improve the accuracy and reliability of NVBs as predictors for treatment response, future studies may benefit from standardising and controlling confounding variables and clinician and rater’s blindness to patient’s clinical or treatment status. Due to the sample sizes being small for most of the studies, future research would benefit from observing changes in NVBs on a larger scale of patients with a range of symptom severities and longer follow up periods, with strictly controlled treatment regimens and a control group where possible. We also noted a lack of studies on clinical deterioration. Designing future studies to focus on potential markers of early deterioration and relapse, rather than just clinical improvement, is an important area of further research and will further enable us to understand the utility of NVB biomarkers in either case.

Furthermore, if manual coding methods are used in future studies, having multiple raters, adequate training to improve inter-rater reliability and ensuring that interviews are consistent in timings and context may improve the reliability and validity of results. However, consideration should be given to the use of AI-based technology and automated methods to detect and analyse NVBs, reducing the potential of human rater bias (80) and enhancing the speed and efficiency of analysis. Incorporating AI-based technology could also help to enable remote monitoring and larger scale, cross-cultural studies taking place (81). At the same time, it is critically important to establish the reliability, validity and ethical considerations of AI-based technology and the collection of sensitive behavioural data (82, 83). Given the novelty of the field, data protection and ethical frameworks for the collection of NVB data and development of AI-based tools are currently lacking but essential for the sustainable widespread use of such technologies that preserve patient autonomy (72, 75).

Conclusion

The current evidence for the utility of NVBs in monitoring responses to treatment in serious mental illness is limited, variable and mostly inconclusive. Indeed, the literature in its current format is insufficient to describe an observable behavioural profile of clinical improvement in response to treatment in SMI patients. Few behaviours may be associated with clinical improvement in depressed patients such as head movements, quantity of movements and smiling which could be further explored. There is a paucity of research in other populations such as in patients with schizophrenia or bipolar disorder. Future research would significantly benefit from large-scale, high-quality studies, supported by automated analysis to enable the potential of NVBs to act as reliable markers (and if possible as early indicators), for treatment response to be robustly assessed.

Data availability statement

The original contributions presented in the study are included in the article/Supplementary Material. Further inquiries can be directed to the corresponding author.

Author contributions

HG: Writing – review & editing, Investigation, Writing – original draft, Conceptualization, Data curation, Methodology, Formal Analysis. EV: Investigation, Writing – original draft, Conceptualization, Validation, Formal Analysis, Writing – review & editing, Methodology. AK: Validation, Conceptualization, Supervision, Writing – review & editing, Writing – original draft, Methodology. SP: Writing – review & editing, Conceptualization, Supervision, Writing – original draft, Validation.

Funding

The author(s) declared that financial support was not received for this work and/or its publication.

Acknowledgments

The open access fee was paid from the Imperial College London Open Access Fund.

Conflict of interest

SP reports grants and honoraria outside the submitted work.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Generative AI statement

The author(s) declare that no Generative AI was used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2025.1643042/full#supplementary-material

References

1. Ekman P and Friesen WV. The Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and Coding. In: The Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and Coding. Berlin, Boston: De Gruyter Mouton (1981). p. 57–106. doi: 10.1515/9783110880021.57

Crossref Full Text | Google Scholar

2. Geerts E and Brüne M. Ethological approaches to psychiatric disorders: focus on depression and schizophrenia. Aust N Z J Psychiatry. (2009) 43:1007–15. doi: 10.1080/00048670903270498

PubMed Abstract | Crossref Full Text | Google Scholar

3. Grant EC. An ethological description of non-verbal behaviour during interviews. Br J Med Psychol. (1968) 41:177–84. doi: 10.1111/j.2044-8341.1968.tb02022.x

PubMed Abstract | Crossref Full Text | Google Scholar

4. von Känel S, Nadesalingam N, Alexaki D, Baumann Gama D, Kyrou A, Lefebvre S, et al. Measuring catatonia motor behavior with objective instrumentation. Front Psychiatry. (2022) 13. doi: 10.3389/fpsyt.2022.880747

PubMed Abstract | Crossref Full Text | Google Scholar

5. Kishimoto T, Takamiya A, Liang K, Funaki K, Fujita T, Kitazawa M, et al. The project for objective measures using computational psychiatry technology (PROMPT): Rationale, design, and methodology. Contemp Clin Trials Commun. (2020) 19:100649. doi: 10.1016/j.conctc.2020.100649

PubMed Abstract | Crossref Full Text | Google Scholar

6. Moon E, Yang M, Seon Q, and Linnaranta O. Relevance of objective measures in psychiatric disorders—Rest-activity rhythm and psychophysiological measures. Curr Psychiatry Rep. (2021) 23:85. doi: 10.1007/s11920-021-01291-3

PubMed Abstract | Crossref Full Text | Google Scholar

7. Habert J, Katzman MA, Oluboka OJ, McIntyre RS, McIntosh D, MacQueen GM, et al. Functional recovery in major depressive disorder: focus on early optimized treatment. Prim Care Companion CNS Disord. (2016) 18. doi: 10.4088/PCC.15r01926

PubMed Abstract | Crossref Full Text | Google Scholar

8. Liu-Seifert H, Adams DH, and Kinon BJ. Discontinuation of treatment of schizophrenic patients is driven by poor symptom response: a pooled post-hoc analysis of four atypical antipsychotic drugs. BMC Med. (2005) 3:21. doi: 10.1186/1741-7015-3-21

PubMed Abstract | Crossref Full Text | Google Scholar

9. Troisi A. Ethological research in clinical psychiatry: the study of nonverbal behavior during interviews. Neurosci Biobehav Rev. (1999) 23:905–13. doi: 10.1016/S0149-7634(99)00024-X

PubMed Abstract | Crossref Full Text | Google Scholar

10. Brüne M. On aims and methods of psychiatry – a reminiscence of 50 years of Tinbergen’s famous questions about the biology of behavior. BMC Psychiatry. (2014) 14:1695. doi: 10.1186/s12888-014-0364-y

PubMed Abstract | Crossref Full Text | Google Scholar

11. Shafir T, Tsachor RP, and Welch KB. Emotion regulation through movement: unique sets of movement characteristics are associated with and enhance basic emotions. Front Psychol. (2016) 6. doi: 10.3389/fpsyg.2015.02030

PubMed Abstract | Crossref Full Text | Google Scholar

12. García-Díaz S. Movement and emotions. Res Investig Sports Med. (2018) 2:1–3. doi: 10.31031/RISM.2018.02.000550

Crossref Full Text | Google Scholar

13. Wu L, Huang R, Wang Z, Selvaraj JN, Wei L, Yang W, et al. Embodied emotion regulation: the influence of implicit emotional compatibility on creative thinking. Front Psychol. (2020) 11:1822. doi: 10.3389/fpsyg.2020.01822

PubMed Abstract | Crossref Full Text | Google Scholar

14. Dibeklioğlu H, Hammal Z, and Cohn JF. Dynamic multimodal measurement of depression severity using deep autoencoding. IEEE J Biomed Health Inform. (2018) 22:525–36. doi: 10.1109/JBHI.2017.2676878

PubMed Abstract | Crossref Full Text | Google Scholar

15. Ike KGO, de Boer SF, Buwalda B, and Kas MJH. Social withdrawal: An initially adaptive behavior that becomes maladaptive when expressed excessively. Neurosci Biobehav Rev. (2020) 116:251–67. doi: 10.1016/j.neubiorev.2020.06.030

PubMed Abstract | Crossref Full Text | Google Scholar

16. Girard JM, Cohn JF, Mahoor MH, Mavadati SM, Hammal Z, and Rosenwald DP. Nonverbal Social Withdrawal in Depression: Evidence from manual and automatic analysis. Image Vis Comput. (2014) 32:641–7. doi: 10.1016/j.imavis.2013.12.007

PubMed Abstract | Crossref Full Text | Google Scholar

17. Lavelle M, Healey PGT, and McCabe R. Nonverbal behavior during face-to-face social interaction in schizophrenia: A review. J Nerv Ment Dis. (2014) 202:47. doi: 10.1097/NMD.0000000000000031

PubMed Abstract | Crossref Full Text | Google Scholar

18. Geerts E, Bouhuys N, and Van den Hoofdakker RH. Nonverbal attunement between depressed patients and an interviewer predicts subsequent improvement. J Affect Disord. (1996) 40:15–21. doi: 10.1016/0165-0327(96)00037-7

PubMed Abstract | Crossref Full Text | Google Scholar

19. Ekman P and Friesen W. Facial action coding system: a technique for the measurement of facial movement. Palo Alto, CA: Consulting Psychologists Press (1978). doi: 10.1037/t27734-000

Crossref Full Text | Google Scholar

20. van Kuilenburg H, Wiering M, and den Uyl M. A model based method for automatic facial expression recognition. In: Proceedings of the 16th European conference on Machine Learning. Springer-Verlag, Berlin, Heidelberg (2005). p. 194–205. doi: 10.1007/11564096_22

Crossref Full Text | Google Scholar

21. Cohn JF, Ambadar Z, and Ekman P. Observer-based measurement of facial expression with the Facial Action Coding System. In: Handbook of emotion elicitation and assessment. Oxford University Press, New York, NY, US (2007). p. 203–21.

Google Scholar

22. Fujiwara K, Bernhold QS, Dunbar NE, Otmar CD, and Hansia M. Comparing manual and automated coding methods of nonverbal synchrony. Commun Methods Meas. (2021) 15:103–20. doi: 10.1080/19312458.2020.1846695

Crossref Full Text | Google Scholar

23. Han M-M, Li X-Y, Yi X-Y, Zheng Y-S, Xia W-L, Liu Y-F, et al. Automatic recognition of depression based on audio and video: A review. World J Psychiatry. (2024) 14:225–33. doi: 10.5498/wjp.v14.i2.225

PubMed Abstract | Crossref Full Text | Google Scholar

24. Astafeva D, Gayduk A, Tavormina G, Syunyakov T, Chigareva O, Markina E, et al. Neuronetwork approach in the early diagnosis of depression. Psychiatr Danub. (2023) 35:77–85.

PubMed Abstract | Google Scholar

25. Barnett I, Torous J, Staples P, Sandoval L, Keshavan M, and Onnela J-P. Relapse prediction in schizophrenia through digital phenotyping: a pilot study. Neuropsychopharmacology. (2018) 43:1660–6. doi: 10.1038/s41386-018-0030-z

PubMed Abstract | Crossref Full Text | Google Scholar

26. De Angel V, Lewis S, White K, Oetzmann C, Leightley D, Oprea E, et al. Digital health tools for the passive monitoring of depression: a systematic review of methods. NPJ Digit Med. (2022) 5:1–14. doi: 10.1038/s41746-021-00548-8

PubMed Abstract | Crossref Full Text | Google Scholar

27. Annen S, Roser P, and Brüne M. Nonverbal behavior during clinical interviews: similarities and dissimilarities among schizophrenia, mania, and depression. J Nerv Ment Dis. (2012) 200:26. doi: 10.1097/NMD.0b013e31823e653b

PubMed Abstract | Crossref Full Text | Google Scholar

28. Balsters MJH, Krahmer EJ, Swerts MGJ, and Vingerhoets AJJM. Verbal and nonverbal correlates for depression: A review. Curr Psychiatry Rev. (2012) 8:227–34. doi: 10.2174/157340012800792966

Crossref Full Text | Google Scholar

29. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. (2021) 372:n71. doi: 10.1136/bmj.n71

PubMed Abstract | Crossref Full Text | Google Scholar

30. Jones IH and Pansa M. Some nonverbal aspects of depression and schizophrenia occurring during the interview. J Nerv Ment Dis. (1979) 167:402–9. doi: 10.1097/00005053-197907000-00002

PubMed Abstract | Crossref Full Text | Google Scholar

31. Ouzanni M, Hammady H, Fedorowicz Z, and Elmagarmid A. Rayyan — a web and mobile app for systematic reviews. Syst Rev. (2016) 5. doi: 10.1186/s13643-016-0384-4

PubMed Abstract | Crossref Full Text | Google Scholar

32. Critical Appraisal Skills Programme. (CASP) Cohort Study Checklist. (2018). Available online at: https://casp-uk.net/casp-tools-checklists/ (Accessed June 8, 2023).

Google Scholar

33. Haddaway NR, Page MJ, Pritchard CC, and McGuinness LA. PRISMA2020: An R package and Shiny app for producing PRISMA 2020-compliant flow diagrams, with interactivity for optimised digital transparency and Open Synthesis. Campbell Syst Rev. (2022) 18:e1230. doi: 10.1002/cl2.1230

PubMed Abstract | Crossref Full Text | Google Scholar

34. Bhatia S, Goecke R, Hammal Z, and Cohn JF. (2019). Automated measurement of head movement synchrony during dyadic depression severity interviews, in: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), . pp. 1–8. doi: 10.1109/FG.2019.8756509

PubMed Abstract | Crossref Full Text | Google Scholar

35. Bouhuys AL and Albersnagel FA. Do interactional capacities based on observed behaviour interfere with improvement in severely depressed patients? A longitudinal study. J Affect Disord. (1992) 25:107–16. doi: 10.1016/0165-0327(92)90073-F

PubMed Abstract | Crossref Full Text | Google Scholar

36. Bouhuys AL, Beersma DG, and van den Hoofdakker RH. Observed behaviors during clinical interviews predict improvement in depression. Acta Psychiatr Belg. (1986) 86:699–720.

PubMed Abstract | Google Scholar

37. Bouhuys AL, Beersma DGM, van den Hoofdakker RH, and Roossien A. The prediction of short- and long-term improvement in depressive patients: Ethological methods of observing behavior versus clinical ratings. Ethol Sociobiol. (1987) 8:117–30. doi: 10.1016/0162-3095(87)90024-0

Crossref Full Text | Google Scholar

38. Fiquer JT, Boggio PS, and Gorenstein C. Talking bodies: Nonverbal behavior in the assessment of depression severity. J Affect Disord. (2013) 150:1114–9. doi: 10.1016/j.jad.2013.05.002

PubMed Abstract | Crossref Full Text | Google Scholar

39. Fiquer JT, Moreno RA, Canales JZ, Cavalcanti A, and Gorenstein C. Is nonverbal behavior in patients and interviewers relevant to the assessment of depression and its recovery? A study with Dutch and Brazilian patients. Psychiatry Res. (2017) 250:59–64. doi: 10.1016/j.psychres.2017.01.053

PubMed Abstract | Crossref Full Text | Google Scholar

40. Fisch HU, Frey S, and Hirsbrunner HP. Analyzing nonverbal behavior in depression. J Abnorm Psychol. (1983) 92:307–18. doi: 10.1037/0021-843X.92.3.307

PubMed Abstract | Crossref Full Text | Google Scholar

41. Fisher H, Reiss PT, Atias D, Malka M, Shahar B, Shamay-Tsoory S, et al. Facing emotions: between- and within-sessions changes in facial expression during psychological treatment for depression. Clin Psychol Sci. (2023) 12:840–54. doi: 10.1177/21677026231195793

Crossref Full Text | Google Scholar

42. Gaebel W and Wölwer W. Facial expressivity in the course of schizophrenia and depression. Eur Arch Psychiatry Clin Neurosci. (2004) 254:335–42. doi: 10.1007/s00406-004-0510-5

PubMed Abstract | Crossref Full Text | Google Scholar

43. Girard JM, Cohn JF, Mahoor MH, Mavadati S, and Rosenwald DP. Social risk and depression: evidence from manual and automatic facial expression analysis. Proc Int Conf Autom Face Gesture Recognit Int Conf Autom Face Gesture Recognit. (2013), 1–8. doi: 10.1109/FG.2013.6553748

PubMed Abstract | Crossref Full Text | Google Scholar

44. Jiang Z, Harati S, Crowell A, Mayberg HS, Nemati S, and Clifford GD. Classifying major depressive disorder and response to deep brain stimulation over time by analyzing facial expressions. IEEE Trans Biomed Eng. (2021) 68:664–72. doi: 10.1109/TBME.2020.3010472

PubMed Abstract | Crossref Full Text | Google Scholar

45. Liu D, Liu B, Lin T, Liu G, Yang G, Qi D, et al. Measuring depression severity based on facial expression and body movement using deep convolutional neural network. Front Psychiatry. (2022) 13:1017064. doi: 10.3389/fpsyt.2022.1017064

PubMed Abstract | Crossref Full Text | Google Scholar

46. Mackintosh JH, Kumar R, and Kitamura T. Blink rate in psychiatric illness. Br J Psychiatry J Ment Sci. (1983) 143:55–7. doi: 10.1192/bjp.143.1.55

PubMed Abstract | Crossref Full Text | Google Scholar

47. McCall MV, Riva-Posse P, Garlow SJ, Mayberg HS, and Crowell AL. Analyzing non-verbal behavior throughout recovery in a sample of depressed patients receiving deep brain stimulation. Neurol Psychiatry Brain Res. (2020) 37:33–40. doi: 10.1016/j.npbr.2020.05.002

PubMed Abstract | Crossref Full Text | Google Scholar

48. Meya U and Renfordt E. Can changes in eye-contacts predict therapeutic outcome in schizophrenic patients undergoing neuroleptic treatment? Results of a preliminary study. Pharmacopsychiatry. (1986) 19:429–33. doi: 10.1055/s-2007-1017282

PubMed Abstract | Crossref Full Text | Google Scholar

49. Paulick J, Rubel JA, Deisenhofer A-K, Schwartz B, Thielemann D, Altmann U, et al. Diagnostic features of nonverbal synchrony in psychotherapy: Comparing depression and anxiety. Cogn Ther Res. (2018) 42:539–51. doi: 10.1007/s10608-018-9914-9

Crossref Full Text | Google Scholar

50. Schneider F, Ellgring H, Friedrich J, Fus I, Beyer T, Heimann H, et al. The effects of neuroleptics on facial action in schizophrenic patients. Pharmacopsychiatry. (1992) 25:233–9. doi: 10.1055/s-2007-1014412

PubMed Abstract | Crossref Full Text | Google Scholar

51. Ulrich G and Harms K. A video analysis of the non-verbal behaviour of depressed patients before and after treatment. J Affect Disord. (1985) 9:63–7. doi: 10.1016/0165-0327(85)90011-4

PubMed Abstract | Crossref Full Text | Google Scholar

52. World Health Organization. International classification of diseases: [9th] ninth revision, basic tabulation list with alphabetic index. Geneva: World Health Organization. (1978), 331. Available online at: https://iris.who.int/handle/10665/39473.

Google Scholar

53. Ramseyer F and Tschacher W. Nonverbal synchrony in psychotherapy: Coordinated body movement reflects relationship quality and outcome. J Consult Clin Psychol. (2011) 79:284–95. doi: 10.1037/a0023419

PubMed Abstract | Crossref Full Text | Google Scholar

54. Porcelli S, Van Der Wee N, van der Werff S, Aghajani M, Glennon JC, van Heukelum S, et al. Social brain, social dysfunction and social withdrawal. Neurosci Biobehav Rev. (2019) 97:10–33. doi: 10.1016/j.neubiorev.2018.09.012

PubMed Abstract | Crossref Full Text | Google Scholar

55. Allen NB and Badcock PBT. The social risk hypothesis of depressed mood: evolutionary, psychosocial, and neurobiological perspectives. Psychol Bull. (2003) 129:887–913. doi: 10.1037/0033-2909.129.6.887

PubMed Abstract | Crossref Full Text | Google Scholar

56. Sloman L, Gilbert P, and Hasey G. Evolved mechanisms in depression: the role and interaction of attachment and social rank in depression. J Affect Disord. (2003) 74:107–21. doi: 10.1016/S0165-0327(02)00116-7

PubMed Abstract | Crossref Full Text | Google Scholar

57. Hale WW, Jansen JH, Bouhuys AL, and van den Hoofdakker RH. Depression relapse and ethological measures. Psychiatry Res. (1997) 70:57–64. doi: 10.1016/S0165-1781(97)03060-6

PubMed Abstract | Crossref Full Text | Google Scholar

58. Bos EH, Geerts E, and Bouhuys AL. Non-verbal interaction involvement as an indicator of prognosis in remitted depressed subjects. Psychiatry Res. (2002) 113:269–77. doi: 10.1016/S0165-1781(02)00268-8

PubMed Abstract | Crossref Full Text | Google Scholar

59. Fiquer JT, Moreno RA, Brunoni AR, Barros VB, Fernandes F, and Gorenstein C. What is the nonverbal communication of depression? Assessing expressive differences between depressive patients and healthy volunteers during clinical interviews. J Affect Disord. (2018) 238:636–44. doi: 10.1016/j.jad.2018.05.071

PubMed Abstract | Crossref Full Text | Google Scholar

60. Troisi A, Pasini A, Bersani G, Grispini A, and Ciani N. (1989) 17:129–36. x. J. Affect. Disord.

Google Scholar

61. Ranelli CJ and Miller RE. Behavioral predictors of amitriptyline response in depression. Am J Psychiatry. (1981) 138:30–4. doi: 10.1176/ajp.138.1.30

PubMed Abstract | Crossref Full Text | Google Scholar

62. Bos EH, Bouhuys AL, Geerts E, van Os TWDP, and Ormel J. Stressful life events as a link between problems in nonverbal communication and recurrence of depression. J Affect Disord. (2007) 97:161–9. doi: 10.1016/j.jad.2006.06.011

PubMed Abstract | Crossref Full Text | Google Scholar

63. Brüne M, Sonntag C, Abdel-Hamid M, Lehmkämper C, Juckel G, and Troisi A. Nonverbal behavior during standardized interviews in patients with schizophrenia spectrum disorders. J Nerv Ment Dis. (2008) 196:282–8. doi: 10.1097/NMD.0b013e31816a4922

PubMed Abstract | Crossref Full Text | Google Scholar

64. Worswick E, Dimic S, Wildgrube C, and Priebe S. Negative symptoms and avoidance of social interaction: A study of non-verbal behaviour. Psychopathology. (2018) 51:1–9. doi: 10.1159/000484414

PubMed Abstract | Crossref Full Text | Google Scholar

65. Chapellier V, Pavlidou A, Maderthaner L, von Känel S, and Walther S. The impact of poor nonverbal social perception on functional capacity in schizophrenia. Front Psychol. (2022) 13. doi: 10.3389/fpsyg.2022.804093

PubMed Abstract | Crossref Full Text | Google Scholar

66. Pappa S and Dazzan P. Spontaneous movement disorders in antipsychotic-naive patients with first-episode psychoses: a systematic review. Psychol Med. (2009) 39:1065–76. doi: 10.1017/S0033291708004716

PubMed Abstract | Crossref Full Text | Google Scholar

67. Bozzatello P, Bellino S, and Rocca P. Predictive factors of treatment resistance in first episode of psychosis: A systematic review. Front Psychiatry. (2019) 10. doi: 10.3389/fpsyt.2019.00067

PubMed Abstract | Crossref Full Text | Google Scholar

68. Ayehu M, Shibre T, Milkias B, and Fekadu A. Movement disorders in neuroleptic-naïve patients with schizophrenia spectrum disorders. BMC Psychiatry. (2014) 14:280. doi: 10.1186/s12888-014-0280-1

PubMed Abstract | Crossref Full Text | Google Scholar

69. Mayo Clinic. Haloperidol (Oral Route) Side Effects - Mayo Clinic. Available online at: https://www.mayoclinic.org/drugs-supplements/haloperidol-oral-route/side-effects/drg-20064173 (Accessed June 8, 2023).

Google Scholar

70. Moncrieff J, Gupta S, and Horowitz MA. Barriers to stopping neuroleptic (antipsychotic) treatment in people with schizophrenia, psychosis or bipolar disorder. Ther Adv Psychopharmacol. (2020) 10:2045125320937910. doi: 10.1177/2045125320937910

PubMed Abstract | Crossref Full Text | Google Scholar

71. Na I, Aldrees A, Hakeem A, Mohaisen L, Umer M, AlHammadi DA, et al. FacialNet: facial emotion recognition for mental health analysis using UNet segmentation with transfer learning model. Front Comput Neurosci. (2024) 18. doi: 10.3389/fncom.2024.1485121

PubMed Abstract | Crossref Full Text | Google Scholar

72. Löchner J, Carlbring P, Schuller B, Torous J, and Sander LB. Digital interventions in mental health: An overview and future perspectives. Internet Interv. (2025) 40:100824. doi: 10.1016/j.invent.2025.100824

PubMed Abstract | Crossref Full Text | Google Scholar

73. Bond RR, Mulvenna MD, Potts C, O’Neill S, Ennis E, and Torous J. Digital transformation of mental health services. NPJ Ment Health Res. (2023) 2:13. doi: 10.1038/s44184-023-00033-y

PubMed Abstract | Crossref Full Text | Google Scholar

74. Vudathaneni VKP, Lanke RB, Mudaliyar MC, Movva KV, Mounika Kalluri L, and Boyapati R. The impact of telemedicine and remote patient monitoring on healthcare delivery: A comprehensive evaluation. Cureus. (2024) 16:e55534. doi: 10.7759/cureus.55534

PubMed Abstract | Crossref Full Text | Google Scholar

75. Torous J, Linardon J, Goldberg SB, Sun S, Bell I, Nicholas J, et al. The evolving field of digital mental health: current evidence and implementation issues for smartphone apps, generative artificial intelligence, and virtual reality. World Psychiatry. (2025) 24:156. doi: 10.1002/wps.21299

PubMed Abstract | Crossref Full Text | Google Scholar

76. Baranyi G, Fazel S, Langerfeldt SD, and Mundt AP. The prevalence of comorbid serious mental illnesses and substance use disorders in prison populations: a systematic review and meta-analysis. Lancet Public Health. (2022) 7:e557–68. doi: 10.1016/S2468-2667(22)00093-7

PubMed Abstract | Crossref Full Text | Google Scholar

77. Peham D, Bock A, Schiestl C, Huber E, Zimmermann J, Kratzer D, et al. Facial affective behavior in mental disorder. J Nonverbal Behav. (2015) 39:371–96. doi: 10.1007/s10919-015-0216-6

Crossref Full Text | Google Scholar

78. Mueser KT, Salyers MP, Rosenberg SD, Goodman LA, Essock SM, Osher FC, et al. Interpersonal trauma and posttraumatic stress disorder in patients with severe mental illness: demographic, clinical, and health correlates. Schizophr Bull. (2004) 30:45–57. doi: 10.1093/oxfordjournals.schbul.a007067

PubMed Abstract | Crossref Full Text | Google Scholar

79. Reilly S, Olier I, Planner C, Doran T, Reeves D, Ashcroft DM, et al. Inequalities in physical comorbidity: a longitudinal comparative cohort study of people with severe mental illness in the UK. BMJ Open. (2015) 5:e009010. doi: 10.1136/bmjopen-2015-009010

PubMed Abstract | Crossref Full Text | Google Scholar

80. Mijwil MM, Aggarwal K, Sonia S, Al-Mistarehi A-H, Alomari S, Gök M, et al. Has the future started? The current growth of artificial intelligence, machine learning, and deep learning. Iraqi J Comput Sci Math. (2022) 3:115–23. doi: 10.52866/ijcsm.2022.01.01.013

Crossref Full Text | Google Scholar

81. McDuff D, Girard JM, and Kaliouby Re. Large-scale observational evidence of cross-cultural differences in facial behavior. J Nonverbal Behav. (2017) 41:1–19. doi: 10.1007/s10919-016-0244-x

Crossref Full Text | Google Scholar

82. Saheb T, Jamthe S, and Saheb T. Developing a conceptual framework for identifying the ethical repercussions of artificial intelligence: A mixed method analysis. J AI Robot Workplace Autom. (2022) 41:1–19. doi: 10.69554/FFZO4203

Crossref Full Text | Google Scholar

83. Renier LA, Schmid Mast M, Dael N, and Kleinlogel EP. Nonverbal social sensing: what social sensing can and cannot do for the study of nonverbal behavior from video. Front Psychol. (2021) 12. doi: 10.3389/fpsyg.2021.606548

PubMed Abstract | Crossref Full Text | Google Scholar