Predicting symptom response and engagement in a digital intervention among individuals with schizophrenia and related psychoses

Introduction: Despite existing work examining the effectiveness of smartphone digital interventions for schizophrenia at the group level, response to digital treatments is highly variable and requires more research to determine which persons are most likely to benefit from a digital intervention.

Materials and methods: The current work utilized data from an open trial of patients with psychosis (N = 38), primarily schizophrenia spectrum disorders, who were treated with a psychosocial intervention using a smartphone app over a one-month period. Using an ensemble of machine learning models, pre-intervention data, app use data, and semi-structured interview data were utilized to predict response to change in symptom scores, engagement patterns, and qualitative impressions of the app.

Results: Machine learning models were capable of moderately (r = 0.32–0.39, R² = 0.10–0.16, MAE_norm = 0.13–0.29) predicting interaction and experience with the app, as well as changes in psychosis-related psychopathology.

Conclusion: The results suggest that individual smartphone digital intervention engagement is heterogeneous, and symptom-specific baseline data may be predictive of increased engagement and positive qualitative impressions of digital intervention in patients with psychosis. Taken together, interrogating individual response to and engagement with digital-based intervention with machine learning provides increased insight to otherwise ignored nuances of treatment response.

Introduction

Schizophrenia, a primary psychotic disorder, is a serious mental illness characterized by debilitating symptoms including delusions, hallucinations, disorganized speech and behavior, and diminished emotional expression (1, 2). Current studies estimate that schizophrenia affects up to 0.64% of the United States population (3–5). However, despite schizophrenia’s comparatively low incidence (6), it had an economic burden of $155.7 billion in 2013 (7), and remains a major contributor to the global burden of disease with two-thirds of affected individuals experiencing persistent symptoms following treatment (1, 8). Further, schizophrenia is often comorbid with anxiety disorders and depression (9, 10). Thus, illness burden is of particular concern, with symptom severity shown to be negatively correlated with physical health, psychological health, and relationships, and one in ten completing suicide (11, 12). Despite recent improvements in diagnostic accuracy and treatment efficacy, schizophrenia continues to negatively impact patients’ overall quality of life (11), highlighting the necessity for additional efforts to address treatment response at the individual level.

Currently, the mainstay treatment for schizophrenia includes the use of antipsychotics, often coupled with regular psychosocial interventions (13, 14). Psychosocial interventions can be offered as individual (supportive counseling, personal therapy, social skills therapy, vocational rehabilitation therapy); cognitive-behavioral; and group (interactive/social therapy) interventions (14). Despite the relative efficacy of these combined approaches, the side-effects of antipsychotic treatment can prove debilitating to schizophrenia patients. Antipsychotic side-effects may include physical symptoms, such as movement disturbances, metabolic derangements and weight gain, sedation, and drooling (15, 16, 17–19), as well as emotional and cognitive blunting (20). As a result, the integration of alternative interventions may prove useful in reducing the side-effect burden of antipsychotics.

Research has shown promise in the use of scalable digital therapeutics in patients living with serious mental disorders, such as schizophrenia and bipolar disorder (21). Although few studies have directly analyzed the efficacy of digital interventions for schizophrenia, existing research has suggested that this method of intervention may be efficacious for schizophrenia treatment or management. For example, PRIME, or Personalized Real-time Intervention for Motivational Enhancement, is a mobile application designed to supplement antipsychotic and psychotherapeutic treatments for schizophrenia. The intervention was found to be effective in improving the mood and motivation of young patients with schizophrenia. Further, PRIME users experienced improvements in depression, self-efficacy, and reward learning (22, 23). Similarly, App4Independence (A4i), a community-centric app designed for schizophrenia patients, offers forums, appointment scheduling, and text-based functions aimed at improving illness self-management (24). Like PRIME, A4i showed modest reductions in symptoms of depression, further pronounced after controlling for gender, age, and other baseline symptoms (24).

Treatment of schizophrenia remains a challenge, partially owing to its highly heterogeneous nature (25–29), with little known of personalized prognostic and treatment factors. With growing numbers of digital treatment options for mental health in the context of this heterogeneity, it is important to understand not only the group efficacy of such interventions, but also the profile of those individuals most likely to benefit. With such an understanding, individuals could be effectively “matched” to the specific interventions most likely to improve their symptoms.

Machine learning, coupled with highly dimensional datasets, like A4i, is uniquely positioned to address these challenges, having shown promise in constructing personalized models, with a capacity to accurately predict individual-level treatment response (30, 31). Specific to individuals with schizophrenia, an extensive corpus of research has shown machine learning efficacious in classifying schizophrenia and psychosis-related symptoms from neuroimaging data (32–34), qualitative social media information (35, 36), and passively collected smartphone data. Notably, virtual communication was positively associated with increased negative affect measures (37), highlighting the necessity to interrogate the driving factors in engagement with digital interventions in individuals with schizophrenia. Furthermore, machine learning has the capacity to model complex relationships in large datasets, with model introspection made possible by high power computing methods, such as Shapley Additive Explanations (SHAP) (38). Methods, like SHAP, address the “black box” problem of machine learning by offering a manner of visually representing the directionality and magnitude of those features most important in the model’s outcome prediction. This allows for the development of downstream digital biomarkers and phenotypes of psychiatric disorders.

In the current study, we utilize data from an open trial delivering the A4i intervention to persons with schizophrenia, schizoaffective disorder, and related psychoses that found modest symptom improvement at the group level (24), to better understand personalized markers of digital intervention engagement and response. We hypothesized (1) unique baseline patient characteristics paired with machine learning would moderately predict (r > 0.3) individual symptom response to, engagement with, and sentiment toward A4i (an app-delivered, digital intervention) (2) specifically, we hypothesized that people with higher affective symptoms (e.g., depression) and lower psychotic symptoms would have the most robust use and response, resulting in more positive sentiment toward the app; this hypothesis can be contextualized in research which suggests better overall prognosis and treatment response for patients with higher affective symptoms (39). Further, (3) we hypothesized that persons with higher interpersonal sensitivity would have the strongest response to the intervention (change in composite BSI score), given the built-in peer-engagement application feature, which was reported in the original study as a user-reported strength of the A4i intervention (24). To evaluate these hypotheses, we developed three machine learning pipelines aimed at modeling the relationship between baseline patient characteristics and (i) response to the digital intervention, (ii) level of engagement with the intervention, and (iii) user sentiment toward the intervention.

Materials and methods

Participants

Participants (N = 38, 2.6% transgender, 71.1% men, 26.3% women, age_mean = 31.42 ± 8.60) were included in the final study population based on previously described inclusion and exclusion criteria (24). The initial study design consisted of a 3–4 week engagement with the A4i app with pre-post assessments. The study was reviewed and approved by an institutional Research Ethics Board; participants provided written consent, and the protocol was registered with clinicaltrials.gov (NCT03649815) (24).

Intervention

The A4i functionality included personalized prompts, activity scheduling, connections to social engagement resources, evidenced-based content tailored to management of psychotic symptoms, a peer engagement network, daily wellness check-ins, and passively collected phone-use information, used as a proxy for sleep and activity (24).

Data collection and outcome metrics: Quantitative data set

Participants provided demographic information, mobile technology use information, and completed quantitative pre-post symptom and intervention engagement metrics (24). The quantitative metrics included: (a) the Brief Adherence Rating Scale (BARS) to examine implications of A4i for medication use (40), (b) the Person Recovery Outcome Measure (PROM) to assess degree of engagement in the recovery process (41), and (c) the Brief Symptom Inventory (BSI) to assess psychiatric symptoms pre and post-intervention. The BSI comprises domains measuring: psychoticism, somatization, depression, hostility, phobia, obsessive-compulsive, anxiety, paranoia, and interpersonal sensitivity (42). Additionally, A4i usage data from each participant during the trial was passively collected, including: (i) a count of participants’ total active interaction with the app, (ii) the number of days each participant engaged with the app, (iii) the participants’ average interaction with the app per day, and (iv) app usage categorized as “low” or “high.” Participant BSI total and subdomain scores were included in the present analysis; BARS scores, PROM scores and demographic information were not.

Post-intervention semi-structured interview

A semi-structured interview was completed at the post-A4i use assessment, which included a series of seven questions providing qualitative feedback from the participants on the functionality, effectiveness, and overall experience interacting with the app (e.g., “What were your favorite features of the app?”) (24). The complete semi-structured interview is provided in the Supporting information section of the original publication (24).

Semi-structured interview response sentiment quantification

We extracted participants’ overall response sentiment from the semi-structured interviews. Individual responses to all questions were concatenated using Python (v3.8.3) for uniformity across participants (43). Overall response sentiment was derived from the concatenated qualitative data, using the Python package TextBlob (Version 0.16.0) to assess polarity (i.e., the valence of the participants responses on a –1 to 1 scale, with a lower score reflecting a more negative statement and a higher score reflecting a more positive statement) (44).

Data preprocessing

Baseline BSI total score, baseline BSI subcategory scores, and passively collected A4i use metric features included in modeling were individually standardized resulting in a μ = 0 and σ = 1 within that feature. Feature standardization has been shown to increase model efficiency and accuracy in machine learning approaches (45), and provides a consistent value range for features when considering their relative influence on a model’s predictions.

Theoretical machine learning modeling framework

We implemented a hypothesis driven framework via the utilization of three separate ensemble machine learning models to interrogate individual-level factors driving (1) app efficacy, (2) app engagement, and (3) qualitative app impressions (see Table 1). The machine learning models implemented in this study were as follows:

1. Symptom Severity Change: Fourteen features, including baseline BSI total score, subcategory scores, and passively collected A4i use metrics were used to predict change in BSI total score (Table 1, Model 1). Change in BSI total score was measured as the difference between baseline BSI total score and post-intervention BSI total score (e.g., a negative change reflects an overall decrease in reported symptoms).

2. A4i Engagement: Ten features, including baseline BSI total score and subcategory scores were used to predict a participants’ overall interaction with the A4i app (Table 1, Model 2).

3. Intervention Impressions: Ten features, including baseline BSI total score and subdomain scores were used to predict an individual participant’s sentiment toward the intervention (Table 1, Model 3). A participants’ sentiment was represented by the polarity score of their concatenated semi-structured interview responses.

TABLE 1

Table 1. Machine learning model corresponding hypotheses, features and outcomes.

Practical machine learning model framework

All machine learning modeling followed the same nested leave-one-out (LOO) cross-validation framework (46). A nested cross-validation framework in machine learning is efficacious in allowing for unbiased performance estimates, regardless of sample size (47). In this process, one subject was completely held out, while the rest of the subjects were used as part of a simple LOO cross-validation approach to tune the hyperparameters of the model. This process was repeated N times so each subject was held out at least once. We used an ensemble approach, whereby distinct machine learning models (i.e., linear models, tree based models, and multilayer perceptrons) were individually trained on the data; an approach which has been shown to consistently outperform base algorithms in mental health disorder related outcomes (48). Predictions from these models were used as inputs to a final “deciding” model, which returned a consensus score. The specific modeling architecture and hyperparameters of the Symptom Severity Change, A4i Engagement, and Intervention Impressions models are provided in Supplementary Table 1. The ensemble pipeline’s predictions were evaluated against the observed values to determine the correlative strength (r), the proportion of the variance in the outcome explained by the model’s predictions (R²), and the normalized mean absolute error (MAE_norm) of the respective ensemble model. The normalized mean absolute error was calculated by dividing the mean absolute error by the range of the observed outcome, offering an outcome-agnostic representation of the model’s mean absolute error.

Model explainability

We implemented Shapley Additive Explanations (SHAP) to aid in model interpretability by evaluating the top five most influential features in each of the three models. Intuitively, SHAP allows for model introspection by iteratively perturbing the input data and assessing how this affects the output (38). In this way, the process can determine feature importance, as well as the marginal contribution of each independent variable to the predicted outcome at the patient level, represented by an individual values positioning on the x-axis of Figure 2. Using this information, SHAP can estimate relative feature importance, directional relationships between predictors and outcomes, as well as different order interactions between variables.

Results

Predictive performance and interpretability: Symptom severity change

Participants’ A4i use metrics and baseline BSI scores (Table 1, Model 1) were capable of moderately predicting change in symptom severity (e.g., Pre-Post BSI score difference, where a negative change corresponds to a decrease in BSI score, and thus reduced overall symptoms) (r = 0.32, R² = 0.10, MAE_norm = 0.29) (Figure 1A). Model introspection via SHAP suggested that the most influential feature for predicting change in BSI score was interpersonal sensitivity on the baseline BSI (Figure 2A), where participants with high baseline interpersonal sensitivity were predicted to have a greater reduction in psychiatric symptomatology. Furthermore, lower psychotic and obsessive compulsive traits were predictive of a reduction in psychiatric symptomatology across the intervention. These findings directly address study hypothesis (1), that unique patient characteristics will moderately predict app response, and more specifically study hypothesis (3), which suggested interpersonal sensitivity as a positive prognostic marker for A4i response.

FIGURE 1

Figure 1. Model(s) actual versus predicted values plotted with respective correlative strength. (A) Baseline BSI total score, subcategory scores, and passively collected A4i use metrics were used to predict change in BSI total score. (B) Baseline BSI total score and subcategory scores were used to predict a participants’ overall interaction (visualized as the log-transformation) with the A4i app. (C) Baseline BSI total score and subdomain scores were used to predict an individual participant’s sentiment toward the intervention. r, Pearson’s correlation coefficient.

FIGURE 2

Figure 2. The top 5 most influential features by model. Individual dot color corresponds to the value of the feature, and location on the x-axis corresponds to that point’s relative impact on the model output [e.g., a high-feature value (red) with a corresponding high x-axis value (SHAP value) represents a point that strongly, positively influences the model’s outcome prediction]. (A) The most influential features from baseline BSI total score, subcategory scores, and passively collected A4i use metrics for predicting change in BSI total score. A positive x-axis value (SHAP value) corresponds to an increase in overall symptoms. (B) The most influential features from baseline BSI total score and subcategory scores for predicting a participants’ overall interaction) with the A4i app. A positive x-axis value (SHAP value) corresponds to an increased interaction with A4i. (C) The most influential features from baseline BSI total score and subdomain scores for predicting an individual participant’s sentiment toward the intervention. A positive x-axis value (SHAP value) corresponds to an increase in qualitative A4i impressions.

Predictive performance and interpretability: App4Independence engagement

Baseline symptom severity (measured by pre-intervention BSI scores) moderately predicted participant engagement across the A4i intervention (r = 0.39, R² = 0.16, MAE_norm = 0.16) (Figure 1B). The BSI subdomain depression was the most influential feature, where participants with high baseline depression interacted with the app more during the intervention; however, overall BSI score showed an inverse relationship, where participants with lower overall total symptoms were predicted to have greater app interaction. These findings directly address study hypothesis (1), that that unique patient characteristics will moderately predict person-level app engagement, and more specifically hypothesis (2) that higher affective symptoms would drive A4i response.

Predictive performance and interpretability: Intervention impressions

Baseline BSI scores moderately predicted valence of interview responses (r = 0.34, R² = 0.12, MAE_norm = 0.14) (Figure 1C), with BSI depression scores having the greatest importance. Specifically, participants with high baseline depression were more positive when discussing the app in the semi-structured interview. Interestingly, similar to the A4i engagement model, overall BSI score showed an inverse relationship to the BSI depression subdomain, with lower overall BSI being predictive of more positive qualitative impressions of the intervention. These findings directly address study hypothesis (1), that unique patient characteristics will moderately predict overall app sentiment and, as in section “Predictive performance and interpretability: A4i engagement” – hypothesis (2) – that higher affective symptoms would drive A4i response.

Discussion

This study demonstrates the capacity of unique patient-level factors to predict response to a digital treatment among patients with schizophrenia, schizoaffective disorder, and related psychoses. Important factors included interpersonal sensitivity, psychotic traits, depressive traits, and overall symptom severity (as determined by baseline BSI), as well as digital intervention interaction metrics passively collected during the study. Subsequent analysis established factors associated with participant interaction, engagement and general attitudes toward the digital therapeutic intervention. As a whole, this work aimed to investigate unique patient markers of digital treatment response, as well as highlight those factors most important in predicting high engagement among persons with psychosis. Our results contribute to ongoing development and implementation of mental health digital interventions by identifying unique patient markers to suggest intervention response as well as engagement.

Higher interpersonal sensitivity

The symptom severity change model (Table 1, Model 1) predicted that participants with high baseline interpersonal sensitivity would benefit more from the A4i intervention. This finding may suggest that high interpersonal sensitivity corresponds to increased participant responsiveness to the community-centric platform of the intervention. The role of patient interaction is highlighted by the users during the semi-structured interviews, with one patient responding that the function of the A4i app is to serve as “a safe space online app based community platform where psychosis can be met with care and empathy.” (24). Thus, in line with findings that lack of interpersonal relationships is known to be a significant contributor to reduced quality of life among individuals with schizophrenia (49), patients that actively interacted with the A4i community responded better to the intervention. Further, these findings are congruent with the A4i intervention goals which sought to target interpersonal aspects of psychotic disorders (24).

A second hypothesis for higher emotional sensitivity predicting better A4i response involves a potential association between interpersonal-affective sensitivity and psychotic symptom disorder severity in persons with psychosis. Interpersonal hypersensitivity has been shown characteristic of prodromal psychosis in clinically high risk patients (50). By contrast, patients at later illness stages show a generalized deficit in affect recognition, not characteristic of their earlier-illness-stage counterparts (51). Moreover, fine recognition of sad and neutral affective states have been inversely correlated with measures of disorganization in schizophrenia (51). Taken together these findings suggest (1) interpersonal and affective sensitivity as a potential surrogate marker of earlier-, or prodromal-, stage illness, which is likely to be more responsive to intervention (52) and (2) higher interpersonal sensitivity as potential marker of lower disorganization in psychotic illness, implying a greater capacity to participate and benefit from psychosocial treatments, such as A4i.

Lower psychotic and obsessive compulsive traits

Conversely, participants with lower baseline psychotic traits and/or lower baseline obsessive compulsive traits were also predicted to benefit from the A4i intervention. Psychosis is often characterized by marked perceptual disturbances and disorganization, which may suggest that individuals who were more organized were able to engage more effectively and consistently with the intervention. Notably, individuals at risk of psychosis are found to experience difficulties with interpersonal relationships, manifesting as an inability to communicate distressing psychological experiences to others (53). These troubles with communication and experiential expression likely also affect patients with high levels of psychotic traits resulting in difficulties engaging with other users of A4i, and thus the intervention overall. Similarly, intervention and community engagement may have proven difficult for patients’ with obsessive compulsive traits, as these may involve intrusive, and distressing thoughts. Patients with OCD often have difficulties recognizing affective social cues, regulating emotion (54), and communicating (55). These difficulties likely also exist in individuals with primary or secondary psychoses who have obsessive compulsive traits, preventing effective engagement with socially dependent interventions, such as A4i.

App interaction and feedback

Notably, higher baseline depression predicted both higher A4i engagement and more satisfaction with the app (as measured by post-intervention feedback polarity, see Figures 2B,C). This finding is consistent with research to date demonstrating the prognostic value of affective symptoms (e.g., depression) in schizophrenia (39, 56, 57). In particular, comorbid depressive symptoms have been associated with more positive outcomes, including fewer hospitalizations and fewer illness relapses in schizophrenia (39). Considered in the context of the present study results, we hypothesize depression to likewise represent a marker of ability to engage with a multi-feature digital intervention. Intuitively, it follows that individuals who had greater app engagement also had greater symptom improvement (see Figure 2A) and therefore would have greater satisfaction with the app (see Figure 2C).

In contrast to depression, higher overall symptom severity at baseline predicted lower A4i engagement and less favorable sentiment toward the app. We hypothesize individuals with greater symptom severity had more disorganization and executive functioning impairment, making it difficult to fully engage with a multi-feature digital intervention, like A4i. It follows that patients who interacted less with the app would have less symptom improvement (see Figure 2A) and therefore would have less favorable impressions (see Figure 2C). These findings are important in light of evidence suggesting overall lower digital app engagement among populations with schizophrenia, likely skewed by small subgroups of heavily engaged participants (58). Understanding the individual-level characteristics that drive engagement and sentiment toward digital technologies is essential for future mental health app development and implementation.

Limitations

Despite the strengths of leveraging a LOO cross-validation machine learning framework to investigate unique patient markers of digital treatment response for individuals with schizophrenia, there are a number of limitations concerning the study sample and design that should be addressed. (1) The original study used a 38-person sample drawn from urban Canadian residents, limiting generalizability of the reported (2) This study did not conduct long-term patient follow-up, an important aspect of comprehensive treatment analyses, thus it is unknown whether the observed improvements in schizophrenia psychopathology will persist for these patients. As such, the long-term efficacy of A4i, and the long-term importance of the identified unique patient markers, cannot be evaluated. (3) Due to the method of analysis, the present results only reflect predictive capacity, not causality. (4) While the BSI captures positive and negative symptoms associated with psychotic disorders, structured interviews specific to psychosis and schizophrenia (e.g., PSYRATS-D) were not included. (5) The present analyses did not incorporate demographic information or lifestyle-related information, and thus cannot account for the manner in which demographic features or living and work environments influenced participant engagement with the A4i intervention.

Conclusion

The present study sought to interrogate the A4i app as a digital intervention for schizophrenia patients via an ensemble LOO machine learning approach, allowing insight into the most influential, unique patient characteristics for predicting intervention response and engagement. Notably, high interpersonal sensitivity was predictive of total symptom reduction across the digital intervention, and high depression was predictive of increased digital intervention engagement and positive qualitative impressions. Taken together, these findings highlight the necessity for patient-level interrogation of treatment efficacy in mental health, particularly for schizophrenia where clinical presentation is heterogeneous. Future work should build upon the present findings to consider how individual demographic characteristics (e.g., gender, race) may influence differential engagement with a digital intervention, particularly in individuals with psychosis. Additionally, future work should aim to extend the current methodology to more traditional interventions for individuals with schizophrenia (e.g., psychosocial interventions combined with antipsychotic use) to identify unique patient characteristics predictive of intervention response, and thus tailor treatment type based on an individual’s clinical presentation.

Data availability statement

Publicly available datasets were analyzed in this study. This data can be found here: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0219491. All relevant data are within the supporting information material.

Ethics statement

The studies involving human participants were reviewed and approved by Research Ethics Board, Centre for Addiction and Mental Health. The patients/participants provided their written informed consent to participate in this study.

Author contributions

GP, MH, and NJ contributed to conceptualization and design of the study. GP and MN implemented the modeling and analyses. GP, MH, MN, JM, and NJ wrote sections of the manuscript. All authors contributed to manuscript revision, read, and approved the submitted version.

Funding

This work was funded by the National Institute of Mental Health and the National Institute of General Medical Sciences under grant 1 R01 MH123482-01.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2022.807116/full#supplementary-material

References

1. Saha S, Chant D, Welham J, McGrath J. A systematic review of the prevalence of schizophrenia. PLoS Med. (2005) 2:e141. doi: 10.1371/journal.pmed.0020141

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Tandon R, Gaebel W, Barch DM, Bustillo J, Gur RE, Heckers S, et al. Definition and description of schizophrenia in the DSM-5. Schizophr Res. (2013) 150:3–10. doi: 10.1016/j.schres.2013.05.028

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Desai PR, Lawson KA, Barner JC, Rascati KL. Estimating the direct and indirect costs for community-dwelling patients with schizophrenia: schizophrenia-related costs for community-dwellers. J Pharmaceut Health Serv Res. (2013) 4:187–94. doi: 10.1111/jphs.12027

CrossRef Full Text | Google Scholar

4. Kessler RC, Birnbaum H, Demler O, Falloon IRH, Gagnon E, Guyer M, et al. The prevalence and correlates of nonaffective psychosis in the National Comorbidity Survey Replication (NCS-R). Biol Psychiatry. (2005) 58:668–76. doi: 10.1016/j.biopsych.2005.04.034

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Wu EQ, Shi L, Birnbaum H, Hudson T, Kessler R. Annual prevalence of diagnosed schizophrenia in the USA: a claims data analysis approach. Psychol Med. (2006) 36:1535–40. doi: 10.1017/S0033291706008191

PubMed Abstract | CrossRef Full Text | Google Scholar

6. McGrath J, Saha S, Chant D, Welham J. Schizophrenia: a concise overview of incidence, prevalence, and mortality. Epidemiol Rev. (2008) 30:67–76. doi: 10.1093/epirev/mxn001

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Cloutier M, Aigbogun MS, Guerin A, Nitulescu R, Ramanakumar AV, Kamat SA, et al. The economic burden of schizophrenia in the United States in 2013. J Clin Psychiatry. (2016) 77:764–71. doi: 10.4088/JCP.15m10278

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Bell CC. DSM-IV: diagnostic and statistical manual of mental disorders. JAMA. (1994) 272:828. doi: 10.1001/jama.1994.03520100096046

CrossRef Full Text | Google Scholar

9. Achim AM, Maziade M, Raymond E, Olivier D, Merette C, Roy M-A. How prevalent are anxiety disorders in Schizophrenia? A meta-analysis and critical review on a significant association. Schizophr Bull. (2011) 37:811–21. doi: 10.1093/schbul/sbp148

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Häfner H, Maurer K, Trendler G, an der Heiden W, Schmidt M, Könnecke R. Schizophrenia and depression: challenging the paradigm of two separate diseases—A controlled study of schizophrenia, depression and healthy controls. Schizophr Res. (2005) 77:11–24. doi: 10.1016/j.schres.2005.01.004

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Singh P, Midha A, Chugh K, Solanki R. Schizophrenia: impact on quality of life. Indian J Psychiatry. (2008) 50:181. doi: 10.4103/0019-5545.43632

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Siris SG. Suicide and schizophrenia. J Psychopharmacol. (2001) 15:127–35. doi: 10.1177/026988110101500209

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Kane JM, Marder SR. Psychopharmacologic treatment of schizophrenia. Schizophr Bull. (1993) 19:287–302. doi: 10.1093/schbul/19.2.287

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Patel KR, Cherian J, Gohil K, Atkinson D. Schizophrenia: overview and treatment options. P T. (2014) 39:638–45.

Google Scholar

15. Burke RE, Fahn S, Jankovic J, Marsden CD, Lang AE, Gollomp S, et al. Tardive dystonia: Late-onset and persistent dystonia caused by antipsychotic drugs. Neurology. (1982) 32:1335–1335. doi: 10.1212/WNL.32.12.1335

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Caroff SN, Mann SC, Campbell EC, Sullivan KA. Movement disorders associated with atypical antipsychotic drugs. J Clin Psychiatry. (2002) 63(Suppl 4):12–9.

Google Scholar

17. De Fazio P, Gaetano R, Caroleo M, Cerminara G, Maida F, Bruno A, et al. Rare and very rare adverse effects of clozapine. Neuropsychiatr Dis Treat. (2015) 11:1995–2003. doi: 10.2147/NDT.S83989

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Lieberman JA, Stroup TS, McEvoy JP, Swartz MS, Rosenheck RA, Perkins DO, et al. Effectiveness of antipsychotic drugs in patients with chronic schizophrenia. New Engl J Med. (2005) 353:1209–23. doi: 10.1056/NEJMoa051688

PubMed Abstract | CrossRef Full Text | Google Scholar

19. van Harten PN, Hoek HW, Kahn RS. Acute dystonia induced by drug treatment. BMJ. (1999) 319:623–6.

Google Scholar

20. Moritz S, Andreou C, Klingberg S, Thoering T, Peters MJV. Assessment of subjective cognitive and emotional effects of antipsychotic drugs. Effect by defect? Neuropharmacology. (2013) 72:179–86. doi: 10.1016/j.neuropharm.2013.04.039

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Biagianti B, Hidalgo-Mazzei D, Meyer N. Developing digital interventions for people living with serious mental illness: Perspectives from three mHealth studies. Evid Based Ment Health. (2017) 20:98–101. doi: 10.1136/eb-2017-102765

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Schlosser D, Campellone T, Kim D, Truong B, Vergani S, Ward C, et al. Feasibility of PRIME: a cognitive neuroscience-informed mobile app intervention to enhance motivated behavior and improve quality of life in recent onset schizophrenia. JMIR Res Protocols. (2016) 5:e77. doi: 10.2196/resprot.5450

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Schlosser DA, Campellone TR, Truong B, Etter K, Vergani S, Komaiko K, et al. Efficacy of PRIME, a mobile app intervention designed to improve motivation in young people with schizophrenia. Schizophr Bull. (2018) 44:1010–20. doi: 10.1093/schbul/sby078

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Kidd SA, Feldcamp L, Adler A, Kaleis L, Wang W, Vichnevetski K, et al. Feasibility and outcomes of a multi-function mobile health approach for the schizophrenia spectrum: App4Independence (A4i). PLoS One. (2019) 14:e0219491. doi: 10.1371/journal.pone.0219491

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Buchanan RW, Carpenter WT. Domains of psychopathology: an approach to the reduction of heterogeneity in schizophrenia. J Nerv Mental Dis. (1994) 182:193–204.

PubMed Abstract | Google Scholar

26. Buchsbaum MS, Haier RJ. Biological homogeneity, symptom heterogeneity, and the diagnosis of schizophrenia*. Schizophr Bull. (1978) 4:473–5. doi: 10.1093/schbul/4.4.473

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Carpenter WT, Kirkpatrick B. The heterogeneity of the long-term course of schizophrenia. Schizophr Bull. (1988) 14:645–52. doi: 10.1093/schbul/14.4.645

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Lang FU, Kösters M, Lang S, Becker T, Jäger M. Psychopathological long-term outcome of schizophrenia—A review. Acta Psychiatr Scand. (2013) 127:173–82. doi: 10.1111/a.12030

CrossRef Full Text | Google Scholar

29. Wolfers T, Doan NT, Kaufmann T, Alnæs D, Moberget T, Agartz I, et al. Mapping the heterogeneous phenotype of schizophrenia and bipolar disorder using normative models. JAMA Psychiatry. (2018) 75:1146. doi: 10.1001/jamapsychiatry.2018.2467

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Meinlschmidt G, Tegethoff M, Belardi A, Stalujanis E, Oh M, Jung EK, et al. Personalized prediction of smartphone-based psychotherapeutic micro-intervention success using machine learning. J Affect Disord. (2020) 264:430–7. doi: 10.1016/j.jad.2019.11.071

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Nemesure MD, Heinz MV, Klein R, McFadden J, Jacobson NC. Predictive modeling approach to evaluate individual response to a physical activity digital intervention for subjects with major depressive disorder. PsyArXiv [Preprint]. (2021). Available online at: https://psyarxiv.com/3kjyh/ (accessed on June 06, 2022). doi: 10.31234/osf.io/3kjyh

CrossRef Full Text | Google Scholar

32. Chand GB, Dwyer DB, Erus G, Sotiras A, Varol E, Srinivasan D, et al. Two distinct neuroanatomical subtypes of schizophrenia revealed using machine learning. Brain. (2020) 143:1027–38. doi: 10.1093/brain/awaa025

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Schnack HG, Nieuwenhuis M, van Haren NEM, Abramovic L, Scheewe TW, Brouwer RM, et al. Can structural MRI aid in clinical classification? A machine learning study in two independent samples of patients with schizophrenia, bipolar disorder and healthy subjects. Neuroimage. (2014) 84:299–306. doi: 10.1016/j.neuroimage.2013.08.053

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Veronese E, Castellani U, Peruzzo D, Bellani M, Brambilla P. Machine learning approaches: from theory to application in schizophrenia. Computat Math Methods Med. (2013) 2013:867924. doi: 10.1155/2013/867924

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Bae YJ, Shim M, Lee WH. Schizophrenia detection using machine learning approach from social media content. Sensors. (2021) 21:5924. doi: 10.3390/s21175924

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Birnbaum ML, Ernala SK, Rizvi AF, De Choudhury M, Kane JM. A collaborative approach to identifying social media markers of schizophrenia by employing machine learning and clinical appraisals. J Med Internet Res. (2017) 19:e289. doi: 10.2196/jmir.7956

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Wang R, Wang W, Obuchi M, Scherer E, Brian R, Ben-Zeev D, et al. On predicting relapse in schizophrenia using mobile sensing in a randomized control trial. Proceedings of the 2020 IEEE International Conference on Pervasive Computing and Communications (PerCom). Austin, TX: IEEE (2020). p. 1–8. doi: 10.1109/PerCom45495.2020.9127365

CrossRef Full Text | Google Scholar

38. Lundberg SM, Lee S-I. A unified approach to interpreting model predictions. Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook, NY: Curran Associates Inc (2017). p. 4765–74.

Google Scholar

39. Arrasate M, González-Ortega I, García-Alocén A, Alberich S, Zorrilla I, González-Pinto A. Prognostic value of affective symptoms in first-admission psychotic patients. Int J Mol Sci. (2016) 17:1039. doi: 10.3390/ijms17071039

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Byerly MJ, Nakonezny PA, Rush AJ. The Brief Adherence Rating Scale (BARS) validated against electronic monitoring in assessing the antipsychotic medication adherence of outpatients with schizophrenia and schizoaffective disorder. Schizophr Res. (2008) 100:60–9. doi: 10.1016/j.schres.2007.12.470

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Barbic SP, Kidd SA, Durisko ZT, Yachouh R, Rathitharan G, McKenzie K. What are the personal recovery needs of community-dwelling individuals with mental illness? Preliminary findings from the Canadian personal recovery outcome measurement (C-PROM) Study. Can J Commun Ment Health. (2018) 37:29–47. doi: 10.7870/cjcmh-2018-005

CrossRef Full Text | Google Scholar

42. Derogatis LR, Melisaratos N. The brief symptom inventory: an introductory report. Psychol Med. (1983) 13:595–605.

Google Scholar

43. Rossum G, van Drake FL, Van Rossum G. The Python Language Reference (Release 3.0.1 [Repr.]). Wilmington: Python Software Foundation (2010).

Google Scholar

44. Loria S. Textblob Documentation. (2018). Available online at: https://buildmedia.readthedocs.org/media/pdf/textblob/latest/textblob.pdf (accessed April 26, 2020).

Google Scholar

45. Thara DK, PremaSudha BG, Xiong F. Auto-detection of epileptic seizure events using deep neural network with different feature scaling techniques. Pattern Recogn Lett. (2019) 128:544–50. doi: 10.1016/j.patrec.2019.10.029

CrossRef Full Text | Google Scholar

46. Webb GI, Sammut C, Perlich C, Horváth T, Wrobel S, Korb KB, et al. Leave-one-out cross-validation. In: C Sammut, GI Webb editors. Encyclopedia of Machine Learning. Berlin: Springer (2011). p. 600–1. doi: 10.1007/978-0-387-30164-8_469

CrossRef Full Text | Google Scholar

47. Varma S, Simon R. Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics. (2006) 7:91. doi: 10.1186/1471-2105-7-91

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Tao X, Chi O, Delaney PJ, Li L, Huang J. Detecting depression using an ensemble classifier based on Quality of Life scales. Brain Informat. (2021) 8:2. doi: 10.1186/s40708-021-00125-5

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Stevens AK, McNichol J, Magalhaes L. Social relationships in schizophrenia: a review. Pers Ment Health. (2009) 3:203–16. doi: 10.1002/pmh.82

CrossRef Full Text | Google Scholar

50. Masillo A, Day F, Laing J, Howes O, Fusar-Poli P, Byrne M, et al. Interpersonal sensitivity in the at-risk mental state for psychosis. Psychol Med. (2012) 42:1835–45. doi: 10.1017/S0033291711002996

PubMed Abstract | CrossRef Full Text | Google Scholar

51. Romero-Ferreiro MV, Aguado L, Rodriguez-Torresano J, Palomo T, Rodriguez-Jimenez R, Pedreira-Massa JL. Facial affect recognition in early and late-stage schizophrenia patients. Schizophr Res. (2016) 172:177–83. doi: 10.1016/j.schres.2016.02.010

PubMed Abstract | CrossRef Full Text | Google Scholar

52. McGlashan TH, Johannessen JO. Early detection and intervention with schizophrenia: rationale. Schizophr Bull. (1996) 22:201–22. doi: 10.1093/schbul/22.2.201

PubMed Abstract | CrossRef Full Text | Google Scholar

53. Byrne R, Morrison AP. Young people at risk of psychosis: a user-led exploration of interpersonal relationships and communication of psychological difficulties: young people at risk of psychosis. Early Intervent Psychiatry. (2010) 4:162–8. doi: 10.1111/j.1751-7893.2010.00171.x

PubMed Abstract | CrossRef Full Text | Google Scholar

54. Jansen M, Overgaauw S, De Bruijn ERA. Social cognition and obsessive-compulsive disorder: a review of subdomains of social functioning. Front Psychiatry. (2020) 11:118. doi: 10.3389/fpsyt.2020.00118

PubMed Abstract | CrossRef Full Text | Google Scholar

55. Cullen B, Samuels J, Grados M, Landa R, Bienvenu OJ, Liang K-Y, et al. Social and communication difficulties and obsessive-compulsive disorder. Psychopathology. (2008) 41:194–200. doi: 10.1159/000120988

PubMed Abstract | CrossRef Full Text | Google Scholar

56. McCabe MS, Fowler RC, Cadoret RJ, Winokur G. Symptom differences in schizophrenia with good and poor prognosis. Am J Psychiatry. (1972) 128:1239–43. doi: 10.1176/ajp.128.10.1239

PubMed Abstract | CrossRef Full Text | Google Scholar

57. Morrissette DA, Stahl SM. Affective symptoms in schizophrenia. Drug Discovery Today. (2011) 8:3–9. doi: 10.1016/j.ddstr.2011.10.005

CrossRef Full Text | Google Scholar

58. Torous J, Staples P, Slaters L, Adams J, Sandoval L, Onnela J, et al. Characterizing smartphone engagement for schizophrenia: results of a naturalist mobile health study. Clin Schizophr Relat Psychoses. (2017) [Online ahead of print]. doi: 10.3371/CSRP.JTPS.071317.

PubMed Abstract | CrossRef Full Text | Google Scholar