What Works and What Doesn’t Work? A Systematic Review of Digital Mental Health Interventions for Depression and Anxiety in Young People

Abstract

Background: A major challenge in providing mental health interventions for young people is making such interventions accessible and appealing to those most in need. Online and app-based forms of therapy for mental health are burgeoning. It is therefore crucial to identify features that are most effective and engaging for young users.

Objectives: This study reports a systematic review and meta-analysis of digital mental health interventions and their effectiveness in addressing anxiety and depression in young people to determine factors that relate to outcomes, adherence, and engagement with such interventions.

Methods: A mixed methods approach was taken, including a meta-analysis of 9 randomized controlled trials that compared use of a digital intervention for depression in young people to a no-intervention control group, and 6 comparing the intervention to an active control condition. A thematic analysis and narrative synthesis of 41 studies was also performed.

Results: The pooled effect size of digital mental health interventions on depression in comparison to a no-intervention control was small (Cohen’s d = 0.33, 95% CI 0.11 to 0.55), while the pooled effect size of studies comparing an intervention group to an active control showed no significant differences (Cohen’s d = 0.14, 95% CI -.04 to 0.31). Pooled effect sizes were higher when supervision was involved (studies with no-intervention controls: Cohen’s d = 0.52, 95% CI 0.23 to 0.80; studies with active control: Cohen’s d = 0.49, 95% CI -0.11, 1.01). Engagement and adherence rates were low. Qualitative analysis revealed that users liked interventions with a game-like feel and relatable, interactive content. Educational materials were perceived as boring, and users were put off by non-appealing interfaces and technical glitches.

Conclusions: Digital interventions work better than no intervention to improve depression in young people when results of different studies are pooled together. However, these interventions may only be of clinical significance when use is highly supervised. Digital interventions do not work better than active alternatives regardless of the level of support. Future interventions need to move beyond the use of digital educational materials, considering other ways to attract and engage young people and to ensure relevance and appeal.

Introduction

In Australia, approximately 8% of young people between 11–17 years of age meet the DSM criteria for major depressive disorder (MDD), while about 20% report high levels of psychological distress (1). The rates of MDD may be as high as 11% in youths in the U.S. (2). In fact, suicide is the second leading cause of death among 15–29-year-olds globally (3). In addition, depression is highly under-diagnosed and thousands who fall outside these statistics experience its debilitating effects on functioning at an important developmental stage. Depression and other mental illnesses affect the social and intellectual development of young people, reducing engagement with education, and if untreated, can become lifelong disabilities (4).

Despite the importance of addressing mental illness early only 20–40% of youths in need in Australia (1) and 25% of youths in the U.K. receive professional help (5). This low engagement with mental health services appears to occur for a variety of reasons: the lack of motivation inherent in conditions such as depression (6), low rates of mental health literacy (7), and the stigma, discrimination and embarrassment surrounding mental illness (8). Young people are also still developing skills in executive functioning such as self-monitoring and organization, which are necessary to identify a mental health problem and obtain support (9).

Although few young people seek professional help, during episodes of depression consumption of media such as music, internet, and television increases (10). Thus Digital Mental Health Interventions (DMHIs) are increasingly of interest as a solution to the low help-seeking and uptake rates of professional mental health services. Studies tend to support the effectiveness of self-help mental health programs whether digital or otherwise, which can be as effective if not more effective than face-to-face delivery (11). Numerous studies have demonstrated the usefulness of web-based programs (12, 13). Young people report feeling more comfortable discussing sensitive and personal issues in the relative anonymity of an online context and use the internet as a major source of mental health information (14, 15). Mobile “apps” are proving particularly useful for administering DMHIs because of the widespread ownership of mobile phones, with the majority of young people in the U.S. reporting almost constant usage of smartphones (16). Several reviews of smartphone applications for mental health across age groups have reported positive benefits (13, 17).

Notably, however, many apps for depression and anxiety that are currently available are not evidence-based and may thus actually be harmful to people with mental illness (18). Even among those purporting to be drawn from evidence-based therapies such as cognitive behavioural therapy, only a small percentage actually contain the core principles of those therapeutic traditions (19). Furthermore, Hollis and colleagues (20) in their meta-review reported that while there is some evidence in support of the effectiveness of DMHIs for depression and anxiety in young people, studies are methodologically limited making it difficult to draw clear conclusions. Furthermore, they suggest the need for identification of the components that make DMHIs effective such as human interaction.

In fact, human interaction has been identified as an important factor influencing effectiveness and engagement with DMHIs (21), but it may also detract from the cost-effectiveness of DMHIs in comparison to face-to-face treatment (20). Furthermore, other disadvantages to the inclusion of social features exist, such as unhelpful advice from peers, and the possibility that some youths may feel afraid to share personal problems even anonymously.

The aim of the current review, therefore, is to examine the literature about DMHIs to address mental health in young people. We have focused on depression and anxiety as these are among the most prevalent mental health conditions experienced by young people and often co-occur with many other disorders (22–24). Specifically we aimed to investigate:

• Do DMHIs reduce anxiety and depression in young people aged 12–25 compared to no intervention or an active control group?

• How effective are DMHIs in reducing anxiety and depression in young people when interaction with the intervention is unsupervised?

• What features and components of DMHIs are most liked or disliked by young users?

Method

Study Design

Given the focus in the current review on both effectiveness and engagement, it was expected that the literature reviewed could include both quantitative and qualitative data. Therefore, a mixed methods approach was selected. Mixed methods reviews attempt to combine looking at ‘what works’ with ‘how and why it works’, combining varied research methods in the analysis or in the types of studies reviewed (25). Mixed method reviews can provide a more holistic understanding than a meta-analysis alone since they are able to integrate a wider variety of studies and provide insights into mechanisms and processes.

Identification and Selection of Studies

A systematic search was conducted in PsychInfo, PubMed, Proquest, and Web of Science using the following terms in the title, abstract and subject descriptors: “mobile”, “application”, “smartphone”, “mobile phone”, “cell phone”, “text message”, “internet-administered therapy”, “computer-aided therapy”, “online” AND “depression”, “anxiety” AND “youth”, “young person”, “adolescent”. The initial search returned 4,828 articles. With duplicates deleted this number was reduced to 3,352.

Identified references were screened according to the following inclusion criteria: (1) participants aged 12 to 25, (ii) interventions targeting depression or anxiety, (iii) interventions delivered by computer, on smartphones, or online, (iv) studies published between 2007 and 2017. We also excluded reviews, opinion, or discussion pieces and unpublished works. While quality assessment was part of the review process, studies were not excluded on the basis of study type or quality since the mixed methods approach taken allowed the inclusion of varied methodologies.

After title screening by 1 researcher 184 abstracts were uploaded to Covidence, an online platform for conducting systematic reviews (http://www.covidence.org). These were scrutinised by 2 researchers. Once agreement was reached on eligibility, 68 articles remained. Full text appraisals were then conducted by 2 researchers and a further 27 articles not meeting the inclusion criteria were excluded, leaving 41 in the review (Figure 1 and Supplementary Materials).

Figure 1

Quality Assessment and Data Extraction

Two researchers independently conducted the data extraction. Information about the characteristics of the studies, participants, interventions, and final outcomes were entered into a pre-established template in Covidence. Qualitative data from the studies relating to usability and appeal of the interventions was also extracted. The quality of studies was assessed using the Joanna Briggs Institute’s (26) critical appraisal tools and the CONSORT-EHEALTH Checklist (V.1.6.1) (27). Studies were considered methodologically sound if they had a matched control group, pre-post data, and randomization, and were only included in the meta-analyses if they met these criteria. However, since this is a mixed-methods review, studies not meeting the criteria for inclusion in the meta-analysis were retained in the study to form part of the narrative synthesis. Risk of bias was assessed using the standard Cochrane Risk of Bias tool (28) in Covidence, which poses questions about aspects of trial design, conduct, and reporting which the user rates as ‘Low’ or ‘High’ risk of bias for each study. Single cohort studies and RCTs were assessed using different indicators of bias under the 5 categories of possible bias as outlined in the Cochrane Handbook of Systematic Reviews (29), and a overall assessment of bias as ‘Low’ or ‘High’ given for each study. Where not enough details were reported in the study to assess risk of bias, this was labeled ‘Unclear’. Again, however, studies with some risk of bias were not excluded from the analyses but sub-group comparisons were conducted to assess whether results differed depending on bias.

Analysis

Meta-Analysis

Due to a lack of comparable randomized controlled trials (RCTs) for anxiety, the meta-analysis focused on DMHIs targeting depression. For each RCT of a DMHI for depression the effect size indicating the difference between the intervention group and the control group at post-test was calculated in OpenMeta Analyst (30). Effect size calculations (Cohen’s d or standardized mean differences) were conducted using the means and standard deviations of post-test scores on instruments measuring symptoms of depression such as the depression subscale of the Depression, Anxiety, Stress, Scale (DASS, 31) or the Centre for Epidemiological Studies Depression Scale (CES-D, 32). Scores on these measures immediately after the study were used rather than longitudinal follow-up scores due to a lack of consistent reporting across the studies. Sample sizes used were the number of participants with complete data in each group rather than intention-to-treat numbers. If effect sizes could not be calculated due to a lack of information, the study was excluded from the effectiveness analysis. Pooled mean effect sizes were calculated using the random effects model as considerable heterogeneity between studies was expected. These were calculated separately for studies using a no-intervention control group and those using an active control group. Subgroup analyses were conducted to investigate the impact of other variables on effectiveness by testing for significant differences between subgroups using a fixed effects model. In particular a sub-group analysis according to risk of bias was conducted to assess how study design contributed to outcomes. Since one area of interest in this review was to investigate whether DMHIs are useful for the high percentages of young people not obtaining professional help, we also conducted a sub-group analysis according to the level of supervision or interaction involved in the treatment condition.

Narrative Synthesis

All studies including those not in the meta-analysis were evaluated using a narrative synthesis model (33). Qualitative data included both results of interviews with participants, and descriptions of survey results relating to the appeal of DMHIs. For example, some studies reported direct comments from participants about their experiences using the DMHI, while others reported levels of agreement with statements about appeal and quality. These data were coded by 2 researchers using standard techniques for thematic analysis to generate an understanding of features that are appealing and aspects that promoted engagement and adherence (34). Codes were grouped to detect patterns, and themes were identified and defined. Once consensus was reached the lead author prepared a narrative analysis, which was checked independently by the other authors.

Results

Study and Participant Characteristics

The 41 studies included 11 from Australia, 10 from the U.S., 12 from other English speaking countries, 5 from Northern Europe, 2 from Asia, and one from South America. The majority of studies were RCTs (n = 27) (Table 1), while 13 were single cohort studies (including 4 with pre-post designs), and one used a case study methodology. Most of the studies used participants recruited from educational institutions (n = 20), 9 were conducted in mental health care settings, 4 with the general community, 4 in primary care settings, 2 in hospitals, and one in a youth organisation. One study did not report recruitment methods or participant information in enough detail to determine the setting.

Studies included participants with no specific mental health symptoms at baseline (n = 12), some with varying levels of depression (mild to moderate n = 7, moderate to severe n = 3, severe = 1, all levels n = 6), others with diagnosed MDD (n = 2) or suicidal risk (n = 5). Participants with varying levels of anxiety were also a focus in some studies (mild n = 1, mild-mod n = 2, mod to severe n = 1, severe n = 1, all levels n = 1). Two studies looked at people with a variety of mental illnesses, and another focused on mood issues in people with traumatic brain injury (Table 1).

Intervention Characteristics

Overall, 32 different DMHIs were investigated across the 41 papers (Table 2). Several DMHIs were evaluated in multiple studies including Bite Back (n = 2), CATCH-IT (n = 2), Master Your Mood (n = 2), MoodGym (n = 3), Reframe-IT (n = 3), SPARX (n = 2), and MEMO (n = 2). Several of these papers reported results from the same data sets (62, 63, 73, 74), but reported on different aspects of the study and therefore both papers were included in the review. Most of the DMHIs drew on established therapeutic models, primarily Cognitive Behavioural Therapy (CBT) (n = 28) or a combination of CBT with other models.

The technologies utilized in the various DMHIs included some phone-based interventions such as text-messages (n = 4) and smartphone applications containing assessment tools and/or psychoeducational materials (n = 3). The majority of DMHIs were web-based (n = 30), including many with online modules, learning materials or activities (n = 24), group chats or courses (n = 2), online forums (n = 2), and online chat facilities with a mental health professional (n = 2). Others were computer-based but not online, including games (n = 5) and psychoeducational computer programs (n = 2).

Many DMHIs included learning modules (n = 18), interactive learning activities (n = 6), psychoeducational materials in a variety of formats including text and video (n = 7), or game-based learning activities (n = 4). Additional features included regular inspiring messages (n = 1), challenges (n = 3), mood tracking (n = 3), or diary/journals (n = 2). Four studies included DMHIs with an accompanying workbook for participants or their parents. Only 11 of the studies reviewed included DMHIs that were entirely self-help and were completed in the participant’s own time. The rest of the studies involved interaction with a mental health professional or completion of the intervention in some kind of supervised setting. Two of these were completed in hospitals, one with some minimal supervision from a health professional (36). Others were completed in a school setting (n = 10). Some of the studies completed at school involved a high level of supervision (n = 5) such as in studies where the intervention was presented by the school wellbeing staff (47), the classroom teacher (39), in the presence of the research team (62, 63), or in class groups (65). Other school-based studies involved lower levels of interaction with a therapist or the research team (n = 3) such as moderation of an online discussion board by a therapist (38), initial training completed at school but the intervention otherwise used in the students’ own time (44), and where the intervention was accessed at school with minimal supervision by the school counselor (68). Studies outside of school settings included DMHIs that could be completed at home in the participant’s own time, but included interactions with a therapist such as sending reminders or text messages (n = 4), or participating in online group courses or chats (n = 9).

Effectiveness of DMHIs

The effectiveness of various DMHIs in treating symptoms of depression was compared to a control group in 15 studies (Table 3). Nine of these studies compared DMHIs to no intervention (a waitlist control group), while five of the studies compared DMHIs to an active control in which some alternative online materials were used, including one with some psycho-educational content (68), and one contained a Treatment As Usual (TAU) comparison group in which face-to-face counseling was offered (56). Since the TAU group in this case included active treatment it was combined in analysis with the active control groups. The pooled effect size of studies comparing the intervention group to a no-intervention group (n = 9) was 0.33 (95% CI 0.11 to 0.55) (Figure 2), suggesting that DMHIs have a small effect size when compared to a no intervention control group, while the pooled effect size of studies comparing the intervention group to an active control group (n = 6) was 0.14 (95% CI -0.04 to 0.31). Thus this review did not find a difference in outcomes between DMHIs and active controls, including a mixture of usual care for depression and non-depression specific interventions (Figure 3). Heterogeneity was relatively high (I^{2 =} 70%) and statistically significant (p < .001). Two studies had negative effect sizes indicating that the control group had lower depression scores at post-test than the intervention group (59, 74). Reid and colleagues (59) evaluated the effectiveness in comparison to a waitlist control group of a smartphone application that allowed self-assessment on 8 domains of mood and functioning, referring this information to general practitioners for medical review. No significant effect on depression was found at post-test, but increased emotional self-awareness was reported. Whittaker and colleagues (74) similarly used a phone-based approach, delivering multimedia messages based on CBT and comparing this to use of similar multimedia messages with no focus on depression. The authors concluded that there was no evidence of benefits superior to the active comparison program with content about healthy behaviours.

Figure 2

Figure 3

Sub-group analyses were conducted to investigate the effect of therapist interactions and study completion settings on the outcomes in the 9 studies that compared the intervention to no intervention (Table 3), and the 6 studies comparing the intervention to an active control group. Studies were categorised as having High levels of human interaction (H) if they involved direct contact with a therapist or were completed in supervised settings such as a lab, clinic or school (47, 51, 55, 65, 66, 68, 70). They were categorised as Low interaction (L) (43, 49, 53) if they had some limited interaction such as regular emails, text messages, or optional opportunities to contact a therapist, or No interaction (N) (52, 54, 56, 59, 74) if they did not involve any interaction with a mental health professional and were completed unsupervised in personal time.

For studies comparing the DMHI to no intervention, the pooled effect size was smallest in the No interaction group (d = 0.04), and also small in the Low interaction group (d = 0.16), while the High interaction group returned a medium effect size (d = 0.52) (Figure 4). This indicates that DMHIs were mostly effective when they involved high levels of human interaction. The DMHIs in the No interaction group that did have a positive effect size were highly interactive, containing multimedia lessons (52), interactive online exercises (54), and game-based challenges and puzzles aiming to improve mental health literacy (56). Levin and colleagues (52) did not include direct conversations with mental health professionals. However, the system did send automatically generated emails that were customized based on participants’ earlier input. This could have given the illusion of human interaction, thereby increasing the effectiveness of the program despite there being no direct personal communication. No significant differences were found according to intervention type or severity of symptoms at baseline.

Figure 4

Similarly, for the studies with active comparison groups, the pooled effect size was again smallest in the No interaction group (d = 0.08), and also small in the Low interaction group (d = 0.21), while the High interaction group returned a medium effect size (d = 0.49) (Figure 5). Both of the studies in the High interaction group were completed in school classroom settings (55, 68). Thus, across both active and no intervention control groups, effect sizes reached a moderate size only when there was a high level of therapist interaction or supervision in the study design.

Figure 5

Risk of Bias

The proportion of studies at high/unclear risk of bias was: 34% selection bias (e.g. randomization or allocation concealment), 63% detection bias (e.g. blinding of outcome assessment), 41% attrition bias, and 31% selective reporting (Table 4). Overall 76% of the studies included in the review as a whole, and all but 4 studies (43, 66, 68, 74) included in the meta-analysis had some risk of bias. Sub-group analyses were performed on the 15 studies included in the meta-analyses according to type of bias. Significant differences were found for selection bias with the low risk of bias group having a lower pooled effect size (d = 0.01, 95% CI -0.05 to 0.24) than the unclear or high-risk group (d = 0.44, 95% CI 0.22 to 0.69). Studies with low risk of detection bias also had a lower pooled effect size (d = .09, 95% CI -0.08 to 0.26) than the unclear or high-risk group (d = .40, 95% CI 0.18 to 0.62).

Attrition, Adherence, and Engagement

Attrition rates were defined as the number of participants who completed the study as a percentage of the participants who commenced the intervention (Table 5).Information about adherence and engagement (how much those who completed the study engaged with the intervention), tended to be reported differently across papers. For example, some papers reported module completion rates (36). Others reported time spent on a website (43). Overall 16 (39%) of the studies had attrition rates over 20%, the level broadly considered indicative of possible attrition bias (76) (Table 5). In several of these studies, while attrition rates were high, they were equal between groups (for e.g. 50) suggesting that drop out rates related more to recruitment methods than to non-engagement. However, this was not always the case and in many studies, even among those with low study attrition, engagement tended to be low, with participants completing less than half of the intervention components (36, 45, 49, 51, 53, 73). The majority of these studies were again ones that involved completion in one’s own time. For example, Ip and colleagues (2016) reported low drop out rates and a small effect size. However, participants only completed roughly three of 10 modules and spent about 39 minutes on the website over 4 months, suggesting relatively low engagement.

Qualitative data from all papers were categorized according to features of the DMHIs that were liked by participants, features that were disliked (Table 6), and features predicting adherence. There were four key categories of data in relation to liked features. The first category related to social support. Several studies reported that participants had found it useful to be in contact with professionals. Participants in one study who had access to a trained supporter in addition to regular text messages reported that they “liked talking to someone who was friendly” (35, p. 101). Participants in one study of a game-based CBT intervention called Pesky gNATs (41) reported liking the fact that doing it on a computer was “not as full on as face-to-face” (p. 15). However this was not perceived by all to be preferable to in-person contact. For example, two studies that evaluated an interactive CBT-based fantasy game called SPARX, both found that some participants preferred face-to-face support from a therapist (36, 56). Similarly, some participants in an online CBT-based group course reported a preference for a face-to-face version of the course (45).

For other participants, it was the opportunity to connect with peers who were experiencing similar difficulties that was helpful. Gerrits and colleagues (56) reported that participants “found chatting to be a pleasant and positive way to talk about being down and their feelings of depression” (p. 6). Similarly, Horgan and colleagues (48) studied the impact of an online forum and reported that participants found it “good to say what was going on aloud (albeit in writing)” (p. 87). One participant in the same study stated: “Its about empathy and the realization that you’re not alone. That others are feeling the same way you do and are having trouble coping” (p. 87).

For some participants the primary attraction of DMHIs were their online or computer-based nature. In contrast to users noted above who reported a preference for face-to-face contact, for many users a key benefit of DMHIs was the privacy they afforded. One participant in an online self-help program, Crystal, stated: “You can kind of do it in a secluded area where nobody is watching you … the privacy is kind of like a really big appeal” (37, p. 28). In one study of an online peer-support website, anonymity allowed participants to share details that they had never shared before, and in fact had “put a lot of effort into hiding” (48, p. 87). DMHIs also had the advantage of fitting into the daily routines of users, connecting with current interests (40), and helping “to bring back a sense of normality” (36, p. 290). A participant named Rob stated: “Most teens are always on the internet … while you’re on say Facebook or something, you can just open up another tab” (37, p. 28) It can also be accessed from a variety of locations such as school, home or in a clinic and participants could “learn by myself and at my own pace” (56, p. 7). Participants reported that it was “fun to be able to do it on a computer” (41, p. 13).

Other participants commented on particularly useful content. Participants reported that the DMHIs “showed me things I didn’t know” (56, p. 7), and helped them learn more about mental health (58). Appreciation was expressed for content that helped participants to learn specific techniques such as problem solving and anger control (71), or challenging negative thoughts (37).

Another major category of the data related to the look and feel of the DMHIs. Participants preferred situations, characters, or avatars that were relatable. For example, participants reported that it was helpful when the focus was “situations any teenager goes through” such as school and interpersonal relationships (37, p. 27). Conversely, several studies reported that drop-outs occurred when the content “did not seem relevant for them” (54, p. 7). Other features that participants reported liking included interactive activities (58, 62), and video components (62, 73). Similar comments were made in relation to DMHIs with a game-like feel. Participants stated that this made engaging with the DMHIs fun (36, 41). It was also important to the users that DMHIs were aesthetically appealing and easy to use and navigate.

One of the most prominent features that participants reported disliking was the educational content of many DMHIs. For example in their case study of use of a smartphone app for anxiety, Whiteside and colleagues (72) reported that the participant “appeared less engaged and interested in the background educational content” (p. 86). Multiple other studies reported similar comments by participants, particularly non-completers (36, 45, 54, 56, 58, 73). Educational modules were viewed as too long (37), “tough and sometimes quite tiring” (45, p. 6), “tedious and laborious” (38, p. 6). Some participants argued that it would be more convenient to be able to tailor modules to one’s own needs: “I didn’t like that you couldn’t skip out of something if you already understood the concept”(58, p. 163). Burckhardt and colleagues (2015) suggested that more structured settings and dose effects may have contributed to their negative results, since other studies have found that the number of activities participants are required to do can reach a saturation point (77).

A noteworthy point was that numerous participants reported that the DMHIs often felt too juvenile or patronising. Participants did not enjoy using DMHIs that seemed like they were designed for younger children (41, 68). One participant suggested: “make it more grown up” (41, p. 14).

Technical glitches and difficulties navigating sites were also frequently cited as reasons for low adherence and engagement (37, 54, 68, 73). Participants stated that DMHIs should be improved to make them “comparable to commercially available games” (36, p. 290). Others reported disliking particular aesthetic features such as the colour scheme and a lack of variety of icons, cartoons, and diagrams (37).

Factors predicting adherence. Only four studies reported predictors of adherence to the DMHIs. Neil and colleagues (57) compared a school-based completion setting for MoodGYm to a community setting, finding that a school-based setting predicted greater adherence. Gender was also a consistent predictor of adherence, with females being more likely to complete compared to males (51, 57, 70). Mental health also played a role, with higher pre-test scores in depression (57), a longer history of mood disorders (51), or low scores in anxiety at pre-test (70) predicting greater adherence.

Discussions

This review aimed to determine the types of DMHIs that are effective in treating depression and anxiety in young people and the components of these interventions most associated with positive outcomes and engagement. Overall, studies in relation to depression demonstrated a small effect size in favour of DMHIs when interventions were compared to no intervention. While this might not always reflect a clinically significant level of change, it suggests that such DMHIs may be of value in the context of public health and preventative interventions. On the other hand, studies comparing DMHIs to active control conditions were not effective. In fact, in two studies the control group actually had lower depression levels at post-test than the intervention group. Both studies included phone based interventions: an app that referred users for medical review, and a program of multimedia mobile phone messages. However, given that there was a risk of bias in many studies included in the meta-analyses, these results should be interpreted with caution. In fact, the two studies in the meta-analysis that reported negative effect sizes were two of the only three studies assessed as having low risk of bias. Studies which did not involve blinding of either group allocation or of outcome assessment tended to have higher effect sizes than studies with low risk of bias, indicating that methodological limitations of the studies reviewed likely inflated the larger effect sizes.

Of further importance in our findings was the fact that only DMHIs involving regular interactions with a therapist or that were completed in a supervised setting reached a moderate effect size in comparison to a no-intervention control group, while DMHIs that involved educational programs completed in the participant’s own time were not found to be effective in this study. This suggests that currently available DMHIs may not be effective in causing clinically detectable levels of change unless they involve a high level of supervised use or therapist involvement. These results reflect the overarching significance of human interaction in psychological interventions (78). However, the preference among some participants for human contact revealed in this review existed in tension with the need for privacy and anonymity, suggesting that there is a need for more effective design of DMHIs to fill a gap that traditional face-to-face therapies do not.

Despite this, adherence and engagement rates tended to be low in many studies particularly those where interventions were completed in their own time. This reinforces the idea that many DMHIs are most likely to be useful for people already receiving mental health support or at least those not averse to doing so. However, DMHIs completed in settings such as schools, labs, or clinics cannot reliably indicate their effectiveness in reaching young people outside of these settings. Where DMHIs were completed in their own time or did not include interaction with a mental health professional, effectiveness was much reduced. Nevertheless, for many users it is the anonymity and privacy afforded by the online context that holds the greatest appeal. Therefore, there is a need to balance the competing advantages of anonymity and social support. Other studies have similarly concluded that social networking features in DMHIs are a “gamble” due to the potential for both negative and positive effects (79).

These results indicate two distinct needs in DMHI development. Firstly, a dire need exists to increase the appeal of DMHIs so as to reach the 80% of young people who are not already obtaining professional help. These young people may not understand that their symptoms indicate the need for mental health assistance. Other barriers such as a lack of energy or motivation to engage with complex tasks, or a fear of the stigma of mental illness may prevent them from accessing even DMHIs where such are overtly about mental health or are educational in nature. Research indicates that young men are particularly unlikely to receive professional help (80), which makes the findings in this review that males are less likely to engage with DMHIs than females particularly disturbing. The development of DMHIs that build on the existing interests of young people in non-confronting ways may be of more appeal to this group. DMHIs that particularly cater to the interests of young men are especially needed. Only highly interactive DMHIs involving multimedia materials or game-based activities were successful in studies with low levels of human interaction, suggesting that these types of features are highly appealing.

Secondly, for DMHIs designed to help young people who are already seeking or willing to seek professional assistance but prefer to do so in the relative anonymity of digital settings, it is clear that the help of schools and mental health professionals is a crucial part of the roll out of such interventions. Similarly, some scholars have recommended a model of ‘supportive accountability’, in which accountability to a supporter or coach can enhance adherence to eHealth interventions (81).

The need for further refinement to available DMHIs was confirmed by our qualitative analysis which revealed factors involved in high drop out rates and low engagement rates in numerous studies. In general, participants liked the online or computer based formats and game-like feel of some DMHIs, particularly when the content was interactive, had relatable situations or characters, and had appealing aesthetic features. However, feedback by study participants or people who withdrew from the studies referred to the boring nature and hard work involved in the online learning modules. Participants found DMHIs with non-appealing interfaces, frequent technical glitches, or material that seemed too juvenile to be off-putting. Other studies have similarly found that making DMHIs easy to use and to navigate is important to users (82), and in fact criteria like this are commonly used to evaluate usability and appeal (83).

Although some participants seemed to appreciate the opportunity to learn various psychological skills to improve their wellbeing, many disliked the high educational focus of the interventions, the fact that they seemed designed for children much younger, and that they did not match commercially available programs in quality. This highlights the need for future DMHIs to consider the opinions of young people closely in their design. There is a need for feedback from young people and co-design methods to ensure that both content and aesthetics are appealing to the target audience. There is also a need for DMHI developers to ensure that materials are not ‘dumbed down’ and that they are presented in a way that does not feel like hard work but that builds on the natural interests of young people. This again presents a challenge for developers, to balance the need for simplicity of use with age-appropriate content. This is especially important for DMHIs designed to address depression and anxiety since these conditions are associated with both a lack of motivation (84) and impaired concentration (85), making such users a particular challenge to engage.

The current study was limited by the search terms used. Future reviews should also include search terms such as “internet-delivered”, “computer”, or “computerised”, since this could have picked up a broader range of studies in the current review. Nevertheless, this review demonstrates that while somewhat effective for those who use them, DMHIs fail to appeal to a large proportion of young people. In fact, when compared to active comparison groups including online materials with no psycho-educational content, DMHIs had only minimally better effects. As yet there seems to be a dearth of DMHIs that are likely to attract the large numbers of young people with mental illness who are not already open to receiving professional help. There is thus a need for urgent attention to developing high quality DMHIs that address the weaknesses and focus on the strengths identified, to help young people currently in the shadows to access appealing and accessible tools for managing their mental health. There is also a need for methodologically robust double-blinded RCTs to be designed to provide more stringent testing of the effectiveness of such interventions.

References

• 1

  LawrenceDJohnsonSHafekostJBoterhoven De HannKSawyerMAinleyJet al. The mental health of children and adolescents. Report on the second Australian Child and Adolescent Survey of Mental Health and Wellbeing. Canberra: Department of Health (2015).

  • Google Scholar

• 2

  MerikangasKRHeJBursteinMSwansonSAAvenevoliSCuiLet al. Lifetime prevalence of mental disorders in U. S. adolescents: results from the national comorbidity survey replication - adolescent supplement (NCS-A). J Am Acad Child AdolescPsychiatry (2010) 49(10):980–9. doi: 10.1016/j.jaac.2010.05.017

  • CrossRef
  • Google Scholar

• 3

  WHO. Depression and other common mental disorders: Global health estimates (2017). Available from: https://apps.who.int/iris/bitstream/handle/10665/254610/WHO-MSD-MER-2017.2-eng.pdf?sequence=1.

  • Google Scholar

• 4

  MeadeTDowswellE. Adolescents’ health-related quality of life (HRQoL) changes over time: a three year longitudinal study. Health Qual Life Outcomes (2016) 14(14). doi: 10.1186/s12955-016-0415-9

  • CrossRef
  • Google Scholar

• 5

  DaviesS. Annual report of the chief medical officer 2013, public mental health priorities: Inesting in the evidence. London, UK: Department of Health (2014).

  • Google Scholar

• 6

  HershenbergRSatterthwaiteTDDaldalAKatchmarNMooreTMKableJWet al. Dimiinished effort on a progressive ration task in both unipolar and bipolar depression. J Affective Disord (2016) 196:97–100. doi: 10.1016/j.jad.2016.02.003

  • CrossRef
  • Google Scholar

• 7

  ColesMERavidAGibbBGeorge-DennDBronsteinLRMcLeodS. Adolescent mental health literacy: young people’s knowledge of depression and social anxiety disorder. J Adolesc Health. (2016) 58(1):57–62. doi: 10.1016/j.jadohealth.2015.09.017

  • CrossRef
  • Google Scholar

• 8

  ClementSSchaumanOGrahamTMaggioniF. What is the impact of mental health-related stigma on help-seeking? A systematic review of quantitative and qualitative studies. Psychol Med (2015) 45(1):11–27. doi: 10.1017/S0033291714000129

  • CrossRef
  • Google Scholar

• 9

  AndersonVAAndersonPNorthamEJacobsRCatroppaC. Development of executive functions through late childhood and adolescence in an Australian sample. Dev Neuropsychol (2001) 20(1):385–406. doi: 10.1207/S15326942DN2001_5

  • CrossRef
  • Google Scholar

• 10

  Dillman CarpentierFRBrownJDBertocciMSilkJSForbesEEDahlRC. Sad kids, sad media? Applying Mood Management Theory to depressed adolescents’ use of media. Media Psychology (2008) 11:143–66. doi: 10.1080/15213260701834484

  • CrossRef
  • Google Scholar

• 11

  CuijpersPDonkerTvan StratenA. Is guided self-help as effective as face-to-face psychotherapy for depression and anxiety disorders? A systematic review and meta-analysis of comparative outcome studies. Psychol Med (2010) 40(12):1943–57. doi: 10.1017/S0033291710000772

  • CrossRef
  • Google Scholar

• 12

  AndrewsGCuijpersPCraskeMGMcEvoyPTitovN. Computer therapy for the anxiety and depressive disorders is effective, acceptable and practical health care: a meta-analysis. PLOS ONE (2010) 5(10):e13196. doi: 10.1371/journal.pone.0013196

  • CrossRef
  • Google Scholar

• 13

  DonkerTPetrieKProudfootJClarkeJBirchM-RChristensenH. Smartphones for smarter delivery of mental health programs: a systematic review. J Med Internet Res (2013) 15(11):239–51. doi: 10.2196/jmir.2791

  • CrossRef
  • Google Scholar

• 14

  BurnsJMDavenportTADurkinLALuscombeGMHickieIB. The internet as a setting for mental health service utilisation by young people. Med J of Aus (2010) 192(11 Suppl):s22–6. doi: 10.5694/j.1326-5377.2010.tb03688.x

  • CrossRef
  • Google Scholar

• 15

  WebbMBurnsJMCollinP. Providing online support for young people with mental health difficulties: challenges and opportunities explored. Early Intervention Psychiatry (2008) 2(2):108–13. doi: 10.1111/j.1751-7893.2008.00066.x

  • CrossRef
  • Google Scholar

• 16

  LenhartA. Teens, Social Media & Technology Overview. Washington DC: Pew Internet and American Life Project (2015).

  • Google Scholar

• 17

  FirthJTorousJNicholasJCarneyRRosenbaumSSarrisJ. Can smartphone mental health interventions reduce symptoms of anxiety? A meta-analysis of randomized controlled trials. J Affective Disord (2017) 218:15–22. doi: 10.1016/j.jad.2017.04.046

  • CrossRef
  • Google Scholar

• 18

  NicholasJBoydellKChristensenH. mHealth in psychiatry: time for methodological change. Evid Based Mental Health (2016) 19(2):33–4. doi: 10.1136/eb-2015-102278

  • CrossRef
  • Google Scholar

• 19

  HuguetARaoSMcGrathPJWozneyLWheatonMConrodJet al. A systematic review of cognitive behavioral therapy and behavioral activation apps for depression. PloS one (2016) 11(5):e0154248. doi: 10.1371/journal.pone.0154248

  • CrossRef
  • Google Scholar

• 20

  HollisCFalconerCJMartinJLWhittingtonCStocktonSGlazebrookCet al. Annual research review: Digital health interventions for children and young people with mental health problems - a systematic review. J Child Psychol Psychiatry (2017) 58(4):474–503. doi: 10.1111/jcpp.12663

  • CrossRef
  • Google Scholar

• 21

  RickwoodDBradfordS. The role of self-help in the treatment of mild anxiety disorders in young people: an evidence-based review. Psychol Res Behav Manage (2012) 5:25–36. doi: 10.2147/PRBM.S23357

  • CrossRef
  • Google Scholar

• 22

  PedrelliPNyerMZulaufCWilensT. College students: mental health problems and treatment considerations. Acad. Psychiatry (2014) 39(5):503–11. doi: 10.1007/s40596-014-0205-9

  • CrossRef
  • Google Scholar

• 23

  OrmelJRavenDvan OortFHartmanCAReijneveldSAVeenstraRet al. Mental health in Dutch adolescents: a TRAILS report on prevalence, severity, age of onset, continuity and co-morbidity of DSM disorders. Psychol Med (2015) 45:345–60. doi: 10.1017/S0033291714001469

  • CrossRef
  • Google Scholar

• 24

  NaughtonMFAMayberyDJGoodyearM. Prevalence of mental illness within families in a regional child-focussed mental health service. Int J Mental Health Nurs (2018) 27:901–10. doi: 10.1111/inm.12386

  • CrossRef
  • Google Scholar

• 25

  GrantMBoothA. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Inf Libraries J (2009) 26(2):91–108. doi: 10.1111/j.1471-1842.2009.00848.x

  • CrossRef
  • Google Scholar

• 26

  Joanna Briggs Institute. Critical Appraisal toolshttp://joannabriggs.org/research/critical-appraisal-tools.html (2018). [Available from: http://joannabriggs.org/research/critical-appraisal-tools.html.

  • Google Scholar

• 27

  EysenbachG. CONSORT E-HEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions. J Med Internet Res (2011) 13(4):e126. doi: 10.2196/jmir.1923

  • CrossRef
  • Google Scholar

• 28

  HigginsJAltmanDGotzschePCJuniPMoherDOxmanAet al. The cochrane collaboration’s tool for assessing risk of bias in randomised trials. BMJ (2011) 343:d5928. doi: 10.1136/bmj.d5928

  • CrossRef
  • Google Scholar

• 29

  HigginsJGreenS, eds. Cochrance Handbook for Systematic Review of Interventions. The cochrane collaboration (2011).

  • Google Scholar

• 30

  WallaceBCDahabrehIJTrikalinosTALauJTrowPSchmidCH. Closing the gap between methodologists and end-users: R as a computational back-end. J Stat Software (2012) 49(5)1–15. doi: 10.18637/jss.v049.i05

  • CrossRef
  • Google Scholar

• 31

  HenryJDCrawfordJR. The short-form version of the depression anxiety stress Scales (DASS-21): construct validity and normative data in a large non-clinical sample. Br J Clin Psychology. (2005) 44(2):227–39. doi: 10.1348/014466505X29657

  • CrossRef
  • Google Scholar

• 32

  RadloffLS. The CES-D scale: a self-report depression scale for research in the general population. Appl Psychol Meas (1977) 3:385–401. doi: 10.1177/014662167700100306

  • CrossRef
  • Google Scholar

• 33

  Dixon-WoodsMCaversDAgarwalSAnnandaleEArthurTHarveyJet al. Conducting a critical interpretive review of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol (2006) 6:(35). doi: 10.1186/1471-2288-6-35

  • CrossRef
  • Google Scholar

• 34

  BraunVClarkeV. Using thematic analysis in psychology. Qual Res Psychol (2006) 3:77–101. doi: 10.1191/1478088706qp063oa

  • CrossRef
  • Google Scholar

• 35

  AnstissDDaviesA. ‘Reach Out, Rise Up’: the efficacy of text messaging in an intervention package for anxiety and depression severity in young people. Child Youth Serv Rev (2015) 58(C):99–103. doi: 10.1016/j.childyouth.2015.09.011

  • CrossRef
  • Google Scholar

• 36

  BobierCStasiakKMountfordHMerrySMoorS. When ‘e’ therapy enters the hospital: examination of the feasibility and acceptability of SPARX (a cCBT programme) in an adolescent inpatient unit. Adv Mental Health (2013) 11(3):286–92. doi: 10.5172/jamh.2013.11.3.286

  • CrossRef
  • Google Scholar

• 37

  BradleyKLRobinsonLMBrannenCL. Adolescent help-seeking for psychological distress, depression, and anxiety using an Internet program. Int J Mental Health Promotion. (2012) 14(1):23–34. doi: 10.1080/14623730.2012.665337

  • CrossRef
  • Google Scholar

• 38

  BurckhardtRManicavasagarVBatterhamPJMillerLMTalbotELumA. A web-based adolescent positive psychology program in schools: randomized controlled trial. J Med Internet Res (2015) 17(7):e187. doi: 10.2196/jmir.4329

  • CrossRef
  • Google Scholar

• 39

  CalearALChristensenHMackinnonAGriffithsKM. Adherence to the MoodGYM program: outcomes and predictors for an adolescent school-based population. (Report). J Affective Disord (2013) 147(1-3):338. doi: 10.1016/j.jad.2012.11.036

  • CrossRef
  • Google Scholar

• 40

  CarrascoAE. Acceptability of an adventure video game in the treatment of female adolescents with symptoms of depression. Res Psychother (2016) 19:10–8. doi: 10.4081/ripppo.2016.182

  • CrossRef
  • Google Scholar

• 41

  ChapmanRLoadesMO’ReillyGCoyleDPattersonMSalkovskisP. ‘Pesky gNATs’: investigating the feasibility of a novel computerized CBT intervention for adolescents with anxiety and/or depression in a Tier 3 CAMHS setting. tCBT (2016) 9:e35. doi: 10.1017/S1754470X16000222

  • CrossRef
  • Google Scholar

• 42

  ChenRYFeltesJRTzengWSLuZYPanMZhaoNet al. Phone-based interventions in adolescent psychiatry: a perspective and proof of concept pilot study with a focus on depression and autism. JMIR Res Protoc (2017) 6(6):e114. doi: 10.2196/resprot.7245

  • CrossRef
  • Google Scholar

• 43

  ClarkeGKelleherCHornbrookMDebarLDickersonJGullionC. Randomized effectiveness trial of an internet, pure self-help, cognitive behavioral intervention for depressive symptoms in young adults. Cognit Behav Ther (2009) 38(4):222–34. doi: 10.1080/16506070802675353

  • CrossRef
  • Google Scholar

• 44

  de VoogdELWiersRWZwitserRJSaleminkE. Emotional working memory training as an online intervention for adolescent anxiety and depression: a randomised controlled trial. Aust J Psychol (2016) 68:228–38. doi: 10.1111/ajpy.12134

  • CrossRef
  • Google Scholar

• 45

  GerritsRSVan Der ZandenRAPVisscherRFMConijnBP. Master your mood online: a preventive chat group intervention for adolescents. Aust J Adv Mental Health (2007) 6(3):152–62. doi: 10.5172/jamh.6.3.152

  • CrossRef
  • Google Scholar

• 46

  GladstoneTMarko-HolguinMHenryJFogelJDiehlAVan VoorheesBW. Understanding adolescent response to a technology-based depression prevention program. J Clin Child Adolesc Psychol (2014) 43(1):102–14. doi: 10.1080/15374416.2013.850697

  • CrossRef
  • Google Scholar

• 47

  HetrickSEYuenHPBaileyECoxGRTemplerKRiceSMet al. Internet-based cognitive behavioural therapy for young people with suicide-related behaviour (Reframe-IT): a randomised controlled trial. Evid Based Mental Health (2017) 20(3):76. doi: 10.1136/eb-2017-102719

  • CrossRef
  • Google Scholar

• 48

  HorganAMcCarthyGSweeneyJ. An evaluation of an online peer support forum for university students with depressive symptoms. Arch Psychiatr Nursing (2013) 27(2):84–9. doi: 10.1016/j.apnu.2012.12.005

  • CrossRef
  • Google Scholar

• 49

  IpPChimDChanKLLiTMHHoFKWVoorheesBWet al. Effectiveness of a culturally attuned Internet-based depression prevention program for Chinese adolescents: a randomized controlled trial. Depress Anxiety (2016) 33(12):1123–31. doi: 10.1002/da.22554

  • CrossRef
  • Google Scholar

• 50

  KingCAEisenbergDZhengKCzyzEKramerAHorwitzAet al.Online suicide risk screening and intervention with college students: a pilot randomized controlled tria. J Consult Clin Psych (2015). 83(3):630. doi: 10.1037/a0038805

  • CrossRef
  • Google Scholar

• 51

  KramerJConijnBOijevaarPRiperH. Effectiveness of a web-based solution-focused brief chat treatment for depressed adolescents and young adults: randomized controlled trial. J Med Internet Res (2014) 16(5):e141. doi: 10.2196/jmir.3261

  • CrossRef
  • Google Scholar

• 52

  LevinMEPistorelloJSeeleyJRHayesSC. Feasibility of a prototype web-based acceptance and commitment therapy prevention program for college students. J Am Coll Health (2014) 62(1):20–30. doi: 10.1080/07448481.2013.843533

  • CrossRef
  • Google Scholar

• 53

  LillevollKRVangbergHCBGriffithsKMWaterlooKEisemannMR. Uptake and adherence of a self-directed internet-based mental health intervention with tailored e-mail reminders in senior high schools in Norway. BMC Psychiatry (2014) 14(1):14–. doi: 10.1186/1471-244X-14-14

  • CrossRef
  • Google Scholar

• 54

  ManicavasagarVHorswoodDBurckhardtRLumAHadzi-PavlovicDParkerG. Feasibility and effectiveness of a web-based positive psychology program for youth mental health: randomized controlled trial. J Med Internet Res (2014) 16(6):e140. doi: 10.2196/jmir.3176

  • CrossRef
  • Google Scholar

• 55

  MelnykBMAmayaMSzalachaLAHoyingJTaylorTBowersoxK. Feasibility, acceptability, and preliminary effects of the COPE online cognitive-behavioral skill-building program on mental health outcomes and academic performance in freshmen college students: a randomized controlled pilot study. J Child Adoles Psychiatr Nurs (2015) 28(3):147–54. doi: 10.1111/jcap.12119

  • CrossRef
  • Google Scholar

• 56

  MerrySNStasiakKShepherdMFramptomCFlemingTLucassenMFG. The effectiveness of SPARX, a computerised self help intervention for adolescents seeking help for depression: randomised controlled non-inferiority trial. Br Med J (Online). (2012) 344:e2598. doi: 10.1136/bmj.e2598

  • CrossRef
  • Google Scholar

• 57

  NeilABatterhamPChristensenHBennettKGriffithsK. Predictors of adherence by adolescents to a cognitive behavior therapy website in school and community-based settings. J Med Internet Res (2009) 11(1):e6. doi: 10.2196/jmir.1050

  • CrossRef
  • Google Scholar

• 58

  PintoMDGreenblattAMHickmanRLRiceHMThomasTLClochesyJM. Assessing the critical parameters of eSMART-MH: a promising avatar-based digital therapeutic intervention to reduce depressive symptoms. Perspect Psychiatr Care (2016) 52(3):157–68. doi: 10.1111/ppc.12112

  • CrossRef
  • Google Scholar

• 59

  ReidSKauerSHearpsSJCrookeAHKhorASanciLet al. A mobile phone application for the assessment and management of youth mental health problems in primary care: health service outcomes from a randomised controlled trial of mobiletype. BMC Family Pract (2013) 14:84. doi: 10.1186/1471-2296-14-84

  • CrossRef
  • Google Scholar

• 60

  RiceSGleesonJDaveyCHetrickSParkerALedermanRet al. Moderated online social therapy for depression relapse prevention in young people: pilot study of a next generation online intervention. Early Intervention Psychiatry (2016) 12(4):613–25. doi: 10.1111/eip.12354

  • CrossRef
  • Google Scholar

• 61

  RickhiBKania-RichmondAMoritzSCohenJPaccagnanPDennisCet al. Evaluation of a spirituality informed e-mental health tool as an intervention for major depressive disorder in adolescents and young adults - a randomized controlled pilot trial. BMC Complement Altern Med (2015) 15:450. doi: 10.1186/s12906-015-0968-x

  • CrossRef
  • Google Scholar

• 62

  RobinsonJHetrickSCoxGBendallSYungAPirkisJ. The safety and acceptability of delivering an online intervention to secondary students at risk of suicide: findings from a pilot study. Early Intervention Psychiatry (2015) 9(6):498–506. doi: 10.1111/eip.12136

  • CrossRef
  • Google Scholar

• 63

  RobinsonJHetrickSCoxGBendallSYuenHPYungAet al. Can an Internet-based intervention reduce suicidal ideation, depression and hopelessness among secondary school students: results from a pilot study. Early Intervention Psychiatry (2016) 10(1):28–35. doi: 10.1111/eip.12137

  • CrossRef
  • Google Scholar

• 64

  SaulsberryAMarko-HolguinMBlomekeKHinkleCFogelJGladstoneTet al. Randomized clinical trial of a primary care internet-based intervention to prevent adolescent depression: one-year outcomes. J Can Acad Child Adolesc Psychiatry = J de l’Acad Canadienne Psychiatr l’enfant et de l’adolesc (2013) 22(2):106.

  • Google Scholar

• 65

  SekizakiRNemotoTTsujinoNTakanoCYoshidaCYamaguchiTet al. School mental healthcare services using internet-based cognitive behaviour therapy for young male athletes in Japan. Early Intervention Psychiatry13(1):79–85. doi: 10.1111/eip.12454

  • CrossRef
  • Google Scholar

• 66

  SmithPScottREshkevariEJattaFLeighEHarrisVet al. Computerised CBT for depressed adolescents: randomised controlled trial. Behav Res Ther (2015) 73:104–10. doi: 10.1016/j.brat.2015.07.009

  • CrossRef
  • Google Scholar

• 67

  SpenceSHDonovanCLMarchSAndersonREGambleAProsserSet al. A Randomized controlled trial of online versus clinic-based CBT for adolescent anxiety. (Author abstract)(Report). J Consult Clin Psychol (2011) 79(5):629. doi: 10.1037/a0024512

  • CrossRef
  • Google Scholar

• 68

  StasiakKHatcherSFramptonCMerrySN. A pilot double blind randomized placebo controlled trial of a prototype computer-based cognitive behavioural therapy program for adolescents with symptoms of depression. Behav Cogn Psychother (2014) 42(4):385–401. doi: 10.1017/S1352465812001087

  • CrossRef
  • Google Scholar

• 69

  Taylor-RodgersEBatterhamPJ. Evaluation of an online psychoeducation intervention to promote mental health help seeking attitudes and intentions among young adults: randomised controlled trial. J Affective Disord (2014) 168(C):65–71. doi: 10.1016/j.jad.2014.06.047

  • CrossRef
  • Google Scholar

• 70

  van Der ZandenRKramerJGerritsRCuijpersP. Effectiveness of an online group course for depression in adolescents and young adults: a randomized trial. J Med Internet Res (2012) 14(3):e86. doi: 10.2196/jmir.2033

  • CrossRef
  • Google Scholar

• 71

  WadeSLWalzNCCareyJMcMullenKMCassJMarkEet al. A randomized trial of teen online problem solving: Efficacy in improving caregiver outcomes after brain injury.(Report). Health Psychol (2012) 31(6):767–76. doi: 10.1037/a0028440

  • CrossRef
  • Google Scholar

• 72

  WhitesideSPHAleCMVickers DouglasKTiedeMSDammannJE. Case examples of enhancing pediatric OCD treatement with a smartphone application. Clin Case Stud (2014) 13(1):80–94. doi: 10.1177/1534650113504822

  • CrossRef
  • Google Scholar

• 73

  WhittakerRMerrySStasiakKMcDowellHDohertyIShepherdMet al. MEMO-a mobile phone depression prevention intervention for adolescents: development process and postprogram findings on acceptability from a randomized controlled trial. J Med Internet Res (2012) 14(1):e14. doi: 10.2196/jmir.1857

  • CrossRef
  • Google Scholar

• 74

  WhittakerRStasiakKMcDowellHDohertyIShepherdMChuaSet al. MEMO: an mHealth intervention to prevent the onset of depression in adolescents: a double-blind, randomised, placebo-controlled trial. J Child Psychol Psychiatry (2017) 58(9):1014–22. doi: 10.1111/jcpp.12753

  • CrossRef
  • Google Scholar

• 75

  WojtowiczMDayVMcGrathPJ. Predictors of participant retention in a guided online self-help program for university students: prospective cohort study. J Med Internet Res (2013) 15(5):e96. doi: 10.2196/jmir.2323

  • CrossRef
  • Google Scholar

• 76

  DumvilleJCTorgersonDJHewittCE. Reporting attrition in randomised controlled trials. Br Med J (2006) 332(7547):969–71. doi: 10.1136/bmj.332.7547.969

  • CrossRef
  • Google Scholar

• 77

  SchuellerSMParksAC. Disseminating self-help: positive psychology exercises in an online trial. J Med Internet Res (2012) 14(3):e63. doi: 10.2196/jmir.1850

  • CrossRef
  • Google Scholar

• 78

  BucciSSeymour-HydeAHarrisABerryK. Client and therapist attachment styles and working alliance. Clin Psychol Psychother (2015) 23(2):155–65. doi: 10.1002/cpp.1944

  • CrossRef
  • Google Scholar

• 79

  SingletonAAbelesPSmithIC. Online social networking and psychological experiences: the perceptions of young people with mental health difficulties. Comput Hum Behav (2016) 61:394–403. doi: 10.1016/j.chb.2016.03.011

  • CrossRef
  • Google Scholar

• 80

  BurnsJDavenportTRicciCEmmaBBlanchardMHickieI. It’s one ‘smart’ solution: using new and emerging technologies to support the mental health and wellbeing of young men. Dev Pract Child Youth Fam Work J (2014) 40:6–19.

  • Google Scholar

• 81

  MohrDCCuijpersPLehmanK. Supportive accountability: a model of providing human support to enhance adherence to eHealth interventions. J Med Internet Res (2011) 13(1):e30. doi: 10.2196/jmir.1602

  • CrossRef
  • Google Scholar

• 82

  CraneDGarnettCBrownJWestRMichieS. Factors influencing usability of a smartphone app to reduce excessive alcohol consumption: think aloud and interview studies. Front Public Health (2017) 5(39). doi: 10.3389/fpubh.2017.00039

  • CrossRef
  • Google Scholar

• 83

  StoyanovSRHidesLKavanaghDJZelenkoOTjondronegoroDManiM. Mobile App Rating Scale: a new tool for assessing the quality of health mobile apps. JMIR mHealth uHealth (2015) 3(1):e27. doi: 10.2196/mhealth.3422

  • CrossRef
  • Google Scholar

• 84

  LoonenAJMIvanovaSA. Circuits regulating pleasure and happiness in major depression. Med Hypothesis (2016) 87:14–21. doi: 10.1016/j.mehy.2015.12.013

  • CrossRef
  • Google Scholar

• 85

  BoschlooLvan BorkuloCDBorsboomDSchoeversRA. A prospective study on how symptoms in a network predict the onset of depression. Psychother Psychosom (2016) 85:183–4 doi: 10.1159/000442001.

  • CrossRef
  • Google Scholar