Development and Feasibility of an Online Brief Emotion Regulation Training (BERT) Program for Emerging Adults

Mental wellness is a critical component of healthy development in emerging adulthood and serves to protect against stress and promote resilience against psychopathology. Emotion regulation is a key mechanism for effective prevention because of its role in socio-emotional competence and its transdiagnostic significance for psychopathology. In this feasibility study, a brief, time and cost-effective emotion regulation training program for emerging adults (BERT) was developed and tested using the RE-AIM framework. Importantly, building interventions within the context of an implementation framework, such as the RE-AIM framework, enhances the chances that an intervention will be able to scale out and scale up. First, the brainwriting premortem method was utilized to refine program content, conducting focus groups a priori to identify potential program failures prior to program implementation. Undergraduate students (n = 12) attended four focus groups presenting initial program content. Four clinicians were also interviewed to determine program barriers. Qualitative analyses aggregated participant feedback to identify compliments, changes, and concerns about BERT and critical feedback was immediately implemented prior to initial testing. BERT was rooted in cognitive-behavioral practices and informed by the Gross model of emotion regulation. The 5-week program was then examined in a college sample (N = 42) to evaluate implementation (low attrition, high content engagement, favorable attitudes, low incidence of technical errors, costs), reach (enrollment and completion demographics comparable to the population in which recruitment took place), and efficacy (positive change in emotion regulation pre- to post-program). Of the recruited participants, 36 remained in the study where 27 completed at least 80% of program content. Repeated-measures ANOVAs exhibited significant improvements in emotion regulation, psychological distress, and negative affectivity, suggesting promising initial efficacy. Initial data provide support for feasibility and a future randomized control trial. BERT has potential significance for promoting healthy development as its brief electronic format reduced barriers and the program development process incorporated stakeholder feedback at multiple levels to inform better implementation and dissemination.

Introduction

Mental wellness is a critical piece of healthy development in emerging adulthood, a time when youth transition from adolescence to adulthood (1, 2). For optimal functioning, emerging adults must utilize appropriate and effective regulatory strategies (3). Healthy emotional functioning is particularly relevant to this population as they take on new roles, solidify identities, and face more responsibility (4–6). This developmental period brings unique stressors and increased independence, making it difficult to translate skills from late adolescence into early adulthood (7, 8). This “perfect storm” is evident in the transition to college, when students often leave home for the first time facing new expectations for their socioemotional and executive functioning furthering inconsistent mental wellbeing (9–11).

Most college students are emerging adults [18-26 years of age] and are likely to experience violence, accidents, and substance use as well as psychological distress based on the collegiate schedule [i.e., multiple exams, course load, financial obligations; (12, 13)]. Without skills to cope with these stressors, emerging adults experience elevated risk for chronic illness, suicide, and addiction (14–16). Emotion regulation (ER), which encompasses responses to emotions and the ability to control emotion processes, is a foundational skill to target as it is critical for healthy development (17). ER deficits are associated with multiple disorders including depression, anxiety, substance-use, personality, eating, and somatoform disorders (18). Building ER skills through brief intervention has the advantage of targeting a common mechanism and resiliency factor underpinning multiple disorders, thereby broadening the range of impact (19, 20).

In emerging adulthood, optimal ER relates to college transitions through social competence, self-control, and academic achievement (21). ER is an excellent treatment target in this population because of the widespread effects of dysregulation, especially when emotional suppression can have negative influences on social functioning (22, 23). Focusing on this underlying transdiagnostic mechanism can help prevent multiple negative outcomes. Though existing interventions are efficacious (e.g., Emotion Regulation Therapy, Dialectical Behavior Therapy, Mindfulness-Based Stress Reduction), they often require significant clinician or instructor training and sustained participant engagement. A community-based prevention program targeting ER may achieve meaningful changes in ER that promote participants' wellbeing while reducing time and cost barriers for both instructors/clinicians and participants.

The Gross (24) model of ER provides a foundation for understanding the pathway of emotion generation and regulation. Gross (24) describes a path to an emotional response in which a situation calls forth attention, leading to appraisal and an eventual response. Along this pathway, there are five regulatory strategies: situation selection (e.g., choosing to engage or avoid), situation modification (e.g., changing a situation), attentional deployment (e.g., shifting attentional resources), cognitive change (e.g., modifying thoughts), and response modulation (e.g., adapting behaviors) (25). These may be implemented at any stage of emotion expression to change an emotional state. Houck et al. (26) extrapolated this model to develop a 12-week (24 session) adolescent ER program that successfully reduced risky sexual behaviors. These promising results support the applicability relevance of the Gross (24) model for emerging adults (26), potentially more than adolescents, as cognition and cognitive control become more critical for regulating emotions in emerging adulthood (27).

Mental health is often divided into mental illness and wellness, though these are better conceptualized as ends of a spectrum than dichotomous categories (3). Considerable research on prevention and intervention promotes wellness in emerging adulthood above the absence of disease (28, 29). As a system, colleges have invested in campus counseling and academic centers to provide individual treatment (30–32). Current investments in university counseling centers and mental health prevention programs remain insufficient as more than 25% of college students in need do not have adequate access mental health services (15).

Developing prevention programs requires careful consideration of program delivery, engagement, and reception by the intended audience (33). Online programs require considerations of scalability and accessibility in addition to program effectiveness, as dropout and other barriers such as self-reliance, awareness of symptoms, inadequate knowledge of psychological resources, and stigma may reduce impact (34–36). The RE-AIM framework was developed as a public health guideline for creating effective treatments that can be scaled out and implemented on a large scale (37, 38). Reach (e.g., recruit large/representative portion of target populations), Efficacy (e.g., intervention impact on behavioral target), Adoption (e.g., which settings and staff initiate/participate in program delivery), Implementation (e.g., fidelity to delivery and costs), and Maintenance (e.g., extending treatments broadly, long term effects) are described as the critical pieces of RE-AIM for developing interventions that can be adopted on a larger scale (37, 39).

Ensuring appropriate Implementation, Reach, and Efficacy requires actively engaging the target population and testing the program for intended and unintended consequences, eliciting feedback at multiple levels. Brief ER programs have yet to be developed for emerging adults; therefore, this study takes careful steps to produce an effective and sustainable practical program. After initial content development, scalability was enhanced by acquiring expert and target population feedback. Incorporating feedback from the onset helps ensure that the program will be adopted and well-received. This framework is consistent with participatory action research, integrating reflection and participant input while seeking to reduce inequity and appropriately engage target populations (40).

Intervention efforts are shifting to brief and electronic formats to combat costs and address growing needs for treatment (41–43). Electronic formats for treatment are increasingly popular for youth raised in a digital world (44). Electronic interventions can often overcome typical barriers to care (e.g., accessibility, cost, stigma) at the cost of a potential loss of connection and accountability that mental health professionals provide (45–47). Furthermore, online treatments simplify the incorporation of measurement feedback systems [MFS; (48)]. In a community-based program, incorporating a MFS for continuous progress monitoring can help rapidly identify participant experience problems and diagnose program component reception in the moment. Brief online interventions have proven efficacious in college students for alcohol addiction (49–53), depression (54–56), and anxiety (57–60). Research comparing internet-based and face-to-face clinical interventions has shown equivalent overall effects, though acceptance of online interventions varies by clinician (41, 61–63).

The overall goal of this project was to systematically develop an online brief emotion regulation training (BERT) program designed to improve ER for emerging adults. Since BERT is a novel ER intervention delivered entirely online, this study aims to examine BERT's initial feasibility in preparation for a future randomized controlled trial. Additionally, it aimed to conduct preliminary tests of its Implementation, Reach and Efficacy. This study focused on the creation of BERT utilizing the brainwriting premortem method to refine program content to prepare for initial testing. The program was then tested in a college sample to evaluate implementation in delivering program content, reach to the recruited population, and preliminary efficacy through improved ER. This study was approved by the university institutional review board (IRB#20-018).

Intervention Development

To develop program content and design, the first author coordinated a team of data scientists, interaction and user experience designers, psychologists, and public health experts. The design team was mindful of guidelines for online design to meet standards of the 1990 Americans with Disabilities Act to ensure broad accessibility. The following sections outline the design of the BERT program and its four primary components: Emotion Regulation Orientation (ERO), Emotion Regulation Training (ERT), Self-Monitoring (SM), and Ecological Momentary Assessment (EMA) based on the Gross model (24).

Emotion Regulation Orientation

Participants completed an interactive 30-min online ERO at baseline describing the Gross (24) model of ER (see Supplementary Figure 1) and mental wellness resources (e.g., academic counseling, psychological support).

Emotion Regulation Training

Following the ERO, participants began the 5-week ERT. Each week, the ERT used short activities to break down a single ER strategy typically taught in one therapy session. Monday provided psychoeducation, Tuesday applied the strategy to daily life, Wednesday aimed to increase knowledge of the strategy, and Thursday's activity enhanced flexible implementation of these strategies (see Table 1 for ERT exercises).

TABLE 1

Table 1. Daily emotion regulation training (ERT) program content is outlined below.

Self-Monitoring

Participants filled out weekly SM surveys at the start of each ERT week measuring emotional distress, substance use, vitality, exercise, sleep, progress toward goals, and presence/absence of major stressors that could affect functioning. Critical scores triggered delivery of help-seeing resources.

Ecological Momentary Assessment

Additionally, EMAs had participants label their emotions and rate their stress in the moment after ERT exercises, though they could be accessed at any time to track their daily mood. The EMA survey was open daily from 8 a.m. until 10 p.m.

Intervention Dosage

After initial ERO, participants were asked to complete ~50 min of content in total per week (ERT), in addition to ~25 min of measurement per week. Altogether, the full treatment dose was ~6.75 h of intervention.

Preliminary Study

Premortem Focus Groups

Following recommendations for early engagement of program stakeholders (64) pre-implementation focus groups were conducted for a priori identification of program failures (65). Brainwriting premortem methods are more advantageous for evaluating a plan's success than pro/con generation, as they specifically address how a program is designed to fail (66). The brainwriting premortem approach provides an opportunity for psychological safety where data are collected in written form. To accomplish virtual brainwriting premortem within the COVID-19 pandemic, undergraduate focus groups were extended to a 1.5-h online session via videoconferencing software and anonymous contributions to a shared Google document. Clinicians attended 45-min individual interviews via videoconferencing software with notes taken by a research assistant during the interview.

Participants included 12 undergraduate students from three universities across four focus groups that took place between June and August 2020. The sample was predominantly female [n = 10; male (n = 1), trans male (n = 1)], heterosexual [n = 8; bisexual (n = 2)], queer (n = 1), lesbian (n = 1)], and white [n = 7; Asian (n = 2), American Indian/Alaskan Native/Latinx (n = 1), multiracial (n = 2)]. All were full-time students and were in their 2nd (n = 2), 3rd (n = 2), 4th (n = 7), or 5+ year (n = 1), with 2 transfer students. In addition, four white women clinicians from early- to mid-career were interviewed of whom two had doctorates in clinical psychology (PhD) and two were licensed clinical social workers (LCSW). All had experience working with emerging adults. Undergraduates and clinicians were all compensated with $10 Amazon gift cards for their participation.

The first author led a team of five trained undergraduate coders to identify themes regarding potential barriers. For each theme, changes to the program were identified in accordance with participant feedback, or a pragmatic or theory-based reason was noted for changes that could not be made (see Table 2). Themes included various skills (e.g., acceptance, coping), language (e.g., increase clarity, inclusivity), timing (e.g., longer or shorter), and external supports (e.g., discussion boards) among others. Though all specific program elements could not be presented in the focus groups and interviews due to time limitations, some themes (29% of undergraduate themes, 26% of clinician themes) were already included in the initial program design. All other themes that could be immediately addressed were incorporated in BERT before initiating the pilot study described below. On a follow-up questionnaire, participants rated their likeliness of completing program content as high (M = 4.00, SD = 1.13, range = 2–5 on a 5-point scale), and likelihood of recommending BERT to others even higher (M = 4.67, SD = 0.65, range = 3–5 on a 5-point scale).

TABLE 2

Table 2. Number of changes across undergraduate focus groups and clinician interviews.

Method

Participants

A priori power analyses were conducted with G Power v3.1.9.2 (67) determining that a sample size of 36 yields power of 0.90 to complete planned repeated measures ANOVAs. As such, a new sample of undergraduate students (N = 42) aged 18–23 years (M = 18.88; SD = 1.25) was recruited in September 2020 to continue evaluating Implementation, Reach, and Efficacy. Participants were predominantly female (n = 36), non-Hispanic white (n = 31), heterosexual (n = 38), and first-year students (n = 22; see Table 3). Participants were recruited through Introductory Psychology classes using an internal extra credit system, academic classes, diversity offices, student centers, women's centers, and wellness centers to encourage recruitment of participants from diverse backgrounds (e.g., underrepresented racial/ethnic groups, sexual minorities, low income) to better evaluate Reach. Students enrolled in undergraduate coursework who did not receive a ceiling score on the DERS were eligible for participation, which took place entirely online. Participants provided informed consent electronically and were compensated with $10 Amazon gift cards at the end of the study. Recruitment completed in October 2020 when participants who enrolled in the study were engaged in the structured program content from October to December 2020. Program content was delivered synchronously to participants who all started week 1 of the ERT simultaneously.

TABLE 3

Table 3. Demographics for initial enrollment compared to the final sample.

Measures

Attitudes

Participants filled out a questionnaire on their attitudes about BERT during the final survey. These questions were free-response and Likert-scale items which are presented in Supplementary Table 1.

Difficulties With Emotion Regulation Scale

This 36-item self-report questionnaire (68) is designed to measure ER in six subscales: non-acceptance of emotional responses (NONACCEPT; α = 0.87–0.91), difficulty engaging in goal-directed behavior (GOALS; α = 0.87–0.89), impulse control difficulties (IMPULSE; α = 0.75–0.82), lack of emotional awareness (AWARE; α = 0.86–0.92), limited access to ER strategies (STRAT; α = 0.87–0.91), and lack of emotional clarity (CLARITY; α = 0.80–0.82), as well as the total score (α = 0.93–0.94), all of which showed good to excellent internal consistency across time points (see Table 4).

TABLE 4

Table 4. Pilot study descriptive statistics and scale reliabilities for the full sample prior to data analysis or manipulation.

Emotion Regulation Questionnaire

This 10-item self-report questionnaire (69) has two subscales: cognitive reappraisal (CR; α = 0.70–0.89) and expressive suppression (ES; α = 0.76–0.82; Table 4) which showed good internal consistency across time points.

Brief Adjustment Scale

This 6-item self-report questionnaire (70) measures overall psychological functioning. It showed good to excellent internal consistency across time points (α = 0.84–0.91; see Table 4).

Modified Differential Emotion Scale

This 20-item self-report measure (71) assesses positive and negative affective experiences. Good to excellent internal consistency across time points was shown for the positive (α = 0.92–0.99) and negative affect (α = 0.85–0.90) scales (see Table 4).

COVID-19 Scales

This series of measures (72) was adapted from recommended CDC scales to measure impacts due to the COVID-19 pandemic. Three scales were utilized in this study: Perceived Coronavirus Threat Questionnaire (PCTQ; α = 0.82), Coronavirus Impacts Questionnaire (CIQ; α = 0.70), and the Coronavirus Experiences Questionnaire (CEQ; α = 0.69). All exhibited acceptable to good internal consistency. The CIQ is divided into three subscales: financial (α = 0.54), resource (α = 0.90), and psychological (α = 0.83). The CEQ is divided into 3 subscales: personal diagnoses/symptoms (α = 0.55), proximity to others (α = 0.86), and news (α = 0.60; see Table 4).

Procedure

Participants completed initial (prior to ERO), midpoint (week 3), and final (following week 5) Qualtrics surveys to evaluate impacts of BERT on ER and psychological distress. All participants received content for ERT Week 1 on the same date, to ensure all participants were completing the program on the same timeline. For self-monitoring, participants completed a brief questionnaire including the BASE-6 and vitality subscale of the Thriving Scale (73) at the start of each week. Automatic feedback was provided with interpretation, graphing scale scores and self-monitoring reports of sleep and substance use. Graphed individual feedback could be viewed by participants on a password protected, Google Apps webpage which ingested individual participant data from Qualtrics (see Supplementary Figure 2). Every day participants received the mDES as an EMA to capture daily mood. This measurement was designed to enhance emotion identification and emotional granularity. Alongside this measurement, participants also engaged in the 5-week ERT curriculum. Participation was recorded through their responses to the Qualtrics survey, serving as a manipulation check. In other words, BERT engagement was tracked by participants' responses to the daily program content.

Results

Data Cleaning

Data was screened for univariate and multivariate outliers; none were identified. Three participants did not complete the midpoint survey. Little's MCAR Test was performed to determine if values were missing at random [${\chi }_{(147,\mathit{\text{N}}=42)}^2$ = 106.80; p = 0.995]. Missing midpoint values were replaced by carrying forward initial survey values, providing a conservative estimate assuming no change. Two participants who completed BERT did not complete the final survey, with an additional participant missing parts of the survey. Little's MCAR Test was performed [${\chi }_{(129,\mathit{\text{N}}=30)}^2$ = 104.57; p = 0.944] and missing final values for these 3 participants were replaced by carrying forward midpoint survey values, or initial survey values if they did not complete the midpoint survey. The overall attrition rate was 11.8%. Participants initiated BERT if they completed the ERO prior to the first week of the ERT (34 of the 42 participants) and completed BERT if they completed program content through Week 5 (n = 30), though 2 completers did not complete the final survey (see CONSORT diagram, Figure 1).

FIGURE 1

Figure 1. Consort diagram to examine BERT completion rates and attrition.

Cost

BERT development took a substantial amount of time and had associated costs. Approximately 250 person hours were required for programming and graphic design, in addition to content development time. There were insufficient funds to build this program as an app-based software. Downstream costs associated with the Google Apps Script (GAS) platform included collaborative efforts between the grant team and five undergraduate students to ensure the program was delivered with fidelity. Approximately 1–2 h per day were required by team members to oversee the data collection process, send program content, and manage email reminders.

Enrollment

Females were represented in the study at a significantly higher rate (86%) than in the university [43%; ${\chi }_{(1,\mathit{\text{N}}=42)}^2$ = 14.13; p < 0.00; see Table 3]. The ethnoracial distribution of this sample did not significantly differ from the university population. Because the university only reports ethnicity separately from race, results were extrapolated from two calculations: (1) assuming that no White students enrolled in the university during the semester the study took place were Hispanic/Latinx, and (2) assuming that all Hispanic/Latinx students enrolled in the university during the study took place were White. The true value for the population will fall somewhere between these two values. In both cases, the actual numbers of ethnoracial minority students (ERM; n = 13) and non-Hispanic White students (NHW; n = 29) were not significantly different than expected based on university demographics ([1] ERM = 13.26, NHW = 28.74; ${\chi }_{(1,\mathit{\text{N}}=42)}^2$ = 0.007; p = 0.931; [2] ERM = 16.8, NHW = 25.2; ${\chi }_{(1,\mathit{\text{N}}=42)}^2$ = 1.433; p = 0.231). Thus, the success of Reach was mixed, with representation adequate by race/ethnicity, but not ideal by gender.

Engagement

Chi-square analyses were conducted to examine whether there were demographic differences between participants who completed the initial survey and continued on to BERT compared with those who did not initiate BERT. Completion rates for each part of the program are shown in Table 5. Though 38 participants started the ERO, 34 completed it, with 32 beginning the ERT. There were no significant differences based on minority status [${\chi }_{(1,\mathit{\text{N}}=42)}^2$ = 35; p = 0.55], gender [${\chi }_{(1,\mathit{\text{N}}=42)}^2$ = 1.19; p = 0.28], or first year status [${\chi }_{(1,\mathit{\text{N}}=42)}^2$ = 0.02; p = 0.90]. These results support the Reach of BERT. To explore the data, separate logistic regressions were conducted to examine whether there were study variable differences between participants who completed the initial survey and continued to BERT compared with those who did not initiate BERT. There were no significant differences based on age [${\chi }_{(1,\mathit{\text{N}}=42)}^2$ = 0.68; p = 0.17], BASE-6 [${\chi }_{(1,\mathit{\text{N}}=42)}^2$ = 0.13; p = 0.72], DERS subscales [${\chi }_{(6,\mathit{\text{N}}=42)}^2$ = 2.90; p = 0.82], or ERQ subscales [${\chi }_{(2,\mathit{\text{N}}=42)}^2$ = 0.58; p = 0.75], mDES scales [${\chi }_{(4,\mathit{\text{N}}=42)}^2$ = 3.25; p = 0.52; see Table 5].

TABLE 5

Table 5. Number of completed surveys or program content for the ecological momentary assessment (EMA), emotion regulation training (ERT), and self-monitoring (SM).

Technical errors for collecting compliance, had ERT completion rates based on 18 exercises (M = 16.57; SD = 2.81). Nineteen participants completed 100% of the ERT content, with 26 (78.1%) completing more than 75% of the ERT exercises. Participants may not have had equal opportunities to complete all 5 SM surveys due to a technical error in Week 4, so the 75% criterion was set at 3 SM completions with 85.3% of participants completing more than 75% of the SM opportunities. There were 35 EMA opportunities (daily for 5-weeks), and the average number of EMAs completed was 28 (80% completion rate), and 70.6% (n = 20) of participants exceeded a 75% completion rate. Three participants completed the EMA more than 35 times. Thus, engagement data support Implementation of BERT.

Attitudes

Descriptive statistics for participants' attitudes about BERT reported on the final survey are presented in Supplementary Table 1. Overall, participants' responses suggest favorable attitudes, as all positively worded items had mean scores above the mid-points on their respective scales and all negatively worded items had mean scores below the mid-points on their scales. Notably, all components of BERT were reported as helpful (Ms = 5.29–5.79 on a 7-point scale). There were varied responses to the ease of remembering the program concepts (M = 4.61; SD = 1.52 on a 7-point scale) and application to daily life (M = 4.89; SD =.96 on a 7-point scale). Participants reported that it was not difficult to keep up with program demands (M = 3.36; SD = 1.34 on a 7-point scale), that they were engaged (M = 6.11; SD = 0.79 on a 7-point scale), and the wording was easy to understand (M = 6.04; SD =0.69 on a 7-point scale). Participants reported they would be somewhat likely to recommend BERT to someone else (M = 3.86; SD = 0.59 on a 5-point scale). All of this supports the Implementation of BERT.

Independent samples t-tests were conducted to examine whether there were race/ethnicity (minority vs. White) differences in participants' favorable attitudes toward BERT on the follow-up survey. There were no significant differences in attitudes about helpfulness (ps = 0.07–0.93). There were no differences in difficulty remembering program concepts [t₍₂₅₎ = −0.93; p = 0.16], applicability to daily life [t₍₂₅₎ = −1.27; p = 0.22], keeping up with program demands [t₍₂₅₎ = 0.80; p = 0.71], engagement [t₍₂₅₎ = −0.46; p = 0.60], understanding wording [t₍₂₅₎ = −0.77; p = 0.15], or recommending the program to others [t₍₂₅₎ = 0.13; p = 0.97]. A similar t-test could not be conducted to test for gender differences because too few men completed BERT. The absence of differential attitudes based on ethnic/racial identity suggests no barriers to Reach.

Emotion Regulation

As the Bonferroni correction was utilized, the corrected alpha for the DERS was 0.007 and for the ERQ was 0.025. Repeated measures ANOVA showed overall difficulties with emotion regulation (DERS total score) significantly decreased from baseline to follow-up [F_(2,58) = 9.07; p < 0.000; partial η η² = 0.24; power = 0.97], consistent with hypotheses. Based on the Bonferroni-corrected alpha, no significant change was noted for the subscales: limited access to emotion regulation strategies [F_(2,58) = 5.07; p = 0.009; partial η² = 0.15; power = 0.80]; non-acceptance of emotional responses [SD = 4.15; F_(2,58) = 5.11; p = 0.009; partial η² = 0.13; power = 0.78]; difficulty engaging in goal-directed behavior [F_(2,58) = 5.64; p = 0.009; partial η² = 0.29; power = 0.82]; impulse control difficulties [F_(2,58) = 4.52; p = 0.015; partial η² = 0.14; power = 0.75]; lack of emotional awareness (F_(2,58) = 0.14; p = 0.870; partial η² = 0.01; power = 0.07]; lack of emotional clarity [F_(2,58) = 3.82; p = 0.028; partial η² = 0.12; power = 0.67].

On the ERQ, cognitive reappraisal skills significantly improved from baseline to follow-up [F_(2,58) = 7.73; p < 0.001; partial η² = 0.21; power = 0.94], consistent with hypotheses, but there was not power to detect a similar statistically significant reduction in emotion suppression over time [F_(2,58) = 2.49; p = 0.092; partial η² = 0.08; power = 0.48].

Psychological Distress

Repeated measures ANOVA showed that psychological distress assessed on the BASE-6 significantly decreased from baseline to follow-up [F_(2,58) = 8.42; p < 0.001; partial η² = 0.23; power = 0.96]. Additionally, the predicted cutoff score for this measure is 19 (70), suggesting that the decrease from 20.50 to 14.97 also is a clinically meaningful decrease in scores.

mDES

As the Bonferroni correction was utilized, the alpha for the mDES was 0.025. Changes in mDES total positive and negative affect scores were examined between baseline and follow-up with a repeated-measures ANOVA to determine broader changes in affectivity following BERT engagement. There was no change from baseline to follow-up in positive affect [F_(1,29) = 0.02; p = 0.882; partial η² = 0.00; power = 0.05]. There was a significant decrease in negative affect over time [F_(1,29) = 13.07; p = 0.001; partial η² = 0.31; power = 0.94].

COVID-19

The COVID-19 pandemic caused a range of interference in students' lives as evidenced by scores on the PCTQ, CIQ, and CEQ (see Table 6). When compared to outcome variables, the CIQ was positively correlated with DERS, BASE-6, and mDES and the CEQ was positively correlated with the DERS (see Table 6).

TABLE 6

Table 6. Descriptive statistics on the short CDC COVID-19 scales: PCTQ, Perceived Coronavirus Threat Questionnaire; CIQ, Coronavirus Impacts Questionnaire; CEQ, Coronavirus Experiences Questionnaire.

Discussion

An online Brief Emotion Regulation Training (BERT) program was developed and evaluated within the RE-AIM framework (37) to ensure this program is both efficacious and scalable. BERT was adapted based on focus groups and clinician feedback prior to pilot testing to determine its Implementation, prospective Reach, and Efficacy. Though there was substantial attrition between soliciting interest and initiating BERT, Implementation was mostly supported, and initial Efficacy was promising. BERT demonstrated Reach regarding race/ethnicity, but not gender, suggesting the importance of future program development exploring the barriers and preferences of undergraduate men for developing emotion regulation skills.

BERT's strength is as a brief and intensive online intervention that does not require the presence of a clinician. Taking a transdiagnostic and preventive lens, BERT focuses on ER as an underlying mechanism impacting a wide range of psychopathology in emerging adulthood. Additionally, BERT is sensitive to a health equity framework (74), attempting to circumvent traditional barriers to care and provide equitable care to vulnerable populations. The brief and intensive nature of this program is designed to be more immersive than traditional interventions (42). Each component was designed to mimic face-to-face evidence-based treatments. BERT was designed to build emotion regulation strategies guided by the Gross (24) process model of ER. Influenced by a cognitive-behavioral perspective, the program incorporates third-wave components (i.e., mindfulness, acceptance). Specifically, the orientation provides a grounding in psychoeducation on ER and mental health, the weekly self-monitoring is a weekly symptom check-in akin to measurement-based care (75), the EMA is intended to enhance emotional clarity, and the ERT is aimed at facilitating daily ER skills practice. While these components were delivered together, future iterations will disentangle these components to understand which ones are most significantly contributing to change.

Premortem Focus Groups

Incorporating stakeholders to identify changes and concerns early in BERT's development provided invaluable input for developing a sensitive and targeted treatment program. Pragmatically, the brainwriting premortem simplified qualitative data collection, as participants shared and responded to ideas in writing. Many suggested changes were already in the program, but critical observations by participants were incorporated prior to pilot launch, including steps for behavior change, types of coping, windows of completion, sensitive language, and general clarity. Further BERT development will integrate ancillary premortem and initial feedback to enhance the applicability of program content.

Reach

Promising initial results supported BERT's utility. Recruitment showed initial interest in BERT, though ~1/3 of students screened did not continue to the initial survey, roughly consistent with dropout rates in studies of smartphone apps for depression (76). Most participants who engaged past the first week of ERT were still active in the final week. There was mixed evidence for BERT's Reach. Students with minoritized racial and ethnic identities participated at rates that would be expected given the university distribution, representing a strength in Reach. Ensuring equitable Reach is especially important because these students are likely to face race-related stressors that increase health risks (77). Strengthening emotion regulation skills may ameliorate physiological and psychological mechanisms for the detrimental effects of race-related stressors, thereby promoting health equity (Peterson et al., 2020). The female-dominated program enrollment did not adequately represent the 57% male overall student population. Gender biases may reduce male-identifying individuals' participation in a program explicitly focused on emotion, as there is still stigma around attending to emotional wellbeing in male-identifying individuals (78–80). There were too few male participants to test gender differences in engagement and favorable attitudes. There were no gender differences in attrition and no race/ethnicity differences in engagement, attrition, or favorable attitudes.

Implementation

Implementation was supported from participants' engagement, low attrition once initiating BERT, and favorable attitudes. However, there were time costs, technological errors, and known technological limitations that posed barriers to implementation and retention. Relying on Qualtrics and GAS instead of mobile apps limited the program design, preventing use of SMS and push notifications that could have simplified content access and reduced implementation team time cost. Personalizing program reminders and having participants choose their notification preferences (i.e., text, push notification, email) would improve accessibility and reduce confusion. Moreover, participants all had to begin the program at the same time, which could have constrained Reach due to delayed initiation or period-specific life circumstances.

Engagement measures including attrition (12%) and mean content completion (92%) compare favorably with app- or smartphone-based interventions for health behaviors and disease management (76, 81–83). BERT completers generally considered the program helpful, reporting positive impacts on emotional and mental wellbeing. The online format ensured that all participants completed measurements within the same timeframe and allowed participants to choose the location and timing of survey completion within administration windows.

Efficacy

This study sought to examine initial efficacy to determine whether further randomized controlled trials are indicated. Initial Efficacy of BERT was promising as there was improvement in DERS total and ERQ cognitive reappraisal scores. No significant changes were seen in DERS subscale scores which may reflect low power due to the small sample size or broad, rather than targeted, change in ER. Still, BERT shows significant improvements in overall ER and cognitive reappraisal, suggesting that foundational ER skills are increasing even if participants are not recognizing or reporting an increase in specific ER components. Interestingly, the mDES exhibited significant reductions in negative emotion but no significant change in positive emotion; this may have been limited by higher reports of positive emotion at baseline. Decreases in negative affectivity could result from increased regulation or management of negative emotional states. Unfortunately, a virtual platform does not allow for in the moment clinical judgment. Future research could expand measurement to include emotionally salient tasks, changes in risk-taking, and executive functioning tasks to better understand how BERT impacts specific ER components or skills.

Psychological distress also saw significant reductions, a powerful result in the context of a cascading “mental health pandemic” resulting from the isolating effects of social distancing and quarantining during COVID-19 (84). It is interesting that on average baseline scores on the BASE-6 were clinically elevated and returned to below the clinical cutoff after the BERT intervention. This secondary effect highlights BERT's prevention value and further supports focus on ER as a critical transdiagnostic factor underlying psychopathology in emerging adults. At this stage, it is challenging to conclusively determine efficacy since there is no control group. Future studies will focus on testing with a waitlist control to ensure that changes are not due to random effects or time-series variation.

COVID-19

An unavoidable limitation was conducting this work during the COVID-19 pandemic., which significantly impacted general mental health and wellbeing (85). As individuals were forced to stay indoors and away from others for safety, ER may have been especially critical to cope with distress. Participants who experienced greater adversity from the pandemic had worse ER and psychological outcomes. Interestingly, receiving a diagnosis of COVID-19 or experiencing symptoms of COVID-19 related to a decrease in anxiety. It is possible that after having the illness, there was less anxiety as there was a reduced fear of becoming infected, or individuals may have felt a sense of relief from the potential to re-open networks after acquiring antibodies. Notably, there were no necessary modifications to the proposed BERT for it to be implemented during the pandemic, highlighting the flexibility of online interventions.

Limitations

Despite strong recruitment efforts, only 42 participants enrolled in BERT, limiting statistical power. The small, somewhat homogenous, sample also raises generalizability concerns about findings. Additionally, since this program is designed as a preventive intervention it is difficult to disentangle potential adverse effects, though none were observed during this study. Occasional technological errors in content delivery may have also impacted observed results, though statistically more likely to have biased effect estimates downward than upward. Since BERT was not yet fully automated, manual email reminders were sent to keep participants on track. Occasionally, difficulties in loading the participant homepage impacted data collection, making it impossible to know if participants were 100% compliant with program content. Replicating testing of BERT with a larger and more diverse sample and with fully automated program delivery are thus important next steps. This stage of program development focused primarily on implementation and feasibility, thus excluding a control group comparison to maximize statistical power. Future work will benefit from a waitlist control design.

Conclusions and Future Directions

Overall, results are promising for transdiagnostic treatment approaches to prevention like BERT. Approaching treatment development with the end in mind allows for critical examination of shortcomings and growth areas of BERT. While the online format lends itself to better fidelity, technological barriers still impeded implementation. By identifying and addressing dissemination barriers, BERT can be adapted to facilitate widespread program uptake. BERT takes an upstream approach to addressing a key mechanism underlying multiple forms of psychopathology, allowing it to effectively prevent a range of negative outcomes while promoting wellbeing. BERT can be widely disseminated because it is cost-effective, adaptable, and does not require significant clinician burden. BERT has great potential as a cost-effective tool to promote regulation and mitigate distress in emerging adulthood. Accessibility by individuals with disabilities was emphasized throughout the design process. Future development aims to test BERT with more ethnoracial and gender identity diverse samples to ensure an equitable program. When unmanaged, poor ER can manifest in deleterious ways (e.g., impulsive behaviors, outbursts, criminal behaviors, dropout) with a host of negative associated consequences (86). Proactively promoting ER strategies rather than reactively treating observed deficits can reduce individual and societal costs of mitigating the effects of dysregulation by preventing its emergence. BERT may also be modified for adolescent populations with the hopes of improving ER, reducing risk, and fostering resiliency in vulnerable youth. Integrating BERT into school settings may allow for increased accessibility and accountability, with professional support more readily available for youth in need of assistance with emotion identification and skill development. Additionally, BERT may be useful as a tool for individuals in need who are still waiting to receive clinical care. BERT has the potential to serve as a cost-effective prevention method for improving ER in vulnerable populations, by circumventing traditional barriers to care.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

The studies involving human participants were reviewed and approved by Virginia Tech Institutional Review Board. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

AG came up with the intellectual property and BERT intervention as well as the primary manuscript drafts, which was supervised by LC and JD. SH provided consultation regarding implementation science. The data bridge team including AB, NP, MS, and TE assisted in the development of the technological components of BERT and assisted with data collection. All authors assisted with editing drafts of the manuscript. All authors contributed to the article and approved the submitted version.

Funding

This project was funded by the Southern Regional Education Board Dissertation Award, the Thomas H. Ollendick Ph.D. and Mary Catherine Haley Ollendick Graduate Fellowship in Clinical Child Psychology, and the Institute for Creativity, Arts, and Technology (ICAT) at Virginia Tech SEAD Grant. Open access publishing fees were supported by the Virginia Tech Open Access Subvention Fund.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpubh.2022.858370/full#supplementary-material

References

1. Park MJ, Mulye TP, Adams SH, Brindis CD, Irwin CE Jr. The health status of young adults in the United States. J Adolesc Health. (2006) 39:305–17. doi: 10.1016/j.jadohealth.2006.04.017

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Wood D, Crapnell T, Lau L, Bennett A, Lotstein D, Ferris M, Kuo A. Emerging adulthood as a critical stage in the life course. In: Halfon N, Forrest CB, Lerner RM, Faustman EM, editors. Handbook of Life Course Development. Springer Nature (2018) 123–43. doi: 10.1007/978-3-319-47143-3_7

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Westerhof GJ, Keyes CL. Mental illness and mental health: the two continua model across the lifespan. J Adult Dev. (2010) 17:110–9. doi: 10.1007/s10804-009-9082-y

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Bonnie RJ, Stroud CE, Breiner HE. Investing in the Health and Well-Being of Young Adults. Washington, DC: National Academies Press (2015).

PubMed Abstract | Google Scholar

5. Schulenberg JE, Sameroff AJ, Cicchetti D. The transition to adulthood as a critical juncture in the course of psychopathology and mental health. Dev Psychopathol. (2004) 16:799–806. doi: 10.1017/S0954579404040015

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Thomas C, Zolkoski S. Preventing stress among undergraduate learners: the importance of emotional intelligence, resilience, and emotion regulation. Front Educ. (2020) 5:1–8. doi: 10.3389/feduc.2020.00094

CrossRef Full Text | Google Scholar

7. Roisman GI, Masten AS, Coatsworth JD, Tellegen A. Salient and emerging developmental tasks in the transition to adulthood. Child Dev. (2004) 75:123–33. doi: 10.1111/j.1467-8624.2004.00658.x

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Schwartz SJ. Turning point for a turning point: advancing emerging adulthood theory and research. Emerging Adulthood. (2016) 4:307–17. doi: 10.1177/2167696815624640

CrossRef Full Text | Google Scholar

9. Gatto AJ, Miyazaki Y, Cooper LD. Help Me Help Myself: Examining an electronic mental health self-monitoring system in college students. Higher Educ. (2020) 83:1–20. doi: 10.1007/s10734-020-00646-8

CrossRef Full Text | Google Scholar

10. Scales PC, Benson PL, Oesterle S, Hill KG, Hawkins JD, Pashak TJ. The dimensions of successful young adult development: a conceptual and measurement framework. Appl Dev Sci. (2016) 20:150–74. doi: 10.1080/10888691.2015.1082429

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Stephanou K, Davey CG, Kerestes R, Whittle S, Pujol J, Yücel M, et al. Brain functional correlates of emotion regulation across adolescence and young adulthood. Hum Brain Mapp. (2016) 37:7–19. doi: 10.1002/hbm.22905

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Arnett JJ. Human Development: A Cultural Approach, 2nd ed. Australia: Pearson (2016).

Google Scholar

13. Kroger J. Introduction: identity development through adulthood. Identity Int J Theory Res. (2002) 2:1–5. doi: 10.1207/S1532706XID0201_01

CrossRef Full Text | Google Scholar

14. Auerbach RP, Mortier P, Bruffaerts R, Alonso J, Benjet C, Cuijpers P, et al. WHO world mental health surveys international college student project: prevalence and distribution of mental disorders. J Abnorm Psychol. (2018) 127:623–38. doi: 10.1037/abn0000362

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Blanco C, Okuda M, Wright C, Hasin DS, Grant BF, Liu SM, et al. Mental health of college students and their non–college-attending peers: Results from the national epidemiologic study on alcohol and related conditions. Arch Gen Psychiatry. (2008) 65:1429–37. doi: 10.1001/archpsyc.65.12.1429

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Hunt J, Eisenberg D. Mental health problems and help-seeking behavior among college students. J Adolesc Health. (2010) 46:3–10. doi: 10.1016/j.jadohealth.2009.08.008

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Eisenberg N, Spinrad TL. Emotion-related regulation: sharpening the definition. Child Dev. (2004) 75:334–9. doi: 10.1111/j.1467-8624.2004.00674.x

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Berking M, Wupperman P. Emotion regulation and mental health: recent findings, current challenges, and future directions. Curr Opin Psychiatry. (2012) 25:128–34. doi: 10.1097/YCO.0b013e3283503669

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Fernandez KC, Jazaieri H, Gross JJ. Emotion regulation: a transdiagnostic perspective on a new RDoC domain. Cognit Ther Res. (2016) 40:426–40. doi: 10.1007/s10608-016-9772-2

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Messer SB, Wampold BE. Let's face facts: common factors are more potent than specific therapy ingredients. Clin Psychol Sci Pract. (2002) 9:21–5. doi: 10.1093/clipsy.9.1.21

CrossRef Full Text | Google Scholar

21. O'Connor M, Sanson A, Hawkins MT, Letcher P, Toumbourou JW, Smart D, et al. Predictors of positive development in emerging adulthood. J Youth Adolesc. (2011) 40:860–74. doi: 10.1007/s10964-010-9593-7

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Mouatsou C, Koutra K. Emotion regulation in relation with resilience in emerging adults: the mediating role of self-esteem. Curr Psychol. (2021) 1–14. doi: 10.1007/s12144-021-01427-x

CrossRef Full Text | Google Scholar

23. Srivastava S, Tamir M, McGonigal KM, John OP, Gross JJ. The social costs of emotional suppression: a prospective study of the transition to college. J Pers Soc Psychol. (2009) 96:883–97. doi: 10.1037/a0014755

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Gross JJ. Handbook of Emotion Regulation. 2nd ed. New York, NY: Guilford Publications (2014).

Google Scholar

25. Gross JJ, Thompson RA. Emotion regulation: conceptual foundations. In: Gross JJ, ed. Handbook of Emotion Regulation. New York, NY: Guilford Publications (2007). p. 2–24.

PubMed Abstract | Google Scholar

26. Houck CD, Hadley W, Barker D, Brown LK, Hancock E, Almy B. An emotion regulation intervention to reduce risk behaviors among at-risk early adolescents. Prevent Sci. (2016) 17:71–82. doi: 10.1007/s11121-015-0597-0

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Taber-Thomas B, Pérez-Edgar K. Emerging adulthood brain development. In: The Oxford Handbook of Emerging Adulthood, ed J. J. Arnett. New York, NY: Oxford University Press (2015). p. 126–41.

Google Scholar

28. Banks BM. Meet them where they are: an outreach model to address university counseling center disparities. J College Stud Psychother. (2019) 34:1–12. doi: 10.1080/87568225.2019.1595805

CrossRef Full Text | Google Scholar

29. Lockard AJ, Hayes JA, Locke BD, Bieschke KJ, Castonguay LG. Helping those who help themselves: does counseling enhance retention? J Counsel Dev. (2019) 97:128–39. doi: 10.1002/jcad.12244

CrossRef Full Text | Google Scholar

30. Kadison R, DiGeronimo TF. College of the overwhelmed: The campus mental health crisis and what to do about it, Vol. 6. San Francisco, CA: Jossey-Bass (2004) p. 296.

Google Scholar

31. O'Keeffe P. A sense of belonging: improving student retention. College Stud J. (2013) 47:605–13. Available online at: https://www.ingentaconnect.com/content/prin/csj/2013/00000047/00000004/art00005

Google Scholar

32. Xiao H, Carney DM, Youn SJ, Janis RA, Castonguay LG, Hayes JA, et al. Are we in crisis? National mental health and treatment trends in college counseling centers. Psychol Serv. (2017) 14:407–15. doi: 10.1037/ser0000130

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Leon AC, Davis LL, Kraemer HC. The role and interpretation of pilot studies in clinical research. J Psychiatr Res. (2011) 45:626–9. doi: 10.1016/j.jpsychires.2010.10.008

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Fernández-Álvarez J, Díaz-García A, González-Robles A, Baños R, García-Palacios A, Botella C. Dropping out of a transdiagnostic online intervention: a qualitative analysis of client's experiences. Internet Interventions. (2017) 10:29–38. doi: 10.1016/j.invent.2017.09.001

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Gulliver A, Griffiths KM, Christensen H. Perceived barriers and facilitators to mental health help-seeking in young people: a systematic review. BMC Psychiatry. (2010) 10:1–9. doi: 10.1186/1471-244X-10-113

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Kessler RC, Berglund PA, Bruce ML, Koch JR, Laska EM, Leaf PJ, et al. The prevalence and correlates of untreated serious mental illness. Health Serv Res. (2001) 36:987–1007. Available online at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1089274/

PubMed Abstract | Google Scholar

37. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. (1999) 89:1322–7. doi: 10.2105/AJPH.89.9.1322

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Balis LE, Strayer TE III, Ramalingam N, Harden SM. Beginning with the end in mind: contextual considerations for scaling-out a community-based intervention. Front Public Health. (2018) 6:357. doi: 10.3389/fpubh.2018.00357

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health. (2019) 7:64. doi: 10.3389/fpubh.2019.00064

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Baum F, MacDougall C, Smith D. Participatory action research. J Epidemiol Community Health. (2006) 60:854–7. doi: 10.1136/jech.2004.028662

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Andersson G, Cuijpers P, Carlbring P, Riper H, Hedman E. Guided Internet-based vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: a systematic review and meta-analysis. World Psychiatry. (2014) 13:288–95. doi: 10.1002/wps.20151

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Schleider JL, Dobias ML, Mullarkey MC, Ollendick T. Retiring, rethinking, and reconstructing the norm of once-weekly psychotherapy. Adm Policy Ment Health Mental Health Serv Res. (2020) 48:4–8. doi: 10.31234/osf.io/svbym

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Tate DF, Zabinski MF. Computer and Internet applications for psychological treatment: update for clinicians. J Clin Psychol. (2004) 60:209–20. doi: 10.1002/jclp.10247

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Prensky M. Digital natives, digital immigrants part 1. On the horizon. (2001) 9:1–6. doi: 10.1108/10748120110424816

CrossRef Full Text | Google Scholar

45. Atkinson NL, Gold RS. The promise and challenge of eHealth interventions. Am J Health Behav. (2002) 26:494–503. doi: 10.5993/AJHB.26.6.10

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Marks IM, Cavanagh K, Gega L. Computer-aided psychotherapy: revolution or bubble? Br J Psychiatry. (2007) 191:471–3. doi: 10.1192/bjp.bp.107.041152

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Newman MG. Technology in psychotherapy: an introduction. J Clin Psychol. (2004) 60:141–5. doi: 10.1002/jclp.10240

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Bickman L. A measurement feedback system (MFS) is necessary to improve mental health outcomes. J Am Acad Child Adolesc Psychiatry. (2008) 47:1114–9. doi: 10.1097/CHI.0b013e3181825af8

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Fleming M, Manwell LB. Brief intervention in primary care settings. Alcohol Res Health. (1999) 23:128–37.

Google Scholar

50. Heather N. Effectiveness of brief interventions proved beyond reasonable doubt. Addiction. (2002) 97:293–4. doi: 10.1046/j.1360-0443.2002.0105a.x

PubMed Abstract | CrossRef Full Text | Google Scholar

51. Hutton A, Prichard I, Whitehead D, Thomas S, Rubin M, Sloand E, et al. mHealth interventions to reduce alcohol use in young people: a systematic review of the literature. Compreh Child Adolesc Nurs. (2020) 43:171–202. doi: 10.1080/24694193.2019.1616008

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Larimer ME, Cronce JM, Lee CM, Kilmer JR. Brief intervention in college settings. Alcohol Res Health. (2004) 28:94–104. Available online at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6601644/

Google Scholar

53. Terlecki MA, Buckner JD, Larimer ME, Copeland AL. Brief motivational intervention for college drinking: The synergistic impact of social anxiety and perceived drinking norms. Psychol Addict Behav. (2012) 26:917–23. doi: 10.1037/a0027982

PubMed Abstract | CrossRef Full Text | Google Scholar

54. Church D, De Asis MA, Brooks AJ. Brief group intervention using Emotional freedom techniques for depression in college students: a randomized controlled trial. Depress Res Treat. (2012) 2012:1–7. doi: 10.1155/2012/257172

PubMed Abstract | CrossRef Full Text | Google Scholar

55. Geisner IM, Varvil-Weld L, Mittmann AJ, Mallett K, Turrisi R. Brief web-based intervention for college students with comorbid risky alcohol use and depressed mood: does it work and for whom? Addict Behav. (2015) 42:36–43. doi: 10.1016/j.addbeh.2014.10.030

PubMed Abstract | CrossRef Full Text | Google Scholar

56. Schleider JL, Mullarkey MC, Fox KR, Dobias ML, Shroff A, Hart EA, et al. A randomized trial of online single-session interventions for adolescent depression during COVID-19. Nat Hum Behav. (2022) 6:258–68. doi: 10.1038/s41562-021-01235-0

PubMed Abstract | CrossRef Full Text | Google Scholar

57. Brunyé TT, Mahoney CR, Giles GE, Rapp DN, Taylor HA, Kanarek RB. Learning to relax: evaluating four brief interventions for overcoming the negative emotions accompanying math anxiety. Learn Individ Differ. (2013) 27:1–7. doi: 10.1016/j.lindif.2013.06.008

CrossRef Full Text | Google Scholar

58. Chaló P, Pereira A, Batista P, Sancho L. Brief biofeedback intervention on anxious freshman university students. Appl Psychophysiol Biofeedback. (2017) 42:163–8. doi: 10.1007/s10484-017-9361-5

PubMed Abstract | CrossRef Full Text | Google Scholar

59. Tatum T, Lundervold DA, Ament P. Abbreviated upright behavioral relaxation training for test anxiety among college students: initial results. Int J Behav Consult Ther. (2006) 2:475. doi: 10.1037/h0101001

CrossRef Full Text | Google Scholar

60. Call D, Miron L, Orcutt H. Effectiveness of brief mindfulness techniques in reducing symptoms of anxiety and stress. Mindfulness. (2014) 5:658–68.

Google Scholar

61. Perle JG, Langsam LC, Randel A, Lutchman S, Levine AB, Odland AP, et al. Attitudes toward psychological telehealth: current and future clinical psychologists' opinions of Internet-based interventions. J Clin Psychol. (2013) 69:100–13. doi: 10.1002/jclp.21912

PubMed Abstract | CrossRef Full Text | Google Scholar

62. Van Doorn M, Nijhuis LA, Egeler MD, Daams JG, Popma A, Van Amelsvoort T, et al. Online indicated preventive mental health interventions for youth: a scoping review. Front Psychiatry. (2021) 12:562–81. doi: 10.3389/fpsyt.2021.580843

PubMed Abstract | CrossRef Full Text | Google Scholar

63. Wentzel J, van der Vaart R, Bohlmeijer ET, van Gemert-Pijnen JE. Mixing online and face-to-face therapy: how to benefit from blended care in mental health care. JMIR Mental Health. (2016) 3:e9. doi: 10.2196/mental.4534

PubMed Abstract | CrossRef Full Text | Google Scholar

64. Brownson RC, Colditz GA, Proctor EK. Dissemination and Implementation Research in Health: Translating Science to Practice. New York, NY: Oxford University Press (2018).

Google Scholar

65. Gilmartin H, Lawrence E, Leonard C, McCreight M, Kelley L, Lippmann B, et al. Brainwriting premortem: a novel focus group method to engage stakeholders and identify pre-implementation barriers. J Nurs Care Qual. (2019) 34:94–100. doi: 10.1097/NCQ.0000000000000360

PubMed Abstract | CrossRef Full Text | Google Scholar

66. Veinott B, Klein GA, Wiggins S. Evaluating the effectiveness of the PreMortem technique on plan confidence. In: Proceedings of the 7th International ISCRAM Conference (May 2010) (2010).

Google Scholar

67. Faul F, Erdfelder E, Buchner A, Lang AG. G^* Power (Version 3.1. 9.2). Kiel: University of Kiel (2014).

68. Gratz KL, Roemer L. Multidimensional assessment of emotion regulation and dysregulation: development, factor structure, and initial validation of the difficulties in emotion regulation scale. J Psychopathol Behav Assess. (2004) 26:41–54. doi: 10.1023/B:JOBA.0000007455.08539.94

PubMed Abstract | CrossRef Full Text | Google Scholar

69. Gross JJ, John OP. Individual differences in two emotion regulation processes: implications for affect, relationships, and well-being. J Pers Soc Psychol. (2003) 85:348–62. doi: 10.1037/0022-3514.85.2.348

PubMed Abstract | CrossRef Full Text | Google Scholar

70. Cruz RA, Peterson AP, Fagan C, Black W, Cooper L. Evaluation of the brief adjustment scale−6 (BASE-6): a measure of general psychological adjustment for measurement-based care. Psychol Serv. (2019) 19:1541–59. doi: 10.1037/ser0000366

PubMed Abstract | CrossRef Full Text | Google Scholar

71. Fredrickson BL, Tugade MM, Waugh CE, Larkin GR. What good are positive emotions in crisis? A prospective study of resilience and emotions following the terrorist attacks on the United States on September 11th, 2001. J Person Soc Psychol. (2003) 84:365–76. doi: 10.1037/0022-3514.84.2.365

CrossRef Full Text | Google Scholar

72. Conway III LG, Woodard SR, Zubrod A. Social Psychological Measurements of COVID-19: Coronavirus Perceived Threat, Government Response, Impacts, and Experiences Questionnaires [Preprint]. (2020).

Google Scholar

73. Porath C, Spreitzer G, Gibson C, Garnett FG. Thriving at work: toward its measurement, construct validation, and theoretical refinement. J Organ Behav. (2012) 33:250–75. doi: 10.1002/job.756

CrossRef Full Text | Google Scholar

74. Peterson A, Charles V, Yeung D, Coyle K. The health equity framework: a science-and justice-based model for public health researchers and practitioners. Health Promot Pract. (2020). doi: 10.1177/1524839920950730

PubMed Abstract | CrossRef Full Text | Google Scholar

75. Scott K, Lewis CC. Using measurement-based care to enhance any treatment. Cogn Behav Pract. (2015) 22:49–59. doi: 10.1016/j.cbpra.2014.01.010

PubMed Abstract | CrossRef Full Text | Google Scholar

76. Torous J, Lipschitz J, Ng M, Firth J. Dropout rates in clinical trials of smartphone apps for depressive symptoms: a systematic review and meta-analysis. J Affect Disord. (2020) 263:413–9. doi: 10.1016/j.jad.2019.11.167

PubMed Abstract | CrossRef Full Text | Google Scholar

77. Williams DR. Stress and the mental health of populations of color: advancing our understanding of race-related stressors. J Health Soc Behav. (2018) 59:466–85. doi: 10.1177/0022146518814251

PubMed Abstract | CrossRef Full Text | Google Scholar

78. Galdas PM, Cheater F, Marshall P. Men and health help-seeking behaviour: literature review. J Adv Nurs. (2005) 49:616–23. doi: 10.1111/j.1365-2648.2004.03331.x

PubMed Abstract | CrossRef Full Text | Google Scholar

79. Latalova K, Kamaradova D, Prasko J. Perspectives on perceived stigma and self-stigma in adult male patients with depression. Neuropsychiatr Dis Treat. (2014) 10:1399–405. doi: 10.2147/NDT.S54081

PubMed Abstract | CrossRef Full Text | Google Scholar

80. Vogel DL, Wester SR, Hammer JH, Downing-Matibag TM. Referring men to seek help: the influence of gender role conflict and stigma. Psychol Men Masc. (2014) 15:60–7. doi: 10.1037/a0031761

CrossRef Full Text | Google Scholar

81. Egilsson E, Bjarnason R, Njardvik U. Usage and weekly attrition in a smartphone-based health behavior intervention for adolescents: pilot randomized controlled trial. JMIR Format Res. (2021) 5:e21432. doi: 10.2196/21432

PubMed Abstract | CrossRef Full Text | Google Scholar

82. Iribarren SJ, Akande TO, Kamp KJ, Barry D, Kader YG, Suelzer E. Effectiveness of mobile apps to promote health and manage disease: systematic review and meta-analysis of randomized controlled trials. JMIR mHealth uHealth. (2021) 9:e21563. doi: 10.2196/21563

PubMed Abstract | CrossRef Full Text | Google Scholar

83. Meyerowitz-Katz G, Ravi S, Arnolda L, Feng X, Maberly G, Astell-Burt T. Rates of attrition and dropout in app-based interventions for chronic disease: systematic review and meta-analysis. J Med Internet Res. (2020) 22:e20283. doi: 10.2196/20283

PubMed Abstract | CrossRef Full Text | Google Scholar

84. Choi KR, Heilemann MV, Fauer A, Mead M. A second pandemic: mental health spillover from the novel coronavirus (COVID-19). J Am Psychiatr Nurses Assoc. (2020) 26:340–3. doi: 10.1177/1078390320919803

PubMed Abstract | CrossRef Full Text | Google Scholar

85. Kumar A, Nayar KR. COVID 19 and its mental health consequences. J Ment Health. (2020) 180:817–8. doi: 10.1001/jamainternmed.2020.1562

PubMed Abstract | CrossRef Full Text | Google Scholar

86. Weiss NH, Sullivan TP, Tull MT. Explicating the role of emotion dysregulation in risky behaviors: a review and synthesis of the literature with directions for future research and clinical practice. Curr Opin Psychol. (2015) 3:22–9. doi: 10.1016/j.copsyc.2015.01.013

PubMed Abstract | CrossRef Full Text | Google Scholar