Predictors of treatment dropout in patients with posttraumatic stress disorder due to childhood abuse¹

Abstract

Background:

Knowledge about patient characteristics predicting treatment dropout for post-traumatic stress disorder (PTSD) is scarce, whereas more understanding about this topic may give direction to address this important issue.

Method:

Data were obtained from a randomized controlled trial in which a phase-based treatment condition (Eye Movement Desensitization and Reprocessing [EMDR] therapy preceded by Skills Training in Affect and Interpersonal Regulation [STAIR]; n = 57) was compared with a direct trauma-focused treatment (EMDR therapy only; n = 64) in people with a PTSD due to childhood abuse. All pre-treatment variables included in the trial were examined as possible predictors for dropout using machine learning techniques.

Results:

For the dropout prediction, a model was developed using Elastic Net Regularization. The ENR model correctly predicted dropout in 81.6% of all individuals. Males, with a low education level, suicidal thoughts, problems in emotion regulation, high levels of general psychopathology and not using benzodiazepine medication at screening proved to have higher scores on dropout.

Conclusion:

Our results provide directions for the development of future programs in addition to PTSD treatment or for the adaptation of current treatments, aiming to reduce treatment dropout among patients with PTSD due to childhood abuse.

1. Introduction

Posttraumatic stress disorder (PTSD) has a major impact on social and occupational functioning and individuals’ quality of life (1). In addition, individuals with PTSD are at great risk of attempting suicide (2–4). Although many treatments targeting PTSD have proven to be effective [e.g., (5)], a recent meta-analysis of randomized controlled trials (RCTs) found an average dropout rate of 21% for guideline-recommended PTSD treatments (6). This is problematic as untreated PTSD compared to treated PTSD may lead to a worse prognosis, and many societal consequences (7). Considering the societal impact of PTSD, it is important to ensure that individuals suffering from PTSD complete their treatment because evidence-based PTSD treatments significantly improve their prognosis (7). Completing treatment may be of particular importance for those with a history of childhood abuse since they are at risk of displaying more severe symptoms of PTSD or developing symptoms of Complex PTSD [ISTSS, 2012; (8)].

In order to establish a Complex PTSD diagnosis individuals need to experience symptoms of “Disturbances in Self-Organization” [DSO; i.e., problems with affect regulation, negative self-concept, and interpersonal problems; (9)], in addition to all diagnostic criteria of PTSD. It has been suggested that existing evidence-based trauma-focused therapies may lead to less favorable outcomes and more dropout in patients with a history of childhood abuse (10–12), and that these patients are at risk of developing Complex PTSD (8). Previous attempts to decrease the dropout rate from PTSD treatments in this target group by adding treatment programs that specifically target Complex PTSD symptoms in addition to actual trauma-focused treatment (11, 13) did not lead to less dropout compared to only applying direct trauma-focused treatment (14–16). The identification of patient characteristics that predict early treatment termination is essential for developing new target strategies to prevent dropout.

Most meta-analyses studying potential predictors of dropout regarding PTSD treatments focus on the kind of therapy as a predictive factor [i.e., (17–21)]. As far as we know, only Varker et al. (6) performed a meta-analysis in which they also considered patient characteristics in the comparison of several PTSD treatments among patients with military and civilian trauma. PTSD chronicity, PTSD severity, medication use, age, employment status, relationship status, sex, baseline depression scores, or baseline anxiety scores were included separately as possible predictors, but none of the included patient characteristics were found to be predictive of dropout. To this end, given the scarcity of studies and variables considered so far and the lack of meta-analytic support for specific comorbid symptoms as potential predictors for treatment dropout, more research on patient characteristics predicting dropout is needed. As, until now, not one specific characteristic has been found to be predictive of dropout, it may be even more important to determine whether a combination of characteristics may be predictive.

The purpose of the present study was to identify patients who are at risk of dropping out, and to identify patient characteristics that predict the dropout of PTSD treatments in patients with PTSD related to childhood abuse. To achieve this aim, we used machine learning techniques. One advantage of machine learning techniques is that all available pretreatment variables and variable combinations are examined (22), thus not limiting the number of possible predictors. Another advantage is that it is particularly appropriate for identifying small effects in predicting outcomes (22). For the current study, the dataset of a multi-center randomized controlled trial was used, that compared a phase-based treatment with a direct-trauma-focused treatment in patients with PTSD related to a history of childhood abuse [Van Vliet et al., 2017; (16)].

2. Method

2.1. Design and participants

A total of 121 patients were recruited in two mental health organizations; Dimence GGZ and GGZ Oost-Brabant. Included patients were aged between 18 and 65 years, and diagnosed with PTSD based on the Clinician-Administered PTSD scale for DSM-5 [CAPS-5; (23)]. Participants had to be a victim of repeated sexual and/or physical abuse before the age of 18 by a caretaker or a person in a position of authority, which was identified by the Life Events Checklist for the DSM-5 [LEC-5; (23)]. Patients were excluded when they did not master the Dutch language sufficiently, or in case of acute suicidality for which direct crisis intervention was needed, as assessed by an item of the Beck Depression Inventory-II [BDI; (24)]. In addition, patients were excluded if they had received at least eight sessions of any well-evaluated treatment for PTSD in the past year, reported being a victim of ongoing physical and/or sexual abuse, reported severe use of alcohol or drugs, or had an intellectual disability. The study design was registered in a national trial registry² and approved by the medical ethics committee Twente NL.³ Further details on the trial methodology and patient sample can be found in the study protocol (25) and main outcome paper of the study (16).

2.2. Interventions

After patients were found to be eligible for participation in the study, they were randomized to two treatment conditions: both contained 16 sessions of EMDR (Eye Movement Desensitization and Reprocessing) as the trauma-focused treatment, and in one condition these sixteen sessions of EMDR were preceded by eight sessions of the first phase treatment STAIR (in total 24 sessions). Both the STAIR and the EMDR therapy were delivered twice a week for 90 min. STAIR was conducted according to the program described by Cloitre and her colleagues (11). EMDR therapy was conducted according to the protocol by Shapiro using the Dutch translation of the treatment protocol (26). Before actual treatment, patients first received one 90 min session consisting of psycho-education in which a hierarchy of relevant traumatic experiences was determined. For a full description of the two treatments, see Van Vliet et al. (25).

2.3. Measures

2.3.1. Drop out

Patients were considered to have dropped out if they discontinued treatment prematurely after the first session, which included psychoeducation and case conceptualization just before the actual treatment began. This applies to cases where the patient did not complete the total number of treatment sessions as per the study’s requirements [see the study protocol; (25)], and failed to complete treatment for all the traumas that were selected for treatment during the first treatment session. The outcome variable was a dichotomous variable: dropout versus completer status.

2.3.2. Pre-treatment variables

Tables 1 and 2 show an overview of the pre-treatment variables and the differences between the two groups (dropout versus completer status). The reasons given by the patients for dropout are shown in Supplementary Table S1.

2.3.2.1. Demographic characteristics

The following demographic characteristics were determined in the study: Gender, level of education, employment status, marital status, and age.

2.3.2.2. PTSD variables

PTSD symptom severity at the start of the treatment was measured with the CAPS-5 (23). The CAPS-5 is a clinical interview that includes 20 items on a 5-point Likert scale, resulting in a total score of between 0 and 80. The CAPS-5 has good psychometric properties [(27); see for the Dutch version (28)]. The inter-rater reliability was assessed by calculating the interclass correlation coefficient, which was 0.999, which is an excellent score.

2.3.2.3. Suicidality

Item 9 of the BDI-II (24) was used to measure suicidality. The scale of this item ranges from 0 to 3, with 0 for the absence of suicidal thoughts, 1 for indicating the presence of suicidal thoughts, but no intention to carry them out, and, 2 for indicating suicidal thoughts accompanied by a clear intention to commit suicide. Patients were excluded when they scored a 3 on this scale, which said that they would commit suicide whenever they had the chance. In that case they were assigned to a direct crisis intervention. In the current analysis, suicidality was used as a dichotomous value: absence (score of 0) or presence of suicidality (score of 1 or 2).

2.3.2.4. Borderline personality disorder

The Structured Clinical Interview for DSM-IV Axis II personality disorders [SCID-II interview; (29, 30)] was used to determine the presence of a borderline personality disorder. The psychometric properties are fair to good for this instrument.

Self-Injury: The severity of self-injury was determined by items 97 and 98 of the Dutch version of the SCID-II interview (30), with 1 for absent, 2 for doubtful, and 3 for present.

2.3.2.5. Dissociative symptoms

The severity of dissociative symptoms was indexed using the Dissociative Experiences Scale (DES-II; (31); Cronbach’s α = 0.93 in the present study at baseline). This is a 28-item self-report questionnaire with scores ranging from 0 to 100 (32). The presence of a dissociative subtype of PTSD was determined using the CAPS-5 (23).

2.3.2.6. Complex PTSD

The presence and severity of Complex PTSD was measured by the Structured Interview for Disorders of Extreme Stress [SIDES; (33)], the 38-item version developed by Ford et al. (34). The SIDES has good psychometric properties as a dichotomous measure in determining whether Complex PTSD is present or not [SIDES Manual by (35)].

2.3.2.7. Interpersonal difficulties

Interpersonal difficulties were indexed using the Inventory of Interpersonal Problems [IIP; (36)]. The psychometric properties of the IIP are satisfying (37). The IIP contains 32 items that can be scored on a 5-point scale from 0 (not at all) to 4 (very strongly). The reliability at baseline in this study was high (Cronbach’s α = 0.85).

2.3.2.8. Emotion regulation

Difficulties in emotion regulation were measured with the Difficulties in Emotion Regulation Scale [DERS; (38)], a questionnaire that has been validated in clinical populations (38, 39) and nonclinical populations (38, 40). Each item of the DERS is rated on a 5-point scale. The reliability in this study at baseline was high (Cronbach’s α = 0.92).

2.3.2.9. Problems In self-esteem

To index problems in self-esteem the self-esteem subscale of the Posttraumatic Cognitions Inventory [PTCI; (41)] was used. Items are scored on a Likert scale from 1 (“I totally disagree”) to 7 (“I totally agree”), and psychometric properties for the Dutch version (42) are good. The PTCI score for self-esteem showed a high reliability at baseline in this study (Cronbach’s α = 0.94).

2.3.2.10. General psychopathology

The Brief Symptom Inventory (43, 44) was used to measure the severity of general psychopathology symptoms. The severity of each item can be rated on a 5-point scale from 0 (not at all) to 4 (a lot). The Dutch version has good psychometric properties [(45); Cronbach’s α = 0.95 at baseline in the present study].

3. Statistical analysis

All analyses were carried out using RStudio (46). The R code used is available in the Supplementary Material.

3.1. Data pre-processing

Following Cohen et al. (47) individuals with less than 50% missing pre-treatment values were included in this study. As a result, ten individuals were excluded. For the remaining 111 participants, variables with missing data were imputed using a random forest imputation algorithm [R package ‘MissForest’, (48)]. The benefits of this approach are that no pre-processing is required and that it is robust for noisy data and multicollinearity, so that it can be applied to mixed data types (48, 49). The imputation method was verified by applying this method to the non-missing dataset with completely at random removed values. After imputing the missing values, the performance was assessed using the normalized root mean squared error (NRMSE) and the proportion of falsely classified (PFC), which is defined by comparing the complete values with the imputed values (48).

In case of highly correlated variables (cor. > 0.70), one of the variables was dropped to avoid redundancy and multicollinearity. The decision which variable to remove was made by the research team. Outliers were winsorized and continuous variables with a non-normal distribution were transformed using the Box-Cox method [(50); R package ‘Caret’; (51)]. Finally, continuous variables were standardized and categorical variables were centered (see Table 3 for details about transformations for each variable). This data pre-processing procedure resulted in a dataset of 111 participants with 22 pre-treatment variables. Dropouts did not differ significantly between patients who received STAIR-EMDR (21.8%) and EMDR only therapy [16.1%; ꭓ2 (1) = 6.00, p = 0.440]. A prognostic model for dropout was developed independently of treatment conditions because the subsample of dropouts was too small to create a prescriptive model for dropout depending on the treatment conditions.

3.2. Imbalanced dataset

After data pre-processing, the ratio between dropout and completers was checked, because it was expected that the dataset would be imbalanced due to a relatively smaller proportion of dropouts compared to completers. This is of importance as an imbalanced dataset causes classification performance problems in machine learning algorithms (52). To deal with this class imbalance, the proven effective synthetic minority oversampling technique (SMOTE) was applied (53). This method over-samples the minority class (i.e., dropout), by artificially creating new samples using the nearest neighbours of the cases, and under-samples the majority class (53).

3.3. Model building

For the dropout prediction, a model was developed using Elastic Net Regularization [ENR, R package ‘glmnet’, (54)]. ENR is a combination of ridge regression and lasso regression where alpha is the tuning parameter between ridge regression (alpha = 0) and lasso [alpha = 1; (55)]. Another tuning parameter is the lambda which determines the shrinkage or penalty of the coefficients, the larger its value the stronger the shrinkage (55). The alpha and lambda were determined using 10-fold cross-validation, where the selected alpha was based on the highest area under the curve [AUC; (56)]. After determination of the alpha and lambda, the final model was built.

3.4. Model evaluation

The model was evaluated on the initial (imbalanced) dataset using three different measures: accuracy, the area under the curve (AUC), and the Brier score. The selected variables in the final model were evaluated for significance and variable importance. The variable importance was calculated using the vip package (57). The accuracy is the percentage of correct dropout predictions ranging from 0% (worst prediction) to 100% (best prediction). The AUC refers to the overall performance of a classifier. When the AUC is 1, predictions are 100% correct, and when the AUC is 0, all predications are incorrect (56). The Brier score can be described as a parameter that measures the accuracy of probabilistic predictions ranging from 0 (best prediction) to 1 [worst prediction; (58)].

4. Results

4.1. Data pre-processing

The total sample consisted of 111 patients. From that number, 80 individuals (72%) had no missing data and the percentage of missing baseline variables was 2.5 percent. All patients who dropped out prematurely and were assessed after dropping out still fulfilled the diagnostic criteria for PTSD. The number of sessions before dropping out was registered and the median of the number of sessions before dropping out was 4 sessions. The missing values were imputed and the performance of the imputation method was acceptable (NRMSE = 0.15; PFC 0.26).

4.2. Imbalanced dataset and dropout rate

The dropout rate in this study was 19%, indicating that the distribution between dropout and completers was unequal, leading to an imbalanced dataset. To balance the dataset, SMOTE was applied. After applying SMOTE, the dropout rate was 43%.

4.3. Model building

The final ENR model selected the following variables for dropout risk: suicidality, benzodiazepine medication use at screening, education, gender, sexual abuse, Borderline personality disorder, PTCI Self-Esteem, SIDES, DERS, complex PTSD, BSI, DES, and CAPS-5 scores.

4.4. Predictor evaluation

The best ENR model (alpha = 0.25, lambda = 0.18) correctly predicted 81.6% of all individuals (accuracy). For this model, the AUC was 0.85 and the Brier score was 0.31, indicating sufficient predictions. The sensitivity and specificity are 75 and 87%, respectively. Predictors were evaluated based on both significance and variable importance. Based on significance (i.e., value of p lower than 0.05), the main variables for risk of dropout in the model were gender, education, suicidality, score on the DERS, score on the BSI and benzodiazepine medication use at screening. The (i.e., an importance score greater than or close to 1), the main variables of the model were suicidality, education, and benzodiazepine use at screening (for a total overview of the variable importance, see Table 4). More specifically, low education level, presence of suicidal thoughts, and not using benzodiazepine medication during screening were most strongly related to dropout risk according to both significance and variable importance. Given our focus on prediction, we will further focus on these three variables with the highest variable importance score and not on all significant variables.

5. Discussion

The purpose of the present study was to identify patients with PTSD related to childhood abuse who are at risk of dropping out of treatment. Therefore, we applied machine learning techniques to analyze all available pre-treatment variables from a recent RCT. The best model was able to correctly predict dropout in 81.6% of the individuals. We consider our research endeavor to be a promising approach for identifying individuals at risk of dropout and, to this end, could be considered a first step in developing models for predicting patient groups that are at an elevated risk of dropout. This makes it possible to develop interventions that support these patients during treatment and prevent them from dropping out; for example, extra nursing support or treatment in clinical settings.

Our model evaluation indicated that the within-sample predictions were accurate based on three different measures (accuracy, AUC, and Brier score). To this end, the combination of pretreatment suicidality, low education, and non-use of benzodiazepines was found to be a risk factor for dropout. Interestingly, from their meta-analysis Guina et al. (59) concluded that the use of benzodiazepines is contraindicated for the treatment of PTSD. They found that the use of benzodiazepines is associated with poor treatment outcomes. To this end, the lack of experienced treatment benefits and the inability to use benzodiazepines as missed opportunities to ease and regulate feelings of tension may explain why patients discontinued therapy. Also, regarding the other predictors, it is conceivable that this could have prompt the patient to discontinue treatment early. This holds true, for instance, if one conceives suicidality as an indicator of feelings of entrapment or hopelessness, and a low level of education as a possible indicator of higher trait impulsivity or low frustration tolerance. However, before speculating too much on how these factors together might explain the increased risk of treatment dropout, more research is needed to first try and replicate these results and rule out the possibility of chance. Our method is the first step toward developing personalized medicine to prevent dropouts from PTSD treatment. Our findings further suggest that one of the ways to reduce dropout rates may be to intensify trauma treatments, as most patients in this study dropped out in an early stage (between the third and fourth treatment sessions) and other studies found that a more intensive PTSD treatment can lead to a lower dropout rate [for example, (60, 61)]. Clearly, replication studies are needed to provide reliable evidence for out-of-sample prediction accuracy for use in clinical practice.

The first strength of this study is that, to our knowledge, this is the first study to analyze predictors of dropout among patients with PTSD related to childhood abuse using machine learning, which has the advantage that it takes into account all possible pre-treatment variables. A second strength is that we handled an imbalanced dataset by applying the proven effective synthetic minority oversampling technique [SMOTE; (53)], which provides more opportunities to compare dropouts with completers, as in most studies, there is a large difference between the sample sizes of both groups. As in any study, some limitations of this study need to be noted. The first limitation is that the study sample was quite small, thereby limiting the possibility of distinguishing between the two treatment conditions (62). Future studies should focus on examining patient characteristics that might predict dropout by machine learning, using larger patient samples and including more types of trauma-focused therapies. A second limitation is the missing data for which we had to impute the data, leading to less reliable outcomes. However, the performance of our imputation method proved reliable (53), which was verified in this study. Third, we were not able to externally validate the data on an independent dataset to answer the question of whether this model leads to significant effects in new samples (63, 64). Before these outcomes are confirmed by external validation, the results cannot be generalized to clinical practice. Although we attempted to overcome this using 10-fold cross-validation, it is unknown how this model will perform on a truly independent dataset. For example, Isaksson et al. (65) argued that cross-validation was unreliable for classifying small samples.

In conclusion, this study identified a combination of variables predicting the dropout rate of patients with PTSD due to childhood abuse in trauma-focused treatments. A challenging task for future research is to examine whether these results can be replicated in larger patient samples. Another challenge is to examine these potential dropout predictors in a more profound way; although our study may help identify patients who are at risk of dropping out of therapy, the results do not reveal the mechanisms that explain the elevated risk. Experimental studies are required to elucidate the exact mechanisms involved, which could be fundamental for future preventive interventions. Developments in the field of machine learning are moving rapidly, and for follow-up research, it may be interesting to look at other models that use tidy modeling.

Funding

Funding for this study was provided by Stichting tot Steun VCVGZ, Dimence Mental Health Care, and the Dutch EMDR association.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

• 1.

  AlonsoJAngermeyerMCBernertSBruffaertsRBrughaTSBrysonHet al. Disability and quality of life impact of mental disorders in Europe: results from the European study of the epidemiology of mental disorders (ESEMeD) project. Acta Psychiatr Scand. (2004) 420:38–46. doi: 10.1111/j.1600-0047.2004.00329.x

  • CrossRef
  • Google Scholar

• 2.

  KesslerRC. Posttraumatic stress disorder: the burden to the individual and to society. J Clin Psychiatry. (2000) 61:4–12; discussion 13-4.

  • Google Scholar

• 3.

  PanagiotiMGoodingPATarrierN. A meta-analysis of the association between posttraumatic stress disorder and suicidality: the role of comorbid depression. Compr Psychiatry. (2012) 53:915–30. doi: 10.1016/j.comppsych.2012.02.009

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 4.

  SareenJCoxBJSteinMBAfifiTOFleetCAsmundsonGJ. Physical and mental comorbidity, disability, and suicidal behavior associated with posttraumatic stress disorder in a large community sample. Psychosom Med. (2007) 69:242–8. doi: 10.1097/PSY.0b013e31803146d8

  • CrossRef
  • Google Scholar

• 5.

  MavranezouliIMegnin-ViggersODalyCDiasSWeltonNStocktonSet al. Psychological treatments for post-traumatic stress disorder in adults: a network meta-analysis. Psychol Med. (2020) 50:542–55. doi: 10.1017/S0033291720000070

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 6.

  VarkerTJonesKAArjmandHAHintonMHilesSAFreijahIet al. Dropout from guideline-recommended psychological treatments for posttraumatic stress disorder: a systematic review and meta-analysis. J Affective Disorders Reports. (2021) 4:100093–3. doi: 10.1016/j.jadr.2021.100093

  • CrossRef
  • Google Scholar

• 7.

  UsmanMRehmanABakhtawarNBhattiABH. Prognosis of PTSD in treated vs. Non-Treated Groups J Pakistan Psychiatric Soc. (2015) 12:39.

  • Google Scholar

• 8.

  RinkJLipinskaG. Evidence of distinct profiles of ICD-11 post-traumatic stress disorder (PTSD) and complex PTSD in a south African sample. Eur J Psychotraumatol. (2020) 11:1818965. doi: 10.1080/20008198.2020.1818965

  • CrossRef
  • Google Scholar

• 9.

  World Health Organization. The ICD-11 for mortality and morbidity statistics. World Health Organization. Guidelines for the management of conditions specifically related to stress. Geneva: World Health Organization (2018).

  • Google Scholar

• 10.

  CloitreM.CourtoisC. A.FordJ. D.GreenB. L.AlexanderP. (et al.)BriereVan der HartO. (2012). The ISTSS expert consensus treatment guidelines for complex PTSD in adults. Available at: https://terrorvictimresponse.ca/wp-content/uploads/ISTSS-Expert-Concesnsus-Guidelines-for-Complex-PTSD-Updated-060315.pdf

  • Google Scholar

• 11.

  CloitreMKoenenKCCohenLRHanH. Skills training in affective and interpersonal regulation followed by exposure: a phase-based treatment for PTSD related to childhood abuse. J Consult Clin Psychol. (2002) 70:1067–74. doi: 10.1037/0022-006X.70.5.1067

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 12.

  KaratziasTMurphyPCloitreMBissonJRobertsNShevlinMet al. Psychological interventions for ICD-11 complex PTSD symptoms: systematic review and meta-analysis. Psychol Med. (2019) 49:1761–75. doi: 10.1017/S0033291719000436

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 13.

  CloitreMStovall-McCloughKCNoonerKZorbasPCherrySJacksonCLet al. Treatment for PTSD related to childhood abuse: a randomized controlled trial. Am J Psychiatr. (2010) 167:915–24. Available from: 10.1176/appi.ajp.2010.09081247. doi: 10.1176/appi.ajp.2010.09081247

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 14.

  OprelDACHoeboerCMSchoorlMDe KleineRACloitreMWigardIGet al. Effect of prolonged exposure, intensified prolonged exposure and STAIR+prolonged exposure in patients with PTSD related to childhood abuse: a randomized controlled trial. Eur J Psychotraumatol. (2021) 12:1851511. doi: 10.1080/20008198.2020.1851511

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 15.

  RaabeSEhringTMarquenieLArntzAKindtM. Imagery Rescripting versus STAIR plus imagery Rescripting for PTSD related to childhood abuse: a randomized controlled trial. J Behav Ther Exp Psychiatry. (2022) 77:101769. doi: 10.1016/j.jbtep.2022.101769

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 16.

  Van VlietNIHuntjensRJCVan DijkMKBachrachNMeewisseMLDe JonghA. Phase-based treatment versus immediate trauma-focused treatment for post-traumatic stress disorder due to childhood abuse: randomised clinical trial. British J Psychiatry Open. (2021) 7:E211. doi: 10.1192/bjo.2021.1057

  • CrossRef
  • Google Scholar

• 17.

  BissonJIEhlersAMatthewsRPillingSRichardsDTurnerS. Psychological treatments for chronic post-traumatic stress disorder. Systematic review and meta-analysis. Br J Psychiatry J Ment Sci. (2007) 190:97–104. doi: 10.1192/bjp.bp.106.021402

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 18.

  BradleyRGreeneJRussEDutraLWestenD. A multidimensional meta-analysis of psychotherapy for PTSD. Am J Psychiatry. (2005) 162:214–27. doi: 10.1176/appi.ajp.162.2.214

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 19.

  HembreeEAFoaEBDorfanNMStreetGPKowalskiJTuX. Do patients drop out prematurely from exposure therapy for PTSD?J Trauma Stress. (2003) 16:555–62. doi: 10.1023/B:JOTS.0000004078.93012.7d

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 20.

  ImelZELaskaKMJakupcakMSimpsonTL. Meta-analysis of dropout in treatments for posttraumatic stress disorder. J Consult Clin Psychol. (2013) 81:394–404. doi: 10.1037/a0031474

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 21.

  LewisCRobertsNPGibsonSBissonJI. Dropout from psychological therapies for post-traumatic stress disorder (PTSD) in adults: systematic review and meta-analysis. Eur J Psychotraumatol. (2020) 11:1709709. doi: 10.1080/20008198.2019.1709709

  • CrossRef
  • Google Scholar

• 22.

  ChekroudAMBondarJDelgadilloJDohertyGWasilAFokkemaMet al. The promise of machine learning in predicting treatment outcomes in psychiatry. World psychiatry: official J World Psychiatric Association (WPA). (2021) 20:154–70. doi: 10.1002/wps.20882

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 23.

  WeathersF. W.BlakeD. D.SchnurrP. P.KaloupekD. G.MarxB. P.KeaneT. M. (2013). The life events checklist for DSM-5 (LEC-5). National Center for posttraumatic stress disorder. Available online at: https://www.ptsd.va.gov/

  • Google Scholar

• 24.

  BeckATSteerRABrownGK. Manual for the Beck depression inventory II. San Antonio, TX: Psychological corporation (1996).

  • Google Scholar

• 25.

  Van VlietNIHuntjensRJCVan DijkMKde JonghA. Phase-based treatment versus immediate trauma-focused treatment in patients with childhood trauma-related posttraumatic stress disorder: study protocol for a randomized controlled trial. Trials. (2018) 19:1–10. doi: 10.1186/s13063-018-2508-8

  • CrossRef
  • Google Scholar

• 26.

  De JonghATen BroekeE. Handboek EMDR, een geprotocolleerde behandelmethode voor de gevolgen van psychotrauma [EMDR manual: A Protocolized treatment method for the consequences of Psychotrauma]. 6th ed. Amsterdam: Benelux Pearson Education BV (2013).

  • Google Scholar

• 27.

  WeathersFWBovinMJLeeDJSloanDMSchnurrPPKaloupekDGet al. The clinician-administered PTSD scale for DSM-5 (CAPS-5): development and initial psychometric evaluation in military veterans. Psychol Assess. (2017) 30:383–95. doi: 10.1037/pas0000486

  • CrossRef
  • Google Scholar

• 28.

  BoeschotenMAVan der AaNBakkerATer HeideFJJHoofwijkMCJongedijkRAet al. Development and evaluation of the Dutch clinician-administered PTSD scale for DSM-5 (CAPS-5). Eur J Psychotraumatol. (2018) 9:1546085. doi: 10.1080/20008198.2018.1546085

  • CrossRef
  • Google Scholar

• 29.

  FirstMBSpitzerRLGibbonMWilliamsJBWBenjaminLS. Structured clinical interview for DSM-IV Axis II personality disorders (SCID-II). Washington, DC: American Psychiatric Press (1997).

  • Google Scholar

• 30.

  WeertmanAArntzADreessenLVan VelzenCVertommenS. Short-interval test-retest interrater reliability of the Dutch version of the structured clinical interview for DSM-IV personality disorders (SCID-II). J Personal Disord. (2003) 17:562–7. doi: 10.1521/pedi.17.6.562.25359

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 31.

  CarlsonEBPutnamFW. An update on the dissociative experiences scale. Dissociation. (1993) 6:16–27.

  • Google Scholar

• 32.

  Van IJzendoornMHSchuengelC. The measurement of dissociation in normal and clinical populations: meta-analytic validation of the dissociative experiences scale (DES). Clin Psychol Rev. (1996) 16:365–82. doi: 10.1016/0272-7358(96)00006-2

  • CrossRef
  • Google Scholar

• 33.

  ScoboriaAFordJLinHFrismanL. Revision of the structured interview for disorders of extreme stress (SIDES): an exploratory and confirmatory factor analytic approach. Assessment. (2008) 15:404–25. doi: 10.1177/1073191108319005

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 34.

  FordJStocktonPKaltmanSGreenBL. Disorders of extreme stress (DESNOS) symptoms are associated with type and severity of interpersonal trauma exposure in a sample of healthy young women. J Interpers Violence. (2006) 21:1399–416. doi: 10.1177/0886260506292992

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 35.

  SpinazzolaJ. (2019). Structured interview for disorders of extreme stress (SIDES) & self-report inventory for disorders of extreme stress (SIDES-SR). Unpublished manuscript Available at https://complextrauma.org/wp-content/uploads/2019/03/SIDES-Manual-Spinazzola-2019.pdf

  • Google Scholar

• 36.

  HorowitzLMAldenLEWigginsJSPincusAL. Inventory of interpersonal problems. London: Psychological Corporation (2000).

  • Google Scholar

• 37.

  BarkhamMHardyGEStartupM. The IIP-32: a short version of the inventory of interpersonal problems. Br J Clin Psychol. (1996) 35:21–35. doi: 10.1111/j.2044-8260.1996.tb01159.x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 38.

  GratzKLRoemerL. Multidimensional assessment of emotion regulation and dysregulation: development, factor structure, and initial validation of the difficulties in emotion regulation scale. J Psychopathol Behav Assessment. (2004) 26:41–54. doi: 10.1023/B:JOBA.0000007455.08539.94

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 39.

  FoxHCAxelrodSRPaliwalPSleeperJSinhaR. Difficulties in emotion regulation and impulse control during cocaine abstinence. Drug Alcohol Depend. (2007) 89:298–301.

  • Google Scholar

• 40.

  JohnsonKAZvolenskyMJMarshallECGonzalezAAbramsKVujanovicAA. Linkages between cigarette smoking outcome expectancies and negative emotional vulnerability. Addict Behav. (2008) 33:1416–24. doi: 10.1016/j.addbeh.2008.05.001

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 41.

  FoaEBEhlersAClarkDMTolinDFOrsilloSM. The posttraumatic cognitions inventory (PTCI). Dev validation Psychol Assessment. (1999) 11:303–14. doi: 10.1037/1040-3590.11.3.303

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 42.

  Van EmmerikAAPSchoorlMEmmelkampPMGKamphuisJH. Psychometrische kenmerken van de Nederlandstalige Posttraumatische Cognities Inventarisatielijst (PTCI). Gedragstherapie [Behav Therapy]. (2007) 40:269–84.

  • Google Scholar

• 43.

  De BeursE. Brief symptom inventory. Handleiding. Leiden: PITS (2006).

  • Google Scholar

• 44.

  DerogatisLR. Brief symptom inventory. Baltimore, MD: Clinical (1975).

  • Google Scholar

• 45.

  De BeursEZitmanFG. De brief symptom inventory (BSI): De betrouwbaarheid en validiteit van een handzaam alternatief voor de SCL-90 [reliability and validity of a handy alternative for the SCL-90]. Maandblad Geestelijke Volksgezondheid [monthly magazine for mental health]. (2006) 61:120–41.

  • Google Scholar

• 46.

  RStudio Team. RStudio: integrated development for R. Oxford: Oxford University Press (2020).

  • Google Scholar

• 47.

  CohenZDKimTTVanHLDekkerJJDriessenE. A demonstration of a multi-method variable selection approach for treatment selection: recommending cognitive–behavioural versus psychodynamic therapy for mild to moderate adult depression. Psychother Res. (2019) 1–14:1563312. doi: 10.1080/10503307.2018.1563312

  • CrossRef
  • Google Scholar

• 48.

  StekhovenDJBühlmannP. Missforest—non-parametric missing value imputation for mixed-type data. Bioinformatics. (2012) 28:112–8. doi: 10.1093/bioinformatics/btr597

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 49.

  TangFIshwaranH. Random Forest missing data algorithms. Stat Analysis Data Mining. (2017) 10:363–77. doi: 10.1002/sam.11348

  • CrossRef
  • Google Scholar

• 50.

  BoxGCoxD. An analysis of transformations. J R Statistical Soc Series B (Methodological). (1964) 26: 211–52. doi: 10.1111/j.2517-6161.1964.tb00553.x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 51.

  KuhnM. Building predictive models in R using the caret package. J Stat Softw. (2008) 28:1–26. doi: 10.18637/jss.v028.i05

  • CrossRef
  • Google Scholar

• 52.

  JapkowiczNStephenS. The class imbalance problem: a systematic study. Intelligent Data Analysis. (2002) 6:429–49. doi: 10.3233/IDA-2002-6504

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 53.

  ChawlaNVBowyerKWHallLOKegelmeyerWP. SMOTE: synthetic minority over-sampling technique. J Artif Intell Res. (2002) 16:321–57. doi: 10.1613/jair.953

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 54.

  FriedmanJHastieTTibshiraniR. Regularization paths for generalized linear models via coordinate descent. J Stat Softw. (2010) 33:1. PMID:

  • Pubmed Abstract
  • Google Scholar

• 55.

  KuhnMJohnsonK. Applied predictive modeling. New York: Springer (2013).

  • Google Scholar

• 56.

  JamesGWittenDHastieTTibshiraniR. An introduction to statistical learning with applications in R. New York: Springer (2003).

  • Google Scholar

• 57.

  GreenwellBMBoehmkeBC. Variable importance plots—an introduction to the vip package. The R J. (2020) 12:343–66. doi: 10.32614/RJ-2020-013

  • CrossRef
  • Google Scholar

• 58.

  BrierGW. Verification of forecasts expressed in terms of probability. Mon Weather Rev. (1950) 78:1–3. doi: 10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 59.

  GuinaJRossetterSRDeRhodesBJNahhasRWWeltonRS. Benzodiazepines for PTSD: a systematic review and meta-analysis. J Psychiatr Pract. (2015) 21:281–303. doi: 10.1097/PRA.0000000000000091

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 60.

  HoppenTHKipAMorinaN. Are psychological interventions for adult PTSD more efficacious and acceptable when treatment is delivered in higher frequency? A meta-analysis of randomized controlled trials. J Anxiety Disord. (2023) 95:102684. doi: 10.1016/j.janxdis.2023.102684

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 61.

  Van WoudenbergCVoorendonkEMBongaertsHZoetHAVerhagenMLeeCWet al. Effectiveness of an intensive treatment programme combining prolonged exposure and eye movement desensitization and reprocessing for severe post-traumatic stress disorder. Eur J Psychotraumatol. (2018) 9:1487225. doi: 10.1080/20008198.2018.1487225

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 62.

  LuedtkeASadikovaEKesslerRC. Sample size requirements for multivariate models to predict between-patient differences in best treatments of major depressive disorder. Clin Psychol Sci. (2019) 7:445–61. doi: 10.1177/2167702618815466

  • CrossRef
  • Google Scholar

• 63.

  BleekerSMollHSteyerbergEDondersADerksen-LubsenGGrobbeeDet al. External validation is necessary in prediction research: a clinical example. J Clin Epidemiol. (2003) 56:826–32. doi: 10.1016/S0895-4356(03)00207-5

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 64.

  Van BronswijkSCBruijniksSJLorenzo-LuacesLDerubeisRJLemmensLHPeetersFPet al. Cross-trial prediction in psychotherapy: external validation of the personalized advantage index using machine learning in two Dutch randomized trials comparing CBT versus IPT for depression. Psychother Res. (2021) 31:78–91. doi: 10.1080/10503307.2020.1823029

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 65.

  IsakssonAWallmanMGöranssonHGustafssonMG. Cross-validation and bootstrapping are unreliable in small sample classification. Pattern Recogn Lett. (2008) 29:1960–5. doi: 10.1016/j.patrec.2008.06.018

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 66.

  FoaEBRiggsDSDancuCVRothbaumBO. Reliability and validity of a brief instrument for assessing posttraumatic stress disorder. J Trauma Stress. (1993) 6:459–73. doi: 10.1002/jts.2490060405

  • Pubmed Abstract
  • CrossRef
  • Google Scholar