This review was prospectively registered on the International Prospective Register of Systematic Reviews (PROSPERO) under the registration number: CRD42020159702. The protocol for this systematic review and meta-analysis has been published previously (Perez et al., 2021) and uploaded in PDF format to PROSPERO.

We considered all EEG-NFB reports involving humans with at least one ID diagnosis per the Diagnostic and Statistical Manual of Mental Disorders [DSM; (American Psychiatric Association [APA], 2013)] or the International Classification of Diseases [ICD; (World Health Organization [WHO], 2018)] with no exclusion by language, locality, ethnicity, age, or sex. Regarding outcome assessments, all reports were required to include data from at least one clinical rating scale assessing one or more core symptoms of the disorder(s) under investigation. To minimize bias and control for non-specific effects, we included only randomized, double-blind (participants and outcome assessors), sham/placebo-controlled (i.e., feedback contingent on either a random signal, the activity from a different person’s brain, or a signal from the participant’s own brain derived from a region unrelated to the condition under study) reports. Concomitant interventions were permitted provided they were identical for both active and sham groups.

Studies eligible for review were identified in a literature search from earliest dates within multiple databases (Table 1). Of note, we decided to remove database limit options from our protocol in order to broaden the scope of our searches. Databases were last searched on 23 November 2021. Considering the known importance of including unpublished data in systematic reviews (Trespidi et al., 2011), we also searched various clinical trial registries including ClinicalTrials.gov¹, the World Health Organization’s International Clinical Trials Registry Platform (ICTRP)², and the Australia New Zealand Clinical Trials Registry (ANZCTR)³ to identify completed, unpublished trials. Notably, the ICTRP indexes trial registrations from 17 registries around the world. All registries were last accessed on 23 November 2021. Additionally, reference lists of included articles and relevant systematic reviews were manually screened to identify additional studies.

Literature search strategies were developed using medical subject heading (MeSH) and text words related to IDs and neurofeedback. The search strategies were developed by author TMP with guidance from the University of Otago’s Health Sciences librarian. A detailed account of the search strategy for each database and registry can be found in Supplementary Material.

TMP collated the list of possible records for inclusion and exported them from each database to EndNote (version X9.2) where duplicates were located using EndNote’s duplicate identification strategy (i.e., identifying references in a library of the same reference type with matching Author, Year, and Title fields) and then removed manually. Two independent reviewers (TMP and JM) screened titles and abstracts for eligibility. In cases of disagreement, consensus on articles to assess for eligibility was reached by discussion between TMP and JM. When disagreements couldn’t be resolved, a third team member (DBA) was enlisted to make the final decision. TMP and JM then independently assessed full-text reports appearing to meet the inclusion criteria or when there was any uncertainty. Again, in cases of disagreement, consensus on inclusion/exclusion was reached via discussion and, when needed, DBA was consulted to make the final decision. When necessary to resolve questions regarding eligibility, TMP sought additional information from study authors via a maximum of three electronic (i.e., email and/or ResearchGate) requests. Notably, two non-English language reports [(Biriukova et al., 2005; Eskandari et al., 2014)] were translated during our full-text eligibility assessments. One report [(Biriukova et al., 2005)] was entered into Google Translate by our team and the subsequent output was validated by the study’s last author. The other (Eskandari et al., 2014) was translated by a native speaker here at the University of Otago. Reasons for excluding trials were recorded. Neither TMP nor JM were blinded to the journal titles, trial authors, or institutions.

Data was extracted by TMP and independently verified by JM. In cases of disagreement, consensus was reached via discussion. For all three included reports, trial authors were successfully contacted via email for clarification of trial details and/or to obtain missing data.

Our primary outcome of interest was clinician ratings or self/parent/teacher reports of ID-related symptomatology. In trials incorporating multiple domains (e.g., clinician ratings and self/parent/teacher reports), a single scale was selected based on a hierarchy (i.e., clinician > self > parent > teacher). When multiple rating scales within a given domain were assessed, the validated scale querying the most core/central feature(s) of the condition under study (as determined by our content expert PG) was selected. In cases where multiple values for a single scale (i.e., total vs. sub-scale scores) were assessed, the total scores were used. Notably, as it has been postulated that longer-term outcomes may help to clarify the issue of specificity (Van Doren et al., 2019), in trials with multiple post-treatment data collection time-points, scores obtained furthest from treatment termination were given preference. To date, standard EEG-NFB protocols have not been established for the treatment of IDs (Banerjee and Argáez, 2017), therefore, no protocols were excluded.

The risk of bias for each eligible report was assessed using the Cochrane Risk of Bias tool version 2 (RoB 2.0) for randomized trials which covers 5 domains (domain 1: risk of bias arising from the randomization process; domain 2: risk of bias due to deviations from the intended interventions; domain 3: risk of bias due to missing outcome data; domain 4: risk of bias in measurement of the outcome; domain 5: risk of bias in the selection of the reported result) (Sterne et al., 2019). Two reviewers (TMP and JM) independently applied the tool to each trial and recorded supporting information/justifications for risk of bias judgments (low, some concerns, high) in each domain. Further, an overall summary risk of bias judgment (low, some concerns, high) was made for each report determined by the highest risk of bias level across all 5 domains. All decisions were guided by the published criteria for judging the risk of bias (Higgins et al., 2021). If there was insufficient detail reported in the study, the original study investigators were contacted via email for more information. Disagreements were resolved in discussion and, in cases where consensus was not reached, a third team member (DBA) was enlisted to make the final decision.

Because the included trials utilized different measurement scales to assess clinical outcomes, standardized mean differences (SMDs) and 95% confidence intervals (CIs) were calculated using Hedges’ adjusted g in RevMan (version 5.4.1) and presented in a forest plot. In our forest plot, an SMD of 0 is represented by a black vertical line with negative (left-sided) and positive (right-sided) values favoring sham and active groups, respectively. Each trial’s SMD point estimate and 95% CI are represented by a green vertical dashes and bilateral black horizontal lines, respectively. The SMD expresses the size of the intervention effect relative to the variability observed under the assumption that between-study standard deviation variations reflect differences in measurement scales rather than variability in the study populations or reliability in outcome measures (Deeks et al., 2021). SMDs of 0.2, 0.5, and 0.8 are generally interpreted to reflect small, medium, and large effects sizes, respectively (Cohen, 1988).

The limited overall number of eligible reports (and participants), large degree of inter-trial heterogeneity, and restricted span of ID populations with published/posted outcome data (i.e., PTSD and OCD) precluded a robust quantitative synthesis investigating the potential for EEG-NFB specificity in the treatment of IDs.

To assess for reporting biases (e.g., selective non-publication and selective non-reporting of results), two independent reviewers (TMP and JM) searched various registries for unpublished trials as well as to compare published trial report outcomes to outcomes specified in their registrations to help guide ratings for domain 5 (i.e., risk of bias in the selection of the reported result) of the RoB 2.0. When registrations were unavailable, we compared the congruency of the section “Materials and Methods” and section “Results” for each trial. Disagreements were resolved in discussion and, when necessary, DBA was consulted to make the final decision.

Two independent reviewers (TMP and JM) assessed the certainty in the body of the evidence as it related to the trials that contributed data to the meta-analysis using the Grading of Recommendations, Assessment, Development and Evaluations (GRADE). A certainty ranking of high (there is a lot of confidence that the true effect lies close to that of the estimated effect), moderate (the true effect is probably close to the estimated effect), low (the true effect might be markedly different from the estimated effect.), or very low (the true effect is likely to be substantially different from the estimated effect) was assigned by the software based on the reviewers assignments across a number of domains including study design (i.e., randomized trial or observational study), risk of bias (not serious, serious, or very serious), inconsistency (not serious, serious, or very serious), indirectness (not serious, serious, or very serious), imprecision (not serious, serious, or very serious). We used the methods and recommendation described in the GRADE handbook (Schünemann and Oxman, 2013). Disagreements were resolved in discussion and, when necessary, DBA was consulted to make the final decision. A GRADE profile (v2) was generated using GRADEpro GDT software (Evidence Prime, 2020). Where necessary, we provided explanations for our assignments using footnotes to aid the reader’s understanding of the results.

Our search uncovered 8,887 records in databases, registries, and relevant reference lists (Figure 1). After duplicates were removed, we screened 6,405 records, from which we reviewed 17 full-text documents. Ultimately, four reports met our eligibility criteria.

Following full-text assessments, we excluded 13 studies from our review (Watson and Herder, 1980; Rice et al., 1993; Biriukova et al., 2005; Iranian Registry of Clinical Trials, 2008; Choi et al., 2011; Eskandari et al., 2014; Sadjadi and Hashemian, 2014; Hashemian and Sadjadi, 2015; Mennella et al., 2017; Noohi et al., 2017; Viereck et al., 2017; Esmaeeli, 2021; Hou et al., 2021) for various reasons (Figure 1) which included a lack of randomization (Watson and Herder, 1980), double-blinding ((Rice et al., 1993; Biriukova et al., 2005; Iranian Registry of Clinical Trials, 2008; Hou et al., 2021), sham EEG-NFB controls (Choi et al., 2011), ID diagnosis (Mennella et al., 2017) and (Viereck et al., 2017), or missing details regarding the control group and/or blinding (Eskandari et al., 2014; Sadjadi and Hashemian, 2014; Hashemian and Sadjadi, 2015; Noohi et al., 2017; Esmaeeli, 2021). With respect to these latter five trials excluded due to missing information, one failed to provide the trialist’s contact information (Esmaeeli, 2021), whereas, for the other four, we made multiple attempts to contact the authors via email and/or ResearchGate, however, they failed to respond to our requests.

An detailed overview of the eligible trials (ClinicalTrials.gov, 2000a; Nederlands Trial Register, 2004; Kopřivová et al., 2013; Nicholson et al., 2020) is presented in Table 2. As can be seen, there is significant clinical (e.g., population) and methodological (e.g., clinical outcome scale) heterogeneity across trials.

We used the RoB 2.0 to assess risk of bias for the eligible trials with posted/published outcome data (Figure 2). The overall risk of bias was deemed moderate (i.e., “some concerns”) for two of the trials (Kopřivová et al., 2013; Nicholson et al., 2020) and high for the third (ClinicalTrials.gov, 2000a) suggesting some methodological heterogeneity across trials. Concerns included a lack of pre-registered protocols and/or analysis plans with which to compare the final reports and outcomes.

Figure 3 displays summary statistics [CFB mean and SD, sample size (Total)] for the sham and genuine EEG-NFB groups along with the SMD and 95% CI of the continuous outcome (reduction in internalizing symptoms) for each study. Of note, one eligible trial [i.e., (Nederlands Trial Register, 2004)] is absent due to a failure to post/publish results. Unfortunately, the lead trialist was unwilling to share unpublished data. For all trials with final results, trialists were successfully contacted via email for additional data not provided in the original report. Baseline outcome measures were reported to be equivalent between genuine and sham groups in the two published studies (Kopřivová et al., 2013; Nicholson et al., 2020) and were not reported in the unpublished trial (ClinicalTrials.gov, 2000a). Across trials, point estimates consistently demonstrated small to moderate effect sizes favoring genuine over sham EEG-NFB, however, in all cases 95% CIs were relatively wide and spanned the null line. Specifically, SMDs [95% CIs] were 0.42 [−0.52, 1.37] (Kopřivová et al., 2013), 0.46 [−0.11, 1.03] (ClinicalTrials.gov, 2000a), and 0.33 [−0.33, 0.99] (Nicholson et al., 2020). The two published trials [i.e., (Nicholson et al., 2020) and (Kopřivová et al., 2013)] did assess for differential targeted EEG-learning with group point estimates suggesting a trend toward greater improvements (i.e., reductions) in the trained EEG variable(s) in the genuine EEG-NFB groups relative to sham, however, correlation analyses between changes in the targeted EEG variable(s) and clinical outcome measures were not performed.

Overall, we feel the risk of selective outcome reporting was low as all trials’ methods and results sections were congruent and well-accepted scales were generally incorporated for the conditions under study, however, there remains a risk of selective analysis reporting due to a lack of pre-specified analysis plans per trial registry records or published protocols. Additionally, we have concerns surrounding publication bias as we were unable to include the outcome data from one of the completed trials meeting our eligibility criteria [i.e., (Nederlands Trial Register, 2004)] because they have yet to be made publicly available despite being “terminated [in 2017] because of disappointing mid trial results” (personal communication with the lead trialist, 7 July 2020).

As seen in the GRADE profile (Table 3), although the certainty of the evidence is initially rated “high” due to the incorporation of the three RCTs with published/posted outcomes, it was subsequently downgraded three steps to “very low” due to (1) “serious” concerns surrounding the cumulative risk of bias, (2) “serious” concerns regarding the imprecision (i.e., wide 95% CIs), and (3) “strongly suspected” publication bias (i.e., no published/posted outcomes from the fourth eligible trial).