Expanding the reach of evidence-based mental health interventions to private practice: Qualitative assessment using a policy ecology framework

Introduction

Evidence-based interventions (EBIs) are infrequently used in routine clinical care settings. Prior research has largely focused on strategies to improve the implementation of EBIs in publicly-funded mental health settings. However, less is known about the implementation of EBIs in private practice mental health settings, where there is also a research-practice gap. Private practice settings represent a large sector of the mental health workforce (1), including a plurality (44.8%) of psychologists (2), and serve a large portion of people with private insurance. Treatment access disparities are particularly wide among individuals with public insurance, but privately-insured individuals also face significant barriers to accessing care (3, 4). Estimates indicate that ~40% of youth with private insurance do not receive needed mental health services (5, 6) and that these families face significant barriers to receiving mental health care (7, 8). The number of unmet mental health needs, especially for anxiety and depression, has only been exacerbated as a result of the COVID-19 pandemic (9). Thus, identifying strategies to increase EBI use in private practice settings may improve access to care for a large portion of individuals in need of mental health services.

Existing research in private practice settings provides some evidence that there are unique challenges to EBI implementation that may be specific to this setting. For instance, in one study, private practice clinicians were found to hold more negative global attitudes toward EBIs than those working in public outpatient settings (10). One interpretation the authors provide for this finding is that private practice clinicians may have chosen this setting to allow them more autonomy and fewer mandates related to the types of interventions they are expected to deliver. Another study, focused on evidence-based assessment, found lower rates of evidence-based assessment use among private practice clinicians compared to clinicians in other settings (11). Supervisory practices reflect these general trends, with fewer references to EBIs and less supervision time spent discussing them among clinicians working in private practice settings (12). Prior studies have not specifically examined determinants of EBI use in private practice settings, which is a necessary first step to inform future work focused on supporting EBI implementation in this setting.

One intervention that is particularly underutilized is exposure therapy for anxiety, which has strong evidence for its efficacy but is rarely used in practice settings (13). A study of private practice clinicians working in Germany found that issues related to the practicability of exposure (e.g., feasibility of conducting exposures in session), negative beliefs about exposure, and distress for the therapist in delivering exposure therapy were barriers to its delivery (14). These findings are consistent with previous research indicating that therapists' negative beliefs about exposure are a primary barrier to its delivery (15–17). Although some interventions have been developed to directly address these negative beliefs (18), insufficient access to training in exposure is another commonly endorsed barrier (19). Even when therapists do receive training, actual use of exposure remains somewhat limited (20). Receiving consultation after training is one promising method that has been shown to increase use of exposure-based treatments (21). However, therapists in private practice have been found to use sub-optimal exposure techniques, such as assigning client self-directed exposure rather than conducting exposures in session (22). This may be due to difficulty accessing ongoing consultation, or a result of various other factors such as competing demands and limited organizational support.

Existing research on determinants of EBI use more broadly have identified an array of clinician- and organization-level barriers that interfere with implementation (23–26). For instance, organizational factors, such as proficient culture, leadership, and presence of champions influence the implementation of EBIs (27). Clinician-level barriers have also been identified as predicting EBI implementation, such as competing responsibilities and lack of training (28, 29). Even when clinicians do access training, one-time or even intensive trainings are not sufficient to lead to sustainable behavior change (30–32). In addition to these clinician- and organization-level considerations, there are also even broader contextual determinants that influence EBI implementation. In their “policy ecology” framework, Raghavan et al. (33) highlight the importance of considering multiple levels of the ecology of implementation, ranging from the organizational context to the larger social context in which implementation takes place, to ensure successful implementation of EBIs. This framework incorporates factors such as incentive strategies for policymakers and payers to improve EBI implementation. According to the policy ecology framework, efforts to sustainably implement EBIs will require implementation strategies that expand beyond clinical factors and include a focus on systemic or ecological determinants. Such determinants have been increasingly studied in public sectors [e.g. (34)], but have not been specifically examined in private practice settings.

The policy ecology framework highlights four key levels that influence implementation, including the organizational context, the agency (regulatory) context, the social context, and the political context, as shown in Figure 1. The organizational context refers to the mental health practice in which treatment is delivered, which for private practitioners may consist of one individual or a large group practice. Specific considerations within organizations include: (1) the costs of delivery that organizations accrue (e.g., for ongoing supervision), which are typically not reimbursed, as well as (2) continuing education, which is often provided through or subsidized by organizations. Policy levers at the organizational level may include adjusting state licensing boards' requirements and expectations regarding continuing education to emphasize EBIs. However, therapists who work in private practice may face financial barriers to accessing continuing education and place less of an emphasis at the organizational level on implementing new EBIs (35). For instance, in an assessment of psychologists' use of outcome measures (36), clinicians in private practice settings were more likely to use outcome measures for clinical purposes, whereas clinicians in other settings (e.g., schools, community mental health, outpatient clinics) endorsed using these measures for clinical and business reasons (i.e., requirements by the work setting). Understanding how contexts for private practice clinicians may differ from other settings can inform the tailoring of implementation strategies.

FIGURE 1

Figure 1. A policy ecology of implementation. Reproduced from (33); distributed under the creative commons attribution license.

The second level of the policy ecology is the agency (regulatory) context, which refers to payers (i.e., insurance companies) or states. Specific considerations within this level include: (1) fiscal incentives, such as pay-for-performance and public recognition for providing EBIs; and (2) payer restrictions, such as requirements for prior authorization or limits to the number and frequency of therapy sessions. In contrast with clinicians who work in publicly-funded agencies, private practice clinicians in the United States are not required to accept insurance. This is likely to have implications for how private practice clinicians interface with payers, including having more flexibility with the decision about whether to panel with insurance companies. If payers are to have an influence on the practice of private practitioners, they will likely need to incentivize providers to accept health insurance plans via strategies such as higher reimbursement rates, coverage of more sessions, and reduced administrative burden.

The political context level refers to legislative and advocacy efforts that may affect EBI use. From a consumer perspective, this may include efforts related to improving accessibility and affordability of mental health treatment for consumers. In public agencies, research has examined how policy mandates related to EBI use influence clinicians' behavior. In general, policy makers are urged to avoid strict policy mandates and instead consider how to balance EBI expectations with available support from a system (33, 34). One study demonstrated modest gains in cognitive behavioral therapy use following a system-wide initiative focused on supporting EBI implementation, a finding that was moderated by organizational culture (37). Given that many policy mandates may be less applicable to private practice clinicians, especially those who do not accept insurance, other issues at the political level may have more relevance. For instance, EBI training is not consistently required during pre-service training for doctoral and master's level clinicians (38). Existing accreditation standards for graduate programs may make it challenging to prioritize EBI training (39, 40). Thus, efforts focused on changing accreditation standards and increasing the emphasis on EBI-based training in pre-service settings may be one relevant lever at the political level that could influence private practice clinicians' behavior (39).

Finally, the social context level refers to public perceptions of EBIs, especially related to stigma and discrimination, as well as consumer demand for EBIs. A systematic review of parents' perceptions of barriers and facilitators to seeking mental health treatment demonstrated that parents frequently reported a lack of knowledge about where or how to seek treatment (41). Direct-to-consumer marketing is one approach that has been proposed to address stigma and increase consumer demand for EBIs (42), which in turn may motivate private practice clinicians to provide them.

Although the levels of the policy ecology framework were originally developed with public mental health settings in mind, the present study assesses how each of its levels might also apply to private practice mental health settings. To date, incentive strategies beyond providing training and technical assistance are rarely used (43) and limited research has examined the barriers to EBI adoption in private practice settings. The present study uses the policy ecology framework (33) to assess private practice therapists' perceptions of multi-level contextual factors that influence their use of EBIs broadly, and exposure therapy specifically. Exposure therapy was selected as a specific example of an EBI given that it has been one of the most difficult to implement (13, 22). Although an array of intervention-specific clinician- and organization-level barriers to exposure implementation have been identified, there may also be opportunities to engage implementation strategies at a broader ecological level to increase adoption and sustained use of exposure therapy. The primary aim of the present study was to conduct qualitative interviews with private practice mental health clinicians to identify incentive structures that may affect EBI implementation, with a particular emphasis on exposure therapy for anxiety disorders. Mixed methods analyses were used to examine differences between therapists in solo vs. group private practice settings, given that each practice structure is likely to present unique implementation considerations. In addition, we examined whether qualitative responses differed based on therapists' perceived level of organizational support for implementing exposure therapy.

Method

Participants

Participants include therapists (N = 20) with (1) an advanced degree in a mental health field who (2) work in private practice settings. Given that exposure therapy is a particularly difficult-to-implement EBI, we were interested in understanding responses from clinicians who had and had not sought out explicit training in exposure therapy. Thus, purposive sampling was used to identify approximately equal numbers of participants with and without previous training in exposure therapy. The final sample included n = 9 (45%) therapists who had previously participated in full day or longer training focused on exposure therapy and n = 11 (55%) who had never attended an exposure-focused training.

Measures

De-identified survey data were collected and stored using Research Electronic Data Capture (REDCap), a HIPAA compliant web-based survey platform (44, 45).

Demographics form

A demographics questionnaire assessed participants' age, gender, race, and education. It also assessed topics related to the participants' work setting, theoretical orientation, and level of familiarity with exposure therapy.

Organizational innovation-specific capacity for exposure (OISCE)

The OISCE (23) assesses therapists' perceptions of organizational policies and procedures for supporting exposure use in their clinical setting with responses rated from 0 (not at all) to 4 (extremely). The measure includes 17 items that cut across five domains of interest, including supervisory support, collaboration, organizational policies, resources, and emphasis on exposure.

Qualitative interviews

A semi-structured qualitative interview guide (see Supplementary Material) included prompts about participants' experience using EBIs in their practice, as well as their training and consultation needs for EBIs. Given that exposure therapy is a particularly underused EBI with strong evidence for its efficacy, therapists were also specifically asked about their use of and training in exposure therapy. In addition, participants were asked about barriers and facilitators to using exposure and other EBIs. Questions were developed based on the policy ecology framework (33), which includes factors at the organizational, agency, political, and social context levels.

Procedure

All study procedures were approved by the Lifespan Institutional Review Board (IRB). Participants with previous training were recruited by contacting therapists who completed prior studies at the Pediatric Anxiety Research Center (PARC) and agreed to be contacted for future research. Additional participants with and without previous training were recruited by sending emails to local (New England) professional listservs and contacting private practice clinicians on referral lists maintained at Bradley Hospital. Local private practice agencies who had publicly available contact information were also emailed and asked to distribute information about the survey to providers in their practice. Finally, participants who completed the qualitative interview were asked if they were willing to forward information about the study to their colleagues as part of a “snowball” recruitment method.

Recruitment of participants was informed by the Dillman Tailored Design Survey Method (46, 47). Initial contact included a phone call or email to potential participants, as well as two follow-up emails sent 1 week after the initial contact requesting participation in the study. Participants who indicated interest in the study were emailed a REDCap link to complete online quantitative measures prior to qualitative interviews. Then, participants were contacted to complete the qualitative interview. Our recruitment methods do not allow us to estimate how many people may have received information about the study. However, a total of 39 people initiated the survey questionnaires, and 24 of them completed the survey measures and provided their contact information. Four of those participants were either not available or did not respond to attempts to schedule the qualitative interview. All 20 participants who were scheduled for qualitative interviews completed them as scheduled. Interviews were audio recorded and conducted one-on-one with participants by phone until thematic saturation (i.e., no emergence of new concepts) (48) was reached. Completion of qualitative and quantitative measures took a total of ~60–90 min. There was no additional follow-up with participants after completion of these interviews. Interviews were conducted by a female postdoctoral fellow (HEF) and a female advanced doctoral student who did not have previous relationships with the study participants. Participants were told that interviewers were affiliated with PARC and interested in understanding factors that affect clinicians' use of evidence-based interventions, including exposure therapy. The first author (HEF) has previous experience leading qualitative studies and provided training to the graduate student to conduct interviews. Interviews were transcribed using NVivo transcription services, checked by undergraduate research assistants, and spot checked by the first author (HEF).

Qualitative coding and data analysis

Qualitative interviews were analyzed in NVivo using content analysis (49) informed by Raghavan and colleagues' (33) policy ecology Framework. A priori codes included: (1) organizational context and its two subcodes: costs of delivery and continuing education; (2) agency (regulatory) context and its two subcodes: fiscal incentives and payer restrictions; (3) social context; and (4) political climate. Coders included the first author (HEF) and a research assistant (LM) who received training in qualitative analysis from the first author. The coders collaboratively reviewed six transcripts to inform their iterative development of a codebook. They then independently applied codes to two transcripts to determine initial interrater reliability (kappa = 0.64). Finally, both coders independently coded transcripts with 20% overlap (n = 4 transcripts) to assess final reliability (kappa = 0.82). At each stage of coding, disagreements were resolved through discussion and consensus.

Thematic analysis of codes was conducted in Excel by the same authors who completed coding of transcripts (HEF and LM). Codes were examined collaboratively to identify patterns and key themes through discussion. Mixed methods analyses were used to integrate quantitative and qualitative findings using a convergent design in which quantitative and qualitative data were merged (50). Quantitative data (i.e., solo vs. group practice; high vs. low organizational support for exposure based on a median split on the OISCE) was entered into NVivo as attributes of each participant and used to categorize and compare themes among subgroups. Once exported into Excel, content analysis was used to identify themes for each code. Then, coders collaboratively created brief written summaries for each theme and for each quantitative variable (i.e., solo private practice, group private practice, high organizational support, low organizational support). These summaries were compared to identify differences in qualitative responses for each quantitative variable. All qualitative analyses adhered to consolidated criteria for reporting qualitative research (COREQ) (51).

Results

Participants were predominantly female (n = 18, 90%) and white (n = 18, 90%). The majority had doctorate degrees (n = 13, 65%) and worked in a solo private practice (n = 11, 55%). Scores on the OISCE indicated overall low levels of organizational support for exposure therapy (M = 1.71, SD = 0.93). Clinicians with high organizational support for exposure had scores above the median of 1.85, indicating responses of “somewhat” or above. See Table 1 for additional demographic information. Table 2 includes an overview of study codes, themes, and example quotes, as well as suggested implementation strategies to address barriers related to each theme.

TABLE 1

Table 1. Participant demographic characteristics (N = 20).

TABLE 2

Table 2. Reported barriers and facilitators and potential implementation strategies.

Organizational context

Clinicians described varying levels of organizational support for EBI use. Among private practice clinicians, the “organizational context” refers to the individual practitioner, as well as the setting (e.g., geographical location, office space, proximity to colleagues) in which they work. Organizations that placed more emphasis on EBI use were described as facilitating EBI implementation. Specifically, organizations that offered EBI-consistent in-house training (e.g., through case conferences) fostered a “push for evidence-based interventions” (P1226). Many clinicians described working in organizations that support the implementation of cognitive behavioral therapy (CBT) principles while also emphasizing that “the patient's needs come first” (P1004) and that CBT should be applied flexibly. Some clinicians described their organizations as supporting EBI use conceptually, but not supporting specific aspects of EBIs (e.g., inability to hold sessions in the community or travel with clients during sessions). Supervisors were generally described as being supportive of EBI use. Similarly, clinicians endorsed that the presence of colleagues who also use EBIs can support EBI implementation.

Clinicians identified some organizational contextual factors that were specific to implementing exposure therapy. For instance, clinicians mentioned that having colleagues who understand and can help with exposure can facilitate exposure implementation. One clinician described, “when I had to do an exposure or when I've been asked to be part of an exposure, people in the office are really willing and able to be a part of that” (P1616). Clinicians also reported that space constraints affect their ability to conduct exposure work in the office. One clinician described, “I think that exposure therapy is best done if the clinician is able to go out with the person and there's more intensive therapy or if it's a home-based program. I don't feel like in private practice I would feel as comfortable” (P2605). Clinicians described that having businesses nearby that are willing to assist with exposures can facilitate the completion of exposure work outside of the office. Finally, clinicians noted that it is helpful to have knowledge of local providers who use exposure if they need to refer to a specialist, but that referral options are often limited [e.g., “There is nobody in (geographic region) who does hard core exposure stuff” (P1058)].

Mixed methods analyses

There were notable differences in descriptions of organizational support for EBI use between clinicians in solo vs. group practice. As might be expected, clinicians working in group practices more often reported that supervisor and colleague support are available to them, whereas clinicians working in solo practice hardly mentioned supervisor support for EBI use and varied more in their reports of the availability of colleagues who support EBI use. Clinicians in group practice more often mentioned that being busy (e.g., scheduling back-to-back sessions) makes it difficult to leave the office for exposure. However, as one clinician stated, “having a lot more people around, in my opinion, makes it easier to do exposure” (P2621). Clinicians working in solo practice more often described leaving the office to meet with clients, although they still cited some barriers and restrictions. One clinician in solo practice mentioned that not having other clinicians in the office is a barrier to the completion of social exposures. Although they navigate around this by identifying social exposures that can be completed with just the clinician and the patient, they said, “it would be great to have other people, unfamiliar people, that [the patient] could interact with” (P4610).

There were also differences in descriptions of organizational support for EBI use for clinicians with high and low organizational support. Clinicians with high organizational support provided examples of the ways in which their organization supported EBI use, including providing “in-house” continuing education and consultation on EBIs, as described in more detail below. Few such examples were given by clinicians with low organizational support. Clinicians with high organizational support noted the importance of supervisors for support and education in EBIs, whereas clinicians with low organizational support did not often discuss the role of supervisor support in EBI implementation. Clinicians with high organizational support spoke about how their colleagues give “a lot of feedback that is often guided by research or something people have done in their own practice” (P1226). They also described having colleagues to help with in-office exposures. In contrast, clinicians with low organizational support spoke about their setting feeling like “you're just here on an island” (P2605).

Continuing education

Clinicians described that the availability of organization-sponsored training (e.g., refresher courses on certain topics) would be helpful to facilitate EBI implementation. Clinicians described that having training “in-house” would facilitate training attendance, and that attending training with colleagues would facilitate group discussion after training. On the other hand, clinicians noted that informal training through case discussions within their practice may not be as helpful as formal trainings. One clinician described, “I would probably [want] a little more formal than what my practice is doing now in terms of training. I don't mind case conferences, but honestly, I don't always go” (P2621). The same clinician described that informal training is “helpful to some degree,” (P2621) but may be less effective if the group is too large or more focused on “brainstorming” (P2621) rather than concrete suggestions. Clinicians indicated that compensation for attending training would facilitate training attendance and subsequent EBI implementation; they described that the cost of getting training includes both the actual cost of the workshop, as well as the lost income from not seeing patients during that time. Clinicians also said that receiving compensation in the form of continuing education units (CEUs) would facilitate training attendance. Clinicians highlighted the value of consultation to support ongoing EBI implementation after training. They described that it is easy to forget training content over time, and that isolated training workshops are often not sufficient to sustain EBI implementation without ongoing consultation. Clinicians indicated that both peer and expert consultation can be helpful, and that it is valuable to hear a diversity of perspectives through consultation.

Clinicians cited various sources through which they receive training. One common example was state-wide, discipline-specific organizations such as the Rhode Island Psychological Association as well as national organizations such as Anxiety and Depression Association of America (ADAA). Clinicians reported that training resources from professional organizations can be helpful, although the cost can be a barrier. Clinicians described online resources as a lower-cost (or no-cost) way of seeking training and continuing education, including professional mailing lists, listservs, special interest groups, Google Scholar, and Facebook groups. These resources were described as helpful to support EBI implementation, particularly when a clinician has a question about a specific clinical topic. Lastly, clinicians noted that being affiliated with an academic institution (e.g., as an adjunct professor) provides access to additional resources such as journal articles, training, and grand rounds that can facilitate continuing education and EBI implementation.

Mixed methods analyses

There were some differences in descriptions of continuing education between clinicians in solo vs. group practice. On one hand, clinicians in group practice reported that the presence of other clinicians made it more likely that they could find someone who could provide consultation. On the other hand, scheduling conflicts and busy schedules (“we joke we're passing ships in the night;” P6303) were described as practical barriers to receiving consultation from colleagues in group practice. Clinicians in solo practice cited specific professional organizations they have joined and specific conferences they have attended. For instance, one solo practice clinician described, “I joined… ADAA and I just started being very alert to opportunities” (P1058). Clinicians in both groups described being aware of training resources, but those in solo practice described more actively using resources outside of their organization. In terms of differences by level of organizational support, clinicians with high organizational support reported finding consultation that was helpful—even if it was outside of their organization [“In [name of professional group], we have a good amount of people you can always consult with” (P8907)]. In contrast, clinicians with low organizational support reported more mixed success in finding helpful consultation, as indicated by statements such as, “there are definitely times when I go, ‘hmm I wish there was someone I could run this by”' (P4610).

Costs of delivery

In addition to the financial considerations related to payer restrictions, clinicians cited costs associated with EBI delivery. Clinicians described that collateral contact with clients, insurance companies, other providers, and hospitals can occupy a lot of time that clinicians are spending unpaid. Clinicians mentioned that they try to avoid collateral contact when possible (e.g., by not having an office phone), but that some collateral contact is unavoidable. Clinicians also highlighted that preparing before a session requires time and effort that is unpaid. Clinicians cited supplies (e.g., rewards for children) as a cost that they incur without reimbursement, and that having supplies available and/or reimbursement for supplies would be helpful. Clinicians specifically described that conducting exposure in session requires additional preparation time, session time, materials, and ability to leave the office. One clinician said, “In order to use [exposure], there's a ton of prep work that goes into it” (P2605).

Mixed methods analyses

There were few differences in descriptions of cost of delivery between clinicians working in solo vs. group practice. Clinicians working in solo practice and clinicians with low organizational support mentioned wanting reimbursement for preparation time, as indicated with comments such as, “Money for planning time would change my behavior in that I would probably [treat] more children… because they're harder to plan for” (P2804). In contrast, clinicians in group practice or with high organizational support rarely mentioned this. Similarly, clinicians with low organizational support mentioned the cost of purchasing supplies [“I end up going out and buying things for kids (like) toys and prizes and all that” (P2804)], whereas no clinicians with high organizational support mentioned this.

Agency (regulatory) context

Payer restrictions

Clinicians described that payers generally want providers to use EBIs, particularly CBT. Respondents noted frustration that payers may prioritize the delivery of intervention components over allocating session time to build rapport. They reported that payers rarely provide reimbursement for longer (e.g., 60- to 90-min) or more frequent sessions, even though “45 min is not sufficient time” (P1226). Clinicians described that longer session time would be especially valuable for child patients and for conducting exposures in session. Clinicians indicated that they often opt to schedule clients back-to-back in order to maximize profit. They described variability in the amount of reimbursement provided by different payers, such that clinicians are less inclined to accept certain insurance providers who reimburse less for the same service. One clinician said that she only takes one insurance and “a sliding scale” (P1526). Clinicians also mentioned occasionally seeking certification in order to “re-negotiate reimbursement with insurance companies” (P8907). Lastly, clinicians indicated that high health insurance copays and deductibles are a barrier to families' treatment access.

Mixed methods analyses

There were no differences in descriptions of payer restrictions between clinicians who reported high vs. low organizational support; however, there were several differences between solo vs. group practice. Clinicians working in solo practice mentioned the administrative burden of dealing with insurance companies (e.g., authorization procedures, potential audits), which was not mentioned by clinicians working in group practice. Clinicians in solo practice also talked in detail about billing codes and dollar amounts from different insurance companies [e.g., “Insurance company name]'s low payment is a barrier… Even for a 90837 (billing code), it's significantly lower (than other companies)” (P1058)]. This topic was not discussed by clinicians working in group practice. Clinicians in solo practice made comments about what insurance companies want them to be doing (e.g., which billing codes to use, regulations related to frequency and duration of sessions) and more often indicated that they interpreted the billing guidelines more flexibility (e.g., billing for an in-office session even if part of the session took place in the community). Clinicians in group practice did not make comments about specific payers (e.g., which companies reimburse for what billing codes), but indicated an understanding of broad limitations of billing [e.g., “Insurance companies don't reimburse me for me to travel to (the client's) house and travel back” (P1226)].

Fiscal incentives

Clinicians described fiscal incentives that may incentivize EBI implementation. Clinicians had varied opinions about whether increased payment to deliver EBIs compared to other interventions would motivate them to implement EBIs; some clinicians reported that increased pay would motivate EBI use, whereas others said that they would use EBIs regardless of pay rate. Clinicians noted that payment for training would incentivize EBI use, given that cost is a barrier to attending training and that time spent in training is time spent not seeing clients. One clinician explained, “What complicates this is this kind of fee-for-service insurance-based care. Time is money. So every hour spent in a case conference or in a training session or whatever it may be is an hour less of a clinician seeing a patient” (P2621). Clinicians also mentioned that certification may in some cases incentivize training attendance. Some clinicians described that certification can provide specialized training, allow for opportunities to become a supervisor, and increase referrals to their practice, whereas others described that certification is not necessary in order to achieve specialized training, may be overly rigid and expensive, and may limit the type of referrals a provider receives. Clinicians reported that receiving CEUs is necessary but is not a primary motivator for how and why clinicians opt to attend training.

Mixed methods analyses

There were some differences in descriptions of fiscal incentives between clinicians working in solo vs. group practice, but no differences between clinicians with high vs. low organizational support. Clinicians working in solo practice indicated that they might be responsive to fiscal incentives (e.g., being compensated by insurance payers at a higher rate). For instance, one clinician said, “We live in a world that is monetized, so definitely increasing payment would [motivate me to use EBIs]” (P8907). In contrast, clinicians working in group practice made statements suggesting that they use EBIs regardless of fiscal incentives, such as, “If insurance would pay an extra 10 dollars per session for using motivational interviewing or something. No, I don't care about that” (P2605). Lastly, clinicians working in solo practice discussed various benefits and limitations of certification, whereas only one group practice clinician mentioned certification and said that it did not affect their practice.

Social context

Responses related to social context included several themes, including consumer beliefs and education about EBIs, consumer demand for EBIs, and consumer reactions to EBIs presented in treatment. Clinicians noted that certain geographical locations may affect beliefs about certain interventions. For example, one clinician noted, “The whole town is crazy about internal family systems treatment” (P1058). Related to this, consumer demand for EBIs was described as variable and guided by clients' understanding about EBIs. Clinicians noted that consumers who are educated about EBIs may be more likely to request them in treatment, which may increase the likelihood that a clinician will implement them. On the other hand, many clinicians noted that families who do their own research may “come in… with some notions [about EBIs] that are accurate and some that are not” (P4610). Lastly, clinicians described that clients have mixed reactions to EBIs introduced in treatment. Some clients have positive reactions to research-based treatment “because people like to hear that things are going to work” (P6303), whereas others “might not be receptive or compliant to the treatment” (P1226) or may be “extremely resistant” (P8907) if they “just need like an open ended session and they just need to vent or they don't want the therapy to feel as formal as maybe it could be” (P1004). Clinicians described that getting client buy-in for EBIs can be difficult and noted the importance of tailoring the EBI to the individual for this reason.

Clinicians noted that consumers' understanding of and familiarity with exposure therapy varied, such that some clients “do their own research before coming in” (P4610) whereas others “don't necessarily know to come in and ask for exposure” (P3004). One clinician said, “I think the biggest problem that I encounter is that people are not compliant with it. Like people don't want to do it because it is too scary… but when we explain the rationale of these interventions to patients or clients, it makes perfect sense to them” (P1226).

Mixed methods analyses

There were minimal differences in perceptions of social context affecting EBI implementation for therapists in solo vs. group practice. Clinicians in group practice talked slightly more often about using psychoeducation to build client buy-in to EBIs. Differences by level of organizational support were also minimal. Clinicians with high organizational support more consistently described having clients ask them for certain types of treatment [e.g., “Some people come in and [exposure] is what they want” (P1616)]. Clinicians with low organizational support had more mixed views on whether people ask for certain types of treatment. For instance, one clinician said, “I think some adults … have done their homework and have heard that [CBT] is a recommended treatment modality for what they're coming for. But most of them are not [asking for CBT] because I see a lot of kids and teenagers” (P6303).

Political climate

No participants made comments about the ways in which current or past political climates have influenced their use of EBIs, including exposure.

Discussion

This study examined an understudied area of implementation research by focusing on implementation determinants in private practice mental health settings guided by the policy ecology framework. Results from this study will inform future efforts to implement EBIs in private practice settings, where a large proportion of individuals seek mental health treatment. Responses from qualitative interviews highlighted the unique considerations for this setting and potential ways to tailor implementation strategies to increase clinicians' use of EBIs, including exposure therapy. Findings demonstrate how broadening our assessment of determinants by using a policy ecology framework may also inform future implementation strategies. Specifically, findings highlight the importance of tailoring implementation strategies to address organizational, agency, and social factors specific to private practice to increase EBI implementation in this setting. Participants did not specifically identify ways in which the political context, including state or federal policies, influenced their use of EBIs; however, political-level changes would likely influence some of the themes that were described by clinicians. Mixed methods analyses identified how EBI determinants may differ for solo vs. group private practice, as well as for organizations with high vs. low organizational support. Potential implementation strategies that address these policy ecology-informed determinants are discussed below and presented in Table 2.

Given evidence that organizational culture influences EBI implementation across various healthcare settings (27), we were particularly interested in understanding how organizational context functioned as a determinant of EBI implementation in private practice settings. Studies of organizational culture and climate typically differentiate between individual and organizational levels of analysis (52). However, there are many private practice organizations that consist of only one individual. Thus, we sought to examine how individuals in such settings describe their organizational context and its influence on their EBI use. The range of responses about the level of organizational support for exposure (OISCE scores) among solo private practice clinicians (range = 0.00–3.06) highlights the fact that there are differences in organizational constructs even across settings that are comprised of one individual. Although it is possible that these variations reflect individual differences, they likely also reflect the larger context in which clinicians work (i.e., geographical setting, theoretical orientation of colleagues, funding structure). An example of an organizational characteristic more commonly noted by clinicians working in solo private practice organizations was that flexibility within their work setting may facilitate EBI implementation. For instance, they described having more control over their schedules and the location in which they deliver services. They also endorsed more flexibility in terms of treatment delivery, such as being able to meet clients in the community for sessions. In contrast to this, clinicians working in group private practice described having access to resources that are, by definition, not available in solo practice, such as colleagues to discuss cases with and staff who manage billing. Clinicians working in solo private practice offered ideas for implementation strategies that might address some of the barriers specific to this setting, such as building peer networks and forming external peer consultation groups to supplement the lack of colleague support. Future studies using social network analysis of clinicians in private practice may provide insights into how this is currently happening or ways in which existing networks could be enhanced to support peer consultation groups across providers (53).

For clinicians in both solo and group practice settings, there was variability in reports of organizational support for exposure implementation, which likely influences clinicians' perceptions of and use of EBIs (54). Overall, quantitative scores on the measure of organizational support for exposure (OISCE) in this sample were consistent with community settings (M = 1.21, SD = 0.86) and lower than anxiety specialty clinics [M = 3.62; SD = 0.34; (52)]. These relatively low scores suggest that existing organizational supports may be inadequate to support exposure implementation for many private practice clinicians. Consistent with the organizational factors expected to support exposure implementation (23), clinicians in practices with high organizational support described having more access to EBI training, as well as supervisory and colleague support for discussing cases. Improving access to training is likely to address multiple barriers, including: (1) increasing knowledge and (2) addressing misconceptions/beliefs about exposures that may interfere with their use (22). Group practices may benefit from ensuring that opportunities for training incorporate supervisors, consistent with recommendations from previous research [e.g., (55)]. This is likely to be more challenging for solo practices in which supervisors are unlikely to be present. As noted above, an alternative might be to create peer networks of expertise to connect clinicians across solo practices who aim to deliver EBIs. Evidence suggests that the presence of colleague or supervisor support for implementing EBIs may increase EBI use, even for clinicians who do not directly participate in EBI implementation initiatives (37). This further highlights the importance of fostering organizational cultures that support EBIs, which may look different for private practice compared to publicly-funded mental health settings.

Notably, even organizations that were described as supporting EBI use conceptually did not provide all of the necessary resources for clinicians to deliver them. Specifically, clinicians described organizational policies that restricted their ability to hold longer sessions or meet clients in the community to conduct exposures. Thus, organizations that are interested in implementing and sustaining EBI use may benefit from a regular review of organizational policies to ensure that they align with expectations that EBIs are prioritized. However, many organizational determinants stem from barriers that were identified at the agency or regulatory level, as discussed in more detail below. Even if organizations adopt policies that support EBI implementation, such as allowing clinicians to leave the office for therapy sessions or to travel with clients in the community, payer restrictions may interfere. Clinicians highlighted how factors such as the inability to bill insurance companies for longer sessions or for sessions conducted outside of the office can make it particularly challenging to implement EBIs, a common concern identified in previous research on implementation of exposure therapy (24). Creation of billing codes that permit travel and longer session lengths would likely address these barriers. This may be particularly warranted given growing evidence supporting home-based delivery of EBIs such as exposure (56–58). Providing increased education to care managers about EBIs may also facilitate the creation of such billing codes.

In this vein, responses related to the agency-context (i.e., payer restrictions and fiscal incentives) indicated that clinicians working in private practice may be disincentivized from engaging in activities for which they are not reimbursed. Clinicians described having difficulty attending training in EBIs, providing collateral contact, and conducting long enough sessions to meet clients' needs. Clinicians working in solo private practice also described being disincentivized from accepting insurance given variable or poor rates of reimbursement and the administrative burden of doing so (59). These findings are largely consistent with previous research examining EBI implementation determinants in other routine mental health care settings. For instance, Okamura et al. (60) found that EBI delivery in public sector mental health settings can incur significant costs to therapists and organizations, which may serve to disincentivize implementation. The same is likely true of EBI use in private practice settings, especially given that private practice organizations do not receive public sector financial support. Costs associated with EBI implementation were also found to vary by intervention (61), which is consistent with our finding that exposure therapy has unique challenges to its implementation. Overall, respondents in this study indicated a strong desire for compensation for services they are already providing (e.g., collateral contact), but varied in their reports of whether compensation could motivate them to change their current practice and incorporate new EBIs.

Efforts to reduce discrepancies in reimbursement across insurance providers and compensate clinicians for training and for clinical services they are already providing (e.g., collateral contact) is one potential method to increase EBI use. Previous research examining stakeholder preferences related to implementation strategies has demonstrated that stakeholders across several groups (i.e., clinicians, supervisors, agency executives, payers) agree that financial incentives are the most useful category of implementation strategies (61, 62). In terms of specific financial incentives, payers would prefer to provide compensation for EBI delivery rather than preparation time (61). Providing an enhanced reimbursement rate for EBI use compared to other interventions may serve to encourage EBI implementation for some clinicians. However, in past research and in the current study, clinicians also identified a strong desire to be compensated for EBI preparation time and training, which has been rated as a less preferred option by payers (61). A potential lower-cost alternative to compensation for preparation time that might facilitate EBI implementation would be to allot funding for therapy supplies. This might be a more feasible incentive that payers could offer to support clinicians in implementing EBIs. Alternatively, the creation and dissemination of freely available and easily accessible (e.g., online) toolkits that incorporate active components, such as reminders, have been shown to support the implementation of EBIs [e.g., (63)]. Furthermore, advocates of “digital apothecaries,” or online repositories for digital interventions (e.g., websites, mobile apps, digital tools) suggest that such resources may be particularly helpful for supporting private practice clinicians and their clients in using EBIs (64). Future implementation research should examine whether tailored online toolkits and digital apothecaries reduce barriers to EBI use and defray costs related to supplies. It is also worth noting that many clinicians mentioned that physical supplies are necessary when working with children (e.g., toys, games, prizes). In a group practice, it might be feasible for clinicians to pool funds for shared supplies. However, in solo practice this may be more challenging. Future implementation research should examine whether stakeholder preferences for financial incentives lead to meaningful behavior change, and whether lower-cost interventions such as providing supplies are effective at changing behavior or maintaining use of EBIs in private practice settings.

In addition to organizational- and agency-level factors, social context was also described as a determinant of EBI implementation. Specifically, clinicians reported that they may be more likely to implement EBIs if patients request them or respond positively to their introduction in treatment. This is consistent with prior research indicating that concerns about patient and caregiver reactions to an EBI (exposure therapy) are a common barrier to implementation (22). Consumer demand for and reaction to EBIs may be influenced by consumer education, geographical location, and culture. Respondents suggested that some patients may respond more negatively to exposure therapy compared to other EBIs. These findings are consistent with existing literature suggesting the need for additional efforts to increase consumer education about and demand for EBIs, including exposure therapy. Direct-to-consumer marketing strategies (42) aimed at increasing consumer demand for EBIs may incentivize more clinicians to use EBIs. Importantly, such marketing strategies may be most effective if tailored to specific subgroups of consumers (65). Thus, continued efforts are needed to examine how such direct-to-consumer marketing strategies should be tailored to consumers receiving services in public vs. private practice mental health settings.

Although clinicians were asked about determinants at the political level of the policy ecology, there were no themes that emerged in this category. One possibility is that considerations related to legislation were perceived by clinicians as too distal to have a direct influence on their behavior. Responses tended to focus on more proximal determinants of clinicians' use of EBIs in their current setting, such as the types of referrals they receive and the geographical location in which they work. While some participants mentioned whether their graduate training emphasized EBIs, they did not describe how this might be linked to political or administrative issues, such as accrediting requirements. Another possible explanation for the lack of findings in this domain is that private practice clinicians perceived the political context as being less relevant to their practice given their decision to operate in private practice settings; they may have selected this setting for its decreased regulatory requirements compared to publicly-funded mental health settings. More focused inquiry into how specific political issues, such pre-service training, loan forgiveness programs, and mental health parity might affect private practice clinicians should be the focus of future research.

The recommendations generated from our findings are a first step toward increasing EBI use in private practice settings. A key strength of this study is the application of an existing implementation framework (i.e., policy ecology framework) to a novel context (i.e., private practice mental health). Given the proliferation of implementation frameworks, applying existing frameworks to new contexts has been identified as a priority for implementation research (66). Additional methodological strengths of this study include the use of purposive sampling to recruit clinicians with and without training in a difficult-to-implement EBI, exposure therapy. Similarly, the inclusion of clinicians working in different types of private practice settings (i.e., solo vs. group private practice) provides insight into EBI implementation in a range of settings in which clinicians might work.

Results should also be interpreted in the context of limitations. The small sample size and lack of ethnic and racial diversity of participants included in this study may limit the generalizability of the study's results. In particular, there are likely to be barriers faced by clinicians of color and their patients that may not be reflected in findings, such as the role of racial discrimination and bias, the need for resources in other languages, and other systemic considerations not represented in these results. Participants included in this study were clinicians located in New England, where clinical practice may differ from other geographical regions. For example, clinicians working in private practice in New England may accept public insurance, which differs from other regions of the United States. Additionally, all clinicians included were those who were willing to participate in a research study, and many had previously participated in a clinician training research study conducted by the same organization. Willingness to participate in a research study may suggest that these clinicians are open to research more broadly and may hold more positive attitudes toward EBIs than clinicians who are unwilling to participate in a research study. In addition, this study did not measure actual EBI or exposure use in clinicians' routine clinical practice. Future research should quantitatively assess whether the identified implementation determinants influence clinicians' EBI use. Finally, limited information was collected on the exact nature of previous training in exposure therapy, and clinicians likely varied widely in the amount of training they have previously received in EBIs.

Despite its limitations, this study provides novel information about the multi-level factors that influence the implementation of EBIs in private practice and supports the use of a policy ecology framework to inform the generation of setting-specific implementation strategies. Respondents in this study cited various organizational, agency-related, and social barriers to the implementation of EBIs in routine clinical practice, which informed suggestions for implementation strategies that may address these barriers. Future research should examine the feasibility, acceptability, and effectiveness of the suggested implementation strategies to increase EBI use in private practice. Future research should also examine the cumulative effects of multiple implementation strategies to target different ecological levels and maximize the likelihood of EBI implementation.

Data availability statement

The datasets presented in this article are not readily available due to the sensitive and potentially identifiable nature of interview transcript data. Requests to access the datasets should be directed to HF, hannah_frank@brown.edu.

Ethics statement

The studies involving human participants were reviewed and approved by Lifespan Institutional Review Board. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

Author contributions

HF conceived of and designed the research study, acquired and analyzed the data, interpreted the data, drafted the manuscript, and substantially revised it. LM analyzed and interpreted the data and drafted sections of the manuscript. JF and KB helped conceive of and design the research study and substantially revised the manuscript. All authors approved the submitted version, have agreed to be accountable for the contributions, and attest to the accuracy and integrity of the work, even aspects for which the authors were not personally involved.

Funding

This study was provided by a grant from the National Institute of Mental Health awarded to HF (T32MH019927, PI, Spirito). The funding bodies did not have any direct role in the manuscript idea or writing of the manuscript.

Acknowledgments

We are grateful to several people for making this study possible, including Kaitlyn Schneider for her role in conducting qualitative interviews for this study, Dr. Wendy Plante for her assistance with study recruitment, and Dr. A. Rani Elwy for her input on the qualitative methods and design of this study. We also thank the clinicians who agreed to participate in this study for their time and effort.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frhs.2022.892294/full#supplementary-material

References

1. Harrington JA. Contemporary issues in private practice: spotlight on the self-employed mental health counselor. J Ment Health Counsel. (2013) 35:189–98. doi: 10.17744/mehc.35.3.8742717176154187

CrossRef Full Text | Google Scholar

2. Hamp A, Stamm K, Lin L, Christidis P. 2015 APA Survey of Psychology Health Service Providers. Washington, DC: American Psychological Association (2016). doi: 10.1037/e512852016-001

PubMed Abstract | CrossRef Full Text | Google Scholar

3. DeVoe JE, Wallace L, Selph S, Westfall N, Crocker S. comparing type of health insurance among low-income children: a mixed-methods study from Oregon. Matern Child Health J. (2011) 15:1238–48. doi: 10.1007/s10995-010-0706-4

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Wray CM, Khare M, Keyhani S. Access to care, cost of care, and satisfaction with care among adults with private and public health insurance in the US. JAMA Netw Open. (2021) 4:e2110275. doi: 10.1001/jamanetworkopen.2021.10275

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Howell EM. Access to Children's Mental Health Services Under Medicaid SCHIP. (2004). Retrieved from http://webarchive.urban.org/publications/311053.html

Google Scholar

6. Whitney DG, Peterson MD. US national and state-level prevalence of mental health disorders and disparities of mental health care use in children. JAMA Pediatr. (2019) 173:389. doi: 10.1001/jamapediatrics.2018.5399

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Busch SH, Barry CL. Does private insurance adequately protect families of children with mental health disorders? Pediatrics. (2009) 124 (Suppl. 4):S399–406. doi: 10.1542/peds.2009-1255K

PubMed Abstract | CrossRef Full Text | Google Scholar

8. DeRigne L. What are the parent-reported reasons for unmet mental health needs in children? Health Soc Work. (2010) 35:7–15. doi: 10.1093/hsw/35.1.7

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Vahratian A, Blumberg SJ, Terlizzi EP, Schiller JS. Symptoms of anxiety or depressive disorder and use of mental health care among adults during the COVID-19 pandemic — United States, August 2020–February 2021. MMWR Morb Mortal Wkly Rep. (2021) 70:490–4. doi: 10.15585/mmwr.mm7013e2

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Rye M, Friborg O, Skre I. Attitudes of mental health providers towards adoption of evidence-based interventions: relationship to workplace, staff roles and social and psychological factors at work. BMC Health Serv Res. (2019) 19:110. doi: 10.1186/s12913-019-3933-4

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Cho E, Tugendrajch SK, Marriott BR, Hawley KM. Evidence-Based assessment in routine mental health services for youths. PS. (2021) 72:325–8. doi: 10.1176/appi.ps.201900537

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Bailin A, Bearman SK, Sale R. Clinical supervision of mental health professionals serving youth: format and microskills. Adm Policy Ment Health. (2018) 45:800–12. doi: 10.1007/s10488-018-0865-y

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Becker-Haimes EM, Okamura KH, Wolk CB, Rubin R, Evans AC, Beidas RS. Predictors of clinician use of exposure therapy in community mental health settings. J Anxiety Disord. (2017) 49:88–94. doi: 10.1016/j.janxdis.2017.04.002

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Pittig A, Kotter R, Hoyer J. The struggle of behavioral therapists with exposure: self-reported practicability, negative beliefs, and therapist distress about exposure-based interventions. Behav Ther. (2019) 50:353–66. doi: 10.1016/j.beth.2018.07.003

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Deacon BJ, Farrell NR, Kemp JJ, Dixon LJ, Sy JT, Zhang AR, et al. Assessing therapist reservations about exposure therapy for anxiety disorders: the therapist beliefs about exposure scale. J Anxiety Disord. (2013) 27:772–80. doi: 10.1016/j.janxdis.2013.04.006

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Farrell NR, Deacon BJ, Kemp JJ, Dixon LJ, Sy JT. Do negative beliefs about exposure therapy cause its suboptimal delivery? An experimental investigation. J Anxiety Disord. (2013) 27:763–71. doi: 10.1016/j.janxdis.2013.03.007

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Harned MS, Dimeff LA, Woodcock EA, Contreras I. Predicting adoption of exposure therapy in a randomized controlled dissemination trial. J Anxiety Disord. (2013) 27:754–62. doi: 10.1016/j.janxdis.2013.02.006

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Farrell NR, Kemp JJ, Blakey SM, Meyer JM, Deacon BJ. Targeting clinician concerns about exposure therapy: a pilot study comparing standard vs. enhanced training. Behav Res Ther. (2016) 85:53–9. doi: 10.1016/j.brat.2016.08.011

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Wolitzky-Taylor K, Chung B, Bearman SK, Arch J, Grossman J, Fenwick K, et al. Stakeholder perceptions of the barriers to receiving and delivering exposure-based cognitive behavioral therapy for anxiety disorders in adult community mental health settings. Community Ment Health J. (2019) 55:83–99. doi: 10.1007/s10597-018-0250-z

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Trivasse H, Webb TL, Waller G. A meta-analysis of the effects of training clinicians in exposure therapy on knowledge, attitudes, intentions, and behavior. Clin Psychol Rev. (2020) 80:101887. doi: 10.1016/j.cpr.2020.101887

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Foa EB, McLean CP, Brown LA, Zang Y, Rosenfield D, Zandberg LJ, et al. The effects of a prolonged exposure workshop with and without consultation on provider and patient outcomes: a randomized implementation trial. Implementation Sci. (2020) 15:1–4. doi: 10.1186/s13012-020-01014-x

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Reid AM, Guzick AG, Fernandez AG, Deacon B, McNamara JPH, Geffken GR, et al. Exposure therapy for youth with anxiety: utilization rates and predictors of implementation in a sample of practicing clinicians from across the United States. J Anxiety Disord. (2018) 58:8–17. doi: 10.1016/j.janxdis.2018.06.002

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Becker-Haimes EM, Byeon YV, Frank HE, Williams NJ, Kratz HE, Beidas RS. Identifying the organizational innovation-specific capacity needed for exposure therapy. Depress Anxiety. (2020) 37:1007–16. doi: 10.1002/da.23035

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Reid AM, Bolshakova MI, Guzick AG, Fernandez AG, Striley CW, Geffken GR, et al. Common barriers to the dissemination of exposure therapy for youth with anxiety disorders. Community Ment Health J. (2017) 53:432–7. doi: 10.1007/s10597-017-0108-9

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Ringle VA, Read KL, Edmunds JM, Brodman DM, Kendall PC, Barg F, et al. Barriers to and facilitators in the implementation of cognitive-behavioral therapy for youth anxiety in the community. PS. (2015) 66:938–45. doi: 10.1176/appi.ps.201400134

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Stewart RE, Chambless DL, Stirman SW. Decision making and the use of evidence-based practice: is the three-legged stool balanced? Pract Innov. (2018) 3:56–67. doi: 10.1037/pri0000063

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Li S-A, Jeffs L, Barwick M, Stevens B. Organizational contextual features that influence the implementation of evidence-based practices across healthcare settings: a systematic integrative review. Syst Rev. (2018) 7:72. doi: 10.1186/s13643-018-0734-5

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Finch J, Ford C, Grainger L, Meiser-Stedman R. A systematic review of the clinician related barriers and facilitators to the use of evidence-informed interventions for post traumatic stress. J Affect Disord. (2020) 263:175–86. doi: 10.1016/j.jad.2019.11.143

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Kilbourne AM, Beck K, Spaeth-Rublee B, Ramanuj P, O'Brien RW, Tomoyasu N, et al. Measuring and improving the quality of mental health care: a global perspective. World Psychiatry. (2018) 17:30–8. doi: 10.1002/wps.20482

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Frank HE, Becker-Haimes EM, Kendall PC. Therapist training in evidence-based interventions for mental health: a systematic review of training approaches and outcomes. Clin Psychol Sci Pract. (2020) 27:e12330. doi: 10.1111/cpsp.12330

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Chu BC, Crocco ST, Arnold CC, Brown R, Southam-Gerow MA, Weisz JR. Sustained implementation of cognitive-behavioral therapy for youth anxiety and depression: long-term effects of structured training and consultation on therapist practice in the field. Prof Psychol Res Pr. (2015) 46:70–9. doi: 10.1037/a0038000

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Thomassin K, Marchette LK, Weisz JR. Practitioners' use and evaluation of transdiagnostic youth psychotherapy years after training and consultation have ended. Adm Policy Ment Health. (2019) 46:821–32. doi: 10.1007/s10488-019-00962-x

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Raghavan R, Bright CL, Shadoin AL. Toward a policy ecology of implementation of evidence-based practices in public mental health settings. Implementation Sci. (2008) 3:26. doi: 10.1186/1748-5908-3-26

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Powell BJ, Beidas RS, Rubin RM, Stewart RE, Wolk CB, Matlin SL, et al. Applying the policy ecology framework to Philadelphia's behavioral health transformation efforts. Adm Policy Ment Health. (2016) 43:909–26. doi: 10.1007/s10488-016-0733-6

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Stewart RE, Chambless DL, Baron J. Theoretical and practical barriers to practitioners' willingness to seek training in empirically supported treatments. J Clin Psychol. (2012) 68:8–23. doi: 10.1002/jclp.20832

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Hatfield DR, Ogles BM. Why some clinicians use outcome measures and others do not. Adm Policy Ment Health. (2007) 34:283–91. doi: 10.1007/s10488-006-0110-y

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Beidas RS, Williams NJ, Becker-Haimes EM, Aarons GA, Barg FK, Evans AC, et al. A repeated cross-sectional study of clinicians' use of psychotherapy techniques during 5 years of a system-wide effort to implement evidence-based practices in Philadelphia. Implementation Sci. (2019) 14:67. doi: 10.1186/s13012-019-0936-9

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Weissman MM, Verdeli H, Gameroff MJ, Bledsoe SE, Betts K, Mufson L, et al. National survey of psychotherapy training in psychiatry, psychology, and social work. Arch Gen Psychiatry. (2006) 63:925–34. doi: 10.1001/archpsyc.63.8.925

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Becker-Haimes EM, Okamura KH, Baldwin CD, Wahesh E, Schmidt C, Beidas RS. Understanding the landscape of behavioral health pre-service training to inform evidence-based intervention implementation. PS. (2019) 70:68–70. doi: 10.1176/appi.ps.201800220

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Bertram RM, Charnin LA, Kerns SEU, Long ACJ. Evidence-Based Practices in North American MSW Curricula. Res Soc Work Pract. (2015) 25:737–48. doi: 10.1177/1049731514532846

CrossRef Full Text | Google Scholar

41. Reardon T, Harvey K, Baranowska M, O'Brien D, Smith L, Creswell C. What do parents perceive are the barriers and facilitators to accessing psychological treatment for mental health problems in children and adolescents? A systematic review of qualitative and quantitative studies. Eur Child Adolesc Psychiatry. (2017) 26:623–47. doi: 10.1007/s00787-016-0930-6

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Becker SJ. Direct-to-Consumer marketing: a complementary approach to traditional dissemination and implementation efforts for mental health and substance abuse interventions. Clin Psychol Sci Pract. (2015) 22:85–100. doi: 10.1111/cpsp.12086

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Stewart RE, Marcus SC, Hadley TR, Hepburn BM, Mandell DS. State adoption of incentives to promote evidence-based practices in behavioral health systems. PS. (2018) 69:685–8. doi: 10.1176/appi.ps.201700508

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. (2009) 42:377–81. doi: 10.1016/j.jbi.2008.08.010

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Harris PA, Taylor R, Minor BL, Elliott V, Fernandez M, O'Neal L, et al. The REDCap consortium: building an international community of software platform partners. J Biomed Inform. (2019) 95:103208. doi: 10.1016/j.jbi.2019.103208

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Dillman DA, Smyth JD, Christian LM. Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method. Hoboken, NJ: John Wiley and Sons (2014).

Google Scholar

47. Hoddinott SN, Bass MJ. The dillman total design survey method. Can Fam Phys. (1986) 32:2366–8.

PubMed Abstract | Google Scholar

48. Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health Serv Res. (2007) 42:1758–72. doi: 10.1111/j.1475-6773.2006.00684.x

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. (2005) 15:1277–88. doi: 10.1177/1049732305276687

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Creswell JW. Revisiting mixed methods and advancing scientific practices. In: Hesse-Biber SN, Johnson RB, editors. The Oxford Handbook of Multimethod and Mixed Methods Research Inquiry. New York, NY: Oxford University Press (2015). doi: 10.1093/oxfordhb/9780199933624.013.39

CrossRef Full Text | Google Scholar

51. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. (2007) 19:349–57. doi: 10.1093/intqhc/mzm042

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Williams MT, Farris SG, Turkheimer EN, Franklin ME, Simpson HB, Liebowitz M, et al. The impact of symptom dimensions on outcome for exposure and ritual prevention therapy in obsessive-compulsive disorder. J Anxiety Disord. (2014) 28:553–8. doi: 10.1016/j.janxdis.2014.06.001

PubMed Abstract | CrossRef Full Text | Google Scholar

53. Valente TW, Palinkas LA, Czaja S, Chu KH, Brown CH. Social network analysis for program implementation. PLoS ONE. (2015) 10:e0131712. doi: 10.1371/journal.pone.0131712

PubMed Abstract | CrossRef Full Text | Google Scholar

54. Aarons GA, Sawitzky AC. Organizational culture and climate and mental health provider attitudes toward evidence-based practice. Psychol Serv. (2006) 3:61–72. doi: 10.1037/1541-1559.3.1.61

PubMed Abstract | CrossRef Full Text | Google Scholar

55. Dorsey S, Pullmann MD, Deblinger E, Berliner L, Kerns SE, Thompson K, et al. Improving practice in community-based settings: a randomized trial of supervision – study protocol. Implementation Sci. (2013) 8:89. doi: 10.1186/1748-5908-8-89

PubMed Abstract | CrossRef Full Text

56. Barbot B, Bick J, Bentley MJ, Balestracci KMB, Woolston JL, Adnopoz JA, et al. Changes in mental health outcomes with the intensive in-home child and adolescent psychiatric service: a multi-informant, latent consensus approach. Int J Methods Psychiatr Res. (2016) 25:33–43. doi: 10.1002/mpr.1477

PubMed Abstract | CrossRef Full Text | Google Scholar

57. Henggeler SW. multisystemic therapy: an overview of clinical procedures, outcomes, and policy implications. Child Psychol Psychiatry Rev. (1999) 4:2–10. doi: 10.1017/S1360641798001786

CrossRef Full Text | Google Scholar

58. Rowa K, Antony MM, Swinson RP. Exposure and response prevention. In: Psychological Treatment of Obsessive-Compulsive Disorder: Fundamentals and Beyond. Washington, DC: American Psychological Association (2007). p. 79–109. doi: 10.1037/11543-004

CrossRef Full Text | Google Scholar

59. Bowers TG, Knapp S. Reimbursement issues for psychologists in independent practice. Psychother Priv Pract. (1993) 12:73–87. doi: 10.1300/J294v12n03_10

CrossRef Full Text | Google Scholar

60. Okamura KH, Benjamin Wolk CL, Kang-Yi CD, Stewart R, Rubin RM, Weaver S, et al. The price per prospective consumer of providing therapist training and consultation in seven evidence-based treatments within a large public behavioral health system: an example cost-analysis metric. Front Public Health. (2018) 5:356. doi: 10.3389/fpubh.2017.00356

PubMed Abstract | CrossRef Full Text | Google Scholar

61. Candon M, Williams N, Zentgraf K, Buttenheim A, Bewtra M, Beidas RS, et al. Variation in stakeholder preferences for implementing evidence-based practices in behavioral health care. PS. (2022) 5. doi: 10.1176/appi.ps.202100453

PubMed Abstract | CrossRef Full Text | Google Scholar

62. Williams NJ, Candon M, Stewart RE, Byeon YV, Bewtra M, Buttenheim AM, et al. Community stakeholder preferences for evidence-based practice implementation strategies in behavioral health: a best-worst scaling choice experiment. BMC Psychiatry. (2021) 21:74. doi: 10.1186/s12888-021-03072-x

PubMed Abstract | CrossRef Full Text | Google Scholar

63. Yamada J, Shorkey A, Barwick M, Widger K, Stevens BJ. The effectiveness of toolkits as knowledge translation strategies for integrating evidence into clinical care: a systematic review. BMJ Open. (2015) 5:e006808. doi: 10.1136/bmjopen-2014-006808

PubMed Abstract | CrossRef Full Text | Google Scholar

64. Muñoz RF, Chavira DA, Himle JA, Koerner K, Muroff J, Reynolds J, et al. Digital apothecaries: a vision for making health care interventions accessible worldwide. Mhealth. (2018) 4:18. doi: 10.21037/mhealth.2018.05.04

PubMed Abstract | CrossRef Full Text | Google Scholar

65. Becker SJ, Helseth SA, Frank HE, Escobar K, Weeks B. Parent preferences and experiences with psychological treatment: results from a direct-to- consumer survey using the marketing mix framework. Prof Psychol Res Pr. (2018) 49:167–76. doi: 10.1037/pro0000186

PubMed Abstract | CrossRef Full Text | Google Scholar

66. Nilsen P. Making sense of implementation theories, models and frameworks. Implementation Science. (2015) 10:53. doi: 10.1186/s13012-015-0242-0

PubMed Abstract | CrossRef Full Text