Using normalization process theory to inform practice: evaluation of a virtual autism training for clinicians

Background There is growing demand for developmental and behavioral pediatric services including autism evaluation and care management. Clinician trainings have been found to result in an increase of knowledge and attitudes. This study utilizes Normalization Process theory (NPT) to evaluate a clinician training program and its effects on practice. Methods The year-long virtual training program about autism screening and care management included didactic portions and case presentations. Focus groups and interviews were conducted with primary care clinicians (n = 10) from community health centers (n = 6) across an urban area five months post-training. Transcripts were deductively coded using NPT to uncover barriers to implementation of autism screening and care, benefits of the training program, and areas for future training. Results Participants were motivated by the benefits of expanding and improving support for autistic patients but noted this effort requires effective collaboration within a complex network of care providers including clinicians, insurance agencies, and therapy providers. Although there were support that participants could provide to families there were still barriers including availability of behavior therapy and insufficient staffing. Overall, participants positively viewed the training and reported implementing new strategies into practice. Conclusion Despite the small sample size, application of NPT allowed for assessment of both training delivery and implementation of strategies, and identification of recommendations for future training and practice sustainability. Follow-up focus groups explored participants' practice five months post-program. Variations in participants' baseline experience and context at follow-up to enable application of skills should be considered when using NPT to evaluate clinician trainings.


Introduction
For the past decade, autism diagnosis and demand for developmental-behavioral pediatric care has been steadily rising with the most recent autism prevalence reported at 1 in 36 children (1).Increased demand of developmental services combined with disinvestment (2) and disruptions from the COVID-19 pandemic (3) has led to service delays.This is problematic given that benefits of early autism diagnosis and evidence-based care are well established (4,5).Although screening is recommended to occur between 18 and 24 months of age (6), most children receive diagnosis after their third birthday, which is the cutoff for state-funded early intervention programs (7).Access to autism developmental care is especially strained in low-income and minoritized communities partly due to shortage of pediatric specialists (8).
A proposed strategy to address gaps in services is training more clinicians to conduct developmental screening and evaluations (9) instead of waiting for referral to specialists.The Extension for Community Healthcare Outcomes (ECHO ® ) has shown promise in equipping primary care clinicians (PCC) with skills and knowledge on a variety of clinical areas (10,11) including autism.The ECHO model consists of a Hub institution comprised of topic experts responsible for delivering trainings and Spoke institutions comprised of PCCs (10) less expert in the topic.Researchers found that autism-focused ECHO programs resulted in increased participant knowledge and confidence (12-16).For example, Mazurek et al. (2019) found that a 12-month ECHO program increased clinicians' ability and selfefficacy to screen for autistic patients in Missouri, United States (13).There has been little evidence, however, that autism-focused ECHO programs result in practice change (17).Most studies on ECHO programs focusing on autism used a pre/post design but did not qualitatively assess practice change at follow-up (13)(14)(15)(18)(19)(20).Evaluating both pre/post knowledge/skills changes and application of practice within the service delivery setting are necessary to understand training impact.
The current study uses Normalization Process Theory (NPT) (21) as a framework to guide the Boosting Capacity to Screen and Care for Underserved Autistic Children ECHO Program (BCAEP), a training designed to enhance autism screening and care in primary care settings.The BCAEP training was conducted virtually during the COVID-19 pandemic and supported PCCs during major care disruptions.NPT posits that there are four components of practice change, namely coherence (i.e., what the new practice is), cognitive participation (i.e., one's and others' roles in the new practice), collective action (i.e., steps needed to accomplish the new practice), and reflexive monitoring (i.e., evaluation of the new practice) (21).NPT emphasizes individual and collective behaviors, in addition to attitudes and beliefs, which aids practice-focused queries (22).Past studies have utilized NPT to assess new practices in healthcare settings (22,23).This study advances previous research by applying NPT to evaluate delivery and effectiveness of PCC trainings in urban safety-net settings as well as the extent to which learned skills and knowledge are applied in practice months after the training.

Program description
The BCAEP training was conducted virtually between November 2020 and October 2021 over 12 60-min sessions.Participants (n = 47) represented PCCs in seven health centers in the Greater Boston Area caring for safety-net populations.Trainings were facilitated by a senior developmental behavioral pediatrician with over 30 years of experience and an advanced practice clinician with 19 years of experience; both were affiliated with a safety-net academic medical institution.Each session consisted of didactic lectures and deidentified patient case discussions as per the ECHO model (10).Topics of the didactic lectures were determined based on an initial participant survey and were adjusted according to participants' interests.Examples of lecture topics were administration of different autism screening tools, engaging with patients in a culturally sensitive manner, and communicating an initial autism diagnosis to families.

Study design
This study was part of a larger mixed methods evaluation of the BCAEP training and represents the qualitative component of the evaluation.Focus groups and interviews with BCAEP PCCs (n = 10) were conducted to contextualize quantitative survey responses and provide actionable practice recommendations.Participants were asked to complete a brief pre-focus group assessment that gathered information about their practice (Table 1); nine responded to the survey and ten participated in focus groups.There were 30 pre-test (before first session), 19 mid-point (after sixth session), and 17 post-test (after twelfth session) survey responses, with nine matches from pre-test to post-test.Survey findings indicated an increase in participants' reported knowledge and self-efficacy in administering autism screeners and managing care in post-test compared to pre-test.Each of these quantitative findings warranted further exploration qualitatively and at a time when participants had a chance to implement the training learnings in practice.
Focus groups and interviews were conducted virtually approximately 6-months post-training using Zoom or in-person depending on participants' availability.This study was reviewed and approved as exempt by the Boston University Institutional Review Board (IRB #H-40718).

Participants
Participants for the qualitative assessments were recruited through convenience sampling of PCCs who attended the training.Participants represented PCCs from six out of seven (85.7%) eligible health centers.Recruitment emails consisting of abbreviated consent forms were sent.Participants were asked to complete a brief pre-focus group assessment that gathered information about their practice (Table 1).

Data collection
Semi-structured focus groups and interviews were conducted in March 2022 using a guide (Appendix A) to gather information on individual level and clinic-level application of strategies taught in the BCAEP training as well as opinions about the training delivery.Questions were informed by survey findings, NPT constructs (21), and outcomes indicated on the logic model.Focus groups were conducted by two evaluation team members; a lead moderator and note-taker.One member was present at all focus group and interview sessions for consistency.Audio recordings were transcribed and verified by two different members for accuracy.

Analysis
Deductive coding (24) using NPT constructs (21) and content analysis (25) were conducted on six transcripts.The coding team consisted of three evaluation team members who were trained to conduct qualitative analysis using NVivo12 (26).Each transcript was coded by two members.All three members then met to discuss discrepancies by consensus with the third member serving as tiebreaker if consensus was not reached.

Results
From the nine pre-focus group assessments, participants included physicians (n = 4), nurse practitioners (n = 4), and administrative leadership (n = 1) with most participants having 3-5 years (n = 3) and 11-20 (n = 3) years of experience.Specialties included pediatrics (n = 7), psychiatry (n = 1), and integrated behavior health (n = 1).On average, participants attended 10 out of 12 training sessions.Themes were organized by NPT constructs and presented with frequencies and illustrative quotes (Table 2) in order to provide a framework for actionable recommendations.
The following sections consisted of quotes coded with the corresponding constructs.NPT consisted of four constructs, each with four sub-constructs ( 21) that start with an understanding of the new practice (i.e., coherence), operationalizing change (i.e., cognitive participation), implementing the change (i.e., collective action), and ending with its evaluation (i.e., reflexive working) (22).While all constructs contributed to the findings and subsequent recommendations, reflexive working and collective action, given the role of clinicians to implement the concepts and assess the utility in practice, were most often mentioned.

Coherence
Coherence occurs when an individual attempts to make sense of the new practice (21).This construct manifests through an individual's perception on a practice as well as their motivation and role for the new practice implementation.

Differentiation
In the current study, differentiation occurred when participants discussed how strategies taught in the training differed from their current practice (21).For example, one participant shared that "in the old model, we could put in a referral to [external institution] developmental-behavioral peds" but there are "kids [who] might be better served if we can keep them sort of inhouse."Another participant mentioned how behavioral observations may differ depending on the child's age and that "having more concrete examples… maybe also seeing… the videos… [of] kids of different ages and genders" could be helpful to deepen their understanding of screening procedures.Additionally, one participant described how the training reminded them to score screenings "right away when it's fresh.""Being able to think and understand how pediatricians approach a patient, as well as the concerns and challenges they face in thinking from that behavioral health lens….What could we do?What could we hold with them so they're not holding it alone?"110:

Internalization
"… I think we're more able to discuss what ABA therapy looks like.I think… we're getting trained on level two screeners in two weeks, and gonna hopefully be using that to do some diagnosis… onsite…" 18 Cognitive participation involves problem identification, collaborations within a clinic to work towards solving the problem, and how clinicians can confidently sustain their practices.Participants identified the pieces that need to come together to lead to positive change in their individual and clinic-level practice.This thinking involves identifying the individuals who will bring about this change and the context in which they will be doing so.Further, participants reflected on whether they are confident that they have the tools and people necessary to exact change Initiation Participants recognize long wait times and gaps between appointments as barriers in their practice.Participants reflected on the gap between their current practice and new strategies introduced in BCAEP.Many BCAEP participants were building from practices that they already had in place following the training.However, others were just getting started with thinking about how to change their practice to better serve autistic patients.

109:
"… I guess when you go through med school, there's still definitely that lingering altruism of "see one, do one, teach one".And I trained in the CARS as a fellow, probably, you know, actually performed it with some small [number of patients], and then I was in the real-world practicing and using it…" 107: practice entails (21).Motivations to engage in the training included an increase in patients with developmental support needs and reinforcement of previously learned concepts.
Autism diagnosis and care management appeared to be important and timely skills to refine as one participant shared that they had "been in practice for 25 years and I never had this many autistic kids" and thus developmental care had "really taken up a great part of my practice."Moreover, autism as a diagnosis had evolved considerably in the past few decades and this training allowed one clinician to "completely [learn] some things that I had half learned… and I unlearned a couple of things that were… wrong… on top of experience with those evaluations 20 years ago… I learned how things had been refined."

Individual specification
Participants' roles and backgrounds in developmental care varied but they seemed to fall into two groups, namely specialists who could use tools and strategies introduced in the training to strengthen current practices and non-specialists who were looking for ways to improve care for autistic patients as they wait for a specialist appointment.
One participant mentioned that they "don't work with autism, so… [the training] was a way of getting… more than just book knowledge…" They described a desire to be better able to recognize "signs we might want to be noticing… And… how do I take that information from the trainings and put it into some practice?"Several participants discussed patient education with one participant describing they were looking for "tools and small things that we can support parents on doing to help their child in the interim [while waiting for Applied Behavior Analysis or ABA]?"

Communal specification
Participants discussed shared goals with colleagues, patients, and families.For example, one participant wanted to be better able to support parents who are "expressing concern or follow [ing] through on the conversation the pediatrician is having.So, we are not diagnosing but we're at least being able to provide bidirectional information."Participants described ways to improve their understanding of other staff roles.One participant expressed a desire to be "able to think and understand how pediatricians approach a patient,… What could we do?What could we hold with them so they're not holding it alone?"

Cognitive participation
Participants engaged in problem identification, collaborations required to solve such problem, and discussed sustainability of the new practice.Participants identified the gap between the current and new practice, reorganized their work accordingly, and reflected on their ability to implement the new practice.

109:
"I valued… the case presentations.This is gonna be such a psychiatrist answer… we think about things like the process vs. the content.And the content was obviously very informative and useful and filed away in some way.But it's more… that one level up.The process of [going] through a case presentation from another pediatrician at another health center about a tough diagnostic disclosure with a family with language and cultural and whatever barriers.Like that experience… and that one level up learning… I found the most valuable."102: "… I loved the videos, whenever those were brought in, I thought they were just such a great teaching tool like when [trainer] would show examples for instance of children that had autism or had of different ages.That was great because we obviously can't practice on Zoom or in these ECHOs but maybe a couple videos of observing different secondary screeners would be great, or screeners in general.I think just the video may be just a great tool."102: "…I would love like somewhat, so um maybe screeners available either in a virtual format or paper format kind of stocked in every room." 100 Reconfiguration Reconfiguration occurs when participants reflect on current or future/planned changes to their practice because of the BCAEP.

Initiation
Participants reflected on the gap between their current practice and new strategies introduced in the training.When asked about potential care improvements, one participant "foresee[s] that the majority of the time, when we have a child who has some developmental or other concerns that bring autism…, the child would have an answer or at least a next-step plan within two months."Another participant reflected on expanding their clinic's autism screening capacity by "dedicat[ing] some sort of FTE [full time equivalent] resource."This was plausible because their clinic was larger "compared to the other health centers…" and thus could "make decisions that smaller health centers can't make."

Enrollment
Participants discussed how they collaborated and learned, such as having autism screening trainings and separate groups to target unique aspects of autism diagnosis and care.One participant shared that their clinic was "doing a training… [for] our primary care providers in the RITA-T [Rapid Interactive Screening Test for Autism in Toddlers] screener."Similarly, another participant shared their clinic's desire to have PCCs "… doing [developmental] tests" in-house before making a specialist referral, which may involve long wait times.

Legitimation
Participants discussed experiencing increased confidence in administering and advocating for autism screenings as they reflected on the value of strategies taught in the training.One participant "did start doing CARS [Childhood Autism Rating Scale] evaluations after the [training] started, partially because I just had more confidence in doing it."Another participant described how PCCs in their workplace "are increasingly confident in using the autism word…"

Activation
Participants described their decision in enacting strategies introduced in the training.Specifically, how these strategies fit into clinic workflow by allowing PCCs to collaborate more efficiently and getting crucial steps in autism care done promptly.One participant shared that their clinic "ha[s] gotten more efficient, and the diagnosis is done quicker…" and "the referral… through the community health worker team is happening faster." The knowledge of different services needed for different age groups was also conducive to efficient care as described by a participant who was "better able to talk about what [services are] available in the moment…"

Collective action
Participants shared about implementing strategies introduced in the training within the context of current systems (e.g., referral process), barriers (e.g., the pandemic), time and resource allocations, and systems of accountability (e.g., staff roles and responsibilities).

Interactional workability
Autism diagnosis and care involve multiple collaborators (e.g., school, insurance agency).The efficiency of interactions with such collaborators affected the implementation of strategies introduced in the training.Barriers included long wait times, complexity of electronic medical record (EMR) systems, billing delays, and limited bandwidth.
One participant described "bewildering random delay as to whether… billing agencies will accept [autism] diagnosis."Even when patients "have the same [government] insurance" and presented with "the same [clinician] doing the same test with the same letter template," they may receive varying responses based on the agencies' understanding of whether they would receive reimbursement.Additionally, EMR systems were described as "not talking to one another… So, if you refer to developmental peds [with a different EMR system], I'm not sure how quickly you're actually being able to see [the referral]."Consequently, timely referrals did not always result in timely services.

Relational integration
Participants discussed structure and accountability within clinics needed for successful referrals to long-term services such as behavior therapy.One participant "would love someone to walk [families] through this [ABA] process because… you see [families] time and time after that and you're like 'Where are you in the process?' 'Oh,… we haven't really started or… we got stuck at paperwork which is… the very beginning [step].'"This issue did not only occur with ABA, as the participant continued that they "refer[red] a patient to EI [Early Intervention] and then like nothing happens."

Skillset workability
Participants described the distribution of responsibilities in implementing a new practice.Expanding training for autism screening beyond specialists could be key to timely referrals and care.One participant said, "If [screening and referral] was all in the hands of one person to…, it can become unwieldy and time intensive," causing the "diagnostic journey [to take] eight months… of [the child's] development."Multiple participants mentioned that it was helpful to have other PCCs who were able to conduct screenings.One participant felt "very fortunate to have [two colleagues conduct screenings]" as it "helped… take some of the stress of the long wait of getting an evaluation in our developmental clinic."

Contextual integration
Participants discussed barriers to autism screening and care such as gaps between appointments in pediatric patients.One participant described that "[pediatric visits] schedule goes from every two months to… every year… very quickly…."Due to age restrictions for some services, "that kid [becomes] too old for EI, [but] too… young for school."Combined with "specialists already hav[ing] a one year waiting list," the participant expressed a desire to "build something in our practice to catch [kids waiting for services]."establish connection with key collaborators.Past studies indicated that improvements were needed in follow up care of children who screened positive for autism (34,35).Although additional research is needed to examine factors affecting low referral rates to follow up services (35), increased understanding of services may enhance PCCs' ability to navigate and advocate for timely service receipt.
Additionally, participants reflected on action steps they took to implement strategies introduced in the training, which were coded using the collective action construct.Barriers to autism screening and care were also discussed such as gaps between pediatric visits, EMR complexities, and inconsistent insurance requirements, all of which could contribute to service delays and aligned with prior research (9).Some barriers were outside clinicians' control, yet participants were committed to improving care whenever possible.Considerable investment is needed to expand the developmental-behavioral pediatric workforce and services (36), but enhancing aspects of care within a clinician's control may be one step closer to short-term care improvement.
For example, easy access of screening materials (e.g., printouts in exam rooms, centralized location of digital copies), clear followup protocols with defined responsibilities such that screenings have actionable outcomes, and enhanced patient education.Enhanced longitudinal family-clinician rapport that could result from these care improvements may further facilitate family's engagement with clinician recommendations, as found within the context of Latinx families (37).
Lastly, the training was well-received as evident from transcript text coded using the reflexive working construct.Participants cited the feeling of community with other attendees.Connection with other clinicians were found to be facilitators of clinician wellbeing (38,39), which is crucial to maintain for an increasingly strained workforce (40,41).The live, synchronous format of the training where trainers could engage attendees in real time appeared to be key in fostering such connection.Moreover, participants reported renewed focus on autism and decreased tolerance of wait times as they now had the tools and strategies to help remedy the situation in the short-term.
There were several limitations to this study.First, there was limited transferability of findings due to small sample size (n = 10) and self-selected participants from an urban area in northeastern United States.Participants, however, represented most of the eligible health centers (85.7%) with varying patient populations.Moreover, there was a large range in years of experience despite similar levels of prior interest in autism.Second, clinicians who participated in focus groups may be subject to social desirability bias as many knew and worked with each other.Findings however were gathered from a mix of focus groups and personal interview data.

Utility of NPT
NPT provided a helpful framework to organize qualitative data into actionable recommendations.Qualitative data analysis revealed participants' motivations, attitudes, and perceived barriers regarding autism screening and care management strategies taught in the training.Findings could guide efforts to enhance and sustain implementation of autism screening and care management in diverse urban settings.NPT could also be used to guide evaluations of other clinician training programs in addition to implementation of specific protocols in healthcare settings (23).
Similar to findings of a review of studies using NPT, we found overlaps between NPT constructs that made it difficult to assign a single construct to our data (22).For example, activation under cognitive participation and individual appraisal under reflexive working.Activation occurred when participants decided to enact a new practice whereas individual appraisal occurred when participants evaluated the value of a new practice after they had enacted it (21).However, participants tended to discuss these topics simultaneously; their decision to enact a new practice was implied by their reasoning involving the value of such practice.Although the overlap did not impact actionable recommendations out of the analyses, it is important to consider when using NPT.
Moreover, because participants varied in their baseline knowledge and experience with autism care, the novelty of their practice change and whether it was influenced by the training was unclear.As such, it was difficult to assign data to the construct differentiation under coherence, which referred to the contrast between existing vs. new practice.Another construct that was challenging to apply was systemization under reflexive working, which referred to participants' way of evaluating a new practice (21).The definition could apply to both formal (i.e., quality improvement studies) and informal (i.e., conversations) evaluation methods, but participants tended to share their thoughts about the training without sharing specifically how they gathered data to come to their conclusions.It was largely implied that they evaluated the training through personal observations and informal conversations with colleagues.Lastly, the current study focused on two innovations, namely the tools and strategies introduced in the training as well as the training delivery itself (i.e., virtual, year-long, inter-professional training).Careful attention was paid to specify which innovation codes applied.

Conclusion
PCC training on autism screening and care management could potentially address service access issues.There were distal barriers outside of a clinician's control, but equipping clinicians with knowledge and self-efficacy about autism care may help address proximal barriers within their control.NPT allowed for detailed assessment of process and outcome evaluations for a PCC training program, identification of gaps, and practice recommendations.Moreover, NPT was useful in highlighting both the process (training delivery) and outcome (practice change) aspects of evaluation and providing a framework for delivering recommendations to program implementers.Lastly, NPT could be used as a guiding framework for other clinician training programs, however, defining the new practice of interest may need to be further clarified when working with a participant group with varying baseline knowledge.
Appendix A. Focus group guide with sample questions/probes Part I: individual-level application of ECHO Autism strategies (Suggested time: 15 minutes)

TABLE 1
Focus group and interview participants characteristics (N = 9) a .

TABLE 2
Qualitative findings from focus groups and interviews with primary care clinicians organized using normalization process theory.
support them with management afterwards."108: "…I've been in practice for 25 years.And I never had this many autistic kids, and I'm not talking about the subtle ones, I'm talking the ones that are so obvious.I don't know what's going on but it's really, it's really taken up a great part of my practice, so this was terrific."101: "… I would have loved to… [learn more about] the ABA portion itself.I find it very difficult to address [ABA] with parents… because… I don't really know how ABA.I mean I have an idea.But really, because parents ask and say 'Okay, what is ABA and how is it going to get better with ABA?' And that's something I'd love to try to explain better…" 105: "So I think that all of it was… more learning for me.Because we don't work with autism, so… it was a way of getting more than just reading, more than just book knowledge, and thinking about what [we are] looking for.So, for the team that's working with the [young children], or even someone who is a teenager: what are those… signs we might want to be noticing that we can then loop back and say, 'Hey, we've been spending this amount of time, and this is what we noticed'.And so, for me, this is about how do I take that information from the trainings and put it into some practice?"105: "Our family partners are pretty amazing in the work that they're doing.And I would love to learn more about what Early Intervention is doing on… Are there these little tidbits that we can do in primary care?We're not gonna be able to do what in-home… or true ABA or anything like that [does].But are there tools and small things that we can support parents on doing to help their child in the interim?" 34 (Continued)

TABLE 2 Continued
patients, and families.Participants discussed how to collaborate with colleagues to improve clinical systems and be on the same page about a patient's diagnosis.Participants found it useful to hear about challenging cases that came up in other healthcare centers and how other clinicians addressed the situation in informing their own understanding of these screening tools.Further, participants expressed a desire to better understand the roles of others acting in the field (e.g., pediatricians, ABA providers) to work toward a common goal.105: "Or if parents are expressing a concern being able to support or follow through on the conversation the pediatrician is having.So, we're not diagnosing but we're at least being able to provide bidirectional information.And I think that's what really important."109: "I think that I've felt more confident through hearing other health centers talk about maybe those challenging cases….How do you talk about what ASD [autism spectrum disorder] is….When we look at it through a lens of a biomedical sort of Western medicineinformed diagnostic entity….but I'm seeing a newly immigrated from Albania family that doesn't speak English… In their culture, there's an idea that… not talking until the age of five is actually pretty typical, and [the family is] not concerned about that at all….We need to somehow meet each other in the middle… and do what we think is in the child's best interest while honoring that family's values….I think the ECHO was also really useful to hear different clinics talk about those types of dilemmas, and this is how I would talk to the parent about that…" 105: CARS evaluations after the program started, partially because I just had more confidence in doing it… The CARS doesn't have an official training per se.I think that part of the reason I started doing them during the program was cause I just, it gave me a little bit more confidence, and partially because that's just kind of when things fell into place."110:I'vebeenat[communityhealth center] for three years.In the early months, it could feel like,… I might see a kid over three sessions get the assessment and the diagnosis done.Involve the referrals team, and it, it might be like four, five months until we feel like that kid finally gets connected with a, you know, ABA team.And that's the first way to answer your question is that I think we've gotten more efficient, and the diagnosis is done quicker….I have some kids that the diagnosis letter is written, you know, within a week.I saw them, and the diagnosis is clear.We make the diagnosis, and then the referral to those services… through the community health worker team is happening faster, so that's one improvement is just the speed of all that."102:"…Betterable to talk about like what's available in the moment… For instance, Early Intervention for the younger kids or school and getting them set up at schools, and then again like the ABA for afterwards."Collectiveactionrefers to the action steps that participants need to take to implement new practices they learned from the ECHO training program.This involves interacting with the systems already in place (e.g., referral process), barriers (e.g., COVID), allocating time and resources, and building systems of accountability (e.g., defining staff roles and responsibilities) to ensure that these changes are implemented effectively.We still are stuck with this bewildering random delay as to whether basically billing agencies will accept those diagnosis.So, our patients all have the same MassHealth insurance.They all have C3… and so we'll refer someone to ABA through an agency that has a short waiting list.And they'll accept our diagnosis because they think that they'll get paid eventually through MassHealth.Then, they'll get a big waiting list, and we'll go to a different agency, and the different agency won't accept our diagnosis because they think they won't get paid through MassHealth.Now that's only because understanding what MassHealth will do and won't do is… a bewildering sort of mess.But we have the same person doing the same test with the same letter template… They're literally all… quite the same… and we get different responses… and that's because agencies don't know what's going to happen, right?And no agency is going to do… six weeks' worth of work and then… feel like they're not going to get paid for it… Although… it would surprise me if any of the agencies could actually trace back any of their individual units of work to… real specific dollars that come in… for services because we certainly can't at the health center.##So…wehave…no idea whether… we're being reimbursed for the service codes and… diagnosis… for the visits [and] for the screening that… we're doing."107:"…Becausebeyondnot being able to match people with ABA therapy… the real troubles that we've had over the last couple of years is that our local Early Intervention program has been almost missing in action.I mean… [state] school department certainly has been missing in action, right?[Moderator 1: yeah] So… we have to coordinate with those two agencies and….I mean I… know why our local Early Intervention program was missing in action 'causeit was all the same things that we were going through also…" 105: If it was all in the hands of one person to be responsible for all of those pieces, I think it can become a little unwieldy and time intensive.And then you're back to the timeline of… we eventually got the kid where they needed to be, but that diagnostic journey took eight months, and that's eight months of their development… So that's the main thing is I think that[integrated behavior health program], multidisciplinary approach at [community health center] is gonna help us continue to improve our autism assessment process."109: "There's talk at the clinic of expanding the training for other providers including the pediatricians themselves to become more skilled with the RITA-T and the CARS [Moderator1: Yeah].Training them in how to do the diagnostic… letter… that insurance will accept… in order to get ABA, the IEP, all those services going… And that sort of to get better as a clinical team… [in] involving behavioral health staff, involving primary care provider, to… maybe catch those kids that fall under the bucket." is that the visit… schedule goes from every two months to… every year… very quickly….. Then what happens is that kid is too old for Early Intervention, [but] too… young for school….And the specialists already have a one year waiting list for kids that are 19 months old….So now, we want to build something in our practice to catch that… but we don't have enough kids in our practice to build that system-to run 1,000 kids through in that age group, you know what I mean?So… the math doesn't work for practices I guess."102: "… We're in kind of like a paper world.So, you get the paper screener… and then it's put into the computer, or you can yourself put it in or have your medical assistant [or] whoever.Sometimes there are lags or drops in [that] process.I think the ultimate goal and what most clinics should be really doing in this day and age is… everything should… just [be inputted] directly into the computer….This is probably off topic, but a lot of practices are like having parents screen things prior or in the waiting room in a virtual kind of format.So, we weren't there yet so we were kind of talking about that process."103: I mean we… were scheduled with-I mean, for us in, at [community health center], the nurse practitioners get a little bit longer [visit time].But even still people run late, it takes time to room, then you got your next patient waiting.And so, it's really hard.And… to do a full autism evaluation and ADOS or… whatever it is they're doing now, it's like an hour, right?And so, it's obviously really hard to… get a full picture of a kid's development in… a 15-minute timeframe.And so, I do feel like the training has helped in terms of making my time with a family more efficient and trying to figure only three pediatricians and me… and we are teeny tiny.We have two exam rooms each, and we are all four of us in a tiny office.So, we don't have tool kits.I guess Dr. ### has her own, she's our newest… provider, but feel like we work in silos with regard to… autism, how we approach autistic kids because we have kind of systems in place…" 32 Reflexive working occurs when participants evaluate the training that they received as well as the implementation of strategies that they learned during the BCAEP... four to six hours a week to just do… the screening the screening diagnostic, the secondary screening diagnostic tests, wait, the secondary screening tests… Because we wanted to try to move kids faster through the early intervention… and school bottleneck… And that's working reasonably well… which is a big… improvement from… before… As a result of that I believe that we've been able to get kids seen at specialty programs or by neurologists more quickly than if we had not been doing that."we got to know each other… you could tell… some personalities throughout the end for sure.And I love the fact that I could see into the thoughts and experiences of others in my very close vicinity that I never would have met, absolutely never would met otherwise especially post-pandemic… And to hear maybe some of what other practices are struggling with or doing well with so that whole like community feel I thought was great."103: "…It's always nice to hear about how things run differently in different clinics, because you start to realize well, maybe we could be more efficient in this way.You know, we could… adopt some of the… processes that are used in other clinics.And so… I feel like it's nice, … not only for getting to know folks in the community, but also… sharing ideas about what's worked, what hasn't worked… in their clinics and ways to improve ours." the other health centers… that were involved in the ECHO, [we are] so much bigger… that we should be better able to… make a change like that.Whether we actually are good enough to actually make a change like that, is… a whole different issue you know what I mean?But… we can make decisions that smaller health centers can't make…."110:"Oh,howI[determine whether new protocols are working for my community health center]?I would foresee that the majority of time when we have a child who has some developmental or other concerns that bring autism…-the child would have an answer, or at least a next-step plan within two months.You wouldn't have a child that's eighteen months old and wait for a diagnosis or at least for a next-step screener for more than two months.[Moderator2:Mhm.Okay.]Orolder"Listen, it is glaringly obvious that this is a child who has autism.And in the old model, we could put in a referral to [outside institution] developmental-behavioral peds, and know that eventually they get to a great clinician who's gonna do the assessment, get that diagnosis, and then get them the services that they need.But that bucket of kids might be better served if we can keep them sort of in-house with the [community health center]'s team and use these tools that, you know, [psychiatrist] was using….There [are] probably some kids that the PCPs more and more could and should be doing those tests, and that's what [my clinic] is trying to, to do moving forward." 106: "As I said, we're sort of working on, we had just started trying to become more autism friendly and… then the pandemic came.So, I think that sort of a combination of that, plus the ECHO plus [clinician]'s mini fellowship.[Clinician] and I are working on making almost sort of a mini developmental clinic day type thing so that we have much better, we have more wraparound services than we had before."110:"Iwouldn't say that we've implemented any direct change yet.But we're doing a training on April 13th with a… developmental pediatrician who was at [academic institution] and is now at [hospital] training our primary care providers in the RITA-T screener.[Moderator2:Mhm] With the plan that we'll start a practice of, um, using the RITA-T to decide who is-has a high enough score that that could be consistent with a diagnosis for autism and actually writing the letter to diagnose within our primary care practice, and who's in this kind of middle of the road area where maybe a developmental pediatrician" [106: Your EHR doesn't talk to people?You don't have that?]105:Haha,we have EPIC, but care everywhere has its limitations, right?So, if you refer to developmental peds, I'm not sure how quickly or what you're actually being able to see.And we have EPIC OCHIN.We're not fortunate to have EPIC EPIC, so.[106:You know, it's better than when the EMRs didn't talk to each other.]105:True.True.(Laughing)"106:diagnosis and write a letter support for ABA and I send a parent out and there's… nobody to support them.I know we have the autism team but they are now limited in how.., this is maybe internal to our clinic or our institution but… I would love someone to walk [families] through this [ABA] process because… you see them time and time after that and you're like 'Where are you in the process?' 'Oh,… we haven't really started or… we got stuck at paperwork which is… the very beginning [step]'….But that may require funding for somebody in.. or, I don't know if there's an online format for someone.. some website that teaches you how to get that stuff that you need, you know even from EI, if not for the.. our patient navigators that are in clinic that are wonderful, and I message them cause… you refer a patient to EI and then like nothing happens."105:"Andthen I would second what you said about the ABA and I'm trying tothink about would it be beneficial for… some of our staff to be trained to some degree in it so that we can do some little tidbits of support."106:"We'recontinuing to have, unfortunately, a lot of trouble getting kids hooked in with ABA, and part of it is that MassHealth now is claiming that I am not qualified to do autism evaluations.[Moderator 1: Why is that?]I think that there may have been a misinterpretation at some point about part of their blurb which says "expertise" out: Am I concerned?Am I not concerned?And… what is the concern?…Is it really just… a language delay or is there some additional piece, like the social communication, things like that?" 104: "There's

TABLE 2 Continued
After the… training, [I] made a point of using the word[autism].And…I don't know if I was shy about it before or not, but what I learned was that most parents were thinking it anyway, and if you didn't say it, then they continued to worry inside that they might be crazy or something, because… almost none of them rebelled against it.You know, they almost all, like, appreciated hearing that."110:"…SoI guess the only thing I'll say is that it made us even less tolerant of the wait times and moving more things and like doing more work to kind of move heaven and earth to not let the wait times be a barrier for our patients… That's probably the way that the ECHO impacted things." 1. What skills are you applying, meaning what are you able to do differently because of what you learned from the ECHO Autism Program?Are any skills missing from what you received in the ECHO Autism Program? (Probe for screening, referral, treatment) 2. What knowledge did you gain, meaning what did you learn that you can apply to your practice and relay information to your patients?Are any pieces of knowledge missing?(Screening tools, referral protocols, resources) 3. How did the program change your perspective on the education and care of your patients?(Probe for comfort of using terminology, confidence in referring, managing care) Part II: clinic-level application of ECHO Autism strategies (Suggested time: 15 minutes) 1.How has your clinic's practice changed because of your participation in in the ECHO Autism Program?Who have been implementing these changes?(Probe for administrative support, connection/community within the practice) 2. What are some of the benefits to implementing the strategies from the ECHO Autism Program training given your current clinic context?Who are the recipients of these benefits?(Probe for EMR supports, protocols for referral and screening, sense of community) 3. How do you determine if strategies from the ECHO Autism Program training are working for your clinic?(Probe for patient satisfaction, staff satisfaction, burnout factors (e.g., feeling worthwhile at work, work is satisfying/ meaningful, they are contributing professionally in ways they value) Part III: opinions about the ECHO Autism training (Suggested time: 10 minutes) 1.What parts of the ECHO Autism Program training do you value the most?(Probe for parts that can be transferrable to other clinics and avenues.) 2. What could have been done differently during the training to better meet the learning objectives?Reflect on your experiences with case presentation and discussion.(Probe for support from clinic administration, further resources, etc.)