A general framework for fidelity assessment

Abstract

A general framework for fidelity assessment is outlined to establish common concepts and common measures related to the science of innovations, implementation, and scaling. Fidelity is defined as the extent to which the essential components of an innovation are used in practice. The function of fidelity assessment is to measure the presence and strength of the essential components of an innovation. The general framework identifies the dimensions of fidelity as context, content, and competence, and identifies direct observation, record reviews, and asking others as methods to measure the presence and strength of the essential components. Previous frameworks and uses of fidelity at scale are reviewed. Multifaceted fidelity assessments are used to illustrate the categories in the general framework for fidelity assessment. The general framework can be useful to support quality assurance and sustainable scale-up of interventions, especially for those provided to adversity-affected communities and in low-resource settings.

Introduction

The interest in evidence-based innovations can be traced to a review of therapy outcome studies by Eysenck (1) who concluded that, “the existence and effectiveness of [therapy] is…unsupported by any scientifically acceptable evidence.” The review was the beginning of decades of research to develop effective innovations and implementation processes (2–5). The quest to produce scientifically acceptable evidence focused attention on the research designs and statistical methods used to generate evidence (6, 7). As improved research methods were used to study outcomes, internal validity (the basic minimum without which any experiment is uninterpretable) and external validity (the question of generalizability) of the innovation (the independent variable) became topics of concern (7). In this context, Yeaton and Sechrest (8) pointed out that consistent monitoring is needed to detect the degree to which treatment is delivered as intended. For an innovation that is not used with integrity “its failure to have any effects becomes totally uninteresting” (p. 161).

As a result of early efforts to define innovations and evaluate outcomes, fidelity of the use of innovations emerged as an important part of the evidence-based movement. Over the years, fidelity has been the subject of systematic reviews (9–18). A recent systematic review of fidelity assessments related to nonspecialists who deliver interventions in lower resource settings concluded that more precise terminology is needed across the implementation and fidelity literatures and greater clarity is needed regarding concepts such as competence and adherence (19).

The goal of this paper is to summarize efforts to define and categorize fidelity and to propose a general framework for fidelity assessment that can serve as a guide for summaries of the literature and for the development of future assessments. Examples of fidelity assessments are provided to illustrate the categories in the general framework.

Fidelity and innovations

Fidelity is defined as the extent to which the essential components of an innovation are used in practice (20). Innovations are used in practice by “intended users” such as therapists, implementation specialists, improvement managers, or others. The fidelity assessment is linked to outcomes. The purpose of evidence-based programs and other innovations is to provide benefits to individuals and populations. Thus, fidelity links the presence and strength of the essential components with beneficial outcomes. These factors are shown in Figure 1. What is intended? Are the intended components assessed? Does doing what is intended matter? Fidelity assessment provides assurance that the innovation is used as intended and that outcomes reasonably can be attributed to the innovation.

Figure 1

Innovations are defined by their essential components. The function of fidelity is to assess the presence and strength of the essential components of an innovation. As noted in Figure 1, the essential components are the aspects of an innovation that have to be there or the innovation is not the innovation. For example, a plan-do-study-act-cycle improvement process has five essential components (PDSAC). The plans vary widely depending on the problem to be solved, cycles may be short-term or occur over many years, and so on. However, it is not an improvement process unless all five components are used as intended (21, 22).

In Figure 1, the innovation refers to human behavior that is interactive and reactive (23). The related literature and the iterative process for establishing the essential components of an innovation have been described (24). The interactive nature of innovations involving human behavior complicates the life of behavioral scientists and practitioners, and it makes fidelity assessment critical for science and practice.

For science, fidelity provides assurance that the procedures described in the methods section were carried out as intended as noted by Mediavilla, García-Vázquez (25) in their experiments with the Doing What Matters in Times of Stress program. This has been referred to as procedural fidelity (26, 27). If the fidelity assessment indicates that the essential components are present, then the outcomes can be related to the intended essential components (28–31). Of course, it is quite possible to have well-established essential components and a good fidelity assessment and also have poor outcomes. For example, Acarturk, Kurt (32) used a well-described group problem solving intervention with Syrian refugees in urban settings. Fidelity was assessed at 78% but depression and anxiety outcomes were not improved.

Even so, the clarity of what was done and what was found provides a useful starting point. As research progresses, the essential components can be revised and refined (33) and the fidelity assessment can be revised and refined (34). As research progresses, an innovation is developed so that the outcomes become more positive and certain. When there is sufficient evidence of the essential components-fidelity-outcomes link, then the innovation is a candidate for scaling in practice.

For practice, as the innovation is replicated and scaled to achieve meaningful benefits for a population, fidelity provides assurance that the essential components of an innovation are present and at sufficient strength to produce beneficial outcomes in each use (35–37). The established essential components-fidelity-outcomes link means that fidelity assessment is the primary measure of scaling success. If fidelity scores are high, then outcomes should be quite good. This is important because fidelity can be known in the present time while outcomes likely will not be realized until some future time.

The importance of fidelity in practice is highlighted when tasks are shifted to non-specialists or when conditions relevant to an innovation are changed in a planned way (19, 38). This may be required in low resource environments or when availability of candidates to fill required roles is limited (39). Fidelity assessment is the sensor to detect divergence from the intended use of the essential components of an innovation. It separates the who and the what: even though who is using the innovation to provide services may change in a planned way, what needs to be provided still must meet fidelity requirements so that intended outcomes can be achieved.

Fidelity dimensions and measures

A framework is a mid-level theory (40), a conceptual framework to account for current evidence and to provide testable statements to produce new evidence (41). Current evidence regarding the fidelity dimensions and measures proposed in the general framework is derived from often cited approaches to fidelity assessment and from the dimensions identified in systematic reviews of the literature regarding the measurement and uses of fidelity.

Various frameworks have been proposed to define fidelity dimensions and measures. The first framework for assessing fidelity was described by Hall and Loucks (42) as part of the Concerns-Based Adoption Model (CBAM). The CBAM Innovation Configuration framework is used to operationalize the essential components of an innovation to distinguish “the behaviors of innovation users and nonusers” (p. 265). The CBAM framework continues to be advocated as a way to define (innovation configuration) and assess innovations (43–45).

Subsequent fidelity frameworks continued to focus on the essential components of innovations (46). Waltz, Addis (47) described the dimensions of fidelity as adherence and competence and noted that “competence presupposes adherence, but adherence does not necessarily imply competence” (p. 620). Perepletchikova, Treat (48) used the same adherence-competence dimensions in their review of fidelity in psychotherapy outcome studies. Dane and Schneider (13) further expanded the adherence dimension of fidelity to include exposure, quality of delivery, participant responsiveness, and program differentiation. A systematic review and analysis of 20 fidelity frameworks found “constructs relating to the delivery of an intervention as intended (i.e., adherence, dose, quality) are at the heart of the concept of fidelity” (14).

Mowbray, Holter (49) provided the next advance in fidelity frameworks, adding structure and process as dimensions in addition to the essential components of the innovation. Century, Rudnick (46) also referred to structure as the method for service delivery (e.g., one hour verbal exchange in a clinic) and process as the way the services are delivered (e.g., rehearse new behavior). Fidelity criteria include a) dosage (the length, intensity, and duration of services), b) content, procedures, and activities over the length of the service, c) roles, qualifications and abilities of staff, and d) inclusion/exclusion characteristics for the target population. Carroll, Patterson (50) focused on dosage or exposure (the amount of an intervention received by participants) and adherence (the frequency, duration of the intervention being delivered), and added another dimension named coverage (“whether all the people who should be participating in or receiving the benefits of an intervention actually do so”).

Measurement of the dimensions of fidelity has received attention as well (17). Waltz, Addis (47) encouraged assessment of behaviors that are essential to the treatment as well as behaviors that are proscribed. Mowbray, Holter (49) described fidelity measures that include observations by experts, documentation in program or client records, and interviews or surveys of individual receiving services or closely related to them. Sanetti and Collier-Meek (26) and Bergmann, Niland (34) have described and compared various direct observation and permanent product methods to assess adherence and fidelity. The review conducted by Bond, Simmons (19) found most fidelity assessments include direct observation of performance, even in high stress environments. Schoenwald and Garland (16) conducted a review of the literature and identified 249 unique fidelity measurement methods related to a wide variety of treatment models, recipients, and practitioners. They found that 71.5% of the measures were based on direct observation.

Mowbray, Holter (49) encouraged correlating fidelity and outcomes. Correlating fidelity and outcomes separates form (e.g., frequency, duration, exposure, dosage, content) and function (e.g., credible assessment of essential components that are known and that have a beneficial outcome when used as intended). In their review, Dane and Schneider (13) found that 13 (8%) of 162 outcome studies analyzed the relationship between fidelity and outcomes. Schoenwald and Garland (16) found about 10% of the studies in their review reported fidelity-outcome relationships. In an analysis of fidelity-outcome relationships, Strain (51) “found that providers need to approximate high fidelity before significant changes in child behavior are evident” (p. 110). This is the first suggestion that there may be a curvilinear relationship between fidelity and outcomes, with desired outcomes being achieved only after fidelity is above 80%. Brunk, Chapman (52) found a non-linear pattern for fidelity with the MST Program Performance Index (PPI). An, Dusing (53) suggested setting a predetermined benchmark for fidelity (≥80%) to ensure the independent variable is available prior to conducting an experiment.

In summary, previous frameworks have identified adherence, competence/quality of delivery, structure, and process as dimensions of fidelity, with exposure/dosage/content, quality of delivery, and roles/qualifications of staff as subcategories. Measurement is with direct observation, information from documentation and records, or interviews or surveys of individuals receiving services or closely related to those individuals.

Considering the dimensions of fidelity

While there is growing clarity about the dimensions that define fidelity, the factors outlined in Figure 1 offer clarity about what fidelity is not. For example, in the literature cited above, Dane and Schneider (13) proposed participant responsiveness and program differentiation as dimensions of fidelity. Carroll, Patterson (50) added a dimension named coverage (“whether all the people who should be participating in or receiving the benefits of an intervention actually do so”). In a review of the literature, Toomey, To (14) identified 20 fidelity frameworks and all the frameworks considered fidelity as a multi-faceted concept comprised of multiple dimensions. The identified dimensions were delivery (e.g., adherence, exposure, and quality of delivery), participants (e.g., receipt of, responsiveness to, and engagement with an intervention), design (e.g., theoretical fidelity, intervention differentiation, and intervention complexity), training (e.g., to facilitate their delivery of the intervention as intended), and context (e.g., recruitment of intervention recipients, organizational characteristics, patient and provider socioeconomic status).

Fidelity in everyday use

While frameworks for fidelity assessment were being proposed for outcome studies, evidence-based innovations that had been established, replicated, and scaled were using fidelity assessments on a broad scale. Several examples are described in this section. While the innovations vary widely, in each case the fidelity assessment is linked directly to the unique essential components of the innovation.

For example, the Teaching-Family Model (54, 55) began in 1967 at Achievement Place, a group home for adjudicated youth referred by the juvenile court. Initial attempts to replicate the evidence-based program produced mixed results. A fidelity assessment was developed (56) to discriminate successes from failures and has been in use for over 50 years in every Teaching-Family Site (57, 58). Additional information on fidelity related to this program is presented in later sections of this paper.

Assertive Community Treatment (ACT) is another early example of an evidence-based innovation (59, 60). ACT provides intensive support for adults with serious co-occurring mental health and substance use disorders so that they can function in the community and avoid hospitalization. The Dartmouth Assertive Community Treatment Scale (DACTS) (61) was developed to assess the fidelity of the use of the essential components of ACT. Ongoing use of fidelity assessment is reported by Harrison and Taylor (62) who describe 10 years of DACTS fidelity data for all ACT programs in one state mental health system. DACTS includes 28 items that assess organizational processes, team staffing, and services provided by ACT. Using standard protocols, trained assessors rate information taken from several sources including reports of program behavior from program supervisors or staff; documents reflecting program authority, responsibility, policies, and procedures; management information system or other sources of quantitative data on staffing, clientele, and services. Ratings are made based on a 5-point anchored scale for each of the 28 items. For example, one item rates in-vivo Services defined as “Program works to monitor status and develop living skills in community rather than in office.” The anchored rating scale for that item is: Score of 1 = less than 20% time in the community, 2 = 20%–39%, 3 = 40%–59%, 4 = 60%–79%, 5 = 80% of total service time in community. The fidelity assessment items relate to adherence and structure measured by ratings of information obtained from documentation and records and from interviews or surveys of individuals.

Multisystemic therapy® (MST) is an intensive, short-term, homebased treatment program for adjudicated youths who are serious offenders (63, 64). A fidelity assessment called the Therapist Adherence Measure (TAM) was developed (65, 66). The TAM is used in each MST replication as MST has scaled in the US and Europe. TAM includes 26 items that relate to the use of the nine essential components of MST. After a therapist's home visit a designated (non-clinical) staff person conducts a telephone interview with the caregiver (usually a parent) and (separately) with the referred youth. Each individual rates each question on a 5-point scale from 1 (Not at all) to 5 (Very much). The data are entered into the MST online data collection and reporting system. The benchmark for MST fidelity is 16/26 items scored as 5 (Very Much). Sample questions are: “My family knows exactly which problems we were working on.” “The therapist tried to change some ways that family members interact with each other.” “The therapist understood what is good about our family.” The TAM also includes questions about proscribed practices: “The therapy session included a lot of irrelevant small talk (chit-chat).” “There were awkward silences and pauses during the session.” The fidelity assessment items relate to competence measured by ratings from those who have directly interacted with a practitioner.

Positive Behavior Interventions and Supports (PBIS) is a whole school intervention to reduce discipline problems and improve academic achievement (67). A fidelity assessment called the School-wide Evaluation Tool (SET) was developed to assess the use of the essential components of PBIS (68). The data are entered into the PBIS online data collection and reporting system. Ongoing use of fidelity assessment is reported by McIntosh, Mercer (36) who conducted an analysis of annual SET fidelity data from 5,331 schools for 5 years. SET includes 28 items organized into 7 subscales, each subscale relating to an essential component of PBIS. A designated staff person in a school collects the information and enters the fidelity data (along with student data) into the PBIS national data system where scores are analyzed, graphed, and available to each reporting school. Data are collected through a review of documents, observations, and interviews or surveys of staff and students. The items are scored on a 3-point anchored scale (0 = not in place, 1 = partially in place, 2 = fully in place). The benchmark for fidelity is an overall score of 80%. Sample questions are: “Is there documentation that staff has agreed to 5 or fewer positively stated school rules/ behavioral expectations?” (0 = no; 1 = too many/negatively focused; 2 = yes). “Are the agreed upon rules & expectations publicly posted in 8 of 10 locations?” (0 = 0–4; 1 = 5–7; 2 = 8–10). “Do 90% of staff asked agree with administration on what problems are office-managed and what problems are classroom–managed?” (0 = 0%–50%; 1 = 51%–89%; 2 = 90%–100%). “Can 90% of staff asked report that there is a school-wide team established to address behavior support systems in the school?” (0 = 0%–50%; 1 = 51%–89%; 2 = 90%–100%). The fidelity assessment items relate to adherence and process measured by ratings of information obtained from documentation and from interviews or surveys of individuals.

The Parent Management Training-Oregon™ (PMTO) program is a family-based parent training intervention for children/youth 2–18 years of age with disruptive and anti-social behaviors (69). A fidelity assessment called the FIMP (Fidelity of Implementation) was developed (70). The FIMP is used in each PMTO replication and has been scaled in Michigan (71, 72) and nationally in Norway where fidelity, implementation processes, and outcomes were continually assessed for more than 10 years (73–76). The coded items relate to the five essential components of PMTO intervention. Coding is based on direct observation of videotaped intervention sessions. Therapist behavior is scored on a 9-point scale where 1–3 is “needs work,” 4–6 is “acceptable,” and 7–9 is “good work.” The benchmark for fidelity is a score of 7–9 (good work). Reliable coding requires about 40 h of training for fidelity assessors. Therapist behavior that is observed and coded on the videotapes includes: Demonstrates integration of PMTO tools. Manages orderly flow. Assesses skills and fills in gaps. Prevents/manages conflict. Overall accomplishment of goals. The fidelity assessment items relate to competence measured by direct observation of practitioner behavior by trained assessors.

Active Implementation Specialists™ conduct coaching for competence, an implementation innovation (77). Coaching for competence is based on direct observation of practitioners using an interaction-based program (78, 79). Newly trained practitioners may falter with the words to say and things to do and often do not recognize the cues (good judgement) to use (or not use) parts of the program. Errors of omission are difficult to detect without direct observation of the practitioner and the recipient interacting with one another. Direct observation also provides information on the quality components (tone of voice, proximity, respectful language, body movements, use of humor) and other nuances that may influence outcomes (80). Implementation Teams use the best practices for coaching to assess fidelity of coaching behavior. Two Implementation Team members directly observe the coach and review relevant records. Each item is scored as fully in place, partially in place, or not in place. The items relate to coaching frequency, documentation, and meetings. Expected frequency of coaching is set in advance and is adjusted as practitioners gain experience and grow in their skill competency (e.g., pre-fidelity, after meeting fidelity criteria twice). Required documentation (e.g., written feedback after each coaching visit) and the timeline for submission by the coach (to whom, by when) is specified. Within 24 h of a direct observation the coach meets with the practitioner, provides a verbal summary of the practitioner's use of the essential components, and engages in practice to criterion of one or two skills that need improvement. A brief written summary of observations and recommendations is provided to the practitioner within five working days. The fidelity assessment items relate to adherence information obtained from documentation and competence measured by direct observation of coach's behavior.

In summary, in these diverse examples fidelity is linked to the essential components of each of these innovations and is sufficiently practical to be assessed continually in replications and scaled use over extended periods of time. While not reviewed here, in each case the fidelity scores have been correlated with intended outcomes to establish the essential components-fidelity-outcomes relationship.

A general framework for fidelity assessment

Previous frameworks and the use of fidelity assessments in standard practice provide a firm foundation for a general framework for fidelity assessment. Given the critical role of fidelity in science, implementation, and scaling (20, 23, 81–83), this paper outlines a general framework for fidelity assessment.

Fidelity is defined as the extent to which the essential components of an innovation are used in practice (20, 24). The function of fidelity assessment is to ensure the presence and strength of the essential components of an innovation. The test of a fidelity assessment is the relationship between fidelity and outcomes (84). A strong correlation (±0.70) between fidelity scores and outcome scores indicates the essential components have been sufficiently identified, adequately measured, and meaningfully related to desired outcomes (24, 85).

The general framework for fidelity assessment provides a universal way to categorize existing fidelity assessments and a guide for developing new assessments. It is intended to clarify “what is fidelity” and “how to measure fidelity” and to establish a common set of concepts to integrate views of fidelity. In recent years, innovation fidelity has been discriminated from implementation fidelity, where implementation fidelity refers to the use of the essential components of an implementation process such as training and coaching processes for practitioner competency development, or processes to create readiness for leaders to engage in organization change so that programs can be used effectively. In the general framework for fidelity assessment, fidelity dimensions and measures apply equally to innovation fidelity and implementation fidelity. Use of the common concepts and common categories of measures are intended to facilitate future reviews and advances pertaining to fidelity, implementation, and scaling.

As outlined in Table 1, in the general framework for fidelity assessment there are three dimensions of fidelity (context, content, and competence) and three ways to assess those dimensions (direct observation, record review, ask others).

Dimensions of fidelity

• Context: Context relates to establishing the conditions under which an innovation is able to be used. This includes prerequisite conditions that need to be in place regarding setting, qualifications, or preparation so that the essential components can be used. For example, availability of equipment, location of services provided, timing, scheduling, or staff qualifications may relate to the presence or use of the essential components and be part of a fidelity assessment.

• Content: Content relates to ensuring the presence of an innovation. This includes the extent to which the essential components are used, referenced, monitored, or accessed and may include assessing the absence of activity or content that is proscribed or should be avoided. For example, indicators of using the procedures related to the essential components, attention to prescribed topics, sequencing of activities, and participation by certain people may be part of a fidelity assessment.

• Competence: Competence relates to the skillful use of an innovation. This includes the extent to which the competencies related to the essential components are used skillfully in interactions with others and may include indicators of the tone of voice or judgement or nuanced uses of essential components.

Measures of fidelity

• Direct Observation: Practitioner interactions with others can be observed with the fidelity assessor present (face-to-face) or using recordings of live interactions (video or audio) to detect and record information related to the use of the essential components of an innovation.

• Record Reviews: Information related to the use of essential components of an innovation can be available in some records (electronic or paper documents) such as treatment plans, progress notes, checklists, personnel records, and so on.

• Ask Others: Practitioner interactions can be assessed by interviews or surveys of opinions or experiences of those who directly interact with a practitioner and can reflect their personal experience and knowledge related to the practitioner's use of the essential components of an innovation.

The context dimension may include items that assess aspects related to the use of an innovation but not the essential components themselves. This was referred to as structure in other fidelity frameworks. For example, air quality and cleanliness may be assessed as part of a fidelity assessment related to a surgical procedure (86). They may not be essential components of the innovation as performed by a surgeon but they enable or facilitate the benefits of a surgical innovation as it is used.

The content dimension is related to what has been referred to as adherence or procedural fidelity where components of an innovation are scored as present or absent. For example, fidelity assessed the practitioners’ use of the instructional components of discrete-trial instruction (e.g., “present materials as written,” “provide praise contingent on correct response,” “record data before presenting the next trial”). The fidelity assessors scored the occurrence–nonoccurrence of each instructional component as a practitioner interacted with a child with autism (34).

The competence dimension goes beyond content and assesses the quality of practitioner behavior as it relates to using the essential components. For example, the competence of the use of motivational interviewing (MI) in clinical treatment for addictions was assessed (87). Fidelity assessment focused on 10 MI consistent items (those prescribed in MI) and 10 MI inconsistent items involving interventions antithetical to MI (e.g., unsolicited advice). Observers rated each MI consistent item for competence (skill of delivery, with 1 = very poor to 7 = excellent). Competence ratings were anchored with guidelines that reflected higher and lower levels of skill as the therapists performed each item. “Competence presupposes adherence, but adherence does not necessarily imply competence” (47).

The reliability of fidelity measures is assessed with interobserver agreement as two assessors simultaneously and independently observe and rate live observations, extract information from records, and conduct interviews or surveys (88, 89). The goal is to achieve an 80% benchmark for interobserver agreement. The validity of fidelity measures in assessed by relating fidelity scores with outcome scores. The goal is to achieve a benchmark correlation of ±0.70 between fidelity and outcomes.

For record reviews, the linkages between practitioner behavior and the record need to be established to ensure the record is a reasonable and reliable indicator of the presence, absence, or strength of actual practitioner behavior. For example, the record of a treatment visit may have a checkmark next to “capitalized on five or more teaching opportunities when using role play” or a record of staffing may indicate “the practitioner met 80% of all qualifications at the end of training.” The validity of the records must be established sufficiently prior to using the records as data sources for fidelity assessments.

For asking others, the validity of the information is assumed. If a parent rates the practitioner one way or the other, then it is assumed that the parent's rating is an accurate reflection of the parent's experience with the practitioner. Given this assumption, a fidelity assessment specifies who should be asked to ensure that each person being asked has sufficient exposure to the practitioner to make an informed judgement. For example, a fidelity assessor may interview each parent who was present during a therapy session or survey each teacher who is in direct contact with a practitioner at least once each week.

The dimensions and measurement methods are outlined in the 3X3 matrix in Table 1. For the fidelity frameworks reviewed in the prior section, structure, quality of delivery, and roles or qualifications of staff are Context dimensions. Adherence, process, exposure, dosage, and content are Content dimensions. And competence and quality of delivery are Competence dimensions. The data sources suggested by Mowbray, Holter (49) are used in the general framework shown in Table 1. Note that self-report typically is unreliable (81, 90) and is not included as a data source.

The general framework for fidelity assessment provides a way to categorize existing fidelity assessments. In Table 2, the fidelity assessments in everyday use referenced in a previous section are used to illustrate the categories in the fidelity matrix. Assertive Community Treatment (ACT) is based on a team of 10 qualified professionals (e.g., nurse, substance abuse specialist, vocational specialist, psychiatrist) who engage and support up to 100 adults with severe mental health and substance abuse problems who live in the community. The Dartmouth Assertive Community Treatment Scale (DACTS) items assess the Context in which treatment is provided based on evidence of staffing ratios, location of services, and the kinds of services.

Positive Behavior Interventions and Supports (PBIS) is a whole school intervention and the School-wide Evaluation Tool (SET) items assess the collective behavior of staff who generate school-wide expectations for behavior, systematically teach those expectations to students, and constructively deal with exceptions. Fidelity assessment is based on evidence of the Content of PBIS found in school records or by asking students and staff or by looking for posters in high-use locations (e.g., cafeteria).

The Parent Management Training-Oregon™ (PMTO) therapist records every session for Direct Observation of Competence. A fidelity assessor analyzes the video recording of therapist and family interactions to assess a therapist's use of the PMTO essential skills and nuances in response to the ongoing behavior of family members. Multisystemic therapy® (MST) is an intensive, short-term, homebased treatment program. A fidelity assessor Asks Others about Competence. After a home visit, a fidelity assessor contacts parents and the referred youth to rate the therapist's behavior as they experienced it during the home visit.

As noted previously, these fidelity assessments have been in use for many years in thousands of replications of the various evidence-based innovations. The fidelity items, measurement methods, preparation of fidelity assessors, and shared data systems provide a sound starting point for the development of any new fidelity assessment.

Fidelity assessment example: goals, essential components, skills

The relationships among the goals, essential components, and fidelity assessment methods for an evidence-based program are illustrated in this section.

A multifaceted fidelity assessment was developed for the Teaching-Family Model and has been in use in every Teaching-Family organization for over 50 years (57, 58). The goals of the Teaching-Family program are to provide treatment that is humane, effective, individualized, satisfactory to consumers, cost efficient, and replicable. The essential components were identified and operationalized in usability testing from its inception in 1967 through 64 attempted replications by 1975. The essential components of treatment are relationship development, teaching appropriate alternative behavior, motivation systems. self-determination, and counseling. Clinical judgement also is an essential component with a focus on morals (right, wrong), ethics (good, bad), and judgement (good sense) as Teaching-Family practitioners use the treatment components in their interactions with others. The essential components have been operationalized and the skills are taught in training and coaching and assessed regularly (24, 91). Multifaceted fidelity assessments are especially important in high risk circumstances, for example, when care is provided to dependent youths who are involuntarily placed in a relatively private residential treatment setting (58).

Fidelity assessment for the Teaching-Family Model was designed to assess the presence and strength of the essential components of the treatment program as they were used by Teaching-Parents (Teaching-Family practitioners). The complexity of the essential components of the Teaching-Family Model requires direct observation to detect their use or misuse, and to assess the nuances related to judgement (e.g., when and with whom to use/not use skills; missed opportunities to use skills). In addition, it is not enough to change youth behavior in the group home. Asking others provides an indication of the impact on (generalization of) youth behavior at home, at school, and in the community.

The goals of being cost efficient and replicable are not part of the fidelity assessment since they are not within the purview of individual Teaching-Parents. Independent analyses have established cost efficiency (92, 93) and ability to replicate the Teaching-Family Model (57, 94, 95).

Each fidelity assessment item asks, “How satisfied are you that…?” For the youths, parents, teachers, and others in the “asking others” column of Table 3, “satisfaction” is an undefined term. It is up to each individual to decide whether he or she is satisfied or not according to his or her own experiences and perceptions. If one person is less than satisfied, then that rating and his or her comments are taken seriously no matter how others might judge the reasoning.

For fidelity assessors, “satisfaction” is defined and observations are calibrated during the development of assessors and checked again by calculating interobserver agreement between the two assessors who conduct an in-home visit (77). During the development of assessors and periodically thereafter, the accuracy and quality of the comments accompanying a satisfaction rating also are the subject of discussion and agreement. This is important because the comments made by assessors are used to inform the Teaching-Family practitioner and coach so that excellent use of the essential components is reinforced with examples, and specific areas for improvement are noted with rationales.

As illustrated by the Teaching-Family example, fidelity assessment items relate directly to the skills needed to carry out the essential components of the innovation (e.g., relationship development, teaching appropriate alternative behavior) and can be summarized to assess the broader goals of the innovation as well (e.g., humane, effective, individualized). Components of the multifaceted Teaching-Family fidelity assessment fit the categories of the general framework and illustrate the use of the general framework.

Conclusion

Fidelity assessment is not a luxury or a burden. Fidelity is required in research to establish the evidence for the presence and strength of the independent variable (the innovation) so that outcomes can be related to the intended essential components described in a methods section. In subsequent research, fidelity is the standard to meet when attempting to replicate and scale an innovation. Fidelity assessment ensures the essential components are present and strong enough to produce expected outcomes in each use of an innovation.

The general framework for fidelity assessment reflects the accumulation of knowledge over the past several decades and clarifies the dimensions of fidelity and the measures of fidelity. Each dimension and each measurement relates to the essential components of the innovation. The 3X3 matrix provides common concepts and common language to enhance the value of systematic reviews of the literature and inform the development of new fidelity assessments.

Science is based on the accumulation of knowledge and knowledge is based on facts (96, 97). With common concepts, common language, and common measures, the science of innovations, implementation, and scaling can crowdsource the development and systematic accumulation of knowledge. The general framework for fidelity assessment is a step down the science to service path.

Limitations

The general framework includes predictions and testable statements that may give the appearance of certainty where the evidence is not yet available. The general framework describes categories of dimensions and measures of fidelity. The definitions of categories in the framework do not explicate alternative views or tradeoffs among the views. The measurement categories are not explained in terms of validity or reliability. Finally, the dimensions, measures, and concepts in the general framework are oriented to developing and scaling innovations to achieve socially significant benefits for individuals and populations. Concepts oriented to basic research may require different views of fidelity, its measurement, and its meaning.

References

• 1.

  EysenckH. The effects of psychotherapy: an evaluation. J Consult Psychol. (1952) 16:319–24. 10.1037/h0063633

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 2.

  LipseyM. What do we learn from 400 research studies on the effectiveness of treatment with juvenile delinquents. In: McGuireJ, editor. What Works: Reducing Reoffending. Chichester: John Wiley (1995). p. 63–78.

  • Google Scholar
  • View reference in article

• 3.

  GlasgowREKlesgesLMDzewaltowskiDABullSSEstabrooksP. The future of health behavior change research: what is needed to improve translation of research into health promotion practice?Ann Behav Med. (2004) 27:3–12. 10.1207/s15324796abm2701_2

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 4.

  MihalicSFaganAIrwinKBallardDElliottD. Blueprints for Violence Prevention. Washington, DC: U.S. Department of Justice, Office of Justice Programs, Office of Juvenile Justice and Delinquency Prevention (2004).

  • Google Scholar
  • View reference in article

• 5.

  Handbook on Implementation Science, eds. Nilsen P, and Birken SA, Cheltenham, UK: Edward Elgar Publishing (2020).

  • Google Scholar
  • View reference in article

• 6.

  AltmanDGSchulzKFMoherDEggerMDavidoffFElbourneDet al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med. (2001) 134(8):663–94. 10.7326/0003-4819-134-8-200104170-00012

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 7.

  CampbellDTStanleyJC. Experimental and Quasi-experimental designs for Research. Chicago: Rand McNally & Company. (1963).

  • Google Scholar
  • View reference in article

• 8.

  YeatonWHSechrestL. Critical dimensions in the choice and maintenance of successful treatments: strength, integrity, and effectiveness. J Consult Clin Psychol. (1981) 49:156–67. 10.1037/0022-006X.49.2.156

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 9.

  MoncherFJPrinzRJ. Treatment fidelity in outcome studies. Clin Psychol Rev. (1991) 11:247–66. 10.1016/0272-7358(91)90103-2

  • CrossRef
  • Google Scholar
  • View reference in article

• 10.

  GreshamFMGansleKANoellGH. Treatment integrity in applied behavior analysis with children. J Appl Behav Anal. (1993) 26(2):257–63. 10.1901/jaba.1993.26-257

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 11.

  NaleppaMJCagleJG. Treatment fidelity in social work intervention research: a review of published studies. Res Soc Work Pract. (2010) 20(6):674–81. 10.1177/1049731509352088

  • CrossRef
  • Google Scholar
  • View reference in article

• 12.

  DurlakJADuPreEP. Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol. (2008) 41:327–50. 10.1007/s10464-008-9165-0

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 13.

  DaneAVSchneiderBH. Program integrity in primary and early secondary prevention: are implementation effects out of control?Clin Psychol Rev. (1998) 18(1):23–45. 10.1016/S0272-7358(97)00043-3

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 14.

  ToomeyEToDNathanNByrneMLorencattoFMatvienko-SikarKet al. Addressing fidelity within complex health behaviour change interventions: a scoping review of fidelity frameworks and models. Health Psychol Rev. (2025) 20:1–24. 10.1080/17437199.2025.2534001

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 15.

  RookesTASchragAWaltersKArmstrongM. Measures of fidelity of delivery and engagement in self-management interventions: a systematic review of measures. Clinical Trials. (2022) 19(6):665–72. 10.1177/17407745221118555

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 16.

  SchoenwaldSKGarlandAF. A review of treatment adherence measurement methods. Psychol Assess. (2013) 25(1):146–56. 10.1037/a0029715

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 17.

  ScheirerMARezmovicEL. Measuring the degree of program implementation:a methodological review. Eval Rev. (1983) 7(5):599–633. 10.1177/0193841X8300700502

  • CrossRef
  • Google Scholar
  • View reference in article

• 18.

  CalsynRJ. A checklist for critiquing treatment fidelity studies. Ment Health Serv Res. (2000) 2(2):107–13. 10.1023/A:1010109205676

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 19.

  BondLSimmonsESabbathEL. Measurement and assessment of fidelity and competence in nonspecialist-delivered, evidence-based behavioral and mental health interventions: a systematic review. SSM Popul Health. (2022) 19:101249. 10.1016/j.ssmph.2022.101249

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 20.

  FixsenDL. Fidelity, not adaptation, is essential for implementation. Front Health Serv. (2025) 5:1–8. 10.3389/frhs.2025.1575179

  • CrossRef
  • Google Scholar
  • View reference in article

• 21.

  TaylorMJMcNicholasCNicolayCDarziABelDReedJE. Systematic review of the application of the plan–do–study–act method to improve quality in healthcare. BMJ Qual Saf. (2014) 23(4):290–8. 10.1136/bmjqs-2013-001862

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 22.

  McNicholasCLennoxLWoodcockTBellDReedJE. Evolving quality improvement support strategies to improve plan–do–study–act cycle fidelity: a retrospective mixed-methods study. BMJ Qual Saf. (2019) 28(5):356–65. 10.1136/bmjqs-2017-007605

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 23.

  FixsenDLVan DykeMKBlaseKA. Is implementation science a science? Not yet. Front Public Health. (2024) 12:1–13. 10.3389/fpubh.2024.1454268

  • CrossRef
  • Google Scholar
  • View reference in article

• 24.

  FixsenDLVan DykeMKBlaseKA. Establishing usable innovations. Front Health Serv. (2026) 5:1–12. 10.3389/frhs.2025.1745148

  • CrossRef
  • Google Scholar
  • View reference in article

• 25.

  MediavillaRGarcía-VázquezBMcGreevyKRUnderhillJBayónCBravo-OrtizM-Fet al. Beyond effectiveness in eHealth trials: process evaluation of a stepped-care programme to support healthcare workers with psychological distress (RESPOND-HCWs). Digit Health. (2024) 10:20552076241287678. 10.1177/20552076241287678

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 26.

  SanettiLMHCollier-MeekMA. Increasing the rigor of procedural fidelity assessment: an empirical comparison of direct observation and permanent product review methods. J Behav Educ. (2014) 23(1):60–88. 10.1007/s10864-013-9179-z

  • CrossRef
  • Google Scholar
  • View reference in article

• 27.

  KodakTBergmannSWaiteM. Treatment integrity and procedural fidelity. In: LuiselliJK, editor. Applied Behavior Analysis Advanced Guidebook (2nd edn). San Diego: Academic Press (2023). p. 33–62.

  • Google Scholar
  • View reference in article

• 28.

  DobsonLCookT. Avoiding type III error in program evaluation: results from a field experiment. Eval Program Plann. (1980) 3:269–76. 10.1016/0149-7189(80)90042-7

  • CrossRef
  • Google Scholar
  • View reference in article

• 29.

  ForsetlundLTalsethKOBradleyPNordheimLBjorndalA. Many a slip between cup and lip: process evaluation of a program to promote and support evidence-based public health practice. Eval Rev. (2003) 27(2):179–209. 10.1177/0193841X02250528

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 30.

  BondGRDrakeRE. Assessing the fidelity of evidence-based practices: history and current Status of a standardized measurement methodology. Adm Policy Ment Health. (2020) 47(6):874–84. 10.1007/s10488-019-00991-6

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 31.

  BellgAJBorrelliBResnickBHechtJMinicucciDSOryMet al. Enhancing treatment fidelity in health behavior change studies: best practices and recommendations from the NIH behavior change consortium. Health Psychol. (2004) 23(5):443–51. 10.1037/0278-6133.23.5.443

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 32.

  AcarturkCKurtGİlkkurşunZDe GraaffAMBryantRCuijpersPet al. Effectiveness of group problem management plus in distressed Syrian refugees in türkiye: a randomized controlled trial. Epidemiol Psychiatr Sci. (2024) 33:e43. 10.1017/S2045796024000453

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 33.

  ForgatchMS. Implementation as a second stage in prevention research. Prev Treat. (2003) 6(3):Article 24. 10.1037/1522-3736.6.1.624c

  • CrossRef
  • Google Scholar
  • View reference in article

• 34.

  BergmannSNilandHGavidiaVLStrumMDHarmanMJ. Comparing multiple methods to measure procedural fidelity of discrete-trial instruction. Educ Treat Children. (2023) 46:1–20. 10.1007/s43494-023-00094-w

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 35.

  SigmarsdóttirMRodríguezMMDGewirtzARainsLTjadenJForgatchMS. Five-year fidelity assessment of an evidence-based parenting program (GenerationPMTO): inter-rater reliability following international implementation. BMC Health Serv Res. (2023) 23(1):576. 10.1186/s12913-023-09611-4

  • CrossRef
  • Google Scholar
  • View reference in article

• 36.

  McIntoshKMercerSHNeseRNTGhemraouiA. Identifying and predicting distinct patterns of implementation in a school-wide behavior support framework. Prev Sci. (2016) 17(8):992–1001. 10.1007/s11121-016-0700-1

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 37.

  RahmanMAshrafSUnicombLMainuddinAKMParvezSMBegumFet al. WASH Benefits Bangladesh trial: system for monitoring coverage and quality in an efficacy trial. Trials. (2018) 19(1):360. 10.1186/s13063-018-2708-2

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 38.

  Shiroya-WandabwaMKabueMKasungamiDWambuaJOtienoDWakaCet al. Coaching community health volunteers in integrated community case management improves the care of sick children under-5: experience from bondo, Kenya. Int J Integr Care. (2018) 18(4):5. 10.5334/ijic.3971

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 39.

  DohrnJNzamaBMurrmanM. The impact of HIV scale-up on the role of nurses in South Africa: time for a new approach. J Acquir Immune Defic Syndr. (2009) 52(1):S27–S9. 10.1097/QAI.0b013e3181bbc9e4

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 40.

  ICEBeRG Group. Designing theoretically-informed implementation interventions. Implement Sci. (2006) 1(4):1–8. 10.1186/1748-5908-1-4

  • CrossRef
  • Google Scholar
  • View reference in article

• 41.

  CarpianoRMDaleyDM. A guide and glossary on postpositivist theory building for population health. J Epidemiol Community Health. (2006) 60:564–70. 10.1136/jech.2004.031534

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 42.

  HallGLoucksSF. A developmental model for determining whether the treatment is actually implemented. Am Educ Res J. (1977) 14(3):263–76. 10.3102/00028312014003263

  • CrossRef
  • Google Scholar
  • View reference in article

• 43.

  ChinmanMHunterSBEbenerPPaddockSMStillmanLImmPet al. The getting to outcomes demonstration and evaluation: an illustration of the prevention support system. Am J Community Psychol. (2008) 41:206–24. 10.1007/s10464-008-9163-2

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 44.

  GreenhalghTRobertGMacFarlaneFBatePKyriakidouO. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. (2004) 82(4):581–629. 10.1111/j.0887-378X.2004.00325.x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 45.

  HallGHordSM. Implementing Change: Patterns, Principles and Potholes. 4th edBoston: Allyn and Bacon (2011).

  • Google Scholar
  • View reference in article

• 46.

  CenturyJRudnickMFreemanC. A framework for measuring fidelity of implementation: a foundation for shared language and accumulation of knowledge. Am J Eval. (2010) 32(2):199–218. 10.1177/1098214010366173

  • CrossRef
  • Google Scholar
  • View reference in article

• 47.

  WaltzJAddisMEKoernerKJacobsonNS. Testing the integrity of a psychotherapy protocol: assessment of adherence and competence. J Consult Clin Psychol. (1993) 61(4):620–30. 10.1037/0022-006X.61.4.620

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 48.

  PerepletchikovaFTreatTAKazdinAE. Treatment integrity in psychotherapy research: analysis of the studies and examination of the associated factors. J Consult Clin Psychol. (2007) 75(6):829–41. 10.1037/0022-006X.75.6.829

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 49.

  MowbrayCTHolterMCTeagueGBBybeeD. Fidelity criteria: development, measurement, and validation. Am J Eval. (2003) 24(3):315–40. 10.1177/109821400302400303

  • CrossRef
  • Google Scholar
  • View reference in article

• 50.

  CarrollCPattersonMWoodSBoothARickJBalainS. A conceptual framework for implementation fidelity. Implement Sci. (2007) 2(1):40. 10.1186/1748-5908-2-40

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 51.

  StrainPS. Personal thoughts on early childhood special education research: an historical perspective, threats to relevance, and call to action. J Early Interv. (2018) 40(2):107–16. 10.1177/1053815117750411

  • CrossRef
  • Google Scholar
  • View reference in article

• 52.

  BrunkMChapmanJESchoenwaldSK. Defining and evaluating fidelity at the program level: a preliminary investigation. Zeitschrift für Psychologie. (2014) 222:22–9. 10.1027/2151-2604/a000162

  • CrossRef
  • Google Scholar
  • View reference in article

• 53.

  AnMDusingSCHarbourneRTSheridanSM. What really works in intervention? Using fidelity measures to support optimal outcomes. Phys Ther. (2020) 100(5):757–65. 10.1093/ptj/pzaa006

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 54.

  PhillipsEL. Achievement place: token reinforcement procedures in a home-style rehabilitation setting for “pre-delinquent” boys. J Appl Behav Anal. (1968) 1:213–23. 10.1901/jaba.1968.1-213

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 55.

  WolfMMKiriginKAFixsenDLBlaseKABraukmannCJ. The teaching-family model: a case study in data-based program development and refinement (and dragon wrestling). J Organ Behav Manage. (1995) 15:11–68. 10.1300/J075v15n01_04

  • CrossRef
  • Google Scholar

• 56.

  BraukmannCJFixsenDLKiriginKAPhillipsEAPhillipsELWolfMM. Achievement place: the training and certification of teaching-parents. In: WoodWS, editor. Issues in Evaluating Behavior Modification. Champaign, IL: Research Press (1975). p. 131–52.

  • Google Scholar

• 57.

  FixsenDLBlaseKA. The teaching-family model: the first 50 years. Perspect Behav Sci. (2018) 42(2):189–211. 10.1007/s40614-018-0168-3

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 58.

  FixsenDLBaronRLDalyDLTylerPM. Sustaining fidelity for 50 years: boys town and the teaching-family model. Resid Treat Child Youth. (2025) 42(4):518–41. 10.1080/0886571X.2025.2543391

  • CrossRef
  • Google Scholar

• 59.

  SteinLITestMA. Alternatives to Mental Hospital Treatment. New York: Plenum Press (1978).

  • Google Scholar

• 60.

  SteinLITestMAMarxAJ. Alternative to the hospital: a controlled study. Am J Psychiatry. (1975) 132(5):517–22. 10.1176/ajp.132.5.517

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 61.

  TeagueGBBondGRDrakeRE. Program fidelity in assertive community treatment: development and use of a measure. Am J Orthopsychiatry. (1998) 68(2):216–32. 10.1037/h0080331

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 62.

  HarrisonJTaylorH. Ten years of implementation: assertive community treatment and integrated dual disorder treatment in a statewide system. Arch Psychol. (2019) 3(2):1–15. 10.31296/aop.v3i2.92

  • CrossRef
  • Google Scholar

• 63.

  HenggelerSWMeltonGBSmithLASchoenwaldSKHanleyJ. Family preservation using multisystemic therapy: long-term follow-up to a clinical trial with serious juvenile offenders. J Child Fam Stud. (1993) 2:283–93. 10.1007/BF01321226

  • CrossRef
  • Google Scholar

• 64.

  HenggelerSWMeltonGBSmithLA. Family preservation using multisystemic therapy: an effective alternative to incarcerating serious juvenile offenders. J Consult Clin Psychol. (1992) 60:953–61. 10.1037/0022-006X.60.6.953

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 65.

  HenggelerSWMeltonGBBrondinoMJSchererDGHanleyJH. Multisystemic therapy with violent and chronic juvenile offenders and their families: the role of treatment fidelity in successful dissemination. J Consult Clin Psychol. (1997) 65(5):821–33. 10.1037/0022-006X.65.5.821

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 66.

  SchoenwaldSKHenggelerSWBrondinoMJRowlandMD. Multisystemic therapy: monitoring treatment fidelity. Fam Process. (2000) 39(1):83–103. 10.1111/j.1545-5300.2000.39109.x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 67.

  HornerRHSugaiGHornerHF. A school-wide approach to student discipline. School Administrator. (2000) 57(2):20–4.

  • Google Scholar

• 68.

  HornerRHToddAWLewis-PalmerTIrvinLKSugaiGBolandJB. The school-wide evaluation tool (SET): a research instrument for assessing school-wide positive behavior support. J Posit Behav Interv. (2004) 6(1):3–12. 10.1177/10983007040060010201

  • CrossRef
  • Google Scholar

• 69.

  ForgatchMSDeGarmoDSBeldavsZG. An efficacious theory-based intervention for stepfamilies. Behav Ther. (2005) 36(4):357–65. 10.1016/S0005-7894(05)80117-0

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 70.

  ForgatchMSPattersonGRDeGarmoDS. Evaluating fidelity: predictive validity for a measure of competent adherence to the Oregon model of parent management training. Behav Ther. (2005) 36(1):3–13. 10.1016/S0005-7894(05)80049-8

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 71.

  WotringJHodgesKXueYForgatchM. Critical ingredients for improving mental health services: use of outcome data, stakeholder involvement, and evidence-based practices. Behav Ther. (2005) 28(7):154–8.

  • Google Scholar

• 72.

  HoltropKPiehlerTFMillerDYoungDTsengC-FGrayLJ. The effectiveness of GenerationPMTO during sustained implementation in the public mental health system: a single-arm open trial evaluation. Behav Ther. (2024) 55(2):248–62. 10.1016/j.beth.2023.06.004

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 73.

  ForgatchMSDeGarmoDS. Sustaining fidelity following the nationwide PMTO™ implementation in Norway. Prev Sci. (2011) 12(3):235–46. 10.1007/s11121-011-0225-6

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 74.

  OgdenTBjørnebekkGKjøbliJPatrasJChristiansenTTaraldsenKet al. Measurement of implementation components ten years after a nationwide introduction of empirically supported programs – a pilot study. Implement Sci. (2012) 7:49. 10.1186/1748-5908-7-49

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 75.

  TommeraasTOgdenT. Is there a scale-up penalty? Testing behavioral change in the scaling up of parent management training in Norway. Adm Policy Ment Health. (2016) 44:203–16. 10.1007/s10488-015-0712-3

  • CrossRef
  • Google Scholar

• 76.

  BashaSAJCaiQRodriguezMMDGewirtzAHSigmarsdóttirMDeGarmoDSet al. Stability amidst change in the measurement of implementation fidelity over time. Prev Sci. (2026) 26:1251–62. 10.1007/s11121-025-01864-1

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 77.

  FixsenDLBlaseKAVan DykeMK. Implementation Practice and Science. 1st edChapel Hill, NC: Active Implementation Research Network, Inc. (2019). p. 378.

  • Google Scholar

• 78.

  SnyderKVarshneyNRoseberryNBradleySMillionC. Pyramid model coaching characteristics and TPOT/TPITOS observation scores: an exploratory analysis. J Early Child Teach Educ. (2025):1–14. 10.1080/10901027.2025.2603594

  • CrossRef
  • Google Scholar

• 79.

  SawyerMR. Using behavioral skills training and coaching with preservice teachers: an evaluation of impact on implementation fidelity. Educ Sci. (2025) 15(2):228. 10.3390/educsci15020228

  • CrossRef
  • Google Scholar

• 80.

  JensenPSWeersingRHoagwoodKEGoldmanE. What is the evidence for evidence-based treatments? A hard look at our soft underbelly. Ment Health Serv Res. (2005) 7(1):53–74. 10.1007/s11020-005-1965-3

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 81.

  SchoenwaldSKGarlandAFChapmanJEFrazierSLSheidowAJSoutham-GerowMA. Toward the effective and efficient measurement of implementation fidelity. Adm Policy Ment Health. (2011) 38(1):32–43. 10.1007/s10488-010-0321-0

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 82.

  MillerJMMillerHV. Rethinking program fidelity for criminal justice. Criminol Public Policy. (2015) 14(2):339–49. 10.1111/1745-9133.12138

  • CrossRef
  • Google Scholar

• 83.

  SanettiLMHCookBGCookL. Treatment fidelity: what it is and why it matters. Learn Disabil Res Pract. (2021) 36(1):5–11. 10.1111/ldrp.12238

  • CrossRef
  • Google Scholar

• 84.

  WilderDAAtwellJWineB. The effects of varying levels of treatment integrity on child compliance during treatment with a three-step prompting procedure. J Appl Behav Anal. (2006) 39(3):369–73. 10.1901/jaba.2006.144-05

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 85.

  U.S. Government Accounting Office. Program evaluation: A variety of rigorous methods can help identify effective interventions. (2009). Available online at:http://gao.gov/products/GAO-10-30 (Accessed November 24, 2010).

  • Google Scholar

• 86.

  SadrizadehSAganovicABogdanAWangCAfshariAHartmannAet al. A systematic review of operating room ventilation. J Build Eng. (2021) 40:102693. 10.1016/j.jobe.2021.102693

  • CrossRef
  • Google Scholar

• 87.

  GibbonsCJCarrollKMBallSANichCFrankforterTLMartinoS. Community program therapist adherence and competence in a motivational interviewing assessment intake session. Am J Drug Alcohol Abuse. (2010) 36(6):342–9. 10.3109/00952990.2010.500437

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 88.

  HartmannDP. Considerations in the choice of interobserver reliability estimates. J Appl Behav Anal. (1977) 10(1):103–16. 10.1901/jaba.1977.10-103

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 89.

  CooperJHeronTHewardW. Applied Behavior Analysis. 3rd edHoboken, NJ: Pearson Education (2019). p. 912.

  • Google Scholar

• 90.

  Copur-GencturkYThackerI. A comparison of perceived and observed learning from professional development: relationships among self-reports, direct assessments, and teacher characteristics. J Teach Educ. (2021) 72(2):138–51. 10.1177/0022487119899101

  • CrossRef
  • Google Scholar

• 91.

  PhillipsELPhillipsEAFixsenDLWolfMM. The Teaching-Family handbook. 2nd edLawrence, KS: University Press of Kansas (1974).

  • Google Scholar

• 92.

  WeinrottMRJonesRRHowardJR. Cost-effectiveness of teaching family programs for delinquents: results of a national evaluation. Eval Rev. (1982) 6(2):173–201. 10.1177/0193841X8200600202

  • CrossRef
  • Google Scholar

• 93.

  Washington State Institute for Public Policy. Teaching-Family Model Group Homes (vs. Other Group Homes) for Court-involved youth. Olympia, WA: Washington State Institute for Public Policy (2024).

  • Google Scholar

• 94.

  FixsenDLBlaseKATimbersGDWolfMM. In search of program implementation: 792 replications of the teaching-family model. Behav Anal Today. (2001/2007) 8(1):96–110. 10.1037/h0100104

  • CrossRef
  • Google Scholar

• 95.

  TylerPMDalyDLBaronRLFixsenDL. FiftyYears of the teaching-family model at boys town: keys to sustainability. Resid Treat Child Youth. (2025) 42(4):1–28. 10.1080/0886571X.2025.2543400

  • CrossRef
  • Google Scholar

• 96.

  WoottonD. The Invention of Science: A new History of the Scientific Revolution. New York: Harper Collins (2015).

  • Google Scholar

• 97.

  KuhnTS. The Structure of Scientific Revolutions. Chicago: University of Chicago Press (1970).

  • Google Scholar