Parents influence children's affective and cognitive development, with lifelong impacts (1, 2). Nurses play an important role in intervening early to support parenting and promote healthy child development (3, 4). Despite recognizing the need to deliver vital parenting programs, monitoring fidelity has been historically ignored (5, 6). Fidelity refers to the extent to which a healthcare program follows an explicit set of criteria specifically designed for its particular program model (7–9). However, defining and operationalizing program fidelity for parenting programs is difficult due to their interactive and dynamic nature (10, 11). Attempts by program developers to assure adherence to their programs include the creation of training and protocol manuals, but these alone may not be sufficient to ensure fidelity of implementation (8, 12). With increased demands for early intervention to be widely delivered by non-specialists, rigorous yet pragmatic strategies for maintaining program fidelity are needed.

In the United States and Canada, the development and implementation of parenting programs to bolster healthy development of children has increased during the last 20 years (13). There is a growing awareness of the significance of developing and applying fidelity measures to evaluate the implementation of such programs (12, 14, 15). However, practical guidelines and exemplars are lacking. This paper describes how to develop and conduct an evaluation of a program fidelity checklist, including the step-by-step developmental process our research team used for the Attachment and Child Health (ATTACH™) parenting program. ATTACH™ is a psycho-educational parenting program that fosters parental reflective function (RF), the ability of parents to envision mental states in themselves and their children to promote healthy child development (16, 17). Compared to routine care, ATTACH™ has been shown to be effective in randomized controlled trials and quasi-experimental studies (16, 17); however, program fidelity assessment was needed to contextualize results and assure ongoing internal validity in support of wider implementation. Thus, the purpose of our paper is to demonstrate the step-by-step process for development of a fidelity checklist, specifically for ATTACH™, and to provide preliminary data on its utility.

To develop and test the fidelity checklist for ATTACH™, we evaluated existing measures for monitoring fidelity of parenting programs. We determined their applicability to the ATTACH™ program and identified challenges that needed to be resolved for our fidelity checklist. Then we created the ATTACH™ fidelity checklist by tailoring standard recommendations (18, 24, 30, 32, 34, 44–47) to ATTACH™'s guiding theory and program structure.

Table 5 shows an example of three ATTACH™ sessions assessed using the ATTACH™ fidelity assessment checklist. We selected the six sessions from three participants (both dyadic and triadic) to illustrate the full scope of assigned ratings and how the checklist may be used for monitoring fidelity and training. For example, for the first participant (No. 4), the overall quality index rating was 4.5 for both dyadic and triadic sessions. This rating was based on the 96–100% adherence on content and process fidelity, adequate pacing of program delivery, and high rating on participant's responsiveness for dyadic session. For the second participant (No. 11), an overall rating of 5 for dyadic session reflected completion of all elements of content fidelity (100%), process elements (100%) with no missed opportunities, completion of the session within time (adequate pacing), and high rating on participant's responsiveness for triadic session. For the third participant (No. 15), the overall rating was 4 for dyadic session, and adherence to content fidelity was 93% meaning that the facilitator skipped two items and took longer than the protocol required to complete the dyadic session.

Table 6 indicates the adherence and competence of the ATTACH™ facilitators. Under adherence to program content elements, the percentage for implemented as intended was 96% for both dyadic and triadic sessions. The percentages for elements implemented as not intended was low, as expected, between 2 and 3%. Under adherence to program process elements, the fidelity for repeat/paraphrase/probing was 99%, while skipped opportunities was 0 as expected. For pacing, the mean was 2.16 for dyadic and 2.3 for triadic sessions out of possible 3. For participant responsiveness the mean was 3 for both dyadic and triadic sessions out of possible 3. The overall quality index for program delivery varied between “very good” to “excellent,” with a mean of 4.3/5. The coders who coded the videos provided positive feedback on the checklist describing that they found user friendly. There were minor changes to address clarifications requested by the coders. For example, our preliminary checklist only included what the parent like/disliked about the play session. However, coders noticed that in later sessions the facilitator changed from asking what the parent like/disliked about the play session to moments the parent felt they were connected/disconnected with their child. We revised our checklist accordingly. Evaluation of the sessions from the ATTACH™ from earlier to later participants showed that the facilitators improved over time, as demonstrated by the improved fidelity ratings. The ATTACH™ fidelity assessment checklist could be used for training and assessing fidelity. The ICC for agreement between the two coders for the three elements of the fidelity checklist (adherence to the ATTACH™ content elements, process elements, and overall quality index) was good to excellent (ICC = 0.85–1.00), suggesting strong inter-rater agreement.

The ATTACH™ Fidelity Assessment checklist was developed to evaluate facilitators' adherence and competence in implementing an evidence-based, psycho-educational parenting program to high-risk families. We found that facilitators adhered to the program content and process fidelity close to 100%. They adequately paced or completed the sessions within 30–45 min without dragging or rushing and maintained excellent participant responsiveness. The overall quality index of program delivery ranged from 4 to 5 on a 5-point Likert scale (1 = poor, 5 = excellent). Coders for the ATTACH™ sessions helped to further refine the checklist by requesting minor changes.

Supported by previous research, we developed the ATTACH™ fidelity checklist to measure the elements of adherence to content and process, as well as participant responsiveness (44, 47, 55, 60, 62–64). Overall pacing of program delivery, used in our checklist, has never been included in previous parenting program fidelity checklists, to our knowledge. This item was adapted from fidelity measures of psycho-educational interventions (66). While overall fidelity of program delivery was only rated in a few parenting programs and mostly based on content validity (14, 55, 57), we created an overall quality index rating from psycho-educational intervention fidelity assessment based on both content and process elements (66, 70, 76–78). As ATTACH™ is a psycho-educational parenting program, we deemed it important to include both pacing and overall assessment items to ensure we capture the essence of program delivery.

As we reviewed in Table 1, most of the extant fidelity checklists of the parenting programs have limitations as they are developed to minimize or eliminate the possibility that a practitioner will miss or fail to perform one or more steps or actions. However, assessing adherence to process elements or competence by simple using “Yes” and “No” categories may not be the best answer to meet all needs, specifically in case of psycho-educational interventions such as ATTACH™ (44, 57, 66). We therefore also applied insights from other, non-parenting intervention fidelity measures (such as Song et al. and Miller et al.) to fully comprehend and assess process elements or competence. We also provided an overall quality index of the intervention delivery, as supported by other fidelity literature of parenting interventions and non-parenting interventions (14, 57, 66, 68).

The ATTACH™ fidelity checklist will be specifically important in evaluating the efficacy of the program on outcomes, exploring ways to improve the program, and learning how to overcome challenges and barriers to program implementation (8, 55). The checklist may also improve the efficiency of training facilitators and systematic evaluation of their adherence and competence (79). Assessing competence of facilitators in program delivery is crucial because a facilitator may implement all the content of the program, but in a non-prescribed manner that can result in low efficacy of the program (14). Any deviated delivery of a program element and not using positive communication behaviors can be discouraged and corrected by assessing the program sessions by using our fidelity checklist.

Although only two facilitators delivered the ATTACH™ sessions evaluated here, the approach of monitoring and assessment is applicable to multiple facilitators. While we developed the fidelity checklist for training and monitoring fidelity in research contexts, it has not been evaluated in a clinical setting. Although inter-rater reliability was high (80), only two highly trained coders were assessed. Inter-rater reliability with clinical raters and a larger sample of sessions is needed to confirm these results. A second limitation is the lack of variability in some of the scores (e.g., Participant Responsiveness) of fidelity. Although this indicates overall good adherence and competence in delivering the ATTACH™ intervention, which is a positive finding, lack of variability in fidelity items limit the ability to understand the distinct items that may influence outcomes (12). This should be discussed as a limitation and potential future direction for research. Further work may focus on greater variability in a sample of community-based facilitators, suggesting that it might be valuable to retain these measures until the measure can be validated in a community implementation setting.

Questions remain concerning whether variations in program implementation should be evaluated as part of program fidelity by taking participants' responsiveness into account as moderators of the program effects on outcomes, or both. In future research, weights for different elements may be taken into consideration when critical and core elements have been identified; however, this could be difficult to implement in clinical settings. There are important cost considerations in using fidelity measures (81), as we found that evaluation of an hour-long single session consumed up to 2.5 h for coding and 30 min for scoring by an experienced coder/rater. Other fidelity measures (e.g., 58, 60) did not involve rating a full intervention session, instead they rated a session segment that took as little as 5 min, as a solution to potential time and cost issues in the discussion. Given the creation and testing phase of ATTACH™, we did not know whether significant differences exist between the major program elements (video feedback and RF exercises). Thus, we treated all steps equally in importance and considered them to be delivered within a specific range of time set for the intervention within the protocol.

In future, an extension of this research would examine links between program fidelity and targeted outcomes (e.g., increased fidelity scores may yield more favorable effects or outcomes) to help ascertain if the benefits outweigh the costs (62, 82, 83). This further step would provide evidence for validity of the ATTACH™ fidelity checklist. Nonetheless, measuring fidelity is arguably even more important as interventions are implemented in community settings outside of research studies, with less control and oversight over fidelity, and decreases in fidelity are linked to reduced intervention outcomes [e.g., (84)].

The program content elements are unique to the ATTACH™ program; therefore, the process of developing and evaluating a fidelity checklist is specifically described for the ATTACH™ parenting program. However, the ATTACH™ fidelity checklist described in this paper, may inform and be adapted to fidelity evaluation tools for other parenting program. High fidelity scores in ATTACH™ sessions provided an evidence of validity for the measure. Establishing treatment differentiation between the ATTACH™ and other manualised interventions would provide another future direction or next step in validation of this measure (33). We have designed a reliable measure and shown how nurses may operationalize program adherence and competence to evaluate facilitators' adherence and competence so that elements of the implementation of parenting programs are more evident. Our evaluation showed the ATTACH™ intervention fidelity was high. Development of checklists like the ATTACH™ fidelity assessment checklist enables the systematic evaluation of program delivery and identification of therapeutic components that enable targeted efforts at improvement.

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author/s.

The studies involving human participants were reviewed and approved by Conjoint Health Research Ethics Board, University of Calgary, Calgary, Canada. The patients/participants provided their written informed consent to participate in this study.

LA: formulated the research question and oversaw all aspects of manuscript preparation from literature review, found all of the relevant articles for fidelity assessment checklist development, developed and refined the checklist with the help of KB and NL, conducted the data analysis, interpreted the results, and undertook writing and submission. KB: helped oversee the research question, literature review, development of checklist, interpretation of the results, and writing. CE: helped oversee the literature review and reviewed the relevant articles, interpreted the results, and writing. MH: helped review the relevant articles, development and refinement of checklist, interpreted the results, and writing. NL: helped formulate the research question, oversaw the literature review and reviewed the relevant articles, helped in the development and refinement of fidelity checklist, interpretation of the results, and writing. All authors have read and approved the final manuscript.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.