Use of the reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) framework to guide iterative adaptations: Applications, lessons learned, and future directions

Introduction

There is emerging consensus among implementation scientists that adaptations to interventions and implementation strategies are inevitable, can be beneficial or detrimental, and need to be carefully documented and better understood (1–7). Implementation science theories and frameworks have been widely used to plan and tailor interventions and implementation strategies (8–10), and some models such as the Implementation Outcomes Framework (IOF) (11) and the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) (12) framework to guide outcomes evaluation. However, there has been little iterative use of these frameworks during the implementation phase or to guide adaptations (13). While Kirk et al. have developed the Model for Adaptation Design and Impact (MADI) (14) to comprehensively characterize adaptations and how they impact outcomes and general guidance has been provided in ADAPT-ITT (15), very few empirical studies have evaluated the actual impact of adaptations qualitatively or quantitatively (16–18). The ADAPT (7) guidance explicitly calls for continuous iterative review of emerging adaptations but did not identify tools or resources for doing this. Below we describe Iterative RE-AIM that shares many similarities with these three approaches but is distinct because of its specific and comprehensive use of an implementation science outcomes framework (RE-AIM); focus on rapid repeated adaptations; and provision of specific directions, tools, survey items and feedback displays.

Real world application of complex health interventions (19, 20) in complex, dynamic systems guarantees that a program will seldom be implemented in diverse non-research settings exactly as planned, no matter how comprehensive and well done the planning. This is especially true of interventions that prescribe specific actions as in a detailed protocol, although likely less so for interventions providing more general guidelines (21) or key “functions” for the intervention (22). Thus, adaptations during implementation are ubiquitous and it would be advantageous if the adaptations could be guided by a conceptual framework and based on data rather than guess work. Our group has published initial work using the RE-AIM framework to guide adaptations (13).

For background, RE-AIM, and its more recent expansion into the Practical Robust Implementation and Sustainability Framework (PRISM), are implementation science frameworks that focus on key outcomes necessary to produce population impact and contextual factors that influence these outcomes. RE-AIM (13, 23) is one of the most widely used implementation science frameworks to assess implementation outcomes and PRISM, which includes RE-AIM dimensions, adds a focus on key contextual factors related to these outcomes. The PRISM domains include individual and organization/setting characteristics; individual and organization/setting perceptions of the intervention; external environmental factors (e.g., relevant policies, reimbursement issues, community influences); and implementation and sustainability infrastructure (e.g., resources, job responsibilities, and processes to support implementation).

In this paper, we present Iterative RE-AIM as a method for assessing progress toward goals set for RE-AIM outcomes, prioritizing areas needing improvement, and identifying adaptation strategies for these areas. Iterative RE-AIM then monitors improvements following these changes. The process is undertaken on multiple occasions (i.e., iteratively) to continue refining intervention delivery. The methods and key functions of Iterative RE-AIM are described in the Methods and results section and Table 1 but in brief, it provides a concrete, structured way to engage implementation team members and to foster discussions of both progress to date and current priorities using RE-AIM as a framework. Team members individually complete a brief survey about their perceptions of both progress and their priorities across the five RE-AIM dimensions of reach, effectiveness, adoption, implementation, and maintenance at that point in time. Results are then integrated and summarized for discussion at an upcoming meeting. These discussions are based on a “gap analysis” of the dimensions on which there is the greatest difference between priority and progress to develop consensus strategies for adaptations to enhance progress on one or two key RE-AIM areas. This process is then repeated at a frequency tailored to the given project.

TABLE 1

Table 1. Key functions of iterative RE-AIM.

Based on encouraging results from initial application of Iterative RE-AIM across multiple projects in a Veteran's Administration Quality Enhancement Research Implementation Initiative (13), we have recently used or are using Iterative RE-AIM in several additional projects. The purposes of this paper are to: summarize four recent and ongoing applications of Iterative RE-AIM to guide adaptations; describe key findings and lessons learned in these applications; and identify directions for future research and practice using Iterative RE-AIM to assess and guide adaptations.

Methods and results

Iterative RE-AIM functions, process and resources

Iterative RE-AIM key functions

There was some variation in the specific activities and processes used across the case studies in this paper. However, as shown in Table 1, across projects there are several key functions of Iterative RE-AIM that were accomplished in different forms in different projects.

Iterative RE-AIM process

To accomplish the key functions described in Table 1, Iterative RE-AIM is conducted using the steps and processes described below. Table 2 summarizes the key steps, implementation strategies, and activities involved in Iterative RE-AIM. The key steps are: (1) At an initial meeting with the identified implementation team members there is a general overview of the Iterative RE-AIM process, review of pragmatic RE-AIM definitions and how they are operationalized, and general discussion of the program or evidence-base practice (EBP) involved; (2) Team members each provide a confidential rating on the importance (group member perceptions) of and progress on all RE-AIM dimensions (using actual data when possible) and results are analyzed and summarized in a way that protects the identify of individuals; (3) The team reviews, reflects on and discusses the ratings using visual displays that summarize ratings; (4) The team identifies one or two RE-AIM dimensions on which to focus, and identifies adaptations (implementation strategies) to address these areas; (5) The agreed upon adaptations and implementation strategies are implemented and short term impact is evaluated; and (6) Future meetings are held approximately every 1–2 months, repeating steps 1–5, which allows for changes in goals and implementation strategy adaptations based on progress.

TABLE 2

Table 2. Steps and component activities in applying Iterative RE-AIM.

Iterative RE-AIM resources

To facilitate conduct of Iterative RE-AIM, several key resources are available. These materials are publicly available at https://re-aim.org/applying-the-re-aim-framework/re-aim-guidance/use-during-implementation/ and include:

• Introductory and educational Iterative RE-AIM materials to orient team members to the concepts in RE-AIM and outline the process used;

• Iterative RE-AIM worksheets containing brief survey questions for team members to record their scores on progress and importance;

• Iterative RE-AIM gap analysis tool (in Excel) to calculate group scores and develop summary reports;

• Sample visual displays of results of the above ratings;

• Action planning forms to provide a written record of the adaptations and implementation strategies planned;

• An Iterative RE-AIM evaluation form to assess the usefulness and impact of the Iterative RE-AIM process.

Case studies

This article is not a standard quantitative report of a trial nor an in-depth qualitative study. Rather it is a compilation of results, experiences and lessons learned across a variety of different applications of Iterative RE-AIM. Thus, instead of following a traditional reporting system or results section, we have organized each case report using the headings of: Description of program and use of Iterative RE-AIM; and Findings. This is summarized in Table 3. This is followed by a section on Crosscutting Lessons Learned.

TABLE 3

Table 3. Summary of current and ongoing Iterative RE-AIM projects.

Hypertension control in Guatemala

Description of the program and use of Iterative RE-AIM

During a hypertension control study in 5 departments (provinces) and 36 districts of Guatemala, we used PRISM/RE-AIM for planning and evaluation and assessed dimensions and aspects of context at multiple time points (24). As described below PRISM, the Practical Robust Implementation and Sustainability Model (23) focuses on key contextual factors related to RE-AIM outcomes. We recognized from the outset that it would be important to prioritize the assessment of delivery of five implementation strategies and aspects of context regularly during the 18-month study. Prior to implementation we conducted a needs assessment (25) in which we identified routine assessment of availability of medications in health posts/centers as a top priority to review at monthly meetings throughout the study. The implementation phase of the study began in 3 provinces in the Eastern part of the country and subsequently in 2 provinces in the Western Highlands. We developed implementation tracking forms (for the RE-AIM Implementation domain) that were filled out by implementers (Ministry of Health staff, primarily auxiliary nurses within intervention districts). Local-level project evaluators, assigned to cover two districts each, captured data using forms to assess key aspects of context within health posts and centers (availability of medications, blood pressure monitors, and staff turnover). The team met regularly with the Ministry of Health at the central level to be aware of broader contextual changes (e.g., service priorities or trainings that would influence providers' time and availability). At monthly research meetings, we reviewed and reflected on changes in Implementation and medication availability and discussed staff turnover and implications for the PRISM contextual factor of Implementation and Sustainability Infrastructure. We reviewed Reach during initial meetings but decided that it would be difficult to influence that dimension in the short-term even though men were participating at a much lower rate than women. The COVID-19 pandemic began during the rollout of the trial. This resulted in a dramatic change to the context, and the study team and Ministry of Health staff identified the need for major adaptations. Some of the key adaptations that we made were: a change in how training was conducted (from in-person to hybrid) and increased flexibility in providing medications to patients (more than 1-month supply, allowing family members to pick up medications, shifting medications from health centers to posts to cut down on distance that patients needed to travel).

Findings

Due to the large number of sites and long distances between them (anticipated) and disruption during the COVID-19 pandemic (unanticipated), we recognized how important it was to have a system in place to track Implementation and contextual issues. The study team took time to review and reflect on data during monthly meetings, using PRISM/RE-AIM, and we discussed key areas on which to follow up—this usually led to reaching out to different actors in the Ministry of Health at the central, provincial, or local levels. Medication availability and staff turnover were recognized as key. Early during the pandemic, when it was not possible to travel, the team felt disconnected from what was happening in the rural communities many hours from the capital city; the project staff decided to make phone calls to implementers and patients to gain insight into their experiences (26) and to inform adaptations. The qualitative and quantitative data obtained during those phone calls helped the team define next steps at a critical moment of uncertainty.

Hospital based point of care lung ultrasound

Description of the program and use of Iterative RE-AIM

Our 12-month long lung ultrasound implementation pilot was conducted at an academic quaternary care medical center in Aurora, Colorado in response to the COVID pandemic (27). The goal of the pilot was to quickly implement the use of lung ultrasound among hospitalist clinicians caring for adults hospitalized with COVID to conserve personal protective equipment and reduce COVID exposure within the hospital environment caused by use of chest imaging modalities performed by radiology. Iterative RE-AIM was the overarching implementation strategy used in this pilot study. Given the low baseline rate of lung ultrasound use among hospitalists at the beginning of the pilot, the implementation team chose to prioritize the RE-AIM outcome measures of Reach and Adoption. To iteratively measure Reach and Adoption, a novel RE-AIM dashboard was created to display these quantitative measures using data extracted from the EHR and automatically updated every 48 h. While the dashboard took some resources and expertise to build, it required minimal time to use and maintain over the course of the study, providing nearly real-time access to these prioritized implementation outcomes. At twice monthly meetings, updated RE-AIM dashboard data were evaluated and discussed by the implementation team which consisted of 4 hospitalists, to screen for barriers to Adoption and Reach. Qualitative data were collected concurrently through interviews with hospitalist faculty to understand contextual factors and determinants of adoption. Through discussion of this qualitative and quantitative data, the implementation team would come to consensus regarding interval adaptations to on-going implementation strategies.

Findings

Through this project we learned that operational dashboards make iterative assessment of RE-AIM outcomes drawn from EHR data highly feasible, allowing for easy monitoring of both the progress and representativeness of some RE-AIM dimensions and data-driven interim adjustments in implementation strategies. Future work will focus on more formally and systematically incorporating rapid qualitative methods (28) guided by the contextual domains of PRISM into our Iterative RE-AIM process to better understand current barriers to implementation detected by iterative evaluation of data displayed via a RE-AIM dashboard.

Colorectal cancer screening project integrating assessment of PRISM contextual factors

Description of the program and use of Iterative RE-AIM

In this NCI-funded research project focusing on increasing colorectal cancer screening in underserved communities in San Diego County, our team works with a bridge organization and two federally qualified healthcare systems (FQHCs) using a hub (i.e., bridge organization) and spoke (i.e., FQHCs) model as an implementation strategy to increase the completion of colorectal cancer screening and follow up of abnormal screening results. We are using PRISM and a PRISM Fit Assessment at two time points, pre- and mid- implementation with each FQHC separately. At both stages, a survey instrument was administered using REDCap to capture perceptions of representatives of the FQHCs, the bridge organization, and the research team on how likely it is that reach, adoption, implementation, effectiveness, and maintenance of the research program will be optimal. The survey also asked whether the program aligned well with PRISM contextual domains of: perceptions of the diverse partners and patients of the program, characteristics of these diverse partners, the implementation and sustainability infrastructure at their health care center, and the external environment. Data from this survey from multiple participants were summarized in visual displays and summary points and shared during a follow-up meeting including all who completed the survey. The meeting allowed for discussion of areas that scored low consistently, reasons for low scores, and possible adaptations to improve these areas. Key discussion points and related action steps were summarized and shared with each FQHC along with a cross-FQHC summary.

Findings

Data from pre- and mid- implementation assessments indicated that most RE-AIM dimensions and PRISM domains were rated relatively highly on all dimensions. Lower ratings were noted for abnormal FIT follow-up compared to the mailed FIT intervention and for the mid-implementation assessment compared to the pre-implementation assessment. Variation was noted across the health care centers in areas of lower ratings. Some areas like adoption—at the implementer level—were not deemed meaningful because staff at the participating FQHCs did not have a choice of opting out of the program. To address this concern, a “not applicable” answer option was added to the survey questions. It was also indicated that the distinction between general patient population and underserved populations in terms of reach, effectiveness, and maintenance was not as relevant as the FQHCs exclusively serve underserved communities. However, it was noted that the program reach was somewhat limited by only including individuals with health insurance. Key RE-AIM dimensions with lower scores were Reach, Implementation (especially as it relates to cost and resources), and Maintenance. Some PRISM context domains with lower alignment scores included Implementation and Sustainability Infrastructure, Recipient Characteristics—organizational, and External Environment. The follow-up sessions when results were discussed allowed for rich discussions between the research team and implementation partners. Key themes identified included the need to consider sustainment, costs and resources needed to deliver the intervention after the study is completed, strategies to reach patients with no insurance, and the external environment including possible policy impact.

Quadruple aim quality enhancement research initiative (QUERI)

Description of the program and use of Iterative RE-AIM

The goal of the Department of Veterans Affairs (VA) Quadruple Aim QUERI is to enhance Veteran outcomes and experiences, clinician engagement, and reduce the cost of care by providing value-based care coordination between VA and community settings for Veterans and implementation partners using sustainable practices. Veterans who receive care in both VA and community settings (dual-use) are at risk for fragmented, poorly coordinated care across care settings, which may contribute to adverse outcomes, poor experiences, and increased costs of care (29, 30).

We are implementing three evidence-based practices (EBPs), one in each of three different health services projects, all rooted in care coordination models to achieve consistently safe, efficient and effective care for Veterans. These three EBPs offer care coordination programs throughout the continuum of care to facilitate the integration of, and navigation through, healthcare services within and across care settings, to help patients receive the care they need and want without unnecessary duplication of services or unwarranted delay (31).

We are using two evidence-based implementation strategy bundles to guide EBP implementation: Iterative RE-AIM and Relational Facilitation (32), which were developed and tested in our previous work of the Triple Aim QUERI and the Office of Rural Health Rural Transition Nurse Program (33). As with these previous programs (6), we are assessing and guiding implementation and adaptations based on emerging data and changing context through the lens of PRISM (34) while addressing factors impacting the RE-AIM outcomes (13). Both Iterative RE-AIM and Relational Facilitation implementation strategy bundles include a set of transactional (i.e., training, audit and feedback) and transformational (i.e., goal setting, strengthening and sustaining team relationships) strategies.

During pre-Implementation, both implementation strategy bundles are being employed with PRISM to engage multi-level partners and identify relevant contextual factors. Relational Facilitation is being used to assess, guide, and develop high-quality interprofessional cross-setting relationships for the purpose of task integration. Relational Facilitation strategies are being implemented in partnership with clinical intervention leads and adapted based on each site's needs. PRISM is being used repeatedly to inform adaptions so they align with context, beginning with the pre-implementation phase.

Currently, we are implementing both implementation strategy bundles to support the teams to review implementation data and rate progress on RE-AIM outcomes and then reflect on the “gap” between rated importance and their progress. Iterative RE-AIM assessments will guide adaptations and action plans, especially by using evidence to direct efficient decisions about care approaches. Progress at each assessment will guide resource allocation and intensity of Relational Facilitation for the subsequent period. The two implementation bundles support each other and are designed to begin with the lowest intensity of facilitation activities using the “minimal intervention needed for change” approach (35). Based on the results of iterative assessments, progress on outcome measures and priorities of the EBP teams, more intensive activities will be applied in an iterative manner, while tracking time and costs.

Findings

We have faced some challenges as we rolled-out each implementation strategy bundle across the different EBPs at various VA sites. We rolled out Relational Facilitation and Iterative RE-AIM separately, in that order, to minimize the staff burden at the local site, and it remains to be seen what impact this will have. We have found it necessary to adapt the process we had used in our previous research with Iterative RE-AIM to work with implementation partners who have less time and engagement, assess the site's priorities, define, and operationalize RE-AIM outcomes as well as adapt requirements for staff training due to our remote work environment.

Crosscutting lessons learned

Although there were differences across the case studies that utilized Iterative RE-AIM, there were also crosscutting findings that emerged across projects. Table 4 and the text below summarize these findings.

TABLE 4

Table 4. Key crosscutting issues in applying Iterative RE-AIM.

Finding #1: Engagement of the persons implementing, making decisions about, or impacted by a program is both important and challenging

The case studies employed different numbers and types of clinical and community partners, but most often centered on the team directly implementing the program. It is important to have team members share perceptions and agree on priorities, but it is unclear how many perspectives are needed and if these need to be the same persons across all Iterative RE-AIM assessments. Although most recommendations regarding team science (36) stress including a full array recipients (e.g., patients, employees, opinion leaders, organizational decision makers, community representatives) as part of the decision making team, the example cases did not involve all these categories of partners. Congruent with recent emphases and recommendations for complex interventions (20) and adaptations (7) we are finding the level of engagement of multiple implementation partners to be critical for success. However, including a larger number and different types of participants needs to be balanced against the logistics and costs of those members being able to meet regularly to continue the Iterative RE-AIM process over time. It will be informative to see if Iterative RE-AIM applications that involve more partners with more diverse perspectives produce better long-term results than those that do not.

Finding #2: Having real time objective data on RE-AIM outcome for use to evaluate progress is ideal but challenging

Except for the lung ultrasound study, the current Iterative RE-AIM applications did not have real time, objective data on RE-AIM outcomes to evaluate progress. Sometimes project records provided information on Reach or Adoption rates, but many of the ratings of progress were made based on the subjective impressions of the team members. Design and proactive use of process data systems that can be queried to produce frequent updates on issues such as fidelity, adaptations, and representativeness (equity) of RE-AIM results would improve the quality of data available for decision making. Once data on progress on RE-AIM dimensions are available, they need to be summarized and communicated in a way that is readily understood and actionable. Current Iterative RE-AIM projects have used some form of a bar chart as shown in Figure 1, and most participants seem to understand and find these displays useful, with exception that information about variability across raters was unclear for some participants. Newer applications of Iterative RE-AIM are experimenting with different types of visual displays, including giving participants their choice of different data displays.

FIGURE 1

Figure 1. Example of Iterative RE-AIM gap analysis to study discrepancies between importance and progress on RE-AIM outcome dimensions.

Finding #3: Leaders of all the Iterative RE-AIM projects agreed that the implementation team exchanging perspectives on progress and priorities, and then making a group decision about the most appropriate area(s) to target and strategies to use is the core of the Iterative RE-AIM process

These discussions can be rich and enlightening for participants but can also require experienced facilitation if there are large differences in perception, power, or information across team members.

Finding #4: Evaluating the impact of the iterative adaptations selected is challenging and has been suboptimal to date

The primary method to date has been estimating progress at the next meeting across RE-AIM dimensions, but this is non-specific and suffers from the same concerns about data quality noted above. Even with high quality data, without experimentally testing strategies, it is difficult to attribute improvement to use of a strategy separate from numerous other dynamic program and contextual factors (19, 37). This is a conceptual and methodological challenge for all approaches to adaptions, not just Iterative RE-AIM. Since it is impossible in many situations such as our Quadruple Aim QUERI project to separate and independently evaluate the impact of separate implementation strategies, this may never be knowable. It is likely best addressed through mixed methods approaches using proximal quantitative data (such as rapid EHR data) followed by qualitative probes to provide confirmation and contextual understanding.

Finding #5: The amount of time and resources required for Iterative RE-AIM can vary considerably

The case studies vary from a single midpoint use of Iterative RE-AIM to numerous biweekly applications; the number of team members from one or two up to 14; and the work required to prepare data summaries from being very little when automated EHR reports are available to fairly time consuming if ratings from several persons need to be analyzed, integrated and feedback displays produced by hand.

Finding #6: There needs to be a balance between standardization of Iterative RE-AIM processes and adaptation to specific projects and contexts

We find useful the concepts of form and function (22) of adherence to core functions of Iterative RE-AIM as outlined in Table 1, while encouraging tailoring of the specific forms- e.g., data sources, data display choices, which staff to involve, number of iterations. We also experienced some challenges in making decisions about what constitutes an adaptation vs. just a small change that is not intended to improve fit to context.

Discussion and future directions for practice and research

Our findings provide initial results from multiple projects utilizing a relatively new Iterative RE-AIM process to identify priorities, iteratively guide adaptations, and monitor progress over time. In general, use of Iterative RE-AIM was feasible for the projects to implement. The Iterative RE-AIM process revealed new insights for the team so that they could better discern how they wanted their implementation to proceed and what was most important through prioritizing specific RE-AIM dimensions. Progress over time was built into the process through the repeated Iterative RE-AIM cycles which specifically called out how the project was proceeding by gathering both data and perceptions. Just having the structure provided a way for the goals of a program to stay more present in the minds of the implementation teams.

Although the future use of Iterative RE-AIM is promising, there is still much to learn to maximize its effectiveness and efficiency. In this section, we key issues for practice of Iterative RE-AIM and identify opportunities for future research evaluating Iterative RE-AIM (Table 5). First, data availability to inform implementation actions and potential adaptations is important. We anticipate greater availability of EHR based dashboards such as that in our lung ultrasound project as well as close to real time data from ecological momentary assessments in the future. More systematic process data collection and tracking fields in project records on reach, equity, and implementation issues would also be helpful and should be feasible for most projects. Once data are collected, they need to be made available rapidly in easily understood and actionable formats. There is an opportunity to identify innovative ways to display data in visual dashboards and inform high quality data generation for rapid research and adaptations.

TABLE 5

Table 5. Directions for future research and practice.

Second, there is a need to more systematically assess impact and ensure that both specific measures and Iterative RE-AIM priorities reflect issues of the greatest value to the participants and system where the program is being implemented. This could be done by engaging implementation partners in the selection, refinement and use of pragmatic measures to rapidly assess the impact of resulting strategies and adaptations. Third, capturing relevant context to inform program implementation and adaptation should facilitate success. This could be operationalized by developing more explicit guidance for adapting Iterative RE-AIM to context and creating alignment with implementing system resources and relationships. Future research could identify key aspects of context for Iterative RE-AIM using PRISM or other conceptual models. Additionally, rapid qualitative methods (28), used in conjunction with “Iterative PRISM,” have the potential to serve as a powerful tool to acquire timely and actionable information on dynamic context. The information produced should allow for a better understanding of current barriers to RE-AIM outcomes and be used to adapt implementation strategies more effectively. These two innovative methods, both focused on improving the rapidity of implementation research should be explored. Fourth, the intensity and timing of Iterative RE-AIM need to be better understood. Practice applications of Iterative RE-AIM could consider stepped care or minimal intervention needed for change approaches (35). Future research could conduct cost effectiveness and comparative effectiveness studies of different levels of facilitation, which parties are involved, and the timing and frequency of Iterative RE-AIM. Finally, careful tracking and investigation of the impact of Iterative RE-AIM on health equity is needed, assessing both intended and potential unintended consequences.

Our findings should be interpreted in the context from which they were derived. Limitations to this report include that none of the case studies are experimental studies or compare Iterative RE-AIM to other implementation strategies. Also, only two of the case examples are completed projects and the ultimate impact of Iterative RE-AIM and the adaptions conducted is yet unknown. While two of the projects were directed and implemented by researchers other than the original developers of Iterative RE-AIM, all were led by investigators experienced in using RE-AIM. Future research should evaluate the level of expertise (in RE-AIM, implementation science, and group facilitation) required for successful use of Iterative RE-AIM. Strengths of this report include pragmatic use of Iterative RE-AIM across several projects diverse in terms of health care issues, settings, patient and staff characteristics and different forms of Iterative RE-AIM. Iterative RE-AIM appears to be broadly applicable, including during the rapidly changing and challenging context of the COVID-19 pandemic. We hope that by providing details of these different applications of Iterative RE-AIM and making resources to conduct Iterative RE-AIM publicly available https://re-aim.org/applying-the-re-aim-framework/re-aim-guidance/use-during-implementation/ will facilitate replication and investigation of its impact and usefulness in guiding adaptations across a variety of different conditions and contexts. We look forward to hopefully having enough applications of Iterative RE-AIM to conduct a more formal review in the future.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving human participants were reviewed and approved by COMIRB. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

Author contributions

REG and BR conceptualized the Iterative RE-AIM process. CB, MM, and RA developed most of the original materials, methods, and resources. RNG drafted the article and contributed to paper compilation. CB, MM, BR, AM, MF, and JH contributed case study examples. RA, BR, and MM contributed to the data collection and analysis. All authors contributed to the implementation of the projects, interpretation of results, and manuscript revisions.

Funding

The research reported in this publication was supported in part by the National Cancer Institute of the National Institutes of Health (NIH) under Center P50 Grant Award Number: 5P50CA244688. It was also supported in part by the National Institutes of Health Grant Award Numbers: 5UH3CA233314-03 and UL1 TR002535. Also supporting this research in part was the National Heart, Lung, and Blood Institute Grant Award Numbers: K12 HL137862, U01HL138647, and 5K12HL137862. This work was also funded in part by the VA HSRD/QUERI.

Acknowledgments

We would like to extend our gratitude to the Colorado Implementation Science Cancer Control Center.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Author disclaimer

The contents of this work are the authors' sole responsibility and do not necessarily represent the official views of the VA.

Abbreviations

RE-AIM, Reach, Effectiveness, Adoption, Implementationand Maintenance; PRISM, Practical Robust Implementation and Sustainability Infrastructure.

References

1. Chambers DA, Norton WE. The adaptome: advancing the science of intervention adaptation. Am J Prev Med. (2016) 51(4 Suppl. 2):S124–31. doi: 10.1016/j.amepre.2016.05.011

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Stirman SW, Miller CJ, Toder K, Calloway A. Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implement Sci. (2013) 8:65. doi: 10.1186/1748-5908-8-65

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Tabak RG, Hook M, Chambers DA, Brownson RC. The conceptual basis for dissemination and implementation research. In: Brownson RC, Colditz GA, Proctor EK, editor. Dissemination Implementation Research in Health: Translating Science to Practice 2R. 2nd ed. New York, NY: Oxford University Press (2018). p. 73–88.

Google Scholar

4. Tabak RG, Chambers D, Khoong EC, Brownson RC. Models in dissemination and implementation research: useful tools in public health services and systems research. Front Public Health Serv Syst Res. (2013) 2. doi: 10.13023/FPHSSR.0201.08

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Powell BJ, Beidas RS, Lewis CC, Aarons GA, McMillen JC, Proctor EK, et al. Methods to improve the selection and tailoring of implementation strategies. J Behav Health Serv Res. (2015) 44:177–94. doi: 10.1007/s11414-015-9475-6

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Rabin BA, McCreight M, Battaglia C, Ayele R, Burke RE, Hess PL, et al. Systematic, multimethod assessment of adaptations across four diverse health systems interventions. Front Public Health. (2018) 6:102. doi: 10.3389/fpubh.2018.00102

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Moore G, Campbell M, Copeland L, Craig P, Movsisyan A, Hoddinott P, et al. Adapting interventions to new contexts-the ADAPT guidance. BMJ. (2021) 374:n1679. doi: 10.1136/bmj.n1679

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Birken SA, Powell BJ, Shea CM, Haines ER, Alexis Kirk M, Leeman J, et al. Criteria for selecting implementation science theories and frameworks: results from an international survey. Implement Sci. (2017) 12:124. doi: 10.1186/s13012-017-0656-y

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Strifler L, Cardoso R, McGowan J, Cogo E, Nincic V, Khan PA, et al. Scoping review identifies significant number of knowledge translation theories, models, and frameworks with limited use. J Clin Epidemiol. (2018) 100:92–102. doi: 10.1016/j.jclinepi.2018.04.008

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Rabin B, Glasgow R. Dissemination & Implementation Models in Health Research Practice. University of California San Diego, University of Colorado Anshcutz Medical Campus (2022). Available online at: https://dissemination-implementation.org/.

Google Scholar

11. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administr Policy Mental Health Mental Health Serv Res. (2010) 38:65–76. doi: 10.1007/s10488-010-0319-7

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health. (2019) 7:64. doi: 10.3389/fpubh.2019.00064

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Glasgow RE, Battaglia C, McCreight M, Ayele RA, Rabin BA. Making implementation science more rapid: use of the RE-AIM framework for mid-course adaptations across five health services research projects in the Veterans Health Administration. Front Public Health. (2020) 8:194. doi: 10.3389/fpubh.2020.00194

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Kirk MA, Moore JE, Wiltsey Stirman S, Birken SA. Towards a comprehensive model for understanding adaptations' impact: the model for adaptation design and impact (MADI). Implement Sci. (2020) 15:56. doi: 10.1186/s13012-020-01021-y

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Wingood GM, DiClemente RJ. The ADAPT-ITT model: a novel method of adapting evidence-based HIV interventions. J Acquir Immune Defic Syndr. (2008) 47(Suppl. 1):S40–6. doi: 10.1097/QAI.0b013e3181605df1

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Aschbrenner KA, Mueller NM, Banerjee S, Bartels SJ. Applying an equity lens to characterizing the process and reasons for an adaptation to an evidenced-based practice. Implement Res Pract. (2021) 2:1–8. doi: 10.1177/26334895211017252

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Aschbrenner KA, Bond GR, Pratt SI, Jue K, Williams G, Banerjee S, et al. Evaluating agency-led adaptions to an evidence-based lifestyle intervention for adults with serious mental illness. Implement Res Prac. (2020) 1:2633489520943200. doi: 10.1177/2633489520943200

CrossRef Full Text | Google Scholar

18. Coury J, Miech EJ, Styer P, Petrik AF, Coates KE, Green BB, et al. What's the “secret sauce”? How implementation variation affects the success of colorectal cancer screening outreach. Implement Sci Commun. (2021) 2:5. doi: 10.1186/s43058-020-00104-7

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. (2013) 8:117. doi: 10.1186/1748-5908-8-117

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Skivington K, Matthews L, Simpson SA, Craig P, Baird J, Blazeby JM, et al. A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. BMJ. (2021) 374:n2061. doi: 10.1136/bmj.n2061

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Hawe P, Shiell A, Riley T. Complex interventions: how “out of control” can a randomised controlled trial be? BMJ. (2004) 328:1561–3. doi: 10.1136/bmj.328.7455.1561

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Perez Jolles M, Lengnick-Hall R, Mittman BS. Core functions and forms of complex health interventions: a patient-centered medical home illustration. J General Internal Med. (2019) 34:1032–8. doi: 10.1007/s11606-018-4818-7

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf. (2008) 34:228–43. doi: 10.1016/S1553-7250(08)34030-6

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Paniagua-Avila A, Fort MP, Glasgow RE, Gulayin P, Hernández-Galdamez D, Mansilla K, et al. Evaluating a multicomponent program to improve hypertension control in Guatemala: study protocol for an effectiveness-implementation cluster randomized trial. Trials. (2020) 21:509. doi: 10.1186/s13063-020-04345-8

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Fort MP, Mundo W, Paniagua-Avila A, Cardona S, Figueroa JC, Hernández-Galdamez D, et al. Hypertension in Guatemala's public primary care system: a needs assessment using the health system building blocks framework. BMC Health Serv Res. (2021) 21:908. doi: 10.1186/s12913-021-06889-0

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Hernández-Galdamez D, Mansilla K, Peralta AL, Rodríguez-Szaszdi J, Ramírez JM, Roche D, et al. Monitoring study participants and implementation with phone calls to support hypertension control during the COVID-19 pandemic: the case of a multicomponent intervention trial in Guatemala. Glob Heart. (2021) 16:77. doi: 10.5334/gh.954

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Maw AM, Morris MA, Glasgow RE, Barnard J, Ho PM, Ortiz-Lopez C, et al. Using Iterative RE-AIM to enhance hospitalist adoption of lung ultrasound in the management of patients with COVID-19: an implementation pilot study. Implement Sci Commun. (2022) 3:89. doi: 10.1186/s43058-022-00334-x

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Palinkas LA, Mendon SJ, Hamilton AB. Innovations in mixed methods evaluations. Annu Rev Public Health. (2019) 40:423–42. doi: 10.1146/annurev-publhealth-040218-044215

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Peterson K, Anderson J, Bourne D, Boundy E. Scoping Brief: Care Coordination Theoretical Models and Frameworks. VA Evidence Synthesis Program Evidence Briefs. VA Evidence Synthesis Program Reports. Washington, DC (2011).

Google Scholar

30. McDonald KM, Sundaram V, Bravata DM, Lewis R, Lin N, Kraft SA, et al. Closing the Quality Gap: A Critical Analysis of Quality Improvement Strategies, Vol 7. Care Coordination. AHRQ Technical Reviews. Rockville, MD(2007).

Google Scholar

31. Cordasco KM, Hynes DM, Mattocks KM, Bastian LA, Bosworth HB, Atkins D. Improving care coordination for veterans within VA and across healthcare systems. J Gen Intern Med. (2019) 34(Suppl. 1):1–3. doi: 10.1007/s11606-019-04999-4

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Huber TP, Rodriguez HP, Shortell SM. The influence of leadership facilitation on relational coordination among primary care team members of accountable care organizations. Health Care Manage Rev. (2020) 45:302–10. doi: 10.1097/HMR.0000000000000241

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Burke RE, Kripalani S, Vasilevskis EE, Schnipper JL. Moving beyond readmission penalties: creating an ideal process to improve transitional care. J Hosp Med. (2013) 8:102–9. doi: 10.1002/jhm.1990

PubMed Abstract | CrossRef Full Text | Google Scholar

34. McCreight MS, Rabin BA, Glasgow RE, Ayele RA, Leonard CA, Gilmartin HM, et al. Using the Practical, Robust Implementation and Sustainability Model (PRISM) to qualitatively assess multilevel contextual factors to help plan, implement, evaluate, and disseminate health services programs. Transl Behav Med. (2019) 9:1002–11. doi: 10.1093/tbm/ibz085

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Glasgow RE, Fisher L, Strycker LA, Hessler D, Toobert DJ, King DK, et al. Minimal intervention needed for change: definition, use, and value for improving health and health research. Transl Behav Med. (2014) 4:26–33. doi: 10.1007/s13142-013-0232-1

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Emmons KM, Viswanath K, Colditz GA. The role of transdisciplinary collaboration in translating and disseminating health research: lessons learned and exemplars of success. Am J Prev Med. (2008) 35(2 Suppl.):S204–10. doi: 10.1016/j.amepre.2008.05.009

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Shelton RC, Chambers DA, Glasgow RE. An extension of RE-AIM to enhance sustainability: addressing dynamic context and promoting health equity over time. Front Public Health. (2020) 8:134. doi: 10.3389/fpubh.2020.00134

PubMed Abstract | CrossRef Full Text | Google Scholar