Applying the updated MRC framework for developing and evaluating complex interventions with integrated implementation conceptual knowledge: an example using NeuroRehabilitation OnLine

Background: The updated 2021 UK Medical Research Council (MRC) Framework offers a valuable guide for implementation scientists to navigate the challenges of the development and evaluation of complex interventions. However, despite extensive citations, there is limited evidence of how the MRC Framework has been used in its entirety and limited integration with relevant implementation conceptual knowledge. To address this, we demonstrate the application of the updated MRC Framework incorporating implementation science frameworks, strategies, and outcomes. This example uses a telerehabilitation intervention, NeuroRehabilitation OnLine (NROL), implemented within an existing healthcare system.

Methods: Within a clinical-academic partnership, we completed the MRC Framework checklist, and the context was described using the updated Consolidated Framework for Implementation Research (CFIR). We used a deliberative process to operationalise the MRC phases: adaptation of NROL based on the ADAPT guidance and establishing the feasibility of NROL through concurrent implementation and evaluation. Phases are described in two iterations: within a single service and then when scaled up as a regional innovation. Stakeholders were involved throughout. Implementation strategies were identified using the CFIR-Expert Recommendations for Implementing Change (CFIR-ERIC) matching tool. Proctor's implementation outcomes were selected for the evaluation.

Results: The MRC Framework provided a useful structure when applied iteratively to address key uncertainties for implementation. Stakeholder co-production was integral to all phases, in both iterations. An additional sustainment phase was added to the framework, reflecting that the value proposition discussions with decision-makers inevitably culminated in decision points. This guided decision-making for NROL to be scaled up. Logic Models were co-produced and iterated to depict programme theory and formalise the integration of implementation conceptual knowledge.

Conclusion: Synergistic in nature, the MRC Framework benefitted the conceptualisation of implementation through the use of its phases, and implementation science knowledge was useful in enacting the core elements within the MRC Framework. This example of application will be directly relevant to the field of rehabilitation and build transferable knowledge to enrich implementation research and practice.

Background

The implementation-practice gap is well documented, with implementation scientists tasked with studying methods to promote the uptake of evidence-based practices. However, despite this important mission, the field faces significant criticisms. Most notably, these include its lack of relevance and timely application in implementation practice, with an overabundance of theories and frameworks, resulting in fragmented understanding, inconsistent use, and static rather than evolving theory, which all present barriers to advancing knowledge and practice (1–3).

In response to these challenges, this paper proposes the use of the updated United Kingdom (UK) Medical Research Council (MRC) Framework for developing and evaluating complex interventions (hereafter “MRC Framework”) as an overarching structure implementation scientists could use to guide their research (4–7). The MRC Framework was published in 2000 (4) with revision in 2006 (5) to offer a systematic architecture to the entire process of developing, evaluating, and implementing complex interventions. From 2006, the framework has consisted of non-linear phases, to develop/identify (adapt), assess feasibility, and implement and evaluate an intervention. The MRC Framework underwent substantial updates in 2021 (6), increasing its scope to a broader range of research perspectives (efficacy, effectiveness, theory-based, and systems) and to include six core elements that should be considered at each phase: identify key uncertainties, engage stakeholders, consider context, develop and refine programme theory, refine the intervention, and economic considerations (see Figure 1). The evolution of the framework reflects a growing sophistication and understanding of developing and evaluating complex interventions, providing a pluralistic guide for intervention implementation and evaluation. The MRC Framework is widely recognised and recommended, for example, with over 16,500 citations (8), and the UK National Institute for Health and Care Research recommends its use in their funding guidance (9). The guidance offers several case study examples (7), as does a recent discussion paper (10), but these are primarily of single phases. There is a need for further operational guidance with research that progresses through all phases of the framework (11).

Figure 1

Figure 1. “Updated MRC Framework for developing and evaluating complex interventions”. Context, any feature of the circumstances in which an intervention is conceived, developed, evaluated, and implemented; programme theory, describes how an intervention is expected to lead to its effects and under what conditions—the programme theory should be tested and refined at all stages and used to guide the identification of uncertainties and research questions; stakeholders, those who are targeted by the intervention or policy, involved in its development or delivery, or more broadly those whose personal or professional interests are affected (that is, who have a stake in the topic)—this includes patients and members of the public as well as those linked in a professional capacity; uncertainties, identifying the key uncertainties that exist, given what is already known and what the programme theory, research team, and stakeholders identify as being most important to discover—these judgments inform the framing of research questions, which in turn govern the choice of research perspective; refinement, the process of fine tuning or making changes to the intervention once a preliminary version (prototype) has been developed; economic considerations, determining the comparative resource and outcome consequences of the interventions for those people and organisations affected. Reproduced with permission from “Framework for developing and evaluating complex interventions” by Skivington et al., licensed under CC BY 4.0.

The potential to combine the updated MRC Framework with developments in implementation science warrants consideration. There are many parallels between the MRC Framework and implementation research, including an emphasis on the importance of context and iteration and that implementation interventions tend to be complex interventions. There has been a call for the co-existence of multiple paradigms in the field of implementation science, with a plethora of theories, frameworks, and models that need to go beyond application with reverence (1). It is acknowledged that the MRC Framework, as a living document, should stimulate debate (12) and be used flexibly. Implementation conceptual knowledge has been combined previously (13–16), but to our knowledge, has not been integrated with the updated MRC Framework within an implementation research study. By leveraging the strengths of the MRC Framework and incorporating implementation science theories, a coherent, comprehensive approach to research could advance the field and better bridge the gap between implementation research and practice.

We provide a worked example of iteratively applying the updated MRC Framework to the implementation of a telerehabilitation intervention, NeuroRehabilitation OnLine (NROL), which, in the UK National Clinical Guideline for Stroke, is promoted as an exemplar innovation for delivering remote rehabilitation (17). NROL is a group-based multidisciplinary telerehabilitation approach for patients undergoing stroke and neurorehabilitation that has been implemented within the National Health System (NHS) in a region of the UK (18). Within rehabilitation literature, akin to wider health research, reference to the MRC Framework to date is predominantly related to methodology commentary or mentioned in passing without discussion on how the researchers applied the framework (19). We aim to fill a gap in evidence by using a real-world example of innovation implementation and scale-up of NROL to demonstrate the application of the updated MRC Framework with the integration of implementation science knowledge.

Methods

The updated MRC Framework (Figure 1) was applied based on guidance (6, 7). We used a deliberative process to operationalise the MRC phases in their entirety: adaptation of NROL based on the ADAPT guidance (20) and establishing the feasibility of NROL through concurrent implementation and evaluation.

We completed the MRC Framework checklist (Table 1) (7) to expand on the core elements, and used implementation conceptual knowledge to help enact these, including the Consolidated Framework for Implementation Research (CFIR-CFIR) as a commonly used determinant framework for context that has been associated with effective implementation (21, 22), the CFIR-Expert Recommendations for Implementing Change (CFIR-ERIC) matching tool to match identified CFIR-based contextual factors to relevant key ERIC implementation strategies for further stakeholder engagement (23–26), and Proctor's outcomes as an established taxonomy to conceptualise implementation success and inform addressing key uncertainties (27). The inherent relationships were formalised, guided by the Implementation Research Logic Model (IRLM) (13, 14), to depict elements of programme theory [determinants (CFIR), strategies (ERIC), mechanisms, and outcomes (Proctor's)] and provide a clear, visual summary for diverse stakeholders. The production of a logic model aligns with programme theory as a core element of the MRC Framework. The process was iterative and refined based on ongoing stakeholder input.

Table 1

Table 1. MRC Framework checklist for NROL.

Context

As a core element, context is described below, with considerations summarised in the MRC checklist (Table 1).

NROL is a group-based, real-time telerehabilitation approach with dedicated technology assistance. The approach was originally developed in 2020 as a standalone intervention by clinical academics in London in response to the pandemic (28). From 2021, the approach was integrated within the NHS in a region of the UK to complement in-person therapy. Targeted therapy and community peer-support groups are delivered within recurring 6-week blocks, with groups facilitated by multidisciplinary team members from the existing therapy workforce coordinated by an NROL operational team. For comprehensive detail please refer to the NROL description using the Template for Intervention Description and Replication (TIDieR) checklist (18).

NROL within the NHS was implemented in two iterations over 4 years, first as a single-service level and then expanded to a regional innovation. For the first iteration (January 2021–March 2022) (29), the single service was a community-based neurological rehabilitation service in the northwest of England (East Lancashire Hospitals NHS Trust) consisting of two teams, the NeuroRehabilitation Team and Stroke Therapy Team. For the second iteration (April 2022 to ongoing) (30), the regional neurological services included were community-based stroke and/or neurorehabilitation services from four NHS Trusts (i.e., partnered Trusts) within the Lancashire and South Cumbria Integrated Care System in the northwest of England (East Lancashire Hospitals NHS Trust, University Hospitals of Morecambe Bay NHS Foundation Trust, Blackpool Teaching Hospitals NHS Foundation Trust, and Lancashire and South Cumbria NHS Foundation Trust). All the services provide multidisciplinary rehabilitation across community settings to adults who have a sudden onset or progressive/intermittent neurological condition.

Stakeholder groups have evolved over time and have included a clinical-academic partnership including clinical academics (Lancaster University), NHS management (Service leads), NHS therapy staff (deliverers, referrers), NROL Operational team (Manager, Technology Support, administrator), healthcare decision-makers (including Commissioners), and academics. Other key stakeholder groups include patients and families, third-sector organisations (SameYou Charity and Stroke Association), and the original standalone intervention developers (National Hospital for Neurology and Neurosurgery and University College London).

Stakeholder co-production

The application of the MRC Framework was guided by co-production principles, where power is shared, and all team members contribute their perspectives and skills (31, 32). We respected and valued everyone's knowledge equally, fostering reciprocal relationships where everyone benefits from collaboration. By prioritising relationship-building and maintaining open communication, we ensured this project remained grounded in real-world experiences and served the needs of all stakeholders relevant to the implementation of NROL. Adaptation, implementation, and evaluation involved stakeholder co-production activities, using targeted strategies identified using the CFIR-ERIC matching tool, with relevant CFIR domains and constructs discussed and agreed upon within the clinical-academic partnership.

Positionality

Our clinical-academic project team, embedded within the UK NHS in the northwest of England, embodies a unique perspective shaped by the inclusion of dual clinical-academic roles since 2019. Led by two physiotherapists (LC, SA) with stroke rehabilitation clinical practice and research experience, and expertise in implementation science and complex intervention research. We acknowledge the influence of our backgrounds on our viewpoints, and our dual roles provide the opportunity to be a conduit between the research and clinical practice communities. As part of the NHS ecosystem, we capitalised on internal knowledge, emphasising the importance of partnership in all project phases and decisions, and did not have a pre-determined project endpoint at the start. One co-author (JR-M), an author of the MRC Framework and an implementation framework, provided oversight and critique from a methodological and theoretical perspective.

Results

The MRC Framework was applied in two iterations, first as a single-service intervention (January 2021–March 2022) and then as a regional multi-service innovation (April 2022 to ongoing). An additional “Sustainment” phase was included, with key decision points identified.

We have called our approach the “advanced” MRC Framework for ease of reference. An overview is shown in Figure 2 and detailed further below within the phase-specific considerations section.

Figure 2

Figure 2. Advanced MRC Framework. Overview of the application of an advanced MRC Framework, showing two iterations: as a single-service intervention and regional multi-service innovation for NROL, NeuroRehabilitation OnLine. Advancements to the updated MRC include a sustainment phase. Integrated implementation conceptual knowledge includes ERIC, CFIR, and IRLM.

Stakeholder co-production

Key stakeholder groups engaged in multiple co-production activities to adapt, implement, and evaluate the intervention. These included identifying influential contextual factors, agreeing upon implementation strategies, selecting relevant implementation outcomes, and developing the logic models.

The contextual factors informed by the CFIR that were identified by stakeholders as influential during co-production activities were as follows:

- Characteristics of NROL: complexity, relative advantage, and adaptability.

- Outer setting: strategic fit, guidelines, pandemic, financial climate, and charitable funding.

- Inner setting (service and regional level): culture, leadership engagement, and available resources.

- Characteristics of individuals (patient and staff): knowledge and beliefs, self-efficacy, and capability.

Further details are provided within the NROL logic models (Figure 3) and in NROL publications (18, 29, 30).

Figure 3

Figure 3. NROL Logic Models. Co-produced logic models showing two iterations: as a single-service intervention (a) and regional multi-service innovation (b). NROL, NeuroRehabilitation OnLine; NHS, National Health Service; CFIR, Consolidated Framework for Implementation Research; ERIC, Expert Recommendations for Implementing Change; SOP, standard operating procedure; SSNAP, Sentinel Stroke National Audit Programme.

The CFIR-ERIC matching tool identified 17 and 18 key ERIC strategies for the single-service and regional NROL iterations, respectively. Each co-production activity employed several of the ERIC strategies identified as important to support effective implementation. Illustrative examples of activities and contributions for each stakeholder group are provided in Table 2. Of note, the clinical-academic partnership was a key enabler of the co-production activities, as it was the basis of an established trusted relationship, facilitating meaningful stakeholder engagement with open and regular communication and supporting collaborative resource production and knowledge exchange.

Table 2

Table 2. Stakeholder groups, co-production activities, and contributions.

Phase-specific considerations

Operationalising the MRC Framework phases (see Figure 2) was dynamic and overlapping, and core elements were enacted using implementation concepts. A summary of the main considerations for the phases is provided below, highlighting the similarities and differences in focus between implementation at the single-service and regional levels and how these influenced working through the phases. For more detail, the phases have been described in a stepwise format together with relevant ERIC implementation strategies and involved stakeholder groups in the Supplementary Material.

Adaptation

The focus of this phase in both iterations was to ensure an intervention-context fit and plan for a successful implementation within the existing NHS healthcare system. The implementation strategies for both the single-service and regional levels had several similarities. Both prioritised building coalitions by leveraging existing clinical-academic partnerships and expanding involvement to include multiple stakeholders. Conducting local needs assessments and assessing readiness was integral, providing insights into alignment with strategic priorities, understanding context and contextual differences, and identifying staff and approval requirements. Initially, a learning collaborative was established, which agreed on implementation outcomes and co-produced a logic model for implementation in the single service (see Figure 3a). For the regional scale-up, the collaborative was extended to incorporate representatives from all the involved community-based neurological services from the four partnered NHS Trusts and management with overarching regional responsibility, and a new iteration of the logic model was co-produced for the regional multi-service approach (see Figure 3b).

Feasibility, implementation, and evaluation

To address the key uncertainties, NROL was sequentially implemented with targeted ERIC strategies. The similarities between the iterations included the acquisition of approvals, allocation of key roles, staff training, and undertaking of an evaluation with knowledge shared. In addition, accessing funding was crucial for initiating, sustaining, and scaling up operations. In the single-service iteration, efforts were concentrated locally, whereas, in the regional iteration, system-level coordination of services was required. This was needed to ensure overarching approvals and buy-in, formalisation of roles and responsibilities across services, and collective adoption. Evaluation in both iterations was mixed methods. A summary of the evaluation of NROL implemented at a single service has been published, with NROL demonstrating positive outcomes and opinions (29). The formative evaluation of the regional innovation had a shift in focus of implementation outcomes as agreed by the stakeholders with an emphasis of evaluation on understanding system-level efficiencies alongside delivery metrics (30).

Sustainability, sustainment, and decision point(s)

Sustainability can be defined as the “continued capacity to deliver the innovation, continued delivery of the innovation, and continued receipt of benefits” (33). Sufficient emphasis should be placed on the factors that may contribute to sustainment throughout the process of adaptation, implementation, and evaluation, including the identification of facilitators, barriers and value. To reflect its importance, we deemed sustainment to warrant a distinct phase and added this in our use of the MRC Framework. The sustainment phase refers to when the intervention is supported and maintained with continued delivery, allowing for modifications and tailoring as necessary (34). Sustainability prioritised communication and relationships between stakeholders, including decision-makers, to facilitate the continued use and impact of the intervention. The coalition, in particular clinical-academic staff, management, and champions, played a pivotal role in strategic alignment and identifying opportunities for visibility. Funding and contracting in both iterations used evaluation findings to identify advantages, resources, and barriers, with the knowledge shared and interpreted by stakeholders. A key difference lay in the scale of stakeholder engagement. The regional approach extended the stakeholder group to include regional healthcare system commissioners and executive boards and required business cases.

It was recognised that framework phases are often time-limited by factors such as staffing and financial resources and inevitably culminate in decision points. For example, decisions had to be reached as to whether to move forward, and, e.g., how to continue current provision, discontinue, or scale up. Although decision points were evident in both iterations, differences in decision-makers reflected the widening scope, with engagement changing from service level to regional executive boards. In this example, a first major decision point was required at the end of single-service funding for NROL, and then again at a regional level (see Figure 2). The different strategic contexts altered the value proposition, with the initial decision point discussing the potential for NROL scale-up to a regional entity and the second reflecting the aspiration of commissioning for sustainment of the regional NROL innovation.

Implementation research logic models

Elements of the programme theory were described visually using co-produced logic models, with final versions detailed for the single-service and regional multi-service iterations (Figure 3). These bring together the use of the CFIR and ERIC strategies, and implementation outcomes and highlight the shift in focus to the system level when NROL was scaled up to a regional level.

Discussion

This publication presents a novel approach to the use of the MRC Framework in combination with implementation conceptual knowledge. To advance the MRC Framework, we integrated the CFIR, ERIC strategies, and selected Proctor's implementation outcomes as methods to enact several of the framework's core elements. This was collated and visualised guided by Implementation Research Logic Models. In addition, we incorporated a focus on sustainability by adding a distinct phase dedicated to sustainment. Furthermore, we acknowledge the existence of decision points that may lead to new iterations of the MRC Framework application, facilitating further progression through the phases to scale up an intervention. This example should assist implementation research and practice by demonstrating how paradigms can be combined, providing a concrete illustration of the application of all phases of the MRC Framework as an intervention is scaled up and offering a valuable reference for future work.

A challenge of the MRC Framework is that it lacks some operational guidance, with core elements that need to be considered at each phase stated but with minimal detail as to how to apply these. We have provided an example of how the operationalisation of the framework could be achieved. The MRC Framework provided an overarching document for framing stakeholder co-production activities around essential considerations for adaptation, implementation, and evaluation. As encouraged (7, 10), we completed an MRC checklist and updated this over time, with the optimal timing and level of detail of this checklist guided by pragmatic decisions and resources. In agreement with the recommendation by Levack et al., we found the checklist a useful tool to frame our research methods. Integrating implementation conceptual knowledge was helpful, particularly regarding the core elements of “considering context,” “engaging stakeholders,” and “developing and refining programme theory.” The iterative nature of the MRC Framework allowed for responsiveness to emerging challenges and opportunities, but we also highlighted the need for clear decision points regarding intervention continuation, modification, or cessation based on the value proposition, including evaluation outcomes.

Responding to suggestions by others (7, 10, 19), we aimed to incorporate a tangible, usable example illustrating the application of the framework. However, it is unclear whether the process or reporting enhanced the effectiveness of our implementation efforts and offset the resources, time, and expertise required, or simply increased the complexity involved. The process of grappling with this example prompted us to consider implementation and sustainability factors early in our planning process but also facilitated a more systematic process that likely improved our overall approach and helped in building stakeholder relationships. This tension between the use of theory-driven methods vs. practical usefulness is a challenge in many areas of implementation research (2). However, we feel the trade-off in resources may yield further benefits in future iterations, scale-out, and subsequent applications in other settings and topics.

Acknowledging the complementary nature, and distinctions, between implementation and sustainability is imperative in ensuring the longevity of interventions within complex systems. Sustainability has gained increasing consideration in implementation science. The updated MRC Framework (7) places limited direct emphasis on sustainability, with the exception that the ADAPT guidance (20) is highlighted in the adaptation phase, and implementing and maintaining the intervention at scale is mentioned. Implementation and sustainment are interconnected and need to be planned for; therefore, there is potential for the MRC Framework to further embrace sustainment perspectives. It is noted that sustainment typically places a heavier emphasis on factors within the outer setting, navigating beyond the immediate organisational boundaries and encompassing socioeconomic, cultural, environmental, and strategic contexts (35, 36). These factors encompass elements such as policy alignment, community engagement, resource availability, and the long-term commitment of stakeholders, extending the scope of considerations beyond the implementation phase. Recognising the increased attention on sustainability, alongside existing implementation frameworks, presents an opportunity to build on established knowledge rather than “reinvent the wheel” (1, 2). We deemed the integration of sustainment into the MRC Framework was sufficient and avoided introducing another framework and its language.

Our systems-based perspective reflected the requirement to understand how the intervention/innovation could fit with a system/s, with principles of co-production through a clinical-academic partnership paramount. What contributes to co-production is difficult to define. We have given some examples of co-production activities but acknowledge that not all activities are tangible. Co-production and implementation are more processes than a list of activities, hindering replication. Work by Metz et al. and others has highlighted the importance of relational aspects in implementation (37–39). Arguably the “who” might be equally as important as the “what.” In our advanced MRC framework (Figure 2), we have reflected the importance of stakeholder engagement as an overarching principle when considering the remaining core elements.

This example demonstrates progression through adaptation, implementation, evaluation, and sustainment phases, and the clinical-academic partnership was a key mechanism in our successful evolution. Specifically, it was significant that the clinical-academic project team was embedded in the NHS, and they were able to leverage internal knowledge and take opportunistic approaches to inform key decision-makers. When the decision to scale up the intervention to a regional level was made, stakeholders were broadened and arguably influenced by pre-existing relationships and credibility. The time and effort for this relational engagement is significant, though it is often not described or resourced with the advantages taken for granted, which has implications for generalisability. With research co-production a neglected pathway to impact (39), there is a key role for clinical-academic partnerships in driving the implementation and evaluation of complex interventions which warrants further exploration and recognition.

We needed to involve the right people with insight and authority, supplied with the right knowledge, to enable decisions on sustainment. The dynamic nature of real-world implementation meant that a sustainability outcome was not always predictable, and this uncertainty necessitated continual flexibility in our approach. The concept of value proposition was included as a new addition to this advanced MRC Framework within the sustainment phase. This was provided by the components of an evaluation summary and by the resource and strategy overview tailored to this phase to enable an informed discussion. Further, deciding when and how to move on to the next phases within the MRC Framework, especially given phase overlap, can be hard to determine. The MRC Framework acknowledges that there are often trade-offs between precise, narrow-focused answers and broader, complex inquiries and “that at any phase key core elements should be considered to guide decisions as to whether or not the research should proceed to the next phase, return to a previous phase, repeat a phase or be aborted” (7). This is described most clearly in the feasibility phase, with progression criteria now generally a given within feasibility protocols. Our example demonstrated an addition with clear major decision points, a crucial juncture where decisions needed to be made about whether to continue, modify, expand, or stop the intervention based on evaluations and available information.

Limitations

While our utilisation of the MRC Framework provided a structured architecture to guide our implementation efforts, it is important to acknowledge the limitations and challenges encountered during its application. We applied the MRC core elements to varying degrees; for example, we had a limited focus on health economics. Minimal resources were available for this, and ongoing work has secured funding to undertake further economic analysis. There is a dependency here, in that the resource for the health economic analysis was not available until the acceptability and value of the intervention had been demonstrated, but this is being increasingly focused on by decision-makers and may become crucial for decision points going forward. Our use of the MRC Framework involved modification. We merged, tweaked, and sometimes even diverged from the reported phases to suit the needs of our context and stakeholders, as implementation evidence would guide us to do. This flexibility, while necessary for practical application, introduced variability in adherence to the framework, though arguably it was used flexibly as intended. We included implementation conceptual knowledge within our advanced MRC Framework, such as the CFIR, ERIC, and Proctor's outcomes, but other implementation knowledge could equally be used, for example, a framework to describe context, such as the Implementation in Context (ICON) Framework (40). Similarly, other evaluation frameworks, such as RE-AIM or PRECEDE-PROCEED, or broader frameworks, such as the Exploration, Preparation, Implementation, Sustainment (EPIS) Framework, could be used (16, 41–44). We invite others to try applying the MRC Framework in their implementation efforts and provide feedback on their experiences, and we will continue to refine our work for other regions and clinical areas as it progresses.

Conclusion

In conclusion, our study contributes to the implementation science literature by demonstrating a novel integration of implementation knowledge with the MRC Framework, providing an example of the original and revelatory use of these related but separate evidence bases and answering the call to advance and evolve existing frameworks rather than duplicating existing evidence (1, 2, 7, 22). By addressing key challenges collectively within a clinical-academic partnership, we provide insights that can inform future research and practice in the field of implementation science and practice. Our experience underscores the importance of balancing theoretical frameworks with practical considerations to ensure relevance and effectiveness in implementation science endeavours. This application example will be directly relevant to the field of rehabilitation and build transferable knowledge to enrich implementation research and practice.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Author contributions

LC: Conceptualization, Data curation, Formal analysis, Methodology, Project administration, Resources, Visualization, Writing – original draft, Writing – review & editing. SA: Conceptualization, Data curation, Formal analysis, Methodology, Project administration, Visualization, Writing – original draft, Writing – review & editing. JR-M: Conceptualization, Validation, Writing – original draft, Writing – review & editing.

Funding

The author(s) declare that financial support was received for the research and/or publication of this article. NROL implementation and evaluation was supported by a generous donation from the charity SameYou (Charity number 1170102) and NHS England Stroke Quality Improvement for Rehabilitation (SQuIRe) Catalyst funding. The funders had no role in writing the manuscript.

Acknowledgments

We convey our appreciation to the individuals (staff, patients, and carers) involved in the stroke and neurorehabilitation services that contribute to NROL from East Lancashire Hospitals NHS Trust, University Hospitals of Morecambe Bay NHS Foundation Trust, Blackpool Teaching Hospitals NHS Foundation Trust, and Lancashire and South Cumbria NHS Foundation Trust. We extend our gratitude to SameYou for their generous support helping make NROL possible. Further, we acknowledge the University of Central Lancashire, a higher education institute involved in NROL within the iterations described.

Conflict of interest

JR-M is a co-author of the MRC Framework for developing and evaluating complex interventions and an Editorial Board member of Implementation Science.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Generative AI statement

The author(s) declare that no Generative AI was used in the creation of this manuscript.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frhs.2025.1562627/full#supplementary-material

Abbreviations

CFIR, Consolidated Framework for Implementation Research; ERIC, Expert Recommendations for Implementing Change; MRC, Medical Research Council; NHS, National Health Service; NROL, NeuroRehabilitation OnLine; SOP, standard operating procedure; SSNAP, Sentinel Stroke National Audit Programme; TIDieR, Template for Intervention Description and Replication.

References

1. Kislov R, Pope C, Martin GP, Wilson PM. Harnessing the power of theorising in implementation science. Implement Sci. (2019) 14(1):103. doi: 10.1186/s13012-019-0957-4

PubMed Abstract | Crossref Full Text | Google Scholar

2. Wilson P, Kislov R. Implementation Science. Cambridge: Cambridge University Press (2022).

Google Scholar

3. Jensen TM, Metz AJ, Disbennett ME, Farley AB. Developing a practice-driven research agenda in implementation science: perspectives from experienced implementation support practitioners. Implement Res Pract. (2023) 4:26334895231199063. doi: 10.1177/26334895231199063

PubMed Abstract | Crossref Full Text | Google Scholar

4. Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, et al. Framework for design and evaluation of complex interventions to improve health. Br Med J. (2000) 321(7262):694–6. doi: 10.1136/bmj.321.7262.694

PubMed Abstract | Crossref Full Text | Google Scholar

5. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M, et al. Developing and evaluating complex interventions: the new medical research council guidance. Br Med J. (2008) 337:a1655. doi: 10.1136/bmj.a1655

PubMed Abstract | Crossref Full Text | Google Scholar

6. Skivington K, Matthews L, Simpson SA, Craig P, Baird J, Blazeby JM, et al. A new framework for developing and evaluating complex interventions: update of medical research council guidance. Br Med J. (2021) 374:n2061. doi: 10.1136/bmj.n2061

PubMed Abstract | Crossref Full Text | Google Scholar

7. Skivington K, Matthews L, Simpson SA, Craig P, Baird J, Blazeby JM, et al. Framework for the development and evaluation of complex interventions: gap analysis, workshop and consultation-informed update. Health Technol Assess. (2021) 25(57):1–132. doi: 10.3310/hta25570

PubMed Abstract | Crossref Full Text | Google Scholar

8. Google Scholar. Available at: https://scholar.google.com/ (Accessed March 19, 2025).

Google Scholar

9. National Institute for Health and Care Research. HEE/NIHR Advanced Clinical and Practitioner Academic Fellowship Round 4 Guidance Notes 2023. Available at: https://www.nihr.ac.uk/documents/heenihr-advanced-clinical-and-practitioner-academic-fellowship-round-4-guidance-notes/34339#mrcnihr-complex-intervention-guidance

Google Scholar

10. Skivington K, Craig N, Craig P, Rycroft-Malone J, Matthews L, Simpson SA, et al. Introducing the revised framework for developing and evaluating complex interventions: a challenge and a resource for nursing research. Int J Nurs Stud. (2024) 154:104748. doi: 10.1016/j.ijnurstu.2024.104748

PubMed Abstract | Crossref Full Text | Google Scholar

11. Dichter MN, Muller M, Mohler R, Balzer K, Richards DA. Misinterpreting the MRC framework on complex interventions. Int J Nurs Stud. (2023) 141:104478. doi: 10.1016/j.ijnurstu.2023.104478

PubMed Abstract | Crossref Full Text | Google Scholar

12. UK Research and Innovation. New Framework on Complex Interventions to Improve Health 2021. Available at: https://www.ukri.org/news/new-framework-on-complex-interventions-to-improve-health/

Google Scholar

13. Smith JD, Li DH, Rafferty MR. The implementation research logic model: a method for planning, executing, reporting, and synthesizing implementation projects. Implement Sci. (2020) 15(1):84. doi: 10.1186/s13012-020-01041-8

PubMed Abstract | Crossref Full Text | Google Scholar

14. Czosnek L, Zopf EM, Cormie P, Rosenbaum S, Richards J, Rankin NM. Developing an implementation research logic model: using a multiple case study design to establish a worked exemplar. Implement Sci Commun. (2022) 3(1):90. doi: 10.1186/s43058-022-00337-8

PubMed Abstract | Crossref Full Text | Google Scholar

15. Hunter SC, Morgillo S, Kim B, Bergström A, Ehrenberg A, Eldh AC, et al. Combined use of the integrated-promoting action on research implementation in health services (i-PARIHS) framework with other implementation frameworks: a systematic review. Implement Sci Commun. (2025) 6:25. doi: 10.1186/s43058-025-00704-1

PubMed Abstract | Crossref Full Text | Google Scholar

16. Moullin JC, Dickson KS, Stadnick NA, Rabin B, Aarons GA. Systematic review of the exploration, preparation, implementation, sustainment (EPIS) framework. Implement Sci. (2019) 14:1–16. doi: 10.1186/s13012-018-0842-6

PubMed Abstract | Crossref Full Text | Google Scholar

17. Royal College of Physicians. National Clinical Guideline for Stroke for the United Kingdom and Ireland. 2023. Available at: https://www.strokeguideline.org/app/uploads/2023/04/National-Clinical-Guideline-for-Stroke-2023.pdf

Google Scholar

18. Ackerley S, Wilson N, Boland P, Peel R, Connell L. Neurorehabilitation OnLine: description of a regional multidisciplinary group telerehabilitation innovation for stroke and neurological conditions using the template for intervention description and replication checklist. Digit Health. (2024) 10:20552076241252263. doi: 10.1177/20552076241252263

PubMed Abstract | Crossref Full Text | Google Scholar

19. Levack WM, Gross DP, Martin RA, Every-Palmer S, Kiekens C, Cordani C, et al. Designing studies and reviews to produce informative, trustworthy evidence about complex interventions in rehabilitation: a narrative review and commentary. Eur J Phys Rehabil Med. (2024) 60(6):1088. doi: 10.23736/S1973-9087.24.08459-4

PubMed Abstract | Crossref Full Text | Google Scholar

20. Moore G, Campbell M, Copeland L, Craig P, Movsisyan A, Hoddinott P, et al. Adapting interventions to new contexts-the ADAPT guidance. Br Med J. (2021) 374:n1679. doi: 10.1136/bmj.n1679

PubMed Abstract | Crossref Full Text | Google Scholar

21. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. (2009) 4:50. doi: 10.1186/1748-5908-4-50

PubMed Abstract | Crossref Full Text | Google Scholar

22. Damschroder LJ, Reardon CM, Widerquist MAO, Lowery J. The updated consolidated framework for implementation research based on user feedback. Implement Sci. (2022) 17(1):75. doi: 10.1186/s13012-022-01245-0

PubMed Abstract | Crossref Full Text | Google Scholar

23. Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, Chinman MJ, Smith JL, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the expert recommendations for implementing change (ERIC) study. Implement Sci. (2015) 10:109. doi: 10.1186/s13012-015-0295-0

PubMed Abstract | Crossref Full Text | Google Scholar

24. Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the expert recommendations for implementing change (ERIC) project. Implement Sci. (2015) 10:21. doi: 10.1186/s13012-015-0209-1

PubMed Abstract | Crossref Full Text | Google Scholar

25. CFIR Research Team. Consolidated Framework for Implementation Research 2023. Available at: https://cfirguide.org/ (Accessed December 05, 2024).

Google Scholar

26. Waltz TJ, Powell BJ, Fernandez ME, Abadie B, Damschroder LJ. Choosing implementation strategies to address contextual barriers: diversity in recommendations and future directions. Implement Sci. (2019) 14(1):42. doi: 10.1186/s13012-019-0892-4

PubMed Abstract | Crossref Full Text | Google Scholar

27. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. (2011) 38(2):65–76. doi: 10.1007/s10488-010-0319-7

PubMed Abstract | Crossref Full Text | Google Scholar

28. Beare B, Doogan CE, Douglass-Kirk P, Leff AP, Ward N. Neuro-Rehabilitation OnLine (N-ROL): description and evaluation of a group-based telerehabilitation programme for acquired brain injury. J Neurol Neurosurg Psychiatry. (2021) 92(12):1354–5. doi: 10.1136/jnnp-2021-326809

PubMed Abstract | Crossref Full Text | Google Scholar

29. Ackerley S, Wilson N, Boland P, Read J, Connell L. Implementation of neurological group-based telerehabilitation within existing healthcare during the COVID-19 pandemic: a mixed methods evaluation. BMC Health Serv Res. (2023) 23(1):671. doi: 10.1186/s12913-023-09635-w

PubMed Abstract | Crossref Full Text | Google Scholar

30. Ackerley S, Mason T, Partington A, Peel R, Vernon H, Connell LA. Can We Enhance Neurorehabilitation Through Regional Implementation of Group-Based Telerehabilitation? A Mixed Methods Evaluation of NeuroRehabilitation OnLine (NROL). medRxiv. 2025:2025.03.10.25323664.

Google Scholar

31. Hickey G, Brearley S, Coldham T, Denegri S, Green G, Staniszewska S, et al. Guidance on Co-Producing a Research Project. 2018. Southampton: INVOLVE.

Google Scholar

32. Hoekstra F, Mrklas K, Khan M, McKay R, Vis-Dunbar M, Sibley K, et al. A review of reviews on principles, strategies, outcomes and impacts of research partnerships approaches: a first step in synthesising the research partnership literature. Health Res Policy Syst. (2020) 18:1–23. doi: 10.1186/s12961-020-0544-9

PubMed Abstract | Crossref Full Text | Google Scholar

33. Urquhart R, Kendell C, Cornelissen E, Madden LL, Powell BJ, Kissmann G, et al. Defining sustainability in practice: views from implementing real-world innovations in health care. BMC Health Serv Res. (2020) 20(1):87. doi: 10.1186/s12913-020-4933-0

PubMed Abstract | Crossref Full Text | Google Scholar

34. Birken SA, Haines ER, Hwang S, Chambers DA, Bunger AC, Nilsen P. Advancing understanding and identifying strategies for sustaining evidence-based practices: a review of reviews. Implement Sci. (2020) 15:1–13. doi: 10.1186/s13012-020-01040-9

PubMed Abstract | Crossref Full Text | Google Scholar

35. Palinkas LA, Spear SE, Mendon SJ, Villamar J, Reynolds C, Green CD, et al. Conceptualizing and measuring sustainability of prevention programs, policies, and practices. Transl Behav Med. (2020) 10(1):136–45. doi: 10.1093/tbm/ibz170

PubMed Abstract | Crossref Full Text | Google Scholar

36. Zurynski Y, Ludlow K, Testa L, Augustsson H, Herkes-Deane J, Hutchinson K, et al. Built to last? Barriers and facilitators of healthcare program sustainability: a systematic integrative review. Implement Sci. (2023) 18(1):62. doi: 10.1186/s13012-023-01315-x

PubMed Abstract | Crossref Full Text | Google Scholar

37. Metz A, Jensen T, Farley A, Boaz A, Bartley L, Villodas M. Building trusting relationships to support implementation: a proposed theoretical model. Front Health Serv. (2022) 2:894599. doi: 10.3389/frhs.2022.894599

PubMed Abstract | Crossref Full Text | Google Scholar

38. Boaz A, Baeza J, Fraser A, Persson E. It depends’: what 86 systematic reviews tell us about what strategies to use to support the use of research in clinical practice. Implement Sci. (2024) 19(1):15. doi: 10.1186/s13012-024-01337-z

PubMed Abstract | Crossref Full Text | Google Scholar

39. Rycroft-Malone J, Graham ID, Kothari A, McCutcheon C. Research coproduction: an underused pathway to impact. Int J Health Policy Manag. (2024) 13:8461. doi: 10.34172/ijhpm.2024.8461

PubMed Abstract | Crossref Full Text | Google Scholar

40. Squires JE, Graham ID, Santos WJ, Hutchinson AM. The implementation in context (ICON) framework: a meta-framework of context domains, attributes and features in healthcare. Health Res Policy Syst. (2023) 21(1):81. doi: 10.1186/s12961-023-01028-z

PubMed Abstract | Crossref Full Text | Google Scholar

41. Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health. (2019) 7:64. doi: 10.3389/fpubh.2019.00064

PubMed Abstract | Crossref Full Text | Google Scholar

42. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. (1999) 89(9):1322–7. doi: 10.2105/AJPH.89.9.1322

PubMed Abstract | Crossref Full Text | Google Scholar

43. Porter CM. Revisiting precede–proceed: a leading model for ecological and ethical health promotion. Health Educ J. (2016) 75(6):753–64. doi: 10.1177/0017896915619645

Crossref Full Text | Google Scholar

44. Green LW, Kreuter MW. Health Program Planning: An Educational and Ecological Approach. New York: McGraw-Hill (2005).

Google Scholar

45. Cramer SC, Dodakian L, Le V, See J, Augsburger R, McKenzie A, et al. Efficacy of home-based telerehabilitation vs in-clinic therapy for adults after stroke. JAMA Neurol. (2019) 76(9):1079–87. doi: 10.1001/jamaneurol.2019.1604

PubMed Abstract | Crossref Full Text | Google Scholar

46. Khan F, Amatya B, Kesselring J, Galea M. Telerehabilitation for persons with multiple sclerosis. Cochrane Database Syst Rev. (2015) 2015(4):CD010508. doi: 10.1002/14651858.CD010508.pub2

PubMed Abstract | Crossref Full Text | Google Scholar

47. Laver KE, Adey-Wakeling Z, Crotty M, Lannin NA, George S, Sherrington C. Telerehabilitation services for stroke. Cochrane Database Syst Rev. (2020) 1(1):CD010255. doi: 10.1002/14651858.CD010255.pub3

PubMed Abstract | Crossref Full Text | Google Scholar

48. East Lancashire Hospitals NHS Trust. NROL Video resources. Available at: https://www.youtube.com/playlist?list=PLDmWlZwMvNkTIlNXmTMEIv2ADO7QOIlH4 (Accessed March 19, 2025).

Google Scholar

49. SameYou. NROL SameYou. Available at: https://www.sameyou.org/nrol (Accessed March 19, 2025).

Google Scholar