Using a “Kickoff” to build implementation partner teams and action plans for active implementation of a quality improvement project

Abstract

Introduction:

A strategy for transitioning implementation successfully from pre-implementation to active implementation is to hold a team “kickoff.” The objectives of this manuscript are: (1) to present the frameworks that guided the development of the Protocol-guided Rapid Evaluation of Veterans Experiencing New Transient neurological symptoms (PREVENT) kickoff strategy, (2) describe design elements of the kickoff and how they contribute to achieving PREVENT kickoff aims; forming teams, developing an action plan, and launching active implementation (3) examine the perceived usefulness of those kickoff design elements toward achieving kickoff aims.

Methods:

PREVENT was a stepped-wedge trial to improve the quality of Transient Ischemic Attack (TIA) care at six Veterans Affairs (VA) medical centers. The PREVENT kickoff was designed from multiple frameworks: theory of change principles for process improvement; Consolidated Framework for Implementation Research (CFIR); social learning models; and systems redesign. Data collected included pre-kickoff planning documents and post-kickoff debriefs from the PREVENT national team, Audience Response System (ARS) data, post-kickoff site participant evaluations and semi-structured interviews.

Results:

Site team participants reflected positively on the framework driven, interactive and interpersonal design elements, team building, and action plan exercises, and found them useful for a successful project launch. In-person and hybrid set-up of the kickoff, interactive elements, and team formation activities emphasized the quality problem, and motivated site implementation providers to plan for stroke/TIA care improvement.

Conclusions:

Implementation team kickoffs during pre-implementation are a useful approach to inform and engage local clinical teams and to form plans for practice changes to improve clinical care.

Clinical Trial Registration:

clinicaltrials.gov, identifier NCT02769338.

Introduction

Evidence-based programs typically move from pre-implementation to active implementation when key implementers are oriented, trained, and equipped to begin implementation. One strategy for launching implementation is to convene a team “kickoff.” (1–4) Kickoffs, or similar events which launch projects, appear across several fields including: implementation research (1, 5), community based or multi-stakeholder research (6), project management (7), and systems engineering [e.g., Rapid Process Improvement Workshops (RPIW), Lean Six Sigma, Systems Redesign] (1, 8, 9). Kickoff events commonly include (1): interactive design elements (1, 6), introduction and orientation to relevant content (1, 6), tools and products to be used during active implementation (1, 6), team-building activities (1, 6), and strategies for planning or charter creation (1, 6, 7). However, few empirical studies have examined how kickoffs are created and tailored, especially for health care settings, and few offer any perspective of the teams' clinical providers and staff on its usefulness for planning quality improvement (5).

Our team had previously used kickoff meetings using Systems Redesign elements of collaborative learning where concurrent teams across sites kickoff together to bridge pre-implementation to implementation (3). However, we noticed gaps in team development and subsequent practice changes using this approach. Therefore, we designed a team kickoff building upon our prior work and using theoretical frameworks to promote teamwork and practice changes, while tailoring the delivery to each team based on baseline team development, organization, relationships and experiences.

This manuscript focuses on the development and evaluation of the PREVENT kickoff strategy (1). The specific aims are: (1) to present the development of the PREVENT kickoff design guided by frameworks, (2) describe the kickoff strategy design elements, and (3) evaluate the usefulness of those kickoff elements in forming site teams, informing the development of site action plans, and moving those teams into active implementation with the “buy-in” to implement PREVENT.

Materials and methods

The PREVENT study

PREVENT was a stepped wedge trial with three waves; two sites per wave (n = 6 facilities overall); each facility had a one-year active implementation period that was started by a kickoff (1). PREVENT was designed to facilitate clinical teams' evidence-based practice to improve the quality of care for patients with transient ischemic attacks (TIA) at Veteran Affairs (VA) facilities across the U.S (2). PREVENT focused on seven processes of TIA and stroke care that have been associated with improved patient outcomes. The PREVENT intervention included five components: a quality of care reporting system, clinical programs, professional education, electronic health record tools, and quality improvement support (1, 3). The PREVENT implementation timeline included four main elements: pre-implementation phases with facility baseline interviews, and early facilitation between national QI implementation team and prospective site partner participants; a site kickoff event that gathered site participants to form a local implementation team, develop an action plan, and launch active implementation; the 1-year active implementation phase involving facilitation, learning collaboratives, and access to data reporting and other resources; and a 6-month sustainment period with reduced facilitation and collaborative opportunities.

The PREVENT national QI implementation team (national team) was multi-disciplinary with members from general internal medicine, implementation and social sciences, nursing, quality management, and data science backgrounds. Their role was to evaluate and facilitate with site partner participants to implement PREVENT and improving seven processes of care via site visits, kickoffs, regular check-ins, and collaborative calls throughout the PREVENT evaluation (1). Based on prior stroke-related QI projects (3), the national team recognized the need for an implementation strategy that would effectively form local site teams delivering TIA and stroke care, equipped with the knowledge and action plan to launch active implementation at their site.

A full-day, in-person team kickoff meeting was conducted at each of the six PREVENT sites to gather participants from diverse clinical settings and transition sites from pre-implementation to active implementation. The local site Champion (the leader for site implementation of PREVENT) identified site participants from across disciplines and services (e.g., neurology, pharmacy, emergency medicine, radiologists, hospitalists, and other primary care staff) who were invited to the kickoff. Site partner participants (site team) of PREVENT were responsible for attending the kickoff, developing their sites action plan for implementing PREVENT, participating in collaborative calls, and reflecting on facility data about seven processes of TIA care. The PREVENT study was approved by the Indiana University institutional review board and the Richard L. Roudebush Department of Veterans Affairs (VA) research and development committee (protocol #1511914238).

Conceptual frameworks guiding kickoff design

Four conceptual frameworks informed the PREVENT kickoff design seeking to enhance team formation and activate implementation: Theory of Change (10–12), Social Learning Model (13), Systems Redesign (SR) (9, 14), and Consolidated Framework for Implementation Research (CFIR) (1, 15). The frameworks were operationalized via agenda items and kickoff design elements (features that were incorporated into the kickoff design but are not necessarily agenda items).

The main aims of the kickoff were to launch active implementation, facilitate a diverse group of clinical providers to form a team to engage in other collaborative PREVENT implementation strategies (4, 16), and develop a site partner-determined action plan to improve TIA care at their facility. Table 1 lists kickoff agenda items and design elements along with their associated conceptual framework that item operationalized and intended application. Figure 1 features a site kickoff agenda overlayed with operationalized frameworks.

Figure 1

Theory of change principles of process improvement

Theory of change identifies and utilizes mechanisms to facilitate change in a system or setting through explicit declaration of an end goal and process (4). Theory of change principles relevant to process change, implementation, and PREVENT's program needs included understanding context and preconditions, providing measures and outcomes, capacity building of team members, alliance strengthening among all parties, and convincing change agents to facilitate implementation (17). The kickoff itself was the initial operationalization of mechanisms of change in the local site processes and system to implement and sustain improvements for TIA and stroke care. Theory of change is applied with almost every agenda item from interactive team activities and planning to strengthen alliances, to conducting in-person meetings and opportunities to review and reflect on facility data to convince change agents (See Table 1).

Social learning model

The PREVENT team adopted a social learning model approach to education (13, 14) which emphasizes observing and modeling; where new information is presented with opportunities to practice and apply information being modeled for observers. The social learning model provided an approach for engaging site staff to build their own plan for implementing PREVENT. Specifically, team examples from previous work were presented to socially model team formation and practices changes including a video from a successful VA stroke team discussing how they work together to improve stroke care processes (See Table 1).

Systems redesign

Systems Redesign (SR) focuses on making systematic changes to improve patient care from an effectiveness, efficiency, and quality standpoint (9, 14, 18). SR is a widespread process improvement program within VHA, and the implementation team had used SR approaches successfully in other QI projects (19, 20). PREVENT sought to improve processes of TIA care within the individual facility's systems of stroke care. Although SR projects include multi-day events, we selected applicable SR processes and activities that could be included in a single-day kickoff. The specific SR activities were selected based on experience with their use from other projects, pilot-testing, and alignment with kickoff and PREVENT program aims (18). SR activities such as identifying barriers to ideal processes, brainstorming solutions, and mapping solutions onto an impact-effort matrix were used in the kickoff to guide site team members to collaboratively develop their site-specific action plan (See Table 1) (7, 9, 21).

Consolidated framework for implementation research (CFIR)

The Consolidated Framework for Implementation Research (CFIR) provides typology for implementation mechanisms within a health services setting; constructs and mechanisms commonly used in successful implementation (22). We used CFIR both to guide implementation development and as an evaluating framework for all of PREVENT implementation. Results from previous stroke teams practice changes and PREVENT baseline interview data directed the national team to focus on operationalizing four key CFIR constructs associated with successful implementation: Planning, Goals and Feedback, Reflecting and Evaluating, and Supporting Champions (1, 23). During the PREVENT kickoff each key construct along with other identifiable constructs (e.g., mission alignment, relative priority, available resources, and others) were demonstrated with applied examples, for instance reviewing facility data operationalizes Reflecting and Evaluating and creating the action plan operationalizes Planning (see Table 1).

Kickoff agenda

All design elements and agenda items were used to meet the objectives of the kickoff. To convey importance of PREVENT, the PREVENT national team conducted the kickoff in-person with some remote staff using videoconferencing in a single full working day. Due to the constraints on clinicians schedules the kickoff was limited to a single day, and time had to be used wisely to get necessary work done to have the site team complete an action plan.

The kickoffs introduced the PREVENT protocol and major components for improving TIA care (1). Site team members were guided through an interactive electronic data dashboard and resource site that site teams could use to monitor their performance data, implementation progress, and access resources. After, site staff and providers were presented with the rationale for focusing on the seven TIA processes of care, external facilitation methods, tools and resources, and quality metrics to measure TIA care. The site team then used SR activities (e.g., identifying barriers to ideal processes, brainstorming solutions, and mapping solutions onto an impact-effort matrix) to discuss local barriers, identify actionable tasks and needs to improve TIA care at their site and incorporate into their action plan. In line with social learning model, SR strategies allowed site staff and providers to engage in the learning process, and moving from learning and observing to planning and goal setting (also a CFIR construct) and preparing for implementing PREVENT with the added layer of simultaneously forming at team (see Table 1; Figure 1).

Site staff and providers comforts and conveniences

To make for an experientially positive kickoff, comforts and conveniences for the site staff and providers, and activities to encourage site staff and providers' engagement were included in the kickoff design. For example, the kickoffs were conducted in-person at each VA facility at a time that was most convenient for personnel at that site. Being in person helped convey the importance of PREVENT, but also to build an interpersonal connection between the national team and the site team. We also explicitly recognized that the facility providers and staff were the local experts most equipped to identify the barriers, and with the most knowledge to implement PREVENT at their site. The national team attempted to avoid a hierarchical approach to the site team to present and demonstrate where needed, but mostly allow the site team to direct preparing for active implementation.

Interpersonal communication conveyed the national team's commitment to engage site teams as full partners. Engagement with site teams required substantial investments in: communication with site staff and providers, planning and coordination between the national team and the site facility, and logistics. Site leadership (i.e., chief-of-staff) were encouraged to attend the kickoff to convey the importance of implementing the PREVENT program and facilitate “buy-in” among site team members. Certificates and tokens of appreciation were given to the kickoff partners to further mark the significance of the kickoff, emphasize a partnership to implement PREVENT, and deputize the newly formed team to implement PREVENT.

Evaluation

Data collection

To evaluate the PREVENT kickoffs, we collected data from planning and debriefing documents from the national team and data collected from 51 site team kickoff participants from across the six sites. There were five main sources of data: (1) Pre-kickoff planning materials, (2) post-kickoff debriefing sessions, (3) during kickoff participant feedback from the Audience Response System, (4) post-kickoff evaluations, and (5) semi-structured interviews collected during active implementation.

Pre-kickoff planning materials consisted of memos and meeting notes from the national team documents outlining ideas, agenda items, site information related to planning, creating, and preparing kickoffs at sites. Post-kickoff debriefing sessions (n = 6) were conducted for each site by the national team, which were recorded and transcribed. These sessions were opportunities for the PREVENT national team to describe what went well, what needed improvement, early impressions of a site team and note participant dynamics, and to discuss next steps for each site.

An all-digital, real-time feedback method, an Audience Response System (ARS), was used during the kickoff to engage staff and providers and elicit their perspectives. ARS questions had multiple choice responses designed to rate kickoff elements. Kickoff participants responded to questions anonymously using remote control “clickers.” Responses from ARS participants were downloaded for analysis.

Kickoff evaluations (n = 32) consisted of five open-ended questions designed for kickoff participants to provide feedback on what they appreciated and gained from the kickoff, and describe next steps for active implementation at their site (see Supplementary Appendix A). The evaluations were administered at the end of the kickoff meeting. Evaluation responses were anonymous and handwritten, but later transposed into an Excel (Microsoft) spreadsheet.

Semi-structured, audio recorded, interviews were conducted in-person with site staff and providers (n = 32) at 6 months after a site's kickoff, during active implementation. Interview questions were asked about the impact of the team kickoffs on local active implementation. Interviews were recorded and transcribed for thematic coding analysis.

Mixed methods analysis

Analysis of the data aimed to: (1) map the frameworks of the kickoffs and PREVENT to design of the kickoff and its agenda items, (2) understand the kickoffs development and usefulness from two separate perspectives; the national team and site team (3) identify if and how the agenda items and incorporated frameworks helped meet the kickoff main objectives, i.e., forming teams and developing action plans.

Kickoff planning materials were analyzed by converting documents into an NVivo file for analysis. Pre-kickoff planning documents went through an open coding process (24) to map overarching, pre-identified frameworks and some emergent sub-frameworks and constructs. Planning material coding was then triangulated with coded post-kickoff debriefs and six-month semi-structured interviews.

Post-kickoff debrief transcripts were open coded in NVivo to identify content related to kickoff planning, adaptations, and theories or frameworks, as well as descriptions of the perceived reception of the kickoff (e.g., relevance for site personnel, critical or positive feedback).

Semi-structured interviews were performed by the national team. Interview questions were focused on kickoff implementation and site staff's experiences. Interviews were transcribed and coded independently by two members of the evaluation/research team merged with partner coders, and consensus coded for agreement on use of the codebook. The codebook was based on implementation domains, principally from CFIR, and project-specific codes related to PREVENT (e.g., kickoff). Additionally, a round of categorical coding was applied to the six-month interviews, where participant signifiers and quantifiable codes, codes such as clinical role, Likert question scores, provider years in practice, etc. were applied to interviews to provide further cross-analysis with other codebooks (24).

ARS responses were tabulated to show the totals on kickoff elements site personnel found most useful across sites (SB, BH, TMD). Post-kickoff evaluation responses were collected and compared across sites and reviewed for common responses about site staff members' perceptions of the kickoff (SB). Perceptions and preferences from evaluations were compared across tables with ARS response tables. SB reviewed and conducted open coding of post-kickoff debrief transcripts for kickoff data related to underlying theories, planning and adaptation elements for kickoffs, and site perceptions of the kickoff. SB analyzed kickoff code reports from 6-month interview transcripts, conducted additional categorical coding and comparing coded 6-month interviews to kickoff coded excerpts from pre-kickoff planning material and post-kickoff debriefs. Transcript coding and analysis of 6-month, post-kickoff debriefs, and pre-kickoff planning documents was performed using NVivo12 software for data coding and analyses (25).

Results

The results show how each respective kickoff design element and framework contributed toward each of the main aims of the kickoffs: developing an action plan, forming collaborative teams and partnerships, and launching active implementation. Kickoff design elements often operationalized more than one framework, each agenda item and design element worked toward achieving multiple aims at a time. Additionally, site team data shows which design elements were more useful, appreciated, or helped generate “buy-in” for participants to begin active implementation.

Participants

There was a total of 51 site team kickoff participants across six sites. There were 32 unique site staff and providers who participated in anonymous post-kickoff evaluations (n = 32) and 6-month interviews (n = 32); an average of 5 (range 3–9) site personnel from each VA site (n = 6) completed evaluations and six-month interviews. ARS responses were aggregated across site attendees; responses to ARS were received from participants who were present and willing to respond when questions were presented during kickoffs.

Planning and action plan

Several activities went into planning for improving the quality of acute TIA care at each local site. Planning was an operationalized CFIR construct, and opportunities to plan implementation of PREVENT at each site were intentionally woven throughout the kickoff agenda (see Table 1; Figure 1). A main objective of the kickoff was the development of a site-specific action plan with inputs from the newly formed multi-disciplinary team. SR activities like diagraming barriers, facilitators, and goals on an impact-effort matrix, and making an action plan were all a part of Planning at the site. The way the kickoffs provided the opportunity to plan as a multi-disciplinary team was appreciated by participating personnel. From ARS responses, site participants were asked about their top preferred kickoff elements, and half of the top responses were related to team planning and developing the action plan; “Team Brainstorming solutions”—18.68%, “Team Developing Plan Do Study Act/Action Plan”—14.02% (see Table 2).

Team forming

The six sites had varied associations and teams among site staff prior to their kickoffs. Some had established and collaborative neurology teams before the PREVENT kickoff. Other sites had staff meeting each other for the first time at the kickoff. The national team commented on debriefs about the positive team dynamics either being evident right at the beginning of kickoffs or being formed throughout the kickoff. Team formation suffered when facility personnel at some sites were unable to attend the kickoff meeting, or when facility staff had to come in and out of the full day meeting to attend to other clinical duties. From ARS data, “Meeting with other staff members from my facility,”—16.15% was one of the top four preferred kickoff elements (see Table 2). In Table 3, site staff and providers expressed that the kickoff was an excellent opportunity for the different site personnel to meet with each other, “[get] everybody on the same page,” and get different perspectives from different providers for planning. In fact, some sites expressed that they wished they had had more relevant site staff and providers who were not present at the kickoff.

“Buy-in” among facility personnel

The team kickoff structure was consistent across sites with adjustments made based upon lessons learned from post-kickoff debriefs and individual site needs. Minor changes were made to improve the flow of the meeting and allocating time for agenda items, engaging specific site personnel, setting up at venues within different VAMCs, and highlighting opportunities for improvement based on baseline quality performance. Interviewed providers stated that kickoffs and design considerations helped build “buy-in” among site personnel to improve the seven TIA processes of care. “Buy-in” is the state in which the national team was able to emphasize the importance of implementing PREVENT and improving the seven TIA processes of care, and the site team were convinced and committed to implementing PREVENT at their site. Being shown how effective PREVENT and improving the seven TIA processes of care could be for improving care for Veterans with stroke or TIA helped kickoff participants, as they stated, “buy into” PREVENT (see Table 3). From interview data, kickoff elements that helped convince skeptical site staff and providers included the in-person meeting, being presented with site data (Reflecting and Evaluating), considerations for site staff and providers comforts and other intangibles, and the physical set up of the kickoff venues (see Table 3). Furthermore, ARS data's top response for preferred kickoff elements was “Reviewing facility performance data on the hub”—31.83%, demonstrating how demonstrating how reflecting and evaluating activities and presenting site's performance data was for participants (See Table 2).

Discussion

The PREVENT national team drew on several frameworks related to process or system change and their own experiences with previous program launch events to develop the PREVENT site kickoff strategy, demonstrating an intentional comprehensive design. Moreover, the PREVENT national team conducted kickoff participant evaluations to actively improve iterations of the kickoff for successive PREVENT sites; and to evaluate the recipients' perceptions of the intentionally designed kickoff. The results demonstrate the effectiveness in the kickoffs: every site developed a facility-specific action plan, and the events promoted team formation and buy-in to implement PREVENT.

Using an intentionally designed kickoff as an implementation planning meeting is aligned with the literature on implementation program launches (7, 26). Within the field of implementation science, several distinct implementation strategies align with kickoff design elements, such as providing education, training on a specific aspect of the evidence based practice, creating new local teams, or having a local setting action plan for moving to active implementation (26). For PREVENT, the kickoffs helped with forming the site teams and set up the teams for later collaborative strategies like collaborative calls and facilitator engagement that contributed to PREVENT's overall success (16). The site action plans from the kickoffs helped some site clinical champions to press their site team to implement all their original goals before active implementation phase ended (27).

The Intervention for Stroke Improvement using Redesign Engineering (INSPIRE) featured similar implementation strategies as PREVENT, primarily with a program initiating “face-to-face collaborative training” for VHA site stroke teams (20). INSPIRE held a multi-day, learning collaborative with external facilitation bringing together six site teams at a centralized location e.g., developing Plan-Do-Study-Act cycles, experiential learning, identifying operational barriers (20). The PREVENT kickoff built upon lessons learned and experiences from INSPIRE including the need to tailor the kickoff to each site team and then build the collaborative from across these individual site teams. Although literature describes the INSPIRE training strategies and elements (19), no literature details the frameworks or rationale for including specific components, nor the perception of participants.

Mendel et al. 2011 described a kickoff utilized as a “community engagement” strategy for a community-partnered participatory research QI program delivering depression interventions, Community Partners in Care (CPIC) (6). The kickoff discussed was a “three-quarter day” and featured modes of engagement appropriate for the community setting to educate and get community members to engage with provided depression interventions (6). Although some participants were affiliated with healthcare settings, the kickoff and CPIC were aimed to serve and meet participants outside of a healthcare system (6). The CPIC study described guiding frameworks for how the kickoff was designed including agenda items, logistics, and evaluation of kickoff participants' perspectives (6). The CPIC team articulated similar concerns as the PREVENT investigators in terms of addressing hierarchical dynamics between facilitators and participants.

Within Veteran Health Administration (VHA), Pittman et al. 2021 had discussed using Rapid Process Improvement Workshops (RPIWs) at multiple VHA sites to implement eScreening (8). The eScreening RPIWs were three-day events to plan and launch implementation for multiple VHA sites concurrently (8). Pittman et al. offered details on their agenda, and involved evaluations that described the perspective of participants on the RPIW, its elements, and impact on implementation (8). The RPIW, while determined to be useful, was also not utilized by all sites.

PREVENT was a QI program operating in a structured healthcare system that took place at VHA facilities across the US. PREVENT modeled the kickoff to align with VHA standards and practices for process improvement, while contending with varying local settings and multiple types of providers. The PREVENT kickoff strategy offered an approach for team implementation within diverse healthcare settings with action plans for successful implementation in each setting.

Limitations

The kickoff was a key implementation strategy for launching active implementation for PREVENT at each participating site during pre-implementation. However, there was no control group; therefore, we are unable to evaluate how effective the kickoffs were in comparison to an alternative approach. The team kickoffs were a component of a larger set of implementation strategies previously reported (1, 2, 16, 21, 27).

Conclusion

The PREVENT kickoff implementation strategy succeeded at launching active implementation at each site and contributed to individual sites' implementation success by bringing together diverse sets of site providers and staff to form collaborative teams with the knowledge and buy-in to implement PREVENT. The kickoffs guided site teams to build site-specific action plans, prepared to begin active implementation.

References

• 1.

  Bravata DM Myers LJ Homoya B Miech EJ Rattray NA Perkins AJ et al The protocol-guided rapid evaluation of veterans experiencing new transient neurological symptoms (PREVENT) quality improvement program: rationale and methods. BMC Neurol. (2019) 19(1):294. 10.1186/s12883-019-1517-x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 2.

  Bravata DM Myers LJ Perkins AJ Zhang Y Miech EJ Rattray NA et al Assessment of the protocol-guided rapid evaluation of veterans experiencing new transient neurological symptoms (PREVENT) program for improving quality of care for transient ischemic attack: a nonrandomized cluster trial. JAMA Netw Open. (2020) 3(9):e2015920. 10.1001/jamanetworkopen.2020.15920

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 3.

  Bravata DM Myers LJ Reeves M Cheng EM Baye F Ofner S et al Processes of care associated with risk of mortality and recurrent stroke among patients with transient ischemic attack and nonsevere ischemic stroke. JAMA Netw Open. (2019) 2(7):e196716. 10.1001/jamanetworkopen.2019.6716

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 4.

  DuBow WM Litzler E . The development and use of a theory of change to align programs and evaluation in a complex, national initiative. Am J Eval. (2019) 40(2):231–48. 10.1177/1098214018778132

  • CrossRef
  • Google Scholar

• 5.

  Goldberg DG Owens-Jasey C Haghighat S Kavalloor S . Implementation strategies for large scale quality improvement initiatives in primary care settings: a qualitative assessment. BMC Prim Care. (2023) 24(1):242. 10.1186/s12875-023-02200-8

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 6.

  Mendel P Ngo VK Dixon E Stockdale S Jones F Chung B et al Partnered evaluation of a community engagement intervention: use of a kickoff conference in a randomized trial for depression care improvement in underserved communities. Ethn Dis. (2011) 21(3 Suppl 1):S1–78–88.

  • Pubmed Abstract
  • Google Scholar

• 7.

  Tereso A Ribeiro P Fernandes G Loureiro I Ferreira M . Project management practices in private organizations. Proj Manag J. (2019) 50(1):6–22. 10.1177/8756972818810966

  • CrossRef
  • Google Scholar

• 8.

  Pittman JOE Rabin B Almklov E Afari N Floto E Rodriguez E et al Adaptation of a quality improvement approach to implement eScreening in VHA healthcare settings: innovative use of the lean six sigma rapid process improvement workshop. Implement Sci Commun. (2021) 2(1):37. 10.1186/s43058-021-00132-x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 9.

  Veterans Healthcare Administration. VA TAMMCS: Improvement Framework Guidebook. Washington, DC: US Department of Veterans Affairs (2011). p. 13–57.

  • Google Scholar

• 10.

  Reinholz DL Andrews TC . Change theory and theory of change: what’s the difference anyway?Int J STEM Educ. (2020) 7(1):1–12. 10.1186/s40594-020-0202-3

  • CrossRef
  • Google Scholar

• 11.

  Breuer E Lee L De Silva M Lund C . Using theory of change to design and evaluate public health interventions: a systematic review. Implement Sci. (2016) 11:1–17. 10.1186/s13012-016-0422-6

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 12.

  Breuer E De Silva MJ Fekadu A Luitel NP Murhar V Nakku J et al Using workshops to develop theories of change in five low and middle income countries: lessons from the programme for improving mental health care (PRIME). Int J Ment Health Syst. (2014) 8:15. 10.1186/1752-4458-8-15

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 13.

  Torre DM Daley BJ Sebastian JL Elnicki DM . Overview of current learning theories for medical educators. Am J Med. (2006) 119(10):903–7. 10.1016/j.amjmed.2006.06.037

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 14.

  Daggett VS Woodward-Hagg H Damush TM Plue L Russell S Allen G et al Health care systems redesign project to improve dysphagia screening. J Nurs Care Qual. (2015) 30(3):226–32. 10.1097/NCQ.0000000000000096

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 15.

  Damush TM Miech EJ Rattray NA Homoya B Penney LS Cheatham A et al Implementation evaluation of a complex intervention to improve timeliness of care for veterans with transient ischemic attack. J Gen Intern Med. (2021) 36(2):322–32. 10.1007/s11606-020-06100-w

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 16.

  Penney LS Homoya BJ Damush TM Rattray NA Miech EJ Myers LJ et al Seeding structures for a community of practice focused on transient ischemic attack (TIA): implementing across disciplines and waves. J Gen Intern Med. (2021) 36:313–21. 10.1007/s11606-020-06135-z

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 17.

  Asada Y Gilmet K Welter C Massuda-Barnett G Kapadia DA Fagen M . Applying theory of change to a structural change initiative: evaluation of model communities in a diverse county. Health Educ Behav. (2019) 46(3):377–87. 10.1177/1090198118818233

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 18.

  Hagg HW Workman-Germann J Flanagan M Suskovich D Schachitti S Corum C et al Implementation of systems redesign: approaches to spread and sustain adoption. In: HenriksenKBattlesJBKeyesMAGradyML, editors. Advances in Patient Safety: New Directions and Alternative Approaches (Vol. 2: Culture and Redesign). Rockville, MD: Agency for Healthcare Research and Quality (US) (2008).

  • Google Scholar

• 19.

  Williams J Leamy M Bird V Le Boutillier C Norton S Pesola F et al Development and evaluation of the INSPIRE measure of staff support for personal recovery. Soc Psychiatry Psychiatr Epidemiol. (2015) 50(5):777–86. 10.1007/s00127-014-0983-0

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 20.

  Williams L Daggett V Slaven JE Yu Z Sager D Myers J et al A cluster-randomised quality improvement study to improve two inpatient stroke quality indicators. BMJ Qual Saf. (2016) 25(4):257–64. 10.1136/bmjqs-2015-004188

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 21.

  Damush TM Penney LS Miech EJ Rattray NA Baird SA Cheatham AJ et al Acceptability of a complex team-based quality improvement intervention for transient ischemic attack: a mixed-methods study. BMC Health Serv Res. (2021) 21(1):453. 10.1186/s12913-021-06318-2

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 22.

  Damschroder LJ Aron DC Keith RE Kirsh SR Alexander JA Lowery JC . Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. (2009) 4:50. 10.1186/1748-5908-4-50

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 23.

  Rattray NA Damush TM Luckhurst C Bauer-Martinez CJ Homoya BJ Miech EJ . Prime movers: advanced practice professionals in the role of stroke coordinator. J Am Assoc Nurse Pract. (2017) 29(7):392–402. 10.1002/2327-6924.12462

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 24.

  Saldaña J . The Coding Manual for Qualitative Researchers. 3rd ed. Thousand Oaks, CA: Sage Publications Ltd (2016).

  • Google Scholar

• 25.

  NVivo Qualitative Data Analysis Software. Version 12 ed. Burlington, MA: QSR International Pty Ltd (2018). Available at: https://www.qsrinternational.com/

  • Google Scholar

• 26.

  Powell BJ Waltz TJ Chinman MJ Damschroder LJ Smith JL Matthieu MM et al A refined compilation of implementation strategies: results from the expert recommendations for implementing change (ERIC) project. Implement Sci. (2015) 10:21. 10.1186/s13012-015-0209-1

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 27.

  Bravata DM Miech EJ Myers LJ Perkins AJ Zhang Y Rattray NA et al The perils of a “my work here is done” perspective: a mixed methods evaluation of sustainment of an evidence-based intervention for transient ischemic attack. BMC Health Serv Res. (2022) 22(1):857. 10.1186/s12913-022-08207-8

  • Pubmed Abstract
  • CrossRef
  • Google Scholar