Understanding implementation research collaborations from a co-creation lens: Recommendations for a path forward

Introduction

Implementation strategies describe actions for promoting the uptake of evidence-based practices (EBPs), programs and policies (1). Implementation strategies often rely on multi-stakeholder collaborations to foster buy-in, inform implementation, and increase sustainment of EBPs (2). The Implementation Science (IS) field is re-assessing and broadening its approach to stakeholder engagement to incorporate the perspectives of a range of contributors, including patients, consumers, health professionals, and policy makers (hereafter called “stakeholders”) to tailor strategies to local contexts. Explicit concerns for involving individuals who experience health disparities, health injustices, and inequitable representation in the planning and implementation of EBPs and policies are increasingly central to such efforts.

Advancing health and social equity is critical to meeting IS goals of promoting action-based pragmatic research and closing the evidence-to-practice gap (3). Equity-centered IS entails naming researchers' assumptions, identifying our differences and systematic accountings of power relationships influencing study designs and decision-making (4). Equity-centered IS requires interaction with broader groups of stakeholders to identify, measure implementation strategies and outcomes, and to have an accurate understanding of how local contexts impact implementation (5–8).

We propose a perspective foregrounded in the concept of co-creation (i.e., synergistic approach to goal attainment) (9, 10), offering five principles to guide structuring implementation collaborations in research. Co-creation emerged in early late 1990s and early 2000s from business management (11) and has gained traction in healthcare and implementation research (12) as it has been heralded as a novel solution to the research to practice gap (13–15). We present three federally-funded implementation research studies as case examples to illustrate the application of these co-creation principles that are informed by the Exploration, Preparation, Implementation, Sustainment (EPIS) Framework (6). EPIS is a framework that guides the examination of determinants of implementation at the inner, outer and bridging levels and through four iterative phases as included in this name (16). Lastly, we offer recommendations for promoting co-creation in IS research.

State of the literature on stakeholder collaboration during implementation

Successful design and deployment of implementation strategies typically require coordinated action among organizations, groups, and individuals. Participatory approaches used by IS researchers include implementation mapping (17–19), user-centered design (19), group prioritization processes (20), community advisory boards and expert panels (21–23). IS researchers have also adopted engagement approaches from community-based participatory research (CBPR) to enhance the active inclusion of all relevant stakeholders in IS work. CBPR necessitates shared leadership and co-learning relationships among researchers and community partners (24). CBPR elucidates the benefits of involving end-users in research planning and implementation of EBPs and other innovations, reducing stakeholder power differentials, and illuminating key factors to address health equity efforts (24).

Current collaborative approaches in IS lack concrete guidance on synergistically integrating all stakeholders' expertise, values, and priorities for the joint, integrated creation of knowledge. IS researchers would benefit from such guidance given variations in stakeholder backgrounds and lived experience, professional roles, access to resources (e.g., fiscal and material capital), and thus privilege and power represented at the table (3). We must better understand collaboration processes in IS that often unfold within complex contexts with stakeholders who may not share the same priorities. We must enhance our knowledge of how implementation collaborations operate in these contexts and ways to optimize them to benefit diverse stakeholders, including end-users who are often not included or meaningfully involved in collaborative processes.

Consequences of “collaborations” lacking co-creation

Stakeholder engagement, governance arrangements, and building capacity for productive and successful co-creation can be challenging (14). Improved understanding of successful approaches for developing researcher and community stakeholder collaborations into co-creation partnerships is critical to achieving meaningful implementation and sustainment outcomes. Guidance on nurturing such collaborations should address power imbalances and support communication and trust among stakeholders (3). In co-creation relationships, all stakeholders ideally participate in and share control throughout all phases of research. This approach contrasts with traditional research dynamics that position researchers as external experts and gatekeepers of information. Such dynamics can perpetuate power differentials and information asymmetries between researchers and community stakeholders (25). Power imbalances may be heightened in research conducted with socially and economically marginalized communities (26). Power imbalances contribute to low acceptability, appropriateness and adoption of new practices, or abandonment of new practices soon after a study concludes (27, 28). Stakeholders lacking meaningful engagement in decision-making and implementation may be left without a clear understanding of how their participation contributes to results (29). Sustainment is challenged when stakeholders identify little value in or from their engagement in an implementation collaborative. Conversely, partnerships built on power-sharing and democratic principles can promote multilevel buy-in, capacity for change, and encourage adoption of new practices (30).

The absence of clear co-creation principles in IS fosters insufficient knowledge about individual actions and responsibilities for achieving implementation goals (31). Role ambiguity engenders confusion about what researchers or community partners are expected to do, curtailing their ability to improve or sustain EBP use (32). When researcher-provided resources recede at the end of the implementation phase, community stakeholders may struggle to organize necessary supports and sustain EBPs (32). Green et al. contend that “without significant changes, the adoption of co-production on its own will not lead to significant changes,” meaning we must become more intentional in overcoming challenges and applying knowledge from stakeholders (33).

These challenges may result in selecting EBPs that fit poorly with community needs, resources and priorities. Fit—the perceived appropriateness of the intervention/implementation strategy and setting—is central to implementation success (16). Researchers and community stakeholders must meaningfully consider their context when selecting an EBP, and be prepared to adapt interventions and strategies to accommodate for changing contextual influences (30, 34). Collaborations that support highly generalizable approaches to intervention design and testing may not fit well with local conditions, and thwart uptake (35).

Proposed solution: Specifying practical and research applications of co-creation

Although IS underscores collaboration's value in implementation research, it lacks guidance for nurturing collaborative efforts, and ensuring they reflect contributions and meaningful participation from all stakeholders. Despite growing awareness that IS must proactively engage with health equity (4), IS researchers struggle with structuring equitable, collaborative processes to support transformative impacts through successful implementation. We draw from organizational research, community-engaged studies, and patient-centered care to argue that the concept of co-creation in implementation collaborations can catalyze contextually relevant insights and approaches to help reach expected outcomes. We next describe co-creation and the application of five co-creation principles through case examples.

Concept of co-creation

Co-creation is the process of convening a diversity of stakeholders who are willing to share their knowledge, skillsets and resources to spark synergy and persevere to an end-result surpassing the sum of its parts (5, 6, 36). The goal is for these partners to contribute to the planning, design, testing, and implementation of the services they fund, deliver, or receive (37, 38). Although there is limited outcomes research on co-creation, current evidence suggest that co-creation leads to stakeholder trust, equitable contributions, and a sense of ownership (39), and in turn to quality research as well as meaningful research which meets individual and community expressed needs and goals (21). This concept is referred to as “co-design” or “co-production,” terms often used inter-changeably in the literature as they focus on jointly producing, designing or creating (e.g., knowledge to be applied, such as an intervention prototype or research design) (38, 40–44). We present co-creation as a multi-dimensional concept for “all things co” that necessitates meaningful engagement among stakeholders (45). This type of engagement requires co-creators (particularly researchers) to grapple with what it means “to open up their processes” to forge effective partnerships with different stakeholders (46). Members of ‘all things co' processes are often specified as stakeholders with relevant and unique expertise and experience to contribute. Graham et al. assert that participatory co-production processes are critical to advancing the science of evaluating stakeholder engagement (47, 48).

Much like CBPR, co-creation research is driven by power-sharing governance arrangements (e.g., partnership agreements) between stakeholders. It is guided by end-users who are experts by experience (49), meaning partners whose lived realities enable them to share knowledge, values, and needs that are often not known or fully appreciated by program developers or researchers. This type of exchange is often characterized as local end user-driven collaboration (50). Stakeholder voices and contributions are engaged at the behavioral, cognitive and/or emotional levels and shaped by the group's motivation(s) for collaborating (51).

Pearce et al. clarify that the co-creation of new knowledge for health interventions must address conceptual ambiguity and the pragmatics of participation by proposing core principles (rigorous research methods and embeddedness) (41). In a special issue of Evidence & Policy on co-creation, Metz [(32), p. 333] assert that the “legitimacy of co-creation approaches is underpinned by explicit core values and assumptions about how affected parties will be involved in the work.” To follow on this recommendation, we draw from a growing body of work (9, 25, 37–39, 52) to assert the following five principles of a co-created collaborative process to enhance implementation efforts:

1. Equity: This principle calls for greater equity in relationship-building among stakeholders, with end-user knowledge and experience being valued equally with that of professionals. By evoking equality, we do not naively assume each stakeholder holds equal power in collaboration. Rather all stakeholders in an implementation pursuit deserve equitable access to shared responsibility, decision-making power, and the resources required for participation. Equitable access recognizes that participation needs may differ across stakeholders based on individual (e.g., culture, preferences, and values), organizational (e.g., professional roles) and contextual characteristics of the implementation environment (e.g., resource-rich vs. underserved). Facilitators of co-creation group processes are tasked with promoting a more active role among implementers and end users in research (50). Equity is supported through access to information, networks and resources, transparency, and value alignment (9). Equity in relationships during co-creation engagement promotes trust and lead to meaningful engagement among non-academic partners and to higher engagement in the research process (53, 54).

Application in IS: This principle is applied by convening collaborations with multi-disciplinary academic researchers, implementers (e.g., service providers), end-users (e.g., patients, clients), and other relevant stakeholders (e.g., community leaders, policy makers) based on the nature of the effort (25). It is also reflected in re-designed governance structures before, during and after the implementation process and in stakeholders, especially researchers, striving to become more self-aware of implicit bias possibly affecting attitudes, interactions, and fundamentals (55).

2. Reflexivity: This principle acknowledges that researchers (and other co-creation partners) strive to be aware of and analyze how their positions within collaborative research processes may influence its dynamics, including how stakeholders interact with one another and engage in implementation (5, 56). Reflexivity is seen as critical to situating positionality and power within the collaborative, likely reducing stigma and promoting respect for all perspectives and values (57). This principle also supports sustainability and long-term goal setting as well as growth of partners' networks over time (57, 58).

Application in IS: Reflexivity requires making time and space for ongoing group reflections to identify and redress power imbalances and processes for sharing information and making decisions, and to recognize limitations of using dominant frameworks, including unintended consequences of well-intended research for diverse implementers and end-users, and social dynamics shaping our collaborations (57, 59, 60).

3. Reciprocity & Mutuality: This principle concerns the degree to which stakeholders are open and interested in learning from each other, referred as the “knowledge appetite” (50). Relationships are perceived and experienced as mutually beneficial through the combined and generative knowledge and the deepened connections and networks developed among all partners (50). This reciprocity leads to perceived stakeholder ownership of the collaboration process. This value can foster accountability, co-learning and learning transfer in a bi-directional fashion between researchers and other partners (32, 53).

Application in IS collaborations: Reciprocity and mutuality is promoted through the inclusion of stakeholders in power-sharing governance arrangements (36), and by researchers communicating the evidence base for potential implementation strategies to inform decision-making (25). Reciprocity can be achieved when co-creation stakeholders collectively create products useful to all partners including community-facing materials (e.g., toolkits, brochures) beyond research manuscripts.

4. Transformative & Personalized: This principle refers to a collaborative process that benefits the study while also offering an individual experience that is enriching, given the emphasis on end-user orientation through use-value and empathy (9). Research is perceived as having room for new possibilities because of the collaborative process. When this principle is met, it is easier to obtain buy-in and support from implementers (50). This principle can foster activation and self-advocacy among patients, families, and community members as a result of their works side-by-side with researchers (59). In addition, promoting an understanding of each partner's motivations for joining a collaborative, and opening the space for them to take on or lead roles that align with those motivations can become a transformative and meaningful participation for individuals (61).

Application in IS collaborations: This principle necessitates increasing knowledge and skills among non-academic stakeholders to relevant theory and research methods (25), and by organizing knowledge and skill-building activities during the Preparation phase, such as IS training boot camps. Training may also need to focus on increasing contextual knowledge and engagement skills for co-creation initiatives among researchers. Stakeholders are empowered to develop their own solutions (53) by participation in identifying and selecting interventions/implementation strategies and desired outcomes. These efforts create not only research value but also individual and community value.

5. Relationships Facilitated: Relationship structures (e.g., partnerships) or procedures (e.g., agreements on roles/responsibilities) are developed collectively to support a co-creation implementation collaborative. Participation is encouraged and facilitated through organizations and social networks and by creating explicit spaces and time for partners to network, invite their own networks to contribute to the implementation process at key phases of the project, and by formalizing roles and responsibilities in writing such as through Memorandum of Understanding or MOUs (50). Relationships are joint, reciprocal and fostered through iterative group processes, active communication, and/or engagement. Facilitated relationships promote trust, shared power, and problem-solving orientations necessary to sustain implementation efforts (61).

Application in IS: This principle is applied by structuring collaborations with diverse and inclusive implementation networks (which requires periodically reflecting on which stakeholders are not at the table and need to be (re)invited to participate), facilitating interdependence by engaging and using mutual resources across all stakeholders, and by building cooperative inter-organizational relationships through participation agreements (9).

Application of principles through EPIS

A key recommendation for using IS frameworks is to establish and maintain community stakeholder partnerships (62). The co-creation concept fits well with existing frameworks. One prime example is EPIS, which encourages stakeholder engagement across the implementation ecosystem to facilitate efforts longitudinally and contextually (16, 63). For EPIS, co-creation is a bridging factor necessitating collaboration among stakeholders in the ecosystem's outer and inner contexts to shape an innovation's adoption and scale-up (64). Bridging factors are the relational ties, arrangements, and processes serving as the connective tissue across and between contexts (64).

Co-creation principles can inform feedback-driven collaborations throughout EPIS phases to increase synergy and equity (Figure 1). Although the three case studies below all used the EPIS to guide the collaborative process, co-creation principles are transferable to other frameworks. An engagement process driven by co-creation principles compels us to critically look at power among partners and how it manifests across each of the EPIS phases. One example is the work of Stanton et al. (3) who offer critical questions to pose across the implementation phases as a way to more intentionally address power in implementation collaborations, and we would add co-creation principles.

FIGURE 1

Figure 1. Co-creation EPIS model (65).

A description of three federally-funded implementation studies are presented in the next section as case examples.

Example # 1: Participatory implementation approaches to advance health equity for gender diverse and sexual minority (GSM) students.

“Reducing LGBTQ+ Adolescent Suicide (RLAS),” or RLAS for short, is a cluster-randomized study that uses a multisectoral community-academic partnership (CAP) involving stakeholders from schools, intermediary organizations, state government, and research institutions. The trial operationalized EPIS using the Dynamic Adaptation Process (DAP) (66), a data-driven implementation planning methodology that was used to facilitate uptake of interventions to enhance school climates and reduce suicidal behaviors for GSM high school students in New Mexico. This inclusive planning methodology made it possible for the CAP to convene implementation resource teams of educators, health professionals, and youth in 19 high schools. As described below, the CAP-provided feedback and technical assistance. The teams engaged in iterative assessment and planning processes to build school capacity, and select and implement interventions/implementation strategies, working closely with researchers to co-create and deploy locally responsive educational materials, tools, and action plans to introduce inclusive practices in socially-conservative school communities.

Levels of partner engagement in RLAS spanned the modes of “involve,” “collaborate,” and “empower” on the Spectrum of Public Participation continuum (67). The process of context-driven adaptation and site-specific tailoring placed IRTs in the highest level of involvement (“empower”) because members were charged with all final decisions regarding implementation. While they were engaged throughout the study, the participation of CAP members ranged from the “involve” to the “collaborate” modes. National organizations were consulted on training materials and data collection, and provided critical information about outer-context efforts concerning school health and GSM advocacy. State agencies were similarly engaged, yet more directly involved in shaping study objectives, providing resources to school sites, and responding to and applying study findings.

They benefited directly from engagement with implementation sites, as RLAS allowed for increased access to school settings that were otherwise difficult to reach. National- and state-level members were more closely aligned with the “involve” mode in that their guidance influenced RLAS, as the core study team maintained continued dialogue with them throughout the course of their work. The members with a more intermediary function were closely aligned with the “collaborate” mode, in that they partnered closely with study coaches, IRTs, and schools to shape implementation on a local level.

Two primary, yet not unsurmountable, challenges affected participation in RLAS. First was staff turnover at all levels. For example, turnover in schools (e.g., constantly changing administration and IRT membership) could undermine progress in implementing GSM supportive practices. Although turnover in the IRTs exerted the greatest direct impact on implementation, personnel at intermediary organizations and state agencies also changed over this 5-year study. Time emerged as a second factor affecting IRTs specifically. As school staff were already stretched thin, time for IRT members to meet, plan, and carry out action items to support implementation came to represent a scarce, highly valuable resource.

Three main facilitating factors balanced the above challenges. First, coaches were key to establishing and maintaining connections among schools, study staff, and intermediary organizations, providing guidance to IRTs, recruiting new IRT members, obtaining administrative buy-in, and ushering resources from outer to inner contexts. Second, the team structure of IRTs and their ability to evolve according to the needs of schools was a boon to sustaining implementation progress despite changes in membership. Third, the personal and institutional relationships fostered through the CAP, coaches, and schools allowed for problem-solving, mutually beneficial leveraging of resources, and tailoring supports to school-based needs, including addressing challenges of time constraints and changes in staff.

Example # 2: Implementation mapping to co-create protocols for supporting state-mandated screening of children for Adverse Childhood Experiences (ACEs).

ACEs are potentially traumatic events occurring before age 18, such as maltreatment or exposure to violence (68, 69). ACEs screening identifies these events and their associated health and wellbeing outcomes. In 2020, California issued an “ACEs Aware” policy that reimburses primary care clinics for annual patient screenings. In partnership with a health system serving over 6,000 children annually, this randomized trial is testing the impact of ACEs screenings on child service access and outcomes as well as the role of a multi-faceted implementation strategy in supporting such screenings for children ages 0–5 years. The co-creation process involved: (a) the clinical partner bringing their identified service gap to academic partner (i.e., need to address patient trauma) to co-develop a plan of action, (b) establishing a Trauma-Informed Care (TIC). Workgroup comprised of clinical staff, providers and managers to address this gap during the Exploration phase; (c) bridging this gap by adopting the state's ACEs Aware policy; and (d) undertaking participatory implementation mapping (70) (i.e., six step iterative and systemic collaborative approach to develop, select and/or tailor multi-level implementation strategies) to co-create implementation strategies for screenings and protocols for delivering trauma-informed care for future pilot-testing. The type of engagement for this project, based on the Spectrum of Public Participation continuum, falls within the collaboration and empowerment levels. The researchers partnered with healthcare administrators, service providers, program managers, members of the Information Technology and Quality Improvement departments, and caregivers of child patients to make decisions on every phase of the research process. That is, partners were collaborators of researchers in 2019 during the exploration implementation phase when the healthcare system was considering and ultimately adopted the ACEs Aware program, in 2020 during grant proposal writing and in 2021–2022 once the project received federal funding, during the planning process and pilot testing of the implementation strategy.

In addition, partners were empowered to make final decisions on which challenges and aligned implementation strategies to focus on, how to structure the activities for implementation of the ACEs screenings, how to organize the planning groups in terms of structure and process, and to have the power to request changes to data collection timeline, as possible by the funding agency, to accommodate significant changes within their system (e.g., high turnover) as well as changes to the ACEs policy requirements or other external events (i.e., inner or outer contexts). During the engagement process, the identification of challenges was complemented with acknowledgment of facilitators or assets within the partner healthcare system. Main challenges faced by partners were the high turnover and the financial and personal impact of the COVID-19 pandemic (i.e., 2020–2022) for implementers, leadership and caregivers of child patients.

Departure from key partners significantly impact engagement and the co-creation process as their expertise, gained knowledge and experiences and support is lost during a period of time or permanently if the position is not filled as it happened often with our healthcare partners. Nonetheless, facilitators that were leveraged to inform strategies included the use of implementation mapping (70) that allowed partners to work within smaller workgroups during the planning phase. This engagement early on during exploration and preparation phases (16) facilitated partners' ability to fill in relatively quickly for colleagues no longer at the organization. In addition, having two co-leads (aka champions) who were internal healthcare personnel co-lead the project along with the research team starting during grant proposal writing facilitated troubleshooting, decision-making, and coordination.

Example # 3: CO-CREATE: co-creating a COVID-19 testing program to promote health equity in a U.S./Mexico border region.

CO-CREATE is a rapid response, mixed methods implementation research study funded by the National Institutes of Health Rapid Acceleration of Diagnostics for Underserved Populations (RADx-UP) initiative to co-design and implement a culturally responsive and competent COVID-19 testing program for San Diego communities near the U.S./Mexico border (71). Co-creation drives this community engagement project through several methods: (a) a Community Advisory Board of community health workers, healthcare providers and administrators, and public health researchers who developed a project-driving theory of change and engaged in Appreciative Inquiry, to evaluate selection and implementation of co-created COVID-19 public health strategies (71); (b) qualitative brainwriting data collection sessions with patients and providers to identify and address implementation barriers; (c) partnership and co-leadership of all project activities with a federally-qualified health center to promote successful implementation and refinement of the testing program. For this project, the level of partner engagement fell between the collaborate and empower modes on the Spectrum of Public Participation continuum. Specifically, through the Theory of Change process (71), the Community and Scientific Advisory Board members were invited to collaborate with community and academic organizing team to identify root causes of inequitable COVID-19 testing and to develop community-vetted solutions to mitigate these inequities. After completion of the Theory of Change, the Community and Scientific Advisory Board has been engaged in an Appreciative Inquiry process to guide the implementation and evaluation of the identified solutions from the Theory of Change (71).

A primary facilitator was the community partner leaders of the project, the Global Action Research Center, who are an intermediary non-profit organization with strong and enduring relationships with community-based and ethnically-based community organizations in the region. The Global Action Research Center identified and invited the community leaders and health workers who were members on the Community and Scientific Advisory Board. They also primarily led each meeting, which fostered trust among the Board members and with the academic research team that organized the project. Another facilitator was the ongoing and multi-method evaluation of engagement that the project team undertook. This included ethnography and survey measurement of partner engagement after each meeting that allowed for near real-time assessment to inform changes needed within Board meetings to promote equitable and meaningful engagement.

Balanced with these facilitators were two primary and naturally-occurring challenges. First, to ensure equitable participation of Board members who represented the Latino, Spanish-speaking communities that were prioritized for the project, the Community and Scientific Advisory Board was structured to host live Spanish language interpretation and translation at each meeting. While this incurred more costs and reserved time for interpretation and translation, this was critical for important community perspectives to be shared. Second, because the levels of engagement were within the collaborate and empower modes, this required significant resources in terms of person-hours and fiscal costs. An analysis of the community engagement resource needs and costs are reported elsewhere (71).

Table 1 provides an overview of how each of the EPIS phases can be approach through a co-creation lens. For each phase(s), we also provide an overview of the activities used by the case examples to meet each of the co-creation principles.

TABLE 1

Table 1. Implementation research collaboration summary case analysis from a co-creation lens and guided by the EPIS framework.

EPIS sustainment phase

From a co-creation lens, the focus should be on supporting partners to lead their own engagement process locally to maintain goals achieved and to continue the implementation collaborative, if relevant. This work entails the exploration of alternative funding opportunities and new partnerships based on the shared commitment to addressing emergent and dynamic needs. Some of the case example projects described here are further along in the Sustainment Phase than others. All projects are currently seeking to maintain funding and developing new or complementary projects with partners. Connections developed in previous EPIS phases have allowed for continued resource provision after the withdrawal of study support. These connections also serve as a springboard for co-designing new initiatives. During this phase, attention to dissemination practices that adhere to each co-creation principles as a guide (e.g., sharing with equity, reciprocity, and mutuality) may involve concrete activities, including efforts to gradually shift control and decision-making to local champions through implementation coaching and feedback, co-presentations at academic and community forums, and new training opportunities for partners grounded in emerging needs and priorities from the co-created implementation process.

Discussion

In this commentary, we argue for co-creation in IS collaborations using five principles and by linking IS activities linked to each principle. These principles are transferable to any research area to enable a synergistic collaborative process. They can also foster longer-term relationships that can support resource intensive implementation efforts and sustainment of new practices. Thus, it is critical for researchers, implementers, and community partners to engage in co-creation to identify the need for change, the research-practice gap to address, prepare for and implement new practices, and sustain efforts long-term. The rich and inter-dependent knowledge that a co-created process promotes across diverse stakeholders is critical for ensuring fit and relevance to local contexts.

We recognize the challenges to co-creation in IS, with available infrastructure and time varying dramatically among academics, funders, and community-based stakeholders. Co-creation should not be expected to be a tidy process—it requires time, compromise and means IS researchers might need to step outside their comfort zones. We need to embrace rather than eschew the tensions possibly arising through co-creation (38), as they likely comprise a source of creativity and new ideas to plan for successful implementation. Furthermore, there is also a need for researchers to be clear on what exactly needs to be co-created and to balance a co-created process with the expectations of rigorous scientific endeavors (73). Co-creation in IS will require re-evaluating prioritization of academic knowledge and frameworks that do not align with or are irrelevant for community partners, especially given cultural, language, and social differences. Last, promoting implementation co-created implementation collaborations require specific knowledge and skills need to be incorporated into existing IS training.

In this paper, we make a case for further developing the concept of co-creation in IS with the goal of answering an ultimate question:

Does a co-created implementation collaboration provide stakeholders, especially end-users and community partners, with a deepened capacity to advocate for quality services, and as defined by local communities?

IS researchers may be familiar with system-level challenges to collaboration, but less aware of dynamics specific to local contexts. This blind spot can compromise their understanding of barriers, facilitators, mechanisms of change associated with implementation. Co-creation will enable such insights, enhancing the scientific value of our evaluations, our ability to replicate outcomes, and increasing the potential for achieving health equity and social justice through successful implementation of needed interventions.

In this paper, we present co-creation as a multidimensional concept and identified five concrete principle that were illustrated through three case examples. These principles have pragmatic values as they can be transferable across groups, topics and systems. Implementation collaboratives can use a prioritization approach to selecting all or the relevant co-creation principles, and as standard goals for the group. Then, partners can identify concrete activities that will allow them to achieve each of those goals and that align with the EPIS phases as presented in Table 1. This mapping of standard goals and tailored activities can facilitate ongoing monitoring and evaluation of the co-creation process through rapid iterative cycles (74). We also recommend complementing this approach by raising Stanton et al. (3) power-based questions for each of the EPIS implementation phases and as a way to translate co-creation from a high face validity value in engagement research to a formal and more standard practice in implementation research.

Prior research have used qualitative interviewing and ethnographic approaches to describe how co-creation builds co-creative relationships that support ongoing collaboration and problem-solving to sustain and scale out implementation efforts (53, 58, 61). However, there is still a dearth of outcomes research on co-creation (32). Mixed methods research is needed to simultaneously measure and explain the impact of co-creation on implementation proximal and distant outcomes such as partner experience, adoption, appropriateness, feasibility and sustainment. Last, future research should explore the alignment of co-creation with spectrums of engagement such as the IAP2 model (67). From that model, if collaboration and empowerment are seen as suitable levels of engagement to be achieved by a group, co-creation could be the vehicle to reach that goal.

An IS approach foregrounded in co-creation will help us better elucidate aspects of collaborations that adhere to co-creation principles, and whether outcomes are achieved through synergistic and equitable approaches among diverse stakeholders. It is our aspirational goal that the co-creation principles we described will inform current efforts to assess the quality of co-produced research (71), and that they will become a more normative and explicit application in IS research.

Author's note

There is growing interest for the field of implementation science research to promote the active engagement of diverse and often disenfranchised stakeholder voices. Including multiple perspectives can increase project buy-in, promote adherence to implementation protocols, improve the selection of meaningful outcomes, and overall sustainment of evidence-based practices. Currently, there is no clear guidance to help implementation science (IS) researchers identify and build collaborative approaches guided by co-creation principles for working with the community. This manuscript provides background on IS collaborations and identify the following co-creation principles relevant to the field and informed by organization, community-engaged and patient-centered literatures: (1) equity, (2) reflexivity, (3) reciprocity and mutuality, (4) transformative and personalized collaborative processes, and (5) relationships facilitated through inclusive networks, iterative processes, and active communication. Applications of these principles are further described using three IS research case examples. Principles for structuring co-creation collaborations will help researchers and community members engaged in implementation science research to spark synergy, address imbalances in power, privilege, knowledge, and access to resources. These principles will help researchers and community members collaborate in meaningful, equitable ways. Additionally, these principles provide the field with guidance that can be tested for its impact on producing meaningful, co-creation collaborations.

Data availability statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.

Author contributions

MP conceptualized the original idea for this manuscript and led the manuscript drafting and revision process. All authors drafted text, contributed original ideas across different sections of the manuscript, revised, and approved of the final manuscript text.

Funding

This work was partially supported by the National Institute of Mental Health R21MH123835 (MP); National Institute of Child Health and Human Development (R01HD083399; CW); NIMH K23 MH110602 (NS); R34 MH120190 (NS); P42 ES010337-19S2 RADx-UP Supplement (NS). JH works at the Center for Development, Evaluation, Complexity, and Implementation in Public Health Improvement (DECIPHer) which was funded by Welsh Government through Health and Care Research Wales; K01 DA056838 (EC); R01 DA049891 (EC and GA); P50 MH113662 (RL-H); P50 MH126231 (NS and GA). EC and RL-H are fellows, MP and NS are alumni, and GA is core faculty with the Implementation Research Institute (R25 MH080916). EC is also a fellow with the Lifespan/Brown Criminal Justice Research Training Program (R25 DA037190).

Acknowledgments

We are grateful for Tomashu Kenyatta Jones, Ph.D., for his valuable comments during the early discussions of this topic. We also thank Ani Movsisyan, Ph.D., for kindly sharing a comprehensive co-production/creation bibliography and Joelle Robinett for help with the bibliography.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. (2013) 8:139. doi: 10.1186/1748-5908-8-139

PubMed Abstract | CrossRef Full Text

2. Hurlburt M, Aarons GA, Fettes D, Willging C, Gunderson L, Chaffin MJ. Interagency collaborative team model for capacity building to scale-up evidence-based practice. Child Youth Serv Rev. (2014) 39:160–8. doi: 10.1016/j.childyouth.2013.10.005

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Stanton MC, Ali SB, Sustain Center Team. A typology of power in implementation: building on the exploration, preparation, implementation, sustainment (EPIS) framework to advance mental health and HIV health equity. Implement Res Prac. (2022) 3:263348952110642. doi: 10.1177/26334895211064250

CrossRef Full Text | Google Scholar

4. Brownson RC, Kumanyika SK, Kreuter MW, Haire-Joshu D. Implementation science should give higher priority to health equity. Implement Sci. (2021) 16:26. doi: 10.1186/s13012-021-01097-0

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Etherington N, Rodrigues IB, Giangregorio L, Giangregorio L, Graham ID, Graham ID, et al. Applying an intersectionality lens to the theoretical domains framework: a tool for thinking about how intersecting social identities and structures of power influence behaviour. BMC Med Res Methodol. (2020) 20:145. doi: 10.1186/s12874-020-01056-1

PubMed Abstract | CrossRef Full Text | Google Scholar

6. True G, Alexander LB, Fisher CB. Supporting the role of community members employed as research staff: perspectives of community researchers working in addiction research. Soc Sci Med. (2017) 187:67–75. doi: 10.1016/j.socscimed.2017.06.023

PubMed Abstract | CrossRef Full Text | Google Scholar

7. True G, Davidson L, Facundo R, Meyer DV, Urbina S, Ono SS. “Institutions don't hug people:” a roadmap for building trust, connectedness, and purpose through photovoice collaboration: J Hum Psychol. (2019) 61:365–404. doi: 10.1177/0022167819853344

CrossRef Full Text | Google Scholar

8. Sheridan S, Schrandt S, Forsythe L, Hilliard TS, Paez K. A Advisory Panel on Patient Engagement (2013 inaugural panel). The PCORI engagement rubric: promising practices for partnering in research. Ann Fam Med. (2017) 15:165–70. doi: 10.1370/afm.2042

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Ranjan KR, Read S. Value co-creation: concept and measurement. J Acad Mark Sci. (2016) 44:290–315. doi: 10.1007/s11747-014-0397-2

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Edquist C. Systems of Innovation: perspectives and challenges. In:Fagerberg J, Mowery D, Nelson R, editors. Oxford Handbook of Innovation. Oxford: Oxford University Press (2005). p. 181–208.

11. Prahalad CK, Ramaswamy V. Co-creating unique value with customers. Strategy Leadersh. (2004) 32:4–9.

Google Scholar

12. Heimburg DV, Cluley V. Advancing complexity-informed health promotion: a scoping review to link health promotion and co-creation. Health Promot Int. (2021) 36:581–600. doi: 10.1093/heapro/daaa063

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Boaz A, Hanney S, Borst R, O'Shea A, Kok M. How to engage stakeholders in research: design principles to support improvement. Health Res Policy and Syst. (2018) 16:60. doi: 10.1186/s12961-018-0337-6

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Rycroft-Malone J, Burton CR, Bucknall T, Graham ID, Hutchinson AM, Stacey D. Collaboration and co-production of knowledge in healthcare: opportunities and challenges. Int J Health Policy Manag. (2016) 5:221. doi: 10.15171/ijhpm.2016.08

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Jull J, Giles A, Graham ID. Community-based participatory research and integrated knowledge translation: advancing the co-creation of knowledge. Implement Sci. (2017) 12:150. doi: 10.1186/s13012-017-0696-3

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. (2011) 38:4–23. doi: 10.1007/s10488-010-0327-7

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Fernandez ME, ten Hoor GA, van Lieshout S, Rodriguez SA, Beidas RS, Parcel G, et al. Implementation mapping: using intervention mapping to develop implementation strategies. Front Public Health. (2019) 7:158. doi: 10.3389/fpubh.2019.00158

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Ibekwe LN, Walker TJ, Ebunlomo E, Ricks KB, Prasad S, Savas LS, et al. Using implementation mapping to develop implementation strategies for the delivery of a cancer prevention and control phone navigation program: a collaboration with 2-1-1. Health Promot Pract. (2022) 23:86–97. doi: 10.1177/1524839920957979

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Woodward EN, Willging C, Landes SJ, Hausmann LRM, Drummond KL, Ounpraseuth S, et al. Determining feasibility of incorporating consumer engagement into implementation activities: study protocol of a hybrid effectiveness-implementation type II pilot. BMJ Open. (2022) 12:e050107. doi: 10.1136/bmjopen-2021-050107

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Powell BJ, Beidas RS, Lewis CC, Aarons GA, McMillen JC, Proctor EK, et al. Methods to improve the selection and tailoring of implementation strategies. J Behav Health Serv Res. (2017) 44:177–94. doi: 10.1007/s11414-015-9475-6

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Pérez Jolles M, Martinez M, Garcia SJ, Stein GL, Thomas KC. Involving Latina/o parents in patient-centered outcomes research: contributions to research study design, implementation and outcomes. Health Expect. (2017) 20:992–1000. doi: 10.1111/hex.12540

PubMed Abstract | CrossRef Full Text | Google Scholar

22. HEALing Communities Study Consortium. The HEALing (helping to end addiction long-term SM) communities study: protocol for a cluster randomized trial at the community level to reduce opioid overdose deaths through implementation of an integrated set of evidence-based practices. Drug Alcohol Depend. (2020) 217:108335. doi: 10.1016/j.drugalcdep.2020.108335

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Waltz TJ, Powell BJ, Matthieu MM, Chinman MJ, Smith JL, Proctor EK, et al. Innovative methods for using expert panels in identifying implementation strategies and obtaining recommendations for their use. Implement Sci. (2015) 10 (Suppl. 1):A44. doi: 10.1186/1748-5908-10-S1-A44

CrossRef Full Text | Google Scholar

24. Sánchez V, Sanchez-Youngman S, Dickson E, Burgess E, Haozous E, Trickett E, et al. CBPR implementation framework for community-academic partnerships. Am J Community Psychol. (2021) 67:284–96. doi: 10.1002/ajcp.12506

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Leask CF, Sandlund M, Skelton DA, Altenburg TM, Cardon G, Chinapaw MJM, et al. Framework, principles and recommendations for utilising participatory methodologies in the co-creation and evaluation of public health interventions. Res Involv Engage. (2019) 5:2. doi: 10.1186/s40900-018-0136-9

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Israel BA, Schulz AJ, Parker EA, Becker AB. Critical issues in developing and following community-based participatory research principles. In:Minkler M, Wllerstein N, editors. Community-Based Participatory Research for Health. San Francisco, CA: Jossey-Bass (2008). p. 47–62.

Google Scholar

27. Gilmore B, Vallières F, McAuliffe E, Tumwesigye NM, Muyambi G. The last one heard: the importance of an early-stage participatory evaluation for programme implementation. Implement Sci. (2014) 9:137. doi: 10.1186/s13012-014-0137-5

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Harding T, Oetzel J. Implementation effectiveness of health interventions for indigenous communities: a systematic review. Implement Sci. (2019) 14:76. doi: 10.1186/s13012-019-0920-4

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Kano M, Willging CE, Rylko-Bauer B. Community participation in New Mexico's behavioral health care reform. Med Anthropol Q. (2009) 23:277–97. doi: 10.1111/j.1548-1387.2009.01060.x

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Goodyear-Smith F, Jackson C, Greenhalgh T. Co-design and implementation research: challenges and solutions for ethics committees. BMC Med Ethics. (2015) 16:78. doi: 10.1186/s12910-015-0072-2

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Mañas MA, Díaz-Fúnez P, Pecino V, López-Liria R, Padilla D, Aguilar-Parra JM. Consequences of team job demands: role ambiguity climate, affective engagement, and extra-role performance. Front Psychol. (2018) 8:2292. doi: 10.3389/fpsyg.2017.02292

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Metz A. Implementation Brief: The Potential of Co-Creation in Implementation Science. National Implementation Research Network. Available online at: https://nirn.fpg.unc.edu/sites/nirn.fpg.unc.edu/files/resources/NIRN-Metz-ImplementationBreif-CoCreation.pdf

Google Scholar

33. Green G, Boaz A, Stuttaford M. Public participation in health care: exploring the co-production of knowledge. Front Sociol. (2019) 4:73. doi: 10.3389/fsoc.2019.00073

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Aarons GA, Green AE, Palinkas LA, Self-Brown S, Whitaker DJ, Lutzker JR, et al. Dynamic adaptation process to implement an evidence-based child maltreatment intervention. Implement Sci. (2012) 7:32. doi: 10.1186/1748-5908-7-32

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Hawe P, Shiell A, Riley T. Complex interventions: how “out of control” can a randomised controlled trial be? BMJ. (2004) 328:1561. doi: 10.1136/bmj.328.7455.1561

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Greenhalgh T, Jackson C, Shaw S, Janamian T. Achieving research impact through co-creation in community-based health services: literature review and case study. Milbank Q. (2016) 94:392–429. doi: 10.1111/1468-0009.12197

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Halvorsrud K, Kucharska J, Adlington K, Rüdell K, Brown Hajdukova E, Nazroo J, et al. Identifying evidence of effectiveness in the co-creation of research: a systematic review and meta-analysis of the international healthcare literature. J Public Health. (2021) 43:197–208. doi: 10.1093/pubmed/fdz126

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Hawkins J, Madden K, Fletcher A, Midgley L, Grant A, Cox G, et al. Development of a framework for the co-production and prototyping of public health interventions. BMC Public Health. (2017) 17:689. doi: 10.1186/s12889-017-4695-8

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Ind N, Iglesias O, Schultz M. Building brands together: emergence and outcomes of co-creation. Calif Manag Rev. (2013) 55:5–26. doi: 10.1525/cmr.2013.55.3.5

CrossRef Full Text | Google Scholar

40. Mauser W, Klepper G, Rice M, Schmalzbauer BS, Hackmann H, Leemans R, et al. Transdisciplinary global change research: the co-creation of knowledge for sustainability. Curr Opin Environ Sustain. (2013) 5:420–31. doi: 10.1016/j.cosust.2013.07.001

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Pearce T, Maple M, Shakeshaft A, Wayland S, McKay K. What is the co-creation of new knowledge? A content analysis and proposed definition for health interventions. Int J Environ Res Public Health. (2020) 17:2229. doi: 10.3390/ijerph17072229

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Whicher A, Crick T. Co-design, evaluation and the Northern Ireland innovation lab. Public Money Manage. (2019) 39:290–9. doi: 10.1080/09540962.2019.1592920

CrossRef Full Text | Google Scholar

43. Cepiku D, Giordano F. Co-production in developing countries: insights from the community health workers experience. Public Manage Rev. (2014) 16:317–40. doi: 10.1080/14719037.2013.822535

CrossRef Full Text | Google Scholar

44. McLean RKD, Carden F, Graham ID, Aiken AB, Armstrong R, Bray J, et al. Evaluating research co-production: protocol for the research quality plus for co-production (RQ+ 4 Co-Pro) framework. Implement Sci Commun. (2022) 3:28. doi: 10.1186/s43058-022-00265-7

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Lee SM, Olson DL, Trimi S. Co-innovation: convergenomics, collaboration, and co-creation for organizational values. Manag Decis. (2012) 50:817–31. doi: 10.1108/00251741211227528

CrossRef Full Text | Google Scholar

46. Ostrom E. Crossing the great divide: coproduction, synergy, and development. World Dev. (1996) 24:1073–87. doi: 10.1016/0305-750X(96)00023-X

CrossRef Full Text | Google Scholar

47. Graham ID, Mccutcheon C, Kothari A. Exploring the frontiers of research co-production: the integrated knowledge translation research network concept papers. Health Res Policy Syst. (2019) 17:88. doi: 10.1186/s12961-019-0501-7

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Dungan R, Angove R, Cope E, Peay H. Engagement Science: Introducing Inclusive Research Practices & Potential Impacts. AcademyHealth Blog (2018). Available online at: https://academyhealth.org/blog/2019-01/engagement-science-introducing-inclusive-research-practices-potential-impacts (accessed March 18, 2022).

Google Scholar

49. Mayer C, McKenzie K. “…it shows that there's no limits”: the psychological impact of co-production for experts by experience working in youth mental health. Health Soc Care Community. (2017) 25:1181–9. doi: 10.1111/hsc.12418

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Heaton J, Day J, Britten N. Collaborative research and the co-production of knowledge for practice: an illustrative case study. Implement Sci. (2016) 11:20. doi: 10.1186/s13012-016-0383-9

PubMed Abstract | CrossRef Full Text | Google Scholar

51. Frow P, Nenonen S, Payne A, Storbacka K. Managing co-creation design: a strategic approach to innovation. Br J Manag. (2015) 26:463–83. doi: 10.1111/1467-8551.12087

CrossRef Full Text | Google Scholar

52. Bonsu SK, Darmody A. Co-creating second life: market—consumer cooperation in contemporary economy. J Macromarket. (2008) 28:355–68. doi: 10.1177/0276146708325396

CrossRef Full Text | Google Scholar

53. Yazejian N, Metz A, Morgan J, Louison L, Bartley L, Fleming WO, et al. Co-creative technical assistance: essential functions and interim outcomes. Evid Policy. (2019) 15:339–52. doi: 10.1332/174426419X15468578679853

PubMed Abstract | CrossRef Full Text | Google Scholar

54. Loignon C, Dupéré S, Leblanc C, Truchon K, Bouchard A, Arsenault J, et al. Equity and inclusivity in research: co-creation of a digital platform with representatives of marginalized populations to enhance the involvement in research of people with limited literacy skills. Res Involv Engagem. (2021) 7:70. doi: 10.1186/s40900-021-00313-x

PubMed Abstract | CrossRef Full Text | Google Scholar

55. van Nunspeet F, Ellemers N, Derks B. Reducing implicit bias: how moral motivation helps people refrain from making “automatic” prejudiced associations. Transl Issues Psychol Sci. (2015) 1:382–91. doi: 10.1037/tps0000044

CrossRef Full Text | Google Scholar

56. Snell-Rood C, Jaramillo ET, Hamilton AB, Raskin SE, Nicosia FM, Willging C. Advancing health equity through a theoretically critical implementation science. Transl Behav Med. (2021) 11:1617–25. doi: 10.1093/tbm/ibab008

PubMed Abstract | CrossRef Full Text | Google Scholar

57. Denzongpa K, Nichols T, Morrison SD. Situating positionality and power in CBPR conducted with a refugee community: benefits of a co-learning reflective model. Reflective Prac. (2020) 21:237–50. doi: 10.1080/14623943.2020.1733955

CrossRef Full Text | Google Scholar

58. Manganelli, A. Realising local food policies: a comparison between Toronto and the Brussels-Capital Region's stories through the lenses of reflexivity and co-learning. J Environ Policy Plan. (2022) 22:366–80. doi: 10.1080/1523908X.2020.1740657

CrossRef Full Text | Google Scholar

59. Skerletopoulos L, Makris A, Khaliq M. “Trikala quits smoking”: a citizen co-creation program design to enforce the ban on smoking in enclosed public spaces in Greece. Soc Mark Q. (2020) 26:189–203. doi: 10.1177/1524500420942437

CrossRef Full Text | Google Scholar

60. Finley EP, Huynh AK, Farmer MM, Bean-Mayberry B, Moin T, Oishi SM, et al. Periodic reflections: a method of guided discussions for documenting implementation phenomena. BMC Med Res Methodol. (2018) 18:153. doi: 10.1186/s12874-018-0610-y

PubMed Abstract | CrossRef Full Text | Google Scholar

61. Sherriff SL, Miller H, Tong A, Williamson A, Muthayya S, Redman S, et al. Building trust and sharing power for co-creation in aboriginal health research: a stakeholder interview study. Evid Policy. (2019) 15:371–92. doi: 10.1332/174426419X15524681005401

PubMed Abstract | CrossRef Full Text | Google Scholar

62. Moullin JC, Dickson KS, Stadnick NA, Albers B, Nilsen P, Broder-Fingert S, et al. Ten recommendations for using implementation frameworks in research and practice. Implement Sci Commun. (2020) 1:42. doi: 10.1186/s43058-020-00023-7

PubMed Abstract | CrossRef Full Text | Google Scholar

63. Moullin JC, Dickson KS, Stadnick NA, Rabin B, Aarons GA. Systematic review of the exploration, preparation, implementation, sustainment (EPIS) framework. Implement Sci. (2019) 14:1. doi: 10.1186/s13012-018-0842-6

PubMed Abstract | CrossRef Full Text | Google Scholar

64. Lengnick-Hall R, Stadnick NA, Dickson KS, Moullin JC, Aarons GA. Forms and functions of bridging factors: specifying the dynamic links between outer and inner contexts during implementation and sustainment. Implement Sci. (2021) 16:34. doi: 10.1186/s13012-021-01099-y

PubMed Abstract | CrossRef Full Text | Google Scholar

65. Becan JE, Bartkowski JP, Knight DK, Wiley TR, DiClemente R, Ducharme L, et al. A model for rigorously applying the Exploration, Preparation, Implementation, Sustainment (EPIS) framework in the design and measurement of a large scale collaborative multi-site study. Health Justice. (2018) 6:1–14. doi: 10.1186/s40352-018-0068-3

PubMed Abstract | CrossRef Full Text | Google Scholar

66. Willging CE, Green AE, Ramos MM. Implementing school nursing strategies to reduce LGBTQ adolescent suicide: a randomized cluster trial study protocol. Implementation Sci. (2016) 11:145. doi: 10.1186/s13012-016-0507-2

PubMed Abstract | CrossRef Full Text | Google Scholar

67. International Association for Public Participation. IAP2 Spectrum of Public Participation. International Association for Public Participation (2018). Available online at: https://cdn.ymaws.com/www.iap2.org/resource/resmgr/pillars/Spectrum_8.5x11_Print.pd (accessed September 14, 2022).

Google Scholar

68. Boullier M, Blair M. Adverse childhood experiences. Paediatr Child Health. (2018) 28:132–7. doi: 10.1016/j.paed.2017.12.008

CrossRef Full Text | Google Scholar

69. Perreira KM, Ornelas I. Painful passages: traumatic experiences and posttraumatic stress among immigrant Latino adolescents and their primary caregivers. Int Migr Rev. (2013) 47:12050. doi: 10.1111/imre.12050

PubMed Abstract | CrossRef Full Text | Google Scholar

70. Pérez Jolles M, Fernández ME, Jacobs G, De Leon J, Myrick L Aarons GA. Using implementation mapping to develop protocols supporting the implementation of a state policy on screening children for adverse childhood experiences in a system of health centers in inland Southern California. Front Public Health. (2022) 10:876769. doi: 10.3389/fpubh.2022.876769

PubMed Abstract | CrossRef Full Text | Google Scholar

71. Stadnick NA, Cain KL, Oswald W, Watson P, Ibarra M, Lagoc R, et al. Co-creating a theory of change to advance COVID-19 testing and vaccine uptake in underserved communities. Health Serv Res. (2022) 51:149–57. doi: 10.1111/1475-6773.13910

PubMed Abstract | CrossRef Full Text | Google Scholar

72. Goodman MS, Ackermann N, Bowen DJ, Thompson V. Content validation of a quantitative stakeholder engagement measure. J Commun Psychol. (2019) 47:1937–51. doi: 10.1002/jcop.22239

PubMed Abstract | CrossRef Full Text | Google Scholar

73. Williams O, Sarre S, Papoulias SC, Knowles S, Robert G, Beresford P, et al. Lost in the shadows: reflections on the dark side of co-production. Health Res Policy Syst. (2020) 18:1–10. doi: 10.1186/s12961-020-00558-0

PubMed Abstract | CrossRef Full Text | Google Scholar

74. Pérez Jolles M, Lengnick-Hall R, Mittman BS. Core functions and forms of complex health interventions: a patient-centered medical home illustration. J Gen Intern Med. (2019) 34:1032–8. doi: 10.1007/s11606-018-4818-7

PubMed Abstract | CrossRef Full Text | Google Scholar