The role of patients, caregivers, and communities in Learning Health Systems: a narrative review

Abstract

Learning Health Systems (LHSs) seek to continuously generate and apply evidence in clinical practice. Most Canadian LHS models emphasize engagement with patients, caregivers, and communities (herein contributors). Yet, there is limited guidance about how engagement works in these dynamic systems and how it may differ from engagement in other settings, for example patient-oriented research and quality improvement. This review examines engagement activities in existing and emerging LHSs for insights into the roles that contributors play in creating patient-oriented and equitable LHSs. A narrative review was conducted using the PerSPEcTiF framework. Search terms were identified for three domains: contributors, LHSs operating in direct patient care settings, and active engagement. Four databases (PubMed-MEDLINE, CINAHL, PsycINFO, Embase) were searched in December 2022. Articles were screened using a domain-based rubric and sampled for richness. Data were extracted including who was engaged, when, where, and how. Engagement activities were coded inductively, then deductively using the International Association for Public Participation (IAP2) Spectrum of Public Participation. An advisory group, including a Patient Partner, provided input at several stages in the project. Thirty-six articles describing engagement in 30 LHSs were included. In all, 192 engagement activities were coded to create a taxonomy of engagement; 139 activities were also coded to the IAP2 Spectrum. Contributors' influence over decision-making was often unclear or limited, with engagement frequently occurring after LHS implementation. However, LHSs also provided contributors with opportunities to engage in deliberative system design and effect change through distributed leadership. Ten contributor roles were synthesized, serving three functions: “System Shapers” (designing and defining LHSs), “Community and Capacity Builders” (expanding and supporting LHSs), and “Implementers” (hands-on efforts). This review provides an overview of engagement in LHSs, demonstrating how these practices can both build on and be constrained by engagement traditions in patient-oriented research and quality improvement. Findings offer a starting point for designing meaningful contributor roles and highlight opportunities to reimagine engagement practices by embedding contributors within systems and engaging communities beyond patient care settings.

1 Introduction

Learning Health Systems (LHSs) are flexible models of care that seek an ever-closer union between healthcare, research, and quality improvement, treating the provision of healthcare as a means of generating and applying evidence quickly in medical practice (1). There is no single definition of LHSs (2) but, essentially, these healthcare systems are designed “to continuously, routinely, and efficiently study and improve themselves” [(3), p. 1], through a “virtuous cycle of health improvement” [(1), p. 45]. LHS models often call for engagement with patients, caregivers, and communities (herein contributors) [e.g., (4, 5)], but effective engagement requires better understanding both of how LHSs work in practice, and how best to engage these important contributors within these models.

First proposed by the now National Academy of Medicine in 2007 (6),¹ LHSs emerged at a time of intense optimism about the transformative power of technology within healthcare (7). Under this model, medical care is seen as a way to generate data to improve quality of care, produce innovative treatments, and lower costs (8)—a data-driven vision for achieving the “quadruple aim” of improving patient and provider experience, bettering population health, and reducing healthcare expenditures (4). As a result, patients often feature in LHS literature as either sources of routine data [e.g., (9)] or ready participants for research [e.g., (10)], including examining the necessity (or not) of their consenting to various uses of their healthcare information [e.g., (11–13)]. By extension, considerable effort has focused on the potential for health data to support an active role for patients in managing their health [e.g., (14)]. This narrow focus on patients as “donors’ of data” [(15), p. 104] or recipients of evidence-driven insights into their own care may limit opportunities to engage patients in LHS design and implementation (15, 16), with few functioning LHSs reporting aspects of patient-oriented research (17).

There is a growing interest in LHSs in Canada, which is influenced by several factors including the potential to harness institutional synergy amongst Canadian health system actors including funders, policy-makers, and academics (4), and the desire to use rapid learning and adaptation to respond to rising pressures on publicly-funded provincial health systems—a process arguably accelerated by the COVID-19 pandemic (18, 19). Canadian LHS models emphasize an active role for patients and/or connections to community beyond direct patient care settings [e.g., (4, 5)]. Lavis et al. (5) identify “engaged patients” as one of seven core characteristics of a LHS. Similarly, Menear et al. (4) argue that a “transformative feature of LHSs is their emphasis on patient and community engagement” (p. 5), which fits within the “Social” pillar of their model. Thus, attempts at LHS implementation in Canada tend to recognize engagement as ensuring that LHSs are “anchored on patient needs, perspectives and aspirations” [(18), p. 8].

In Canada, LHSs are also being designed within an established and evolving culture of patient engagement in health research (20), largely driven by the Canadian Institutes of Health Research (CIHR) Strategy for Patient-Oriented Research (SPOR). First introduced in 2011, this national policy framework supports active and meaningful engagement of patients as partners in all aspects of health research (21). Patient-oriented research (POR) is distinct but related to other engagement traditions, for example patient-centered outcomes research in the United States (US) [e.g., (22)] and Patient and Public Involvement in the United Kingdom (UK) [e.g., (23, 24)]. Akin to approaches Madden and Speed (25) describe as “pragmatic and outcome orientated” (p. 3), POR aims to embed lived experience in applied research by connecting patients with other health system stakeholders,² early and often, to improve the relevance of research and speed its application in practice (21).

In 2022, the CIHR, Canadian provincial governments and partners jointly announced more than $250M CDN of additional funding for SPOR SUPPORT Units (27–34), with the stipulation that LHSs become a core component of each Unit's work (35). This closely tied patient engagement to LHSs and provided resources for supporting LHS implementation within and across Canadian jurisdictions. Several SPOR SUPPORT Units have created LHS models for their contexts (36–38). Building on POR traditions, these interpretations also emphasize an active role for contributors and prioritize health equity (36–38).

Despite these favourable conditions—an appetite for LHSs, availability of resources, a tradition of patient engagement, and the conceptual importance placed on engagement—there remain gaps in understanding about how best to engage patients in LHSs such that engagement is also seen as a barrier to LHS implementation (4, 17). Even while affirming its importance, Menear et al. (4) acknowledge that there is limited guidance about how to adapt existing engagement practices specifically within LHSs, including “few mechanisms…to ensure strong patient involvement in health system design and priority setting” (p. 9). Others have noted lack of guidance and limited roles as persistent hurdles to meaningful engagement within LHSs (16, 17). Thus, engagement reflects a broader “gap between the promise and practice of LHSs” [(39), p. 4], whereby a largely theoretical literature describes components of LHSs without a clear description of how to best implement them (17). This creates a two-fold challenge, first to understand how the LHS operates, and then how best to engage contributors within this context. This narrative review seeks to close this gap by collating and critically analysing examples of engagement activities within LHSs, examining contextual factors that impact engagement, and describing the roles that contributors play within various LHSs, with particular attention to how they may differ from engagement in patient-oriented research.

2 Methods

This review explores a variable and subjective phenomenon (engagement) as it occurs within complex systems (LHSs)—a combination that makes for a highly heterogenous literature that is not amenable to rigid systematic review methods characterized by pre-defined research questions and strict protocols (40). Instead, this narrative review draws on Boell and Cecez-Kecmanovic's hermeneutic method (41), a flexible approach that focuses on interpretative understanding developed through progressive engagement with the literature (40, 41). This study was undertaken in Nova Scotia, Canada. Handwritten reflexive journals, maintained by RG, provided a structured way to iteratively explore the literature (e.g., noting common themes, key questions) and increased awareness of her positionality, for example how past experiences as a patient and current work in patient-oriented research influenced the analysis, as well as how personal privilege creates blind spots towards inequities (42). Key steps are summarized below; activities were often iterative and overlapping. All steps were performed by a student lead (RG), with input from supervisors (JW, TF), and a Stakeholder Advisory Group (DR, MS, SD, LM, KH).

2.1 Stakeholder engagement

A Stakeholder Advisory Group (SAG) was created to seek specific perspectives, identify gaps in local knowledge, avoid duplication of efforts, and connect findings to practice (21, 23, 43, 44). Recruitment was purposive and limited to five members, including a patient partner, engagement staff, and an LHS researcher. All members were offered an honorarium. Prior to convening, all members completed a short online survey in Survey Monkey (45) to learn about their experience and interest in the topic. Survey results informed the agenda for a virtual meeting held in November 2022, where SAG members helped refine the research question and revise a list of search terms. This session informed the search strategy, which was emailed to all SAG members for review. Preliminary findings were also presented to two SAG members (DR, SD) in April 2023, when a “member reflections” exercise was performed. Stakeholder engagement and its impact are reported throughout and summarized in Supplementary Material S1 using the Guidance for Reporting Involvement of Patients and the Public short form (GRIPP2-SF) standard template (46).

2.2 Literature searches

An initial search was performed in October 2022. Both PROSPERO and PubMed-MEDLINE were searched for literature reviews (of any type) on engagement in LHSs (Supplementary Material S2). None were found, although several covered engagement in other settings [e.g., (47–51)]. While narrative review does not require it (40, 52), a more structured search strategy was developed to create an entry point into the variable LHS literature. The search strategy, summarized below, was reviewed by two health librarians.

2.2.1 Selecting search terms and defining concepts

The core search strategy hinges on three core concepts—contributors, engagement practices, and LHSs—each of which is described by a range of terms. Many of these terms have overlapping meanings, subtle context-specific nuances (2, 53), and/or politicized interpretations (54, 55). Diversity of terms can lead to unclear meaning and fragmentation in the literature (56, 57). Defining terms and identifying potential synonyms increases clarity and consistency, and enables a more comprehensive search strategy (58). Drawing on prior knowledge, a list of search terms was drafted then revised based on the literature on patient engagement (47, 49, 50) and LHSs (2, 59), as well as review protocols on engagement-related topics (58, 60, 61). This list was refined with feedback from the SAG. Notably, SAG members pointed to established, if siloed and distinct, traditions of engagement within research and quality improvement. To increase the relevance of search results, the setting was restricted to terms specific to learning health systems, organizations, and/or networks. This approach is consistent with previous research (17, 59), while also allowing some flexibility to capture different approaches to LHS implementation at the meso- and macro-level. Advocacy-related terms were excluded, as the SAG did not see this as a core function of engagement within local health systems. Finally, taking a similar approach to Chrysikou et al. (58), terms were grouped into “domains of terms.” Domains of terms were mapped against the first three components of the PerSPEcTiF question framework: perspective (Per), setting (S), phenomenon of interest (P), environment (E), comparison (c), timing (Ti) and findings (F) (Table 1). This question framework accounts for context in complex interventions (62), and provided structure for subsequent data extraction and analysis.

2.2.2 Searches, screening, and sampling

Searches were performed for the terms included in each core domain—Perspective (contributors), Setting (LHSs in direct patient care settings), and Phenomenon (active engagement)—and were limited to the title, abstract and keywords. Where available, Medical Subject Headings (MeSH) terms were included. The initial search strategy was developed for PubMed-MEDLINE (Supplementary Material S2), and adapted for CINAHL, PsycINFO, and Embase. Databases were searched in December 2022. Search syntax translations and search results were tracked in Microsoft Excel (63) (Supplementary Material S3). Targeted keyword searches were used to hand search highly-relevant journals: Learning Health Systems, Health Expectations, Research Involvement and Engagement, Health Research Policy and Systems, and BMC Health Services Research. Additionally, MS shared a collection of peer-reviewed literature that was tagged for patient engagement in another LHS review (17, 59). To maintain a manageable sample, all grey literature was excluded.

The core search results, as well as peer-reviewed literature recommended by MS and/or identified during the initial search, were uploaded to Covidence, a web-based software platform, for deduplication and screening (64). Screening was complicated by the heterogeneity of the literature and, as a result, inclusion and exclusion criteria were refined through three rounds of title and abstract screening. To speed the process, studies likely to meet inclusion criteria were tagged “highly relevant” and moved directly to full-text screening. Gradually, through this familiarization process, a domain-based screening rubric was developed (Table 2).

Given the large number of papers meeting the refined inclusion criteria (n = 63), a sampling strategy was devised based on “thick description” (65, 66). Here, “thick descriptions” are loosely defined as those that offer sufficient detail to contribute to meaningful interpretations of the literature and, employing a common strategy, “thick descriptions” were contrasted against “thin descriptions” as relative measures of the level detail (67). All sources meeting the inclusion criteria were exported from Covidence (64) to Microsoft Excel (63), and assessed for richness using a 2 × 2 matrix based on the detail provided about the LHS and engagement activities. This sampling strategy allowed for the examination of contributor roles in different settings, contrasting different approaches to engagement. RG performed screening and sampling, with a small number of sources additionally screened and assessed for richness by TF and JW.

2.3 Analysis and interpretation

A data extraction template was developed in Covidence (64) and trialled on five sources. Following the hermeneutic method, data extraction was an iterative, multi-step process. For each source, the annotated copy from the full-text screening was printed for “close reading,” and the data extraction form was completed in Covidence (64). Emerging findings were added to post-it notes on a wall to organize, cluster, and capture relationships between observations. After this process was complete for each individual source, extracted data for all included sources were exported from Covidence (64) into Microsoft Excel (63) for aggregate analysis and comparison across sources.

Specific examples of engagement activities were coded inductively for each source. Multi-faceted activities were double-coded, for example a Patient and Family Advisory Council that contributed to recruitment was coded as both an advisory council and recruitment (68). During aggregate analysis, activities were coded progressively within and across sources. Similar activities were then consolidated to create a taxonomy of engagement. As a crude measure of frequency, sources were counted once for each type of engagement activity that they described. After inductive coding, the International Association for Public Participation (IAP2) Spectrum of Public Participation (69), herein IAP2 Spectrum, was used to deductively code individual engagement activities. The IAP2 Spectrum runs a gamut from “Inform” to “Empower,” with each successive position signalling contributors' increasing influence over decision-making (69). This framework was recommended by the SAG for analysing the nature of the engagement. Sources were counted once for each type of engagement activity that coded to a specific position on the IAP2 Spectrum (69). During aggregate analysis, coding was compared across sources for consistency and to determine the position of similar activities on the IAP2 Spectrum (69). Activities with insufficient detail to reasonably place on the IAP2 Spectrum (69) were coded as unclear.

A virtual “member reflections” exercise was performed in April 2023 to allow for collaborative input into coding engagement activities (66). Initial findings were presented to two SAG members (DR, SD), who then independently sorted 20 examples of engagement activities to a position on the IAP2 Spectrum (69) using the Optimal Workshop online closed card sort (70). Following a discussion about coding differences, RG reviewed the coding of all engagement activities. Finally, RG synthesized roles based on the coded descriptions of engagement activities, and notes and observations on engagement practices captured throughout the analysis.

3 Results

Of the 1,430 records identified, 63 sources met the inclusion criteria. After assessing for richness, 36 sources providing “thick descriptions” of engagement were included. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram (71) was adapted to summarize the screening, sampling, and selection process (Figure 1). Findings characterize the literature and report on who is being engaged, where, when and how.

Figure 1

3.1 Study selection and characteristics

3.1.1 Limited focus on engagement

The literature was highly heterogenous (Table 3), with only a third of sources (n = 12) focused primarily on reporting engagement activities. In addition to providing rich detail, these sources surfaced tensions around representation (68), between individual experiences and population-level data (72, 73), trust and different ways of knowing (15, 74, 75), and also documented barriers and facilitators to engagement (15, 68, 72, 73, 75–78). Yet, half these sources offered scant contextual information on the LHSs (15, 73, 75–77, 79). This highlights the value—and relative scarcity—of “thick descriptions” of engagement within the LHS literature (even when selected for), and the separation of rich detail on engagement from contextual information about the LHSs where these activities take place. Conversely, most sources describe a particular LHS, with varying levels of detail about engagement reported as a part of a broader program of work (n = 20). A small number of sources tackling aspects of LHS design (n = 4) also met the inclusion criteria by including examples of engagement in patient care settings.

3.1.2 Largely narrative reporting with limited co-authorship by contributors

Most papers (n = 22) took the form of an experiential report that describes the firsthand experience of an LHS (68, 72, 74, 75, 78–95). Sometimes experiential reports were published explicitly as “use cases” (95), “real-world examples” (81), case studies (68, 92) or commentary (80), but often without providing specific methods [e.g., (78, 79, 92)]. Notably, only nine sources explicitly reported contributors as co-authors in-text and/or in the author information (15, 76, 77, 79, 80, 87, 96–98). While this suggests limited co-authorship by contributors, this may be an underestimate. For example, two sources used group authorship—CERTAIN Collaborative (72), “the pSCANNER team” (89)—where a full list of people involved was included at the end of the paper (whether those listed included contributors was unclear). Contributors may also identify based on professional roles within health systems but also have lived experience [e.g., (77)]. Still, this suggests that contributors may be infrequently involved in reporting engagement practices within LHSs.

3.1.3 Young and fragmented

All included sources were published in 2013 or later, with most sources (n = 31) published in 2017 or later. Nearly a third of sources (n = 10) were published in 2022, the year that the searches were performed. While the recency of the literature may improve its relevance to the current policy window, the sequential delays between the coining of LHS in the 2007 National Academy of Medicine (NAM) report (6) (the definition most cited amongst the sampled literature), the development of LHS models with an active role for patients, and the reporting of patient engagement activities may signal a tendency for engagement to be added in after implementation, developing over time as LHSs mature (72, 79, 82). Additionally, much of the literature is fragmented between journals with different topic areas, with 36 sources published in 26 journals. Only one source was published in a journal focused specifically on engagement (Health Expectations)—suggesting that the academic literature describing engagement activities within LHSs is relatively isolated from the broader peer-reviewed literature on engagement.

3.1.4 Variable use of theory

Theory is used in the literature to situate engagement within LHSs; guide engagement activities; organize collaboration; implement and scale interventions; and critically examine social processes (Table 4). Despite the existence of numerous patient engagement frameworks (48), engagement functions were generally described as a component of broader LHS models. Only two sources used engagement frameworks to guide activities (72, 76), with a further two sources highlighting engagement frameworks that helped orient publicly-funded health systems towards engagement over time, namely the Strategy for Patient-Oriented Research (SPOR) in Saskatchewan, Canada (80) and the Person-Centred Practice Framework in Sweden (78).

More commonly, organizational theory is used to guide collaboration, and implementation science frameworks used to implement and scale interventions. Notably, there is a small cluster of sources with overlapping authorship that describe the use of “Actor-Oriented Architecture” (88, 90, 99), developed by Fjeldstad et al. (100), to theorize aspects of collaboration. Amongst these, Murray et al. (88) emphasize that organizations can be designed to support people to prioritize and solve problems—reflecting distributed leadership as a mechanism for collaborative priority setting. Overall, there is a notable lack of critical analysis of social processes. Milligan et al. (97) offer a strong critique, arguing that the “rather narrow orientation toward clinical research and health service delivery neither adequately captures the breadth and depth of relevant theory from other research traditions…nor promotes a model that prioritizes nonclinical or tacit forms of knowledge” (p. 2). There are, however, a handful of exceptions drawing on critical social theory to challenge inequities arising from colonialism, social marginalization, and/or established hierarchies of knowledge (15, 95, 97).

3.2 Findings about engagement practices

3.2.1 Who is engaged in LHSs?

Most sources reported engaging more than one type of contributor (n = 30). Patients or people with lived experience of a health condition (n = 34) were engaged most often, followed closely by family and informal caregivers (n = 29) (Figure 2). Engaging contributors alongside other stakeholders was seen as advantageous, for example, in describing the design of the Collaborative Chronic Care Network (C3N), Seid et al. (90) noted that “taking a multi-stakeholder perspective forced the design team to consider the new system not as a system for doctors or a system for patients, but rather as a system for people” (p. 7)—an approach aligned with the quadruple aim central to the value proposition for many LHSs (4) and fitting pragmatic approach common to POR (21).

Figure 2

Engaging with communities was less common (n = 12) but also provided the only two instances where patients were not engaged. Community engagement focused on addressing persistent health inequities rooted beyond healthcare systems (97, 101). Notably, the term “community” was used variably, often making it difficult to determine who was being involved. Perhaps because the concept is nebulous and the practicalities of engaging large groups challenging (90), community perspectives were typically accessed via individuals: community leaders (83), organizers (90), representatives and/or members (75, 82, 85, 95, 101)—raising the thorny issue of representation (75). There were less frequent references to broader community outreach (75, 93) and partnering with community organizations (95), and these approaches were most common in LHSs with strong traditions of participatory community-based research (75, 101). Nonetheless, “community” tends to be treated as cogent “categories of shared identity” [(101), p. 2], often with scant reference to tensions within and between groups or the potential for intersecting identities.

Community engagement was also characterized by more a relational approach. For example, Seid et al. (90) frame LHS models as capable of appealing to solidarity, and thus supporting a shift from “health care as a transaction to shared work” (p.5). However, this focus on shared enterprise does not necessarily challenge the underlying assumptions about health or ways of knowing within these systems. Milligan et al. (97) emphasize how Indigenous knowledge systems, including a relational approach to well-being, can be at odds with colonial health systems that are typically rooted in Western traditions and derived from a biomedical model that privileges individual patient autonomy. This epistemic relationality presents a more fundamental challenge for LHSs, and the ways in which contributors' expertise and knowledge is valued within them. Notably, references to “publics” were rare (n = 5), amorphous, and often presumed, e.g., Levin et al. (84) indicated “patient partners provided a welcomed perspective on concerns and questions relevant for members of the patient community and the greater public” (p. 5). As a result, publics were not considered a distinct category for analysis.

Finally, examining who was being engaged provided insight into the potential nature of their contributions. There was considerable variation in the terms used to describe contributors, which were grouped to reflect their perceived expertise (Table 5). While the emphasis is clearly on lived experience of patients and caregivers, less common terms highlight potential shifts, for example a growing emphasis on cultural and/or organizational knowledge and relationships [e.g., Elders and Knowledge Keepers (97), veterans (102)]. Despite relatively few search terms emphasizing the relationship to the LHS, several were identified in the literature, e.g., actors (95, 100), coproducers (77), or “improvers” (74)—highlighting experience with the LHS as a sought-after form of expertise.

3.2.2 When did engagement take place in LHSs?

Timing of engagement was difficult to discern and largely inferred from the description of engagement activities. It was generally possible to determine whether engagement first occurred before or after the initial LHS implementation, with an even split between LHSs that engaged contributors from the outset or during early implementation versus those that engaged after implementation (Figure 3). This breakdown suggests that patients are often not default players, despite suggestions otherwise [e.g., (99)].

Figure 3

The exception was large publicly-funded LHSs, most of which initiated some form of engagement prior to LHS implementation (73, 80, 84, 89, 97, 98, 102, 103). The connection appears more tenuous at other levels of implementation, but there may be complex relationships at play. For example, all four meso-level LHSs that initiated engagement prior to implementation tended to be nested in larger organizational settings with established engagement traditions (77, 93, 96, 99, 104). This suggests that timing of engagement could be influenced by the presence of ongoing relationships and existing capacity for engagement. However, it is difficult to tell how much these findings are impacted by lack of reporting about the timing of engagement and/or when a health system starts using LHS terminology to describe itself.

3.2.3 Where were contributors being engaged?

Contributors were engaged in 30 LHSs providing direct patient care (Supplementary Material S4). Most sources reported on LHSs operating in a single, high-income country: USA (n = 22), Canada (n = 6), United Kingdom (n = 3), or Sweden (n = 2). Exceptions included two sources offering examples from the US and Sweden (77, 99), and sources reporting on the Starzl Network for Excellence in Pediatric Transplantation, an international collaboration with partners in the US and Canada (87, 91), and ImproveCareNow (ICN), which has members in US, UK, Belgium, and Qatar (68, 74, 81, 90, 99, 105).

Clearly, engagement is happening in a variety of LHSs, operating in a range of care settings, serving diverse populations, and implemented at different levels, from smaller projects to large-scale networks. Half of the LHSs (n = 15) focused on specific diseases (68, 74, 81, 90, 99, 105) or services for specific disease (77, 99). Two LHSs focused on creating infrastructure: a learning network supporting discrete projects (83) and an LHS creating technical infrastructure to support data sharing between three large health networks (89). Additionally, some larger networks aimed to create unified standards of care across multiple organizations (87, 91). Nearly a third of LHSs (n = 7) were categorized as having more than one focus, for example Early Intervention Services for Psychosis in Québec is a disease-specific LHS implemented within a publicly-funded provincial health system (103).

Similarly, the boundaries between meso- and macro-levels of implementation are porous, for example some LHSs categorized as meso-level operate as single health systems but involve networks of facilities [e.g., (86, 94)]. Boundaries between LHSs are often grey. The Cincinnati Children's Hospital, an LHS early adopter in the US, acts as a common denominator for multiple LHSs, including operating internal services as LHS (Diabetes Centre), hosting larger networked LHS (ICN), and providing mentorship and support for LHS spin-offs (C3N). Nesting LHSs within a single health system was evident in other sources too, for example, the STRONG STAR Consortium (102) and Veteran Affairs Evidence Synthesis Program (73)—both positioned as LHSs, but taking different approaches to engagement within Veteran's Affairs (itself an LHS). These findings suggest that engagement is present in LHSs of all shapes and sizes, and points to opportunities to nest LHS approaches—providing learning opportunities within and between LHSs.

Unsurprisingly, given the variability of LHS implementation, several sources emphasized the need to tailor engagement to specific LHS settings (83, 94, 97, 101). Yet, there were no obvious connections between LHS focus or level of implementation and contributors' roles or the timing of engagement—with one notable exception. LHSs categorized as “publicly-funded, macro-level LHS” (n = 7) tended to focus on power-sharing and/or partnership as key motivators for engagement (73, 78, 80, 84, 89, 97, 98, 102, 103), as opposed to more pragmatic contributions to bounded projects and, as noted previously, all but one initiated engagement activities prior to LHS implementation (Figure 3). Analysing a diverse sample of LHSs helped to identify shared features, present across various settings, that differentiate LHSs from more siloed traditions of research and quality improvement (Table 6). These features impacted engagement—both positively and negatively—and thus represent important contextual considerations for designing patient-oriented LHSs and engagement practices within them.

3.2.4 How were contributors engaged in LHSs?

Engagement activities (n = 192) were grouped into a taxonomy of engagement outlining seven functions (Table 7). All sources reported more than one type of activity. The most common engagement activities have strong traditions in patient-oriented research and/or quality improvement: advisory councils (n = 20); informal roles for individuals (n = 15); patient surveys (n = 16); training and education (n = 13); working groups (n = 13); creating resources (n = 9); providing testimonials (n = 7). These activities are consistent with the tendency for LHSs to build on internal research and quality improvement work [e.g., (75, 101)] or be designed de novo by partners drawing on existing traditions of engagement [e.g., (80, 90)]. Confirming the barrier noted by Menear et al. (4), priority setting activities were rare (n = 4). The lack of engagement in priority setting may be the product of delayed engagement, for example half of large networked LHSs reported on engagement activities occurring post-implementation (Figure 3), and is consistent with challenges noted in a CIHR SPOR evaluation (106). However, the relative rarity of these activities could also reflect a lack of reporting about timing of engagement and/or these functions may be assumed within references to other activities, e.g., governance committees.

The taxonomy of engagement provides practical examples of how contributors are being engaged in LHS, but also highlights shifts in engagement methods, particularly towards co-production (15, 76, 84, 88, 92, 95–98, 104) and, to a lesser extent, participatory methods (15, 95) that support community engagement. Existing approaches are evolving, e.g., the growing use of social media (18, 68, 73, 90, 107) and a shift from deficit-based training towards mentorship (74, 79, 103). However, newer approaches are less common, and the lines between old and new ways of engaging are often unclear. For example, in some instances, social media was used to involve patients in dialogue, e.g., the creation of an online community with “more than 150 people posting more than 1,700 messages” in a “private social media site” [(90), p. 5]. Conversely, social media channels were more commonly used to disseminate information [e.g., (18, 73)].

Nearly two-thirds of the 192 engagement activities were mapped to a position on the IAP2 Spectrum (n = 139) (Figure 4). The aggregate data suggests contributors' influence over decision-making remains limited in LHS settings, with only a small number of activities coded to “Empower” (n = 14). Roughly half coded along the low- and middle-spectrum: “Inform” (n = 11), “Consult” (n = 24) and “Involve” (n = 49). While the descriptions of these activities suggest that their influence over decision-making is relatively bounded, SAG members noted that these activities can still serve important functions and be appropriate, if aligned with engagement goals. More than a quarter of engagement activities were coded unclear (n = 53)—highlighting the need for more descriptive and evaluative reporting of engagement practices.

Figure 4

The breakdown of IAP2 Spectrum coding by activity provides a more nuanced view. Clearly design matters, with the same activity often occupying different places on the IAP2 Spectrum (Figure 5). This variation usually reflects differences between LHSs but, on rare occasions, the same type of activity occupied different places on the IAP2 Spectrum within the same LHS. These differences suggest that the nature of engagement cannot be taken for granted. For example, while co-production is billed as fundamentally collaborative, it was often invoked without details about the methods involved. As a result, about half of these general references were coded as unclear (84, 88, 92, 96, 104). Conversely, some references to co-production impacted decision-making sufficiently to be coded “Empower” (76, 95, 97), but the participatory methods associated with these approaches were coded as either “Involve” or “Collaborate”—suggesting co-production can also be more than the sum of its methodological parts.

Figure 5

The influence of the contextual features that impact engagement in LHSs (Table 6) is also apparent in the ways contributors are involved. For example, while governance committees often clearly described their decision-making authority (nearly all coded “Empower”), contributors may be ascribed influence over decision-making by virtue of their membership regardless of how much influence they exert. Similarly, influence over decision-making was far less clear for advisory groups than for working groups. Yet, decisions (albeit not usually at the system-level) tended to be made and implemented directly by working groups, perhaps an illustration of the assertion that distributed leadership can provide a path to meaningful engagement within LHSs [e.g., (75, 87, 95)].

The transition from informal to formal roles for individual contributors also provided opportunities for distributed leadership. Informal roles are more common (n = 15), but often had vague responsibilities and little or no reference to compensation. The scope of work for these roles tends to be more loosely defined (if at all) and/or their relationship to the broader organizations less clear. In contrast, four sources describe paid or funded formal roles for contributors (75, 86, 99, 102) that had clearer scopes (only one was coded unclear). Formal roles tended to move beyond narrative work and advisory functions to exert control over research, engagement, and/or knowledge translation activities, for example “Patient Supporters” in the Self-Dialysis Unit at Ryhov Hospital conceived the unit, co-designed the service, and trained patients to perform self-dialysis (99). Formal roles can also allow contributors to shape LHSs without assuming leadership positions, for example as “Community Research Associates” who combine research skills with community knowledge to perform a narrow range of tasks (75). These examples illustrate how contributors can be embedded within LHSs and guide the daily work of research and health system improvement.

Additionally, the active involvement of contributors in data-related functions appeared to be both constrained and evolving. Half of attempts to reflect contributor perspectives were consultative (n = 8) (18, 72, 74, 84–86, 96, 103), and accomplished primarily through patient/client surveys and standardized data collection tools (e.g., Patient-Reported Outcome Measures or, less frequently, Patient-Reported Experience Measures). There is a tension between bespoke or open measures that capture personal experience and standardized measures, which can be more cleanly collected, easily compared, and used to generate population-level data (72, 98). Knowles et al. (15) argue the need “to expand big data of informatics with the rich data of narrative and experience” (p. 112)—a position echoed in the call for data that is meaningful to both clinicians and patients (104) and capable of empowering people experiencing inequities (95). This process could involve engagement focused not only on what data is collected and how it is interpreted, but also by whom. There are indications that this transition is happening within LHSs through formal roles for contributors, for example Myers et al. (93) describe peer researchers gathering and interpreting data. There were also several engagement activities that actively involved contributors in data-related functions by helping to select and/or co-develop symptom questionnaires and survey tools (72, 83, 84), or through usability testing (103) and co-developing designing data dashboards (104).

Notably, despite excluding advocacy-related terms from the search syntax, advocacy activities were described in a small number of sources. For example, a petition jointly created by physicians and parents “to support prioritizing children for pediatric donor livers” received more than 15,000 signatures, demonstrating the “power of a unified voice” to “facilitate collection action” [(87), p. 423].

3.2.5 What role do contributors play in LHSs?

Ten distinct roles for contributors were synthesized, serving three broad functions within LHSs (Table 8). Contributors acted as System Shapers (designing and defining LHSs), Community and Capacity Builders (expanding and supporting LHSs), and Implementers (undertaking hands-on tasks) (Figure 6). Roles were flexible, varied, and often overlap with individuals sometimes performing several functions, along with the ability to come and go from roles. Three LHSs are associated with a single role for contributors (81, 85, 104); all other LHSs were associated with multiple roles.

Figure 6

For LHSs described in more than one source, e.g., ImproveCareNow (ICN) (68, 74, 81, 90, 99, 105), it was common for different sources to describe different roles. Differences in contributor roles between sources could be due to the focus of the publication but, in at least some cases, sources indicated roles that changed over time. For example David et al. (68) describes self-directed changes to the ICN Patient Advisory Council structure and membership engagement, and how this adaptive process helped “meet identified community needs” (p. 8). Therefore, roles described for each LHS are unlikely to be exhaustive but rather provide a snapshot of engagement and, just like engagement activities, offer opportunities to tailor engagement practices to specific LHS settings. There is considerable flexibility for those designing LHSs, both in addressing system needs and matching contributors with roles that suit their interests and skills.

In line with the dearth of activities coded to “Empower” on the IAP2 Spectrum, the most common roles related to the pragmatic “Implementers” function and are associated with a narrower scope of engagement—either as “Consultants” (n = 20 LHSs) or “Workers” (n = 18 LHSs). Conversely, the roles associated with “System Shapers” and “Community and Capacity Builders” suggest intriguing possibilities to reimagine engagement to take advantage of the unique features of LHSs (Table 6). For example, within the “System Shapers” category, “Leaders” did not necessarily have influence over system-level decision-making but were able to effect change through distributed leadership. For example, formal roles such as “funded investigators” (86) represent a devolution of powers akin to what Hughes and Duffy (108) describe as “user-led research,” where people with lived experience direct work including defining projects, seeking funding and publishing results. Defined to reflect distributed leadership, “Leaders” was the third most common role, present in nearly half of LHSs (n = 15 LHSs)—suggesting an expanding and possibly bottom-up influence for contributors over these dynamic health systems. Other examples include “Advocates” which reflect connections between LHSs and broader social context; “Service Designers” which reflects the embeddedness of contributors within clinical care settings; and “Reservoirs” which are well-suited to nested and networked approaches.

“Reservoirs,” where formal or informal affiliation connects contributors to LHSs, were also a key strategy for embedding contributors, and a way to balance between rapid change and potential longevity of LHSs (68, 74, 78–80, 90, 92, 95, 99, 105). Access to experienced contributors was seen as helpful for people new to engagement and a route to including more diverse perspectives (84). Reservoirs may also support flexibility, allowing contributors to take on and resign different roles over time. Practically speaking reservoirs were created via social processes (establishing and nurturing relationships), capacity development (training and mentoring), and organizational infrastructure, for example supportive policies, hiring engagement staff, formalizing contributor roles, routine communications (e.g., mailing lists), and patient registries. The impact of reservoirs depends on how they are used and maintained, and the ways in which contributors are called up. For example, patient registries can be passive pools occasionally dipped into (75, 78) or used to more proactively engage contributors, e.g., the Diabetes Centre Patient Registry, which is used to identify at-risk patients to involve and potentially support with an innovation fund (95). Having supportive organizational structures with engagement staff, e.g., dedicated office (76) or platform (80), is noted as a facilitator but warrants further exploration.

A closely related strategy involved embedding contributors as “Peer supporters,” whose involvement helped create engagement-ready environments. “Peer supporters” were involved in various peer functions, e.g., supporting engagement including through recruitment (75), onboarding (87), and helping other contributors to participate in or manage engagement activities and governance functions (79, 88, 89, 93). Peer functions were largely performed via groups, for example coalitions (80), collaboratives (76), networks (79, 98, 102). While mostly positioned as beneficial to the LHSs, contributors also expressed a desire for peer communities that provide forums to share experiences and support each others' engagement efforts (80). Peer functions were positioned near-universally as positive and supportive. For example, contributor involvement in recruitment is presented in a largely uncritical way with a focus on the benefits of attracting and supporting new contributors [e.g., (68, 75, 79, 80, 89)] and rarely problematized the potential bias for recruiting narrow perspectives. There was no discussion of what Vinson [(109), p. 3] calls “peer regulation”—a process of “orienting members’ action toward the stated values of a group”—and, by extension, the potential for “peer policing” or informal social control over contributors. More work is needed to understand how peers will relate to each other, as well as other health system actors, particularly those mediating peer functions such as engagement staff.

4 Discussion

This review provides an overview of engagement practices within existing and emerging LHSs. Unsurprisingly, given the flexibility of the LHS concept, the peer-reviewed literature on the topic is highly heterogenous and fragmented. In line with LHS models, engagement was frequently reported as a component of a larger programme of work, often limiting the detail provided. Despite these challenges, 192 engagement activities from 30 LHSs were examined to create a taxonomy of engagement (Table 7). Additionally, ten flexible and overlapping contributor roles were synthesized, which serve three core functions: “System Shapers” (designing and defining LHSs), “Community and Capacity Builders” (expanding and supporting LHSs), and “Implementers” (hands-on efforts). Taken together, these findings provide useful examples of how and why contributors have been engaged in LHSs, drawn from and applicable to a range of settings.

Critically examining engagement practices in LHSs demonstrates how engagement in these settings both builds on and is constrained by engagement traditions in research and quality improvement, which can be siloed (80) and vary in the extent of contributor involvement [e.g., (47, 48, 56)]. Having a culture of and capacity for engagement can be beneficial [e.g., (68, 72, 80, 86, 88)], but scaffolding to established practices risks perpetuating tokenism and other poor engagement practices (20, 106, 110, 111), as evidenced by the relative scarcity of activities coding to “Empower” on the IAP2 Spectrum (69). Few priority setting activities were identified and, when reported, the timing of engagement (after implementation in half of LHSs), suggests that contributors are not always included by default, despite these systems operating in direct patient care settings. These findings affirm and elaborate the documented “promise-practice” gap when it comes to engagement in LHSs (4, 16, 17).

There is also considerable latitude to define and refine engagement within the complex settings typical of LHSs. The nature of engagement activities is not inherent, with the same types of activities mapping to different positions on the IAP2 Spectrum (69)—variability that highlights the importance of how activities are structured and supported. Engagement activities are evolving, with the creation of formal roles, shifts towards mentorship over deficit-based training, and an emphasis on co-production including expanded roles for contributors in data-related functions. Contributors often play multiple roles within a single LHS, with new roles emerging that embed contributors within LHSs—particularly those associated with the “System Shapers” and “Community and Capacity Builders” functions.

In line with recent recommendations that contributors “be present and participate in” LHS design (112), embedded contributors become available to participate in deliberative system design, for example helping to network nested approaches; exercising distributed leadership; and nurturing relationships with other health system stakeholders.³ This relational approach makes them a part of “relational and self-organizing systems” [(113), p. 665], and aligns with a growing treatment of engagement as a landscape for mutual exploration (114), and complex, cooperative ecosystems mediated by dynamics between various actors (115). Embedding contributors could also strengthen and expand the small but distinct group of contributors recognized for their relationship to the LHSs and, by extension, knowledge of the health system. This shift would tap into a growing desire to capitalize on institutional knowledge held by experienced contributors (20), and increased calls for “systemic integration” of engagement to support LHS approaches [(116), p. 5] and for patient partner compensation (106, 116–118).

Yet, the relative scarcity of embedded approaches suggests that, while promising, embedding contributors within LHSs remains a largely untapped strategy. Considerable efforts have focused on embedding researchers in Canadian LHSs [e.g., (119)], but more work is needed to explore the practicalities of embedding contributors, for example organizational policies and compensation. There is also a need to evaluate contributors' desire for and experience of embedded functions, and to understand the perspectives of other health system actors, including engagement staff. This is particularly important given the potential of embedding contributors to exacerbate barriers to engagement, notably misunderstandings about contributor roles (79), challenges in adjusting to new roles (72), and creating burdens on contributors (104). When developing these new roles and ways of working, embedded contributors will need to navigate changing relationships with other health system actors, as well as with their peers. These interactions are shaped by the ways in which their knowledge and expertise are framed within LHSs (44, 120–122), values (113, 123), and by power dynamics (25). Although rarely invoked in the sampled LHS literature, social theory could help to explicate these complex social processes and provide useful lenses for further elaborating the social struggles that arise when embedding contributors.

Alongside the inwards shift to embed contributors, there is also an outwards expansion characterized by community engagement beyond direct patient care settings. In the sampled literature, community-based approaches often focus on addressing inequities and are distinct from patient and caregiver involvement. This aligns with calls for equity-centred engagement of distinct stakeholder groups to accelerate learning in LHSs (124), but this shift has important implications in the Canadian context. First, while the SPOR definition of engagement includes “affected communities” (Table 1), engagement has typically focused on individual patients and/or caregivers, with expertise derived from lived experience of health conditions and/or receiving health care. Recruitment largely relies on self-selection (125), with patients brought into stages of research, often to inform or consult on activities (106)—and not necessarily using participatory methods. Expanding engagement through existing networks may prove insufficient due to a lack of diversity amongst the existing patient partner community in Canada, which is skewed to older, white females (20). Furthermore, considerable effort will likely be needed to build relationships with equity-denied communities, and there were relatively few examples of LHSs engaging beyond the health system apart from a handful with cross-sectoral partnerships [e.g., (79, 86, 88, 95)].

By contrast, LHSs with a focus on equity tended to use participatory methods aimed at engaging and building relationships with communities, who bring experience beyond health and healthcare (75, 97, 101). This shift could be informed by the discourse about learning (121), and reflects “‘learning with’ community [which] entails authentic partnership, power-sharing and the co-production of knowledge” rather than more extractive and reductive approaches to “learning from” communities [(126), p. 2]. Additionally, the sampled literature suggests a need to engage communities in processes of unlearning. Beyond identifying and discontinuing practices that do not improve patient outcomes (82, 97), unlearning recognizes a need to “disassemble systems that have produced inequities, reimagine and create ones that produce equity, and overcome the inertia of the status quo” [(95), p. 4]. This involves examining not only patient outcomes but also changing “practices or policies that resulted in [the] current state” of the healthcare system [(95), p. 4]. This disassembly needs to address epistemic injustice or the undervaluing of some forms of knowledge (15)—a difficult task for health systems adherent to the hierarchy of knowledge associated with evidence-based medicine (127–129). The social processes underpinning learning and unlearning from communities will be important considerations for LHSs tackling inequities (93), particularly when engaging equity-denied communities where historic injustices compound current inequities (97), such that there is a need for “relationship repair” [(101), p. 7].

Therefore, the shift towards community engagement could bring the current pragmatic rationale for engagement into tension with more democratic rationales (25, 44), which enlist “broader social and ethical narratives around democratic representation, transparency, accountability, responsibility and the redressing of power imbalances” [as cited in (25, p. 6)]. As Dhamanaskar et al. (125) noted, the historical mixing of patient and public engagement obscures important theoretical differences in pragmatic patient-oriented approaches and more democratically motivated public engagement (124). This suggests that LHSs with strong POR traditions may need to rethink not just who is engaged but how and why, as well as what expertise contributors bring—particularly if aiming to improve health equity.

4.1 Strengths and limitations

Combining the hermeneutic method, with its focus on interpretive understanding, and the structure of the PerSPEcTiF framework was a key strength of this study. Coding and quantifying data added insight into the type, frequency, and nature of engagement activities, particularly through the application of the IAP2 Spectrum (69). This approach was flexible enough to include sources that supported critical reflections on how engagement differs in various LHS settings. However, to maintain a manageable scope, grey literature was excluded. This is an important limitation as strict word counts in peer-review publications may limit the level of detail provided about engagement practices and may bias the sample towards research, as opposed to other engagement activities that may not as routinely result in peer-reviewed publications, for example quality improvement initiatives. Additionally, screening, sampling, and analysis were also highly subjective; another reviewer may have selected a different sample, possibly arriving at different conclusions. The Stakeholder Advisory Group (SAG) helped to incorporate the perspectives of patients, engagement staff, and researchers, improving rigour and enriching the findings with alternative views (66), and increasing the relevance of findings (130).

Findings about individual engagement practices are also constrained by a tendency for easy measures of engagement (131) and “descriptive rather than evaluative” reporting (25, p. 2). This complicates efforts to understand the nature and scope of engagement activities, how they proceed over time, and their context (131, 132). Additionally, there is a relatively positive perspective on engagement that may arise from selecting for LHSs that value engagement enough to be doing it and/or publication bias. Where possible, sources describing engagement within the same LHS were selected to provide additional detail, and efforts were made to engage with critical themes. There are additional limitations arising from the literature itself. The variability in terms could lead to language-related blind spots. Publics were rarely engaged and/or essentially assumed via participation of patients, such that there was insufficient information to consider this group in the analysis. There was also insufficient detail to code the sociodemographic characteristics of contributors—information that is needed to better understand and support equitable engagement practices (106). Furthermore, the LHSs described in this sample operate in a small number of developed countries. This could impact on the applicability of findings in lower- and middle-income countries. Unique insights from these settings may also be missing, for example community-based health workers could provide useful models for formalizing roles and/or creating contributor reserves (133). Additionally, co-authorship by contributors was rare, which is out-of-step with increased calls for contributor acknowledgement (117, 118), and could materially impact how engagement practices are reported, as contributors experience engagement and view outcomes differently than other stakeholders [e.g., (106)].

Despite these limitations, narrative review “deals in plausible truth” [(40), p. 3]. This study offers an overview of engagement within emerging and existing LHSs—not an exhaustive account. If anything, this work reinforces that there is no complete sample or single way to do engagement in LHSs. It does, however, provide a snapshot of existing practices at the time of the search that highlights unique opportunities to design meaningful engagement within LHSs (Table 9), at a time when institutions are grappling with the realities of LHS implementation and SPOR is undergoing a refresh including a potential shift towards broader engagement and increasing community involvement (141).

4.2 Conclusions

Most LHS models call for an active role for patients and communities, yet contributor roles have arguably remained limited with few mechanisms for engagement in priority setting and system design. To help bridge this gap, this narrative review provides an overview of engagement practices in existing and emerging LHSs. Overall, engagement practices varied across the literature, with room to improve the role of contributors in LHSs. Emerging roles suggest new opportunities for contributors to participate in deliberative system design and challenge persistent health inequities. Future research should expand and explicate contributor roles, paying particular attention to embedded opportunities and community engagement, as well as examining the impact of engagement activities on LHS outcomes and exploring negative and/or unintended consequences of engagement.

References

• 1.

  Friedman CP Rubin J Brown J Buntin M Corn M Etheredge L et al Toward a science of learning systems: a research agenda for the high-functioning learning health system. J Am Med Inform Assoc. (2015) 22(1):43–50. 10.1136/amiajnl-2014-002977

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 2.

  Pomare C Mahmoud Z Vedovi A Ellis LA Knaggs G Smith CL et al Learning health systems: a review of key topic areas and bibliometric trends. Learn Health Syst. (2022) 6(1):e10265. 10.1002/lrh2.10265

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 3.

  Friedman CP Allee NJ Delaney BC Flynn AJ Silverstein JC Sullivan K et al The science of learning health systems: foundations for a new journal. Learn Health Syst. (2017) 1(1):e10020. 10.1002/lrh2.10020

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 4.

  Menear M Blanchette MA Demers-Payette O Roy D . A framework for value-creating learning health systems. Health Res Policy Syst. (2019) 17(1):79. 10.1186/s12961-019-0477-3

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 5.

  Lavis JN Gauvin FP Mattison CA Moat KA Wilson MG Reid RJ . Rapid Synthesis: Creating Rapid-learning health Systems in Canada. Hamilton, Canada: McMaster Health Forum (2018). Available online at:https://www.mcmasterforum.org/docs/default-source/product-documents/rapid-responses/creating-rapid-learning-health-systems-in-canada.pdf?sfvrsn=102554d5_5(Accessed November 27, 2022).

  • Google Scholar

• 6.

  Institute of Medicine (US), OlsenLAisnerDMcGinnisJM. The Learning Healthcare System: Workshop Summary. Washington, DC: The National Academies Press (2007). 10.17226/11903

  • CrossRef
  • Google Scholar

• 7.

  Friedman CP Rubin JC Sullivan KJ . Toward an information infrastructure for global health improvement. Yearb Med Inform. (2017) 26(1):16–23. 10.15265/IY-2017-004

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 8.

  Krumholz HM . Big data and new knowledge in medicine: the thinking, training, and tools needed for A learning health system. Health Aff (Millwood). (2014) 33(7):1163–70. 10.1377/hlthaff.2014.0053

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 9.

  Noritz G Boggs A Lowes LP Smoyer WE . “Learn from every patient”: how a learning health system can improve patient care. Pediatr Qual Saf. (2018) 3(5):e100. 10.1097/pq9.0000000000000100

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 10.

  DeJesus RS Bauer KW Bradley DP Haller I Bradley SM Schroeder DR et al Experience and expectations of patients on weight loss: the learning health system network experience. Obes Sci Pract. (2019) 5(5):479–86. 10.1002/osp4.364

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 11.

  Faden RR Kass NE . Learning health care and the obligation to participate in research. Hastings Cent Rep. (2022) 52(3):29–31. 10.1002/hast.1393

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 12.

  Cumyn A Barton A Dault R Cloutier AM Jalbert R Ethier JF . Informed consent within a learning health system: a scoping review. Learn Health Syst. (2020) 4(2):e10206. 10.1002/lrh2.10206

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 13.

  Cumyn A Barton A Dault R Safa N Cloutier AM Ethier JF . Meta-consent for the secondary use of health data within a learning health system: a qualitative study of the public’s perspective. BMC Med Ethics. (2021) 22(1):81. 10.1186/s12910-021-00647-x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 14.

  Oliver BJ Nelson EC Kerrigan CL . Turning feed-forward and feedback processes on patient-reported data into intelligent action and informed decision-making: case studies and principles. Med Care. (2019) 57(Suppl 51):S31–7. 10.1097/MLR.0000000000001088

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 15.

  Knowles SE Allen D Donnelly A Flynn J Gallacher K Lewis A et al Participatory codesign of patient involvement in a learning health system: how can data-driven care be patient-driven care? Health Expect. (2022) 25(1):103–15. 10.1111/hex.13345

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 16.

  Lee-Foon NK Smith M Greene SM Kuluski K Reid RJ . Positioning patients to partner: exploring ways to better integrate patient involvement in the learning health systems. Res Involv Engagem. (2023) 9(1):51. 10.1186/s40900-023-00459-w

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 17.

  Somerville M Cassidy C Curran JA Johnson C Sinclair D Elliott Rose A . Implementation strategies and outcome measures for advancing learning health systems: a mixed methods systematic review. Health Res Policy Syst. (2023) 21(1):120. 10.1186/s12961-023-01071-w

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 18.

  Cassidy C Sim M Somerville M Crowther D Sinclair D Rose AE et al Using a learning health system framework to examine COVID-19 pandemic planning and response at a Canadian health centre. PLoS One. (2022) 17(9 September):e0273149. 10.1371/journal.pone.0273149

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 19.

  McMahon M Nadigel J Thompson E Glazier RH . Informing Canada’s health system response to COVID-19: priorities for health services and policy research. Healthc Policy. (2020) 16(1):112–24. 10.12927/hcpol.2020.26249

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 20.

  Abelson J Canfield C Leslie M Levasseur MA Rowland P Tripp L et al Understanding patient partnership in health systems: lessons from the Canadian patient partner survey. BMJ Open. (2022) 12(9):e061465. 10.1136/bmjopen-2022-061465

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 21.

  Government of Canada CIHR. Strategy for Patient-Oriented Research - Patient Engagement Framework [Internet]. Canadian Institutes of Health Research (CIHR). (2014). Available online at:https://cihr-irsc.gc.ca/e/48413.html(Accessed May 27, 2019; Accessed November 27, 2022).

  • Google Scholar

• 22.

  Patient-Centered Outcomes Research Institute (PCORI). The PCORI Approach to Patient-Centered Outcomes Research [Internet]. Washington, DC: PCORI. (n.d.). Available online at:https://www.pcori.org/engagement/research-fundamentals/pcori-approach-pcor(Accessed January 22, 2024).

  • Google Scholar

• 23.

  Pandya-Wood R Barron DS Elliott J . A framework for public involvement at the design stage of NHS health and social care research: time to develop ethically conscious standards. Res Involv Engagem. (2017) 3(1):6. 10.1186/s40900-017-0058-y

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 24.

  UK Standards for Public Involvement [Internet]. UK Public Involvement Standards Development Partnership. (2019). Available online at:https://www.invo.org.uk/wp-content/uploads/2019/11/UK-standards-for-public-involvement-v6.pdf(Accessed January 22, 2024).

  • Google Scholar

• 25.

  Madden M Speed E . Beware zombies and unicorns: toward critical patient and public involvement in health research in a neoliberal context. Front Sociol. (2017) 2:7. 10.3389/fsoc.2017.00007

  • CrossRef
  • Google Scholar

• 26.

  Akl EA Khabsa J Petkovic J Magwood O Lytvyn L Motilall A et al “Interest-holders”: a new term to replace “stakeholders” in the context of health research and policy. Cochrane Evid Synth Methods. (2024) 2(11):e70007. 10.1002/cesm.70007

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 27.

  Government of Canada CIHR. The Government of Canada and Maritime Provinces put Patients First with new Investment. Fredericton, NB: Canadian Institutes of Health Research (CIHR) (2022). Available online at:https://www.canada.ca/en/institutes-health-research/news/2022/03/the-government-of-canada-and-maritime-provinces-put-patients-first-with-new-investment.html(Accessed December 24, 2022).

  • Google Scholar

• 28.

  Government of Canada CIHR. New Investment in Patient-oriented research in Manitoba. Winnipeg, MB: Canadian Institutes of Health Research (CIHR) (2022). Available online at:https://www.canada.ca/en/institutes-health-research/news/2022/06/new-investment-in-patient-oriented-research-in-manitoba.html(Accessed December 24, 2022).

  • Google Scholar

• 29.

  Government of Canada CIHR. Government of Canada and Alberta are Putting Patients First with new Research Investment. Edmonton, AB: Canadian Institutes of Health Research (CIHR) (2022). Available online at:https://www.canada.ca/en/institutes-health-research/news/2022/08/government-of-canada-and-alberta-are-putting-patients-first-with-new-research-investment.html(Accessed December 24, 2022).

  • Google Scholar

• 30.

  Government of Canada CIHR. Government of Canada and British Columbia are Putting Patients First with new Research Investment. Victoria, BC: Canadian Institutes of Health Research (CIHR) (2022). Available online at:https://www.canada.ca/en/institutes-health-research/news/2022/08/government-of-canada-and-british-columbia-are-putting-patients-first-with-new-research-investment.html(Accessed December 24, 2022).

  • Google Scholar

• 31.

  Government of Canada CIHR. Government of Canada and Newfoundland and Labrador are Putting Patients First with new Research Investment. St. John’s, NFLD: Canadian Institutes of Health Research (CIHR) (2022). Available online at:https://www.canada.ca/en/institutes-health-research/news/2022/07/government-of-canada-and-newfoundland-and-labrador-are-putting-patients-first-with-new-research-investment.html(Accessed December 24, 2022).

  • Google Scholar

• 32.

  Government of Canada CIHR. Governments of Canada and Ontario and Provincial Partners Renew Investment in Research That Puts Patients First. Ottawa, ON: Canadian Institutes of Health Research (CIHR) (2022). Available online at:https://www.canada.ca/en/institutes-health-research/news/2022/10/governments-of-canada-and-ontario-and-provincial-partners-renew-investment-in-research-that-puts-patients-first.html(Accessed December 24, 2022).

  • Google Scholar

• 33.

  Government of Canada CIHR. New Investment in Patient-oriented research in Saskatchewan. Saskatoon, SK: Canadian Institutes of Health Research (CIHR) (2022). Available online at:https://www.canada.ca/en/institutes-health-research/news/2022/06/new-investment-in-patient-oriented-research-in-saskatchewan.html(Accessed December 24, 2022).

  • Google Scholar

• 34.

  Government of Canada CIHR. The Governments of Canada and Québec and Provincial Partners Renew Research Investment That Puts Patients’ needs First. Ottawa, ON: Canadian Institutes of Health Research (CIHR) (2022). Available online at:https://www.canada.ca/en/institutes-health-research/news/2022/12/the-governments-of-canada-and-quebec-and-provincial-partners-renew-research-investment-that-puts-patients-needs-first.html(Accessed December 24, 2022).

  • Google Scholar

• 35.

  Government of Canada Canadian Institutes of Health Research (CIHR). SPOR SUPPORT Units (2012). Available online at:https://cihr-irsc.gc.ca/e/45859.html(Accessed April 24, 2023).

  • Google Scholar

• 36.

  Unité de Soutien SSA Québec. Learning Health System [Internet]. (n.d.). Available online at:https://ssaquebec.ca/wp-content/uploads/2022/03/SSA_rectangle_details_twitter_pwpoint_avec-logo-SSA_en.png(Accessed January 8, 2023).

  • Google Scholar

• 37.

  Maritime SPOR SUPPORT Unit (MSSU). Supporting Maritime Learning Health Systems. (n.d.). Available online at:https://mssu.ca/research/learning-health-systems/(Accessed September 1, 2023).

• 38.

  Reid R Wodchis P Kuluski K Lee-Foon N Lavis N Rosella L et al A learning health system adoption engine that integrates research and health systems. Institute for Better Health Brief 1. Mississauga: Institute for Better Health, Trillium Health Partners (2023). Available online at:https://ossu.ca/wp-content/uploads/Learning-Health-System-Brief-27Oct2023.pdf

  • Google Scholar

• 39.

  Hardie T Horton T Thornton-Lee N Home J Pereira P . Developing Learning Health Systems in the UK: Priorities for Action. London, UK: The Health Foundation (2022). Available online at:10.37829/HF-2022-I06

  • CrossRef
  • Google Scholar

• 40.

  Greenhalgh T Thorne S Malterud K . Time to challenge the spurious hierarchy of systematic over narrative reviews?Eur J Clin Invest. (2018) 48(6):e12931. 10.1111/eci.12931

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 41.

  Boell S Cecez-Kecmanovic D . A hermeneutic approach for conducting literature reviews and literature searches. Commun Assoc Inf Syst. (2014) 34(1):257–86. 10.17705/1CAIS.03412

  • CrossRef
  • Google Scholar

• 42.

  Meyer K Willis R . Looking back to move forward: the value of reflexive journaling for novice researchers. J Gerontol Soc Work. (2019) 62(5):578–85. 10.1080/01634372.2018.1559906

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 43.

  Staley K Barron D . Learning as an outcome of involvement in research: what are the implications for practice, reporting and evaluation?Res Involv Engagem. (2019) 5(1):14. 10.1186/s40900-019-0147-1

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 44.

  Martin GP . ‘Ordinary people only’: knowledge, representativeness, and the publics of public participation in healthcare. Sociol Health Illn. (2008) 30(1):35–54. 10.1111/j.1467-9566.2007.01027.x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 45.

  Survey Monkey Online Survey Software. San Mateo, USA: Momentive. Available online at:https://www.surveymonkey.com/(Accessed December 31, 2022).

  • Google Scholar

• 46.

  Staniszewska S Brett J Simera I Seers K Mockford C Goodlad S et al GRIPP2 Reporting checklists: tools to improve reporting of patient and public involvement in research. Br Med J. (2017) 358:j3453. 10.1136/bmj.j3453

  • CrossRef
  • Google Scholar

• 47.

  Bombard Y Baker GR Orlando E Fancott C Bhatia P Casalino S et al Engaging patients to improve quality of care: a systematic review. Implement Sci. (2018) 13(1):98. 10.1186/s13012-018-0784-z

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 48.

  Greenhalgh T Hinton L Finlay T Macfarlane A Fahy N Clyde B et al Frameworks for supporting patient and public involvement in research: systematic review and co-design pilot. Health Expect. (2019) 22(4):785–801. 10.1111/hex.12888

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 49.

  Brett J Staniszewska S Mockford C Herron-Marx S Hughes J Tysall C et al Mapping the impact of patient and public involvement on health and social care research: a systematic review. Health Expect. (2014) 17(5):637–50. 10.1111/j.1369-7625.2012.00795.x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 50.

  Harrison JD Auerbach AD Anderson W Fagan M Carnie M Hanson C et al Patient stakeholder engagement in research: a narrative review to describe foundational principles and best practice activities. Health Expect. (2019) 22(3):307–16. 10.1111/hex.12873

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 51.

  O’Connor S Hanlon P O’Donnell CA Garcia S Glanville J Mair FS . Understanding factors affecting patient and public engagement and recruitment to digital health interventions: a systematic review of qualitative studies. BMC Med Inform Decis Mak. (2016) 16(1):120. 10.1186/s12911-016-0359-3

  • CrossRef
  • Google Scholar

• 52.

  Sutton A Clowes M Preston L Booth A . Meeting the review family: exploring review types and associated information retrieval requirements. Health Inf Libr J. (2019) 36(3):202–22. 10.1111/hir.12276

  • CrossRef
  • Google Scholar

• 53.

  Grundmann R . The problem of expertise in knowledge societies. Minerva. (2017) 55(1):25–48. 10.1007/s11024-016-9308-7

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 54.

  Scarff JR . What’s in a name? The problematic term “provider”. Fed Pract. (2021) 38(10):446–8. 10.12788/fp.0188

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 55.

  Richards DP . The patient as person: an update. Br J Sports Med. (2020) 54(23):1376–1376. 10.1136/bjsports-2020-102418

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 56.

  Manafo E Petermann L Mason-Lai P Vandall-Walker V . Patient engagement in Canada: a scoping review of the ‘how’ and ‘what’ of patient engagement in health research. Health Res Policy Syst. (2018) 16(1):5. 10.1186/s12961-018-0282-4

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 57.

  Proctor EK Powell BJ McMillen JC . Implementation strategies: recommendations for specifying and reporting. Implement Sci. (2013) 8(1):139. 10.1186/1748-5908-8-139

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 58.

  Chrysikou E Hernandez-Garcia E Minetou L Savvopoulou E Rehn J Lad H . Current trends in patient involvement and engagement with people with dementia in built environment design research: a mixed-methods systematic review. PROSPERO: International prospective register of systematic reviews. (2021). Available online at:https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42021259928(Accessed December 30, 2022).

  • Google Scholar

• 59.

  Somerville M Cassidy C Curran J Rothfus M Sinclair D Elliott Rose A . What implementation strategies and outcome measures are used to transform healthcare organizations into learning health systems? A mixed-methods review protocol. Health Res Policy Syst. (2022 Sep 6) 20(1):97. 10.1186/s12961-022-00898-z

  • CrossRef
  • Google Scholar

• 60.

  Wearn A Kaner E Ramsay S Aquino MRJ Alderson H Cheetham M et al Evaluating process and outcomes of public involvement in applied health and social care research: a rapid systematic review. PROSPERO: International prospective register of systematic reviews. (2022). Available online at:https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42022310210(Accessed February 14, 2023, Accessed November 27, 2022).

  • Google Scholar

• 61.

  Macklin J Shahid N Pinto A . Patient Engagement in the Development of Artificial Intelligence (AI)-Based Applications in Health Care: a Systematic Review. PROSPERO: International prospective register of systematic reviews. (2020). Available online at:https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42020180739(Accessed November 27, 2022).

  • Google Scholar

• 62.

  Booth A Noyes J Flemming K Moore G Tunçalp Ö Shakibazadeh E . Formulating questions to explore complex interventions within qualitative evidence synthesis. BMJ Glob Health. (2019) 4(Suppl 1):e001107. 10.1136/bmjgh-2018-001107

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 63.

  Microsoft Excel [Internet]. Redmond, USA: Microsoft. Available online at:https://www.microsoft.com/en-ca/(Accessed October 10, 2025).

  • Google Scholar

• 64.

  Covidence Systematic Review Software [Internet]. Melbourne, Australia: Veritas Health Innovation. Available online at:https://support.covidence.org/help/how-can-i-cite-covidence(Accessed March 01, 2024).

  • Google Scholar

• 65.

  Geertz C . The Interpretation of Cultures. New York: Basic Books (1973). Thick Description: Toward an Interpretive Theory of Culture [Internet]; p.310-323. Available online at:https://philpapers.org/archive/GEETTD.pd

  • Google Scholar

• 66.

  Tracy SJ . Qualitative quality: eight “big-tent” criteria for excellent qualitative research. Qual Inq. (2010) 16(10):837–51. 10.1177/1077800410383121

  • CrossRef
  • Google Scholar

• 67.

  Ponterotto J . Brief note on the origins, evolution, and meaning of the qualitative research concept thick description. Qual Rep. (2006) 11(3):538–49. 10.46743/2160-3715/2006.1666

  • CrossRef
  • Google Scholar

• 68.

  David J Berenblum Tobi C Kennedy S Jofriet A Huwe M Kelekian R et al Sustainable generation of patient-led resources in a learning health system. Learn Health Syst. (2021) 5(3):e10260. 10.1002/lrh2.10260

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 69.

  Core Values, Ethics, Spectrum – The 3 Pillars of Public Participation. International Association for Public Participation [Internet]. (n.d.). Available online at:https://www.iap2.org/page/pillars(Accessed November 27, 2022).

  • Google Scholar

• 70.

  Optimal Workshop Card Sorting Software [Internet]. Wellington, New Zealand: Optimal Workshop; Available online at:https://www.optimalworkshop.com/(Accessed May 01, 2023).

  • Google Scholar

• 71.

  Page MJ McKenzie JE Bossuyt PM Boutron I Hoffmann TC Mulrow CD et al The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Br Med J. (2021) 372:n71. 10.1136/bmj.n71

  • CrossRef
  • Google Scholar

• 72.

  Devine EB Alfonso-Cristancho R Devlin A Edwards TC Farrokhi ET Kessler L et al A model for incorporating patient and stakeholder voices in a learning health care network: washington state’s comparative effectiveness research translation network. J Clin Epidemiol. (2013) 66(8 Suppl):S122–129. 10.1016/j.jclinepi.2013.04.007

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 73.

  Gierisch JM Hughes JM Williams JW Gordon AM Goldstein KM Williams JWJ . Qualitative exploration of engaging patients as advisors in a program of evidence synthesis: cobuilding the science to enhance impact. Med Care. (2019) 57:S246–52. 10.1097/MLR.0000000000001174

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 74.

  Keck C Hartley DM Havens M Margolis PA Seid M . Getting what is needed, when it’s needed: sharing information, knowledge, and know-how in a collaborative learning health system. Learn Health Syst. (2021) 5(3):e10268. 10.1002/lrh2.10268

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 75.

  Skelton JA Daniel SS Tapp H Moore KR Mann-Jackson L Alonzo J et al Developing and sustaining a community advisory committee to support, inform, and translate biomedical research. J Clin Transl Sci. (2023) 7(1):e20. 10.1017/cts.2022.473

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 76.

  Grob R Gleason K McLean P McGraw S Solomon M Joffe S . Patients’ roles in governance of learning: results from a qualitative study of 16 learning healthcare systems. Learn Health Syst. (2022) 6(1):e10269. 10.1002/lrh2.10269

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 77.

  Elwyn G Nelson E Hager A Price A . Coproduction: when users define quality. BMJ Qual Saf. (2020) 29(9):711–6. 10.1136/bmjqs-2019-009830

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 78.

  Hedberg B Wijk H Andersson Gare B Petersson C . Shared decision-making and person-centred care in Sweden: Exploring coproduction of health and social care services. Z Evidenz Fortbild Qual Im Gesundheitswesen. 2022;171[(Hedberg) School of Health and Welfare, Jonkoping University, Sweden]:129–34.

  • Google Scholar

• 79.

  Eidson T Hess A Hess T Kelly A . Family engagement in the autism treatment and learning health networks. Pediatrics. (2020) 145:30–4. 10.1542/peds.2019-1895F

  • CrossRef
  • Google Scholar

• 80.

  Teare GF Keller M Hall D . Bringing together research and quality improvement: the Saskatchewan approach. Healthc Q. (2018) 21:56–60. 10.12927/hcq.2018.25637

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 81.

  Moloney RM Tambor ES Tunis SR . Patient and clinician support for the learning healthcare system: recommendations for enhancing value. J Comp Eff Res. (2016) 5(2):123–8. 10.2217/cer.15.67

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 82.

  Lindsell CJ Gatto CL Dear ML Buie R Rice TW Pulley JM et al Learning from what we do, and doing what we learn: a learning health care system in action. Acad Med. (2021) 96(9):1291. 10.1097/ACM.0000000000004021

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 83.

  Holve E Segal C . Infrastructure to support learning health systems: are we there yet? Innovative solutions and lessons learned from American recovery and reinvestment act CER investments. J Comp Eff Res. (2014) 3(6):635–45. 10.2217/cer.14.72

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 84.

  Levin A Malbeuf M Hoens AM Carlsten C Ryerson CJ Cau A et al Creating a provincial post COVID-19 interdisciplinary clinical care network as a learning health system during the pandemic: integrating clinical care and research. Learn Health Syst. (2023) 7(1):e10316. 10.1002/lrh2.10316

  • CrossRef
  • Google Scholar

• 85.

  Nash DM Brown JB Thorpe C Rayner J Zwarenstein M . The alliance for healthier communities as a learning health system for primary care: a qualitative analysis in Ontario, Canada. J Eval Clin Pract. (2022) 28(6):1106–12. 10.1111/jep.13692

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 86.

  Davis FD Williams MS Stametz RA . Geisinger’s effort to realize its potential as a learning health system: a progress report. Learn Health Syst. (2021) 5(2):e10221. 10.1002/lrh2.10221

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 87.

  Perito ER Squires JE Bray D Bucuvalas J Krise-Confair C Eisenberg E et al A learning health system for pediatric liver transplant: the starzl network for excellence in pediatric transplantation. J Pediatr Gastroenterol Nutr. (2021) 72(3):417–24. 10.1097/MPG.0000000000002974

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 88.

  Murray DS Anixt JS Coury DL Kuhlthau KA Seide J Kelly A et al Transforming an autism pediatric research network into a learning health system: lessons learned. Pediatr Qual Saf. (2019) 4(2):e152. 10.1097/pq9.0000000000000152

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 89.

  Ohno-Machado L Agha Z Bell DS Dahm L Day ME Doctor JN et al pSCANNER: patient-centered scalable national network for effectiveness research. J Am Med Inform Assoc. (2014) 21(4):621–6. 10.1136/amiajnl-2014-002751

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 90.

  Seid M Dellal G Peterson LE Provost L Gloor PA Fore DL et al Co-Designing a collaborative chronic care network (C3N) for inflammatory bowel disease: development of methods. JMIR Hum Factors. (2018) 5(1):e8. 10.2196/humanfactors.8083

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 91.

  Squires JE Logan B Lorts A Haskell H Sisaithong K Pillari T et al A learning health network for pediatric liver transplantation: inaugural meeting report from the starzl network for excellence in pediatric transplantation. Pediatr Transplant. (2019) 23(6):e13528. 10.1111/petr.13528

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 92.

  Harris JG Bingham CA Morgan EM . Improving care delivery and outcomes in pediatric rheumatic diseases. Curr Opin Rheumatol. (2016) 28(2):110. 10.1097/BOR.0000000000000257

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 93.

  Myers RE DiCarlo M Romney M Fleisher L Sifri R Soleiman J et al Using a health system learning community strategy to address cancer disparities. Learn Health Syst. (2018) 2(4):e10067. 10.1002/lrh2.10067

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 94.

  Kraft S Caplan W Trowbridge E Davis S Berkson S Kamnetz S et al Building the learning health system: describing an organizational infrastructure to support continuous learning. Learn Health Syst. (2017) 1(4):e10034. 10.1002/lrh2.10034

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 95.

  Parsons A Unaka NI Stewart C Foster J Perez V Jones NHY et al Seven practices for pursuing equity through learning health systems: notes from the field. Learn Health Syst. (2021) 5(3):e10279. 10.1002/lrh2.10279

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 96.

  Van Citters AD Kennedy AM Kirkland KB Dragnev KH Leach SD Buus-Frank ME et al Prioritizing measures that matter within a person-centered oncology learning health system. JNCI Cancer Spectr. (2022) 6(3):pkac037. 10.1093/jncics/pkac037

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 97.

  Milligan C Mantla R Blake G Zoe JB Steinwand T Greenland S et al Health system learning with indigenous communities: a study protocol for a two-eyed seeing review and multiple case study. Health Res Policy Syst. (2022) 20(1):65. 10.1186/s12961-022-00873-8

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 98.

  Sivan M Greenhalgh T Darbyshire JL Mir G O’Connor RJ Dawes H et al LOng COVID multidisciplinary consortium optimising treatments and servIces acrOss the NHS (LOCOMOTION): protocol for a mixed-methods study in the UK. BMJ Open. (2022) 12(5):e063505. 10.1136/bmjopen-2022-063505

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 99.

  Fjeldstad ØD Johnson JK Margolis PA Seid M Höglund P Batalden PB . Networked health care: rethinking value creation in learning health care systems. Learn Health Syst. (2020) 4(2):e10212. 10.1002/lrh2.10212

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 100.

  Fjeldstad ØD Snow CC Miles RE Lettl C . The architecture of collaboration. Strateg Manag J. (2012) 33(6):734–50. 10.1002/smj.1968

  • CrossRef
  • Google Scholar

• 101.

  Irby MB Moore KR Mann-Jackson L Hamlin D Randall I Summers P et al Community-Engaged research: common themes and needs identified by investigators and research teams at an emerging academic learning health system. Int J Environ Res Public Health. (2021) 18(8):3893. 10.3390/ijerph18083893

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 102.

  Haibach JP Hoerster KD Dorflinger L McAndrew LM Cassidy DG Goodrich DE et al Research translation for military and veteran health: research, practice, policy. Transl Behav Med. (2021) 11(2):631–41. 10.1093/tbm/ibz195

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 103.

  Ferrari M Iyer S LeBlanc A Roy MA Abdel-Baki A . A rapid-learning health system to support implementation of early intervention services for psychosis in Quebec, Canada: protocol. JMIR Res Protoc. (2022) 11(7):e37346. 10.2196/37346

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 104.

  Dixon-Woods M Campbell A Chang T Martin G Georgiadis A Heney V et al A qualitative study of design stakeholders’ views of developing and implementing a registry-based learning health system. Implement Sci. (2020) 15(1):16. 10.1186/s13012-020-0976-1

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 105.

  Gremyr A Andersson Gare B Thor J Elwyn G Batalden P Andersson AC . The role of co-production in learning health systems. Int J Qual Health Care. (2021) 33:ii26–32. 10.1093/intqhc/mzab072

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 106.

  Government of Canada CIHR. Evaluation of the Strategy for Patient-Oriented Research (SPOR) [Internet]. Canadian Institutes of Health Research (CIHR). (2023). Available online at:https://cihr-irsc.gc.ca/e/53635.html(Accessed January 1, 2024).

  • Google Scholar

• 107.

  Harris J Croot L Thompson J Springett J . How stakeholder participation can contribute to systematic reviews of complex interventions. J Epidemiol Community Health. (2016) 70(2):207–14. 10.1136/jech-2015-205701

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 108.

  Hughes M Duffy C . Public involvement in health and social sciences research: a concept analysis. Health Expect. (2018) 21(6):1183–90. 10.1111/hex.12825

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 109.

  Vinson AH . Culture as infrastructure in learning health systems. Learn Health Syst. (2021) 5(3):e10267. 10.1002/lrh2.10267

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 110.

  Richards DP Poirier S Mohabir V Proulx L Robins S Smith J . Reflections on patient engagement by patient partners: how it can go wrong. Res Involv Engagem. (2023) 9(1):41. 10.1186/s40900-023-00454-1

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 111.

  Johannesen J . “The Trouble with Patient and Public Involvement (PPI)”—Keynote at Cochrane Colloquim 2018 [Internet]. (2018). Available online at:https://johannesen.ca/2018/09/the-trouble-with-patient-and-public-involvement-ppi-keynote-at-cochrane-colloquium-2018/(Accessed March 8, 2022).

  • Google Scholar

• 112.

  Fournier-Tombs M Bédard S Menear M L’Espérance A . Roadmap for partnering with patients, families and caregivers in a learning health system [internet]. Centre of Excellence on Partnership with Patients and the Public and Unité de soutien SSA Québec. (2022). p. 1–9. Available online at:https://ssaquebec.ca/wp-content/uploads/2023/05/P269-FINAL-ANG-v4-1.pdf(Accessed January 05, 2024).

  • Google Scholar

• 113.

  Greenhalgh T Engebretsen E Bal R Kjellström S . Toward a values-informed approach to complexity in health care: hermeneutic review. Milbank Q. (2023) 101(3):646–74. 10.1111/1468-0009.12656

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 114.

  L’Espérance A Bogdanova E Canfield C Levasseur MA Del Grande C Abelson J et al Learning Together Evaluation framework for Patient and Public Engagement (PPE) [Internet]. Centre of Excellence on Partnership with Patients and the Public. (2023). Available online at:https://ceppp.ca/en/resources/learning-together-evaluation-framework-for-patient-and-public-engagement-ppe-in-research/(Accessed January 5, 2024).

  • Google Scholar

• 115.

  Boivin A Dumez V Castonguay G Berkesse A . The ecology of engagement: fostering cooperative efforts in health with patients and communities. Health Expect. (2022) 25(5):2314–27. 10.1111/hex.13571

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 116.

  Chudyk AM Stoddard R Duhamel TA Andreas B Ashe MC Daly-Cyr J et al Future directions for patient engagement in research: a participatory workshop with Canadian patient partners and academic researchers. Health Res Policy Syst. (2024) 22(1):24. 10.1186/s12961-024-01106-w

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 117.

  Oliver J Lobban D Dormer L Walker J Stephens R Woolley K . Hidden in plain sight? Identifying patient-authored publications. Res Involv Engagem. (2022) 8(1):12. 10.1186/s40900-022-00346-w

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 118.

  Richards DP Birnie KA Eubanks K Lane T Linkiewich D Singer L et al Guidance on authorship with and acknowledgement of patient partners in patient-oriented research. Res Involv Engagem. (2020) 6(1):38. 10.1186/s40900-020-00213-6

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 119.

  Petrie S Cheng I McMahon M Lavis JN . Future leaders in a learning health system: exploring the health system impact fellowship. Healthc Manage Forum. (2023) 37(3):151–5. 10.1177/08404704231216951

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 120.

  Rowland P Forest PG Vanstone M Leslie M Abelson J . Exploring meanings of expert and expertise in patient engagement activities: a qualitative analysis of a pan-Canadian survey. SSM - Qual Res Health. (2023) 4:100342. 10.1016/j.ssmqr.2023.100342

  • CrossRef
  • Google Scholar

• 121.

  Rowland P Fancott C Abelson J . Metaphors of organizations in patient involvement programs: connections and contradictions. J Health Organ Manag. (2021) 35(2):177–94. 10.1108/JHOM-07-2020-0292

  • CrossRef
  • Google Scholar

• 122.

  Knowles SE Walkington P Flynn J Darley S Boaden R Kislov R . Contributors are representative, as long as they agree: how confirmation logic overrides effort to achieve synthesis in applied health research. Health Expect. (2022) 25(5):2405–15. 10.1111/hex.13555

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 123.

  Gradinger F Britten N Wyatt K Froggatt K Gibson A Jacoby A et al Values associated with public involvement in health and social care research: a narrative review. Health Expect. (2015) 18(5):661–75. 10.1111/hex.12158

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 124.

  Reid RJ Wodchis WP Kuluski K Lee-Foon NK Lavis JN Rosella LC et al Actioning the learning health system: an applied framework for integrating research into health systems. SSM - Health Systems. (2024) 2:100010. 10.1016/j.ssmhs.2024.100010

  • CrossRef
  • Google Scholar

• 125.

  Dhamanaskar R Boothe K Massie J You J Just D Kuang G et al Trends in government-initiated public engagement in Canadian health policy from 2000 to 2021. Healthc Policy. (2024) 20(Special Issue):17–35. 10.12927/hcpol.2024.27416

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 126.

  Milligan C Berta W . Reimagining community relationships for organizational learning: a scoping review with implications for a learning health system. BMC Health Serv Res. (2021) 21(1):1–10. 10.1186/s12913-021-06640-9

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 127.

  Ashcroft RE . Current epistemological problems in evidence based medicine. J Med Ethics. (2004) 30(2):131–5. 10.1136/jme.2003.007039

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 128.

  Tonelli MR . The philosophical limits of evidence-based medicine. Acad Med. (1998) 73(12):1234. 10.1097/00001888-199812000-00011

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 129.

  Greenhalgh T Wieringa S . Is it time to drop the ‘knowledge translation’ metaphor? A critical literature review. J R Soc Med. (2011) 104(12):501–9. 10.1258/jrsm.2011.110285

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 130.

  Whitty CJM . What makes an academic paper useful for health policy?BMC Med. (2015) 13(1):301. 10.1186/s12916-015-0544-8

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 131.

  Russell J Fudge N Greenhalgh T . The impact of public involvement in health research: what are we measuring? Why are we measuring it? Should we stop measuring it?Res Involv Engagem. (2020) 6(1):63. 10.1186/s40900-020-00239-w

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 132.

  Murdoch J Paparini S Papoutsi C James H Greenhalgh T Shaw S . Mobilising context as complex and dynamic in evaluations of complex health interventions. BMC Health Serv Res. (2023) 23(1430). 10.1186/s12913-023-10354-5

  • CrossRef
  • Google Scholar

• 133.

  Boyce MR Katz R . Community health workers and pandemic preparedness: current and prospective roles. Front Public Health. (2019) 7:62. 10.3389/fpubh.2019.00062

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 134.

  MacQueen KM McLellan E Metzger DS Kegeles S Strauss RP Scotti R et al What is community? An evidence-based definition for participatory public health. Am J Public Health. (2001) 91(12):1929–38. 10.2105/AJPH.91.12.1929

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 135.

  “Public.” Merriam-Webster.com Dictionary, Merriam-Webster [Internet]. Available online at:https://www.merriam-webster.com/dictionary/public(Accessed November 27, 2022).

  • Google Scholar

• 136.

  Psek WA Stametz RA Bailey-Davis LD Davis D Darer J Faucett WA et al Operationalizing the learning health care system in an integrated delivery system. EGEMS Wash DC. (2015) 3(1):1122. 10.13063/2327-9214.1122

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 137.

  Committee on the learning health care system in America, institute of medicine (US). In: SmithMSaundersRStuckhardtLMcGinnisJM, editors. Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. Washington, DC: National Academies Press (2013). p. 436. Available online at:http://www.ncbi.nlm.nih.gov/books/NBK207225/(Accessed July 19, 2023).

  • Google Scholar

• 138.

  Carman KL Dardess P Maurer M Sofaer S Adams K Bechtel C et al Patient and family engagement: a framework for understanding the elements and developing interventions and policies. Health Aff (Millwood). (2013) 32(2):223–31. 10.1377/hlthaff.2012.1133

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 139.

  Steels S Ainsworth J van Staa TP . Implementation of a “real-world” learning health system: results from the evaluation of the connected health cities programme. Learn Health Syst. (2021) 5(2):e10224. 10.1002/lrh2.10224

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 140.

  Steels S van Staa T . Evaluation protocol of the implementation of a learning healthcare system in clinical practice: the connected health cities programme in the north of England. BMJ Open. (2019) 9(6):e025484. 10.1136/bmjopen-2018-025484

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 141.

  SPOR Strategy and Governance Refresh Steering Committee. Pathways to Impact: Refreshing Canada’s Strategy for Patient-Oriented Research (SPOR). Ottawa (Canada): National Alliance of Provincial Health Research Organizations (NAPHRO). (2025). Available online at:https://www.naphro.ca/spor(Accessed March 31, 2025).

  • Google Scholar

• 142.

  Giacomantonio R . The Role of Patients, Caregivers, and Communities Within Learning Health Systems [dissertation/master’s Thesis]. Oxford (UK): University of Oxford (2024).

  • Google Scholar