Is Co-production Just a Pipe Dream for Applied Health Research Commissioning? An Exploratory Literature Review

Introduction

The Promise of Co-production

Internationally, the idea of “co-production” has become more popular in health research because of the promise of partnership between researchers and patients to create research that focuses on patient's needs. Patient and Public Involvement (PPI) at an early stage in deciding what research should be funded can improve the quality and impact of research. However, internationally there are very few examples of research commissioners involving patients or the public in decisions about research. This can leave members of the public feeling unheard or excluded by professionals.

Research commissioning is the most important stage of the research process for patients and the public to be involved as it gives the greatest potential to shape research agendas and to influence research funding (Oliver, 1996). However, research on decision making about future research priorities shows this rarely involves patients or the public. Decisions are more often made on the basis that technical rationalization of what research should be done, is more applicable than what is important to end users of research outputs.

Internationally in health services research PPI is widely recognized as being essential to the development of quality health services that are fit for purpose (Minogue and Girdlestone, 2010). Compared to health service delivery, PPI in health research management is globally a more recent movement and set of practices (Abrahams et al., 2004; Elberse et al., 2012; Gagnon et al., 2014; NIHR, 2015; PCORI, 2018).

Involving patients and the public in research, and especially in the early phases of research commissioning, such as research question or topic identification, priority setting, prioritization, and developing calls or advertisements for funding is thought to be crucial to overcome differential priorities between research funders, pharmaceutical companies and researchers, and the priorities of clinicians, patients and the public (Caron-Flinterman et al., 2005; Crowe et al., 2015). The consequences, as Chalmers and Glasziou (2009) describe of poor involvement of relevant stakeholders such as clinicians and patients in priority setting is an estimated avoidable waste of 85 per cent of global health research funding (Minogue et al., 2018).

Defining PPI and Co-production

The history of involving the public in service provision in the UK, one of the earliest adopters of PPI, was catalyzed by the rise in consumerist thinking in the 1960s and 1970s, and democratic or rights-based approaches that arose thereafter (Ridley et al., 2002). Under the UK Health and Social Care Act 2001 publicly-funded organizations have a duty to involve the public in the planning and provision of health services.

In the UK in 2006 the National Institute for Health Research (NIHR) was established with a mandate to involve patients and the public in commissioning and delivering publicly-funded applied health research. The organization “consumers in research” now known as INVOLVE, a national advisory group for PPI, also joined the NIHR in the same year. A legacy of this organization is its widely used definition of PPI, which we utilize in this paper:

“Research being carried out “with” or “by” members of the public rather than “to”, “about” or “for” them” (INVOLVE¹).

There is variation internationally in definitions, models and ways of thinking about PPI. There is for example no agreed nomenclature with participation, engagement and involvement often being used interchangeably. There is also great variation, dependant on the country's historical development of democracy, in the mix of institutionalized vs. contestory forms of involvement in healthcare (Slutsky et al., 2016). Within the UK context, involvement within health research funding tends be embedded within institutionalized mechanisms and processes.

Theoretically there are different levels at which people can be involved, as highlighted in Hogg's (1999) models of involvement in service development which closely relate to the INVOLVE levels of involvement in research (consultation, collaboration, user-led and co-production). Paternalistic models of involvement, assume that professionals know best, and hence lend themselves to involvement at the consultative level. The Partnership models of involvement lend themselves more to collaborative approaches to involvement. The Consumerist model describes consumers in charge or user-driven or controlled involvement. Finally the Autonomy model emphasizes the importance of valuing individuals and the different perspectives patients and professionals bring, and is closely aligned with involvement at the co-produced level as defined by INVOLVE (Hickey et al., 2018).

Co-produced research harnesses the principles of sharing of power, including all perspectives and skills, respecting values and the knowledge of all those working together on the research, reciprocity and building and maintaining relationships. However, this understanding of co-production, while acknowledged to be valuable, has been criticized as being idealistic given current cultural, institutional and regulatory constraints (Madden and Speed, 2017; Green and Johns, 2019; Paylor and McKevitt, 2019).

Previous Research

The evidence base for PPI, and especially effective co-produced approaches in the early phases of research commissioning is underdeveloped (Nilsen et al., 2006; Oliver et al., 2008), especially when compared to PPI elsewhere in research (Shippee et al., 2015) or health services commissioning (Sheaff et al., 2015). A rapid review carried out by Manafò et al. (2018), which we include in this review, utilized rapid review methodology to explore existing evidence around the different approaches that could be utilized to enable PPI in priority setting in health ecosystems and health research. There is a need to further explore these and other different approaches and mechanisms, and the influence and impact PPI might have in the early phases of the research commissioning context (Staniszewska et al., 2011). This information could inform innovative collaborative and co-produced approaches which maximize the benefits of PPI through the research commissioning process.

PPI is perceived to be particularly challenging in the commissioning of clinical research, which might not have direct relevance to human health or patient outcomes, due to its early placement in the applied health research translational pathway (Caron-Flinterman et al., 2005; Dobbs and Whittaker, 2006). Some researchers might assume that patients may be put off engaging in such research due to finding science boring, irrelevant, or intimidating (Dobbs and Whittaker, 2006). Other researchers may be apprehensive because PPI can mean a different way of working that challenges established notions of professionalism (Thompson et al., 2009).

Concerns about tokenism and meaningful PPI are found throughout the research literature but are used as a catchall term that may not fully convey the limiting forces of professional power. Tokenism can be defined as the policy or practice of making only a symbolic effort to involve people (Domecq et al., 2014) or failure to develop approaches that enable people to contribute in meaningful ways (Supple et al., 2015). Unequal power relations between experts and the public can be challenging for both parties, and co-production and power sharing may be an unfulfilled ideological goal.

Aims of the Review

The aim of this exploratory literature review was to draw on international health research literature to explore some of the contextual complexities and the potential challenges of PPI in the early stages of research commissioning, with a particular focus on early translational applied health research.

The questions we explored were (a) whether and how the public can be involved in the co-production of knowledge in research commissioning? (b) What are the specific challenges in the context of basic and early phase translational applied health research? The paper draws on the findings of the exploratory literature review to address these questions.

We used the notion of co-production to consider how research might overcome differentials in power between professional and public members, which may limit meaningful PPI. Drawing on examples and findings from the literature, in the discussion, we suggest possible ways forward for innovation and improvement of meaningful PPI.

Our focus is the potentially challenging field of commissioning early phase applied health research because it is here that commissioning is far less likely to involve PPI than in the later phases of the “bench-to-bedside' research process (Callard et al., 2012). The reasons for which we will also explore.

Methods

Approach

An exploratory literature review was carried out between May and August 2018. Owing to the disparate and scarce nature of evidence on PPI in research commissioning, a systematic review was unlikely to yield useful results that can inform practice. Therefore, an exploratory approach was chosen to seek out relevant published literature to allow us to consider the issues and challenges of PPI in research commissioning. The method is illustrated by Figure 1.

FIGURE 1

Figure 1. Flow chart of exploratory review method.

We sought information about how to enable meaningful and effective approaches to involvement, as well as clarification about the meaning of tokenistic PPI in this context. We were interested in learning about ways of working that enable patients/public representatives to contribute to decision-making processes and the types of impact that PPI can have. The study team included two public contributors who were consulted throughout study.

Inclusion/Exclusions

The review explored issues about PPI in the commissioning of health research, including health services, health care, public health, clinical, and biomedical research. Included articles were those that addressed issues about: (i) any type of patients and public groups involved and their roles e.g., public reviewers, patient representatives or lay members, (ii) contexts of involvement in stages of the commissioning process, (iii) approaches to involvement, for example commenting on commissioning materials or involvement in face-to-face meetings, informing decisions, or shared decision-making practices, (iv) evidence of influence or impact of involvement on commissioning decisions, practices, or outcomes.

We sought journal articles (including empirical studies and literature reviews) and gray literature (including reports, discussion papers, commentary, and opinion pieces) where these offered useful insights and learning and were published in the English language.

Due to the limitations of time and resources we excluded articles published in other languages. We excluded articles that did not relate to health research commissioning, for example PPI in commissioning social care research or health professional education.

Search Strategy

The search strategy was to identify relevant evidence and information using:

• web-based searches of Web of Science, Google Scholar and PubMed to search the international scholarly literature; explore related works, citations, authors, and publications; and the retrieval of documents through online libraries or on the web.

• searches of the INVOLVE Evidence Library for gray literature e.g., PhD studies, organizational reports, and bibliographies.

• searches for NIHR unpublished reports and documents relating to PPI in commissioning.

Key Search Terms

Searches used the key term “patient and public involvement in research commissioning' and variations on the term (e.g., patient involvement in funding agencies). A comprehensive search drew on the search terms used by Brett et al. (2014) in their systematic review of the impact of PPI. It combined sets of terms including and relating to patient and public involvement (consumer, citizen, client, carer, lay, service users, survivor, stakeholder, family, relative); type of involvement (particp^*, collaborat^*, engage^*, partner^*, consult^*, evaluat^*) and commissioning (funding agencies, research briefs, research funding, identifying research priorities, research priority setting, scoping review). MeSH terms were used to expand the searches (patients, public, economics, research, funding).

Data Extraction

Identified articles deemed to be relevant to the aim of the review were retrieved in full for analysis. Data were extracted into themed categories in Microsoft Word and key data extracted included the following: the author; the year and country; the aims or focus of the article; the methods used for PPI; the type of patients or groups of the public involved; key issues, findings or implications.

Analysis

The approach to the analysis was to explore and identify themes in the data (Denzin and Lincoln, 2005) reflecting the aims of the review to explore some of the contextual complexities and the potential challenges of PPI in the early stages of research commissioning. We read each article and considered the main issues raised in relation to the questions of whether and how the public can be involved and specific challenges associated with involvement in the commissioning context. As issues were identified, these were given a code (a title phrase or word representing the issue), and in this way a code framework was developed from the data to indicate patterns across the data (Braun and Clarke, 2006). Codes were grouped together into emerging themes (purpose, benefits, challenges, facilitators) with links to the original sources (Denzin and Lincoln, 2005). In the analysis the notion of co-production was used as a lens through which to consider issues of power (Hickey et al., 2018) between professionals and public members. For example we looked for examples of power sharing in the data, e.g., new roles and responsibilities of PPI members, evidence of shared decision-making, and approaches to supporting positive interactions and communication. Tables were used to present synthesized themes and links to original sources.

Rigor

A study protocol for the review was developed and revised by team members, including identification of databases to be searched and key search terms. Strategies for minimizing biases in the search strategy were as follows. (a) One team member independently cross-checked a sample of 20 returned papers against included/exclusion criteria. (b) Members of the team discussed and reached agreement on the importance of emerging themes in the analysis. (c) Inclusion and use of gray literature to extend the searches beyond peer reviewed articles.

Results

The review identified 74 relevant papers, reports and articles about PPI in health research commissioning. The results of the review confirmed the lack of published material specifically around PPI in the early phases of the commissioning processes of early phase applied health or basic health research. The review did yield results on PPI in commissioning of applied health research that was further along the translational pathway. Here we present summary results of the main findings with some representative references to the body of literature from the review.

The structure of the results is presented according to four themes that emerged:

• Reasons for PPI in research commissioning

• Benefits of PPI at strategic levels of research commissioning

• Contributions of patients and members of the public

• Improving PPI in research commissioning.

Reasons for PPI in Research Commissioning

The review demonstrated that PPI in research commissioning predominantly operated within a paternalistic model, with public members being consulted rather than more inclusively involved in the commissioning processes as co-creators of knowledge and co-producers of commissioning decisions and processes. Reasons for PPI were rarely given or explained, which could reflect the fact that PPI is often a requirement of being awarded central funding in the UK context. However, this is not the case in other countries or for all health research that is funded by other means.

Benefits of PPI at Strategic Levels of Research Commissioning

Despite operating within a paternalistic environment, several benefits to involving patients and the public, beyond getting them to provide views about priorities for research, were identified in the literature. These are summarized in Table 1 and include research priorities becoming more relevant to users; broader perspectives being brought into commissioning decisions; research being more likely to be ethical, inclusive and fair; the contribution of public contributors' skills and knowledge to commissioning decisions; and encouragement of PPI in funded research.

TABLE 1

Table 1. Benefits of PPI in research commissioning.

Contributions of PPI Members

The review also highlighted specific activities and contributions patients and public members make to the overall commissioning process. These have been summarized in Table 2 and include identifying topics, prioritizing topics, assessment, review of evidence, synthesizing results, and writing research briefs.

TABLE 2

Table 2. Contributions of PPI members to research commissioning.

Improving PPI in Research Commissioning

The review discovered that new priority setting projects are being developed around the world (in the UK, US, Australia, Netherlands, and Canada) to build partnerships between patients and professionals (Bragge et al., 2011; Gagnon et al., 2014; Tong et al., 2015; Pratt et al., 2016; Ghisoni et al., 2017; Abma, 2018; JLA, 2018; Manafò et al., 2018) (see Table 3).

TABLE 3

Table 3. Priority setting approaches that involve patients and the public.

Manafò et al. (2018) review of these priority-setting approaches concluded they are inclusive and objectively based, while being specific to the priorities of stakeholders engaged in the process. Key limitations identified were a lack of evaluation data on the success and extent to which patients were engaged, issues pertaining to feasibility of stakeholder engagement, coordination, communication, and limited resources.

Evaluation of nine projects that used the Dialogue Model (Abma et al., 2015) found patient involvement in agenda-setting is not automatically followed by patient involvement in programming and implementation. The authors recommend that support is needed during the process to organize patient involvement and adapt organizational structures like review procedures. Facilitating factors for success of the model include the importance of ownership; the value of dialogue for personal and mutual understanding; relational empowerment and critical awareness raising among patients; the importance of responsibility, responsiveness and trust; support in working with co-researchers; and the issue of representation (Abma, 2018).

Gagnon and colleagues of the Canadian Health Technology Assessment (HTA) programme have generated a conceptual framework for interventions to promote patient involvement in the early stages of HTA (Gagnon et al., 2014). Outcomes of PPI are evaluated with patients and their representatives using interviews and observations. These priority-setting projects and activities are promising but more needs to be done to test them out in different research funding contexts and particularly in early translational applied health research commissioning.

The review found examples of ways to facilitate PPI in commissioning, which could be utilized for the identification and prioritization stages of the process within early stage applied health research. In summary, these are:

• Planning for meaningful involvement all the way through the commissioning process (Oliver et al., 2004, 2009)

• Finding ways to expand opportunities for wider and effective participation and engagement with the public (Willis, 1995; Abelson et al., 2003; Oliver et al., 2008; INVOLVE, 2012; Morrow et al., 2013; Rikkers et al., 2015; Franck et al., 2018; Rawson et al., 2018; Simpson et al., 2018; Truitt et al., 2018)

• Building positive attitudes toward PPI as well as positive relations between stakeholders (Pittens et al., 2014; Abma et al., 2015; Abma, 2018). This could be facilitated by developing guidance, training and support for patient and the public contributors, Chairs of commissioning bodies and teams, including opportunities for shared learning (Boote et al., 2002; Caron-Flinterman et al., 2005; Oliver et al., 2008; INVOLVE, 2012)

• Encouraging organizations to assess the quality and impact of public involvement in commissioning (Oliver et al., 2015)

• Supporting commissioning teams to assess and provide feedback about processes and outcomes (O'Donnell and Entwistle, 2004; Howe et al., 2017).

Discussion

Variation in Opportunities for PPI in Commissioning

The review reveals a story of PPI opportunities for involvement in commissioning that ranges from ineffectual tokenism to meaningful co-creation of knowledge. Our findings suggest that while some research funders are fully committed to PPI at every stage, others have not given sufficient consideration to the benefits of PPI identified in this review. Indeed our findings do little to contest previous observations that commissioners may be concerned that PPI will distort research agendas (O'Donnell and Entwistle, 2004). Improved utilization of the review identified activities that patient and public representatives can be involved in by research funders would move the involvement model and levels from one of paternalism and consultation to one that is partnership-based and collaborative.

Most research funding organizations are open to asking patients to submit their views about priorities for research (e.g., a website where people can make suggestions for research), and some organizations go out and engage patients and groups of the public about their views about research needs. While the review highlighted novel and effective approaches to priority setting that include patients and the public, it also demonstrated that there is relatively little evidence, beyond identification and prioritization of research topics (e.g., James Lind Alliance Priority Setting Partnerships), of wide-spread co-production or co-creation in the development of prioritized research areas and funding calls.

The study by van Bekkum et al. (2016) which looked at ten UK agencies that fund health or medical research found involvement was not routinely incorporated into the planning of funding calls and there was little evidence of PPI being driven by democratic imperatives or rights-based arguments. Agencies and commissioning groups working within specific areas of health and medicine tend to promote particular definitions and practices which determine the boundaries in which researchers in these areas understand and practice PPI (van Bekkum et al., 2016). Professionals may be generally in favor of PPI but may believe that ultimately decisions about which research gets funded should be made by the professionals who are held accountable for these decisions (Oliver et al., 2004).

There are some strong examples of how the public can be involved in the co-production of knowledge in research commissioning. For example, some UK research funders, such as the NIHR and Medical Research Council, and US funders such as the Patient Centered Outcomes Research Institute, have research management frameworks for PPI which may include patients and members of the public being asked to review documentation that support prioritization of research topics or act as members of research prioritization committees (Oliver et al., 2009). However, even within this framework, it appears that some commissioning activities (e.g., defining assessment criteria, reviewing evidence, synthesizing results, writing documents for the consideration by committees, and funding decisions) may be undertaken by professionals without public input. Power is therefore balanced more toward researchers and funding organization staff than patients and public representatives. This is often the case for basic and the early applied health research commissioning context.

The Effects of Power Differentials

The review did not identify literature that focused on early stage commissioning processes for basic or early phase applied health research. The literature reveals some of the specific challenges in the context of basic and early phase translational applied health research. Perhaps most significant, is that professional skepticism and resistance manifest in subtle yet powerful ways that can limit co-production to a pipe dream (Chase et al., 2000). Even though the usefulness of patients' experiential knowledge alongside professional and clinical knowledge is widely accepted (Boote et al., 2002; Brett et al., 2014), it can be less clear how to integrate this type of knowledge into decision-making (Caron-Flinterman et al., 2005), to share ownership of decisions, and to assess decision-making effectiveness (Entwistle and O'Donnell, 2003). Researchers and funders may therefore employ tokenistic PPI, especially within the UK context where PPI is either increasingly encouraged or mandated.

The literature indicates that tokenism can be caused by lack of awareness or resistance to involvement amongst professionals, but can also be caused by practices, systems, structures or cultures that exclude patients and the public from contributing in meaningful ways (Supple et al., 2015). The technical nature of early phase translational research and the bureaucratic nature of commissioning may be a reason why the public are excluded from some commissioning activities. However, the literature demonstrates that public contributors' understanding of the technical clinical subjects, the language and science are not a necessary barrier to involvement.

When investigating patient and public involvement in biomedical research, Caron-Flinterman et al. (2005) asserted that training may support patients and the public to understand highly scientific or technical research. Further widespread use of non-technical language by professionals and plain English summaries may better enable involvement. Training for commissioning teams could cover inclusion strategies in patient–expert partnerships thereby enabling a better platform for both parties to effectively communicate and contribute to collaborative or co-produced approaches (Elberse et al., 2011).

Areas for Innovation and Improvement

Commissioning research requires informed judgements to be made about what research is important, and could lead to potentially significant results and impactful outcomes (Oliver et al., 2009). A sole focus on PPI as a participatory right endangers the involvement process into becoming a tokenistic activity that is consultative at best. Previous discursive papers on PPI suggests three different lines of thinking about the reasons for PPI in commissioning. These are: moral (to assure participative rights to involvement) (Boote et al., 2002; Coulter, 2004), methodological (to improve the quality and relevance of research to society) (Fisher, 2002; Chalmers and Glasziou, 2009), and impact (health, political, legislative, economic and societal impact). Moral or rights-based arguments suggest that PPI should be integral to research from the earliest stages as an intrinsic participatory right (Boote et al., 2002; Coulter, 2004). Methodological and impact based motivations, on the other hand, do not necessarily recommend involvement through the whole processes where it does not add value. It remains imperative that commissioners embed the moral or participatory rights-based driver as a key underlying factor that propels involvement in the system. Additionally an effective commissioning system must also consider and harness the methodological and impact drivers and benefits of PPI, such as those identified in this review, to create buy-in from all stakeholders.

Increasingly commissioning bodies are recognizing that the issue of what constitutes a rational discourse for future research, is a complex interplay of issues about how principles of patient need and rights translate into research contexts. Arguments against PPI warn against the lack of objectivity, possible bias, and individual self-interest of members of the public when it comes to making decisions about the allocation of research funds. Notions of the rights of the public to participate in all areas of health care—captured in the phrase “nothing about us without us”—are undermined by the apparent irrationality of involving members of the public in rational decisions about the allocation of research funds based on gaps in the evidence base and the feasibility, methods, and merit of the science in question.

Preoccupation with representation issues and concerns about the professionalization of lay members has directed too much attention to questions about the effectiveness of individual PPI representatives. Instead, PPI could be improved by examining the presuppositions and validity dimensions of everyday communication (normalized discourse) between professionals and PPI members. In relation to PPI in research commissioning this could include using reflective studies, to activate reflection on the unease, tensions and concerns about tokenism.

Improving opportunities for PPI requires the provision of meaningful spaces for dialogue, exchange and decision-making that suit different types of professionals and PPI representatives, as well as the public more generally. Early explicit exploration of different PPI roles and contributions with members of the public may assist effective participation and satisfaction. Singular PPI models are unable to effectively respond to the pluralism in experiences, values and opinions that different members of society hold.

Much could be gained from the involvement of third sector groups with local, regional or sector-wide views. Other approaches could be e-consultation or crowd-sourcing research topics and prioritizing them with a virtual public and professional community of practice, democratic prioritization (through voting), use of social media, or holding James Lind Alliance style priority-setting and consensus-building exercises to identify and prioritize areas of future focus (Rawson et al., 2018; Simpson et al., 2018; Truitt et al., 2018). In their review, Oliver et al. (2008) suggest a particularly fruitful method for involving the public in setting large-scale research agendas. The method was a combination of collaboration and consultation, with lay people taking leading roles in consulting peers in their networks.

There is a need for more innovative thinking about ways to relate to “seldom heard” and “hard to reach” populations, such as black and minority ethnic groups and persons with disabilities, by diversifying languages and mechanisms of communication. Creating mechanisms for engagement in commissioning that are more inclusive of diversity (e.g., by age, gender, ethnicity, socioeconomic background, and other characteristics) and reach out to wider groups of patients and the public (e.g., different experiences of health and illness, different patient groups, carers and those who are well) can help to stimulate interest and participation in commissioning. Combining different approaches can bring a more diverse range of people and their perspectives and views to the commissioning process that are more representative of diverse service user needs and priorities (Oliver et al., 2008).

More could be done to find ways to talk about complex technical ideas and research methods in accessible plain English (and to celebrate those professionals who find comprehensible expression), and to raise awareness of behavior that intimidates, side-lines or stigmatizes individuals. If we do not want PPI to be tokenistic in this area, it is important to develop policy, standards, guidance, roles, training, information, communication technologies and digital platforms (e.g., websites and social media) to support patient and public involvement in different research commissioning activities.

It is vital for people who find themselves occupying positions of power in the commissioning system to turn a critical eye toward the system. Research areas that appear to be far removed from immediate patient benefit due to being positioned early in the applied research translational pathway, especially need to better engage the public. Those in power should seek to show how the system is responsive to societal needs, for example showing the impact of commissioned research on patients or other beneficiaries (Pramesh et al., 2016). Therefore, a key issue for funders going forward is how to build capacity to adapt and absorb change brought about through co-production and the co-creation of new ways of commissioning.

Limitations

This review does not cover some of the practical challenges of PPI funders may face, including access and issues of reimbursement and payment. These issues, in different contexts, have been explored elsewhere in the literature and guidance to overcome some of these challenges is available from INVOLVE (Snape et al., 2014). The main limitation of the review is the focus on professionally defined commissioning approaches and models. It does not include lay groups taking the initiative through user-led research, or commissioning practices of user-led research organizations.

Limitations of the literature reviewed are the deficit of high-quality research studies (no trials were identified), the reliance on literature reviews, and small-scale evaluation studies carried out on single units or programmes. While international literature was included, differences in language and terminology of involvement, engagement and participation between countries are a limitation of the searches. Including languages other than English would have reduced bias but this was not possible within the limited resources for the review.

Conclusions

Although the public are involved in some countries, at some stages of the research commission process, it is clear that the process of agreeing research priorities is a long way from being co-produced and can be tokenistic. Tokenism can be caused by lack of awareness or resistance to involvement amongst professionals, but it can also be caused by highly structured commissioning systems, technically defined subject areas, and tasks that may exclude patients and the public from contributing in meaningful ways.

Addressing concerns about tokenism requires commissioners to critically reflect on current PPI practices and to devise ways of working that are meaningful and worthwhile for everyone involved. PPI could change from a minimal and minor role to a true partnership role, if improvements were made to communication, practices, systems, structures and cultures that stop patients and the public from contributing in meaningful ways.

If we want to avoid tokenism in PPI, it is important that commissioning organizations develop mechanisms to enable commission teams to secure the involvement of patients and the public through a range of options for engagement and involvement, including use of face-to-face methods and digital platforms. New, more distributed approaches to commissioning could be based on collaboration or partnership models, which bring together patients, carers and clinicians to create truly co-produced research agendas.

Data Availability

No datasets were generated or analyzed for this study.

Author Contributions

All authors are part of the project team comprising the larger work programme from which this paper draws its data. All authors contributed to the development of the paper, with DT and EM taking primary leadership in drafting and co-authors (LW, DL) providing detailed feedback on drafts.

Funding

The literature review was conducted as part of a work programme around an internal evaluation in the UK National Institute of Health Research Evaluation Trials and Studies Coordinating Center (NETSCC).

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We would like to thank members of project team of the NETSCC internal evaluation and in particular Alice Hawliczek (NETSCC) and Jane Putsey (Public Member).

Footnotes

1. ^INVOLVE. What is public involvement in research? Retrieved from: http://www.invo.org.uk/find-out-more/what-is-public-involvement-in-research-2/

References

Abelson, J., Eyles, J., McLeod, C. B., Collins, P., McMullan, C., and Forest, P. G. (2003). Does deliberation make a difference? Results from a citizens panel study of health goals priority setting. Health Policy 66, 95–106. doi: 10.1016/S0168-8510(03)00048-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Abma, T. (2005). Patient participation in health research: research with and for people with spinal cord injuries. Qual. Health Res. 15, 1310–1328. doi: 10.1177/1049732305282382

PubMed Abstract | CrossRef Full Text | Google Scholar

Abma, T. (2018). Dialogue and deliberation: new approaches to including patients in setting health and healthcare research agendas. Action Res. 1476750318757850. doi: 10.1177/1476750318757850

CrossRef Full Text | Google Scholar

Abma, T., Pittens, C. A. C. M., Visse, M., Elberse, J. E., and Broerse, J. E. W. (2015). Patient involvement in research programming and implementation: a responsive evaluation of the dialogue model for research agenda setting. Health Expect 18, 2449–2464. doi: 10.1111/hex.12213

PubMed Abstract | CrossRef Full Text | Google Scholar

Abrahams, N., Adhikari, R., Bhagwat, I. P., Christofides, N., Djibuti, M., Dyalchand, A., et al. (2004). Changing the debate about health research for development. international health research awards recipients. J. Public Health Policy 25, 259–287. doi: 10.1057/palgrave.jphp.3190028

PubMed Abstract | CrossRef Full Text | Google Scholar

Andejeski, Y., Breslau, E. S., Hart, E., Lythcott, N., Alexander, L., Rich, I., et al. (2002). Benefits and drawbacks of including consumer reviewers in the scientific merit review of breast cancer research. J. Womens Health Gend. Based Med. 11, 119–136. doi: 10.1089/152460902753645263

PubMed Abstract | CrossRef Full Text | Google Scholar

Boote, J., Dalgleish, M., Freeman, J., Jones, Z., Miles, M., and Rodgers, H. (2014). ‘But is it a question worth asking?’ a reflective case study describing how public involvement can lead to researchers' ideas being abandoned. Health Expect 17, 440–451. doi: 10.1111/j.1369-7625.2012.00771.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Boote, J., Telford, R., and Cooper, C. (2002). Consumer involvement in health research: a review and research agenda. Health Policy 61, 213–236. doi: 10.1016/S0168-8510(01)00214-7

PubMed Abstract | CrossRef Full Text | Google Scholar

Brady, L., and Preston, J. (2017). Evaluating the Extent and Impact of Young People's Involvement in National Institute for Health Research (NIHR) Studies: An Assessment of Feasibility. Report of a project commissioned by the James Lind Initiative. Retrieved from http://generationr.org.uk/?p=1375 (accessed July 9, 2018).

Google Scholar

Bragge, P., Clavisi, O., Turner, T., Tavender, E., Collie, A., and Gruen, R. L. (2011). The global evidence mapping initiative: scoping research in broad topic areas. BMC Med. Res. Methodol. 11:92. doi: 10.1186/1471-2288-11-92

PubMed Abstract | CrossRef Full Text | Google Scholar

Braun, V., and Clarke, V. (2006). Using thematic analysis in psychology. Qual. Res. Psychol. 3, 77–101. doi: 10.1191/1478088706qp063oa

CrossRef Full Text | Google Scholar

Brett, J., Staniszewska, S., Mockford, C., Herron-Marx, S., Hughes, J., Tysall, C., et al. (2014). Mapping the impact of patient and public involvement on health and social care research: a systematic review. Health Expect 17, 637–650. doi: 10.1111/j.1369-7625.2012.00795.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Callard, F., Rose, D., and Wykes, T. (2012). Close to the bench as well as at the bedside: involving service users in all phases of translational research. Health Expect 15, 389–400. doi: 10.1111/j.1369-7625.2011.00681.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Caron-Flinterman, J. F., Broerse, J. E. W., and Bunders, J. F. G. (2005). The experiential knowledge of patients: a new resource for biomedical research? Soc. Sci. Med. 60, 2575–2584. doi: 10.1016/j.socscimed.2004.11.023

PubMed Abstract | CrossRef Full Text | Google Scholar

Chalmers, I., and Glasziou, P. (2009). Avoidable waste in the production and reporting of research evidence. Lancet 374, 86–89. doi: 10.1016/S0140-6736(09)60329-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Chase, D., Milne, R., Stein, K., and Stevens, A. (2000). What are the relative merits of the sources used to identify potential research priorities for the NHS HTA programme? Int. J. Technol. Assess. Health Care 16, 743–750. doi: 10.1017/S0266462300102028

PubMed Abstract | CrossRef Full Text | Google Scholar

Coulter, A. (2004). Perspectives on health technology assessment: response from the patient's perspective. Int. J. Technol. Assess. Health Care 20, 92–96. doi: 10.1017/S0266462304000856

PubMed Abstract | CrossRef Full Text | Google Scholar

Crocker, J. C., Boylan, A. M., Bostock, J., and Locock, L. (2017). Is it worth it? patient and public views on the impact of their involvement in health research and its assessment: a UK-based qualitative interview study. Health Expect. 20, 519–528. doi: 10.1111/hex.12479

PubMed Abstract | CrossRef Full Text | Google Scholar

Crowe, S., Fenton, M., Hall, M., Cowan, K., and Chalmers, I. (2015). Patients', clinicians' and the research communities' priorities for treatment research: there is an important mismatch. Res. Involv. Engage. 1:2. doi: 10.1186/s40900-015-0014-7

CrossRef Full Text | Google Scholar

Denzin, N., and Lincoln, Y. (2005). “Introduction: the discipline and practice of qualitative research,” in The Sage Handbook of Qualitative Research 3rd ed, eds N. K. Denzin, and Y. S. Lincoln (Thousand Oaks, CA: Sage, 1–32.

Dobbs, T., and Whittaker, I. (2006). Patient and public involvement in basic science research - are we doing enough? BMJ Opin. Available online at: https://blogs.bmj.com/bmj/2016/05/11/ppi-in-basic-science-research-are-we-doing-enough

Domecq, J. P., Prutsky, G., Elraiyah, T., Wang, Z., Nabhan, M., Shippee, N., et al. (2014). Patient engagement in research: a systematic review. BMC Health Serv. Res. 14:89. doi: 10.1186/1472-6963-14-89

PubMed Abstract | CrossRef Full Text | Google Scholar

Elberse, J. E., Caron-Flinterman, J. F., and Broerse, J. E. (2011). Patient-expert partnerships in research: how to stimulate inclusion of patient perspectives. Health Expect 14, 225–239. doi: 10.1111/j.1369-7625.2010.00647.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Elberse, J. E., Pittens, C. A., de Cock Buning, T., and Broerse, J. E. (2012). Patient involvement in a scientific advisory process: setting the research agenda for medical products. Health Policy 107, 231–242. doi: 10.1016/j.healthpol.2012.05.014

PubMed Abstract | CrossRef Full Text | Google Scholar

Entwistle, V., Calnan, M., and Dieppe, P. (2008). Consumer involvement in setting the health services research agenda: persistent questions of value. J. Health Serv. Res. Policy 13(Suppl. 3), 76–81. doi: 10.1258/jhsrp.2008.007167

PubMed Abstract | CrossRef Full Text | Google Scholar

Entwistle, V., and O'Donnell, M. (2003). Research funding organisations and consumer involvement. J. Health Serv. Res. Policy 8, 129–131. doi: 10.1258/135581903322029458

PubMed Abstract | CrossRef Full Text | Google Scholar

Fisher, M. (2002). The role of service users in problem formulation and technical aspects of social research. Soc. Work Educ. 21, 305–312. doi: 10.1080/02615470220136885

CrossRef Full Text | Google Scholar

Franck, L. S., McLemore, M. R., Cooper, N., De Castro, B., Gordon, A. Y., Williams, S., et al. (2018). A novel method for involving women of color at high risk for preterm birth in research priority setting. J. Vis. Exp. 131:56220. doi: 10.3791/56220

CrossRef Full Text | Google Scholar

Fudge, N., Wolfe, C. D. A., and McKevitt, C. (2007). Involving older people in health research. Age Ageing 36, 492–500. doi: 10.1093/ageing/afm029

PubMed Abstract | CrossRef Full Text | Google Scholar

Gagnon, M. P., Candas, B., Desmartis, M., Gagnon, J., Roche, D. L., Rhainds, M., et al. (2014). Involving patient in the early stages of health technology assessment (HTA): a study protocol. BMC Health Serv. Res. 14, 273–273. doi: 10.1186/1472-6963-14-273

PubMed Abstract | CrossRef Full Text | Google Scholar

Gagnon, M. P., Desmartis, M., Lepage-Savary, D., Gagnon, J., St-Pierre, M., Rhainds, M., et al. (2011). Introducing patients' and the public's perspectives to health technology assessment: a systematic review of international experiences. Int. J. Technol. Assess. Health Care 27, 31–42. doi: 10.1017/S0266462310001315

PubMed Abstract | CrossRef Full Text | Google Scholar

Gamble, C., Dudley, L., Allam, A., Bell, P., Goodare, H., Hanley, B., et al. (2014). Patient and public involvement in the early stages of clinical trial development: a systematic cohort investigation. BMJ Open 4:e005234. doi: 10.1136/bmjopen-2014-005234

PubMed Abstract | CrossRef Full Text | Google Scholar

Ghisoni, M., Wilson, C. A., Morgan, K., Edwards, B., Simon, N., Langley, E., et al. (2017). Priority setting in research: user led mental health research. Res. Involve. Engage. 3:4. doi: 10.1186/s40900-016-0054-7

PubMed Abstract | CrossRef Full Text | Google Scholar

Gooberman-Hill, R., Horwood, J., and Calnan, M. (2008). Citizens' juries in planning research priorities: process, engagement and outcome. Health Expect 11, 272–281. doi: 10.1111/j.1369-7625.2008.00502.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Green, G., and Johns, T. (2019). Exploring the relationship (and power dynamic) between researchers and public partners working together in applied health research teams. Front. Sociol. 4:20. doi: 10.3389/fsoc.2019.00020

CrossRef Full Text | Google Scholar

Greenhalgh, T., Ovseiko, P. V., Fahy, N., Shaw, S., Kerr, P., Rushforth, A. D., et al. (2017). Maximising value from a United Kingdom biomedical research centre: study protocol. Health Res. Policy Syst. 15:70. doi: 10.1186/s12961-017-0237-1

PubMed Abstract | CrossRef Full Text | Google Scholar

Hailey, D., and Nordwall, M. (2006). Survey on the involvement of consumers in health technology assessment programs. Int. J. Technol. Assess. Health Care 22, 497–499. doi: 10.1017/S0266462306051427

PubMed Abstract | CrossRef Full Text | Google Scholar

Hewlett, S., Wit, M., Richards, P., Quest, E., Hughes, R., Heiberg, T., et al. (2006). Patients and professionals as research partners: challenges, practicalities, and benefits. Arthritis Rheum. 55, 676–680. doi: 10.1002/art.22091

PubMed Abstract | CrossRef Full Text | Google Scholar

Hickey, G., Brearley, S., Coldham, T., Denegri, S., Green, G., Staniszewska, S., et al. (2018). Guidance on Co-producing a Research Project. Southampton: INVOLVE.

Hogg, C. (1999). Patients, Power and Politics: From Patient to Citizens. London: Sage.

Howe, A., MacDonald, H., Barrett, B., and Little, B. (2006). Ensuring public and patient participation in research: a case study in infrastructure development in one UK research and development consortium. Primary Health Care Res. Develop. 7, 60–67. doi: 10.1191/1463423606pc269oa

CrossRef Full Text | Google Scholar

Howe, A., Mathie, E., Munday, D., Cowe, M., Goodman, C., Keenan, J., et al. (2017). Learning to work together – lessons from a reflective analysis of a research project on public involvement. Res. Involve. Engage. 3:1. doi: 10.1186/s40900-016-0051-x

PubMed Abstract | CrossRef Full Text | Google Scholar

Husereau, D., Boucher, M., and Noorani, H. (2010). Priority setting for health technology assessment at CADTH. Int. J. Technol. Assess. Health Care 26, 341–347. doi: 10.1017/S0266462310000383

PubMed Abstract | CrossRef Full Text | Google Scholar

Hutchison, K., Rogers, W., and Entwistle, V. A. (2017). Addressing deficits and injustices: the potential epistemic contributions of patients to research. Health Care Anal. 25, 386–403. doi: 10.1007/s10728-016-0323-5

PubMed Abstract | CrossRef Full Text | Google Scholar

INVOLVE (2012). Tip Sheets: Recruiting Members of the Public to Get Involved in Research Funding and Commissioning Processes. Available online at: http://www.invo.org.uk/wp-content/uploads/2012/04/Recruitment-tips-sheet.pdf (accessed July 9, 2018).

JLA (2018). James Lind Alliance Priority Setting Partnerships. Available online at: http://www.jla.nihr.ac.uk/news-and-publications/psp-articles-and-publications.htm (accessed July 9, 2018).

Google Scholar

Lindenmeyer, A., Hearnshaw, H., Sturt, J., Ormerod, R., and Aitchison, G. (2007). Assessment of the benefits of user involvement in health research from the warwick diabetes care research user group: a qualitative case study. Health Expect 10, 268–277. doi: 10.1111/j.1369-7625.2007.00451.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Madden, M., and Speed, E. (2017). Beware zombies and unicorns: toward critical patient and public involvement in health research in a neoliberal context. Front. Sociol. 2:7. doi: 10.3389/fsoc.2017.00007

CrossRef Full Text | Google Scholar

Manafò, E., Petermann, L., Vandall-Walker, V., and Mason-Lai, P. (2018). Patient and public engagement in priority setting: a systematic rapid review of the literature. PLoS ONE 13:e0193579. doi: 10.1371/journal.pone.0193579

PubMed Abstract | CrossRef Full Text | Google Scholar

Manikam, L., Shah, R., Reed, K., Santini, G., and Lakhanpaul, M. (2017). Using a co-production prioritization exercise involving South Asian children, young people and their families to identify health priorities requiring further research and public awareness. Health Expect 20, 852–861. doi: 10.1111/hex.12524

PubMed Abstract | CrossRef Full Text | Google Scholar

McCormick, S., Brody, J., Brown, P., and Polk, R. (2004). Public involvement in breast cancer research: an analysis and model for future research. Int. J. Health Services 34, 625–646. doi: 10.2190/HPXB-9RK8-ETVM-RVEA

PubMed Abstract | CrossRef Full Text | Google Scholar

Menon, D., and Stafinski, T. (2011). Role of patient and public participation in health technology assessment and coverage decisions. Expert Rev. Pharmacoecon. Outcomes Res. 11, 75–89. doi: 10.1586/erp.10.82

PubMed Abstract | CrossRef Full Text | Google Scholar

Minogue, V., Cooke, M., Donskoy, A.-L., Vicary, P., and Wells, B. (2018). Patient and public involvement in reducing health and care research waste. Res. Involve. Engage. 4:5. doi: 10.1186/s40900-018-0087-1

PubMed Abstract | CrossRef Full Text | Google Scholar

Minogue, V., and Girdlestone, J. (2010). Building capacity for service user and carer involvement in research: the implications and impact of best research for best health. Int. J. Health Care Qual. Assur. 23, 422–435. doi: 10.1108/09526861011037470

PubMed Abstract | CrossRef Full Text | Google Scholar

Moran, R., and Davidson, P. (2011). An uneven spread: a review of public involvement in the national institute of health research's health technology assessment program. Int. J. Technol. Assess. Health Care 27, 343–347. doi: 10.1017/S0266462311000559

PubMed Abstract | CrossRef Full Text | Google Scholar

Morgan, L. J., Chambers, R., Banerji, J., Gater, J., and Jordan, J. (2005). Consumers leading public consultation: the general public's knowledge of stroke. Fam. Pract. 22, 8–14. doi: 10.1093/fampra/cmh709

PubMed Abstract | CrossRef Full Text | Google Scholar

Morris, C., Shilling, V., McHugh, C., and Wyatt, K. (2011). Why it is crucial to involve families in all stages of childhood disability research. Dev. Med. Child Neurol. 53, 769–771. doi: 10.1111/j.1469-8749.2011.03984.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Morrow, E., Cotterell, P., Robert, G., Grocott, P., and Ross, F. (2013). Mechanisms can help to use patients' experiences of chronic disease in research and practice: an interpretive synthesis. J. Clin. Epidemiol. 66, 856–864. doi: 10.1016/j.jclinepi.2012.12.019

PubMed Abstract | CrossRef Full Text | Google Scholar

NIHR (2015). Going the Extra Mile: Improving the Nation's Health and Wellbeing Through Public Involvement in Research. Available online at: https://www.nihr.ac.uk/patients-and-public/documents/Going-the-Extra-Mile.pdf (accessed July 7, 2018).

NIHR BRCU (2017). NIHR Biomedical Research Centres and Units Annual Reports 2016/17. Available online at: https://www.nihr.ac.uk/about-us/how-we-are-managed/managing-centres/nihr-central-commissioning-facility/ccf-ppi/ppie-annual-reports.htm (accessed July 9, 2018).

NIHR CLAHRC (2017). Collaboration for Leadership in Applied Health Research and Care East of England. Patient and Public Involvement in Research Handbook. Available online at: http://www.clahrc-eoe.nihr.ac.uk/wp-content/uploads/2018/01/CLAHRC-EoE-PPI-IN-Research-Handbook_December-2017.pdf (accessed July 8, 2018).

Nilsen, E. S., Myrhaug, H. T., Johansen, M., Oliver, S., and Oxman, A. D. (2006). Methods of consumer involvement in developing healthcare policy and research, clinical practice guidelines and patient information material. Cochrane Database Syst. Rev. 3:Cd004563. doi: 10.1002/14651858.CD004563.pub2

CrossRef Full Text | Google Scholar

O'Donnell, M., and Entwistle, V. (2004). Consumer involvement in decisions about what health-related research is funded. Health Policy 70, 281–290. doi: 10.1016/j.healthpol.2004.04.004

PubMed Abstract | CrossRef Full Text | Google Scholar

Oliver, S. (1996). The progress of lay involvement in the NHS research and development programme. J. Eval. Clin. Pract. 2, 273–280. doi: 10.1111/j.1365-2753.1996.tb00057.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Oliver, S., Armes, D. G., and Gyte, G. (2009). Public involvement in setting a national research agenda: a mixed methods evaluation. Patient 2, 179–190. doi: 10.2165/11314860-000000000-00000

PubMed Abstract | CrossRef Full Text | Google Scholar

Oliver, S., Clarke-Jones, L., Rees, R., Milne, R., Buchanan, P., Gabbay, J., et al. (2004). Involving consumers in research and development agenda setting for the NHS: developing an evidence-based approach. Health Technol. Assess 8, 1–148, iii–iv. doi: 10.3310/hta8150

PubMed Abstract | CrossRef Full Text | Google Scholar

Oliver, S., and Gray, J. (2006). A Bibliography of Research Reports About Patients', Clinicians' and Researchers' Priorities for New Research. London: James Lind Alliance.

Google Scholar

Oliver, S., Liabo, K., Stewart, R., and Rees, R. (2015). Public involvement in research: making sense of the diversity. J. Health Serv. Res. Policy 20, 45–51. doi: 10.1177/1355819614551848

PubMed Abstract | CrossRef Full Text | Google Scholar

Oliver, S., Rees, R. W., Clarke-Jones, L., Milne, R., Oakley, A. R., Gabbay, J., et al. (2008). A multidimensional conceptual framework for analysing public involvement in health services research. Health Expect. 11, 72–84. doi: 10.1111/j.1369-7625.2007.00476.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Parsons, S., Thomson, W., Cresswell, K., Starling, B., and McDonagh, J. E. (2017). What do young people with rheumatic disease believe to be important to research about their condition? a UK-wide study. Pediatr. Rheumatol. Online J. 15:53. doi: 10.1186/s12969-017-0181-1

PubMed Abstract | CrossRef Full Text | Google Scholar

Paylor, J., and McKevitt, C. (2019). The possibilities and limits of “Co-producing” research. Front. Sociol. 4:23. doi: 10.3389/fsoc.2019.00023

CrossRef Full Text | Google Scholar

PCORI (2018). How We Select Research Topics. Available online at: https://www.pcori.org/research-results/about-our-research/how-we-select-research-topics (accessed July 9, 2018).

Pittens, C. A. C. M., Elberse, J. E., Visse, M., Abma, T. A., and Broerse, J. E. W. (2014). Research agendas involving patients: Factors that facilitate or impede translation of patients' perspectives in programming and implementation. Sci. Public Policy 41, 809–820. doi: 10.1093/scipol/scu010

CrossRef Full Text | Google Scholar

Pramesh, C. S., Venkataramanan, R., Suvarna, V., Goel, N. S., Lakshman, S., Venkatesh, V., et al. (2016). Involvement of general public in biomedical research. Perspect. Clin. Res. 7, 152–155. doi: 10.4103/2229-3485.192029

PubMed Abstract | CrossRef Full Text | Google Scholar

Pratt, B., Merritt, M., and Hyder, A. A. (2016). Towards deep inclusion for equity-oriented health research priority-setting: a working model. Soc. Sci. Med. 151, 215–224. doi: 10.1016/j.socscimed.2016.01.018

PubMed Abstract | CrossRef Full Text | Google Scholar

Rawson, T. M., Castro-Sanchez, E., Charani, E., Husson, F., Moore, L. S. P., Holmes, A. H., et al. (2018). Involving citizens in priority setting for public health research: implementation in infection research. Health Expect. 21, 222–229. doi: 10.1111/hex.12604

PubMed Abstract | CrossRef Full Text | Google Scholar

Resnik, D. (2001). Setting biomedical research priorities: justice, science, and public participation. Kennedy Inst. Ethics J. 11, 181–204. doi: 10.1353/ken.2001.0017

PubMed Abstract | CrossRef Full Text | Google Scholar

Rhodes, P., Nocon, A., Booth, M., Chowdrey, M. Y., Fabian, A., Lambert, N., et al. (2002). A service users' research advisory group from the perspectives of both service users and researchers. Health Soc. Care Commu. 10, 402–409. doi: 10.1046/j.1365-2524.2002.00376.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Ridley, J., and Jones, L. Scottish Health Feedback. (2002). User and Public Involvement in Health Services: A Literature Review. Edinburgh: Project Report. NHS Scotland/Scottish Executive.

Rikkers, W., Boterhoven de Haan, K., Lawrence, D., McKenzie, A., Hancock, K., Haines, H., et al. (2015). Two methods for engaging with the community in setting priorities for child health research: who engages? PLoS ONE 10:e0125969. doi: 10.1371/journal.pone.0125969

PubMed Abstract | CrossRef Full Text | Google Scholar

Royle, J., and Oliver, S. (2004). Consumer involvement in the health technology assessment program. Int. J. Technol. Assess. Health Care 20, 493–497. doi: 10.1017/S0266462304001412

PubMed Abstract | CrossRef Full Text | Google Scholar

Shah, S. G., and Robinson, I. (2007). Benefits of and barriers to involving users in medical device technology development and evaluation. Int. J. Technol. Assess. Health Care 23, 131–137. doi: 10.1017/S0266462307051677

PubMed Abstract | CrossRef Full Text | Google Scholar

Sheaff, R., Charles, N., Mahon, A., Chambers, N., Morando, V., Exworthy, M., et al. (2015). NHS commissioning practice and health system governance: a mixed-methods realistic evaluation. Health Serv. Deliv. Res. 3. doi: 10.3310/hsdr03100

CrossRef Full Text | Google Scholar

Shippee, N. D., Domecq Garces, J. P., Prutsky Lopez, G. J., Wang, Z., Elraiyah, T. A., Nabhan, M., et al. (2015). Patient and service user engagement in research: a systematic review and synthesized framework. Health Expect 18, 1151–1166. doi: 10.1111/hex.12090

PubMed Abstract | CrossRef Full Text | Google Scholar

Simpson, S., Cook, A., and Miles, K. (2018). Patient and public involvement in early awareness and alert activities: an example from the United Kingdom. Int. J. Technol. Assess. Health Care 34, 10–17. doi: 10.1017/S0266462317004421

PubMed Abstract | CrossRef Full Text | Google Scholar

Slutsky, J., Tumilty, E., Max, C., Lu, L., Tantivess, S., Hauegen, R. C., et al. (2016). Patterns of public participation: opportunity structures and mobilization from a cross-national perspective. J. Health Organ. Manag. 30, 751–768. doi: 10.1108/JHOM-03-2016-0037

CrossRef Full Text | Google Scholar

Smith, E., Ross, F., Donovan, S., Manthorpe, J., Brearley, S., Sitzia, J., et al. (2008). Service user involvement in nursing, midwifery and health visiting research: a review of evidence and practice. Int. J. Nurs. Stud. 45, 298–315. doi: 10.1016/j.ijnurstu.2006.09.010

PubMed Abstract | CrossRef Full Text | Google Scholar

Smith, E., Ross, F. M., Mackenzie, A., and Masterson, A. (2005). Developing a service-user framework to shape priorities for nursing and midwifery research. J. Res. Nurs. 10, 107–118. doi: 10.1177/136140960501000101

CrossRef Full Text | Google Scholar

Snape, D., Kirkham, J., Britten, N., Froggatt, K., Gradinger, F., Lobban, F., et al. (2014). Exploring perceived barriers, drivers, impacts and the need for evaluation of public involvement in health and social care research: a modified Delphi study. BMJ Open 4:e004943. doi: 10.1136/bmjopen-2014-004943

PubMed Abstract | CrossRef Full Text | Google Scholar

Staley, K. (2009). Exploring Impact: Public Involvement in NHS, Public Health and Social Care Research. Eastleigh: INVOLVE.

Staniszewska, S., Adebajo, A., Barber, R., Beresford, P., Brady, L.-M., Brett, J., et al. (2011). Developing the evidence base of patient and public involvement in health and social care research: the case for measuring impact. Int. J. Consum. Stud. 35, 628–632. doi: 10.1111/j.1470-6431.2011.01020.x

CrossRef Full Text | Google Scholar

Supple, D., Roberts, A., Hudson, V., Masefield, S., Fitch, N., Rahmen, M., et al. (2015). From tokenism to meaningful engagement: best practices in patient involvement in an EU project. Res. Involve. Engage. 1:5. doi: 10.1186/s40900-015-0004-9

CrossRef Full Text | Google Scholar

Thompson, J., Barber, R., Ward, P. R., Boote, J. D., Cooper, C. L., Armitage, C. J., et al. (2009). Health researchers' attitudes towards public involvement in health research. Health Expect 12, 209–220. doi: 10.1111/j.1369-7625.2009.00532.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Tong, A., Chando, S., Crowe, S., Manns, B., Winkelmayer, W. C., Hemmelgarn, B., et al. (2015). Research priority setting in kidney disease: a systematic review. Am. J. Kidney Dis. 65, 674–683. doi: 10.1053/j.ajkd.2014.11.011

PubMed Abstract | CrossRef Full Text | Google Scholar

Truitt, A. R., Monsell, S. E., Avins, A. L., Nerenz, D. R., Lawrence, S. O., Bauer, Z., et al. (2018). Prioritizing research topics: a comparison of crowdsourcing and patient registry. Qual. Life Res. 27, 41–50. doi: 10.1007/s11136-017-1566-9

PubMed Abstract | CrossRef Full Text | Google Scholar

van Bekkum, J. E., Fergie, G. M., and Hilton, S. (2016). Health and medical research funding agencies' promotion of public engagement within research: a qualitative interview study exploring the United Kingdom context. Health Res. Policy Syst. 14:23. doi: 10.1186/s12961-016-0093-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Viswanathan, M., Ammerman, A., Eng, E., Garlehner, G., Lohr, K. N., Griffith, D., et al. (2004). Community-based participatory research: assessing the evidence. Evid. Rep. Technol. Assess. (Summ). 99:1–8. doi: 10.1037/e439622005-001

CrossRef Full Text | Google Scholar

Willis, K. (1995). Imposed structures and contested meanings: Policies and politics of public participation. Austr. J. Soc. Issues. 30, 211–227. doi: 10.1002/j.1839-4655.1995.tb01036.x

CrossRef Full Text | Google Scholar