The impact of the European Association of Endoscopic Surgery research grant scheme—a mixed qualitative quantitative methodology study protocol

1. Introduction

The European Association of Endoscopic Surgery (EAES) is a surgical society who promotes the development and expansion of minimally invasive surgery to surgeons and surgical trainees (1). It does so through its activities in education, training, and research. The EAES research committee aims to promote the highest quality clinical research in endoscopic and minimally invasive surgery. They have provided grant funding since 2009 in education, surgery, and basic science (2). Despite the success and longevity of the scheme, the academic and non-academic impact of the research funding scheme has not been evaluated.

The importance of establishing the impact of a research funding scheme is multifaceted as clinicians, policymakers, and researchers benefit from strategic investment to promote high-quality research (3). Research funding has been found to contribute directly to successful academic publications and conference presentations, and facilitate progression of successful applicants into academic career paths (4, 5). Correspondingly, research stakeholders and independent charities have introduced funding policies to promote what is frequently referred to as scientific excellence (6). Providing competitive funding based on the potential to produce high quality academic output is a popular method of funds allocation (7), but this may to some extent ignore the societal relevance and the need for research to produce positive change beyond the constraints of academia (6).

The established research funding process based on the notion of “excellence”, is used widely, at least in the western world (6); however, it has occasionally been under scrutiny. Authors argue that the concept of excellence is dependent upon whomever defines it, as a validated definition of this within academia does not exist (8, 9). Others believe that it promotes a culture of competition and an outcome focused ethos, often at odds with impactful research outputs (10). Moreover, there have been reports that up to 85% of clinical, health services, and basic science research is avoidably wasted (11).

The Economic and Social Research Council (ESRC) defines research impact as “the demonstrable contribution that excellent research makes to society and the economy” (12). The impact of research can be divided into: Instrumental impact, i.e., influencing policy, practices or services, conceptual impact; understanding policy matters and reframing debate and building capability through technical or personal skill development (12).

Although several models have been proposed for assessing research impact (13), a review by Raftery et al. (14) identified the “payback model” by Buxton and Hanney (15) as the dominant one. Impact according to this model focuses in four areas:

(i) Knowledge production (e.g., publications, presentations),

(ii) Research capacity building (e.g., training new academics)

(iii) Informing policy and product development (e.g., influencing policies and guidelines)

(iv) Health system benefits (e.g., cost savings), economic benefits (e.g., commercial spin-outs) (15).

This model combines areas of academic excellence such as presentations and publications, as well as impact outside academia (e.g., informing policy and economic benefits) (15).

Though identifying the impact of research is important, it is equally vital to identify factors promoting or impeding high-quality research (3). A Lancet series recommends that efforts should be made to identify factors associated with successful research initiatives, recognising the importance of ensuring that research investments yield productive initiatives (11). Farrokhyar et al. (3) showed that assessing the scientific productivity of organisational grant funding can identify the characteristics of grant awardees, as well as the influence of funding on career growth (3).

The International School on Research Impact Assessment (ISRIA) was established by research organisations recognising the need for research impact strategies and uniform assessment efforts based on the robust and reproducible evaluation methods (16). ISRIA developed a series of best practice guidelines (17), encouraging organisations to (1) analyse the research context, (2) reflect on the purpose of RIA, (3) identify stakeholders' needs, (4) engage with the research community, (5) choose appropriate conceptual frameworks, (6) choose appropriate evaluation methods and data sources, (7) choose indicators and metrics responsibly, (8) consider ethics and conflicts of interest, (9) communicate results, and (10) share learning (17).

In line with the ISRIA guidelines the EAES research committee is embarking on this project aiming to evaluate the academic and non-academic impact of their research funding scheme and to identify barriers and facilitators for successful completion of research projects and for achieving the good impact.

For this project that is kindly funded by EAES; The aims and objectives of the project are as follows:

• Primary Aim: Assess the short, long and real world impact of the EAES funding scheme

○ Objective 1: Assess the academic impact of the EAES research funding scheme

○ Objective 2: Assess the non-academic impact of the EAES research funding scheme

• Secondary Aim: Identify barriers and facilitators for achieving good impact.

○ Establish the completion rate of projects funded

○ Identify facilitators for achieving high impact

○ Identify barriers for the completion of research projects and hence achieving high impact

2. Methods and analysis

Research impact can be established with in-depth analysis of the areas described above which can be roughly categorised to academic impact and impact outside of academia. Such an attempt would involve both qualitative and quantitative methods. This would include semi-structured interviews with researchers (18), questionnaires and assessing objective, quantified measures of impact (e.g., number of publications). In addition to the above, through mixed methodology, further impact areas can be identified.

This will be a mixed qualitative, quantitative study, as it will encompass both semi-structured interviews and a questionnaire.

The study will be performed in several stages.

2.1. Stage 1: preparing the questionnaire

The survey will be designed with the Checklist for Reporting Results of Internet E-Surveys (CHERRIES) developed by the Enhancing the QUAlity and Transparency Of health Research Network (Equator) (19) in mind. This consists of eight topics:

1. Design: The target population will be previous recipients of the EAES research grants. They will be invited directly via email.

2. Institutional approval, informed consent and data protection: This project has been approved by Hull and York Medical School Ethics Committee. Participants will be kindly requested to sign an informed consent. It will be made clear that participation is voluntary and that they can withdraw from the project whenever they wish, without having to provide a reason. However, any data they may have provided up to the point of withdrawal may be used in the final analysis. No identifiable data will be stored and the responses to the survey will be anonymous.

3. Development of questionnaire. Based on the four areas proposed above, the steering committee of this project is expected to reach a consensus on questionnaire questions. The process will involve each member of the steering committee independently reviewing a pre-drafted list of questions. Supplementary Appendix 1 demonstrates the list of draft questions for the committee's review. The committee will then have a series of virtual meetings, until an absolute (100%) consensus is reached. The number of questions is not rigidly defined, therefore, if the members of the steering group feel that questions need to be added or removed, this can be done after the necessary consensus is reached.

4. Recruitment process and description of the sample having access to the questionnaire. The participants will be recruited after email invitation through the EAES mailing list.

5. Survey administration. The study will be created and disseminated through Google® docs. The link of the survey will be shared with investigators via email. As mentioned above, participation will be voluntary. Participants will be given an incentive to complete the survey, as one of the responders will be randomly selected for a travel grant to the next EAES conference. The survey will remain open for 4 weeks. The number of items will be determined at the designing process however, the entirety of the questionnaire will be in one page only. This will allow participants the ability to go back and change their answers, they can also scroll back and review the final version of their answers before they submit the completed questionnaire.

6. Response rate. This will be easily calculated as the proposed participant number will be known at the beginning of the survey.

7. Preventing multiple entries from the same individual. Each responder will have to provide details about their project therefore, duplication is unlikely. However, IP identification will be used to ensure there is no duplication no two entries from the same IPs will be allowed.

8. Analysis. Incomplete questionnaires are not expected as all questions requiring an answer are made compulsory, therefore the user cannot progress to the next question unless the previous one was answered. The answers will be extracted in an Excel (Microsoft®) sheet, analysed and presented with appropriate descriptive statistics.

2.2. Stage 2: delivering the questionnaire

Grant recipients of the past years will be invited to answer the questionnaire. This will be done through the emailing list of EAES. The participants will be asked whether they would like to volunteer to participate in phase 2 which will be semi-structured interviews aiming to further define questionnaire answers.

2.3. Stage 3: preparation and delivery of semi-structured interviews

The steering committee of the project will be asked to review the raw data from the survey results and suggest 5 questions which they feel will further define the results of the survey. Once consensus is established, semi-structured interviews will be conducted with 6–10 randomly selected survey responders stemming from the novel method of information power for sample sizing in qualitative research methodology by Malterud and colleagues (20). The interviews will be conducted by researchers with previous relevant experience.

The transcripts of the interviews will be transcribed verbatim and analysed by two independent assessors (Marina Yiasemidou, Judith Long) who have previous experience with such analysis.

Target population:

Past EAES Grant recipients will be asked to reply to the questionnaire developed by the project's steering committee. Responders will be invited to participate in semi-structured interviews. Participation will be voluntary.

2.4. Data collection and analysis

2.4.1. Survey

Data collection will be performed using a bespoke spreadsheet. Besides the answers of the survey, demographics and project characteristics will be collected.

The results of the survey will be reported according to the Checklist for Reporting Results of Internet E-Surveys (CHERRIES) developed by the Enhancing the QUAlity and Transparency Of health Research Network (Equator) (19).

2.4.2. Semi-structured interviews—thematic analysis

The thematic analysis of the semi-structured interviews will be completed in the following steps: (i) familiarisation (ii) coding (iii) generating themes (iv) reviewing themes (v) naming themes (vi) write up of results (18). Familiarisation will occur during the transcribing of the interviews. Coding will involve highlighting sections of the transcript and generating “codes” to describe their content. For example, if the transcript states, “I am not sure” the code can be “uncertainty”. During the next step of generating themes, the codes created will be assessed for patterns amongst them. This process will generate themes; a broader concept than codes. On most occasions, several codes will be combined into a single theme (18). Following this, the themes will be reviewed by the two assessors (MY, JL) to ensure that they are useful and accurate representations of the data.

This may result in themes being divided, combined, discarded or generated. This process will be ultimately validated by the steering committee. The themes developed will then need to be named in a succinct and precise way. The last step of the process will be the write up of the report, which will be prepared and delivered to the steering committee for careful scrutiny (18).

2.5. Analysis

2.5.1. Semi-structured interviews—thematic analysis

The themes will be reviewed by the two assessors (MY, JL), a report will be generated and presented to the steering committee. The entire process will be overseen and validated by the steering committee.

2.5.2. Questionnaire

The results of the questionnaire will be presented using descriptive statistics such as frequencies and percentages.

3. Discussion

This study will determine whether the EAES research funding scheme had a significant positive impact on research output, career progression but also non-academic output such as change in clinical guidelines, healthcare quality and cost-effectiveness improvement.

This project is also expected to identify facilitators and barriers to successful completion of projects and to achieving high impact. This will inform EAES and the rest of the surgical and academic communities as to how clinicians would like to be supported when conducting research. There should also be a positive and decisive change towards removing factors that hinder the timely and impactful completion of projects. A final report with a list of recommendations to address key issues derived from the questionnaire and semi-structured interviews will be shared with the organizers of the EAES trainee grant funding committee.

Ethics statement

The studies involving human participants were reviewed and approved by Hull York Medical School. The patients/participants provided their written informed consent to participate in this study.

Author contributions

AM, BH, JK, JL, MM, JW, MY conception and design of study and preparation and editing of manuscript. NF, TA, IC, MY supervision and editing of manuscript. All authors contributed to the article and approved the submitted version.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fsurg.2023.1197103/full#supplementary-material

References

1. Mahendran B, Celentano V, Soltes M, Popa D, Adamina M, Sanz CM, et al. EAES online educational resources: a survey of the membership of the European association for endoscopic surgery (EAES). Surg Endosc. (2021) 35(5):2059–66. doi: 10.1007/s00464-020-07602-8

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Francis N, Kazaryan AM, Pietrabissa A, Goitein D, Yiannakopoulou E, Agresta F, et al. A research agenda for the European association for endoscopic surgeons (EAES). Surg Endosc. (2017) 31(5):2042–9. doi: 10.1007/s00464-017-5531-z

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Farrokhyar F, Bianco D, Dao D, Ghert M, Andruszkiewicz N, Sussman J, et al. Impact of research investment on scientific productivity of junior researchers. Transl Behav Med. (2016) 6(4):659–68. doi: 10.1007/s13142-015-0361-9

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Shalhoub J, Cromwell D, Coomer M, Alderson D. The impact of the royal college of surgeons of England research fellowship scheme. Ann R Coll Surg Engl. (2016) 98(6):431–5. doi: 10.1308/rcsann.2016.0163

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Hure AJ, Ramanathan S, Deeming S, Figtree GA, Jennings G, Webster J, et al. Translation and impact of funded Australian cardiovascular research: a review with perspective. Heart Lung Circ. (2021) 30(10):1442–8. doi: 10.1016/j.hlc.2021.04.025

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Scholten W, Franssen TP, van Drooge L, de Rijcke S, Hessels LK. Funding for few, anticipation among all: effects of excellence funding on academic research groups. Sci Public Policy. (2021) 48:265–75. doi: 10.1093/scipol/scab018

CrossRef Full Text | Google Scholar

7. Cremonini L, Horlings E, Hessels LK. Different recipes for the same dish: comparing policies for scientific excellence across different countries. Sci Public Policy. (2018) 45:232–45. doi: 10.1093/scipol/scx062

CrossRef Full Text | Google Scholar

8. JS. Against excellence. The guardian (2014).

9. Science needs to redefine excellence. Nature. (2018) 554(7693):403–4. doi: 10.1038/d41586-018-02183-y

CrossRef Full Text | Google Scholar

10. Moore S, Neylon C, Paul Eve M, O’Donnell DP, Pattinson D. “Excellence R US”: university research and the fetishisation of excellence. Palgrave Commun. (2017) 3:16105. doi: 10.1057/palcomms.2016.105

CrossRef Full Text | Google Scholar

11. Moher D, Glasziou P, Chalmers I, Nasser M, Bossuyt PMM, Korevaar DA, et al. Increasing value and reducing waste in biomedical research: who’s listening? Lancet. (2016) 387(10027):1573–86. doi: 10.1016/S0140-6736(15)00307-4

PubMed Abstract | CrossRef Full Text | Google Scholar

12. (ESRC) EaSRC. Defining impact (2022). Available at: https://www.ukri.org/councils/esrc/impact-toolkit-for-economic-and-social-sciences/defining-impact/

13. Boulding H, Kamenetzky A, Ghiga I, Ioppolo B, Herrera F, Parks S, et al. Mechanisms and pathways to impact in public health research: a preliminary analysis of research funded by the national institute for health research (NIHR). BMC Med Res Methodol. (2020) 20(1):34. doi: 10.1186/s12874-020-0905-7

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Raftery J, Hanney S, Greenhalgh T, Glover M, Blatch-Jones A. Models and applications for measuring the impact of health research: update of a systematic review for the health technology assessment programme. Health Technol Assess. (2016) 20(76):1–254. doi: 10.3310/hta20760

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Buxton M, Hanney S. How can payback from health services research be assessed? J Health Serv Res Policy. (1996) 1(1):35–43. doi: 10.1177/135581969600100107

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Kamenetzky A, Hinrichs-Krapels S. How do organisations implement research impact assessment (RIA) principles and good practice? A narrative review and exploratory study of four international research funding and administrative organisations. Health Res Policy Syst. (2020) 18(1):6. doi: 10.1186/s12961-019-0515-1

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Adam P, Ovseiko PV, Grant J, Graham KEA, Boukhris OF, Dowd AM, et al. ISRIA statement: ten-point guidelines for an effective process of research impact assessment. Health Res Policy Syst. (2018) 16(1):8. doi: 10.1186/s12961-018-0281-5

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Methodology thematic analysis. Available at: https://www.scribbr.com/methodology/thematic-analysis/ (Accessed November 9, 2020).

19. Eysenbach G. Improving the quality of web surveys: the checklist for reporting results of internet E-surveys (CHERRIES). J Med Internet Res. (2004) 6(3):e34. doi: 10.2196/jmir.6.3.e34

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Malterud K, Siersma VD, Guassora AD. Sample size in qualitative interview studies: guided by information power. Qual Health Res. (2016) 26(13):1753–60. doi: 10.1177/1049732315617444

PubMed Abstract | CrossRef Full Text | Google Scholar