One Coin, Two Sides: Eliciting Expert Knowledge From Training Participants in a Capacity-Building Program for Veterinary Professionals

Introduction

Scientific research may include non-academic participants in the research process to improve the quality and impact of studies (1–3). There are many paradigms, methodologies, and purposes for utilizing such approaches. This paper focuses on the elicitation of knowledge from subject matter experts, whose estimation or judgment of fact-based matters is used to answer the research question (3, 4). This approach is utilized when available data are scarce, unrepresentative, or inadequate to describe the processes and systems being studied. “Expert” in this usage refers to a person who can provide information about the question based on their experience with the subject matter of interest (5, 6).

Expert elicitation is increasingly common within veterinary science, although used less frequently than in other fields. A search on Web of Science for “expert knowledge” OR “expert elicitation” OR “expert judgment” returned 60 articles (out of 708,779) within the category of Veterinary Sciences, 30 of which were published since 2017. When accounting for the total number of articles in each Web of Science category, the same search string occurred 10 times more frequently within Environmental Sciences (1,232/1,489,989) and 12 times more frequently for Ecology (599/591,636). The purposes of expert knowledge in veterinary publications include estimation of parameter values (7, 8), ranking of risk factors or criteria (9–11), enhancing or interpreting available data (12–14), or developing an instrument for use by practitioners (15, 16). Many applications are in data-scarce environments, but there are also cases where expertise is used to make sense of or add rigor to abundant or heterogeneous data sources (14, 17).

When expert knowledge is utilized as a source of information, there are limitations and potential pitfalls (18). People have restricted mental models, poor causal reasoning, and are prone to a litany of biases (4, 19). Estimating probabilities and quantifying uncertainty require training distinct from subject matter expertise (4). Rigorous and structured procedures for participant selection, knowledge elicitation and interpretation, and study validation are crucial to ensure the quality of study conclusions (3, 4, 20).

Structured procedures and training of participants can help to alleviate bias but may be inconvenient or impractical, especially when working with subject matter experts from outside of academia. Elicitations may be carried out in a restricted time period (e.g., embedded within a workshop or conference) or through long-distance interactions. Including participants who are “boots on the ground” practitioners or community members can be challenging if they have limited time available for the activity and a steeper learning curve with respect to the research and elicitation methods. Subject matter experts may not have an academic understanding of the techniques being used, which can impede effective communication and impact the quality of the results if adequate training is not provided.

In this paper, we present a case of expert elicitation embedded within an extended training course for veterinary professionals as an example of overcoming some of these constraints. The coupling of the two activities may create a synergy between research and training which enriches the outcomes and expands the impact of each component, creating a whole greater than the sum of the parts. First, we give a brief overview of the training program, research objectives, and expert elicitation activities performed. Then, we describe the observed outcomes and character of this approach, perceived to be beneficial and synergistic. Finally, we discuss considerations for future opportunities.

Research and Training Overview

The research objective was to quantify and analyze the risk for transmission of foot and mouth disease (FMD) associated with the export of beef produced in Kenyan and Ugandan cattle systems. FMD is a highly infectious transboundary disease of cattle and other livestock and wildlife species (21) and is endemic to East African countries (22, 23). In order to model that risk, it was necessary to understand the underlying processes and the values of key variables. Most of those data are not published; people who work in those beef cattle systems provided expertise and guidance to build, quantify, and validate the risk assessment model.

The elicitation was carried out within 2 concurrent cohorts of ProgRESSVet: a systematic education program for building professional capacity of veterinarians in Kenya and Uganda delivered by the University of Minnesota Center for Animal Health and Food Safety (CAHFS) (24). Participants for the program in each country were required to have a degree in veterinary medicine and experience in the field. There were 13 veterinarians from Kenya, with an average of 13 (range of 2–29) years of experience working in animal health and/or production. The Ugandan cohort had 10 participants, with an average of 7 years of experience (range 2–15 years).

ProgRESSVet training programs are tailored to address gaps identified in the OIE (World Organization for Animal Health) Performance of Veterinary Services Pathway (25) for each country or region of implementation. The programs are designed per Fink (26) to build individual capacity to generate lasting change in participants, thereby building the technical, collaborative, and systems-thinking capacity of the Veterinary Services (VS) to ultimately improve the health and well-being of the communities and countries where they work (24). ProgRESSVet was first offered in 2017 and in 2018 in the Latin American region. ProgRESSVet Uganda and ProgRESSVet Kenya were launched in 2020, incorporating new educational elements based on results of formative and summative education evaluation from the previous Latin America program.

The guided risk assessment and elicitation was one of three activities integrated into the curriculum, which we called Test Drives (Figure 1). The Test Drives included participants in the process of data collection and synthesis about questions relevant to their own communities without requiring them to autonomously direct their own analyses. These activities were conceptualized to achieve research objectives during the challenges of covid-19 restrictions and were then recognized as an opportunity to support knowledge application.

FIGURE 1

Figure 1. Research and training activities were carried out within 2 concurrent cohorts of ProgRESSVet: one in Kenya and one in Uganda. Participants completed 5 months of online coursework followed by 6 months developing proposals to support the trade of animals and animal products. The guided risk assessment was one of three “Test Drive” activities integrated into the curriculum which included participants in the process of data collection and synthesis about questions relevant to their own communities. The Test Drives, including the guided risk assessment, were part of the training program; participants could opt in for their contributions to be used for research purposes. All training and research activities were carried out separately in each country.

Prior to the Test Drives, including the guided risk assessment, participants had completed 5 months of online coursework (Figure 1), including modules on risk analysis applied to animal health, food safety, and international trade. For the next 6 months, participants would develop proposals to support the trade of animals and animal products. Each portion of the training was structured and delivered by the same team of researchers and faculty. The guided risk assessment was part of the training program; participants could opt in for their contributions to be used for research purposes and 100% of enrolled individuals in each country chose to do so.

The details of the elicitation procedures and results are described elsewhere (27). The approach followed a modified version of the Delphi method, a technique for obtaining the consensus of a group of experts (28), and was carried out independently with the participants from Uganda and from Kenya (n = 10 and n = 13, respectively). First, participants individually worked through a series of open-ended questions in which they described the system, identified important variables and relationships, and critiqued a preliminary scenario tree and risk model structure. Next, also individually, they estimated the distributions for key parameter values. Both questionnaires are included as Supplementary Materials 1, 2. Those responses were synthesized and presented in a group discussion with each cohort in order to reach consensus on the meaning and values of key variables. Each participant received a final report with an accessible summary of the discussion and had the opportunity to comment with any additional suggestions or concerns.

Research Process and Outcomes

The novelty of this approach was the use of an education program to support the elicitation activity and research objectives. Structured protocols recommend training experts in the elicitation approach and rationale being used (3, 29). Such training is thought to reduce apprehension, increase understanding of the process, provide motivation, identify biases among the experts, and provide guidelines for working between the facilitators and experts (4). However, practical constraints may preclude the incorporation of training into the research activities.

By embedding the elicitation within an extended educational program, several of these objectives were achieved. After 6 months of partnership (including adaptations on both sides to continue the program through covid-19 uncertainty), the experts (veterinary participants) and researchers (education team) had a collaborative working relationship with established norms and patterns. The researchers supported the participants in developing proposal ideas, which may have helped to convey the team's interest and investment in the individual and institutional impact to result from the program. The participants in each country knew one another through interactive ProgRESSVet activities, including pre-covid in-person workshops and a program discussion thread on the WhatsApp platform.

The education program also provided subject matter training for the exercise. The participants discussed the importance of the problem (the control challenges and trade repercussions of endemic FMD) throughout the courses. The curriculum included 5 weeks on risk analysis including probability and scenario trees, and the elicitation activities included supplemental training on these topics. The participants were well-versed in both “the how” and “the why” of the research question.

The ongoing engagement (in contrast to a single day or workshop) enabled an iterative process of elicitation, consultation, and consensus. Participants allocated a suggested 6–10 h per week to the program and were offered continuous professional development credit. This may have increased their motivation and time available to submit thorough and thoughtful responses. And the platform of a training program supported inclusion of expert participants who were on-the-ground practitioners across a variety of regions and roles in Kenyan and Ugandan livestock systems.

The attributes of the data collected—elicited, analyzed, and evaluated separately for Kenya and Uganda—reflects the value of this approach. Responses provided extensive descriptions of cattle health, production, and handling relevant to the research question. Candid discussions reflected participant perspectives of how the animal health system does work, not merely how it should work, including contrasts between distinct settings (e.g., feedlot vs. pastoralist). They provided insights about causal relationships based on firsthand experience, including the actions, motivations, and incentives of key actors. Participants took the option of responding “no answer” to some questions and/or focusing on specific production systems, suggesting to the researchers that they did not feel pressured to provide information beyond the extent of their experience.

As a result, valuable parameters were quantified by expert knowledge where there otherwise were no available data, and participant expertise improved the structure and specification of the risk model used to represent the system (17). Participants contributed information that otherwise may have been neglected and corrected errors in the researcher's thinking. For example, they highlighted the need to specify both disease diagnosis and appropriate follow-up action to define infected cattle as detected. They described scenarios in which the sale of cattle for meat may be correlated with the probability of having disease, and consequently an additional set of parameters was included to represent disease prevalence among animals which had been sold (rather than assuming animals chosen for sale would be selected at random). Both of these issues were raised by multiple individuals in each country.

Synergistic Character

This coupled approach of training and expert elicitation yielded benefits beyond the research results. We would characterize the elicitation in this context as synergistic learning (26), complementing and enhancing the educational material rather than “stealing time” away from training. The Test Drives are intended to contribute to ProgRESSVet learning objectives by enabling participants to apply the tools presented to their own work, to have an expanded view of food systems and their roles, and to value the critical use of evidence for decision-making.

Participant responses to the end-of-program evaluation (supplied anonymously) support the perceived value of the Test Drive activities in contributing to these objectives. Several respondents said they had already applied the principles and skills from the Test Drives, including for work related to covid-19, animal disease control strategies, enhanced safety of meat, managing animal health challenges with limited resources, and even for embarking on a family project. Others commented on changes in their perspectives, including how to consider stakeholders affected by an issue, new understanding of regional and international trade, the multidimensional nature of livestock health challenges, finding common ground among partners with diverse perspectives, and sharing knowledge with other members of a One Health district task force. [The program evaluation asked about the suite of three Test Drive activities as a whole, so these responses describe skills and perspective garnered from the guided risk assessment as well as two other applied activities whose outputs were not used for research (Figure 1)].

We believe the impact of this approach can transcend that of elicitation or training activities alone to produce benefits for the research and training team, the participants and their community, and the network of both (Figure 2). The experience and insights have contributed to the evolving culture of practice and specifically the education and training model at CAHFS: reinforcing and clarifying the ProgRESSVet approach as a collaborative engagement with peers from a diverse set of background experiences, cultures, and knowledge, focused on meeting local needs through building local capacity. The hope and intention is that participants were empowered by generating and synthesizing shared knowledge about the problems and processes studied, building individual and institutional capacity to address specific and unknown future challenges. Finally, the engagement helped create a network of professionals from both the university and Veterinary Services who can continue to work and learn together.

FIGURE 2

Figure 2. Benefits of combined elicitation and training activity embedded within an education program. Level 1: Both objectives (elicitation and education) can be achieved within a single activity. Level 2: Each attribute (elicitation and education) of the activity enhances the other, contributing to improved achievement of each. For example, the coupling of the two activities enabled extended opportunities for subject-matter training and a relationship of mutual respect to be the foundation for the elicitation process. The participatory research activities reinforced knowledge synthesis objectives of the educational program. Level 3: The synergy between the two concurrent objectives may produce benefits which transcend either independent activity: solutions and ideas built by local professionals, evolving collaborative research and training approaches, and a network of diverse academic and practicing professionals.

Future offerings of the ProgRESSVet curriculum will maintain the Test Drive approach and the education team will continue reporting related educational modifications and outputs pursuant to a robust understanding of the method's potential.

Discussion

The coupling of research activities with capacity-building of health professionals has been applied previously (30, 31), though we have not seen a model in which the same individuals occupy the role of both trainees and contributors of expert knowledge. The ProgRESSVet and Test Drive approach is unique in that expert elicitation activities are embedded and structurally scaf folded within a broader training program, serving to complement the capacity-building objectives while eliciting and activating the expertise of the participants.

We believe this is a valuable approach with flexibility to adapt to particular settings and constraints. However, it is important to be aware of limitations or potential pitfalls. For example, in our case the experts were all veterinarians and nearly all employed in the public sector. A wider diversity of value chain actors would have provided more perspectives contributing to the research and to the discussion of local issues among participants (5). Our structured elicitation and consensus process was heavily facilitated; a constructivist approach with a more open-ended, participant-driven dialogue would favor a different paradigm of research themes and shared learning (2, 32).

The design and implementation of a similar program will require evaluation of the components (the participants, training, and research or elicitation activities) and how they fit together. Practitioners should weigh the value and tradeoffs of possible program designs, considering available resources, existing infrastructure, and their highest priority objectives. The research requiring participant input needs to be carefully aligned with participant expertise and experience. The type and scope of participatory research activity should be guided by the educational approach in order to complement other training elements. The research activity must be realistic given the duration of the training program and the relationships that will be established before launching the elicitation exercises. Time and effort required (of the participants and of the academic team) should be considered, including sequential or iterative steps for the research process.

As with any research method, it is critical to use systematic and robust methods for expert elicitation in order to obtain results that can withstand “close interrogation” and “independent validation,” two facets of reproducibility (33, 34). Rigorous approaches emphasize the inclusion of multiple and diverse experts and the use of a structured protocol for the phases of knowledge elicitation, aggregation, and validation (6, 29); the specific character of those methods may be situation-specific (35, 36). There is much yet to be studied about the nature of expert elicitation approaches that alleviate bias to obtain accurate and well-calibrated results (4).

Research studies that embed expert elicitation into a training program as described here should be designed to produce rigorous results, and may have opportunities to validate those results through repetition over multiple training cohorts. In addition, it may be possible to assess the impact of the coupled approach on the quality of research outputs, furthering the field's understanding of the practice and methodology of expert elicitation (4). For example, the impact on quantitative parameter estimates could be studied in the future by eliciting the parameterization from each participant before and after the training program. Another area of research could be to assess the relationship between responses and certain features of the participants (e.g., gender, age, years of experience). It may be expected that the training approach results in less variation in the responses, compared to gathering data in the absence of a training program, and may be less biased by external factors.

We have demonstrated the opportunity to gather information from subject matter experts in a way that enhances the research process and outputs while at the same time educating and training participants. In our experience, combining both objectives in a single set of activities served to reinforce each component. The participants, before their formal role as “experts,” were trained in the methods and rationale of risk analysis and had developed a relationship of mutual respect with the academic team members. Conversely, the experience of switching roles and interacting (with the subject matter and with each other) in a new way provided an opportunity for significant learning for the participants, pushing them beyond consumption of information or hypothetical scenarios into a realm of application to their actual communities and challenges, while able to sit in the seat of expertise to “test drive” research and analytic methodologies without the full expectation of designing and managing a project on their own. This combined approach has the potential to generate benefits for the academic team as well as the participants and their communities that transcend what any individual activity or institution would produce alone.

Data Availability Statement

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author/s.

Ethics Statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

JA drafted the manuscript. JA, MP, AP, and MO'B contributed to manuscript revision, read, and approved the submitted version. All authors contributed to the development and implementation of the project described.

Funding

This project has been supported in part by the Bill & Melinda Gates Foundation (Grant No. OPP1211169). Under the grant conditions of the Foundation, a Creative Commons Attribution 4.0 Generic License has already been assigned to the author accepted manuscript version that might arise from this submission. Additional support was received from the USDA National Needs (Grant No. 2014-38413-21825) and from the MnDRIVE Global Food Ventures Graduate Student Professional Development award.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

The authors would like to acknowledge and thank the veterinarians who participated in the 2020 cohorts of ProgRESSVet Uganda and ProgRESSVet Kenya for their partnership and contributions to the activities described.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fvets.2021.729159/full#supplementary-material

Supplementary Material 1. Short answer survey.

Supplementary Material 2. Quantitative survey.

References

1. Lang DJ, Wiek A, Bergmann M, Stauffacher M, Martens P, Moll P, et al. Transdisciplinary research in sustainability science: practice, principles, and challenges. Sustain Sci. (2012) 7:25–43. doi: 10.1007/s11625-011-0149-x

CrossRef Full Text | Google Scholar

2. Balazs CL, Morello-Frosch R. The three Rs: how community-based participatory research strengthens the rigor, relevance, and reach of science. Environ Justice. (2013) 6:9–16. doi: 10.1089/env.2012.0017

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Knol AB, Slottje P, Van Der Sluijs JP, Lebret E. The use of expert elicitation in environmental health impact assessment: a seven step procedure. Environ Health. (2010) 9:1–16. doi: 10.1186/1476-069X-9-19

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Hanea AM, Hemming V, Nane GF. Uncertainty quantification with experts: present status and research needs. Risk Anal (2021). doi: 10.31219/osf.io/vk3bf. [Epub ahead of print].

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Burgman M, Carr A, Godden L, Gregory R, McBride M, Flander L, et al. Redefining expertise and improving ecological judgment. Conserv Lett. (2011) 4:81–7. doi: 10.1111/j.1755-263X.2011.00165.x

CrossRef Full Text | Google Scholar

6. Burgman MA, McBride M, Ashton R, Speirs-Bridge A, Flander L, Wintle B, et al. Expert status and performance. PLoS ONE. (2011) 6:1–7. doi: 10.1371/journal.pone.0022998

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Verdugo C, Zimin-Veselkoff N, Gardner IA. Mardones FO. Expert elicitation of the diagnostic performance of two tests for Bacterial Kidney Disease (BKD) surveillance in Atlantic salmon (Salmo salar L) broodstock in Chile. Aquaculture. (2020) 525:735274. doi: 10.1016/j.aquaculture.2020.735274

CrossRef Full Text | Google Scholar

8. Beck-Johnson LM, Hallman C, Miller RS, Portacci K, Gorsich EE, Grear DA, et al. Estimating and exploring the proportions of inter- and intrastate cattle shipments in the United States. Prev Vet Med. (2019) 162:56–66. doi: 10.1016/j.prevetmed.2018.11.002

PubMed Abstract | CrossRef Full Text | Google Scholar

9. McEachran MC, Sampedro F, Travis DA, Phelps NBD. An expert-based risk ranking framework for assessing potential pathogens in the live baitfish trade. Transbound Emerg Dis. (2020). doi: 10.1111/tbed.13951. [Epub ahead of print].

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Muellner P, Hodges D, Ahlstrom C, Newman M, Davidson R, Pfeiffer D, et al. Creating a framework for the prioritization of biosecurity risks to the New Zealand dairy industry. Transbound Emerg Dis. (2018) 65:1067–77. doi: 10.1111/tbed.12848

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Brookes VJ, Ward MP. Expert opinion to identify high-risk entry routes of canine rabies into Papua New Guinea. Zoonoses Public Health. (2017) 64:156–60. doi: 10.1111/zph.12284

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Squarzoni-Diaw C, Arsevska E, Kalthoum S, Hammami P, Cherni J, Daoudi A, et al. Using a participatory qualitative risk assessment to estimate the risk of introduction and spread of transboundary animal diseases in scarce-data environments: a spatial qualitative risk analysis applied to foot-and-mouth disease in Tunisia 2014–2019. Transbound Emerg Dis. (2020) 68:1966–78. doi: 10.1111/tbed.13920

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Sharifi I, Aflatoonian MR, Daei Parizi MH, Hosseininasab A, Mostafavi M, Bamorovat M, et al. Visceral leishmaniasis in southeastern Iran: a narrative review. Iran J Parasitol. (2017) 12:1–11.

PubMed Abstract | Google Scholar

14. Faverjon C, Bernstein A, Grütter R, Nathues C, Nathues H, Sarasua C, et al. Transdisciplinary approach supporting the implementation of a big data project in livestock production: an example from the swiss pig production industry. Front Vet Sci. (2019) 6:1–11. doi: 10.3389/fvets.2019.00215

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Patyk KA, Duncan C, Nol P, Sonne C, Laidre K, Obbard M, et al. Establishing a definition of polar bear (ursus maritimus) health: a guide to research and management activities. Sci Total Environ. (2015) 514:371–8. doi: 10.1016/j.scitotenv.2015.02.007

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Comin A, Grewar J, Schaik GV, Schwermer H, Paré J, El Allaki F, et al. Development of reporting guidelines for animal health surveillance—AHSURED. Front Vet Sci. (2019) 6:426. doi: 10.3389/fvets.2019.00426

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Grant C, Lo Iacono G, Dzingirai V, Bett B, Winnebah TRA, Atkinson PM. Moving interdisciplinary science forward: Integrating participatory modelling with mathematical modelling of zoonotic disease in Africa. Infect Dis Poverty. (2016) 5:17 doi: 10.1186/s40249-016-0110-4

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Sutherland WJ, Burgman M. Use experts wisely. Nature. (2015) 256:317–8. doi: 10.1038/526317a

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Vennix JA. Group Model Building: Facilitating Team Learning Using System Dynamics. Wiley (1996) p. 312.

20. Gustafson LL, Gustafson DH, Antognoli MC, Remmenga MD. Integrating expert judgment in veterinary epidemiology : Example guidance for disease freedom surveillance. Prev Vet Med. (2013) 109:1–9. doi: 10.1016/j.prevetmed.2012.11.019

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Brito BP, Rodriguez LL, Hammond JM, Pinto J, Perez AM. Review of the global distribution of foot-and-mouth disease virus from 2007 to 2014. Transbound Emerg Dis. (2017) 64:316–32. doi: 10.1111/tbed.12373

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Mwiine FN, Velazquez-Salinas L, Ahmed Z, Ochwo S, Munsey A, Kenney M, et al. Serological and phylogenetic characterization of foot and mouth disease viruses from Uganda during cross-sectional surveillance study in cattle between 2014 and 2017. Transbound Emerg Dis. (2019) 66:2011–24. doi: 10.1111/tbed.13249

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Nthiwa D, Bett B, Odongo D, Kenya E, Wainaina M, Grazioli S, et al. Seroprevalence of foot-and-mouth disease virus in cattle herds raised in Maasai Mara ecosystem in Kenya. Prev Vet Med. (2020) 176:104929. doi: 10.1016/j.prevetmed.2020.104929

PubMed Abstract | CrossRef Full Text | Google Scholar

24. O'Brien MK, Wuebbolt Macy K, Pelican K, Perez AM, Myhre Errecaborde K. Transforming the one health workforce: lessons learned from initiatives in Africa, Asia and Latin America. Rev Sci Tech. (2019) 38:239–50. doi: 10.20506/rst.38.1.2956

PubMed Abstract | CrossRef Full Text | Google Scholar

25. OIE (World Organisation for Animal Health). PVS Pathway. Available online at: https://www.oie.int/en/what-we-offer/improving-veterinary-services/pvs-pathway (accessed June 21, 2021)

26. Fink LD. “A Taxonomy of significant learning,” In: Creating Significant Learning Experiences : An Integrated Approach to Designing College Courses San Francisco, CA: Wiley (2013). p. 31–66.

27. Adamchick J, Rich KM, Perez AM. Self-Reporting of Risk Pathways and Parameter Values for Foot and Mouth Disease in Slaughter Cattle from Alternative Production Systems By Kenyan and Ugandan veterinarians. University of Minnesota Digital Conservancy. (2021). Available online at: https://hdl.handle.net/11299/223215

Google Scholar

28. Okoli C, Pawlowski SD. The Delphi method as a research tool: An example, design considerations and applications. Inf Manag. (2004) 42:15–29. doi: 10.1016/j.im.2003.11.002

CrossRef Full Text | Google Scholar

29. Drescher M, Perera AH, Johnson CJ, Buse LJ, Drew CA, Burgman MA. Toward rigorous use of expert knowledge in ecological research. Ecosphere. (2013) 4:1–26. doi: 10.1890/ES12-00415.1

CrossRef Full Text | Google Scholar

30. Motta P, Garner G, Hòvari M, Alexandrov T, Bulut A, Fragou IA, et al. Framework for reviewing livestock disease reporting systems in high-risk areas: assessing performance and perceptions towards foot and mouth disease reporting in the Thrace region of Greece, Bulgaria and Turkey. Transbound Emerg Dis. (2019) 66:1268–79. doi: 10.1111/tbed.13143

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Ramsay A, Harries AD, Zachariah R, Bissell K, Hinderaker SG, Edginton M, et al. The structured operational research and training initiative for public health programmes. Public Heal Action. (2014) 4:79–84. doi: 10.5588/pha.14.0011

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Schwandt TA. “Participatory Action Research (PAR),” In: The SAGE Dictionary of Qualitative Inquiry Thousand Oaks, CA: SAGE Publications, Inc., (2014). p. 221–222.

Google Scholar

33. Bello NM, Renter DG. Invited review: reproducible research from noisy data: revisiting key statistical principles for the animal sciences. J Dairy Sci. (2018) 101:5679–701. doi: 10.3168/jds.2017-13978

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Begley CG, Ioannidis JPA. Reproducibility in science: improving the standard for basic and preclinical research. Circ Res. (2015) 116:116–26. doi: 10.1161/CIRCRESAHA.114.303819

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Iglesias CP, Thompson A, Rogowski WH, Payne K. Reporting guidelines for the use of expert judgement in model-based economic evaluations. Pharmacoeconomics. (2016) 34:1161–72. doi: 10.1007/s40273-016-0425-9

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Drescher M, Edwards RC. A systematic review of transparency in the methods of expert knowledge use. J Appl Ecol. (2019) 56:436–49. doi: 10.1111/1365-2664.13275

CrossRef Full Text | Google Scholar