Users’ perception of the OH-EpiCap evaluation tool based on its application to nine national antimicrobial resistance surveillance systems

Abstract

Introduction:

Antimicrobial resistance (AMR) is a One Health (OH) challenge. To achieve or maintain an effective and efficient AMR surveillance system, it is crucial to evaluate its performance in meeting the proposed objectives, while complying with resource restrictions. The OH-EpiCap tool was created to evaluate the degree of compliance of hazard surveillance activities with essential OH concepts across the following dimensions: organization, operational activities, and impact of the surveillance system. We present feedback on the application of the OH-EpiCap tool from a user’s perspective, based on the use of the tool to evaluate nine national AMR surveillance systems, each with different contexts and objectives.

Methods:

The OH-EpiCap was assessed using the updated CoEvalAMR methodology. This methodology allows the evaluation of the content themes and functional aspects of the tool and captures the user’s subjective experiences via a strengths, weaknesses, opportunities, and threats (SWOT) approach.

Results and Discussion:

The results of the evaluation of the OH-EpiCap are presented and discussed. The OH-EpiCap is an easy-to-use tool, which can facilitate a fast macro-overview of the application of the OH concept to AMR surveillance. When used by specialists in the matter, an evaluation using OH-EpiCap can serve as a basis for the discussion of possible adaptations of AMR surveillance activities or targeting areas that may be further investigated using other evaluation tools.

1. Introduction

International organizations are calling for a One Health (OH) approach to tackle antimicrobial resistance (AMR). The One Health High Level Expert Panel (OHHLEP) defines OH as “an integrated, unifying approach that aims to sustainably balance and optimize the health of people, animals, and ecosystems” recognizing that the health of these populations is closely linked and interdependent (1).

AMR genes and microbes know no border, and certain antimicrobial agents are cross used in humans, animals, and plants. Hence, AMR is one of the quintessential examples of a OH challenge (2). Therefore, an integrated, multisectoral approach is necessary to address the issue (3, 4). Integrated surveillance, according to Aenishaenslin et al., is the “systematic collection, analysis, interpretation of data, and dissemination of information collected from different components of a system to provide a global, multidisciplinary, multi-perspective understanding of a health problem and to inform system-based decisions” (5). These actions should be coordinated between the human, animal and environmental sectors (6).

The application of this concept to national surveillance systems is essential to better understand AMR emergence and dissemination and to sustain risk mitigation decisions (7). The OHHLEP has recently released a theory of change for OH that can help to support intersectoral collaboration in national strategies for OH challenges, including those aiming to keep antimicrobials (AM) effective for future generations of people and animals. This theory of change includes the goals, objectives, desired impact at country level, intermediate outcomes, and related functions (1).

Conducting regular evaluations of a surveillance system’s processes and performance is crucial to assess if the established objectives are being met in the most cost-effective way (8). OH initiatives should preferably be evaluated using a methodology that targets all disciplines encompassed and estimate the potential added value of the current approach over a less integrated one (9). The objectives of the evaluation should be made clear from the start, and an overview of the systems’ surveillance components should be produced to guide it, and to balance the objectives of the evaluation with the available resources to perform it (8).

The international network CoEvalAMR was established in 2019 with the goal of providing guidance to help users in choosing an assessment tool from a catalog of tools available to evaluate antimicrobial use (AMU) and AMR surveillance systems (10). Moreover, the network aimed to guide future applications and improvement of the tools assessed and the development of new tools. To meet these aims, a methodology focusing on the users’ perception of the tool was developed during Phase 1 of the CoEvalAMR network (11). The original methodology was used by Sandberg et al. (11) to provide feedback on six different evaluation tools based on their application in eight countries. Based on the experience gained, the methodology has recently been updated and further refined, as part of the work undertaken in Phase 2 of the CoEvalAMR network (12). The methodology encompasses the evaluation of descriptive and functional aspects, together with an assessment of content themes and questions on strengths, weaknesses, opportunities, and threats (SWOT) (12).

OH-EpiCap is among the catalog of tools being assessed in Phase 2 of the CoEvalAMR network. This tool has recently been developed by the MATRIX consortium, funded by the One Health European Joint Program to systematize the characterization of epidemiological surveillance activities in a national surveillance system (13). OH-EpiCap is presented as an easy-to-apply tool, covering previously overlooked aspects such as the impact of integrated surveillance. More specifically, the purpose of OH-EpiCap is to facilitate the evaluation and reinforcement of national capacities and capabilities for OH integrated surveillance of zoonotic hazards (13).

In this study, we applied and evaluated OH-EpiCap using the updated CoEvalAMR user’s perception methodology and presented feedback on the application of the OH-EpiCap tool to nine national AMR surveillance systems, with different monitoring contexts and objectives.

2. Materials and methods

2.1. Description of OH-EpiCap

The OH-EpiCap tool is composed of three thematic domains (called “dimensions”), each with four different targets that are again segmented into four indicator questions, leading to a total of 48 indicators, briefly presented in Table 1. Each indicator is scored from 1 (no compliance) to 4 (full compliance), with the possibility to select “non-applicable” in case the indicator is not relevant to the system under evaluation. All indicators have the same weight, and for each target, the average value of the indicators’ scores is converted into a target score (13).

Different respondents can have diverging opinions on the scoring of the indicators that compose OH-EpiCap, according to their backgrounds, perceptions, and expectations. To reduce the possible bias that the subjectivity of the scoring method may create, a consensus among respondents within one working group is required to select a final score among those described in the scoring guide (13).

The tool also includes a graphical interface developed in RShiny, where the results of the evaluation are presented in a dashboard that can be exported as a report. The OH-EpiCap tool is available on the following website: https://freddietafreeth.shinyapps.io/OH-EpiCap/.

2.2. Data collection

The nine surveillance systems evaluated were selected by members of the CoEvalAMR network. The selection was made by convenience of the members, due to direct acquaintance with the systems evaluated or close personal contacts. The evaluations were conducted from August to November 2022.

The number of respondents involved in the evaluation of each case study varied from one to five; these respondents are referred to as “assessors” throughout the text. The assessors filled in the OH-EpiCap evaluation questions during either a single or repeated workshop session that lasted a total of 2–8 h. All assessors involved had expertise in AMR surveillance in the country they represented for this study, scoring the indicator questions according to their own experience or knowledge from previous activities. This methodology makes the evaluation outputs somewhat subjective. In the country case studies that were conducted by more than one assessor, the subjectivity was reduced because of the requirement to reach consensus within the group of assessors who formed part of the country case. Whenever needed, additional experts and information sources were consulted.

The OH-EpiCap tool was used to evaluate national AMR surveillance systems in Bangladesh, Belgium, Canada, Denmark, France, Italy, Norway, Portugal and the United Kingdom (Table 2). The number of assessors and their affiliation, the type of workshop conducted, and the total duration of the evaluation are described for each country in Supplementary Table S1. The surveillance system evaluated in each country including its main aims can be found in Table 2.

2.3. Data analysis

The updated CoEvalAMR users’ perception methodology was used to evaluate the OH-EpiCap tool (13). The methodology consists of a series of questions related to: (1) a general description of the case study and the tool, (2) two standardized scoring schemes, one regarding functional aspects, and another for content themes, as well as (3) a SWOT analysis (12). The functional aspects encompassed in the methodology are grouped into: Ease of use, Scope, Prerequisites before use, Time and resources, and Outputs. The content themes related to the tool’s scope are: AMU and AMR, Collaboration, Resources, Output and use of information, Integration, Adaptivity, Technical operations, Impact and Governance. The definitions of the content themes and functional aspects can be consulted in Alban et al. (12).

Both functional aspects and content themes of OH-EpiCap tool were scored semi-quantitatively using a scale from 1 to 4 or “non-applicable.” Groups composed of several functional aspects or content themes were averaged. Next, the median, maximum and minimum of the scores given by the assessors for functional aspects and content themes were displayed in a radar diagram in Figures 1A,B, respectively. Due to the skewness of the distribution of the answers’ scores, which were not normally distributed, the decision was made to show the median of the scores. Microsoft Excel^® was used for data analysis and visualization of the outputs.

Figure 1

The SWOT analysis was undertaken to capture the assessors’ subjective experiences when applying OH-EpiCap. More specifically, the following wording accompanied each component: Strengths: “The strengths of this tool are,” Weaknesses: “The weaknesses of this tool are,” Opportunities: “The added value(s) of using this tool is” and Threats: “This tool might be criticized because of.” A qualitative analysis of the feedback provided by the assessors was performed following the same principles as described by Sandberg et al. (11), which were based on grounded theory (15): all individual sentences were collected, then, sentences with similar content were simplified and condensed into one sentence. The synthesis was performed by three of the assessors and later verified by the remaining assessors.

3. Results

3.1. Functional aspects

Regarding ease-of-use, OH-EpiCap scored highly due to its user-friendly interface with checkboxes to answer the questions that formed these indicators. The scope of the tool is defined as the ability to address the stated evaluation objectives and is further divided into the content themes evaluated (12). OH-EpiCap was not created with the objective of covering all the national capacities and capabilities for OH integrated surveillance. OH-EpiCap does not cover certain content themes in the level of detail perceived as relevant by the assessors, even for a macro evaluation, as addressed in section 3.2.

The OH-EpiCap tool is free of use. As for prerequisites to use it, no previous data collection is required, and the answers can be given based on the evaluators’ experience connected with the surveillance system. Most indicators require that the evaluation is conducted with specialists in the surveillance system, or that they are consulted in the process, given that an in-depth perspective of the whole surveillance system is needed. No training is necessary to get acquainted with the tool.

OH-EpiCap can—in most cases—be successfully applied by a small group composed of for example three or four persons, providing that the group can form a clear cross-sectoral picture of the surveillance system. Based on our experience, and depending on the expertise of the stakeholders gathered, the evaluation can be conducted in half a day or slightly longer. If additional stakeholders need to be consulted after the initial workshop, the evaluation process will be prolonged. In the case that supplementary information that may impact a given answer is gathered via extra communications, outside of the stakeholder workshop, it should be further discussed with all the assessors.

The graphical outputs generated by the tool were found to provide an easily accessible overview of the responses given. However, given the superficiality of the evaluation content (Table 1), the output of the evaluation need to be further discussed and investigated with relevant actors before it can be translated into specific changes in the surveillance system. Please see the section 3.2 for an elaboration of this issue.

3.2. Content themes

The tool does not encompass indicators specifically addressing AMU and AMR surveillance. Even though not covered to a complete extent, OH-EpiCap still provides an overview of the thematic areas connected with the human and budget resources needed to maintain the surveillance activity, as well as the collaboration in the governance structures of the system and in the technical surveillance activities. It also encompasses indicators related to the possible adaptation of the surveillance activities to new challenges and in an efficient manner. The overall impact of the surveillance system is also covered, but without details on how the information generated by the surveillance system could lead to changes in the health outputs. Moreover, the tool does not go into details in the governance domain, specifically the accountability of stakeholders, the coordination of activities and the transparency of processes are only superficially covered.

3.3. SWOT analysis

The subjective experience of the application of OH-EpiCap by the assessors captured via SWOT analysis is presented in Table 3 in a summarized format.

4. Discussion

4.1. Overall perception of the tool

During the development of OH-EpiCap, several pilot applications on various surveillance activities were conducted. Due to the generic design of OH-EpiCap, it has been successfully applied to surveillance activities connected with food-borne hazards, such as Salmonella, Campylobacter, Listeria, and other zoonotic hazards such as Chlamydia psittaci (15). With this study, we illustrate its application to the evaluation of integrated surveillance systems for AMR.

According to the information collected in the nine case studies, OH-EpiCap can provide an overview of several crucial topics connected with AMR integrated surveillance, even though the tool was not specifically designed to evaluate these activities. The OH-EpiCap tool provides a summary assessment of the three dimensions targeted, which cover most of the elements that are important for assessing integrated surveillance systems, as described in the Integrated surveillance systems evaluation (ISSE) framework (5). The ISSE framework identified five levels of assessment for such surveillance systems, which include the integration of a OH approach, the production of integrated information and expertise, the generation of actionable knowledge, the influence on decision-making and the contribution to desirable outcomes. Evaluating these five levels in a comprehensive manner requires considerable time and resources, and OH-EpiCap constitutes a good first step toward evaluation of all of them.

Simplistic design and user friendliness, without requiring training of evaluators, are highly appreciated, not just by our assessors but also among users in general as shown in a survey recently undertaken among surveillance program practitioners and evaluators (16). To make the workshop more time efficient, it is recommended that at least one of the evaluators gets acquainted with the indicator questions and clarifies any possible doubts before organizing a session with the specialists involved in the evaluation and other relevant actors.

The outputs generated by OH-EpiCap may not lead directly to actions, however these can provide the basis for discussing further improvements with relevant stakeholders, as presented in a case study by (17). The MATRIX project also encompassed other activities that are complementary to the development of OH-EpiCap, such as a “Roadmap to develop national One Health Surveillance” which aims to function as a guideline for the development of OH Surveillance activities according to needs and resources in different countries (18).

An evaluation using OH-EpiCap can be conducted in a short period of time and with a small group of stakeholders, making it feasible to conduct an evaluation in situations with low resources. Moreover, evaluations can be done recurrently, when changes are implemented, benchmarking the system with itself over time. This can be made easily as OH-EpiCap contains benchmarking functionalities. These functionalities were not investigated in the present study, because of the different aims and purposes of the systems evaluated as noted in Table 2. For example, the Danish DANMAP program serves the purpose of integrated monitoring of AMU and AMR for both the animal and human sectors. In contrast, the Italian ClassyFarm encompasses mainly farm-level risk categorization components (e.g., biosecurity and animal welfare, besides AMR and AMU) whereas AMR surveillance in the human sector is conducted by different Italian institutions (19). Given the above-mentioned differences in the aims of the surveillance activities which we evaluated, indicator questions connected to real-time response capacity were considered not relevant in some surveillance activities.

AMR surveillance systems are complex and encompass multiple hazards, e.g., surveillance of clinical isolates in human health, bacterial isolates from animals at slaughter lines or in slurry, or AMR genes in sewage systems, each with their own particularities and logistics (5). So, when answering some of the questions representing an individual indicator in OH-EpiCap, interpretations need to be considered. This approach can justify the application of OH-EpiCap to several surveillance components, while focusing on one hazard at a time.

We applied the OH-EpiCap tool in nine different countries, by different native language users, providing important feedback to the developers regarding the phrasing of the indicator questions. We found that most of the indicator questions were considered simple and straight-forward. However, considering the expected worldwide application of the tool by users, who may have different use of the English language and, hence, familiarity with the terminology used, materials should be developed to unequivocally clarify the meaning of all indicators. With the publication of case studies evaluations and the scientific paper accompanying the tool (13), this should be accounted for. At the time of writing, the OH-EpiCap tool was still in a Beta Version, so the phrasing of indicators was not final.

4.2. Contribution of OH-EpiCap to the evaluation of integrated AMR surveillance systems

Except for Bangladesh, all country cases presented here were conducted in high-income countries. Hence, we have only limited experience regarding the applicability of the tool to low- and middle-income countries (LMICs). In LMICs, AMR surveillance is often hindered by deficient health system governance and restrictive financing of health data producing systems and laboratory capacities (20, 21). In addition, more efforts are needed to improve the capacity, quality standards, and integration of AMR surveillance in LMICs, which often have focus on human health. Due to its generic design, OH-EpiCap does not require that an integrated surveillance system is already established. However, at least primary surveillance activities need to be established and run. If this is not the case in a country, engagement in other tools such as the FAO Progressive Management Pathway for Antimicrobial Resistance (FAO-PMP-AMR), which aims to guide countries in the implementation of national action plans against AMR and early surveillance efforts (22), may be considered.

By highlighting components which may be improved in a hazard integrated surveillance system, OH-EpiCap can be considered as a valuable new addition to the current catalog of tools to evaluate integrated AMR surveillance systems (11). Moreover, OH-EpiCap can act as a simple gateway to conduct a more in-depth evaluation of certain surveillance system components as considered relevant. This may be done by using other pre-established tools designed to evaluate OH integration, such as the Evaluation of Collaboration for Surveillance (ECoSur) or The Network for Evaluation of One Health (NEOH).

The ECoSur tool has been developed to facilitate an in-depth analysis of the organization and functioning of collaboration taking place in a multisectoral surveillance system, aiming to evaluate the overall quality and relevance of such collaboration in meeting the objectives envisioned by stakeholders to produce the expected outputs of the program (23). From a user’s perspective, this tool gives a detailed evaluation of multisectoral collaboration in OH surveillance activities, however it requires a high level of abstraction to understand the indicator questions listed in the tool. Still, conducting a full ECoSur evaluation is rewarding regarding quality of output, but remains time and resource demanding (11).

The NEOH tool allows the evaluation of the coherence between operational and organizational aspects of OH activities, with the aim of identifying the added value of the integration across disciplines and sectors (24). From a user’s perspective, this tool is a comprehensive, multi-faceted fit for a transversal and detailed analysis of OH initiatives. However, conducting an evaluation using NEOH may be difficult and time consuming given that users should have specific training in systems thinking to make the most of it (11).

One of the ongoing activities in the CoEvalAMR network aims to simplify the application of the NEOH and ECoSur tools, using a modular approach. Given the complexity of evaluating integrated AMR surveillance systems, this could be of great value, targeting the evaluation to certain components which need to be prioritized.

Within the CoEvalAMR network, case studies have already been conducted from a user’s perspective on the application of the ECoSur (25) and NEOH tools (26–29). Other tools and frameworks that have been specifically designed to evaluate integrated AMR surveillance have also been evaluated: the FAO-PMP-AMR tool (30–33) as mentioned above (34); the FAO Assessment Tool for Laboratories and AMR Surveillance Systems (FAO-ATLASS) (35) developed to facilitate the assessment and definition of targets to improve national AMR surveillance systems in the food and agriculture sectors (36) and the ISSE framework (37, 38) developed to structure an assessment of the added value of integration in AMR surveillance systems (39). The interactive selection tool developed by the CoEvalAMR network can help users to select an appropriate tool for their needs (40).

5. Conclusion

The OH-EpiCap tool is a new addition to the portfolio of existing tools to evaluate integrated AMR surveillance systems. It provides a brief macro-overview of relevant OH topics, such as the perceived added value of establishing a OH team as a governance structure. This can serve as a basis to discuss possible adaptations of AMR surveillance activities, or targeting areas that may be further investigated using other established tools. It is free and easy to use, does not require training, and can be performed in less than a day provided that the group performing the evaluation has detailed knowledge on the surveillance system to be evaluated.

Funding

This study was funded by the Canadian Institutes for Health Research through the Joint Programming Initiative on Antimicrobial Resistance (JPIAMR). The Portuguese case study was supported by funding from the European Union’s Horizon 2020 Research and Innovation programme under grant agreement No 773830: One Health European Joint Programme (MATRIX project).

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

• 1.

  One Health High-Level Expert Panel (OHHLEP). Strategic framework for collaboration on antimicrobial resistance – together for one health. (2022). Available at: https://www.who.int/publications/i/item/9789240045408

  • Google Scholar

• 2.

  Pinto FerreiraJBattagliaDDorado GarcíaATempelmanKBullonCMotriucNet al. Achieving antimicrobial stewardship on the global scale: challenges and opportunities. Microorganisms. (2022) 10:1599. doi: 10.3390/microorganisms10081599

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 3.

  RobinsonTPBuDPCarrique-MasJFèvreEMGilbertMGraceDet al. Antibiotic resistance is the quintessential one health issue. Trans R Soc Trop Med Hyg. (2016) 110:377–80. doi: 10.1093/trstmh/trw048

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 4.

  CaniçaMManageiroVJones-DiasDClementeLGomes-NevesEPoetaPet al. Current perspectives on the dynamics of antibiotic resistance in different reservoirs. Res Microbiol. (2015) 166:594–600. doi: 10.1016/j.resmic.2015.07.009

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 5.

  AenishaenslinCHäslerBRavelAParmleyEJMediouniSBennaniHet al. Evaluating the integration of one health in surveillance Systems for Antimicrobial use and Resistance: a conceptual framework. Front Vet Sci. (2021) 8:611931. doi: 10.3389/fvets.2021.611931

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 6.

  QueenanKHäslerBRushtonJ. A one health approach to antimicrobial resistance surveillance: is there a business case for it?Int J Antimicrob Agents. (2016) 48:422–7. doi: 10.1016/j.ijantimicag.2016.06.014

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 7.

  StärkKDCArroyo KuribreñaMDauphinGVokatySWardMPWielandBet al. One health surveillance - more than a buzz word?Prev Vet Med. (2015) 120:124–30. doi: 10.1016/j.prevetmed.2015.01.019

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 8.

  NielsenLRAlbanLEllis-IversenJMintiensKSandbergM. Evaluating integrated surveillance of antimicrobial resistance: experiences from use of three evaluation tools. Clin Microbiol Infect. (2020) 26:1606–11. doi: 10.1016/j.cmi.2020.03.015

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 9.

  LégerAStärkKDCRushtonJNielsenLR. A one health evaluation of the University of Copenhagen Research Centre for control of antibiotic resistance. Front Vet Sci. (2018) 5:194. doi: 10.3389/fvets.2018.00194

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 10.

  Home Page. CoEvalAMR. Available at: https://coevalamr.fp7-risksur.eu/

• 11.

  SandbergMHespAAenishaenslinCBordierMBennaniHBergwerffUet al. Assessment of evaluation tools for integrated surveillance of antimicrobial use and resistance based on selected case studies. Front Vet Sci. (2021) 8:620998. doi: 10.3389/fvets.2021.620998

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 12.

  AlbanLBordierMHäslerBCollineauLTomassoneLBennaniHet al. Capturing systematically users’ experience of evaluation tools for integrated AMU and AMR surveillance. Front Vet Sci. (2023) 10:1107122. doi: 10.3389/fvets.2023.1107122

  • CrossRef
  • Google Scholar

• 13.

  TegegneHABogaardtCCollineauLCazeauGLaillerRReinhardtJet al. OH-EpiCap: a semi-quantitative tool for the evaluation of one health epidemiological surveillance capacities and capabilities. Front Public Health. (2023) 11:1053986. doi: 10.3389/fpubh.2023.1053986

  • CrossRef
  • Google Scholar

• 14.

  HenauxVTegegneHBogaardtCLaillerRCollineauLPradaJ. Deliverable D-JIP-MATRIX-WP4.2 OH-EpiCap tool and tutorial. (2022). Available at: https://zenodo.org/record/7006654

  • Google Scholar

• 15.

  KhanSN. Qualitative research method: grounded theory. Int J Bus Manag. (2014) 9:1–10. doi: 10.5539/ijbm.v9n11p224

  • CrossRef
  • Google Scholar

• 16.

  RüeggSRAntoine-MoussiauxNAenishaenslinCAlbanLBordierMBennaniHet al. Guidance for evaluating integrated surveillance of antimicrobial use and resistance. CABI One Heal. (2022) 2022:1–13. doi: 10.1079/cabionehealth.2022.0007

  • CrossRef
  • Google Scholar

• 17.

  MouraPHøgBBAlbanLSönksenUWDuarteASRSandbergM. Evaluating the OH-EpiCap tool using the Danish integrated surveillance programme for AMU and AMR as a case study, Submitted to Frontiers in Public Health on the 19 of December 2022.

  • Google Scholar

• 18.

  EvesCFriesemaISkjerdalOTHolmbergMÅgrenELopez de AbechucoEet al. Deliverable D-JIP-MATRIX-WP5.1 - report on requirement analysis for OHS roadmap template. (2022). Available at: https://zenodo.org/record/6504418

  • Google Scholar

• 19.

  The AR-ISS Surveillance System. Available at: https://www.epicentro.iss.it/en/antimicrobial-resistance/surveillance-ar-iss

• 20.

  IskandarKMolinierLHallitSSartelliMHardcastleTCHaqueMet al. Surveillance of antimicrobial resistance in low- and middle-income countries: a scattered picture. Antimicrob Resist Infect Control. (2021) 10:63–19. doi: 10.1186/s13756-021-00931-w

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 21.

  GandraSAlvarez-UriaGTurnerPJoshiJLimmathurotsakulDvan DoornHR. Antimicrobial resistance surveillance in low- and middle-income countries: progress and challenges in eight south Asian and southeast Asian countries. Clin Microbiol Rev. (2020) 33:1–29. doi: 10.1128/CMR.00048-19

  • CrossRef
  • Google Scholar

• 22.

  Food and Agriculture Organization of the United Nations (FAO). FAO progressive management pathway for antimicrobial resistance (FAO-PMP-AMR) | antimicrobial resistance | food and agriculture Organization of the United Nations. Available at: https://www.fao.org/antimicrobial-resistance/resources/tools/fao-pmp-amr/en/

• 23.

  BordierMDelavenneCNguyenDTTGoutardFLHendrikxP. One health surveillance: a matrix to evaluate multisectoral collaboration. Front Vet Sci. (2019) 6:109. doi: 10.3389/fvets.2019.00109

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 24.

  RüeggSRNielsenLRButtigiegSCSantaMAragrandeMCanaliMet al. A systems approach to evaluate one health initiatives. Front Vet Sci. (2018) 5:23. doi: 10.3389/fvets.2018.00023

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 25.

  BordierMarionDelavenneCamilleGoutardFlavieHendrikxP. User feedback on ECoSur tool applied to the multisectoral surveillance system for antibiotic resistance in Vietnam. (2020). Available at: https://guidance.fp7-risksur.eu/wp-content/uploads/2020/06/Case-study-EcoSur_tool-applied-to-ABR-surveillance-in-Vietnam.pdf

  • Google Scholar

• 26.

  HespAylaBergwerffUrsulavan SchaikG. User feedback on NEOH tool applied to monitoring of antimicrobial resistance and antibiotic usage in animals in the Netherlands (MARAN). (2020). Available at: https://guidance.fp7-risksur.eu/wp-content/uploads/2020/06/Case-study_NEOH_tool_applied-to-MARAN_the-Netherlands.pdf

  • Google Scholar

• 27.

  NielsenLiza RosenbaumLisAlbanEllis-Iversen JohanneSM. User feedback on NEOH tool applied to AMR in Salmonella isolated from pigs-a part of the DANMAP system general information. (2020). Available at: https://guidance.fp7-risksur.eu/wp-content/uploads/2020/06/Case-study-NEOH-tools_applied-to-Danmap-in-Denmark.pdf

  • Google Scholar

• 28.

  FilippitziMaria-EleniChantziarasIliasMoussieauxNA. User feedback on NEOH tool applied to AMR surveillance in Belgium general information. (2020). Available at: https://guidance.fp7-risksur.eu/wp-content/uploads/2020/06/Case-study-NEOH_tool_applied-to-NAP-in-Belgium.pdf

  • Google Scholar

• 29.

  NorströmM. User feedback on NEOH applied to monitoring programme for AMR in animals, food and feed in Norway (NORM-VET) general information. (2020). Available at: https://guidance.fp7-risksur.eu/wp-content/uploads/2020/06/Case-study-report-template_NEOH_Norway.pdf

  • Google Scholar

• 30.

  NorströmM. User feedback on FAO PMP tool applied to the monitoring programme for AMR in animals, food and feed in Norway general information. (2020). Available at: https://guidance.fp7-risksur.eu/wp-content/uploads/2020/06/Case-study-FAO-PMP-tool_applied-to-NORMVET-in-Norway.pdf

  • Google Scholar

• 31.

  TomassoneLaurade MeneghiD. User feedback on FAO PMP tool applied to the ClassyFarm programme in Italy general information. (2020). Available at: https://guidance.fp7-risksur.eu/wp-content/uploads/2020/06/Case-study-FAO-PMP-tool_applied-to-ClassyFarm-in-Italy.pdf

  • Google Scholar

• 32.

  FilippitziMaria-EleniChantziarasIliasMoussieauxNA. User feedback on FAO PMP tool applied to AMR surveillance in Belgium general information. (2020). Available at: https://guidance.fp7-risksur.eu/wp-content/uploads/2020/06/Case-study-FAO-PMP_tool_applied-to-NAP-in-Belgium.pdf

  • Google Scholar

• 33.

  NielsenLRAlbanLEllis-IversenJet al. Evaluating integrated surveillance of antimicrobial resistance: experiences from use of three evaluation tools. Clin Microbiol Infect. (2020) 26:1606–1611.

  • Google Scholar

• 34.

  FAO. FAO progressive management pathway for antimicrobial resistance (PMP-AMR). Available at: https://www.fao.org/antimicrobial-resistance/resources/tools/fao-pmp-amr/en/

• 35.

  LizaRosenbaum NielsenAlbanLisEllis-IversenJohanneSandbergM. User feedback on ATLASS tool applied to AMR in Salmonella isolated from pigs-a part of the DANMAP system general information. (2020). Available at: https://guidance.fp7-risksur.eu/wp-content/uploads/2020/06/Case-study-ATLASS_applied-to-Danmap-in-Denmark.pdf

  • Google Scholar

• 36.

  FAO. FAO assessment tool for laboratories and AMR surveillance systems (ATLASS). Available at: https://www.fao.org/antimicrobial-resistance/resources/tools/fao-atlass/en/

• 37.

  AenishaenslinCécileVernetMichèleParmleyJaneRavelAndréHäslerB. User feedback on ISS-AMR framework for the evaluation of the impacts on decision-making of the Canadian integrated program for antimicrobial resistance surveillance general information. (2020). Available at: https://guidance.fp7-risksur.eu/wp-content/uploads/2020/06/Case-study-AMR-ISS-tools_applied-to-CIPARS-in-Canada.pdf

  • Google Scholar

• 38.

  BennaniHoudaHäslerBarbaraCornelsenL. User feedback on AMR-ISS tool applied to evaluation of the surveillance system for AMR and AMU in the United Kingdom from a one health perspective general information. (2020). Available at: https://guidance.fp7-risksur.eu/wp-content/uploads/2020/06/Case-study-AMR_ISS_tool-applied-to-AMR-AMU-surveillance-in-UK.pdf

  • Google Scholar

• 39.

  AenishaenslinCHäslerBRavelAParmleyJStärkKBuckeridgeD. Evidence needed for antimicrobial resistance surveillance systems. Bull World Health Organ. (2019) 97:283–9. doi: 10.2471/BLT.18.218917

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 40.

  Support for selecting an evaluation tool – CoEvalAMR surveillance evaluation. Available at: https://guidance.fp7-risksur.eu/welcome/selection-tool/

  • Google Scholar