Developing a Survey Tool to Assess Implementation of Evidence-Based Chronic Disease Prevention in Public Health Settings Across Four Countries

Background: Understanding the contextual factors that influence the dissemination and implementation of evidence-based chronic disease prevention (EBCDP) interventions in public health settings across countries could inform strategies to support the dissemination and implementation of EBCDP interventions globally and more effectively prevent chronic diseases. A survey tool to use across diverse countries is lacking. This study describes the development and reliability testing of a survey tool to assess the stage of dissemination, multi-level contextual factors, and individual and agency characteristics that influence the dissemination and implementation of EBCDP interventions in Australia, Brazil, China, and the United States. Methods: Development of the 26-question survey included, a narrative literature review of extant measures in EBCDP; qualitative interviews with 50 chronic disease prevention practitioners in Australia, Brazil, China, and the United States; review by an expert panel of researchers in EBCDP; and test-retest reliability assessment. Results: A convenience sample of practitioners working in chronic disease prevention in each country completed the survey twice (N = 165). Overall, this tool produced good to moderately reliable responses. Generally, reliability of responses was higher among practitioners from Australia and the United States than China and Brazil. Conclusions: Reliability findings inform the adaptation and further development of this tool. Revisions to four questions are recommended before use in China and revisions to two questions before use in Brazil. This survey tool can contribute toward an improved understanding of the contextual factors that public health practitioners in Australia, Brazil, China, and the United States face in their daily chronic disease prevention work related to the dissemination and implementation of EBCDP interventions. This understanding is necessary for the creation of multi-level strategies and policies that promote evidence-based decision-making and effective prevention of chronic diseases on a more global scale.


INTRODUCTION
Chronic diseases are a threat to global health, in developed and developing countries alike, accounting for 60% of deaths worldwide (1). The medical costs and loss of productivity related to chronic diseases are a great financial burden to individuals and economies (1). Evidence-based chronic disease prevention (EBCDP) interventions are effective tools for preventing chronic diseases (2). However, studies among U.S. and European public health practitioners indicate that only 56-64% of chronic disease prevention interventions currently in use are evidencebased (3,4), while estimates of use of EBCDP interventions in lower and middle income countries are unknown. Studies in Australia and the United States have identified multilevel contextual factors that influence the dissemination and implementation (D&I) of EBCDP interventions. Examples of these contextual factors include individual-and agency-level capacity characterized by the training, structure, material and human resources at hand that hinder or facilitate the use of EBCDP interventions (2,(5)(6)(7). Additional work has addressed some of the contextual barriers by training practitioners on the evidence-based decision-making process, specifically clarifying the reasons for selecting EBCDP interventions and outlining how to find the interventions and resources to support effective implementation and quality improvement (3,4,7). These studies report increases in the D&I of EBCDP interventions among practitioners who attended the trainings. Research on Canadian public health departments has identified tailored messaging as an effective method for promoting the D&I of evidence-based interventions (8), and examined the pathways through which evidence is shared through organizational systems (9). These contextually specific findings inform next steps in addressing barriers and promoting evidence-based decision-making across the Canada. Little is known about these contextual factors that influence the D&I of EBCDP interventions in developing countries, nor the similarities and differences of contextual factors across countries. Several studies call for global strategies to improve the D&I of EBCDP interventions in order to more effectively reduce chronic diseases around the world (10)(11)(12). Reviews of measures used to assess the contextual factors that influence the D&I of EBCDP interventions highlight a lack of psychometric testing of the existing measures and room for improvement among those that have been tested (13)(14)(15). To assess cross-country contextual factors and inform globallyfocused recommendations for facilitating the D&I of EBCDP interventions, a single survey tool that can be used across multiple, diverse countries is needed.
This study provides a detailed overview of the development and test-retest reliability of a survey tool to measure the stage of dissemination, multi-level contextual factors, and individual and agency characteristics that influence the D&I of EBCDP interventions in Australia, Brazil, China, and the United States. These countries were chosen for several reasons including, their leadership in distinct regions of the world (16)(17)(18)(19)(20), differences on contextual variables of interest (e.g., sociocultural, political/economic) (21), and high prevalence of chronic diseases (22). The World Health Organization reports from 2014 showed that the large majority of deaths in each of the four countries was due to chronic diseases (91% in Australia, 88% in the United States, 87% in China, and 74% in Brazil) (22). Further, based on the few studies of the D&I of EBCDP from Brazil and China (23,24), compared with the many from Australia and the United States (25)(26)(27)(28)(29), Brazil, and China were selected as countries likely in earlier stages of dissemination of EBCDP than Australia and the United States.

Survey Tool Development
Development of the 26-question survey occurred in several stages. First, a guiding framework was developed based on previous work (30,31) of the research team (see Figure 1). This framework informed subsequent stages of survey tool development, ensuring that qualitative interview questions and initial survey drafts were literature-based and comprehensive from the outset.
Second, a narrative literature review of extant measures in EBCDP was carried out in order to identify relevant questions and gaps in the D&I of EBCDP literature (2,6,(31)(32)(33)(34)(35). Third, between February and July 2015 semi-structured interviews of public health practitioners in Australia (n = 13), Brazil (n = 9), China (n = 16), and the United States (n = 12) were conducted by trained researchers. Practitioners were identified through purposive sampling based on their employment at agencies responsible for the prevention of chronic disease in each country, including community health services, regional health departments, and non-government organizations (Australia); the ministry of health and local health departments (Brazil); hospitals, community health centers, and the Centers for Disease Control and Prevention (China); and local health departments (United States). The interviews were performed in English, Chinese, or Portuguese, audio recorded, transcribed, translated to English by two bi-lingual research team members (n = 25) when appropriate, and analyzed using deductive, hierarchical coding in NVivo version 10.
Forth, drafts of the survey underwent expert review by 13 chronic disease prevention researchers and were translated forward and backward to Chinese and Portuguese from English. Survey questions were organized into one of the five stages of dissemination or as multi-level contextual factors seen in Figure 1. Individual and agency characteristics were also included. Seven response items were deemed non-applicable or inappropriate for China contexts, but were included in the survey for the other three countries. These response items and the resulting tool can be found in Table 1.
Fifth, research team members in each country recruited public health practitioners working in chronic disease prevention, primarily on the local and regional levels, in each of the four countries to complete the survey. Samples of practitioners from various regions of each country were identified through national databases and networks of chronic disease prevention practitioners between November 2015 and April 2016. Public health systems across countries varied so much that there was no equivalent sampling method that worked for all four countries. In the United States, a stratified (by region) random sample of chronic disease prevention practitioners from a national database received up to three emails and two follow-up telephone calls requesting participation in the electronic survey (58% response rate). In Australia, up to two emails requesting participation in the electronic survey were sent to all chronic disease practitioners in a national registry (18% response rate). In Brazil, the same protocol as was followed in the United States was used, but with an additional follow-up telephone call (46% response rate). In China, a convenience sample of practitioners working within a network of community hospitals received one email and one follow-up telephone call requesting participation in the electronic survey (87% response rate). All surveys were delivered by an email embedded link and completed electronically. Upon completion of the survey, all respondents were asked to retake the survey two to three weeks later for test-retest reliability testing purposes. This process was repeated until each respondent to the survey had been contacted twice, requesting them to retake the survey. Calculating Cohen's kappa and Intraclass correlation coefficients (ICC) ranging from 0.50 to 0.70 require a sample size of 25-50 test-retest pairs, respectively (38), thus 25 pairs were the minimum, but 50 pairs were the goal. During data collection, political events in Brazil affected the work lives of many Brazilian chronic disease practitioners and made recruitment of Brazilian practitioners extraordinarily difficult (39,40). The data collection period was extended for research team investigators in Brazil in order to reach the minimum sample size.
This study was carried out in accordance with the committee responsible for human experimentation (institutional and national) and with the World Medical Association's Declaration of Helsinki with informed consent from all subjects. After reading the electronic informed consent document, subjects indicated their consent by selecting a radial button at the bottom of the informed consent document that read, "I consent to participate in this research study." Additional written documentation of consent was waived and the protocol was approved by Table 2 shows frequency counts for each response option by country, the first time respondents completed the survey. Item responses vary in prevalence from zero endorsements to endorsement from a large majority of a county's sample. The test-retest reliability coefficients and percentage agreement by question and country appear in Table 3. Of the seven questions with ordinal response options assessed using ICC, six and seven demonstrated good to moderate reliability among practitioners from Australia and the United States, respectively, whereas three questions among practitioners from Brazil and China demonstrated good to moderate reliability. Six of those seven questions were also assessed using percentage agreement. Six and five of the questions demonstrated good to moderate percentage agreement among practitioners from Australia and the United States, respectively, whereas three questions among practitioners from Brazil and one among practitioners from China demonstrated moderate percentage agreement at best.
Of the 11 questions whose response options were dichotomized and assessed using mean Cohen's kappa, few questions among practitioners across all four countries showed moderate mean reliability at best (Australia, N = 2; Brazil, N = 1; China, N = 1; United States, N = 3). Mean percentage agreement told a different story for these 11 questions. All but one question showed good mean percentage agreement among practitioners from Australia and the United States. Seven and five questions showed good mean percentage agreement among practitioners from Brazil and China, respectively. The remaining of the 11 questions across the countries showed moderate mean percentage agreement.
The following four questions produced less than moderately reliable responses based on both ICC and percentage agreement among practitioners in China: Personal use of repositories to find evidence-based interventions; Workplace staff use of repositories to find evidence-based interventions; Frequency that programs end that should have continued; and Frequency that programs continue that should have ended. Two of those questions (Workplace staff use of repositories to find evidence-based interventions, and Frequency that programs end that should have continued) produced less than moderately reliable responses among practitioners from Brazil based on both measures of reliability as well.

DISCUSSION
The development and reliability testing of this survey tool are important early steps toward facilitating population-level research that can increase our knowledge of country-specific and cross-country contextual factors that influence the D&I of EBCDP interventions and, in turn, begin to inform more global strategies for improving the D&I of EBCDP. This study, novel in its common methods across countries, showed that the measurement tool produced moderate to good reliability of responses, with at least one measure of reliability, among 14 of the 18 questions across all four countries.
Reliability findings inform the adaptation and further development of this tool. For example, the authors recommend revising the four questions pertaining to personal and workplace staff use of repositories for finding evidence-based interventions and frequency that programs end or continue without warrant before further use among practitioners in China and Brazil. The poor reliability of responses produced from these questions among practitioners from Brazil and China reflect a difference in how they relate to the content of the questions, compared with practitioners from Australia and the United States. This difference may highlight meaningful differences within contexts with respect to D&I processes and structures. For instance, practitioners in countries for which EBCDP is in an earlier stage of dissemination tend to be less knowledgeable about key concepts of EBCDP, making the questions conceptually more difficult and in turn negatively influencing the reliability of their responses (43). Another potential contributing factor to the lower reliability among responses from practitioners in Brazil and China is that the survey tool had to be translated from English to Chinese and Portuguese. Tanzer and Sim review international guidelines on translating and adapting measures across cultural contexts, and this study reflects well the best practices for developing a relevant survey tool for use in the four intended countries (44). For instance, bilingual researchers from each of the four cultural perspectives, as well as public health practitioners working in the chronic disease prevention context in each country were involved in the development of the questions, response options, translations, and reliability testing. Despite steps that the research team took to minimize mis-translation, the meaning of each question and response option becomes one layer removed from its original, intended meaning after translation. Next steps for informing further adaptation of the survey tool should include validity testing  In a few of their programmatic areas 9 (23.      among chronic disease prevention practitioners in Australia, Brazil, China, and the United States, ideally in representative samples (45). There was low prevalence (N < 5) for many response options and the items with low prevalence varied by country. According to Sim and Wright, low prevalence has stifling effects on Cohen's kappa coefficients, but inflating effects on percentage agreement (46). Low prevalence likely contributed to the low kappa coefficients and comparatively higher percentage agreement found in this study. A larger sample of practitioners across all four countries with more diversity of experiences may improve the variability of responses and the accuracy of reliability findings. Response items with low prevalence of endorsements may also reflect response items that are less applicable to practitioners' experiences in that particular country. Use of this survey tool in a larger, randomly selected sample of chronic disease practitioners in each country would clarify this conjecture.

Strengths and Limitations
This study responds well to a U.S. federal report that called for additional research focused on the experiences and perspectives of key stakeholders in evidence-based intervention delivery, in order to better facilitate the sustainability of interventions (47). The questions within this survey tool reflect critical contextual factors based on the literature, qualitative interviews of public health practitioners, and expert review (2,5,6). This survey tool allows researchers to proceed with research on the D&I of EBCDP interventions on a more global scale than was previously available. To our knowledge, this is the first study of its kind that used common methods across four countries. The research team had particular trouble recruiting retest respondents in Brazil due to significant political unrest that affected public health practitioners at the time of the request (39,40). This contributed to the longer duration between test and retest and the smaller sample from Brazil compared with the other three countries. Additionally, his survey tool demonstrated lower reliability of responses among practitioners from Brazil and China compared with those from Australia and the United States. Lastly, a convenience sampling approach was carried out in some of the countries to recruit chronic disease prevention practitioners serving local or regional jurisdictions. Such a sampling method introduces potential selection bias and is unlikely to produce representative samples of all chronic disease prevention practitioners in each country. However, the intention of the present study was not to test hypotheses or provide prevalence estimates, which would have required using methods to address sampling error (46). Acknowledging these limitations of the sampling approach, the researcher team ensured that the selected sample included practitioners from various regions of each country, and provided distributions of all survey responses as well as demographic characteristics of the sample.

CONCLUSION
This survey tool allows cross-country data collection that can contribute toward an improved understanding of the contextual factors that public health practitioners in Australia, Brazil, China, and the United States face in their daily chronic disease prevention work. This understanding is necessary for the creation of multi-level strategies and policies that promote evidence-based decision-making and effective prevention of chronic diseases on a global scale.

ETHICS STATEMENT
This study was carried out in accordance with the committee responsible for human experimentation (institutional and national) and with the World Medical Association's Declaration of Helsinki with informed consent from all subjects. After reading the electronic informed consent document, subjects indicated their consent by selecting a radial button at the bottom of the informed consent document that read, I consent to participate in this research study. Additional written documentation of consent was waived and the protocol was approved by The University

AUTHOR CONTRIBUTIONS
EB contributed to the conception and design of the study, interpretation of data, and drafting of the full manuscript. XY and AdR contributed to the analysis and interpretation of data and drafting of the Statistical Analyses. KS and RB contributed to the conception and design of the study, interpretation of data, and drafting of the Discussion. ZW, PS, TP, RA, and RR contributed to the conception and design of the study. All authors contributed to manuscript revision, read and approved the submitted version.