In United States healthcare delivery organizations, 40% of patients do not receive evidence-based care, ~25% of patients are harmed, and at least 30% of annual healthcare spending is wasted (1–3). Harm events include medication errors, injuries, infection, prolonged hospitalization, or other unintended consequences (2). At the same time, the scientific community publishes more than 140 clinical trials and 80 systematic evidence reviews every single day (4, 5). Still, it takes decades and millions of dollars for a new scientific discovery to become part of clinical practice (6–9). For example, over the past two decades numerous randomized control trials have demonstrated the effectiveness of collaborative care models for depression and dementia. Nevertheless, the probability of patients receiving these evidence-based models is very low (10–13). Determinants that prevent the adoption of evidence-based practices include inconsistent internal demands, stakeholder resistance, unstable leadership, and inadequate resources (14).

To address the gap between knowledge and practice, healthcare delivery organizations must acquire workforces with competency in designing, implementing, and diffusing evidence-based healthcare solutions. In 2015, the Indiana University Center for Health Innovation & Implementation Science (CHIIS) launched a Professional Certificate Program in Innovation and Implementation Sciences as an attempt to build a network of Agile Change Conductors. This graduate educational program sought to train established healthcare professionals, consisting of clinicians, educators, researchers, and administrators, to be able to leverage insights from Agile Science to design, implement, scale, sustain, and diffuse evidence-based care solutions within their local complex adaptive healthcare delivery organizations (15). Derived from the root of the word “Agile”, which refers to speed and adaptability, Agile Science is an evolving and adaptive discovery and acquisition process for understanding, predicting, and steering the behavior of an individual human or the behavior of a social human organization, such as a healthcare delivery organization (16–18). The program seeks to equip clinicians and healthcare leaders with tools, processes, and strategies to create demand, encourage positive behaviors, foster a reinforcing culture, and address setbacks quickly and with precision to create multidisciplinary systemic change.

The purpose of this paper is to describe and evaluate the effectiveness of the Graduate Certificate from its inception in the Fall of 2015 until the Summer of 2021 and discuss areas for quality improvement.

The first six cohorts included 25 physicians, nine nurses, 14 masters-level administrators, three social workers, and nine other clinically trained personnel. Participants for this study included all 60 past students of the program. Because the first cohort graduated in 2016, there were a total of six cohorts with varying amounts of participants in each, the most recent having graduated in August of 2021. Of the 60 alumni contacted, 52 completed the survey yielding an 87% response rate. Sixty percent of the graduates were female and 30% were of an under-represented minority (African American, Asian, Hispanic). Using the Net Promotor Scale, 36 participants (69%) were promoters of the program with a score of 9 or 10; one participant (2%) was considered a detractor with a score below 7; and finally, 15 participants (29%) were neutral with a score of 7 or 8. Thus, the Net Promoter Score was 67 (promoters minus detractors). Neutral and detractor participants came from every cohort and a diverse range of healthcare occupations.

In terms of the professional benefit of the program (see Table 1), graduates agreed that the certificate provided significant benefits for advancing their professional career [4.308 with a standard deviation (SD) of 0.612]; expanded their professional network (4.615, SD of 0.530); had a large impact on the effectiveness of their leadership (4.288, SD of 0.667), their change management (4.365, SD of 0.742), and their communication (4.392, SD of 0.666); and finally the graduates agreed that Agile Science has been applicable in their field (4.438, SD of 0.727).

Table 2 summarizes the competencies of the graduates in being Agile Change Conductors using a scale from 0 to 10 where 10 is indicative of reaching mastery. The total average score for each of the 26 different competencies ranged from 5.37 (SD of 2.80) for writing grants or business proposals to 7.77 for self-awareness (SD of 1.96). There were no statistically significant differences between the 2016 and 2021 cohorts (p = 0.52). Existing competency data taken prior to course completion existed for the 2020–2022 cohorts (n = 34). There were no statistically significant differences between pre-completion scores over the 3 years (p = 0.99). Twenty-two out of the 26 competencies saw a statistically significant increase from pre to post training proficiency (n = 18). The exceptions were in writing papers, writing grants or business proposals, statistical analysis to pick up signal from noise, and traditional research methodology.

Regarding the frequency of utilizing the tools, processes, and strategies obtained from the program (see Table 3), the average scores ranged from monthly to weekly use. Graduates claimed to use Agile processes (Innovation, Implementation, or Diffusion), storytelling, and nudging weekly. The majority of graduates deliberately brand themselves and review their playbooks monthly.

Based on information from publicly available databases, 65% of graduates had a promotion and/or have since pursued an additional degree or professional training. The average number of followers was 846 based on available LinkedIn™ accounts. The average number of years in their current position was 4.11 years (SD of 4.73). For career advancements, the average number increased from 0.25 one year out to two changes 6 years out (see Table 4).

We found that the IIS Certificate Program has impacted graduates in terms of their professional careers, network expansion, change management skills, communication, and leadership. The program was also successful in building a network of Agile Change Conductors who are strong promoters of the program and competent in implementing and diffusing evidence-based care in their local complex adaptive healthcare delivery organizations. This indicates the Certificate Program is an effective way to create Agile Change Conductors capable of designing, implementing, and diffusing evidence-based healthcare solutions.

We used the Net Promoter score to measure the graduates' attitudes toward the program. A score above 70 is considered excellent (28). The program had a Net Promoter score of 67 indicating overall satisfaction with having taken the Certificate Program. Coupling this score with the high impact of the certificate on professional careers, the high competency of the graduates, and their high use of the tools, processes, and strategies obtained via the program exemplifies the success of the program in building an integrated network of Agile Change Conductors.

The frequency of usage section in the survey helps monitor the influence of the program on the daily activities of graduates. While the scores had more variation between graduates, the results were still overwhelmingly positive. No question had an average score below 2.5, which translates onto monthly use. Weekly use was the most common response for using Agile Processes (Implementation, Innovation, and Diffusion), storytelling, and nudging, demonstrating a high amount of practice. The integration of course material into the habitual lives of the graduates is a testament to sustained impact that goes beyond knowledge acquisition.

Majority of graduates did not have an active LinkedIn™ account prior to enrollment in the Certificate Program. Students without accounts were strongly encouraged to create a profile for the purposes of branding and networking for resources and discovery. Therefore, 92% of the graduates having an active account with an average of over 800 connections reasonably demonstrates the Certificate Program expanded professional networks. From profiles, there was professional advancement of most members (65%), in line with most graduates agreeing that the program benefited them professionally through position changes, continued education advances, and/or achievements.

When looking at the success of the program in creating well-rounded Agile Change Conductors equipped with transferable change management skills, the proficiency test indicates graduates are competent and nearing mastery. For each of the 26 competencies, the data were skewed left with averages showing nearly all members were more competent than not. The standardized increase in competency for nearly all skills of an Agile Change Conductor in a subsample of the study population is indicative of some causal benefit from the program. Proficiency did not appear to decline, given no variation between the 2016 cohort surveyed 6 years following graduation and the 2021 cohort surveyed 1 year following graduation. There was no decline in competency for either the 2020 or 2021 cohorts despite the shift to fully virtual learning. These results indicate satisfactory improvements in each of the intended competencies, highlighting the Graduate Certificate Program has thus far met its professed goals of training members in innovation and implementation science.

The University of California San Francisco Certificate Program in Implementation Science has similarly seen moderate to high confidence in 12 self-assessed competencies (n = 54, 2008–2017), coming out in strong support of the benefits of graduate-level training in implementation science career development (29). These results match the current study's moderate to high levels of mastery for all competencies. The University of South Florida offers an Implementation Science Graduate Certificate Program through The Institute for Translational Research Education in Adolescent Drug Abuse (30). Using semi-constructed interviews (n = 58), the program has been found to create a robust and transferable skillset for researchers and healthcare practitioners (30). Despite the success of the IU-CHIIS Graduate Certificate Program and similar ones across the nation, there is still demand for an increases in the number of advanced education programs in innovation, implementation, and diffusion sciences to systematically increase the use of evidence-based health practices (27). Continued expansion of Agile Diffusion and improvements in developing signal-noise detection, leadership, and branding skills will prove crucial for steering information and behavior change in widespread complex adaptive human networks (24). Disseminating the IU-CHIIS curriculum seeks to provide a framework for future educational programs capable of creating Agile Change Conductors within the healthcare delivery system.

Competencies with the lowest averages were the ability to write grants or business proposals and leading 150 people toward a goal, both of which have larger amounts of variability likely due to how applicable these specific competencies are to some professions compared to others. Areas without significant improvements also included writing papers and traditional research methodology. Selective bootcamps on Agile Science specified for professionals in healthcare delivery business operations, implementation and innovation science research, or healthcare administrative leadership are advised for directed aptitude development in drafting business proposals, writing papers or research methodology, and grant writing or leading 150 people toward a goal, respectively. Signal-noise detection was another Agile competency with space for improvement. Continued research at IU-CHIIS on Agile analytics, with better ways to distinguish signals from noise will help address such gaps. The development and addition of an immersive simulation-based portion of the curriculum in the near future will likely provide further opportunities to develop mastery of the core principles of Agile Science while providing objective data on the ability of graduates to optimally capitalize on their training. A qualitative study interviewing past graduates will be needed to provide direct feedback and recommendations for future years.

There remain several limitations to this study. First, since the primary source of data was a survey with subjective responses, this provides room for bias in the responses. Responses may be less critical of the program. The survey was de-identified but not anonymous due to the necessity of knowing cohort years and to help monitor which alumni had yet to complete the survey. However, data was analyzed after being de-identified, blinding researchers to identifiable information. Furthermore, even though the response rate was high (87%), the survey lacked eight responses to make the data complete. It is therefore possible the data is not completely representative of all past cohorts, especially early cohorts with lesser response rates. A significant limitation of this study is the lack of baseline competency data for cohorts 2016–2019, greatly reducing the comparative aspect of this study. Majority of conclusions are made using descriptive statistics. While the absence of variation in competency across cohorts may help generalize the 2020–2021 pre-post training findings more broadly, the small sample size limits direct causal attributions. Additionally, there are likely large parts of careers not posted on LinkedIn™. It is difficult to know the extent to which career advances seen online were due to the program, as is also the case with follower counts. Despite these limitations, the data overwhelmingly points to a beneficial impact of the program.

In conclusion, the Certificate Program in Health Innovation and Implementation Sciences has been successful in creating Agile Change Conductors who have been able to act upon the set of learned tools, processes, and strategies to transform the healthcare delivery system.

The original contributions presented in the study are included in the article/Supplementary material, further inquiries can be directed to the corresponding author.

The studies involving human participants were reviewed and approved by Indiana University Institutional Review Board (#15824). The patients/participants provided their written informed consent to participate in this study.

JM: primary author of the paper, designed, conducted, and analyzed the study, drafted and edited the paper, and developed all supplementary materials. MA: developed idea and initiative for paper and edited paper and survey throughout the process. AO'B, TB, RA, and DS: editors for the paper and made significant contributions to the rhetoric and outstanding ideas presented in the final version. MB: principle investigator, helped design, conducted, analyzed, and wrote about the study. All authors contributed to the article and approved the submitted version.