ADHS and the curriculum workgroup worked together to develop program and learner evaluations of the Arizona Pain and Addiction Curriculum that would be administered annually.

The program evaluation collected information from each program on their progress, challenges and best practices of implementation. Although curriculum management and evaluation are considered in most accreditation standards, no such existing program evaluation was identified for a curriculum on pain and addiction, nor cross-program evaluations of pain and addiction curricula implementation. The curriculum developed in Massachusetts was reported to be implemented in all four medical schools, but no details on the assessment were provided.

The program evaluation was designed to assess curriculum implementation, with a goal of identifying successes and challenges in order to shape future curricular iterations. Given the public health interest in a statewide uptake in order to transform education and the provider practices that follow, questions were developed to assess the degree of implementation, difficulty of implementation and methods of content delivery. Given the novel material and approach, questions also included assessment of faculty development and exploration of learner's didactic dissonance (disconnect between what is taught and what is seen in practice). And last, given the role of the pharmaceutical industry on the opioid epidemic and the public health goal of preventing similar epidemics, questions were developed to assess programmatic approaches to learner's experience with industry.

Questions were refined by the curriculum workgroup during two in-person meetings in 2018 with consideration to relevance and brevity. The final program evaluation consisted of 20 questions (including multiple choice, Likert scales, and open-ended questions) that collected information about: Degree of implementation, Difficulty of implementation, Delivery methods (including those in the Toolbox for Operationalization), Faculty Development, Didactic dissonance and Discussion Opportunities (“6 Ds” of curriculum implementation) (see Appendix). Several questions touched on implementation of adult learning theories. The program evaluation was reviewed and edited by reviewers from Johns Hopkins and the Mayo Survey Research Center before administration (see Acknowledgments). The first program evaluation to be administered in 2019 was to serve as a baseline, presumably before the Arizona Pain and Addiction Curriculum was fully implemented across programs statewide.

The first learner evaluation was simultaneously administered in 2019 with a survey that collected information from learners about their knowledge, attitudes and plans for treating patients with pain and/or addiction, and these results will be presented in a separate paper.

On April 4, 2019, the director of the Arizona Department of Health Services sent an email to the deans and curricular representatives of each health educational program in Arizona, requesting prompt and thorough completion of the program evaluation. On April 5, 2019 and May 15, 2019, links to the evaluation were sent out to the same program recipients by another physician at the Arizona Department of Health Services. The program evaluation was expected to be completed by one faculty member in each program familiar with the Arizona Pain and Addiction Curriculum by June 15, 2019.

Completion of the evaluation was not mandatory, but communication from the Arizona Department of Health Services stressed that strong, statewide data would benefit the state and its educational approach.

Qualtrics software (Qualtrics, Provo, UT) was used to collect responses. Responses were processed and analyzed using SAS 9.4 (SAS Institute, Cary NC). Simple frequencies and percentages were completed for each response question. Additional processing was performed to produce interquartile and Heat Map findings in Figure 1.

In total, 14/18 Arizona health education programs completed the Arizona Pain and Addiction Curriculum program survey, representing 78% of programs in the state. Every type of health education program in Arizona (MD, DO, NP, PA, DMD, ND and DPM) was represented in the responses (Table 1).

The number of students that received components of the Arizona Pain and Addiction Curriculum peaked in the second and third years of training (Table 2). Of note, not all programs contain a fourth year of training.

10/14 (71%) of programs responded that all ten Core Components of the Arizona Pain and Addiction Curriculum were included in their curriculum in the past academic year. Within the options of Fully Implemented, Partially Implemented or Not at All Implemented, 50% or more of programs reported having fully implemented all but one of the ten Components (Table 3).

Thirteen of the 14 programs completed the implementation section. The majority of programs reported that Components were not difficult to implement. Of the 130 total component responses, 103/130 (79%) were reported as “Extremely easy,” “Moderately easy,” or “Neither easy nor difficult. Component 10 had the greatest level of reported difficulty by programs; more than one-third of responding programs indicated that Component 10 was “Extremely difficult” or “Moderately difficult” to implement (Figure 1). Although the free-text answers did not specifically reference the difficulty of Component 10, they did mention struggles with faculty development, culture change, and finding sufficient time required to teach the material. On the other hand, there were also comments that implementation was not difficult and that the curriculum and accompanying Faculty Guide were clear.

All responding programs reported implementing Components through lectures (7/13, 54%), online modules (2/13, 15%), or a combination of both lecture and online modules (4/13, 31%). In the free-text section asking for best practices, programs reported successful models of partnering with addiction clinics and specifically choosing to focus on pain and addiction content during the clerkship years.

77% (10/13) of programs reported not having a process to ensure clinical rotation supervisors are teaching content consistent with the Arizona Pain and Addiction Curriculum (Figure 2).

77% (10/13) of programs reported not discussing students' observations after clinical rotations about pain and addiction care, and how it may or may not differ from what was taught in the Arizona Pain and Addiction Curriculum (Figure 2).

Programs reported 2,407 total students having received specific didactic training about the potential influence of industry on practice, with the plurality in Year 2 (Year 1-566; Year 2-873; Year 3-598; Year 4-370).

However, most programs did not report specifically asking students about their interactions with industry representatives upon completion of clinical rotations (77%, 10/13). In the free-text section asking about unique examples of training on industry influence, several programs wrote that they did not have any; another suggested that students are welcome to interact with industry with their preceptor's approval and the program “[does] not seek to control that.” One osteopathic program described having a specific lecture about industry influence during orientation and embedding further industry topics during Pharmacology units.

Program participation in the baseline evaluation was notable and included the majority of programs in the state and representation from every program type. Over 2,000 students received at least some component of the curriculum already by spring 2019, suggesting widespread implementation across Arizona.

It was unexpected to find that most programs reported having fully implemented each Component of the Curriculum. This either reflects program's unexpectedly strong momentum to make curriculum changes or that the questions were phrased broadly as to ask about the primary concept of each Component rather than the more nuanced Objectives under each Component. Results from future evaluations may help to clarify this and what “fully implemented” means to each program.

Most programs did not find the Arizona Pain and Addiction Curriculum difficult to implement. The highest difficulty ratings for Component 10 implementation is understandable yet unfortunate, as skepticism and solving problems at a systems-level are difficult to canonize but essential for future improvement.

One identified area of growth is the need to implement adult learning principles. The use of lectures for all components was predominant, with fewer programs reporting use of standardized patients, small group activities, or other examples from the Toolbox of Operationalization. This is important given the transformative nature of the curriculum and the recognition that more active learning approaches result in more profound levels of internalization and integration for learners. Recognizably, these approaches such as small group discussions, interactions with patient panels (patients recovering from pain and/or addition), and rotations with community mutual support groups (e.g., Alcoholics Anonymous or Narcotics Anonymous) are more difficult for programs to organize and maintain than a single didactic lecture.

Most schools did not have a system in place to ensure that faculty were familiar and teaching concepts in line with the Arizona Pain and Addiction Curriculum, but we hope this improves over time. Faculty development and familiarity with the new and transformative concepts in the curriculum is key to its success, and the Faculty Guide was created to help with that. Annual meetings with the curriculum workgroup will guide the production of further tools to assist faculty comfort with the new material. It remains to be seen the degree of engagement the faculty will have with the comprehensive material presented in the Faculty Guide and methods recommended in the Toolbox for Operationalization.

There is also an opportunity for programs to delve deeper into the concept of “didactic dissonance.” This curriculum reflects a modern approach to pain and addiction and a cultural transformation from prior teaching, and learners are expected to experience some degree of disconnect between what is taught in the curriculum and what they see in practice. By including the concept of didactic dissonance in the Toolbox for Operationalization, the curriculum aims to turn this ostensible barrier to education into an opportunity to deepen and reinforce learning and cultural transformation. Others have published on the hidden curriculum's exposure to learners (11) and the American College of Physicians recommends that faculty “encourage reflection and discussion of positive and negative behaviors in the training environment” (12). The impact of this curriculum is likely to be diminished if the didactic dissonance is not openly addressed.

Similarly, addressing the impact of the pharmaceutical and device industries on students during their experience on clinical rotations, particularly with external preceptors where school policies on pharmaceutical influence are not observed, would be beneficial. Contact with industry influences trainee and health professional behavior (13) and was an important factor in the development of the opioid epidemic (14). Without circling back to discuss experiences that students have with industry, this is likely a missed opportunity to reset standards. There were very few examples of best practices on pharmaceutical relationship education; only one osteopathic program actively addressed industry influence throughout their 4 year curriculum.

One limitation on this study includes the timing of the evaluation. Different health education programs in Arizona vary in terms of starting and ending dates and may have affected the degree of program implementation.

Another limitation is response bias, as the respondents were not blinded. It is possible that the programs wanted to show how their program was “putting its best foot forward.”

A final limitation is the lack of standardization of who completed the evaluation. Anecdotal reports of who completed the evaluation on behalf of the program ranged from the original curriculum workgroup to other deans that had not participated in the development or implementation. The evaluation was straightforward and rapid if the person completing it was familiar with the Curriculum; it was onerous and long if the person was not. This may have impacted the quality of results.