ORIGINAL RESEARCH article
Sec. Implementation Science
Extent, Type and Reasons for Adaptation and Modification When Scaling-Up an Effective Physical Activity Program: Physical Activity 4 Everyone (PA4E1)
- 1School of Medicine and Public Health, University of Newcastle, Callaghan, NSW, Australia
- 2Hunter New England Population Health, Wallsend, NSW, Australia
- 3Hunter Medical Research Institute, New Lambton Heights, NSW, Australia
- 4Priority Research Centre for Health Behaviour, University of Newcastle, Callaghan, NSW, Australia
Background: Few studies have described the extent, type and reasons for making changes to a program prior to and during its delivery using a consistent taxonomy. Physical Activity 4 Everyone (PA4E1) is a secondary school physical activity program that was scaled-up for delivery to a greater number of schools. We aimed to describe the extent, type and reasons for changes to the PA4E1 program (the evidence-based physical activity practices, implementation support strategies and evaluation methods) made before its delivery at scale (adaptations) and during its delivery in a scale-up trial (modifications).
Methods: The Framework for Reporting Adaptations and Modifications-Enhanced (FRAME) was used to describe adaptations (planned and made prior to the scale-up trial) and modifications (made during the conduct of the trial). A list of adaptations was generated from a comparison of the efficacy and scale-up trials via published PA4E1 protocols, trial registrations and information provided by trial investigators. Monthly trial team meetings tracked and coded modifications in “real-time” during the conduct of the scale-up trial. The extent, type and reasons for both adaptations and modifications were summarized descriptively.
Results: In total, 20 adaptations and 20 modifications were identified, these were to physical activity practices (n = 8; n = 3), implementation support strategies (n = 6; n = 16) and evaluation methods (n = 6, n = 1), respectively. Few adaptations were “fidelity inconsistent” (n = 2), made “unsystematically” (n = 1) and proposed to have a “negative” impact on the effectiveness of the program (n = 1). Reasons for the adaptations varied. Of the 20 modifications, all were “fidelity consistent” and the majority were made “proactively” (n = 12), though most were “unsystematic” (n = 18). Fifteen of the modifications were thought to have a “positive” impact on program effectiveness. The main reason for modification was the “available resources” (n = 14) of the PA4E1 Implementation Team.
Conclusions: Adaptations and modifications to public health programs are common. Modifications have the potential to impact the implementation and effectiveness of programs. Our findings underscore the importance of comprehensive reporting of the extent, type and reasons for modifications as part of process evaluations, as this data may be important to the interpretation of trial findings.
Clinical Trial Registration: https://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?id=372870, Identifier ACTRN12617000681358.
Physical activity has extensive benefits for health and society (1, 2). One in four adults and four in five adolescents globally are insufficiently active to meet aerobic physical activity guidelines (3, 4). While there are an abundance of evidence-based programs to address physical activity, many of these have been tested under optimal research conditions (5, 6), and few have successfully targeted adolescent physical activity (7, 8). As such, many programs previously tested have utilized technical expertise, skills, resources and infrastructure that are not common in real-world operational environments where they are intended to be implemented (9, 10). Further, research trials often recruit participant groups that differ markedly from those of the target population (11).
As physical activity programs examined in research trials are often unsuitable for replication in more real-world environments, they are frequently changed by end-users as part of efforts to make them more suitable for implementation and scale-up (9, 12). These changes can take two forms – adaptations to a program prior to it being delivered, and modifications that occur during the delivery of the program. Adaptation (8, 13) has been defined as a process of thoughtful and deliberate alteration to the design or delivery of a program, with the goal of improving its fit or effectiveness in a given context (13). Program adaptations can include both adaptations to the evidence-based practices and/or to the implementation support strategies provided to increase adoption of the practices in the setting (such as training for clinicians or teachers who will be delivering the program). Adaptations include those to core components of the program, cultural adaptations, mode of delivery adaptations, target audience adaptations and service setting adaptations (14). Modification has been defined as encompassing any change to a program, whether deliberately and proactively, or in reaction to unanticipated challenges that arise in the context of its delivery (13). Adaptations and modifications can also be made to evaluation methods.
Systematic reviews demonstrate that program adaptations are ubiquitous as part of efforts to scale-up programs in practice. For example, a systematic review of physical activity programs (8) reported that 100% of programs made adaptations to the program tested in an efficacy trial prior to undertaking a trial of its scale-up. The majority of adaptations focused on the “delivery mode” of programs (8, 15, 16), such as giving preference to online or telephone over face-to-face delivery modes, which are often undertaken to enable greater program reach (8). Understanding program adaptations and modifications is important as they can have significant implications to the effectiveness of programs (9, 12, 17). They have been attributed, in part, to a phenomenon labeled “voltage drop” whereby the effects of a program are reduced by 25–50% when they are implemented at scale in real world contexts (8, 16, 17). However, they have also been hypothesized to improve the impact of programs. For example, improvements may be made by allowing tailoring of evidence-based programs and their implementation (i.e., the local culture, historical context, priorities and availability of funding, staffing and resources), strengthening key program components, reducing inequities by improving its cultural relevance, or reducing relative costs via delivery using less expensive modalities (12). Understanding the nature of program adaptations and modifications is also important for the development of explanations about how they may impact program implementation and outcomes as part of trial process evaluations (18).
The Framework for Reporting Adaptations and Modifications-Enhanced (FRAME) (13, 19) was recently developed to support the consistent documentation and reporting of program adaptations and modifications. It provides a taxonomy of classifying adaptations and modifications (13) including what is adapted/modified, the nature of the adaptation/modification, who participated in the adaptation/modification decision, for whom/what is the adaptation/modification made and when it occurred. Despite the existence of FRAME and the need for consistent reporting, both adaptations made prior to program delivery and modifications made during the implementation are often poorly described in research reports (13, 15, 20, 21). That is, individual trials seldom report adaptations for scale-up (prior to program delivery) using consistent taxonomies (13), instead trials rely on descriptions of adaptations that can't be compared between trials (13).
A systematic review of 42 evidence-based public health programs that reported adaptations to the evidence-based program practices (15) found that the most frequent types of adaptation were tailoring (93%) or adding elements (71%). Most commonly these adaptations were to content (100%), context (95%), cultural (74%) and/or delivery (62%). While the review provides useful insights into the frequency of adaptations to the evidence based program practices, it does not explicitly include adaptations to the implementation support strategies used or the evaluation methods (13, 22). Also, the authors relied on published papers to retrospectively code adaptations to evidence-based program practices (15). A limitation of relying on published papers is that sometimes the extent, types, context and reasons for adaptations and modifications may be unclear or absent completely from these documents, remaining instead with those people involved in the scale-up process (15). Additionally, it is also unclear in many programs what modifications occur during delivery, and to the authors' knowledge, no physical activity studies have used a consistent taxonomy to report modifications during program delivery (23). Importantly, prior studies have also not routinely reported who was responsible for program adaptations or modifications, why these were undertaken, and if they were considered to contribute, or detract, from the effects of the program. Such information could be used to help interpret trial findings in implementation-effectiveness studies (13, 22, 24).
In the absence of well-described adaptations prior to delivery and modifications during delivery, we present here a descriptive study of the adaptations and modifications made in the scale-up of an evidence based physical activity program targeting adolescents, Physical Activity 4 Everyone (PA4E1). PA4E1 is a secondary school physical activity program. After an efficacy trial, PA4E1 showed positive results (25–29), PA4E1 was adapted in preparation for scale-up (30). The PA4E1 program includes both an evidence-based program (consisting of seven school physical activity practices) and seven implementation support strategies offered to help schools implement these physical activity practices (implementation support). The aims of the current paper are:
1. To describe the extent, type and reasons for adaptations to PA4E1 that were made for scale-up to the physical activity practices, implementation support strategies and evaluation methods.
2. To describe the extent, type and reasons for modifications during the PA4E1 scale-up trial made to the physical activity practices, implementation support strategies and evaluation methods.
This research has been conducted and reported in accordance with the requirements of the Standards for Reporting Implementation Studies (StaRI) Statement (Additional File 1) and Template for Intervention Description and Replication (TIDieR) checklist (Additional File 2).
The efficacy and scale-up trials have been registered at ACTRN12612000382875 and ACTRN12617000681358, respectively. Ethical approvals were sought from Hunter New England Human Research Ethics Committee (Ref No. 11/03/16/4.05), University of Newcastle (Ref No. H-2011-0210), NSW Department of Education and Communities (SERAP 2011111), Maitland Newcastle Catholic School Diocese, Broken Bay Catholic School Diocese, Lismore Catholic School Diocese, Armidale Catholic School Diocese, and the Aboriginal Health and Medical Research Council (AHMRC).
Stages of Physical Activity 4 Everyone (PA4E1)
An outline of the physical activity practices (evidence based program) for both the efficacy and scale-up trials are shown in Figure 1. The implementation support strategies offered to schools are outlined for both trials in Figure 2 (26, 27, 29–32). As is best practice in implementation science (33), we distinguish between program components, separating the evidence-based program practices (the physical activity practices) from the implementation support strategies, which are designed to assist schools to implement the physical activity practices.
Efficacy Trial (2012–2014)
The evaluation methods of the PA4E1 efficacy trial have been reported in a trial protocol (26). Briefly, the PA4E1 efficacy trial was a 2-year (2012–2014) cluster randomized controlled trial involving 10 low-socioeconomic Australian secondary schools (five per group). Hunter New England Local Health District (HNELHD) led PA4E1 (34), supported by two other local health district delivery partners (26, 34) in a research-practice partnership with the University of Newcastle and New South Wales (NSW) Department of Education. PA4E1 had positive effects on students' device-measured moderate-to-vigorous physical activity and unhealthy weight gain (25–27, 29) and was deemed cost-effective (28). The PA4E1 program consisted of seven physical activity practices (Figure 1) to support students to be more physically active (the evidence-based program) and six implementation support strategies designed to embed the physical activity practices within the school environment (Figure 2).
Adaptation Process (2017)
Adaptations were made in 2017 with the goal of scaling up the program (physical activity practices and implementation support strategies) employed in the efficacy trial and testing the effects again as part of a scale-up trial. The process has been reported in more detail elsewhere (30). The scale-up adaptation process sought to retain the effects of the original program by retaining components deemed as core (physical activity practices and implementation support strategies) (25–27, 29) while enabling greater reach (scaling up to more schools). Briefly, to adapt PA4E1 for scale-up, we used a four-stage iterative scale-up process, based on a review of existing models and factors for scaling up public health programs (35) and a scoping review of frameworks for adapting public health programs (36). Firstly, we identified barriers and enablers to the physical activity practices and implementation support strategies. Second, we mapped the identified barriers to the Theoretical Domains Framework (37) and the Behavior Change Wheel (38). Thirdly, we prioritized the components of the program from the perspective of a health service requiring its delivery at scale, considering variables such as affordability, practicability and acceptability (39). Finally, the PA4E1 Expert Advisory Group (comprised of senior health service staff, senior academics, NSW Education sector partners and the PA4E1 project staff) reviewed the prioritized program (physical activity practices and implementation strategies) and made the final judgement regarding the design and components of the resulting PA4E1 program (30).
Scale-Up Trial (2017–2019)
The scale-up trial was a type III hybrid implementation-effectiveness trial (24). Methods have been reported in both a trial protocol (30) and a process evaluation protocol (32). The scale-up trial (2017–2019) tested the adapted PA4E1 program in a larger number of low-socioeconomic secondary schools (n = 49) across a larger geographic area (HNELHD leading three other local health districts, in a research-practice partnership with the University of Newcastle and NSW Department of Education) (34). Program schools were offered seven implementation support strategies (incorporating 23 sub-strategies) to support their adoption of seven physical activity practices (30–32). The scale-up trial recruited 49 schools (24 program, 25 control) (31).
Defining Adaptation and Modification
We operationally defined changes to PA4E1 (inclusive of the physical activity practices, implementation support strategies and the evaluation methods) temporally, as either adaptations which were planned and made prior to the scale-up trial or modifications which were made during the conduct of the trial. The methods are reported by aim. Aim one focuses on adaptations (prior to scale) and aim two focuses on modifications (during implementation of the program in the scale-up trial).
Measures and Procedures
The measures and procedures are reported by aim.
Aim 1: To describe the extent, type and reasons for adaptations to PA4E1 that were made for scale-up to the physical activity practices, implementation support strategies and evaluation methods.
We used the FRAME framework to code the adaptations to the PA4E1 program made for scale-up (13). We applied the FRAME framework coding to the evidence-based practices (physical activity practices), the implementation support strategies (13) and the evaluation methods. The FRAME framework was used to provide a discrete set of codes for each category of adaptation (see Table 2 for a list of codes for each category). Additional coding categories (outlined below) were developed by the author team in line with methods from Rabin et al., (23) to report free-text descriptions of each adaptation.
Table 2 shows the adaptation categories utilized and the response codes. Briefly, these are:
• Description of the adaptation (open text)*
• What component of the PA4E1 program was the adaptation made to (the physical activity practices, implementation support strategies or evaluation methods)?
• Was the adaptation proposed to have a positive, null or negative impact on the program effectiveness at the time of adaptation?
The following categories were coded for each adaptation from the FRAME framework (6):
• Were adaptations systematic or unsystematic?
• Note: Based on the Model for Adaptation and Impact (MADI) framework (20), we revised the FRAME framework (13) terminology to remove the code “planned” and replace it with the code “systematic.” This emphasizes the importance of how the modification was made (i.e., was it done using a systematic process which involved the use of theory to make the adaptation?). We operationalized the code “systematic” and applied it to adaptations that used both theory and a process to make the adaptation. By contrast, “unsystematic” was the code assigned when either a theory was not used or a process was not used to make the adaptation.
• Were adaptations proactive or reactive?
• Who participated in the decision to adapt? (e.g., program manager, individual practitioners such as in-School Champion)
• What was the goal of adaptation? (e.g., improve feasibility, reduce cost)
• What was adapted? [i.e., context, training and evaluation, implementation and scale-up activity, content (including aspects of the way content was delivered)]
• Context adaptations were made to what? (e.g., format, setting)
• At what level of delivery did adaptations occur? (e.g., unit level - an individual school)
• What is the nature of the content adaptation? (e.g., adding elements, shortening/condensing)
• Were adaptations fidelity consistent (relationship to core elements retained) or fidelity inconsistent?
• Reasons for the adaptation? (including socio-political, organization/setting, provider and recipient reasons).
Certain coding categories were considered “not applicable” to the evaluation methods (as outlined in Table 2).
Firstly, MM identified a list of adaptations between the efficacy and scale-up trial on physical activity practices, implementation support strategies and evaluation by triangulating data from a number of sources. The first source was published PA4E1 research papers: the trial protocol (26), 12 and 24-month outcome papers (27, 29) and cost-effectiveness paper for the efficacy trial (28); and the trial protocol (30), process evaluation protocol (32) and 12-month practice outcome paper (31) for the scale-up trial. Secondly, we drew upon the trial registries for both trials. Finally, to provide important context and resolve discrepancies in recorded adaptations, we drew on the historical knowledge of three authors (LD, EC and RS) who were involved in both trials through meetings with the lead author (MM).
To refine the initial codes, a consensus meeting was held between MM, TM, LD, RS, and EC. Following discussion, MM then finalized the coding for each of the adaptations. Finally, the final codes were agreed upon by email by MM, TM, LD, RS, and EC. Descriptive statistics for each adaptation category were calculated (e.g., the number and type of adaptations that were made).
To synthesize the adaptations, we calculated descriptive statistics for each adaptation category (e.g., the number and type of adaptations that were made) overall, and for adaptations made to physical activity practices, to implementation support strategies and to evaluation.
Aim 2: To describe the extent, type and reasons for modifications during the PA4E1 scale-up trial made to the physical activity practices, implementation support strategies and evaluation methods.
We used the FRAME framework to describe and code modifications made during the delivery of the scaled-up PA4E1 program to the physical activity practices, the implementation support strategies and the evaluation methods.
We used the method as outlined in the process evaluation protocol (32). Throughout the 24-month program (2017–2019), Support Officers involved in the delivery of the program to schools and the PA4E1 Management Team involved in the day-to-day operations of the project continually tracked modifications to PA4E1 during the program in “real-time” by adding them to a Microsoft Excel spreadsheet (23). A monthly meeting (up to 60 min) was held between the PA4E1 Management Team (including at least one Support Officer) to code the identified modifications onto the Stirman et al., (13) framework for modifications in the same spreadsheet (19). MM subsequently updated the coding to reflect the additional categories included within the updated framework by Stirman et al., the FRAME framework (13, 19). This method of ‘real-time ongoing coding’ has previously been found to be feasible to track modifications (23). We used the same coding categories from Aim 1 for each modification (replacing the word adaptation for modification), as well as coding two additional categories. For the category of proposed impact on the program effectiveness at the time of modification (i.e., “positive”, “null” or “negative”), we coded this based on the predicted impact of the modification at the time of coding, rather than the actual or measured impact of the modification.
• Reason for the modification (open text)
• School term the modification was first applied (within the nine school term program).
At the end of the program, the final codes were collated by MM. The final codes were discussed and refined (by RS, EC, LD, TM) to reach consensus. Descriptive statistics for each modification category were calculated (e.g., the number and type of modifications that were made) overall and separately for physical activity practices, implementation support strategies and evaluation methods.
Results are reported by adaptations (aim 1) and modifications (aim 2), respectively.
Aim 1 was to describe the extent, type and reasons for adaptations to PA4E1 that were made for scale-up to the physical activity practices, implementation support strategies and evaluation methods. Table 1 descriptively summarizes the main program adaptations from efficacy trial to scale-up trial for individual physical activity practices and implementation support strategies, including the codes “fidelity consistent or fidelity inconsistent”, “systematic or unsystematic” and “proposed positive, negative or null impact on the project” (as described in the methods). For a more expanded description of adaptations made for the scale-up trial and their coding, see Additional File 3.
Table 1. Adaptations to each specific component of PA4E1 efficacy trial made for the scale-up trial, including adaptations to individual school physical activity practices and implementation support strategies offered to schools.
Number of Adaptations
A total of 20 adaptations were made to scale-up PA4E1. Eight adaptations were to the school physical activity practices, six to the implementation support strategies and the remaining adaptations were to the evaluation methods (n = 6). Table 2 summarizes the codes according to the FRAME framework for physical activity practices, implementation support strategies and evaluation methods, respectively (13). Additional File 3 outlines each individual adaptation in detail and includes the full set of codes outlined in the methods.
Table 2. FRAME framework (13) modification codes for adaptations from efficacy to scale-up trial.
The vast majority of adaptations were coded as “systematic” (n = 19), as they were made during the theory-informed iterative scale-up process. By definition, all adaptations (n = 20) were made “proactively” rather than in response to an unknown event or circumstance. Two adaptations were deemed “fidelity inconsistent”, as the core elements or functions had changed as a result of the adaptation (13). Of the 14 adaptations to the practices and strategies, 12 were proposed to have a positive impact on program effectiveness. All adaptations involved both the “Program Manager” and “Treatment/Intervention Team” in the decision-making process, which included the PA4E1 Implementation Team (inclusive of the program manager, project staff and the expert advisory group) as described in the methods.
Types of Adaptations
The goals and types of adaptations varied. The most common goals were to “improve fit with recipients” (n = 8), “improve feasibility” (n = 5), “improve effectiveness/outcomes” (n = 4) and to “reduce cost” (n = 3). Most adaptations were to “content” (n = 13) and “implementation and scale-up activities” (n = 12). Fewer adaptations were to “training and evaluation” (n = 6) or “contextual” (n = 1). All adaptations (n = 20) were made at the “target intervention group” level, meaning that adaptations applied to all schools receiving the program, rather than certain schools or local health districts (e.g., “individual”, “cohort” or “individual practitioner” level). Content adaptations varied, including “substituting” (n = 8) “tailoring/tweaking/refining” (n = 3) and “shortening/condensing” (n = 3).
Reasons for Adaptations
The reasons for adaptations included the broad context of having “funding or resource availability/allocation” (n = 3) and also the PA4E1 Implementation Team having “available resources (funds, staff, technology, space)” (n = 6). School and in-School Champions (i.e., “recipients”) reasons included cultural or religious norms (n = 4), “physical capacity” (n = 2), “motivation and readiness” (n = 2) and “access to resources” (n = 2).
Aim 2 was to describe the extent, type and reasons for modifications during the PA4E1 scale-up trial made to the physical activity practices, implementation support strategies and evaluation methods. Table 3 provides a brief description of each modification made to the physical activity practices and implementation support strategies during the scale-up trial delivery.
Table 3. Modifications to each component of PA4E1 scale-up trial made during the delivery of the program (practices and implementation support strategies).
Number of Modifications
A total of 20 modifications were made during the delivery of the scale-up trial of PA4E1 from 2017–2019. Of these, 16 modifications were made to the implementation support strategies, three to the physical activity practices and one to the evaluation methods. All modifications were deemed “fidelity consistent” and most modifications were proposed to have a positive impact on the effectiveness of the program (n = 15). Table 4 summarizes the modification codes according to the FRAME framework (13). Additional File 4 outlines each individual modification in detail and includes the full set of codes (as outlined in the methods).
Types of Modifications
The vast majority of modifications were “unsystematic” (n = 18), because most modifications did not make deliberate use of theory to make the modification. Two modifications made to support sustainability used theory and a specific process, and therefore were deemed systematic. Most modifications (n = 12) were made “proactively”, however eight modifications were made in response to an unknown event or circumstance and were therefore coded as “reactive”. Most modifications involved the “program manager” (n = 17). “Individual practitioners” participated in half of all decisions (n = 10). Other groups and individuals participated in the decision making process less frequently, including the whole PA4E1 Implementation Team “treatment/intervention team” (n = 3), local health districts “administrator” (n = 2) and in-School Champion “recipients” (n = 1).
The goals and types of modifications varied. The most common goals were to “improve fit with recipients” (n = 7), “increase satisfaction” (n = 6), “improve effectiveness/outcomes” (n = 5), “increase reach or engagement” (n = 4) and to “increase retention” (n = 4). All modifications were to either “content” (n = 14) or “training and evaluation” (n = 6). The nature of content modifications varied, including “adding elements” (n = 5), “tailoring/tweaking/refining” (n = 3) and “removing/skipping elements” (n = 2). Most modifications (n = 15) were made across all schools, in all local health districts, and were therefore coded the “target intervention group” level. The remaining modifications occurred at particular local health districts “cohort level” (n = 3) or individual schools “clinic/unit level” (n = 4).
Reasons for Modifications
Reasons for modifications were primarily related to the “available resources” (funds, staff, tech, space) (n = 14) of the PA4E1 Implementation Team, i.e., the “provider”. The second most common reason was schools and in-School Champions (i.e., recipient) “motivation and readiness” (n = 8). Further reasons for modification were few, but included “historical context” (n = 1) and the “social context” (n = 1) surrounding the “PA4E1 Implementation Team”.
To the authors' knowledge, this is the first study to use a taxonomy to comprehensively report the number, types and reasons for both adaptations made to scale-up a program and also the modifications that occurred during the delivery of the scaled-up program. Our findings show that 20 adaptations were made to the PA4E1 program for scale-up (26, 27, 29), including eight adaptations to the physical activity practices, six to the implementation support strategies and six to the evaluation methods. Most adaptations were proposed to have a positive impact on the effectiveness of the program (n = 12). Additionally, 20 modifications were made during the delivery of the scaled-up program, of which 16 were proposed to have a positive impact on the effectiveness of the program. Most modifications were to the implementation support strategies (n = 16). Given that the use of adaptation and modifications data is being encouraged to explain the findings of scale-up trials (20), the findings of this study provide valuable data, together with detailed process evaluation data (32) to help to explain the findings of the scale-up trial (18, 32).
“Funding and resource availability” was a common reason adaptations and modifications were made. The occurrence of adaptations as part of a research trial, involving the original trial developers, and with a good understanding of the programs and its mechanism of effect may also explain the frequency with which they were fidelity consistent, and thought to have a beneficial impact. The findings underscore the importance of selecting programs that are congruent with the available resources to deliver them at scale, and in doing so, reduce the need for significant adaptations. The use of scalability assessment tools may assist policy makers and practitioners can assist with this process (43).
Both adaptations and modifications to the scale-up of PA4E1 were primarily made to “improve fit with recipients” (i.e., schools). Most adaptations were coded as “fidelity consistent” and “systematic” (i.e., informed by theory and used a process). Such findings are perhaps unsurprising, given they occurred in the context of a funded trial, involving the original trial developers with a good understanding of the program and its mechanism and who employed a considered process to informing adaptations (30). It is also consistent with previous reviews of public health programs (15). Adaptability is a key component of scalable programs, where optimal adaptations are those which are made to fit different contexts and environments while retaining fidelity consistency (12, 13, 21, 43).
Modifications, in contrast to adaptations, were found to be “unsystematic”, likely reflecting the rapid and reactive contexts in which these changes were made by school staff (in-School Champions) and practitioners delivering the program (Support Officers, Program Managers). Previous research into public health prevention programs has similarly found that modifications were unsystematic, suggesting they may be more likely to detract from intervention core functions, resulting in negative impacts on implementation-effectiveness outcomes (44, 45). However, modifications to the program were all coded as “fidelity consistent”, and so retain the intended core functions of the program (outlined in Additional File 3). School staff may have, as a result of the training undertaken as part of the scale-up strategy, or via their existing tacit knowledge, have a good understanding of how the program may impact on the intended outcome and been mindful of this when undertaking modifications. Further research is warranted to explore and better explain such findings.
Reviews of scaled-up physical activity (8) and obesity programs (16) that characterized the nature of adaptations made for scale-up, concluded that adaptations to the “mode of delivery” of programs were particularly prevalent (8, 14, 16). Similarly, we found 13 content adaptations to scale-up PA4E1 and 14 content modifications during delivery of the scaled-up PA4E1.While changes to the delivery modes or content modifications may be perceived as fidelity consistent or improving the overall impacts of a program, for example, by increasing reach and the number of people who may benefit, they may also reduce the absolute effect size of a program (8, 46, 47). That is, the “voltage drop” phenomena whereby the effect sizes of physical activity programs are reduced at scale, may be acceptable from a population perspective if the scaled-up program is capable of reaching and so benefiting (due to delivery mode adaptations) more people, at lower relative cost. Taking a population-level perspective is therefore important when weighing and assessing the potential impact of adaptations or modifications.
Strengths and Limitations
It has been recommended that researchers consider potential causal pathways of modifications, considering both the intended and unintended impacts of modifications on outcomes (20). In line with our process evaluation protocol (32), we have comprehensively described the extent, type and reasons for adaptations and modifications to PA4E1. A strength of this study is the use of real-time tracking of modifications during delivery to record deviations from the planned protocols, which is expected during trial delivery but often not documented well. Indeed, we found the method to be feasible and informative within our study. We found coding using the FRAME to initially be quite difficult, despite the existence of a coding manual (48). We would suggest to future researchers to consider annotating the FRAME framework coding for their own context (13, 48). We also emphasize the importance of going beyond published papers to generate a list of adaptations. By also using the knowledge of those involved in both the efficacy and scale-up trials, we were able to code more accurately the reasons for adaptation. However, a limitation of our research is that we were not blinded to the outcomes of the implementation-effectiveness trial outcomes, which may have influenced our interpretation (18). Additionally, although this study drew upon historical knowledge of those involved in delivering both the efficacy and scale-up trials, it is possible that some specific details were forgotten, given that the efficacy trial was completed in 2014 (7 years ago). Future studies reporting adaptations should therefore aim to do so prospectively (13, 18). Finally, we used the FRAME to report adaptations and modifications to the physical activity practices, implementation support strategies and evaluation methods. Subsequent to our data analysis, the FRAME-IS (Implementation Strategy) was released which is designed for implementation support strategies and organized into modules. The use of both the FRAME (13) and FRAME-IS (22) may have improved our coding of adaptations and reduced the frequency of consensus meetings required. Additional support to use FRAME to code adaptations and modifications to evaluation methods may also be useful.
Adaptations and modifications to public health programs are common. Modifications have the potential to impact the implementation and effectiveness of programs. Our findings underscore the importance of comprehensive reporting of the extent, type and reasons for modifications as part of process evaluations, as this data may be important to the interpretation of trial findings. Making modifications that retain core components but better suit a particular context (program adaptability) is considered to be an important component of successfully scaled-up programs. However, it will be important for future programs to identify project management strategies to mitigate the occurrence of reactive operational modifications that are fidelity inconsistent.
Describing the extent, type and reasons for adaptations and modifications made to public health programs provides valuable process evaluation data to help explain the findings of the program. For example, the data may be used to explain the expected reduction in effect size when they are scaled-up. The comprehensive and transparent description of adaptations and modifications will assist us to generate hypotheses relating to the trial process evaluation and implementation outcome data, which will be explored further.
Data Availability Statement
The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author/s.
The studies involving human participants were reviewed and approved by Hunter New England Area Human Research Ethics Committee. Written informed consent from the participants' legal guardian/next of kin was not required to participate in this study in accordance with the national legislation and the institutional requirements.
MM: conceived the design of the study, led the data analysis, and led the development of the manuscript. MM, JW, EC, RS, and LW: developed the research questions, these were further refined by MM, EC, and LW. RS, EC, LW, and JW: obtained funding for the research. MM: led the ongoing data collection, with significant contributions from EC, RS, TM, and LD. MM, EC, RS, TM, and LD: were involved in the coding of modifications. All authors contributed to the interpretation of findings, provided critical comment on multiple versions of the manuscript, and approved the final manuscript.
This project is funded by the NSW Ministry of Health, Translational Research Grant Scheme. The NSW Ministry of Health has not had any role in the design of the study as outlined in this protocol and will not have a role in data collection, analysis of data, interpretation of data and dissemination of findings. This work was also supported by Cancer Council New South Wales. The project received infrastructure support from the Hunter Medical Research Institute (HMRI). RS is supported by a NHMRC TRIP Fellowship (APP1150661). LW is supported by a NHMRC Career Development Fellowship (APP1128348), Heart Foundation Future Leader Fellowship (101175) and a Hunter New England Clinical Research Fellowship.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
The authors would like to acknowledge the staff from the Central Coast, Mid-North Coast, South-West Sydney and Hunter New England Local Health Districts involved in delivering this trial.
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frhs.2021.719194/full#supplementary-material
AHMRC, Aboriginal Health and Medical Research Council; FRAME, Framework for Reporting Adaptations and Modifications-Enhanced; FRAME-IS, Framework for Reporting Adaptations and Modifications-Enhanced - Implementation HMRI: Hunter Medical Research Institute; HNELHD, Hunter New England Local Health District; MADI, Model for Adaptation Design and Impact; NESA, New South Wales Education Standards Authority; NSW, New South Wales; PA, Physical Activity; PA4E1, Physical Activity 4 Everyone; PE, Physical Education; RT4T, Resistance Training 4 Teens; SAAFE, Supportive, Active, Autonomous, Fair and Enjoyable; SERAP, State Education Research Applications Process.
2. The International Society for Physical Activity and Health (ISPAH). The Bangkok Declaration on Physical Activity for Global Health and Sustainable Development. Br J Sports Med. (2017) 51:1389–91. doi: 10.1136/bjsports-2017-098063
3. Guthold R, Stevens GA, Riley LM, Bull FC. Global trends in insufficient physical activity among adolescents: a pooled analysis of 298 population-based surveys with 1.6 million participants. Lancet Child Adolesc Health. (2019) 4:23–35. doi: 10.1016/S2352-4642(19)30323-2
4. Guthold R, Stevens GA, Riley LM, Bull FC. Worldwide trends in insufficient physical activity from 2001 to 2016: a pooled analysis of 358 population-based surveys with 1.9 million participants. Lancet Glob Health. (2018) 6:e1077–e86. doi: 10.1016/S2214-109X(18)30357-7
5. Finch M, Jones J, Yoong S, Wiggers J, Wolfenden L. Effectiveness of centre-based childcare interventions in increasing child physical activity: a systematic review and meta-analysis for policymakers and practitioners. Obes Rev. (2016) 17:412–28. doi: 10.1111/obr.12392
6. Yoong SL, Wolfenden L, Clinton-McHarg T, Waters E, Pettman TL, Steele E, et al. Exploring the pragmatic and explanatory study design on outcomes of systematic reviews of public health interventions: a case study on obesity prevention trials. J Public Health. (2014) 36:170–6. doi: 10.1093/pubmed/fdu006
7. Love R, Adams J, van Sluijs EMF. Are school-based physical activity interventions effective and equitable? A meta-analysis of cluster randomized controlled trials with accelerometer-assessed activity. Obes Rev. (2019). 20:859–70. doi: 10.1111/obr.12823
8. Lane C, McCrabb S, Nathan N, Naylor P-J, Bauman A, Milat A, et al. How effective are physical activity interventions when they are scaled-up: a systematic review. Int J Behav Nutr Phys Act. (2021) 18:16. doi: 10.1186/s12966-021-01080-4
9. Milat AJ, King L, Bauman AE, Redman S. The concept of scalability: increasing the scale and potential adoption of health promotion interventions into policy and practice. Health Promot Int. (2013) 28:285–98. doi: 10.1093/heapro/dar097
11. Averitt AJ, Weng C, Ryan P, Perotte A. Translating evidence into practice: eligibility criteria fail to eliminate clinically significant differences between real-world and study populations. NPJ Digit Med. (2020) 3:67. doi: 10.1038/s41746-020-0277-8
13. Wiltsey Stirman S, Baumann AA, Miller CJ. The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implement Sci. (2019) 14:58. doi: 10.1186/s13012-019-0898-y
15. Escoffery C, Lebow-Skelley E, Haardoerfer R, Boing E, Udelson H, Wood R, et al. A systematic review of adaptations of evidence-based public health interventions globally. Implement Sci. (2018) 13:125. doi: 10.1186/s13012-018-0815-9
16. McCrabb S, Lane C, Hall A, Milat A, Bauman A, Sutherland R, et al. Scaling-up evidence-based obesity interventions: a systematic review assessing intervention adaptations and effectiveness and quantifying the scale-up penalty. Obes Rev. (2019) 20:964–82. doi: 10.1111/obr.12845
17. Movsisyan A, Arnold L, Evans R, Hallingberg B, Moore G, O'Cathain A, et al. Adapting evidence-informed complex population health interventions for new contexts: a systematic review of guidance. Implement Sci. (2019) 14:105. doi: 10.1186/s13012-019-0956-5
19. Stirman SW, Miller CJ, Toder K, Calloway A. Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implement Sci. (2013) 8:65. doi: 10.1186/1748-5908-8-65
20. Kirk MA, Moore JE, Wiltsey Stirman S, Birken SA. Towards a comprehensive model for understanding adaptations' impact: the model for adaptation design and impact (MADI). Implement Sci. (2020) 15:56. doi: 10.1186/s13012-020-01021-y
21. Yoong SL, Bolsewicz K, Grady A, Wyse R, Sutherland R, Hodder RK, et al. Adaptation of public health initiatives: expert views on current guidance and opportunities to advance their application and benefit. Health Educ Res. (2020) 35:243–57. doi: 10.1093/her/cyaa014
22. Miller CJ, Barnett ML, Baumann AA, Gutner CA, Wiltsey-Stirman S. The FRAME-IS: a framework for documenting modifications to implementation strategies in healthcare. Implement Sci. (2021) 16:36. doi: 10.1186/s13012-021-01105-3
23. Rabin BA, McCreight M, Battaglia C, Ayele R, Burke RE, Hess PL, et al. Systematic, multimethod assessment of adaptations across four diverse health systems interventions. Front Public Health. (2018) 6:102. doi: 10.3389/fpubh.2018.00102
24. Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. (2012) 50:217–26. doi: 10.1097/MLR.0b013e3182408812
25. Hollis JL, Sutherland R, Campbell L, Morgan PJ, Lubans DR, Nathan N, et al. Effects of a “school-based” physical activity intervention on adiposity in adolescents from economically disadvantaged communities: secondary outcomes of the “Physical Activity 4 Everyone” RCT. Int J Obes. (2016) 40:1486–93. doi: 10.1038/ijo.2016.107
26. Sutherland R, Campbell E, Lubans DR, Morgan PJ, Okely AD, Nathan N, et al. A cluster randomised trial of a school-based intervention to prevent decline in adolescent physical activity levels: study protocol for the “Physical Activity 4 Everyone” trial. BMC Public Health. (2013) 13:57. doi: 10.1186/1471-2458-13-57
27. Sutherland R, Campbell L, Lubans D, Morgan P, Okely AD, Nathan N, et al. “Physical Activity 4 Everyone” school-based intervention to prevent decline in adolescent physical activity levels: 12 month (mid-intervention) report on a cluster randomised trial. Br J Sports Med. (2015). 50:488–95. doi: 10.1136/bjsports-2014-094523
28. Sutherland R, Reeves P, Campbell E, Lubans DR, Morgan PJ, Nathan N, et al. Cost effectiveness of a multi-component school-based physical activity intervention targeting adolescents: the “Physical Activity 4 Everyone” cluster randomized trial. Int J Behav Nutr Phys Act. (2016) 13:94. doi: 10.1186/s12966-016-0418-2
29. Sutherland RL, Campbell EM, Lubans DR, Morgan PJ, Nathan NK, Wolfenden L, et al. The Physical Activity 4 Everyone cluster randomized trial: 2-year outcomes of a school physical activity intervention among adolescents. Am J Prev Med. (2016) 51:195–205. doi: 10.1016/j.amepre.2016.02.020
30. Sutherland R, Campbell E, Nathan N, Wolfenden L, Lubans DR, Morgan PJ, et al. A cluster randomised trial of an intervention to increase the implementation of physical activity practices in secondary schools: study protocol for scaling up the Physical Activity 4 Everyone (PA4E1) program. BMC Public Health. (2019) 19:883. doi: 10.1186/s12889-019-6965-0
31. Sutherland R, Campbell E, McLaughlin M, Nathan N, Wolfenden L, Lubans DR, et al. Scale-up of the Physical Activity 4 Everyone (PA4E1) intervention in secondary schools: 12-month implementation outcomes from a cluster randomized controlled trial. Int J Behav Nutr Phys Act. (2020) 17:100. doi: 10.1186/s12966-020-01000-y
32. McLaughlin M, Duff J, Sutherland R, Campbell E, Wolfenden L, Wiggers J. Protocol for a mixed methods process evaluation of a hybrid implementation-effectiveness trial of a scaled-up whole-school physical activity program for adolescents: Physical Activity 4 Everyone (PA4E1). Trials. (2020) 21:268. doi: 10.1186/s13063-020-4187-5
33. Pinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G, Griffiths CJ, et al. Standards for reporting implementation studies (StaRI): explanation and elaboration document. BMJ Open. (2017) 7:e013318. doi: 10.1136/bmjopen-2016-013318
34. Wolfenden L, Yoong SL, Williams CM, Grimshaw J, Durrheim DN, Gillham K, et al. Embedding researchers in health service organizations improves research translation and health service performance: the Australian hunter New England population health example. J Clin Epidemiol. (2017) 85:3–11. doi: 10.1016/j.jclinepi.2017.03.007
36. Escoffery C, Lebow-Skelley E, Udelson H, Boing EA, Wood R, Fernandez ME, et al. A scoping study of frameworks for adapting public health evidence-based interventions. Transl Behav Med. (2018) 19:1–10. doi: 10.1093/tbm/ibx067
40. Lubans DR, Lonsdale C, Cohen K, Eather N, Beauchamp MR, Morgan PJ, et al. Framework for the design and delivery of organized physical activity sessions for children and adolescents: rationale and description of the “SAAFE” teaching principles. Int J Behav Nutr Phys Act. (2017) 14:24. doi: 10.1186/s12966-017-0479-x
41. Lubans DR, Morgan PJ, Callister R, Collins CE, Plotnikoff RC. Exploring the mechanisms of physical activity and dietary behavior change in the program x intervention for adolescents. J Adolesc Health. (2010) 47:83–91. doi: 10.1016/j.jadohealth.2009.12.015
42. Kennedy SG, Smith JJ, Morgan PJ, Peralta LR, Hilland TA, Eather N, et al. Implementing resistance training in secondary schools: a cluster randomized controlled trial. Med Sci Sports Exerc. (2018) 50:62–72. doi: 10.1249/MSS.0000000000001410
43. Lee K, Milat A, Grunseit A, Conte K, Wolfenden L, Bauman A. The intervention scalability assessment tool: a pilot study assessing five interventions for scalability. Public Health Res Pract. (2020) 30:3022011. doi: 10.17061/phrp3022011
44. Carvalho ML, Honeycutt S, Escoffery C, Glanz K, Sabbs D, Kegler MC. Balancing fidelity and adaptation: implementing evidence-based chronic disease prevention programs. J Public HealthManag Pract. (2013) 19:348–56. doi: 10.1097/PHH.0b013e31826d80eb
45. Mackie TI, Ramella L, Schaefer AJ, Sridhar M, Carter AS, Eisenhower A, et al. Multi-method process maps: an interdisciplinary approach to investigate ad hoc modifications in protocol-driven interventions. J Clin Transl Sci. (2020) 4:260–9. doi: 10.1017/cts.2020.14
46. Stewart AL, Gillis D, Grossman M, Castrillo M, Pruitt L, McLellan B, et al. Diffusing a research-based physical activity promotion program for seniors into diverse communities: CHAMPS III. Prev Chronic Dis. (2006) 3:A51.
47. Hardy LL, Mihrshahi S, Gale J, Nguyen B, Baur LA, O'Hara BJ. Translational research: are community-based child obesity treatment programs scalable? BMC Public Health. (2015) 15:652. doi: 10.1186/s12889-015-2031-8
48. Stirman. FRAME Coding Manual. (2020). Available online at: http://med.stanford.edu/fastlab/research/adaptation.html. (accessed April 22, 2021).
Keywords: process evaluation, scale-up, fidelity, implementation science, physical activity, modification, adaptation, schools
Citation: Mclaughlin M, Campbell E, Sutherland R, McKenzie T, Davies L, Wiggers J and Wolfenden L (2021) Extent, Type and Reasons for Adaptation and Modification When Scaling-Up an Effective Physical Activity Program: Physical Activity 4 Everyone (PA4E1). Front. Health Serv. 1:719194. doi: 10.3389/frhs.2021.719194
Received: 08 June 2021; Accepted: 30 July 2021;
Published: 17 November 2021.
Edited by:Ann Catrine Eldh, Linköping University, Sweden
Reviewed by:Siw Carlfjord, Linköping University, Sweden
Johanna Fritz, Mälardalen University College, Sweden
Copyright © 2021 Mclaughlin, Campbell, Sutherland, McKenzie, Davies, Wiggers and Wolfenden. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Matthew Mclaughlin, firstname.lastname@example.org
†ORCID: Matthew Mclaughlin orcid.org/0000-0003-2870-8556
Elizabeth Campbell orcid.org/0000-0002-7265-5407
Rachel Sutherland orcid.org/0000-0003-0470-7663
Tom McKenzie orcid.org/0000-0001-5132-6279
Lynda Davies orcid.org/0000-0002-2747-5589
John Wiggers orcid.org/0000-0001-6361-9685
Luke Wolfenden orcid.org/0000-0002-6178-3868