Supporting Teachers in Improving Formative Decision-Making: Design Principles for Formative Assessment Plans

Formative assessment is considered as one of the most effective interventions to support teacher decision-making and improve education and student learning. However, formative assessment does not always meet these expectations. In order to be effective, formative assessment activities should be consciously and coherently planned aligned with other aspects of the curriculum and the decisions teachers wish to make based on these activities. While there is sufficient support for teachers to design formative assessment activities, no guidelines exist to help them tie these different activities together in an effective way. To support teachers in designing formative assessment plans informing formative decision-making, this study focused on the creation of a set of design principles. These design principles for formative assessment plans were formulated based on expert interviews and subsequently evaluated by future users. The result is a set of eight design principles that can be used and validated in educational practice.


INTRODUCTION
Assessment is used formatively when teachers and/or students interpret and use the evidence about student achievement to make formative decisions, decisions about the next steps in teaching and learning (Black and Wiliam, 2009). For example, decisions about adjusting lessons, how to differentiate, if students are ready for a new subject or what is the best way to support student learning at a given time. Accordingly, formative assessment embodies all activities that students and teachers undertake to elicit evidence to establish where students are in their learning in order to inform education. Teachers interpret and use this information to "make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions they would have taken in the absence of this information" (Black and Wiliam, 2009, p. 9).
In fact, formative assessment is a continuous dialogue between students and teachers about three questions (Wiliam and Thompson, 2007;Black and Wiliam, 2009;Wiliam, 2011): 1) Where is the learner going? 2) Where is the learner now? 3) What is necessary to bridge the gap between where the learners are in their learning and where they are going?
Many studies have presented strategies that help to answer these three questions. Strategies associated with formative assessment include: identifying and making explicit learning objectives and success criteria; elicitation of evidence of students' understanding or learning; interpretation of the elicited information against the learning objectives and/or success criteria; providing students with feedback, and follow-up actions taken by the student and/or teacher to improve teaching and learning (Ruiz-Primo and Furtak, 2007;Antoniou and James, 2014;Veugen et al., 2021). Continuously answering these questions, using these strategies, helps teachers to better meet students' needs and to increase students' involvement in their own learning process (Black and Wiliam, 2010). As a result, formative assessment is seen as one of the most effective interventions to improve education and increase student learning (Briggs et al., 2012;Christoforidou et al., 2014;Offerdahl et al., 2018).
Formative assessment has an intuitive appeal and the potential effectiveness is widely acknowledged (Furtak and Ruiz-Primo, 2008;Black and Wiliam, 2009;Offerdahl et al., 2018), nevertheless empirical evidence about the effect of formative assessment on student learning is variable. Empirical studies that investigated the effect sizes of formative assessment on student learning vary in methods and outcomes (Black and Wiliam, 2010;Kingston and Nash, 2011;Briggs et al., 2012;Offerdahl et al., 2018;Gu, 2021). Offerdahl et al. (2018) suggest that differences in enactment by teachers explain a main part of the differences in effect sizes of formative assessment on student learning. Perspectives on formative assessment, context or formative assessment proficiency and literacy can all influence this enactment (Deneen et al., 2019;Earle, 2021;Gu, 2021;Yan et al., 2021). Apart from these factors improving enactment of formative assessment starts with answering the question how teachers can best design and implement formative assessment to have it really contribute to better or better founded formative decision-making. This article will try to answer this question by focusing on teacher activities in designing and implementing formative assessment.
Formative assessment is best considered as an ongoing process to inform and support teaching and learning (Earle, 2021;Gu, 2021;Veugen et al., 2021). While achieving learning objectives usually exceeds a lesson, teachers working with formative assessment also need to exceed lessons planning these activities. During a series of lessons they have to keep checking whether objectives are reached or not and for what reasons, followed by deciding what this means for their teaching. When teachers want to design formative assessment, they, therefore, should plan series of connected formative activities instead of individual activities to support their lessons (Furtak et al., 2016). These formative activities should be constructively aligned with the objectives and planned lessons. Especially this connection between formative assessment activities and the link with the rest of the curriculum seems important but also hard to accomplish in classroom practice. Many studies that investigated formative assessment conclude that extra attention for the integration, coherency and alignment of formative assessment activities in classroom practice is needed to be more effective (Gulikers et al., 2013;Wylie and Lyon, 2015;Van Den Berg, 2018).
For planning these connected series of formative assessment activities, Wiliam (2013Wiliam ( , 2014 advocates decision-driven data collection. In decision-driven data collection future formative decisions are the starting point for planning formative assessment activities (Wiliam, 2013(Wiliam, , 2014Moss, 2020). It differs from one of the most well-known forms of formative assessment, namely data-based decision making (Schildkamp and Kuiper, 2010;Van der Kleij et al., 2015;Heitink et al., 2016). Since data-based decision making starts from existing data, Wiliam (2013) argues that this might, in certain situations, be unsuitable or just too late for formative decision-making about for instance lesson preparations or last-minute adjustments in instruction. Accordingly, he suggests making plans of actions, a blueprint of formative learning activities, that incorporate the strategies for collecting evidence of learning as well as what will be done with this information when it is collected based on future formative decisions (Wiliam, 2013). Formative assessment plans for decisiondriven data collection, as suggested by Wiliam (2013), can also accommodate the integration, coherency and alignment in advance to support teachers in implementing formative assessment that informs their formative decision-making. So far, formative assessment, however, has predominantly been planned, executed and investigated in singular formative assessment activities. As a result, we see a lot of examples and tools for teachers to design formative assessment activities but few examples or guidelines to help them tie these different activities together in an effective way in formative assessment plans.
Hence, the research question in this study is: What are design principles for formative assessment plans that help teachers to make better founded formative decisions in classroom practice about the next steps in teaching and learning?

MATERIALS AND METHODS
This study has an educational design research approach and intends to develop a first prototype set of design principles for future use and empirical testing in schools for secondary education. For this design study, the three steps for educational design research, defined by McKenney and Reeves (2019), were followed (see Figure 1).
In the first step, analysis and exploration, interviews with three different groups of experts were used to gather the first ideas about design principles for formative assessment plans. Subsequently, in the design and construction phase, these expert interviews were followed by a thematic analysis of the interview data to design and construct a set of design principles. Finally, as part of the concluding step of evaluation and reflection, future users evaluated the constructed design principles. Teachers from four schools for secondary education evaluated the design principles during group interviews, which resulted in the final adjustments of the design principles. To uphold the participative character of educational design research, future users were involved as members of the expert groups and as informants in the evaluation phase. As experiential experts in the expert group interviews, they helped constructing the first ideas about design principles for formative assessment plans and made a first step in analyzing the outcomes. In the evaluation phase, future users were asked to evaluate the design principles.

Analysis and Exploration: Expert Interviews
Three interviews were planned with heterogeneous groups of experts on the subject formative assessment. In total, twenty participants were experts from both research and practice, all involved with formative assessment. The educational researchers were selected as experts when they conducted research on formative assessment but also had their own formative assessment teaching practice. Teachers were selected as experts when they worked in one of four participating schools and had demonstrable experience using formative assessment in their classrooms. One school-policy maker was selected because she was responsible for implementing formative assessment in one of these four schools. Additionally, two teacher educators were selected who were involved in the development of a minor for formative assessment. Table 1 shows the combination of experts in each expert group.
The purpose of these expert interviews was to reach agreement among the participants of each group about what they thought were critical aspects a formative assessment plan should have to be effective. These critical aspects will be clustered, first within groups and then across groups, and used in a later stage as a starting point to formulate design principles. To promote consensus, the interviews were organized as group decision rooms where the discussion was supported by a digital group support system (Fjermestad and Hiltz, 2000;Pyrko et al., 2019). This support system offers participants the possibility to answer questions individually and digitally through a device followed by a group discussion about how to cluster all given answers. By clustering their own answers during the interview within the

Groups Participants
Group 1 Three educational researchers, one teacher-educator, and one school policy maker Group 2 Two educational researchers, three teachers from one school for secondary education Group 3 Nine teachers from two school for secondary education, one teacher-educator group, participants were directly involved in the first phase of data-analysis. All subsequent steps that were taken in the expert interviews are presented in Table 2.
Through these steps, each expert interview has generated a list of clusters of critical aspects for formative assessment plans. Going forward in this article, these clusters of aspects will be referred to as features of formative assessment plans.

Design and Construction: Thematic Analysis
The features of formative assessment plans that were suggested through the expert interviews were collected and used in a design session with three of the four authors of this paper and two of their colleague researchers. The purpose of this design session was to synthesize the outcomes from the three expert group interviews and develop a first draft of design principles for formative assessment plans. Thematic analysis is systematic but always subjective (Bearman and Dawson, 2013). Therefore, the researchers who joined the thematic analysis during this design session were invited since they all had experience with investigating formative assessment, knew the project well but operated at different levels of distance in the project. Intersubjectivity was sought by combining the common knowledge of the objectives in the current study with the quality of these researchers with different backgrounds, experiences, and perspectives on formative assessment. To systematically generate design principles from the collected interview data the first five phases of thematic analysis as described by Nowell et al. (2017) were followed during this session. At the start, to get familiarized with all data, each feature, that was a result of the expert interviews, was put on an individual card and these cards were laid down on a large table. The researchers studied these cards, clustered the features that were similar into themes, and named each of these themes. Thereafter, in the next phase of analysis, these themes were critically reviewed by the researchers by questioning if each theme of features was explicitly applicable for designing formative assessment plans. This final critical review resulted in 10 final themes of features of formative assessment plans. These remaining 10 themes were then formulated as design principles, in phase five, using the following structure based on Van den Akker's (2013) suggestion on how to formulate design principles: If you want to design formative assessment plans • Then you are best advised to give these plans the following characteristics These design principles were provided with a description of what this would mean in practice and used as input for the group interviews with future users. In these group interviews, future users will evaluate this draft version of the design principles for formative assessment plans based on transparency, completeness, usability, and suitability for teaching practice.

Evaluation and Reflection: Group Interviews
Four group interviews were set up to evaluate the draft version of the design principles for formative assessment plans. The group interviews were organized with future users originating from four schools for secondary education. Each group consisted of five to eight teachers from the same school. In two cases, a school leader also joined the interview (see Table 3).
The teachers and school leaders were questioned about recommendations regarding transparency, usability, completeness, and suitability of the design principles for school practice. The participants had received the design principles in advance. First, the participants were asked to write down all recommendations they could think of to improve the design principles. Secondly, they were asked to decide what facet of the design principles would improve if this recommendation was followed. Facets they could choose from were transparency, usability, completeness, or suitability. Subsequently, they were asked to give explanations of their recommendations and the improvements they would expect. The interview transcripts were analyzed through thematic analysis (Nowell et al., 2017). Recommendations for improvements from the interviews were coded and clustered into themes by the first author. Before defining and naming them, three of the four authors reviewed the initial themes and subthemes. Each theme of recommendations was also linked to the corresponding facets of the design principles that would improve most when the recommendations of that theme were adopted in the design principles. Thereafter findings were used to reflect on and improve the design principles for formative assessment plans.

RESULTS
In this section, the outcomes of this study are presented through the subsequent steps that were taken to answer the research question.
1. Analysis and exploration: expert interviews 2. Design and construction: thematic analysis 3. Evaluation and reflection: group interviews with future users Since the results will be presented in subsequent steps, they will reveal the creation as well as the evolution of the design principles for formative assessment plans so far, as advised by Bakker (2019).
Step 1: Analysis and Exploration: Expert Interviews Table 4 shows the findings from the three expert interviews. The first expert group resulted in nine features that a formative assessment plan should have and the second and third expert group resulted in 11 features. The findings show some overlap between features that were mentioned in more than one group. The features goal orientation, alignment, giving insight in learning progress, leaving room for improvement, consciously and logical structured and involving competent teachers were mentioned in two or three expert groups. Because there were differences in descriptions and/or individual answers/aspects that were linked to these overlapping features, the features mentioned in each group will be regarded and used as unique features at this point of the study. Consequently, the expert interviews resulted in a set of 31 unique features that a formative assessment plan should have according to the participants. The features are presented in Table 4 in order of importance as ranked by the participants of the concerning groups.
Step 2: Design and Construction: Thematic Analysis The 31 unique features that experts believe a formative assessment plan should have were used in a thematic analysis to derive themes for design principles.
The first column of Table 5 shows the first themes that were made based on the features from the expert interviews. The second column represents the outcome of the critical review. And the third column shows the design principles that were formulated based on Van den Akker's structure for design principles (2013).
The second column of Table 5 shows that two of the initial themes were not adopted. The active role of students, theme five, was not adopted because on closer inspection this was not considered as a feature specifically applicable for designing formative assessment plans. Theme eight, about prerequisites, was not adopted because it also did not represent a design principle for formative assessment plans rather conditions that should be in place before working with formative assessment plans.
The critical review also resulted in two themes being split up. On closer inspection theme alignment actually consisted of two themes: "Alignment of formative assessment activities inside and outside of the plan and with the rest of the curriculum" (3) and "Integration of formative assessment plans into curriculum/lesson plans" (4). And providing insight in learning processes was split up in two more specific themes: "Provide insight for teachers" (9) and "Provide insight for students" (10).
Overall, the thematic analysis resulted in 10 design principles for formative assessment plans.

Step 3: Evaluation and Reflection: Group Interviews
The draft version of the design principles was evaluated on their transparency, usability, completeness, or suitability by 23 teachers, four school leaders and one school policy maker during four group interviews (see Table 3). Through thematic analysis, three different types of recommendations to improve the design principles for formative assessment plans were found. These three themes were found in all interviews regardless of composition or context of the interviews.
The main points of improvement future users suggested for the 10 design principles were: -Improve ambiguous writing -Improve style and structure -Improve content Table 6 shows examples for each of these three themes. According to the participants, improving ambiguous writing through more accessible concrete language enriched with practical examples and images would help make the design principles more suitable and usable for teachers in secondary education.
To improve style and structure participants suggested to put design principles in a chronological order and to present the design principles as an easy to use tool: roadmap, format, checklist, digital tool, or menu. Participants mainly linked these recommendations to enhancing transparency and usability of the design principles. *Italic text means that this feature was also mentioned in one of the other expert groups.

Themes of features Critical review Design principles (Draft to evaluate in group interviews)
If you want to design formative assessment plans • then you are best advised to give these plans the following characteristics. . ..

Consists of consciously chosen formative learning activities (4)* Adopted
The plan has a logical and clear structure of formative assessment activities that build on from each other and are evenly spread. This means neither too much nor too little and sufficient variation in the formative assessment activities.

Is transparent (1) Adopted
The plan must be transparent to all stakeholders. This means that those involved are aware and understand how formative assessment will be executed and why.
3. Is aligned (1) Adopted and split up in two design principles The plan consists of formative assessment activities that are aligned with each other, other formative assessment plans and the rest of the curriculum.
The plan is integrated as much as possible into the curriculum/series of lessons.

Decision-driven (1) Adopted
The plan consists of formative assessment activities that are consciously planned and chosen in the light of future decisions. It is clear in advance how the information provided by the formative learning activities will be used in making choices and decisions in (supporting) Students' learning.
5. Active role students (1) Not adopted The plan is flexible. Meaning that the plan creates room for moments of contingency. These moments of contingency can later be used to follow up on the information provided by formative assessment activities.

Leaves room for improvement (4) Adopted
The plan leaves room for improvement and development of students. This means that after each formative activity there must be opportunity for all students to improve. Follow-up activities should be deliberately planned in order for students to use feedback. The formative assessment activity is therefore not only checking and concluding, but must be able to contribute to the next step in learning.

Is goal-oriented (3) Adopted
The plan is a set of consciously chosen formative assessment activities that are tied together by the same (or an overlap of) learning objectives.

Provides insight in learning (5) Adopted and split up in two design principles
The plan provides insight into the learning processes of students at various times (how is a student doing, what development becomes visual, and what are the learning needs).
The plan provides students with Insight into their own learning process at various times through feedforward of teachers or directly through formative assessment activities (how am I doing, what have I done so far, how can I continue?).
*The numbers in parentheses refer to the number of original features that were clustered under this category.

Recommendation Examples of evaluation design principles Examples of recommendations
Improve ambiguous writing School B: "The document is still a bit vague, I miss a concrete explanation of activities" School C: "It wasn't always clear. We both read it in different ways" School C: "Based on what you are trying to say here, also include an example of a teacher who is literally designing a lesson. So you can picture it a little better." School B: "You shouldn't use words as 'clear' and 'maybe.' I really think formulation can be more concrete" Improve style and structure School B: "If it is now (structured) in the right order, I think 10 is really much too late. Because 10 may move all the way forward, if you ask me, because that is about communication with others" School D: "but in terms of how (the set of design principles) is designed. If I have to work with this and then make a program (. . .) For me it's much better to work point by point School A: just a very clear order of you going to do this first and then this and then this, which helps a lot. Chronology is important School B: "A lot of words, a more schematic display is preferable." Improve content School D "ten principles of which we say 'hey this one is almost the same as the others' Meaning that they overlap" School A: "Especially merge (design principles). Off course automatically a few principles will disappear when you put things together. (. . .) Two and three can be put together" Finally, participants thought that merging of overlapping texts and design principles would improve content and advance transparency, usability, and suitability of the design principles.

Final Adaptions Design Principles
To improve the design principles the following five actions were undertaken as a response to the outcomes of the group interviews.
(1) Checking the design principles to see whether they could be formulated more concretely and to the point, (2) Checking the design principles for clear and consistent use of concepts, (3) Reviewing the design principles for overlap and repetition and merging if possible, (4) Reviewing the design principles to shorten sentences and texts where possible, and finally (5) Putting the design principles in a chronological order for designing formative assessment plans. The result of these actions is presented in Table 7 and shows that part of design principle Prototype design principles 1. Use a set of learning objectives and lesson plans as a starting point 2. Choose formative assessment activities that match the learning objectives that you are aiming for and the decisions you want to make 3. Plan formative assessment activities equally divided over time and in a way that they can build on from each other 4. Choose formative assessment activities that provide you with rich information about student learning and the necessary next steps in education and learning 5. Plan time, space and opportunity for students to improve their learning based on the outcome of formative assessment activities 6. Leave room for moments of contingency in formative assessment-and lesson plans 7. Align a formative assessment plan with other formative assessment activities that are taking place before, parallel or after this plan.
8. The plan must be transparent and feasible to all stakeholders seven, from the draft version, is merged with draft version's principles 9 and 10 into design principle four. Draft design principles five and eight are combined in new principle two.
The suggestions made about a more schematic design and including examples and visuals in order to increase the usability will be considered at a later point in time when the design principles will be incorporated in a practical tool.

DISCUSSION
Formative assessment does not live up to the expectations when it is not carefully and coherently planned and constructively aligned with the rest of the curriculum (Wiliam and Thompson, 2007;Wiliam, 2013). In this study, a set of design principles for formative assessment plans was developed to support teachers in planning formative assessment activities coherently for the sake of well-informed formative decision-making.
Well-articulated design principles provide insight into the purpose and advised characteristics of an intervention accompanied by guidelines how to design this intervention, procedures, and conditions for implementation, all supported by empirical and theoretical arguments (Plomp and Nieveen, 2013;Bakker, 2019). It is important for future users that design principles provide this rich information to understand the value of the design principles together with when, why and how they work.
The design principles that are a result of the current study provide information about characteristics that formative assessment plans should have as well as procedures how to design these formative assessment plans. Often these characteristics and procedures can be recognized in literature regarding formative assessment activities that apparently often applies for formative assessment plans as well. In the next paragraph, the eight design principles will be used to give a preview on how these principles could be used in a design process.
The first four principles for formative assessment plans echo the importance of formative assessment activities to be aligned, coherent, and part of decision-driven data collection in order to be effective (Biggs, 1996;Wiliam, 2013;Furtak et al., 2016). As a result, principle 1 advises teachers to use the learning objectives and existing lesson plans as starting point for their design of a formative assessment plan. Starting from learning objectives ensures that student learning is perceived in the light of learning processes toward general learning objectives instead of focusing on good or wrong answers (Coffey et al., 2011). Starting from existent lesson plans makes it easier for teachers to embed formative assessment activities in existing teaching processes and use existing learning activities as proof of learning to inform teaching (Earle, 2021). Principle 2 recommends decision-driven data collection (Wiliam, 2013(Wiliam, , 2014Moss, 2020). Teachers determine in advance at which moments there is a need to make a decision about the next steps in teaching or learning with regards to the learning objectives. For example, decisions about adjusting lessons, how to differentiate, if students are ready for a new subject or what is the best way to support student learning at a given time. For these specific moments, teachers deliberately plan formative assessment activities that provide rich information about student learning on the defined learning objectives and helps inform the specified decisions (Principles 3 and 4). Deliberately planning these moments and formative activities linked to decisions and objectives ensures coherency and the possibility of formative activities to build on from each other.
Principle 5, 6, and 7 focus on how to make sure that the formative assessment plans leave room for improvements in teaching and learning. Formative assessment can only be effective if it results in a well-informed follow up and feedback can only become valuable for learners when they get opportunities to use it (Winstone and Boud, 2020). Concretely this means that after each activity that provides information about student learning teachers should plan time and possibilities for themselves and students to act upon this information (principles 5 and 6). Teachers must be able to adjust their lesson plans and students must be given the possibility to use feedback. Students should be provided with opportunities to improve their learning within the formative assessment plan. A recent study by Veugen et al. (2021) shows that teachers who use formative assessment mainly experience difficulties in making adjustments based on the outcomes of student learning. They do not always feel capable to make these adjustments. Therefore, a formative assessment plan should leave room for adjustments in teachers' instruction as well as the adjustments students want to make in their learning based on feedback they have received.
Principle 1 through 6 can be worked out in a timeline or added to a plan for a series of lessons. The final design principles, principles 7 and 8, focus on a final check of the formative assessment plan when everything is planned. Principle 7 challenges teachers to perceive their designed formative assessment plan in larger context while principle 8 focuses on the check for transparency and feasibility of formative assessment plans to be useful and beneficial for all stakeholders.
Looking back at what defines the quality of design principles we can see that these eight design principles give information about procedures and characteristics that can help to design formative assessment plans. Nevertheless, thorough empirical support for these design principles lacks as the current study only consisted of theoretical evaluation with future users. Teachers have not had the chance to use these design principles in practice yet. Further investigation of the value and prescriptive validity of these principles in classroom practice is needed. A second limitation in this study is that, although this is an educational design study, future users were not part of all steps in the research process. Future users did prepare the thematic analysis in step two by clustering their answers during the expert interviews, however, the actual analysis in step two was conducted solely by researchers.
Bakker (2019) advises researchers, whenever they present design principles as outcomes of design research, to be explicit about the nature of the design principles. Are they values, criteria, predictions or advice (Bakker, 2019). At this moment these design principles contribute to existing literature on formative assessment by advising how to design formative assessment plans coherently, decision-driven and with successive and ongoing formative cycles. The design principles that are the outcome of the current design-study can be seen as an advice for teachers who want to design formative assessment plans. This might change into more prescriptive design principles in the future based on repeated cycles of design research. Nevertheless, the purpose will never be to formulate these design principles as strict guidelines (Havnes et al., 2012). The main goal is that these design principles can support teachers now and in the future to design decisiondriven formative assessment that informs their teaching and improves learning.

DATA AVAILABILITY STATEMENT
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

ETHICS STATEMENT
The studies involving human participants were reviewed and approved by the Ethische Commissie Onderzoek HAN university of applied sciences. The participants provided their written informed consent to participate in this study.

AUTHOR CONTRIBUTIONS
JS was responsible for the research work and preparing the manuscript. TS-M, CV, and DJ-T were involved in publication planning, providing advice on directions for the research and manuscript, reading of drafts and giving suggestions for changes and feedback on the final manuscript.

FUNDING
This work was supported by the Taskforce for Applied Research SIA, or Regieorgaan SIA (Dossier no. RAAK.PRO03.057).