The convergence of the aforementioned research, and the evidence it produced, was used to shape the teacher evaluation policy that was central to RTTT. To understand why and how these particular lines of research were used, we borrow from two theoretical perspectives—one that considers policy formation in general terms (McDonnell and Weatherford, 2020) and one that considers the sociopolitical context of teaching from a critical race perspective (Nasir et al., 2016). Together, these perspectives help us better understand why certain research evidence was so salient in policy formation, why other research was not attended to, and, ultimately, why the research that guided policy was insufficient to adequately satisfy the ambitious policy goals of RTTT.

McDonnell and Weatherford (2020) described the strategic use of evidence by policymakers to achieve political objectives given a set of goals and beliefs about how best to achieve those goals. In that context, they argued that it was important to understand what evidence is given attention as well as who is engaged in the production and use of evidence. The who includes:

Nasir et al. (2016) argued that, in order to have a comprehensive understanding of teaching, one must take into account the multi-level context in which teaching is situated. Yet, research on teaching and the resulting policies have often ignored such complexity. They further contended that the research and policies over the recent past have been guided by particular kinds of framing of the problems to be addressed.

In Nasir et al.’s (2016) framework, there are three levels of context that need to be addressed in any full analysis of teaching. First, there are broad economic and policy macro-trends that include: significant and growing economic inequality; the paradox of increasing racial and ethnic diversity in American schools combined with increasing social class segregation in society and schools; and marketized neoliberalism (bringing free-market principles to social issues). The second level includes ways that schools and districts adapt to these broader economic and policy macro-trends. The third level focuses on how these other levels influence the nature of instruction and learning environments that students, and particularly marginalized students, encounter. The focus on accountability testing, for example, often results in low-skill test preparation teaching for marginalized students.

Nasir et al. (2016) also adopted Hand et al.’s (2012) conception of operating frames “as a way to examine and reorganize race and power within learning environments. Power plays out in everyday social interaction as individuals become attuned to, coordinate and mobilize around frames they engage in during moments of interaction” (Hand et al., 2012, p. 251). The first frame they identify is one of colorblindness, a view that “minimizes the existence or consequentiality of race and views policy solutions as best when universal in nature” (Nasir et al., 2016, p. 354). A second frame is meritocracy, one that ascribes accomplishment as solely due to the actions of individuals and “allows policy makers to act without acknowledging the systemic nature of racial disparities and diverts attention to the choices of individual actors” (Nasir et al., 2016, p. 353). The final frame, also located in their multi-level hierarchy is neoliberalism, which has led to the marketization of schooling and “emanates from three decades of policy that positioned the private sector to be superior to the public sector in providing more efficient social services” (Nasir et al., 2016, p. 355).

Indeed, accountability efforts, particularly those involving the federal government, have engendered significant debate about their role in supporting education as a public good. While the dominant policy argument has long been that accountability efforts exist to improve education and support the enterprise as a public good, others have been far more critical. They have taken more critical stances, such as those embodied by Nasir et al. (2016), to argue for a much more nuanced understanding of how accountability efforts have also served to diminish education as a public good (see Anagnostopoulos et al., 2013).

The research that guided teacher evaluation policy, and how it was conducted and by whom, can help us make sense of how the policy took shape, why the policy was problematic in its uptake by states and districts, and, ultimately, why the initiatives were largely abandoned or dramatically reduced in scope. Using the frameworks provided by both Nasir et al. (2016) and McDonnell and Weatherford (2020), we highlight key aspects of research development and use.

The guiding principles of teacher evaluation grew out of the dominant framing noted by Nasir et al. (2016) that has directed policy perspectives on education for the last several decades. Embedded within this work was the meritocratic perspective that teachers are the primary agent associated with student growth and that their relative success is deserved and an outcome of choices and actions by individuals (teachers and administrators). The VAM models were proffered as ways of overcoming the influence of any contextual factors and, thus, were designed to be pure measures of a teacher’s contribution. By controlling for factors such as race and socio-economic status, these models also subscribed to the framing of colorblindness—that evaluation scores are fair estimates of a teacher’s quality regardless of a teacher’s (or their students’) background. Neo-liberal framing was evident throughout the system in hiring and retention policies as well as in the various pay-for-performance schemes that were linked to teacher evaluation.

These framings had important consequences for the kinds of research that was done, who did the research, and how the work was supported. Research on teacher effects was almost always guided by researchers (e.g., Kane et al., 2008; Rockoff et al., 2011) who adopted the three frames identified by Nasir et al. (2016). These researchers, often educational economists, were focused on identifying the “effects” of teaching by adopting methods that were designed to control for contextual effects rather than trying to understand their influence.

As McDonnell and Weatherford (2020) argued, there are multiple actors involved in how research shapes policy and vice versa. The emergence of teacher evaluation policy, including its central features, represented the confluence of a strategic use of evidence to achieve a particular set of objectives. By the time RTTT was developed, the lines between researchers, policy entrepreneurs, and translators and disseminators/advocates had become highly blurred (see DeBray and Houck, 2011). Reckhow and colleagues described how think tanks, foundations, government policymakers, and researchers set a research agenda and policy coordinated to elevate teacher evaluation (see Reckhow and Tompkins-Stange, 2018; Reckhow et al., 2021). All of these players, likewise, were guided by the three frames identified by Nasir et al. (2016). The Bill and Melinda Gates Foundation and the U. S. Department of Education were the two primary drivers of this work. The Gates Foundation funded research, supported intensive district-level reform efforts, provided advocacy, and worked with the U. S. Department of Education. The U. S. Department of Education, through the RTTT program as well as through funding from the Institute of Education Sciences (IES), not only led the policy initiative but was instrumental in leading advocacy efforts (e.g., Duncan, 2009) and funding programs of research that were supportive of the endorsed teacher evaluation efforts. RTTT was driven by a set of core beliefs about public schools, teachers and teacher unions, neo-liberal approaches to the marketization of education, and concerns about academic performance by students in marginalized communities, along with the emergence of research that offered potential solutions.

In 2009, the U. S. Department of Education announced the RTTT competition and invited states to compete for funds to support educational reform (U. S. Department of Education, 2009). The initiative was based on a theory of action that improved teacher quality would lead to improved student learning. Theories of action specify a cause-and-effect relationship between a policy intervention and a set of desired outcomes (e.g., McDonald, 2009). As articulated by Gitomer and Bell (2013), teacher evaluation was championed as improving teacher quality through four complementary drivers. First, teacher evaluation served an accountability purpose in which teachers (and principals) could be held accountable for student performance. Second, evaluation could support what came to be called the strategic use of human capital. In a market-based approach, evaluation results could be used to guide a system of incentives and disincentives to manage the supply of teachers by increasing the supply of effective teachers and removing less effective teachers (e.g., Gordon et al., 2006; Heneman et al., 2006). A third purpose was to improve individual teacher and institutional capacity by including direct measures of classroom instructional quality that could be used as a tool for providing feedback to teachers (e.g., Borko, 2004; Johnson et al., 2004; Kardos and Johnson, 2007). Finally, teacher evaluation could be used to support evidence-based instructional policy by determining the efficacy of particular policies and interventions (e.g., Rowan et al., 2004, 2009; Rowan and Correnti, 2009).

There were two components related to teacher evaluation that all proposals needed to satisfy:

Specific criteria that had to be met included:

By 2011, 19 states received RTTT funding. However, far more states and localities (42 in total) adopted these policies in order to obtain waivers from the NCLB mandates that were still in effect (Gitomer and Marshall, in press). While states, and often, districts within states, varied in how they developed the specifics of their systems, Gitomer and Marshall (in press) described the key features of all systems, their technical limitations, and how they varied across and within states.

One requirement for determining any teacher growth measure is the necessity of defining which students’ growth scores should be used in determining a teacher’s value-added. The realities of schooling made this a non-trivial problem, and the solutions varied greatly. For example, how should students with high levels of absenteeism be treated? If students move between schools multiple times across the year, how should they be treated in the VAM models? What about situations in which multiple teachers are responsible for the students in a particular classroom (e.g., special education)? Of course, less stable student populations are typically associated with schools with high proportions of minoritized and economically insecure students (see Everson, 2017).

A second issue concerned the inclusion of particular test score results for each teacher. In some models, teachers had evaluation scores that included test results for which they ostensibly had no teaching responsibility.

Third, states had to decide which measures contributed to an evaluation system. Almost all states included a student growth measure and a classroom observation score. But other measures, including student learning objectives (SLOs), principal rating, overall achievement levels of grades and/or schools, and student surveys, were also used in some systems.

Fourth, states needed to decide how the scores from different measures were aggregated for a final evaluation score. Aggregation methods could be compensatory, in which each component is weighted in a linear combination of scores, and a total score is used to determine the appropriate evaluation category for an individual. Another option was to use a conjunctive model, in which a minimum score is required for each of the constituent measures. How scores were weighted in the overall model depended on how highly particular measures were valued relative to others, as well as how much variation was associated with particular measures. Measures that have scores that vary more across individuals will have a greater influence on overall evaluative judgments than measures on which most individuals receive the same score, even if the latter measures are assigned nominal weights.

Fifth, states differed in both the consequences and supports given for particular evaluation scores. Typically, an ineffective rating was associated with some type of probationary status for the first year, which then would require some type of additional professional development and support.

Sixth, measures, particularly those associated with classroom observations, required some type of training of principals and other administrators. While researchers have given great attention to observer training, calibration, and overall quality control of scores (National Research Council, 2008; Bell et al., 2014), in practice, these procedures were often compromised as school districts did not have the time or resources to undertake the kinds of procedures that had been used to validate measures from research studies.

RTTT advocated the use of growth measures to overcome inherent problems associated with making any relative judgments of teachers based on their students’ achievement status by separating the effects of teachers from other factors such as demographics, resources, and student prior achievement. The basic logic was that any attributions to teacher effectiveness must be made with respect to the relative year-to-year growth in student achievement. A broad range of growth models were used, including different versions of VAM, as well as a related method, Student Growth Percentiles (SGP; Betebenner, 2009). All growth models required multiple years of student test data linked to individual teachers.

Research on growth models made clear that precise, causal estimates of a teacher’s contribution to student learning were very fragile. Rowan and Raudenbush (2016) provided a detailed overview of the challenges in using growth models to make high-stakes decisions about teachers. Reardon and Raudenbush (2009) explained how the fundamental statistical assumptions that are foundational to these models can never be satisfied. Studies have revealed how relative estimates of teacher quality can shift dramatically because of using different estimation techniques (Goldhaber and Theobold, 2013) or different achievement measures (Lockwood et al., 2007; Grossman et al., 2014). Multiple studies have shown that VAM estimates can be statistically biased toward classrooms that have students with higher levels of prior achievement, a situation that growth models were supposed to overcome (Rothstein, 2009, 2017; Raudenbush, 2013). Researchers also found that VAM scores in one testing domain (e.g., reading) could be influenced by the quality of teaching in another domain (e.g., mathematics) (Koedel, 2009).

Scholars in measurement and statistics issued several statements to caution about the use of these models for high-stakes decisions. Baker et al. (2010) produced a consensus statement of several leading educational scholars that cautioned the use of these measures and also highlighted potential unintended consequences, including discouraging teachers from wanting to work in schools with students who had the most academic needs. The American Statistical Association (2014) also released a statement, recognizing the value of VAM to help understand the relationship of different factors to student outcomes when results are aggregated across teachers but also cautioning against using these models to make strong causal statements about individual teachers. Other cautionary and critical statements were made by the National Association of Secondary School Principals (NASSP) in 2015 (see https://www.nassp.org/top-issues-in-education/position-statements/ for the most recent version, updated in 2019; National Association of Secondary School Principals, 2019) and the American Educational Research Association (American Educational Research Association Council, 2015). Several lawsuits challenging the consequences of teacher evaluation efforts were also instituted (see Paige and Amrein-Beardsley, 2020).

As much as growth measures based on student achievement scores were central to this evaluation movement, the fact is that a very large proportion of teachers did not have testing data that would be appropriate for estimating student growth. Testing was only federally mandated, for example, in grades 3–8 mathematics and reading, meaning that teachers in earlier and later grades, as well as those who taught other subjects, would not have students who had multiple years of testing data to analyze. Certain states did, however, impose more encompassing testing requirements.

Thus, in order to address the legislative mandate that teacher evaluation needed to include a “student growth measure,” most states adopted SLOs for teachers in non-tested subjects, but many states also used them for all teachers as a complementary measure of student growth. SLOs are a locally determined evaluation of teacher effectiveness by which measurable targets for student achievement are set following an analysis of baseline data. Essentially, SLOs include some prior to instruction measure of student understanding (pre-test) and a post-instruction measure or assessment. The extent to which those targets are met is then used to evaluate the teacher. Within this common definition, specific features of the SLO process have varied substantially (see Crouse et al., 2016).

An SLO consists of three components. The first is the population of students it covers—is the teacher evaluated on the basis of performance by all students in all classrooms and subjects taught by the teacher or just a subset (e.g., only mathematics or reading for an elementary teacher, only one section of a course for a secondary teacher)? The second component is the target of the SLO—do all teachers with the same teaching assignment in a school or district have the same target, or is greater variability part of the design? In addition, the meaningfulness of SLO-based scores is largely a function of the quality control procedures used in the implementation of the SLO process (Crouse et al., 2016).

The third component is the assessment to measure student learning. SLO assessments include locally generated measures as well as standardized, externally developed assessments. Often, classroom assessments such as portfolios or some type of performance assessment are used.

While little research about the quality of SLO measures was done, Crouse et al. (2016) described the inherent problems of using SLOs as a measure of student growth in teacher evaluations. They argued that the validity of such measures for evaluating and comparing teachers could not be justified because of the idiosyncratic nature of their design and implementation. They also pointed out that making causal attributions to a teacher was problematic in light of external factors such as district curriculum, outside tutoring, and student background characteristics that can influence student outcomes. Finally, the use of SLOs was highly variable across states and districts. In some cases, all teachers needed to have an SLO as part of their evaluation. In other instances, only those teachers who did not have standardized test-based growth scores were required to have SLOs. Because the distribution of scores for test-based growth models and SLOs tends to be different, the net effect is that overall evaluation scores could be lower for teachers who have growth estimates based entirely or, in part, on standardized tests as compared with those only having SLOs as their growth measure component.

Structured observation protocols, originally designed as tools for professional development (e.g., Danielson, 2007; Pianta et al., 2008), soon became the object of study in research and a key component of teacher evaluation systems under RTTT. These protocols were created around particular views of teaching that drew on research and were organized along sets of cognitive, social, emotional, and classroom management dimensions of instructional quality.

The protocols adopted for teacher evaluation systems were designed to be used across grades and subject areas. Each protocol provided guidelines for how to observe a period of classroom instruction, how to code what was observed, and how to score instruction for the set of criteria that were described in the protocol’s scoring rubric (see Bell et al., 2012).

Scores typically involved some form of aggregation of dimensional scores into a total lesson score as well as aggregation of scores across multiple lessons. The management of observations and recording and maintaining of data within school systems was often done with the assistance of commercial observation tools that were designed specifically to support teacher evaluation processes.

Research has shown the limitations of observation protocols in assuring precise and valid estimates of teacher quality. For one, many factors, other than the quality of the instruction itself, can influence the scores for a particular observation, most especially the observers themselves. Research efforts have tried to moderate these sources of error through careful training and monitoring of observers, using multiple and different observers across multiple observations, and ensuring that there were no conflicts of interest between the observer and the observed that might bias scoring (see Bell et al., 2012, 2014).

As observation measures were used in evaluation systems, it became clear that findings from research studies did not generalize to practice settings. Observation scores in practice are uniformly higher than scores from research studies, for example. Scores in research studies that typically fell in the 2–3 range on 4-point scales fell between 3 and 4 when used in practice (see Sartain et al., 2010; Briggs et al., 2014).

Of course, conditions within practice settings were quite different as observers were not disinterested parties. They knew the teachers and worked with them as part of a professional staff (Harris et al., 2014; Kraft and Gilmour, 2017; Donaldson and Woulfin, 2018), and they gave higher scores for teachers they worked with than for teachers with whom they were not familiar (Ho and Kane, 2013). School administrators must conduct observations by statute, regardless of how well qualified they are to score. Typically, fewer observations were conducted in school evaluations than in research studies, and it was very rare for any system to include multiple observers.

It also became clear, in both research studies and studies of observation in practice, that personal characteristics of the teacher, and especially the students, affected observation scores. There have been consistent findings that teachers of students with weaker academic profiles are assigned lower observation scores (Gitomer et al., 2014; Campbell and Ronfeldt, 2018). In addition, Steinberg and Sartain (2021) found that observation scores of Black teachers were substantially lower than scores for White teachers and that those differences could be accounted for by the achievement levels of their students. Campbell and Ronfeldt (2018) found that male teachers tended to have lower than expected scores than female teachers and that scores were also lower than expected in classrooms with higher concentrations of Black, Latin*¹, male, and low-performing students, a result also found by Garrett and Steinberg (2015).

One goal of teacher evaluation was to differentiate teachers based on their effectiveness. However, Kraft and Gilmour (2017) and Stecher et al. (2018) found that evaluation score distributions were largely unchanged from the findings of Weisberg et al. (2009). Grissom and Loeb (2017) noted that principals would give higher scores in an accountability context than they would for professional development. Not surprisingly (see Rowan and Raudenbush, 2016), evaluators in professional contexts consider many factors aside from the performance itself in making ratings (Harris et al., 2014; Donaldson and Woulfin, 2018). Two explanations for the failure to identify weak teachers are (1) inconsistent training of evaluators (i.e., school leaders); and (2) the difficulty of negatively evaluating colleagues.

If the end goal for improving teacher quality through teacher evaluation is that students would benefit, results were disappointing. Stecher et al. (2018) observed null effects in terms of mathematics and English language arts (ELA) achievement in three large school districts across the 6 years of an intensive push to embed teacher evaluation systems. Bleiberg et al. (2021) conducted a cross-state analysis of student achievement by examining test scores before and after each state implemented their evaluation system. They also found null effects that did not vary over time since implementation.

Critical to the theory of action underlying this policy was the idea that weaker teachers could be replaced with more effective teachers. Substantial effects were found in Washington D. C. (Dee and Wyckoff, 2017; James and Wyckoff, 2020). While studies with other samples found changes in the teaching force, although to a lesser degree (e.g., Grissom and Bartanen, 2019; Nguyen et al., 2019; Cullen et al., 2021), Stecher et al. (2018) found null effects.

One of the key policy mechanisms for improving teaching quality was to provide better and more targeted professional development. Kraft and Gilmour (2017) and Stecher et al. (2018) did not find any evidence of such improvement and attributed this to the inherent tension between evaluations being used for accountability and high-stakes evaluations on the one hand and then being used for professional development on the other. In such cases, the accountability uses typically dominated and crowded out the professional development messages.

Arguably the most important goal of this educational reform initiative was to improve the quality of teaching in schools that had histories of poor academic performance. Schools in urban and impoverished communities were of particular interest as those areas were the face of the U. S. education crisis (Cuban, 1989). These districts typically had high proportions of Black, Latin*, and Indigenous students, English language learners (ELLs), students considered in need of special education services, as well as the highest proportion of minoritized teachers (Boyd et al., 2010; Ronfeldt et al., 2016; D’Amico et al., 2017).

Again, Washington D. C. was relatively unique in making progress toward these goals, but this was an exception. In most targeted districts, teachers were disincentivized from working in low-performing schools. As we have discussed, teachers of students with weaker academic profiles fare more poorly on teacher evaluations (Drake et al., 2019). The bias that has been observed in these systems across measures is alarming.

There are multiple reasons why teachers of students with weaker academic profiles fare more poorly in these evaluation systems. As previously mentioned, there appears to be some statistical biases in the growth model estimates. Additionally, some low-achieving students have high levels of absenteeism, yet their test scores contribute as much to a teacher’s estimate as those of students who rarely miss school. Cowen (2017) found that unhoused students, more likely to be Black and Latin*, are much more transient, almost always impoverished, and have lower achievement levels (i.e., classroom assessments and standardized test scores). Yet, teachers of these students are unfairly treated identically in the growth estimate models.

Classroom observations raise a number of additional issues with respect to equity. Jacob and Walsh (2011), Gitomer et al. (2014), Garrett and Steinberg (2015), Campbell and Ronfeldt (2018), and Steinberg and Sartain (2021) have all found that observation scores are systematically lower for teachers who teach students with weaker academic profiles.

Additionally, Black teachers are more likely to receive lower observation scores (Campbell and Ronfeldt, 2018; Steinberg and Sartain, 2021), and Black teachers who work in schools with predominantly White staff are more likely to receive lower evaluation ratings than those who work at schools with mostly Black colleagues (Drake et al., 2019). Campbell (2020) found that Black women received lower observation scores than White women, even when accounting for other measures of teaching quality, especially in schools where the race of the evaluator differed from that of the teacher.

Unfortunately, there is a paucity of research on the effects of teacher evaluation policies on teachers of students who represent the full range of students in American schools. However, there have been several studies that have discussed the complexity of conducting evaluations of teachers of special education students (Jones et al., 2022) and of English language learners (Turkan and Buzick, 2016).