Artificial Test-Takers as Transformed Controls: Measuring SAT Difficulty Drift and Student Performance

Abstract

Abstract Introduction: Standardized test score trends are widely used to track student performance and inform policy, but they are difficult to interpret when exam content changes over time. We introduce an artificial test-taker framework that uses a fixed large language model as a stable benchmark to measure SAT Math difficulty drift and construct difficulty-adjusted measures of student performance. Methods: We built a longitudinal SAT Math item bank from SATs spanning 2007–2023. For each year, we generated 50 bootstrapped SAT forms that match the year-specific section blueprint and administered all items to GPT-4 under a fixed set of parameters and training to develop counterfactuals. We combine our difficulty benchmarks with national SAT Math scores released by the College Board to assess robustness to compositional changes. Results: The artificial test-taker framework indicates a statistically significant decline in SAT Math difficulty of 0.21σ relative to 2012. After adjusting for test difficulty using the transformed-control benchmark, student performance declines by 34 points in Average Difference in Scores (ADS) from 2012 to 2023. Heterogeneity analyses show that these declines are not uniform across racial groups. Discussion: Artificial test-takers provide a scalable, protocol-invariant audit of longitudinal comparability when traditional equating is infeasible, opaque, or incomplete. Our findings imply that evolving SAT Math content can mask substantial underlying performance decline and can differentially obscure trends across student subgroups. More broadly, transformed-control designs using AI offer a tool for benchmarking educational outcomes and for separating changes in measured performance from changes in the measurement instrument itself.