The Validity of Standardized Tests for Evaluating Curricular Interventions in Mathematics and Science

The literature expresses concern about the use of standardized tests as outcome measures for evaluating the educational impact of new curricular interventions. The concern is that the standardized test may measure the wrong outcome and the evaluation will misjudge the intervention, derailing educational reform. In spite of these concerns, the literature contains almost no empirical information about the use or validity of standardized tests as outcome measures. In this talk, I discuss three studies that explore the prevalence and validity of standardized tests for evaluating curricular interventions in mathematics and science. The studies examine the alignment between the tests and the interventions and the issues that arise with low alignment. The first study is a secondary analysis of Institute of Education Sciences (IES)-funded research in mathematics and science education. The analysis shows that the use of standardized tests as outcome measures is common and frequently leads to measurement problems. In the second study, a simulation experiment models the relationship between alignment and the sensitivity of an evaluation for detecting treatment effects. The results produce minimum benchmarks for alignment, demonstrating that standardized tests typically lack the alignment to serve as valid measures in impact evaluations. The third study is a quasi-experimental evaluation that documents the educational effectiveness of an innovative mathematics intervention for teaching rational number to English learners in 3rd and 4th grade, while solving the alignment problem by coordinating results from two achievement tests and a theoretical mechanism of action for the intervention. Taken together, the studies expose basic problems with applied measurement in education and point to practical solutions grounded in changes to policy and the use of specific research methods.

Joshua Sussman is a postdoctoral scholar at the BEAR center. His research focuses on applied educational measurement, such as the validity of outcome measures in evaluations of curricular effectiveness, communication of complex psychometric information to stakeholders, and the construction of new assessments in areas such as early childhood development. He is currently writing about (a) various aspects of the challenges of selecting valid outcome measures to evaluate the educational impact of new curricular interventions in mathematics and science and (b) evaluation of a mathematics curriculum that supports learning in language inclusive classrooms. He is a credentialed school psychologist in California and received his Ph.D. in School Psychology (2016) from the University of California, Berkeley supported by an Institute of Education Sciences Predoctoral Fellowship in the Research in Cognition and Mathematics Education program.

Tuesday, November 14, 2017 - 2:00pm
PDF icon Sussman presentation1.21 MB