Evaluating the Treatment Effect in the ADM Study and Lord's Paradox

I present my findings from an analysis of the effects of a Data Modeling curriculum designed to improve statistical reasoning skills, as well as general math achievement when compared to the existing curriculum. The data were collected from cluster-randomized trials, wherein schools from districts were assigned to treatment or control at random within each district and pretests and posttests were administered to students before and after the treatment. Two main approaches for analyzing such data are (1) to regress the change from pretest to posttest on the treatment indicator; (2) to regress posttest on the treatment indicator and pretest. These two approaches can yield conflicting results. Lord (1967) warned of this problem decades ago and started a debate that continues to the present day. In this presentation, I first discuss the apparent paradox at the heart of this issue. Then I elaborate on both of these approaches, examine the appropriateness of each for analyzing the treatment effect, and discuss these two approaches from the latent variable and multilevel modeling perspectives.

Perman Gochyyev is a research psychometrician at the University of California, Berkeley, at the Berkeley Evaluation and Assessment Research (BEAR) Center. Perman received his PhD in Quantitative Methods and Evaluation from UC Berkeley in 2015. His research focuses on latent variable and multilevel modeling, multidimensional and ordinal IRT models, latent class models, and issues related to causal inference in behavioral statistics.

Tuesday, October 17, 2017 - 2:00pm
PDF icon Gochyyev presentation1.35 MB