Tuesday, October 29, 2024
2:00 - 4:00 PM (PDT) at Berkeley Way West 1207 and via Zoom
Open to GSE faculty, students, and community.
Request a zoom link from convenors@bear.berkeley.edu
Abstract:
Analyses of heterogeneous treatment effects (HTE) are common in applied causal inference
research. However, when outcomes are latent variables assessed via psychometric
instruments such as educational tests, standard methods ignore the potential HTE that may
exist among the individual items of the outcome measure. Failing to account for "item-level"
HTE (IL-HTE) can lead to both underestimated standard errors and identification challenges
in the estimation of treatment-by-covariate interaction effects. We demonstrate how Item
Response Theory (IRT) models that estimate a treatment effect for each assessment item can
both address these challenges and provide new insights into HTE generally. This study
articulates the theoretical rationale for the IL-HTE model and demonstrates its practical value
using 73 data sets from 46 randomized controlled trials containing 5.8 million item responses
in economics, education, and health research. Our results show that the IL-HTE model
reveals item-level variation masked by single-number scores, provides more meaningful
standard errors in many settings, allows for estimates of the generalizability of causal effects
to untested items, resolves identification problems in the estimation of interaction effects, and
provides estimates of standardized treatment effect sizes corrected for attenuation due to
measurement error.
About the authors:
Joshua Gilbert is a PhD student in Education Policy and Program Evaluation at the Harvard
Graduate School of Education, where he works with James Kim and Luke Miratrix. His
research interests include the intersection of causal inference and psychometric methods, and
his work has been published in journals such as Developmental Psychology, Journal of
Educational Psychology, Journal of Educational and Behavioral Statistics, Behavior Research
Methods, Journal of Research on Educational Effectiveness, Applied Measurement in
Education, Epidemiologic Methods, and others.