Improving the Estimation of Site-Specific Effects and their Distribution in Multisite Trials [Online]


Modeling site-specific effects using observed data is a critical component in understanding the results of multisite trials. A standard approach leveraging Bayesian methods is to rely on Gaussian distributional assumptions and to use the posterior means (PM) of the random effects. The standard approach can be misleading, however, in the estimation of individual site-specific effects and their empirical distribution and ranks. In this talk, we review the following two strategies developed to improve inferences regarding site-specific effects: (a) relaxing the normality assumption by flexible modeling of the random-effects distribution using Dirichlet process mixture (DPM) models, and (b) replacing the choice of PM as the summary of the posterior by alternative estimators, such as the constrained Bayes (CB) or the triple-goal (GR) estimators. We then examine when and to what extent the two strategies and combinations thereof work or fail under varying conditions. We found that the informativeness of the data is the most influential factor for all inferential goals and determines the effects of other factors. When the data are uninformative, specifying a flexible DPM model is not in general an effective strategy compared to a Gaussian model. A simple parametric model combined with a posterior summary method targeted toward an inferential goal performs better under the uninformative data environment. When data is informative, however, DPM models tend to outperform a Gaussian model particularly in estimating percentiles of the underlying distribution.

JoonHo Lee is an Assistant Professor in Educational Research at the University of Alabama. Lee studies quantitative methods in educational research, with a focus on multilevel and longitudinal modeling, modern approaches to causal inference, and quantile regression. His methodological research centers on understanding and measuring treatment effect heterogeneity in experiments and quasi-experiments in education. His applied research uses the quantitative methods to explore why, or through what processes, new school funding typically fails to narrow student achievement gaps. Lee is a 2019-2020 American Educational Research Association (AERA) Dissertation Fellow and he earned his Ph.D. in Quantitative Methods and Evaluation at the University of California at Berkeley in 2020.

Tuesday, November 24, 2020 - 2:00pm
Online session
PDF icon Presentation slides1.09 MB