Tuesday, March 15, 2022
2:00 - 4:00 PM (PST) on Zoom
Abstract:
Understanding the "fit" of models designed to predict binary outcomes has been a long-standing problem. We propose a flexible, portable, and intuitive metric for quantifying the change in accuracy between two predictive systems in the case of a binary outcome, the InterModel Vigorish (IMV). The IMV is based on an analogy to well-characterized physical systems with tractable probabilities: weighted coins. The IMV is always a statement about the change in fit relative to some baseline---which can be as simple as the prevalence---whereas other metrics are stand-alone measures that need to be further manipulated to yield indices related to differences in fit across models. Moreover, the IMV is consistently interpretable independent of baseline prevalence. We illustrate the flexible properties of this metric in numerous simulations and showcase its flexibility across examples spanning the social, biomedical, and physical sciences. The IMV allows for precise answers to questions about changes in model fit in a variety of settings in a manner that we think will be useful for furthering research with binary outcomes.
About the speaker:
Ben Domingue is an assistant professor in the Graduate School of Education at Stanford University. He is interested in how student outcomes are leveraged to inform our understanding of student learning, teacher performance, and the efficacy of other programs. He has a particular interest in the technical issues that make it challenging to draw simple inferences from such student outcomes. While not analyzing item response data, he may be found thinking about the implications for social science of the sudden increase in our capacity to measure human DNA and the promise and pitfalls associated with how this new data may change our understanding of human behavior.