Respondent Speed in the Context of Adaptive Testing

As our ability to collect data about respondents increases, approaches for incorporating ancillary data features such as response time are of heightened interest. Models for response time have been advanced, but relatively limited large-scale empirical investigations have been conducted. We take advantage of a unique and massive dataset---data from computer adaptive administrations of the NWEA MAP assessment in two states consisting of roughly 1/4 billion item responses---containing both item responses plus response times to shed light on important test-taker behaviors. We focus on two behaviors in particular. The first, response acceleration, is a reduction in response time for responses that occur relatively late on the assessment. We further note that such reductions are heterogeneous as a function of estimated ability and that reductions in response time on later items lead to reductions in their probability of correct responses relative to expectation. The second, heterogeneous processing, suggests that response time has a surprising relationship with the ultimate response depending on the underlying difficulty of the particular item for an individual. This finding has potential connections to the nascent literature on different within-person response processes. These empirical findings offer potential insight on how response times could be used to improve measurement processes. Please attend if you are interested in response time or just want to tell me what I should be doing with an item response dataset of this magnitude!

Ben Domingue is an assistant professor in the Graduate School of Education at Stanford University. He is interested in how student outcomes are leveraged to inform our understanding of student learning, teacher performance, and the efficacy of other programs. He has a particular interest in the technical issues that make it challenging to draw simple inferences from such student outcomes. While not analyzing item response data, he may be found thinking about the implications for social science of the sudden increase in our capacity to measure human DNA and the promise and pitfalls associated with how this new data may change our understanding of human behavior.

Tuesday, April 28, 2020 - 2:00pm
Berkeley Way West