Tuesday, May 3, 2022
2:00 - 4:00 PM (PDT) at Berkeley Way West 1217 and on Zoom
Abstract:
Careless and insufficient effort responding (C/IER) poses a major threat to the quality of questionnaire data. Traditional indicator-based procedures for its detection are limited in that they are only sensitive to specific types of C/IER behavior, such as straight-lining or rapid responding, and do not allow to take the uncertainty of C/IER classification into account. I will present two approaches that leverage response times to overcome these limitations. Both allow considering the uncertainty in C/IER identification and are agnostic towards the specific types of C/IER patterns in the data. The first approach is a model-based procedure that incorporates theoretical considerations on attentive and C/IER behavior on questionnaires. C/IER is identified on the item-by-respondent level. In doing so, this approach supports a fine-grained identification of C/IER and a more nuanced understanding of response behavior. The second approach is a scalable two-step screen-time-based weighting procedure that downweighs responses according to the probability of stemming from C/IER. The procedure can feasibly be integrated with common analysis workflows for large-scale survey data. I will provide illustrations of both approaches using PISA student background questionnaire data.
About the speaker:
Esther Ulitzsch is a Research Associate at the Leibniz Institute for Science and Mathematics Education in Kiel, Germany. She holds a PhD from Freie Universität Berlin. Her current work focuses on the development of methods for the analysis of process data from large-scale assessments, ranging from psychometric models to exploratory sequence mining techniques. Her dissertation on the use of response times for modeling missing values was awarded the NCME Brenda Loyd Dissertation Award and the Gustav A. Lienert Doctoral Thesis Award from the Expert Group of Methods and Evaluation of the German Psychological Society.