What Makes Reading Difficult? An Investigation of the Contribution of Passage, Task and Reader Characteristics on Item Difficulty Using Explanatory Item Response Models

Reading comprehension (RC) is often viewed as a multi-faceted, multi-layered construct (Broek & Espin, 2012; Duke, 2005; Graesser & McNamara, 2011; Perfetti & Stafura, 2014), which manifests through complex interactions among three broad factors: the reader, the passage, and the task, all residing in a particular socio-cultural context (RAND Reading Study Group, 2002). Drawing on the item difficulty modeling paradigm, this study examined how these three factors as well as their interactions affected comprehension difficulty. Specifically, the study used explanatory item response models (De Boeck & Wilson, 2004) to analyze a vertically-scaled item response matrix from an operational online assessment, which included a wide range of readers (n=10,547) as well as of passages (n=48), covering grades 1 through 12+. Analyses indicated that it is text features, as measured by computational text analyzers, rather than task features as coded by human raters, that explained over half of the variance in item difficulty, after controlling for student general vocabulary knowledge. Specifically, sentence length, word frequency, syntactic simplicity, and temporality (i.e., the extent to which the text has time markers) were found to significantly affect comprehension difficulty in both model building and cross validation analyses. Further, small but significant interaction effects were found, indicating that these textual effects were moderated by student general vocabulary knowledge as well as task demands as captured by item types. In general, readers with higher vocabulary knowledge benefitted more from traditional textual affordances (e.g., shorter sentences, familiar words) than their peers with lower vocabulary knowledge, especially when questions asked them to recall specific localized information without accessing the source passage. However, a reverse effect was found with temporality: passages with more time markers helped low vocabulary readers, while it was low temporality passages that helped high vocabulary readers. The implications of these findings as well as their limitations will be discussed as they relate to the measurement of RC and text complexity, as well as to instructional practice.

Yukie Toyama is currently a postdoctoral scholar at BEAR Center, University of California, Berkeley. She received a Ph.D. in Quantitative Methods and Evaluation from the University of California Berkeley’s Graduate School of Education in 2019. She is interested in researching text and language complexity involved in testing to inform the field about test accommodations and item designs to make assessments fairer and accessible to struggling readers, including English Language learners. She is also interested in investigating growth in student reading ability using IRT models.

Tuesday, December 3, 2019 - 2:00pm
Berkeley Way West
PDF icon Presentation slides1.76 MB