What Machines Can Learn About the Humans Who Train Them: Application of Rater Modeling in Automated Essay Scoring


Scoring student work using machine learning (ML) algorithms has become increasingly common. The training data used for supervised learning typically consists of judgments of human raters collected on a sample of student work. Human raters, however, are subject to defects of inconsistency and bias. This study examines how such defects among individual raters impact overall measures of rating quality and the performance of automated scoring engines. Furthermore, the feasibility and utility of learning (i.e., modeling) the behavioral profiles of individual raters, as a component of engine training, is explored. The hierarchical rater model (HRM) framework is employed for simulating rater errors, and an ML-based automated scoring engine is developed using typical elements for text processing, tokenization, feature extraction, and modeling via gradient-boosted regression trees. Research questions are addressed through a simulation study and by examining scored data from the International Corpus of Learner English (ICLE). Machines that appropriately recognize and account for the particular humans that train them are shown to provide superior scores, especially under a subset of the studied conditions. Implications and possible applications are discussed.

Richard Patz is an educator, researcher and consultant who specializes in educational assessment and statistics. He has held numerous scientific and executive positions in the educational testing industry, and he has held visiting scholar appointments at UC Berkeley and Stanford University, and graduate faculty status at the University of Massachusetts. He currently serves as a Distinguished Research Advisor at the BEAR Center and on the faculty of UC Berkeley’s Fall Program for Freshmen, where he teaches mathematics and statistics. He also maintains an active portfolio of consulting projects with innovative education and research organizations.

Rich earned his masters and Ph.D. degrees in statistics from Carnegie Mellon University, and a bachelor’s degree in mathematics from Grinnell College. A former high school mathematics teacher, he has an abiding interest in education and human development that animates his work. He has served in numerous volunteer and leadership roles, and is a past president of the National Council on Measurement in Education.

An author of numerous journal articles and book chapters, Rich’s scholarship has focused on quantitative methods in the social sciences, and methodological and applied research in educational testing and measurement. His practical experience includes the development and utilization of advanced technologies in support of teaching and learning.

Tuesday, September 17, 2019 - 2:00pm
Berkeley Way West
PDF icon Presentation slides1.38 MB