Angela Haydel DeBarger, Ph.D.
SRI International

PADI: Defining the evidence to be gathered

Linda Morell, Ph.D.

Larry Gallagher, Ph.D. and Geneva Haertel, Ph.D.
SRI International

Larry Hamel
Codeguild, Inc.

PADI: Defining the purpose of the assessment system

Michelle Riconscente, University of Maryland

A series of 4 hands-on workshops demonstrating the PADI approach to designing assessments, assessment systems, and assessment tasks will be presented throughout the Fall BEAR Seminar Series. These workshops present a framework and software tools to design assessments in which meaningful inferences about student progress and status can be drawn. The framework is particularly useful when assessment data involve multiple interrelated measures and/or sequential responses.

The workshops will help participants understand the inferential structures that underlie all assessments, how to evaluate such structures, and how to design structures that make sense of student responses consistent with a particular assessment purpose. Participants who bring a laptop with wifi capability will be able to use the PADI software; others can design using print documents.

This first workshop focuses on constructing an assessment argument for measuring particular aspects of what students know and can do. Once the assessment purpose is established, assessment tasks and inferential structures can be designed consistent with that purpose. We introduce three components of the PADI Design System: the Design Pattern, the Task Template, and the Student Model. These components are used to record the purpose underlying an assessment and for subsequent alignment of assessment tasks with that purpose. These subsequent steps are illustrated in the following workshops. Participants are encouraged to bring an assessment that they would like to model using PADI techniques and tools.

Download the Workshop 1 slides.

PADI: Defining the evidence to be gathered

Angela Haydel DeBarger, Ph.D., SRI International

This workshop introduces participants to the PADI approach for designing task formats including the content that is presented to respondents, the work products that respondents produce, and contextual conditions that influence either what is presented or what is produced. Both dynamically-generated and static content will be described for systems that adhere to Evidence Centered Design principles and the Four-Process Delivery Model. The PADI Design System components presented in this workshop include Materials and Presentation Requirements, Work Products, and Task Model Variables. Participants are encouraged to bring assessment tasks, particularly complex tasks, that they would like to model using PADI techniques and tools.

top

Validity Evidence Based on Response Processes: An Examination of Student Contribution and the Validation Process

Linda Morell, Ph.D.

Researchers in the areas of psychology and education have been exploring the intersections among aspects of validity, educational measurement, and cognitive theory for at least half a century. Linda Morell’s dissertation looked at these areas and collected information to address two related research questions: (1) how can respondents contribute to the validation process in ways other than providing traditional “subject” information, and (2) how can evidence gathered regarding respondents’ opinions inform the validation argument? A mixed model conceptual framework was used to address the research questions; and evidences were collected through a variety of methods including, traditional paper and pencil tests, surveys, think-alouds, and exit interviews of fifth and sixth grade students, as well as interviews with teachers and science experts. Findings will be discussed during her presentation as well as a brief discussion of how she completed her work.

top

Measuring Measuring

Brent Duckor, Ph.D.

top

PADI: Defining the interpretations that can be made from response data

Larry Gallagher, Ph.D. and Geneva Haertel, Ph.D., SRI International
and Larry Hamel, Codeguild, Inc.

This workshop will illustrate how assessment designers link the responses that students produce with the measurement objectives of the assessment. Assessment coherence is established by specifying how the work products developed by students are evaluated to provide evidence of particular knowledge, skills or abilities. The PADI Design System components introduced in this workshop include Evaluation Phases, Observable Variables, and Measurement Models. We also introduce Gradebook software that automates implementation of certain types of Evaluation Phases. Participants are encouraged to bring examples of student work to consider how evaluation phases, observable variables, and measurement models might be defined and represented in the PADI Design System.

top

Evaluating Differential Gender Performance on Large -Scale Assessments

Ou (Lydia) Liu, Ph.D.

Efforts to determine and explain performance differences on large-scale assessments are often thwarted by two factors: (1) assessments have multidimensional subscales and (2) examinees are of differing background characteristics. This study presents an innovative approach to evaluating the concerns of differential gender performance on large-scale math assessments. Using several latent trait modeling approaches (i.e., Rasch model, Rasch DIF model, latent regression, mixture model), the content specifications of an assessment can be linked to the cognitive and noncognitive aspects of examinee latent traits. This new approach is illustrated with a real data example using the 2000 and 2003 PISA data sets. The analyses focused on gender differences on mathematics achievement. Three research questions were investigated. First, how do gender differences vary across math content domains and item types? Second, do items display differential item functioning and differential facet functioning? Third, how do the two genders differ in noncognitive variables (i.e. motivation, self-efficacy) and how do the two genders differ in learner characteristics and do learner characteristics affect math performance? The investigations yielded interesting findings.

top

PADI: A Complete Example

Cathleen A. Kennedy, Ph.D., BEAR Center

This concluding workshop of the PADI series provides an end-to-end example of how an online assessment was designed, calibrated, and delivered to students using PADI techniques and tools. The assessment measured two dimensions of knowledge and assessment tasks involved multiple, interdependent responses. The assessment system itself was deployed on a non-PADI system, but accessed PADI software located on the BEAR Center server to compute proficiency estimates. These estimates were then immediately presented to students as they interacted with the online assessment. In this session we introduce three new software tools: a wizard for designing assessments, the Calibration Engine for computing item and model parameters, and the Scoring Engine for producing proficiency estimates. Participants are encouraged to bring the work they’ve completed in previous workshop sessions to try out the wizard for generating new assessments from existing components.

top