BEAR Events Calandar

Spring 2001

BEAR Seminar Members
"Reports from the Evaluators' Institute "

The Evaluators' Institute provides short-term professional development courses for practicing evaluators.  Attendees of the 2001 San Francisco workshops report what they learned, which topics are "hot", and how they evaluated the evaluation training.

Theo Linda Dawson, Ph.D., UC Berkeley
"The Shape of Learning: A Lifespan Cognitive-Development Perspective"

In order to address several questions about conceptual development in the moral domain, we gathered 747 Heinz and Joe (Form A, Colby & Kohlberg, 1987) and Picnic interviews (Damon, 1980) from 8 different studies of moral judgement. Respondents were from 4 to 86 years of age and represent a range of socio-economic groups. Interviews were scored for their hierarchical complexity (stage) employing Commons' Hierarchical Complexity Scoring System (HCSS) in order to examine the major postulates of stage theory. Patterns of performance indicate that (1) stages of development are apparent in performance; (2) the sequence of acquisition is invariant; (3) there are seven stages from age 4 to middle age; (4) development proceeds in a series of spurts and plateaus; and (5) patterns of performance are similar for childhood and adulthood stages.

Lynde Paule , GSE Evaluation Unit, UC Berkeley
"Work In Progress: A Survey Instrument for Evaluating a GSE Scholarship"

An evaluation plan for the Flanders Fellowship will be described. A draft of a survey instrument for use in the evaluation will presented and discussed.

Paul De Boeck, Department Psychology, University of Leuven, Belgium
"Categories versus Dimensions as Structures for Psychological and Educational Data"

In different fields of psychology and education dimensional models are often used. The implication of their use is that the differences between persons and groups are gradual and quantitative. Alternatively, in a categorical model, the differences are no longer gradual or quantitative (but qualitative instead). An approach will be presented to differentiate between the use of categories and dimensions as structural entities to explain differences. For example, are personality disorders categorical or dimensional, are age groups differences categorical or dimensional?

David Sul, UC Berkeley
"Work in Progress: An Evaluation of the Santa Clara University Calculus Readiness Examination

Santa Clara University is a private liberal arts college located in Santa Clara, CA. During summer orientation, freshmen students planning to enroll in Calculus are asked to take an on-line assessment test to determine their "readiness" for Calculus. This presentation will focus on an analysis being completed for the Department of Mathematics and Computer Science on the exam. Methods employed include those of both Classical Test Theory and Item Response Theory.

John Gargani& > Linda Morell, UC Berkeley
"Reports from the ConferenceEvaluating Social Programs & Problems: Visions for the New Millennium "

Michael Scriven,> Joseph S. Wholey, David Fetterman , Yvonna Lincoln ,Donna M. Mertens,>Stewart I. Donaldson, Thomas D. Cook, and others articulated their visions for evaluation.  Topics included improving accountability, constructivist evaluation,empowerment evaluation,theory-driven evaluation, inclusive evaluation, and transdisciplinary views of evaluation.

Paul Holland, ETS, &Machteld Hoskens , Ph.D. UC Berkeley
"Classical Test Theory as a First Order Approximation to Item Response Theory with Application to Scaling and Linking Tests."

We give an account of Classical Test Theory (CTT) in terms of the more fundamental ideas of Item Response Theory (IRT). This shows that we may view CTT as a mean and variance (i.e., "first-order") approximation to a very general version of IRT that contains many specific IRT models as special cases. This approach views classical test theory as a very general version of IRT, and the commonly used IRT models as detailed elaborations of CTT for special purposes. We then use this approach to CTT to find some general results regarding scaling and linking of the scores from different tests. This leads us to a new measure of the effects on inferences from test scores that result from attempts to link tests that were not developed with linking in mind. In addition we develop IRT analogues to the Dorans and Holland measures of test equatability. We illustrate the accuracy of the first-order theory using simulated data from the Rasch model, and illustrate the effect of population differences using a set of real data.

Juliet Popper Shaffer, Professor, UC Berkeley
"Analyzing Educational Surveys: Experiences at NCES, ETS, and NISS"

The technical and practical challenges (and delights) of large-scale educational surveys are outlined by someone who's "been there and done that!"

Eva Ponte , Ph.D., UC Berkeley
Presentation of Dissertation

I hope to accomplish a couple of things during this talk.  First, I will touch on my dissertation work.  Second, I will reflect on the making of it as a way to offer some insights to students who find themselves at this stage of their work.

In the first part of this presentation I will offer an account of how a teacher and a researcher (myself) collaboratively designed a portfolio assessment.  Specifically, I will discuss how our views about teaching, learning, language, and assessment influenced the design process of the portfolio assessment.  I will also describe how this teacher implemented the portfolio assessment in his classroom and how he made sense of its actions within the context of the classroom.  Finally I will briefly address changes in the teachers’ understanding about instruction and assessment and the student’s learning as a result of the use of portfolio assessment in this classroom. I will then talk about “The making of my Dissertation.”  I will discuss how I chose the topic for my dissertation, how I went about gathering and analyzing my data, how I organized my schedule to write my dissertation, and how my advisors guided my work. I hope this presentation will provide some useful insights into the use of performance assessment in the classroom and will offer some helpful hints to students about completing their dissertations.

Diane Allen, UC Berkeley Ph.D. Student

"From Admission to Clinician:  Predicting Success in a Graduate Education Program for Health Care Professionals"
Of what use are undergraduate grade point averages (GPA) and Graduate Record Exam (GRE) scores in determining how an applicant will perform in graduate school?  These and other admissions factors were investigated in relation to the performance of physical therapy students in their professional
education.  Failure to progress in the curriculum exacts a toll on faculty members and students alike, in emotion, time, and budgeting.  If we could predict more accurately which applicants are likely to succeed, or to have difficulties, we might avoid this toll, or prepare ourselves better to promote success despite the challenges.

Sungsook Kim, Korea Institute of Curriculum & Evaluation (KICE)
"A further interpretation of analyzing performance assessments : A comparison of G theory and MFRM (Multi-Facet Rasch Model)"

The purpose of this study is to compare the results of analyzing data from a performance assessment (written composition) for a college entrance exam using two methods, Generalizability theory and the Multifacet Rasch Model (MFRM). The view that G theory and MFRM are alternative solutions to the same measurement problem, in particular rater effects, is seen to be an illusion. G theory provides a general summary including the relative influence of each facet on a measure and the reliability of decision estimating from data collection. MFRM concentrates on the individual examinee and provides a measure that is as fair as possible to derive from the data. In addition, the study presents a further interpretation based on understanding the merit of two methods for the multifaceted of performance assessments.

Robert McCarthy, UC Berkeley Survey Research Center
"Survey Research: Data Management Issues and Strategies"

Robert McCarthy has worked with the Survey Research Center at UC Berkeley since 1978.  Currently he is responsible for the day-to-day operation of the Data Management Unit of the Center's Survey Services Facility.  He will discuss some recent projects and general data management issues and strategies.