Jooyong Park, Visiting Scholar UC Berkeley

Constructing Your Own Answers on a Multiple-choice Test?

Mark Wilson, Min-joung Kim, Kavita Seeratan, Kristen Burmester and Mary Full, BEAR Center

BEAR Seminar - Assessing Data Modeling and Statistical Reasoning

Nathaniel Brown, BEAR Center

Chris Harrison, Kings College London

A Path to Deep Conceptual Understanding: The Meaning Equivalence Instructional and Assessment Methodology

Kavita Seeratan, BEAR Center

Current methods (e.g., multiple-choice) available for the assessment of learning are plagued with problems and often do not reveal depth of comprehension of the learned material. Frequent criticisms regarding their heavy reliance on memory-based or procedural knowledge, has provided impetus for the development of alternative theoretically valid, psychometrically authenticated methods for probing understanding.

Discourse theory researchers propose that the ability to mentally represent a given meaning in a variety of ways is a pre-requisite for, and a marker of, deep comprehension. It is this experimental paradigm that underpins the Meaning Equivalence (ME) methodology. The ME method aims to evaluate deep comprehension of newly acquired concepts in a given domain by appraising learners' ability to recognize and produce multiple representations of content that encode equivalence-of-meaning (Shafrir, 1999; Sigel, 1999). ME is also believed to allow learners to begin thinking in ways that promote deep processing and understanding, prerequisites for successful knowledge transfer (Seeratan, 2006). Other pedagogical advantages of the method include its capacity to identify learners' strengths and weaknesses; facilitate instruction and remediation; and be objective, practical, and technological scalable.

This workshop will describe the design, development, application, various evaluative implementations of the ME method, as well as implications and next steps.

Constructing Your Own Answers on a Multiple-choice Test?

Jooyong Park, Visiting Scholar UC Berkeley

The word “construct” does not seem to go with a multiple-choice (MC) test, because examinees respond to MC items by choosing among ready-made options rather than constructing their own answers. In this presentation, I will introduce a new system, called the Constructive Multiple-choice Testing (CMT) system, that forces examinees to do exactly that. The CMT system combines the strength of the constructed response and the MC formats using computer technology.

The CMT system has two versions. In the implicit version of the system, by being given only the stem of a multiple-choice problem, examinees are called upon to generate the answer as if they were solving an open-ended problem. Once the examinee is ready to respond to the question, the examinee can signal to the computer to show the options by clicking the mouse. The options are presented for a short duration; just enough time for the examinee to check his or her answer against the options and choose one. This procedure allows for the activation of relevant information from the examinee’s memory as in a short-answer test, but permits objective scoring. In the explicit version of the system, examinees are required to answer to the same question twice: first, by typing in their answer, and second, by choosing among MC options for the same question.

Empirical results and implications from experiments using the two different versions of the CMT system and future research direction will be discussed.

top

The Assessing Data Modeling and Statistical Reasoning Project

Mark Wilson, Min-joung Kim, Kavita Seeratan, Kristen Burmester and Mary Full,
BEAR Center

The Assessing Data Modeling and Statistical Reasoning project is an interdisciplinary collaboration between practitioners and researchers with expertise in the following areas: learning, assessment, instruction, and technological innovations. This project involves the development of a hands-on curriculum and an embedded assessment system, used concurrently to increase diagnostic and instructional capacity in an area vital to education in both mathematics and science: data modeling and statistics.

Our overarching goal is to investigate how reasoning in data modeling and statistics develop and co-evolve in students' thinking in elementary and middle schools. For this purpose we have developed a set of 7 constructs related to data modeling and statistics: Theory of Measurement, Modeling Measurement, Data Display, Meta-representational Competence, Conceptions of Statistics, Informal Inference, and Chance.

In this presentation we will give an overview of the project, goals, and people involved with it. Next, we will demonstrate how constructs were used to create an assessment system that is embedded in, and aligned with, the curriculum. Finally, we will describe aspects of the diverse working communities that are affected through the process of implementing the BEAR system.

top

The Carbon Cycle Project

Karen Draney, BEAR Center

The Carbon Cycle Project, a joint project between Michigan State University and the BEAR Center at UC Berkeley, involves the development of a research-based learning progression for an important topic in the life, physical, and earth sciences: Transformations of matter and energy in biogeochemical systems, focusing on the ecological carbon cycle and human effects on that cycle. These processes are a fundamental part of the K-12 science curriculum, and their importance is likely to grow in the future. Human populations draw sustenance through these transformations of matter and energy. As our populations grow and our technologies become more powerful, our responsibilities for maintaining these systems will grow, too.

We are currently developing learning progressions, and accompanying teaching experiments, for investigating student learning of several key concepts involved in the carbon cycle. These include the tracing of matter through systems, the growth of biological and engineered systems, energy, inquiry, and citizenship. In this presentation, we focus on the first of these. 

We have examined student responses to carbon cycle items at several levels of schooling, from elementary to high school, and have used the patterns we have noticed to design the progress variables that form our learning progression. We will be discussing the development process, and the current versions of the progress variables.

We are currently designing additional items on a variety of scales, including molecular level changes, cellular metabolic processes, macroscopic processes, and large-scale processes in ecosystems. We are ensuring that our items have several levels of scaffolding, from those in which students are explicitly reminded to explain matter changes at other than the macroscopic level, to those in which students are asked about processes for which a satisfactory answer would require more than one level, but are not explicitly reminded to do this. We thus hope to examine the process by which students come to understand how these various levels are connected.

In addition, we will soon be engaging in a number of teaching experiments. These will involve week-long curricular units, centered around the carbon cycle. As part of these experiments, we will be administering pretests and posttests, and at least one embedded activity, centered around the learning progression and associated items. We will use the data gathered to understand how students' thinking changes with small amounts of focused instruction, and to further refine our understanding of the learning progression.

Download the Standard Setting powerpoint presentation.

top

Methodology in Conceptual Change Research: Characterizing and Measuring Explanations of Chemical Phenomena

Nathaniel Brown, BEAR Center

Research on conceptual change suffers from a lack of agreement over the nature of conceptual understanding of scientific phenomena, with several competing and often explicitly contradictory theories coexisting in the literature (e.g., intuitive theories, p-prims, mental simulations, etc.). The argument is made that the state of the field is due, in part, to four methodological issues: lack of attention to situated, distributed, and interactive aspects of cognition, insufficient precision in describing conceptual activity, incomparability of results across samples and contexts, and poor accounting for students that are exceptions to general trends. A research agenda is described that addresses these issues by combining the qualitative methods of conversation analysis with the quantitative methods of objective Rasch measurement. This integrated approach is illustrated in the characterization and measurement of undergraduate students' explanations of chemical phenomena involving dynamic equilibrium. Explanations are characterized as narrative sequences, co-constructed by listener and speaker, supported by illative chains of kinesthetic, visual, and verbal cognitive elements. Explanations are measured to have relatively shallow depth and narrow breadth in the particular contexts in which they were produced, speaking against theories of conceptual change that imply coherent structures of conceptual understanding.

top

Exploring Classroom Assessment Practices

Christine Harrison
King's College London

The KOSAP (King's-Oxfordshire-Summative-Assessment-Project) is a developmental-research project that has worked with a small group of teachers who are well-versed in formative assessment practices, and has helped these teachers of English and mathematics explore their classroom assessment practices. While the teachers' desire and intention in their instruction is to use assessment practices to drive learning, they also needed at times to take snap-shots of where a child is in their learning and consider what progress the child is making. This information can then be utilized for a variety of summative purposes. This study has looked at how the teachers of English and mathematics come to view their summative practices. It outlines some of the changes that they attempted, the dilemmas they encountered and the teacher learning that took place, while trying to provide a robust summative system that still allowed their formative practices to work effectively.

top