The Measure STEM Caliper Development Initiative [Online]

Abstract

The 89 international STEM Learning Ecosystems (https://stemecosystems.org/) coordinate learning in Science, Technology, Engineering, and Mathematics via a wide range of educational programs also encompassing the arts, coding, social skills, literacy, and other fields. In 2015, educators, policymakers, funders, and other key stakeholders convinced of the value of cross-sector collaborations defined five pillars of ecosystem success: partnerships, measures, administrative systems, teaching and learning practices, and workforce development connections. In 2019, the ecosystem success construct was mapped in terms of the development of stakeholder participation.

Items were written to tap a sequence proceeding through phases of being Informed, Consulted, Involved, Collaborating, and Empowered (ICICE). Mapping this construct will support development of reports ecosystems can use to orient themselves in relation to where they have been, where they want to be, and where to go next on their success journey, as well as being able to see where they stand in relation to each other. Existing items from an earlier ecosystem assessment were revised and augmented, four parallel forms with common items were devised, and an online administration system was created in early 2020. Items were categorized by ICICE levels, and as addressing the conditions for success specified in one of the five pillars. Four forms each tapping all ICICE levels and success pillars were linked via 31 common items, with 63 or 64 items on each form in total.

Data collecting wrapped up on 8 June 2020 with 283 measured responses; by-item sample sizes range from 29 to 274. Overall measurement separation reliability was 0.94-0.96, and Cronbach's alpha (KR-20) was 0.96, with satisfactory model fit. Participant measures were highly correlated (most 0.90 and higher) across items located on different forms, associated with different success pillars, and across the five ICICE levels. Item calibrations were also highly correlated across participant subsamples (separated by gender, ethnicity, ecosystem, form, etc.). Though additional data will be needed for items with fewer than 150 respondents, the stage is set for deployment of an item bank supporting adaptive administration and/or equated forms.

The common metric for outcome assessment and improvement will be augmented by reports not only mapping the formative trajectory toward participant empowerment, but also by highlighting subgroup differential item functioning. The special utility of these reports will be realized in the way some demographic groups may tend to rate items associated with one or another pillar unexpectedly low or high. By-subgroup standardized residuals could be used to alert leaders to improvement opportunities contingent on special weaknesses. A gender or ethnic group might disagree in response to partnership items, for instance, which tend to garner highly agreeable responses. Failure to engage in partnership fundamentals is likely to negatively impact more difficult challenges to ecosystem success included in other pillars. Having a common language for comparing outcomes and learning within and across ecosystems may prove key to enhanced outcomes.

Jan Morrison is the Founder and Senior Partner of TIES – Teaching Institute for Excellence in STEM. She has worked with multiple foundations, the NSF, and the National Academy of Engineering in their STEM education initiatives, efforts that included being a reviewer of the Next Generation Science Standards. Jan has served as an advisor with the White House and Department of Education for the past four administrations. Currently, Jan and TIES lead the National STEM Funders Network, a collaboration of more than twenty-eight STEM funders in Canada, Israel, Kenya, Mexico, and the USA.
William P. Fisher, Jr. is a Senior Scientist at the Research Institute of Sweden in Gothenburg, and a Research Associate with the BEAR Center in the Graduate School of Education at the University of California, Berkeley. He consults independently via Living Capital Metrics LLC, and as a partner with INNORBIS.com, on projects in education, health care, and sustainable development globally.

William P. Fisher, Jr. is a Senior Scientist at the Research Institute of Sweden in Gothenburg, and a Research Associate with the BEAR Center in the Graduate School of Education at the University of California, Berkeley. He consults independently via Living Capital Metrics LLC, and as a partner with INNORBIS.com, on projects in education, health care, and sustainable development globally.

Date: 
Tuesday, September 1, 2020 - 2:00pm
Building: 
Online session
Room: 
Zoom meeting
AttachmentSize
PDF icon Presentation slides1.29 MB