The Center for Elementary Mathematics and Science Education (CEMSE) at the University of Chicago will conduct a full-scale 2-year study to demonstrate and further develop and test the theory upon which the suite of Fidelity of Implementation (FOI) instruments is based by exploring the practical use of these tools. This mixed-method study grows out of CEMSE's past NSF-funded studies focusing on the development of innovative tools for evaluating the implementation of reform-based K-8 mathematics and science instructional materials programs. The goals of this project are: 1) operationalizing the process of instrument adaptation to a locally-developed science program; 2) further refining and developing the FOI instruments through field testing; 3) exploring methodologies for instrument use in evaluating the effect of program implementation on student outcomes; and 4) sharing, disseminating, and communicating knowledge about measurement of implementation of science and mathematics instructional materials through both online and site-based collaborative discussions with practitioners, evaluators and researchers.
This project will involve three mid-sized school districts located in Champaign, IL, Naperville, IL, and Newton, MA, that use Everyday Mathematics (EM) and a reform-based science curriculum. Within the participating districts, the investigators will use a purposeful sampling strategy to select 17 to 20 elementary schools that best represent the district in terms of school size, student demographics, and/or student achievement. In year 1, the existing FOI instruments for EM will be administered to all K-5 teachers and school leaders in the participating schools. The instruments will also be adapted, field tested, and revised for measurement of implementation of the science curriculum. In Year 2, a revised set of FOI instruments for science will be administered to participating teachers and school leaders. Data analysis will include exploring the status and types of EM and science materials implementation. Standardized state test scores and grades will be collected within all participating classrooms to link status and types of instructional materials use to student achievement.
Underlying this study is a belief that efforts to improve evaluation of the implementation of educational innovations are only truly effective when evaluation tools and data are usable and useful to the audiences for which they are intended. There exists a continuing need in the field for research instruments that describe, measure, and explain specific aspects of mathematics and science instructional materials implementation in order to understand their effectiveness and to inform ongoing improvement efforts in STEM education. The need for more rigorous measurement of mathematics and science instructional materials is also particularly relevant to school and district leaders seeking to bring effective programs to their schools. These FOI instruments will enable stakeholders to obtain data at a level of specificity that can inform ongoing improvements and identify elements of their curricula that are leading to the desired practices and outcomes. This study will ultimately contribute to the broader field of STEM education by furthering an understanding of the ways in which an innovative approach to FOI measurement may be applied in school settings while creating and sharing knowledge about the evaluation of implementation of STEM instructional materials.
Normal 0 false false false EN-US JA X-NONE Intellectual merit: This project will contribute to the development of knowledge in several areas. First, findings from this work will create a solid foundation for clearly articulating how instructional programs are implemented, why they are implemented in certain ways, and how that implementation affects outcomes. The Outlier Research & Evaluation team at the University of Chicago collected data from school/district leaders and teachers in a school district to determine the most essential elements of their mathematics and science programs. The team also collected data on the "factors" (the contexts and conditions) to determine how they affected their implementation of those elements. Building from earlier work describing "innovations" in education, Outlier worked with the district leadership and curriculum coordinators for math and science to describe their district’s "models" for mathematics and their "locally developed" science curriculums. The team organized those components into two groups – structures and interactions – and placed them within an implementation framework that was developed for reform based mathematics and science innovations. We learned that our framework was consistent with the local science curriculum, placing two new components into our existing framework. Clearly articulating the components of the mathematics and science models helped us to measure the implementation status of such components and to investigate which components are most important for accomplishing desired student outcomes. Second, we developed and validated instruments (e.g., questionnaires, logs, observations, interviews) that measured program implementation and account for the effect of influential factors related to program outcomes. As such, we have addressed a need in the field of evaluation and research for more tools that describe, measure, and explain specific aspects of mathematics and science instructional materials implementation. This provides evaluators and researchers with the necessary tool to rigorously describe and understand innovations’ effectiveness and to inform ongoing improvements in both curriculum designs. These instruments together with the Implementation and Factor Frameworks are able to provide evaluators with a widely applicable, comprehensive approach for understanding the implementation process within and across a variety of innovations. This helps to compare findings across programs or accumulate knowledge on enacted program elements within evaluations of reform-based STEM interventions. Broader impacts: This project has led to the articulation of Outlier’s implementation approach to evaluation. In particular, we offer evaluators and program implementers a systematic approach to evaluation that is built on well-established theory about measurement and implementation. Specifically, our approach will assist evaluators and practitioners to understand the type of implementation that is occurring and to assess how implementation has occurred. Our approach to evaluation provides a means to measure implementation in a wide range of settings through the identification of context-specific components and factors. This approach can be widely applied not only to mathematics and science instructional programs, but to other disciplines as well. This project has also benefited researchers and evaluator, in that we provide them practical guidelines for working with practitioners. In particular, practitioners in STEM program evaluation and research have little incentive to participate in time and resource-intensive data collection unless there are meaningful benefits. Evaluators and researchers have to develop studies that will be of interest and address the practical needs of stakeholders. This was accomplished by building rapport with the study participants, understanding what they want to learn about their intervention, answering questions they have, and providing findings and products that are useful to them.
