Education Development Center, Inc. has proposed a three-year REESE Frontier Research empirical project (Measurement, Modeling, and Methods) to develop a training manual and facilitate a national field test of the Inquiry Science Instruction Observation Protocol (ISIOP). The ISIOP was developed with NSF funding. The proposed project will develop observation protocol and scoring rubrics that will be tested for use by science education evaluators. The main objective of the project is to improve the measurement technology and tools for capturing variation in scientific inquiry instruction via classroom observation.
The field test will involve 40 evaluation professionals (20 teams) at different schools across the country. They will generate an observation data set of 80 paired observations. The field test will use a stratified sampling strategy to maximize the representation of schools, classrooms and students across the nation in the study. This data set will be used to demonstrate a scientifically sound validity argument supporting the intended interpretation of the scoring rubrics for measuring the nature and extent of scientific inquiry instructional practices exhibited in middle-grades science classroom teaching.
The products generated from the project include (1) an observation protocol and scoring rubrics that have been tested for usability and utility across a range of science education program evaluators in various settings; (2) a user?s manual that clearly describes the development process, the derivation of the evidence of reliability and validity, procedural information for using and scoring the protocol, guidelines for establishing inter-rater reliability using the included videotaped examples of teaching, and suggestions about how to coordinate the use of the ISIOP with other instruments to produce a comprehensive research or evaluation plan.
The major goal of the Inquiring into Science Instruction Observation Protocol (ISIOP) Field Test project was to improve the ability of science education researchers and evaluators to comprehensively and quantitatively capture elements of secondary science classroom instruction. The project built on prior efforts to develop constructs and items that were both useful for researchers and observable. The field test engaged researchers from around the country in training and using the tool in their own projects in order to ensure that the tool can capture science instructional practices across a range of settings. Using data collected from the field test, the team further refined the ISIOP materials and established evidence of the reliability and validity of the constructs and indicators. The products generated from the project include (1) an observation protocol and scoring rubrics that have been tested for usability and utility across a range of science education program evaluators in various settings; (2) a user’s manual that clearly describes the development process, the derivation of the evidence of reliability and validity, procedural information for using and scoring the protocol, guidelines for establishing inter-rater reliability using the video and transcript examples, and suggestions about how to analyze ISIOP data to produce a comprehensive research or evaluation plan; and (3) a project website that contains all ISIOP products. In science, it is advances in tools and instruments that often drive new conceptualizations of old problems leading to discoveries of new understanding. The necessity of rigorous evaluation instruments that can assist in the interpretation of how instructional practices relate to student achievement is particularly urgent for science education. With the emphasis on scientific practices in the Next Generation Science Standards, the pedagogical approaches to teach about and through scientific investigation will continue to play a significant role in science teaching and learning across all science disciplines. However, there are few rigorously tested tools for conducting classroom observations in science, and of these, none captures detailed and nuanced information about the specific components of instructional practice that facilitate scientific inquiry learning like the ISIOP does. Observation instruments typically receive little attention by researchers and evaluators who regularly develop and test the psychometric properties of research tools, even though classroom observation is often a key element in research and evaluation plans. This project provided a much needed tool, tested training procedures, and set of supportive methodological resources that will significantly add to the capacity of education researchers and evaluators of science programs to provide detailed, valid, and reliable information about the programs that they study.
