This collaborative project involving the University of Colorado at Denver and the University of Washington at Seattle in conjunction with Facet Innovations, will build a framework for addressing the use of contextualized items in the assessment of STEM learning. The primary goal is to systematically investigate the effects of characteristics of contextualized items on student performance to strengthen practices in science assessments, ensure fairness in science testing, and increase support for both assessment and instructional purposes. Test items with contexts are called contextualized items which include supplemental information that precedes or follows a test item question. Such information may include a description of a lab setup, a natural phenomenon, or a practical problem often depicted as a scenario, background, vignette, or cover story. The project findings will help to understand how students make sense of contextualized items focusing on complex scientific concepts that they usually encounter in science assessments. Currently, contextualized items are constructed from either conventional wisdom or non-contextualized item writing rules. Such items could mislead students to attend to irrelevant information or interfere with the targeted construct, and, therefore lead to inaccurate inferences about student learning. This project will develop a framework for developing items to help address this problem. Approximately, 70 classroom teachers and 4800 students, in secondary grades, will participate in the study.
The project will offer a theoretical articulation of the characteristics of contextualized items and empirically test the effects on student performance. It seeks to address four gaps in the literature on contextualized items: (1) insufficient knowledge about how to conceptualize construct-relevant contextualized items; (2) lack of research on contextualized items in science; (3) lack of research that systematically studies the characteristics of contexts and evaluates their effects on student performance; and (4) the need for studies that examine differential effects related to subgroups of students to gain a greater clarity of what types of contexts affect whom. The research design and data analyses will be guided by three research questions: (1) what are critical context characteristics that may affect student performance and should therefore be considered when developing science test items? (2) what are context characteristics associated with construct-relevant variance? (3) what context characteristics are associated with differential student performance patterns due to gender, ELL status, and socioeconomic-status variables?
The project will take place over three years through a two phase process. During Phase 1 the project will refine a proposed theoretical framework that will identify the item context characteristics and articulate the item development guidelines. During Phase 2, the goal is to apply the framework by selecting, revising, and developing science items with varying profiles of contexts; conduct field tests of the items; and perform a range of psychometric and statistical procedures with test scores, and qualitative analyses of students' cognitive interview responses and teacher interviews. Items resulting from this process will aim to evoke students' stored knowledge relevant to the content and/or process skills targeted. The project will involve a team of researchers specialized in assessment development and validation, science education, content knowledge, linguists, and expert classroom teachers. The item development approach and items generated from this project will have immediate implications for researchers and practitioners in science education nationally and internationally.
