This project addresses the need for effective approaches for assessing science knowledge among English language learners (ELLs) who are tested in English. The project will investigate the effectiveness of vignette illustrations in test items as a form of testing accommodation for ELLs. In a vignette-illustrated item, the illustration is not necessary to understand the text of the item and the text of the item makes no explicit reference to the illustration. Since they can be added to the text of existing items, vignette illustrations are less likely than other forms of testing accommodation to alter the construct that items are intended to measure.
The project will investigate how vignette illustrations minimize the impact of limited English proficiency on student performance in science tests. Also, it will attempt to identify basic principles for designing illustrated test items. ELL and non-ELL middle-school students will be given items of three types (vignette-illustrated items whose illustrations are designed systematically, vignette-illustrated items whose illustrations are created arbitrarily, and items without illustrations) at two levels of distance to the enacted curriculum (close and distal). Diverse forms of analysis (including statistical analyses and verbal protocols) will determine: (1) whether and how ELL and non-ELL students differ significantly on the ways in which they use vignettes to make sense of items, (2) whether the use of vignettes reduces test score differences due to language factors between ELL and non-ELL students, and (3) whether the level of distance of the items moderates the effectiveness of vignette-illustrated items.
While illustrations are frequently used in test items, there is limited guidance in the assessment development literature on how to approach illustrations. Furthermore, the value of illustrations as a resource for ensuring that ELL students understand what a given item is about and what the item asks them to do has not been systematically investigated. Semiotics, cognitive psychology, linguistics, and social cultural theory are brought together to develop systematic procedures for developing illustrations as visual supports in tests.
The project will contribute to enhanced practice in both classroom and large-scale assessment primarily of ELL students. Vignette illustrations have the potential to become a low-cost, easy-to-implement form of testing accommodation for ELLs. The set of principles for the proper design and use of vignette illustrations as a form of testing accommodation for ELLs could be used by states, teachers and the testing industry.
