The Common Core Learning Standards (CCLS), adopted by 44 states and the District of Columbia, define the skills a student should demonstrate by the end of each grade. One key skill emphasized by CCLS is reading ability, which is the precursor for learning in all content areas. In New York State, students in grades 3-8 take an English Language Arts (ELA) test each spring to measure their CCLS achievement in reading. An ELA test contains both multiple choice questions and open-ended questions based on short text passages; to do well, students should be able to read a passage closely for textual evidence and draw logical inferences from it. To report the results, the number of correct student responses is converted into a scale score; this in turn is divided into four performance levels: NYS Level 1 for well below proficient, NYS Level 2 for partially proficient, NYS Level 3 for proficient, and NYS Level 4 for exceptional in grade-level standards. Schools arrange academic intervention services for students whose performance level is either NYS Level 1 or NYS Level 2. To drive change in students who are at risk for not meeting academic expectations, the Response-to-Intervention model aims to deliver instructions as a function of these assessment outcomes. But the PIs argue that a single performance score as the assessment outcome is often insufficient for identifying underlying learning problems, especially in reading comprehension. In this exploratory project they will focus on discovering error patterns in assessment outcomes at the lexical level, in the expectation these will ultimately lead to improved understanding of how the raw data from a pool of underperforming text-based analytic reading assessments can be transformed into an informative and understandable structure for delivery of effective reading comprehension interventions. Project outcomes will complement the current scoring system by supporting diagnosis at an individual level, and by facilitating grouping of students with similar reading disabilities in the same intervention group in order to optimize school teaching resources. The approach will also support a data-driven instruction framework by maximizing the information gain from each test, which can result in fewer tests taken and more hours for teaching per school year.
This is an interdisciplinary collaboration between a computer scientist (Tsai) and an expert in literacy education (Zakierski). PI Tsai will be responsible for computer algorithm development and data analysis, whereas PI Zakierski will be in charge of data collection and evaluation of the proposed approach based on findings in literacy and pedagogy. The team will build a database containing words with lexical properties from literature for children up to grade 3, assessment materials from NYS ELAs, and intervention records. They will annually collect ELA assessment materials from a pool of approximately 120 third grade students with performance at NYS Level 2 or below, for both research development and evaluation. They will develop a computer-aided intervention system that performs data-mining on underperforming individual ELA assessment materials to discover error patterns, which should assist the teacher in identifying a student's underlying reading comprehension problems in order to prepare a more effective instruction plan. And they will evaluate the performance by doing both formative and summative assessments, the former to consist of questionnaires for teachers and mock ELA tests for students taken during the period of intervention, and the latter being the real ELA tests in April following the intervention. From the computer science perspective, the main challenge is the small size of the dataset. The PIs will develop new techniques that are domain-knowledge driven for performing meaningful analysis to discover error patterns in such situations; if successful, the approach will open the door to broad research opportunities in other cases where "small data" is easier to come by. In addition, the exploration of data mining on literacy education itself will constitute a unique contribution, since the marriage of the two fields has not yet received much attention from the research community and there are many interesting questions waiting to be addressed using computational approaches.
