The primary purpose of the research is to investigate the extent to which a performance standard can be set for an assessment instrument using a diagnostic cognitive model (DCM). The proposed Pathway research project is to use of an emerging psychometric methodology to investigate different approaches to establish a performance standard on an instrument for algebra readiness. The proposed research will use TIMSS algebra items and involve a secondary analysis of the 2003 TIMSS U.S. Grade 8 data. The research will focus on the skills and knowledge required for algebra readiness.
This research project involves both quantitative and qualitative studies. The qualitative study provides strong evidence to support the findings of the quantitative investigations. The use of Diagnostic Cognitive Models to inform standard setting is exciting because it offers both a clearly defined methodology, but also because it promises diagnostic information regarding the strengths and weaknesses of each student so that teachers can design a specific remediation to meet each student's needs.
To date there is no research that has examined the topic of setting performance standards using a diagnostic cognitive model (DCM). This research project offers a promising way forward for improving processes of standard setting, such that they rely less on the inscrutable judgments of panel members, and relatively more on factors that can be easily measured and related to test outcomes, such as DCM. The approach is potentially transformative not only for its potential to change how standard setting is conducted, but also for its potential to change the relationship between standards and teachers' ability to intervene effectively in the classroom, by targeting aspects of the DCM that best predict algebra readiness. The findings of the research project have broader implications for both instruction and policy. This research project cannot only advance our understanding of related issues in educational measurement and mathematics education, but also establish a model for two camps to communicate and work together for value-added research endeavor.
This research project resulted in the development of a standard setting method designed to be used with tests that fit a cognitive diagnostic model (CDM). Rather than a single test score, CDMs provide diagnostic skill profiles of student strengths and weaknesses. These models have become the focus of considerable interest in light of the recent push to have K-12 students ready for college and careers. In recent psychometric research, CDMs have been used successfully to (a) identify the cognitive skills that underlie learning in specific subject areas and (b) develop predictive models that can inform the measurement of those skills. A standard setting method to determine a score that meets a certain performance standard (e.g. proficient) on a CDM-based test, however, had not been developed previously. In this study, we developed the Diagnostic Profiles (DP) standard setting method to determine a cut score on a test that measures readiness to take high school Algebra I. The DP method focuses on profiles of skills that are required to meet the performance standard, in this case algebra readiness. The algebra readiness test was created using items and data from the 2003 grade 8 TIMSS, and the C-RUM model was used to calibrate item characteristics. The key task for standard setting panelists is to judge whether or not each skill profile indicated that the performance standard has been reached, which meant, in this case, whether or not specific combinations of skills, such as skill in number operations, fractions, decimals, and/or patterns, meant that a student was ready to take Algebra I. Our results showed that the DP method resulted in greater consensus in determining a cut score on the test than the traditional Angoff standard setting method. This is an important finding for a standard setting method because it means that there was greater agreement among subject matter experts and therefore a more defensible cut score. Additionally, the DP method was easy to implement, well understood by the panelists, and took approximately the same amount of time as the Angoff method. In addition to these findings, our study provided insight into the specific subject matter at hand. Previously, there had surprisingly little research on what skills students need to have to be ready for Algebra I, nor was there little guidance on how to determine if students were ready. Also, the DP method could be readily applied to any subject area in K-12 and higher education in which the subject matter is considered multifaceted and decisions need to be made about students on the basis of test scores. Finally, we presented our results at the 2013 Annual Meeting of the National Council on Measurement (NCME) and have submitted a manuscript from this study to a peer-reviewed journal. The conference paper and some of the panel meeting materials have been made available publicly on a website (http://filebox.vt.edu/users/gskaggs/nsf/nsf-home.html).
