This collaborative research project implements a comprehensive, multidisciplinary approach to the design and validation of concept inventories (CIs). The purpose is to braoden and enhance the effective use of concept inventories in STEM education for formative and summative assessment. To accomplish this goal the project will focus on a specific concept inventory, the Thermal Transport Concept Inventory (TTCI). The data will be collected from the varied undergarduate student samples. The research would develop an explicit set of facets of understanding for diagnostic measurement and reporting, would gather multi-level empirical data of student performance on the TTCI (including student protocol and interview studies, and large scale data for psychometric modeling), would revise both the items and facets based upon these empirical data analyses, and would gather further data on the revised instrument. The data on TTCI will be analyzed in terms of the underlying conceptual knowledge and skills that it taps in critical areas of science and engineering. Additional empirical data on student reasoning and performance will be analyzed applying modern psychometric analysis techniques focused on diagnostic modeling. Research outcomes expected are: (1) an improved TTCI available through the web and accompanied by interpretive tools for use by faculty; (2) an in-depth, multi-level analysis of the assessment validity and diagnostic capacity of the TTCI; (3) a comprehensive inventory of possible formative and summative uses of CIs, including what is needed by STEM practitioners to make good use of CIs; and (4) a research and development framework for designing, developing, evaluating, improving, and/or implementing CIs in other STEM areas.
In general, the study will contribute to the better understanding of student learning and more authentic diagnostic and formative assessment, leading to better learning and teaching of important concepts in STEM. The use of the TTCI will elicit student misconceptions in the thermal and transport engineering sciences such as fluid mechanics, heat transfer, and thermodynamics. The TTCI particularly focuses on concepts deemed important but difficult for undergraduate engineering students. Thus, the proposed research is likely to lead to expanded uses of the TTCI and other CIs for formative classroom purposes as well as for a variety of summative purposes. This project are both an expansion of the psychometric modeling and methods for applying diagnostic models to CIs, and an expanded assessment validity in the area of measurement and assessment of key concepts in science and engineering.
This was a Collaborative Project involving University of Illinois at Chicago (UIC - Lou DiBello and Jim Pellegrino co-PIs) NSF Project # 0918552; Colorado School of Mines (CSM – Ron Miller PI) NSF Project # 0918436; and Purdue University (Ruth Streveler PI) NSF Project # 0918531. Goal. The overarching goal was to implement a comprehensive, multidisciplinary approach to the design and validation of concept inventories (CIs) capable of transforming their use in STEM classrooms and enhancing their effectiveness and impact. Multiple disciplines of STEM content knowledge, STEM teaching and learning, cognitive science, educational measurement, and psychometric modeling, were applied to two specific CIs, the Thermal and Transport Concept Inventory (TTCI) (Streveler, Olds, & Miller, 2003), and the Concept Assessment Tool for Statics (CATS) (Steif & Dantzler, 2004). Primary research questions: --What do the TTCI and CATS really measure relative to their conceptual foundations, their measurement claims and their intended uses? --Do TTCI and CATS items and distractors reliably diagnose conceptual understanding and robust misconceptions? Can new items and distractors improve the diagnostic functioning and utility of the TTCI? --How should instructors think about CIs for instruction? How do CI concepts correspond to the standard topics taught in a typical STEM course? How can CIs be used formatively in classrooms? What impact do such formative uses have on student learning and performance? Significant Accomplishments --Multiple quantitative and qualitative analyses were completed of both the CATS and TTCI inventories. --A new approach applied the Evidence Centered Design (ECD) procedure to expand and measurably improve the existing TTCI. --Research outcomes were extensively disseminated. Key Outcomes and Other Achievements: Using ECD, a new version of the TTCI was developed and tested with think-aloud studies and a small sample of on-line performance data. A framework was demonstrated for both qualitative and quantitative analyses of CI data to investigate the validity of CI claims. Impacts: Engineering Education: CIs are increasingly popular in STEM education settings and yet relatively little has been known about their validity and instructional utility. As part of an ongoing discussion among engineering educators about possibilities for extending CI uses beyond the common pre-posttest comparisons for evaluating engineering curriculum and instruction, project outcomes and products provide a comprehensive theoretical and analytical framework for analyzing multiple aspects of the validity of CIs. Our approach includes an analytical framework that is grounded in a guiding validity perspective directly applicable to typical CI uses and to expanded diagnostic reporting for formative classroom uses. Educational Assessment: Recent developments in educational assessment stress the relationships among curriculum, instruction and assessments to directly inform classroom teaching and learning. This project extends classical and current thinking about assessment validity from large scale assessments to assessments used in authentic classrooms. The theoretical framework applied and documented here has the potential to assist STEM practitioners at all levels including instructors, departments, and curriculum developers to improve their understanding of what makes quality assessment and effect assessment enactment by instructors. Educational Measurement and Psychometrics: We demonstrate how psychometrics contributes to quality STEM assessment. We developed multidimensional, diagnostic frameworks to guide evaluation and improvement of CIs. Learning Sciences: Assessment forms a central core of the sciences of learning within disciplines, and an improved framework for classroom assessment validity contributes to that core. Other Disciplines: The work performed under this grant has implications for other areas of practice and research that depend on or incorporate assessments such as K12 curriculum development. Typical CI uses of multiple choice questions to test for deeper conceptual understanding and for misconceptions could be extended to other types of assessment. In the future the use CI-type measures of conceptual understanding of important and difficult concepts and key misconceptions in STEM areas could be used to improve the success rate of students in central, foundational courses. And that could lead to US institutions graduating more engineers who have a deeper understanding of underlying concepts. Impact on society: The work of this project may impact the continuing national concern about improving education and preparation of scientists and engineers capable of meeting the challenges of a high technology dominated future. The use of instruments like the TTCI, especially with the benefit of principled design approaches such as ECD and suitably vetted for aspects of validity by both qualitative and quantitative empirical methods could contribute to the US producing more engineering graduates who can demonstrate a deeper understanding of key foundational concepts in critical engineering and science disciplines, and that could increase the chances of building a diverse domestic pool of engineers able to tackle the Grand Challenges of engineering.
