This collaborative research project implements a comprehensive, multidisciplinary approach to the design and validation of concept inventories (CIs). The purpose is to braoden and enhance the effective use of concept inventories in STEM education for formative and summative assessment. To accomplish this goal the project will focus on a specific concept inventory, the Thermal Transport Concept Inventory (TTCI). The data will be collected from the varied undergarduate student samples. The research would develop an explicit set of facets of understanding for diagnostic measurement and reporting, would gather multi-level empirical data of student performance on the TTCI (including student protocol and interview studies, and large scale data for psychometric modeling), would revise both the items and facets based upon these empirical data analyses, and would gather further data on the revised instrument. The data on TTCI will be analyzed in terms of the underlying conceptual knowledge and skills that it taps in critical areas of science and engineering. Additional empirical data on student reasoning and performance will be analyzed applying modern psychometric analysis techniques focused on diagnostic modeling. Research outcomes expected are: (1) an improved TTCI available through the web and accompanied by interpretive tools for use by faculty; (2) an in-depth, multi-level analysis of the assessment validity and diagnostic capacity of the TTCI; (3) a comprehensive inventory of possible formative and summative uses of CIs, including what is needed by STEM practitioners to make good use of CIs; and (4) a research and development framework for designing, developing, evaluating, improving, and/or implementing CIs in other STEM areas.
In general, the study will contribute to the better understanding of student learning and more authentic diagnostic and formative assessment, leading to better learning and teaching of important concepts in STEM. The use of the TTCI will elicit student misconceptions in the thermal and transport engineering sciences such as fluid mechanics, heat transfer, and thermodynamics. The TTCI particularly focuses on concepts deemed important but difficult for undergraduate engineering students. Thus, the proposed research is likely to lead to expanded uses of the TTCI and other CIs for formative classroom purposes as well as for a variety of summative purposes. This project are both an expansion of the psychometric modeling and methods for applying diagnostic models to CIs, and an expanded assessment validity in the area of measurement and assessment of key concepts in science and engineering.
This Collaborative Project involved the University of Illinois at Chicago (Lou DiBello and Jim Pellegrino co-PIs) NSF Grant # 0918552; the Colorado School of Mines (Ron Miller PI) NSF Grant # 0918436; and Purdue University (Ruth Streveler PI) NSF Grant # 0918531. Goal. The overarching goal was to demonstrate a comprehensive, multidisciplinary approach to the design and validation of concept inventories (CIs) capable of enhancing their effectiveness and impact and transforming their use in STEM classrooms. Two specific CIs, the Thermal and Transport Concept Inventory (TTCI) (Streveler, Olds, & Miller, 2003), and the Concept Assessment Tool for Statics (CATS) (Steif & Dantzler, 2004) were investigated, applying expertise from the multiple disciplines of STEM content knowledge, STEM teaching and learning, cognitive science, educational measurement, and psychometric modeling. Primary research questions: --What do the TTCI and CATS really measure relative to their conceptual foundations, their assessment claims and their intended uses? --Do TTCI and CATS items and distractors reliably diagnose conceptual understanding and robust misconceptions? Can new items and distractors improve the diagnostic functioning and utility of the TTCI? --How should instructors think about the use of CIs for instruction? How do CI concepts correspond to the standard topics taught in a typical STEM course? How can CIs be used formatively in classrooms? What impact do such formative uses have on student learning and performance? Significant Accomplishments --Multiple quantitative and qualitative analyses of both the CATS and TTCI inventories furthered our understanding of how to use and interpret students’ scores on these inventories. --The Evidence Centered Design (ECD) procedure was used to clarify the domain concepts measured by the TTCI, and to write new items to expand and improve the TTCI. --Research outcomes were extensively disseminated through journal submissions, national and international conferences, and book chapters. Key Outcomes and Other Achievements: A new version of the TTCI was developed iteratively, using the ECD approach and incorporating findings from think-aloud studies and online TTCI data from university students. A framework was developed and documented that incorporates both qualitative and quantitative CI data to investigate the validity of CI claims relative to intended uses. Impacts: Engineering Education: CIs are increasingly popular in STEM education settings and yet relatively little is known about their validity and instructional utility. As part of an ongoing discussion among engineering educators about extending CI uses beyond the common pre-posttest comparisons, our project outcomes provide a comprehensive theoretical and analytical framework for analyzing multiple aspects of the validity of CIs. Our approach is grounded in a guiding validity perspective that is directly applicable to typical CI uses and that addresses issues of expanded diagnostic reporting for formative classroom uses. Educational Assessment: Recent developments in educational assessment stress the relationships among curriculum, instruction and assessments. This project extends classical and current thinking about the validity of large-scale assessments to classroom assessments that can directly inform teaching and learning. Our theoretical framework can assist STEM practitioners at all levels including instructors, departments, and developers of both curricula and assessments to improve their understanding of what makes quality assessment and support instructors use of assessments as part of their instructional practice. Educational Measurement and Psychometrics: Research outcomes demonstrate how psychometrics contributes to quality STEM assessment. A validity perspective and related multidimensional diagnostic framework was developed to guide evaluation and improvement of CIs. Learning Sciences: Assessment forms a core of the sciences of learning within disciplines, and an improved framework for classroom assessment validity contributes to that core. Other Disciplines: This work has implications for other areas of practice and research that depend on or incorporate assessments such as K-12 curriculum development. Typical CI uses of multiple choice questions to test for deeper conceptual understanding and for misconceptions could be extended to assessments in other disciplines. Future CI-type measures of conceptual understanding of important and difficult concepts and diagnosis of key misconceptions could be used to improve the success rate of students in foundational courses in STEM areas. This could allow US institutions to graduate engineers with a deep understanding of disciplinary concepts. Impact on society: The work of this project may help improve education and preparation of scientists and engineers to meet the challenges of a technology and knowledge-based economy. The use of instruments like the TTCI, especially with the benefit of principled design approaches such as ECD, and suitably vetted for aspects of validity by both qualitative and quantitative empirical methods, could contribute to the US producing more engineering graduates who can productively use key foundational concepts in engineering and science disciplines. That could increase the chances of building a diverse domestic pool of engineers able to tackle the Grand Challenges of engineering.
