The investigators propose to develop a new method of STEM assessment--epistemic network analysis (ENA)--that focuses not only on whether students master specific scientific facts, math skills, or engineering concepts, but also on whether and how students link the skills, knowledge, identity, values, and epistemology of a STEM practice into a coherent way of thinking about complex STEM problems. The approach links assessment data to modeling using tools similar to social network analysis.
The investigators describe ENA as potentially transformational because it is in its early stages, and involves a radically different and interdisciplinary approach to the problems of STEM assessment.
The researchers will use data collected from two epistemic games, Digital Zoo (engineering) and Urban Science (urban planning), both funded by a previous NSF award. For each game, the investigators have data from 12 players in Grades 8 and 9 from the Madison, Wisconsin area. These data will serve as proof of concept for the development of the assessment prototype and for the development of the statistical methods that are necessary to make a social networking component possible. Two outcomes of this research can be expected: (1) a proof of concept prototype method for assessing a network of knowledge that students bring to a given problem, and (2) the methods and measurement development that makes such assessment systems possible and potentially useful in other contexts.
This project was funded by the EArly-concept Grants for Exploratory Research or EAGER. In this project, we developed new statistical tools to measure STEM learning in the 21st century. Today, work that requires only basic skills flows overseas where labor is cheaper, and complex and meaningful STEM thinking means linking skills and knowledge in the context of real-world problems and situations. Problem solving in real STEM practices is characterized by knowledge and skills, to be sure, but also by the way those skills are connected to each other, and to the values and ways of making decisions in STEM fields. In this project we developed of a new method of STEM assessment—called Epistemic Network Analysis (ENA) that focuses not on whether students master specific scientific facts, math skills, or engineering concepts, but on whether and how students link the skills, knowledge, identity, values, and epistemology of a STEM practice into a coherent way of thinking about complex STEM problems. In this project we linked prior work on an innovative theory of STEM thinking with the mathematical and conceptual tools of social network analysis to create a new conceptual and statistical approach to the measurement of STEM thinking and STEM learning. We developed ENA as an assessment tool in the context of a particular theory of learning (the epistemic frame hypothesis) that applies to a specific kind of STEM learning computer game (epistemic games). However, we want to emphasize that ENA is an approach to assessment that can be used in any situation of complex STEM thinking where the connections between things being learned are more important than isolated pieces themselves. The cognitive model developed in this project will help educators in a wide variety of fields analyze and improve STEM education by providing a means to dynamically assess the development of complex STEM thinking. The tools we have developed contribute to the field of network analysis, and our development process will enhanced the skills and career trajectories of five young investigators. During the project we completed several studies to validate the use of ENA techniques in discourse during online learning. For example, one study of a game designed for first year engineering students showed that ENA can identify students who made multiple connections between engineering design and other engineering skills and knowledge. Moreover, those students were more likely to view engineering positively at the end of the game. ENA is currently being used by researchers at the University of Wisconsin, the University of Missouri, the University of Massachusetts, the University of Memphis, and the University of Georgia. Based on the success of the current research, a follow-up project funded by the National Science Foundation is building a user-friendly version of ENA and a user-community for the tool.
