The goal of the project is to understand the multidisciplinary and interdisciplinary (MIDR) nature of education and learning research and how it has developed over time, with a focus on cognitive science, an area of particular importance to education research. Investigators from Search Technology Inc will study proposals funded by two NSF education research programs (ROLE and REESE). Using bibliometric methods they devised, they will examine the references cited in the proposals to characterize the MIDR nature of the projects. They will also analyze the publications produced by the projects and track the dissemination of that research knowledge via the papers that cite those publications. The investigators will focus additionally on the work of key cognitive scientists and highly-cited ROLE/REESE-derived publications. Using social network analysis and science mapping, the investigators will track STEM community engagement with this educational research.
The project seeks to advance measurement, analysis, and visualization tools to help quantify science and science policy. In particular, these offer empirical evidence on much-touted, but little measured, MIDR. The results should have important implications for STEM education researchers who wish to understand trends in their fields, for cognitive science and education research communities that have traditionally not engaged one another's literatures, and for science policy analysts and program developers.
This EAGER project set out to learn about cross-disciplinary research knowledge transfers relating to Science, Technology, Engineering and Math (STEM) Education. It focused on NSF Research and Evaluation on Education in Science and Engineering ("REESE") Program awards, their outputs (publications), and outcomes (citations to those publications). Much of the project work has been on gathering and treating data relating to a set of REESE awards. Early work included development of text analysis methods to extract information from REESE proposals, particularly concerning the sources that they referenced. This is challenging in that formatting is not consistent and the text is in pdf documents. These data offer potential value in enabling study of the sources of knowledge upon which REESE research draws. We are particularly interested in learning about which disciplines contribute to this STEM Education research. That information can then be linked to their research emphases and the disciplinary distribution of outputs. For instance, do researchers who draw heavily upon scientific research [as indexed in the Web of Science (WOS), the world’s leading database of papers published in some 12,000 leading journals] go on to publish findings in WOS journals? Is their research more influential (as indicated by citation intensity)? And what fields cite their research (an indicator of its influence)? Another line of inquiry explored information resources that collect & index research outputs to track REESE outputs. We examined Scopus and WorldCat (particularly to get at books) data, but determined to concentrate on WOS in the current studies. A second set of data challenges concerned identifying research outputs (papers) associated with REESE support. While WOS now provides a field of information on "funding acknowledgements," this proved woefully inadequate in identifying papers supported by this NSF program. We were able to identify >1000 papers (in various forms), as reported by PIs, attributed to the 183 REESE awards funded in 2006, 2008, and 2010, that we are studying. Extensive checking of those indicated papers located 274 journal articles and conference papers indexed by WOS, deriving from this NSF funding. A third set of data challenges entailed tracking down the papers that reference those 274 REESE-derived publications. Our colleague devised a macro (software routine) to facilitate this process. We were thereby able to retrieve, check, and clean 2064 WOS abstract records of these citing papers. These enable tracking research knowledge diffusion from the REESE support, with particular interest in the disciplines involved. We are particularly studying the diffusion of research knowledge across fields. We began by using the WOS subject categories. Those did not do a good job of distinguishing Education-related research concentrations. We therefore worked closely with NSF colleagues to generate "Tailored Web of Science Categories" (TWCs) that more validly group research contributions. This work had to be revised as WOS changed its subject categories in 2012. We are now well-positioned for further analyses of cross-disciplinary research knowledge diffusion. A journal article presented the main previous research findings in analyzing ROLE and REESE awards, outputs, and impacts: Porter, A.L., Schoeneck, D.J., Roessner, D., and Garner, J. (2010). Practical research proposal and publication profiling, Research Evaluation, 19(1), 29-44. Our 2013 results were shared via three conference presentations: Porter, A.L., Schoeneck, D.J., Solomon, G., Lakhani, H., and Dietz, J. (2013), Measuring and mapping interdisciplinarity: Research & evaluation on education in science & engineering ("REESE") and STEM, American Education Research Association Annual Meeting, San Francisco (Apr.). Porter, A.L., Schoenek, D.J., O’Brien, J.J., Solomon, G., Lakhani, H., and Dietz, J.S., Connections: Patterns of links between STEM education research and other disciplines, Atlanta Conference on Science and Innovation Policy, Atlanta, 2013 (Sep.). Porter, A.L., Schoeneck,D.J., Solomon, G., Lakhani, H., Dietz, J., and O’Brien, J., Assessing Science, Technology, Engineering and Mathematics (STEM) Education (ED) research, Annual Conference of the American Evaluation Association, Washington, DC, 2013 (Oct.). Highlights include: Papers citing the REESE-supported papers are distributed widely across the disciplines. Figure 1 shows a science overlay map. The black background of 224 nodes represent the WOS categories. Their location and interconnections reflect how frequently they cite each other, based on the entire 2010 WOS Science and Social Science Citation Indexes. The colored nodes overlaid on them show the concentrations of citations to the REESE-derived research (274 papers). Larger nodes indicate more cites. REESE research staunchly engages the Cognitive Sciences. Figure 2 shows the top TWCs of the 274 publications and of the 2064 papers that cite them. Note the remarkable engagement of various Psychology subfields and other Cognitive Science domains. There are some interesting differences in the concentrations of REESE-derived publications and of the citations to them. Note the relatively higher citation densities for Neurosciences and Social Psychology. We aspire to continue these analyses to gain understanding of the "connections" among STEM Education, Cognitive Sciences, and related fields of research.
