This project is exploring the existence of indicators that are evidence of improved student learning resulting from the National STEM Digital Library (NSDL). In particular it is studying whether focusing on a particular Library collection and using multiple methods to measure impact will provide insight into improving the design of the Library. Evidence of student impact is difficult to develop. This was the conclusion of a January 2009 "Science" article, "NSF Rethinks Its Digital Library." This article stated that there is no evidence to date that the NSDL has improved student learning. Despite this conclusion, members of this project's study team accumulated anecdotal evidence of positive impact on student learning while working with the NSDL project, "Where Have We Come From and Where Are We Going? Learning the Lessons and Disseminating Exemplary Practices from the Projects of the NSDL."
This study is examining a coherent section of the NSDL, the "Premier Award for Excellence in Engineering Education Courseware," sponsored by the NSDL Engineering Pathway. It is employing the case study research method. The rationale for this approach is to focus on a collection that has a high possibility of providing data on student impact and, further, that may provide some useful insight into effects on student learning. Over the last 12 years, the Premier Award competition, and its precursor (NEEDS), reviewed 100 courseware submissions and selected 21 award winning courseware creations and eight finalist candidates. The award winning courseware has been disseminated widely. Premier Award winning materials have been the most frequently downloaded materials on the NEEDS/Engineering Pathway website. Over 20,000 Premier Award CDs have been distributed and workshops and seminars for engineering educators have been held. Hence the case study of the Premier Award has the potential of discovering a large user population. In general, identifying faculty users has been a significant challenge for NSDL research. The Premier Award provides an opportunity to reach a population of users and learn from them. The research is addressing the following questions: 1. How has receipt of the Premier Award impacted the awardees' career paths and why? 2. How has the quality of the courseware submitted changed over time? 3. How has the award winning courseware affected or impacted student learning? The project is studying the population of faculty that is using courseware available through NSDL (via the NEEDS/Engineering Pathway). The findings from this research will lay the groundwork for future in-depth, longitudinal studies of impact on teaching and learning that reach across the sciences, technology, engineering and mathematics.
The purpose of 'Learning from the Best' project was to study the impact of the Premier Award for Engineering Excellence in Engineering Education Courseware, a 15-year effort to disseminate high quality digital learning resources available through the NSDL. The Premier Award (PA) was developed "to recognize high-quality, non-commercial courseware designed to enhance engineering education". Courseware (a term that has been superseded by the more general "software") is computer-based educational material that can be used to assist engineering students in their learning. During this study, the research team's priority was to gain insight into factors that affected changes in engineering education and to identify shifts in how innovations in technology and teaching were recognized or rewarded. Results of the research suggest that: The PA provided a critical path to legitimize the development and use of innovative courseware by faculty and student developers, bearing out the hopes of the PA program founders. The quality of courseware shifted over time, with demonstration of the impact on student learning and/or teaching replacing simple usability tests of usability. Student learning was positively impacted in terms of the outcomes: attitude towards the course, motivation to learn, faster learning, engagement with course content and retention in the course. The results seem to differ with more impact shown prior to 2009, but decreasing thereafter. Lower impact may be the result of the ubiquitous use of computers in education and daily life. (No causal effects were studied as a part of this research.) PA winning faculty members and instructors implemented extensive and continual efforts to encourage broad adoption of their innovation. No single method was shown as most effective; adoption was strongly impacted by the sustainability of the innovation. As a case study, the results cannot be easily generalized across all populations, types of institutions and types of reward systems. However, the results provide an in-depth picture of the nuances associated with use of PA winning courseware by both developers and adopters. The research opens a window to understanding the kinds of challenges faculty members face in creating and propagating these materials. It also suggests that as courseware (software) and computers become more ubiquitous, faculty may be rewarded less for its use. This may be the result of this type of use is no longer viewed as innovative. The researchers were able to reach adopters of the innovations to learn more about how and why they use this courseware. This is a unique addition to the literature of adoption, especially since some of the courseware is ‘old’ by today’s standards. By identifying these users, and collecting the case study data, the research lays valuable groundwork for future, more in-depth longitudinal studies of the impact of these innovations on teaching and learning. The methods used in this study are highly qualitative in nature and demonstrate how they might be used to study similar questions in other disciplines outside of engineering and the sciences.
