The objective of this project is to investigate whether NSDL tools and services support efficient selection of digital resources for educational purposes and whether these tools and services promote successful learning with these resources as measured by users' cognitive processes. The expected outcomes include a detailed analysis of the learning outcomes associated with two major NSDL tools that target the identification and contextual use of resources: NSDL General Search and NSDL Science Literacy Maps.
This project, conducted by the University of Utah, provides critical data about how digital library tools can promote effective learning with distributed learning objects on the Web. Phase 1 of the project uses a laboratory study to examine the influence of NSDL tools on pre-service teachers as they search for and use educational digital resources, Phase 2 brings in-service teachers into the lab, assessing their cognitive processes and outcomes as they use NSDL to perform a series of educational tasks, and Phase 3 moves to the classroom, assessing the impact of NSDL tools on student thinking and learning.
The broader impact of this project is a better understanding of the educational effectiveness of NSDL technologies and resources as well as recommendations on more effective tool and service design to promote positive impact on learning processes and outcomes. The evaluation materials and protocols are published for reuse in other digital library research.
Overview Despite significant national investment in the development of large-scale, educational digital libraries to support high-quality science instruction, little prior evidence as to the educational impact of these digital libraries has been established. The goal of this project was to explore the educational impact of the National STEM Digital library (NSDL) by examining the ways in which NSDL’s two major (and most-utilized) tools – NSDL General Search and NSDL Science Literacy maps – influenced pre-service and practicing teachers’ identification, processing, and use of digital resources for instruction. NSDL General Search is a keyword-based search tool that allows users to search for and access digital resources catalogued in the NSDL. NSDL Science Literacy Maps provide structured access to NSDL digital resources using node-link diagrams (much like concept maps). These node-link diagrams provide an organized representation of domain-relevant learning goals within a STEM topic area; clicking a learning goal (represented in a node) retrieves NSDL digital resources related to that learning goal. In order to identify the unique effects of NSDL tools, they were compared to a popular, commercial search engine (Google.com). Because NSDL provides educators with access to high-quality, relevant digital materials for science, technology, engineering, and mathematics (STEM) instruction, this project sought to assess NSDL impact on three major areas of educational importance: (1) cognitive processes during search for, evaluation of, and selection of digital resources, (2) cognitive processes associated with use of NSDL resources for instructional planning and educator learning, and (3) teachers’ development and implementation of student-centered science instruction. Research Across the project, four major studies were completed. Two studies focused on pre-service teachers’ use of NSDL to select and learn from relevant digital resources during instructional planning. The remaining studies examined the impact of NSDL on practicing teachers’ selection of digital resources and planning/implementation of student-centered, inquiry-based instruction. Research activities used a variety of research methodologies to gather rich and detailed data on teachers’ cognitive processes and online activities. These methodologies included: eyetracking, think-aloud protocols, cognitive load measurement, screen recordings, surveys, semi-structured interviews, and domain knowledge assessment. Key Findings Pre-Service Teachers. For pre-service teachers, NSDL tools were found to provide significant support for pre-service teachers when they provided a structured overview of domain information and reduced the demands associated with creating search queries. NSDL Science Literacy Maps promoted more efficient identification of relevant STEM digital resources compared to the NSDL General Search. NSDL Science Literacy Maps were as efficient as a very familiar and frequently-used commercial search engine: Google.com NSDL Science Literacy Maps eased the cognitive load associated with digital resource retrieval as pre-service teachers planned instruction. Pre-service teachers experienced significantly lower cognitive effort when using NSDL Science Literacy Maps compared to NSDL General Search or Google.com. NSDL Science Literacy Maps promoted deeper consideration of STEM content during instructional planning. When using NSDL Science Literacy Maps, pre-service teachers expressed significantly more thoughts related to reasoning/reflection about STEM content compared to when they were using Google.com. Practicing Teachers. For practicing teachers, NSDL was found to provide significant support for the development of student-centered, inquiry-based classroom instruction and to improve teachers’ technology-based teaching practices. Specifically: Teachers rated STEM digital resources found using NSDL.org as significantly more accurate and well-aligned to state standards than resources found using Google.com Science teachers agreed that using the NSDL Science Literacy Maps changed their approaches to technology-supported instruction and strongly agreed that using NSDL Science Literacy Maps changed the types of digital resources that they integrated into their lessons. When comparing a typical classroom lesson with a lesson developed using NSDL Science Literacy Maps, teachers reported significantly greater adherence to "reformed" teaching practices that reflect national efforts to improve science instruction and support student learning/engagement. Surveys completed following classroom teaching showed significant differences between a typical science lesson and an NSDL-supported lesson. Science lessons prepared with the NSDL Science Literacy Maps were significantly more likely to include student-centered and inquiry-based activities. Conclusions The National STEM Digital Library has a demonstrable and positive educational impact. Pre-service teachers benefitted from NSDL tools that structure access to STEM digital resources via graphical organizers that depict key learning goals in a domain. NSDL Science Literacy Maps facilitated efficient access to digital resources, reduced cognitive effort associated with digital search and retrieval, and promoted deeper processing of domain concepts. For practicing science teachers, NSDL had significant and positive impacts on their abilities to develop student-centered, inquiry-based science instruction. Although NSDL’s keyword search interface showed few advantages over an existing commercial search engine, NSDL Science Literacy Maps showed strong and robust support for beginning and experienced educators. Development efforts within educational digital libraries should focus on expanding access to graphical or interactive tools that structure access to digital materials based on organized, conceptually-meaningful representations of domain ideas.
