This research project plans to refine, field test and establish the psychometric properties of a formal assessment tool, Dimensions of Sucess (DOS) for STEM Learning, in out-of-school time (OST) settings. With the help of this tool, formal observations on various dimensions of programming and learning of informal science can be conducted. The goal of the project is to assess the tool's psychometric properties, primarily its construct validity, so the instrument can be used widely to assess the program quality. Phase 1 will focus on iteratively refining and revising the DOS for STEM learning, building on existing experience with the protocol. Reliability analysis on the instrument will be condusted using a sample of educators. In Phase 2, student and teacher self-report surveys will provide additional data, which will be analyzed to support the further validation of the instrument. Observations of 100 OST educators with their students on two occasions by two randomly assigned observers will allow for full analysis of the tool and for the development of the training program for observers during Phase 2.
The project will have significant implications for the quality and evaluation of OST programs. At the local level, DOS will enable practitioners and administrators of programs to improve program quality and professional development. The project will yield a valid instrument that can serve as a standard for assessing OST STEM programs to improve program quality and to compare programs across sites. Thus, larger studies of OST STEM programs will be possible, contributing to the shared knowledge base. The program will offer training via the internet to program staff, researchers and evaluators, and will provide support for the use of the instrument, so that the instrument can be used widely. The training support will build capacity for program development and evaluation in the field of informal science learning.
The Dimensions of Success (DoS) observation tool was developed to measure STEM program quality in out-of-school time. The tool includes 12 dimensions that fall into two domains: Activity Environment and STEM Learning and Practices. PEAR has prepared a technical report for distribution to the OST field, which summarizes the development of the instrument and findings from the initial study that included 284 observations in the field across 58 STEM programs in four geographic regions. Data were collected by 46 trained observers who worked in pairs. This report is the initial step in developing a validity argument for the instrument. It will also share current uses of the tool as well as future work to further develop an understanding of its applicability in the field. After-school science programming takes many shapes and sizes, structures and approaches: the DoS tool offers one way to systematically discuss the quality of these experiences among colleagues, researchers, staff, and other stakeholders so practitioners can create the best learning opportunities for youth. Furthermore, funding structures for many STEM programs are now dependent on clear evidence of quality, best practices, and efforts to improve STEM activity offerings. A tool like DoS is critical in meeting this need. Now that it has been piloted widely, DoS will be used by program leaders and practitioners in a wide spectrum of informal science programs, including private and public school afterschool programs, charter schools, stand-alone organizations, local affiliates of larger organizations, girl-only programs, etc. Thus, DoS will become a versatile tool for assessing a variety of informal science learning experiences for today's youth. The DOS tool connects the fields of education and science, focusing on pedagogy within informal learning environments. DoS is unique in that it can be used across a range of scientific disciplines. For example, afterschool programs that focus on physics concepts, chemistry processes, engineering principles, or environmental science mechanisms can all be observed using DoS--therefore, it can help define how quality looks different but contributes to informal learning across scientific disciplines. We believe project findings will encourage a wider exposure of youth, and particularly underrepresented groups, to STEM concepts and careers, as the DoS tool will help programs increase the quantity of STEM offerings while also increasing the quality of the learning experiences. Development of the DoS tool will benefit research and education in the following ways. First, the education field will have a comprehensive tool for informal STEM education, which can be applied in a variety of settings. Second, the outcomes of the project will greatly contribute to the knowledge-base of ISE research- a field, in which more rigorous research is needed, and evidence-based findings are crucial to move forward. DoS will also contribute to institutional development of many research and evaluation organizations, as the availability of a psychometrically sound instrument will advance their work. In addition, the training program currently in development will be a web-based training prototype including multiple videos, real-time discussions between raters, and establishing IRR. Contributions beyond science and engineering include recommendations and contribution to the fields of positive youth development, adolescent psychology, informal learning infrastructure, formal and informal science teaching approaches (pedagogy), etc. With much funding going to in-school time and assessments that focus more on content learning and learning gains or academic success, there is a need for this work to highlight the contribution of informal science to student engagement, motivation, interest, and ability to both learn and USE scientific concepts. The findings of the current project will contribute to knowledge in the field of informal science education and assessment. After surveying a large set of afterschool programs, the need for a general tool that can identify program quality and allow for comparisons across settings was highlighted. Furthermore, funding structures for many STEM programs are now dependent on clear evidence of quality, best practices, and efforts to improve STEM activity offerings. A tool like DoS is critical in meeting this need. Now that it has been piloted widely, DoS will be used by program leaders and practitioners in a wide spectrum of informal science programs, including private and public school afterschool programs, museum and science center-based afterschool programs, charter schools, stand-alone organizations, local affiliates of larger organizations, girl-only programs, etc. Thus, DoS will become a versatile tool for assessing a variety of informal science learning experiences for today's youth.
