The STEP special project examines the challenges facing principal investigators and evaluators in efforts to assess the impact of individual STEP Type 1 projects and to define, construct, and conduct appropriate comparison group studies. The project empirically tests the feasibility and usefulness of collecting, analyzing, and sharing a common set of data to evaluate the impact of activities at the individual project level and to assess the effectiveness and impact of a set of proven practices across a well defined set of institutions for comparison purposes. The project pairs an interrupted time series design with a case control design to identify relevant distinctions between the experiences of students who are participating in project activities and the experiences of students who are not participants. The research integrates institutional administrative data with survey data from student participants to examine the effectiveness and impact of first-year experiences on student success. The project develops and provides a set of strategies and data resources from which institutions with similar characteristics can draw to assess the impact of the STEP program on baccalaureate degree attainment. Working closely with STEP Type 1 projects, this effort has the potential to develop strategies for the design and execution of evaluation efforts across multiple projects; provide project level data for project improvement and documentation of successes; provide information on project challenges and opportunities; provide strong evidence for project impact; and provide high quality data resources for other researchers interested in student retention in science, technology, engineering, and mathematics.
In this study, we partnered with four universities that had STEP (Science, Technology, Engineering, and Mathematics Talent Expansion Program) Type 1 projects to assess the feasibility of establishing benchmark indicators to understand what works across studies. The goal of the STEP Type 1 projects was to increase the number of students entering and graduating in STEM (Science, Technology, Engineering, and Mathematics) majors. The goal of this study was to determine the feasibility of coordinating data across projects about ways that colleges and universities can increase the number of students both becoming STEM majors and receiving STEM degrees. Students from four universities were surveyed over a two-year period using a control and treatment group design to compare students who were participating in STEP Type 1 programs to similar eligible non-participants from the year prior to the launch of the project. Students were surveyed in the fall and spring of both years. The questionnaire included questions about the student’s choice of major, the experiences of the student in that major, and variables that influenced the student to continue in the major and to switch to another major. We included the following topics: high school math and science coursework, college major and science coursework, time spent doing homework, GPA and academic degree expected, expected salary after leaving college, relationships with teachers and fellow students, questions about personality such as persistence, grit, and parents’ education level. We gained valuable knowledge about how to better implement the full-scale study. For example, we refined the content of the questionnaire and improved the follow-up survey, improved the training methods of site coordinators (who made phone calls to students who had not completed the survey) at our partner institutions, and evaluated incentives to motivate students to complete the survey. This study provided the foundation of the full-scale study (Building on STEP to Understand Variation in STEM Entry and Persistence), also funded by NSF. We applied the knowledge we gained and the preliminary results from this feasibility study to improve the research design of the full-scale study.
