There is a need at NSF and in the field for a set of guidelines that would direct the evaluation process for projects whose purpose is to develop a more diverse STEM workforce. The purpose of this project is to create guidelines that would be used by PIs and proposal developers to write effective evaluation plans in their proposals; by evaluators to plan, implement and conduct rigorous project evaluations; and by Program Officers and reviewers to judge the merits of evaluation sections of proposals or to provide guidance for prospective or current projects. comprehensive guide, population-focused tip sheets, interactive internet materials and a wiki about conducting evaluations with diverse populations including considerations centering on gender, race/ethnicity, and disabilities. Consideration is also given to the setting of the evaluation (e.g., institutional type, program type). The materials will include information about designing, implementing and assess the quality of projects and activities that focus on broaden participation in STEM education, especially those targeting post-high school participants. The final products will be unique because they will specifically address evaluation issues related to diverse populations such as under represented minoritis, females, individuals with disabilities, and individuals in multiple categories.
This project builds on previous NSF-funded work by the PIs individually and together. In addition, several specific NSF-funded or developed documents have led the way in formulating the idea and design for this project. Examples are the Framework for Evaluating Impacts of Broadening Participation Projects: A Report from a National Science Foundation Workshop, co edited by Beatriz Chu Clewell and Norman Fortenberry under Contract Number GS-10F-0482P by the Division of Research on Learning, the Indigenous Evaluation Framework Report by LaFrance and Nichols (Grant 0438720), and the Mixed Methods User-Friendly Handbook which was updated in 2010 (Contract REC 99-12175). The specific objectives of the project delineate the plan for the project, including a needs assessment with stakeholder groups, a literature review, the development and beta testing of materials, revision and field testing, and dissemination. The project includes an Advisory Board that has the expertise to provide important input to the process. The project also includes an external, independent evaluator and an evaluation plan that considers the progress and outcomes of the project.
This project is a unique contribution to the fields of evaluation, broadening participation, and STEM education.
Outcomes The primary project outcome is a user-friendly website, BeyondRigor.org, that provides evaluators and others with ways to better design and carry out evaluation with diverse populations. While the original goal was to provide this for evaluators of programs to increase STEM workforce diversity, the resulting research based tips and information have value for a much broader population. These tips help evaluators and others understand differential impacts of some evaluation strategies on participants from different demographic groups and how evaluations can be designed to minimize or even eliminate these effects. BeyondRigor.org also informs program officers about evaluation and what is needed to have better evaluations with diverse populations and better monitoring of their resource investments. A third section of BeyondRigor.org provides evaluators and others will easy to access information from seven existing resources on STEM Broadening Participation (BP) evaluations. Two of the guiding realities behind this project are that "all populations are diverse" and that "each individual is diverse". That while populations may be homogeneous in one area but they are not homogeneous in all areas and that each individual encompasses a variety of demographic characteristics seems self-evident. Making these realities explicit and showing how they need to be considered in evaluation (and educational research) design, analysis and interpretation has the potential to change the approach to evaluation. For example, an evaluation of a STEM workforce development program that targets African Americans needs to look for diversity within that population and see if there are other factors, such as family income, that need to be considered in the analysis. A program that "works for girls" tells us little about whether it will work for specific subgroups of girls. The major outcome of this project, is a way for evaluators to tailor their evaluations to better determine "what works for whom in what context." Intellectual Merit Key to being able to broaden participation in STEM, is to know what works for whom in what context. Without good evaluation this is not going to happen. It is not enough for evaluation to be methodologically rigorous. That is necessary but not sufficient to learn about what works for different groups of people. Evaluation methods need to be targeted toward the needs, issues and goals of different subgroups, if they are not then the results can be incomplete and even inaccurate. To help make this happen, the following resources were developed, reviewed and refined: with the assistance of over 30 principal investigators, evaluators and current and past NSF program officers, a series of tips and rationales for evaluators, principal investigators and others to use to improve evaluations with diverse populations tied to collecting more accurate data, using more appropriate analysis and collecting the right data. with the assistance of members of different groups and people with expertise working with specific populations, an overview of context and overviews of 11 of the contextual factors that those doing STEM evaluations over diverse populations need to consider. with the assistance of former NSF program officers, a series of tools for funders to use to improve evaluations at five different points in the grant process: the RFP/GA, Proposal Review, Grant Award, Project Implementation and Final Report. a resources section which provides readers with explicit, useful information on evaluation basics within the context of programs and projects to improve the quality, quantity and diversity of the STEM workforce. This project advanced the conceptual framework that evaluators bring to their work, including reminding them that the realties that all populations are diverse and that each individual is diverse needs to be considered in evaluation design, analysis and interpretation. Broader Impacts To improve broadening participation efforts, one of the five areas of broader impacts, is the very reason for this project’s existence. BeyondRiger.org has developed and is disseminating materials to improve evaluations with diverse groups. Unlike many projects that target one group; this project targets a variety of groups including gender, disability and race/ethnicity, further increasing its impact. The website, www.BeyondRigor.org is now available and is being used. Its tips were tested in workshops in two universities attended by faculty, evaluators and graduate students. Participants reported that about 30% of the tips were new to them and that they would now start using 60% of the tips. As evaluators, program officers and others continue to access and use the resources in the BeyondRigor.org, the impact will grow.
