This exploratory study will investigate what is needed to increase the capacity of STEM education evaluators to use a systems-oriented approach to evaluation. The work will be guided by two research questions: 1) Are core system evaluations more effective than evaluations without an intentional system orientation? and 2) How does ECLIPS effectively build the capacity of evaluators to conduct core system evaluations? Evaluators of ITEST and ATE projects will be guided in revising their evaluations to employ a more systems-oriented approach to evaluation. Results from the project on systems-oriented evaluation and what is needed for evaluators to learn how to use this approach will be informative to evaluators of the full range of STEM projects. Researchers will be from InSites, a support network for educational change, located in Fort Collins, Colorado.
Six to eight STEM education evaluators of ITEST and ATE projects will be identified to participate in the study. These evaluators will work with guides and others who are knowledgeable about systems-oriented evaluation to revise existing evaluation plans. Much of the interactions will be through monthly webinars, annual meetings, and individual interaction with guides. Sessions will be recorded and interactions among participants and leaders will serve as a major source of data. In addition, a pre and post questionnaire will be administered to the STEM education evaluators. Other STEM evaluators and interested parties will be used to validate inferences made.
Outcomes from the project will include six to eight STEM education evaluators who are more knowledgeable about systems-oriented evaluations, a written guide on how others can be educated in using such an approach, and published articles on the process. The findings will be particularly relevant to STEM project evaluators and evaluators in general.
