Professional development of teachers is a strong component in the theory of action of improving student outcomes, yet such efforts are rarely rigorously evaluated. One of the problems of robust evaluation of professional development is that these programs are largely conducted with teachers in their school or district settings rather than being randomly assigned individually to different activities. This requires that the evaluation design be developed around group randomized trials, and the field of educational evaluation lacks important knowledge of how to structure such evaluations. This project is developing estimates of the empirical benchmarks of design parameters to be used to structure evaluations of whole school professional development efforts in mathematics. The project uses the large data sets that have been accumulated using the Mathematical Knowledge of Teaching (MKT) measures to develop estimates of the variation in teacher pedagogical content knowledge in mathematics to provide values that can be used to estimate sample size and other design parameters in evaluation design. Researchers from the College of Education at Wayne State University, the Educational Testing Service and Michigan State University are conducting this research.
This project leverages the data collected from multiple professional development projects for mathematics teachers that have used the MKT under the auspices of the researchers that have been collected in an online Teacher Knowledge Assessment System from over 10,000 teachers in over 1150 schools. The data from the MKT is combined with data that is available publically in the Common Core of Data about the schools and districts within which the professional development was conducted and additional data that have been collected by the MKT researchers. The researchers use random intercept hierarchical linear models to estimate the design parameters for four different group random trial designs. They examine the impact of several covariate sets of parameters on the variance components and the corresponding estimates of intraclass correlation. The resulting parameters are reported in association with the conditions of the schools and districts in which the data were generated.
The findings from this study are reported in the professional literature on research and evaluation design. In addition, professional development workshops disseminate the findings and the processes used to develop them at national educational research and evaluation meetings. In addition, the researchers include their findings in updates of the Optimal Design software that is used by multiple researchers and evaluators to structure the design of their own studies. Too frequently, researchers do not have sufficient knowledge of the potential contextual issues surrounding the schools and districts in which they work to make accurate estimates of the parameters that they should use to determine important characteristics of their research such as sample size. The findings of this study provide more realistic estimates of the design parameters that researchers and evaluators need to conduct their own studies.
