The gold standard of research on the impact of educational interventions is a randomized controlled experiment where students or classrooms are randomly assigned to a treatment or control context. However, randomized experiments cannot always be used for ethical or logistical reasons. This study will contribute to the on-going research that identifies the conditions under which nonrandomized experiments can yield valid answers to questions about effectiveness of interventions. Within-study comparisons compare results from a randomized experiment to results from a nonrandomized experiment to understand the conditions under which results from the latter might approximate results from the former. Typically, such studies examine whether the results from both experiments are in the same direction, are both statistically significant (or both not), or differ significantly from each other. These studies usually yield only a binary conclusion that the results either do or do not correspond. The researchers in this study will use a Bayesian approach to develop more nuanced conclusions about the probability that the two designs match in terms of the estimated outcome. These more nuanced conclusions will provide better standards by which the validity of evidence from studies of effective educational practices will be determined.
The researchers will use datasets from three major kinds of quasi-experimental designs - the regression discontinuity design, the interrupted time series design and the nonequivalent control group design. They will explore how Bayesian approaches should be structured by investigating the nature of the parameters that need to be established. The analysis for the Bayesian correspondence analyses will be WinBUGS, a free software program for conducting Bayesian inference. However, the researchers will also explore the development of the syntax for computing such analyses with other statistical programs to expand the platforms within which such studies might be performed. The researchers will publish program syntax and findings for the major variants of within-study comparisons to make the Bayesian correspondence criteria they identify more accessible to education researchers.
