The goal of this SBIR is to develop an online tool, Synthetic Controls, to help local evaluators and others charged with demonstrating evidence of effectiveness of drug prevention programs obtain evidence for assessing the effectiveness of their interventions. Synthetic Controls will provide useful analyses and simultaneously relieve local evaluators of the burden of having to identify, convene, maintain, and administer surveys to a comparison group. The system will meta-analytically combine archived data from previously administered surveys - including evaluation data from intervention groups of relevant programs and from control groups where available - to establish counterfactuals for local evaluators. Thus, rather than relying on gold-standard methods (e.g. randomized control trials) for assessing impact, the goal of this approach will be to provide a statistically reliable method for determining the impact of local programs by comparing current performance against benchmarks set by an accumulated database of previous evaluations. During Phase I, we were able to demonstrate the feasibility of using an extant database for completing a pilot test of proposed methods. We refined statistical methods for completing analyses using referent data. We completed extensive analyses to demonstrate the feasibility and utility of the proposed methods. During Phase II, we propose to complete the following tasks: 1. We will design a database that will efficiently reconstitute various configurations of data to allow users to select appropriate comparison groups. 2. We will develop data analysis algorithms that will construct appropriate comparisons for behavioral and mediating variable analyses. 3. We will develop protocols to automatically link the application to data collected by local evaluations conducted using Evaluation Lizard. 4. We will develop protocols to provide a method to import data collected by local evaluations conducted using other (not Evaluation Lizard) services. 5. We will design a Synthetic Controls user interface that will allow clients to register, manage projects, complete analyses, and download reports. 6. We will conduct an alpha test to debug the system. 7. We will conduct a beta test to demonstrate that the system will work with a variety of potential clients.
Currently, local evaluations of drug prevention programs lack the ability to assess effectiveness because few, if any, include control or comparison groups against which to establish the degree to which they have reduced the rate of drug use onset. This project will provide an innovative strategy whereby local drug prevention evaluations can compare their findings with the findings of previous local evaluations. This will allow programs to assess their relative effectiveness.
| Hansen, William B; Derzon, James H; Reese, Eric L (2014) A Synthetic Comparator Approach to Local Evaluation of School-Based Substance Use Prevention Programming. Eval Health Prof 37:258-82 |
