The project comprises methodological research to aid the analysis of comparative studies; that is, studies comparing "treated" subjects to controls in order to estimate effects of the treatment. Its starting point is an influential approach associated with Donald Rubin and Paul Rosenbaum, among others. In this approach, if such a study can be assumed latently randomized -- conditional on the covariate, treatment is assigned as if at random -- then the sample is reorganized into suitably similar blocks, and later analysis proceeds as if assignment to treatment had been manifestly at random, at least within these blocks. Techniques with which to reorganize samples in this way, such as matching and propensity scoring, have received much methodological attention in recent years, but whether and when it is inferentially valid to treat the transformed sample as if it had manifestly been randomized remains incompletely understood. In propensity-score matching, for example, extant methodological literature is relatively silent on estimation uncertainty in the propensity score, on imprecision in matching on it, and on how to determine whether these errors may be large enough to invalidate the adjustment. The primary aim of the present project is to use new characterizations of these errors, which are compatible with widely used forms of propensity stratification, to develop diagnostics for the suitability of a given propensity matching or stratification. Secondarily, the project assumes that manifestly randomized studies, along with those being treated as such, are to be analyzed with methods that rely only on properties of randomization, rather than model-based methods. These methods differ from those most commonly used in the social sciences to analyze comparative studies, but they can build on and borrow strength from better-known methods as this project will also make clear. The project's methodological advances will be demonstrated in social science applications, and will be made freely available in high-quality open source software.
A motivating premise of the project is that good statistical methods not only better meet the needs of specialists who use them but also enable nonspecialists to more readily and accurately appraise the quantitative evidence those specialists produce. The research therefore concerns itself with a type of statistical adjustment, matching, that is distinguished for its simplicity and appeal to non-technical audiences. Because it and related methods are already quite popular, the methods and methodological extensions developed by the project will be immediately and directly applicable to any number of empirical investigations in the social and behavioral sciences, many of which have public health or public policy implications. These methods' emphasis on diagnostics, and on statistical conclusions based on the same independence assumptions that the diagnostics seek to corroborate, may help to demystify the statistical component of empirical investigations to which they contribute, and to make transparent the evidential strengths and weaknesses of particular studies. The project also seeks to make the methods themselves more broadly available. By generating freely available software, it will enable all who are so inclined to use the techniques in their own analyses. By including undergraduate and graduate students in the research, it contributes to the training of future quantitative social science methodologists. Because it includes a plan for special marketing of this opportunity to underrepresented minority college students, it complements efforts by others to cultivate research expertise among underrepresented groups.
