Advantages of observational studies (Oss) over randomized controlled trials (RCTs) include lower cost, greater timeliness, and patients more representative of those usually treated. Unfortunately, researchers and clinicians frequently dismiss OSs because of concerns that unrecognized confounding factors may frequently and unpredictably distort the results. The empirical basis of these concerns, however, comes from methodologically flawed assessments of OSs. In addition the OSs evaluated in these assessments were primarily performed before the 1980s. Recent observational studies may be better. The proposed research will update the comparison of OSs and RCTs and examine characteristics of OSs that may affect comparability with RCTs. Potential treatment comparisons to be studied will be identified from a literature search for OSs comparing two treatments that were published after 1984. For each of these treatment comparisons we will search for all OSs and RCTs published afterl965 that compared the same two treatments using the same outcomes. Two types of analyses will be performed: 1) The first type of analysis will consider each treatment comparison separately. Within each comparison the combined effect size of the OSs will be tested for a difference with the combined effect size of the RCTs. 2) The OS from all treatment comparisons will be included in the second type of analysis. For these analyses we will test characteristics of the OSs for an association with a discrepancy measure, i.e. the difference between the effect size of the individual OS and the combined effect size of the comparable RCTs. Examples of characteristics of observational studies to be evaluated are l) objective characteristics of the OS (e.g. differences between treatment groups in measured baseline risk factors or sources of subjects), 2) subjective assessment by medical specialists that physicians will choose one specific treatment rather than another on the basis of patient risk, and 3) subjective assessment by the co-investigators of the quality of aspects of the OS. The number of clinically rich electronic data bases is rapidly increasing. This study could help inform the appropriate use of these databases for evaluating medical therapy.
