Natural Experiments and RCT Generalizability: The Women's Health Initiative Principal Investigator: Vogt, William B This application addresses Broad Challenge Area (07) Enhancing Clinical Trials and specific challenge topic 07-AG-103 Development of methodologies and scientific tools for improving and/or assessing the external validity of randomized clinical trial (RCT) results to known populations. Much clinical research seeks to infer the effects of a treatment in a target population. The randomized controlled trial (RCT) is the """"""""gold standard"""""""" from which to make this inference. However, the average treatment effect among individuals enrolled in the trial may differ from the average treatment effect in the target population: the RCT may lack generalizability. Non-generalizability may arise if the study treatment differs from the target population's treatment or if subjects enrolled in the RCT differ from the target population. One rarely commented upon issue is that the relevant target population differs depending on the policy goals of the analyst. An additional challenge to the external validity of RCTs comes from drop-in, drop-out, and attrition from the study arms. We suggest statistical techniques to overcome these two threats to external validity and apply our techniques to the example of the Women's Health Initiative Estrogen + Progestin (WHI E+P) RCT. To deal with the variety of potential target populations and with the non-representativeness of RCT study populations for these targets, we suggest re-weighting techniques from the statistical literature. To deal with the intent-to-treat problem, we suggest the use of instrumental variables techniques from the econometric literature. The conjunction of those two techniques is novel;however, the greatest novelty of our proposal arises from the fact that we will validate these new techniques by examining how well they predict the effects of withdrawing menopausal hormone therapy (HT) from a large number of women in a well-known natural experiment. Novel methods to detect and correct for threats to external validity of RCTs such as ours have enormously important scientific and clinical significance. Our project is scientifically significant because methods, like the ones we propose, to address RCT external validity are underdeveloped and not routinely applied in the analysis of randomized trials in the medical literature. Improved methods to analyze the generalizability of RCTs have the potential to greatly increase the return from the large investment in RCTs made by both the public and by private companies by improving methods of statistical inference. Specifically, we aim 1) To use a large insurance claims database and the Medical Expenditure Panel Survey to characterize the population of women receiving menopausal hormone therapy prior to July, 2002 and the population of """"""""news responders"""""""" who discontinued hormone therapy after July, 2002 and to compare these populations to those enrolled in the Women's Health Initiative Estrogen + Progestin trial;2) To use reweighting techniques in order to assess the clinical effects the Women's Health Initiative Estrogen + Progestin trial results would predict in these populations;and 3) To use the natural experiment created in July 2002 to assess the actual effects of discontinuation among news responders and to compare this to the predicted effect.
We suggest statistical techniques to overcome two threats to the external validity of a randomized controlled trial and apply our techniques to the example of the Women's Health Initiative Estrogen + Progestin study. New methods to detect and correct for threats to external validity of randomized controlled trials have enormously important scientific and clinical significance since these trials are expensive to conduct and central to so many policy and clinical decisions and since these threats are so rarely analyzed. Improved methods to analyze the generalizability of RCTs have the potential to greatly increase the return from the large investment in RCTs made by both the public and by private companies by improving methods of statistical inference.
