Researchers often wish to synthesize data from independent studies, using methods such as systematic review, meta-analysis, and expert elicitation. Research synthesis presents difficult methodological questions and approaches have evolved differently across academic disciplines. Both the choice of synthesis method and its application can affect the conclusions, with important implications for decision making. This workshop explores these issues in the context of environmental, health, and safety policy, where such methods may affect conclusions about the toxicity of a substance, the relationship between exposure and health risks, the monetary value of risk reductions, and other policy-relevant research questions.
The goal of the project is to improve the methods used for research synthesis so as to promote evidence-based decision making. The objectives include: (1) increasing cross-disciplinary communication and collaboration on methodological issues by bringing together experts from diverse fields to address common problems; (2) defining more rigorously the types of problems and types of data for which different synthesis methods are most appropriate, alone or in combination; (3) developing innovative approaches for addressing specific challenges in applying these methods; and (4) identifying areas where further cross-disciplinary work may be particularly fruitful. The project includes three phases: developing background papers, convening an expert workshop, and disseminating results.
How to best synthesize research evidence across studies is a pervasive issue that arises in almost every discipline. Practices vary across fields, however, due in part to differences in the problems addressed and the data available. As a result, there are many opportunities for researchers to learn from other disciplines and to collaborate to address methodological challenges. With support from NSF and others, the Harvard Center for Risk Analysis (HCRA) sponsored a workshop in October 2013 to address these issues. The workshop brought together experts from diverse fields to discuss a variety of research-synthesis methods, including systematic review, meta-analysis, and expert elicitation. It focused on using these methods to estimate parameters in policy-analytic models and other applications. Below, we describe the project then briefly summarize the key findings. Project Summary This project included three phases. First, we commissioned papers from leading experts in diverse disciplines, including risk assessment, toxicology, program evaluation, and economic analysis. The papers are posted on the HCRA website (www.hsph.harvard.edu/hcra/research-synthesis-project) and address the use of research-synthesis methods in environmental protection, food safety, health care, and other policy areas. The two-day workshop was the second phase. The first day was open to the public and included presentation and discussion of the commissioned papers. The authors were asked to address four questions: 1) What criteria should be used to evaluate the applicability of different research-synthesis methods to particular types of problems and data? 2) What particular characteristics of the problem and data make the research-synthesis method(s) you address particularly well (or poorly) suited for that context? 3) What are the strengths and limitations of the outputs provided, and the implications for their use in policy analysis? 4) What are the most important research needs, in terms of methodological development, given your findings? A videotape of the proceedings is available on the HCRA website, along with copies of the slides and other materials. The second day was limited to the paper authors and focused on developing recommendations based on the previous day’s discussion. The third phase is submitting the papers for publication. The papers are currently undergoing review and revision and several of them will ultimately be published as a special issue of Risk Analysis. (Some papers were rejected by the journal and some authors chose to submit their papers to other journals.) Major Conclusions Some of the major conclusions are: Clearly defining the problem to be addressed is an important first step. In addition to identifying the quantity to be estimated, this step requires understanding decision-maker and stakeholder needs as well as the time and resources available. The second step should be a systematic, qualitative review of the literature. At minimum, this review should include a clearly-defined protocol for identifying studies and criteria for evaluating them. The details of the approach should be tailored to the nature of the data and the problem as well as the time and resources available. The decision to conduct a meta-analysis should be made on a case-by-case basis. In some cases, qualitative review may be all that is needed or appropriate to address the problem. In others, meta-analysis may be a useful tool for exploring variation in the research results or developing a reduced-form model to apply the results to other contexts. Structured expert elicitation is useful when some information is available but is not adequate to estimate the parameter of interest without the application of expert judgment, perhaps because results must be extrapolated from a study to a policy context. Such elicitations must be carefully designed and implemented to ensure valid responses and document the basis for the estimates. A general theme was that the methods used should be "fit for the purpose." The methods to be used, and how they are applied, need to be tailored to the particular context. All methods, regardless of the degree to which they involve quantitative or qualitative analysis, require some judgment. Thus the decisions made by the researchers should be clearly documented along with related assumptions. Finally, regardless of the field, inconsistent reporting of study methods, assumptions, and results often inhibits the use of synthesis methods. Establishment of reporting standards that are acceptable to researchers working in particular fields would promote more widespread and effective use of these methods.
