This project investigates the process by which scientists attempt to assess and summarize--accurately, fairly, objectively--the scientific knowledge that society needs to make informed judgments about complex issues. How do scientists evaluate the reliability of their colleagues? research, understand its limits and degrees of uncertainty, and come to consensus (or not)? How do scientists respond to the subtle or overt pressures that arise when they know the world is watching the outcome of the process? What factors internal to the process may lead to systematic bias, error, or distortion? Periodically, scientists come together to present scientific information at the request of policy-makers and for the benefit of the general public, particularly in the area of environmental science relevant to policy. Recent examples include the National Acid Precipitation Assessment Program, the Ozone Trends Panel, and the Intergovernmental Panel on Climate Change, organized to summarize and assess the state of the art with respect to acid rain, global ozone depletion, and anthropogenic climate change, respectively. Such assessments consume large amounts of time, money, and manpower, and they can play a major role in policy debates. Therefore, it is important to understand how they work (or fail to work). Yet, few scholars have studied them.
Broader Impact
Understanding the internal dynamics of the assessment process puts us in a better position to judge the quality of any particular assessment, as well as to suggest potential means of improvement, and, ultimately, to provide an improved foundation for science-based policy. The presumption that assessments should influence policy assumes that the quality of these assessments is high, but that assumption has rarely been closely examined.
Science assessment has largely been a black box to date. Yet this approach to developing a relevant scientific basis for public policy has become not only pervasive but a critical influence on research agendas. The process has been studied from a political science perspective, i.e., what are the characteristics of an assessment process which is successful in terms of external influence. However, there has been no study of how evaluation,negotiation,deliberation, and decision on consensus about scientific evidence in actually carried out within assessments. We organized an unusual multidisciplinary team to carry out the research, including a physical scientist, an historian, and a philosopher at the senior level, and two ethnographers and two historians at the junior level. The methods included interviews of participating scientists, archival research on documents related to or directly produced by assessments, and direct observation in the case of National Research Council (NRC) panels. We focused mainly upon three assessment processes: the US National Acid Deposition Assessment Program, carried on over more than a decade, which both supported and assessed research; a series of ozone assessments, particularly those under the auspices of the World Meteorological Organization (WMO) beginning in 1985; and a set of assessments of the stability of the ice sheets in the context of predicting sea level rise, especially the assessments of the Intergovernmental Panel on Climate Change which began in 1988. Studies of a fourth set of assessments, those of the NRC, are still underway. A fundamentally unique aspect of this project was the decision to try to study assessments via ethnographic observations and our initial implementation of this approach at the NRC. As indicated above, this has not previously been done for assessment panels to our knowledge. The importance of this approach is not only to transcend the limitations of interviews and archival material, but also to give us direct and synchronous access to the institutional rules, regulations,structures, and individuals who create the conditions under which the assessment proceeds. Our key findings were these: 1. Particularly based on the study of NAPAP and supported by our recent observations of NRC panels, we concluded that institutional factors are critical for determining the scope and sometimes the conclusions of an assessment. 2. Based particularly on the study of the WMO ozone assessment and an early NRC assessment in 1983, we concluded that participants sometimes develop new knowledge ad hoc to fill gaps in the existing literature in order to meet the needs of the policy-maker clients. 3. On the whole, particularly in the case of IPCC, assessments tend to be cautious in their findings. Whether this is an inevitable result of consensus processes or a deeper inclination of scientists needs to be studied more carefully, hopefully through our ongoing ethnographic work. However, the evidence base illustrates that in the case of IPCC, projections of climate outcomes have generally been on the mark or indicated a lower level of change than actual outcomes years later.
