Experts are often asked to provide judgments to inform both private sector and public policy decisions. Such judgments may be used alone or with scientific models to estimate the probability of events such as changes in energy markets, levels of future carbon dioxide emissions, global temperature change, or the number of hurricanes to make landfall in the United States. Expert judgments are essential because past data may either be unavailable or not directly relevant due to changing conditions. From psychological research, however, we know that when making such probability judgments, people use mental short-cuts, or heuristics. The heuristics skew how people express judgments, resulting in unintentional biases in probabilities that systematically distort an individual's stated probabilities. These cognitive biases are the focus of the research, rather than intentional biases in which expressed probabilities are deliberately distorted in order to game the system. Minimizing cognitive biases in expert probabilities is essential when the probabilities are inputs to scientific models or to decisions that must be made without waiting for perfect information.
The objective of this research is to develop mathematical models and statistical procedures with which an analyst can estimate the degree of bias for an individual and thereby quantify adjustments that would eliminate those biases. The research focuses on three cognitive biases: overprecision, the tendency to be too sure that a particular event will occur; partition dependence, in which judged probabilities depend inappropriately on how the range of the uncertain variable is divided; and carryover, an ordering effect in which an individual?s stated probabilities may be affected by previous judgments. The bias measurement and debiasing methods are to be developed and tested in experimental settings using a large group of participants. The experimental results will show the extent of the biases under various circumstances, and the effectiveness of the method for removing the bias. This research will enable experts to provide probabilities that better represent their beliefs and knowledge, undistorted by bias, when engaged in public- and private-sector risk analyses. Potential applications include decisions in which data scarcity, coupled with high stakes, make the use of expert judgments essential. These include many areas of business decision making, as well as high-stakes policy decisions concerning, for instance, climate change and terrorism risk.
Experts are often asked to provide judgments to inform both private sector and public policy decisions. Such judgments may be used alone or with scientific models to estimate the probability of changes in energy markets, food borne illness incidences, or the number of hurricanes to make landfall in the U.S. Such expert probability judgments are essential because past data may either be unavailable or not directly relevant due to changing conditions. When developing these probability judgments, experts use mental short-cuts or heuristics. These heuristics affect the probabilities stated by the experts, leading to the unintentional inclusion of biases in their judgments. Although debiasing procedures exist, few if any are especially appropriate when using a survey approach to obtain the expert’s information. Our goal was to develop statistical procedures to identify and estimate the degree of bias in probabilistic judgments. We focused on two distinct but important categories of biases that have been identified in the behavioral decision-making literature: A pair of biases–partition dependence and carryover–that affect expressed probabilities of uncertain events when asking redundant questions. The partition bias reflects a tendency to assign probabilities that are insufficiently adjusted from equal weighting among alternatives, while carryover bias complicates the estimation of partition bias because experts tend to try to be consistent with previous judgments, even if flawed. The overprecision bias, which affects expressed confidence intervals for uncertain variables. Experts usually give intervals that are too narrow, so that actual events tend to fall outside of those intervals more often than assessed by the expert. We developed models that represent the expressed probability or confidence interval as a weighted sum of the expert’s actual belief and other quantities that are hypothesized to affect and bias the expressed judgment. Then by statistically extracting those weights from a series of answers from the expert, the hypothesized unbiased beliefs can be recovered. This general approach was tested using data bases of responses to in-person and web-based questionnaires. Partition Dependence and Carryover When asked for the probability of some event X happening versus not happening, experts tend to give probabilities that are too close to 50:50. Consequently, expressed distributions depend too much on arbitrary choices concerning how the variable is divided into classes or ranges. Such partition dependence can be identified by a sort of triangulation, by asking people several questions about differently defined categories or ranges. However, identifying and removing partition dependence is made difficult by the tendency for experts either to state a number similar to the last answer given ("previous response carryover"), or to combine previous responses and the laws of probability to calculate a response ("logical carryover"). We developed a model in which the stated probability is a weighted sum of all possible anchors, including the expert’s actual opinion, a 50:50 split (or other obvious splits), and previous responses, plus a random error. By asking several questions involving different categories, we can estimate the actual beliefs and the weights upon the other anchors using a weighted least-squares approach. We tested the approach using databases of probability assessments by students and assessments by a broad range of individuals via a web-based survey. Partition dependence and carryover were found for most questions and people. We examined whether the biases could be avoided by adjusting the ordering of questions or alternative questionnaire formats and confirmed that this could reduce, but not eliminate, those biases. Because partition dependence was found to persist even in the best case, we conclude that the statistical procedure is useful for suggesting debiased probabilities. Overprecision Overprecision often occurs when experts assess confidence intervals. For instance, an expert’s stated 90% interval may include the actual values only half of the time. In contrast, a "well-calibrated" expert’s intervals would enclose nine out of ten actual values. Our procedure corrects this bias by asking questions about uncertain variables whose outcomes are known and then statistically expanding or contracting the assessed intervals so that the adjusted intervals are well-calibrated. This same adjustment can then be applied to expressed intervals for variables that are inputs to a risk analysis. We used this approach to correct overprecision in an existing database of expert assessments of thermal performance of buildings. We demonstrated the effectiveness of the procedure, and out-of-sample validation confirmed that the corrected distributions were better calibrated. We also noted that the bias is strongly asymmetric, in that the upper and lower confidence limits are usually biased by substantially different amounts. Conclusion Our research has refined a general mathematical procedure to identify and correct for individual or interacting biases in expert elicitation survey data. Using our approach to compare the results of the stated probabilities with the debiased probabilities would allow a researcher to determine whether the results of a study or a decision are sensitive to such differences in the probability values.
