The focal problem of the proposal is significant and a matter of broad interest: how to bring second order beliefs (beliefs about beliefs) to bear on first order beliefs. Such issues arise, for example, in the context of legal assessments of evidence where jurists are called on to consider general considerations of reliability when adjudicating specific eye witness evidence.
Intellectual Merit: Although probabilistic approaches to rational belief have become very popular, there is no standard way of using these models to represent the first-order/second-order interactions such as the one mentioned above. A successful account of these interactions would represent a major achievement in probabilistic understanding of rational belief. The project promises to provide a formula for how second order beliefs affect first order beliefs.
Broader Impact: Among the potential broader impacts of the results of this project is a potential for providing a basis for a view of science that is neither dogmatic nor skeptical, which could mitigate the polarization of opinion exemplified in the so-called "science wars". These results might also be used to understand how jurors can take proper account of their own fallibility in doing their job thereby providing the project with obvious social importance. Other broader impacts may result from the publication of a planned book whose audience includes students in courses in criminology and philosophy of science. Insofar as the book will be studied by college students in various majors, its impact will reach well beyond professionals in the investigator's field. The project may also have a significant effect beyond the usual academic circles through the production of pamphlets that are addressed to policymakers, jurors, and the public at large that address the public perception of science (for example, how to reconcile the reliability of science with its fallibility), the dynamics of group decision-making by juries, and the proper way to evaluate forensic evidence, expert testimony, and eyewitness reports. For a project in philosophy, the proposed project has unusually broad impact.
This project has made contributions in three areas: general theory of rationality, replies to skeptical arguments about error, and the effect of calibration on equitable treatment of testimony. The question I addressed in the theory of rationality is how to use evidence about reliability, yours or someone else’s, to revise your beliefs about the world. If you are sure from witnessing a murder that the defendant is the guilty party, and then learn of the psychological evidence that human beings are generally overconfident in eyewitness testimony, how confident should you now be about the defendant’s guilt? This new information was evidence about you, not about the murder. We feel that it is relevant to what you should believe about the murder, but why? Similarly, when you learn that scientists have made mistakes, what effect should that have on your believing the next thing they say? The fact that they’ve made a mistake in the past is not relevant to the substance of a new hypothesis. In the standard probabilistic (Bayesian) theory of rationality – which is used widely in philosophy and the social sciences – such "second-order" evidence cannot affect your primary beliefs, because of idealizing assumptions I have generalized away from. I defined a rule, called Re-Cal, that says that you should revise your confidence in p to what your best evidence says is your reliability in p-like matters; this amounts to aiming for the well-known state of calibration, where your confidence matches your reliability. I derived Re-Cal from an already accepted more general principle of rationality, and thereby provided an explanation why the rule should be followed. With a collaborator I proved a convergence theorem that says that if you were to follow this rule then in the long run your confidence in p would match the objective probability of p. The rule is the first explicit and fully general re-calibration rule in the form of a conditionalization; as such it provides the answer for what re-calibration should be for a Bayesian. This formulation of re-calibration avoids standard objections to the idea, and escapes the threats of inconsistency that had led to resistance to even trying to develop an explicit rule in second-order probability. The rule has practical applications since we have recently become inundated from empirical psychology with information about human beings’ judgments and proclivities to action that doesn’t match our expectations about ourselves. We need to learn how to use it in assessing, for example, eyewitnesses. Also, the general public perpetually has difficulty learning of the fallibility of science without falling into a generic mistrust. The answer in both cases is a proportionality that is captured in Re-Cal: one mistake is not evidence for unreliability, but five mistakes out of ten assertions is; adjust confidence to reliability. The rule I defined can be understood intuitively and applied in both contexts. Second, I have evaluated five skeptical arguments discussed among philosophers that have strong negative consequences about the possibility of science and mathematics, and I found in each case that errors have been made about error, the growth of error over steps of reasoning, or the relevance of past science to present science. For example, in the pessimistic induction over the history of science, mistakes are made about the relevance of the failures of past scientists to the legitimacy of our confidence in our own science. Third, I’ve found empirical evidence, in two experiments, that gender has a systematic effect on calibration level – relation of confidence to accuracy – with women learning and men not learning, to be less overconfident. (The effect is causal, not merely correlational.) The significance of this can be seen in light of well-known psychological evidence that on average, by default, human beings believe confident people more. This means that in a population where individuals are trying to accumulate credibility, the underconfident and calibrated will be at an initial disadvantage compared to the overconfident, even if they are equally reliable. This would be a systematic source of an uneven playing field in any area where advancement depends on credibility, which includes all of academia, and if my experiments are right it would differentially affect women. I have argued that the only way to remedy this unfairness is by methods of blinding ourselves to a candidate’s confidence in the evaluation of performance, and teaching everyone calibration in an explicit way as part of basic education. The findings on the new re-calibration rule and its applications in science and the public discussion of science are discussed in a broadly accessible way in an article in the December 2012 issue of the magazine International Innovation, which has a wide readership over many fields and professions.
