The success of democracies, markets and organizations often depends upon accurate forecasts by collections of individuals. Collective forecasts can involve a handful of people on a board of directors forecasting business trends, hundreds of people in congress making a policy forecast, or millions of people determining prices through a stock market. In each case, more accurate forecasts lead to better outcomes and forecasts that are way off the mark can lead to crises.
In this project, the research team will develop a new analytic framework for understanding collective forecasting that combines models from cognitive psychology and mathematics. In this framework, individuals construct models that they use to make forecasts. These models rely on attributes -- components of reality that the individual sees as most relevant in the unfolding future. For example, an individual might see a company's spending on research and development as portending better long run growth.
In this approach, different people choose non-overlapping sets of attributes creating a diversity of forecasts. Collective accuracy hinges on a combination of diversity -- seeing different parts of the problem and connections between them -- and sophistication. This approach extends the standard statistical framework in which collective accuracy arises through the cancellation of errors.
By unpacking the contributions of individual cognitive depth and collective diversity on forecast accuracy, the framework can be used to examine the knotty problem of how to create incentives to produce collective accuracy. Accurate groups, teams, or societies need individual level sophistication and collective diversity. How much of each will depend on the forecasting task. An effective incentive structure must therefore take into account the complexity of the task. To test incentive structures, the team will develop a suite of forecasting problems with real world features. Ideally, those structures that perform best can be tested in real applications.
The ability of a collection of people to make accurate assessments and forecasts underpins the well functioning of democracies, markets, and organizations. The stated goal of this project was to advance our understanding of the characteristics of wise crowds and how to create environments in which they would be more likely to emerge. We can report three contributions from this project. First, we were able to develop a framework based on two types of signals with which we can reconcile existing findings in economics, political science, computer science, and psychology. We find that in each of these disciplines wise crowds rely on a combination of individual ability and collective diversity. By diversity here, we mean different ways of seeing and understanding the world and different mental models. One of the PIs on this project testified before a Congressional Subcommittee to this effect -- that accurate predictions required a diversity of models and that perhaps contributed to our collective failiure to anticipate recent crises. Second, we found that maintaning cognitive diversity requires substantial potential diversity and a culture of experimentation. By that we mean for a group to maintain say five ways of thinking, they may need to have ten ways of thinking at their disposal. Diversity maintenance also requires that individuals experiment with new models. We also found, quite to our surprise, that isolating groups did not have as large an effect as has been believed. This counter-intuitive finding arises because smaller groups can maintain less diversity. When a group is divided in half, the resulting smaller groups lack the capacity to maintain the same level of diversity as the large group. Thus, for a society or an organization to maintain sufficient diversity, they need to be exposed to multiple ways of thinking and and also need a culture of experimentation - a willingness to try and explore new ways of thinking. Third, though the research project consisted of theoretical research intended for an academic audience, our research had a surprisingly large impact. Ideas from this project formed a central part of an online course that has attracted more than a quarter million students world wide. Owing to our findings on the role of diversity of thought, this research has been presented to literally scores of for profit and non-profit agencies, as well as high schools and universities. The PIs have also communicted with staff in the United States government on issues related to this work.
