Back to the Future

Philip Tetlock& Dan Gardner. Super-Forecasting: The Art and Science of Prediction

Tetlock and Gardner’s book is a popular account of the Good Judgement Project, a tournament on forecasting geopolitical events which ran from 2011 to 2015. Of interest to, well, anybody interested in making forecasts.

About 2800 forecasters were asked to provide predictions for more than 500 questions. The forecasters had to assign probabilities to whether specific outcomes (e.g. the number of Syrian refugees crossing a certain threshold) would occur within a defined time frame (e.g. the next three months). While it is impossible to evaluate a single probabilistic forecast, a series of forecasts can be evaluated according to two criteria: calibration- when forecasters assign 20%, 30%, or 40% probabilities, these events should happen 20%, 30%, or 40% of times, and resolution- forecasters should assign high probabilities to events that eventually do happen and low probabilities to events that don’t.

Superforecasting is a successor to Expert PoliticalJudgement (EPJ), a book based on a forecasting tournament Tetlock ran in the 1980s and 1990s. EPJ presented two major findings: the average forecaster did no better than randomly guessing outcomes would have done, and experts who based their judgment on many different analytical approaches (whom Tetlock labelled “foxes”) were more accurate than experts whose thinking was based on one big idea or ideology (“hedgehogs”).  Tetlock’s current book takes this investigation into the sources of variation in forecasting skill one step further. He and his co-author Gardner focus on “superforecasters”, the best participants in the tournament who did significantly better than chance or other forecasters. Superforecasters scored consistently well over the course of the tournament, with roughly 70% of forecasters in one year keeping this status in the next year as well.

Tetlock demonstrates that superforecasters have a couple of intellectual characteristics in common: higher scores on intelligence and knowledge tests, numerical literacy, and active open mindedness. What is most striking about superforecasters, however, is that they also share a certain approach to forecasting: 

• they first unpack every question into its components, separating between components that can be known and those that cannot (e.g. which scenarios are imaginable for the U.S. to intervene in Syria?);
• they then start by taking an outside view, i.e. establish a baseline probability for the event to be forecast as part of a larger class of events (e.g. what is the baseline probability for the U.S. to militarily intervene in a civil war in the Middle East?);
• they then take an inside view i.e. take into account the particularities of the case in question as well as the various causal factors at play (e.g. which specific attributes of the Syrian crisis might drive the U.S. to intervene?); 
• they make sure to compare their view to other views;
• and, finally, they synthesize all the information into a probability judgement.

Superforecasters also react to news in line with Bayes’ theorem, i.e. they neither under- nor over-react to new evidence, but instead update their assessments based on their prior beliefs as well as the diagnostic value of new information. Finally, superforecasters were even better when put together in teams. Teams of superforecasters were good at avoiding both extremes that may affect teamwork- internal wars and group think- and managed to outperform prediction markets.

Tetlock also compares his results with the black swan approach to forecasting popularized by Nassim Taleb. Black swans are extreme and improbable, but highly consequential events (e.g. World War I). According to Taleb, historical probabilities (possible ways in which history could unfold) have a fat-tailed, not a normal distribution, meaning that black swans are vastly more likely to occur than is often assumed. Against this objection, Tetlock puts forward that most of history proceeds incrementally rather than in the form of a constant stream of extreme events. Moreover, the consequences of black swan events take time to develop and can be broken down into distinct questions that can be forecast. While it would for instance have been impossible to predict the storming of the Bastille, some of the events that followed would at least have been partially predictable.

Overall, a great project and a fascinating read. Tetlock and Gardner do a good job creating an accessible account of what may easily be one of this decade’s most important social science projects. However, while the book has been compared to Daniel Kahneman’s “Thinking. Fast and Slowly”, it does not entirely live up to the comparison. First, there is of course a huge difference in terms of the scope and wealth of material covered. Kahneman covered multiple decades of research in cognitive psychology, while Tetlock focuses on one research project. Moreover, while Kahneman also aimed to reach a broader audience, he seemed to have more confidence in his readers grasping complex issues than Tetlock and Gardner. Some chapters, for instance on the book’s implications for executive leadership style, are nice to read, but remain quite superficial and do not add much to the book’s central message. In spite of these minor quibbles, this is still an important and informative book. Hoping that there will be a follow-up more geared towards an academic audience, though…
Next on my social science reading list is Fukuyama's "Political Order and Political Decay", possibly to be followed by McAdam, Tarrow and Tilly's "Dynamics of Contention" and Della Porta's "Clandestine Political Violence"....

Random movie association:

Back to the Future about 2015…