Is Forecasting Political and Economic Events Feasible?
Many researchers argue that forecasters are prone to multiple errors - they are biased, tend to be overconfident, and they differ a lot between each other creating as a result high level of forecasting noise. You would find equally confident forecasts that Ukraine is going to win the war as those who claim otherwise, same goes for US President elections, FX rates, stock indices etc.
It has been therefore an area of focus for many researchers and one of the most notable work on the quality of forecasting was performed by the Good Judgment Project led by Philip Tetlock (2011-2015).
During the project several thousand volunteers were making forecasts of the different political and economic events. Interestingly, a small group (around 2%) consistently outperformed the average forecaster. They were called by the project team - "superforecasters" and they even did significantly better than average for intelligence community analysts - a group trained to make accurate forecasts with access to classified information.
What Sets Superforecasters Apart?
Superforecasters distinguish themselves not only through high general intelligence but in how they apply their intellect:
They excel in analytical and probabilistic thinking.
They structure and break down complex problems.
They consistently apply the "outside view," focusing on base rates in their analyses.
They readily update their judgments in response to new information, embracing a cognitive style characterized by "active open-mindedness" - Tetlock calls it Perpetual-Beta" - degree to which one is committed to updating his beliefs and self-improvement. They like particular style of thinking: try, fail, analyse, adjust, try again...
"When the facts change, I change my mind. What do you do?" Maynard Keynes
This ability to change your judgment does not strike like high level of self-confidence but ability to form strong opinions and stay behind them is not a good predictor of forecasting capabilities. On the contrary, constant search for new data, even a certain level of doubt in a general feasibility of forecasting is a desired characteristic in this field.
Practical Applications from the Good Judgment Project
My personal involvement in the Good Judgment Project as a volunteer and recipient of the superforecaster badge in 2015 has profoundly influenced my perspective. I've grown more skeptical of individual forecasting feasibility and more appreciative of collective intelligence approaches.
One effective strategy highlighted in the project is the aggregation of multiple independent estimates, an universally applicable decision hygiene technique. This method proves most effective when combining judgments that are both independent and complementary, ensuring that forecasters do not influence each other and bring diverse skill sets to the table.
Eliciting and aggregating judgments that are both independent and diverse will be often the easiest, cheapest, and most broadly applicable decision hygiene strategy.
Implementing Decision Hygiene in Your Meetings
Mini-Delphi is a simple version of the formal method for aggregating diverse views, which can be applied in the single meeting. Here is how it works:
Participants produce separate (and silent) estimates.
They explain and justify their estimates.
Finally all participants can make a final estimate in response to the estimates and explanations of others.
The consensus judgment is the average of the individual estimates obtained in the second round.
This process, easily adaptable for decision-making meetings, emphasizes independent and diverse judgment, fostering richer insights and more robust conclusions.
Resources:
Kahneman, D., Sibony, O., & Sunstein, C. R. (2021). Noise: A flaw in human judgment. Hachette UK.
Comments