Philip Tetlock, a professor at the University of Pennsylvania and co-author of Superforecasting: The Art and Science of Prediction, shares his insights on the art of forecasting.
He discusses the role of forecasters, the importance of cognitive diversity, and the challenges of predicting social events.
Influence of Tetlock’s Work
Tetlock’s work has had a significant impact on political figures, particularly in the UK.
Dominic Cummings, a senior adviser to Boris Johnson, is a fan of his work and has even appointed a super forecaster to his team.
However, there is a deep intellectual divide between social scientists and conservatives.
Accountability and Forecasting
While accountability in forecasting can lead to evasion and self-deception, forecasting tournaments offer a unique form of accountability where accuracy is paramount.
This can help shift incentives towards accuracy and away from ideological bias.
Identifying Talent and the Role of Super Forecasting
Identifying talent, particularly in environments where super bright people are not super rare, is a challenge.
Super forecasting could be a useful technique for predicting how successful people will be, based on the overlap between super forecasting and intelligence, and the overlap between intelligence and success in various professional lines of work.
The best forecasters aren’t just intelligent, but fox-like integrative thinkers capable of navigating values that are conflicting. – Philip Tetlock
Forecasting tournaments create a very stark monistic type of accountability in which one thing and only one thing matters and that is accuracy. – Philip Tetlock
Beyond Accuracy in Forecasting
Forecasters are expected to provide more than just accurate predictions.
They also serve as sources of ideological assurance, entertainment, and regret minimization.
The most effective forecasters are integrative thinkers who can navigate conflicting values and differentiate between varying levels of uncertainty.
Cognitive Diversity in Forecasting
Cognitive diversity is a critical component in forecasting.
A diverse group of forecasters can produce a composite forecast that outperforms the majority.
This is especially true when there is convergence among diverse observers, indicating that the weighted average composite is likely too conservative.
Challenges in Predicting Social Events
Predicting social events is inherently challenging.
The world’s predictability is uncertain, and simple extrapolation algorithms are often difficult to surpass.
However, knowing when to adjust the trend is crucial.
The Role of the CIA
The CIA should be value-neutral, providing impartial, apolitical advice.
Forecasting tournaments could be a useful tool for the CIA, as they incentivize accuracy above all else.
However, reforming such an established institution presents challenges.
The Role of Counterfactuals
Counterfactuals play a significant role in forecasting and policy arguments.
They are essential in learning from history and are often employed in debates to make rhetorical points.
However, their unresolvable nature makes them a challenging aspect of forecasting.
Cognitive Style and Political Ideology
Politicians who can confront cognitive dissonance between their values and engage in complex synthetic thinking are likely to be better forecasters.
However, fluid intelligence is a more powerful predictor of forecasting accuracy.
Objective Metrics for Superior Counterfactual Forecasts
Tetlock’s focus program aims to develop objective metrics for identifying individuals and methods that generate superior counterfactual forecasts.
These forecasts are used in simulated worlds where history can be rerun to assess the probability distributions of possible outcomes.
Linking Counterfactual Reasoning and Conditional Forecasts
The program also aims to link the sophistication of counterfactual reasoning about the past to the subtlety and accuracy of conditional forecasts about the future.
As people become better counterfactual reasoners, there should be less ideological polarization in their counterfactual beliefs.
Limitations of Machine Learning and AI
Despite advancements in machine learning and artificial intelligence, these technologies are not yet capable of outperforming human intelligence in certain areas, particularly those posed by the intelligence community.
