What’s Wrong With Expert Predictions

I wonder if the eighth circle of hell is the most crowded. In Dante's Divine Comedy that's where fortune-tellers existed. The soothsayers “had their faces twisted toward their haunches and found it necessary to walk backward, because they could not see ahead … and since he wanted so to see ahead, he looks behind and walks a backward path.” Poetic justice.

The July edition of Cato Unbound focused on expert predictions. Because humans are wired to avoid uncertainty we are drawn to anyone claiming to know what the future will resemble. While experts sound confident about an uncertain future, they nonetheless have difficulty beating random guesses about future events.

Once we understand that experts are not that reliable at predicting the future, the question arises: How can we do better? That was one of the questions the panel at Cato tried to address.

While some events will always be unpredictable, Dan Gardner (Future Babble: Why expert predictions fail and why we believe them anyway; my notes) and Philip Tetlock (Expert Political Judgment: How Good Is It? How Can We Know?), suggest a few ways that experts might still be able to improve.

Overcoming Our Aversion to Acknowledging Our Ignorance

Dan Gardner and Philip E. Tetlock review the not-too-promising record of expert predictions of political and social phenomena. The truth remains that for all our social science, the world manages to surprise us far more often than not. Rather than giving up or simply declaring in favor of populism, however, they suggest several ways to improve expert predictions, including greater attention to styles of thinking as well as a “forecasting tournament” in which different methodologies will compete against one another to gain empirical data about the process. Still, they concede that our ability to predict the future will probably always be sharply limited.

Who Cares about Forecast Accuracy?

Robin Hanson argues that most people aren’t interested in the accuracy of predictions because predictions often aren’t about knowing the future. They are about affiliating with an ideology or signaling one’s authority. The outcomes of predictions have nothing to do with either, of course, especially in the present. He suggests that one way to make predictions more accurate might be to lift both the social stigma and legal prohibitions against gambling. Unlike mere predictions, wagers carry real consequences for those who make them. Which, Hanson argues, they should.

In Defense of the Hedgehogs

John H. Cochrane offers a limited defense of the hedgehogs: Economics is full of uncertainty because the agents within the system are aware of the theories and possible actions of the other agents. Trying to capture all of them produces a hopeless muddle. Instead, what are needed are explanations of principle and the tendencies that arise all other things being equal. This calls for a hedgehoggy worldview after all. “Especially around policy debates,” he argues, “keeping the simple picture and a few basic principles in mind is the only hope.”

Fox-Hedging or Knowing: One Big Way to Know Many Things

We should not be surprised when experts fail to predict the future, says Bruce Bueno de Mesquita. Expertise doesn’t mean good judgment; rather, expertise is an accumulation of many facts about a subject. That we commonly prefer the pronouncements of experts suggests a bias in favor of “wisdom” and against the scientific method. He argues that statistically rigorous game theory can do better by examining the beliefs and objectives of major players in a given situation, and he welcomes forecasting tournaments as a means of refining the method.

Read this if you want to learn more about foxes and hedgehogs.

Discover what you've been missing and subscribe to Farnam Street via twitteremail, or RSS.