Dan Gardner's Future Babble explains why expert predictions often fail and why we believe them anyway.
Not only are experts wrong, the vast majority preform worse than random chance. And these failures don't seem to matter. In our fast media twitter cutlure, we rely on experts more than ever to shap our thoughts. While scary, ultimately, our world view is often shaped by people who are less accurate than a coin toss.
The first part of the answer lies in the nature of reality and the human brain. The world is complicated – too complicated to be predicted. And while the human brain my be magnificent, it is not perfect, thanks to a humble of cognitive wiring that makes systematic mistakes. Try to predict an unpredictable world using an error-prone brain and you get the gaffes that litter history.
As for why we believe expert predictions, the answer lies ultimately in our hard-wired aversion to uncertainty. People want to know what's happening now and what will happen in the future, and admitting we don't know can be profoundly disturbing. So we try to eliminate uncertainty however we can. We see patters where there are none. We treat random results as if they are meaningful. And we treasure stories that replace the complexity and uncertainty of reality with simple narratives about what's happening and what will happen. Sometimes we create these stories ourselves, but, even with the human mind's bountiful capacity for self-delusion, it can be hard to fool ourselves into thinking we know what the future holds for the stock market, the climate, the price of oil, or a thousand other pressing issues. So we look to experts. They must know. … And thanks to the news media's preference for the simple and dramatic, the sort of expert we are likely to hear from is confident and conclusive. They know what will happen; they are certain of it. We like that because that is how we want to feel. And so we convince ourselves that these wise men and women can do what wise men and women have never been able to do before. Fundamentally, we believe because we want to believe.
Bottom line, Gardner gives a good intro on why the future is impossible to predict. I enjoyed reading the book.
Still curious? There are other books on this subject too: Wrong: Why experts* keep failing us–and how to know when not to trust them; Being Wrong; and Manias, Panics, and Crashes.