Tag: Dan Gardner

Philip Tetlock: Ten Commandments for Aspiring Superforecasters

The Knowledge Project interview with Philip Tetlock deconstructs our ability to make accurate predictions into specific components. He learned through his work on The Good Judgment Project.

In Superforecasting: The Art and Science of Prediction, Tetlock and Dan Gardner (his co-author), set out to distill the ten key themes that have been “experimentally demonstrated to boost accuracy” in the real-world.

1. Triage

Focus on questions where your hard work is likely to pay off. Don’t waste time either on easy “clocklike” questions (where simple rules of thumb can get you close to the right answer) or on impenetrable “cloud-like” questions (where even fancy statistical models can’t beat the dart-throwing chimp). Concentrate on questions in the Goldilocks zone of difficulty, where effort pays off the most.

For instance, don't ask, “Who will win the world series in 2050?” That's impossible to forecast and unknowable. The question becomes more interesting when we come closer to home. Asking in April who will win the World Series for the upcoming season and how much justifiable confidence we can have in that answer is a different proposition. While we have low confidence in who will win, we can have a lot more than trying to predict the 2050 winner. At the worst we can narrow the range of outcomes. This allows us to move back on the continuum from uncertainty to risk.

Certain classes of outcomes have well-deserved reputations for being radically unpredictable (e.g., oil prices, currency markets). But we usually don’t discover how unpredictable outcomes are until we have spun our wheels for a while trying to gain analytical traction. Bear in mind the two basic errors it is possible to make here. We could fail to try to predict the potentially predictable or we could waste our time trying to predict the unpredictable. Which error would be worse in the situation you face?

2. Break seemingly intractable problems into tractable sub-problems.

This is Fermi-style thinking. Enrico Fermi designed the first atomic reactor. When he wasn't doing that he loved to tackle challenging questions such as “How many piano tuners are in Chicago?” At first glance this seems very difficult. Fermi started by decomposing the problem into smaller parts and putting them into the buckets of knowable and unknowable. By working at a problem this way you expose what you don't know or, as Tetlock and Gardner put it, you “flush ignorance into the open.” It's better to air your assumptions and discover your errors quickly than to hide behind jargon and fog. Superforecasters are excellent at Fermi-izing — even when it comes to seemingly unquantifiable things like love.

The surprise is how often remarkably good probability estimates arise from a remarkably crude series of assumptions and guesstimates.

3. Strike the right balance between inside and outside views.

Echoing Michael Mauboussin, who cautioned that we should pay attention to what's the same, Tetlock and Gardner add a historical perspective:

Superforecasters know that there is nothing new under the sun. Nothing is 100% “unique.” Language purists be damned: uniqueness is a matter of degree. So superforecasters conduct creative searches for comparison classes even for seemingly unique events, such as the outcome of a hunt for a high-profile terrorist (Joseph Kony) or the standoff between a new socialist government in Athens and Greece’s creditors. Superforecasters are in the habit of posing the outside-view question: How often do things of this sort happen in situations of this sort?

The planning fallacy is a derivative of this.

4. Strike the right balance between under- and overreacting to evidence.

Belief updating is to good forecasting as brushing and flossing are to good dental hygiene. It can be boring, occasionally uncomfortable, but it pays off in the long term. That said, don’t suppose that belief updating is always easy because it sometimes is. Skillful updating requires teasing subtle signals from noisy news flows— all the while resisting the lure of wishful thinking.

Savvy forecasters learn to ferret out telltale clues before the rest of us. They snoop for nonobvious lead indicators, about what would have to happen before X could, where X might be anything from an expansion of Arctic sea ice to a nuclear war in the Korean peninsula. Note the fine line here between picking up subtle clues before everyone else and getting suckered by misleading clues.

The key here is a rational Bayesian updating of your beliefs. This is the same ethos behind Charlie Munger's thoughts on killing your best loved ideas. The world doesn't work the way we want it to but it does signal to us when things change. If we pay attention and adapt we let the world do most of the work for us.

5. Look for the clashing causal forces at work in each problem.

For every good policy argument, there is typically a counterargument that is at least worth acknowledging. For instance, if you are a devout dove who believes that threatening military action never brings peace, be open to the possibility that you might be wrong about Iran. And the same advice applies if you are a devout hawk who believes that soft “appeasement” policies never pay off. Each side should list, in advance, the signs that would nudge them toward the other.

There are no paint-by-number rules here. Synthesis is an art that requires reconciling irreducibly subjective judgments. If you do it well, engaging in this process of synthesizing should transform you from a cookie-cutter dove or hawk into an odd hybrid creature, a dove-hawk, with a nuanced view of when tougher or softer policies are likelier to work.

If you really want to have fun at meetings (and simultaneously decrease your popularity with your bosses) start asking what would cause them to change their mind. Never forget that having an opinion is hard work. You really need to concentrate and rag on the problem.

6. Strive to distinguish as many degrees of doubt as the problem permits but no more.

This could easily be called nuance matters. The more degrees of uncertainty you can distinguish the better.

As in poker, you have an advantage if you are better than your competitors at separating 60/ 40 bets from 40/ 60— or 55/ 45 from 45/ 55. Translating vague-verbiage hunches into numeric probabilities feels unnatural at first but it can be done. It just requires patience and practice.

7. Strike the right balance between under- and overconfidence, between prudence and decisiveness.

Superforecasters understand the risks both of rushing to judgment and of dawdling too long near “maybe.” They routinely manage the trade-off between the need to take decisive stands (who wants to listen to a waffler?) and the need to qualify their stands (who wants to listen to a blowhard?). They realize that long-term accuracy requires getting good scores on both calibration and resolution— which requires moving beyond blame-game ping-pong. It is not enough just to avoid the most recent mistake. They have to find creative ways to tamp down both types of forecasting errors— misses and false alarms— to the degree a fickle world permits such uncontroversial improvements in accuracy.

8. Look for the errors behind your mistakes but beware of rearview-mirror hindsight biases.

It's easy to justify or rationalize your failure. Don't. Own it and keep score with a decision journal. You want to learn where you went wrong and determine ways to get better. And don't just look at failures. Evaluate successes as well so you can determine when you were just plain lucky.

9. Bring out the best in others and let others bring out the best in you.

Master the fine art of team management, especially perspective taking (understanding the arguments of the other side so well that you can reproduce them to the other’s satisfaction), precision questioning (helping others to clarify their arguments so they are not misunderstood), and constructive confrontation (learning to disagree without being disagreeable). Wise leaders know how fine the line can be between a helpful suggestion and micromanagerial meddling or between a rigid group and a decisive one or between a scatterbrained group and an open-minded one.

10. Master the error-balancing bicycle.

Implementing each commandment requires balancing opposing errors. Just as you can’t learn to ride a bicycle by reading a physics textbook, you can’t become a superforecaster by reading training manuals. Learning requires doing, with good feedback that leaves no ambiguity about whether you are succeeding—“ I’m rolling along smoothly!”— or whether you are failing—“ crash!”

As with anything, doing more of it doesn't mean you're getting better at it. You need to do more than just go through the motions.  The way to get better is deliberate practice.

And finally …

“It is impossible to lay down binding rules,” Helmuth von Moltke warned, “because two cases will never be exactly the same.” Guidelines (or maps) are the best we can do in a world where nothing represents the whole. As George Box said: “All models are false. Some are useful.”


Mark Steed, a former member of The Good Judgment Project offered us 13 ways to make better decisions.

Why expert predictions fail and why we believe them anyway


Dan Gardner's Future Babble explains why expert predictions often fail and why we believe them anyway.

Not only are experts wrong, the vast majority preform worse than random chance. And these failures don't seem to matter. In our fast media twitter cutlure, we rely on experts more than ever to shap our thoughts. While scary, ultimately, our world view is often shaped by people who are less accurate than a coin toss.

The first part of the answer lies in the nature of reality and the human brain. The world is complicated – too complicated to be predicted. And while the human brain my be magnificent, it is not perfect, thanks to a humble of cognitive wiring that makes systematic mistakes. Try to predict an unpredictable world using an error-prone brain and you get the gaffes that litter history.

As for why we believe expert predictions, the answer lies ultimately in our hard-wired aversion to uncertainty. People want to know what's happening now and what will happen in the future, and admitting we don't know can be profoundly disturbing. So we try to eliminate uncertainty however we can. We see patters where there are none. We treat random results as if they are meaningful. And we treasure stories that replace the complexity and uncertainty of reality with simple narratives about what's happening and what will happen. Sometimes we create these stories ourselves, but, even with the human mind's bountiful capacity for self-delusion, it can be hard to fool ourselves into thinking we know what the future holds for the stock market, the climate, the price of oil, or a thousand other pressing issues. So we look to experts. They must know. … And thanks to the news media's preference for the simple and dramatic, the sort of expert we are likely to hear from is confident and conclusive. They know what will happen; they are certain of it. We like that because that is how we want to feel. And so we convince ourselves that these wise men and women can do what wise men and women have never been able to do before. Fundamentally, we believe because we want to believe.

Bottom line, Gardner gives a good intro on why the future is impossible to predict. I enjoyed reading the book.

Still curious? There are other books on this subject too: Wrong: Why experts* keep failing us–and how to know when not to trust themBeing Wrong; and Manias, Panics, and Crashes.