Are experts Wrong?

To read the factoids David Freedman rattles off in his book Wrong is terrifying. He begins by writing that about two-thirds of the findings published in the top medical journals are refuted within a few years. It gets worse. As much as 90% of physicians' medical knowledge has been found to be substantially or completely wrong. In fact, there is a 1 in 12 chance that a doctor's diagnosis will be so wrong that it causes the patient significant harm.

Time conducted an interview with David and I think you'll be interested in some of the discussion.

In Wrong you write about the “Wizard of Oz” effect. Basically, from a young age we're taught to think that someone else always knows best. First our parents, then our teachers, and so on.

The fact of the matter is, unless you're the smartest person in the world, there is someone out there who knows more than you do. So it's not that we want to discard expertise — that would be reckless and dangerous. The key becomes, how do we learn to distinguish between expertise that's more likely to be right and expertise that's less likely to be right?

And how do we go about that?
It would be nice if we could look at the experts' track record and look at all their pronouncements to see what percentage were right. But we can't do that, so you have to play a sort of statistics game here and ask the question, “What does better advice have in common?” so we can look for those features. Or, conversely, “What does bad advice have in common?” so we can avoid it.

What have you learned about bad advice?
Bad advice tends to be simplistic. It tends to be definite, universal and certain. But, of course, that's the advice we love to hear. The best advice tends to be less certain — those researchers who say, ‘I think maybe this is true in certain situations for some people.' We should avoid the kind of advice that tends to resonate the most — it's exciting, it's a breakthrough, it's going to solve your problems — and instead look at the advice that embraces complexity and uncertainty.

So we essentially just blindly follow experts?
That's exactly what it is. And there are certain experts who, not only is their advice very resonant, but they themselves are very resonant. Some experts project tremendous confidence. They have marvelous credentials. They can be very charismatic — sometimes their voice just projects it. Some experts get very, very good at this stuff. And what do you know? It really sort of lulls us into accepting what they say. It can take a while to actually think about it and realize their advice makes no sense at all.

You found some cases of experts who willingly discarded data that didn't fit with the conclusion they were after?
That is a huge understatement — it is almost routine. Now, let me point out that it's not always nefarious. Scientists and experts have to do a certain amount of data sorting. Some data turns out to be garbage, some just isn't useful, or it just doesn't help you answer the question, so scientists always have to edit their data, and that's O.K. The problem is, how can we make sure that when they're editing the data, they're not simply manipulating the data in the way that helps them end up with the data they want? Unfortunately, there really aren't any safeguards in place against that. Scientists and other experts are human beings, they want to advance their careers, they have families to support, and what do you know, they tend to get the answers they chase.

Continue Reading @ Time

David's book: Wrong: Why experts* keep falling us-and how to know when not to trust them.
Being Wrong: Adventures in the Margin of Error