Measurements That Mislead

Tests are often spectacularly bad at predicting performance in the real world. So why do we still cling to them? Because they give us a strong number—something precise.

This creates what Malcolm Gladwell calls “mismatch problems“—when the criteria for evaluating job candidates is out of step with the reality of the job demands.

Jonah Lehrer adds to the literature around the mismatch problem in this weeks WSJ.

The reason maximal measures are such bad predictors is rooted in what these tests don't measure. It turns out that many of the most important factors for life success are character traits, such as grit and self-control, and these can't be measured quickly.

Consider grit, which reflects a person's commitment to a long-term goal. As Angela Duckworth, a psychologist at the University of Pennsylvania, has demonstrated, levels of grit consistently predict levels of achievement, such as graduation from West Point and success in the National Spelling Bee.

The problem, of course, is that students don't reveal their levels of grit while taking a brief test. Grit can only be assessed by tracking typical performance for an extended period. Do people persevere, even in the face of difficulty? How do they act when no one else is watching? Such traits often matter more than raw talent. We hear about them in letters of recommendation, but hard numbers take priority.

The larger lesson is that we've built our society around tests of performance that fail to predict what really matters: what happens once the test is over.

Continue Reading