In their 1978 paper Performance Sampling in Social Matches, researchers March and March discussed the implications of performance sampling for understanding careers in organizations. If you hire/fire or promote people you’ll probably want to read this post.

Considerable evidence exists documenting that individuals confronted with problems requiring the estimation of proportions act as though sample size were substantially irrelevant to the reliability of their estimates. But how can we apply this information in an organizational setting?

False Record Effect. A group of managers of identical (moderate) ability will show considerably variation in their performance records in the short run. Some will be found at one end of the distribution and will be viewed as outstanding; others will be at the other end and will be viewed as ineffective. The longer a manager stays in a job, the less the probable difference between the observed record of performance and actual ability. Time on the job increased the expected sample of observations, reduced expected sampling error, and thus reduced the change that the manager (or moderate ability) will either be promoted or exit.

Hero Effect. Within a group of managers of varying abilities, the faster the rate of promotion, the less likely it is to be justified. Performance records are produced by a combination of underlying ability and sampling variation. Managers who have good records are more likely to have high ability than managers who have poor records, but the reliability of the differentiation is small when records are short.

Disappointment effect. On the average, new managers will be a disappointment. The performance records by which managers are evaluated are subject to sampling error. Since a manager is promoted to a new job on the basis of a good previous record, the proportion of promoted managers whose past records are better than their abilities will be greater than the proportion whose past records are poorer. As a result, on the average, managers will do less well in their new jobs than they did in their old ones, and observers will come to believe that higher level jobs are more difficult than lower level ones, even if they are not.

…The present results reinforce the idea that indistinguishability among managers is a joint property of the individuals being evaluated and the process by which they are evaluated. Performance sampling models show how careers may be the consequences of erroneous interpretations of variations in performance produced by equivalent managers. But they also indicate that the same pattern of careers could be the consequence of unreliable evaluation of managers who do, in fact, differ, or of managers who do, in fact, learn over the course of their experience.

But hold on a second before you stop promoting new managers (who, by definition, have a limited sample size). I’m not sure that sample size is the right way to think about this.

Consider two people: Manager A and Manager B who are up for promotion. Manager A has 10 years of experience and is an “all-star” (that is great performance with little variation in observations). Manager B, on the other hand, has only 5 years of experience but has shown a lot of variance in performance. If you had to hire someone you’d likely pick A. But its important not to misinterpret the results of March and March and dig a litte deeper.

What if we add one more variable to our two managers. Manager A’s job has been “easy” whereas Manager B took a very “tough” assignment. With this in mind, it seems reasonable to conclude that Manager B’s variance in performance could be explained by the difficulty of their task. Additionally, this could explain the lack of variance in Manager A’s performance. With only two variables, I’ve already complicated the findings to the point where they can be questioned.

Some jobs are tougher than others. If you don’t factor in degree-of-difficulty you’re missing something big and sending a message to your workforce that discourages people from taking difficult assignments. Current management theory, on the other hand, advocates that you want your best people in your most difficult jobs.

Most organizations I’ve seen discourage their best people from the most difficult jobs. Occasionally, those people seek out the challenge regardless of organizational incentives. If they succeed in a difficult assignment they are a “hero.” If they fail, however, they are a mistake. That same person, however, could have been assured “hero” status if they took an easy job.