Over 400,000 people visited Farnam Street last month to learn how to make better decisions, create new ideas, and avoid stupid errors. With more than 100,000 subscribers to our popular weekly digest, we've become an online intellectual hub. To learn more about what we do, start here.

Intuitions about sample size: the empirical law of large numbers

Some studies have indicated that people are good intuitive statisticians.

That is, we don't fall victim to the base rate fallacy and we take sample size into account.

Another group of studies, however, claims exactly the opposite: we fail to account for sample size and thus fall victim to the base rate fallacy.

Who are we to believe?

In their research Peter Sedlmeier and Gerd Gigerenzer propose a reasonable alternative that explains the contradictory results. They argue that “common intuitions about sample size conform to the empirical law of large numbers” and “that this law works only for one group of sample-size problems.”

“If,” they write, “this conjecture is valid, one should find that frequency distribution problems have been typically used by those who reported that people attend to sample size, and sampling distribution problems by those who concluded that people largely ignore sample size.”

Basically they believe that human intuition conforms to the law of large numbers.

Why do people sometimes attend to sample size and sometimes not?

The empirical law of large numbers is not to be confused with the (mathematical) law of large numbers. The mathematical law of large numbers is about a situation in which the sample size approaches infinity, whereas none of the studies reviewed here deals with this situation, but with finite sample sizes. Nevertheless, several researchers have described people's reasoning as following the ‘law of large numbers' (if they attend to finite sample sizes) or as violating it (if they do not). Because this misconception is widespread, we clarify in the Appendix what the law of large numbers is, why it does not apply to this research on sample size, and which mathematical results do apply.

The empirical law of large numbers therefore leads us to distinguish between two kinds of tasks: (1) frequency distribution tasks, in which participants judge how well a sample mean (a mean of Q frequency distribution) estimates a population mean and (2) sampling distribution tasks, in which participants judge the variance of sampling distributions. These two kinds of tasks have rarely been distinguished in research on intuitions about sample size, leading us to derive the prediction that studies reporting attention to sample size used frequency distribution tasks, while those reporting disregard of sample size used sampling distribution tasks.

Frequency Distributions and Sampling Distributions

A frequency distribution is a distribution of values from one sample. The overall range of values is divided into categories and the number of cases in each category is recorded. An example including a quantitative variable is the frequency distribution of heights in a sample of Italian men, where the categories might be 160 cm, 161 cm, 162 cm, and so on; an example with a qualitative (binary) variable is the distribution of male and female births during one day at a certain hospital. We will use the term ‘sampling distribution' for a distribution of means from independent samples of fixed size, drawn from the same population. A sampling distribution is not about the frequency of observations in different categories but about the frequency (or probability) of sample means falling into different categories.' The height distribution of 100 randomly sampled Italian men is a frequency¬†distribution; the distribution of height means in repeated random samples of 100 Italian men is a sampling distribution.

The difference between the variance of frequency and sampling distributions is particularly evident in the limiting case in which the sample includes the whole population. In such a case, a frequency distribution will be identical to the population distribution. A sampling distribution, however, will ultimately converge into a distribution concentrated at a single value: all sample means will be identical to the population mean, and the variance of the sampling distribution will be zero. We will now examine whether the distinction between frequency and sampling distributions can account for a substantial part of the inconsistent results.