Daniel Kahneman, answered from the texts and cited to the page.
The law of small numbers is the name Amos and I gave to a systematic error in intuition about random sampling: people expect small samples to behave like large ones, and they are wrong in ways that matter.1 The clearest demonstration of this came from our first study together. We asked researchers — statisticians, members of the Mathematical Psychology Group and the American Psychological Association — a simple question: if you have already obtained a significant result with 20 subjects, what is the probability that a new group of 10 subjects will also yield a significant result?
The median answer was around.85. The correct answer is closer to.48.2 Only 9 of our 84 respondents gave anything in that range.3 These were not naive undergraduates; they were trained scientists who work with samples for a living, and their intuitions were badly miscalibrated. The underlying belief we identified is this: people treat a sample drawn from a population as highly representative of that population in all essential characteristics.4
From that premise it follows — wrongly — that any two samples from the same population should closely resemble each other. The gambler's fallacy is one expression of this; the exaggerated faith in the replicability of a single significant result is another.5 By the time I wrote about this in Thinking, Fast and Slow, I could place the error inside a larger picture.
The exaggerated faith in small samples is one instance of a more general tendency: we attend to the content of what we observe and pay too little attention to the reliability of the observation.6 We jump to causal stories. A county with high kidney cancer rates sits in a rural area — we reach for an explanation involving lifestyle or diet. A county with low rates sits in a rural area too — we reach for a different explanation.
Both patterns may be nothing more than the consequence of small populations producing extreme results more often than large ones do.7 The causal story feels satisfying, but it explains a statistical artifact. Knowing that large samples are more precise than small ones is not the same as having internalized what that means. Most people, if pressed, will assent to the proposition.
But if you cannot immediately see that "large samples are more precise" and "small samples yield extreme results more often" are two statements of the same fact, you have not understood it in any way that will protect your judgment.8
Our thesis is that people have strong intuitions about random sampling; that these intuitions are wrong in fundamental respects; that these intuitions are shared by naive subjects and by trained scientists; and that they are applied with unfortunate consequences in the course of scientific inquiry.Judgment under Uncertainty, the Kahneman and Tversky Chapters, p. 21
The first study that Amos and I did together showed that even sophisticated researchers have poor intuitions and a wobbly understanding of sampling effects.Thinking, Fast and Slow
If you feel that the probability is around.48, you belong to a minority. Only 9 of our 84 respondents gave answers between.40 and.60. However,.48 happens to be a much more reasonable estimate than.85.Judgment under Uncertainty, the Kahneman and Tversky Chapters, p. 21
We submit that people view a sample randomly drawn from a population as highly representative, that is, similar to the population in all essential characteristics. Consequently, they expect any two samples drawn from a particular population to be more similar to one another and to the population than sampling theory predicts, at least for small samples.Judgment under Uncertainty, the Kahneman and Tversky Chapters, p. 21
The first hypothesis was advanced to explain the common belief that chance processes are self-correcting, the exaggerated faith in the stability of results observed in small samples, the gambler's fallacy, and related biases in judgments of randomness.Judgment under Uncertainty, the Kahneman and Tversky Chapters, p. 68
The exaggerated faith in small samples is only one example of a more general illusion—we pay more attention to the content of messages than to information about their reliability, and as a result end up with a view of the world around us that is simpler and more coherent than the data justify.Thinking, Fast and Slow
Statistics produce many observations that appear to beg for causal explanations but do not lend themselves to such explanations. Many facts of the world are due to chance, including accidents of sampling. Causal explanations of chance events are inevitably wrong.Thinking, Fast and Slow
Even now, you must exert some mental effort to see that the following two statements mean exactly the same thing: Large samples are more precise than small samples. Small samples yield extreme results more often than large samples do.Thinking, Fast and Slow