Heading image for MA03: Survivor Bias

When we measure only the survivors, the mathematics can make failure look like success.

Imagine evaluating the performance of investment funds over twenty years. You calculate their average returns and discover impressive results.

But what happened to the funds that closed during those twenty years?

If your calculation excludes them, your results may be seriously misleading.

This is survivorship bias, a statistical error caused by studying only the observations that remain after a selection process.

The Mathematics of Survivorship Bias

In Fooled by Randomness, Nassim Nicholas Taleb explains how survivorship bias leads us to mistake luck for skill. [1]

Consider his discussion of investment fund managers.

Some managers achieve outstanding returns. Others perform poorly and disappear. When researchers examine only the surviving funds, they exclude part of the original population.

The resulting average may be artificially high.

Mathematically, the distinction is between two quantities:

  • Average return of surviving funds
  • Average return of all funds that originally existed

Their difference measures survivorship bias in the reported average.

When poorly performing funds are more likely to disappear, the survivors’ average return will generally overstate the performance of the original population.

This is a form of selection bias. The sample is not representative because inclusion depends on the outcome being measured.

Why This Matters

Taleb examines The Millionaire Next Door, a study of wealthy Americans that identifies habits associated with financial success.

He questions what happens when we study only those who became wealthy.

What about people who followed similar habits but failed to accumulate wealth?

Without including them, we cannot reliably determine how much of the observed success came from the habits themselves, rather than luck or favourable circumstances.

The statistical consequences are measurable.

In 1992, Stephen Brown and colleagues demonstrated mathematically that selecting only surviving investment funds can create the appearance of predictable performance, even when the underlying returns do not contain that predictability. [2]

In 1996, Elton, Gruber, and Blake examined mutual fund performance using records that included funds that had disappeared. They demonstrated that excluding those funds distorted reported performance. [3]

The lesson is not that every survivor is lucky or that successful strategies are ineffective.

The problem is that a biased sample cannot reliably tell us how effective a strategy is.

Even collecting more observations will not necessarily solve the problem if the selection process remains biased.

The Practical Principle

When evaluating statistical evidence:

  1. Define the population. What complete group are we trying to understand?
  2. Identify the selection process. Which observations were included or excluded?
  3. Find the missing data. Were failures, closures, or unsuccessful outcomes removed?
  4. Recalculate the average. How does the result change when missing observations are included?
  5. Test the conclusion. Does the apparent pattern survive after correcting for selection bias?

The reliability of a statistical conclusion depends not just on how much data we collect, but on which data we are missing.

References

[1] Taleb, N. N. (2001; revised 2004). Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets. Chapter 8, “Too Many Millionaires Next Door.” Random House.

[2] Brown, S. J., Goetzmann, W., Ibbotson, R. G., & Ross, S. A. (1992). “Survivorship Bias in Performance Studies.” The Review of Financial Studies, 5(4), 553–580. https://doi.org/10.1093/rfs/5.4.553

[3] Elton, E. J., Gruber, M. J., & Blake, C. R. (1996). “Survivor Bias and Mutual Fund Performance.” The Review of Financial Studies, 9(4), 1097–1120. https://doi.org/10.1093/rfs/9.4.1097