Do you have a cat? You might want to think about moving to a flat on the sixth floor or higher. The study shows that cats surviving falls from floors below the sixth tend to get more serious injuries than the ones falling from higher floors. It might be because at the fifth floor level cats reach the maximum speed (precisely, the so called terminal velocity), after which they relax, reducing the impact of the fall.
Or it might be simply because most cats falling from floors above the sixth don’t survive, which means they aren’t brought to a vet and their injuries aren’t taken into account in the study.
The survivorship bias is a common error in the analysis of data. It happens when we ignore an important group of data, which can lead to erroneous conclusions. We strive for success, so we analyse successful [people, companies, cats…] to borrow some of their features. However, what we’d be better off doing is to look at those who fail.
Imagine a wartime. You’re responsible for armouring the bombers. First idea? Use as much armour as possible, we want to keep the planes safe! However, this would make the bomber much heavier and harder to maneuver. On the other hand, using too little or no armour at all decreases the already slim chances of pilot’s survival. How to find the sweet spot?
That’s the question posed to the Statistical Research Group (SRG) formed at Columbia University (New York City) during the World War II. This team of excellent statisticians was working on various mathematical problems encountered during the war. By “excellent” I really mean excellent, as the group included for example Milton Friedman (Nobel Memorial Prize in Economic Sciences, 1976), Norbert Wiener (MIT mathematician and the pioneer of cybernetics) or Jimmie Savage (one of the creators of decision theory). However, probably the most outstanding scientist in the whole SRG was Abraham Wald. Paradoxically, as a Jewish immigrant from the enemy country (remember that we’re talking about the wartime) he was not allowed to produce any official reports within the group. And there’s so much he needed to communicate!
To answer the question Wald asked the army to collect some data from planes returning from the battle field. They recorded which parts of the bomber were damaged the most. It turned out that the bullet holes clustered around the center, the tail gunner and the wings. So that’s where one should put the armour, right?
Abraham Wald suggested the completely opposite. He told the military to leave the usually damaged parts unarmed and put the armour where one finds bulled holes least often, i.e. on the engines.
Did he go completely crazy? Why would one waste the armour on parts that don’t need it? Here’s where the survivorship bias comes into play. Remember that the army could only collect the data from the planes that returned. What we’re missing are the planes that were too damaged to come back from the enemy’s territory. This means that bombers can survive getting shot around the center, the tail gunner and the wings; it’s the damage of engines that bring the planes down. Engines are the parts that one should protect.
This worked. Wald’s idea was so good (and simple) that it’s been used during the wars in Korea and Vietnam as well. His awareness of survivorship bias saved many, many lives and bombers. This shows that equations and logical thinking can be more powerful than bombs and guns.
The survivorship bias appears in many aspects of life, not only in the military (or cats’ injuries). For example, companies are often judged based on the performance of their mutual funds. Imagine a company with an amazing grow. A good investment, hm? Don’t forget about the funds you don’t see! In order to judge the company’s performance, you need not only the data on its successes but also failures. How many mutual funds have died?
Or imagine you’re choosing a high school for yourself or your child who dreams of studying at Harvard. If out of ten best students at your favourite faculty of Harvard six come from the high school X, you might be tempted to choose this school. But what about all other kids from high school X? Were they as successful as the lucky six or did most of them fail to get to good universities?
Next time you want to base your decision on the past successes, think about the cats falling from the seventh floor. Maybe the apparent failure is what you actually want.