What you don’t see

Do you have a cat? You might want to think about moving to a flat on the sixth floor or higher. The study shows that cats surviving falls from floors below the sixth tend to get more serious injuries than the ones falling from higher floors. It might be because at the fifth floor level cats reach the maximum speed (precisely, the so called terminal velocity), after which they relax, reducing the impact of the fall.

cat
Does falling from higher floors decrease cats’ injuries?

Or it might be simply because most cats falling from floors above the sixth don’t survive, which means they aren’t brought to a vet and their injuries aren’t taken into account in the study.

The survivorship bias is a common error in the analysis of data. It happens when we ignore an important group of data, which can lead to erroneous conclusions. We strive for success, so we analyse successful [people, companies, cats…] to borrow some of their features. However, what we’d be better off doing is to look at those who fail.

Imagine a wartime. You’re responsible for armouring the bombers. First idea? Use as much armour as possible, we want to keep the planes safe! However, this would make the bomber much heavier and harder to maneuver. On the other hand, using too little or no armour at all decreases the already slim chances of pilot’s survival. How to find the sweet spot?

That’s the question posed to the Statistical Research Group (SRG) formed at Columbia University (New York City) during the World War II. This team of excellent statisticians was working on various mathematical problems encountered during the war. By “excellent” I really mean excellent, as the group included for example Milton Friedman (Nobel Memorial Prize in Economic Sciences, 1976), Norbert Wiener (MIT mathematician and the pioneer of cybernetics) or Jimmie Savage (one of the creators of decision theory). However, probably the most outstanding scientist in the whole SRG was Abraham Wald. Paradoxically, as a Jewish immigrant from the enemy country (remember that we’re talking about the wartime) he was not allowed to produce any official reports within the group. And there’s so much he needed to communicate!

To answer the question Wald asked the army to collect some data from planes returning from the battle field. They recorded which parts of the bomber were damaged the most. It turned out that the bullet holes clustered around the center, the tail gunner and the wings. So that’s where one should put the armour, right?

Survivorship-bias
The clusters of bullet holes on the returning planes. By McGeddon [CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0)%5D, from Wikimedia Commons.
Abraham Wald suggested the completely opposite. He told the military to leave the usually damaged parts unarmed and put the armour where one finds bulled holes least often, i.e. on the engines.

Did he go completely crazy? Why would one waste the armour on parts that don’t need it? Here’s where the survivorship bias comes into play. Remember that the army could only collect the data from the planes that returned. What we’re missing are the planes that were too damaged to come back from the enemy’s territory. This means that bombers can survive getting shot around the center, the tail gunner and the wings; it’s the damage of engines that bring the planes down. Engines are the parts that one should protect.

Abraham_Wald
Abraham Wald, the man who didn’t fall for the survivorship bias. By Konrad Jacobs, Erlangen, Copyright is MFO [CC BY-SA 2.0 de (https://creativecommons.org/licenses/by-sa/2.0/de/deed.en)%5D, via Wikimedia Commons.
This worked. Wald’s idea was so good (and simple) that it’s been used during the wars in Korea and Vietnam as well. His awareness of survivorship bias saved many, many lives and bombers. This shows that equations and logical thinking can be more powerful than bombs and guns.

The survivorship bias appears in many aspects of life, not only in the military (or cats’ injuries). For example, companies are often judged based on the performance of their mutual funds. Imagine a company with an amazing grow. A good investment, hm? Don’t forget about the funds you don’t see! In order to judge the company’s performance, you need not only the data on its successes but also failures. How many mutual funds have died?

Or imagine you’re choosing a high school for yourself or your child who dreams of studying at Harvard. If out of ten best students at your favourite faculty of Harvard six come from the high school X, you might be tempted to choose this school. But what about all other kids from high school X? Were they as successful as the lucky six or did most of them fail to get to good universities?

Next time you want to base your decision on the past successes, think about the cats falling from the seventh floor. Maybe the apparent failure is what you actually want.

Advertisements

1 thought on “What you don’t see

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Violent metaphors

Thoughts from the intersection of science, pseudoscience, and conflict.

Our World in Data

Maths is fun. And fun is maths!

FlowingData

Strength in Numbers

Paul Cairney: Politics & Public Policy

Professor of Politics and Public Policy, University of Stirling

To tylko teoria

Maths is fun. And fun is maths!

Mathspig Blog

Just another WordPress.com weblog

CoolStatsBlog

Economics and statistics in plain English

Nintil

To estimate, compare, distinguish, discuss, and trace to its principal sources everything

Infinity Plus One

math is awesome

Quomodocumque

Math, Madison, food, the Orioles, books, my kids.

PhD Life

a blog about the PhD student experience, University of Warwick

The Thesis Whisperer

Just like the horse whisperer - but with more pages

Next Scientist

Digital Hacks To Be A Better Scientist

Daily Writing Tips

Maths is fun. And fun is maths!

Count Bayesie - A Probability Blog

Maths is fun. And fun is maths!

Eventually Almost Everywhere

A blog about probability and olympiads by Dominic Yeo

MARCOMMUNIQUE

Marketing and Communication for Nonprofit, Education, and Science

%d bloggers like this:
search previous next tag category expand menu location phone mail time cart zoom edit close