How Favouring The Winners Can Make You Lose

When planes were coming back from combat in World War II, they were riddled with bullet holes.

The powers that be asked scientists and researchers to figure out where to add armour to minimize losses.

Armour adds weight which slows the aircraft down and limits the distance it can fly so adding armour is done very sparingly.

An analysis was done on the planes and the spread of damage looked like this.

They assumed that to decrease losses they should be adding more armour over the wings and tail where all the bullet holes are.

A Mathematician Walks Into A Bar

Abraham Wald didn’t think so.

Wald was a member of the Statistical Research Group, which used statistical analysis to explore wartime challenges.

His insight was that the study that created the damage map above, only used planes that came back. They had survived the missions.

This meant that the sample information that was being used was an incomplete data set and didn’t paint an accurate picture of the damage the planes were experiencing.

Survivor Bias

Survivor Bias is a logical error that creates incorrect conclusions by focusing on the people or things that made it past a specific selection process.

This overlooks what gets ignored. A lot of what is ignored is not easily visible so therefore it doesn’t exist.

In Wald’s case it was the planes that didn’t make it back.

He proposed the opposite recommendation, reinforcing the areas that had no damage on the damage map. He recommended adding armour to the cockpit and engines.

A Nice Story

We all fall for the nice story. Bill Gates dropped out of University to become one of the richest people in the world.

So drop out of school and become a success right?

This leaves out a lot of other information. How many people dropped out of university and didn’t become successful? How many people got into Harvard, like Gates did. Or had already had years learning about computers and had a huge headstart in what would become the defining industry of his time?

Making conclusions with misleading or incomplete data is bad, what is worse is getting data that is biased towards a particular outcome.

We do this all the time. We want to hire people who have worked in good companies or who have gone to good schools. Surely, the other businesses and schools only picked the best.

But they can make mistakes, and many people that got jobs at good firms and good universities turn out to be not as good as first thought.

Another nice story is that Wald did what is said above. He did not. It is a good internet story about a maths genius telling some military guys what to do.

He did apply mathematics to the problem but survivor bias was already known at this point.

Good Questions Beat Good Conclusions

Remember the most important thing is to be answering the right question.

Sometimes you don’t have an answer to a good question and it takes a lot of work to figure it out so a good conclusion to a question you do know sounds like a better (easier) option.

To be answering questions you don’t know you need to find out good information and also address what information you don’t know.

---