Survivor Bias

I found this (probably apocryphal) anecdote demonstrating survivor bias which I think makes the point with more clarity than the classical example of mutual funds:

Every day in World War II, bombers would fly from England to drop bombs on targets in continental Europe. Some would never return, some others would make it home on a wing and a prayer, shot full of holes from German fighters and anti-aircraft guns.

In order to improve the odds of survival, the allies decided to improve the armor plating on their bombers. But since weight is important on airplanes, they only wanted to add armor to those places on the airplane where it would actually help. So the engineers began a research project to determine where on the aircraft it would be most useful to have additional armor.

They diligently measured the locations of bullet holes on damaged bombers, compiled the statistical data (today we would say they built a database, but back then it was done by hand), and discovered clear patterns: bombers were much more likely to have bullet holes in the wings, tail surfaces, and in the tail gunner’s position. Holes in the cockpit and fuel tanks were relatively rare.

So the engineers made their recommendation: add armor plating to the wings, tail surfaces, and tail gunner’s position, since those were the locations on the plane most likely to be hit by German fire.

Then a statistician looked at the data, and realized that the engineers had come to exactly the wrong conclusion: because the engineers could only examine bombers which made it home safely, the right thing to do was to put armor plating where the engineers didn’tfind find any bullet holes, over the fuel tanks and cockpit.