Protecting what isn't damaged

It's World War II and your job is to help the military devise a strategy for reducing the shockingly high loss rate of planes in battle. Dozens and dozens of planes are being lost due to ground-based enemy anti-aircraft weapons, as well as in air combat.

And of the planes that do make it back to their air bases safely, most have received at least some damage, with many of the damaged planes requiring substantial repairs to make them air-worthy again.

You show up to the air base, and as you begin examining the damaged planes you make an interesting observation - most of the planes that made it back have sustained damage to the wings, fuselage, and fuel systems, but most do not exhibit signs of damage in the engines or front of the cockpits.

A bunch of shot-up planes but a fairly consistent of measurable and repeatable characteristic - damaged fuselages but not engines. Wings that have sustained hits but with clean and intact cockpits.

Your recommendation to the military brass to reduce the rate and number of lost planes?

Well it seems intuitive that better armor and protection on the parts that have sustained the most damage would be the best strategy. I mean, you have evidence all around you - blown apart wings, fuel systems, etc. These parts are obviously sustaining heavy damage in battle, and need shoring up.

Makes sense, right?

Except that it is almost completely wrong, and due to the research and conclusions made in WWII by Abraham Wald, the opposite of the best strategy.

Wald concluded that the Air Force shouldn't arm or add protection to the areas of the planes that sustained the most damage on the ones that came back. By virtue of the fact that they planes came back at all, those parts of the planes could sustain damage.

Wald's insight, that the holes from flak and bullets on the bombers that did return represented the areas where they were able to take damage led him to conclude that these patches were the weak spots that led to the loss of a plane if hit, and that they must be the parts to be reinforced. 

Wald's suggestion an recommendation seemed unconventional, but only if you could get past what you could 'see', a bunch of blown apart wings and fuselages; and think about what you couldn't see, the planes that crashed as a result of the damage they sustained.

The big lesson or takeaway from this tale?  As usual, probably not much of one, with the possible exception is that it serves as a compelling reminder not to always focus on the obvious, the apparent, and what seems like the easy explanation.

Note - some of Wald's notes on this research can be found here.

Comments

[Mark]

One of my favorite statistical theory stories (I know, I know). You're absolutely right about the takeaway - we can't just look at the obvious and think that's the answer (although it can be); we have to consciously remind ourselves of what isn't visible as well. Our brains have developed through natural selection to zero in and quickly identify patterns in what we sense, especially what we see and that can trap our thinking. This story provides a good lesson that we have to do careful thinking in order to use our tools effectively. The Atlantic recently had an article about this and four other statistics problems that also offer important lessons.

[Steve]

Thanks Mark - I know you would appreciate this! It is a cool story, maybe because it presents a simple answer to what had to be seen as an incredibly hard problem. Thanks for the comments!