Auditors, Airplanes, and Bullet Holes

Here's a fascinating article by Chris Dobbert on how engineers approached adding armor to WWII bombers. The short version goes something like this...

Too many planes were getting shot down so the air force wanted to add armor for additional protection. Armor is heavy, you can't add too much without negatively impacting performance, maneuverability, and range, so the critical question is where do you place the armor? As planes came back, engineers mapped out where the most bullet holes were assuming that's where most airplanes got shot. The common thought pattern was place the armor where the greatest concentration of bullet holes are.

Which is completely, 100% wrong. That's true only if you understand the data in the context of a full population. If you're not looking at all planes, e.g. only the planes that made it back to base, you're ignoring all the planes that got shot down. In fact, the right answer is to add armor to all the places where there were no bullet holes.

Lot's of parallels here for professional accountants and auditors. It's often all to easy to focus on areas where data exists and ignore things where data is hard to come by. Ideally, you'd start with all accounting evidence and conduct full population substantive testing to identify exactly where the risk is and quantify how much is there. Then auditors could structure their procedures to focus on the risk.

Or said a different way, place the armor in exactly the right location.

‍