Diversity, testing, and how bugs often go unnoticed

Recently I was talking to a friend who told me that he was in the market for a new car. My friend, who has been a loyal driver of a particular brand of vehicle for many years, for argument's sake let's call it Lexura (not the actual brand, but since I don't want to get hassled by any PR folks, I am making up this brand). When I asked him if he was considering the latest Lexura luxury model, a brand new one for 2015 that has generated significant buzz and some really stellar initial reviews, my friend surprised me by saying 'No', that he had soured on the Lexura brand.

When I asked why the conversation went more or less as follows:

Him: 'I like Lexura, I really do. But the last two Lexura's I owned have had the exact same problem. When it is raining, and I have the windshield wipers on, the water comes right inside the driver's window. I am always getting wet.'

Me: 'Why would you have the window open if it is raining so much that you have to have the wipers on? That seems a little odd. Most people you know, close the windows when it's raining.

Him: 'Well, usually I do. But sometimes I am smoking. And I like to keep the window open when I am smoking in the car. And then if it is raining the water comes right in off the wipers.'

A sort of odd story, and immediately after hearing the 'when I am smoking in the rain the wipers get me wet' take from my friend I starting thinking about software, (and really any other kind of product), that is developed, tested (significantly), and then is released into the wide, strange, and harsh world of customers and users.

My friend should not, on paper anyway, be driving in the rain, wipers on, driver's side window rolled down, getting wet. It does not really make much sense to most of us. It's raining. Roll up the window, dummy.

But if you are a smoker, then it is pretty common that when you are smoking and driving that you want/need the window down, at least part of the way. I guess even smokers don't really want to be trapped in a small, enclosed area with their own second hand smoke. So they open the window. And most of the time it works out just fine. Unless it's raining and your Lexura has the bad habit of directing water off of the windshield wipers right into the open window and in your face.

So let's spin this back to technology and testing and think about how/why 'bugs' like the Lexura spitting water back through the driver's window (let's just assume that it is actually, a bug for now), can make it past the testers and developers and engineers and make it into the world. Could it be, perhaps, that no one on the Lexura design/dev team was a smoker? That driving in the rain with the wipers on and window open was never actually tested, as it would have never occurred to the non-smokers at Lexura that this was actually a thing that would be important to some customers? That this possible lack of 'diversity' in the makeup of the Lexura team led to a bug that was only likely to be experienced by customers whose specific issues were not adequately represented at Lexura?

This is kind of a odd story, as I mentioned above, but I think there is something important here nonetheless.

First, it is almost impossible in the design, development, and testing processes of software, hardware, or products of any sort to test everything, every potential use case that is possible. It just cant' be done. Bugs will results, often from customers using the product in a way the builders never considered or even could have reasonably imagined.

And second, 'diversity', at least the way we usually think about it, is often a very incomplete way to frame the noble notion of ensuring all important and representative voices are heard. Because every time you think you have incorporated ideas and points of view from all the necessary constituencies, one new one you never thought about raises a hand, and wants to be heard.

Even smokers who drive in the rain with the windows open.

Have a great week.