I recently had the opportunity to participate in a panel discussion on the topic of advanced analytics and data driven businesses for a gathering of senior executives from very large, mostly traditional enterprises.
One of the other speakers, a senior partner at a prestigious strategy consulting firm, kicked off by lamenting the lack of data driven decision making he sees at the senior level of most organizations. He described a typical meeting where someone on the team presents data, and then the group reverts to the HPPO decision rule, (i.e. the Highest Paid Person’s Opinion.)
I couldn’t help but note that I find the distinction between data and intuition a troubling false dichotomy. Why?
While there’s no question that many decisions are made without data (or even contrary to data), it’s equally dangerous to suggest that decisions must be made based only on data that is overtly considered. Put differently, the answer to the question “Is intuition at odds with data driven decision making?” should be a resounding “No.”
In Blink, The Power of Thinking Without Thinking, Malcolm Gladwell captures the essence of why this is true:
“...our world requires that decisions be sourced and footnoted, and if we say how we feel, we must also be prepared to elaborate on why we feel that way...We need to respect the fact that it is possible to know without knowing why we know and accept that - sometimes - we're better off that way.”
More bluntly, as supreme court judge Potter Stewart replied after struggling to create a concise definition of pornography, “I know it when I see it.”
The human brain is the most powerful data collection and processing system in the world; every day, it enables us to process and interpret the world around us, even if we’re unaware of what factors are influencing us or are unable to express the rules by which we’re making a decision. That leads us to the false dichotomy of relying - seemingly blindly - on expert experience, or making a “quantitative” decision based on “hard data.” Never mind that until recently, this data likely only represented a sliver of the complexity that exists in everyday life or business. While of course not universally true, in many cases that highly paid person is in that position as a result of experience -- experience resulting in both exposure to a broader universe of influences and the development of better decision making criteria.
The recent advances in “Big Data” technology -- the ability to collect, store, and process vast amounts of data -- has enabled us to assemble an increasingly comprehensive record of our daily interactions with the world. This record now resembles human memory, (i.e. a store of information regardless of whether it is explicitly deemed to be useful.)
It’s precisely this conundrum that has ignited the resurgence of interest in artificial intelligence and machine learning. As The Economist recently noted, “Machine learning is exactly what it sounds like: an attempt to perform a trick that even very primitive animals are capable of, namely learning from experience.” When coupled with this rapidly expanding digital record of our daily lives, the power of machine learning becomes its ability to observe the patterns of human decision making and infer the complex rules by which decisions are being made. By understanding which outcomes were good vs those that were less good, we can create a synthetic expertise approaching that of expert humans.