As CIOs continue to weather the Covid crisis, there is tremendous value in the insights and priorities shared in the most recent CanadianCIO Census. A snapshot of day-to-day business hurdles, it pays special attention to the positive impact of data analytics, and the erroneous notion that it cannot be measured.
“This is a classic problem,” said management consultant Doug Hubbard, creator of the Applied Information Economics method, and author of How to Measure Anything, a book that contends that anything can be measured, including things deemed unmeasurable, such as technology ROI, organizational flexibility, and customer satisfaction.
JOIN DOUG AT ANALYTICS UNLEASHED
Speaking with ITWC CIO Jim Love, in advance of keynoting ITWC’s Analytics Unleashed conference on Sept. 23, 2021, Hubbard countered the claim that it is impossible to assess the value of analytics, citing a wealth of examples in which even simple statistical models are found to be superior to expert intuition. “If you can observe something, you can measure it,” he concluded. “That’s why I called my book How to Measure Everything, and not How to Measure Some Things.”
According to Hubbard, part of the problem is that most companies adopt analytics without having a baseline – a practice he compares to starting a weight loss program without weighing yourself at the outset. “Companies that weren’t tracking their performance before, don’t really see the change in performance,” he said.
In response to a comment from Love about the failure by some companies to measure the right things, Hubbard explained that people aren’t just measuring the wrong things; they are measuring almost exactly the wrong things, and most notably for IT, costs rather than benefits. “Compare almost any two measurements that an organization might conduct and ask yourself, which one do I spend more time measuring and which one is more uncertain most of the time,” he suggested. “Those will be inverted. It’s not just a little bit wrong. It’s a lot wrong.”
Compounding this lack of judgement in what gets measured is the realization that people are spending huge amounts of time doing measurements that are fundamentally useless. The fix, from Hubbard’s perspective, is to change the way decisions are made about what to measure. When faced with a big decision analysis problem, rather than asking what they know how to measure, people should ask what they need to measure. If they don’t know how to do the measurement required, it’s worth a small amount to either train someone to do it or hire a consultant.
In Love’s opinion, the real value of analytics is the reduction in uncertainty, especially when it comes to revealing benefits as opposed to costs. He agreed with Hubbard that companies hold algorithms to higher standards than human intuition, even though there is good evidence that an algorithm will, in many, many circumstances, outperform even expert judgement.
Similarly, Hubbard said people mistakenly believe that things that are uncertain are harder to measure, yet mathematically speaking, the opposite is true. “The more uncertainty you have about a thing, the more uncertainty reduction you get from the first few observations,” he argued. “If you know almost nothing, almost anything will tell you something. That’s the way to think about it.”
Asked to reflect on the often-voiced issue of not having enough data to conduct a rigorous analysis, Hubbard said people have more data than they think they need. “We don’t need to know everything,” agreed Love. “What we need is enough to reduce our level of uncertainty.”
For Hubbard, it is not about whether a model is perfect, but rather about which models are measurably less wrong than others. “If that’s the criteria, people end up making very different choices,” he said. “If models were never wrong – if they were exactly right – we wouldn’t even call them models, we’d call them reality.”
The best way forward, in the estimation of both Hubbard and Love, is to approach all models, including personal judgements, with some degree of skepticism. The good news, they say, is that there is plenty of data for making informed decisions. The bad news is that not measuring the benefits of analytics will rob organizations of the opportunity to do things differently and achieve their desired outcomes.
Join Jim Love and Doug Hubbard for more about measuring analytics at Analytics Unleashed, a new ITWC event focusing on how organizations can unleash the real potential of data analytics.