by Robyn Bolton | Jul 24, 2022 | Customer Centricity, Innovation, Leadership, Metrics, Stories & Examples, Strategy
Many years ago, Clay Christensen visited his firm where I was a partner and told us a story*.
“I imagine the day I die and present myself at the entrance to Heaven,” he said. “The Lord will show me around, and the beauty and majesty will overcome me. Eventually, I will notice that there are no numbers or data in Heaven, and I will ask the Lord why that is.”
“Data lies,” the Lord will respond. “Nothing that lies can be in Heaven. So, if people want data, I tell them to go to Hell.”
We all chuckled at the punchline and at the strength of the language Clay used (if you ever met him, you know that he was an incredibly gentle and soft-spoken man, so using the phrase “go to Hell” was the equivalent of your parents unleashing a five-minute long expletive-laden rant).
“If you want data, go to Hell.”
Clay’s statement seems absolutely blasphemous, especially in a society that views quantitative data as the ultimate source of truth:
- “In God we trust. All others bring data.” W. Edward Deming, founding Father of Total Quality Management (TQM)
- “Above all else, show the data.” – Edward R. Tufte, a pioneer in the field of data visualization
- “What gets measured gets managed” – Peter Drucker, father of modern management studies
But it’s not entirely wrong.
Quantitative Data’s blessing: A sense of safety
As humans, we crave certainty and safety. This was true millennia ago when we needed to know whether the rustling in the leaves was the wind or a hungry predator preparing to leap and tear us limb from lime. And it’s true today when we must make billion-dollar decisions about buying companies, launching products, and expanding into new geographies.
We rely on data about company valuation and cash flow, market size and growth, and competitor size and strategy to make big decisions, trusting that it is accurate and will continue to be true for the foreseeable future.
Quantitative Data’s curse: The past does not predict the future
As leaders navigating an increasingly VUCA world, we know we must prepare for multiple scenarios, operate with agility, and be willing to pivot when change happens.
Yet we rely on data that describes the past.
We can extrapolate it, build forecasts, and create models, but the data will never tell us with certainty what will happen in the future. It can’t even tell us the Why (drivers, causal mechanisms) behind the What it describes.
The Answer: And not Or
Quantitative data Is useful. It gives us the sense of safety we need to operate in a world of uncertainty and a starting point from which to imagine the future(s).
But, it is not enough to give the clarity or confidence we need to make decisions leading to future growth and lasting competitive advantage.
To make those decisions, we need quantitative data AND qualitative insights.
We need numbers and humans.
Qualitative Insight’s blessing: A view into the future
Humans are the source of data. Our beliefs, motivations, aspirations, and actions are tracked and measured, and turned into numbers that describe what we believed, wanted, and did in the past.
By understanding human beliefs, motivations, and aspirations (and capturing them as qualitative insights), we gain insight into why we believed, wanted, and did those things and, as a result, how those beliefs, motivations, aspirations, and actions could change and be changed. With these insights, we can develop strategies and plans to change or maintain beliefs and motivations and anticipate and prepare for events that could accelerate or hinder our goals. And yes, these insights can be quantified.
Qualitative Insight’s curse: We must be brave
When discussing the merit of pursuing or applying qualitative research, it’s not uncommon for someone to trot out the saying (erroneously attributed to Henry Ford), “If I asked people what they wanted, they would have said a horse that goes twice as fast and eats half as much.”
Pushing against that assertion requires you to be brave. To let go of your desire for certainty and safety, take a risk, and be intellectually brave.
Being brave is hard. Staying safe is easy. It’s rational. It’s what any reasonable person would do. But safe, rational, and reasonable people rarely change the world.
One more story
In 1980, McKinsey predicted that the worldwide market for cell phones would max out at 900,000 subscribers. They based this prediction on solid data, analyzed by some of the most intelligent people in business. The data and resulting recommendations made sense when presented to AT&T, McKinsey’s client.
Five years later, there were 340,213 subscribers, and McKinsey looked pretty smart. In 1990, there were 5.3 million subscribers, almost 6x McKinsey’s prediction. In 1994, there were 24.1M subscribers in the US alone (27x McKinsey’s global forecast), and AT&T was forced to pay $12.6B to acquire McCaw Cellular.
Should AT&T have told McKinsey to “go to Hell?” No.
Should AT&T have thanked McKinsey for going to (and through) Hell to get the data, then asked whether they swung by earth to talk to humans and understand their Jobs to be Done around communication? Yes.
Because, as Box founder Aaron Levie reminds us,
“Sizing the market for a disruptor based on an incumbent’s market is like sizing a car industry off how many horses there were in 1910.”
* Except for the last line, these probably (definitely) weren’t his exact words, but they are an accurate representation of what I remember him saying
by Robyn Bolton | Apr 1, 2021 | Innovation, Metrics
Of all the facets of innovation, innovation metrics may be the most requested, studied, and debated.
This is not surprising given that companies need to justify the billions of dollars they spend every year on “innovation” and the best way to do that is through a sound set of metrics with proven predictive power and relevant benchmarks.
However, despite the need, and decades of work to address it, a satisfying answer to “what innovation metrics should we use?” is as elusive as ever.
The key word there is “satisfying”
Innovation metrics exist. There are probably hundreds of them.
Innovation metrics are in use. Maybe even at your company.
But one universal set of metrics with proven predictive power and relevant benchmarks? Nope, that doesn’t exist.
And it shouldn’t exist.
What innovation is, why it is pursued, and the investment of resources and time required varies from industry to industry and company to company. So, it makes no sense to apply a standard set of metrics to a custom approach and activity.
The trap of “satisfying” metrics
“What gets measured gets managed” is a trap (and also not true and Peter Drucker never said it).
It’s a trap because, as noted by monetary theorist Charles Goodhart noted, “Any statistical regularity will tend to collapse once pressure is placed upon it for control purposes.”
Or as us common folk (and anthropologist Marilyn Strathern) say, “When a measure becomes a target, it ceases to be a good measure.”
It is not satisfying when ideas become cobras
Although Goodhart’s Law was first articulated in 1975, it’s been in action for centuries. Consider the story of the British government’s bounty on cobras back when India was still a colony – to reduce the cobra population, the British government offered to pay people for every dead cobra they brought in. The money was good and eventually, people started raising cobras for the sole purpose of killing them to collect the county.
This is exactly what happens with most innovation metrics.
Because companies are so keen to measure and track progress, they set metrics they can measure. Those measurable metrics quickly become targets.
For example, one of my clients was eager to “fill their innovation funnel” and so they set a metric for the number of new ideas submitted each quarter. It didn’t take long before the innovation team spent most of the last day of each quarter frantically submitting “new ideas” so that they could hit the target.
The next day, they would go into the system and reject all the frantically submitted ideas because the ideas didn’t meet established qualitative criteria for strategic fit, scalability, or business potential.
At the end of the year, management was dumbfounded – despite getting thousands of ideas, the team had yet to pilot an idea, let alone launch a new business and generate revenue.
Metrics aren’t satisfying if they aren’t effective in achieving your ultimate goal
Was killing lots of cobras Britain’s ultimate goal? No.
They wanted to decrease the cobra population.
But killing cobras was a key step in decreasing the population and it was easier and faster to measure and track the number of cobras killed.
Was getting lots of ideas my client’s ultimate goal? No.
They needed to increase revenue by 50% in the next 5 years.
But getting ideas was a key step in creating a new stream of revenue and it was easier and faster to measure and track the number of ideas submitted.
How to Create Effective Innovation Metrics
State the ONE Why.
This is the hardest part of setting metrics because there are lots of reasons to pursue innovation. But we have to remember that innovation is a means to an end and effective metrics focus on the end, not the means.
I like to use the 5 Whys to get to the ultimate goal. Here’s a rough approximation (with completely made-up numbers) of the conversation I had with the client mentioned above:
- ME: Why do you want # of new ideas in the funnel each quarter?
- Client: Because we need that many to get 4 new products to pilot
- Me: Why do you need to pilot 4 new products?
- Client: Because we have a 50% pilot success rate
- Me: OK, so you need to launch 2 new products. Why two?
- Client: Because our new products generate $100M in Yr 1 revenue
- Me: Why do you need $200M in new revenue each year?
- Client: Because we have a $1B gap between what we promised to shareholders and what we’re on track to deliver in 5 years
Their ultimate goal was to close a $1B revenue gap in 5 years, not generate # new ideas per year.
Set interim milestones.
Achieving an ultimate goal takes years of hard work and you may need to pivot or readjust expectations along the way. For that reason, it’s important to set interim milestones, not as decision points but as markers for when to pull your head out of the day-to-day details and assess whether or not you are on track and, if not, what you need to change.
Using the same client as an example, we could have decided that $1B in new revenue in 5 years meant that we needed to generate $200M in new revenue every year. But, as we talked through things, we realized that it would take time to fill the innovation funnel AND build the organizational capability to develop, test, and launch new products. As a result, we should set revenue goals that increased each year, reflecting the organization’s increasing innovation effectiveness.
Explore the MANY Hows.
Innovation may be one way to achieve your ultimate goal but there are lots of others and it’s important to stay open to all of them.
For my client, we quickly realized that it wasn’t reasonable to rely solely on organic innovation to close the $1B revenue gap. Instead, we needed to explore multiple approaches from organic innovation to M&A and everything in between.
In closing…
It’s easy to fall into the trap of “satisfying” metrics that measure activity.
It’s a bit harder to evade the trap by identifying effective metrics that measure progress to an ultimate goal.
But, I can assure you, achieving that ultimate goal is way beyond satisfying.
