AI is everywhere: in our workplaces, homes, schools, art galleries, concert halls, and even neighborhood coffee shops. We can’t seem to escape it. Some hope it will unlock our full potential and usher in an era of creativity, prosperity, and peace. Others worry it will eventually replace us. While both outcomes are extreme, if you’ve ever used AI to conduct research with synthetic users, the idea of being “replaced” isn’t so wild.
For the past month, I’ve beta-tested Crowdwave, an AI research tool that allows you to create surveys, specify segments of respondents, send the survey to synthetic respondents (AI-generated personas), and get results within minutes.
Sound too good to be true?
Here are the results from my initial test:
150 respondents in 3 niche segments (50 respondents each)
51 questions, including ten open-ended questions requiring short prose responses
1 hour to complete and generate an AI executive summary and full data set of individual responses, enabling further analysis
The Tool is Brilliant
It took just one hour to gather data that traditional survey methods require a month or more to collect, clean, and synthesize. Think of how much time you’ve spent waiting for survey results, checking interim data, and cleaning up messy responses. I certainly did and it made me cry.
The qualitative responses were on-topic, useful, and featured enough quirks to seem somewhat human. I’m pretty sure that has never happened in the history of surveys. Typically, respondents skip open-ended questions or use them to air unrelated opinions.
Every respondent completed the entire survey! There is no need to look for respondents who went too quickly, chose the same option repeatedly, or abandoned the effort altogether. You no longer need to spend hours cleaning data, weeding out partial responses, and hoping you’re left with enough that you can generate statistically significant findings.
The Results are Dangerous
When I presented the results to my client, complete with caveats about AI’s limitations and the tool’s early-stage development, they did what any reasonable person would do – they started making decisions based on the survey results.
STOP!
As humans, we want to solve problems. In business, we are rewarded for solving problems. So, when we see something that looks like a solution, we jump at it.
However, strategic or financially significant decisions should neverrely ona single data source. They are too complex, risky, and costly. And they definitely shouldn’t be made based on fake people’s answers to survey questions!
They’re Also Useful.
Although the synthetic respondents’ data may not be true, it is probably directionally correct because it is based on millions and maybe billions of data points. So, while you shouldn’t make pricing decisions based on data showing that 40% of your target consumers are willing to pay a 30%+ premium for your product, it’s reasonable to believe they may be willing to pay more for your product.
The ability to field an absurdly long survey was also valuable. My client is not unusual in their desire to ask everything they may ever need to know for fear that they won’t have another chance to gather quantitative data (and budgets being what they are, they’re usually right). They often ignore warnings that long surveys lead to abandonment and declining response quality. With AI, we could ask all the questions and then identify the most critical ones for follow-up surveys sent to actual humans.
We Aren’t Being Replaced, We’re Being Spared
AI consumer research won’t replace humans. But it will spare us the drudgery of long surveys filled with useless questions, months of waiting for results, and weeks of data cleaning and analysis. It may just free us up to be creative and spend time with other humans. And that is brilliant.
In 2020, the International Standards Organization, most famous for its Quality Management Systems standard, published ISO 56000, Innovation Management—Fundamentals and Vocabulary. Since then, ISO has released eight additional innovation standards.
But is it possible to create international standards for innovation, or are we killing creativity?
That’s the question that InnoLead founder and CEO Scott Kirsner and I debated over lunch a few weeks ago. Although we had heard of the standards and attended a few webinars, but we had never read them or spoken with corporate innovators about their experiences.
So, we set out to fix that.
Scott convened an all-star panel of innovators from Entergy, Black & Veatch, DFW Airport, Cisco, and a large financial institution to read and discuss two ISO Innovation Standards: ISO 56002, Innovation management – Innovation management systems – Requirements and ISO 56004, Innovation Management Assessment – Guidance.
The conversation was honest, featured a wide range of opinions, and is absolutely worth your time to watch.
Here are my three biggest takeaways.
The Standards are a Good Idea
Innovation doesn’t have the best reputation. It’s frequently treated as a hobby to be pursued when times are good and sometimes as a management boondoggle to justify pursuing pet ideas and taking field trips to fun places.
However, ISO Standards can change how innovation is perceived and supported.
Just as ISO’s Quality Management Standards established a framework for quality, the Innovation Management Standards aim to do the same for innovation. They provide shared fundamentals and a common vocabulary (ISO 56000), requirements for innovation management systems (ISO 56001 and ISO 56002), and guidance for measurement (ISO 56004), intellectual property management (ISO 56005), and partnerships (ISO 56003). By establishing these standards, organizations can transition innovation from a vague “trust me” proposition to a structured, best-practice approach.
The Documents are Dangerous
However, there’s a caveat: a little knowledge can be dangerous. The two standards I reviewed were dense and complex, totaling 56 pages, and they’re among the shortest in the series. Packed with terminology and suggestions, they can overwhelm experienced practitioners and mislead novices into thinking they have How To Guide for success.
Innovation is contextual. Its strategies, priorities, and metrics must align with the broader organizational goals. Using the standards as a mere checklist is more likely to lead to wasted time and effort building the “perfect” innovation management system while management grows increasingly frustrated by your lack of results.
The Most Important Stuff is Missing
Innovation is contextual, but there are still non-negotiables:
Leadership commitment AND active involvement: Innovation isn’t an idea problem. It’s a leadership problem. If leadership delegates innovation, fails to engage in the work, and won’t allocate required resources, you’re efforts are doomed to fail.
Adjacent and Radical Innovations require dedicated teams: Operations and innovation are fundamentally different. The former occurs in a context of known knowns and unknowns, where experience and expertise rule the day. The latter is a world of unknown unknowns, where curiosity, creativity, and experimentation are required. It is not reasonable to ask someone to live in both worlds simultaneously.
Innovation must not be a silo: Innovation cannot exist in a silo. Links must be maintained with the core business, as its performance directly impacts available resources and influences the direction of innovation initiatives.
These essential elements are mentioned in the standards but are not clearly identified. Their omission increases the risk of further innovation failures.
Something is better than nothing
The standards aren’t perfect. But one of the core principles of innovation is to never let perfection get in the way of progress.
Now it’s time to practice what we preach by testing the standards in the real world, scrapping what doesn’t work, embracing what does, and innovating and iterating our way to better.
Doing nothing fuels creativity and innovation, but that fuel is wasted if you don’t put it to use. Idleness clears the mind, allowing fresh ideas to emerge, but those ideas must be acted upon to create value.
Why is doing something with that fuel so difficult?
Don’t blame the status quo.
The moment we get thrown back into the topsy-turvy, deadline-driven, politics-navigating, schedule-juggling humdrum of everyday life, we slide back into old habits and routines. The status quo is a well-known foe, so it’s tempting to blame it for our lack of action.
But it’s not stopping us from taking the first step.
We’re stopping ourselves.
Blame one (or more) of these.
Last week, I stumbled upon this image from the Near Future Laboratory, based on a theory from psychologist Mihaly Csikszentmihalyi’s book Flow:
There’s a lot going on here, but four things jumped out at me:
When we don’t have the skills needed to do something challenging, we feel anxiety
When we don’t feel challenged because our skills exceed the task, we feel boredom
When we don’t feel challenged and we don’t have the skills, we feel apathy
When we have the skills and feel challenged, we are in flow
Four different states. Only one of them is positive.
I don’t love those odds.
Yet we live them every day.
Every day, in every activity and interaction, we dance in and through these stages. Anxiety when given a new project and doubt that we have what it takes. Boredom when asked to explain something for the 82nd time to a new colleague and nostalgia for when people stayed in jobs longer or spent time figuring things out for themselves. Sometimes, we get lucky and find ourselves in a Flow State, where our skills perfectly match the challenge, and we lose track of space and time as we explore and create. Sometimes, we are mired in apathy.
Round and round we go.
The same is true when we have a creative or innovative idea. We have creative thoughts, but the challenge seems too great, so we get nervous, doubt our abilities, and never speak up. We have an innovative idea, but we don’t think management will understand, let alone approve it, so we keep it to ourselves.
Anxiety. Boredom. Apathy.
One (or more) of these tells you that your creative thoughts are crazy and your innovative ideas are wild. They tell you that none of them are ready to be presented to your boss with a multi-million-dollar funding request. In fact, none of them should be shared with anyone, lest they think you, not your idea, is crazy.
Then overcome them
I’m not going to tell you not to feel anxiety, boredom, or apathy. I feel all three of those every day.
I am telling you not to get stuck there.
Yes, all the things anxiety, boredom, and apathy tell you about your crazy thoughts and innovative ideas may be true. AND it may also be true that there’s a spark of genius in your crazy thoughts and truly disruptive thinking in your innovative ideas. But you won’t know if you don’t act:
When you feel anxious, ask a friend, mentor, or trusted colleague if the challenge is as big as it seems or if you have the skills to take it on.
When you feel bored, find a new challenge
When you feel apathetic, change everything
Your thoughts and ideas are valuable. Without them, nothing changes, and nothing gets better.
You are a natural-born problem solver. From the moment you were born, you’ve solved problems. Hungry? Start crying. Learning to walk? Stand up, take a step, fall over, repeat. Want to grow your business? Fall in love with a problem, then solve it more delightfully than anyone else.
Did you notice the slight shift in how you solve problems?
Initially, you solved problems on your own. As communication became easier, you started working with others. Now, you instinctively collaborate to solve complex problems, assembling teams to tackle challenges together.
But research indicates your instincts are wrong. In fact, while collaboration can be beneficial for gathering information, it hinders the process of developing innovative solutions. This counterintuitive finding has significant implications for how teams approach problem-solving.
What a Terrorism Study Reveals About Your Team
In a 2015 study, researchers used a simulation developed by the U.S. Department of Defense to examine how collaboration impacts the problem-solving process. 417 undergrads were randomly assigned to 16-person teams with varying levels of “interconnectedness” (clarity in their team structure and information-sharing permissions) and asked to solve aspects of an imaginary terrorist attack scenario, such as identifying the perpetrators and target. Teams had 25 minutes to tackle the problem, with monetary incentives for solving it quickly.
Highly interconnected teams “gathered 5 percent more information than the least-clustered groups because clustering prevented network members from unknowingly conducting duplicative searches. ‘By being in a cluster, individuals tended to contribute more to the collective exploration through information space—not from more search but rather by being more coordinated in their search,’”
The Least Interconnected teams developed 17.5% more theories and solutions and were more likely to develop the correct solution because they were less likely to “copy an incorrect theory from a neighbor.”
How You Can Help Your Team Create More Successful Solutions
You and your team rarely face problems as dire as terrorist attacks, but you can use these results to adapt your problem-solving practices and improve results.
Work together to gather and share information. This goes beyond emailing around research reports, interview summaries, and meeting notes. “Working together” requires your team to take action, like conducting interviews or writing surveys, with one another in real-time (not asynchronously through email, text, or “collaboration” platforms).
Start solving the problem alone. For example, at the start of every ideation session, I ask people to spend 5 minutes privately jotting down their ideas before group brainstorming. This prevents copying others’ theories and ensures all voices are heard. (not just the loudest or most senior)
Invite the “Unusual Suspects” into the process. Most executives know that diversity amplifies creativity, so they invite a mix of genders, ages, races, ethnicities, tenures, and industry experiences to brainstorming sessions. While that’s great, it also results in the same people being invited to every brainstorm and, ultimately, creating a highly interconnected group. So, mix it up even more. Invite people never before invited to brainstorming into the process. Instead of spending a day brainstorming, break it up into one-hour bursts at different times of the day.
Are You Willing to Take the Risk?
For most of your working life, collaboration has been the default approach to problem-solving. However, this research suggests that rethinking when and how to leverage collaboration can lead to greater success.
Making such a change isn’t easy – it invites skepticism and judgment as it deviates from the proven “status quo” process.
Are you willing to take that risk, separating information gathering from solution development, for the potential of achieving better, more innovative outcomes? Or will you remain content with “good enough” solutions from conventional methods?
“Time is a flat circle. Everything we have done or will do we will do over and over and over and over again – forever.”
– Rusty Cohle, played by Matthew McConaughey, in True Detective
For the whole of human existence, we have created new things with no idea if, when, or how they will affect humanity, society, or business. New things can be a distraction, sucking up time and money and offering nothing in return. Or they can be a bridge to a better future.
As a leader, it’s your job to figure out which things are a bridge (i.e., innovation) and which things suck (i.e., shiny objects).
Innovation is a flat circle
The concept of eternal recurrence, that time repeats itself in an infinite loop, was first taught by Pythagoras (of Pythagorean theorem fame) in the 6th century BC. It remerged (thereby proving its own truth) in Friedreich Nietzsche’s writings in the 19th century, then again in 2014’s first season of True Detective, and then again on Monday in Jamie Dimon’s Annual Letter to Shareholders.
Mr. Dimon, the CEO and Chairman of JPMorgan Chase & Co, first mentioned AI in his 2017 Letter to Shareholders. So, it wasn’t the mention of AI that was newsworthy. It was how it was mentioned. Before mentioning geopolitical risks, regulatory issues, or the recent acquisition of First Republic, Mr. Dimon spends nineparagraphs talking about AI, its impact on banking, and how JPMorgan Chase is responding.
Here’s a screenshot of the first two paragraphs:
He’s right. We don’t know “the full effect or the precise rate at which AI will change our business—or how it will affect society at large.” We were similarly clueless in 1436 (when the printing press was invented), 1712 (when the first commercially successful steam engine was invented), 1882 (when electricity was first commercially distributed), and 1993 (when the World Wide Web was released to the public).
Innovation, it seems, is also a flat circle.
Our response doesn’t have to be.
Historically, people responded to innovation in one of two ways: panic because it’s a sign of the apocalypse or rejoice because it will be our salvation. And those reactions aren’t confined to just “transformational” innovations. In 2015, a visiting professor at Kings College London declared that the humble eraser (1770) was “an instrument of the devil” because it creates “a culture of shame about error. It’s a way of lying to the world, which says, ‘I didn’t make a mistake. I got it right the first time.’”
Neither reaction is true. Fortunately, as time passes, more people recognize that the truth is somewhere between the apocalypse and salvation and that we can influence what that “between” place is through intentional experimentation and learning.
JPMorgan started experimenting with AI over a decade ago, well before most of its competitors. As a result, they “now have over 400 use cases in production in areas such as marketing, fraud, and risk” that are producing quantifiable financial value for the company.
It’s not just JPMorgan. Organizations as varied as John Deere, BMW, Amazon, the US Department of Energy, Vanguard, and Johns Hopkins Hospital have been experimenting with AI for years, trying to understand if and how it could improve their operations and enable them to serve customers better. Some experiments worked. Some didn’t. But every company brave enough to try learned something and, as a result, got smarter and more confident about “the full effect or the precise rate at which AI will change our business.”
You have free will. Use it to learn.
Cynics believe that time is a flat circle. Leaders believe it is an ever-ascending spiral, one in which we can learn, evolve, and influence what’s next. They also have the courage to act on (and invest in) that belief.
What do you believe? More importantly, what are you doing about it?
AI is NOT a substitute for person-to-person discovery conversations or Jobs to be Done interviews.
But it is a freakin’ fantastic place to start…if you do the work before you start.
Get smart about what’s possible
When ChatGPT debuted, I had a lot of fun playing with it, but never once worried that it would replace qualitative research. Deep insights, social and emotional Jobs to be Done, and game-changing surprises only ever emerge through personal conversation. No matter how good the Large Language Model (LLM) is, it can’t tell you how feelings, aspirations, and motivations drive their decisions.
Then I watched JTBD Untangled’s video with Evan Shore, WalMart’s Senior Director of Product for Health & Wellness, sharing the tests, prompts, and results his team used to compare insights from AI and traditional research approaches.
In a few hours, he generated 80% of the insights that took nine months to gather using traditional methods.
Get clear about what you want and need.
Before getting sucked into the latest shiny AI tools, get clear about what you expect the tool to do for you. For example:
Provide a starting point for research: I used the free version of ChatGPT to build JTBD Canvas 2.0 for four distinct consumer personas. The results weren’t great, but they provided a helpful starting point. I also like Perplexity because even the free version links to sources.
Conduct qualitative research for me: I haven’t used it yet, but a trusted colleague recommended Outset.ai, a service that promises to get to the Why behind the What because of its ability to “conduct and synthesize video, audio, and text conversations.”
Synthesize my research and identify insights: An AI platform built explicitly for Jobs to be Done Research? Yes, please! That’s precisely what JobLens claims to be, and while I haven’t used it in a live research project, I’ve been impressed by the results of my experiments. For non-JTBD research, Otter.ai is the original and still my favorite tool for recording, live transcription, and AI-generated summaries and key takeaways.
Visualize insights:Mural, Miro, and FigJam are the most widely known and used collaborative whiteboards, all offering hundreds of pre-formatted templates for personas, journey maps, and other consumer research templates. Another colleague recently sang the praises of theydo, an AI tool designed specifically for customer journey mapping.
Practice your prompts
“Garbage in. Garbage out.” Has never been truer than with AI. Your prompts determine the accuracy and richness of the insights you’ll get, so don’t wait until you’ve started researching to hone them. If you want to start from scratch, you can learn how to write super-effective prompts here and here. If you’d rather build on someone else’s work, Brian at JobsLens has great prompt resources.
Spend time testing and refining your prompts by using a previous project as a starting point. Because you know what the output should be (or at least the output you got), you can keep refining until you get a prompt that returns what you expect. It can take hours, days, or even weeks to craft effective prompts, but once you have them, you can re-use them for future projects.
Defend your budget
Using AI for customer research will save you time and money, but it is not free. It’s also not just the cost of the subscription or license for your chosen tool(s).
Remember the 80% of insights that AI surfaced in the JTBD Untangled video? The other 20% of insights came solely from in-person conversations but comprised almost 100% of the insights that inspired innovative products and services.
AI can only tell you what everyone already knows. You need to discover what no one knows, but everyone feels. That still takes time, money, and the ability to connect with humans.
Run small experiments before making big promises
People react to change differently. Some will love the idea of using AI for customer research, while others will resist with. Everyone, however, will pounce on any evidence that they’re right. So be prepared. Take advantage of free trials to play with tools. Test tools on friends, family, and colleagues. Then underpromise and overdeliver.
AI is a starting point. It is not the ending point.
I’m curious, have you tried using AI for customer research? What tools have you tried? Which ones do you recommend?
