I Sent a Survey to AI, and the Results were Brilliant… and Dangerous
AI is everywhere: in our workplaces, homes, schools, art galleries, concert halls, and even neighborhood coffee shops. We can’t seem to escape it. Some hope it will unlock our full potential and usher in an era of creativity, prosperity, and peace. Others worry it will eventually replace us. While both outcomes are extreme, if you’ve ever used AI to conduct research with synthetic users, the idea of being “replaced” isn’t so wild.
For the past month, I’ve beta-tested Crowdwave, an AI research tool that allows you to create surveys, specify segments of respondents, send the survey to synthetic respondents (AI-generated personas), and get results within minutes.
Sound too good to be true?
Here are the results from my initial test:
- 150 respondents in 3 niche segments (50 respondents each)
- 51 questions, including ten open-ended questions requiring short prose responses
- 1 hour to complete and generate an AI executive summary and full data set of individual responses, enabling further analysis
The Tool is Brilliant
It took just one hour to gather data that traditional survey methods require a month or more to collect, clean, and synthesize. Think of how much time you’ve spent waiting for survey results, checking interim data, and cleaning up messy responses. I certainly did and it made me cry.
The qualitative responses were on-topic, useful, and featured enough quirks to seem somewhat human. I’m pretty sure that has never happened in the history of surveys. Typically, respondents skip open-ended questions or use them to air unrelated opinions.
Every respondent completed the entire survey! There is no need to look for respondents who went too quickly, chose the same option repeatedly, or abandoned the effort altogether. You no longer need to spend hours cleaning data, weeding out partial responses, and hoping you’re left with enough that you can generate statistically significant findings.
The Results are Dangerous
When I presented the results to my client, complete with caveats about AI’s limitations and the tool’s early-stage development, they did what any reasonable person would do – they started making decisions based on the survey results.
STOP!
As humans, we want to solve problems. In business, we are rewarded for solving problems. So, when we see something that looks like a solution, we jump at it.
However, strategic or financially significant decisions should never rely ona single data source. They are too complex, risky, and costly. And they definitely shouldn’t be made based on fake people’s answers to survey questions!
They’re Also Useful.
Although the synthetic respondents’ data may not be true, it is probably directionally correct because it is based on millions and maybe billions of data points. So, while you shouldn’t make pricing decisions based on data showing that 40% of your target consumers are willing to pay a 30%+ premium for your product, it’s reasonable to believe they may be willing to pay more for your product.
The ability to field an absurdly long survey was also valuable. My client is not unusual in their desire to ask everything they may ever need to know for fear that they won’t have another chance to gather quantitative data (and budgets being what they are, they’re usually right). They often ignore warnings that long surveys lead to abandonment and declining response quality. With AI, we could ask all the questions and then identify the most critical ones for follow-up surveys sent to actual humans.
We Aren’t Being Replaced, We’re Being Spared
AI consumer research won’t replace humans. But it will spare us the drudgery of long surveys filled with useless questions, months of waiting for results, and weeks of data cleaning and analysis. It may just free us up to be creative and spend time with other humans. And that is brilliant.
ISO Innovation Standards: The Good, the Bad, and the Missing
In 2020, the International Standards Organization, most famous for its Quality Management Systems standard, published ISO 56000, Innovation Management—Fundamentals and Vocabulary. Since then, ISO has released eight additional innovation standards.
But is it possible to create international standards for innovation, or are we killing creativity?
That’s the question that InnoLead founder and CEO Scott Kirsner and I debated over lunch a few weeks ago. Although we had heard of the standards and attended a few webinars, but we had never read them or spoken with corporate innovators about their experiences.
So, we set out to fix that.
Scott convened an all-star panel of innovators from Entergy, Black & Veatch, DFW Airport, Cisco, and a large financial institution to read and discuss two ISO Innovation Standards: ISO 56002, Innovation management – Innovation management systems – Requirements and ISO 56004, Innovation Management Assessment – Guidance.
The conversation was honest, featured a wide range of opinions, and is absolutely worth your time to watch.
Here are my three biggest takeaways.
The Standards are a Good Idea
Innovation doesn’t have the best reputation. It’s frequently treated as a hobby to be pursued when times are good and sometimes as a management boondoggle to justify pursuing pet ideas and taking field trips to fun places.
However, ISO Standards can change how innovation is perceived and supported.
Just as ISO’s Quality Management Standards established a framework for quality, the Innovation Management Standards aim to do the same for innovation. They provide shared fundamentals and a common vocabulary (ISO 56000), requirements for innovation management systems (ISO 56001 and ISO 56002), and guidance for measurement (ISO 56004), intellectual property management (ISO 56005), and partnerships (ISO 56003). By establishing these standards, organizations can transition innovation from a vague “trust me” proposition to a structured, best-practice approach.
The Documents are Dangerous
However, there’s a caveat: a little knowledge can be dangerous. The two standards I reviewed were dense and complex, totaling 56 pages, and they’re among the shortest in the series. Packed with terminology and suggestions, they can overwhelm experienced practitioners and mislead novices into thinking they have How To Guide for success.
Innovation is contextual. Its strategies, priorities, and metrics must align with the broader organizational goals. Using the standards as a mere checklist is more likely to lead to wasted time and effort building the “perfect” innovation management system while management grows increasingly frustrated by your lack of results.
The Most Important Stuff is Missing
Innovation is contextual, but there are still non-negotiables:
- Leadership commitment AND active involvement: Innovation isn’t an idea problem. It’s a leadership problem. If leadership delegates innovation, fails to engage in the work, and won’t allocate required resources, you’re efforts are doomed to fail.
- Adjacent and Radical Innovations require dedicated teams: Operations and innovation are fundamentally different. The former occurs in a context of known knowns and unknowns, where experience and expertise rule the day. The latter is a world of unknown unknowns, where curiosity, creativity, and experimentation are required. It is not reasonable to ask someone to live in both worlds simultaneously.
- Innovation must not be a silo: Innovation cannot exist in a silo. Links must be maintained with the core business, as its performance directly impacts available resources and influences the direction of innovation initiatives.
These essential elements are mentioned in the standards but are not clearly identified. Their omission increases the risk of further innovation failures.
Something is better than nothing
The standards aren’t perfect. But one of the core principles of innovation is to never let perfection get in the way of progress.
Now it’s time to practice what we preach by testing the standards in the real world, scrapping what doesn’t work, embracing what does, and innovating and iterating our way to better.
The Cost of Silence: Why Neglecting New Hires’ Ideas Hurts Revenue
Stop me if this sounds familiar. A new hire bounces into your office and, with all the joy and enthusiasm of a new puppy, rattles off a list of ideas. You smile and, just like with new puppies, explain why their ideas won’t work, and encourage them to be patient and get to know the organization.
Congratulations! You just cost your company money. Not because the new hire’s idea was the silver bullet you’ve been seeking but because you taught them that it’s more critical for them to do their jobs and maintain the status quo than to ask questions and share ideas.
If that seems harsh, read the new research from Harvard Business School professor Amy Edmondson.
Year 1: Rainbows and Unicorns (mostly)
From 2017 through 2021, Dr. Edmonson and her colleagues collected data from over 10,000 physicians. Using biannual (every two years) surveys, they asked physicians to rate on a 5-point scale how comfortable they felt offering opinions or calling out the mistakes of colleagues or superiors.
It was little surprise that agreement with statements like “I can report patient safety mistakes without fear of punishment” were highest amongst people with less than one year of service at their employer.
These results all come down to one thing: high levels of psychological safety.
Years 2+: Resignation and Unhappiness
However, psychological safety erodes quickly in the first year because:
- There’s a gap between words and actions: When new hires join an organization, they believe what they hear about its culture, values, priorities, and openness. Once they’re in the organization and observe their colleagues’ and superiors’ daily behavior, they experience the disconnect, lose trust, and shift into self-protection mode.
- Their feedback and ideas are rebuffed: This scenario is described above, but it’s not the only one. Another common situation occurs when a new hire responds to requests for feedback only to be met with silence or exasperation, a lack of follow-through or follow-up, or is openly mocked or met with harsh pushback
- Expectations increase with experience: It’s easier to ask questions when you’re new, and no one expects you to know the answers. Over time, however, you are expected to learn the answers and you no longer feel comfortable asking questions, even if there’s no way you could know the answer.
20 years to regain what was lost in 1
According to Edmondson’s research, it takes up to 20 years to rebuild the safety lost in the first year.
As a leader, you can slow that erosion and accelerate the rebuilding when you:
- Recognize the Risk: Knowing that new hires will experience a drop in psychological safety, staff them on teams that have higher levels of safety
- Walk the Talk: Double down on demonstrating the behaviors you want. Immediately act on feedback that points out a gap between your words and actions.
- Ask questions: Demonstrate your openness by being curious, asking questions, and asking follow-up questions. As Edmonson writes, “You are training people to contribute by constantly asking questions.”
- Promises Made = Promises Kept: If you ask for feedback, act on it. If you ask for ideas, act on some and explain why you’re not executing others.
- Be Vulnerable: Admit your mistakes and uncertainties. It sets a powerful example that it’s okay to be imperfect and to ask for help. It also creates an environment for others to do the same.
The Cost of Silence vs. The Cost of Time
Building and maintaining psychological safety takes time and effort. It takes 5 minutes to listen to and respond to an idea. It takes hours to ensure new hires join safe teams. It takes weeks to plan and secure support for post-hackathon ideas.
But how does that compare to 20 years of lost ideas, improvements, innovations, and revenue? To 20 years of lost collaboration, productivity, and peak effectiveness? To 20 years of slow progress, inefficiency, and cost?
How many of your employees stick around 20 years to give you the chance to rebuild what was lost?
Reality Strikes Back: How to Build Innovation Resilience in Uncertainty
“This time feels different.” I’ve been hearing this from innovation practitioners and partners for months We’ve seen innovation resilience tested in times of economic uncertainty and geopolitical volatility. We’ve seen it flourish when markets soar and capital is abundant. We’ve seen it all, but this time feels different.
In fact, we feel a great disturbance in the innovation force.
Disturbances aren’t always bad. They’re often the spark that ignites innovation. But understand the disturbance you must, before work with it you can.
So, to help us understand and navigate a time that feels, and likely is, different, I present “The Corporate Innovator’s Saga.”
Episode I: The R&D Men (are) Aces
(Sorry, that’s the most tortured one. The titles get better, I promise)
A long time ago (1876), in a place not so far away (New Jersey), one man established what many consider the first R&D Lab. A year later, Thomas Edison and his Menlo Park colleagues debuted the phonograph.
In the 20th century, as technology became more complex, invention shifted from individual inventors to corporate R&D labs. By the late 1960s, Bell Labs employed 15,000 people, including 1,200 PhDs. In 1970, Xerox’s famed Palo Alto Research Center (PARC) opened.
Episode II: Attack of the Disruptors
For most of the twentieth century, R&D labs were the heroes or villains of executives’ innovation stories. Then, Harvard Business School professor Clayton Christensen published, The Innovator’s Dilemma: When New Technologies Cause Great Firms to Fail.
He revealed that executives’ myopic focus on serving their best (most profitable) customers caused them to miss new waves of innovation. In example after example, he showed that R&D often worked on disruptive (cheaper, good enough) technologies only to have their efforts shut down by executives worried about cannibalizing their existing businesses.
C-suites listened, and innovation went from an R&D problem to a business one.
Episode III: Revenge of the Designers
Design Thinking’s origins date back to the 1940s, its application to business gained prominence with l Tim Brown’s 2009 book, Change by Design: How Design Thinking Transforms Organizations and Inspires Innovation.
This book introduced frameworks still used today’s: desirability, feasibility, and viability; divergent and convergent thinking; and the process of empathy, problem definition, ideation, prototyping, and testing.
Innovation now required businesspeople to become designers, question the status quo, and operate untethered from the short-termism of business,
Episode IV: A New Hope (Startups)
The early 2000s were a dizzying time for corporate innovation. Executives feared disruption and poured resources into internal innovation teams and trainings. Meanwhile, a movement was gaining steam in Silicon Valley.
Y Combinator, the first seed accelerator, launched in 2005 and was followed a year later by TechStars. When Eric Ries published The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses in 2011, the US was home to nearly 1o0 startup accelerators.
Now, businesspeople needed to become entrepreneurs capable of building, and scaling startups in environments purpose-built to kill risk and change.
In response, companies spun up internal accelerators, established corporate venture capital teams, and partnered with startup studios.
Episode V: Reality Strikes Back
Today, the combination of a global pandemic, regional wars, and a single year in which elections will affect 49% of the world’s population has everyone reeling.
Naturally, this uncertainty triggered out need for a sense of control. The first cut were “hobbies” like innovation and DEI. Then, “non-essentials” like “extra” people and perks. For losses continued into the “need to haves,” like operational investments and business expansion.
Recently, the idea of “growth at all costs” has come under scrutiny with advocates for more thoughtful growth strategies emerging There is still room for innovation IF it produces meaningful, measurable value.
Episode VI: Return of the Innovator(?)
I don’t know what’s next, but I hope this is the title. And, if not, I hope whatever is next has Ewoks.
What do you hope for in the next episode?