See the results of our survey of 300+ innovators on decision making best practices

Read Now

How do market researchers add experimentation to drive better business outcomes?

by Basil Shariff
July 5, 2018

Incumbents across every industry have been put on notice: technology upstarts are taking over the Fortune 500 list at an accelerating pace. No one wants to be the next Blockbuster. But the ability to avoid that fate won’t come from open workspaces, a relaxed dress code, and a bring-your-own-device policy. It requires organizations, from analyst to executive, to internalize the fast-paced world we live in and to adopt modern workflows that enable them to compete.

Look no further than Amazon – in a 2017 letter to shareholders, CEO Jeff Bezos, credited a decision-making model for the company’s ability to deliver value to the market faster:

“Most decisions should probably be made with somewhere around 70% of the information you wish you had. If you wait for 90%, in most cases, you’re probably being slow. Plus, either way, you need to be good at quickly recognizing and correcting bad decisions. If you’re good at course correcting, being wrong may be less costly than you think, whereas being slow is going to be expensive for sure.”

So how do they quickly generate 70% of the information they need to make a decision? “Amazon has a point of view that’s deeply embedded in the company,” Investor’s Business Daily explains. “It will win with the customer by doing a series of small bets that give it insight on how to build that long-term customer relationship.”

Make small bets, via experimentation

Whereas many organizations have deep and meaningful expertise doing large-scale market research, their findings are hitting critical limitations. As someone who did market research for a decade, the last half while at Toyota, I saw this first-hand.

Product teams responsible for introducing new solutions to the market are embracing agile workflows and need to engage in higher-velocity decision-making to inform recurring code sprints. They can’t wait for large-scale studies to be completed and so are left making decisions based on non-contextualized data from past reports at best or gut feelings at worst. In an effort to get closer to Amazon’s benchmark of having 70% of the information needed, some product managers even go to nearby Starbucks to generate feedback from random patrons. It’s not a rigorous practice by any means.

In short, market research is extraordinarily effective for sensing broad marketplace changes and informing the organization about what markets to enter and what go-to-market strategies to deploy. But it simply can’t inform the day-to-day decisions that product teams need to make.

Small bets, via experimentation introduces data to these daily decisions. Designed specifically for continuous iterations, experimentation informs product teams with directional insight into distinct user preferences and behaviors.

Whereas market research can be used for sophisticated initiatives like market segmentation and conjoint analysis studies, experimentation involves more tactical roadmap prioritization activities like concept split tests and usability assessments. Integrated together, market research and experimentation can enable organizations to allocate resources intelligently and efficiently.

Achieve better business outcomes

At Alpha, we offer a platform that drives rapid experimentation for product teams at more than one-third of the Fortune 100, and thus have a front row seat to how organizations are integrating experimentation into their product development lifecycles. They utilize varying workflows, as I’ll describe shortly, but have all seen impressive results:

LendingTree calculated that they are making smarter product decisions 8x more cost-effectively. DE Shaw incubated a startup and validated an initial offering 3x faster than usual. Aetna dramatically increased usability and user engagement. Each increased meaningful metrics, from conversion rates to NPS to revenue.

I think this is a particularly important quote from our podcast interview with Alex Gruber, Director of Product at Vonage:

“Because we can solve many different problems many different ways, but there’s only so many hours in the day, so you have to pick the ones that your customers are feeling today and that are not being solved by other solutions. And finding those are easier when you crowdsource that versus trying to talk to 10 or 20 people. With Alpha you can talk to thousands at a time which is not practical just going to Starbucks – you’d be pretty well caffeinated at that point.”

As noted earlier, this is a critical capability that I felt was lacking in many years doing market research. After delivering world-class insights and analysis to the organization, we would regularly find product roadmaps to be lists of unvalidated pet projects or hunches.

Best practices for running experiments

When it comes to adding experimentation to market research, the biggest mistake you can make is not using each to its strengths and weaknesses. Misusing data can be more dangerous than operating without data. Remember that experimentation is an iterative testing methodology that provides directional insight, whereas market research consists of study-based methodologies to provide statistically significant results.

The quality of the insights generated is directly correlated to the quality and rigor of your methodology. With regard to experimentation, you should ensure that:

  • You are eliminating bias from psychological distance by generating revealed preferences (i.e. authentic reactions) to stimuli rather than stated preferences (i.e. zero-risk answers) to survey questions
  • You are running split tests so that your insights are comparative (e.g. Feature A vs. Feature B) rather than independent (e.g. reactions to just Feature A), especially with smaller samples
  • You are sourcing an audience based on behaviors and problems, and not on preconceived notions of how certain demographics map to preferences

For further reading, we collaborated with academic researchers to develop a broad overview of how to determine when consumer feedback is actionable. Done right, you can combine experimentation with market research to achieve a powerful trifecta:

I also strongly recommend that you listen to our podcast episode (also embedded below for your convenience) with Gregg Archibald and Leonard Murphy, Partners at Gen2 Advisors and GreenBook. In the interview, they share what product managers need to know about market research, how product managers and researchers can collaborate more effectively, and how technology is reshaping the field.

Different models for process ownership

As with most things in business and life, there’s no one-size-fits-all approach. I’ve seen three general models work for adding experimentation to your company’s arsenal of customer-focused initiatives:

  • Product teams have direct access to frameworks, tools, and platforms to run experiments. This works really well when your teams are trained to understand data, empowered to make decisions, and need to move quickly in an emerging or fast-paced market.
  • Research teams incorporate their principles and best practices into the frameworks, tools, and platforms that product teams use. This works really well when research teams communicate the strengths and weaknesses of different methodologies and are aligned with product teams’ objectives and workflows.
  • Research teams act as a hub for running experiments, using their own frameworks, tools, and platforms entirely. This works really well if research teams have the resources and bandwidth to handle and prioritize all the inbound questions and hypotheses.

At Alpha, we offer the flexibility to operate using any of the above models. Hundreds of product, innovation, and research teams run tens of thousands of experiments on our platform every year. I hear time and time again that this is the platform they wish they had for the last decade, and I couldn’t agree more (which is why I’m here)! If you want to learn more about bringing on-demand insights to your organization, schedule a demo below.

Basil Shariff

Basil Shariff is a customer success manager at Alpha, the on-demand insights platform enabling teams to make data-driven decisions about users, products, and new markets.

Get the latest updates from Alpha

Subscribe now to get notified about our latest posts, events, features for product teams.