Case Study: How Alpha enables teams to make smarter product decisions more cost-effectively

Read

How do you define and source an audience for user testing?

by Claire Gerson
December 20, 2017

Continually sourcing users is one of the core challenges for product teams that want to rapidly experiment and validate product decisions. To be effective in most enterprise contexts, you’ll need procedures to mitigate bias and a streamlined technology in order to define and source an audience for ongoing testing.

For product teams just beginning to make the shift to data-driven practices, it’s common to simply get feedback from family, friends, or patrons of a nearby Starbucks. While better than nothing, this technique suffers from extreme bias and can’t be streamlined throughout an entire organization.

More sophisticated teams incorporate feedback checkpoints and surveys into their products, in order to generate insights from existing users. We heavily recommend this approach as it adds significant context beyond what a tool like Google Analytics or Mixpanel can offer.

In many environments however, product teams are unable to generate feedback from existing users. Consider the following common scenarios:

  • In a B2B2C environment, you may not have access to the end user, but are responsible for proving to customers that your product is valuable to end users
  • In a heavily regulated environment, laws may prevent you from collecting data from individuals without compliance teams reviewing and slowing down each step of the process
  • In a high-touch B2B environment, sales teams may closely control customer relationships, and prevent you from easily accessing customers for feedback
  • If you work for a premium brand, you may not want to expose new and sensitive product concepts to existing users, and would prefer to use a lookalike audience

In addition to the examples above, advanced product teams recognize the value in generating insights from new markets and non-users (where there is often the greatest opportunity for learning and revenue). To reach such audiences, you can run ads on sites like Craigslist, LinkedIn, and Facebook. For anything beyond ad hoc testing, you will be able to move much faster and seamlessly by working with an integrated experimentation technology like Alpha.

Reaching your target audience for rapid experimentation

Alpha leverages technology and scale to programmatically source highly targeted audiences quickly and efficiently. We combine panel vendors integrations with our proprietary sourcing techniques to deliver a capability for fast moving product teams to experiment and iterate daily. For Alpha users, it’s as simple as defining your audience in plain-English.

Inputs: Navigate to the bottom of an experiment and click ‘Add Audience.’ When defining your audience, it’s better to think less about ‘personas’ and more about ‘jobs-to-be-done.’ By defining an audience in terms of past experiences, behaviors, and perceptions instead of by demographic factors, you increase the accuracy of your audience and the value of the insights generated. (You can read more about this difference here.)

For example, a bad audience definition would be highly demographic-focused and prescriptive, like:

San Francisco-based couples with newborn child and a FICO score between 640 and 680.

Instead of guessing at a persona that you think represents a target audience, define the necessary activities and knowledge that would make someone qualified to share their feedback. Here’s a good audience definition for the same example:

Wells Fargo customers thinking of purchasing a starter home who will most likely qualify for a low-interest rate mortgage.

Remember, a customer success manager will always review the objective of an experiment to ensure that your audience is well-suited to generate actionable insights. But to speed up the process, it’s best to be proactive about defining appropriate audiences. All audiences can then be accessed via a dropdown for all future tests within that experiment.

Within our testing environment, additional measures filter out respondents who answer questions too quickly or copy and paste responses. Our proprietary NLP techniques analyze responses and filter out respondents who fail to establish expertise in a required area (for example, if someone who claims to be a doctor cannot offer robust answers about a medical field).

Generating on-demand insights

As your product team shifts from ad-hoc or waterfall testing to on-demand insights, the capability to programatically source audiences becomes a competitive advantage. With dynamic sample sizing and automatic segmenting, Alpha continuously ensures that insights are actionable and accessible.

Claire Gerson

Claire Gerson is a product manager at Alpha. She is responsible for the platform’s quantitative analytics solutions.

Get the latest updates from Alpha

Subscribe now to get notified about our latest posts, events, features for product teams.