Case Study: How Alpha enables teams to make smarter product decisions more cost-effectively


How frequently do product managers run experiments and ship new features?

by Sam Henick in Best Practices
August 2, 2017

When it comes to building digital products, there are essentially two ways to get feedback and iterate. You can structure an experiment to prototype and test a product concept with prospective users, or you can measure how a product performs after launching it. Most product managers do some combination of each, and new data sheds light onto why.

In our recent survey of 140 digital product managers, we evaluated how frequently respondents develop new features and run experiments. Steve Blank, one of the pioneers of the ‘Lean’ movement, wrote bluntly in Harvard Business Review about the value of balancing both activities:

Business plans rarely survive first contact with customers. As the boxer Mike Tyson once said about his opponents’ prefight strategies: “Everybody has a plan until they get punched in the mouth.”

Given the uncertainty surrounding the innovation process, product teams consider the costs and risks associated with each product assumption, and make a decision about when to get ‘punched in the face.’ They can do so earlier with an experiment, or later on post-launch.

In a world in which every single feature idea undergoes some validation exercise before being added to the roadmap, you would expect somewhere near a 1:1 relationship between frequency of code releases and experiments. That’s clearly not the case – nor should it necessarily be the case – which means that product teams are prioritizing assumptions to test.

Test it or ship it?

Product teams often go through some sort of explicit or passive triaging exercise, in which they experiment with assumptions that pose the greatest risk and can be tested most efficiently. But when we dug deeper into the data, we uncovered some interesting nuances.

First, we learned that smaller companies release features more frequently than larger companies, which isn’t entirely unexpected:

Smaller companies generally have fewer dependencies, fewer regulatory or organizational bottlenecks, and leaner roadmaps. By that same rationale though, you’d expect larger organizations to experiment more frequently than smaller companies, as an added measure to mitigate the risk of new product development.

What’s interesting though is that such a tradeoff isn’t crystal clear in the data. In fact, we actually can’t determine any correlation at all because of the sudden spike in the percentage of respondents that report having no recurring experimentation frequency as company size gets larger:

Essentially, smaller companies release features more frequently and experiment on a recurring basis, while larger companies release features less frequently and are far more likely to experiment on an ad hoc basis. We don’t know whether large companies experiment more (as we hypothesized) or less. So what gives?

Doing more with more

Perhaps ‘experiment’ is a loaded term meaning different things to different people. It could be that what a small company considers to be an experiment is wildly different than what a large company considers to be an experiment.

Experiments are used as an incremental cost to increase the overall ROI of a project. If a large company spends considerably more to experiment, that adversely impacts the ROI of experimentation, and could mean less frequent efforts. And that appears to be the case:

Moreover, when we investigate the typical makeup of an experiment, more differences begin to emerge. By spending more, larger organizations typically experiment against larger audiences. In terms of techniques, larger companies are nearly twice as likely as smaller companies to emphasize focus groups, which typically require costly third-party agencies to execute.

The fact that larger organizations run more robust experiments shouldn’t come as a surprise, given the layers of scrutiny and consensus building necessary to make decisions in such environments. It also may explain the prevalence of ad hoc experimentation, as more elaborate testing requires more planning and coordination.

Moving forward

The future will be won by customer-centric companies that are the quickest to deliver highly desired solutions to the market. Given internal constraints and checkpoints, it’s unlikely that large organizations will be able to ship new features more frequently. In order to compete with nimbler startups then, product teams at such companies need to be less reactive and instead focus on establishing a recurring and more efficient workflow for experimentation.

New technologies and methodologies offer perhaps the largest opportunity to level up. Fortune 500 organizations that leverage Alpha’s platform typically accelerate their experimentation processes by at least 10x, while simultaneously accessing larger audiences and expanding their use of qualitative and quantitative research techniques. Download our full report for even more insights into product management activities such as roadmapping, reporting, and collaboration.

Sam Henick

Sam Henick is a senior at Columbia University and is a quantitative analyst intern at Alpha. He has worked at prestigious firms such as CATO and FiveThirtyEight.

Get the latest updates from Alpha

Subscribe now to get notified about our latest posts, events, features for product teams.