What are the best quotes about experimentation from the podcast, This is Product Management?

by Benjamin Martinez in Best practices
November 21, 2016

For 18 months, This is Product Management has been a major resource for product teams across industries and company sizes. But at 75 episodes, it can be difficult to keep up. That’s why I’m cataloging some of the best quotes about different topics referenced in the series. For this first post, I highlighted the best quotes about experimentation.

“We’re in the home run business. If you’re in the product management business, you need to be swinging for the fences. You need to be hitting home runs on a consistent basis. There’s only one way to do that [and you’re going to] strike out a lot. The way to get home runs in the innovation business is to get a lot of at bats. The way to do that with limited resources is to run experiments on prototypes. Give your team the opportunity to swing for the fences and think outside of the box.” – Steven Cohn, Founder of Validately

“The way we do it is by putting [the prototype] in front of people one-on-one… Typically if it’s to understand better problems or understand if the solution is actually going to fit the problem, you want to be able to have conversations with people as they’re using the prototype and testing it out.” – Josh Wexler, Head of Product at Yieldmo

“It always starts with a hypothesis, we start with a question and through a series of prototypes, sketches, drawings, and diagrams, we start testing the idea against other ideas and also with consumers to see how those ideas resonate with consumers. We use Alpha for that a lot, every innovation cycle starts with us putting the hypothesis in the Alpha platform and getting a read on those results three days to a week later.” – Flavio Masson, Head of Innovation at AARP Life Reimagined

“Any sentence that begins with the words, ‘I believe’ or ‘we believe’, I can test that hypothesis by asking do you believe that, then your hypothesis is true. Your hypothesis should be in the structure of ‘if this then that’, there has to be some sort of causal relation. If we put up this landing page, some people will sign up, that’s not a hypothesis. A hypothesis is a causal statement and it has to be extraordinarily specific. The metric I will measure is the percentage of people that sign up and I will have a fail condition, what will convince me that the hypothesis is incorrect.” – Tristan Kromer, Lean Consultant

“First of all, it’s a focus on how our customers are making decisions. What is driving them to make decisions, what are their constraints, what are their budget needs, and what are they doing today? I think what typically happens when we go out and talk to customers is we fall prey to a lot of cognitive biases and it’s just because what our brains do. When you start asking [leading questions] you really taint the feedback you’re getting. There’s a need to learn to ask questions that are much more neutral and more focused on current behavior.” – Cindy Alvarez, Director of User Experience of Yammer, a Microsoft Company

“Before we ever do any whiteboarding or wireframes and stuff like that, I always say don’t sketch, don’t wireframe, don’t code, until you’ve talked to people. And so, I think there’s a lot of misconceptions that come from research… and I don’t think it really needs to be that complicated… I sometimes start with a simple Twitter search to see what people are saying, I go on message boards, and go to Quora to see how people might be struggling with this problem now and what are they doing now to solve that problem on their own. Then by doing that research and identifying your audience then you can go out.” – Sarah Doody, UX Consultant

“The biggest excuse is we do everything by the book, we don’t fail, and why would we take that risk for some peanut of a business compared to our enterprise, division, or product line? But from that peanut grows a peanut farm and that’s what the corporation needs to accept and strive for… The other issue we have with big corporations is very few people in the corporation are willing to take the career risk of jumping into some new internal startup team that might fail.” – Bob Dorf, Author and Entrepreneur

“Iteration is about constant improvement. When you’re iterating typically you’re either building something you think is better or you’re building something that will let you learn what might be better. And when I say building, that doesn’t necessarily mean a product or production launch of something. When you’re iterating you should be iterating on your experiments, iterating inside your product, outside your product. Basically iteration should apply to all areas of your business. I see iteration as the practical application of experimentation and continuous improvement. Iteration isn’t about building or launching its about learning and about constantly folding in what you’ve learned into a subsequent test or launch of something.” – Michael Bamberger, Alpha

Become a Product Management Insider

Subscribe now to get actionable insights and the industry’s latest articles in your inbox every week.