Intuit fmr Product Lead on Product Testing: Learn Fast, Fail Small

Product testing is an essential part of building successful products. Former Product Lead at Intuit and current MLSE Director of Product and Design, Pansy Lee had a lot of wisdom on the subject to share with the attendees at her Speaker Series event in Toronto. As always, we suggest watching the entire video of her presentation above.

The highlights of Lee’s Speaker Series presentation are included below.

How to do Product Testing

The number one reason start-ups fails is that there was no market need for the product. 42 percent of start-ups fail for that reason!

1. Define problem & ideal solution
2. Build a hypothesis
3. Identify riskiest assumptions
4. Minimum success criteria
5. Pick experiment
6. Execute & collect data
7. Evaluate – pivot or persevere

How to define a problem and a solution

A great product manager knows how to identify the problems they’re facing and the solutions to them.

Problem

• Who’re you solving for?
• What are they trying to do?
• What is holding them back?

Solution

• In a perfect world…
• 11-star experience (An Airbnb example of exceeding beyond expectations)

Tip: Talk to your customers! It will help you understand why a problem exists.   

How to identify your riskiest assumptions

As you begin product testing it’s a good idea to take note of assumptions you’re making.

List your assumptions: In order for my idea to be successful, the following must be true…

Find your riskiest assumptions

• Has never been done before
• No alternative solution

Tip: Write it all down! Even if it feels trivial, you can eliminate the ones that have already been proven after.

How to Build a Hypothesis

What do you think will happen?

If we _____________ (our action),

we believe _________ (who)

will do _____________ (their action)

because ____________ (reason).

Minimum success criteria

How do you know if you should move forward or move on?

Cost: How much interest or conversions do you need to generate in order to cover the cost of creating this product?

Vs.

Reward: Look for signs of real intent (email, phone # or payment) and find comparable conversion rates from old tests or industry standards.

Execute & collect data

• For how long should you run an experiment?
• How many people should see it?
• What is statistically significant?

It’s all about gaining confidence to make a decision!

10 types of Product Experiments

There are ten different types of product testing you should consider.

1. Adword test: test interest using ads and tracking clicks
2. Beta sign up: landing page that describes your solution and asks for sign ups
3. Shadow button/coming soon: buttons that look like they lead to real products, but actually lead to a coming soon or beta sign up
4. Buy buttons: purchase flows lead to a beta sign up
5. 404 pages: if you only have time to develop a button and not a landing page, track the clicks on the button that lead to an error page
6. A/B tests: take your incoming traffic and show them different variations of content and creative
7. Explainer video: videos that walk through your product/service from clip art and drawing to 3D renderings
8. Wizard of Oz: Real front end, fully manual back end
9. Piecemeal MVP: Product/delivery is stitched together using out of the box software
10. Fundraising: crowdfunding to see if there is enough demand to take prototypes to manufacturing

About the Video:
Speaker: Pansy Lee
Host: