What are those tests?
- Sean Ellias test
- David Cummings Test
WHAT IS THE SEAN ELLIS TEST?
With so many competing definitions, Product/Market Fit can be difficult to get to grips with; it’s both essential to understand and extremely difficult to do so.
It’s for that reason Sean Ellis (DropBox, Lookout, Eventbrite, LogMeIn…) set out to create a straightforward test to help founders identify the crucial moment when their solution finds Product/Market Fit.
Sean’s simple PMF survey takes a step back from complex metrics, and instead focuses on the most revealing data source at any SaaS company’s disposal: their users.
How would you feel if you could no longer use [product]?
- Very disappointed
- Somewhat disappointed
- Not disappointed (it isn’t really that useful)
- N/A – I no longer use [product]
If over 40% of users respond to the survey saying they’d be “Very disappointed” to stop using your product, there’s a great chance your solution has found Product/Market Fit. Why?
After comparing nearly 100 startups, Sean found that those that struggled to reach traction always scored under 40% on this particular test. In contrast, those that managed to gain strong traction always scored over 40%.
Applying this to your business, you have a quick and dirty way of gauging Product/Market Fit.
With enough good-fit users (see below) to survey (Buffer determined that 40-50 responses should carry statistical significance), you can make an informed decision to either double-down on product development, or start to scale-up your sales, marketing and customer success.
I generally recommend to survey the following:
- People that have experienced the core of your product offering.
- People that have used your product at least twice.
- People that have used your product in the last two weeks.
What are the pros of Sean Ellis test?
1) IT’S SUPER SIMPLE
The simpler a metric is, the more likely it is that people will actually use it.
Even the best-intentioned of tech startups end-up with a huge dashboard of KPIs that look useful, without having any real understanding of how to actually use the metrics to help their business grow.
The Sean Ellis Product/Market Fit test sidesteps this problem – creating a metric that’s easy to calculate, easy to understand, and easy to use.
2) IT’S USER FOCUSED
Despite the perceived complexity of trying to determine Product/Market Fit, at it’s heart, it’s pretty simple: we’re trying to find out if people want our product.
…most startups fail not because they don’t manage to develop and deliver a product to the market; they fail because they develop and deliver a product that no customers want or need.Steve Blank
Sean’s test homes in on the people that engage with your product. Instead of fixating on a ton of quantitative, context-less data, it helps founders touch base with reality, and find out how their users and customers actually feel about their product.
Sometimes we lose track of our customers and instead focus on terms, such as “product/market fit” and if we were in closer touch with our customers we would know whether we have it or not.Leo Widrich, Buffer
What are the cons of Sean Ellis’ Product-Market Fit test?
1) INTENTION =/= ACTION
This is a classic problem with any type of survey, from the Sean Ellis test to Net Promoter Scores (also used as a PMF indicator): there’s almost always a disconnect between what people say they’ll do, and what they actually do.
There’s no guarantee that survey responses, no matter how well intentioned, will actually foreshadow action. If a business bets their entire growth strategy on a single survey, they’re exposing themselves to a whole lot of risk.
While surveys are useful, I would much rather take bets on what people are doing, not what they say they would do or are doing.
This is why I think about these surveys more of as a leading indicator of product market fit vs proof of product market fit.Brian Balfour, Reforge
2) IT INDICATES PMF, BUT DOESN’T GUARANTEE
Sean’s testing revealed that a score of 40% or higher correlates with Product/Market Fit – but as my Stats professor used to hammer into us every lecture, correlation does not equal causation.
A positive score on this test strongly indicates PMF, but it doesn’t guarantee it. The market for your product is going to be nuanced and complicated, and it’s entirely possible for a product to excel at this test, only to crash and burn during the attempt to scale.
KNOWLEDGE IS POWER
Any Product/Market Fit test needs to be viewed as an indicator of PMF, and not proof.
It’s impossible for any test to generate a guaranteed Yes/No, because the question we’re trying to ask is so complicated, and the variables we’re trying to measure change so frequently.
As a result, the more information you have, the more you can de-risk the decision to scale.
Though imperfect, the Sean Ellis Product/Market Fit test plays a huge part in calculating PMF, allowing you to take the pulse of the people that will make or break your business: your users.
Tests for determining Product/Market Fit come in all shapes and sizes, from the super simple to the over-complicated.
Today, we’re looking at the pros and cons of one of the most popular: the approach used by Pardot founder, David Cummings.
WHAT IS THE DAVID CUMMINGS TEST?
David’s approach is relatively simple: if you’re able to satisfy these 5 criteria, you have a good indication that you’ve reached Product/Market Fit (PMF).
1. 10+ customers have signed-up in a short period of time (typically 3 – 9 months). Crucially, these need to be legitimate customers, and not ‘friendlies’ – people with an existing relationship to you, or those trying your product as a favour.
2. You have at least 5 customers actively using the product with little/no product customisation. This means that your product is useful in its current state, and doesn’t require extensive modification to make it useful to each new customer.
3. At least 5 customers have used the product for over a month without finding a bug.
4. At least 5 customers use the product in a similar way, and achieve similar results. It’s common to see early-stage customers using your product in all manner of weird, wonderful and unintended ways – but it’s consistent usage patterns that suggest your product is ready to scale.
5. At least 5 customers exhibited a similar acquisition and onboarding process. In addition to using your product in a consistent way, your customers need to sign-up and start using it in a repeatable fashion using it in a repeatable fashion. It’s these repeatable processes that lay the building blocks for scale.
Successfully achieving each of these 5 goals may seem like a far-reach for some early-stage Startups, and that’s intentional.
Achieving Product/Market Fit is an ongoing process, and not a discrete, big bang event. David’s goals are designed as benchmarks for monitoring progress towards Product/Market Fit. Instead of creating a measurement with a binary output, they provide insights into how Product/Market Fit can be continually developed and improved.
There’s an ongoing maturation process that takes time, even with extended resources. As a founder deep within the process of developing product / market fit, it’s useful to refer back to these five ways to assess progress.David Cummings
PROS OF THE DAVID CUMMINGS PRODUCT/MARKET FIT TEST?
1) IT FOCUSES ON QUALITY OVER QUANTITY
The main reason to determine PMF is to calculate when it’s safe to scale your business. To assess this, your total number of customers isn’t as important as the quality of those customers.
For example, you could have a hundred customers actively using your product; but if they’re each using it in a different way, to achieve different goals, it’ll be almost impossible to scale that success.
David’s goals focus explicitly on the quality of customer engagement, more so than the volume of customers. In doing so, it makes sure that the essential building blocks of your business – the product’s key functionality, its reliability, and its ability to solve a valuable problem – are ready to scale.
…signing 10 customers that fit your ideal customer profile is more important than signing a large number of random customers in the near-term…
— Customers will always ask for product enhancements, so it’s key that requests align with the entrepreneur’s vision of the future
— Ideal customers are going to be happier customers and happier customers are going to provide testimonials as well as references for future customers.”David Cummings
2) IT’S EASY TO USE
Great metrics need to be easy to use. By focusing on a small sample size, David’s test makes it extremely easy to get a snapshot of progress towards PMF.
There’s no need to dig out a monstrous spreadsheet, or wait for your customer base to swell to triple digits: you can get a rough-and-ready idea of Product/Market Fit in a few minutes.
Better yet, they’re extremely actionable metrics: if you’ve failed at providing a bug-free experience, you need to hone in on product development; if new customers are missing out on tons of functionality, it’s time to improve your onboarding process.
CONS OF THE DAVID CUMMINGS PRODUCT/MARKET FIT TEST?
1) IT’S LESS APPLICABLE TO B2C
Each of David’s hallmarks of Product/Market Fit hinge on the experiences and engagement of 5-10 customers.
Whilst such a small sample size is great for usability, it does mean that the test is most relevant to the B2B sector. This is particularly true for those targeting the Enterprise, where a viable business can built with relatively few high-value customers; and the insights offered by such a small sample size can be useful enough to extrapolate from.
With a much lower Customer Lifetime Value (CLTV), B2C companies need a much larger number of customers to sustain their growth. To make this model more applicable to B2C SaaS businesses, the sample size needs to be increased (it’s been estimated that a 10x sample size is enough to make these tests useful).
2) IT DOESN’T PROVIDE MUCH MARKET INSIGHT
If we return to the original definition of PMF, there are two criteria a Startup has to satisfy:
- Being in a good market
- Having a product which satisfies that market
With such a small sample size, it’s difficult to extrapolate these findings to learn about the quality of the wider market. Instead, David’s approach does a great job at assessing the second component of Product/Market Fit.
‘Satisfying the market’ goes beyond delivering value to a handful of customers – you need to be consistent and predictable with that value. By focusing on a handful of users, David’s criteria ensure that Startups create value in the right way.
By satisfying each of these five criteria, you can rest assured that your startup is attracting the right types of customers – and crucially, providing value to them in a scalable way.
Ankita Saikia 