The Sean Ellis Test applied to an AI startup product is a one-question survey that measures product-market fit. It asks active users how they would feel if they could no longer use the product.
The Sean Ellis Test is a product survey method named for founder Sean Ellis. For an AI product, it separates polite interest from real dependence. It works best after users have had enough time to form a habit.
Table of Contents
- What the test asks
- How to run it in an AI product
- How to read the answers
- Limits founders should watch
- What the test asks
- How to run it in an AI product
- How to read the answers
- Limits founders should watch
What the test asks
The core question offers three basic choices: very disappointed, somewhat disappointed, or not disappointed. Only regular users should receive it. Casual visitors and one-time triallists give noisy answers. A common benchmark treats 40% "very disappointed" as a strong fit signal.
Treat that number as a guide, not a guarantee. A small, passionate user group can matter more than a broad, indifferent one. The follow-up questions matter as much as the score. Ask what users would use instead and what benefit they would miss most. Those answers point directly to the product's core value.
How to run it in an AI product
Pick users with real exposure to the AI output. For most AI tools, that means people who ran several tasks, prompts, drafts, or searches. Exclude brand-new signups and long-dormant accounts.
Keep the survey short and place it inside the product. Email can work for weekly tools, but in-app delivery usually brings higher response rates. Aim for at least a few dozen responses before acting.
- Survey users after three to ten successful uses, not after signup.
- Ask about the whole workflow outcome, not model quality alone.
- Add one open question about the main benefit.
- Add one open question about the closest alternative.
How to read the answers
Sort responses by use case, frequency, and customer type. An AI writing assistant may delight marketers and disappoint engineers. The average hides that split. Read the open answers for jobs, not adjectives.
"Saves me an hour on client reports" beats "cool AI." Group repeated jobs into two or three clusters. Those clusters define who should get more attention. Cross-check disappointment against retention and repeat use. High disappointment plus repeat use supports further investment. High disappointment without repeat use often means a painful setup, unclear pricing, or missing trust controls.
Limits founders should watch
AI products face novelty bias. Early users may praise a demo, then stop paying when the work feels ordinary. Re-run the test after the novelty fades. The test also misses non-users with buying power. A team member may love the tool while a manager blocks purchase over privacy, accuracy, or cost. Pair the survey with buyer interviews and support tickets.
Use low scores as a repair list. If users would switch back to spreadsheets, manual review, or an older tool, improve reliability, speed, integrations, and review steps before adding new model features.INTRODUCTION: The Sean Ellis Test applied to an AI startup product is a one-question survey that measures product-market fit. It asks active users how they would feel if they could no longer use the product. The Sean Ellis Test is a product survey method named for founder Sean Ellis. For an AI product, it separates polite interest from real dependence. It works best after users have had enough time to form a habit.
What the test asks
The core question offers three basic choices: very disappointed, somewhat disappointed, or not disappointed. Only regular users should receive it. Casual visitors and one-time triallists give noisy answers. A common benchmark treats 40% "very disappointed" as a strong fit signal.
Treat that number as a guide, not a guarantee. A small, passionate user group can matter more than a broad, indifferent one. The follow-up questions matter as much as the score. Ask what users would use instead and what benefit they would miss most. Those answers point directly to the product's core value.
How to run it in an AI product
Pick users with real exposure to the AI output. For most AI tools, that means people who ran several tasks, prompts, drafts, or searches. Exclude brand-new signups and long-dormant accounts.
Keep the survey short and place it inside the product. Email can work for weekly tools, but in-app delivery usually brings higher response rates. Aim for at least a few dozen responses before acting.
- Survey users after three to ten successful uses, not after signup.
- Ask about the whole workflow outcome, not model quality alone.
- Add one open question about the main benefit.
- Add one open question about the closest alternative.
How to read the answers
Sort responses by use case, frequency, and customer type. An AI writing assistant may delight marketers and disappoint engineers. The average hides that split. Read the open answers for jobs, not adjectives.
"Saves me an hour on client reports" beats "cool AI." Group repeated jobs into two or three clusters. Those clusters define who should get more attention. Cross-check disappointment against retention and repeat use. High disappointment plus repeat use supports further investment. High disappointment without repeat use often means a painful setup, unclear pricing, or missing trust controls.
Limits founders should watch
AI products face novelty bias. Early users may praise a demo, then stop paying when the work feels ordinary. Re-run the test after the novelty fades. The test also misses non-users with buying power.
A team member may love the tool while a manager blocks purchase over privacy, accuracy, or cost. Pair the survey with buyer interviews and support tickets. Use low scores as a repair list. If users would switch back to spreadsheets, manual review, or an older tool, improve reliability, speed, integrations, and review steps before adding new model features.