Startup Validation: Why surveys are a bad idea and what to do instead
Trying to better understand your customers or validate a problem? What do you do?
It seems like 9 out of 10 people I meet in my line of work default to surveys. They think surveys are great. You reach many people. You can have statistical significance. It’s easy.
While surveys do have many use cases, they’re not the way to understand your (potential) customer. The only way to do that is to talk to them. Get out of the building and actually talk to people.
If you’re one of the nine people I’ve referred to above, you’d probably ask:
You: So how many should we do then? We need XX to be able to have a good sample.
Lisa: No, you don’t. Qualitative research is.. not about quantity, it’s about quality, just as the name suggests.
You: Okay, so you’re saying we don’t need XX, but how many do we need?
Lisa: It depends.
You: WHAT?? We need the exact number.
Lisa: There is no exact number. It really depends. You’re not after a specific number of responses to claim you’ve done enough. What you’re after is gaining enough insight to feel confident you understand the customers and their problems/ pains.
You: So what’s the ideal number?
Lisa: The ideal number is the one that brings you to the state of so-called saturation. This is when you interview an additional person but anything they tell you about the problem you’re trying to understand, you already know. So you’ve reached the level where you really get it and adding more interviews isn’t going to make much difference.
You: But why can’t we just survey them? Can’t we reach the same level of.. saturation?
Lisa: No, you can’t. There are a couple of challenges with the surveys. First, the questions are set in stone. Let’s say they give you an interesting answer to one of your open-ended questions. And you read through the submission and think “Oh my god, this is so interesting. I wonder why?” — all you can do is to keep wondering. If that was an interview, you could actually ask them. This allows you to learn about things you couldn’t have even thought of before the beginning of the interview. Can’t do that with a survey, can you?
You: Okay but is there anything else?
Lisa: Yes, there is so much more. When interviewing, you are with the person (used to be in the same room, now mostly on Zoom). You can actually observe them. You can see if they’re paying attention (as opposed to just clicking the random buttons because they’re bored), you can see how comfortable or not they are about certain questions and derive a lot of insights from their behaviour as well.
You: Okay. But there’s one thing you can’t take away from the surveys — they can be anonymous!
Lisa: Well, yes, but... you’d think people feel more like sharing because you don’t know who they’re. But I’d argue that in many situations they feel more like sharing because you’ve made them feel comfortable. You’ve built rapport. You’ve shown some human touch and empathy. This got them talking. Telling you something they’d have never told the screen in the survey.
You: Okay, but how about... testing willingness to pay?
Lisa: Don’t even get me started on this one.
You: What? Why? Is that something you can do in an interview as well?
Lisa: No, this is something landing pages are for. But that’s a topic for another time.
If you’re still in doubt, read Talking to Humans. The best piece on.. talking to humans I have ever come across (and it’s free!). I wish everyone I’ve ever mentored or coached has read this. Would make my job so much easier.