SEASON 2 - EPISODE 130 - NOVEMBER 6, 2024

Question Your Customers

Ever filled out a customer survey? For this conversation, Jason Fried and David Heinemeier Hansson, co-founders of 37signals, share their approach to gathering customer feedback. They discuss the timing and frequency of feedback requests, the power of open-ended questions, and how to transform customer language into effective marketing.

Watch the full video episode on YouTube

Key Takeaways

00:37 - A single open-ended question can reveal a wealth of valuable insights
05:51 - Listen for the language customers use in their feedback for possible marketing opportunities
08:42 - Form questions and answer fields that allow customers to use their own words to express what they think
14:18 - Focus on organic feedback rather than quantitative metrics

Links & Resources

Transcript

Kimberly (00:00): Welcome to Rework, a podcast by 37signals about the better way to work and run your business. I’m your host Kimberly Rhodes, and if you are a Basecamp customer, you may have seen a link to a survey at the top of your account. We recently reached out to a few of our customers to get some feedback on the product and we don’t do that often, so I thought it would be a good time for us to talk about it, the reasoning behind it and what exactly we learned. To do that, I’m joined by Jason Fried and David Heinemeier Hansson, co-founders of 37signals. Jason, I’m going to start with you. Have we reached out to customers often to ask them questions? I saw it pop up on Basecamp and I was like, oh, this is new.

Jason (00:37): Yeah, we don’t do it very often over the past couple of months I’ve done it twice. I typically do them when I’m curious about something and there’s something, one thing I’m curious about and what I’ve found is that that is mentally what I’m curious about. I’m curious about this. So lemme ask the one thing, and we tend to get really high response rates because it’s one question and the question is always a question with a big text area and it’s like fill it in. It’s not multiple choice. It’s like whatever you want to say, say. And if you don’t want to say it, you don’t have to say it. And we get somewhere, I think the last one we did, we got about 1600 responses.

Kimberly (01:13): Oh wow.

Jason (01:13): And it was so full of incredibly useful, rich stuff by just asking one question that’s open-ended, that there’s no way I could have mined for that information any other way. In fact, I would’ve gotten less useful information had I asked more questions and been more specific about the kinds of answers I wanted. So just as if you’re going to ask a friend a question, you don’t give them multiple choice options. You just ask a question and then people share. And so typically what I’ll do is it might give an example of how you might respond if I’m looking for a kind of directional type of answer. For example, the last question we asked was about how’d you land on Basecamp? How’d you get here? And I said, like for example, maybe you start with email and then you went to Slack and then you went to something else and then you went to something else and you went to Asana and then you got Basecamp.

(02:04): That could be an example path for how you landed here. And so most people will take that format, essentially that template, and kind of fill it in for themselves. Other people will tell a long story, other people will just say something else. So you can kind of point someone in a relative direction of what you kinda want, but you still want to let them do it themselves. And I have no problem answering questions like that. What I do have a problem with, I bought a car recently and I got a survey from the brand I think it was, or the dealership, I think it’s the brand. And I’m like, okay, I’ll dive in. So I hit the thing and you get a few questions like, okay, I can answer those. And it’s like next. And it’s another few questions and you kind of go through maybe a page or two and you’re like, oh, I get this.

(02:43): This is going to be like 45 questions and they’re all on a zero to 10 scale and I don’t even know what it means to be a seven or an eight. How was your dealership experience? Seven, eight? I don’t even know. And the other thing is of course when you buy a car, they always tell you, hey, they nudge you, hey, hey, I need tens. Give me some tens, we need tens. It’s like the whole thing is such a scam, but most of all, you’re asking me so many questions that in the end, I’m actually not going to finish the survey. I’m not going to submit these answers. You’re going to get nothing from me. You would’ve been better off asking me one thing or three things or whatever, or telling me this is the only page of questions you’re going to get. And okay, there’s six of ‘em, I can do that. But this never ending survey, and this comes from marketing departments and product development departments who feel like they just need so much and they’re may be only going to get one chance so they ask everything. You have as many chances as you want if you ask decent questions occasionally. Just take your time, get good answers from people and get great response rates. And I find it to be a very, very wonderful way to do this.

Kimberly (03:43): So was there an impetus for this survey, a reason why you wanted to know this specific question at this specific time? Or was it just like general curiosity?

Jason (03:52): Yeah, David actually and I were sitting down for lunch with someone and just kind of riffing on some stuff and I just got curious about, we’re talking about marketing and whatnot, how do people get to Basecamp? And I just had this thought about, well, I dunno, let’s find out. I’m like, I think I know, but let’s find out. Let’s find out. And I’m sort of curious to see what products people have gone through because it turns out it’s very interesting. Some people discover Basecamp initially and they just stick with it. Other people start with Basecamp, they leave because they think they need more. It’s amazing. You watch their path through these other prominent tools we’ve all heard of and they come back to Basecamp. It turns out 40 some odd percent, I think it was 41% of people had Basecamp in their thing twice. So they started with something else.

(04:36): They tried Basecamp, they left and they came back. And the things they went through were Monday, Asana, Clickup, Trello, Notion, Slack, some Atlassian products, all the usual suspects, and you could see what they did. They left. You can imagine we need more power, we need more this thing that we think we need. Yeah, go ahead. Go out there and check it out. And they check it out and they go, yeah, we either didn’t need it or it’s way too complicated, or no one’s using the damn thing at all. Or, wow, this is not where I thought it was going to be, and they come back and they circle back. It was just very interesting to see that. The one previous to that I asked, and we asked this every few years, which is what’s changed for the better since you switched to Basecamp?

(05:18): And it’s not about Basecamp, it’s about however you want to answer the question. But it turns out it’s mostly about what people end up answering is like workflows are better, people are communicating more clearly. People are writing things up in ways that are better. There are fewer meetings, all these things. Basecamp doesn’t have a fewer meetings feature, but using Basecamp leads to people having fewer meetings and so they share that kind of information. It’s very, very interesting. So I like the open-ended questions without a distinct answer that I’m fishing for. I just kind of want to know, which is I think a much better way to do it.

David (05:51): What’s so amazing about these open-ended questions is that the customers who answer them will give you the greatest gift of all. They will give you the language by which they think about these problems. That is such a goldmine to everything from how to name a feature to how to market that feature because they’re presenting the problems like they think about them. I remember when we did one of these open-ended inquiries a while back and these sayings would go over and over again like things falling through the cracks. This was then a term we would use in some of the positional material we have on, well, Basecamp is a place where things don’t fall through the cracks or when things fall through the cracks, you need Basecamp. The richness of that unprocessed language, that’s where all the gold is. And it’s just so tragic to me that marketers in particular are willing to trade that because it’s manual.

(06:52): You have to dig through it. You can’t just tabulate it. You can’t just get this junk specificity. You can’t just get like, oh, well the customer satisfaction was 7.253 this quarter. That is a 11.2% increase over last quarter. What the fuck does that even mean? How does that connect to any human in any way that anyone would respond to? But this is the problem of incentives sometimes, right? You’re incentivized like quarter after quarter, you have to show some growth on some KPIs, some metrics. It’s difficult to show that growth and oh, I have a deeper understanding about how our customers think about our problems or how our products are received, but that’s where all the nutrients are. This ultra processed form junk stats that comes out of these multiple choice, these zero to 10, these one to five bars, there’s so little in them, there’s so little nourishment in them. There’s so little you can use them for other than just putting them into some quarterly report that no one’s going to read anyway and it’s not going to have any bearing on whether you can attract more customers in the future or not. It’s all just circumstantial, hyper extracted bullshit. If you instead dive down into these human narratives, a lot of them actually, if you go, I think it’s on basecamp.com, we have these 1000 testimonials that came from one of these questions. I think Jason, wasn’t it the what things got better?

Jason (08:22): Yep.

David (08:24): You can read through that and you can see entire story arcs. You can imagine the entire customer, you can get closer to the customer in that sense, and it’s so valuable when you’re willing to actually engage with the material, not just treated as stats.

Jason (08:43): By the way, that page is at basecamp.com/customers, which has close to a thousand quotes, simply answering the open-ended question. And to David’s point, which I thought was great about this point about the language and the nourishment, if you look at that page, basecamp.com/customers and you scan, I’m not expecting anyone to read the whole thing and the whole point of it being so long is that it’s just supposed to be overwhelming, actually, that’s the idea, but we’ve highlighted some things. Okay, so you look at that data set, right? And then I could have had another data set, which is like how many fewer meetings do you have per week? And it could have said three or four or five, whatever. This could have been a choice, right? There’s no language. There’s no energy or emotion. So I could have shown the data set that we have, which basecamp.com/customers and other data set, which is numbers, which just answers, which is pure statistical data.

(09:35): There’s no richness in that statistical. I can make a chart, I can make a graph, I can put in Excel, I can ask questions about it roughly in trends and percentages and do more or fewer people are avoiding more meetings in the last time we asked this. I can get at all that stuff. I would never take that though. I’d always take the open-ended, use your own words and tell me what you think. Is there more to read? Yes. Can I analyze it? No, not really. I can read it and absorb it and feel it. Yeah, it can be analyzed. I’m sure if you want to look for keywords, how many times is this word mentioned and all the things, whatever. You can do all that too, but it’s actually readable. It’s ingestible. It’s something you can feel and that is where the value is.

David (10:17): I’ll extend this metaphor one step further. What this is doing is it’s training your gut instincts. When you read through all these customer accounts about how they found your product, why the product is valuable, where it’s going, you’re teaching that supercomputer as we’ve talked about, your gut instincts to be in touch with what customers actually want. Can you directly map that to how this feature should go or some growth targets? No, you cannot, but you will be wiser and you will be more in tune with the customer set you have when you have to make those decisions and you won’t be able to trace it back necessarily to one of these thousand anecdotes, but I guarantee you, you will be wiser and more informed and therefore better able to make snap gut instinct decisions when you are making new features, what matters, what doesn’t matter.

(11:08): So much of this is in the subtlety. So much of this is not something you can tap, relate, or even A/B test. Someone actually just asked Jason and I today, Hey, do you do a lot of A/B testing? And I thought about the fact that I don’t think we’ve done an AB test inside the product for probably 10 plus years, and the last time I remember I was doing an A/B test inside the product, I ended up with more questions than answers. So what does it mean that 41% uses it in this way? I don’t know any of your motivations. I don’t know what you’re trying to accomplish. I don’t know where you’re trying to go. I don’t know what to do with this information. Versus when you get the stories about what people are trying to do, you get the motivation behind what they’re trying to accomplish. You can go like, oh, okay, I see what you’re trying to do here. We could actually fix that. We could dive a little deeper and we could make some tweaks and I can satisfy that. I can’t find that out of 41.97.

Kimberly (12:06): I think the other thing that’s interesting about these open-ended questions or just having one single open-ended question is the ability to follow up. So Brian, our head of product, was looking through some of these surveys results and sent me a couple. They were like, hey, Kimberly, is customers asking about videos to understand Basecamp? Well, we can help you with that. Or someone had a question about using Basecamp for event management. We can help you with that. But it wasn’t a question of one through five or tell me what your problem is. It was open-ended, but it gave us an opportunity to follow up with people who we didn’t even know that they had this question. It just gave them a place to put it.

Jason (12:43): Something I did on the what’s changed for the better question. There was technically I think two questions on that, which is like what’s changed for the better? And then the other one was like anything else you want to tell us? Which is not really a question. I mean it is. It’s a question technically, but it’s really like a catchall. Just leave a space for people to say something and they will. A thousand people said something else. Some of them was just wonderful to read. Basecamp changed our business. Can’t believe it. Thank you so much. Other things are like, we’ve been using Basecamp for a number of years, 12 years, 13 years, amazing stuff like that. Other people just share stuff about their company. You don’t even know what people are going to share, which is awesome. It’s more space for people just to say something because a lot of people have something on their mind.

(13:31): You can ask 30 questions in a survey and get less information than asking two questions. And so I feel like we get way more information out of asking two questions or one question than we would if we asked 30 questions because the information we get is what’s truly on their mind, and that’s actually the more valuable stuff, not the stuff that we want to squeeze out of them. I keep going back to this car example because it’s like I don’t actually have an opinion of how the staff treated me. It’s not how I think about that experience. So now I got to make something up and I don’t know, and they were nice people. I guess five, I don’t know. And then you’re just giving them information that’s not even accurate in a sense. I just feel like the best thing you can do is ask fewer questions and have them more open-ended and really try to get to one if you can.

David (14:18): What I love about this example with the car survey, and I received those two both from cars and other experiences, is that you always know when you allow something to be quantified, it’s going to be gamed. And this is exactly what happens with those car surveys. The salesperson knows that there is a metric that they have to hit and it becomes a very circumspect dance. I have to extract a five out of five out of you, otherwise my job is in danger. One of the great examples of this is on Uber, if a driver has, I believe it’s less than 4.7 as their average score, it basically means they’re total shit. The scale doesn’t go from one to five, it goes from 4.7 to five. That’s where all of it is, right? And you just go like, there’s such a poverty in that realm and you can’t really gamify an anecdote.

(15:17): That’s the other thing. It just has built in protection against that. What’s the salesperson going to say something nice about me? But you know what? It’s a lot harder to feign that if you didn’t really mean it or you can be subtle about it in a way where the person reading it can understand what was going on. So I think the open-ended question is simply a more resilient form of surveying. It is more resilient to gamification, to junk stratification, to aggregate averages that compresses all the nutrients into this ultra processed bullshit. The stuff is here. Now whether you’re able to take that and turn it into either product improvements or otherwise, okay, that’s up to you, but at least the opportunities there, the soil is rich.

Jason (16:06): Another way to think about this is the reason why these things have numbers is so they can be processed and they can be turned into charts and put in a presentation that can be presented to the board. If I was the board or whatever, I’d rather take 10 minutes, give me a hundred responses and let’s read ‘em together. That is way more valuable than seeing some chart, but the chart is easy to disseminate. It doesn’t ask anything of anybody else. And so that’s what you end up with. And then also, of course, it can be measured over time. You could say customer satisfaction is up 18% this quarter. What does that even mean? I’m sure many of the questions were on this car survey. I wasn’t even thinking about those things. That wasn’t how my experience was reflected. So yet somehow I would’ve given it a number. It actually is meaningless. It doesn’t mean anything, but now it’s in the data, and so the data represents a lot of nothing actually.

Kimberly (16:59): So let me ask you this. We know what type of questions you like to ask and how many questions you like to ask. How often do you like to ask your customers for their feedback? Is this something you do regularly with all the products? Kind of tell me when this comes up for you guys.

Jason (17:14): Well, first off, we hear from 500 plus people a day via customer service. So there’s always stuff coming in and going out. These kind of pointed questions once or twice a year usually is how I would do it. Maybe three times, maybe? Often two or three. I’m trying to remember the last time I asked something three times in a year, probably, I don’t know, once or twice. So I’d say one or two questions. We’ve done ‘em only for Basecamp, have not done them for HEY. We actually should, and thank you for bringing this up. I wasn’t even thinking about it, but I’m going to do one for HEY as well. What’s cool about Hay is that we could actually hit reply or something and just respond that way. Maybe, maybe not. I’m not sure. That might be kind of a messy thing, but yeah, that’s it though.

Kimberly (17:57): Okay, well thank you for that. We’re going to wrap it up. Rework is a production of 37signals. You can find show notes and transcripts on our website, 37 signals.com/podcast. We’ll video episodes are on YouTube and X. And if you have a question for Jason or David about a better way to work and run your business, leave us a voicemail at 7 0 8 6 2 8 7 8 5 0. You can also text that number or leave us an email at rework@37signals.com.