3 Common Customer Survey Mistakes Marketers Make

Why it pays to ask the right questions – in the right way

Common customer research assumptions and techniques are misguiding your decisions.

With the rise of DIY market research tools like Survey Monkey, Typeform, Delighted and even Google Forms, data can be collected from customers quickly like never before without the need for a market research firm.

Simply load up your customer list, ping them a message from your modified template survey, wait a few days for them to respond, and send the results to your team.

Hey presto, this market research thing is easy!

Except it’s not.

Herein lies the problem and a major mistake inexperienced DIY market researchers make.

Armed with a digital tech-infused, “move fast and break things” enthusiasm, inexperienced employees are routinely making critical flaws in their customer research process. Mistaes that could be prevented by reading even the most basic textbook on marketing research. Unfortunately this mix of prejudice and skill deficiency, is leading to equally flawed management decisions. Decisions which are costing firms millions.

I was lucky enough to work during university at a professional market research firm while studying units on the same subject. Here I quickly learnt, first hand, how easy it was to make major mistakes. These mistakes, if they remain unidentified, can be extremely costly and lead the entire organization in the wrong direction.

NPS®

One of the worst offenders is the routine Net Promoter Score (NPS®) survey sent via email to customers. An example of one of these surveys is featured below via the popular branded distribution tool ‘Delighted’.

This was sent to me repeatedly over the course of a 12 month period as I never responded and fell into some unresponsive pool of unhelpful users who obviously needed reminding.

However, this classic example of an NPS® survey suffers immediately from two major research flaws.

1 – Response bias

If I’m a current customer, am I willing to give an honest answer if I really hate this company? My cultural predisposition will encourage me to be polite and save face, even if I’m displeased and this may affect my ability to given an honest answer. The brand I’m reviewing is directly sending this survey to me instead of via a neutral “third-party” research firm who promises to make the answers anonymous when reporting back to their client. This third-party research approach is standard practice for professional research firms so as to limit this form of response bias.

Here’s another example.

Perhaps an emoji-led approach is superior? This version appeared after I connected to the M Gallery hotel’s network in Paris last year.

Somehow, I already feel better about connecting my current mood (no matter how sour) with a pre-existing emoji expression. Yet, this survey still suffers from the same basic flaws which surprised me, given Accor Group is such as large global company.

2 – Sample bias

The second flaw is sample bias. Sometimes the groups of people who’s opinion ends up being the most valuable, are the exact people who don’t respond. By simply tallying the results from everyone who responded to an email distributed survey, you’ve automatically received a biased sample.

This sample is going to consist of respondents who are likely regular email checkers, have spare time on their hands and have a general affinity to be helpful. What about everyone else?

On multiple occasions I’ve started analyzing results from large samples of email respondents before digging a little deeper into the group who didn’t respond. I’ve followed up with these non-respondents with a phone call and the results from their answers have between so diametrically opposed to the first group it’s not funny.

Yet I understand NPS® surveys are a popular survey to conduct. They provide consistent longitudinal data if used repeatedly over time and form a general yardstick measurement of brand health. Many people however, aren’t aware that Bain and Company invented the term which has led to the widespread adoption and general acceptance within the business community. Perhaps leading to an over reliance on this one example of potential customer surveys.

The problem is that there’s so much more you could ask – far richer questions. If an NPS® is your only form of primary market research, you’re missing out on a lot of valuable insight.

I want to find the answer to a lot of questions beside just a referral tendency. Things like, brand preference, purchase intent and other high value sales determinants.

Accor at least follows up with a secondary question regardless of your response in order to capture a richer set of data.

However, herein lies a second major shortcoming of DIY researchers.

Are you asking the right question?

“Judge a man by his questions, not by his answers” Voltaire

To illustrate this point, I used three Instagram polls to test the popularity of a product category I know a lot about – Grower Champagne.

For the unacquainted, “grower” Champagnes could be best summarized as the equivalent of the craft beer brands of the Champagne world. Smaller producers who grow their own grapes and often use more artisan production methods. Generally these are brands you would never hear about unless you’re really into your Champagne.

The non-grower Champagnes or ‘big brands’ vary depending on which country you reside in but could include Dom Perignon, Moet Chandon, Veuve Clicquot, Taittinger, Pol Roger, Lanson, Charles Cazanove etc.

So I got a friend to poll his 80k following to test whether these smaller brands were popular. Around 400 different people responded and the results were as follows.

Looks pretty definitive doesn’t it?

To be sure, I polled my 8k followers who all specifically follow my account because they have an affinity for Champagne, instead of just wine in general.

To reveal richer insight, I wanted to know more than just ‘liking’. I wanted to test brand ‘preference’ instead, which tends to be a closer determinant of purchase intent. I also added a reference to the ‘big brands’ vs ‘growers’.

Interesting result, and reasonably definitive again. I’ll add that the sample size was around 50, so we’d need to do collect more responses before an accurate comparison can be made, but the results were interesting all the same.

I know that one of the main problems with grower Champagne however, is that they are distributed less widely than the larger brands. They are often harder to find in stores closeby and due to import economies of scale, they can be overpriced relative to their quality.

I also know, that Champagne is a product that people buy to show off to others during special occasions. The “show off” effect is considerably reduced if your friends aren’t familiar with the brand because any sense of prestige is diminished considerably.

So I asked another question with a similar sample size to the same audience.

Now that is interesting.

If I was a grower Champagne brand and didn’t ask this third question, I might be encouraged to make major investments in the market without knowing what is really determining sales. From personal experience, I know this grower market is actually growing in Australia quite rapidly so the macro fundamentals for growth are there.

However, according to the third question, the main challenge grower brands face is not convincing people that grower Champagne is better, but rather convincing them to actually follow through with their preference and purchase.

We would need to discover the reasons why they still purchase the big brands over the little brands – not whether they like or prefer the big brands. Once the answer is found, a strategy can be put in place to change customers’ current behaviour.

Without this, all the advertising and promotion in the world, will be largely ineffective at driving sales.

You could spend millions of your budget raising brand awareness, encouraging influencers to tout your brand in a credible way, creating a shiny VR app, redesigning your websites and many other things, but it’s not going to do much.

Your attention should be instead focused on fixing your distribution problem.

Final Question

So I decided to test this distribution hunch with a further open-ended question where the audience could answer in detail.

“Why do you think people buy less Grower Champagnes?”

This is a sneaky way of asking the question in order to reveal each respondents personal opinion by taking advantage of projection.

These were the answers. Note: RM is a contracted form of Recolant Manipulant meaning a grower Champagne.

• Lack of familiarity & limited access to a solid selection of RM producers
• Probably not as well known by the masses
• Because they are scared to admit they don’t know anything and just listen to rappers
• It’s less accessible and most ppl look for the Marquee Brands. But I’m all for the RM’s.
• Don’t know about it
• Less marketing
• Access/convenience for the general consumer
• Because they are less famous. Today everything is marketing, branding, communication and today the big brands try to have all the champagne market
• Because they do not know it

Takeaways

Before you rip into my research methodology (which I’ll admit needs further validation), remember that I’m using these tests as a very basic example to illustrate common flaws that can have major effects on your research insight.

For Growth and Marketing Staff

If you’re going to conduct market research in your firm, make sure you spend some time thinking about whether you’re asking the right question. At the very least, hire an experienced marketing research consultant to check your work before the campaign launches (you can find them by contacting yours truly).

For Managers and Executives

If you’re in upper management, it pays to query the details of the research findings that are presented to you. Probe a little into the specific way the data was collected and how assumptions may have been made before making critical decisions.

Further Reading & Action

Coke

Even the largest companies in the world get it wrong. Perhaps the most famous example of market research failure affected Coke when they tested ‘New Coke’ before releasing it to the market. Subsequent launches of product extensions such as “Vanilla Coke’ and ‘Coke Life’ have fared better, but further illustrate the purchase resistance their loyal customer base exhibits toward adopting new products. Click here for the full story – new_coke_AMA

Assistance

What are the questions you need answers to in your role? Reach out to our team on +61 3 9021 2210 or register for an invite request to access our new growth platform where you’ll find further reading and more options for improvement within the market research area.

Don’t forget to follow John on LinkedIn and say hello.