- WHAT WE DO
- INSIGHTS
- CLIENT Results
Tab Content One
Tab Content Two
Biases in Market Research
Psychologist Eleanor Longden sought help from a psychiatrist as she tried to deal with symptoms of schizophrenia. When one appointment ran very late, she excused herself, telling the doctor, “I’m reading the news at six.” The psychiatrist duly recorded that Ms. Longden had “delusions of being a television news broadcaster.” The reality was Longden really did have to read the news on a student-run TV station, as she describes in Scientific American Mind. This story is as amusing as it is telling, all the more so because one wouldn’t expect a trained psychiatrist to make assumptions or jump to unwarranted conclusions.
The misinterpretation by the psychiatrist is an example of confirmation bias. She was expecting to find evidence of mental illness in her patient, and as a result viewed the slightly improbable news-reading comment as a delusion. The psychiatrist made no effort to clarify the statement. One quick question to explore the topic, or even a shrink-talk, “Oh?” would have revealed the simple, factual explanation.
If a skilled psychiatrist, trained to look for nuances in patient statements, are marketeers also prone to this type of predisposition?
Errors are often made similar to Longden’s doctor. In interpreting customer feedback, or data from surveys and focus groups, it’s a natural tendency to interpret the data in a way that is consistent with what we believe ourselves. As the data rolls in, we want to blurt out, “I knew it!” or, “I told you so!” Rarely do we look for other ways of viewing the data, particularly explanations that might prove us wrong.
There are lots of ways that market research can go wrong, but this one is particularly subtle because it exploits a natural flaw in human thinking.
So much of today's market research is bad. For example, a CEO had achieved past success in selling commodity products at higher prices by differentiating with service and quality. He was convinced that the same strategy would work another market. When a customer survey showed that “price” ranked behind multiple other factors in importance, no one asked, “Why?” The question was asked in a different way, or it didn't became clear this surprising result be explored in more detail.
Instead, confirmation bias kicked in. The survey was viewed as proof that price wasn’t that important, and that a differentiation strategy would allow at least a modest price premium over less skilled competitors.
The new strategy lasted just a few short weeks. Sales fell, and recovered only when prices were rolled back to competitive levels. In retrospect, price was critically important to customers but was never an issue because everyone charged the same price. The survey results were accurate, but were taken to imply something that wasn’t true.
When doing market research or analyzing customer feedback, one has to be careful to look for alternate ways of interpreting the information--e.g., interjecting dissenting voices, asking "what have we missed", and following what appears to be insignificant data points. In the pricing example, there were a few experienced people who questioned the findings and strategy, but their objections were summarily dismissed as mere resistance to change.
In both of the examples above, just a little more exploration would have revealed that the initial assumptions were incorrect. But, since the statements seemed to confirm closely held beliefs and assumptions, nobody took that extra step.
In the coming years, Big Data, behavior metrics, and other quantitative measurements will give us more data than we’ve ever had before. This wealth of information won’t help the bottom line, though, if we use it to support our current beliefs instead of viewing it thoughtfully and seeking alternative ways to interpret it.
The following are some of the other types of biases we need to be aware of during market research.
Psychologist Eleanor Longden sought help from a psychiatrist as she tried to deal with symptoms of schizophrenia. When one appointment ran very late, she excused herself, telling the doctor, “I’m reading the news at six.” The psychiatrist duly recorded that Ms. Longden had “delusions of being a television news broadcaster.” The reality was Longden really did have to read the news on a student-run TV station, as she describes in Scientific American Mind. This story is as amusing as it is telling, all the more so because one wouldn’t expect a trained psychiatrist to make assumptions or jump to unwarranted conclusions.
The misinterpretation by the psychiatrist is an example of confirmation bias. She was expecting to find evidence of mental illness in her patient, and as a result viewed the slightly improbable news-reading comment as a delusion. The psychiatrist made no effort to clarify the statement. One quick question to explore the topic, or even a shrink-talk, “Oh?” would have revealed the simple, factual explanation.
If a skilled psychiatrist, trained to look for nuances in patient statements, are marketeers also prone to this type of predisposition?
Errors are often made similar to Longden’s doctor. In interpreting customer feedback, or data from surveys and focus groups, it’s a natural tendency to interpret the data in a way that is consistent with what we believe ourselves. As the data rolls in, we want to blurt out, “I knew it!” or, “I told you so!” Rarely do we look for other ways of viewing the data, particularly explanations that might prove us wrong.
There are lots of ways that market research can go wrong, but this one is particularly subtle because it exploits a natural flaw in human thinking.
So much of today's market research is bad. For example, a CEO had achieved past success in selling commodity products at higher prices by differentiating with service and quality. He was convinced that the same strategy would work another market. When a customer survey showed that “price” ranked behind multiple other factors in importance, no one asked, “Why?” The question was asked in a different way, or it didn't became clear this surprising result be explored in more detail.
Instead, confirmation bias kicked in. The survey was viewed as proof that price wasn’t that important, and that a differentiation strategy would allow at least a modest price premium over less skilled competitors.
The new strategy lasted just a few short weeks. Sales fell, and recovered only when prices were rolled back to competitive levels. In retrospect, price was critically important to customers but was never an issue because everyone charged the same price. The survey results were accurate, but were taken to imply something that wasn’t true.
When doing market research or analyzing customer feedback, one has to be careful to look for alternate ways of interpreting the information--e.g., interjecting dissenting voices, asking "what have we missed", and following what appears to be insignificant data points. In the pricing example, there were a few experienced people who questioned the findings and strategy, but their objections were summarily dismissed as mere resistance to change.
In both of the examples above, just a little more exploration would have revealed that the initial assumptions were incorrect. But, since the statements seemed to confirm closely held beliefs and assumptions, nobody took that extra step.
In the coming years, Big Data, behavior metrics, and other quantitative measurements will give us more data than we’ve ever had before. This wealth of information won’t help the bottom line, though, if we use it to support our current beliefs instead of viewing it thoughtfully and seeking alternative ways to interpret it.
The following are some of the other types of biases we need to be aware of during market research.
|
Irrational Escalation
|
This is the tendency to dismiss or ignore new research evidence if it happens to override or undermine a decision that has already been made. For example, if a company has already rolled out production for a new product line, and new research contradicts the actions already taken. The key is to be open minded and realistic relative to new insights.
|
|
Social desirability
|
This is an issue is really an issue on the behavior of research subjects. It is a bias that leads people to make more socially acceptable decisions when in the presence they are being observed or in the presence researcher than if left alone. For example, if a person is being interviewed for testing the effectiveness of a charitable donation program, and they are asked on the ;likelihood of their giving, they are more likely to say yes as they don't want to offend the person on the other end.
You can prevent this bias by testing in a real environment, or by accounting for the skew with some sort of data compensation. |
|
Framing
|
Like with social desirability, framing is a problem with research subjects. It assumes that similar information will yield different conclusions if it is presented in a different way. For example, two advertisements could contain the same three sections; if those three sections are reversed, the audience is getting the same information, but in a different order, and they may draw different conclusions as a result.
This can be compensated for by testing the product or conducting surveys in a randomized order; for example, survey questionnaire randomization. |
Tab Content Three