Types Of Bias To Watch For In Product & UX Research

Any product research requires an extensive amount of resources. There are numerous stages of product research process starting from the brainstorming and ideation phase to final decision making and final product. Cognitive bias is one such component, but an unwelcome one. The fact is that cognitive bias does more harm than good to the process of product research. Let’s shed some light on the topic.

Cognitive bias or reasoning bias, is the inclination, disagreement, approval towards a specific idea or proposition based on one’s principles and situation that is unreasoned. They can exist at all levels - participant, moderator, and researcher, and alter the design persona. Here are some of the types of research bias:

Knowledge Bias

One of the common biases, knowledge bias, is the easiest of intruders in the product research process. The bias comes into play when the respondent in question has prior knowledge about the product and knows a bit about its pros and cons. This prior information about or owning the product makes their judgment subjective. The association with the old product is a hindrance. This association acts as a blinder against the new product. So no matter how good the product is, even if equipped with better features, respondents will not be open to the idea of feedback for a new product. They have an associative response based on the product which hinders objective reasoning. To mitigate this bias, one needs to focus on motivational factors for the respondent to provide an unbiased account.

Confirmation Bias

People love to deliver account for their beliefs and confirmation does just that. It is the inclination nature of humans to stand firm with their beliefs and hypothesis, and evade confrontation to those beliefs as much as possible. However, this evolution characteristic of humans also leads to deliberation. Deliberation to favoring results and disregarding the rest of the plot. Thus, it can hamper a test as personal beliefs might influence the outcome instead of the rational cognition. Also, add to the fact that it is common and can affect both quantitative and qualitative research, the task of refining becomes much more challenging. However, for that to happen, there’s need of large sample space which somewhat guarantees for different feedback because of different mentalities.

Framing Bias

Unlike confirmation bias, framing bias is about the way of presentation rather than beliefs. Framing, as the name suggests, molds responses to a specific frame, i.e., presentation. This type of bias comes in the form of answers that are heavily dependent on the presentation style. Hence, the outcomes differ according to the frame of queries. Framing bias depends more on the moderator or researcher than the responder. Queries and methods used, influence the outcome which solely relies on the researchers. To counter this, one needs to be very careful while framing the questions. There should be any reflection of ideas from the researcher’s or moderator’s perspective. One should avoid leading questions. Instead of asking ‘What you like about the product?’, The question should be ‘What’s your opinion about the product?’. Even ego affects being objective. So putting ego aside is a priority to have a clear vision of presented information. One needs to accept feedbacks, suggestions with an open mind and act accordingly on the information acquired.

Social Desirability Bias

Social Desirability is an evolution consequence. The tendency to be liked and accepted by society is the same thing that hinders research. We are hard-wired to be acknowledged by society. So, no wonder we deviate from our actual necessities just to be accepted in the community. This bias kicks in when questions begin pricking our personal spaces. As one would expect, it all leads to inaccurate reports generated due to inexact responses. From the responder’s perspective, it is the matter of craving something and asking for something else. Unconditional positive regard can be used to control [social bias](https://www.survata.com/market-research/resources/social-desirability-bias/). That is, there is no wrong answer and participant is allowed to express their opinions. Even the questions could evoke third person response. This aids for better gauging of the participant’s feelings because of indirect questioning.

Attentional Bias

Attentional bias is the affinity to pay attention to some things while ignoring others. Attention bias impacts both perception and decision making and becomes worse when there’s a time constraint. We as humans evolved in a certain way to avert danger. Hence, we tend to focus on stimuli that pose a risk to our existence more than anything else. This intense concentration on the danger even affects our memory. Also, add to the fact that attentional bias can be affected by emotional state, it makes our bad choices and decisions. Hence, the feedback that one receives could be a deception.

Cluster Illusion

Cluster illusion is finding patterns in data where there isn’t any present. When working with a small pool of sample, there is a high chance that a set of information project a pattern. This event is generally a rare coincidence which is mistaken for a pattern. Patterns do make it easier to process information. However, one's needs to understand that any pattern is a random event, not the whole result. It is clear that the use of a small sample size does not account for the larger picture. The lack of diversity in sample affects the results/outcomes and might hint towards a different idea. So, not distinguishing pattern and relying solely on results helps to mitigate the bias.

Illusion of Transparency

When there’s an unknown involved, our mind starts assuming things. The illusion of transparency is something that falls into that category. This bias is the case of assuming things that have not been said and overestimating the extent of people's thinking or conveyance. Apart from being verbal, communication occurs through body language, and other non-verbal ways too. When taken into account as a whole, the verbal and non-verbal signs create an illusion of communication. There is an ambiguity of message delivered. The message conveyed, and the understood message could be two different things. Affirmative feedback from the responder about the response helps to make sure the moderator or researcher gets the point.

Bandwagon Effect

Following a trend could sometimes be risky. Just because majority follows something does not make it right. What’s more dubious about a bandwagon is the uncertainty of its existence. It might be a wild act of pouring resources into something that might wither away in the next two or three years. This decision will only result in the outlay of additional resources for adaptation when it dies down. So, instead of focusing on the competition, one needs to focus on its customers for better criticism and insight.

Culture Bias

Where we come from also leads how we make decisions and respond. Making assumptions, be it making decisions or responding to questions, should not be influenced by culture. There are two aspects, ethnocentrism, and cultural relativism. While ethnocentrism talks about judging other cultures using one's culture as a source of reference; cultural relativism makes the expectation of understanding one's culture based on their beliefs and ideas. Culture should not define the design persona. In the case of researchers, they could offset this by unconditional positive regard. In addition to this, one should be well aware of his/her cultural assumptions and make sure it does not intervene in decision making.

Sunk Cost Fallacy

The strong association with things at an emotional level leads to cognitive biasing towards it even though it might cost much damage. In the case of researchers, stretched association with data subtlety influences them. At some point, it might develop into obsession over findings. This higher degree of affection acts when dealing with large sample sizes or data. One can fight this by analyzing data in smaller volumes which helps to concentrate and avert the risk of misdirected outcomes.

Decoy Effect

In some cases, there is subjective participants’ response. Other existing options influence the chosen option, i.e., some options might hold superiority over others. Anything more than two options and decoy effect starts influencing. In the case of two options, they have their superiorities and are independent of each other. Upon introduction of another option, there’s scope of comparison. An option can have a positive effect on some options and an adverse effect on some options. The superiority of options affects in some or other aspect in decision making as a simple task of choosing options becomes a task of weighing relevance with each other.

The Halo Effect

Sometimes there’s the [risk of participants making assumptions based on a single positive attribute. This single factor guides them through the whole testing process which does not help to get a definite opinion. This act is apparent when there is a lack of information or if the respondent is not mentally stable. Tired minds will work the most straightforward way out and, i.e., by assumption. Any little information the respondent gets about the product; they use it as a baton to navigate through any questions that come their way. To get proper feedback, the respondent needs to have full information about the product and not provide answers based on any single attribute. They need to weigh down all the aspects before answering.

Hindsight Bias

It is reliant on the assumption of the respondent. A person’s lack of knowledge paired with their assumption associates reasons for past occurrences without any factual evidence. As discussed earlier, our mind loathes vagueness and hence, assumes things. Although the things might not be right, the person hardly gets to know unless the face a challenge to their reasoning. Hindsight bias is something the respondent might not be aware of, but the researcher needs to be. A researcher needs to be careful about this cognitive bias and assess it with evidence to support it.

Here’s the summary of some of the product research biases that one needs to account for:

Knowledge Bias: The subjective judgment based on prior knowledge of product

Confirmation Bias: Accounting for own beliefs and evading confrontation

Framing Bias: Dependency of responses on the way of presentation

Social Desirability Bias: Response based on the tendency to be liked and accepted by the society

Attentional Bias: Response based on a strong focus on stimuli posing a risk

Cluster Illusion: Finding patterns in data where there isn't any present

The Illusion of Transparency: Difference between the message conveyed and the understood message

Bandwagon Effect: Following a trend because of others doing it

Culture Bias: Responses influenced by cultural perspective

Sunk Cost Fallacy: Biasing due to emotional attachment

Decoy Effect: Superiority or domination of an option over other options

The Halo Effect: Responses based on one positive attribute

Hindsight Bias: Reasoning past events without any factual evidence

Product research is a complex process. The involvement of various components, although helps in the development, but also carry the risk of decision biasing from these factors. The above few biases are some of the common ones that one might encounter when involved in product research and development. That said, any study cannot be entirely unbiased. There will be some amount of bias induced. The key to product development is to recognize and mitigate these biases.

author: patrick algrim
About the author

Patrick Algrim is an experienced executive who has spent a number of years in Silicon Valley hiring and coaching some of the world’s most valuable technology teams. Patrick has been a source for Human Resources and career related insights for Forbes, Glassdoor, Entrepreneur, Recruiter.com, SparkHire, and many more.


Help us by spreading the word