Poorly designed questions and scaling problems can derail your research efforts faster than you can say "the cat in the hat"!
To help you avoid a few of the more common and onerous problems, we will explore two separate but related questionnaire-design issues: matrix questions, especially the big, scary kind, and unbalanced scales, which provide data that is, at best, difficult to analyze and, at worst, useless.
Scary Matrix Questions
First, we examine what we will refer to as scary matrix questions (SMQs). A colleague of mine who likes that phrase said he imagined research professionals texting in acronyms: "OMG did U R the RI with all the SMQs? LOL." (Translation: Oh my god, did you read the research instrument with all the scary matrix questions? Laughing out loud.) Don't be the subject of such laughter.
The use of matrix questions that could scare a respondent into dropping out is counterproductive. Yet matrix questions are used frequently and written with less care than you would write an email.
Those considerable and overwhelming questions are used typically to collect a large number of data points in a relatively small amount of space.
One justification is to create a data set that will provide enormous differentiation using a large set of variables and values. The more likely outcome is tired respondents who straight-line their answers (i.e., select all 3s, 4s, or 5s) as a way to move through the task quickly.
What's the outcome? Rather than a highly differentiated set of responses, the research creates the equivalent of white bread and adds very little to the study's ability to gather insights.
Another rationale often cited is a misguided notion that the number of questions will be fewer and therefore the use of large matrix questions is a good and acceptable approach. Nothing could be further from the truth.
Large matrix questions place a huge burden on respondents. Moreover, in worst-case scenarios (which I have seen more than once), multiple matrix questions appear in one questionnaire.
One case in point had five such questions back-to-back, each with more than 20 rows and 3-5 columns with drop-down menus that asked for scale scores related to satisfaction or capabilities. Those five questions attempted to collect more than 400 data points.
How many respondents do you think finished, and of those who did... how many provided thoughtful answers for each item? There is no way to know for sure. However, I am guessing (and you probably are, too) the odds were not in the researcher's favor.
Much-simpler matrix questions than that example have proven difficult to execute successfully. When you create a matrix question and your battery of attributes is longer then what fits on one screen, that's a red flag.
At the very least, consider dividing the items into two questions; often, there are two constructs or themes that you can use to divide the items logically.
Since many, if not most, matrix questions use scales to collect information, the second part of the discussion is about scale structures.
The Importance of Balanced Scales
The first rule for all questionnaire-design work is, Do not make your respondents work harder than necessary.
That is good general advice when designing a questionnaire, and it is particularly true when you construct questions that use scales. If the scales are hard to use, you are placing an undue burden on respondents. If the scales are awkward, the respondents will be frustrated. If the scales are unbalanced, you will get flawed data.
Three examples (below) will probably clarify those points more effectively than a long description of the issues.
Example 1: A One-Sided, or Skewed, Scale
Let's start with a classic example, which many a survey taker has had to endure. A one-sided, or skewed, scale shown in the following example was taken from a satisfaction study.
It's doubtful the imbalance in the number of positive and negative responses fooled the respondents who took the time to read the questions carefully. Companies that use this tactic fool only themselves and look foolish to the survey taker.
Example 2: The Nonparallel Scale
The next form of an unbalanced scale is the use of nonparallel construction. In the following example, the scale starts with the idea of negative on the left and migrates to the idea of strong on the right. More than likely, the error was simply an oversight by the designer.
However, it can cause problems for respondents and will be a problem downstream for the person analyzing the data. By the way, in case you didn't notice, the nonparallel scale is another example of skewed-scale construction.
Example 3: The Big-Leap Scale
Finally, at least for this article, we have the third unbalanced-scale issue. It features an internal problem that will frustrate many respondents. At first glance, it looks OK, but it has what we will call the "big leap" scale problem. The person writing the questionnaire created a big leap between the first and second scale points and the fourth and fifth scale points.
When respondents are asked a question with that set of items, they are likely to be frustrated because, for some people, the answer they want to give is not on the scale, so none of the options given reflects their experience accurately. In other words, they do not want to say it was "significant" and yet the ROI might be better or worse than "somewhat," leaving them with a dilemma.
* * *
Consider those issues the next time you design a study, or point them out to a colleague if you are asked to review a research instrument. Be gentle, so your colleague hears and understands the issue and can correct it.
Finally, at the risk of being obvious, always provide clear instructions. It is amazing to me the number of times clear and simple instructions are not used. Ironically, the simpler the questionnaire, the more likely it is that instructions are not included.
Perhaps what the researcher wants seems obvious enough to the researcher, but that is a potentially fatal flaw, and the best advice is to never assume anything!
Remember: It is never about what you want; it is always about what the respondent can and is willing to provide you.
If you make the respondent's task impossible or extremely difficult, you will reap what you sow!
