Leading and misleading questions always yield questionable data, based on which you are highly likely to report findings that can misguide stakeholders. Moreover, decisions they make based on such data could cause an organization's failure rather than lead to its success.
Leading Questions
Questions like those in the following examples lead respondents to give answers they might not otherwise provide if given a less-biased question and a set of balanced answer choices. Although these real-world examples in this article may seem a little extreme, they illustrate the point.
Examples of Leading Questions
Q10. When did you (or will you) stop investing in print marketing?
1. Last year
2. This year
3. Next year
Q20. Vendors that offer free upgrades always win higher satisfaction scores than those that don't. How much more satisfied are you when a vendor offers free upgrades?
1. Slightly more satisfied
2. Moderately more satisfied
3. More satisfied
4. Much more satisfied
5. Definitively more satisfied
Assuming a precondition (e.g., a behavior that is no longer practiced, such as discontinuing an investment behavior) and predisposing respondents with an introductory statement, even if true, are common types of leading questions. Avoid both.
Lack of Parallelism
Another way to mislead respondents is to present a confusing question. It's never good if respondents read a question and then say to themselves, What do they want? That can happen if you have not used parallel construction.
Lack of parallel construction between a question and its answer choices can confuse respondents and increase the prob¬ability of invalid data. Parallel construction is not hard to achieve, but researchers must be diligent when writing questions to ensure parallelism.
The following examples illustrate what to watch for and how to write parallel questions.
The list of responses within question Q30 does not match the list of answers the respondents are asked to give. Either the items listed in the question or the answer choices must change.
Examples of Parallel-Construction Problems
Q30. What percent of per-unit product costs are associated with run-time software elements, electronic components, and mechanical components? (Enter percentages in the boxes provided.)
Percent of Per-Unit Product Cost
Software components (per unit royalties) ____ ____%
Components such as semiconductors and boards ____ ____%
Components like physical housing and engines ____ ____%
Other (please specify): ___________________ ____ ____%
Q40. Approximately how many labels does a single barcode scanner in your organization scan per week? (Select one.)
Scans Per Day
1. Less than 25
2. 25 to 49
3. 50 to 99
4. 100 to 249
5. 250 to 499
6. 500 to 999
7. 1,000 or more
8. Don't know
Question Q40 could be very hard to answer and presents more than one problem.
First, if a company owns two or more scanners, the weekly number of scans could be radically different for each type of scanner in use.
Implicitly, you have asked the respondents to choose one scanner. But which one? Should they choose the newest scanner, the most-active scanner, or the fastest scanner, or use some other criterion for selection?
Asking about the average number of scans is another problem, because the average could be for a very heterogeneous scanning environment, rendering an average essentially meaningless. Additional clarity is required if the researcher wants to interpret the data with a degree of accuracy.
Finally, the question asks about barcodes scanned per week, but the answer section says "Scans Per Day"; does the researcher want the respondents to do the math?
More than likely, during questionnaire revisions, the time frame was changed but edits were not made carefully. Regardless, the question is confusing, and it will yield confusing data that researchers cannot interpret with confidence.
The revised versions, Q30R and Q40R, show a solution for each question. However, the revisions are merely one way to address the issues.
Examples With Parallel Construction
Q30R. What percent of per-unit product costs are associated with run-time software unit royalties, electronic components, and mechanical components? (Enter percentages in the boxes provided.)
Percent of Per-Unit Product Cost
Run-time-software unit royalties ____ ____%
Electronic components (e.g., semiconductors, boards) ____ ____%
Mechanical components (e.g., physical housing, engines) ____ ____%
Other (please specify): ___________________ ____ ____%
Q40R. Approximately how many bar-code labels does your organization's newest scanner scan per day? (Select one.)
Scans Per Day
1. Less than 25
2. 25 to 49
3. 50 to 99
4. 100 to 249
5. 250 to 499
6. 500 to 999
7. 1,000 or more
8. Don't know
Mismatches
Another easy mistake to make is to mismatch the question text with the type of question you're asking.
One common example is the use of the term "primary" in a question that is structured as a multiple-response question, thus creating an oxymoron.
Example of a Mismatch
Q50. What is the primary reason your company will add new [type here] partners?
(Select all that apply.)
[Randomize]
1. Company reputation for excellent staff
2. Product complements existing areas of focus
3. Offers high margins
4. Offers marketing services with highest co-op funds
5. Provides alternative product choices
6. Superior service in the form of sales support
7. Best technology in the market
8. None of the above
9. Other (specify) ___________________________
The text says "primary," but the question is structured as a multiple-response question with instructions to "Select all that apply." The word "primary" means "main"—and so, in the example, the one reason or primary reason. If you want more than one answer, don't use the word "primary."
Poor or Awkward Wording
Always work to root out poor and awkward wording (which leaves room for ambiguity) and the use of poor English (e.g., improper verb tense or poor choice of words and phrases, use of double negatives).
Poor writing will undermine your credibility with respondents and your client. Taking the time to be thoughtful, to write a clear and precise question, is its own reward, but it will also pay dividends.
Examples of Poorly Worded and Confusing Questions
Q60. According to you, what are the key factors that contribute to the success of a company in the present-day poor economic conditions?
A more typical and more efficient way to structure that open-ended question is shown in question Q60R. The respondent has clear instructions, and the answer portion of the question is structured to aid the respondent and to make life easier for the person coding the answers.
Q60R. What, in your opinion, are the key factors that contribute to the success of a [type of] company in the present recessionary economy? (Please briefly describe up to three.)
1. _____________________________________________________
2. _____________________________________________________
3. _____________________________________________________
Q70. Rank the following value-added services in terms or order of importance or level of desirability beginning with the most important 1 meaning "most important" through 5 meaning "least important."
Rank (1 to 5)
1. Access to demonstration equipment ____
2. Co-op marketing funds ____
3. Lead generation ____
4. Access to Web tools ____
5. Sales support ____
How would you answer that question? It includes a typo (the first "or" should be an "of") and two different ways in which to rank the services listed (by "order of importance or level of desirability"), and, as if those problems weren't enough, its instructions are awkwardly structured.
Let's take a look at the same question reworked.
Q70R. Rank the following value-added services in terms of their importance to your organization's success, where the most important is ranked 1, the next most important is ranked 2, and the least important is ranked 5.
Rank from 1 to 5:
____ Access to demonstration equipment
____ Co-op marketing funds
____ Lead generation
____ Access to Web tools
____ Sales support
If "level of desirability" were the criterion, it could be substituted for "order of importance," but you have to pick one criterion and go with it. Otherwise, ask about each in separate questions or at least in separate columns, but only if there is a compelling reason to do so.
* * *
As the examples in this article illustrate, it is easy to make errors when developing a questionnaire. However, the errors are not difficult to correct if you know what to look for and if you keep the study objectives in mind.
For more insights on questionnaire design, see Questionnaire Design for Business Research (Tate Publishing, 2010).
