The value of strong questionnaire design, complemented by the task of high-quality sample development, is not fully appreciated. Too often researchers relegate those two essential building blocks of market research to a status secondary to the analytics and presentation of results. Big mistake!
A strong questionnaire design and, specifically, conducting an audit to ensure quality, is the focus of this article.
Questionnaire Design
Research Axiom No. 1: You can never fully recover from a poorly written questionnaire.
No manipulation of data points, regardless of how cleverly done; no amount of analysis, regardless of how brilliantly executed; and no degree of interpretation, regardless of the intellectual insights, can save you from a poor research foundation. Your building will collapse as if shaken by an earthquake!
Questionnaire design is a task repeatedly given insufficient time and attention. Clients and research professionals alike often underestimate the effort it will take to develop a well-structured and concise research instrument.
What amazes me most? Project leaders relegate the task to a status depicted in this attitude: "Once the questionnaire is done, we can get on with the important stuff, like analysis and reporting."
The assumption that analysis work is the essence of the research, and the expectation that interpreting the results is where the mastery of research ultimately lies, is a mystery to me.
Have we not pounded the concept of garbage-in garbage-out into the heads of research students across the globe? Can new Internet tools substitute for critical thinking and the hard work of aligning the research instrument to the purpose of the study to answer the business questions that sponsors paid to learn?
If that sounds like a rant, I'm guilty as charged, but I will end it with an analogy: If you have not studied and then practiced writing poetry, would you expect to publish a book of poems simply because the VP of marketing asked you to? Designing a good-quality research instrument probably takes less talent than being a good poet, but it's close.
What does that have to do with questionnaire auditing? That's simple. If you work hard up front to create a great questionnaire, the audit will be quick and painless. If you don't... well, you already know the answer.
Questionnaire Auditing
Our research on research has shown that even experienced researchers are less diligent about quality than one would hope. Research and questionnaire design, in particular, require not only science but also art; and if the proper fundamentals are not applied, the product is less than artful.
The purpose of an audit is to help you avoid errors in structure and allow you to focus more on the art.
A Questionnaire Quality Control (QQC) audit using a standard set of parameters provides valuable feedback to designers, the project teams, and clients. A good QQC audit has three parts:
- A set of standardized quality checks: (a) Checks that maximize objective assessment and minimize subjective evaluation; (b) A QQC score based on the results of the standardized portion of the audit, taking into account repeat mistakes
- A clear mechanism for testing a questionnaire's overall length: (a) Testing the questionnaire during development; (b) Testing the questionnaire after it is finalized and programmed; (c) Checking the longest paths
- An expert review to examine unique aspects of the questionnaire and to identify issues or problems that might need attention, correction, or enhancement
Questionnaire Design Problems
The QQC audit can identify problems and errors commonly encountered during the task of questionnaire design. However, in our experience, after providing a review of a questionnaire and telling the author it needs work, the person frequently asks, "What's wrong with my questionnaire? It seemed perfectly fine to me."
Authors don't want to hear that what they've created is less than masterful, but there is more to questionnaire design than asking questions the way you do in everyday discourse. A well-structured audit will be helpful and provide actionable guidance to the author(s) of research instruments. It always helps to receive an impartial review.
Here is a partial list of audit items to help answer the "What's wrong with my questionnaire?" question:
- Answers not mutually exclusive
- Double-barrel questions
- Unbalanced scales
- Lack of parallel construction
- Terminology that creates ambiguity
- Definition problems
- Scary matrix questions
- Lack of instructions
- Poor or confusing wording
- Inventory question overload
- High respondent burden (e.g., too much math)
- The overly long research instrument
If you are having trouble designing a questionnaire, you are not alone. Through the application of a multipoint questionnaire audit, we have repeatedly shown that questionnaires do not meet the requirements of sound questionnaire design. Here are three examples of QQC audits we've conducted:
- A director of research from a well-known university had a QQC score of 65.
- The owner of a LinkedIn Group related to market research had a QQC score of 67 (and the questionnaire had fewer than 10 questions).
- The owner of a market-research company repeating a study done more than five times scored 58 on the QQC audit.
No, you are definitely not alone.
Final Message
Questionnaire design is not an end in itself; rather, it is a means to an end.
Its purpose is to illuminate the subject under investigation, to bring forth content that one can use to develop insights into the marketplace. It should help decision makers make better decisions than they otherwise would make if the information were not available.
To accomplish that goal, practitioners must construct questionnaires with great care and skill while avoiding errors throughout the process.
As Albert Einstein once said, "We can't solve problems by using the same kind of thinking we used when we created them."
If you do not care about the quality of the research you conduct, well, shame on you—but at least recognize that a questionnaire of good quality is a necessity for self-preservation. Enough said.
