5 Mistakes That Kill Survey Data (and How to Avoid Them)

Whether you are getting feedback from program participants, members, donors or other stakeholders, an online survey can be the easiest and most efficient way to do it. Of course, online surveys are not a one size fits all solution for collecting data to answer your burning questions, but they can be a cost effective method to gather certain data for your organization. No matter your field or audience, you have to follow some best practices if you want to make use of your data.

At Inciter, we develop surveys for a variety of reasons. And if you already have a survey but think you could be doing more with it, we can help you clean up your instrument and fine tune your questions, too. By asking good questions about what you want to know, who has the information, and what you will do with it once you get it, and then using best practices in survey design that we have honed over the last 20 years, you can get the insights you need.

In this article, we’ll show you some common mistakes to avoid and offer remedies that just might save your data (or your butt when you present the results to the board). We will also show you some hilariously horrible examples of surveys gone wrong from the account @badsurveyQ on Twitter that makes us simultaneously laugh and cringe.

Questionnaire development: there is a wrong way. pic.twitter.com/y1USqEE8Gh Steven R. Shaw (@Shawpsych) December 3, 2018

5 Questions That Kill Survey Data:

1. Leading Questions

Leading questions are biased questions that use subjective adjectives or context-heavy wording. They use a positive or negative tone that influences an informants response. Avoid writing leading questions by leaving out adjectives that elicit a feeling and staying away from wording that implies there is a right answer.

Leading Question: Were you worried about the quality of the course before you signed up for it? Alternative: What were your expectations for the course?

2. Double-Barreled Questions

Double-barreled questions touch on two topics or ask two questions in one. These are fairly common and much like loaded questions, they can appear benign. They ruin data because theres no real way of knowing if the participant was responding to both questions, or just one.

Double-Barreled Question: Do you agree that smoking should be illegal because it causes cancer?

Alternative: Do you believe smoking should be illegal? Then, ask if smoking’s association with cancer is one of the reasons they believe this.

Heres a great example of a completely unanswerable double-barreled question:

pic.twitter.com/tP8NcV9ZXV Eric Winsberg (@ewinsberg) May 21, 2020

3. Loaded Questions

Have you ever been asked a question on a survey that didnt apply to you, but you werent given an option to indicate that? It was probably a loaded question. They are common and often appear benign.

Loaded questions are similar to leading questions. The difference is that leading questions slightly influence a respondents thinking, but loaded questions make an assumption about the participant that can force them to choose a response that inaccurately represents their true reaction. They can assume the respondent has a certain opinion, is familiar with a topic, has had an experience, or possesses a characteristic.

Loaded Question: What time of day does your favorite TV show air?

This question relies on quite a few prerequisites. To give an accurate response, the respondent has to own a TV, watch it regularly, have a favorite TV show, and they have to watch that TV show when it airs. To avoid asking loaded questions, you should ask qualifying questions first. Make sure your participants are only being presented questions that apply to them.

4. Extreme or Absolute Questions

Avoid extreme and absolute words. Using Always and Never in your question limits your response options to Yes and No.

Other words to avoid that arent quite as obvious include only, just, ever, must, and all.

Absolute Question: Do you always floss your teeth every day?

Alternative: How many days a week on average do you floss?

5. Nonsensical or Confusing Questions

Avoid confusing your audience. Don’t assume they will be familiar with jargon and acronyms. Always consider how reasonable your requests are. It’s simple: don’t ask your audience something they don’t know or may not be comfortable giving to you.

We highly encourage you to have your survey thoroughly reviewed and edited by at least one of your team members. Everyone makes mistakes! If your errors go unnoticed, they could cost you your data. Or worse, you could end up on @badsurveyQ like our other favorites below.

I can’t help it. I just love this thing. pic.twitter.com/4i62zsjXAX Dave Kellogg (@Kellblog) June 23, 2020

red lobster helps define who I am pic.twitter.com/PBTIJOYV63 Saint Spaghettio (@tasha_why) January 19, 2020

Let us know if you have any questions or additional topics you’d like us to cover by emailing contact@inciter.io.