Open-Ended Questions vs. Closed-Ended Questions

Posted by Guest Blogger

Jan 28, 2009 8:30:00 AM

Recently we had a user who had written a questionnaire with 40 open-ended questions. It was easy to write and easy to set up in Vovici v4 (every question was an essay question). Alas, it was not at all easy to analyze. Our user had to read a list of verbatim responses for every question, then categorize them and tally the categories.

If open-ended questions are easy to write, but hard to analyze, then closed-ended questions can be hard to write but easy to analyze. The difficulty in creating a closed-ended question is coming up with the appropriate choice list, a list that covers the most common answers and doesn't bias responses.

For instance, it is easy to ask:

what is your favorite color

Do this, though, and you will get results something like this:

beige

blue

Bluw

green

purpel

red

Black

blue

fire engine red

green

purple

red

black

blue

Gray

grey

red

sea green

black

blue

Green

orange

red

sky blue

Blue

blue r0x!!

green

orange

red

yellow


While sorting and categorizing colors is not as difficult as sorting and categorizing lists of products and vendors (something I did a lot of early in my career), it can still be a time-consuming and unnecessary step.

The solution is usually quite simple: give the user a list of choices to select from. For years, when demonstrating the word processor UI of our survey software, I would begin with this example:

Favorite_color_question

Red-Orange-Yellow... that choice list is simply the ROY G. BIV mnemonic I learned for the color spectrum in eighth grade (thank you, Mr. Gesacion!). However, I don't think many people other than Amy Ray and Emily Saliers have "indigo" for a favorite color. Clearly, I never intended to field this particular question, but it does make the point that careful thought must be given to writing the choice list.

For instance, here's an example poll I found that offers too many choices:
Favorite_color_poll

Clearly, this wasn't intended as serious research, but it does illustrate some best practices people often fail to follow when coming up with choices.

Best Practices for Writing Closed-Ended Questions

  1. Make sure the list contains all common choices. Have a co-worker review the list to make sure no obvious choices are missing. How can you ask people their favorite color and leave out one of the primary colors?! A co-worker should have quickly spotted that "Yellow" was missing from the choice list. (For more on this point, see Favorite Color Survey Results: How Short Choice Lists Lead to Wrong Answers.)
  2. Provide the respondent with an "Other - please specify" choice where they can type in their answer if no other choice is appropriate. This is an important "escape valve", so that you don't need to worry if you missed some minor choices. It also lets you purposefully leave out the hot-pink choices. You may need to categorize and tally the answers to this part of the question, if many people select "Other", but usually it's such a small percentage of overall responses that you don't need to.
  3. Try not to clutter the list with unlikely choices. No one selected "Magenta" or "Hot Pink" and only one respondent selected "Maroon" or "Gray". Which choices will be unpopular can be difficult to determine in advance of course; in fact, I just wrote a question on Monday with only four choices: we fielded it Tuesday and one choice was not selected by any of the 140+ respondents. Oops! Some choice lists you will refine as you field the question in subsequent surveys. When in doubt, include the choice.
  4. Arrange the choices in a logical order. Why aren't "Sky Blue", "Lime Green" and "Hot Pink" next to their base colors? Why isn't the list in some type of order, rather than reflecting one choice after the next, stream-of-consciousness style? The appropriate order for your choice list may be numeric order (for salary ranges, for instance) or some other natural order. If nothing suggests itself, sort alphabetically.

Sometimes, though, you simply can't anticipate in advance how respondents might answer. In that case, use an open-ended question but field your survey to a small sample. Once you get about 30 responses, reading the results should help you to craft a good choice list. Then write a fuller questionnaire with what you learned and field it to a larger sample.

Finally, it should be noted that the choose-one-and-specify question is really a hybrid question, mixing a closed-ended list with an open-ended response. As a result, it represents the best of both worlds as a question type. Sometimes you don't have to choose between using open-ended and closed-ended questions!

Topics: survey software

Share this content!

    

Search Verint

Select an RSS Feed

Subscribe to Email Updates

Sort Posts by Author

See All