Questionnaire Design
General Considerations
·
design the questionnaire to fit the medium.
o
phone interviews cannot show pictures.
o
mail or Web survey responders cannot easily ask “What
exactly do you mean by that?”
o
intimate, personal questions are best handled by mail or
computer, where anonymity is most assured.
·
KISS - keep it short and simple.
o
respondents won’t
do a 20-page questionnaire
o
ask yourself what you will do with the information from
each question.
o
place your
questions into three groups:
§
must know,
§
useful to know
and
§
nice to know.
o
discard the last group, unless the previous two groups are
very short.
·
state who you are and why you want the information in the
survey. A good introduction or welcome
message will encourage people to complete your questionnaire.
·
allow a “Don't Know” or “Not Applicable” response to all
questions, except to those in which you are certain that all respondents will
have a clear answer.
·
for the same reason, include “Other” or “None” whenever
either of these is a logically possible answer.
Question Types
·
Researchers use
three basic types of questions:
o
multiple
choice,
o
numeric open
end and
o
text open end (sometimes called "verbatims").
·
Examples of each
kind of question follow:
·
Rating Scales and Agreement Scales are two common types of
questions that some researchers treat as multiple choice questions and others
treat as numeric open end questions. Examples of these kinds of questions are:
·
Question and Answer Choice Order
·
keep in mind when considering Q&A choice order.
o
how order can
encourage people to complete your survey.
o
how the order of questions or the order of answers affect
the results of your survey.
·
ideally, the early questions in a survey should be easy and
pleasant to answer.
o
they encourage people to continue the survey.
o
build rapport with the interviewer.
o
grouping questions on the same topic also makes the
questionnaire easier to answer.
·
leave difficult or sensitive questions until near the end
of your survey.
·
answer order can make individual questions easier or more
difficult to answer.
o
use a logical or
natural order to answer choices
o
present agree-disagree choices in that order. (Presenting in disagree-agree order will seem
odd)
o
positive to negative and excellent to poor scales should be
presented in those orders.
o
in numeric rating scales higher numbers should mean a
more positive or more agreeing answer.
·
Question order
can affect the results in two ways.
o
mentioning
something (an idea, an issue, a brand) in one question makes people think of it
while they answer a later question
§
separating
related questions with unrelated ones can reduce this problem
o
habituation - applies to a series of questions that all have the
same answer choices.
§
some people will
start giving the same answer, w/o considering it
§
people think more when asked the earlier questions in the
series and so give more accurate answers to them.
§
ask a short series of similar questions, then ask one or
more different kinds of questions, and then another short series.
§
change the “positive” answer. This applies mainly to level-of-agreement
questions.
·
“My supervisor
gives me positive feedback
·
“My supervisor
usually ignores my suggestions”).
·
this technique forces the respondent to think more about
each question.
·
answer order can also affect the answers given.
o
people tend to
pick the choices nearest the start of a list when they read a list
o
they tend to pick
the most recent answer when they hear a list
Other
General Tips
·
keep the questionnaire as short as possible.
o
more people will
complete a shorter questionnaire, regardless of the interviewing method. If
a question is not necessary, do not include it.
·
start with a Title (e.g., Leisure Activities Survey).
·
always include a
short introduction
o
who you are and why you are doing the survey.
·
reassure your
respondent that responses will not be revealed, but only combined with many
others
·
include a cover letter with all mail surveys.
o
describe how to return the questionnaire.
o
include the name and telephone number of someone the
respondent can call if they have any questions.
o
include instructions on how to complete the survey itself.
·
appeal to the recipient's sense of altruism ("please
help").
·
if the questions are sensitive in nature, do not have a
space for a name.
o
people may be suspicious and not complete the survey.
·
include all the relevant alternatives as answer choices.
o
leaving out a choice can give misleading results.
o
recent polls that
ask Americans if they support the death penalty "Yes" or
"No" have found 70-75% of the respondents choosing ”Yes.”
o
polls that offer the choice between the death penalty and
life in prison without the possibility of parole show support for the death
penalty at about 50-60%.
o
polls that offer the alternatives of the death penalty or
life in prison without the possibility of parole, with the inmates working in
prison to pay restitution to their victims’ families have found support for the
death penalty closer to 30%.
·
do not put two questions into one.
o
avoid: "Do you buy frozen meat and frozen fish?"
§
"Yes"
can mean meat or fish or both.
§
"Have you
ever bought Product X and, if so, did you like it?"
o
Be as specific:
"Do you ever buy pasta?"
§
yes can include someone who once bought some in 1990.
§
Better:
"Have you bought pasta (other than in a restaurant) in the last three
months?"
·
make sure your questions can accurately tell you what you
want to learn.
o
the way you phrase a question can change the answers you
get.
o
make sure the wording does not favor one answer choice
over another.
·
avoid emotionally charged words or leading questions that
point towards a certain answer.
o
You will get
different answers from asking
§
"What do you
think of the XYZ proposal?"
§
"What do you
think of the Republican XYZ proposal?"
·
avoid technical
terms and acronyms, unless you are absolutely sure that respondents know they
mean
·
make sure your questions accept all the possible answers.
o
"Do you use
regular or premium gas in your car?" does not cover all possible answers.
o
owner may alternate between both types.
o
ignores the possibility of diesel or electric-powered cars.
o
better: "Which type(s) of fuel do you use in your
cars?" The responses allowed might be:
|
·
Regular
gasoline |
·
if you want only one answer from each person, ensure
that the options are mutually exclusive. For example:
o
in which of the following do you live?
|
·
A house |
o
this question ignores the possibility of someone living in
a house or an apartment in the suburbs.
·
rating scale questions (e.g., "If '5' means very good
and '1' means very poor how would rate this product?") are a particular
problem.
o
researchers are very divided on this issue. Many surveys use a
ten-point scale, but evidence suggests anything over a five point scale is
irrelevant.
o
among university graduates a ten point scale will work
well.
o
among people with less than a high school education five
points is sufficient.
o
in third world countries, a three-point scale
(good/acceptable/bad) may be all some respondents can understand.
·
giving a verbal
or written label to each point on a scale, instead of just the endpoints, will
usually yield higher-quality data
·
be sure any rating scale labels are meaningful. For
example:
o
what do you think about product X?
|
It's
the best on the market |
o
a question phrased like above forces most answers into
the middle category, resulting in very little usable information.
·
if you have used a particular scale before and need to
compare results, use the same scale.
o
four on a five-point scale is not equivalent to
eight on a ten-point scale.
o
someone who rates an item "4" on a five-point scale
might rate that item anywhere between "6" and "9" on a
ten-point scale.
·
be aware of cultural factors.
o
in the third world, respondents have a strong tendency
to exaggerate answers.
§
researchers may be perceived as being government agents, with the
power to punish or reward according to the answer given.
·
Always discount
"favorable" answers by a significant factor.
o
The desire to
please is not limited to the third world.
o
people tend to
pick the most positive answer on a scale more often when answering telephone
surveys than other types of surveys
o
there is a tendency to pick agreeing answers on agreement
scales.
o
experiments have shown that more people will agree than disagree.
§
to eliminate this problem ask half if they agree that
"X is good" and the other half if they agree that "X is
bad."
§
then reverse the answers given by the second group.
·
people give answers they feel will reflect well on them.
o
more people say they will vote than actually will vote.
o
more people say they go to museums or libraries than
actually do.
o
this problem is most significant when your respondents are
talking directly to a person.
o
people give more honest answers when answering questions on
a computer. Mail surveys are in-between.
·
in personal interviews it is vital for the interviewer
to have empathy with the interviewee.
o
interviewers should try to "blend" with respondents in
terms of race, language, sex, age, etc.
·
leave your demographic questions (age, gender, income,
education, etc.) until the end of the questionnaire.
o
Exceptions: any
demographic questions that qualify someone to be included in the survey.
·
do not have an interviewer ask a respondent's gender,
unless they really have no idea. Have
the interviewer fill in the answer themselves.
·
paper questionnaires requiring text answers, should always
leave sufficient space for handwritten answers.
·
leave a space at the end of a questionnaire entitled
"Other Comments."
·
consider the layout of your questionnaire especially on paper,
computer direct and Internet surveys.
o
make it attractive, easy to understand and easy to
complete.
o
make it easy for
your data entry personnel
·
Q&A choice
grids are popular with many researchers.
They
o
look attractive and save paper, or computer screen
space.
o
avoid a long series of very repetitive question and answer
choice lists.
·
look at the
following layouts and decide which you would prefer to use:
o
Do you agree,
disagree or have no opinion that this company has:
|
·
A good vacation
policy - agree/not sure/disagree. |
o
alternative
layout is:
o
Do you agree,
disagree or are not sure that this company has:
|
|
Agree
|
Not Sure |
Disagree
|
·
|
|
A
good vacation policy |
3
|
2 |
1 |
·
|
|
Good
management feedback |
3 |
2 |
1 |
·
|
|
Good
medical insurance |
3 |
2 |
1 |
·
|
|
High
wages |
3 |
2 |
1 |
·
|
o
the second
example shows the answer choices in neat columns
o
is easier to read
o
speed data entry, if you are using a paper questionnaire.
Pre-test
the Questionnaire
·
test a questionnaire with a small number of interviews
before conducting your main interviews.
·
you should test the survey on the same kinds of people
you will include in the main study.
o
if that is not possible, at least have a few people,
other than the question writer, try the questionnaire.
o
A test run can
reveal unanticipated problems with question wording, instructions to skip
questions, etc.