Most international development programs involve one or more surveys – whether it’s baseline surveys, endline surveys, needs assessment surveys, or feedback forms from participants. This guide explains how to write clear, concise survey questions that will collect accurate data. The inspiration for many of these tips comes from The Survey Research Handbook by Pamela Alreck and Robert Settle.
This advice is for:
- Basic quantitative surveys such as feedback forms, needs assessments, simple baseline and endline surveys, etc.
- Written surveys completed by individuals who are literate.
This advice is NOT for:
- Complex baseline and endline surveys or research studies.
- Developing new measurement instruments for use in research (e.g. psychological instruments for measuring concepts such as confidence, motivation, etc).
- Graphical surveys completed by people with low literacy.
- Qualitative focus groups or interviews.
Don’t write new questions unless you have to
There are thousands of NGOs, UN agencies and donors doing surveys all the time. Chances are you aren’t the first person to do this type of survey. So before you even consider creating new survey questions, do some research to find out if other people have done something similar which you could use. For example, if you’re planning to measure poverty then the Grameen Foundation already have a standard tool to do this. Or if you want to measure the prevalence of diarrhea the Bristol Stool Scale has been created specifically for this purpose.
The other benefit of using standard questions is that you can compare the data you collect to what other people have collected. For example, if you use the same questions as a national survey then you can compare the results in your target area to the national results.
Write the questions in the local language first
In international development it’s quite common to have managers or technical advisers who don’t speak the local language. If you’re in this position it can be tempting to write the survey questions in your native language (such as English or French) and then have someone translate it.
If you’ve ever been to another country and seen poorly translated signs you’ll understand why this is a very bad idea. Even the best translator can have difficulty getting exactly the right meaning for a question. I’ve even had cases where the translator completely reversed the meaning of the question by accident, making the data useless.
To avoid this problem the questions should be written first in the local language, and then translated into other languages, such as English. If you don’t speak the local language then work with someone who does. Discuss each of the questions verbally in a language you both understand, and then have them write the question in the local language. Once they’ve written the question they can translate it into a language you understand.
Sometimes it’s not possible to write the questions in the local language, particularly if you’re using standard questions from another survey. In that case your best option is to use a process called “back translation”. Start by asking one translator to translate the question into the local language. Then ask a different translator (who hasn’t seen the original) to translate it back again. Then have both translators compare all three versions to identify any discrepancies or problems with meaning (see example below). If it’s a very important survey then you may want to have four translators working independently – two to translate, and two to back translate.
Make sure each question focuses on one specific issue
Before you start writing a question it’s important to be absolutely clear about what you want to know. In the example below, the first question asks about which health centre the participant likes the best. This is not the same as asking about which health centre they are most likely to go to. They may like the health centre in the capital city, but they might not actually go to that centre because it’s too far away or too expensive.
The following example is similar. In the first question someone may say that child care services are very important for society in general, but it might not be important to them specifically if they don’t have children. The second version makes it clear that the question is asking how important it is for them personally. If you think there might be some confusion you can use italics or bold to highlight the key words in the question.
Finally, each question should only ask about one issue. If you need to ask about two issues then use two different questions, as in the example below.
Use simple language
Unless your target audience are technical specialists, survey questions should never include technical terms, acronyms or abbreviations. The language used should be as simple as possible, and suitable for the education level of the target audience. In some cases it may be necessary to use local phrases or colloquial terms to make the questions clear.
If you’re writing the survey in a language supported by Microsoft Word then you can use Word to test your survey’s readability (see instructions here). Word will do the Flesch Reading Ease test on your document which rates text on a 100-point scale. The higher the score, the easier it is to understand. You should aim to score as high as possible on this test, ideally around 80-90 points.
Word will also calculate the Flesch-Kincaid Grade Level. This tells you which U.S. school grade would be able to read the document. For example, a score of 6.0 means that an sixth grader (someone who has finished primary school) can understand the document. If you use this test it’s important to remember that the U.S. school system is not the same as school systems in other countries. For example, if the quality of education is lower in your area then someone who has completed six years of school may only be able to read a survey that is at the fourth grade level according U.S. system.
Don’t include examples
Sometimes it can be tempting to add examples to a question in order to make it easier to understand. The problem is that examples can bias people’s answers. In the first question below people are more likely to remember buying malaria medication because it is included as an example in the question. This will bias the results, so it’s better not to include examples at all in the question.
Don’t expect people to remember too much
A lot of surveys in international development ask people to remember things. This could be how many times they washed their hands yesterday, whether they voted in the last election, or how many days of school their child missed last month. People are notoriously bad at remembering things accurately, particularly when it’s a boring topic (even if you think hand washing is very interesting, chances are they don’t).
Because of this it’s important not to ask people to remember too much. The following table gives some examples of different types of events and how long people are likely to remember them for. You can use this as a guide when developing your questions. For example, it would be reasonable to ask someone how long they spent watching TV yesterday, but you shouldn’t ask how long they spent watching TV in the last month or year.
| Type of event | Examples | Ability to remember |
| Everyday things | Cooking, watching TV, going to school | 1 day – 1 week |
| Common but unusual | Being sick, buying household items | 2 weeks – 1 month |
| Major events | Death, marriage, buying a house | Years |
Keep all questions neutral
If you want people to give honest answers then it’s important to avoid questions that might encourage them to answer a particular way. A leading question is one that suggests a particular answer rather than being neutral. In the following example the incorrect version of the question suggests that the new policy is good, while the correct version is neutral.
You should also avoid emotional or extreme language which makes it difficult for someone to answer a particular way. For example, if someone answered “no” to the first question below it might sound like they want the country to be invaded. The second version of the question is more likely to get an unbiased response.
Hide what is socially desirable
People like to give answers that will make them look good. It’s common for survey participants to overestimate how much exercise they do, and underestimate how much junk food they eat. Because of this it’s important to write survey questions in a way that makes it unclear what the socially desirable answer is.
In the example below men are more likely to answer “yes” to the first question because emphasises the social norm that a man should earn more than his wife. The second version of the question is more likely to get an accurate response because it avoids this.
In some cases you may need to ask about very controversial or socially undesirable things, such as men having sex with men or government officials engaging in corruption. In these cases it can be extremely difficult to get honest answers. One way to increase the number of honest answers is to write the question in a way that normalises the behaviour.
For example, rather than asking “Have you had sex with another man before?” you could ask “How many other men have you had sex with?”. The second version of the question makes it sound like the interviewer is expecting the participant to have had sex with other men, and so the participant is more likely to respond honestly.
Write the introduction carefully
The way a question is written can determine how likely people are to give honest answers. The introduction given at the start of the survey can also impact this. It’s important to explain the purpose of the survey in the introduction, but you should leave out details that might bias people’s answers.
For example, if you are collecting feedback on a particular NGO you might want to avoid telling participants that the survey is being run by that specific NGO because people may be more likely to give positive answers. You should also encourage people to give honest answers by explaining that their real opinion is very important, and ultimately more useful for you.
The introduction should also include a process for getting informed consent. For more information on this see: How to get informed consent.
Continue to Part 2
Photo by plings
