How to pretest and pilot a survey questionnaire

It’s important to test your survey questionnaire before using it to collect data. Pretesting and piloting can help you identify questions that don’t make sense to participants, or problems with the questionnaire that might lead to biased answers. This guide explains how to conduct basic pretesting and piloting for a survey.

This advice is for:

• Basic quantitative surveys such as feedback forms, needs assessments, simple baseline and endline surveys, etc.

This advice is NOT for:

• Complex baseline and endline surveys or research studies.
• Developing new measurement instruments for use in research (e.g. psychological instruments for measuring concepts such as confidence, motivation, etc).
• Qualitative focus groups or interviews.

Any testing is better than no testing

People often think that testing a survey takes a long time. They think they don’t have the time or resources for it, and so they end up just running the survey without any testing. This is a big mistake. Even testing with one person is better than no testing at all. So if you don’t have the time or resources to do everything in this guide, just do as much as you can with what you have available.

As a general rule, you should aim to pretest all your surveys and forms with at least 5 people. Even with this small number of people you’ll be surprised how many improvements you can make. Piloting is only really needed for large or complex surveys, and it takes significantly more time and effort.

Pretesting

Find 5-10 people from your target group

Once you’ve finished designing your survey questionnaire, find 5-10 people from your target group to pretest it. If you can’t get people from your exact target group then find people who are as close as possible. I once designed a survey that was going to be completed by garment factory workers in another province. There wasn’t enough budget available for us to travel to that province to pretest it, so we found some garment factory workers in our own province to test it.

Try to get a range of different people who are representative of your target group. For example, if your target group is young people aged 15-25, try to include some who are younger, some who are older, boys and girls with different socioeconomic backgrounds.

Although 5-10 people might not sound like many, you will usually find that most of them have the same problems with the survey. So even with this small number of people you should be able to identify most of the major issues. Adding more people might identify some additional smaller issues, but it also makes pretesting more time consuming and costly.

Ask them to complete the survey while thinking out loud

Once you’ve found your testers, ask them to complete the survey one at a time (they shouldn’t be able to watch each other complete it). The testers should complete the survey the same way that it will be completed in the actual project. So if it’s an online survey they should complete it online, if it’s a verbal survey you should have a trained interviewer ask them the questions.

While they are completing the survey ask them to think out loud. Each time they read and answer a question they should tell you exactly what comes into their mind. Take notes on everything they say.

Observe how they complete the survey

You should also observe them completing the survey. Look for places where they hesitate or make mistakes, such as the example below. This is an indication that the survey questions and layout are not clear enough and need to be improved. Keep notes on what you observe.

Make improvements based on the results

Once all the testers have completed the survey review your notes from each session. At this point it’s normally clear what the major problems are so you can go about improving the survey to address those problems. Normally this is all that’s needed. However, if major changes are needed to the questions or structure it might be necessary to repeat the pretesting exercise with different people before starting the survey.

Piloting

Select the pilot sample

For large or complex surveys it’s a good idea to do a full pilot before starting actual data collection. To do a pilot you need to test all the survey steps from start to finish with a reasonably large sample. The size of the pilot sample depends on how big your actual sample is, and how many data collectors you have. For a typical baseline or endline survey a sample of around 30-50 people is usually enough to identify any major bugs in the system.

Implement all the steps from start to finish

Start by training your data collectors, if you have them. Then distribute and collect the survey exactly as you would in practice. Enter the completed surveys into the database that you plan to use and then test the analysis that you plan to perform.

Make improvements

Assuming that the survey was pretested, piloting will normally identify practical problems with implementation, rather than problems with the survey design. For example, lack of staff training, challenges with the logistics of distributing and collecting the survey, or errors in data entry. These can then be fixed before you do the actual survey.

Photo by CDKNetwork