Tips for Creating and Administering
Effective Surveys and Reporting Results
Surveys are frequently used to collect information from students, faculty, and other members of the university community regarding their perception of the implementation and impact of programs, initiatives, and activities. This document presents suggestions and research-based techniques for developing survey items, sampling, survey administration, and data use and reporting. These suggestions and techniques are intended to support valid and reliable data collection while reducing undue burden on survey participants.
Before developing a survey, see if the information you want already exists in another form. Often there is institutional data on websites that will answer your questions. Low response rates on surveys suggest that other data collection processes may be more appropriate.
Development of Survey Items
- Be clear about what information you need from the survey and ask questions relevant to that purpose. Stay focused on your original purpose and avoid asking extra questions that are not pertinent to your study.
- Keep the survey short (usually no more than 10 minutes to complete) while maintaining sufficient coverage of each construct.
- Ask one question at a time (avoid double-barreled items).
- Keep the language simple and avoid technical terms or jargon.
- Be specific with response options (e.g., use descriptors rather than numbers to define a rating scale).
- Use open-ended questions sparingly (qualitative analysis can be very time-consuming).
- Test out survey items with a student or colleague to ensure that they are clear and make sense to potential users.
At the planning stage, think through how the survey results will be analyzed and reported and who the audience will be. These issues will also inform the development of the survey. For additional information on survey development, check the following resources:
- Dillman, D. (2012). Don Dillman’s Guiding Principles for Mail and Internet Surveys.
- Hanna, J. (2012). 10 Tips to Improve Your Online Surveys.
- Author Unspecified (2011). Some Thoughts on Writing Questions – Adapted from the work of Don A. Dillman.
- Dillman, D., Smyth, J & Christian, L. (2009). Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method, Hoboken, NJ: John Wiley & Sons, Inc.
Although a survey can be administered to the whole population (e.g., all graduating seniors in a given year at Mason), often researchers have to limit their data collection to a sample due to limited resources. Where a sample is used, appropriate sampling techniques should be adopted to ensure that the survey data collected from the sample can be generalized to the population.
- Define the scope of the population and the total number targeted by the survey, e.g., all senior students living on campus, N=600.
- Determine the sample size using a sample calculator, easily available on the internet.
- Obtain the list of the population along with the contact information; randomly select a sample from the list using the sample size generated by the calculator.
- Where random sampling is not feasible and a convenience sample is used as an alternative (e.g., students enrolled in a course or participants attending an event), the limitation of the sample should be clearly stated in reporting to provide a context for interpreting the survey results.
Use of sampling techniques can become complicated. Remember, we are available for consultation. Feel free to contact us.
Survey administration is a major undertaking when it is conducted on a large scale. The survey administrator needs to plan ahead to make sure an appropriate protocol is followed. Good timing is also needed to achieve optimal return. Here are a few useful tips:
- Remember your data collection may need Institutional Review Board approval. You can find out more information by visiting the Office of Research Integrity & Assurance.
- Obtain approval to use confidential student contact information if needed for survey administration.
- Check the university survey calendar available on this website before planning your survey administration time. Avoid conflicts with major student surveys.
- Communicate clearly upfront to survey targets about the purpose of the survey and how the data will be used, particularly with regard to participant confidentiality.
- Improve participation rate by articulating the benefits of survey participation to respondents in initial invitation, sending reminders to participants, and offering incentives for participants where appropriate.
Data Use and Reporting
The survey administrator and associated office(s) have the responsibility to ensure the safety and confidentiality of the data collected and accuracy in data use. Recommended practices include the following:
- Follow university policy when handling survey data with personally identifiable information. (see http://universitypolicy.gmu.edu/policies/data-stewardship/ for University Policy on research data)
- When individual-level survey data need to be released to support specific activities (e.g., academic advising), make sure training is provided to prospective users on protecting data confidentiality and interpreting data.
- Protect individual confidentiality in reporting. This can be accomplished via reporting aggregated results and by removing personally identifiable information when reporting non-aggregated data or open-ended responses verbatim.
- Report n (i.e., number of respondents) when presenting results.
- Include response rate in survey reporting to provide a context for the results.
- Exercise caution in reporting percentages, particularly when the n is small. Set a threshold for minimum n required for reporting. The rule of thumb is that the n chosen is not likely to reveal an individual respondent when results presented are used with other available information.
The Office of Institutional Effectiveness and Planning is available for consultation on survey research. Feel free to contact us at firstname.lastname@example.org.