The Qualtrics Survey Guide outlines Institutional Research’s standard practices for survey development and distribution via Qualtrics. The Qualtrics Survey Checklist is also available to use as a template for your survey projects.
Please contact Institutional Research at ir@syr.edu if you have any questions or feedback for this page.
Items noted with an A are standards that we apply for accessibility reasons.
Please also refer to the Accessibility Guide for Qualtrics Survey Creators developed by ITS for guidance on survey form accessibility.
Qualtrics also has a Survey Accessibility page with guidance on accessible survey question types.
Qualtrics Survey Guide Contents:
This guide is also available in PDF format.
Survey Development
Project Sharing
Share project with relevant team members. Share with at least one other person. On the Projects page, select Collaborate from the project settings (…).
Question Formatting
- Put scales in order from negative to positive; lowest to highest; least frequent to most frequent, etc.
- Strongly Disagree, Somewhat Disagree, Neither Agree nor Disagree, Somewhat Agree, Strongly Agree
- Never, Rarely, Sometimes, Often
- Rank Order question type – set the format to “Text box” and direct respondents to fill in a numeric rank (e.g., Put a 1 next to the item you most prefer, a 2 next to the item you second-most prefer, etc.). A
- Matrix Table question type: A
- Set format to “Profile”
- In the case of wider matrices, the Profile format will require respondents to horizontally scroll to see choices on the righthand side of scales. This creates two issues: the scrolling may be inaccessible to some respondents, or respondents may not notice scrolling is required to see the righthand scale choices, leaving them unaware of these choices. If the Profile format leads to horizontal scrolling, which can be tested by previewing the survey, either use the Standard Likert format instead, or include an instruction for respondents to scroll to the right to see all response choices.
- If you choose to use the Standard Likert format instead, select the option to “Repeat headers.”
- To keep vertical scrolling to a minimum, avoid having numerous items in one matrix, break up the items into multiple matrices where possible. Repeat instructions for each matrix.
- Set format to “Profile”
- Avoid horizontal scrolling (typically applies to matrix items). A
- Avoid lengthy questions/directions and use plain language. A
Look and Feel
- Theme, Layout, and Style A
- Set Theme to Syracuse Simple
- Set Layout to Classic
- Set Style | Foreground Contrast to High
- Remove “Powered by Qualtrics” footer
- Look and Feel | Style | Custom CSS
- Add this custom code: .Skin #Plug a { display: none !important; }
Anonymous vs. Confidential Surveys
- A survey is anonymous when responses cannot be linked to any respondent identifying information, such as name, email, or Student ID. This is called de-identified data.
- Qualtrics can create an anonymous link that can be sent or displayed to any potential respondents. Under Distributions, select Anonymous link on the lefthand side.
- Qualtrics can anonymize the data if personal links are used. You will still be able to see who completed the survey in the Distribution History, but the data viewed in and downloaded from Qualtrics will not contain identifying information. Under Survey Options | Security | Anonymize responses, move the toggle to On. Ensure this option is saved and selected prior to distributing the survey.
- A survey is confidential when responses are (or can be) linked to respondent identifying information. Any datasets that contain identifying information are kept strictly confidential by those working with the data, and no individually identifying information is shared in any survey results or reporting.
- Personal links created by Qualtrics and distributed by Qualtrics or other methods are confidential but not anonymous, unless the Anonymize responses option is toggled to on (as described above).
Introductory Language
- For most surveys, there should be an introductory message at the start of the survey. Depending on the message that is included in the survey invitation (if there is one), you may want to include:
- Purpose of the survey
- Who it’s being sent to/why they are receiving it
- Who’s collecting the data
- How the data will be used, how will it be reported
- If the data is anonymous or confidential
- Who will have access to the data
- Estimate of how long it will take to complete
- Incentive if there is one
- Who to contact if they have questions about the survey (usually IR) and who to contact if they have questions about the program or content (usually the client).
- Example introductory language:
“Thank you for participating in the [Survey Name]! The purpose of the survey is to [describe purpose]. What you tell us will be used [describe how findings will be used]. The survey may take up to [time estimate] to complete. Your responses to the survey are voluntary and [confidential/anonymous]. Results of the survey will contain no individually identifying information. To begin the survey, please continue to the next page.”
Display Logic
- Avoid “is not selected” logic. Rather, specify which choices should be selected for the question to appear. This will prevent questions from displaying when items are not selected because a previous question was skipped. In the example below, if the first question is skipped, the display logic will still work in the second question. Someone born in the 70s will be presented with a question that does not include their birth year.
- In which decade were you born? (70s, 80s, 90s)
- [DISPLAY IF: 70s is not selected] In which year were you born? (1980, 1981 … 1999)
- Check the “In Page” box after specifying display logic so that the question appears on the same page when the logic is met (rather than appearing on the next page). Do this if it fits the flow of the survey.
Skip Logic
Be careful using skip logic. It’s very limited in capability because you can only specify one condition. Also, if someone does not answer a question that uses skip logic, the logic will not be “activated,” and they will receive questions that maybe should have been skipped for them. The only way to prevent this is to force a response to the question. If it is not a question for which a response is required, skip logic should be avoided in favor of display logic.
Back Button
Enable a “Back” button:
- Survey Options | Responses | Back button
- Set toggle to On
- Note that depending on your block branching logic, respondents may not be able to return to previous pages even if this toggle is set to “on.”
Next and Back Button Text
- Look and Feel | General
- Change next and back buttons to text rather than an arrow symbol:
- “Next” for next button text
- “Back” for back button text
- Change next button text to “Begin Survey” after introductory text (put introductory text in separate block; select the block and Edit Block options will appear on the left; under Format, select Next/Previous button text; enter “Begin Survey” for Next Button Text.
- Change next button text to “Submit Survey” after last question (put last question or set of questions in separate block; select the block and Edit Block options will appear on the left; under Format, select Next/Previous button text; enter “Submit Survey” for Next Button Text.
Time for Partial Respondents
- Allow time for partial respondents to return to complete survey.
- Survey Options | Responses | Incomplete survey responses
- The default is only 1 week. Set to something far in the future (e.g., 6 months).
Specify Survey Display Name
- Survey Options | General | Display name
- Change display name to survey name, prefixed by “Syracuse University” (e.g., Syracuse University Orange Survey).
Sharing Previews
- Publish changes before sharing previews.
- When sharing a survey preview, there are two “Link visibility permission” settings. If you leave it at the default “brand internal only” setting, people will only be able to preview the survey when they are logged onto the SU system via AD (i.e., using an SU machine). Changing the setting to public allows people to access the survey from any computer.
- If you share a preview link, the survey can be accessed multiple times by multiple people which is usually preferable. However, make sure that if the survey has lots of branches and conditional questions, the testers know that they are unlikely to see everything in the survey. If they have not seen the full survey, it is advisable to provide a Word document with the full survey for additional review.
- Make sure to remove all the preview responses before launching the survey for real! (But if you forget, you can clean out the previews based on either the response type field or the date of submission.)
Distribution
Setting Up the Invitation Email
- Qualtrics gives you the option to specify the “from email” and “from name.” Outlook reads the “from email” and uses whatever the associated office/department name is in Outlook.
- For example, Syracuse Abroad goes by “Syracuse Abroad,” but they are in the Outlook address book as “SU Abroad.” So even if you set the “from name” to be “Syracuse Abroad” in Qualtrics, the email will appear to come from “SU Abroad” since this is what is listed in the Outlook address book.
- Recommended to use an email/office that exists; if an email/office that doesn’t exist is desired (e.g., a working group), remember that there is a risk of invitations/reminders being sent to spam/interpreted as phishing; suggest adding “Syracuse University” before non-existent offices. For these emails, leave email as default non-existent email “surveys@syr.edu” and provide sender information.
- Remove the option for people to opt out of the survey. If they truly want to be removed from the survey, they can send an email. Opting out could have the effect of preventing the person from receiving invitations from other surveys in the future.
Invitation Email Standard Language
Include Syracuse University watermark as heading on email. Contents of the invitation email may include the following:
- Purpose of the survey
- Who it’s being sent to/why they are receiving it
- Who’s collecting the data
- How the data will be used, how will it be reported
- If the data is anonymous or confidential
- Who will have access to the data
- Estimate of how long it will take to complete
- Incentive if there is one
- Who to contact if they have questions about the survey (usually IR) and who to contact if they have questions about the program or content (usually the client).
If using personalized links for the survey (attaches respondents to identifying information such as email), be explicit in the invitation email that the link is personalized and they should NOT share their link with anyone else.
Contact List Upload
- After uploading the contact list, check that the count of contacts matches your sample file. If contacts have been dropped in the upload, trying deleting and re-uploading.
- Include EMPLID as an External Reference field in your distribution list. This will ensure we have a consistent identifier for each student in the sample. There is some inconsistency with Qualtrics and tracking the original email address which can affect our response rate calculation.
Adding SSO Authentication
The following Answers page provides guidance for adding SSO authentication to a survey:
Qualtrics Adding SSO Authentication to a Survey
Survey Close
General
- Close survey.
- Download raw data and save file to project folder.
- Download distribution history and save to the project folder.
Response Rate
- Exclude anyone who did not receive the survey invitation from the total sample count (Email Bounced, Email Failed, etc.).
- Include full and partial respondents in the respondent count, but check for those who did not answer any questions (minimum number of questions answered to count as a respondent may vary by survey).
Incentives
- Only consider respondents who completed the full survey for the incentive.
- Notify office manager of incentive winners: provide list of names, emails, what the incentive is, what survey it is for, and who is paying for it.