How to write an outcome survey your participants will actually answer

Start with the decision the survey will inform

Before you write a question, write down what decision the survey supports. “Should we keep the Tuesday evening group?” is a decision. “Learn how participants feel” is not. If your program has a logic model, look at the outcomes column and pick the one or two changes you most need to know about.

Then test every draft question against the decision. If no answer to a question would change what you do, cut it. Some outcomes are also better answered by records you already keep, such as attendance or housing status, than by asking people to report them again. A survey is best for what only participants can tell you: whether services felt useful, respectful, and easy to reach, and how they see their own situation.

Keep it short, ask one thing at a time, use plain words

People are less likely to finish long surveys, as the Australian Institute of Family Studies notes in its guide to writing evaluation questionnaires. Aim for something a person can finish in a few minutes, and treat that time as a gift from the participant.

Each question should carry one idea. Pew Research Center warns against double-barreled questions, which ask about more than one concept and are hard to answer. Imagine a housing program that asks, “Did staff treat you respectfully and help you find a place to live?” Someone who felt respected but never got help has no good answer. Ask two questions. CDC’s plain language guidance makes a similar point about writing: limit each sentence to one idea and choose words your audience knows.

Pew also advises avoiding jargon, unfamiliar abbreviations, and double negatives, and thinking about the education level of the people you’re surveying. Use the words participants use. If staff say “care coordination” and participants say “my case manager,” write “case manager.”

Word questions neutrally and keep answer choices simple

Pew’s overview of question wording notes that poorly worded or leading questions can skew results. The Australian Institute of Family Studies gives “How much did you benefit?” as an example of wording that signals the right answer (AIFS). It assumes a benefit. A neutral version asks whether anything has changed, and lets the person say it got better, stayed the same, or got worse.

A few habits help:

• Keep scales small. Pew recommends no more than four or perhaps five answer choices. Use words for every choice, not just numbers at the ends.
• Use the same scale where you can. Switching scales mid-survey makes people stop and relearn the task.
• Let people skip. Add “prefer not to answer” to sensitive items, and make it clear that skipping is fine.
• Mind the order. Pew finds that earlier questions can influence how people answer later ones. Read your survey top to bottom the way a participant would.
• Use open-ended questions sparingly. One or two (“What is one thing we should change?”) give you the most useful detail.

Use a validated instrument when one exists

Inventing your own questions is slower and riskier than it looks. The Australian Institute of Family Studies says that when a standardized, validated questionnaire exists, using it saves time and increases reliability.

Two places to look. SAMHSA hosts the MHSIP Consumer Survey, which asks mental health service users about satisfaction, outcomes, and social connection. It’s a long form, so ask its publisher before you shorten or reword it. The NIH-supported PROMIS system offers publicly available measures of physical, mental, and social health. HealthMeasures says English paper versions are free to use, while electronic administration and translated versions can involve permissions or fees.

Read the terms of any instrument before you use it, and check who it was tested with. An instrument built for one population may not fit yours.

Ask in a way that respects power and dignity

When staff hand a survey to people they serve, participants may worry that honest answers will affect their services. Pew has found that when people talk with an interviewer, they are more likely to give answers that paint themselves or their communities in a positive light. That research was on phone and web polls, but the lesson carries over: your survey is likely to sound rosier if the person who provided the service is watching the answers.

Some ways to lower the pressure:

• Separate the asker from the provider. Have someone other than the participant’s own worker offer the survey.
• Let people answer privately, on paper in an envelope or on a link they can open on their own time.
• Ask at a natural point, such as a routine check-in or closure, not in the middle of a crisis or at the first visit before trust exists. This is a judgment call, so adjust it for your program.
• Ask about the present and about your service. You rarely need people to revisit painful history to learn whether the program is working.

SAMHSA’s trauma-informed approach gives a useful frame. Its six principles include safety, trustworthiness and transparency, collaboration, and empowerment, voice, and choice. For a survey, that means explaining why you’re asking and what you’ll do with answers, making every question optional, and offering to read the questions aloud. It also means asking participants to help shape the survey.

Be clear about consent, confidentiality, and language

CDC’s evaluation guidance says people who provide data should know their rights, any risks, and how their data will be handled, stored, and used. Put that in a short introduction at the top: who is asking, why, that taking part is voluntary, how long it takes, who will see the answers, and what will not happen, such as losing services.

Be careful with the words “anonymous” and “confidential.” If you collect names, or can link a form to a client record, it isn’t anonymous. In a small program, even a few demographic questions can point to one person. Do not promise more than you can deliver.

Whether HIPAA, state law, a funder’s terms, or research-ethics review applies to your survey depends on your organization and program. Check with counsel or your organization’s approved requirements. This guide can’t answer that for you.

Language matters as much as wording. Pew, when it translates surveys, aims for questions that are comparable in meaning, not simply literal translations, and it has professional linguists review the translations. HHS’s Section 1557 guidance says covered health programs must take reasonable steps to give people with limited English proficiency meaningful access, and points away from relying on unqualified staff as translators. Check whether that applies to you, and plan for the languages your participants use.

Test it with a few people before you launch

Pew calls pretesting an essential step in designing a questionnaire. CDC’s guidance describes piloting instruments with people who have a stake in the results to make sure the questions are clear. The Australian Institute of Family Studies suggests testing with staff or a group of participants (AIFS).

Try it with a handful of people, including a frontline worker and a former participant if you can. Ask them to say each question back in their own words. Note where they pause, ask what a word means, or seem uneasy. Time it. Then revise. An hour of this catches problems that would otherwise show up as blank answers.

What low response rates and small numbers can and can’t tell you

Nonresponse bias happens, in the Census Bureau’s description, when the people who complete a survey differ from those who don’t. The Bureau reported that low-earning households were much less likely to respond to the American Community Survey in 2020, which affected income and poverty estimates. The point for you: who answers matters as much as how many.

AAPOR, the survey research association, notes that a low response rate doesn’t automatically mean poor quality, and that response rates don’t reliably separate accurate data from inaccurate data. But a low rate doesn’t tell you the results are fine, either. It means you can’t tell. Ask who might be missing: people who left the program early, people with the least time, people who are unhappy. Track how many people you asked and how many answered, compare respondents with everyone served on facts you already have, such as length of stay, and say so in your report.

Small numbers need the same honesty. Pew explains that the margin of error is driven primarily by sample size, and that it covers only sampling variability, not nonresponse or social desirability. It also says results for subgroups carry larger margins of error. With twenty responses:

• Report counts. “Fourteen of twenty” is clearer than “70 percent.”
• Show how many answered. Always print the number of responses next to every result.
• Don’t slice. Splitting twenty responses by age or race leaves groups too small to say anything.
• Describe, don’t generalize. “These participants told us” is accurate. “Our participants believe” is not.
• Treat results as a prompt. Use them to start conversations, and pair them with records and staff observation.

Use what you learn, and tell people

Go back to the decision you wrote down at the start. What did the answers say about it, and what did you decide? Write that down next to the results.

Share it with participants too. A short note on a bulletin board or in a group, saying “You told us X, and here’s what we’re changing,” follows the trauma-informed principle of transparency, and it’s the best way to earn an answer next time. If nothing will change, say that honestly.

A practical first step

Write one sentence: the decision this survey will inform. Then draft no more than eight questions that serve it, with each question asking about one thing. Try them out with three people, including one frontline staff member, and cut anything that confuses or doesn’t help you decide. You’ll have a better survey than most, and a much better chance that people answer it.