Can you say your program caused the change? Choosing an evaluation design that fits

Start with the question and who will use the answer

Before you pick a method, decide what you need to know and who will act on it. The CDC’s evaluation framework puts this early: assess the context, and identify the people who might use the findings, such as program staff, leadership, and funders (CDC Step 1). It also asks plainly what money and staff time you have for the evaluation. Only then does it move on to focusing the questions and the design (CDC Step 3).

“Are we delivering the program the way we planned?” is a different question from “Are participants better off?” And both differ from “Would they have been just as well off without us?” Each one calls for a different design.

Three kinds of evaluation, in plain terms

The CDC’s framework describes three types (CDC Program Evaluation Framework Action Guide):

• Process (or implementation) evaluation. Did the program run as planned? Who came, how often, and did staff deliver what the model calls for?
• Outcome evaluation. Did participants reach the outcomes the program intends? The CDC notes that this type cannot, on its own, establish that the program caused them.
• Impact evaluation. How do the outcomes compare with estimates of what would have happened without the program?

Most small programs can do the first two well. The third is the hardest, and it is the one people usually mean when they say “prove it.” Knowing which question you are answering keeps you from promising an impact finding when you have designed an outcome evaluation.

What tracking and pre/post can show, and why change is not proof

The simplest useful design is tracking outcomes over time for the people you serve. A common version is pre/post: measure something at intake and again at exit or follow-up. For a small team, this is often the right starting point. It is affordable, it fits into work staff already do, and it tells you whether participants are moving in the direction you hoped. (For help with the measures themselves, see the survey design guide.)

What it can’t tell you is why they moved. The CDC describes designs without comparison groups as simpler and cheaper, but says they cannot support causal conclusions (CDC). Methodologists list the specific reasons, and they are easy to understand in everyday terms (National Academies, Bridging the Evidence Gap in Obesity Prevention):

• Time and growth. People change on their own. Teenagers mature, job seekers eventually find work, and grief eases. Some of the improvement would have happened anyway.
• Other events. Something unrelated to your program happens between the two measurements: a new benefit, a housing opening, an economic shift, help from another agency.
• Starting at a low point. People often enter a program during a crisis, or because they scored badly on something. Scores that start extreme tend to move back toward ordinary levels, even without help. This is called regression to the mean.
• Who joined in the first place. The GAO warns that when participants are volunteers, they may be more motivated to succeed than people who did not sign up (GAO, Designing Evaluations). Your results may reflect who chose you, not what you did.

None of this means pre/post data is worthless. It means a before-and-after gain is a finding to describe carefully, not proof.

Comparison groups, and why most small programs should not run a randomized trial

A comparison group helps because it gives a sense of what happens to similar people who did not get the program. The CDC’s framework lays out three families of design: experimental designs, where people are randomly assigned to a program or control group; quasi-experimental designs, which compare groups that were not randomly assigned; and observational designs, which use no comparison group (CDC). It notes that randomized designs can show cause and effect but may be costly or raise ethical problems, and that quasi-experimental designs need extra analysis to account for other influences.

A randomized trial needs enough participants, careful assignment, consistent data, and money, and it can require holding back a service from some people. Most small nonprofits can’t do it, and shouldn’t try. Quasi-experimental comparisons are often more feasible than randomized ones (Patricia Rogers, UNICEF), but they usually need a well-matched group and an analyst who can adjust for differences. That is a good moment to call in help.

The CDC adds that no design is best in every circumstance (CDC Program Evaluation Framework, 2024). The right design is the strongest one you can carry out well, ethically, with the people and budget you have.

Say what you know: contribution language and honest limits

If your design can’t show that the program alone caused the change, you can still make a defensible claim. BetterEvaluation describes contribution analysis as a way to ask whether it is reasonable to conclude that the program contributed to the observed changes, while looking at other factors that may explain them (BetterEvaluation). It recognizes that results usually come from several influences working together, not one program acting alone.

In practice, that sounds like this. Imagine a peer-mentoring program that tracks participants’ housing stability at intake and at six months. A careful summary might read: “Most participants were stably housed at six months. This is consistent with our program model, and participants told us the mentoring mattered. We did not compare against people who didn’t take part, so we can’t say how much of the change is due to the program. Other services and a tight rental market may also have played a part.”

That one paragraph says what you found, what you believe, and what you don’t know. The American Evaluation Association’s principles ask evaluators to make clear the limitations of their approach and findings (AEA Guiding Principles). Funders generally read a plain limitations paragraph as a sign of rigor, and a report that overclaims is harder to defend the first time someone asks a follow-up question. Agree with the funder ahead of time on how results will be interpreted, as CDC also advises (CDC Step 5).

Strengthen a modest design with existing evidence and participants’ voices

You don’t have to prove everything from scratch. If your program follows a model that others have studied, that published research can support the case that your approach is a sound bet. CDC suggests reviewing findings alongside the existing evidence base and relevant theory (CDC Step 5). BetterEvaluation lists gathering existing evidence on your theory of change as a step in contribution analysis. Clearinghouses can help you find it. The What Works Clearinghouse, for example, publishes reviews of education research. Look for studies of similar programs with similar participants, and say where yours differs.

Participants’ own accounts are evidence too. The CDC notes that mixed methods, which combine numbers and narrative, can improve accuracy and let each approach make up for the other’s gaps (CDC Step 4). Interviews and open-ended questions can show why a change happened, what else was going on in someone’s life, and which parts of the program mattered. That is exactly the information a pre/post number can’t give you.

Ethics and dignity come before design

Design choices should never cost participants dignity or services. The CDC’s standards say evaluations must protect the dignity, rights, safety, and privacy of participants (CDC Program Evaluation Framework, 2024), and the AEA principles include informed consent and confidentiality (AEA).

In practice:

• Don’t withhold help to build a comparison group. International research ethics guidance for health trials says that, as a general rule, people in a control group should receive an established effective intervention (CIOMS guideline 5). Your program is not a clinical trial, but the spirit applies: don’t turn people away or delay help for the sake of your data.
• Involve the people you serve. BetterEvaluation describes participatory approaches as engaging stakeholders, especially intended beneficiaries, in conducting the evaluation or making decisions about it (BetterEvaluation). Ask participants what outcomes matter and how they’d like to be asked about them.
• Check whether review is required. The federal Common Rule defines research as a systematic investigation, including research development, testing, and evaluation, designed to develop or contribute to generalizable knowledge (45 CFR 46.102). Whether your evaluation falls under that is not something to settle on your own. Ask your funder, and an appropriate review body, before you collect data, especially if you plan to publish or share results widely.

When to bring in an outside evaluator

Consider outside help if a funder requires an independent evaluation, if you want to make an impact claim that needs a comparison group, if the stakes are high, or if no one on staff has time or training to design and analyze the study. A one-time design consult can be enough. You don’t always need a full external evaluation.

Come to the first conversation prepared. A white paper on hiring evaluators suggests putting your program description, evaluation questions, budget range, staff involvement, and reporting expectations in writing, and asking candidates what role your team will play and how they handled an evaluation that went wrong (Usable Knowledge). Add a few questions of your own:

• Which of my questions can your proposed design answer, and which can it not?
• What will you tell us we can and can’t claim from the results?
• How will you involve participants and frontline staff?
• What will we own at the end: the data, the instruments, the write-up?

A practical first step

Write your evaluation question in one sentence, and next to it name who will use the answer. Then write down the strongest claim your current data could support: “participants improved,” “participants improved and say the program helped,” or “the program caused the improvement.” Be honest about which one it is. If the honest answer is the first or second, you don’t need a bigger study. You need a clear outcome measure, consistent collection at intake and follow-up, a few participant conversations, and a limitations paragraph you’d be comfortable defending.