Two Truths and a Lie about Impact Data and Evaluation.
Ready? Here we go:
Staff are under pressure to show results.
Evaluation findings often raise more questions than answers.
The data speaks for itself.
And the answer? You got it: The data speaks for itself.
While we often hear that "the data speaks for itself," the reality in social impact work is that data needs us—human understanding, our values, and our critical thinking—to truly tell the story of our impact and inform future actions.
Ever feel like your organization's data is telling a heroic tale, but deep down you wonder if it’s more fiction than fact? Data isn't just numbers and graphs. It’s the story of your impact. But like any good story, it can be twisted, embellished, or even outright fabricated if we’re not careful. Even with the best intentions (and sometimes, let’s be honest, under pressure to land the next round of funding), it’s alarmingly easy to fall into some common traps.
We’re talking about:
Seeing only what we want to see
Focusing on shiny numbers that don’t actually mean much
Confusing coincidence with causation
And maybe the biggest one? Using evaluation as just another marketing tool
These pitfalls cost organizations credibility, waste precious resources, and—worst of all—get in the way of real change.
I believe honest data is powerful data. Let’s walk through the most common traps and how to avoid them.
1. Confirmation Bias: Seeing What We Want to See
What it is: It’s when we act like selective listeners—embracing data that confirms our beliefs and conveniently ignoring the rest.
Real example: You launch an after-school program. Sign-ups are through the roof. Success, right? Maybe. But did you check whether kids not in the program also improved thanks to, say, a new school-wide initiative? If you didn’t, you might be attributing all the success to your own program just because it fits your narrative.
How to dodge it:
Ask harder questions. “What data could prove we’re wrong?” Then go look for it!
Define success early. Before data collection starts, write down what success actually looks like. This will prevent you from subconsciously highlighting data that confirms your pre-existing beliefs while overlooking contradictory evidence.
Bring in fresh eyes. Invite others to challenge your assumptions—partners, staff from other teams, or even an external evaluator.
2. Misleading Metrics: Shiny Numbers vs. Real Change
What it is: It’s easy to highlight numbers that look good but don’t actually tell you if lives are changing.
Real example: Your job training program ran 50 workshops this quarter. Awesome! But did people actually get jobs? What kind? Are they still employed 6 months later? A high workshop count is nice—but it’s not the full picture.
How to dodge it:
Track outcomes, not just activities. Focus on what changes for people, not just what you did. In other words, outcomes as opposed to just outputs.
Mix quant and qual. Use surveys and stories. Both matter.
Always ask, “Compared to what?” Baselines and comparison groups give your numbers meaning.
3. Correlation ≠ Causation: Just Because It Happened After Doesn’t Mean You Caused It
What it is: When two things happen together, we want to believe one caused the other. But it’s rarely that simple.
Real example: A mentoring program starts around the same time youth unemployment drops. Did the program drive the change? Or was it a new business opening nearby? Or a broader economic shift? Spoiler: It’s probably not the ice cream that causes the spike in summer crime.
How to dodge it:
Use comparison groups. When possible, compare with similar non-participants.
Track external influences. Keep tabs on what else could be affecting your outcomes.
Be careful with your words. Say, “Our program may have contributed to...” rather than “Our program caused...”
4. Evaluation as Marketing: When the Story Comes First, Truth Second
What it is: This one’s sneaky—and common. You’re under pressure to show success, so your evaluation becomes more about looking good than learning honestly.
Real example: A nonprofit launches an environmental initiative and commissions an evaluation that’s really a PR project. Positive findings are plastered on every slide deck. Anything less-than-stellar? Buried. We've seen this pattern too many times at Anthralytic.
Here’s the hard truth: if you go into an evaluation trying to prove something, rather than learn something, you're not evaluating—you’re storytelling. And not in a good way.
How to dodge it:
Be transparent. Evaluations should be about learning. Share the good and the not-so-good.
Separate evaluation from marketing. Let your learning team do their job without the pressure of writing fundraising copy.
Normalize imperfection. Mistakes are opportunities. If your team treats failures as shameful, no one will speak up.
Selection Bias: When Who Participates Skews What You See
What it is: Selection bias happens when the people who choose (or are chosen) to participate in your program are already different in ways that affect your outcomes.
Real example: A mentorship nonprofit reports a 90% graduation rate for program participants. Sounds incredible! But who were those participants? If they signed up voluntarily, they might’ve already been more motivated or had stronger support systems than those who didn’t. That skews your results.
How to dodge it:
Use a comparison group. Match your participants with a group of similar non-participants—like students on a waitlist—to get a clearer picture of your program’s true impact.
Acknowledge what you can’t control. Selection bias isn’t always avoidable, but you can be transparent about its influence.
Design smarter data collection. Whenever possible, collect baseline data and ask questions that help you understand how your participants differ from others.
Keeping It Real: How to Build a Learning Culture That Lasts
If you want to move beyond glossy reports and into meaningful change, here’s how to start:
Hold regular, structured data discussions. Set aside time to go deeper than dashboards. Ask, “What surprises us? What doesn’t fit?”\n
Model humility and curiosity. Leaders: say “I don’t know” more often. Reward honest questions, not just positive results.
Build your team’s skills. Invest in training on data literacy and evaluation basics. A little support goes a long way.
At Anthralytic, we help mission-driven orgs cut through the noise and get to the truth. Because you don’t need perfect data—you need honest data.
And honest data? That’s what drives meaningful impact.
Want to keep it real with your data?
→ Visit Anthralytic to learn how we can help