The Question Every Program Leader Should Be Asking
Are We Making a Difference—Or Just Keeping Busy?
A practical guide to learning what’s working, what’s not, and what to do about it.
What you’ll get from this post:
A plain-language intro to how program teams can track what’s working—and why it matters
Simple ways to learn and adjust (without needing a full-time data person)
Real-world tools to help your team adapt in real time
A better understanding of where AI fits in—and what it can’t do for you
Encouragement to start small, stay curious, and build confidence as you go
You’ve launched a new program. Flyers are printed. Community members are showing up. Your team is working hard and your funders seem happy.
But somewhere in the back of your mind, the question nags:
Is this actually working?
That question is the sign of a good leader and it’s where systematic impact measurement comes in—not as a bureaucratic hoop, but as your program’s early warning system, tuning fork, and compass.
What is impact measurement, really?
I come from international development, where we use a system called Monitoring, Evaluation, and Learning. Sometimes you'll see an extra letter—R for research or A for accountability—but it all comes back to the same goal: understanding and improving impact.
Whether you call it outcomes tracking, performance management, or evaluation1, the core challenge is universal: How do you know if your program is creating the change you promised?
These same principles apply whether you're running a literacy program in Detroit or a health project in rural Texas. My goal is to bring the rigor and flexibility of global evaluation practice into the domestic social sector—so more programs can learn, adapt, and deliver measurable impact with confidence.
Here’s the simplest way to think about it:
Monitoring helps you track what’s happening in real time.
Evaluation helps you understand why.
Learning helps you do better next time.
If you’ve ever adjusted your program based on feedback or asked “what’s working and what’s not?”—you’re already doing this. This post helps you do it with purpose, on purpose.
Why this matters
This isn’t just about collecting data to satisfy funders. It’s about making smarter decisions while your program is still running, when there’s still time to fix gaps (or at least for the next iteration to double down on what’s working).
Learning asks, “How can we get better at creating change?”
Accountability asks, “Are we delivering what we promised?”
Leaders who balance both earn trust—and often, more flexibility and funding.
Done right, this kind of measurement gives you three things every leader craves:
Clarity about what’s working
Confidence when talking to stakeholders
Control to adjust course before you waste time or money
Start with your “why”: The Theory of Change
Before you collect a single data point, pause and ask:
What change are we actually trying to make?
That’s what a Theory of Change helps you map. It’s not a slide deck. It’s a shared understanding of how your work leads to real impact.
If your team can’t explain—in plain language—what the program is trying to achieve and how it gets there, then it’s not doing its job.
Think of it like this:
What are we doing? → What’s changing in people’s lives? → What’s the big-picture difference we’re aiming for?
Then flip it: start from the big goal and work backward. What has to happen first? Those are your outcomes.
🥕 Example: Mobile Food Market
Say you’re running a food trailer that hands out fresh produce in underserved neighborhoods. You’re not just distributing vegetables. You’re increasing access to healthy food, building confidence in cooking, and supporting long-term community health.
That’s your outcome.
Assumption: People will show up to cooking demos.
Reality: They’re scheduled the same night as church—so no one comes.
If you don’t check that assumption, your data might tell the wrong story.
Here’s how it breaks down:
Impact → Better community health
Outcome → More healthy meals at home
Output → 200 families served weekly
Activity → Distribute fresh produce
You don’t need all the answers. But you do need to ask good questions—early and often, especially with the people doing the work on the ground.
A Theory of Change isn’t a prediction. It’s a hypothesis. And a good hypothesis can be tested, refined, and improved.
The Learning Cycle
Think of this as a rhythm:
Plan → Monitor → Reflect → Adapt → Repeat
This isn’t something you tack on at the end. It’s how you run your program from day one.
It shows up in:
Weekly team huddles
Check-in surveys
Quarterly reviews
This is adaptive management—and it’s how good programs become great ones.
Measuring What Matters (Not Just What’s Easy)
Once you’re clear on the change you’re working toward, ask:
How will we know if we’re making progress—and what do we need to track to find out?
Start with basics. Maybe it’s how many cooking demos you held (process). Then look at what changed—like how many families say they’re cooking more at home (outcomes).
But here’s the secret:
Only track what you’ll actually use.
If a number won’t shape a decision, don’t waste time collecting it.
Ask yourself:
“If this number changed a lot, would it change how I manage this program?”
If not, skip it.
From Activity to Impact: The Evaluation Piece
Evaluation isn’t just about what happens at the end. It’s how you check your assumptions, test your strategy, and understand whether you’re truly making a difference.
One simple principle:
Think about the data you’ll need later—now.
That’s where a baseline comes in. It’s just a snapshot of where things stand before your program starts. Without it, you’re guessing.
You don’t need a fancy consultant to do this. Try these low-cost, high-impact approaches:
– Ask a few key questions at the start and again later
– Invite participants to share what’s changed in their lives
– Talk to people who didn’t participate—are they seeing similar changes?
– Don’t aim for perfection—aim for insight
Even lightweight evaluation can help you answer:
– Are we making a difference?
– For whom?
– In what ways?
– Under what conditions?
– And—most importantly—how can we do better?
What About AI?
Let’s talk about the robot in the room.
AI is quietly reshaping how this work gets done—especially for small teams with big goals. Imagine uploading a pile of open-ended survey responses and getting a summary of themes—without hiring a data analyst.
AI can flag patterns, surface hidden trends, or alert you to drops in participation at specific sites.
But here’s the golden rule:
AI should help you see more clearly—not think for you.
You still need to ask smart questions, interpret results, and act based on what your team and community know to be true. AI is a flashlight. You’re still the one steering.
Okay, I’m In. Where Do I Start?
Start small. Seriously.
Sketch your Theory of Change on paper. Pick 3–5 things to track. Set a rhythm for reviewing what you’re learning.
When something surprises you—change something. Then do it again.
Not a report. Not a spreadsheet. A practice.
Build it into your work-plan: Map your theory in month one. Collect basic data. Reflect monthly. Make a change. Improve over time.
Final Thought: This Is Power
If you want to lead programs that get better with every cycle, earn trust without spinning stories, and know you’re not just busy—but effective—this is your path.
This work doesn’t require a PhD. It takes curiosity, courage, and a little structure.
The good news? You’ve already got the first two.
Leading programs that create lasting change? Follow Anthralytic for more insights on strategic program management and impact.
This term has a more specific meaning, but non-profits sometimes use it to refer to the whole system.