Anthralytic’s Substack
Sign in

Be Kind, Rewind

Start with the decision, not the dashboard: how to evaluate AI without letting the available data decide what matters.

Anthralytic's avatar
Anthralytic
Nov 5

Video stores used to put a sticker on every rental: be kind, rewind. It’s good advice for evaluation, too. Start at the end, with the decision you need to make, and work backward.

A principle I return to often in evaluation is that the availability of data shouldn’t determine what gets measured. The decisions an organization needs to make should.

That principle matters more than usual with AI, because the available data is abundant and almost all of it describes the tool. As I’ve laid out in How Many Licks Does It Take?, Where’s the Beef?, and Mikey Likes It, usage reports stop at outputs, and the outcomes that matter sit with the people the work is for.

Thanks for reading Anthralytic’s Substack! Subscribe for free to receive new posts and support my work.

The decisions are concrete

For AI, the decisions are rarely abstract: whether to expand access, redesign a workflow, invest in training, add safeguards, or stop using a tool. Each calls for different evidence.

That might mean comparing work produced with and without AI, tracking time spent correcting and reworking AI output, or examining whether time savings show up in anything clients or staff actually experience. I’ve outlined one approach in Three Moves to Tell If Nonprofit AI Works (and for whom).

Be ready for an unwelcome answer

Starting with the decision also means accepting that the answer might be no. Some uses of AI aren’t worth continuing, however efficient they appear. An evaluation that can’t return that finding isn’t an evaluation. It’s a renewal argument.

In Quantware, Esra Kucukciftci suggests a quick way to see whether you’re headed for one. Take one AI line item from your budget, or one invoice, and write down what the money bought in the document’s own words. Then write one sentence describing what the system actually did for the organization in the same period. Set the two side by side. The distance between them is the gap worth evaluating (p. 19)1.

The framing will arrive before the evidence

Most nonprofits will never run a smart report. They’re available only on Claude Enterprise plans.

That doesn’t mean nonprofits are off the hook. The way smart reports define value (deliverables produced, cost per session, hours saved) is going to travel. It’ll show up as a new line for AI efficiency in a funder’s reporting template. It’ll show up in a vendor case study about a nonprofit that doubled its grant output. It’ll show up in a board deck borrowed from the corporate sector.

My guess is that the framing will get here well before the evidence does. And once a measure is in the template, it’s hard to get out. Nonprofits have seen this before. Outputs became the standard because they were easy to report, and it took the field decades to move past them.

Measurement of AI is becoming more sophisticated. The evidence it produces isn’t becoming more meaningful at the same rate.

In How Many Licks Does It Take?, I described my partner’s well-built dashboard of work per token. It could report how much he produced, how quickly, and at what cost. The outcome indicator he was looking for was never going to be found in the tokens.

Next up: Knowing Is Half the Battle. What this week adds up to, and what to do about it.

Leave a comment

Share


Anthralytic helps organizations connect strategy, evidence, and responsible AI adoption to the outcomes that matter. Human-Led. AI-Powered. Impact-Driven.

black and white rectangular device
Photo by JC Gellidon on Unsplash

1

Quantware quotes and page numbers are from an advance excerpt: Esra Kucukciftci, Quantware (Ideapress, forthcoming 2027).

Discussion about this post

User's avatar

Keep reading

Anthralytic’s Substack

Anthralytic Substack

  • About
  • Archive
  • Recommendations
  • Sitemap
© 2026 Anthralytic · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for writing, podcasting, and media founders.