Anthralytic’s Substack
Sign in

Mikey Likes It!

Why AI’s value shows up outside the tool, and why careful research keeps finding less of it than we feel.

Anthralytic's avatar
Anthralytic
Nov 4

The results that matter for most organizations sit with the people the work is for: the funder who said yes, the family that received services, the staff member whose evening wasn’t consumed by revisions. None of them appear in a transcript.

I’ve argued in How Many Licks Does It Take? and Where’s the Beef? that AI usage data stops at outputs. It can say what was produced, how fast, and at what cost. It can’t say what changed. This piece is about where to look instead, and what the evidence shows when people do.

Thanks for reading Anthralytic’s Substack! Subscribe for free to receive new posts and support my work.

Ask the person eating it

Establishing whether AI contributed to an outcome requires two things the tool’s own records don’t contain. The first is a comparison to what would have happened without AI. The second is the perspective of the people affected.

Neither is exotic. Evaluators collect both routinely. They just have to be collected on purpose, from outside the system being evaluated.

The old Life cereal commercial got this right. Two brothers won’t try the new cereal, so they push it on their little brother Mikey, who hates everything. He eats it. He likes it. Nobody found that out by studying the box. They found out by watching the person eating it.

Where saved time goes is part of the outcome, too. When AI reduces a task from eight hours to two, the six hours are redirected somewhere: toward more output, lower staffing costs, harder problems, or relief for overextended staff. I’ve explored this in Who Is Better Off When AI Speeds Up the Workflow?

Without a deliberate decision, efficiency gains tend to become higher output expectations. Three groups have a stake in that decision: the people an organization serves, the people who work there, and the dashboard. Usage-based reporting measures the dashboard.

Esra Kucukciftci approaches AI from pricing and commercial strategy in her forthcoming book, Quantware (Ideapress, early 2027)1. Her definition of an outcome is explicitly counterfactual: an AI result is worth paying for only if it moved the organization’s results beyond what would have happened without the system (p. 14).

That’s the evaluator’s question, arrived at through contracts and invoices. For nonprofits, it extends one step further. The result that matters often belongs to people who are party to neither the contract nor the invoice.

Look outside the tool and the gains get smaller

When researchers measure work directly rather than reading usage data, they do find gains. Those gains are real, uneven, and concentrated at the level of individual tasks.

A 2025 study in the Quarterly Journal of Economics of 5,172 customer-support agents found that AI assistance increased issues resolved per hour by about 15% on average. Gains were largest for less experienced workers. The most experienced agents saw small speed improvements and, in some cases, modest declines in quality.

A field experiment across 66 firms and 7,137 knowledge workers found that workers who used an AI tool embedded in their email and writing applications spent about two fewer hours per week on email and less time working outside regular hours. The researchers found no change in the quantity or composition of workers’ tasks. Several of the authors were employed by Microsoft, which makes the tool studied. That doesn’t invalidate the findings, but it’s relevant context.

At the organizational level, the signal is weaker. In 1987, the economist Robert Solow observed that “you can see the computer age everywhere but in the productivity statistics.” Swap in AI and the line still holds. An NBER working paper surveying nearly 9,000 senior executives across seven countries, revised in October 2026, found that executives credit AI with raising annual labor productivity growth by about 0.2 percentage points over the past three years. They expect roughly 0.6 over the next three.

Feeling faster isn’t the same as being faster

In METR’s 2025 experiment, experienced open-source developers took 19% longer to complete tasks with AI assistance, yet believed afterward that AI had sped them up. The finding was narrow: experienced developers, familiar codebases, early-2025 tools.

When METR attempted a follow-up with newer tools, it reported in February 2026 that the results had become unreliable, largely because developers declined to participate if it meant working without AI. That refusal is informative. A survey asking what would break without access will produce vivid answers. Those answers describe dependence. They don’t establish value.

Adopting AI, feeling more productive, demonstrating task-level gains, and realizing organizational benefit are four different claims. They’re routinely treated as one.

None of these studies examines nonprofits. They come from call centers, software teams, and corporate knowledge work. Evidence on what AI does to service quality, relationships with the people served, or staff wellbeing in social sector organizations is close to nonexistent.

That gap is part of the argument, not a footnote to it.

Next up: Be Kind, Rewind. Starting with the decision instead of the dashboard.

Leave a comment

Share


Anthralytic helps organizations connect strategy, evidence, and responsible AI adoption to the outcomes that matter. Human-Led. AI-Powered. Impact-Driven.

1

Quantware quotes and page numbers are from an advance excerpt: Esra Kucukciftci, Quantware (Ideapress, forthcoming 2027).

Discussion about this post

User's avatar

Keep reading

Anthralytic’s Substack

Anthralytic Substack

  • About
  • Archive
  • Recommendations
  • Sitemap
© 2026 Anthralytic · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for writing, podcasting, and media founders.