Anthralytic’s Substack
Sign in

How Many Licks Does It Take?

How to measure the efficiency of a self-licking lollipop

Anthralytic's avatar
Anthralytic
Nov 2

My partner, the most skilled AI user I know, recently built a dashboard to track how efficiently he was using it. The dashboard was well designed. Each metric captured some version of output relative to input: think work completed per token consumed.

When I asked what the dashboard told him, the answer was revealing. It told him a great deal about efficiency, but it said nothing about outcomes.

Thanks for reading Anthralytic’s Substack! Subscribe for free to receive new posts and support my work.

Stop me if you’ve heard this one. Across more than 150 posts, I’ve banged on about the difference between counting what goes into the work and measuring what changes because of it. Usually the confusion is between inputs and outcomes. But often it’s efficiency and outcomes, and the distinction matters just as much. The confusion persists because outcomes are hard to identify and measure.

There’s a big difference, and it matters. An efficiency measure describes the ratio between what goes into a process and what comes out of it. It can tell you a system is running fast without telling you whether it’s worth running. An old colleague called this the “self-licking lollipop”: a system that exists mostly to keep itself going. (All right, it’s ice cream cone, but we’ll go with lollipop to stick with the Gen X vibes.) No matter how many licks or how quickly, the question remains: who cares?

That question is what an outcome measure answers. It describes what changed as a result of the work. A process can become dramatically more efficient while that change stays flat, or never arrives.

After our conversation my partner tried to add an outcome indicator. What he came up with didn’t quite hit the level of an outcome, and that’s no knock on him. Building a good one takes a different kind of skill.

A sound outcome indicator asks a lot. It starts with saying what the work is for. Then you need a sign of change that happens outside the work, that you can reasonably trace back to it, and that shows up in time to inform a decision. Evaluators struggle to meet that bar for programs they’ve studied for years.

There’s a bigger problem, though: the data itself. Everything his dashboard can measure happens inside the tool: prompts, tokens, completed tasks. Outcomes happen somewhere else. No amount of skill applied to usage data will show what happened after the work left the screen, and that’s true of any measurement built from a tool’s own records, not just his.

My dear and brilliant friend, Esra Kucukciftci, puts it well in her forthcoming book, Quantware: AI dashboards measure “the traffic around the outcome” (p. 13) with real precision and leave the outcome itself unaccounted for1. She comes at this from pricing and commercial strategy, not evaluation, and lands in exactly the same place.

Timing is part of it, too. The software he’s building isn’t published yet. Whether people start using it would be an early signal, and whether they find it valuable enough to keep using it would be a stronger one, but neither will exist until real people have it in their hands. Until then, the honest move is to name the outcome you’ll watch for and choose a stand-in, like whether a handful of test users can complete the task it was built for.

There’s a second outcome question available right now, though: what building it with AI is doing for him. I’ve watched his learning compound since 2022. That’s valuable, and it’s hard to measure. No dashboard of tokens will capture either one.

The same trap frequently shows up in nonprofit work. Imagine a nonprofit that uses AI to submit 50 grant proposals in a year instead of 20. The dashboard would look great: more proposals out the door, a lower cost for each one, fewer staff hours per submission.

The obvious next step is to check the win rate. But win rate is still efficiency. It tells you how many proposals turned into awards, not what the awards did. Total dollars awarded doesn’t get you there either.

The outcomes are what the money made possible. That might be steadier funding that lets the organization plan beyond the next grant cycle, or money from funders whose priorities fit the mission instead of pulling programs off course. Ultimately, it’s the change the mission makes for the people it serves: families who stay housed, kids who read at grade level. And, like my partner’s learning, the growth of the nonprofit’s staff counts, too. The usage data has no line for any of those.

Here’s a hint: know your why, and your what and how become clear. It doesn’t work in reverse. Start from the how, a dashboard of tokens or proposals, and you can measure it forever without learning whether any of it was worth doing.

That’s the trap my partner ran into, and every AI dashboard runs into it eventually. Efficiency is easy to count because it happens inside the tool. Whether the work was worth doing happens out in the world, with the people it was for.

So here’s how to measure the efficiency of a self-licking lollipop: you don’t. You ask who it’s for.

How many licks does it take? The world may never know.

Leave a comment

Share

Next up: Where’s the Beef? I’ll talk about AI vendors building the kind of reports nonprofits spent decades learning to move past.


Anthralytic helps organizations connect strategy, evidence, and responsible AI adoption to the outcomes that matter. Human-Led. AI-Powered. Impact-Driven.

1

Quantware quotes and page numbers are from an advance excerpt: Esra Kucukciftci, Quantware (Ideapress, forthcoming 2027).

Discussion about this post

User's avatar

Keep reading

Anthralytic’s Substack

Anthralytic Substack

  • About
  • Archive
  • Recommendations
  • Sitemap
© 2026 Anthralytic · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for writing, podcasting, and media founders.