Where’s the Beef?
AI vendors are building the kind of reports nonprofits spent decades learning to move past.
Nonprofits spent decades learning that a count of meals served isn’t a measure of hunger. This week, an AI company shipped a meal counter.
In How Many Licks Does It Take?, I wrote about the gap between measuring how efficiently we use AI and measuring whether any of it matters. That gap isn’t only a problem for individual users with homemade dashboards. It’s now being built into products.
The field already learned this lesson
For much of the twentieth century, social sector reporting meant outputs: meals served, workshops held, clients enrolled. Outputs were what organizations controlled and could document, so outputs became what they reported. The field’s turn toward outcomes was slow and contested. United Way of America’s 1996 guide, Measuring Program Outcomes: A Practical Approach, marked a widely adopted shift, asking funded programs to report what changed for participants rather than how many of them came through the door.
The reason for that shift wasn’t tidiness. Robert Merton named the underlying risk decades earlier: goal displacement, in which the measure gradually replaces the purpose it was meant to track. When organizations are evaluated on outputs, they get very good at producing outputs. Whether anyone is better off becomes a separate question that the system neither asks nor rewards.
Smart reports count what the tool can see
On October 28, Anthropic announced smart reports and in-app surveys for Claude Enterprise. A smart report has Claude read a sample of a team’s transcripts from up to the past 28 days, group the work into workstreams, attach spend to each one, and list the deliverables produced. It also identifies friction: missing connectors, failed tool calls, output that didn’t match the request, rework loops. It counts sessions that produced nothing usable.
Parts of this design deserve credit. Counting failed sessions and rework is more honest than counting tokens. The privacy guardrails are substantive: according to the help center documentation, conversations involving health, union activity, harassment complaints, or personal life are used only in aggregate, and administrators can’t write custom questions about them. I should also note that I write with the assistance of LLMs. I’m not observing this from the outside.
Still, everything in the report comes from inside the conversation. Evaluators separate results into levels because each level answers a different question.
Applied to AI, the levels look like this:
Inputs: tokens, dollars, staff time.
Activities: sessions, prompts, tasks run.
Outputs: what was produced: drafts, code, proposals.
Outcomes: what changed because of that work.
Impact: the longer-term difference, and how much of it AI can reasonably claim.
Efficiency isn’t a level of its own. It’s a ratio across the first three: how much output you get for each unit of input. That’s why it can improve endlessly without ever reaching an outcome.
Usage data covers the first three levels well. It records every session, prices every token, and can describe every deliverable. It stops there, and not because the instruments are immature. Outcomes happen later, in other systems, among other people. A transcript ends when the conversation does.
Shocker right?
In 1984, Wendy’s ran an ad with a fair question about its competitors’ burgers: “Where’s the beef?” A smart report is a generous bun. It has workstreams, spend, deliverables, friction points, and cost per session. What it doesn’t have is the beef.
Smart reports are built entirely from the first three levels, and in at least one place they borrow the language of the fourth. One section of each report is titled “Task outcomes.” The documentation describes it as showing what each session produced. That’s an output labeled as an outcome.
The label matters because it fills the empty slot. A report that appears to cover outcomes gives no one a reason to notice they’re missing. This is goal displacement at the level of vocabulary: once the word for the purpose is attached to the measure, the purpose quietly drops out of view.
Built to find value
The surveys are meant to add the team’s own perspective. Their four default questions ask for the most valuable recent use of Claude, how Claude changed the time a task took, which use a teammate should copy, and what would break first if access disappeared.
Each question presupposes value. None invites a respondent to say a use didn’t help or made the work worse. Administrators can edit the questions, but defaults tend to persist. Estimates of time saved, whether reported by employees or scored by Claude itself, remain perceptions.
The documentation is candid about the reports’ purpose. Among the listed use cases is to “bring evidence to renewals.” That clarifies what the instrument is: a vendor’s tool for helping a customer justify continued spending with that vendor. I’ve written about the broader problem of self-assessment in Grading Their Own Homework.
The narrower point here is that an organization’s question is wider than its vendor’s. The organization needs to know when AI adds little or nothing, and an instrument designed to support renewals isn’t built to surface that finding.
Governance also falls to the customer. Anthropic states that the reports shouldn’t be used to evaluate individual performance or make employment decisions. An administrator setting, off by default, allows report viewers to reveal which employee ran a given session, and each reveal is logged. These are real safeguards. They’re also choices, and organizations need their own rules about who sees these reports, what employees are told, and what the findings may be used for.
AI measurement is at the beginning of a cycle the social sector has already been through. The meal counter is easy to build and easy to read. The work is in refusing to mistake it for the meal.
Next up: Mikey Likes It. Where AI’s value actually shows up, and how to find it.
Anthralytic helps organizations connect strategy, evidence, and responsible AI adoption to the outcomes that matter. Human-Led. AI-Powered. Impact-Driven.


Discussion about this post