The pattern is worth saying out loud, because it is not really about me.

What I built and what happened

The app lays out a set of principles for how AI initiatives in social impact organizations fail, alongside the breaking patterns that show up when those principles are skipped. It is a small educational tool. Each card has a “Suggest Actions” button that calls an AI provider’s API to generate concrete steps an organization could take.

To make those buttons work, the page needed an API key. The right way to handle a key is to keep it on a server, behind a function the browser can talk to, so the key never goes anywhere near the user’s computer. I put the key directly into the HTML, where anyone who opened the page and looked at the source could read it. I’m still learning all of these things.

Eight months later, two notifications arrived in close succession. The first was a billing alert for about thirty dollars in unexpected API charges. The second was a suspension email citing “abusive activity consistent with hijacked resources” and language about the key being harvested from a public source.

Bots crawl the open web for credential strings constantly. Mine was one of millions they found that week.

The cleanup

I revoked the key. I rewrote the demo to call the API from a server-side function. I audited every other project I had built for the same mistake and found that I had reused the same key in several places, which meant the single leak compromised everything it touched. I scrubbed git histories. I treated each small project as a small disaster site, which is what each of them was.

The bill stopped at about thirty dollars. The reason it stopped there is honestly not that I caught it. The provider’s automated abuse detection caught it. The project was attached to a small billing footprint, which kept the blast radius low. Both of those facts belong to the vendor’s infrastructure, not to mine.

That is the part of the story I want to sit with.

The pattern, named by the app it broke

The app I built names four principles that were violated to build it. Build Internal Literacy. Plan for Adversaries. Govern and Minimize Data. Monitor and Recalibrate. Each one fits the story exactly. I do not raise this to be clever about it. I raise it because the pattern is structural rather than personal.

Most of the AI tools small mission-driven organizations are experimenting with right now are built the way mine was. They are built quickly, often by people who are not specialists, in environments where “I will get to it later” is the default operating posture, because the alternative is not building anything at all.

The literacy gap that allowed me to embed an API key in a deployed page is the same gap that allows a small nonprofit to leak donor data through a misconfigured spreadsheet, run a model on biased inputs without noticing, or ship an automation that quietly mishandles edge cases for months before anyone notices. The technical specifics differ. The structural mismatch is identical.

The mismatch is between the speed at which an idea can be tested and the speed at which a tested idea becomes a system that handles real stakes. Demo-grade decisions get made on demo-grade timelines, and then the demo gets deployed and forgotten, and “demo” turns into “production” by accident. There is no moment where someone declares the change of state. The change of state is what the absence of that moment looks like.

What protected me was not me

This is the part that should make any small organization building with AI uncomfortable.

The infrastructure that kept this failure cheap was not infrastructure I designed. It belonged to the vendor. Their abuse detection ran on their schedule, not on mine. The thresholds that triggered the suspension were thresholds they set without asking me. The rate of spend that pushed the system into protective mode was small only because my account happened to be small.

For an organization running on a larger billing footprint, or with a vendor whose detection runs slower, or with a key attached to a higher-traffic project, the same mistake is a much bigger story. The factors that made my version of this story end at thirty dollars are not transferable. They are circumstantial.

Most of what protects small social impact organizations from the worst versions of their AI experiments right now is circumstantial. What stands between a misconfigured demo and a serious incident is mostly the goodwill of automated systems we did not build and do not control.

We do not have the literacy to design the protection ourselves. We do not have the leverage to demand it from vendors. We need to build them both. Myself included.

Leave a comment

Share

Anthralytic helps mission-driven organizations clarify and amplify their impact. We work at the intersection of evaluation, AI governance, and the systems that shape both.

Everyone loves to hate on AI right now.

Here’s a headline from a piece that ran in a Substack weekender yesterday: “My uncle used Claude to write my Nana’s obituary.” The writer is furious, and the headline is meant to make the reader gasp.

There’s a question about AI that almost nobody is asking right now. It’s not whether. It’s how.

Did he just sit with it and talk through what he remembered? Parse through a complicated relationship, find the words for what he couldn’t say out loud? Did he land on the detail that was true: standing at the window, waving, until the car was gone? Or did he type “write an obituary for my grandmother” and submit whatever came back?

Those are not the same thing. “Used AI” covers both. That’s the problem.

The how question is one I see very few people asking. The concerns are loud right now: job displacement, data centers, cognitive offloading, an internet drowning in slop. Those are real. But none of them tell you anything about what the uncle actually did, or whether the obituary was any good. How someone uses a tool is a different question than whether the tool should exist, and we keep flattening them into one.

How matters everywhere, not just in obituaries

My kids’ schools have unclear AI policies or none at all. Teachers tell students not to use AI, then later use AI in the classroom, with no explanation of what distinguishes one from the other. Don’t use it for the essay. Do use it for the research. Why? What’s the principle? Nobody says, because if there’s a policy at all, it was written without one. The students learn that AI is alternately forbidden and assigned, and that adults don’t know the difference. I’ve written before about doing this with my own teenager, working through a paper at the kitchen table while my laptop sat open to Claude. I haven’t fully resolved the contradiction either, and I’ve also questioned who should be doing the teaching when the schools won’t.

The same incoherence shows up in hiring. Many of the job applications I submitted this year, even tech jobs, came with a variation of the same instruction. We want your words to be your own. Do not use AI in this application. Jobs, fellowships, grants. The instruction was everywhere. I used AI on all of them. I sent more than 400 applications over a year and a half. Asking a candidate to do that volume of work without the tools at their disposal is not a measure of authenticity. It’s a measure of who has time. And it’s asking applicants not to use a productivity tool they would almost certainly be asked to use on the job.

The instruction is trying to screen for something real: the difference between someone who did the thinking and someone who didn’t. That’s a legitimate thing to want to know. But you don’t get there by asking the candidate not to use AI and ticking a box to confirm they didn’t. That approach is lazy. Ironically, it is cognitive offloading by the people doing the screening, letting a checkbox do the thinking for them. It is also ineffective, because of course people will use the tools available to them.

Better to judge the output first, and then ask how.

What the how actually looks like

Here’s how I often use AI, since I’m asking you to judge me by the output. I use it in this newsletter. It isn’t a simple prompt like “write me a piece on X.” I use it first as a conversation partner when I have a half-formed idea, and the back-and-forth becomes a process of discovery that helps me understand the situation better. I’ll talk an argument through, push back when the response is sycophantic or imprecise, and come out the other end with a cleaner and truer version of the thing I was already trying to say. Nobody asks whether you talked your ideas through with someone before you wrote them down. The difference is that the conversation partner is AI.

I use it for research. I used it to build things I couldn’t have built as easily on my own. I built an interactive Monday.com version of an Excel Gantt chart that’s changed how I manage projects. I used it on my job applications to assess fit before I invested time, to iterate on materials, and to push back when a draft read like something a machine wrote. When it did, I told the AI exactly how to rewrite it. The process is closer to editing through conversation than to receiving a finished thing. It is still work.

What this actually looks like, in any of these examples, is the same four-step loop: conversation, draft, iteration, critique.

I start with a conversation: what I’m trying to do, who it’s for, what I know about the context, what I’ve already tried. The AI drafts. I read what came back and tell it what’s wrong. The opening is generic. The third paragraph is making an argument I don’t actually believe. This sentence sounds like a machine wrote it.

The iteration is closer to instructing a junior staffer than to receiving a new draft. I name what isn’t working, point to a specific paragraph, explain why the framing falls flat, ask for a specific change. It comes back. I read again. The argument is sharper but the opening still drags, so I say so, and it tries the opening again.

Several passes in, I ask for critique: score this honestly, tell me what a careful reader would catch. On one of those cover letters, I asked whether it was being sycophantic after it scored a draft nearly perfect. It backed down, re-scored honestly, and named the weakness a real reviewer would catch. It’s the exchange that matters. The tool was correctable. I corrected it. It’s not outsourcing and it’s not cognitive offloading. It’s editing, and it’s work. It just looks different, and the output is better for it.

Would we ask someone not to use a template? Would we ask an accountant not to use Excel or Google Sheets because they’re a shortcut? Would we ask someone to do statistical analysis by hand to prove the work is theirs?

On the applications that asked me not to use AI, I disclosed that I had. A short italicized line stating that I used an AI assistant to brainstorm and draft initial structure, and that the analysis, arguments, and final writing are my own. I chose disclosure over concealment because concealment isn’t the honest posture, and because the “no AI” instruction, taken literally, pushes people toward hiding something rather than naming it.

The output tells you

Here’s the thing. You don’t have to restrict AI use, because poorly used AI shows up in the outputs. The unearned transitions. The throat-clearing openers. The vocabulary one notch fancier than how a person usually writes. The confident summary of something the writer never actually engaged with. The list of three when two would have been honest. Anyone who reads enough AI-generated text starts to recognize the cadence. It feels like the prose is performing thought rather than doing it.

Poorly used AI also shows up in what’s wrong on the page. Hallucinated sources. Statistics from the wrong year, presented as current. Citations to studies that don’t exist. This one is harder, because the writer may not catch it, and most readers can’t be expected to fact-check every claim. The AI presents wrong information with the same confidence as right information, and that’s a real tool problem, not just a user problem. I’ve written more about how AI systems break in social impact contexts — gaming, bias amplification, context collapse — and most of those failure modes also live in the outputs if you know what to look for. I’ve written about what AI is actually accelerating in the social sector, and the failure mode I see most is research done at speed without verification. A grant officer who treats AI output as a finished search instead of a starting one will chase a foundation that closed its program two years ago. A good researcher always starts with a question, not an answer. Using AI doesn’t change that. We don’t want to use AI to find evidence for a claim we already have. We want to use it to figure out what’s actually true, and then verify what comes back.

Good AI use shows up too. It shows up in the argument that’s sharper than it would have been. The research that was checked and pushed on, not accepted. The specific detail that nobody could have produced from a generic prompt, because the person sat with the tool long enough to find it. Standing at the window, waving, until the car was gone. That’s a sentence somebody had to remember. The AI didn’t know.

What to do instead

The fix isn’t to restrict AI use. It’s to ask the right questions of the right people. Three actions.

Judge the output. This is the action for anyone reviewing work. Instead of asking “did you use AI?”, ask for the work and read it closely. A hiring manager should look at whether the cover letter actually engages with the role or recites it back. A teacher should ask the student to walk them through their argument. A funder reviewing a proposal should look for the specific detail that proves somebody understood the problem before they wrote about it. The tells are visible to anyone willing to read closely. They were visible before AI existed. And don’t try to outsource the work to an AI detector. Those tools are notoriously unreliable, flag human writing as AI-generated, and miss the careful AI use you actually want to catch.

Disclose how. This is the action for anyone using AI in work that will be read or evaluated. Don’t hide it. A short, specific line is enough: I used an AI assistant to brainstorm and draft initial structure. I used it to sharpen the argument. The analysis, conclusions, and final writing are my own. That turns a hidden practice into a stated one, and it lets the reader weigh what they’re looking at. It also separates you from the person who pasted the prompt and submitted whatever came back. That person won’t disclose anything.

Cite and verify. This is the action for anyone using AI to do research. The AI will give you statistics from the wrong year and citations to studies that don’t exist, with the same confidence as the real ones. Until the tools flag their own uncertainty, the burden is on the writer to check. If you can’t cite it from a real source, don’t claim it. This is the action I want named loudest, because it’s the one most people skip.

The person who did the thinking will produce better work. The person who didn’t will produce slop, or generic prose, or confident wrongness. Those things are distinguishable. What isn’t distinguishable, and shouldn’t be penalized, is whether a thoughtful person used a tool to get there.

And maybe with the uncle, there isn’t a clean answer. Maybe the AI helped him sort through complicated feelings he couldn’t have organized on his own. Maybe it helped him grieve by giving him something to push against. Maybe it even helped him get the obituary written at all, on a deadline, when the emotional weight made the words impossible to find. I don’t know. The niece doesn’t know either. She’s furious because she imagines the worst version. But she doesn’t know which version it was.

And maybe the niece’s anger isn’t really about the AI. Grief looks for somewhere to land, and AI could simply be a convenient target. It’s a pattern I see often. The thing people are angry about and the thing they say they’re angry about don’t always match. AI is loud and unfamiliar and easy to name, and naming it is easier than sitting with the harder feelings underneath. That’s worth saying gently, because the impulse is human, and because none of what I’ve written here is meant to dismiss it.

But it does mean the question on the table should be a fair one. Go back to the uncle. Read the obituary. If the writing carries his mother in it, if the details could only have come from someone who knew her, then he sat with the AI and did the work. If the obituary could have been written about anyone, he didn’t. The writing tells you. You don’t need him to.

That’s the only question worth asking. Not whether. How. And the answer is in front of you, on the page.

The uncle is one example. The schools are another. The applications are a third. The pattern is the same in every case: we’re using a screening question that can’t tell us what we actually need to know, and the cost of getting it wrong is that we punish the people doing careful work and let the careless ones through.

AI isn’t going away. We can keep pushing blindly against it, or we can learn to use it well, cut the slop, and teach other people how to use it well too. That’s the choice on the table. The version where we ban it doesn’t exist. The version where we pretend it isn’t already in everyone’s workflow doesn’t exist either. What exists is a tool that some people are using carelessly and some people are using carefully, and a screening question that can’t distinguish between them. The people who lose out will be the ones who refused to engage with it at all.

So replace the question. Judge the output, disclose how, cite and verify. That’s the version that’s worth our time.

Anthralytic is a strategy and evaluation studio for mission-driven organizations. If you make decisions about resources in the social sector, whether or not you call yourself an evaluator, this newsletter is for you.

I read the announcement twice. The second time, I recognized it. Not from anything in technology. From international development.

I have watched this approach fail before

For most of the last century, international development ran on a particular assumption. “Expertise” lived in the capital, or the donor country, or the consulting firm, and it traveled outward to places that had problems but not answers. The expert arrived with a solution that had worked somewhere else. The local context was treated as a setting for the solution rather than a source of one.

The results are well documented. Schemes that ignored how things were actually done on the ground. Tools nobody adopted. Programs built around problems the community would not have named first. The failures were rarely failures of the tool. They were failures of a direction of travel. Knowledge was assumed to move one way, from the people who held the method to the people who supposedly just had the need.

The field has spent decades trying to unlearn this. Simply put, participatory development was the answer to that problem. It started from a simple reordering: the people who live inside a problem understand it in ways an outsider cannot, and any solution that does not begin with their knowledge will ride roughshod over the parts that matter. Outside expertise still had a place. It just had to earn that place on local terms, in service of knowledge that was already there.

I spent more than a decade inside that world. I know how hard the lesson is to learn and how easily it gets forgotten.

Who has the expertise?

Using AI well is a real expertise. It takes time to build and it does not come free. I am not saying the tool side is easy. I am saying there are two expertises here, the tool and the mission, and the program is built as if there is only one.

But, it does not even fund that one well. The fellow is a young professional with under two years of work experience, selected not for tool expertise but for comfort and judgment from daily use of Claude. The base camp training is a reasonable on-ramp for a newcomer. It is a one week pre-placement training covers prompt design, building with the API, and running an AI evaluation and about five hours a week after. There is no training in the work of the organizations these fellows will enter. It is not the making of an expert in anything. Anthropic is explicit that it is not looking for sector expertise, only comfort and judgment from daily use of Claude. The fellow arrives as fluent as one can get in Claude in a one-week training and learns the mission on the job. So the program does not actually deliver deep tool expertise either. It delivers a junior person with a week of orientation, positioned as the AI authority in the room.

There is a deeper asymmetry underneath the training gap. The tool side, thin as it is, still gets a dedicated person, full-time, for a year, with a salary behind them. The mission side gets no new hire at all. It stays with a staff member who already had a full job, and who now has to add managing that newcomer on top of it. And the fellow will take what they learn and move on when the year ends. The program is not just under-resourcing the domain side. It is treating the domain knowledge as something a smart outsider absorbs on the way to the real work, while the tool gets treated as the expertise worth a salary and a year.

This is familiar. It is the exact assumption international development is still grappling with. The outsider arrives certified in the method, and the place they land is treated as the setting where the method gets applied rather than as knowledge that should shape what gets done in the first place. The method is the expertise. The community is the backdrop. Claude Corps rebuilds that same hierarchy.

The inversion shows up again in who decides what the fellow works on. To be eligible to host, an organization does not even need a project in mind. Anthropic’s own guidance says that if something is slowing your team down but you have not figured out the AI use yet, that is fine, because Anthropic helps with project discovery before the fellowship begins. The whole arrangement runs on an assumption that is never tested: that AI is the answer. Maybe it is, for a given problem. Maybe it is not. But a program that places a Claude-trained fellow first and looks for the problem second has already decided the question that domain knowledge exists to ask. It is a hammer looking for a nail.

The program is built around the model, not the nonprofit

There is an obvious way to build this if the goal is the host’s mission. Take someone the organization already trusts, someone who knows the work, and train them in the tool. That person exists. The program calls them the supervisor. Instead the program imports a newcomer trained in Claude and asks the supervisor to direct them in the margins of an already full job.

The reason it cannot take the obvious path is in the eligibility rules. Anyone over eighteen with under two years of full-time work experience can apply, regardless of educational background. The hard filter is a workforce filter, early-career and new to the workforce, not a mission filter. The program screens out the people best positioned to bridge a tool to a sector, the ones who already carry a sector in their hands, because the fellow being a new entrant to the workforce is not incidental. It is the design.

That points to what the program is actually organized around. Read the structure as the answer to a single question, how does Claude take root in four hundred nonprofits, and every feature falls out cleanly. Fellows trained only in Claude. Hosts required to be paying Claude for Nonprofits customers before they can receive a fellow at all. Workflows built around Claude that keep running after the fellow leaves. A fresh entrant rather than a sector veteran who might conclude the tool fits in a smaller way than hoped. None of these follow from what would most help this organization. All of them follow from how the model spreads.

I have watched this structure before, pointed at a different group. Last fall I wrote about Microsoft, OpenAI, and Anthropic launching an academy to train hundreds of thousands of teachers in AI. I said then that training is often product onboarding in disguise, and that the effort was honestly about two things at once, preparing students and expanding markets. Swap teachers for fellows and classrooms for nonprofits and the shape is the same. The people being trained are real, the help is real, and underneath it the model gets a generation fluent in it as the default way the work is done.

This is not the design of a program whose top level goals are aimed at the intern nor the nonprofit. So the honest version of the question is not whether the fellow benefits, or whether the host benefits. Both can be true. It is whether anyone designing this started from them. The fellow’s career and the host’s mission read like outcomes the program would be glad to produce on the way to the outcome it was built for. When a workforce program is genuinely for its workers, or a capacity program genuinely for the organizations, you can see it in what the design refuses to compromise. Here the thing held constant is the model. Everything else bends around it.

The correction development already found

Participatory development doesn’t tackle this by rejecting outside expertise. It tackles it by changing the direction of travel. The knowledge that organizes the work belongs to the people inside it. Whatever comes from outside has to bend toward that knowledge, not the reverse.

Applied here, the reordering is straightforward. The person bridging AI and a nonprofit’s work should start from deep knowledge of that work: what the organization is trying to do, who it serves, what its data actually represents, where the hours really go, what would break if you automated the wrong thing. From there, they reach toward the tool, learning enough to judge where it fits and where it does not. That judgment cannot be installed in a base camp. It is the accumulated knowledge of the domain, and the program treats it as the part that takes care of itself.

That is not the harder version of Claude Corps. It is the inverse of it. Same two expertises, opposite ordering. I have written before that AI is most useful when you manage it the way you would manage a junior staffer, where your judgment, your framing, and your knowledge of your own context are the things that do not automate. That only works when the judgment comes first and the tool answers to it. Claude Corps runs the relationship the other way.

What it looks like to build from this side

The other ordering is not hypothetical. It is what I am building Anthralytic toward. This piece is not about that, and if you want to read more you can go to anthralytic.com. The point is only that the reverse direction can be built: a platform whose methods a practitioner designed, with a human expert kept in the loop where field knowledge matters most, so the domain knowledge is the thing the tool is built out of rather than something bolted on afterward. The mission knowledge sets the terms and the technology is shaped to serve it. That is the opposite ordering from a fellow trained in a model and sent to find a use for it.

I am not neutral here. I’m building something that argues against the direction Claude Corps travels. And I’m building it on the same models Claude Corps would deploy. The complicity is real and I would rather name it than pretend to stand outside it. But the argument does not rest on my positionality. It rests on a lesson international development paid for over decades and is now positioned to relearn through AI.

If Anthropic wanted to build it the other way

There is a version of this program that starts from the nonprofits. Here is what it would change.

Instead of recruiting newcomers and training them in Claude, borrow the people who already know the work. Every host has staff who understand the mission, the data, the constraints, and the communities. Take one of them out of their regular load for the year, full-time or half-time, and train them in the tool. Have the fellowship pay for the share of their salary the organization is no longer covering. They are already embedded. They already passed the hard test, the one no base camp can teach, which is knowing the work. Add the tool to the expertise rather than adding the expertise to the tool.

Then evaluate it honestly. The current plan puts measurement and evaluation in the hands of the program’s own partners, scoring whether host organizations advanced their missions. An honest design would hold out a control group, a comparable set of organizations that receive no fellow, so the gains can be told apart from what the organizations would have managed anyway. And it would measure more than Claude usage. How much Claude gets used is the easy number, and it is the wrong one. The question is whether outcomes improved for the people the organizations serve, which is the measurement the sector keeps mistaking velocity for. Counting tool adoption and calling it impact is the same error one layer up.

Anthropic can keep its adoption goal. The whole structure is built to drive usage, which is plain enough that I was able to reverse engineer the strategy from the eligibility rules and the host requirements alone. There is nothing hidden about it. But a program cannot serve two masters and measure for only one of them. If helping nonprofits is a real goal alongside spreading the model, then nonprofit outcomes have to be in the measurement as a priority, not as a line Social Finance reports at the end. What gets measured is what the program is actually for. Right now that is adoption.

None of this is more expensive than what is already committed. It is the same money, pointed at the sector instead of the model. It would be harder to scale into a clean replicable unit, and it would be slower, because the people who know the work are not interchangeable the way a base camp cohort is. That difficulty is the tell. The friction is exactly the sector’s real texture, the thing the current design smooths away by importing a standard newcomer instead.

The nonprofit sector does not need the old mistake in a new coat. It needs the bridge built from the side that already understands the work. If Anthropic wants to help the nonprofit world, that is where to start: with the knowledge that is already there, and the honesty to measure whether anything actually changed for the people it was all supposed to be for.

Leave a comment

Share

Anthralytic is a strategy and evaluation studio for mission-driven organizations. If you make decisions about resources in the social sector, whether or not you call yourself an evaluator, this newsletter is for you.

Anthralytic is a strategy and evaluation studio for mission-driven organizations. We work at the intersection of measurement, impact, and decision-making systems, including the ones nobody voted on.

Not a slow rollback. Not a voluntary pause. A letter from Commerce Secretary Howard Lutnick to CEO Dario Amodei, received at 5:21pm ET on June 12, directing Anthropic to suspend all access to Fable 5 and its restricted sibling Mythos 5 for any foreign national, anywhere in the world, including Anthropic’s own employees. Anthropic couldn’t filter users by nationality in real time. So they shut it down for everyone.

Fable 5 was Anthropic’s first public release built on Mythos, its most capable and restricted model class, previously available only to a small group of government-vetted organizations through Project Glasswing. This is the first time the US government has used pre-deployment authority to pull a commercially deployed AI model from the market. The precedent is the story.

The Same Vulnerability, A Different Outcome

The stated reason for the shutdown was a jailbreak. A specific technique for bypassing Fable 5’s safeguards, identified by what David Sacks described as “a highly trusted partner trusted by both Anthropic and the US government.” Sacks, co-chair of the President’s Council of Advisors on Science and Technology, said the fix was straightforward: patch the vulnerability and Fable can go back online.

Anthropic’s response was that the jailbreak was narrow. Not a broad bypass of the model’s capabilities, but a specific technique that worked in limited circumstances. And then the detail that should stop anyone reading this: Anthropic said the same vulnerability existed in other publicly available models, including OpenAI’s GPT-5.5.

Those models weren’t touched.

One company gets a Commerce Secretary letter. A named competitor, with a documented version of the same vulnerability, keeps operating. That is not a safety standard being enforced. A safety standard applies to the hazard, not to the company. When the same hazard produces a shutdown for one firm and nothing for another, what’s being applied is discretion, and discretion wearing a safety justification is harder to challenge than discretion that admits what it is.

Anthropic has called for oversight that is, in their own words, transparent, fair, clear, and grounded in technical facts. What they got was an undisclosed national security concern, a letter with no specifics, and a deadline measured in hours.

The Backdrop Nobody Should Skip

This didn’t happen in a vacuum. In March 2026, the Department of Defense classified Anthropic as a “supply chain risk”, a designation Anthropic was contesting in court. The conflict reportedly centered on Anthropic’s refusal to make Claude available for mass domestic surveillance and fully autonomous weapons systems without restriction.

So when a narrow jailbreak becomes the basis for a global shutdown of a single company’s model three months later, the question of whether this is a genuine security measure or pressure on a company that said no is not paranoid. It’s the obvious question. And the public record cannot answer it, because there is no record. No published finding, no technical disclosure, no process anyone outside two organizations can see.

That opacity is the point. A legitimate safety action can survive being examined. This one is structured so it can’t be.

Is Anthropic a Sympathetic Victim Here?

Worth pausing on, because the answer isn’t clean.

Anthropic confidentially filed its IPO prospectus with the SEC on June 1, eight days before Fable 5 launched. A company heading for a public offering has every reason to look like the responsible adult in the room, and “we called for our own regulation” is a good line for investors nervous about a government crackdown. Some of the safety positioning is, undoubtedly, positioning.

But the record runs deeper than IPO season. Anthropic disclosed that Chinese state-sponsored hackers used Claude Code to run what they called the first documented large-scale cyberattack carried out without substantial human intervention. They didn’t have to. A researcher extracted what Claude itself called a “soul document” from the model weights, and rather than deny it, Anthropic’s character lead confirmed it was real. Dario Amodei has twice called for AI companies, his own included, to be taxed to fund support for workers AI displaces. Days before the shutdown, he published an essay calling for exactly the kind of government authority to block dangerous models that was then used against him.

You don’t have to find Anthropic noble to see the problem. The argument here doesn’t depend on Anthropic’s sincerity. It depends on whether the government followed a process anyone can inspect. It didn’t.

This Is the Dark Mirror of an Argument I’ve Made

I’ve written before that voluntary AI governance can’t handle what’s coming, that we need real oversight with mandatory reporting, audits, and enforced constraints, the way we regulate food safety rather than trusting processors to self-certify. I still believe that.

The Fable 5 episode is what that argument looks like when it goes wrong. Oversight arrived. It just arrived with no published standard, no technical basis anyone can review, no process to challenge, and a curiously selective target. This is the version of government authority that should worry the people who want government authority. Because it discredits the legitimate case for it.

Real oversight and arbitrary power both look like a letter from the Commerce Department. The difference is whether there’s a standard behind the letter, applied evenly, that the public can see. Strip that away and you don’t have governance. You have whoever holds the authority, using it, on whatever basis they choose, against whoever they choose.

What Compliance Buys You

Here’s the question that outlasts this particular shutdown. If the most safety-focused major lab can be pulled from the market on a narrow jailbreak that its competitors share, with no process and no published reason, what exactly does compliant behavior buy a company?

The honest answer right now is: not much. Transparency didn’t protect Anthropic. Disclosure didn’t. Calling for regulation didn’t. The next time a lab finds something alarming in its own model, the Fable 5 precedent is sitting there as a reason to keep quiet, because candor and cooperation were rewarded with a kill switch.

That’s the real cost, and it isn’t Anthropic’s to bear alone. It lands on everyone downstream who depends on these tools and has no say in any of this. Including the social-sector organizations building real work on top of models that can vanish by end of business on a Thursday.

The question of who gets to decide which AI is too dangerous doesn’t have a good answer yet. What this episode tells us is what the answer can’t be allowed to become: a letter, no standard, no process, no appeal, and a different rule depending on which company you are.

Anthralytic is a strategy and evaluation studio for mission-driven organizations. We work at the intersection of measurement, impact, and decision-making systems, including the ones nobody voted on.

Leave a comment

Share

AI is succeeding at accelerating the wrong things. The dominant report measures the acceleration as impact. The companies producing the evidence base are the companies selling the tools. The frame is built so that the question that matters never gets asked.

This piece is the constructive turn, and it offers three things in response. A question worth holding, a practice worth starting, and a demand worth making. One conceptual, one personal, one political.

The risk in a piece like this is naivete, the temptation to overstate what one practitioner or one studio or one newsletter can do. I will try not to. What follows is not the answer but rather the beginning. It is the shape of the answer the field will have to build, and three places to start.

We do not know where the time goes, because no one is asking

A word on evidence first. While drafting the first piece in this series, I went looking for research on where AI-saved time actually goes for nonprofit workers, and what happens to the people the work exists to serve.

The cross-sector evidence is mixed. A Zoom survey early this year found most knowledge workers saving thirty minutes a day or more and using it for breaks and life outside work. A Berkeley Haas ethnographic study in Harvard Business Review found the opposite, a workday that expanded to fill whatever AI freed up. A Workday survey found close to forty percent of saved time going back into fixing what AI got wrong.

The nonprofit-specific evidence is thinner. A 2025 systematic review of AI-assisted case management in social work measured decision accuracy, not relational quality or client experience. The adoption report measures self-reported efficiency. None of it asks where the time went or who is better off.

The honest version is that we do not know, because no one is asking in the right places. What follows is what asking would look like.

A question worth holding: Who is better off?

The first move is conceptual. Run every conversation about AI in your organization through one question before any other. Not whether to use AI. Not whether you are falling behind. When this organization adopts this tool, who is made better off, and how would we know.

Three groups could plausibly benefit: the people the organization serves, the workers doing the work, and the mission itself—the work that does not fit a dashboard but that the organization depends on.

The vendor frame collapses all three into a single number, organizational capability measured as fundraising velocity, and calls it impact. The question of who is better off refuses that collapse. It is the question Nicole Bowman’s Indigenous evaluation work has been asking for decades: evaluation as a relational act, carried out with communities as relatives rather than performed on them as subjects. This is not woo-woo stuff. This is the purpose of the work. Treating the people the work is for as the people who get to say what the work is doing is the move the field has been organized to avoid.

Carry the question into every AI conversation you are in. It changes what gets said.

That is the place for a pre-mortem. Before adopting a tool, imagine it is a year on and the decision went badly, then work backward to name how it failed: whose work got heavier, what got automated that should have stayed human, which relationship with the people you serve quietly thinned. A pre-mortem costs an hour and surfaces the parts of the trade the vendor’s demo is built to leave out.

The tools that most need a pre-mortem are the ones no one ever decided to bring in. Microsoft Copilot does not arrive through a procurement conversation. It arrives switched on, inside a license the organization already pays for, and that is exactly how it slips past the question. A tool you did not choose to adopt is a tool you are adopting anyway. Run the pre-mortem on that one too.

A practice worth starting: ask your workers where the time went

The second move depends on where you sit. If you manage people, you can start it this week, without waiting for a funder or a perfected methodology. Ask the people who report to you where the hours AI is supposed to be saving them are actually going.

Ask carefully, though, because the question is less neutral than it sounds. To the person answering, you control their workload, and admitting that AI saved them three hours can feel like handing you a reason to fill those hours. In this sector especially, saved time has a way of becoming more work, so honesty carries a risk, and you will get the safe answer instead.

Take that risk out. Make it voluntary and anonymous, run it through a survey or a facilitator outside the chain of command, and report back in themes rather than named responses. Say plainly what it is not: not a productivity audit, not an input to anyone’s review, not a pretext to add work or cut a role. Then go first yourself, and name where your own time went.

The questions are simple. Ask whether the work got more fulfilling or less, whether the caseload grew, whether they are working fewer hours or the same hours with more output, and whether AI gave back the relational parts of the job or took them. Ask the people the organization serves a version of the same, where that can be done without imposing: whether they could tell when something was AI-generated, and whether they felt known.

What matters most is what you do with the answers. If someone got an hour back, the test is whether the hour stays theirs. Fill it and you have proven the fear right, and you will not get an honest answer again. The time was supposed to go somewhere that mattered, and the worker is one of those places.

If you do not manage anyone, the practice turns inward. Track where your own saved time goes, and notice when AI quietly widens your scope, when the hour it gave back fills with work you did not used to carry. Protect that time where you can, name the pattern to the people who can change it, and when someone runs the question by you, answer it honestly. The leader’s version only works if someone is willing to tell the truth. None of this is rigorous in the standard sense. It is what one organization, or one person, can do now to refuse the vendor’s frame.

A demand worth making: independent evaluation, funded outside the vendors

The third move is political, and it is not addressed to practitioners. It is addressed to funders, to evaluators willing to organize, and to researchers willing to do the work that does not yet exist.

The most concrete piece of it is this. The sector needs rigorous, nonprofit-specific study of how AI adoption is affecting the workforce, with attention to who. The Berkeley Haas study is the closest analog, and it documented a corporate workforce expanding its workload to fill the time AI freed. The nonprofit sector has nothing equivalent, and the differences matter. The martyr effect does not land evenly. International consultants and local staff. Headquarters and field. Frontline workers, fundraisers, evaluators, directors. Each carries a different expectation of what commitment costs, and AI will land on each one differently. Some will reclaim time. Some will absorb expansion. Some will lose hours to cleanup while held to the same targets. We do not know who. We should.

The method would be longitudinal and mixed, with the qualitative treated as primary, and it would be built with the communities the work serves rather than on them, the relational stance Nicole Bowman’s work has spent years arguing for. Julian King’s Value for Investment is the closest cousin in the standard literature, because it refuses to reduce the evaluative question to cost alone and asks what is worth investing in, by whose criteria, and for whose benefit.

This has to be a demand because it is structural. The evaluation architecture funders spent thirty years building, the one that produces clean metrics on tight reporting cycles, is incompatible with what honest AI assessment requires. The funder has to choose. AI did not create that problem. It made it visible at a speed the field can no longer pretend not to see.

I know where the seams are, and I am building toward the other side

I worked within cooperative agreement structures for years. I knew the evidence base for what worked was being produced by the people who benefited from it working, and I reported against those benchmarks anyway, because that was the work being paid for. I am not outside the system I am describing. I am the person who knows where the seams are.

Anthralytic exists, in part, to do the work the vendor-produced research will not. The Conditions Web maps conditions across eight domains of social reality before an organization designs strategy or evaluation. The How AI Breaks in Social Impact tool teaches the failure modes the sector is currently absorbing without naming. A Rapid Evaluability Scorecard tests whether a program is ready to be evaluated at all, and an Impact Wizard helps a team build a theory of change. These are not the answer. They are free, practitioner-scale moves toward the questions the field’s architecture is not asking. I name this as positioning, not a pitch.

The improvement is real only if it reaches the people it was for

The improvement is real only if it is an improvement for the people it was supposed to be for. The current architecture does not check. The next one has to.

The harder work starts here. With the practitioners willing to ask. With the funders willing to pay. With the communities willing to say what they have seen.

Previous pieces in this series:

Who is Better Off When AI Speeds up the Workflow?

Measuring the Wrong Thing Faster

Grading Their Own Homework

AI is succeeding at accelerating the wrong things. The dominant report measures the acceleration as impact. The companies producing the evidence base are the companies selling the tools. The frame is built so that the question that matters never gets asked.

This piece is the constructive turn, and it offers three things in response. A question worth holding, a practice worth starting, and a demand worth making. One conceptual, one personal, one political.

Subscribe now

The risk in a piece like this is naivete, the temptation to overstate what one practitioner or one studio or one newsletter can do. I will try not to. What follows is not the answer but rather the beginning. It is the shape of the answer the field will have to build, and three places to start.

We do not know where the time goes, because no one is asking

A word on evidence first. While drafting the first piece in this series, I went looking for research on where AI-saved time actually goes for nonprofit workers, and what happens to the people the work exists to serve.

The cross-sector evidence is mixed. A Zoom survey early this year found most knowledge workers saving thirty minutes a day or more and using it for breaks and life outside work. A Berkeley Haas ethnographic study in Harvard Business Review found the opposite, a workday that expanded to fill whatever AI freed up. A Workday survey found close to forty percent of saved time going back into fixing what AI got wrong.

The nonprofit-specific evidence is thinner. A 2025 systematic review of AI-assisted case management in social work measured decision accuracy, not relational quality or client experience. The adoption report measures self-reported efficiency. None of it asks where the time went or who is better off.

The honest version is that we do not know, because no one is asking in the right places. What follows is what asking would look like.

A question worth holding: Who is better off?

The first move is conceptual. Run every conversation about AI in your organization through one question before any other. Not whether to use AI. Not whether you are falling behind. When this organization adopts this tool, who is made better off, and how would we know.

Three groups could plausibly benefit: the people the organization serves, the workers doing the work, and the mission itself—the work that does not fit a dashboard but that the organization depends on.

The vendor frame collapses all three into a single number, organizational capability measured as fundraising velocity, and calls it impact. The question of who is better off refuses that collapse. It is the question Nicole Bowman’s Indigenous evaluation work has been asking for decades: evaluation as a relational act, carried out with communities as relatives rather than performed on them as subjects. This is not woo-woo stuff. This is the purpose of the work. Treating the people the work is for as the people who get to say what the work is doing is the move the field has been organized to avoid.

Carry the question into every AI conversation you are in. It changes what gets said.

That is the place for a pre-mortem. Before adopting a tool, imagine it is a year on and the decision went badly, then work backward to name how it failed: whose work got heavier, what got automated that should have stayed human, which relationship with the people you serve quietly thinned. A pre-mortem costs an hour and surfaces the parts of the trade the vendor’s demo is built to leave out.

The tools that most need a pre-mortem are the ones no one ever decided to bring in. Microsoft Copilot does not arrive through a procurement conversation. It arrives switched on, inside a license the organization already pays for, and that is exactly how it slips past the question. A tool you did not choose to adopt is a tool you are adopting anyway. Run the pre-mortem on that one too.

A practice worth starting: ask your workers where the time went

The second move depends on where you sit. If you manage people, you can start it this week, without waiting for a funder or a perfected methodology. Ask the people who report to you where the hours AI is supposed to be saving them are actually going.

Ask carefully, though, because the question is less neutral than it sounds. To the person answering, you control their workload, and admitting that AI saved them three hours can feel like handing you a reason to fill those hours. In this sector especially, saved time has a way of becoming more work, so honesty carries a risk, and you will get the safe answer instead.

Take that risk out. Make it voluntary and anonymous, run it through a survey or a facilitator outside the chain of command, and report back in themes rather than named responses. Say plainly what it is not: not a productivity audit, not an input to anyone’s review, not a pretext to add work or cut a role. Then go first yourself, and name where your own time went.

The questions are simple. Ask whether the work got more fulfilling or less, whether the caseload grew, whether they are working fewer hours or the same hours with more output, and whether AI gave back the relational parts of the job or took them. Ask the people the organization serves a version of the same, where that can be done without imposing: whether they could tell when something was AI-generated, and whether they felt known.

What matters most is what you do with the answers. If someone got an hour back, the test is whether the hour stays theirs. Fill it and you have proven the fear right, and you will not get an honest answer again. The time was supposed to go somewhere that mattered, and the worker is one of those places.

If you do not manage anyone, the practice turns inward. Track where your own saved time goes, and notice when AI quietly widens your scope, when the hour it gave back fills with work you did not used to carry. Protect that time where you can, name the pattern to the people who can change it, and when someone runs the question by you, answer it honestly. The leader’s version only works if someone is willing to tell the truth. None of this is rigorous in the standard sense. It is what one organization, or one person, can do now to refuse the vendor’s frame.

A demand worth making: independent evaluation, funded outside the vendors

The third move is political, and it is not addressed to practitioners. It is addressed to funders, to evaluators willing to organize, and to researchers willing to do the work that does not yet exist.

The most concrete piece of it is this. The sector needs rigorous, nonprofit-specific study of how AI adoption is affecting the workforce, with attention to who. The Berkeley Haas study is the closest analog, and it documented a corporate workforce expanding its workload to fill the time AI freed. The nonprofit sector has nothing equivalent, and the differences matter. The martyr effect does not land evenly. International consultants and local staff. Headquarters and field. Frontline workers, fundraisers, evaluators, directors. Each carries a different expectation of what commitment costs, and AI will land on each one differently. Some will reclaim time. Some will absorb expansion. Some will lose hours to cleanup while held to the same targets. We do not know who. We should.

The method would be longitudinal and mixed, with the qualitative treated as primary, and it would be built with the communities the work serves rather than on them, the relational stance Nicole Bowman’s work has spent years arguing for. Julian King’s Value for Investment is the closest cousin in the standard literature, because it refuses to reduce the evaluative question to cost alone and asks what is worth investing in, by whose criteria, and for whose benefit.

This has to be a demand because it is structural. The evaluation architecture funders spent thirty years building, the one that produces clean metrics on tight reporting cycles, is incompatible with what honest AI assessment requires. The funder has to choose. AI did not create that problem. It made it visible at a speed the field can no longer pretend not to see.

I know where the seams are, and I am building toward the other side

I worked within cooperative agreement structures for years. I knew the evidence base for what worked was being produced by the people who benefited from it working, and I reported against those benchmarks anyway, because that was the work being paid for. I am not outside the system I am describing. I am the person who knows where the seams are.

Anthralytic exists, in part, to do the work the vendor-produced research will not. The Conditions Web maps conditions across eight domains of social reality before an organization designs strategy or evaluation. The How AI Breaks in Social Impact tool teaches the failure modes the sector is currently absorbing without naming. A Rapid Evaluability Scorecard tests whether a program is ready to be evaluated at all, and an Impact Wizard helps a team build a theory of change. These are not the answer. They are free, practitioner-scale moves toward the questions the field’s architecture is not asking. I name this as positioning, not a pitch.

The improvement is real only if it reaches the people it was for

The improvement is real only if it is an improvement for the people it was supposed to be for. The current architecture does not check. The next one has to.

The harder work starts here. With the practitioners willing to ask. With the funders willing to pay. With the communities willing to say what they have seen.

Previous pieces in this series:

Who is Better Off When AI Speeds up the Workflow?

Measuring the Wrong Thing Faster

Grading Their Own Homework

Leave a comment

Share

Anthralytic is a strategy and evaluation studio for mission-driven organizations. If you make decisions about resources in the social sector, whether or not you call yourself an evaluator, this newsletter is for you.

Last week I asked who is actually better off when the workflow gets faster. Earlier this week I looked at what the dominant report measures and what it cannot. Now the question moves back one more step. Who set the frame in the first place.

In a few months, one set of terms became the way the sector talks about AI: the ninety-two percent who use it, the seven percent who get real impact, the efficiency plateau in between, and the integration that is supposed to close it. All of it now reads as neutral description. It is worth asking who built that frame, because the answer shapes every question that gets asked next.

Before going further, a clarification. I am not against AI. I use it, I enjoy it, and I think it holds real promise for the social sector. That is not the argument. The argument is narrower and easier to miss. The conclusions the sector is absorbing about AI, what works, what success looks like, what to do next, are not independent conclusions. They come from the people selling it.

The report’s publisher sells the cure it prescribes

The 2026 Nonprofit AI Adoption Report is published by Virtuous and Fundraising.AI. Virtuous is a CRM company, and it sells an AI fundraising product, Virtuous Momentum, which the report’s launch material holds up as what integration looks like when it works.

So the report defines the sector’s problem as a failure to integrate, and the thing that fixes a failure to integrate is the integrated product the publisher sells. The seven percent who broke through are, by the report’s own logic, the organizations that did the integration. In many cases they are also the organizations that bought the stack.

This is the part worth slowing down on. The frame does not only describe the sector’s disappointment with AI. It diagnoses it, and the diagnosis points in a single direction. If your AI is underwhelming, the problem is that you have not integrated it deeply enough, and the answer is to integrate it more. The question the first two pieces were built around, whether the thing being accelerated is worth accelerating at all, never comes up. The frame has already converted it into a different question. Not whether this is worth doing, but whether you are doing it well enough.

Blackbaud is the cleanest version of this

Blackbaud runs a research arm, the Blackbaud Institute, which has spent years publishing reports that link digital maturity to fundraising revenue growth. Blackbaud, the company, sells the digital maturity. The research finds that organizations with more integrated technology raise more money, and the company stands ready to sell more integrated technology. The study and the sales sheet are the same document in different formatting.

It is not only Blackbaud. AWS runs free trainings built around case studies of nonprofits using its tools, and Salesforce has folded AI agents into its nonprofit CRM in partnership with OpenAI. The specifics vary. The structure does not. The company that benefits from the conclusion is the company producing the evidence for it.

There is no independent grader, because no one funds one

The obvious response is that someone else should produce the evidence. The trouble is that almost no one does. The infrastructure that would, the academic centers, the intermediaries, the independent evaluation shops, has never been resourced at the scale the question needs, and recent funding cuts have made it thinner rather than stronger.

Even the sector’s most serious equity work runs on the same dependency. Candid’s AI Equity Project found that only 6.9 percent of surveyed nonprofits had internal policies for responsible AI use, and Candid’s own analysis names the reason. Funders keep paying for tools and experimentation rather than governance, policy, or the staff time to learn. They want responsible adoption and they fund the opposite. Mission money inverted, AI edition, the same mechanism I traced through the USAID closure. The dollars that would pay for an independent grader go to the products instead. So the only grader left is the one selling the test prep.

This is not a scandal, it is the default

A scandal is a violation, something that broke a rule and can be exposed and fixed. This broke no rules. No one had to corrupt anything. The only research that gets funded is the research a vendor has a reason to pay for, and the only frame that travels is the one that arrives already written. It is a different kind of problem than a scandal, one with no single bad actor to point to, nothing to expose, and no obvious fix to demand. It looks like information, and it pays for itself, which is why it stays.

The cost lands on people who are not in the room. Nonprofit leaders are making real decisions, what to buy, who to hire, what to cut, on the strength of research designed to sell them software. The communities those organizations serve inherit the decisions. When the evidence base is built by the sellers, the sector loses the ability to tell what works apart from what sells, and it spends scarce money as though the two were the same thing.

The sellers are grading their own homework

Across the three pieces, one mechanism keeps repeating. AI accelerates workflows that were already broken. The acceleration gets counted as impact. And the companies selling the acceleration are the ones producing the evidence that says it worked.

The next piece is about where an honest assessment could happen instead, and what it would take to hold it.

Previous pieces in this series:

Who is Better Off When AI Speeds up the Workflow?

Measuring the Wrong Thing Faster

Forthcoming post in this series:

Three Moves to Tell if Nonprofit AI Works (and for whom)

Anthralytic is a strategy and evaluation studio for mission-driven organizations. If you make decisions about resources in the social sector, whether or not you call yourself an evaluator, this newsletter is for you.

Leave a comment

Share

Measuring the Wrong Thing Faster

Anthralytic helps nonprofits and social impact programs better understand and scale their positive impacts and use AI better so that real people benefit.

Last week I asked who is actually better off when the workflow gets faster. This week the question is narrower and harder. When the field tries to assess whether AI is working at all, what exactly is it measuring.

The headline gap sits between adoption and impact. Ninety-two percent of nonprofits use AI. Only seven percent report a major gain in what their organization can actually do. The report names this the efficiency plateau and reads it as a problem of integration, a sector that picked up the tool without learning to use it well. The real gap is somewhere else. It sits between what the report measures and what would tell us the sector is working.

The numbers are real. The definition underneath them is a choice.

The 2026 Nonprofit AI Adoption Report surveyed 346 nonprofits. Ninety-two percent adoption. Seven percent reporting major improvement in organizational capability. The report calls the space between them the efficiency plateau, and the coverage settled into one explanation. Nonprofits are using AI but not capturing transformation. The fix is better strategy, better governance, better workflows.

That explanation quietly accepts a definition of impact. The sector should not accept it without naming it first.

The report counts velocity and calls it impact

Read the categories carefully. Major improvement in organizational capability is framed around fundraising efficiency, donor segmentation, personalized outreach, the capacity to manage a larger portfolio, the hours saved on a campaign. The report’s own exemplar is a development director who set aside six hours for an email campaign and finished it in twenty minutes.

That is the picture of impact the report offers.

It is a real efficiency gain. It is not a measure of whether the relationship between the organization and the donor grew stronger, whether the fundraising program became more sustainable, whether the staff member is less burned out, or whether the mission moved at all. The thing being measured is velocity in transactional fundraising. AI is very good at that.

When the measure becomes the target, it stops measuring

I have written before about Goodhart’s law, the observation that when a measure becomes a target, it stops being a good measure. The sector spent thirty years building its incentive architecture around fundraising velocity. AI is now arriving inside that architecture and being judged against the targets that architecture set.

So the seven percent are not the organizations that figured out AI. They are the organizations that bought the integrated stack and tuned it to the targets they already had. The report measures their success against the same metrics the sector was already optimizing for.

The efficiency plateau is not a plateau. It is the report reaching the edge of its own measurement. There is nothing left to find beyond the metric it chose to look for. This is Goodhart playing out in real time, in a report that is shaping how the whole sector talks about AI, and no one is naming it.

The questions that would matter are not in the frame

A real assessment would ask a different set of questions. Whether the depth of donor relationships changed. Whether gift officers still remember the names of donors’ children, or have quietly handed that to a CRM field. Whether the fundraising program is more sustainable, in the sense that it could survive a major funder walking away, or whether AI personalization has deepened single-source dependency by raising the cost of reaching any donor not already in the system.

It would ask what happened to staff sustainability. What happened to the relationship between the organization and the communities it serves, as those communities started receiving AI-generated outreach. What happened to the work that never appears in a dashboard at all, the unscheduled phone call, the showing up at the funeral, the slow accumulation of trust.

None of this is in the report. None of it is in the coverage. It is not in the frame.

The comparison the report makes should not pass quietly

The report puts one staffer occasionally using a free chatbot in the same category as organizations running paid platforms, cross-functional teams, and integrated workflows. It calls both of them adoption, and then asks why one shows impact and the other does not.

The seven percent gap may be less an integration story than an artifact of that collapsed category. The seven percent are, by definition, the customers who bought the integrated stack. The ninety-three percent are the rest of the sector. Naming the distance between them the efficiency plateau obscures the simpler fact that the plateau is exactly what the report was built to measure.

I am not standing outside this

I have worked within systems that counted outputs as if they were outcomes, that treated touchpoints as engagement, that measured whatever the funder asked for and reported it back as impact. I did not always see the frame I was working inside. More often than I would like, I did not think to question it.

This is the same move at sector scale. The report measures what its publisher’s product accelerates and calls the acceleration impact. The complicity I am naming is not Virtuous’s. It is the field’s. It is mine.

Indigenous evaluation asks the question the report skips

Nicole Bowman’s work treats evaluation as a relational act, carried out with communities as relatives rather than performed on them as subjects. An honest assessment of what AI is doing to the nonprofit sector would start by asking the people the sector exists to serve whether anything changed for them. The report did not ask. Most of the coverage did not either.

That is the gap.

The report is failing the ninety-three percent

The ninety-three percent are not failing the report. The report is failing them.

Measuring the wrong thing faster does not turn into impact once the measurement clears a threshold. It turns into the thing the field begins to mistake for impact, because the measurement is the part of the work the field can actually see.

Anthralytic is a strategy and evaluation studio for mission-driven organizations. If you make decisions about resources in the social sector, whether or not you call yourself an evaluator, this newsletter is for you.

For the previous piece in this series:

Forthcoming posts in this series:

Grading Their Own Homework

Three Moves to Tell if Nonprofit AI Works (and for whom)

Leave a comment

Share

This was supposed to be a simple question.

I asked Codex: “Can you set up something that reads your Substack piece for users?”

What I meant was: does Substack already have a native listen-to-this-post feature?

What happened instead was much stranger, and much more interesting.

Codex started building.

First, it made a small local app that could take a Substack link, extract the post text, and read it aloud. Then it added a download button, so the post could become a podcast-style audio file.

At that point I had to stop and say: wait. This is cool, but it is not useful if it lives outside Substack. A reader is not going to copy a link, open another app, paste the post, and make their own audio.

So the question changed.

Not: can a reader turn this into audio?

But: can my publishing workflow automatically create an audio version when I publish something new?

That is when this turned from a toy into a workflow.

We connected it to my Substack RSS feed. We set it up to detect new posts. We added text extraction, OpenAI text-to-speech, an outbox folder, metadata, upload notes, and even a desktop shortcut so I could find the audio files later.

Then we tested it on a previous post.

It worked.

A written Substack piece became an audio file.

And here is the irony.

The first post I wanted to turn into audio was called “Who Is Better Off When AI Speeds up the Workflow?”

That piece argued that AI efficiency is not automatically a good thing. The real question is not whether AI makes a workflow faster. The real question is who benefits when it does.

Then I used AI to speed up my own workflow.

So the question applies here too.

Who is better off because this audio workflow exists?

If the answer is “the dashboard,” then this is just another way to produce more content faster. More formats. More outputs. More proof that something happened.

That is not enough.

The only reason this workflow matters is if it helps people engage with the ideas in a format they can actually use. If someone listens while walking, commuting, cooking, resting their eyes, or moving through a day when reading is not possible, then the workflow has created access. If it simply adds another production expectation to my plate, then it has recreated the problem my original post was warning about.

That is the test I want to keep in front of me:

Does this create capacity, or does it create another obligation?

Does it help the reader, or just feed the content machine?

Does it make the work more accessible, or merely faster?

That is the first “wow” for me, but it is a qualified wow.

The second “wow” came after the workflow existed.

I wanted to share the story. I wanted to explain what had happened and why it mattered.

I did not start with a blank page.

I stayed in the same working context and asked Codex to draft a bonus post about how this came to be.

That connects to something I wrote last year called “The Power of Staying in the Thread.” In that piece, I argued that when you keep work in the same AI thread, the model can use the history of the project. You do not have to re-explain the decisions, the voice, the false starts, or the goal.

I still believe that.

But this experience updated the point.

You do not always have to stay in the exact same thread. What matters is that the context can travel.

In this case, Codex knew the original question. It knew I had not meant to build a standalone app. It knew the first voices were terrible. It knew we shifted from a reader tool to a publishing workflow. It knew the API key problem. It knew the RSS feed. It knew the generated audio file. It knew I kept saying: this is not really about Substack. It is about what people can now do with AI.

So when I asked for a bonus post, I did not have to reconstruct the story.

I did not have to say: here is the whole sequence of events, here is what mattered, here is the lesson, here is the earlier post this connects to.

The context was already there.

That is context carry.

Not just staying in one thread forever. Not just memory as a vague feature. Context carry means preserving enough of the project that AI can help you move from doing the work to explaining the work.

The first surprise was that AI helped me build the audio workflow.

The second surprise was that, because the context was still available, AI could help me turn the workflow into this piece.

That distinction matters.

The audio app shows what AI can help you make.

This post shows what happens when the context of making it is still available.

You can build something, reflect on it, turn it into a story, revise the story, and even generate an audio version of the story without starting over each time.

That is not just convenience. It changes the rhythm of work.

The friction drops between idea, prototype, output, reflection, and sharing.

You do not need to begin with “I want to build software.”

You can begin with friction.

This should be easier.

This should happen automatically.

This should become another format.

This should reach people in a different way.

Then you can ask AI to help you work backward from the wish to the workflow.

And if you preserve the context, you can also ask it to help you explain what happened after the workflow exists.

That is the shift.

The question is no longer just: what can AI write for me?

A better question is:

What can AI help me make possible?

And maybe the follow-up is:

Once I make it, how can AI help me understand it, improve it, and share it?

Anthralytic is a social impact strategy and evaluation studio that asks hard questions about who AI benefits, how, why, and when.

In February, Virtuous and Fundraising.AI released the 2026 Nonprofit AI Adoption Report. Gabe Cooper, the CEO of Virtuous, used the launch to call the question of whether nonprofits should use AI largely settled, and to say the real work now is rethinking workflows. In the same press release, the company’s chief AI officer, Nathan Chappell, described the organizations pulling ahead as the ones integrating AI into how decisions get made. That, he said, is where capacity begins to expand.

Two months later, the same Nathan Chappell told the Chronicle of Philanthropy that the productivity paradox is “one of the things I worry about the most with AI in our sector.” His worry was specific. A twenty-dollar tool that makes a worker twenty percent more productive looks like a free extra day of labor, and the temptation is to keep loading that worker until the job no longer fits one person. He named the risk by its real name. Retention.

Same person. Different room. The February version is the one that traveled. This piece starts from the April one.

Subscribe now

The numbers are not the surprising part

The report surveyed 346 nonprofits in late 2025. Ninety-two percent use AI. Seventy-nine percent report small to moderate efficiency gains. Seven percent report major improvements in organizational capability. Eighty-one percent use AI individually, without shared workflows. Forty-seven percent have no AI governance policy. The report calls the space between the ninety-two and the seven an efficiency plateau.

The dominant reading is that ninety-three percent of the sector has an integration problem. The seven percent did the hard work. They built cross-functional teams, wrote governance policies first, went slow before they went fast. Everyone else is one person on the side drafting an appeal.

I want to read the same numbers differently.

The seven percent are succeeding at acceleration, not transformation

The report defines impact as efficiency gain, fundraising velocity, and personalized donor outreach. By that definition, the seven percent are not doing something categorically different from the ninety-three percent. They are doing the same thing at scale, with the friction removed.

So the question worth holding is not whether the seven percent figured out integration. The question is who is better off when the workflow goes faster.

Three groups could plausibly be made better off by AI in this sector. The people the organizations exist to serve. The workers doing the serving. And the organizations themselves, as measured by their dashboards.

The report measures the third. It does not ask about the first two. What follows takes them in turn.

The people being served are not measured

A case manager who finishes intake forms faster has made a real gain. Intake is not the work anyone went into the field to do. The question is where the saved time goes, and whether the client is better off for it.

The nonprofit-specific evidence here is thin to the point of absence. A 2025 systematic review of AI-assisted case management in social work found a small set of empirical studies reporting effective outcomes, but the outcomes measured were things like risk-assessment accuracy and decision support. None of them measured whether case managers had more relational time, or whether clients felt more heard. The Virtuous report measures self-reported efficiency from staff. It does not ask the people the work is for.

If the saved time goes to a tenth intake of the day, the client is processed faster and heard no more. If it goes to a real conversation with one of the ten, the client is genuinely better off. If it goes to fixing what the AI summary got wrong, the client is now at risk of a decision made on bad data. We do not know which of these is happening at any scale, because the people producing the evidence base are not asking.

The extreme version made the news this spring. A Texas mental health nonprofit used an AI grant research tool that surfaced a foundation as a perfect match, presented years-old information as current, and missed that the foundation had changed its mandate long before. Three weeks of application work followed. The counseling program closed two months later. The story comes to us through a press release from a competing AI vendor, which is its own small lesson, but the shape of it is not in dispute. The communities served by that program are measurably worse off. The communities served by the case manager remain unmeasured.

The workers are not measured either

The worker question has the same shape. Where does the time go, and is the worker better off.

The cross-sector evidence is genuinely mixed. On the encouraging side, a Zoom survey early this year found most knowledge workers saving thirty minutes a day or more and spending it on breaks and life outside work, and Workday found that the workers who report good outcomes from AI tend to reinvest saved time in higher-value work. On the harder side, the Berkeley Haas eight-month ethnographic study published in Harvard Business Review in February found that AI did not free up time at all. It expanded what people were willing to take on. Workers moved faster, widened the scope of what counted as their job, and let work seep into lunch and evenings, often without anyone asking. A separate Workday survey of 3,200 employees found that close to forty percent of the time AI saves gets spent fixing what AI got wrong.

None of this is from the nonprofit sector. And nonprofit workers do not enter this period as neutral subjects. They enter it from inside the pattern I have written about before, the martyr effect, where sacrifice is the currency of belonging and giving up rest is the quiet price of staying.

The Berkeley Haas study describes a workforce that was already willing to absorb more. The nonprofit workforce is structurally more willing than that. The worker who says AI freed up an hour, so I am leaving at five today, is making a move the field has spent decades organized to punish.

Whether AI is making nonprofit workers better off or simply accelerating their existing pull toward self-extraction is an empirical question. The report did not ask it. The cross-sector evidence is too mixed to assume the answer. That gap is not incidental. It is a research demand.

Only the dashboard improved

What is left is the third group. The dashboard.

The CRM looks better. The reports look better. The numbers the funder sees look better. The report’s own exemplar is a development director who set aside six hours for an email campaign and finished it in twenty minutes with an AI fundraising tool. That is offered as proof of impact.

It is also exactly what an improved dashboard looks like. The hours are tracked. The output is counted. The conversion rate is measured. Whether the saved time went to the worker, or to a deeper conversation with a donor, or to a colleague who needed help, does not appear anywhere in the report.

The dashboard is better off. The dashboard is not the work.

This is the water the sector already swims in

The sector spent thirty years building its incentive architecture around what the dashboard could see. Donor outreach velocity. Output counts. Overhead ratios. Grant compliance cadence. The work that did not fit a CRM field, the judgment, the listening, the slow trust, the accountability owed to communities, was invisible because the reporting frame could not hold it. The architecture rewarded what it could count.

AI is now arriving inside that architecture and being judged against the same metrics. Of course it is succeeding. It was built to.

This is not a failure of integration. It is success at the wrong thing.

I am not outside this. I built monitoring systems that rewarded throughput. I wrote indicator frameworks that counted contacts and never the depth of what passed between them. The infrastructure now being accelerated is infrastructure I helped lay. The acceleration is not happening to the sector from outside. It is running through the architecture practitioners like me put in place.

Efficiency is the wrong word if the time goes nowhere

Efficiency is the word the report uses. It would be an honest word only if the time saved went somewhere that mattered. To the client, in the form of being more fully heard. To the worker, in the form of going home earlier or being paid for the work they actually do. To the mission, in the form of work the dashboard cannot see.

The time isn’t going there. The thing that wins is the dashboard.

So the real question is not whether AI works. AI works. The question is why we keep optimizing for the dashboard.

Forthcoming posts in this series:

Who is Better Off When AI Speeds up the Workflow?

Measuring the Wrong Thing Faster

Grading Their Own Homework

Three Moves to Tell if Nonprofit AI Works (and for whom)

Leave a comment

Share

Anthralytic is a strategy and evaluation studio for mission-driven organizations. If you make decisions about resources in the social sector, whether or not you call yourself an evaluator, this newsletter is for you.

The International Rescue Committee, with staff on the ground, has named the pattern directly. Years of underinvestment compounded by recent funding cuts have hollowed out eastern DRC’s surveillance capacity, and the outbreak escalated faster because the system that would have detected it earlier had been thinned.

That is the news. What follows is the structural story underneath it, read against the two pieces in which the Secretary of State has, over the past year, introduced and defended the new American aid architecture.

The mechanism was announced in plain English.

On July 1, 2025, the day USAID formally ceased implementing foreign assistance, Secretary of State Marco Rubio published Making Foreign Aid Great Again on the State Department’s Substack. The piece rested on a familiar move. USAID, in Rubio’s telling, had drifted into charity divorced from American interest, and the closure would restore foreign assistance to its proper role as an instrument of American foreign policy. The strawman is worth setting aside for a moment, because the operational promises that came alongside it are what merit re-reading now.

Rubio committed the State Department to “consolidating fragmented appropriations accounts” to create more flexible pools of funds. He committed it to new efficiency criteria that would “measure impact quantitatively.” He committed it to building a “fast feedback loop” through empowered diplomats in regional bureaus, so that programs would track to American interests and partner needs in real time.

Those commitments were not loose. The consolidated pools of funds became the $19.2 billion closeout pool that an April 20 congressional notification later set aside to wind down USAID. Rubio described the new architecture in plain operational terms and promised it would work better than what had come before.

Nine months later, the promises had disappeared.

On April 6, 2026, Rubio published another piece on the same Substack. The Face of America Abroad is a defense of the Foreign Service as the inheritor of a long American diplomatic tradition. The piece moves through Silas Deane’s secret 1776 mission to France, Benjamin Franklin’s negotiations in Paris, and the Treaty of Paris. It celebrates the State Department’s recruitment refresh, the removal of DEI from entrance exams, and a renewed focus on diplomatic history and tradecraft for new officers. It closes by holding up Franklin as the model for the modern Foreign Service Officer.

The word aid does not appear in the piece. There is no accountability report on the new efficiency criteria, no example of the feedback loop in operation, no description of what the consolidated appropriations pools have produced. Nine months after the closure announcement, the deliverable that was promised has dropped out of the rhetoric entirely. What remains is a discourse about American power and the people who project it, with the thing being delivered moved off-stage.

The redirect is doing two jobs at once.

On May 7, CNN reported that the administration plans to redirect roughly $3.2 billion that Congress appropriated for global health and foreign development assistance to pay for the closure costs of USAID. The redirect is one slice of the $19.2 billion closeout pool.

That pool is doing two things at once. It is paying for the institutional dismantling of USAID. It is also paying organizations the US government already owes for work performed, supplies procured, and contracts terminated mid-stream. Both jobs are being done by the same money, through the same mechanism, with funds Congress appropriated for neither.

The $19.2 billion breaks down across three pools. The largest, more than $15 billion, is money the US had committed to multi-year grant agreements with partner governments but had not paid out when those agreements were cancelled. About $625 million is unspent 2024 appropriations. The remaining $3.2 billion is unspent 2025 money from accounts Congress designated for global health and economic development.

The legal piece sits inside that pool. A June 2025 internal estimate reported by Bloomberg Government put litigation costs at roughly $344 million, much of it tied to lawsuits from implementing partners and USAID staff who contend they were terminated illegally. Chemonics International, DAI Global, and others have filed suit alleging hundreds of millions in unpaid invoices, including, in Chemonics’ case, $240 million in committed medicine and health supplies stranded mid-supply-chain.

For a decade I worked for one of those implementing partners. The organizations now negotiating settlements and pursuing lawsuits are former colleagues. This analysis comes from inside that field, not from outside it.

The line that catches outbreaks is the line being closed.

Inside the 2025 portion of that pool, $647 million is in reductions to global health security specifically. That line pays for outbreak surveillance, preparedness, and the laboratory infrastructure that catches a strain like Bundibugyo before it has been circulating for months. The funding that would have detected this outbreak earlier is the funding now paying to close the agency that ran the surveillance.

A former USAID official quoted in the CNN reporting described the agency as the glue. The role was to coordinate health officials, NGOs, and donors so that when an outbreak appeared, the experts arriving on the scene had something to attach to. Without the glue, expertise arrives but cannot pay health workers, cannot move supplies, cannot stand up the operational backbone that a response actually runs on.

This is the part that does not show up on a funding diagram. Surveillance is not a line item; it is a set of relationships that take years to build and minutes to dissolve. The line items describe the activity. The relationships are what make the activity possible.

The feedback loop Rubio promised in July, in plain operational terms, is precisely this kind of infrastructure. It is the surveillance and field-monitoring architecture that would have caught a Bundibugyo strain in eastern Congo before it had been moving through communities undetected for months. The loop he announced is the loop that just got defunded. The only loop running now is between the closure and itself.

I have written budget narratives for surveillance indicators. I have written into log frames the assumption that the institutional glue would be there when the indicator needed to be measured. The line items will reappear in some other agency’s budget eventually. The relationships will not.

The mission money inverted.

Set the closeout ledger against what the same pool of money was appropriated to do. Analysts projecting the impact of the $2 billion health reduction have estimated 121,000 preventable TB deaths, 47,600 preventable malaria deaths, nutritional aid lost for 23 million children, and safe childbirth access for 5.7 million women. Add the Ebola line on top of that ledger: more than 170 deaths so far, with WHO warning that the actual scale of the outbreak is much larger than has been detected.

The detection systems that would have caught the outbreak are the detection systems being paid to be turned off. The mission money is paying for the mission’s shutdown.

Aid was the interest.

The Secretary’s framing for closing USAID rests on a strawman. He describes a charity model that lost sight of American interest. That is not what USAID was.

The agency was created in 1961 under the Foreign Assistance Act and designed from the start to integrate development into foreign policy. For most of its history, US food aid had to be procured from American farmers and shipped on American flag vessels. Development contracts carried Buy American provisions. PEPFAR, the Millennium Challenge Corporation, Feed the Future, and the Development Finance Corporation that the first Trump administration created were all explicitly built around American strategic and economic interest. The interest was the architecture, not an afterthought to it.

I have written grant narratives that named US strategic interest in the same paragraph as community outcomes. The two were never severable in USAID work. That was the design, not the corruption.

A Bundibugyo strain not caught early in eastern Congo can land in an American hospital. Disease surveillance in DRC is one of the most American-interest-aligned line items in the federal budget. The $647 million in global health security reductions is the line item that catches what arrives.

The April piece talks about American power as if it has only ever been projected through diplomats. That has never been true. American power has been projected through aid as a deliberate instrument of foreign policy for more than six decades, and the architecture is being erased from the rhetoric at roughly the same speed it is being closed in the budget. The interest was always there. The closure is what removes it.

The counterfactual does not exist.

The closure mechanics are being measured. The USAID Inspector General has begun a series of audits on asset disposition at terminated missions, with the first published in March 2026. House Foreign Affairs leadership has formally asked GAO to review what the cancellations cost and what capacity remains at State.

What is not being measured is the programmatic counterfactual. The Ebola response will be assessed against what is happening on the ground now. What the response would have looked like with the surveillance intact is unknowable, because the surveillance was the way we would have known.

You can audit the closure of a measurement system. You cannot evaluate the impact of its absence using the system that is gone.

Three pieces, one mechanism.

Two Substack posts, nine months apart. The first announced the architecture and promised accountability through measurement. The second moved to Franklin and stopped mentioning aid altogether. The Ebola outbreak landed between them.

Collapsed. Redirected. Inverted. Indirect-cost-as-structural-failure in its purest form: the agency’s closing pays itself, in the currency of the missions it was supposed to deliver.

Anthralytic is a strategy and evaluation studio for mission-driven organizations. If you make decisions about resources in the social sector, whether or not you call yourself an evaluator, this newsletter is for you.

Leave a comment

Share

The first was an evaluation opportunity. As we talked through what the client was looking for, the shape of what they wanted became clear without anyone saying it out loud: they had a decision to make and they wanted a summative evaluation that arrived at that recommendation. They needed a third-party expert to put it in writing, and the signature was the unspoken deliverable.

I declined. The request did not meet ethical evaluation standards, which require findings to follow the evidence rather than the other way around.

The second was a conversation with a former colleague who now works in public health. She was describing a company trying to win a contract, and to win it the company needs to demonstrate specific statistical benchmarks. They are massaging the data to get there. The benchmarks were set up to standardize quality across providers, but in practice, the colleague said, everyone games them, and the actual quality across providers varies wildly.

By her account, the company in question is actually one of the good ones, with a service that an external evaluation found to exceed expectations. They provider is gaming the benchmark not because their work is poor but because the benchmark cannot capture what their work actually does. But the poor quality providers also do the same thing, so rather than focusing on improving actual quality, they focus on success in gaming the system.

Two different sectors, two different stories, and the same underlying problem sitting underneath both.

The contract refusal was a privilege

The privilege is worth naming out loud, because pretending otherwise is dishonest.

Evaluators have industry standards and ethical codes, but declining work depends on being able to, which depends on other work, on savings, on a household income that does not require this one contract. The evaluator who needs this engagement to pay rent is not always less ethical, but they are, by definition, more constrained, and they are also a more useful case study in what the system actually rewards.

When we talk about evaluator integrity, we tend to treat it as a stance you take. Sometimes it is. More often, it is a luxury good purchased with margin, and writing about ethics without naming that costs nothing and changes nothing.

The fish do not make the water

The public health organization is doing what the system asks of them. Their question is existential, and the intervention they offer might be one of the ones that actually works. If they do not produce the numbers in the right shape, the contract goes elsewhere, the program ends, staff lose jobs, and the people they serve lose the service.

In that frame, massaging data starts to look like keeping an effective program alive. One that they know is working. The data is now load-bearing for survival, not just for learning.

The temptation to round up is not exotic, and it is not a character flaw. It is the rational response of a fish trying to swim in the only water available.

This is a structural problem before it is a character problem. The fish are not the issue. The water is.

The dynamic has a name. Goodhart’s Law says that when a measure becomes a target, it ceases to be a good measure. Campbell’s Law, the social-science cousin, says that the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures. It is the same law in different language, and the field has had fifty years to know it, but keeps building the systems anyway.

I wrote about this dynamic for the AI context last August, in a piece on how people break AI in social impact. That piece looked at how the pattern plays out inside a single model. What follows extends the frame to three sectors I have either worked in or watched closely: international development, education, and healthcare. The public health story sits inside a much larger pattern, and the pattern is the point.

Bad data follows the money in international development

I will start with the field I know best. Justin Sandefur and Amanda Glassman wrote a paper for the Center for Global Development in 2014 called The Political Economy of Bad Data. Their finding, across multiple African countries, was that official statistics systematically exaggerate development progress, and they traced the exaggeration to two specific mechanisms.

The first mechanism is that governments overreport to foreign donors. Their lead example is a results-based aid program that paid for reported vaccination rates, and the reported rates exceeded what independent household surveys could find. The money was tied to the number, and the number obliged.

The second mechanism is that governments are themselves misled by frontline providers. Their example is primary school enrollment, where official numbers diverged sharply from survey estimates after funding shifted from user fees to per-pupil government grants. The schools had been counting students before, more or less honestly, and then the grant changed what the count was for.

USAID’s own Inspector General reached similar conclusions from the inside. Across 21 performance audits in Egypt, Jordan, and the West Bank and Gaza between 2011 and 2013, 71 percent found unreliable data, including overstated indicators and missing documentation. In one case from Ukraine, the implementer overstated an indicator by roughly 100,000 and attributed the overstatement to a typographical error. In Guatemala, another implementer overstated leveraged funds by $3.4 million.

The pattern is not occasional misconduct; it is the predictable output of a funding structure that pays for indicator achievement. Logframes do not cause this. What does is the cost-reimbursement contract, the indicator-tethered disbursement, and the political pressure to show progress, all running at the same time.

Andrew Natsios, who ran USAID from 2001 to 2006, named the dynamic from the inside. He called it “obsessive measurement disorder.” The programs that get measured most precisely, he argued, are the least transformational, while the programs that actually transform anything are the hardest to measure. The aid system, he said, had organized itself around the wrong half of that observation.

Education made evidence into a badge

The Every Student Succeeds Act of 2015 set up a four-tier system for evaluating evidence behind educational programs, with Tier 1 reserved for strong evidence from a randomized trial, Tier 2 for moderate evidence from a quasi-experiment, and the lower tiers for weaker designs. Federal money for certain program categories is tied to using interventions with stronger evidence.

Vendors self-select the tier they claim, and the incentive structure that results is striking. Once a vendor has a single Tier 1 study showing a statistically significant positive effect, additional studies can only weaken the claim. The rational move, then, is to fund one tightly controlled study with extensive implementation support, get the positive result, badge the product, and never study it again. The market fills with Tier 1 claims, and the actual quality among Tier 1 products varies enormously.

The framework was supposed to standardize quality. In practice, it standardized the gaming.

This is the story my colleague was telling me, told in a different sector. Replace ESSA with whatever the equivalent benchmark is for a state public health contract, and the structure is the same. There is a number, that number determines whether the program gets the money, and the shape of the company’s behavior follows.

The older and sharper version of this same pattern is the Atlanta Public Schools cheating scandal, which ended with multiple educators convicted of racketeering for systematically changing student answers on standardized tests. The high-stakes use of a single metric for teacher accountability did not produce better teaching. It produced erased answers.

In healthcare, the metric moves and the reality does not

The Affordable Care Act created the Hospital Readmissions Reduction Program in 2010, under which hospitals with higher-than-expected 30-day readmission rates for selected conditions get penalized through reduced Medicare payments. The program was credited with substantial reductions in readmissions, and for years it was held up as a success of pay-for-performance in healthcare.

In 2019, Ody and colleagues published a paper in Health Affairs showing that the credit was overstated. A coincident change in electronic transaction standards allowed hospitals to document more diagnoses per claim, which meant higher risk scores, which in turn meant lower risk-adjusted readmission rates. The patients were the same, the care was largely the same, and what actually changed was the paperwork. Accounting for that change cuts the apparent decline in risk-adjusted readmissions for targeted conditions by 48 percent, and the authors conclude that either the HRRP had no effect on readmissions, or its effect was roughly half what we thought.

The Veterans Affairs wait-times scandal is the cleanest version of the same dynamic. The VA set a 14-day target for new patient appointments and tied executive bonuses to performance on that measure. Across at least 93 VA facilities, schedulers and managers manipulated the data: they started the clock late, kept secret paper waiting lists, and cancelled and rescheduled appointments to reset the count. One scheduler wrote in an internal email, “Yes, it is gaming the system a bit. But you have to know the rules of the game you are playing, and when we exceed the 14-day measure, the front office gets very upset.”

The Inspector General documented the manipulation in 2014. A follow-up audit in 2022 found the practice continuing under different mechanisms, with a 66-day wait being logged as 43 because of how the start date was defined. A decade on, FOIA records show the gaming persists.

Gaming is the system working as designed

Three sectors, different actors, different stakes, and the same pattern emerges in each. This is not a moral failure of evaluators, vendors, hospitals, or aid implementers. It is the predictable result of designing accountability around single quantitative targets attached to financial consequences, while leaving the underlying conditions of the work unexamined. The field has not lacked for the analysis. It has lacked for any structural willingness to use it.

My refusal of the predetermined evaluation matters. It protects my integrity, and it may have spared a specific program from a third-party signature on a verdict already written. It is the right thing to do. It does not change the structure that produced the request.

For every evaluator who declines, there is another evaluator who needs the work, and the field selects, over time, for the people who will deliver what is asked. That is not because evaluators are weak. It is because the field is funded that way. The same is true of the public health vendor. They can decline to game, but the next vendor will not, and the next vendor will get the contract. The system is not waiting on individual ethics to fix itself.

This is the part that gets uncomfortable for those of us who think of ourselves as on the right side of the question. Our refusal is necessary, and our refusal is not enough.

Some interventions actually end the gaming

If individual integrity is not enough, the next question is what is.

The research and the field experience point at a few real candidates, and the two with the strongest track records are worth developing in some detail.

The first is closure paired with quality alternatives. Charter schools are an imperfect and sometimes maligned case, and the case needs to be told honestly. In states with strong authorizer accountability, low-performing charter schools do close, and at higher rates than equivalent traditional public schools. CREDO’s 2017 Lights Off study tracked 1,522 low-performing schools that closed between 2006 and 2013 across 26 states, and it found that the charter sector closed a higher share of its low-performing schools than the traditional sector did. That is a real structural intervention, and one that is rare in the rest of the social sector, where ineffective programs tend to keep running because no one has both the authority and the willingness to end them.

The same CREDO study, however, found that less than half of displaced students landed in better schools. Closure without somewhere good to go is not accountability; it is dispersal. The intervention is not closure alone. It is closure paired with quality alternatives, and paired with rules that prevent failing programs from authorizer-shopping their way back into the market.

The second is independent verification that the producer cannot prepare for. Sandefur and Glassman flag one working example at the end of the bad-data paper: the World Bank’s Health Results Innovation Trust Fund, which uses small unannounced household surveys to cross-check the administrative data participating facilities report, and applies penalties for over-reporting. In Cameroon, this dropped over-reporting of outpatient consultations by more than 90 percent in less than a year. The verification is built into the funding mechanism, and the people producing the numbers cannot study to the test, because they do not know which numbers will be checked or when the check will arrive.

Neither of these is exotic. Neither requires new theory. They require funders willing to design accountability that cannot be gamed by the people whose careers depend on it.

The harder thing this week was not declining the evaluation. The harder thing is staying clear-eyed about which parts of this system I am still inside even when I decline.

I take contracts, I produce indicators, and I write reports that get read by people making decisions about budgets. I am one of the fish. Refusal is honest. Refusal is partial. Building anything different requires naming the ecosystem the game runs on, not just naming the players in it.

Leave a comment

Share

Anthralytic is a strategy and evaluation studio for mission-driven organizations. This newsletter is a practitioner's thinking-out-loud about evaluation, AI governance, and the systems social impact work runs through. If you know someone who makes decisions about resources in the social sector, whether or not they call themselves an evaluator, this newsletter is for them.

That admission is unusual in print from that quarter of the field. It is also worth taking seriously, because the struggle they name is real and it is not only about discourse.

What the struggle actually is

The field is being asked to evaluate a transition that has not yet stabilized into something evaluable. The objects of evaluation are changing under our feet. In a piece last week, Who’s Left at the Table, I wrote about the early data on where new US foreign assistance dollars have actually gone: to a small handful of multilateral pooled funds, and to bilateral government-to-government arrangements. That shift is not only a change in who receives money. It is a change in what kind of work gets done, by whom, and inside what institutional culture.

When the work changes, evaluating it changes too.

What is happening to the craft

The US implementing partner ecosystem developed a particular set of evaluation conventions over several decades, including utilization-focused evaluation, theory-of-change-driven design, third-party mid-term and final evaluations, MEL teams housed inside implementing organizations, and Collaborating, Learning, and Adapting frameworks for ongoing learning. These conventions did not emerge from nowhere. They emerged from a delivery architecture that needed them and was, in imperfect but real ways, built to use them.

When the delivery architecture changes, the conventions either travel or they do not. They are not changing because the field decided to change them. They are changing because the institutional homes that supported them are getting smaller, fewer, or differently configured.

The cultures receiving the work

The United Nations Office for the Coordination of Humanitarian Affairs runs pooled funds with humanitarian monitoring and evaluation conventions optimized for speed and output reporting. They are designed for crisis response. They are not designed for the kind of longitudinal outcome learning that the US implementing partner ecosystem built over decades.

The Global Fund operates on a vertical-program model with disease-specific indicator frameworks and country reporting systems. Its evaluation culture is real and substantial. It is not the same culture as the broad-spectrum, mixed-methods, utilization-focused tradition.

Bilateral government-to-government arrangements typically have minimal independent third-party evaluation by design. Accountability runs between two states. The affected community is not the audience for the evaluation, if there is one.

None of this is a failure of the new institutional receivers. They have their own histories and conventions, which work for the problems they were built to solve. Those conventions are not, however, interchangeable with the ones the US implementing partner ecosystem developed.

Why the words are getting harder

When CGD writes that they have been struggling to talk about evidence and evaluation, part of what is happening is that those two words now carry different meanings inside the new architectures than they did inside the old one. Value for money looks different through a humanitarian pooled fund than through a USAID cooperative agreement. The same words, with audience-specific meanings, are sliding past each other in the field’s current conversation.

The discourse problem is not that the field forgot how to talk. The referents are moving. The struggle to talk is downstream of that.

A practitioner note

I have built monitoring and evaluation systems inside cooperative agreement structures for years. I have written learning agendas that depended on the continuity of the implementing partner. That continuity is gone in many of the cases I worked in. The systems I built are not as portable into the new architectures as I once assumed they were. The struggle CGD names is not external to me. I am inside it.

Three questions that travel

Three orientations work in any architecture. They are not new. They are the ones the field has been asked to take seriously for a long time and has not always wanted to.

The first is motive: why we are evaluating. Evaluation for control, for justification, and for learning are different practices that produce different power dynamics. The new architectures are arriving with their motives already attached, and naming our own, as evaluators, is now a more visible act than it used to be.

The second is ownership: who owns the knowledge. The implementing partner, the funder, the multilateral pooled fund, the partner government, and the affected community are all candidates. The answer matters because ownership of findings shapes whether findings get used, and by whom.

The third is authorship: who counts as an evidence-maker. Affected communities themselves can be evidence-makers, not only the people the funder hires to study them. Indigenous and participatory traditions have answered this question well for decades. It is harder to ignore in an architecture where the old implementing partner intermediaries are gone.

These orientations are practical, not framework-shaped. They work in cooperative agreements, in pooled funds, in bilateral arrangements, in domestic philanthropy. They survive the architecture shift because they are not architecture-dependent.

CGD’s admission is worth taking seriously. The struggle to talk is real, and it is also structural. Naming what is moving makes the words usable again. The questions that travel are the ones we have been asked to take seriously for a long time. The field’s reorganization may be the moment to actually take them seriously.

Share

Anthralytic is a strategy and evaluation studio that helps mission-driven organizations clarify and amplify their impact.

In late February of this year, in a five-day stretch, the US government signed three new foreign assistance awards that together moved more than two billion dollars. Two went to the United Nations Office for the Coordination of Humanitarian Affairs. One went to the Global Fund to Fight AIDS, Tuberculosis and Malaria. After almost a year of very few sizable new awards, the money started moving again, and it moved to a small number of places.

In April I wrote about the orgs that are missing from the FY26 Food for Progress table and why (Who’s Missing From the FY26 Table). The Center for Global Development released an analysis last month that lets us see the other side of that picture: where the money actually went.

The data, plainly

Charles Kenny at CGD pulled new contracts and assistance awards from USAspending.gov for the main non-military foreign assistance accounts, covering January 20, 2025 through March 30, 2026. In the last year of the Biden administration, the same accounts issued 2,604 new awards totaling $8.881 billion. In the Trump administration’s first 14 months, the same accounts issued 345 new awards totaling $3.482 billion.

The top 10 recipients took 94.8% of those new dollars. UNOCHA alone took 57.5%. UNOCHA, the Global Fund, and the International Organization for Migration together took roughly 80%. If you want to look at the data yourself check out usaspending.org. That is the picture. Money is moving again, and it is moving to a small handful of multilateral institutions and to government-to-government bilateral arrangements.

The frame being applied

Kenny calls this shift “mostly a good thing in and of itself.” That language borrows, deliberately or not, from the localization argument that has been alive in development discourse for at least a decade. Localization advocates have argued for years that fewer, larger, more flexible agreements with actors closer to affected people would be an improvement on the US contractor-and-cooperative-agreement model. That position has integrity and a long literature behind it. I am not going to argue against it.

But the data does not show that movement.

The distinction that matters

A pooled fund managed by UNOCHA in New York and Geneva is not closer to affected people than a cooperative agreement with a US implementer that ran country offices staffed by nationals. A contribution to the Global Fund is not closer to affected people than a maternal and child health program working through district health offices. A bilateral arrangement between two governments routes funds through national systems, which is a real shift, but national governments are not the same as affected communities. Some bilateral arrangements will move decisions and resources closer to people. Others will reinforce the patterns those people have been organizing against for years. The architecture does not produce the outcome on its own.

Localization, as the term has been used, has meant shifting decision-making and resource control closer to the people the work is meant to reach. What the data shows is something different. It shows fewer institutional addresses, mostly in Geneva, Rome, and Atlanta, plus partner government capitals, holding much larger flows.

The geometry

The architecture now has fewer addresses, with larger flows per address. In many cases, the money is more distant, not less, from the people it is meant to reach. The accountability surface is smaller and harder for an affected community to access than it was under the old model, not larger and more open. The new architecture may turn out to be better than the one it is replacing on a range of dimensions, including lower transaction costs, fewer intermediaries taking a cut, more flexibility, and faster response. Those are real possibilities. They are not, on the evidence so far, what localization meant.

A practitioner note

I have been in rooms where the localization argument was made well, and rooms where it was made badly. I have helped write proposals that claimed local ownership while routing most of the real decisions and most of the money through US headquarters. The frame has always been available for use in either direction. The question now is which direction the current consolidation is actually moving in. The early data suggests upward, not outward.

Lingering questions

I don’t know whether the new bilateral country-to-government arrangements will move decision-making toward affected communities, or whether they will replicate the same patterns at a different level of government. I don’t know whether the multilateral pooled funds now receiving most of the money will invest in the partnership and learning infrastructure that affects accountability, or treat affected populations as recipients of disbursements. I don’t know where the new consolidation is functioning closer to true localization, and what makes those cases different from the rest.

What I do know is that the money has started moving, and it is moving to a small number of places. That picture deserves to be looked at on its own terms, before any frame gets placed over it.

Subscribe now

Leave a comment

Share

Anthralytic is a strategy and evaluation studio that helps mission-driven organizations clarify and amplify their impact.

Mike Lee’s recent piece, “A New Blueprint for Big Food,” names the pattern and offers a different wager. Stop fighting the profit motive, he writes. Redirect it. If the financial logic that runs Big Food can be re-aimed at soil health, nutrient density, flavor, and worker welfare, then the activist investor and the sustainability advocate end up pushing for the same outcome. It is a sharp piece, and the move is the right one. Lee takes financial logic seriously as the system to be redirected, refuses the moral lecture that has not worked for forty years, and stays honest about the gaps in his own case.

I want to build on his piece. There is a question sitting underneath it that he does not quite ask, and naming it changes how the wager reads. The question is what food is actually for.

Lee inherits an answer from the system he is trying to redirect: food is a consumer good, grown for sale and accounted for on balance sheets, and the argument is which costs that system should price. Any serious proposal for fixing the food system has to take that reality seriously, and Lee’s does. The question is not whether food operates inside markets. It does. The question is which of food’s functions the market sees and rewards, and which it leaves to whatever happens around the edges.

Food does many things at once. It sustains bodies, but never just bodies. It carries pleasure, but pleasure shaped by what you grew up tasting and where. It connects people: the eaters and the cooks and the growers and the carriers. It marks place: soil and climate, kitchen and table. It carries culture, the recipes and rituals and meanings that travel forward. It expresses sovereignty, the ability of a community to feed itself on terms it chooses. It participates in living systems, drawing from soil and water and weather and labor and feeding back into them. Each function is upstream and downstream of the others. The parts do not run in parallel, they run in relationship. That is what it means to call food mutually causal.

What food is for also varies by what food we are talking about. Wine is for pleasure, ritual, conviviality, the expression of place. Nobody buys wine for nutrient density. Vegetables sit at the other end: nutrient density and accessibility are the dominant functions, with pleasure and culture alongside. Lee’s flavor-revenue path borrows its strongest evidence from wine, where flavor and place have been linked for centuries, and stretches it across categories where the dominant functions are different. The flavor-soil-nutrition chain may hold for one category and break for another. What food is for is not a single answer. It is many answers, depending on what is on the plate, who is doing the eating, where, and under what circumstances.

Any blueprint reaches some of these functions and misses others. Lee’s reaches pleasure and partially reaches nutrition through the flavor-revenue path. It does not, by design, reach connection, place, culture, sovereignty, or living-system participation, except where those happen to be downstream effects of changes in soil and worker welfare. That is not a failure of his blueprint. It is its shape, fitted to the goal he is working from.

His goal, which he does not quite name, is to make Big Food less harmful without replacing it: to keep the consolidated industry and the market system that runs it roughly intact while the financial logic starts rewarding different things. That goal is real and defensible. It is not the only one. Other goals would produce different blueprints: dismantling Big Food, moving food out of market logic, centering food sovereignty, building regional systems on commons or cooperative principles.

Two paths, two theory levels

Lee proposes two paths. The first is what I will call the flavor-revenue path: better soil produces more flavorful and more nutrient-dense food, and eaters reliably pay for flavor, so regenerative agriculture finds its way to revenue through people’s mouths and wallets. The second is the externality-pricing path: get the true cost of conventional food (soil loss, polluted water, healthcare costs, exploited labor) into accounting standards and supplier contracts, and the cheap commodity ingredient stops being cheap.

He bundles them as complementary tools. Pulling them apart and asking whether the parts hold together takes a couple of distinctions worth borrowing from evaluation work.

A theory of change is an explanation of how change happens in a given context. It traces the causal mechanisms that connect where we are to where we want to be, and surfaces the assumptions that have to hold for the connection to work. The term comes out of evaluation theory, particularly Carol Weiss’s writing on theory-driven evaluation. A theory of change is mostly descriptive: this is how the world rearranges itself when the right conditions are in place.

A theory of action is a plan for what specific actors will do to make change happen. It identifies the activities, the actors, and the resources needed to produce an outcome the system would not produce on its own. A theory of action is mostly prescriptive: this is what we will do.

The evaluative question is whether the two align. Does the proposed action actually trigger the change the theory of change describes? Where the action is too narrow for the mechanism the change theory implies, there is an alignment problem worth surfacing.

This gets sharper with a second distinction, from Chris Argyris and Donald Schön. They observed that people and organizations carry two theories of action at once. The espoused theory is the one we offer when asked: the world view and values we believe our behavior is based on. The theory-in-use is the one our actions reveal: the world view and assumptions implied by what we actually do. The two are often misaligned. The most important fact about the gap is that people are usually unaware of it. Argyris’s claim is that effectiveness comes from developing congruence between the two, which means surfacing the gap is constructive work, not gotcha work.

Their example is concrete. A management consultant said he would handle a disagreement with a client by first stating his understanding of the disagreement and then negotiating what data they could agree would resolve it. That was his espoused theory: joint control of the problem. A tape recording of the same consultant in such a situation showed him advocating his own view and dismissing the client’s. That was his theory-in-use: unilateral control. The consultant did not know the gap was there.

A public proposal carries both layers too. There is an espoused theory of change and an espoused theory of action, the version Lee writes for public reading. There is also a theory-in-use embedded in the design, made of the assumptions the blueprint quietly depends on. Whether they align is the evaluation question.

For Lee, the espoused theories sit cleanly with each other on the surface. The espoused theory of change says markets respond to better measurement, regenerative becomes profitable, and Big Food reorganizes around soil, flavor, and worker welfare. The espoused theory of action follows: fund the science, build the measurement tools, change the accounting standards, and wait for the reorganization to follow. Read on the surface, the action plan addresses the conditions the change theory says are needed.

The theory-in-use is harder to see and more interesting. The blueprint as designed depends on a number of assumptions the espoused theory does not name. Markets reliably translate measured signals into desired outcomes, which they often do not, because indicators get captured by the producers being measured. Big Food’s consolidated industry structure is compatible with regenerative supply chains, which it often is not, because consolidation tends to compress regenerative farmers when they get pulled into industrial supply. Measurement infrastructure is a neutral tool rather than a lever controlled by particular actors. Consumer behavior reliably tracks measurable signals at scale, despite the well-documented class stratification of palate. Accounting standards respond to a clear intellectual case, despite the Rockefeller Foundation’s true-cost report sitting in circulation for years without translating into FASB or IFRS movement. The financial logic is the dominant or sufficient lever for change.

None of those assumptions is named in Lee’s piece. All of them are doing structural work in the design. This is not a charge of bad faith. It is the standard pattern Argyris and Schön describe: the blueprint’s author probably could not see the in-use theory if asked, because that is what theories-in-use do.

The forty-year pattern Lee opens his piece with is a useful test case for the misalignment. Annie’s, Stonyfield, Honest Tea: each tried to operate inside an in-use theory that scale and mission would coexist. Scale rewarded itself, and mission got compressed. The pattern is not a fluke. It is what happens when an espoused theory and a theory-in-use are out of alignment, and the actors involved are unaware of the gap.

The misalignment in Lee’s blueprint is between what the proposal says about how change will happen and what the proposal actually depends on. The espoused theory pulls toward an ambitious reorganization of the food system. The theory-in-use defaults to the conditions of the existing food system, with measurement infrastructure as the only adjustable variable. When the in-use assumptions fail, and indicator capture is one well-documented way they fail, the espoused outcome does not arrive. What arrives instead is what the theory-in-use is actually fitted to: a less destructive Big Food that has absorbed regenerative aesthetics without changing structure.

That is the alignment finding. The hidden goal I named earlier is the goal embedded in the theory-in-use. The work the evaluation does is to surface the gap, so that the choice between the espoused theory and the in-use theory becomes a choice we make rather than a default we inherit.

The empirical assumption underneath

The flavor-revenue path also rests on a science question that deserves to be named, even if I cannot settle it here.

Lee’s path depends on a chain of empirical claims, each of which needs its own evidence base. Healthier soil produces more nutrient-dense food. Nutrient density correlates with flavor. Eaters reliably perceive the difference at the bite. Eaters will pay for that perception at scale.

Here is what I think we know, gesturally. The wine and terroir literature is reasonably solid on the relationship between soil, climate, plant stress, and flavor compounds. Plant physiology supports the secondary metabolite story: plants under mild stress do produce more of the polyphenols and aromatics that make food taste like something. The dilution effect in produce, where yields have gone up while nutrient density per unit weight has gone down, is real but contested in its interpretation. The connection between soil health specifically and human nutrition is an emerging field with real research and a long way to go before it can carry procurement or policy weight.

The point is not to settle the science here. Lee himself calls for the kind of research that would test these assumptions at the statistical confidence we apply to drugs, and he is right to. Before measurement infrastructure gets built on top of a partial chain, the chain itself has to be tested. The pieces that are well established need to be defended. The pieces that are partial need controlled studies. The pieces that are gestural need to be flagged as such, not built into the architecture as if they were settled. We are agreeing with Lee on what needs to happen next, and we are agreeing that the work has to happen before the wager scales.

When you optimize for the wrong thing

This is where the practitioner reflex kicks in.

The minute regenerative becomes “a measurable input that can be priced,” the indicator captures the work. I have watched this happen across enough fields, including ones I work in, that it now reads as a near-certainty. Carrots get bred for the spectrometer. Breeding programs, fertilization timing, harvest conditions, and post-harvest handling all get tuned to optimize the signal that drives the price premium. The reading goes up. Whether the carrots are actually more nourishing becomes a separate question that nobody is checking, because the reading has become the proxy.

This is the standard fate of an indicator that gets too important too fast: the number drifts from the thing the number was supposed to track, the spreadsheet still says we are on track, and the underlying thing is somewhere else.

What gets captured by the indicator is whatever goal the indicator was designed to serve, named or unnamed. If the goal is unnamed, the capture is invisible. We optimize for it without knowing we are optimizing for anything in particular. That is the through-line of every measurement system that has been built on top of an unstated objective. It is also why the question I named at the top is not optional. If the goal has not been named, the indicator will name it for us, and the name will be the one with the best signal-to-noise ratio, not the one we would have chosen.

A pre-mortem, in one scenario

A pre-mortem is a planning method from the psychologist Gary Klein. You imagine that the project has failed, then work backward and ask why. The point is to surface the assumptions a plan is hiding by treating failure as the starting condition rather than the worst case. It is one of the few tools I know for finding the hidden goal of a hidden theory of change.

Here is one pre-mortem for Lee’s blueprint, deliberately limited to a single scenario.

It is 2041. The wager worked. Healthier soil reliably produces more flavorful and more nutrient-dense food. The handheld spectrometer is in every grocery store, and true cost accounting moved into financial disclosures a decade ago. The CFO’s spreadsheet rewards regenerative supply chains, and the activist investor pushes for more rotation acres. By every metric in Lee’s piece, this is a win.

Here is what failed. Carrots are bred for the spectrometer. The reading went up year over year, but whether the carrots are actually more nourishing has become a separate question that nobody is checking, because the reading is the proxy. Breeding programs, fertilization timing, harvest conditions, and post-harvest handling all got tuned to optimize the signal. The signal is high. The underlying biology drifted somewhere else.

That is one scenario. Others are easy to imagine. There is the small regenerative farmer who cannot operationalize the new standards as fast as Big Food and gets bought out by the buyer who can. There is the premium-priced flavor that sorts by income and deepens the class divide in what people eat. There is the relationship between an eater and a farmer mediated by a device, where trust now lives in a number rather than in a name. There is the question of governance, where the new accounting standards get written by the same class of people who wrote the old ones, leaving the question of who actually decides about food exactly where it was. Each is its own pre-mortem worth running.

Each is also a different answer to the question of what food is for. The small farmer scenario asks about food as livelihood. The class divide scenario asks about food as access and dignity. The device-mediated relationship asks about food as trust between people. The governance scenario asks about food as something we decide together. None of these questions is forbidden by Lee’s blueprint. None of them is asked by it either.

What the question opens

Holding the objective question open is uncomfortable. It does not deliver a slogan or produce a campaign. What it does is change our relationship with proposals like Lee’s. You can take the wager seriously without absorbing the goal it is fitted to. You can fund the science the flavor path depends on without committing to flavor as the thing food is most for. You can push externality-pricing into accounting standards without conceding that accounting standards are the right place for the question of what we owe to soil and to each other.

The work of evaluation is to make hidden things visible: the goals, assumptions, and trade-offs that shape decisions before anyone realizes a decision is being made. When those are surfaced, the choices that come next are choices people can be held accountable for, instead of choices that get made for us by an objective nobody named.

Lee’s piece is worth taking seriously, and so is the question underneath it. The wager he describes is one good answer to one of the questions food asks of us. The other questions are still here, on the table, waiting for someone to ask them out loud.

Leave a comment

Share

Subscribe now

Anthralytic is a strategy and evaluation studio for mission-driven organizations, working at the intersection of measurement, impact, and the systems that shape how decisions get made.

Other Related Posts

It took me a decade to realize that the systems I had been working with abroad could solve a problem in my own yard.

What I Built When I Got Home

Years after that visit, after a decade of supporting country directors whose programs ran in places where the grid was unreliable and the rainfall sparse, I built a system of my own. I’d rather have bought it but nothing existed on the market that could do what I needed: supply rain water from my barrel with solar power directly to the plants with individual emitters. It was a simple drip system with lines running through the garden beds, the kind of setup a smallholder farmer near the Nile would recognize immediately, because the principles were the same.

I built it because I travel for work and my garden was dying every time I left. I knew what I was building, having thought about systems like it for a decade. What I did not expect was that other people would want one.

But they did. Friends came over for garden parties and stopped to ask about the drip lines, wanting to know how the system worked and whether I could set them up too. Some had never thought about where their irrigation water came from. Others had been trying to grow tomatoes and giving up. All of them wanted what was in my side yard.

That was when I understood that the international agricultural development I had spent my career inside was not as far from home as I had treated it.

Technology Transfer from the Global South

International agricultural development has been working on circular systems for as long as I have been in the field, and longer. It was not a brand but a necessity, because there was no other option.

The water you have is the water that fell. The inputs you have are the inputs you can carry in. The energy you have is the energy you collect. The equipment you have is the equipment you can repair. Smallholder agriculture, in the places I worked, was circular by necessity, and the systems that survived were the ones that closed their own loops.

I worked on circular economy directly during those years, by other names. The portfolio included a public-private partnership on recycling in Egypt, a social cost-benefit analysis of a solid waste management social venture, and research for a family foundation on Tetra Pak. The questions about flexible packaging, recovery, and what to do with what we had been calling waste were not new; they were just elsewhere.

We rarely called any of this what it actually was. Sometimes we called it resilience, sometimes climate-smart agriculture, sometimes appropriate technology. The frame that fit best was circular economy at human scale.

Agricultural technology transfer was supposed to be a one-way pipe, from research institutions and donor countries toward smallholder farmers. It has been flowing both directions for a long time. What I built in my Minneapolis backyard is just the latest piece of evidence.

The Upper Midwest Is Starting to Ask the Same Questions

The questions I worked on abroad are now the questions being asked here.

Climate is changing how water moves through this region, and soil health is back on the agenda after a generation of treating it as a fixed asset. Flexible film waste is becoming visible because it has nowhere else to go. Coalitions are forming around circular economy in food and agriculture, and capital is moving toward work that would not have been funded ten years ago.

Some of what is now being called innovation in domestic agriculture is what international agriculture has been doing for decades, at smaller scale, with less capital, and often with more ingenuity. I say this with affection for both fields, having worked in one for a decade and writing now from inside the other.

What Gets Counted, Gets Funded

The work that is keeping people fed in this region right now is mostly invisible to the people deciding where capital goes.

I am thinking about what grew here during Operation Metro Surge. Mutual aid networks expanded fast, community fridges multiplied, and neighbors shared canned food because it was the middle of winter. Now that it’s spring, neighbors are sharing seeds and seedlings, garden tools and unused rain barrels. Soon we will the fruits — the eggplants, tomatoes, herbs, and zuccinni. Inflation has pushed more people into household food production this year than at any point I can remember, and Operation Metro Surge made what was already happening visible to people who had not been paying attention.

None of this shows up in the data that funders use. There is no indicator for households closing their own water loop, and no line item for the neighbors who shared seedlings or the rain barrel that fed three gardens on the same block. The measurement infrastructure that exists in agriculture was built for commodity markets, supply chains, and donor reporting, not for what is happening on a residential block in Saint Paul or a community garden in North Minneapolis.

This is an evaluation problem with real stakes: what is not measured is not funded, and what is not funded does not scale. The household and mutual-aid scale of circular agriculture is doing significant work right now, in this region, under conditions that make it more necessary every month, and it deserves to be seen.

The next decade of agricultural evaluation has to take this seriously, not to replace what corporate sustainability reporting does but to complete it. The full picture of circular agriculture in the Upper Midwest includes the largest companies in the world and the gardener with a rain barrel, and right now the data only sees one of them.

What I Am Building

My son and I are building Greenway Drip, the simple drip system that started in my backyard and is now being made buildable for other people.

It is a solar powered, rain-barrel fed, drip irrigation system: the kind of system a smallholder farmer near the Nile would recognize, and the kind of thing the people at my garden parties kept asking how to copy. It is small, and that is the point. Circular agriculture has to be able to operate at every scale, including the one where a parent and a kid in a backyard want to grow food without leaving the grid running for three weeks.

It is also a measurement opportunity. A household that captures its own water, runs on its own solar, and grows its own food is a circular system that funders and policymakers cannot currently see, and if we build the tools to see it, the case for supporting it gets easier to make.

The future of circular agriculture in this region will depend on whether it can see itself at every scale. The largest companies in the world are working on it, and the gardener two blocks over is working on it too. The coalitions, capital, and measurement systems that get built in the next few years will decide whether both kinds of work get counted.

I think both kinds should.

For more information on our solar-powered, rain barrel drip irrigation system go to GreenwayDrip.Com and read our Substack, the Drip Drop.

Anthralytic is a strategy and evaluation studio working at the intersection of data, AI, and mission-driven impact. Subscribe for writing on evaluation, agriculture, AI governance, and what it takes to build systems that actually serve the people they claim to.

Share

Leave a comment

Related Posts