I Broke the App I Built With AI to Teach How AI Breaks
There is a new card in the How AI Breaks in Social Impact app I built. I could have added it silently and gone about my business. Maybe I should have. It took me a while to get to it, but the story underneath the card is the story the whole app is about. The structural pattern I named to teach other people is the pattern I was sitting inside of when I built the thing that names it.
The pattern is worth saying out loud, because it is not really about me.
What I built and what happened
The app lays out a set of principles for how AI initiatives in social impact organizations fail, alongside the breaking patterns that show up when those principles are skipped. It is a small educational tool. Each card has a “Suggest Actions” button that calls an AI provider’s API to generate concrete steps an organization could take.
To make those buttons work, the page needed an API key. The right way to handle a key is to keep it on a server, behind a function the browser can talk to, so the key never goes anywhere near the user’s computer. I put the key directly into the HTML, where anyone who opened the page and looked at the source could read it. I’m still learning all of these things.
Eight months later, two notifications arrived in close succession. The first was a billing alert for about thirty dollars in unexpected API charges. The second was a suspension email citing “abusive activity consistent with hijacked resources” and language about the key being harvested from a public source.
Bots crawl the open web for credential strings constantly. Mine was one of millions they found that week.
The cleanup
I revoked the key. I rewrote the demo to call the API from a server-side function. I audited every other project I had built for the same mistake and found that I had reused the same key in several places, which meant the single leak compromised everything it touched. I scrubbed git histories. I treated each small project as a small disaster site, which is what each of them was.
The bill stopped at about thirty dollars. The reason it stopped there is honestly not that I caught it. The provider’s automated abuse detection caught it. The project was attached to a small billing footprint, which kept the blast radius low. Both of those facts belong to the vendor’s infrastructure, not to mine.
That is the part of the story I want to sit with.
The pattern, named by the app it broke
The app I built names four principles that were violated to build it. Build Internal Literacy. Plan for Adversaries. Govern and Minimize Data. Monitor and Recalibrate. Each one fits the story exactly. I do not raise this to be clever about it. I raise it because the pattern is structural rather than personal.
Most of the AI tools small mission-driven organizations are experimenting with right now are built the way mine was. They are built quickly, often by people who are not specialists, in environments where “I will get to it later” is the default operating posture, because the alternative is not building anything at all.
The literacy gap that allowed me to embed an API key in a deployed page is the same gap that allows a small nonprofit to leak donor data through a misconfigured spreadsheet, run a model on biased inputs without noticing, or ship an automation that quietly mishandles edge cases for months before anyone notices. The technical specifics differ. The structural mismatch is identical.
The mismatch is between the speed at which an idea can be tested and the speed at which a tested idea becomes a system that handles real stakes. Demo-grade decisions get made on demo-grade timelines, and then the demo gets deployed and forgotten, and “demo” turns into “production” by accident. There is no moment where someone declares the change of state. The change of state is what the absence of that moment looks like.
What protected me was not me
This is the part that should make any small organization building with AI uncomfortable.
The infrastructure that kept this failure cheap was not infrastructure I designed. It belonged to the vendor. Their abuse detection ran on their schedule, not on mine. The thresholds that triggered the suspension were thresholds they set without asking me. The rate of spend that pushed the system into protective mode was small only because my account happened to be small.
For an organization running on a larger billing footprint, or with a vendor whose detection runs slower, or with a key attached to a higher-traffic project, the same mistake is a much bigger story. The factors that made my version of this story end at thirty dollars are not transferable. They are circumstantial.
Most of what protects small social impact organizations from the worst versions of their AI experiments right now is circumstantial. What stands between a misconfigured demo and a serious incident is mostly the goodwill of automated systems we did not build and do not control.
We do not have the literacy to design the protection ourselves. We do not have the leverage to demand it from vendors. We need to build them both. Myself included.
Anthralytic helps mission-driven organizations clarify and amplify their impact. We work at the intersection of evaluation, AI governance, and the systems that shape both.
