Anthralytic’s Substack
Sign in

What Good AI Use Looks Like: Cases in Critical Thinking

Anthralytic's avatar
Anthralytic
Oct 26

A couple of days ago, a friend asked me to help with a job application because of how much time I’ve spent working with (and talking about) AI. The application asked candidates to use the LLM of their choice to work through a research paper and then submit the conversation in the application.

This was the first time I’ve seen this task in a job application. It’s exactly how employers should be bringing AI into hiring.

In I Used AI to Help Write This Piece. Judge the Output., I argued that the question isn’t whether someone used AI, but how, and that the output usually tells you. This employer went a step further and asked to see the how directly.

Thanks for reading Anthralytic’s Substack! Subscribe for free to receive new posts and support my work.

The conversation shows how someone asks questions, where they push back, what they do when an explanation doesn’t quite fit the source, and whether they end up with an understanding they could defend on their own. That’s critical thinking, made visible. The AI is in the room, and the employer still gets to see the person.

AI isn’t going away and more and more employers are expecting their knowledge workers to use it. The task is a great way to see how effective a candidate is with their use. One addition would make it even better: a timeframe. How long are candidates expected to spend on completing this application? Thirty pages in 30 minutes makes it clear that nobody is expected to read the whole paper, and that AI is part of the assignment. Then the conversation shows how candidates decide where to read closely and where to rely on the tool, which is critical thinking too.

A hidden trap skips the thinking entirely

A job posting that I applied to a while back took the opposite approach—an AI honeypot. Hidden in white text was an instruction telling an LLM to begin the cover letter a particular way. Anyone who pasted the posting into a model and sent back what came out would be exposed.

Claude caught it. Which means the trap mostly measures which model a candidate used and whether that model fell for the injection. It says very little about the applicant’s judgment, and nothing about how they think.

The first employer invited AI into the task and asked to see how candidates used it. This one tried to catch them using it.

Judgment can be taught

The same technique shows up in a very different setting. Julia Díez’s account of teaching AI literacy to teenagers includes a demonstration of an assistant following malicious instructions hidden in a recipe. Her students watched the failure happen, then talked through why it mattered and how to respond.

Other exercises built the same muscle. Students predicted missing words in sentences, which was easy for familiar phrases. Asked for the capital of Atlantis, they produced confident answers to a question with no factual answer. That gave them a direct feel for how a system can generate something plausible without establishing whether it’s true. They also examined scams, explained their judgments, and worked out how to verify what they saw through another channel.

Díez used hidden instructions to build judgment. The hiring screen used them to catch its absence. Her classroom produces exactly the kind of thinker the first employer was looking for. Some of these students are the knowledge workers of the future. They will need to use AI. Díez’s approach appreciates that.

A ban doesn’t teach anyone to think

The honeypot I described above is an elaborate version of something far more common: the “no AI” rule. Both treat AI use as the thing to detect, rather than understanding as the thing to assess. I’ve written before about how these rules show up in schools and job applications, and how little they reveal.

The concern behind them is real: nobody wants work handed in by someone who never engaged with it. But asking AI to explain a concept, generate practice problems, or challenge an interpretation is a different activity from having it complete the assignment. A ban treats them all the same.

Prohibition can tell you whether someone used a tool. It can’t tell you whether they can think with one.

Ask to see the thinking

The examples that work have something in common. In each, the person has something to investigate, explain, and take responsibility for. AI supports the work, and the task is built so the reasoning leaves evidence.

That’s the design principle schools and workplaces need: decide what capability you want people to develop, then build tasks where that capability shows. Sometimes that means unaided practice. Sometimes the AI conversation is the best evidence you’ll get.

The first employer had the right instinct. Ask to see the conversation, and tell people what you expect. Critical thinking is the skill that makes AI useful, so it’s the skill worth assessing.


Anthralytic is a strategy and evaluation studio for mission-driven organizations. If you make decisions about resources in the social sector, whether or not you call yourself an evaluator, this newsletter is for you.

iridescent brain render on blue purple background
Photo by Milad Fakurian on Unsplash

Discussion about this post

User's avatar

Keep reading

Anthralytic’s Substack

Anthralytic Substack

  • About
  • Archive
  • Recommendations
  • Sitemap
© 2026 Anthralytic · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for writing, podcasting, and media founders.