Anthralytic’s Substack
Sign in

Testing How Humans and AI Check Qualitative Interpretations

A companion to the six-part series on AI and qualitative interpretation.

Anthralytic's avatar
Anthralytic
Nov 19

Thanks for reading Anthralytic’s Substack! Subscribe for free to receive new posts and support my work.

AI is getting a great deal of attention at the American Evaluation Association conference, where I am this week. For evaluators using it to interpret interviews and documents, that attention brings a question about evidence. When a model helps us develop themes, suggest explanations, or challenge an interpretation, how do we assess whether the analysis improves?

That question runs through my six-part series on AI and qualitative interpretation. Researchers and models can catch things the other overlooked. They can also reinforce each other’s assumptions. The final piece proposed a study comparing different ways of working together, with independent reviewers assessing whether the resulting interpretations were better supported by the source material. Read the final piece

I built a small pilot to try one part of that design. To compare review arrangements, we need a way to distinguish someone who catches an unsupported claim from someone who objects to everything. We also need to understand what happens when a reviewer recognizes uncertainty but leaves an overstatement unchallenged.

What I tested

The exercise pairs twelve fictional interview excerpts with twenty-four summary claims. Some claims accurately reflect the source. Others go beyond what it supports. Reviewers flag the claims they would challenge and explain their reasoning.

The scorer records correct flags and false alarms, along with explicit uncertain responses. It checks which claims the reviewer identified. Assessing the explanation and any proposed revision requires a further reading of the source.

ChatGPT helped me develop the cases and found all the planted errors when I asked it to review them in the same conversation. I didn’t want ChatGPT grading its own homework, so I gave Claude the packet in a fresh session without the answer key or development discussion. Claude also found all the planted errors and left the supported claims unflagged.

The repository preserves Claude’s individual flags and reasons. The ChatGPT result survives as a reported total. I reconstructed the scoring key afterward with the results already in view, so the public replay allows readers to inspect those judgments and reproduce the scoring. A larger study would need independent reference judgments established before examining the responses. Pilot materials and provenance

What my own answers revealed

I also completed a separate version using different fictional cases. Some of my responses exposed a distinction I had blurred. Being unsure what happened in a program and recognizing that a particular claim exceeds the available evidence are different judgments.

An outcome can remain uncertain while the evidence gives us a clear reason to challenge an interpretation. We may need more information to understand how several conditions interacted, for instance, while already knowing that the source cannot support attributing the outcome to one of them.

Keeping uncertain responses visible helped me examine where my reading stopped. In some cases I had noticed the limits of the evidence without treating those limits as grounds to flag the claim. That creates a specific question for the larger study. Would a model’s challenge help a researcher recognize the overstatement and revise the interpretation?

How this connects to the larger questions

The pilot gives reviewers claims to examine and excerpts against which to examine them. The series also asks about experiences and explanations that disappear before anyone writes a claim. Studying those omissions would require returning to the wider interview material and examining the choices that shaped the analysis.

The question of who judges also remains. In qualitative work, experienced readers can reach different interpretations for defensible reasons. A study would need to preserve those disagreements and the evidence behind them. That would help distinguish a consequential error from an interpretive difference worth exploring.

The study proposed in the finale would follow the exchange between researchers and models. It would preserve initial interpretations, record challenges and responses, and have independent readers assess the resulting revisions. That would let us examine whether working together helped reviewers make supported corrections, where they introduced new problems, and what both continued to overlook.

Try the exercise

If you would like to try it, start with the reviewer packet before looking at the key or saved responses. Record which claims you would challenge and why. Where you hesitate, describe what you think the excerpt establishes and what remains open. Open the reviewer packet

I’d be interested in a claim you found difficult to judge, a disagreement with the key, or a revision you think better reflects the source. If you’re at the conference this week, I’d be glad to talk through it. Those responses would help me examine whether the difficulty lies in the reading, the wording of a claim, or the judgment built into the exercise.

Share

Leave a comment

User's avatar
Join Anthralytic’s subscriber chat
Available in the Substack app and on web
a blue object with a tick on it
Photo by Farhat Altaf on Unsplash

Discussion about this post

User's avatar

Keep reading

Anthralytic’s Substack

Anthralytic Substack

  • About
  • Archive
  • Recommendations
  • Sitemap
© 2026 Anthralytic · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for writing, podcasting, and media founders.